Weirdly Long DNS Switchover?

https://www.lesswrong.com/posts/utPTFBSKRzAFt8PQF/weirdly-long-dns-switchover

Link post

[EDIT: as Jai points out in the FB comments, the problem was likely forgetting IPv6. The AAAA record is now updated, and I should be able to tell in a few hours whether that fixed it.]

I’m helping my dad migrate some code to a new server. It was at

162.209.99.139 for years, but ten days ago he changed his DNS settings to point to

34.199.143.13. Everything looks good to me:

$ dig notes.billingadvantage.com notes.billingadvantage.com. 1800 IN A 34.199.143.13

The TTL is 1800s, or 30min, which agrees with

dns.google. I expected everyone would be moved over within a couple hours, but a week later the old server is still receiving traffic nearly as much as the new:

date old server new server

2021-02-05 943 0

2021-02-06 201 127

2021-02-07 17 108

2021-02-08 364 423

2021-02-09 488 448

2021-02-10 255 503

2021-02-11 281 345

2021-02-12 250 248

2021-02-13 0 88

2021-02-14 0 78

2021-02-15 217 262

2021-02-16 202 287

The old server getting no traffic on the 13th and 14th is probably because that’s the weekend, and the users who happen to be still stuck on the old site aren’t using it on the weekend. I asked one of the users still getting the old server to try rebooting, to no effect.

I thought maybe something was misconfigured with the name servers, but it looks fine:

$ whois billingadvantage.com
| grep ‘Name Server’ Name Server: NS1.ZEROLAG.COM Name Server: NS2.ZEROLAG.COM

$ dig notes.billingadvantage.com
@ns1.zerolag.com notes.billingadvantage.com. 1800 IN A 34.199.143.13

$ dig notes.billingadvantage.com
@ns2.zerolag.com notes.billingadvantage.com. 1800 IN A 34.199.143.13

I’m not seeing any references to the old IP address anywhere.

Any guesses about why the traffic isn’t moving over?

Comment

https://www.lesswrong.com/posts/utPTFBSKRzAFt8PQF/weirdly-long-dns-switchover?commentId=KYBLXiwcEfq4BnnLS

Your A records are fine, but you seem to have changed name servers. Your old NS records are probably cached all over the place; the TTL on those seems to be 48 hours. It looks like the old server (at my-tss.com) is serving the correct data now, but it was probably still serving stale data when you saw the problem. Possibly it took it a while to realize that it wasn’t authoritative for the zone, or possibly there was an update problem.

Generally, it’s better to do the server change and the data change separately. And you have to make sure that the new and old servers are serving the same thing through the full TTL of the old NS records, or at least have the old server definitively reconfigured not to see itself as authoritative so that it can avoid misleading other systems when it gets a query.

Evidence: If I let my system use my ISP’s servers and do dig -t ns bilingadvantage.com., I get the wrong cached data:

; <<>> DiG 9.11.27-RedHat-9.11.27-1.fc33 <<>> -t ns billingadvantage.com ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 38501 ;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 1232 ; COOKIE: b250042072aca8b7ccde8a21602d31062c20fb70fa6f7d57 (good) ;; QUESTION SECTION: ;billingadvantage.com. IN NS

;; ANSWER SECTION: billingadvantage.com. 3600 IN NS dns2.my-tss.com. billingadvantage.com. 3600 IN NS dns1.my-tss.com.

;; Query time: 168 msec ;; SERVER: 127.0.0.1#53(127.0.0.1) ;; WHEN: Wed Feb 17 10:06:46 EST 2021 ;; MSG SIZE rcvd: 122

… but if I go and query the actual GTLD servers, say with dig -t ns @a.gtld-servers.net billingadvantage.com, I get the right data:

; <<>> DiG 9.11.27-RedHat-9.11.27-1.fc33 <<>> -t ns @a.gtld-servers.net billingadvantage.com ; (2 servers found) ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 27939 ;; flags: qr rd; QUERY: 1, ANSWER: 0, AUTHORITY: 2, ADDITIONAL: 3 ;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 4096 ;; QUESTION SECTION: ;billingadvantage.com. IN NS

;; AUTHORITY SECTION: billingadvantage.com. 172800 IN NS ns1.zerolag.com. billingadvantage.com. 172800 IN NS ns2.zerolag.com.