On The Importance of Reverse DNS

I was creating a cache file for webalizer and analog and came to the obvious optimization to select only hosts which received a "200" status code in response to a request (I suppose I could add 304s as well but I'm not sure that would add any value), and then strip out any hosts which made less than 10 requests in a 30 day period. This reduced the IP address list from about 3500 entries across the sites I maintain, to 470.

See updates below...

There appear to be about 140 unique organizations that hit the sites (mainly epcostello.net, frisket.org, and artific.com).

18% of the addresses do not reverse resolve.

Of the 387 addresses which did reverse resolve, 41 (11%) reverse resolve to an address which itself does not forward resolve to anything (that is: address 1.2.3.4 reverse resolves to something.example.com, but something.example.com itself does not resolve back to 1.2.3.4.

MSN has a number of hosts which reverse to a phx.gbl top level domain (64.4.8.113 through .118). That is to say, 64.4.8.113 reverse maps to by1sch4041904.phx.gbl, not .msn.com or .msn.net. I sent a note to the poc for the 64.4.8 network but it appeared to disappear into a black hole. Or they're intentionally reverse mapping to a nonsense domain.

Oddly, there's another 17 hosts which all reverse resolve to msnbot.msn.com, none of which are on the 64.4.8 network. That is to say: 17 hosts, across a number of networks and subnets, all reverse resolve to the same hostname, msnbot.msn.com, this hostname itself does not resolve to anything.

crawler.bloglines.com does not forward resolve to [65.214.39.151], though 65.214.39.151 resolves back to crawler.bloglines.com.

MSN seems to have the largest number of IPs and hostname mismatches or resolution failures.

One of the IBM gateways (I'm guessing in the Southbury, CT data center) reverse resolves [129.33.1.37] to bi01pt1.ct.us.ibm.com, which does not in turn resolve back to 129.33.1.37.

Nothing earth shattering here...most web sites turn off name resolution these days, doing it only in post-processing, or on a specific basis within an application. And no one who is remotely sane turns on HostnameLookups double in their server configurations.

Where it does come into play is if you are using hostnames in access control lists. Unlikely on a totally public site, but if you have a protected area, a semi-private extranet, and you add a Allow from .ibm.com, then anyone who's using that 129.33.1.37 gateway will get bounced, at least from Apache based servers since mod_access will perform a double lookup (at least according to the documentation).

I suppose there are other situations where you might use the hostname to allow access for search engine spiders, where otherwise you might require some other form of authentication (eg: set up a satisfy any block, add allow *.google.com, *.msn.com, *.yahoo.com and then a check for a cookie with a mix of Allow and SetEnvIf rules.

Update 8 July 2006

I noticed more search hits coming in to this article as well as some more comments and thought I'd post this update. I feel confident that the fake .phx.gbl top level domain name is being used by Microsoft/MSN, though I cannot understand why (if you're going to create PTR records, why not make them which your identifiable domain, since anyone can eventually determine who is assigned the address block).

I created the following table of addresses in the 64.4.8.0/255 network with the corresponding PTR records (reverse DNS records), this is a snapshot as of 0347Z on 8 July 2006.

If anyone from Microsoft network operations reads this, would you mind explaining why you're using a fake top level domain for your search engine robot? (It also appears to be used in some MSN/Hotmail mail headers, that's less of a concern or interest to me)

Update: September 6, 2006

• crawler.bloglines.com now forward and reverse resolves to 65.214.44.29.

Posted in

Received: from mail pickup service by hotmail.com with Microsoft SMTPSVC;
	 Thu, 13 Jul 2006 02:33:47 -0700
Message-ID: <BAY103-F4F8E2BA80472462866982AC6E0@phx.gbl>
Received: from nnn.nnn.nnn.nnn by by103fd.bay103.hotmail.msn.com with HTTP;
	Thu, 13 Jul 2006 09:33:43 GMT

bl1sch4061111.phx.gbl - - [24/Sep/2006:03:41:40 +0200] "GET /robots.txt HTTP/1.0
" 200 - "-" "msnbot-media/1.0 (+http://search.msn.com/msnbot.htm)"
bl1sch4061111.phx.gbl - - [24/Sep/2006:03:41:41 +0200] "GET / HTTP/1.0" 200 1165
7 "-" "msnbot-media/1.0 (+http://search.msn.com/msnbot.htm)"