Aliasing, Or "what's The Real Domain Name?"

We've been working on aliasing issues over at our "sitetruth.com", which rates sites based on the business behind them. One key issue for us is finding the "real domain" of a web site. Is it "example.com", "www.example.com", or something like "us.example.com"?

Google has punted on this; they now expect webmasters to go to the "Google control panel" and designate the "real domain". We wanted something automatic.

Here's what we're currently doing: First, we get the "base domain" from the requested domain name. This is what you buy from a registrar. "example.com" and "example.co.uk" are base domain names.

We try to read a web page from the base domain. If that doesn't work, we put "www." in front of that, like a browser does. If that doesn't work, we try whatever domain name (not full URL) the user or search engine gave us as a starting point.

Once we can read a page, we check for redirects. If "example.com" redirects to "www.example.com", we'll treat "www.example.com" as the actual domain. Such redirects have to be within the same base domain; we don't follow inter-domain redirects, because they might be from a third party.

This gives us the "real domain". We then read any SSL certificate associated with that domain, and look through the site content for the name and address of the business. The object of all this is to get a business name and address, so we can start checking business records (licensing, credit, directory, etc.)

Domains like "foo.example.com" are treated as subdomains of "example.com", and share its rating. We have a mechanism for manually specifying that a subdomain needs to be treated as a separate domain, which we use on "redirector subdomains", like "r.aol.com" and "rds.yahoo.com". Redirector scripts are often exploited by phishing sites, so they can put your web site into PhishTank. We notice that and blacklist. (Unless you have an extremely good reason, don't put an "open redirector", a script that will redirect to any URL, on your web site. And if you do, give it its own subdomain. Otherwise, you'll end up being blamed for phishing and will be on various blacklists.)

This handles most of the common cases, but there are a few situations that we don't recognize. Most of these are bad web design.

A common error is to have an SSL certificate for "www.example.com", but a web site at "example.com" that doesn't redirect to "www.example.com". This inconsistency results in invalid SSL certificate usage. If your SSL cert is for "www.example.com", your "example.com" should have a redirect to "www.example.com". Otherwise, the user who types [url] gets an SSL warning of a phony site. You don't want that. We've seen this on a few big-name sites that should know better.

On a related note, wildcard SSL certificates for "*.example.com" don't
cover "example.com". It's possible to get an SSL cert that covers more than one domain, which is the right solution to some of these problems.

Incidentally, we ignore "domain control only" validated certificates. If there's no business identity in the cert, it's meaningless as an indicator of business legitimacy.

We had an amusing issue with "robots.txt" files. There are some sites that use a "robots.txt" file to disallow access to their "example.com" domain, so that search engines will index the "www.example.com" domain. They usually put a redirect on the "example.com" domain, so browsers also go to the "www.example.com" domain. "ibm.com" is set up that way. Originally, we read the "robots.txt" file for "example.com", saw we were locked out, and refused to rate the site. We now look at the header part of the page at "example.com" first, to check for a redirect, and once past the redirect issue, start obeying "robots.txt" before we read any site content.

For another adventure in redirection, try [url]. Yes, Microsoft redirects their "robots.txt" file.

Anyone know of any more special cases we need to handle?