and curious as to why it is downloading your entire site.
you may want to invite the robot to archive your site.
Either way, here is important information you should know about the web-crawling robot:
The robot, which identifies itself as ia_archiver in the HTTP "User-agent" header field, uses a site-by-site crawl strategy. Basically, it starts at the top of the site and drills down, fetching all of the local links that it finds as it goes. There are several advantages to this approach, but chief among them is that it can be as polite as possible to the site being crawled.
We will not archive anything you request to remain private.
All you have to do is tell us. How? By using the Standard for Robot Exclusion (SRE).
The SRE was developed by Martijn Koster at Webcrawler to allow content providers to control how robots behave on their sites. All of the major web-crawling groups, such as InfoSeek and AltaVista, resepect this standard. The Archive strictly adheres to the standard, and extends it in several respects:
Please write Alexa Internet with your questions and concerns.