tag:blogger.com,1999:blog-14857972.post4311544962507447367..comments2023-08-27T00:21:36.908+08:00Comments on Click To Know...! Ask For More...!: How To Ask Alexa Crawler To Crawl Your Site.adminhttp://www.blogger.com/profile/02162709817427625166noreply@blogger.comBlogger2125tag:blogger.com,1999:blog-14857972.post-15456951126820535532009-07-27T05:26:59.981+08:002009-07-27T05:26:59.981+08:00thats all very well but HOW do you get it to recr...thats all very well but HOW do you get it to recrawl your site ?<br />My site is www.freeipod4.me.ukAnonymousnoreply@blogger.comtag:blogger.com,1999:blog-14857972.post-10895147013306645492008-02-18T07:48:00.000+08:002008-02-18T07:48:00.000+08:00ia_archiver has 'accidentally' crawled some privat...ia_archiver has 'accidentally' crawled some private (password protected!!!) admin pages on one of my sites - pages that cause the deletion of stock!<BR/><BR/>It looks to me like ia_archiver not only crawls pages that it finds links to, but also pages it discovers through the toolbar.<BR/><BR/>Don't rely on robots not finding files you don't want them to see - block them with the robots.txt file as a precaution!Keithhttps://www.blogger.com/profile/07998348011624561853noreply@blogger.com