July 01, 2004
New spider from Yahoo.com::[Search Engine]
For Chinese market, Yahoo launch www.yisou.com , which is mainly focus on China/Asia market.
Just after Yisou.com launched, it seems they also start using a new kind of spider, which they even didn't give it name yet.
web2.search.cnb.yahoo.com - - [22/Jun/2004:12:41:54 -0500] "HEAD /gmail HTTP/1.1" 200 - "-" "DeadLinkCheck/0.4.0 libwww-perl/5.69"
This is the first day yahoo spider crawl the website but only check the "DeadLink"
After that, it seems they spend 1 week to process the whole data and then remove the dead links and prepar for crawl the alive links.
web8.search.cnb.yahoo.com - - [01/Jul/2004:15:35:16 -0500] "GET /广州环境污染问题 HTTP/1.1" 200 2108 "-" "Mozilla/4.0"
This obviously a spider since it:
1] Only see these txt/html pages
2] No jpeg/flash and all other media loaded at the same time
3] Dig a website from link to link
4] the ip of this crawl locate: 北京市 Yahoo中国
The wired thing is that they didn't even give it a name, wish I can know the name soon.
Posted at July 1, 2004 04:00 PM by Liang at 04:00 PM | Comments (1) | TrackBack(0) | Booso!| Niu.la收藏!Trackback
You can ping this entry by using http://www.wespoke.com/cgi-bin/mt/mt-tb.cgi/526
