May 19, 2004

Searching result scripts::[Search Engine]

Thanks for Chedong, I remade the script he suggested to me and generate a new script which can find out the keywords and make a easy stastic from access_log:

The searching result for this website seems really funny and we do can find sth interest here: Searching result of wespoke.com.

Here is script
 #!/bin/bash #NAME="access_log.`date +%m-%d-%y`" NAME=$1 grep baidu.com/baidu $NAME > baidu.search cat baidu.search | gawk ' { print $7 } ' | sort | uniq > list.search cat baidu.search | sed "s/\ /#/g" > baidu.tmp mv baidu.tmp baidu.search echo "<html>" echo "===============Baidu=============== " for VAL in `cat list.search` do COUNT=`grep "#$VAL"# baidu.search | wc -l ` echo "<a href=http://blog.wespoke.com/$VAL>[$COUNT] http://blog.wespoke.com/$VAL</a>" echo " " grep "#$VAL"# baidu.search | tr "#" ' '|awk '{print $11}' | perl -pe 's/\\x(\w+)/%\1/gi' |perl -p -e 's/%(..)/pack("c", hex($1))/eg' | perl -pe 's/(.*)?(word=(.*?))[&"].*/$3/gi' > keys.search for VAR in `cat keys.search | sort | uniq ` do KEYCOUNT=`grep $VAR keys.search | wc -l ` echo "[$KEYCOUNT]<a href=http://www.baidu.com/baidu?word=$VAR>$VAR</a>||" done echo " " done rm baidu.search list.search keys.search echo "===============Google=============== " NAME="access_log.`date +%m-%d-%y`" grep google.com $NAME > google.search cat google.search | gawk ' { print $7 } ' | sort | uniq > list.search cat google.search | sed "s/\ /#/g" > google.tmp mv google.tmp google.search for VAL in `cat list.search` do COUNT=`grep "#$VAL"# google.search |grep search | wc -l ` echo "<a href=http://blog.wespoke.com/$VAL>[$COUNT] http://blog.wespoke.com/$VAL</a>" echo " " grep "#$VAL"# google.search | grep search | tr "#" ' ' | awk '{print $11}' | perl -pe 's/\\x(\w+)/%\1/gi' | perl -p -e 's/%(..)/pack("c", hex($1))/eg' | perl -pe 's/(.*)?(q=(.*?))[&"].*/$3/gi' | jv-convert --from utf-8 -to gb18030 > keys.search for VAR in `cat keys.search | sort | uniq ` do KEYCOUNT=`grep $VAR keys.search | wc -l ` echo "[$KEYCOUNT]<a href=http://www.google.com/search?hl=zh-CN&ie=gb2312&c2coff=1&q=$VAR&lr=lang_zh-CN>$VAR</a>||" done echo " " done echo "===============3721=============== " grep 3721.com $NAME > 3721.search cat 3721.search | gawk ' { print $7 } ' | sort | uniq > list.search cat 3721.search | sed "s/\ /#/g" > 3721.tmp mv 3721.tmp 3721.search echo "<html>" for VAL in `cat list.search` do COUNT=`grep "#$VAL"# 3721.search | wc -l ` echo "<a href=http://blog.wespoke.com/$VAL>[$COUNT] http://blog.wespoke.com/$VAL</a>" echo " " grep "#$VAL"# 3721.search | tr "#" ' '|awk '{print $11}' | perl -pe 's/\\x(\w+)/%\1/gi' |perl -p -e 's/%(..)/pack("c",hex($1))/eg' | perl -pe 's/(.*)?((p|name)=(.*?))[&"].*/$4/gi' > keys.search for VAR in `cat keys.search` do echo "<a href=http://seek.3721.com/right.htm?name=$VAR>$VAR</a>||" done echo " " done echo "===============Yahoo=============== " grep search.yahoo.com $NAME > yahoo.search cat yahoo.search | gawk ' { print $7 } ' | sort | uniq > list.search cat yahoo.search | sed "s/\ /#/g" > yahoo.tmp mv yahoo.tmp yahoo.search echo "<html>" for VAL in `cat list.search` do COUNT=`grep "#$VAL"# yahoo.search | wc -l ` echo "<a href=http://blog.wespoke.com/$VAL>[$COUNT] http://blog.wespoke.com/$VAL</a>" echo " " grep "#$VAL"# yahoo.search | tr "#" ' '|awk '{print $11}' | perl -pe 's/\\x(\w+)/%\1/gi' |perl -p -e 's/%(..)/pack("c", hex($1))/eg' | perl -pe 's/(.*)?(p=(.*?))[&"].*/$3/gi' > keys.search for VAR in `cat keys.search | sort | uniq ` do KEYCOUNT=`grep $VAR keys.search | wc -l ` echo "[$KEYCOUNT]<a href=http://cn.websearch.yahoo.com/search/web_cn?stype=&p=$VAR>$VAR</a>||" done echo " " done echo "===============Sina=============== " grep sina.com.cn $NAME > sina.search cat sina.search | gawk ' { print $7 } ' | sort | uniq > list.search cat sina.search | sed "s/\ /#/g" > sina.tmp mv sina.tmp sina.search echo "<html>" for VAL in `cat list.search` do COUNT=`grep "#$VAL"# sina.search | wc -l ` echo "<a href=http://blog.wespoke.com/$VAL>[$COUNT] http://blog.wespoke.com/$VAL</a>" echo " " grep "#$VAL"# sina.search | tr "#" ' '|awk '{print $11}' | perl -pe 's/\\x(\w+)/%\1/gi' | perl -p -e 's/%(..)/pack("c", hex($1))/eg' | perl -pe 's/(.*)?((_searchkey|word)=(.*?))[&"].*/$4/gi' > keys.search for VAR in `cat keys.search | sort | uniq ` do KEYCOUNT=`grep $VAR keys.search | wc -l ` echo "[$KEYCOUNT]<a href=http://search.sina.com.cn/cgi-bin/search/search.cgi?_searchkey=$VAR>$VAR</a>||" done echo " " done echo "</html>" rm google.search list.search keys.search

Posted at May 19, 2004 11:25 AM by Liang at 11:25 AM | Comments (0) | TrackBack(0) | Booso!| Niu.la收藏!

Trackback

You can ping this entry by using http://www.wespoke.com/cgi-bin/mt/mt-tb.cgi/463

搜索引擎研究

May 19, 2004

Searching result scripts::[Search Engine]

Trackback

Comments

Post a comment

搜索引擎研究
Blog::Technology