Blogdriver again :) New problems coming out one by one, yesterday the problem is the Deadlock of our Database. At the same time, there are around 100 users query our database, this number is not large at all. So, I thought it should be the configuration problem in the SQL server. I was proven to be right. Usually, when a database meet deadlock, we need look into detail: 1] Do you set lock in your program? 2] Check if you use too many affairs and open too many connections. 3] Please check the Lock Granularity, table, database? 4] Check the SQL server log, they will show you the exactly which SQL cause deadlock 5] What kind of these deadlock? S(share), Exclusive, IS.. ? and are they make sense? 6] What is the Deadlock Timeout setting too long? Default is -1. Here is a paper about the Deadlock, very useful and have lots detail also:...
In the Apache+tomcat clustering, tomcat server maybe down after several days running, I wrote a script to watch on the free memory and the response of the tomcat. this script will auto restart tomcat when the response is very slow and the free memory is less than 15K. cat watch.sh #!/bin/sh cd /root/bin PID=`ps -aef | grep Xms500M | grep -v grep | gawk '{print $2}'` PID=`expr $PID + 1 - 1` date free echo $PID echo "------------------" if [ $PID -eq 0 ] then sleep 10 /usr/java/tomcat/bin/startup.sh sleep 160 fi while [ 1 ] do date free echo "Tomcat process ID is $PID" wget http://192.168.1.101/jsp/w_blog/blog.jsp -O working.jpg 2>> /dev/null & sleep 120 touch working.jpg SIZE=`du working.jpg | gawk '{print $1}'` if [ $SIZE -le 20 ] then WID=`ps -aef | grep 192.168.1.101 | grep -v grep | gawk '{print $2}'` WID=`expr $WID + 1 - 1` if ! test -z $WID then killall wget fi echo "Tomcat restart checking......" free vmstat FREEMEM=`free | grep Mem | gawk '{print $4}'` if [ $FREEMEM -le 15000 ] then if [ $PID -ne 0 ] then kill -9 $PID ls -l /usr/java/tomcat/bin/shutdown.sh fi /usr/java/tomcat/bin/shutdown.sh sleep 10 /usr/java/tomcat/bin/startup.sh sleep 30 fi date ps -aef | grep -v httpd ls -l PID=`ps -aef | grep Xms500M | grep -v grep | gawk '{print $2}'` PID=`expr $PID + 1 - 1` fi rm -f working.jpg done...
http://www.blogdriver.com switch to shanghai IDC, we have 3 more servers. In the last few days we worked on the clustering and load balancing. It seems working great after we setup Load balancing and clustering. configure: Hardware: dual-cpu xeon. 2.4G 1Gmem Linux server Broadcast: 228.0.0.4 Server: app1 + app2 +app3 Testing: sessions can duplicate in different servers. Tolerance: 2.96 times VS the single server configure file: apache mod_jk , server.xml There is an article I highly recommand:...
since grep can't support binary or extend-coding, I change most of my scripts to pcregrep. To my surprise, the script runing much faster after I switch to pcregrep: Here is a example: mcfarm:~/work/rootfit/background-only-step1/output$ time pcregrep 1000 exper.1.data > /dev/null 0.940u 0.030s 0:00.97 100.0% 0+0k 0+0io 103pf+0w mcfarm:~/work/rootfit/background-only-step1/output$ time grep 1000 exper.1.data > /dev/null 28.350u 0.470s 0:29.45 97.8% 0+0k 0+0io 155pf+0w 0.94 VS 28.35 !!! Generally say, pcregrep is more than 20 times faster then grep. pcregrep searches files for character patterns, in the same way as other grep commands do, but it uses the PCRE regular expression library to support patterns that are compatible with the regular expressions of Perl 5. See pcre(3) for a full description of syntax and semantics. If no files are specified, pcregrep reads the standard input. By default, each line that matches the pat- tern is copied to the standard output, and if there is more than one file, the file name is printed before each line of output. However, there are options that can change how pcregrep behaves....
1] Change '&'#XXXXX;' to Utf-8: perl -p -e 's/&#(.....);/pack("U", $1)/eg' 2] Change \xb?\xb? to GB2312 perl -p -e 's/\\x(..)/pack("c", hex($1))/eg' 3] Change %C2%D2%C2%D7 to gb2312 perl -p -e 's/%(..)/pack("c", hex($1))/eg' basicly, jv-convert is the best tool to convert different formats in UNIX/Linux: jv-convert --from UTF-8 --to gb18030 : jv-convert --help Usage: jv-convert [OPTIONS] [INPUTFILE [OUTPUTFILE]] Convert from one encoding to another. --encoding FROM --from FROM use FROM as source encoding name --to TO use TO as target encoding name -i FILE read from FILE -o FILE print output to FILE --reverse swap FROM and TO encodings --help print this help, then exit --version print version number, then exit...
On April 1st, google announce the launch of their ambitious mail service system—Gmail, which can uniquely offer a mail storage as large as 1G as a free mail. Despite all the obvious feasures of gmail, like large storage mail size, unique mail searching and organizing method, here we will focus on the current mail war among the major email service providers, and also pay special attention to google’s marketing strategy. Email War and Google’s way of marketing After google’s first gunfire in the email war, yahoo is the first to respond by increasing its email storage as large as 100M, 1/10 of Gmail, and promising better virus-protection, and ad-scan service. Followed by Microsoft, the largest mail service provider, plan to increase all their hotmail user’s mail storage to 250 by early July. First rule of marketing is that product should meet the market’s need, and market pull is always stronger superior to product push. For the usual industry cases, product push is only important for new start-up, venture capital and firm with large budget in R&D, while most successful cases are based on carefully serving the well-defined target market segment. Gmail proves its greatness by counter-proving this rule. It chose to enter the email market, which don’t have immediate need for new email account, as we already have at least email account. However, when google came out, we all want a gmail account in action, and we are eager to get others’ invitation. Google’s strategy at this level is simply that once it wants to beat this mature market, it must beat it hardly as unimaginable as 1 G mail. Gmail’s marketing also benefits from the carefully maintained reliable image and reputation. There is still one advanced rule of marketing: to create a demand, even it doesn’t exist now. This demand is usually created by call up consumers’ new awareness or re-examine the original assumption. As derived from the email market, all major mail providers are crazy about expanding their business; Microsoft is slowly gaining large reward from their MSN Messenger Platform, which was launched up years ago, now they are integrating more business factors including ticket and hotel booking, car-renting, dating match etc. varied with local situation. Yahoo with its Yahoo messenger regarded as the follower but actually rival to MSN is far less successful than MSN messenger. When everyone’s mind is occupied with the fast expanding business, google...
There are so many fake referral from Spam website which is try to catch my eyeballs. There are two easy way to ban them: 1] In httpd.conf #################################### SetEnvIfNoCase Referer ".*(casino|gambling|poker|porn|sex|hqsearch|webcamss|rape).*" BadReferrer order allow,deny allow from all deny from env=BadReferrer #################################### 2] Using .htaccess to ban Refer and also Ban IP #################################### SetEnvIfNoCase Referer ".*(casino|gambling|poker|porn|sex|hqsearch|webcamss|rape).*" BadReferrer order deny,allow deny from env=BadReferrer deny from 63.81.44.2 deny from 69.50.191.130 ####################################...
For Chinese market, Yahoo launch www.yisou.com , which is mainly focus on China/Asia market. Just after Yisou.com launched, it seems they also start using a new kind of spider, which they even didn't give it name yet. web2.search.cnb.yahoo.com - - [22/Jun/2004:12:41:54 -0500] "HEAD /gmail HTTP/1.1" 200 - "-" "DeadLinkCheck/0.4.0 libwww-perl/5.69" This is the first day yahoo spider crawl the website but only check the "DeadLink" After that, it seems they spend 1 week to process the whole data and then remove the dead links and prepar for crawl the alive links. web8.search.cnb.yahoo.com - - [01/Jul/2004:15:35:16 -0500] "GET /广州环境污染问题 HTTP/1.1" 200 2108 "-" "Mozilla/4.0" This obviously a spider since it: 1] Only see these txt/html pages 2] No jpeg/flash and all other media loaded at the same time 3] Dig a website from link to link 4] the ip of this crawl locate: 北京市 Yahoo中国 The wired thing is that they didn't even give it a name, wish I can know the name soon....