Too Busy For Words - the PaulWay Blog

Mon 19th Jun, 2006

Watching the search engines

Or: why Google rocks so much.

I love blinkenlights and CPU load graphs and live logs. tail -f was invented for me. So it's been amusing for me, since posting my page collecting all the Linux Australia Q&A session torrents on my local machine, to watch who's been accessing it and when. Specifically, which search engines picked up on it.

I posted it at around 2PM AEST (= GMT+1000) on Saturday 17th. A couple of people who were on the #linux-aus channel picked it up and grabbed the torrents. Someone from Linux Australia - James Purser presumably - blogged it at about 5:30PM that day and there were a few more. Nothing much more exciting happened until about 11AM AEST on Sunday 18th. Google picked it up then - a total of eighteen hours after it was posted on planet.linux.org.au. That is actually pretty quick.

Before any other search engine has time to scan the /~paulway/la page that was posted, Google has scanned it again (at 10:30PM same day). At 1:30 on Monday 19th, Yahoo catches up and indexes the page. During this time, several search engines have indexed various other bits of my local pages - my MythTV notes, a few pictures. IRLbot from irl.cs.tamu.edu has had time to completely stuff up my tracker's URL (dropping the port suffix off the name), so it's asked for a huge number of announce and scrape and stats pages that don't actually exist on the webserver. Forex, whose bot notification sadly leads to an Apache default page, is third. Google, for reasons I start gleefully speculating on, then indexes all the torrent files just in case. No other search engine has hit it in the two days that it's been up so far.

It's little wonder why nothing compares to Google for searching...

Last updated: | path: tech | permanent link to this entry


All posts licensed under the CC-BY-NC license. Author Paul Wayper.


Main index / tbfw/ - © 2004-2023 Paul Wayper
Valid HTML5 Valid CSS!