The first crawling tests have started this week-end, to be exact on Saturday the 15th of November 2008, and have already given quite some good content to start developing the TooT search engine front end. There are a few small issues of course, like every new starting project but we are confident that we can resolve them in a timely manner.

One of these issues is the handling of special characters such as the so called german « Umlauts », which is very important for the search engine because most of the websites in Switzerland are in German as this is Switzerland’s major language. So we must make sure words indexed with these special characters are indexed properly and can be searched for. The same applies to the french language and it’s « accents ».

Terminal screenshot of the TooT crawler live fetching and indexing various pages of swiss websites.

If you have already been TooTed, so to say crawled by us :-), you should see the following browser entry in your web access log files:

Toot/Nutch-0.9 (Toot crawler; http://www.toot.ch/; crawler(at)toot(dot)ch)

You will see the same in your web statistics software. So don’t be frightened, TooT doesn’t hurt, it just indexes your pages in our database and won’t do anything else. Of course, if you see some kind of strange behavior from our crawler don’t hesitate to contact us, but this shouldn’t be the case. 

The next test crawls have been programmed to automatically run in the night, starting at midnight Central European Time. This is to populate our test index with more and more content for better testing.

We will now be working on the front end as well as doing some crawling optimization at the same time and will keep you posted here on this blog about the progress of TooT. So stay tuned…