Slowly we are arriving at the limits of our current hardware due to the increasing amount of time required for crawling Swiss websites. As a matter of fact it is quite normal that the crawling takes more and more time due to the increasing index size and the amount of data gathered. The crawling process itself is still quite fast but merging the update crawls to the master index as well as storing the data takes quite some time now.

The server itself only does the crawling whereas the data crawled by that same server gets stored on a network file server. This is actually already one bottleneck for a search engine as the data should be stored on the same crawling server or cluster of servers to avoid losing performance due to the data continuously being read and written through the network to the file server. Second bottleneck is the memory of the server, which is currently limited to 1 GB. Nowadays 1 GB is really nothing, but the server being already a few years old won’t fit any extra memory. And last but not least bottleneck: the CPU which is always working at full power during the crawling and indexing.

It is not really a big problem to continue running like that but it simply takes longer than we would like it to. Therefore and to speed things up we are looking for a hardware sponsor which could sponsor us a server with the minimum following requirements:

• two Intel Quad Core 2.6 GHz processors
• 8 GB of memory
• two disks of 1 TB in a hardware RAID 1 configuration
• two redundant power supplies

Thinking “Swiss”, as our search engine does, we have inquired last week a Swiss based hardware company called Transtec but unfortunately they already used their whole yearly sponsoring budget. I am guessing that this is due to the CERN with their Large Hadron Collider as Transtec is listed as an official sponsor for this massive project. After reflection, this made me think that the end of a year is not such a prosper time to look for a sponsor. On the other side we are soon in 2009, so let’s think positive. In this enthusiastic state of mind we have reiterated our sponsor query asking this time the computer company DELL and have sent our sponsor inquiry letter this Tuesday. Let’s hope this will work as their PowerEdge servers are really nice.

Just to mention that in counterpart we will offer a prominent place to the hardware sponsor on our future and planned sponsor page. Right now we already have one sponsor for the hosting of our server but I will speak about this sponsor in another separate post.

I will of course let you know as soon as we get an answer on the outcome of our sponsoring inquiry with DELL. Until then our current server will do his best and will be working really hard…