eliminating request duplication

Identifying Database Churn

Dynamic web page generation can significantly tax the processing capacity of a web server, particularly when multiple concurrent users are being served.  In building a typical dynamic page, the web server’s ‘listener’ receives a request for dynamic content and passes that request to the application server.  The application server will then make calls to a database before formatting the response as html.  This process can be extremely processor and I/O intensive; an individual web page may involve as many as 60 round trips to the database, requiring the server to process each action.  If personalization technologies are used, the server load is still further increased.

The 80/20 Rule

Data requests from large populations typically follow the 80-20 rule; 80% of the requested dynamic content is identical, while about 20% is unique.  As a result, web servers are required to process the same data repeatedly.  SpiderCache leverages this repetition by storing requested page instances in a dynamic cache that is located within the site’s web server.  As a result, the server doesn’t need to compile and execute scripts when content is served out of the cache, thus reducing the load on the CPU and memory.