How large RSS reader works (netvibes, Google reader…)
I wonder how web applications like Google Reader, Blogline, techronati works, and what technics they follow to parse millions of RSS feeds using cron job at one time? There is a lot of different techniques... the "worst" one being the one that you describe. (time based polling). The first thing you need to consider is that they may not all do the parsing on the server side. For example, I know that Netvibes was doing the parsing on the client side (but cached the content on the server), so it saved them a lot of resources. This way they would poll feeds only when users asked from them, so