How can I detect and survive being “Slashdotted”?

Question

What\'s a good way to survive abnormally high traffic spikes?

My thought is that at some trigger, my website should temporarily switch into a \"low bandwidth\" mode: swi

Answer 1

I rewrite all URLs referred by several popular sites to be redirected through the coralCDN.

An example for Apache:

<IfModule mod_rewrite.c>
RewriteEngine On
RewriteBase /

RewriteCond %{HTTP_USER_AGENT} !^Googlebot
RewriteCond %{HTTP_USER_AGENT} !^CoralWebPrx
RewriteCond %{QUERY_STRING} !(^|&)coral-no-serve$
RewriteCond %{HTTP_REFERER} ^http://([^/]+\.)?digg\.com [OR]
RewriteCond %{HTTP_REFERER} ^http://([^/]+\.)?slashdot\.org [OR]
RewriteCond %{HTTP_REFERER} ^http://([^/]+\.)?slashdot\.com [OR]
RewriteCond %{HTTP_REFERER} ^http://([^/]+\.)?fark\.com [OR]
RewriteCond %{HTTP_REFERER} ^http://([^/]+\.)?somethingawful\.com [OR]
RewriteCond %{HTTP_REFERER} ^http://([^/]+\.)?kuro5hin\.org [OR]
RewriteCond %{HTTP_REFERER} ^http://([^/]+\.)?engadget\.com [OR]
RewriteCond %{HTTP_REFERER} ^http://([^/]+\.)?boingboing\.net [OR]
RewriteCond %{HTTP_REFERER} ^http://([^/]+\.)?del\.icio\.us [OR]
RewriteCond %{HTTP_REFERER} ^http://([^/]+\.)?delicious\.com
RewriteRule ^(.*)?$ http://example.com.nyud.net/$1 [R,L]
</IfModule>

Answer 2

Use caching!

If you're using WordPress (for example), you can use something like WP-Super-Cache. If you're using regular PHP there are still a number of options you can use including memcache. Or you can just use regular squid proxy style caching.

Any caching you use will help bulletproof (or slashdot/digg-proof) your site :-)

Answer 3

1. install munin to monitor load/memory consumption etc and notify on overloads.
2. install monit to restart apache2 if it crashes
3. install nginx as apache2 frontend, it will massively decrease memory requirements under heavy load

Answer 4

No one has mentioned load balancing... haproxy, etc. Optimize, cache and load balance should survive almost anything. That being said, I'm not sure if stackoverflow is behind a load balancer ;)

Answer 5

Cache... hard. Record hits, and if a spike occurs, write out a completely static copy of the page being hit, then serve that. Cutting DB queries from 100 to 2 with a good caching system can survive a weak slashdotting, but having any DB queries at all will still result in a dead site under serious load that you aren't prepared for.

Answer 6

There are a number of ways this can be done, or at least helped. Search Google for "slashdot-proof" and you'll find a number of them:

• Slashdot-proof your server with FreeCache - Boing Boing
• Simple Thoughts Blog is now Slashdot Proof

etc.