tcpsndbuf        192296     192296    2867477    4096277  151117486

Why was it happening?

After a little research into what a tcpsndbuf is and why this error might arise, it turned out that most people have this problem when tons of emails are being sent out all at once.  If you think this is your problem, check your mail log to see if anything’s fishy.  Look at what was happening email-wise at the time of the “black zone”.  Here’s a post about tracking down a rogue email script.

Another reason this might happen is when large files are being downloaded simultaneously.

Tracking it down

First we checked the mail logs to see if one of our clients had a script that was spamming, or sending out newsletters to a giant email list or some other nefarious plot was at hand.  No dice, they’re good citizens.

Next we checked the bandwidth usage and noticed that one domain in particular was using much more than usual (like 20 times what it usually uses per month).  This site was a blog that ran a weekly feature tracing a single song through all its manifestations being covered by other artists.  Each post had 5 – 10 mp3s. Now we were getting somewhere.

Checking out their apache access logs it turned out that these audio files had been discovered by a website in China.  There were tons requests for the audio files with no referer, originating from an IP in China, or from the site itself.

Fixing It

I’m not trying to hook China up with free Xiu Xiu mp3s so here’s how we stopped them.  Basically we set up an .htaccess file that rejected requests for audio files where the referer didn’t at least have our domain name in it, or that came directly from the offending site.  Here’s the code:

RewriteEngine On
RewriteCond %{REQUEST_URI} /audio/.*
RewriteCond %{HTTP_REFERER} !.*theDomainInQuestion.* [OR]
RewriteCond %{HTTP_REFERER} .*the\.incomingIP\.fromTheChinese\.site.*
RewriteRule .* – [F]

The problem with this method is that it’s pretty strict.  Because it will refuse any downloads that don’t come directly from the domain, this means it will deny links from an RSS feed too.  You could get around this limitation by changing line 3 to something like:

RewriteCond %{HTTP_REFERER} ^$ [OR]

which would prevent requests from no referer, and then tweak it from there, adding more [OR]s to block specific sites.

In consultation with the site owners we decided this strict method was fine.  We added a notice to each post in their RSS feed explaining the problem and notifying readers that the audio files were available on the site.