Maximizing Linux (CentOS) For Maximum Concurrent Users

I am running lighttpd and eccelerator.

I have stripped the php-cgi.

I have tried forking anywhere from 50 to 300 to 1000 fastcgi children with PHP_FCGI_CHILDREN (through lighttpd, I'll attempt to do with same with spawn-fastcgi and fpm)

Server stats:

Intel Core i7 920
12GB DDR
250GB 7.2k RPM SATAII 16MB Cache

I am using to lighttpd's lighttpd-status to estimate concurrent connections.

When I refresh the panel, it shows that there are around 100-150 connections and around 150 requests/s in the last 5 seconds.

My vmstats show that CPU is 98% idle. Blocks written/read is neglible. MySql key_buffer set to 2gb and I'm pretty sure it's not mysql. The overwhelming majority of requests do not access mysql.

EDIT: Uh oh, I just realized that tcp_mem could be a huge bottle neck.

I just set it to:

net.ipv4.tcp_mem = 4096000 87380000 4194304000

It was previously:
net.ipv4.tcp_mem = somenumber somenumber 393,216 <<<--- WTF!

x1000 for my read values (it's an access server only). I can't benchmark the server right now so let me know if you have any suggestions besides this. I do think that this was the problem. When under load images could not be accessed either.