Hitting Connections Per Second Limit Of RedHat EL

I have a powerful 8 core 8gb ram web server with scsi raid drives running RedHat EL 4. This server handles 2,000 - 3,000 HTTP requests per second via Litespeed httpd without strain (over 60%+ CPU idle time during peak load, under 1% IO wait). As the traffic volume continues to increase I've encountered a strange problem, the symptoms of which are as follows:

- About 1/4 or 1/3 of new connections are not answered by the server - they time out.

- All connections that are answered have exactly 3 seconds added to the time it takes to establish connection with the server (can be seen as "Connecting to ..." phase in FireFox). HTTP response times were tested by Pingdom from multiple locations all over the world.

- The problem is either "on" or "off", it is not gradual.

- Server ping is unaffected during the problem - no delay and no packet drops.

- The problem does not happen during off-peak hours of the day.

If litespeed httpd settings are tweaked to keep as many connections as possible in keepalive state for as long as possible, the problem is avoided, while tens of thousands of connections are kept in keepalive state.

Possible causes that were tested and eliminated: PHP/MySQL load (problem applies to static files exactly the same), CPU / IO / RAM, network uplink, hardware firewall, DNS.

This makes me think that there is some kind of bottleneck of how many NEW connections per second the server can accept. By maxing out keepalive quantity and duration I'm reducing the number of new connections per second. This is a temporary fix that will only work up to a certain point.

After investigation, litespeed staff verified that my litespeed configuration was correct and after some testing said that nothing in litespeed was responsible for this limiting factor. Litespeed process uses relatively little CPU and can definitely handle more volume.

Following sysctl.conf values were increased substantially to see if that will make a difference: tcp_max_syn_backlog, tcp_max_tw_buckets, tcp_max_orphans, netdev_max_backlog, somaxconn, file-max. This didn't produce any results. Disabling syncookies didn't help either. dmesg doesn't have any notices of limits being hit or throttles being applied.

Litespeed staff suggests that likely some limit in linux kernel is being reached. The strange 3 second delay does seem like an "intelligent" DDOS protection strategy of some sort. Perhaps this is some kind of kernel level DDOS protection?