Too High Server Load, How To Handle

if upgrading to that new server that I'll mention will probably solve my problems. Whatever help you can provide would be greatly appreciated. Below are the details:

In the GMT evenings and nights my current server gets so loaded that every page load takes 10 - 30 seconds. Even the pure html pages will be so slow to load. It seems that after a certain treshold it just suddenly becomes that much slower. Not much middleground there. I have high MaxClients and ServerLimit values now and the error log doesn't say that they are exceeded anymore but that didn't help enough.

I have a high traffic website that is using latest version of apache (2.2.x) with the prefork MPM and apache is optimized, PHP 5.2.5 and APC 3.0.15.

I get 160,000 - 210,000 pageloads per day. 32,000 - 45,000 visits per day.

Most of its pages are PHP but shouldn't be too CPU or databes intensive. Mysql isn't used and I mostly used sharedmem (php's shm functions) for databases. 2 semaphores are quite heavily used but that can't explain how a few more users would make the server serve pages so much slower.

Swap usage is practically 0 and CPU user % usage is like 1 - 2 % and CPU system % is also about the same even during peak times. However the Average Load or whatever that "top" reports is 6 - 9.

My current server scecs: 1 GB Ram, Pentium D 3 ghz, CentOS 5 32bit fully updated.

I load all pictures and even the stylesheet from a secondary server by using href="$secondaryserverIP..." in the html code, so the main server practically just serves the pages.

My new server will have apache with the worker MPM and latest versions of every software. Also its specs are: 2 GB of RAM, Intel Dual Core Xeon 2.40GHz, CentOS 5.1 32bit fully updated.

I have a sophisticated netstat based ddos script that is an improved version of DDoS Deflate and while some of these slowdowns seem to have been caused by attacks that it then was able to defend me from, most of them are not. I am even protected from users who constantly have 7+ connections to my site and if someone has a way too high number of connections, the script won't even check if it constantly has it and the script just bans that user outright. It probably is banning a bunch of innocent proxy users too but that is a small price to pay.