Is 512 Enough RAM For A Small LAMP Setup

I'm setting up a VPS server on Slicehost, I've followed the guide on howtoforge for debian and have everything installed including ISPConfig3. Everything seems to work fine at the mo but when I check the memory usage, i see that I'm using around 490/500mb of my 512mb setup. Its a Xen setup so I understand this is real memory available but I'm not sure if this is enough.

I will ultimately be running around a dozen sites, some static and some dynamic. There's one Mambo and one Joomla site there but they don't update. The rest are all php/mysql sites that I've built myself but again, they don't change that much. None of the sites currently see more than a couple hundred uniques a day, and some only a handful, so usage is not high. Most of the domains have e-mail accounts attached, but its only moderate usage with around a dozen addys per domain max.

So, do I need more memory or will 512 be enough? Is there anything I can do to reduce the load and give myself some more room?

I'm moving from a shared hosting environment and while I'm comfortable setting up the server, I would rather have a simple mail manager that would give users more control over their accounts, change passwords etc, which I'm hoping ISPConfig3 will be able to help with, but thats probably best discussed in another post.