Server Network Very Slow

problem with my server running CentOS 5.3.

I noticed that there are huge pings to my server from time to time, example:

------------------
64 bytes from HOSTNAME (server-IP): icmp_seq=0 ttl=60 time=2.93 ms
64 bytes from HOSTNAME (server-IP): icmp_seq=1 ttl=60 time=2.70 ms
64 bytes from HOSTNAME (server-IP): icmp_seq=2 ttl=60 time=1901 ms
64 bytes from HOSTNAME (server-IP): icmp_seq=3 ttl=60 time=899 ms
64 bytes from HOSTNAME (server-IP): icmp_seq=5 ttl=60 time=2.69 ms
64 bytes from HOSTNAME (server-IP): icmp_seq=6 ttl=60 time=2.62 ms
64 bytes from HOSTNAME (server-IP): icmp_seq=4 ttl=60 time=2132 ms
64 bytes from HOSTNAME (server-IP): icmp_seq=8 ttl=60 time=2.57 ms
64 bytes from HOSTNAME (server-IP): icmp_seq=7 ttl=60 time=1190 ms
64 bytes from HOSTNAME (server-IP): icmp_seq=10 ttl=60 time=2.65 ms
64 bytes from HOSTNAME (server-IP): icmp_seq=9 ttl=60 time=1048 ms
64 bytes from HOSTNAME (server-IP): icmp_seq=12 ttl=60 time=2.74 ms
64 bytes from HOSTNAME (server-IP): icmp_seq=11 ttl=60 time=1205 ms
------------------

First I thought that it is network related, but most strange for me was that I did not have any packets lose.

Then I tried to ping from my server to other hosts - situation was the same - some ping were good and some were huge (700ms, 800ms, even 2000ms)

I checked:
cat /proc/sys/net/ipv4/netfilter/ip_conntrack_max
and it was 65536

Then I checked:
cat /proc/sys/net/ipv4/netfilter/ip_conntrack_count
and it was ~1600 so normal.

dmesg was showing such errors:

conntrack_ftp: partial 227 3331059707+13

Then I checked ifconfig, here is the output:

------------------
eth0 Link encap:Ethernet HWaddr 00:24:21:57:2B:6F
inet addr:MAIN=IP Bcast:BCAST=IP Mask:255.255.255.192
inet6 addr: fe80::224:21ff:fe57:2b6f/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:170557113 errors:0 dropped:2421127049 overruns:0 frame:0
TX packets:182047660 errors:0 dropped:46 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:2756835074 (2.5 GiB) TX bytes:79640621 (75.9 MiB)
Interrupt:82 Base address:0xe000
------------------

Strange for me was:

RX packets [...] dropped:2421127049
TX packets [...] dropped:46

I did not have such dropped information on all my other servers. Dropped counter for RX was constatnly increasing.

So I decided to restart all services on the server. After restarting network and ipaliases - problem disappeared. RX dropped counter is still rising, but I do not have any slowdowns on the server and pings are normal.

My question is - does anyone could have any idea what can casue my problem and how can I prevent this in the future?