FTP Disconnects...

I am having this really irritating problem. I keep getting disconnected when I FTP into my VPS. First I thought it had to do with my VPS or hosting company but I think it's something else because I've had the same problem at two hosts.

These are the facts:
- Earthlink broadband through Time Warner's cable
- I live in Hawaii and my server is in Texas
- I use SmartFTP as my FTP client
- I have one VPS with Zone.net and one with JaguarPC

This is the error message:
A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.

Observations:
- it disconnects more often when I upload than when I download
- it seems to relate to file quantity rather than size of file

Does anyone know what would cause this? Is it my ISP?