Robots Flood My Server
Jul 26, 2009
I use Outpost Firewall to view active connections to my server. If I don't restart the httpd service on a regular basis my server will grind to a halt from being flooded by robots.
I currently have the service set up to restart at Midnight and Noon every day. Sometimes that's enough, lately it's not. For example, I checked an hour ago and I had 385 connections to httpd. At least 50% of the connections were robots - tons of the same IP addresses and they're just crawling the site.
Almost all of the connections show up as less than 1kb bytes received and 0 bytes sent per connection.
I already have a good 20 connections by these robots and the connection time shows as 11 minutes... I just browsed to a web gallery page on my site figuring that'd be mildly "intensive" on connections with all the thumbnails and my connections aren't lasting more than one minute.
So, what's with all these connections that are lasting 10+ minutes? I've even got one connection that has an Uptime of 30 minutes, bytes sent 65811, bytes received 180. It seems like something with these robots doesn't terminate correctly...
what to do so these connections quit jamming my server up? It's like a very very slow DOS...
View 3 Replies
ADVERTISEMENT
Oct 25, 2009
I'm having a issue with my current robots.txt file , which is not properly handling the requests/ blocking the content to be access . What I want is that to only allow like google bots , yahoo , msn , bing , alexander ranking beside those bots block all other bots . my current file rebots.txt is below
Code:
User-agent: Googlebot
Allow: /
User-agent: googlebot-image
Allow: /
User-agent: googlebot-mobile
Allow: /
User-agent: MSNBot
Allow: /
User-agent: Slurp
Allow: /
User-agent: Teoma
Allow: /
User-agent: twiceler
Allow: /
User-agent: Gigabot
Allow: /
User-agent: Scrubby
Allow: /
User-agent: Robozilla
Allow: /
User-agent: Nutch
Allow: /
User-agent: ia_archiver
Allow: /
User-agent: baiduspider
Allow: /
User-agent: naverbot
Allow: /
User-agent: yeti
Allow: /
User-agent: yahoo-mmcrawler
Allow: /
User-agent: psbot
Allow: /
User-agent: asterias
Disallow:
User-agent: yahoo-blogs
Allow: /
View 5 Replies
View Related
Jan 8, 2008
Googlebot is eating up about 30GB bandwidth per month.
I want to prevent him (and other robots) from spidering:
[url]
and all derivatives thereof (e.g. [url])
Is this possible with robots.txt? How would the code look?
Will this work?
User-agent: *
Disallow: /directory1/page1.php
or do I need to somehow specify a wildcard, e.g.
User-agent: *
Disallow: /directory1/page1.php?*
View 1 Replies
View Related
Mar 21, 2008
1 Do I really need a robots.txt file?
2 Don't misbehaved spiders simply ignore them?
3 For 'disallow', shouldn't I only include urls which are linked from public pages - and not those which I use for testing and which aren't linked-to from any public pages?
4 If I include such urls in 'disallow', aren't I simply alerting spiders (and anyone else who wants to see what sections of my server I don't want known) to stuff they'd otherwise not discover?
View 4 Replies
View Related
Apr 28, 2009
Do you know what is most important search engine robot ips?
I need to all serach engine robot ips too firewall white list.
View 3 Replies
View Related
May 11, 2009
I've been getting a syn flood for the last week or so.
I've pretty much tried everything I could online but have been unsuccessful in stopping them.
I talked to the data center techs and they basically can't put a stop to it either.
Here's a very small portion of my netstat
tcp 0 0 xxx.xxx.xxx.xxx:80 86.50.121.144:8540 SYN_RECV -
tcp 0 0 xxx.xxx.xxx.xxx:80 41.100.16.152:7824 SYN_RECV -
tcp 0 0 xxx.xxx.xxx.xxx:80 52.53.22.7:3146 SYN_RECV -
tcp 0 0 xxx.xxx.xxx.xxx:80 77.217.49.124:1659 SYN_RECV -
tcp 0 0 xxx.xxx.xxx.xxx:80 75.162.93.151:3230 SYN_RECV -
tcp 0 0 xxx.xxx.xxx.xxx:80 113.85.63.249:1656 SYN_RECV -
tcp 0 0 xxx.xxx.xxx.xxx:80 15.253.35.29:8849 SYN_RECV -
tcp 0 0 xxx.xxx.xxx.xxx:80 24.56.59.180:6911 SYN_RECV -
tcp 0 0 xxx.xxx.xxx.xxx:80 33.185.99.83:1917 SYN_RECV -
tcp 0 0 xxx.xxx.xxx.xxx:80 103.5.8.249:4782 SYN_RECV -
root@xxx [/]# netstat -nap |grep SYN |wc -l
2008
The IP's change often and it's not possible to narrow it down.
So far the things I have done;
syn cookies enabled
reduced time out
echo "1" > /proc/sys/net/ipv4/netfilter/ip_conntrack_tcp_timeout_syn_recv
increased
echo "150000" > /proc/sys/net/ipv4/ip_conntrack_max
installed apf but it slowed down the server to a crawl which made my clients really unhappy so had to remove it.
The bandwidth is constantly staying at 30Mbps with slight bumps here and there but every day around 7pm it drops completely to normal levels and the flood stops. It starts back up around 7 in the morning.
View 6 Replies
View Related
Jul 28, 2009
We are currently experiencing an SYN Flood attack on our primary production server and are looking for some help in resolving the issue.
Running:
Novell SUSE Linux Enterprise Server 10.2-64
SuperMicro X7DBR-E Intel Xeon QuadCore DualProc SATA [2Proc]
Processor Intel Xeon-Clovertown 5320L-QuadCore [1.86GHz]
8GB Memory
@ Softlayer DC in Texas.
Need help within the next hour or two. Please ask any necessary follow up questions and how you might go about resolving the issue (i.e. SYN Cookies, etc.)
View 5 Replies
View Related
May 3, 2009
someone decided to attack my webserver and I can't figure out how to block it.
tcp 0 0 localhost:80 207.44.129.88:2138 SYN_RECV
tcp 0 0 localhost:80 207.44.129.88:2243 SYN_RECV
tcp 0 0 localhost:80 213.66.121.211:63372 SYN_RECV
It's literally thousands of those requests overloading apache. The server is fine, the load average is like .8. But none of the website are loading.
We're hosting with ThePlanet, and they're doing a great job at blocking a huge portion of the attack. But we're still getting hit pretty hard. I've got APF installed, and 3 or 4 anti-dos scripts.
Every once in a while a page will load for the websites, I think we've got just under 50 legit connections.
View 14 Replies
View Related
Jun 11, 2008
How can I best work with a syn flood? I've tried the apf, deflate-ddos etc.... and don't work. Even tried litespeed etc but doesn't work against a 90mbps attack.
If I get a few servers, how would I have it setup to best defend?
View 6 Replies
View Related
Jul 25, 2008
one of my server were hit by massive ddos syn atack. target was port 80-apache
i am running centos 5 in xen vps iptables were strong with syn filtering and limit
but...what can do?
View 1 Replies
View Related
Mar 8, 2008
I don't know it anymore. Tried everything. I can not reach my server properly. A lot of time time out.
netstat -anp |grep 'tcp|udp' | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -n
1 127.0.0.1
1 38.99.44.101
1 64.131.90.38
1 78.176.158.75
1 78.179.73.135
1 88.229.70.143
1 88.242.97.204
1 88.247.87.58
1 92.112.211.223
3 78.176.175.136
3 88.229.40.142
3 88.231.180.181
4 78.183.227.146
18 0.0.0.0
348
348 connections from an empty IP? i have tried syn cookies, let the firewall block all ports without results.
View 10 Replies
View Related
May 28, 2008
Can anyone share tips how to prevent DNS flood on a cPanel and Directadmin server platform on Centos?
View 7 Replies
View Related
May 29, 2008
Does keepalive off help in syn flood?
View 6 Replies
View Related
May 4, 2008
My server run after 10h sevices ftp is down (network error: connection timed out). may be flood ftp.
how to Detect flood ftp.
View 4 Replies
View Related
Apr 4, 2008
We got hit with a huge bandwidth bill for last month. It was 4X our usual bill. The ISP said that we were the victim of UDP flood attacks from an outside server. We have a sonicwall router and the firewall seems to be blocking the port that the ISP claims the attacks can from. Is it possible that the attacks would still count towards our bandwidth usage even if the connection is refused by our firewall? Our ISP uses 95th percentile billing.
View 6 Replies
View Related
Oct 30, 2007
One of the servers have 1 account on, but seems like its extremely attacked. I cannot SSH and many packet loss. so I asked softlayer and they access it and said its a SYN Flood as from the /var/log/messages (I cannot see it as the server is not accessable) they put the main public ip under Cisco guard but still didn't help. when I asked for any solution, unfortunaly I were told there isn't and have to wait the attackers to stop as it comes from MANY addresses that iptables even won't help.
Isn't there any solution (software-hardware) to stop that ?
View 14 Replies
View Related
Jun 25, 2008
one of my costumers server is getting ddos attacks. I solved syn and get attacks with litespeed web server but I have another problem. They started to do udp flood. I m losing connection to my server. I bought new server with 1 gbit port for solving it.
View 3 Replies
View Related
Oct 27, 2008
flood in FTP and brute force
all day i receved msgs of BFD someone trying acess server, how to stop it, exemple:
Executed ban command:
/etc/apf/apf -d 221.186.164.233 {bfd.pure-ftpd}
The following are event logs from 221.186.164.233 on service pure-ftpd (all time stamps are GMT -0500):
Oct 25 13:52:37 svr1 pure-ftpd: (?@221.186.164.233) [INFO] New connection from 221.186.164.233
Oct 25 13:52:37 svr1 pure-ftpd: (?@221.186.164.233) [INFO] New connection from 221.186.164.233
Oct 25 13:52:38 svr1 pure-ftpd: (?@221.186.164.233) [WARNING] Authentication failed for user [router] ....
View 0 Replies
View Related
Aug 1, 2007
When I check on port 80 connections, I get a list of few IPs with more than 100 connections.
I need to know which website / specific file being downloaded / URL is the IP accessing to? How can I do that?
View 3 Replies
View Related
Jan 11, 2007
my server under attack with syn flood and i attach the active connection during attack
View 2 Replies
View Related
Jun 20, 2007
some of my friend website is under attack by Ddos ; FLOOD ; and other way to down his site !
how can we save him from these attacks?
else the firewall!
1- is there any program or script to prevent these attacks!
2- is there any rules or script's for Firewall to prevent it!
View 4 Replies
View Related
Jun 5, 2015
I serve large professional documents, and sometimes links to them end up on social media. No big deal, but I think people clicking on them from tose social media sites don't have a clue. They think they're being directed to small page, when in fact they are downloading megabytes of pdf -- myfile.pdf. So what I've started to do is to redirect requests from social media to an archive page, where they can see specifically what document they are trying to get, and recognize its size before they ask for it. No problem, right? I just do
View 10 Replies
View Related
May 8, 2007
I've got a VPS which is serving as the main server for a number of sites. Web Server, SSH Server, and Mail Server.
What I've got running:
Apache2, PHP5, MySQL5, Dovecot, Postfix
One of the sites is a growing forum with a MASSIVE photo album. This is the site where I notice the most slowness.
Changing the server software is not an option - Only optimization.
Quote:
Originally Posted by httpd.conf
ServerTokens OS
ServerRoot "/etc/httpd"
PidFile run/httpd.pid
Timeout 300
KeepAlive Off
MaxKeepAliveRequests 100
KeepAliveTimeout 5
<IfModule prefork.c>
StartServers 8
MinSpareServers 8
MaxSpareServers 13
ServerLimit 256
MaxClients 256
MaxRequestsPerChild 50
</IfModule>
<IfModule worker.c>
StartServers 2
MaxClients 150
MinSpareThreads 25
MaxSpareThreads 75
ThreadsPerChild 25
MaxRequestsPerChild 0
</IfModule>
Listen 80
LoadModule auth_basic_module modules/mod_auth_basic.so
LoadModule auth_digest_module modules/mod_auth_digest.so
LoadModule authn_file_module modules/mod_authn_file.so
LoadModule authn_alias_module modules/mod_authn_alias.so
LoadModule authn_anon_module modules/mod_authn_anon.so
LoadModule authn_dbm_module modules/mod_authn_dbm.so
LoadModule authn_default_module modules/mod_authn_default.so
LoadModule authz_host_module modules/mod_authz_host.so
LoadModule authz_user_module modules/mod_authz_user.so
LoadModule authz_owner_module modules/mod_authz_owner.so
LoadModule authz_groupfile_module modules/mod_authz_groupfile.so
LoadModule authz_dbm_module modules/mod_authz_dbm.so
LoadModule authz_default_module modules/mod_authz_default.so
LoadModule ldap_module modules/mod_ldap.so
LoadModule authnz_ldap_module modules/mod_authnz_ldap.so
LoadModule include_module modules/mod_include.so
LoadModule log_config_module modules/mod_log_config.so
LoadModule logio_module modules/mod_logio.so
LoadModule env_module modules/mod_env.so
LoadModule ext_filter_module modules/mod_ext_filter.so
LoadModule mime_magic_module modules/mod_mime_magic.so
LoadModule expires_module modules/mod_expires.so
LoadModule deflate_module modules/mod_deflate.so
LoadModule headers_module modules/mod_headers.so
LoadModule usertrack_module modules/mod_usertrack.so
LoadModule setenvif_module modules/mod_setenvif.so
LoadModule mime_module modules/mod_mime.so
LoadModule dav_module modules/mod_dav.so
LoadModule status_module modules/mod_status.so
LoadModule autoindex_module modules/mod_autoindex.so
LoadModule info_module modules/mod_info.so
LoadModule dav_fs_module modules/mod_dav_fs.so
LoadModule vhost_alias_module modules/mod_vhost_alias.so
LoadModule negotiation_module modules/mod_negotiation.so
LoadModule dir_module modules/mod_dir.so
LoadModule actions_module modules/mod_actions.so
LoadModule speling_module modules/mod_speling.so
LoadModule userdir_module modules/mod_userdir.so
LoadModule alias_module modules/mod_alias.so
LoadModule rewrite_module modules/mod_rewrite.so
LoadModule proxy_module modules/mod_proxy.so
LoadModule proxy_balancer_module modules/mod_proxy_balancer.so
LoadModule proxy_ftp_module modules/mod_proxy_ftp.so
LoadModule proxy_http_module modules/mod_proxy_http.so
LoadModule proxy_connect_module modules/mod_proxy_connect.so
LoadModule cache_module modules/mod_cache.so
LoadModule suexec_module modules/mod_suexec.so
LoadModule disk_cache_module modules/mod_disk_cache.so
LoadModule file_cache_module modules/mod_file_cache.so
LoadModule mem_cache_module modules/mod_mem_cache.so
LoadModule cgi_module modules/mod_cgi.so
Include conf.d/*.conf
User apache
Group apache
Quote:
Originally Posted by my.cnf
[mysqld]
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
# Default to using old password format for compatibility with mysql 3.x
# clients (those using the mysqlclient10 compatibility package).
old_passwords=1
[mysql.server]
user=mysql
basedir=/var/lib
[mysqld_safe]
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid
View 8 Replies
View Related
Mar 7, 2009
I looked a lot - can not find solution ....
I want to transfer a file from [url]to [url]or [url]Without it will pass my localcomputer (slow upload)
It can be also a script i will install like this one - this is only for images
[url](remote)
View 7 Replies
View Related
Oct 16, 2013
I have Plesk 11.5 (service provider mode) on a Windows 2008 server IIS7.Most of my sites are developed in .asp and therefore i use a custom 500-100.asp error page that check s the IP of the visitor then displays either a friendly error, or if its my IP a full error of what has happened (it also emails me the error). This allows me to debug pages easily whilst developing and to keep an eye on anyone trying SQL Injection hacks on my sites (as the error and email also have session variables and IP address).I dont have root access to the server as it is a Webfusion dedicated server.I have following the Plesk documentation -
1) Switch on custom errors for the subscription
2) Look in virtual directories and navigate to error documents
3) Find the error in question (500:100) and change it to point at either a file or URL
FILE - I had the data centre add in the 500-100.asp error page in to the virtual template so that my page is available in the list of virtual files - this didn't work but that maybe because its not a static page??
URL - when i add the path it says its incorrect, if i add a fully qualified address, it accepts it but it doesn't work.give me a specific example of the URL that can be entered relative to the root as the format in the documentation isn't accepted. The last step is to restart IIS which is also an issue as i cant seem to do this from the Plesk panel..It is as if it isn't catching the 500:100 error, and only catching the general 500 error??
View 1 Replies
View Related
Nov 21, 2006
I am currently running Google Analytics/Urchin 5 (v5.7.02), on a server, the server has started to act up, (on its last legs etc) and now I am trying to transfer the Urchin Software to a new server, where it would work effectively.
However upon installing the urchin software on the new server and running it (localhost:9999), I am presented with An Action Items Page, and these following choices
Obtain Demo License
Buy License
Activate Pre-Purchased License
I choose ‘Activate Pre-Purchased License’ pop in the Serial number and complete the registration then…
---------------------------------------------------------
Urchin Licensing Center -- Error!
An error has occured during your transaction, please use the back button and correct the problem. The specific error message is:
• Unable to generate a license. Some possible reasons:
Your serial code is currently active <<< How do I disable it and use it on another server?
---------------------------------------------------------
So all I want to do is deactivate the serial and reactivate it on another server.
Does anyone have experience with this or a similar problem or have a solution to this problem. Any help be most appriciated.
Or even a Contact Number so that i can get some one over the phone!
View 2 Replies
View Related
Nov 7, 2009
This is the scenario, domain.com are setup on server1, however server2 also has the same profile of domain.com as we use ns3 and ns4 using domain.com. This works fine with the nameserver setup on server2.
However I encounter problems as the emails from server2 won't reach server1 as there are duplicate profile on server2.
My question is how do I setup the DNS in cpanel/whm from server2 so the emails from server2 will reach server1?
Server1 (www.domain.com)
ns1.domain.com
ns2.domain.com
Server2
ns3.domain.com
ns4.domain.com
View 6 Replies
View Related
Apr 22, 2009
I just want to use a server for file sharing, it will have nginx and that's it. I'm looking at centos, or freebsd, but I been using centos forever now and I'm not sure how to use freebsd, should I just stay with centos?
Do I tell my hosting provider to just install the OS and give me ssh action and that's it? Don't install any control panels or any other stuff? I want one domain and one subdomain on it though and ftp action.
View 8 Replies
View Related
May 12, 2009
Remote Spamassassin for Multiple Smartermail Server
I want to setup Remote Spamassassin(On Linux) for Multiple Smartermail servers. I want to the setup the spamassassin on a linux box
How i can setup this with multiple smartermail servers.
View 6 Replies
View Related
