I'm running a web server with mod_evasive and want to know how can I prevent mod_evasive from blocking the googlebot crawler ip address.
Is there a script out there that can detect this crawler and make sure its ip doesnt get blocked by iptables or mod_evasive?
Upon reviewing my modsecurity log today, I found an interesting hit from google.
-------------------
Requesting IP: 66.249.65.67 is http://ws.arin.net/cgi-bin/whois.pl?...t=66.249.65.67
Date: 2006-10-03
Time: 07:10:16
Handler: mod_gzip_handler
Get: /page/index/1&show=25,07,2005?php%20echo%20$bmc_vars%5B'site_url'%5D;%20?%3E/profile.php?id=1
Mod_Security-Message: Access denied with code 406. Pattern match "echo " at THE_REQUEST
Mod_Security-Action: 406
------------
The rule that set off this 406 response was:
SecFilterSelective THE_REQUEST "echo "
What I find interesting is that I do not have any such URL structure on this website that google requested.
I am just wondering whether my idea will work for the google search engine.
Basically, I have my official VPS root (home/admin/public_html/) and this is where my main website will be hosted. However, since my VPS will be used for additional website, I will direct additional domains to it.
My second site hosted on this VPS may have a document root of (home/admin/public_html/advertising/). And my domain will then be setup to have that as its document root.
However, when google searches for my second site (e.g. advertising.com) will it go 'below' the domain root? For example, will it also creep the files under /public_html/ for this domain, even though the domains root is /public_html/advertising/?
edit: Or do people host multiple sites differently? Is this an appropriate method?
I recently found out that the Google robot spidered my site and used up 5.74 GB in bandwidth! The hits were 4498 and I even have it scheduled to return every 2 weeks.
Now my site is down for the rest of the month unless I up my bandwidth. This is the second time this has happened to me in the past year. What is going on?
Googlebot has been absolutely ripping through my bandwidth. This has been on going for many months now, but each month the damage has gotten worse and worse. I have posted about this problem here but with no luck:
http://www.webmastertrader.com/showthread.php?t=389
To demonstrate the problem, I have decided to show you all my server stats for one of my websites.
I have no choice but to ban googlebot from my site and follow this action with a letter to google!
1) Aug: http://www.webmastertrader.com/attac...0&d=1159816704
2) Sep: http://www.webmastertrader.com/attac...1&d=1159816714
3) Oct: http://www.webmastertrader.com/attac...2&d=1159816723
We're only a day and a half into this month and Google's robots have already consumed over 1gig of bandwidth!
Yes, it’s quite true. DreamHost representatives are asking their clients to block GoogleBot trough the .htaccess file, because their websites were “hammered by GoogleBot”.
Dreamhost representatives are also guiding their clients to make all their websites “unsearchable and uncrawlable by search engine robots”, because they cause “high memory usage and load on the server”.
PS: This is NOT a rumor. Multiple people are already complaining about Dreamhost asking clients to block search engine spiders. A post in QuickOnlineTips confirms this too.
Initial news bit via Zoso.
If your server is blocking googlebot from finding your robots.txt file, how do you configure your firewall to unblock it?
I've searched through Google and I've seen may people just say your firewall is blocking it, but none mention how to really stop it from doing that. Like does Google have an IP it uses, and if so, what is the IP you should whitelist for your server?
As I keep getting that message: Network unreachable: robots.txt unreachable
and I'm sure it's due to a firewall issue, just have no idea how to fix that.
One of my customers uses Webmasters Tools from google , looking at what pages he have indexed by Googlebot, found that 180 pages are giving a "DNS lookup timeout" error, tried searching for help on google and the only thing that i found is " We received a timeout on DNS lookup."
DNS are ok, same as the zone file, everything is responding OK, I dont know what can be the issue.. ? any ideas ?
Im using Ensim Pro and RHES 4.
I have recently upgraded to a virtual private server. What is a safe speed for googlebot to crawl my website.
Can you please give me results in the format of:
xxx requests per second
xxx seconds between requests.
Let's say when i first joined the datacenter, i'm given some 16 IPs from 123.123.123.*.
Now that i need more IPs, and they have to give me IPs from 123.123.124.*, am i right to say i need a new switch?
Now is it possible that i link up the new switch together, so that both switch can pick up any available IPs from either range?
am using APF firewall and am getting ddos from these range ips
4.68.25.*
8.0.4.*
8.0.5.*
how could i delete all the range from these ip's?
I set up a forum for a small group of users, so I don't really wish to see spiders or bots on it, so I've put a robots.txt file there to prevent all of them from accessing the forum pages.
I know not all bots follow the robots.txt rule, and these days a really annoying bot called MUNAXNET or Munax AB with IP range 82.99.30.0 - 82.99.30.127 is causing the forum to have extra and unexpected loads.
I've tried to block this IP range with .htaccess and uploaded it to the root of the site a few days ago, here is the content:
<Limit GET HEAD POST>
order allow,deny
deny from 82.99.30.0-82.99.30.127
allow from all
</LIMIT>
However strangely it seems that all of these are not working for this bot, today I saw my forum had 80 users online and that army still keeps coming and browsing all pages of my forums...
I tested the .htaccess with blocking myself, and it actually worked for me, dunno why it's not working for that bot..
we want a dedicated server with a full range of ip with our own company name(or with my name) and abuse mail
+ kvm and apc access
we need a good support too , for example if we want to check our hard disk the provider check it fast
I need to ban IP range and I inserted say ip 12.44.0.0 in the deny_hosts rules, this should ban range from 12.44 but strange is people from that range still be able to access my site, any idea what went wrong?
View 3 Replies View RelatedServer: OS RHEL 4
Web Server: Apache1.3.37
Hi. I want to block a range of IPs. Currently, I use the following command as an effective weapon against the IPs of people I find in the log trying to do bad things.
/sbin/route add -host x.x.x.x reject
Works like a charm. I then put the offending IP in the file /etc/rc.d/rc.local so that it will reload the bad ips when the server reboots next.
I also use the CSF/LFD firewall, and it successfully blocks single, offensive IPs also.
What I need, though, is the ability to block an entire range of IPs. For example, i have a very persistent hacker trying to access from a certain range like so....
255.155.x.x.
The last two numbers are always changing but the first two remain the same.
How do I block this "range" of IPs from accessing my server?
Note, I know how to block a range of IPs in a .htaccess file for a certain account, I put this in the .htaccess file...
deny from 205.196.
But when I try this with /sbin/route, it will not accept the ip. My firewall will also not accept a range of IPs.
Is there any way to block all and just allow certain IP using APF or iptables?
I want to block all the traffic to the server and just alow IP range.
I have a client who needs to block IP range on a windows server. However, he is using Cloud hosting from Rackspace. I guess they are not being corporative in doing so. Anyway to do this without root? Perhaps from the control panel?
I am getting my quote back Tuesday but need a little bargaining power with these guys...
Oakland, Ca datacenter
40mbps, 20A, 42U rack.
What should I be looking at price range here, how much per mbps?
Only info I've seen is from 2003 where people were saying $200/mbps. Obviously prices have come WAY down. I've seen people on here reselling internap bandwidth for $12/mbps, but they might have bought a huge commit.
i have 2 range ip
range 1 is : 111.111.111-119
range2 is : 222.222.222-229
how may i add both range?
i can add for one range,but about 2 rage i can`t
from 36 hours we have really critical issue:
we have 3 server with Liquidweb that have problem to "resolve" a server locate in Italy
Email sent from server with LW to Italy server is not sent and stay in queue.
We try to force delivery and error is:
Message 1Je39R-0000wk-3N is not frozen LOG: MAIN
cwd=/usr/local/cpanel/whostmgr/docroot 4 args: /usr/sbin/exim -v -M
1Je39R-0000wk-3N delivering 1Je39R-0000wk-3N Connecting to italyserver.com
[**.**.**.**]:25 ... failed: Connection timed out (timeout=5m) LOG: MAIN
italyserver.com [**.**.**.**]:25 Connection timed out LOG: MAIN
== email@italyserver.com R=lookuphost T=remote_smtp defer (110): Connection
== timed out
--
We have check, there aren't problem with port and firewall
All seesm ok, for all server
We have try to restart exim, reboot, ..
Emails don't start from 3 server LW to 1 in Italy
We have try to ping from server LW the Italian server and the problem is the same, found IP but 100% packet lost..
If we ping from 3 server LW a IP of server that is "over" the server IT is all ok
The problem seems only with single IP
So, we have try to send from server in Italy to 1 of 3 server under LW.. same problem
------
Message 1JeSBO-0003CD-HO is not frozen LOG: MAIN
cwd=/usr/local/cpanel/whostmgr/docroot 4 args: /usr/sbin/exim -v -M
1JeSBO-0003CD-HO delivering 1JeSBO-0003CD-HO Connecting to
mail.serverlw.it [**.**.***.**]:25 ... failed: Connection timed out
(timeout=5m) LOG: MAIN mail.serverlw.it [**.**.***.**] Connection
timed out LOG: MAIN
== email@serverlw.it R=lookuphost T=remote_smtp defer (110):
== Connection timed out
------
LW support not have suggest solutions and also Italian support not have idea of problem solution
we want a dedicated server with a full range of ip with our own company name(or my name) and our own abuse e-mail address
+ kvm and apc access
we need a good support too , for example if we want to check our hard disk the provider check it fast
any body know anywhere to provide this services with a good price?
I would like to use different Class C IP ranges across my domains for better cross linking. Of course, all domains are closely related to a particular niche.
I am currently on imountain.com, they have been extremely helpful and good. But wondering if same host can provide me different class C IP range?
want to confirm this before i order a new server from them.
or should i just go for another hosting provider? wondering anyone as good as imountain?
Is there any way to get the IP Range of providers like SAVVIS, Global Crossing, or Level3?
View 1 Replies View RelatedSome of our client emails are being blocked and it appears that the whole GNAX range of more than 103680 IP addresses was blocked.
Quote:
As you should know now: It is not you, it is your complete provider which got UCEPROTECT-Level 3 listed.
Your IP xxx.xxx.xxx.xxx was NOT part of a spamrun, but you are the one that has freely chosen your provider.
By tolerating or ignoring that your provider doesn't care about spammers you are indirectly also supporting the global spam with your money.
Seen from this point of view, you really shouldn't wonder about the consequences.
What I want to know is how often do it happens that a datacenter have all it's IP addresses being blocked?
I contacted Gnax about this but their response was merely that they will take steps and the whole range will eventually be delisted. (Obviously the immediate removal fee of $200 something is too much for them)
Ironically merely a few hours AFTER I contacted them I got a "warning" about spam send from one of our clients.
The date of the email was a few weeks AGO and We already long ago warned our LEGIMATE client that such emails are not allowed and the account will be terminated if it happens again. (In other words WE immediately acted against potential spam while Gnax not and then have the nerve to warn US after WE informed them of the blacklisting)
On the one hand I understand that it can be difficult for a very large provider and uceprotect.net seems a bit harse to block ALL IP addresses due to 0.238 % spam ip addresses.
This came as a surprise today, I setup a server-based RSS reader and could not get WHT's forum RSS feeds. A little digging revealed it was the default APF installation that was blocking the 174.0.0.0/8 range, which includes WHT and a chunk of Softlayer's ip range.
The quick fix is easy, just remove that range from the /etc/apf/internals/reserved.networks file and restart, in the latest apf version, I don't know how many apf versions back this block goes.
The APF folks do a fantastic job in keeping APF up to date, but this seems to be recent update to this particular ip range that hasn't made it into APF yet.
I want to stop users from country X from accessing my website, I know I can ban people's IPs but I dont know if it is possible to ban certain geographical area and if so, I haven't got a clue about how to do it.
View 8 Replies View RelatedDoes anyone know how to range ban IPs using IPsec.
I can enter IPs manually but unable to ban an entire RANGE of ips
i.e. For example 172.10.10.10 - 172.1.1.999
Anyone know?
I've read that all ethernet switches in a MST Region need the same Name, Revision number, and list of member vlans for each Instance. So what happens when you need to change the range of VLANs in a MSTI ? Let's say that you need to add a range of vlans to an instance that spans 20 switches? How would you do that?
View 1 Replies View RelatedI've been trying to install a new WHM/cpanel in a clean machine with RAID 1 and in the middle in the install (its a long process as u may know well) throws me some strange messages like this
Apache restart failed. Unable to load pid from pid file and no httpd process found in process list.
If apache restart reported success but it failed soon after, it may be caused by oddities with mod_ssl.
You should run /scripts/ssl_crt_status as part of your troubleshooting process. Pass it --help for more details.
Also be sure to examine apache's variouse log files.
and this:
Dependencies Resolved
=============================================================================
Package Arch Version Repository Size
=============================================================================
Installing:
compat-db i386 4.2.52-5.1 base 1.7 M
db4-devel i386 4.3.29-9.fc6 base 2.0 M
openssl-devel i386 0.9.8b-8.3.el5_0.2 base 1.8 M
pam-devel i386 0.99.6.2-3.26.el5 base 188 k
rpm-build i386 4.4.2-47.el5 base 551 k
zlib-devel i386 1.2.3-3 base 101 k
Installing for dependencies:
e2fsprogs-devel i386 1.39-10.el5_1.1 updates 563 k
elfutils i386 0.125-3.el5 base 163 k
elfutils-libs i386 0.125-3.el5 base 105 k
keyutils-libs-devel i386 1.2-1.el5 base 27 k
krb5-devel i386 1.6.1-17.el5_1.1 updates 1.9 M
libselinux-devel i386 1.33.4-4.el5 base 131 k
libsepol-devel i386 1.15.2-1.el5 base 189 k
Transaction Summary
=============================================================================
Install 13 Package(s)
Update 0 Package(s)
Remove 0 Package(s)
Total download size: 9.3 M
Downloading Packages:
http://mirror.steadfast.net/centos/5..._0.2.i386.rpm: [Errno -1] Package does not match intended download
Trying other mirror.
http://mirrors.rit.edu/centos/5.1/os..._0.2.i386.rpm: [Errno 9] Requested Range Not Satisfiable
Trying other mirror.
http://yum.singlehop.com/CentOS/5.1/..._0.2.i386.rpm: [Errno 9] Requested Range Not Satisfiable
Trying other mirror.
http://centos-distro.cavecreek.net/c..._0.2.i386.rpm: [Errno 9] Requested Range Not Satisfiable
Trying other mirror.
http://ftp.osuosl.org/pub/centos/5.1..._0.2.i386.rpm: [Errno 9] Requested Range Not Satisfiable
Trying other mirror.
http://mirror.trouble-free.net/cento..._0.2.i386.rpm: [Errno 9] Requested Range Not Satisfiable
Trying other mirror.
http://mirrors.unbornmedia.com/cento..._0.2.i386.rpm: [Errno 9] Requested Range Not Satisfiable
Trying other mirror.
http://pubmirrors.reflected.net/cent..._0.2.i386.rpm: [Errno 9] Requested Range Not Satisfiable
Trying other mirror.
http://mirrors.bluehost.com/centos/5..._0.2.i386.rpm: [Errno 9] Requested Range Not Satisfiable
Trying other mirror.
http://ftp.usf.edu/pub/centos/5.1/os..._0.2.i386.rpm: [Errno 9] Requested Range Not Satisfiable
Trying other mirror.
Error Downloading Packages:
openssl-devel - 0.9.8b-8.3.el5_0.2.i386: failure: CentOS/openssl-devel-0.9.8b-8.3.el5_0.2.i386.rpm from base: [Errno 256] No more mirrors to try.
this one is because it cant find the mirrors!
Well there are some other errors, but those worry me the most.
i've done the install as appears in the cpanel website, dunno why i get these errors. Any suggestion is appreciated.
btw, its a centos 5.1 install with no GUI and no extras installed.
Is there some way to trigger a script or send an email if someone from a specified IP or IP range (e.g. 125.125.125.%) accesses an account, or any URL on the server? Perhaps by placing something within .htaccess or httpd.conf or something of that nature.
For instance, an email "$IP is visiting the URI $REQUEST_URI" would be sent, or a PHP script that sends the email would be triggered (though in that case, I'd need some way to tell the script the IP and URI).
And in a related matter, is there any way (perhaps some existing software) to filter the access logs of an account to find all requests by a certain IP address?