Stop Your Server From Blocking Googlebot
May 21, 2008
If your server is blocking googlebot from finding your robots.txt file, how do you configure your firewall to unblock it?
I've searched through Google and I've seen may people just say your firewall is blocking it, but none mention how to really stop it from doing that. Like does Google have an IP it uses, and if so, what is the IP you should whitelist for your server?
As I keep getting that message: Network unreachable: robots.txt unreachable
and I'm sure it's due to a firewall issue, just have no idea how to fix that.
View 5 Replies
ADVERTISEMENT
Oct 3, 2006
Upon reviewing my modsecurity log today, I found an interesting hit from google.
-------------------
Requesting IP: 66.249.65.67 is http://ws.arin.net/cgi-bin/whois.pl?...t=66.249.65.67
Date: 2006-10-03
Time: 07:10:16
Handler: mod_gzip_handler
Get: /page/index/1&show=25,07,2005?php%20echo%20$bmc_vars%5B'site_url'%5D;%20?%3E/profile.php?id=1
Mod_Security-Message: Access denied with code 406. Pattern match "echo " at THE_REQUEST
Mod_Security-Action: 406
------------
The rule that set off this 406 response was:
SecFilterSelective THE_REQUEST "echo "
What I find interesting is that I do not have any such URL structure on this website that google requested.
View 2 Replies
View Related
Mar 22, 2007
I'm running a web server with mod_evasive and want to know how can I prevent mod_evasive from blocking the googlebot crawler ip address.
Is there a script out there that can detect this crawler and make sure its ip doesnt get blocked by iptables or mod_evasive?
View 9 Replies
View Related
Aug 2, 2008
I am just wondering whether my idea will work for the google search engine.
Basically, I have my official VPS root (home/admin/public_html/) and this is where my main website will be hosted. However, since my VPS will be used for additional website, I will direct additional domains to it.
My second site hosted on this VPS may have a document root of (home/admin/public_html/advertising/). And my domain will then be setup to have that as its document root.
However, when google searches for my second site (e.g. advertising.com) will it go 'below' the domain root? For example, will it also creep the files under /public_html/ for this domain, even though the domains root is /public_html/advertising/?
edit: Or do people host multiple sites differently? Is this an appropriate method?
View 4 Replies
View Related
Sep 24, 2005
I recently found out that the Google robot spidered my site and used up 5.74 GB in bandwidth! The hits were 4498 and I even have it scheduled to return every 2 weeks.
Now my site is down for the rest of the month unless I up my bandwidth. This is the second time this has happened to me in the past year. What is going on?
View 14 Replies
View Related
Oct 2, 2006
Googlebot has been absolutely ripping through my bandwidth. This has been on going for many months now, but each month the damage has gotten worse and worse. I have posted about this problem here but with no luck:
http://www.webmastertrader.com/showthread.php?t=389
To demonstrate the problem, I have decided to show you all my server stats for one of my websites.
I have no choice but to ban googlebot from my site and follow this action with a letter to google!
1) Aug: http://www.webmastertrader.com/attac...0&d=1159816704
2) Sep: http://www.webmastertrader.com/attac...1&d=1159816714
3) Oct: http://www.webmastertrader.com/attac...2&d=1159816723
We're only a day and a half into this month and Google's robots have already consumed over 1gig of bandwidth!
View 14 Replies
View Related
Jun 3, 2007
Yes, it’s quite true. DreamHost representatives are asking their clients to block GoogleBot trough the .htaccess file, because their websites were “hammered by GoogleBot”.
Dreamhost representatives are also guiding their clients to make all their websites “unsearchable and uncrawlable by search engine robots”, because they cause “high memory usage and load on the server”.
PS: This is NOT a rumor. Multiple people are already complaining about Dreamhost asking clients to block search engine spiders. A post in QuickOnlineTips confirms this too.
Initial news bit via Zoso.
View 12 Replies
View Related
Jul 4, 2007
One of my customers uses Webmasters Tools from google , looking at what pages he have indexed by Googlebot, found that 180 pages are giving a "DNS lookup timeout" error, tried searching for help on google and the only thing that i found is " We received a timeout on DNS lookup."
DNS are ok, same as the zone file, everything is responding OK, I dont know what can be the issue.. ? any ideas ?
Im using Ensim Pro and RHES 4.
View 1 Replies
View Related
Mar 9, 2008
I recently signed up a new client to my dedicated server - The minute they switched over to my server, it seems that all hell broke loose. (I'm going to refer to them as "Company A")
Company A called me up and said that one of their employees was getting a huge amount of SPAM and that after a day or two, they were having issues with their E-mail.
I looked at my logs and it showed something unusual-
LOGIN FAILED, user=myclientuser@companya.com, ip=[::ffff:XX.XXX.170.47]: 110 Time(s)
When I explained this to Company A, they ran some virus checks on their computers and 3 out of 5 computers had viruses on them.
They claim to have fixed the viruses but now, they cannot send e-mails to specific clients.
I checked their I.P. against blacklists and they are using Comcast cable internet at their location and I cleared their only blacklisting (spamhaus.org).
I'm still getting calls that Company A cannot e-mail a few of their clients and just to make sure it's not JUST them, I tried to send a test e-mail to the same clients as Company A.
The e-mails from me were rejected due to time-out.
HERE IS MY QUESTION:
Is this an issue on MY end that must be taken care of *OR* is it due to the fact that they had viruses on their computers and now they are blocked because the virus tried to attack everybody in THEIR e-mail address book?
None of my other clients are complaining of e-mail issues or that e-mails are getting kicked back. Just Company A.
View 4 Replies
View Related
May 4, 2008
I've been carrying some weird hours lately, so I'm able to see activity on my sites that I normally don't see.
Certain IPs are trying to diddle their things into my server and need the boot. It's not consistent (ie not happening every 5, 10 mins)... it's periodically throughout the months. I'll see an IP I blocked 2 months ago just randomly show up at 4:30am and try accessing the same files it was probing during it's last visit. Assuming this is just some sort of bot, can I block it permanently?
I know APF has a collection system that purges an IP list to keep it from bloating, and I had PSM do some hardening so I'm not entirely sure about the workings of APF firewall. So far my IP blocks are blank (which is a good sign!), but I'd like to add some nuisances to it, to keep their crap from appearing in my error logs anymore as "Denied by Server Configuration"
My question is: Can I block people at server level permanently? I do not want their IP being taken out with the purge list that comes by every so often.
View 4 Replies
View Related
Nov 20, 2008
My server is getting a lot of spam robots registering as fake users into my forums (altho most emails are invalid
View 2 Replies
View Related
Feb 26, 2007
I was wondering if it is possible to block traffic to and from a server with iptables.
Like for example a user transferring files with his ftp client to another server x.x.x.x (FXP)....
been trying with these rules here:
iptables -IINPUT -s x.x.x.x -j DROP
iptables -I OUTPUT -s x.x.x.x -j DROP
iptables -I FORWARD -s x.x.x.x -j DROP
But still the user can transfer to the server destination...
View 0 Replies
View Related
Jul 10, 2007
I have a wierd problem on my server. I'm updating an order with my php shopping cart script and notice some of the orders hang on updating. After research I discovered the orders that wont update are orders that have domain names in the textarea field. More specifically. www.yourdomain.com will just hang if I try to update them.
After further research I discovered the if I put an backslash it'll update no problem. IE: www.yourdomain.com
This only happens with domain names inside a textarea. I'm not sure if it's a php config setting, apache setting, my firewall, mod_evasive, mod_security. I've looked at all of them and can't seem to find what is blocking the script from updating just because of www.domainname.com in a textarea.
View 7 Replies
View Related
Dec 15, 2008
I have recently upgraded to a virtual private server. What is a safe speed for googlebot to crawl my website.
Can you please give me results in the format of:
xxx requests per second
xxx seconds between requests.
View 2 Replies
View Related
May 27, 2009
Is there any way to block or monitor and find scripts such as rapidleech and other torrent upload scripts on a cPanel server?
View 2 Replies
View Related
Feb 1, 2008
how can i remove list of blocking IPs on iptable on my server its over 100 IPs how can i remove it?
View 2 Replies
View Related
Aug 14, 2007
This topic has probably been covered many times. I am trying to block spam bots from direclty accessing scripts on my site. Measures I have taken thus far, also block my site pages that want are designed to use the scripts. This includes Hot Link Protection and a couple of directives in the .htaccess file.
how I can block direct access to my scripts?
View 0 Replies
View Related
Apr 13, 2015
Every time my server tries to send mail to Gmails/yahoos - they reject it with the following message:
Code:
Apr 13 11:35:49 m1370 plesk sendmail[5009]: handlers_stderr: PASS
Apr 13 11:35:49 m1370 plesk sendmail[5009]: PASS during call 'limit-out' handler
Apr 13 11:35:49 m1370 plesk sendmail[5009]: handlers_stderr: SKIP
Apr 13 11:35:49 m1370 plesk sendmail[5009]: SKIP during call 'check-quota' handl er
Apr 13 11:35:49 m1370 postfix/pickup[1759]: C90FA61668: uid=48 from=<apache>
[Code] ....
I tried Qmail, same issue
Code:
Hi. This is the qmail-send program at m1370.contabo.
I'm afraid I wasn't able to deliver your message to the following addresses.
This is a permanent error; I've given up. It didn't work out.
<emailhere@gmail.com>:
2a00:1450:4013:0c00:0000:0000:0000:001b failed after I sent the message.
[Code] .....
Centos 6.6 + Plesk 12 Web Pro Edition
My server certainly did not spam Gmail servers, (my old server works fine sending email t gmail) I fear this has something to do with my "Postfix" configuration.
View 1 Replies
View Related
Jul 14, 2008
How i can stop Spammer from my server?
my control panel is CPANEL !
what software i must install?
View 4 Replies
View Related
Jan 10, 2007
as many of you know, FreeBSD is a stable system... I have many other FreeBSD servers (with the same kernel as this one) that doesn't have problems but this server keeps rebooting once or twice a day (EVERY DAY)
it's just a reboot... something very very similar to someone pushing the reset button
1) messages, security, auth or dmesg has no entries just before the reset, so the kernel is not getting aware the server is rebooting
2) the server comes back after around 10 minutes (reboot time + fsck)
this is happening for long time, so I compiled a new kernel... and the problem didn't stop
I request the datacenter techs to replace hardwares and they told me everything was replaced: motherboard, CPU, memories... and yesterday also the power suply
so I have no other idea on what to do
in fact I have one... setting a nobreak in this server power suply for 2 or 3 days to see if the problem stops, but the datacenter didn't like this idea
View 10 Replies
View Related
May 5, 2009
I've recently had problems where customers will upload PHP scripts that seem to use alot of CPU. I've got PRM installed but when a PHP script uses a lot of CPU, it doesn't seem to kill the processes or do anything to stop it crashing the server. I've checked the logs of PRM and it does kill some processes that use a lot of CPU/RAM though...
The ideal solution would be for PRM or something else to stop people being able to access the script causing excessive CPU/RAM usage. Even suspending for the reason of using excessive CPU/RAM would be sufficient.
For those interested the OS is CentOS 5.3 with cPanel 11, Apache and the latest PHP 5. Average load is always between 0.50 - 1.90.
View 4 Replies
View Related
Mar 30, 2009
in the last 2 weeks has increased the spam mail to external users using our mail accounts.
So a user receives spam believing that it is sent from our sites.
I think the best method is to create a txt file in dns but I have many doubts about how to proceed.
Looking at one of the e-mail back to our mail server I see that emails are sent via outlook.
This is an example of the emails: ...
View 7 Replies
View Related
Jul 7, 2009
My server was hit with flood recently, to the point where I was unable to log in via SSH. Running 'netstat' command showed I was getting flooded with thousands of http requests from China/Saudi Arabia/Korea. I installed APF firewall and added those countries to deny list.
Next day I was hit from Russia and Romania and some others. By reading some posts on this site, on top of APF, I have also installed Dos Deflate. It was working for couple of hours, but then it stopped working. I could not even log in via SSH. My provider told me that APF was using all of the "conntrack" connections. I have increased conntrack connections to 130,000 (I have 4 Gigs of RAM on my server). Is that possible? (I have about 300 IP ranges in my APF deny list).
Next day, I was got hit by different attack: there was 11 Mbps of malicious traffic on average sent to my server. My provider put me behind firewall to mitigate against that kind of attack.
Currently, I am both behind the hardware firewall and I have APF and Dos Deflate running. However my server is not accessible.
When I request, I can log in for couple of minutes, but then I get kicked out.
View 9 Replies
View Related
Nov 18, 2008
My server had been crashing for while with Blue Screen of Death (BSOD) and bug check error code as Stop 0x00000050 PAGE_FAULT_IN_NONPAGED_AREA.
It would literally stop by business till I reboot it again.
So I tried pull up all information I could get and fix this.
Here is what I found-
Possible causes:
A faulty driver recently installed
Faulty RAM
Antivirus
Corrupted NTFS file system
I checked the system logs and found errors related to NTFS. Well, my disk needed a chkdsk /r /f to fix this.
Ran it at the command prompt and since it required a reboot to fix on the system drive (C:), had to reboot. Came back successfully.
It has been 14 days and it has not recurred.
View 2 Replies
View Related
Nov 18, 2008
My server had been crashing for while with Blue Screen of Death (BSOD) and bug check error code as Stop 0x00000050 PAGE_FAULT_IN_NONPAGED_AREA.It would literally stop by business till I reboot it again.So I tried pull up all information I could get and fix this.
Here is what I found-Possible causes:A faulty driver recently installed
Faulty RAM
Antivirus
Corrupted NTFS file system
I checked the system logs and found errors related to NTFS. Well, my disk needed a chkdsk /r /f to fix this.Ran it at the command prompt and since it required a reboot to fix on the system drive (C, had to reboot. Came back successfully.It has been 14 days and it has not recurred.
View 3 Replies
View Related
Dec 6, 2008
i am getting hundreds of theses in my mail log each day, trying different names etc and want to put a stop to them and auto ban the ips.
I have APF
View 2 Replies
View Related
Jan 10, 2007
Today my system which is hosting the site bepenfriends got compramised(win 2k3) and now LT tech guys are working on it to reload the system with a data save. I was not having a hardware firewall which caused this problem. But i had windows firewall, windows malinious software removal tool (defender i haven't installed). I have updated all patches of win2k3 whch was released till today.
Now after restore it will be great work to bring my website back with all those rewritten urls and the softwares and its licenses.
Now please help me out in below stuff.
How to stop further attack and further compramisation of server.
View 9 Replies
View Related
Nov 19, 2007
I'm having a very odd problem with one of my Linux (CentOS) cpanel server, all the server's services (http, ssh, mail, dns, etc) stop responding but the server still responds to ping.
I can't find anything wrong at all on the log files either, and the technicians that manually restart the server have told me that there is no indication of a problem on the screen.
I suspected a hardware issue and had the data center techs run a hardware test on the server but everything cleared ok.
This issue started a couple of weeks ago, no major upgrade or install took place when it started happening. From what i can see the halts are completely random, some times it goes for days without it happening and some times it happens just hours after the reboots.
View 14 Replies
View Related
