Googlebot Hammering My Website!
Oct 2, 2006
Googlebot has been absolutely ripping through my bandwidth. This has been on going for many months now, but each month the damage has gotten worse and worse. I have posted about this problem here but with no luck:
http://www.webmastertrader.com/showthread.php?t=389
To demonstrate the problem, I have decided to show you all my server stats for one of my websites.
I have no choice but to ban googlebot from my site and follow this action with a letter to google!
1) Aug: http://www.webmastertrader.com/attac...0&d=1159816704
2) Sep: http://www.webmastertrader.com/attac...1&d=1159816714
3) Oct: http://www.webmastertrader.com/attac...2&d=1159816723
We're only a day and a half into this month and Google's robots have already consumed over 1gig of bandwidth!
View 14 Replies
ADVERTISEMENT
Sep 24, 2005
I recently found out that the Google robot spidered my site and used up 5.74 GB in bandwidth! The hits were 4498 and I even have it scheduled to return every 2 weeks.
Now my site is down for the rest of the month unless I up my bandwidth. This is the second time this has happened to me in the past year. What is going on?
View 14 Replies
View Related
Dec 12, 2008
So I checked on the Google Webmaster tools, it has a nice graph now saying that it's hit my site 30 times/day for the last month or so. HOWEVER, when I look at my logs (I log googlebot) I have over 17000 page views in the last three days. Already I am using the Agent to remove images and some of the more expensive queries which are too old to be in the cache (Googlebot looks at pages on my site which are very old).
What other things should I be doing...? I appreciate that Google is indexing my site, but this is more of a scrape, than index. Just for this month alone, looking at my stats, I see the following:
144360 10.03% 143949 12.31% 1724938 17.72% 2 0.01% crawl-66-249-66-1.googlebot.com
2 visits, generating 144360 page views!!!!
View 7 Replies
View Related
Oct 3, 2006
Upon reviewing my modsecurity log today, I found an interesting hit from google.
-------------------
Requesting IP: 66.249.65.67 is http://ws.arin.net/cgi-bin/whois.pl?...t=66.249.65.67
Date: 2006-10-03
Time: 07:10:16
Handler: mod_gzip_handler
Get: /page/index/1&show=25,07,2005?php%20echo%20$bmc_vars%5B'site_url'%5D;%20?%3E/profile.php?id=1
Mod_Security-Message: Access denied with code 406. Pattern match "echo " at THE_REQUEST
Mod_Security-Action: 406
------------
The rule that set off this 406 response was:
SecFilterSelective THE_REQUEST "echo "
What I find interesting is that I do not have any such URL structure on this website that google requested.
View 2 Replies
View Related
Mar 22, 2007
I'm running a web server with mod_evasive and want to know how can I prevent mod_evasive from blocking the googlebot crawler ip address.
Is there a script out there that can detect this crawler and make sure its ip doesnt get blocked by iptables or mod_evasive?
View 9 Replies
View Related
Aug 2, 2008
I am just wondering whether my idea will work for the google search engine.
Basically, I have my official VPS root (home/admin/public_html/) and this is where my main website will be hosted. However, since my VPS will be used for additional website, I will direct additional domains to it.
My second site hosted on this VPS may have a document root of (home/admin/public_html/advertising/). And my domain will then be setup to have that as its document root.
However, when google searches for my second site (e.g. advertising.com) will it go 'below' the domain root? For example, will it also creep the files under /public_html/ for this domain, even though the domains root is /public_html/advertising/?
edit: Or do people host multiple sites differently? Is this an appropriate method?
View 4 Replies
View Related
Jun 3, 2007
Yes, it’s quite true. DreamHost representatives are asking their clients to block GoogleBot trough the .htaccess file, because their websites were “hammered by GoogleBot”.
Dreamhost representatives are also guiding their clients to make all their websites “unsearchable and uncrawlable by search engine robots”, because they cause “high memory usage and load on the server”.
PS: This is NOT a rumor. Multiple people are already complaining about Dreamhost asking clients to block search engine spiders. A post in QuickOnlineTips confirms this too.
Initial news bit via Zoso.
View 12 Replies
View Related
May 21, 2008
If your server is blocking googlebot from finding your robots.txt file, how do you configure your firewall to unblock it?
I've searched through Google and I've seen may people just say your firewall is blocking it, but none mention how to really stop it from doing that. Like does Google have an IP it uses, and if so, what is the IP you should whitelist for your server?
As I keep getting that message: Network unreachable: robots.txt unreachable
and I'm sure it's due to a firewall issue, just have no idea how to fix that.
View 5 Replies
View Related
Jul 4, 2007
One of my customers uses Webmasters Tools from google , looking at what pages he have indexed by Googlebot, found that 180 pages are giving a "DNS lookup timeout" error, tried searching for help on google and the only thing that i found is " We received a timeout on DNS lookup."
DNS are ok, same as the zone file, everything is responding OK, I dont know what can be the issue.. ? any ideas ?
Im using Ensim Pro and RHES 4.
View 1 Replies
View Related
Dec 15, 2008
I have recently upgraded to a virtual private server. What is a safe speed for googlebot to crawl my website.
Can you please give me results in the format of:
xxx requests per second
xxx seconds between requests.
View 2 Replies
View Related
May 3, 2008
If I type google.com in my address bar, it forwards me to www.google.com. This is not happening for my website right now. I think its a good idea to do this, since then search engines will have only 1 main URL for the website to index.
My question is:
How do I implement this? I think this may involve mucking with CNAME settings...
View 2 Replies
View Related
May 15, 2009
I want my users to be redirected directly to my forum
so when they type in www.mywebsite.com it will redirect instantly to www.mywebsite.com/forums
I know this can be done on Cpanel... any other ways?
View 7 Replies
View Related
Nov 2, 2009
I m tired with Shared Hosting so i bought one vps but i haven't any knowledge how to run site through vps and how to change nameserver blah blah
I Have Window VPS
Can any one help me step by step for setting up site through window vps ..
View 10 Replies
View Related
Apr 30, 2008
I use Ian Lloyd's book and that's where I found out about this forum. Looks like a great forum.
I downloaded Fliezilla FTP and I'm trying to transfers files from my computer onto an angelfire web site.
Filezilla asks for a server address and I put in the URL address that I registered with angelfire. It then asks me for an administration password, and I put in my password to the angelfire site. I keep getting: Error: Connection to server lost...
Does anyone know what I'm doing wrong here? I would like to use Filezilla to upload my files (web pages) to the angelfire site.
View 20 Replies
View Related
Jan 16, 2007
I hosted website and sometime it is going down. Same server some of my websites working fine. Please just know why going down my website sometimes..
View 7 Replies
View Related
Oct 23, 2008
I have a website which is currently hosted with streamline.net on their shared msql 11 server.
We have had several issues with them over the last few weeks where someone is using most of the server and slowing everyone elses sites down so much so they crash.
This week and weekend are my busiest time of the year (I sell fancy dress) and my site it totally unuseable.
We have phoned them and they have done nothing except ask us for a log which we have provided for short periods of time.
The down time has now got so bad that I have had only 2 sales today. I estimate I am losing approx 400 per day at the moment due to this problem.
Is there anything that I can do urgently to prevent my business from being killed by someone else.
View 11 Replies
View Related
Jul 27, 2007
So I'm interviewing with a company and when I typed in the URL to their website, I was met with a nasty surprise: a "hacked by so and so" message! However, after looking closer, I see that I had accidentally appended a period (".") to the end of the domain name, for example: http://www.example.com./
When I removed the period, the site appeared as normal. I don't know anything about the server other than it's IIS. Is there anything I can suggest to them when I go in to interview? I'd like to point this out to them; it may even help my chances at landing the job! (It's not related to networking, though.)
View 0 Replies
View Related
Jan 6, 2007
Does anybody know how I can mirror a websute using PHP?
I have 2 dif domains on different servers different OS, and I need PHP to collect the page from the other server from another domain.
View 1 Replies
View Related
Jan 10, 2009
Is it possible to buy a dedicated server off eg Dell and host your own website on it from home ie with a www prefix to the url- Do ye know any good tutorial on it. Would 20Mbit bandwidth be enough bandwidth for a fairly busy php ,mysql site? Completely new to this.
View 4 Replies
View Related
Jun 29, 2009
This is probably a pretty complicated answer so please forgive me as I'm a newbie to making my own ecommerce website.
What steps are needed to protect/prevent one's site from being hacked? I have domain privacy (on WhoIs) but I feel this isn't enough.
View 4 Replies
View Related
Jun 27, 2009
I have a domain name websiteexample.com and its hosted with somebody, I have built a new website under a new domain .co.uk on a new server and need to redirect my .com to the .co.uk, how the best way to handle this with serach engines in mind, I dont want to loose my .com listing if possible, is a framed forward a good way?
View 9 Replies
View Related
Jun 2, 2009
I want to have my email and website going to different servers:
At the moment i have it setup so that the nameserver records point to the ip of the website. That means that both site and mail go to the server where i host my site. I have then set up an mx record to point the mail to another server...
however can i make this split at the start?.. i.e. from the domain registar? can i set up an A record to point to the website and an MX record for the mail? instead of pointing them both to the website and then redirecting the mail to another server?
View 5 Replies
View Related
Apr 30, 2009
Just this week, I believe one of my site has been hacked...or potentially my whole server! When accessing the website (a vBulletin forum), instead of going to the main page, we get a screen that looks like Window's "My Computer" and there is a scan running. Firefox has blocked the site for suspicion.
I am stumped. Where to begin? I have full SSH access to my server (after rebooting it). Thank you in advance.
Server: CentOS Linux 4.3
View 10 Replies
View Related
May 18, 2009
I have a wordpress site, and am trying to find someone good and reasonable price to design me a template for the website so I can still use wordpress but have access and control to everything.. I am going to buy a domain as well to hook wordpress up to it so I can edit themes etc.. I was looking at maybe using mmhosting..
Also, I have a forum, but not sure what type of forum I need to be able to arrange it and add and use features don't I can't now, so basically upgrade..
My site, as I said is wordpress but I want it to look like the forums I am using..
View 13 Replies
View Related
Jan 18, 2009
I'm thinking of starting a website that pokes fun at news stories and photos, but I'm a complete rookie at this. All the RSS feeds I found say they are free for non-commercial use. I want to be able to sell t-shirts and caps on my site, or at least advertising. Any ways to use current news and photos that are free or inexpensive?
View 3 Replies
View Related
Jun 29, 2009
if i were to have 1 website get have a DDoS attack to it, would it knock out my server for a while, and can i recover the server by restarting it?
View 14 Replies
View Related
Apr 6, 2009
I've discovered that a site of mine has being completely copied
The sites are in a different country
[url]
[url]
The original site, that is still currently under construction with a lot of copy, rewriting and pages still have
[url]
Now, I don't mind someone taking inspiration or borrowing ideas. I'll admit as a web designer I have 1000's of book marks of sites that I've liked and will look at for ideas and inspiration, but I don't just download and put up.
Now as the sites are in a separate country I don't know yet if it is worth all the hassle as our customer base are separate, but I presume that having identical content means that google ranking could be damaged?
View 14 Replies
View Related
Jun 9, 2009
How to check web hosting of any Website and which hosting is more valuable and why?
View 24 Replies
View Related
Mar 7, 2008
My website down several hours ago, I have submitted two tickets already, but none replied and none contact with me.
my site has a daily 10k+ traffic, I will lose 10 dollar every hour... And there are pending ads. I need to put on.
View 14 Replies
View Related
Aug 31, 2008
I registered my domain with domain.com several weeks ago. Then, when I tried to get web hosting services! but I had the worst experience over the internet by professional internet company.
Anyway, the issue is not that, I did lost my money with this company but I did manage to purchase web hosting services with another web hosting company (webhostingpad.com).
However, now I am stuck with how do I manage to activate my web address at (domain.com) to be directed to my website at (webhostingpad.com).
I tried both companies support with no respond at all.
So, My question is;
What is needed to do now:
How do I activate my web address over the internet?
How to make this address point to my website at webhostingpad.com?
How to make my address shows in all search engines?
View 8 Replies
View Related
Aug 19, 2008
I wan't to start a PTC website. Therefore I need a host with a very fast and stable SQL server.
I don't need much space.
Wich one should I take? There are hundreds of them...
View 14 Replies
View Related
