Is there any way to control the pesky Yahoo bots?
Bots with an IP of 74.6.*.* are endlessly crawling my forums, never seeming to be able to finish their task and using up huge amounts of bandwidth. Ideally I would like to be able to turn them away at the gate but allow others to view the site.
If I block the IP via the control panel they still visit presumably just getting error pages but still using up lots of bandwidth.
I took the 1st one this morning and the 2nd one few hours later. It was filling up my vpss numtcpsock, which slowed down my vps dramtically. Any tips or suggestions? Is there a way to lower the number of numtcpsocks
early morning
Quote:
CLOSE_WAIT
tcp 886 0 74.208.69.213:80 124.43.222.202:2701 CLOSE_WAIT
tcp 886 0 74.208.69.213:80 124.43.222.202:2702 CLOSE_WAIT
tcp 893 0 74.208.69.213:80 124.43.222.202:2703 ESTABLISHED
tcp 0 11950 74.208.69.213:80 58.62.96.22:11072 CLOSE_WAIT
tcp 886 0 74.208.69.213:80 124.43.222.202:2696 CLOSE_WAIT
tcp 0 11950 74.208.69.213:80 58.62.96.22:11073 CLOSE_WAIT
tcp 886 0 74.208.69.213:80 124.43.222.202:2697 CLOSE_WAIT
tcp 292 0 74.208.69.213:80 74.6.72.244:47500 ESTABLISHED
tcp 894 0 74.208.69.213:80 124.43.222.202:2698 CLOSE_WAIT
tcp 892 0 74.208.69.213:80 124.43.222.202:2699 CLOSE_WAIT
tcp 853 0 74.208.69.213:80 87.52.47.152:4663 ESTABLISHED
tcp 393 0 74.208.69.213:80 58.62.96.22:11084 CLOSE_WAIT
tcp 0 0 74.208.69.213:80 85.160.18.247:1598 FIN_WAIT2
tcp 0 11950 74.208.69.213:80 58.62.96.22:11020 LAST_ACK
tcp 0 11855 74.208.69.213:80 222.170.151.73:32432 FIN_WAIT1
tcp 394 0 74.208.69.213:80 58.62.96.22:11085 CLOSE_WAIT
tcp 953 0 74.208.69.213:80 84.144.97.34:61014 ESTABLISHED
tcp 0 52 74.208.69.213:22 74.103.175.48:63101 ESTABLISHED
tcp 951 0 74.208.69.213:80 84.144.97.34:61013 ESTABLISHED
tcp 0 0 74.208.69.213:80 213.6.220.1:46493 TIME_WAIT
tcp 0 11951 74.208.69.213:80 58.62.96.22:11022 LAST_ACK
tcp 0 11978 74.208.69.213:80 124.43.222.202:2694 CLOSE_WAIT
tcp 0 11951 74.208.69.213:80 58.62.96.22:11023 LAST_ACK
tcp 0 11980 74.208.69.213:80 124.43.222.202:2695 CLOSE_WAIT
tcp 394 0 74.208.69.213:80 58.62.96.22:11080 CLOSE_WAIT
tcp 244 0 74.208.69.213:80 86.27.108.29:1643 ESTABLISHED
tcp 394 0 74.208.69.213:80 58.62.96.22:11082 CLOSE_WAIT
tcp 0 11855 74.208.69.213:80 222.170.151.73:18295 FIN_WAIT1
tcp 0 11855 74.208.69.213:80 222.170.151.73:33719 FIN_WAIT1
tcp 1179 0 74.208.69.213:80 85.160.18.247:1606 ESTABLISHED
tcp 0 11854 74.208.69.213:80 222.170.151.73:31560 ESTABLISHED
tcp 0 11950 74.208.69.213:80 58.62.96.22:10996 LAST_ACK
tcp 0 11855 74.208.69.213:80 222.170.151.73:32073 FIN_WAIT1
tcp 0 11855 74.208.69.213:80 222.170.151.73:32075
few hors later:
Quote:
tcp 0 0 74.208.69.213:80 124.43.212.78:3676 SYN_RECV
tcp 0 0 74.208.69.213:80 124.43.212.78:3696 SYN_RECV
tcp 0 0 74.208.69.213:80 124.43.212.78:3737 SYN_RECV
tcp 0 0 74.208.69.213:80 124.43.212.78:3731 SYN_RECV
tcp 0 0 74.208.69.213:80 124.43.212.78:3735 SYN_RECV
tcp 0 0 74.208.69.213:80 124.43.212.78:3723 SYN_RECV
tcp 0 0 74.208.69.213:80 217.77.17.186:1959 SYN_RECV
tcp 0 0 74.208.69.213:80 124.43.212.78:3691 SYN_RECV
tcp 0 0 74.208.69.213:80 124.43.212.78:3671 SYN_RECV
tcp 0 0 74.208.69.213:80 88.102.18.14:3628 SYN_RECV
tcp 0 0 74.208.69.213:80 124.43.212.78:3747 SYN_RECV
tcp 0 0 74.208.69.213:80 124.43.212.78:3757 SYN_RECV
tcp 0 0 74.208.69.213:80 124.43.212.78:3754 SYN_RECV
tcp 0 0 74.208.69.213:80 124.43.212.78:3761 SYN_RECV
tcp 0 0 74.208.69.213:80 201.27.210.29:2540 SYN_RECV
tcp 0 0 74.208.69.213:80 213.216.199.14:37692 SYN_RECV
tcp 0 0 74.208.69.213:80 124.43.212.78:3753 SYN_RECV
tcp 0 0 74.208.69.213:80 72.14.199.72:40365 SYN_RECV
tcp 0 0 74.208.69.213:80 88.146.161.248:19475 SYN_RECV
tcp 0 0 74.208.69.213:80 81.193.196.132:2482 SYN_RECV
tcp 0 0 74.208.69.213:80 82.229.95.240:3246 SYN_RECV
tcp 0 0 74.208.69.213:80 124.43.212.78:3743 SYN_RECV
tcp 0 0 74.208.69.213:80 124.43.212.78:3695 SYN_RECV
tcp 0 23232 74.208.69.213:80 124.43.212.78:3501 ESTABLISHED
tcp 887 0 74.208.69.213:80 124.43.212.78:3693 ESTABLISHED
tcp 897 0 74.208.69.213:80 124.43.212.78:3629 CLOSE_WAIT
tcp 0 11616 74.208.69.213:80 124.43.212.78:3500 ESTABLISHED
tcp 792 0 74.208.69.213:80 124.43.212.78:3628 CLOSE_WAIT
tcp 891 0 74.208.69.213:80 124.43.212.78:3631 CLOSE_WAIT
tcp 890 0 74.208.69.213:80 124.43.212.78:3694 ESTABLISHED
tcp 886 0 74.208.69.213:80 124.43.212.78:3630 CLOSE_WAIT
tcp 800 0 74.208.69.213:80 124.43.212.78:3625 CLOSE_WAIT
tcp 891 0 74.208.69.213:80 124.43.212.78:3624 CLOSE_WAIT
tcp 0 39204 74.208.69.213:80 124.43.212.78:3499 ESTABLISHED
tcp 895 0 74.208.69.213:80 124.43.212.78:3627 CLOSE_WAIT
tcp 0 49173 74.208.69.213:80 124.43.212.78:3562 ESTABLISHED
I would like to ask about the best system or software code used to stop bots and offline down loaders from entering website.
View 2 Replies View RelatedSimple web site:
www.oldWithoutMoney.com
Implemented WordPress a little while ago via cPanel's Fantastico widget -- vanilla implementation.
Just about every day, I get spam comments in the blog's Inbox for moderation.
Was wondering if folks had general tips on how to prevent or minimize this sort of nuisance and make the blog less bot-accessible, and/or where I might read up on ways to do so.
Most of them are from Google and Yahoo...
Server is being heavily loaded beause of this.
I guess blocking crawlers is not the most brilliant
I just received two complaints that my server (71.6.197.244) is trying to run exploits on other people's servers.
I have tried checking my access logs, but am not sure what to look for.
Is this a process, or is it an exploit through a url or a php form?
I have attached the e-mail complaints as txt.
Looking trough my logs I found something that bothers me, there are bots who keep doing requests on my website with pages like /admin or /secure to find vulnerabilities.
It's making about 5-6 requests for unexisting pages every second until it comes to the end of it's dictionary (the pages are even sorted in alphabetical order,
Is there some way to let my Apache server block access to these bots when they make X attemps to see a page who does not exists in a short amount of time? A bit like iptables reject connection if someone tries to log in but fails to do so too many times.
I have a dedicated RHEL server with cPanel and my server loads spikes about +0.4 (out of 2.0) for about 30 mins every 4-6 hours or so. My regular server load is 0.01, because there is barely any traffic on the server yet, but by looking at my top processes in WHM, I can see that the processes that are spiking the Server Load when it is high, is something like:
sshd: [priv] root
sshd: [priv] root
sshd: [priv] root
sshd: [accepted]
sshd: [priv] games
sshd: [priv] news
sshd: [priv] root
sshd: [priv] root
sshd: [accepted]
...something along these lines. And a lot of times there are 10-20 of these sshd processes at one time.
My server is managed and my dedicated server engineer said it was probably a bot trying passwords. He took one of the IP's, said it was from Taiwan, and blocked that IP in iptables.
However, this is still happening constantly with different IP's. Is there a way to prevent this from happening? I'm the only person (and my host) who should be able to login to my server using SSH... however, I don't have a static IP and I work from multiple locations, so only allowing certain IP's won't work for me.
First off, is this normal? Or am I being attacked or what? What can I do to remedy this? It seems the bots haven't successfully logged in, but they are spiking my server load which is NOT what I want.
I am having a problem with blocking bots using .htaccess. I think I tried all possible syntax variants, yet all the bots that I am blocking get HTTP 200 response instead of 403 (I can verify it using access log).
I am using Apache 2.4 running on Ubuntu 14.04.2 with Plesk 12.0.18.
My AllowOverride is set to allow the use of .htaccess files, so .htaccess file gets loaded: when I make an error in .htaccess sysntax I can see the error in the error log and the webpages don't load. Besides, I have some "Deny from [IP address]" directives in the .htaccess and I see that these IPs get HTTP 403 response when access my site.
I spent hours trying different variants of .htaccess syntax (see below) and neither seems to work...
variant 0:
SetEnvIfNoCase User-Agent LivelapBot bad_bot
SetEnvIfNoCase User-Agent TurnitinBot bad_bot
Order allow,deny
Allow from all
Deny from env=bad_bot
[Code] ....
If I know the IP range that I want to block the best option is to block it with IPTABLES. This works well when you want to block entire countries. But what happens when you want to block specific IPs rather than ranges? Is iptables still more effective than "deny from [IP]" in .htaccess? I read that you don't want iptables to grow too big as it slows performance, but I guess it is still more effective than having big .htaccess..?
When it comes to blocking spam bots or referrers, robots.txt is just a suggestion for bots, when I looked at my traffic logs I noticed that most bots don't even look at robots.txt file. As far as I understand the only option here is to use .htaccess
1. I am currently using this in my .htaccess:
SetEnvIfNoCase User-Agent *ahrefsbot* bad_bot=yes
SetEnvIfNoCase Referer fbdownloader.com spammer=yes
...
SetEnvIfNoCase Referer social-buttons.com spammer=yes
Order allow,deny
Allow from all
Deny from env=spammer
Deny from env=bad_bot
2. Apparently, there is another approach as per below:
# Deny domain access to spammers
RewriteEngine on
RewriteBase /
RewriteCond %{HTTP_USER_AGENT} queryseeker [OR]
RewriteCond %{HTTP_REFERER} ^(www.)?.*(-|.)?adult(-|.).*$ [OR]
...
RewriteCond %{HTTP_REFERER} ^(www.)?.*(-|.)?sex(-|.).*$
RewriteRule .* - [F,L]
Which approach is better #1 or #2? Any better alternative?
Finally, somebody suggested that you need to have both (as per example below). Is it true?
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} ^rogerbot [OR]
RewriteCond %{HTTP_USER_AGENT} ^exabot [OR]
RewriteCond %{HTTP_USER_AGENT} ^MJ12bot [OR]
[Code] ....
For several months the Yahoo bot had been controlled to one visit per 40 seconds with a robots.txt page. Today it is not working and creating several page impressions per minute.
Has anyone seen any new advice on how to control this pesky creature?
This is the file content I have been using:
User-agent: Slurp
Crawl-delay: 40
User-agent: *
Disallow:
My server is being pounded with Yahoo bots, close to 500 or more at a time. They are causing my loads at various times of the day to skyrocket, crashing the mysql server, etc. They have been on my site for about 4 days now and I just dont see why that many bots are needed to crawl a single site for several days.
Is there any way around this without banning their ips? Any way to limit them?
i want to preview a file in my browser but so far it s not possible i am using yahoo sitebuilder 2.5 and when i m trying to previw the website an erro appears and says that i havent save my site in the write place.. when i do that it says that i haven still save my site in the right place,, i spend about a week of work on it... also i cant publish it.. is there anyway to c the html code or do somthn else?
View 6 Replies View RelatedI have this strange problem with smartermail, it can send perfectly to hotmail and gmail but when it tried sending to yahoo accounts, there is always a delay and the email end in the bulk folder.
Such problems does not occur with my qmail or icewarp merak. Any idea how to solve? Have already emailed Yahoo hoping to get a whitelisting of my IP. Besides installing domainkeys any other ways?
Has anyone ever used Yahoo to host their website? I have heard some good and more bad comments about them. I am a rookie at this so my site is going to be pretty basic. How do they rank vs. other hosting companies?
Yahoo has an “easy site creator” tool to make a site with a template like process. Do sites that use these rank as well in search engines as sites built without a template?
lately yahoo make advancements in their hosting plan,and i want to choose best of yahoo or bluehost. could anyone give me an opinion.
View 14 Replies View Relatedyahoo offering unlimited bandwidth and space at very low cost around $50 (2000INR in our currency),is it reliable or not?
Because i have a wallpaper site requires some server resources but not so high.
suggest me,can i take this one?
If I'm not mistaken we have reverse DNS setup and no blacklist entries. Is there any reason why welcome emails from our custom CP are being filtered directly into the Yahoo SPAM box?
Here's the full headers from my personal email. This is the exact message a client would be sent.
From HostVentrilo.com Sun Apr 20 20:38:17 2008
Return-Path: <nobody@web.teamspeakhost.com>
Authentication-Results: mta105.rog.mail.scd.yahoo.com from=hostventrilo.com; domainkeys=neutral (no sig)
Received: from 69.93.229.114 (EHLO web.teamspeakhost.com) (69.93.229.114)
by mta105.rog.mail.scd.yahoo.com with SMTP; Sun, 20 Apr 2008 20:38:18 -0700
Received: from nobody by web.teamspeakhost.com with local (Exim 4.68)
(envelope-from <nobody@web.teamspeakhost.com>)
id 1JnmrB-0007AT-P3
for xxxxx@rogers.com; Sun, 20 Apr 2008 23:38:17 -0400
Received: from phpmailer ([67.204.23.77])
by www.hostventrilo.com with HTTP (PHPMailer);
Sun, 20 Apr 2008 23:38:17 -0400
Date: Sun, 20 Apr 2008 23:38:17 -0400
To: Jeff Piper <xxxxx@rogers.com>
From: "HostVentrilo.com" <info@hostventrilo.com>
Subject: Ventrilo Server Information
Message-ID: <edee9a956937977bcb723593ae6938a5@www.hostventrilo.com>
MIME-Version: 1.0
Content-Transfer-Encoding: 8bit
Content-Type: text/plain; charset="utf-8"
Content-Length: 567
My only guess would be the PHP programming of the mailer code?
Some of my emails send to the clients who are using Yahoo's email is stored in their " Bulk " folder, so Yahoo is considering my email as spamer althought i'm not. So do you know how can i fix it ? Do i need to contat Yahoo about this matter ?
View 1 Replies View RelatedNowadays it's everyone's headache for sending mails to hotmail and yahoo i guess. I am having the same problem.
I add the domainkey to domains, but when i send mail to yahoo it drops to spam.
I didn't tried it with hotmail, but reason for dropping spam at yahoo is probably because of no spf record.
I am having a bit trouble with setting spf. I read some documents and my mind is messed up, i can't figure out what to do.
My server's ip address is : xx.xx.xx.xx
My server's hostname is : server.x.com
and i have domains like , y.com, z.com, f.com bla bla that wants to send mail to hotmail.
Those sites are using different ips, but each ip is using more than 1 site,
I read that i need to setup reverse dns, but one article says i should do it for my hostname, other one says for mail server : mail.x.com.
So first question how should i setup reverse dns?
Second question, it says i need to add spf record for my domain, does it mean my hostname, or the domain i want to send mail ?
v=spf1 a mx ptr ~all
and than i should fill form at https://support.msn.com/eform.aspx?p...rid&ct=eformts
and than wait.
So i will be glad if someone can answer my questions and tell me if there is any missing parts.
I am looking for an expert who can help me configure my dedicated server to ensure my email gets delivered to Yahoo.
I have an autoresponder script hosted on the server. The script was previously hosted with another host and I got excellent deliverability to Yahoo, Gmail, Hotmail and everyone else.
Unfortunately the volume of mail was too large for that host, so now I have a dedicated server. Since I moved to the dedicated server (two weeks ago), my mail does not get delivered to Yahoo. It doesn't even go into the junk mail folder. It just disappears.
I am very strictt about using confirmed optin for my lists, so it's not a blacklist issue. Right from day one on the new server, my mail did not reach Yahoo. So it must be the way my server is set up.
If there is someone who knows how to resolve this, I am willing to pay you to solve the problem for me. Or, if there is anyone who can give me some advice, that would be much appreciated.
Im having problem with Yahoo web hosting. All .php pages is not working.. it will give you an error "403 Forbidden".
I cant even access my domain webhosting control panel..
I called Yahoo Customer Service and it will take 2-3 days to fix. Sad to know my website is down almost 1 week.
Can anyone tell me what's really going on with Yahoo Webhosting? Do they care their customers?
I have noticed that Yahoo is refusing emails coming from the IP of my server. I have written them and I got this email, has this happened to you too? Any recommendations? ...
View 9 Replies View RelatedDo you know what is most important search engine robot ips?
I need to all serach engine robot ips too firewall white list.
I have a problem with Yahoo defering my email from servers for weeks!
I've submit to [url]
from
[url]
MANY TIMES! this is getting me frustrated. what's more.. i try to change my SMTP to a new server (fresh IP within same network in the same DC) and it is ALSO deferred. How do I check if this has something to do with a whole block of IP maybe listed in some spam engine or something?
How can i ban Yahoo! Slurp and its IPs using .htacces?
View 3 Replies View RelatedLike so many others I am having the problem in which Hotmail and Yahoo are rejecting emails being sent from my server. I recently changed servers and this is most likely the reason.
While Hotmail hasn't been fixed, they have responded swiftly, usually within 6 hours.
I need to contact Yahoo about it but I can't find any information or forms to fill out. Could you please direct me to the correct URL - their site is a complete maze.
Unfortunately yahoo blocked our server ip, so our customer cant send any email to yahoo!
How can fix this issue? Do you have any email or ticket system in yahoo for contant about this issue?
business is just getting out of hand. I applied several weeks ago for Whitelist status, and my issues finally went away for a little over a week (though I never received a response to my Postmaster requests). But then today -bam- 100% deferrals for going on 18 hours now, not a single message has gone through. And naturally no two Yahoo servers give me the same error message.
So...
At this point I'm ready to contract out my Outbound mail to Yahoo through a whitelisted 3rd party until I can get this resolved on my end. Would this be reasonable? Is anyone else doing this? I worked with an outsourced SMTP provider in another life for an internal company mailing list with good success.
It seems that Yahoo does web hosting, and they offer WordPress as one of their one-click install kinds of things. Given the absolutely absurd responsiveness of yahoo.com, I was curious if anyone actually used them as a webhost?
Their pricing is decent, but I'm wondering about performance? I'm currently with Dreamhost, and I'm okay with them, except that WP performance isn't great (not awful, though, either), despite having wp-cache installed and operational.
So how good it Yahoo's small business webhosting? Their TOS seems pretty generous as well.
[url]
if there is much i can do to get the mail that gets sent from my server to show up in the inbox of users using a free mail service.
View 2 Replies View Related