Apache :: Blocking Bots And Spam Referrers?
Apr 10, 2015
If I know the IP range that I want to block the best option is to block it with IPTABLES. This works well when you want to block entire countries. But what happens when you want to block specific IPs rather than ranges? Is iptables still more effective than "deny from [IP]" in .htaccess? I read that you don't want iptables to grow too big as it slows performance, but I guess it is still more effective than having big .htaccess..?
When it comes to blocking spam bots or referrers, robots.txt is just a suggestion for bots, when I looked at my traffic logs I noticed that most bots don't even look at robots.txt file. As far as I understand the only option here is to use .htaccess
1. I am currently using this in my .htaccess:
SetEnvIfNoCase User-Agent *ahrefsbot* bad_bot=yes
SetEnvIfNoCase Referer fbdownloader.com spammer=yes
...
SetEnvIfNoCase Referer social-buttons.com spammer=yes
Order allow,deny
Allow from all
Deny from env=spammer
Deny from env=bad_bot
2. Apparently, there is another approach as per below:
# Deny domain access to spammers
RewriteEngine on
RewriteBase /
RewriteCond %{HTTP_USER_AGENT} queryseeker [OR]
RewriteCond %{HTTP_REFERER} ^(www.)?.*(-|.)?adult(-|.).*$ [OR]
...
RewriteCond %{HTTP_REFERER} ^(www.)?.*(-|.)?sex(-|.).*$
RewriteRule .* - [F,L]
Which approach is better #1 or #2? Any better alternative?
Finally, somebody suggested that you need to have both (as per example below). Is it true?
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} ^rogerbot [OR]
RewriteCond %{HTTP_USER_AGENT} ^exabot [OR]
RewriteCond %{HTTP_USER_AGENT} ^MJ12bot [OR]
[Code] ....
View 7 Replies
ADVERTISEMENT
Apr 23, 2015
I am having a problem with blocking bots using .htaccess. I think I tried all possible syntax variants, yet all the bots that I am blocking get HTTP 200 response instead of 403 (I can verify it using access log).
I am using Apache 2.4 running on Ubuntu 14.04.2 with Plesk 12.0.18.
My AllowOverride is set to allow the use of .htaccess files, so .htaccess file gets loaded: when I make an error in .htaccess sysntax I can see the error in the error log and the webpages don't load. Besides, I have some "Deny from [IP address]" directives in the .htaccess and I see that these IPs get HTTP 403 response when access my site.
I spent hours trying different variants of .htaccess syntax (see below) and neither seems to work...
variant 0:
SetEnvIfNoCase User-Agent LivelapBot bad_bot
SetEnvIfNoCase User-Agent TurnitinBot bad_bot
Order allow,deny
Allow from all
Deny from env=bad_bot
[Code] ....
View 7 Replies
View Related
Apr 22, 2009
Looking trough my logs I found something that bothers me, there are bots who keep doing requests on my website with pages like /admin or /secure to find vulnerabilities.
It's making about 5-6 requests for unexisting pages every second until it comes to the end of it's dictionary (the pages are even sorted in alphabetical order,
Is there some way to let my Apache server block access to these bots when they make X attemps to see a page who does not exists in a short amount of time? A bit like iptables reject connection if someone tries to log in but fails to do so too many times.
View 6 Replies
View Related
Jun 27, 2009
I managed to block referrers/hotlinkers by htaccess,so images are not visible there anymore,but the problem is i still see them in access log,and it seems that causing large cpu usage of apache when checking daily cpu usage.And those loaded files even doesn't exits anymore.So is it possible to completely block them,so i can not see them in domlogs/access logs?
View 1 Replies
View Related
Apr 7, 2014
I've been trying to figure out some IP blocking with no success. The environment is UNIX and Apache version is 2.2.22-14.
The site is on a hosted solution and doesn't have a firewall due to the virtualization software limitations. I've tried setting something similar to the following:
Code:
<Directory /home/username/mysite.com>
#IP Blocks
Deny from 1.2.3.1/24
Deny from 1.2.4.5
etc..
but with no success. I've also tried it in the <Location> tag with no success.
The way I'm testing this is editing the conf and then bouncing the apache server.
View 4 Replies
View Related
Nov 20, 2008
My server is getting a lot of spam robots registering as fake users into my forums (altho most emails are invalid
View 2 Replies
View Related
Feb 15, 2013
I have a website on a linux-server working fine with PHP/Apache. The page loads a lot of css/js/image-stuff (total 84 requests, 220k), it takes about 4 secs to load via internet.
Now I'm testing the same page locally on a Win7-64-system (Apache 2.2, PHP 5.4). The system is not very slow (8 GB RAM, SSD, i7-CPU), but loading the same page as above takes about 50 secs.
The Q is: What might be the problem?
- I turned off firewall and anti-virus.
- I used mod_status: 150 threads, max. 11 seems to be used during the loading of the page.
- I tried php5apache2_2.dll with TS-PHP 5.4 and mod_fcgid.so with NTS-PHP 5.4, but the loading-time kept almost the same.
Looking at the "network-tab" in FF or Chrome, I found that a lot of subqueries get a timing like this:
Blocking: 11.96 s
Sending: 0
Waiting: 1 ms
Receiving: 6ms
So the loss of time seem to be in the "blocking"-section. I first thought of something like "limited number of TCP-Connections", but as said above, on the same system the page is remotly loaded fast enough almost without these "blocking"-parts.
View 3 Replies
View Related
Jan 21, 2014
I have been trying to solve a big problem for the last 2 weeks with one of our servers.
The client using our system (web based w/ apache and php) is a contact center firm. They have about 120 operators, all connect to our websever with the same IP.
We have been suffering DoS attacks from some of these operators. This are simple, browser attacks , namely 5 or 10 operators will just hold F5 key and bombard the server with requests when they shouldnt.
We did manage to produce a php protection which will recognize the multiple requests and blacklist the user, but its "too late" because the request have already been sent and processed by the webserver.
We use the user ID in the system to control who should be blacklisted, so this is all dependent on our own authentication.
Ideally, we need something EXACTLY like mod_evasive, but for rejecting single requests instead of blocking the IP. Exemplifying : if a user calls the same url, 5 times, in a 3 second spawn, we will reject every next request for 30 seconds, but only the requests by that user.
If the webserver can make any use of it, the user id is stored in a cookie.
View 4 Replies
View Related
Apr 19, 2007
I took the 1st one this morning and the 2nd one few hours later. It was filling up my vpss numtcpsock, which slowed down my vps dramtically. Any tips or suggestions? Is there a way to lower the number of numtcpsocks
early morning
Quote:
CLOSE_WAIT
tcp 886 0 74.208.69.213:80 124.43.222.202:2701 CLOSE_WAIT
tcp 886 0 74.208.69.213:80 124.43.222.202:2702 CLOSE_WAIT
tcp 893 0 74.208.69.213:80 124.43.222.202:2703 ESTABLISHED
tcp 0 11950 74.208.69.213:80 58.62.96.22:11072 CLOSE_WAIT
tcp 886 0 74.208.69.213:80 124.43.222.202:2696 CLOSE_WAIT
tcp 0 11950 74.208.69.213:80 58.62.96.22:11073 CLOSE_WAIT
tcp 886 0 74.208.69.213:80 124.43.222.202:2697 CLOSE_WAIT
tcp 292 0 74.208.69.213:80 74.6.72.244:47500 ESTABLISHED
tcp 894 0 74.208.69.213:80 124.43.222.202:2698 CLOSE_WAIT
tcp 892 0 74.208.69.213:80 124.43.222.202:2699 CLOSE_WAIT
tcp 853 0 74.208.69.213:80 87.52.47.152:4663 ESTABLISHED
tcp 393 0 74.208.69.213:80 58.62.96.22:11084 CLOSE_WAIT
tcp 0 0 74.208.69.213:80 85.160.18.247:1598 FIN_WAIT2
tcp 0 11950 74.208.69.213:80 58.62.96.22:11020 LAST_ACK
tcp 0 11855 74.208.69.213:80 222.170.151.73:32432 FIN_WAIT1
tcp 394 0 74.208.69.213:80 58.62.96.22:11085 CLOSE_WAIT
tcp 953 0 74.208.69.213:80 84.144.97.34:61014 ESTABLISHED
tcp 0 52 74.208.69.213:22 74.103.175.48:63101 ESTABLISHED
tcp 951 0 74.208.69.213:80 84.144.97.34:61013 ESTABLISHED
tcp 0 0 74.208.69.213:80 213.6.220.1:46493 TIME_WAIT
tcp 0 11951 74.208.69.213:80 58.62.96.22:11022 LAST_ACK
tcp 0 11978 74.208.69.213:80 124.43.222.202:2694 CLOSE_WAIT
tcp 0 11951 74.208.69.213:80 58.62.96.22:11023 LAST_ACK
tcp 0 11980 74.208.69.213:80 124.43.222.202:2695 CLOSE_WAIT
tcp 394 0 74.208.69.213:80 58.62.96.22:11080 CLOSE_WAIT
tcp 244 0 74.208.69.213:80 86.27.108.29:1643 ESTABLISHED
tcp 394 0 74.208.69.213:80 58.62.96.22:11082 CLOSE_WAIT
tcp 0 11855 74.208.69.213:80 222.170.151.73:18295 FIN_WAIT1
tcp 0 11855 74.208.69.213:80 222.170.151.73:33719 FIN_WAIT1
tcp 1179 0 74.208.69.213:80 85.160.18.247:1606 ESTABLISHED
tcp 0 11854 74.208.69.213:80 222.170.151.73:31560 ESTABLISHED
tcp 0 11950 74.208.69.213:80 58.62.96.22:10996 LAST_ACK
tcp 0 11855 74.208.69.213:80 222.170.151.73:32073 FIN_WAIT1
tcp 0 11855 74.208.69.213:80 222.170.151.73:32075
few hors later:
Quote:
tcp 0 0 74.208.69.213:80 124.43.212.78:3676 SYN_RECV
tcp 0 0 74.208.69.213:80 124.43.212.78:3696 SYN_RECV
tcp 0 0 74.208.69.213:80 124.43.212.78:3737 SYN_RECV
tcp 0 0 74.208.69.213:80 124.43.212.78:3731 SYN_RECV
tcp 0 0 74.208.69.213:80 124.43.212.78:3735 SYN_RECV
tcp 0 0 74.208.69.213:80 124.43.212.78:3723 SYN_RECV
tcp 0 0 74.208.69.213:80 217.77.17.186:1959 SYN_RECV
tcp 0 0 74.208.69.213:80 124.43.212.78:3691 SYN_RECV
tcp 0 0 74.208.69.213:80 124.43.212.78:3671 SYN_RECV
tcp 0 0 74.208.69.213:80 88.102.18.14:3628 SYN_RECV
tcp 0 0 74.208.69.213:80 124.43.212.78:3747 SYN_RECV
tcp 0 0 74.208.69.213:80 124.43.212.78:3757 SYN_RECV
tcp 0 0 74.208.69.213:80 124.43.212.78:3754 SYN_RECV
tcp 0 0 74.208.69.213:80 124.43.212.78:3761 SYN_RECV
tcp 0 0 74.208.69.213:80 201.27.210.29:2540 SYN_RECV
tcp 0 0 74.208.69.213:80 213.216.199.14:37692 SYN_RECV
tcp 0 0 74.208.69.213:80 124.43.212.78:3753 SYN_RECV
tcp 0 0 74.208.69.213:80 72.14.199.72:40365 SYN_RECV
tcp 0 0 74.208.69.213:80 88.146.161.248:19475 SYN_RECV
tcp 0 0 74.208.69.213:80 81.193.196.132:2482 SYN_RECV
tcp 0 0 74.208.69.213:80 82.229.95.240:3246 SYN_RECV
tcp 0 0 74.208.69.213:80 124.43.212.78:3743 SYN_RECV
tcp 0 0 74.208.69.213:80 124.43.212.78:3695 SYN_RECV
tcp 0 23232 74.208.69.213:80 124.43.212.78:3501 ESTABLISHED
tcp 887 0 74.208.69.213:80 124.43.212.78:3693 ESTABLISHED
tcp 897 0 74.208.69.213:80 124.43.212.78:3629 CLOSE_WAIT
tcp 0 11616 74.208.69.213:80 124.43.212.78:3500 ESTABLISHED
tcp 792 0 74.208.69.213:80 124.43.212.78:3628 CLOSE_WAIT
tcp 891 0 74.208.69.213:80 124.43.212.78:3631 CLOSE_WAIT
tcp 890 0 74.208.69.213:80 124.43.212.78:3694 ESTABLISHED
tcp 886 0 74.208.69.213:80 124.43.212.78:3630 CLOSE_WAIT
tcp 800 0 74.208.69.213:80 124.43.212.78:3625 CLOSE_WAIT
tcp 891 0 74.208.69.213:80 124.43.212.78:3624 CLOSE_WAIT
tcp 0 39204 74.208.69.213:80 124.43.212.78:3499 ESTABLISHED
tcp 895 0 74.208.69.213:80 124.43.212.78:3627 CLOSE_WAIT
tcp 0 49173 74.208.69.213:80 124.43.212.78:3562 ESTABLISHED
View 7 Replies
View Related
Nov 14, 2008
I would like to ask about the best system or software code used to stop bots and offline down loaders from entering website.
View 2 Replies
View Related
Mar 18, 2008
Simple web site:
www.oldWithoutMoney.com
Implemented WordPress a little while ago via cPanel's Fantastico widget -- vanilla implementation.
Just about every day, I get spam comments in the blog's Inbox for moderation.
Was wondering if folks had general tips on how to prevent or minimize this sort of nuisance and make the blog less bot-accessible, and/or where I might read up on ways to do so.
View 5 Replies
View Related
Mar 16, 2008
Most of them are from Google and Yahoo...
Server is being heavily loaded beause of this.
I guess blocking crawlers is not the most brilliant
View 6 Replies
View Related
Feb 19, 2008
I just received two complaints that my server (71.6.197.244) is trying to run exploits on other people's servers.
I have tried checking my access logs, but am not sure what to look for.
Is this a process, or is it an exploit through a url or a php form?
I have attached the e-mail complaints as txt.
View 6 Replies
View Related
Nov 25, 2007
Is there any way to control the pesky Yahoo bots?
Bots with an IP of 74.6.*.* are endlessly crawling my forums, never seeming to be able to finish their task and using up huge amounts of bandwidth. Ideally I would like to be able to turn them away at the gate but allow others to view the site.
If I block the IP via the control panel they still visit presumably just getting error pages but still using up lots of bandwidth.
View 7 Replies
View Related
Apr 26, 2008
I have a dedicated RHEL server with cPanel and my server loads spikes about +0.4 (out of 2.0) for about 30 mins every 4-6 hours or so. My regular server load is 0.01, because there is barely any traffic on the server yet, but by looking at my top processes in WHM, I can see that the processes that are spiking the Server Load when it is high, is something like:
sshd: [priv] root
sshd: [priv] root
sshd: [priv] root
sshd: [accepted]
sshd: [priv] games
sshd: [priv] news
sshd: [priv] root
sshd: [priv] root
sshd: [accepted]
...something along these lines. And a lot of times there are 10-20 of these sshd processes at one time.
My server is managed and my dedicated server engineer said it was probably a bot trying passwords. He took one of the IP's, said it was from Taiwan, and blocked that IP in iptables.
However, this is still happening constantly with different IP's. Is there a way to prevent this from happening? I'm the only person (and my host) who should be able to login to my server using SSH... however, I don't have a static IP and I work from multiple locations, so only allowing certain IP's won't work for me.
First off, is this normal? Or am I being attacked or what? What can I do to remedy this? It seems the bots haven't successfully logged in, but they are spiking my server load which is NOT what I want.
View 8 Replies
View Related
Aug 29, 2007
Yesterday my mail logs started showing many a spam email being sent from my server. There isn't anything mission critical running on it, so I took down qmail until I could find the vulnerability and fix it. But try as I might, I haven't found any conclusive vulnerability, so I thought to ask here where someone with more experience might spot something obvious that I've missed (I'm still somewhat new to this).
Anyway, the qmail logs show that the messages came from uid 48, apache. Log excerpt (sending of first spam mail):
Quote:
Aug 28 11:10:51 host qmail-queue[8056]: mail: all addreses are uncheckable - need to skip scanning (by deny mode)
Aug 28 11:10:51 host qmail-queue[8056]: scan: the message(drweb.tmp.TNDOi2) sent by anonymous@HOSTNAME to SPAMADDRESS should be passed without checks, because contains uncheckable addresses
Aug 28 11:10:51 host qmail: 1188295851.742521 new msg 51970054
Aug 28 11:10:51 host qmail: 1188295851.742679 info msg 51970054: bytes 445 from <anonymous@HOSTNAME> qp 8057 uid 48
Aug 28 11:10:51 host qmail: 1188295851.752799 starting delivery 460: msg 51970054 to remote SPAMADDRESS
Aug 28 11:10:51 host qmail: 1188295851.752933 status: local 0/10 remote 1/20
Unfortunately, my Apache logs have no entries around the time when these messages were sent. There are some suspect "CONNECT" requests scattered throughout the logs, but all are denied with 405's, and none correspond exactly with the time of the spam. Example (from about 3 hours after the spam):
Quote:
210.17.191.242 - - [28/Aug/2007:14:34:43 +0100] "CONNECT 205.158.62.146:25 HTTP/1.0" 405 235 "-" "-"
210.17.191.242 - - [28/Aug/2007:14:34:43 +0100] "PUT [url]
HTTP/1.0" 405 231 "-" "-"
210.17.191.242 - - [28/Aug/2007:14:34:43 +0100] "POST [url]
HTTP/1.0" 200 2 "-" "-"
(The fact that the final query wasn't denied worries me slightly though. Does anyone have any insight?)
I'm not sure where to go from here. I'm concerned about the lack of logs by Apache. There's a nine hour period without any entries; not unusual for my server given that its not very active, but the time when the spam was sent falls in this time period. I've checked for common security issues, but qmail is configured only to relay from localhost, and Apache isn't configured as an open proxy. Are there any other common issues I should check for? Is there any other information I should post here to help identify the problem?
I'm running Apache version 2.0.52, and qmail 1.03.
I'd be very grateful for any help or links to relevant HOWTOs.
View 6 Replies
View Related
Jun 28, 2015
I have a regular visitor from 1999521.videos-for-your-business.com/ and I am aware this is spam from semalt.com, the trouble is I have tried blocking this but it keeps coming back the number 1999521 changes frequently
I have the following in my htaccess file
## STOP REFERRER SPAM
RewriteCond %{HTTP_REFERER} semalt.com [NC,OR]
RewriteCond %{HTTP_REFERER} best-seo-offer.com [NC,OR]
RewriteCond %{HTTP_REFERER} videos-for-your-business.com [NC,OR]
RewriteCond %{HTTP_REFERER} semaltmedia.com [NC,OR]
RewriteCond %{HTTP_REFERER} best-seo-solution.com
RewriteRule .* - [F]
View 3 Replies
View Related
Mar 14, 2008
Not far ago somebody hacked our customer account through the vulnerability in phpBB Album module and uploaded some scripts. Then it started to send nigerian spam using exim and apache. These scripts were found and deleted and the Album module was fully deleted too. But when I look at the processes now I see that exim and httpd still start very often so the system resources are probably overused by them ......
View 7 Replies
View Related
Oct 17, 2013
Microsoft Windows Server 2008 R2 Service Pack 1
Panel version 11.0.9 Update #59, last updated at Oct 3, 2013 02:06 AM
MailEnable version 5
I see in the plesk documentation that the screen to enable SPAM filtering for an individual there is an option to "Move spam to the Spam folder". I don't see that option so I am wondering if it is only available on some versions of Plesk, or in combination with certain mail servers. How to make that option available?
View 3 Replies
View Related
Apr 5, 2007
I've had it with EV1. On any given day we get 30-50 BFD attacks from their servers. That doesn't include the dozens of other types of attempts per day our IPS/IDS catch. We've also traced back client servers that were hacked directly via EV1 servers.
It's obvious that EV1 does little or nothing to stop these issues. We spoke to the FBI about these issues and their comments lead me to believe that EV1 is one of the major sources of these issues and that EV1 has shown little or no effort to curb the problem or cooperate in stopping the issue.
We have elected to now block all all EV1 IPS.
Drastic measures, not really. If they won't take care of their own problems I no longer want them dumped at my door step. I think other hosts might want to think about this.
View 14 Replies
View Related
Apr 28, 2007
At the moment it will block people who login with the wrong username/password 5 times. it also blocks people if they do the wrong email settings.
Is there a way to turn the pop3/email blocking off?
View 3 Replies
View Related
Dec 3, 2006
I am hitting a limit on number of POP signons per hour imposed by my host. I host maybe 10 domains on this account and have 4 or 5 email addresses to monitor for each domain. If I check once every 15 minutes I run up against a limit on the number of POP3 signons permitted by my ip. Add this to having multiple mail clients behind a NAT router and I am beginning to have real problems.
Does anyone else have this issue? Is my only workaround to forward all email to a single account or install a local mail server? Does 100 POP signons an hour from a single IP sound like a lot to anyone? Any advice?
View 6 Replies
View Related
Jul 19, 2008
Let's say you want to protect againts hacking,and using method with simply blocking loading url.So let's say someone hacked your index.html and changed links to lead to his domain.com.Is it possible to block what would be loaded on site ?(to prevent possible future hacking intrusions)
View 6 Replies
View Related
Nov 28, 2008
I have 2 server one is Linux server+Cpanel+CSF firewall where my site is running and one is windows server where my exchange mail server is running .Now thing is that when anyone send mail through my web site (after filling contact form) to me it doesn't come to my email id but when i stop my firewall and then i check contact form and fill it the mail goes to my mail id.
I have php script with SMTP authentication.
which port is blocked in my firewall and after disabling firewall it work.how can i check when firewall is on that time why mails are not coming in my email id that time which port is blocked by firewall.
Allow Port in firewall:-- 25,80,20,21,465,443,110,143
View 10 Replies
View Related
Jul 5, 2007
seems one of my sites has been added to some mega "toplist" site thats bringing in fake traffic to my site which is basically like a DOS attack - over 1000 connections.
coming from
[url]
[url]
[url]
linking to a php file in one of my accounts which has since been removed. however still getting a heck of a lot of hits, they probably all see 404 messages which still causes load on my server.
any suggestions how to fix this? the traffic is referred from above urls but hundreds of ip addresses. is there anyway to blacklist the referrer so people are just blocked, period?
View 6 Replies
View Related
Oct 6, 2007
Running freebsd with pf, and was wondering if there's anything like www.fixingtheweb.info for pf instead of IP tables? Otherwise it'll be a long day
View 1 Replies
View Related
Apr 14, 2007
I had a few sites hacked today. I'm using phpbb (all updates) and, apparently, the only thing they did was to drop the database and replace it with one featuring a single post "advertising" their hacker group. I tried bringing everything back on-line, but they would just attack again and take it down quickly... I'm thinking it's probably just some script kiddies.
They announce themselves as "turkish hackers". Browsing around for their message, I found they attacked quite a few sites. What I was thinking, to help preventing this from happening again, is to ban all visitors from Turkey (none of these sites has a need for them, as they're aimed at a local audience).
Can I do this simply by using "deny from .tr" in htaccess? Or are there any more steps to be taken?
View 6 Replies
View Related