.htaccess Block BaiduSpider Mod_rewrite

Oct 22, 2009

I have tried, without success, to block approximately 50 crawlers of the Baidu search engine, in China; since this web site can only sell magazines to people with U.S. addresses. The top 6 lines of my .htaccess file are at the bottom of this post.

Someone in OLM Support gave me the below link to webmasterworld. Since nothing I have tried so far has worked, I would appreciate someone knowledgeable about mod_rewrite and .htaccess letting me know whether I can place those 3 lines at the very top of my .htaccess file, or, if they need to be placed lower in the file. (If so, where?)

how to keep the Baidu crawlers out. The site is on Shared Hosting, so trying to keep them out with iptables or something else is not an option for me. They do not look at the robots.txt file or sitemap.txt file, they just come in....

[url]

If your site is on an Apache server, you can block using mod_rewrite via .htaccess:

RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} ^Baiduspider [NC]
RewriteRule .* - [F]

I included the [NC] which allows for case differences since at least one of the Baidu bots uses "BaiDuSpider"

deny from baidu.com
deny from crawl.baidu.com
deny from 220.181.7.
deny from 123.125.66.
deny from baiduspider-220-181-7-20.crawl.baidu.com
deny from baiduspider-220-181-7-61.crawl.baidu.com

View 10 Replies


ADVERTISEMENT

.htaccess Mod_rewrite

May 28, 2008

I'm trying to redirect everything except the subdomains from an old domain to a new domain. The following appears to work so far:

Options +FollowSymLinks
RewriteEngine on
RewriteCond %{HTTP_HOST} !^sub1.olddomain.com$ [NC]
RewriteCond %{HTTP_HOST} !^sub2.olddomain.com$ [NC]
RewriteCond %{HTTP_HOST} !^sub3.olddomain.com$ [NC]
RewriteCond %{HTTP_HOST} !^sub4.olddomain.com$ [NC]
RewriteRule ^(.*)$ http://newdomain.com/$1 [R=301,L]

But the subdomains also seem to be loading rather slow. Am I doing anything wrong here? Is there any way to optimise or simplify this (I have around 11 subdomains on the old domain)?

View 1 Replies View Related

Mod_rewrite Rule To Block User Agent

Dec 17, 2008

how can i block this user agent through mod_rewrite
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
and is this agent will affect all the IE 6 users ?
and whats is the difference between the one above and this one :
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; InfoPath.2; .NET CLR 2.0.50727; .NET CLR 3.0.04506.30)?

View 7 Replies View Related

Mod_rewrite Active But Not Reading Htaccess

Jan 18, 2007

although mod_rewrite is active but htaccess i not read why?

and how can i know mod_rewrite is active or not?

(Plesk 7.x)

View 6 Replies View Related

Htaccess Block Spammers

Mar 13, 2007

On my domain access logs, I see a spammer using many different IPs to join my top site list with fake emails. At the end of every line, it contains I am SPAMER! How can I configure htaccess to block this spammer when a request contains that text?

View 3 Replies View Related

Redirect Or Block POST Request With .htaccess

Apr 22, 2009

if there is anyway to block/redirect the POST request to another page using .htaccess ?

The reason is my site is getting hammer with DDOS attack where they target my index page with request like below: ....

View 5 Replies View Related

Apache :: How To Block A Specific (Query String) URL Via HTAccess

Mar 18, 2013

I don't know how to block a specific QUERY STRING url via .htaccess file, well actually I want to block this type of url :

test.php?q=RANDOMTEXT=&tl=The%20path%20ends

View 2 Replies View Related

IIS And Mod_rewrite

Jun 12, 2009

I've just moved a few webistes from Linux to Windows and I'm using IIS to run them. Does anybody know of any free alternatives to ISAPI_REWRITE so I can use mod_rewrite per account?

View 8 Replies View Related

Mod_rewrite

Oct 2, 2007

Its October and I have a Server at home with me running NO CONTROL PANEL... I have been using it for 5 years now... But my problem is my mod_rewrite is not working... I have tried fixing it... But I can't find it in the httpd.conf file where I heard it was at... They said it came by default by Apache..

I'm running Apache2.2.4, Fedora Core 7), PHP 5, MySQL 5.... I don't see the mod_rewrite on there....

View 14 Replies View Related

Mod_rewrite RewriteMap

Dec 15, 2005

I have a list of categories that i want to rewrite the URL to display the names instead.

I'm using this:
RewriteMap catlist txt:C:/web/apache2/htdocs/catnames.txt
RewriteRule ^/cats/(.*)$ /categories.php?catid=${catlist:$1|0}

catnames.txt consists of
CategoryName 6
AnotherCatName 8
OnemoreCatName 9
.....
and so on.

This works fine on my local windows PC since i'm using httpd.conf to set this up.
When i use this on my server under .htaccess, the error log says "RewriteMap" not allowed here.
I it only to find out that i cant use RewriteMap under .htaccess... Is there any other alternative to achieve this without listing all the category names with corresponding rewriteurl statements in my .htaccess file?

View 14 Replies View Related

Installing Mod_rewrite?

Sep 8, 2005

I have installed a Apache server version 1.3.33 on windows XP and would like to enable mod_rewrite so i can rewrite my ugly url´s. How do I install this great feature in Apache so i can test it locally?

Do i just remove # from

#LoadModule rewrite_module modules/mod_rewrite.so
#AddModule mod_rewrite.c

View 2 Replies View Related

Mod_rewrite Tracking

May 28, 2008

I've lots of:

[warn] RewriteCond: NoCase option for non-regex pattern '-f' is not supported and will be ignored.
[warn] RewriteCond: NoCase option for non-regex pattern '-d' is not supported and will be ignored.

Any ideas how to track them?

View 5 Replies View Related

Does Mod_rewrite Consume RAM?

Oct 24, 2009

does mod-rewrite consume ram?

actually found in new vb discussion of seo friendly urls and mod rewrite was an option for most friendly url, so wanted to know if constant usage of mod-rewrite adds additional load to the server or not?

forum which I and my friends use are pretty big more than 6k in a forum and other above 25k, with 250k posts in 1 forum and 600k posts in the other, so just give me an idea on wat can happen due to constant overwritten?

View 8 Replies View Related

Apache And Mod_rewrite

Mar 29, 2009

So I've setup a server heres the spec provided by Webmin

Operating systemUbuntu Linux 8.10

Webmin version1.470

Time on systemSun Mar 29 18:56:12 2009

Kernel and CPULinux 2.6.27-11-server on i686

System uptime5 days, 19 hours, 42 minutes

I'm struggling with mod_rewrite their seems to be no such thing...

Its not mentioned in the apache2.conf or httpd.conf, httpd.conf was blank previously but I added in certain things to get php working but I cannot get mod_rewrite working, which brings the question forward what else am i missing?

I started out with the aim of blocking the ability of viewing directories.

Quote:

Loaded Modules core mod_log_config mod_logio prefork http_core mod_so mod_alias mod_auth_basic mod_authn_file mod_authz_default mod_authz_groupfile mod_authz_host mod_authz_user mod_autoindex mod_cgi mod_deflate mod_dir mod_env mod_mime mod_negotiation mod_php5 mod_setenvif mod_status mod_vhost_alias

So how can I go about getting mod_rewrite setup and/or what ever else I'm missing that I might find essential later on once I have everything setup?

View 4 Replies View Related

Lighttpd Mod_rewrite

May 15, 2009

I am struggling to get this .htaccess file to work with lighttpd, i am not a coder so its very hard for me to fix it, hopefully if someone knows how can tell me how to write rewrite code that will get it working.
{quote}

1. Comment the 2 lines below if the server returns 500 errors!
Options -Indexes
Options +FollowSymLinks

#Uncomment following lines if you want to use image caching!
#<IfModule mod_expires.c>

1. ExpiresActive On
2. ExpiresDefault A1209600
3. ExpiresByType text/html A1
#</IfModule>

1. Uncomment following lines if Apache doesnt support MultiViews!
<IfModule mod_rewrite.c>
RewriteEngine On
1. Uncomment the 2 lines below if you are using www.domain.com # as the baseurl for the site and users access your site # via domain.com (THIS IS REQUIRED FOR JQUERY TO WORK)
#RewriteCond %{HTTP_HOST} ^domain.com [NC]
#RewriteRule ^(.*)$ http://www.domain.com/$1 [L,R=301]

RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule .* loader.php [L,QSA]
</IfModule>

1. Edit below lines and set to
2. ErrorDocument CODE /RELATIVE/error.php
3. If the script is installed in the default document
4. root then relative is null.
#ErrorDocument 401 /error.php
#ErrorDocument 403 /error.php
#ErrorDocument 404 /error.php {/quote}

looks like its calling the rule from loader.php file, now how can i convert this to lighttpd language,

View 4 Replies View Related

Php-cgi Isn't Work With Mod_rewrite

Oct 28, 2009

I have been using apache and php-cgi and it works well together but when I tried to use
mod_rewrite it stopped working and I got "Internal Server Error" result.

Apache error log showed "FastCGI: incomplete headers (0 bytes) received from server"

I think that mod_rewrite rules are ok because when I refer to some HTML file then it all works.

Apache 2.2.14 and php 5.2.11

View 5 Replies View Related

Mod_Rewrite (Enable It)

Mar 23, 2008

I have a VPS and I need to enable mod_rewrite.

Here is my PHP Info: [url]

I read a tutorial but I just don't get it. I would appreciate if you could help me out.

I need it for my Wordpress blog.

View 14 Replies View Related

Mod_Rewrite Subdomain

Aug 31, 2007

I am attempting to redirect any subdomain as follows:

subdomain(dot)domain(dot)com to
domain(dot)com/index.php?user=subdomain

I am having some issues doing this though. None of the samples online seem to work. Do I need a wildcard subdomain? Do I need to do anything to the dedicated linux server I am on? I have other sites on the same server all using mod_rewrite successfully.

Here is what I have so far:

RewriteEngine On
RewriteCond %{HTTP_HOST} !^w{3}. [NC]
RewriteCond %{HTTP_HOST} ^([^.]+).domain.com$ [NC]
RewriteRule ^.*$ /index.php?user=%1 [L]

Is there anything I am missing to get this thing working?

View 3 Replies View Related

PHP-cgi Breaks My Mod_rewrite

Aug 13, 2007

I had the sysadmin install SuPHP - so that my upload script would work properly, however now my mod_rewrite is broken.

You can view the problem here: [url]

It is supposed to show the item with id 29, as you can see. However it shows the gallery index (As if /gallery/ was typed in without the view-29)

The .htaccess file:

Code:
#.htaccess

RewriteEngine On
#This will force trailing slashes
RewriteCond %{SCRIPT_FILENAME} -f [OR]
RewriteCond %{SCRIPT_FILENAME} -d
RewriteRule .* - [L]

#RewriteCond %{REQUEST_URI} !^*(css|png|jpe?g|gif)

RewriteRule ^(.+)/(.+)/?$ $1.php?args=$2 [QSA,L]
RewriteRule ^(.+)$ $1.php [QSA,L,NC]
My very framework relies on the .htaccess working this way... I can turn it off, however I would really rather not - so if you have any idea how I can fix this, please do tell.

And on a related note, is it possible to run PHP under a user and not use cgi_php?

View 14 Replies View Related

Mod_rewrite / Lighttpd

Aug 6, 2007

how can I run a equivalent on lighttpd like this:

# BEGIN WordPress
<IfModule mod_rewrite.c>
RewriteEngine On
RewriteBase /blog/
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule . /blog/index.php [L]
</IfModule>

# END WordPress

View 0 Replies View Related

Url Rewriting Or Mod_rewrite For IIS

Apr 3, 2007

I've got a big .htaccess file which works brilliantly in mod_rewrite for apache. However, the client I want to use it for uses a windows server which is based on IIS.

Does anyone have experience with rewriting urls on IIS. Is it easy, could I easily change this:

RewriteCond %{REQUEST_URI} !^/(.+).php$
RewriteCond %{REQUEST_URI} !^/$
RewriteCond %{REQUEST_URI} !^/products.php$
RewriteCond %{REQUEST_URI} !^/special-offers.php$
RewriteCond %{REQUEST_URI} !^/best-sellers.php$
RewriteCond %{REQUEST_URI} !^/spares.php$
RewriteCond %{REQUEST_URI} !^/images/
RewriteCond %{REQUEST_URI} !^/css/
RewriteCond %{REQUEST_URI} !^/components/
RewriteCond %{REQUEST_URI} !^/Special_Offers
RewriteCond %{REQUEST_URI} !^/Best_Sellers
RewriteCond %{REQUEST_URI} !^/Spares
RewriteRule ^([^/]+)/?$ products.php?show=cats&main_category=$1

Into something that would work. Someone mentioned using asp.net but I've never looked at this before, and can't afford to spend too long this.

Is there a quick/reliable solution (apart from switching webserver).

View 3 Replies View Related

Netfirms Finally Has Mod_rewrite!

May 1, 2006

Hey guys... Not sure if this was posted yet, couldn't find anything on a first-glance search.

Netfirms has finally added mod_rewrite capabilities as well as better .htacces support! Now I'm completely happy. I was considering switching over to Dreamhost due to the fact that I've started running blog software and messy URLs = crap. Now there's no need. That was pretty much the only thing that was holding me back from a full endorsement, so if anyone's looking for a good cheap hosting plan, here's some of the stuff that I'm really happy about for $5 a month:

- quick, personal customer service (I think there's only 1 or 2 guys, and they always seem to be able to answer my questions on the spot)

- mod_rewrite
- .htaccess
- reliable clean FTP (no _vti files or other annoying folders like so many companies these days)
- good online knowledgebase
- nice online site stats viewer
- a bunch of other great stuff that I probably have forgotten about or don't use
- only $5!

Just thought I'd share in case the news hadn't spread yet. Not sure when they added it, but I noticed the "NEW" sticker today.

View 3 Replies View Related

Mod_rewrite And Boolean Logic?

May 2, 2007

If I want to apply a rule if (Condition A and (Condition B OR Condition C)) is true, will the following work?

RewriteCond Condition A
RewriteCond Condition B [OR]
RewriteCond Condition C

Or will the following?

RewriteCond Condition B [OR]
RewriteCond Condition C
RewriteCond Condition A

Or neither? Or both?

View 2 Replies View Related

IIS Equivalent To Apache Mod_rewrite

Jul 14, 2009

I'm moving a framework over to a new server for a client and they're using IIS. Is there an IIS equivalent to the following two lines of mod_rewrite code from Apache?

Code:
RewriteCond %{REQUEST_FILENAME} !-f
RewriteRule ^(.*)$ pamwf.php?PAMWF_PATH_QUERY=$1 [L,QSA]

View 4 Replies View Related

How To Do Mod_rewrite From Web Root To Subfolder

Jun 3, 2009

figure out an .htaccess rewrite rule?

I am trying to rewrite everything in the web root folder to use a /blog/ folder.

So for example, if someone goes to [url] then I need it to actually serve the file from [url]

(But not a redirect.. ideally the browser should still show [url]

Is this possible?

View 1 Replies View Related

Mod_rewrite Strip Off Php Extension

May 15, 2008

I am trying to figure out how to use mod_rewrite to make my php url's friendly. I have read a few articles but none seem to do what I want.

Here is what I want:

/login.php -> /login
/home.php -> /home
/users/update_profile.php -> /users/update_profile
/users/update_profile.php?id=343242 -> /users/update_profile/343242

Basically just strip off the .php extension, so it seems like php files are actually folders.

View 0 Replies View Related

Mod_rewrite - Restart Required

Apr 11, 2008

i have a bunch of rewrite rules that work fine on my dev server but when i upload to production server they dont work. is a restart of apache required?

im on a dedicated box so this shouldnt be a problem, but how do people on shared accounts do this then?

View 1 Replies View Related

Mod_rewrite, Vhosts, And Errorlog

Nov 16, 2007

I tried searching the forums a bit and the Apache manual, but I was unable to find much related to my specific problem.

We have a system such that each user has a 'www' folder in their home directory so that example.com/~user will point to that folder. The user information, however, is not on the www server and instead we use a network file system to access the files in the home folder.

This all works fine and dandy with mod_rewrite. However, we want to allow each user to have access to the error logs, etc. for their own site. The best way to do this seems to simply store it in their home directory.

The problem is that how do you split these files apart. While it would be possible to have a separate process do it, the best way seems to just have each user be their own vhost. mod_vhost_alias seems to be applicable in this situation.

However, mod_vhost_alias relies on either the Host: field or the ip to split queries into their respective vhosts. As said before, our system uses the directory.

So the question is, is there a way to either use mod_rewrite to tell mod_vhost_alias what the vhost's name is. Or, is there a way to split the logs another way.

View 0 Replies View Related

Infinite Loop From Mod_rewrite

Dec 6, 2007

I am having trouble figuring this out.

I am receving this error in my error log:

Code:
[error] [client XX.xxx.XXX.x] Request exceeded the limit of 10 internal redirects due to probable configuration error. Use 'LimitInternalRecursion' to increase the limit if necessary. Use 'LogLevel debug' to get a backtrace.
Now Earlier I had that ip keep accessing my server for an odd reason. So I used mod_rewrite to block its requests.

Code:
RewriteRule file /index.php [R=301,NC]
RewriteRule file2 /index.php [R=301,NC]
RewriteRule file3 /index.php [R=301,NC]
So I am convinced Errors I am getting with these are from bots that are not accepting the redirects?

View 6 Replies View Related

Mod_rewrite To Another Domain On Another Server

Aug 28, 2007

So - I'm trying to do the following: Have multiples domains on multiple servers, with a single server handling them all. I need mod_rewrite to map:
[url] -> [url]

But do it silently - so a [301] Redirect or similar won't do - www.olddomain.com is still the domain I want everyone to use *always*.

I've seen ProxyPass (and mod_proxy) but it appears that these need to be done at an httpd.conf-type level - how can this be achieved with mod_rewrite?

Effectively, I just want *everything* on a particular domain redirected to another domain silently in the background. In actual fact, it'll 99% of the time be on the same server - but I think the principle should be the same anyway.

I've tried:
RewriteEngine on
RewriteRule / [url]

View 14 Replies View Related

Mod_rewrite RegEx Conditionals

Feb 9, 2007

I am using mod_rewrite to rewrite some URL's for a database application. One of my URL's I am rewriting must start (after the base folder) with a folder named after one of the 50 U.S. states (ie. /alabama/, /alaska/, /arizona/, and so on.).

Is it okay to include all 50 of these state names within my RegEx pattern conditional? Or is that way too much info to put into a pattern and be too intensive?

Example:

Code:
^/?(alabama|alaska|arizona|arkansas|california|colorado|connecticut|delaware|florida|georgia|hawaii|idaho|illinois|indiana|iowa|kansas|kentucky|louisiana|maine|maryland|massachussets|michigan|minnesota|mississippi|missouri|montana|nebraska|nevade|new-hampshire|new-mexico|new-york|north-carolina|north-dakota|ohio|oklahoma|oregon|pennsylvania|rhode-island|south-carolina|south-dakota|tennesee|texas|utah|vermont|virginia|washington|west-virigina|wisconsin|wyoming)/([a-zA-Z_]+)/([a-zA-Z_]+)$

View 1 Replies View Related







Copyrights 2005-15 www.BigResource.com, All rights reserved