Robots.txt File

Oct 25, 2009

I'm having a issue with my current robots.txt file , which is not properly handling the requests/ blocking the content to be access . What I want is that to only allow like google bots , yahoo , msn , bing , alexander ranking beside those bots block all other bots . my current file rebots.txt is below

Code:

User-agent: Googlebot

Allow: /

User-agent: googlebot-image

Allow: /

User-agent: googlebot-mobile

Allow: /

User-agent: MSNBot

Allow: /

User-agent: Slurp

Allow: /

User-agent: Teoma

Allow: /

User-agent: twiceler

Allow: /

User-agent: Gigabot

Allow: /

User-agent: Scrubby

Allow: /

User-agent: Robozilla

Allow: /

User-agent: Nutch

Allow: /

User-agent: ia_archiver

Allow: /

User-agent: baiduspider

Allow: /

User-agent: naverbot

Allow: /

User-agent: yeti

Allow: /

User-agent: yahoo-mmcrawler

Allow: /

User-agent: psbot

Allow: /

User-agent: asterias

Disallow:

User-agent: yahoo-blogs

Allow: /

View 5 Replies


ADVERTISEMENT

Do I Really Need A Robots.txt File?

Mar 21, 2008

1 Do I really need a robots.txt file?

2 Don't misbehaved spiders simply ignore them?

3 For 'disallow', shouldn't I only include urls which are linked from public pages - and not those which I use for testing and which aren't linked-to from any public pages?

4 If I include such urls in 'disallow', aren't I simply alerting spiders (and anyone else who wants to see what sections of my server I don't want known) to stuff they'd otherwise not discover?

View 4 Replies View Related

Robots.txt With Wildcard

Jan 8, 2008

Googlebot is eating up about 30GB bandwidth per month.
I want to prevent him (and other robots) from spidering:

[url]

and all derivatives thereof (e.g. [url])

Is this possible with robots.txt? How would the code look?

Will this work?
User-agent: *
Disallow: /directory1/page1.php

or do I need to somehow specify a wildcard, e.g.
User-agent: *
Disallow: /directory1/page1.php?*

View 1 Replies View Related

Google And Yahoo Robots Ips

Apr 28, 2009

Do you know what is most important search engine robot ips?

I need to all serach engine robot ips too firewall white list.

View 3 Replies View Related

Robots Flood My Server

Jul 26, 2009

I use Outpost Firewall to view active connections to my server. If I don't restart the httpd service on a regular basis my server will grind to a halt from being flooded by robots.

I currently have the service set up to restart at Midnight and Noon every day. Sometimes that's enough, lately it's not. For example, I checked an hour ago and I had 385 connections to httpd. At least 50% of the connections were robots - tons of the same IP addresses and they're just crawling the site.

Almost all of the connections show up as less than 1kb bytes received and 0 bytes sent per connection.

I already have a good 20 connections by these robots and the connection time shows as 11 minutes... I just browsed to a web gallery page on my site figuring that'd be mildly "intensive" on connections with all the thumbnails and my connections aren't lasting more than one minute.

So, what's with all these connections that are lasting 10+ minutes? I've even got one connection that has an Uptime of 30 minutes, bytes sent 65811, bytes received 180. It seems like something with these robots doesn't terminate correctly...

what to do so these connections quit jamming my server up? It's like a very very slow DOS...

View 3 Replies View Related

Plesk 12.x / Linux :: Deny User Upload File Via File Manager Or Hidden File Tab?

Feb 10, 2015

I'm build Plesk Panel for Linux and Presence Builder, I don't want my user can upload their website to hosting via File Manager. How can I do it...

View 2 Replies View Related

1.com/file.php, 2.com/file.php Where File.php Is Hosted On Main.com/file.php

May 26, 2008

Say I have 2 websites and they all use file.php which is located on mainserver.com/file.php.

I want to use the file like this:
website1.com/file.php
website2.com/file.php

View 2 Replies View Related

Ffmpeg :: Cannot Open Shared Object File: No Such File Or Directory In Unknown On Line 0

Mar 6, 2008

Rapidly growing error logs showing the same message

$ug-non-zts-20020429/ffmpeg.so' - /usr/local/lib/php/extensions/no-debug-non-zts-20020429//usr/local/lib/php/extensions/no-debug-non-zts-20020429/ffmpeg.so: cannot open shared object file: No such file or directory in Unknown on line 0

root@server [~]# ls /usr/local/lib/php/extensions/no-debug-non-zts-20020429
./ ../ eaccelerator.so*
root@server [~]# ls /usr/local/lib/php/extensions/no-debug-non-zts-20020429
./ ../ eaccelerator.so*

Using cpanel 11 / centos 4

View 1 Replies View Related

How Can I Show Hidden File In Cpanel File Management Tool

Jun 16, 2008

i have a server with centos,

i need to edit the hidden file .htaccess from the file management tool of cpanel,

but the hidden files not shown,

ow can i modify the setting and let the files shown in the file management tool of cpanel?

View 6 Replies View Related

Plesk 12.x / Windows :: STOP Resetting File Permissions On DLL File?

Sep 17, 2014

How can we stop Plesk resetting the file permissions on a dll file that is found in

C:Program Files (x86)

??

Specifically,

we have a file, jmail.dll,

here

C:Program Files (x86)Dimacw3JMail

By default Plesk permissions are set to DENY for PSACLN.

But the JMail plugin cannot work with these permissions !

We change this to be ALLOW for READ & EXECUTE and DENY for WRITE.

And everything works fine.

But every time Plesk does an update it reverts it back !

This means that a number of our customers contact forms stop working !

View 12 Replies View Related

/usr/bin/gzip -p Correct Option To Unzip First File To Second File?

Nov 22, 2008

I'm trying to do this

/usr/bin/gzip -p /home/mysite/public_html/shop/feeds/myfile.xml.zip > /home/mysite/public_html/shop/feeds/myfile.xml

But it just tells me

/usr/bin/gzip: invalid option -- p
X-Powered-By: PHP/5.2.5
Content-type: text/html

How do I find the correct option to unzip first file to second file?

View 8 Replies View Related

Apache :: How To Discover Which Php File Allows Malicious File Upload

Oct 10, 2014

i manage linux apache webserver with a few wordpress blogs and from time to time i see someone inject a malicious .php file into wp-content/uploads/2014/10/ directory.

i think its some bad plugin or theme, but these is more blogs, i ugrade, update, WP, but

how can i setup some monitor to tell me which php file (or even line in php file) injected that malicious .php ? I have linux root access so i can setup anything 

View 3 Replies View Related

Links Files In Linux (file.txt For File.php)

Jul 9, 2007

Today I found some cstomer on the servers make a link for named it file.txt and link it to other customer php file.

so that customer have the ability to show the other custoer file content when visiting the url because it is a text wile originally it is a php file.

the php file was a config file, so now he know the database password , and because he is in the same server he can use that databse.

the question , how to avoide this prolem in the future?

notes , the SuExec is rnning and the open_basedir protection is enabled, but the problem still exists.

View 12 Replies View Related

How Can I Convert A PDF File To A WORD Doc. File

Oct 3, 2009

have some E_Books in PDF I want to convert them to word document format

Also need a share hosting for this

View 14 Replies View Related

Put A .FLV Format File On A Web Page With Out Using The .swf File?

Nov 26, 2008

how to put a .FLV format file on a web page with out using the .swf file?

View 4 Replies View Related

Un-tar Big File

Jul 1, 2009

un-tar big file - big head ache - any way to overcome?

I was un-tarring a file of 6.622gb, the un-tar size should be around 14.4gb I think, most of them are image files .jpg's

and its taking hell a lot of time and not to mention the breaks for un-specified time to process once again,

started decompress around 40-50mins earlier and still the process is on,,.....

using centos5 , ssh - root access

is any there any way I can overcome this problem in future, I can't afford to spend so much time on friends vps just to decompress files?

View 8 Replies View Related

FTD File

Jun 12, 2009

Client asked me:

I'd like you " to move my web sites that I have in a FTD file from were it is now to GoDaddy? I have the accounts already set up all that has to happen is the move."

I must confess that I have no clue what he means by FTD file. What am I missing?

BTW, I have Googled it with no results that make sense to me.

View 5 Replies View Related

/tmp File

Jun 25, 2009

I was playing around with this and that as you do, and somehow changed my /tmp folder from rwxrwxrwt to rwxrwxrwx -

iv tried running chmod 1777 /tmp as someone suggested that but nothing seems to of happened?

View 5 Replies View Related

.tar.tgz File

Jan 30, 2008

I have a web site backup file (let's call it 'filename.tar.tgz') that was generated from a home-grown web hosting panel and is ~1.6GB in size. It is resident on a WinXP computer, but I also have it copied to a *nix machine.

I have attempted to restore the backup using the normal restore process provided by the site admin panel, but it will never complete because of the size of the file. So, I need to retrieve the folders/files from within the 'filename.tar.tgz' file so that I can re-upload the files/folders through normal FTP.

I have had no success extracting the files/folders when using tar, gtar, gunzip, etc on the Linux box. 7Zip won't open it either. The Linux terminal reports a 'stdin: not in gzip format' error when trying to decompress/extract the files.

What I need is the exact syntax (with any switches) that I can use in my Linux Terminal Shell for extracting this archive so that I can access the files within.

View 8 Replies View Related

Where Is The My.cnf File Kept For Plesk?

Oct 2, 2007

Where would the my.cnf file be kept for Plesk?

I found one in /etc, but the max_allowed_packet option was there.

View 2 Replies View Related

PayPal Or CC On File?

Apr 2, 2008

If a hosting company offers both paypal and CC on file (CC on file with auto-subscription/-recurring ability), which payment method do you prefer to use as a customer?

View 14 Replies View Related

403 Forbidden / File Too Big?

Jan 12, 2008

I've been recently trying to move an account between servers, but the backup file is always incomplete. I was told that it is possible there are too many files.

I decided to tar some of them and move manually, but I cannot access the tar file. I already changed all permissions (644), owner, group, but I still get 403 Forbidden error. Is it possible that the file is too big (9 BGs), and if it is, how do I change the file size limit?

View 14 Replies View Related

Cron: How To Run Php File

Feb 7, 2007

My server with cPanel, I'd like run file http://domain.com/file.php at 0h00 everyday, I have set the Cron Job in cPanel :

Code:
0 0 * * * /usr/bin/ehpwget http://domain.com/file.php
but The cron is not working well

Code:
/bin/sh: /usr/bin/ehpwget: No such file or directory

Can any one please let me know how to run a php file with cron.
(as user or root)

View 2 Replies View Related

Clear The File

Apr 29, 2009

How to clear all content in e.g file.txt via ssh(OS is Debian)?

I need a script to merge contents of 2 .txt files, e.g file1.txt and file2.txt to 1 file - file3.txt

View 4 Replies View Related

File Sharing

Mar 8, 2009

Can i share files from linux to linux machine using SAMBA

View 1 Replies View Related

File Bandwidth

Mar 3, 2009

Can i find how much a file consumes bandwidth when the file is executed on the server or when the request is made on the server?

View 6 Replies View Related

PHP File Upload

Jun 8, 2009

I think I messed php config and I can't upload anything with php now
Dir is chmoded on 777 and File_Uploads = On in php.ini

I'm running lsphp5 with suhosin, when I try to import db via phpmyadmin I get error: Uploading is not allowed and when I try to upload some file via php script I can't

View 5 Replies View Related

File Permissions With XP Pro And IIS

May 19, 2009

I need to run my PHP application locally on my XP Pro machine with IIS. Part of my application requires creating files, how do I set write permissions on folders like you can with windows server?

Is this possible with XP pro

View 3 Replies View Related

File Info

May 6, 2009

any idea on this error,? my fiend is having pb with fileinfo after his installation

Warning: finfo_open() [function.finfo-open]: Failed to load magic database at '/usr/share/file/magic.mime'. in /home/tomita/public_html/ff/fileinfo.php on line 2

Warning: finfo_file(): supplied argument is not a valid file_info resource in /home/tomita/public_html/ff/fileinfo.php on line 3

Warning: finfo_close(): supplied argument is not a valid file_info resource in /home/tomita/public_html/ff/fileinfo.php on line 4

these are the errors he see when he use the following code

<pre><?php
$finfo=finfo_open(FILEINFO_MIME);
$FT=finfo_file($finfo,'windows2008.png');
finfo_close($finfo);
print_r($FT);
?>
~

View 5 Replies View Related

WAR File Not Self Extracting

Jun 18, 2009

I have a new installation running tomcat in WHM and it works fine for manuallly uploaded files

But the user have uploaded a .war file that supossedly should autoextract but it doesn't

File is located in the public_html folder, is there something additional I should for this to work?

View 1 Replies View Related







Copyrights 2005-15 www.BigResource.com, All rights reserved