Do I Really Need A Robots.txt File?
Mar 21, 2008
1 Do I really need a robots.txt file?
2 Don't misbehaved spiders simply ignore them?
3 For 'disallow', shouldn't I only include urls which are linked from public pages - and not those which I use for testing and which aren't linked-to from any public pages?
4 If I include such urls in 'disallow', aren't I simply alerting spiders (and anyone else who wants to see what sections of my server I don't want known) to stuff they'd otherwise not discover?
View 4 Replies
ADVERTISEMENT
Oct 25, 2009
I'm having a issue with my current robots.txt file , which is not properly handling the requests/ blocking the content to be access . What I want is that to only allow like google bots , yahoo , msn , bing , alexander ranking beside those bots block all other bots . my current file rebots.txt is below
Code:
User-agent: Googlebot
Allow: /
User-agent: googlebot-image
Allow: /
User-agent: googlebot-mobile
Allow: /
User-agent: MSNBot
Allow: /
User-agent: Slurp
Allow: /
User-agent: Teoma
Allow: /
User-agent: twiceler
Allow: /
User-agent: Gigabot
Allow: /
User-agent: Scrubby
Allow: /
User-agent: Robozilla
Allow: /
User-agent: Nutch
Allow: /
User-agent: ia_archiver
Allow: /
User-agent: baiduspider
Allow: /
User-agent: naverbot
Allow: /
User-agent: yeti
Allow: /
User-agent: yahoo-mmcrawler
Allow: /
User-agent: psbot
Allow: /
User-agent: asterias
Disallow:
User-agent: yahoo-blogs
Allow: /
View 5 Replies
View Related
Jan 8, 2008
Googlebot is eating up about 30GB bandwidth per month.
I want to prevent him (and other robots) from spidering:
[url]
and all derivatives thereof (e.g. [url])
Is this possible with robots.txt? How would the code look?
Will this work?
User-agent: *
Disallow: /directory1/page1.php
or do I need to somehow specify a wildcard, e.g.
User-agent: *
Disallow: /directory1/page1.php?*
View 1 Replies
View Related
Apr 28, 2009
Do you know what is most important search engine robot ips?
I need to all serach engine robot ips too firewall white list.
View 3 Replies
View Related
Jul 26, 2009
I use Outpost Firewall to view active connections to my server. If I don't restart the httpd service on a regular basis my server will grind to a halt from being flooded by robots.
I currently have the service set up to restart at Midnight and Noon every day. Sometimes that's enough, lately it's not. For example, I checked an hour ago and I had 385 connections to httpd. At least 50% of the connections were robots - tons of the same IP addresses and they're just crawling the site.
Almost all of the connections show up as less than 1kb bytes received and 0 bytes sent per connection.
I already have a good 20 connections by these robots and the connection time shows as 11 minutes... I just browsed to a web gallery page on my site figuring that'd be mildly "intensive" on connections with all the thumbnails and my connections aren't lasting more than one minute.
So, what's with all these connections that are lasting 10+ minutes? I've even got one connection that has an Uptime of 30 minutes, bytes sent 65811, bytes received 180. It seems like something with these robots doesn't terminate correctly...
what to do so these connections quit jamming my server up? It's like a very very slow DOS...
View 3 Replies
View Related
Feb 10, 2015
I'm build Plesk Panel for Linux and Presence Builder, I don't want my user can upload their website to hosting via File Manager. How can I do it...
View 2 Replies
View Related
May 26, 2008
Say I have 2 websites and they all use file.php which is located on mainserver.com/file.php.
I want to use the file like this:
website1.com/file.php
website2.com/file.php
View 2 Replies
View Related
Mar 6, 2008
Rapidly growing error logs showing the same message
$ug-non-zts-20020429/ffmpeg.so' - /usr/local/lib/php/extensions/no-debug-non-zts-20020429//usr/local/lib/php/extensions/no-debug-non-zts-20020429/ffmpeg.so: cannot open shared object file: No such file or directory in Unknown on line 0
root@server [~]# ls /usr/local/lib/php/extensions/no-debug-non-zts-20020429
./ ../ eaccelerator.so*
root@server [~]# ls /usr/local/lib/php/extensions/no-debug-non-zts-20020429
./ ../ eaccelerator.so*
Using cpanel 11 / centos 4
View 1 Replies
View Related
Jun 16, 2008
i have a server with centos,
i need to edit the hidden file .htaccess from the file management tool of cpanel,
but the hidden files not shown,
ow can i modify the setting and let the files shown in the file management tool of cpanel?
View 6 Replies
View Related
Sep 17, 2014
How can we stop Plesk resetting the file permissions on a dll file that is found in
C:Program Files (x86)
??
Specifically,
we have a file, jmail.dll,
here
C:Program Files (x86)Dimacw3JMail
By default Plesk permissions are set to DENY for PSACLN.
But the JMail plugin cannot work with these permissions !
We change this to be ALLOW for READ & EXECUTE and DENY for WRITE.
And everything works fine.
But every time Plesk does an update it reverts it back !
This means that a number of our customers contact forms stop working !
View 12 Replies
View Related
Nov 22, 2008
I'm trying to do this
/usr/bin/gzip -p /home/mysite/public_html/shop/feeds/myfile.xml.zip > /home/mysite/public_html/shop/feeds/myfile.xml
But it just tells me
/usr/bin/gzip: invalid option -- p
X-Powered-By: PHP/5.2.5
Content-type: text/html
How do I find the correct option to unzip first file to second file?
View 8 Replies
View Related
Oct 10, 2014
i manage linux apache webserver with a few wordpress blogs and from time to time i see someone inject a malicious .php file into wp-content/uploads/2014/10/ directory.
i think its some bad plugin or theme, but these is more blogs, i ugrade, update, WP, but
how can i setup some monitor to tell me which php file (or even line in php file) injected that malicious .php ? I have linux root access so i can setup anything
View 3 Replies
View Related
Jul 9, 2007
Today I found some cstomer on the servers make a link for named it file.txt and link it to other customer php file.
so that customer have the ability to show the other custoer file content when visiting the url because it is a text wile originally it is a php file.
the php file was a config file, so now he know the database password , and because he is in the same server he can use that databse.
the question , how to avoide this prolem in the future?
notes , the SuExec is rnning and the open_basedir protection is enabled, but the problem still exists.
View 12 Replies
View Related
Oct 3, 2009
have some E_Books in PDF I want to convert them to word document format
Also need a share hosting for this
View 14 Replies
View Related
Nov 26, 2008
how to put a .FLV format file on a web page with out using the .swf file?
View 4 Replies
View Related
Jul 1, 2009
un-tar big file - big head ache - any way to overcome?
I was un-tarring a file of 6.622gb, the un-tar size should be around 14.4gb I think, most of them are image files .jpg's
and its taking hell a lot of time and not to mention the breaks for un-specified time to process once again,
started decompress around 40-50mins earlier and still the process is on,,.....
using centos5 , ssh - root access
is any there any way I can overcome this problem in future, I can't afford to spend so much time on friends vps just to decompress files?
View 8 Replies
View Related
Jun 12, 2009
Client asked me:
I'd like you " to move my web sites that I have in a FTD file from were it is now to GoDaddy? I have the accounts already set up all that has to happen is the move."
I must confess that I have no clue what he means by FTD file. What am I missing?
BTW, I have Googled it with no results that make sense to me.
View 5 Replies
View Related
Jun 25, 2009
I was playing around with this and that as you do, and somehow changed my /tmp folder from rwxrwxrwt to rwxrwxrwx -
iv tried running chmod 1777 /tmp as someone suggested that but nothing seems to of happened?
View 5 Replies
View Related
Jan 30, 2008
I have a web site backup file (let's call it 'filename.tar.tgz') that was generated from a home-grown web hosting panel and is ~1.6GB in size. It is resident on a WinXP computer, but I also have it copied to a *nix machine.
I have attempted to restore the backup using the normal restore process provided by the site admin panel, but it will never complete because of the size of the file. So, I need to retrieve the folders/files from within the 'filename.tar.tgz' file so that I can re-upload the files/folders through normal FTP.
I have had no success extracting the files/folders when using tar, gtar, gunzip, etc on the Linux box. 7Zip won't open it either. The Linux terminal reports a 'stdin: not in gzip format' error when trying to decompress/extract the files.
What I need is the exact syntax (with any switches) that I can use in my Linux Terminal Shell for extracting this archive so that I can access the files within.
View 8 Replies
View Related
Oct 2, 2007
Where would the my.cnf file be kept for Plesk?
I found one in /etc, but the max_allowed_packet option was there.
View 2 Replies
View Related
Apr 2, 2008
If a hosting company offers both paypal and CC on file (CC on file with auto-subscription/-recurring ability), which payment method do you prefer to use as a customer?
View 14 Replies
View Related
Jan 12, 2008
I've been recently trying to move an account between servers, but the backup file is always incomplete. I was told that it is possible there are too many files.
I decided to tar some of them and move manually, but I cannot access the tar file. I already changed all permissions (644), owner, group, but I still get 403 Forbidden error. Is it possible that the file is too big (9 BGs), and if it is, how do I change the file size limit?
View 14 Replies
View Related
Feb 7, 2007
My server with cPanel, I'd like run file http://domain.com/file.php at 0h00 everyday, I have set the Cron Job in cPanel :
Code:
0 0 * * * /usr/bin/ehpwget http://domain.com/file.php
but The cron is not working well
Code:
/bin/sh: /usr/bin/ehpwget: No such file or directory
Can any one please let me know how to run a php file with cron.
(as user or root)
View 2 Replies
View Related
Apr 29, 2009
How to clear all content in e.g file.txt via ssh(OS is Debian)?
I need a script to merge contents of 2 .txt files, e.g file1.txt and file2.txt to 1 file - file3.txt
View 4 Replies
View Related
Mar 8, 2009
Can i share files from linux to linux machine using SAMBA
View 1 Replies
View Related
Mar 3, 2009
Can i find how much a file consumes bandwidth when the file is executed on the server or when the request is made on the server?
View 6 Replies
View Related
Jun 8, 2009
I think I messed php config and I can't upload anything with php now
Dir is chmoded on 777 and File_Uploads = On in php.ini
I'm running lsphp5 with suhosin, when I try to import db via phpmyadmin I get error: Uploading is not allowed and when I try to upload some file via php script I can't
View 5 Replies
View Related
May 19, 2009
I need to run my PHP application locally on my XP Pro machine with IIS. Part of my application requires creating files, how do I set write permissions on folders like you can with windows server?
Is this possible with XP pro
View 3 Replies
View Related
May 6, 2009
any idea on this error,? my fiend is having pb with fileinfo after his installation
Warning: finfo_open() [function.finfo-open]: Failed to load magic database at '/usr/share/file/magic.mime'. in /home/tomita/public_html/ff/fileinfo.php on line 2
Warning: finfo_file(): supplied argument is not a valid file_info resource in /home/tomita/public_html/ff/fileinfo.php on line 3
Warning: finfo_close(): supplied argument is not a valid file_info resource in /home/tomita/public_html/ff/fileinfo.php on line 4
these are the errors he see when he use the following code
<pre><?php
$finfo=finfo_open(FILEINFO_MIME);
$FT=finfo_file($finfo,'windows2008.png');
finfo_close($finfo);
print_r($FT);
?>
~
View 5 Replies
View Related
Jun 18, 2009
I have a new installation running tomcat in WHM and it works fine for manuallly uploaded files
But the user have uploaded a .war file that supossedly should autoextract but it doesn't
File is located in the public_html folder, is there something additional I should for this to work?
View 1 Replies
View Related
