Need A Host Which Doesn't Throttle Spiders
May 6, 2009it seems like shared hosts throttle spiders, im wondering which hosts will let my spiders run at full speed.
View 11 Repliesit seems like shared hosts throttle spiders, im wondering which hosts will let my spiders run at full speed.
View 11 RepliesAs posted in a previous thread I noted how I have been hosting with GeekStorage. After a little bit of investigation I found that a total of 593 (possibly more) website where being hosted on the same server as mine. I'm not sure if this is overselling but because I'm new to all this but I'm pretty sure it is.
Now, I'm looking on switching from GeekStorage to someone who doesn't oversell. I would be fine with 10-20GB of storage and >150GB bandwidth. Budget is 5-10$ a month.
I found myself writing some filters to monitor spider activity across various sites. Unfortunately it appears that an increasing number of time wasters are setting their User Agent to Googlebot and likes.
Googlebot is pretty easy to filter out as given some research it appears that any legit googlebot IP has a valid .googlebot.com PTR that resolves back to the original IP. It does not appear to be as straight forward with other spiders with msnbot topping the hilarity chart with IPs resolving to meaningless .phx.gbl names.
Is anyone aware of any recommendations or trustworthy references that can be used as guidelines to reliably identify legitimate spider request from fakes?
I'm new to linux and handling my dedicated server. I wanted to know how to setup a bandwidth cap/throttle for the users from my site. I wanted to limit the speeds to 300-400kb/s per user/IP. I've looked around but I haven't found anything really that solves my problem just scripts that protect CPU.
View 3 Replies View RelatedLike for example, recently talked to liquidweb. They have a 100mbs uplink port on their servers, but they only let you use 30% of it. But you can upgrade at anytime. Is that normal for hosts to do that?
View 2 Replies View RelatedcPanel server with EXIM.
Seems the standard feature there - number of emails per hour limit is not working in this case of script emailing.
Any way to limit the number of emails sent via .php scripts or/and throttle this emailing speed cause in the case of spamer the server load hit the roof.
so for over 3 weeks now pc-core.net had some problems with their servers and my accounts were just lost and no one answers my support tickets they just erase them, they disabled my account to support so i can only submit support tickets and click the link on the email.
they told me to email cshelpdesk.com to get my account moved to a new server and cshelpdesk is just telling me im SOL. they dont have access to the server i was at and basically to F off.
is there anything i can do? im basically just being ignored and they dont even have a number to them.
my forum is VB and its doesnt sent any email for users!
i check the options and everything looks fine ..
my vps had csf firewall installed ..
the csf.conf is
# Lists of ports in the following comma separated lists can be added using a
# colon (e.g. 30000:35000).
# Allow incoming TCP ports
TCP_IN = "53,1116,80"
# Allow outgoing TCP ports
TCP_OUT = "53,80,3306,1116"
# Allow incoming UDP ports
UDP_IN = "1116,53"
# Allow outgoing UDP ports
# To allow outgoing traceroute add 33434:33523 to this list
UDP_OUT = "53,1116"
# Allow incoming PING
ICMP_IN = "0"
# Set the per IP address incoming ICMP packet rate
# To disable rate limiting set to "0"
ICMP_IN_RATE = "1/s"
# Allow outgoing PING
ICMP_OUT = "1"
# Set the per IP address outgoing ICMP packet rate
# To disable rate limiting set to "0"
ICMP_OUT_RATE = "1/s"
and the version of csf is
[root@www csf]# /usr/sbin/csf -v
csf: v4.56 (generic)
[root@www csf]#
am useing centOS5 and ssh port 1116
am not useing any of control panel or ftp ...
the email i configured it with google.com/a services ..
my sys admin is currently in vacation and ovh staff cant help me much so i really appreciate if someone could help me tackle down this issue. I cant create ftp account at all heres what ovh said
Good morning,
Thank you to trust us, there is not my point of view or that of my colleague problem of the kernel compilation on your server or a file corrupted, and your Apache web server is functional, Status of the launch; after this command, via ssh:
/ etc / init.d / httpd start
Starting httpd: [OK]
We see that the port is open 80:
netstat-tanpu | grep 80
tcp 0 0 0.0.0.0:7080 0.0.0.0: * LISTEN 3071/lshttpd
tcp 0 0 0.0.0.0:80 0.0.0.0: * LISTEN 14349/httpd
From the outside, at this precise moment, we are able to establish a connection on port 80:
Telnet ks357687.kimsufi.com 80
Trying 91.121.148.X ...
Connected to ks357687.kimsufi.com.
Escape character is'^]'.
Get
<! DOCTYPE HTML PUBLIC "- / / IETF / / DTD HTML 2.0 / / EN">
<html> <head>
<title> 501 Method Not Implemented </ title>
</ head> <body>
<h1> Method Not Implemented </ h1>
<p> get to / not supported. <br />
</ p>
<hr>
<address> Apache/2.2.8 (EL) Server at ks357687.kimsufi.com Port 80 </ address>
</ body> </ html>
Connection closed by foreign host.
If I stop the Apache server:
/ etc / init.d / httpd stop
Stopping httpd: [OK]
More than 80 active port or connection possible from the outside:
In this case the connection of the exterior evidemment is not possible:
Telnet ks357687.kimsufi.com 80
Trying 91.121.148.X ...
telnet: Unable to connect to remote host: Connection refused
I am using Windows 2003 Enterprise Edition SP1 and i have recently set the computer up to 4GB RAM. I notice a problem occur error :
When I start the computer, Bios detected all 4GB Ram. However, i have checked Total physical memory in General (My Computer-> Properties) doesnot detect all 4GB Ram. it only detect 3GB Ram.
I have checked that this OS support up to 32GB . Why it doesn't detect all 4GB ?
What happen to it?
Mainboard : Intel chipset 915GL
I did following the instruction in internet (exactly is Microsoft) is /PAE in boot.ini file. But, it doesnt work.
Yesterday we registerd and created 2 domains in our server.
Everything is ok, files uploaded, dns zones created, httpd virtual host created...
The problem is that trying to browse the site trough http://www.site.com redirects me to [url] my main server IP address, this page shows a cPanel Welcome message.
But, using [url], the site is loaded ok, withouth any problems.
DNS zone shows:
Code:
; cPanel 11.11.0-BETA_16977
; Zone file for site.com
$TTL 14400
@ 86400 IN SOA mydns1.wolo.com. admin.hostingserver.com. (
2007092601 ; serial, todays date+todays
86400 ; refresh, seconds
7200 ; retry, seconds
3600000 ; expire, seconds
86400 ) ; minimum, seconds
site.com. 86400 IN NS mydns1.wolo.com.
site.com. 86400 IN NS mydns2.wolo.com.
site.com. IN A IP_ADDRESS
localhost.site.com. IN A 127.0.0.1
site.com. IN MX 0 site.com.
mail IN CNAME site.com.
www IN A IP_ADDRESS
ftp IN A IP_ADDRESS
Apache virtual host code
Code:
<VirtualHost IP_ADDRESS>
ServerAlias www.site.com
ServerAdmin webmaster@site.com
DocumentRoot /home/siteuser/public_html
ServerName site.com
<IfModule mod_suphp.c>
suPHP_UserGroup siteuser siteuser
</IfModule>
<IfModule mod_php4.c>
php_admin_value open_basedir "/home/siteuser:/usr/lib/php:/usr/local/lib/php:/tmp"
</IfModule>
<IfModule mod_php5.c>
php_admin_value open_basedir "/home/siteuser:/usr/lib/php:/usr/local/lib/php:/tmp"
</IfModule>
User siteuser
Group siteuser
BytesLog /usr/local/apache/domlogs/site.com-bytes_log
CustomLog /usr/local/apache/domlogs/site.com combined
Options -ExecCGI -Includes
</VirtualHost>
The email of my client doesnt receive certain emails from certain accounts, including my yahoo.com.ph account. After several tries I received this error instead.
After trying to send mail unsuccedfully several times, this is the error I am getting.
Hi. This is the qmail-send program at yahoo.com.
I'm afraid I wasn't able to deliver your message to the following
addresses.
This is a permanent error; I've given up. Sorry it didn't work out.
:
Sorry, I couldn't find any host by that name. (#4.1.2)
I'm not going to try again; this message has been in the queue too
long.
--- Below this line is a copy of the message.
Quote:
Originally Posted by copy of email
Return-Path:
Received: (qmail 90296 invoked by uid 60001); 25 Jun 2007 04:05:27
-0000
DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws;
s=s1024; d=yahoo.com;
h=X-YMail-OSG:Receivedate:Fromubject:To:MIME-Version:Content-Type:Content-Transfer-Encoding:Message-ID;
b=GYMZ4utbLEJMc+EXHdW5Ng5ZJeCArXAubpEaUtZzZz77STh9HAGNhrFDjgfqNJ5BdE/
SAlCQ78wseeWPTqUFlalE246OSls0L2tnadTvmxHAQiALfIJ/efHw980subD/VVk6c3NZXGKaKn3vQwJ57bHW5a6qsWjazjl6BXp5Q20=;
X-YMail-OSG:
1iaqBZkVM1mfqTKSQPqhJwf61.zXtBuozzwn.p275yXXF_KgFA8taeSyThc.P1xawEKTuVIB
Vgxk18AHBp_TxSZD753CsOSTbU56JD1OgWwBl8GCwZNxI6YhJHPfyg--
Received: from [203.177.91.252] by web32604.mail.mud.yahoo.com via
HTTP; Sun, 24 Jun 2007 21:05:26 PDT
Date: Sun, 24 Jun 2007 21:05:26 -0700 (PDT)
From: *********
Subject: test
To: sales@*********.com
MIME-Version: 1.0
Content-Type: multipart/alternative;
boundary="0-357102911-1182744326=:89673"
Content-Transfer-Encoding: 8bit
Message-ID: <979097.89673.qm@web32604.mail.mud.yahoo.com>
--0-357102911-1182744326=:89673
Content-Type: text/plain; charset=iso-8859-1
Content-Transfer-Encoding: 8bit
test
After migrating from the old ensim to the new "X", everything is working fine, except for the webalizer, I activated webalizer in one domain a few days ago.. and everyday Im trying to see the webalizer information, and it is always displaying this message:
Could not retrieve the requested site report because the Web server logs have not yet been generated. These are generated once a day. Please check again later.
I also tried forcing the log rotating for that site, but the problem still there.
Server is RHES 4 with EnsimPro X, log generation and webalizer is activated for the domain.
I configured monit to send me email alerts when apache is using too many resources but I cannot receive email alerts. I use Qmail.
As for syslog, there is this message:
monit[6473]: Sendmail: error receiving data from the mailserver 'localhost' -- Resource temporarily unavailable
I tried to add for server name the name used in MX record but with the same result.
Btw: other things configured in cron (eg. RKHUNTER) send me alerts OK.
im using centos and a perl script doesnt seem to work on my system.
it gives an error on this line.
use LWP::UserAgent;
$ua = LWP::UserAgent->new;
$ua->agent("MyApp/0.1 ");
BEGIN failed--compilation aborted at line 8.
is it a problem with my perl or is it an error with the script?
I'm on windows vista. I've got an SDSL line into the house with a dedicated IP 62.etc. I've then got a Prestige 791R and all PC's plugged in to that. I've also got a wireless router for my Wii set up.
Basically I've set up this DHCP thingy to start at 192.etc.3 and the Wireless router is at 192.etc.1 and the Prestige is at 192.etc.2 my pc is at 192.etc.9
This is about where my knowledge stops unfortunately.
I have installed apache as the most basic default install I think you can and it seems to be working absolutely fine. I'm assuming I could also install PHP and MySQL etc and wouldn't have too much of a problem, however, my friend cannot seem to access the environment from the WAN IP and when I type in my WAN IP I get the prestige router's control panel.
I read somewhere that I needed to forward port 80, now I can 'open' ports in the prestige control panel but I did try opening port 80 in the control panel and pointing it to my LAN IP (92.etc.9) but it didn't seem to work.
I'm at a bit of a loss so if anyone could point me in the right direction I would be most grateful, just to clarify:
//localhost loads the html file 100%
//192.etc.9 loads the html file 100%
//62.WANIP internally loads my prestige routers control panel
//62.WANIP externally does not load a page (cannot be found)
If you need any more info just let me know and I'll get it.
I have a remote XP PC: 172.16.1.5 OpenVPN connection with route added for 192.168.2.0/24 to go via the VPN
Now on the other end the network consists of:
I have a OpenVPN server inside the lan on 192.168.2.245
Its default gateway is 192.168.2.1
I have 3 Windows Servers, 192.168.2.246, 247 and 248. All gateways are set to 192.168.2.1.
I have a ethernet router on the network, 192.168.2.1, it has a route added for 192.168.0.0/16 to go via 192.168.2.245, and route added for 172.16.0.0/24 to go via 192.168.2.245 also). the 192.168.0.0/16 is incase any other LANs are connected at a later date, if computers saw any packets not on the the 192.168.2.x subnet they would be routed to the default gateway which would then pass them to the OpenVPN router.
The trouble is, i can remotely connect and ping to the OpenVPN router fine and also the ethernet router, however when i ping any of the Windows boxes it times out. But i can open up Remote Desktop and connect to the windows box without problem, infact if i am running 'ping 192.168.2.246 -t' it will suddenly come alive but only after the RDP connection is made.
Is this something funny with the routing? I want to keep the OpenVPN server internal to the network and i appreciate it is hitting the ethernet router then being passed to the OpenVPN but something is weird as it fires up RDP fine but not ping. No firewalls are enabled on any of the boxes. If i log into ssh on the OpenVPN router or ethernet router i can ping from that to the windows boxes fine.
Its as if a ICMP redirect is issued, and all is well after the 1st connection. Not too sure but could anyone be kind enough to enlighten me?
My server which runs cPanel (license issued touchsupport) is having problems with handling emails.
I use an external DNS provider (editdns.net) only for this 1 particular domain for handling many Records and such. Through their panel, I have added my mail server's record, i.e. "mailserver.tld".
Now there is a problem with one part of the email issue. I have email piping setup and 1 webmail account setup too. These two work flawlessly. No issues with them. But email forwarding and default address doesn't work.
I am confident that the email settings are working right because, when my clients send an email, piping works and my support system creates a new ticket. But the email forwarding/Default EMail doesn't.
Not just that, I cannot even connect to the webmail account I created via POP3. I 100% sure that I used the correct login credentials and also tried the "+" and ":" for the username part. Just doesn't work. But I can access and read my emails via the Webmail (squirrelmail).
Please help me out with this as soon as possible.
to upload some flash videos over to our web host.
All the videos have been uploaded to << link removed >> and in there you will seea flash_video directory, in there should be a html file where you can double click and watch the flash video.
Now go ahead and try it, click on the HTML file and click CONTINUE and try clicking on MODULE 1.. You will see a video play to the left but on the right are bunch of POWERPOINT slides that will appear as the guy continues to speak. THAT DOESNT SHOW..
Our website is hosted by xo.com
Dont know much about the plan since its my 2nd day at the job..
But here is the weird part,
i have uploaded the samething - same exact way to another web host and it plays fine, the PPTs show up fine..
What do you guys think it is?
Before someone asks about uploading methods, i tried filezilla, coreftp and cuteFTP using both ASCII and binary methods. Samething..
I'm configuring a website for a client who has moved their web hosting to downtownhost, but is keeping their email hosting with their current provider. They do not want to configure an MX entry on DTH to autoforward email back to their current provider because they don't want their email to pass through DTH.
In order to do this, do I keep the nameserver entries on the current provider the same, and configure an http redirect to point to DTH? Or is there something else I should configure on the current provider?
I'm wondering if you can tell me something about your experience with a host/server in a different country or overseas.
My webpage will be mostly frequented by Australians.
What do you think is the advantage and the disadvantage of having a host in your own country and having one overseas?
I'm thinking about stability, traffic speed, peak hours in different time zones, customer service, ....
What are your experiences?
I was with Blue Host and their support and service was pretty bad. Servers going down all the time, and chat support was terrible. Then I moved to Host Monster and received the same kind of service/support. I later then figured out that Blue Host and Host Monster is runned by the same people. For example, I opened up a tech support ticket with Host Monster and then they replied signing their signature with Blue Host. So I got the same service: terrible tech support and servers kept going down. Now I'm looking for a new web hosting. I a few people here gave Host Gator some good reviews. I hope Host Gator is not run be the same people that runs Blue Host and Host Monster.
View 15 Replies View RelatedDoes anyone have a preference for Host Excellence compared to Blue Host?
Do you think their basic offers are the same?
This is for just a couple of simple domains and sites to be hosted.
with this being my first website goddady's website tonight or there blog site plan seem pretty good as I do not know how to build a professional looking website nor do I have a knowledge of HTML. Are there other hosts that offer better services such as these?
Also I would want to register with 1&1 as to register privately is free but with godaddy it costs around $15 and I've heard of godaddy stealing persons domains when they haven't even expired yet i.e. familyAlbum.com?
Is it allowed for instance to register my domain name with 1&1 and host my site with godaddy?
If you had to chose either Host Gator or Hawk Host which would you go with and why? Don't bring any other hosts into consideration just compare these two. This would be for a standard website, forum, and a few downloads if it makes a difference...nothing too fancy.
View 14 Replies View RelatedI have just made a personal website using photoshop, html, css and php. My site contains an index page, a gallery page with 8 photos, a contact page with a form and an extra page with some quotes. I am planning to use a paid host for the first time. Based on feedbacks from some of my sitepoint friends, I decided to go in for shared hosting. I thought of using Hostdogs.com as they have the option of a monthly billing cycle. I thought that I have the option of changing my host after a month if I am not happy with them. I have also read a lot about Hostgator.com. A lot of sitepoint members seem hooked to it.
1. Can anyone give an opinion as to which host I should go in for? Hostdogs also give a free domain. Is there any reason I should go in for Hostgator instead of Hostdogs?
2. One more question, I have heard that once I have a site running, I can earn money through the advertisements placed on it? How do I place advertisements on my site?
3. One last and very important question is whether it is right to go in for shared hosting at this point of time. I just read this
Quote:
When you look for a suitable host, be sure to check out the amount of bandwidth that will be available to your site. Even a site that doesn't require a great deal of data transfer per month could run slowly if you're hosted on a shared server; particularly if you are on the same server as a more popular or bandwidth-intensive site.
The more sites hosted on a server, the more likely they are to compete with each other for available resources, including bandwidth. This is why you may notice a reduction in server response times during busy periods, or during a sudden peak in traffic at a busier site on a server that you share.
This made me wonder whether I should go in for shared hosting. But then mine is a personal home page which may not have so much of traffic.
4. What does the paragraph below mean? How does one handle ones registration manually?
Self-Management (no cost)
Select this option if you prefer to handle your registration manually.Your web host can establish service with any Top-Level Domain (TLD), international or domestic.
I will be setting up a site which uses ASP.NET 1.1. I know versions 3.0/3.5 are available, but they are not yet V2.0 compliant. Anyway thst is not the issue - just context.
Would like your advice as to whether it would be better to host this site with a company who looks as though they are specialising in ASP; such as DiscountASP, or Softsyhosting, or go with a general Windows hoster such as Steadfast, Fluidhosting, or 3Essentials?
I know nothing about ASP, so I don't know whether it is not at all neccessary to have a host specifically knowledgable about it, as I am unlikely to ever have problems with it, OR; I will most certainly have to address ASP issues sometime, so it would be essential to have support who really knows ASP.
I got the beta of vis studio 2005 about a month ago and happily started coding away. I recently told my host to convert to asp.net 2.0 so I could upload the stuff and have it work and he said he couldnt until the non beta comes out.
Is there anyone out there at the moment who will host asp.net 2.0?
company that will do Web Hosting along with FTP hosting in 1 package?
Looking for full support for SQL, PHP, etc.
FTP Support would be for 2 people in different states to share files for work.
I just want to ask the community of WHT a series of questions about shared and reseller hosting and what you would like out of a shared/reseller host.
What do you look for in a web host (in terms of shared and reseller hosting).
Do you prefer cheap "unlimited" and "unmetered" plans?
Would you like support through live chat, phone, or helpdesk?
What kind of price range would you consider reasonable for a shared hosting plan? a reseller plan?
Would a community forum with support staff and other clients be a plus to you?
Does the site's design and the software it utilizes be a concern to you?
In your opinion, what would make you choose a specific web hosting company over any other?
Anyone have a recent bad experience with 247-host.com?
I'm considering moving my website there.