How To Filter Noisy Rows From Raw Log Files
Aug 31, 2008
in order to analyse traffic from a specific application I'm using, I need to filter out some types of server requests from the raw log files.
the reason: this app makes multiple requests to the same content in 6 sec intervals.
so I want to leave only the first request, that indicated that this item has been requested, and take out all the others, which can't be counted as hits or visits and just create noise data.
I know how to define a "relevant" request, and how to define a "noisy" request.
the question:
how can I make this filtering?
Do I need to run a script on the log files, clean them, and then use the log analyzer (i'm using Web log explorer)?
Or can i use web log explorer to define a filtering template?
this is a very critical demand, so i'd be glad for any suggestions/ideas how to tackle this problem.
View 0 Replies
ADVERTISEMENT
Mar 5, 2008
I want to filter any files uploaded and i have put this line at php.ini
suhosin.upload.verification_script = /my path
But my problem till now i can not make this script. (Disable upload php files)
View 1 Replies
View Related
Jan 21, 2007
I would like to know how to filter more than one NIC with APF firewall... at the moment, it only let you filter Eth0, how do I add Eth1 in it as well?
View 7 Replies
View Related
Jan 22, 2009
I am getting a VPS and the only mail filter option available with this company is SpamAssassin.
Is Spam Assassin's version 3.2.x sufficient to block majority of Spam/Junk emails?
View 9 Replies
View Related
Apr 23, 2008
As a web host or ISP what do you think is best to do? do spam/virus filtering, or don't touch mail and let the user do what they want with it? (no chance of false positives, or lost mail, this way)
I work for an ISP and we have a barracuda and we get tons of calls from customers regarding lost mail and such as when you're filtering such a large varitey of mail its very hard to have a "perfect" filter. Mail that a car dealership gets and what a hospital get is totally different, for example.
Also in terms of web hosting the filtering will put a reasonable load on the server during peak spamming hours.
So just curious, as a ISP/webhost customer do you think your host should do filter or do you rather manage that yourself?
View 10 Replies
View Related
Mar 4, 2007
I sent the below email, it keeps getting caught in the filter
Quote:
Maria,
Here are the username and password for this site.
www.consumerreports.com [url]
user:
pass:
Maybe you can find some more info about the cars you like.
Thanks,
Matthew
Sitebuilder Specialist
[800.729.xxxx ext xxx] or [xxx.625.8546]
email malberty@domain.com
www.anotherdomain.com
Support Contact for any Stewart Related Website:
[url]<[url]
The exim logs show this:
2007-03-04 21:02:53 1HO3TR-0003tu-Ek <= matthew@localdomain.com H=(localhost) [127.0.0.1]:43496 I=[127.0.0.1]:25 P=esmtpa A=fixed_login:matthew@localdomain.com S=1527 id=9bb87bf2893bf84500323930692044ef@localhost T="test" from <matthew@localdomain.com> for matthew@localdomain.com
2007-03-04 21:02:53 cwd=/var/spool/exim 3 args: /usr/sbin/exim -Mc 1HO3TR-0003tu-Ek
2007-03-04 21:02:53 1HO3TR-0003tu-Ek => discarded (system filter)
2007-03-04 21:02:53 1HO3TR-0003tu-Ek Completed QT=0s
2007-03-04 21:02:53 SMTP connection from (localhost) [127.0.0.1]:43496 I=[127.0.0.1]:25 lost
I cannot figure out why the filters is picking this up, nor which WORD is causing this. How would I trace this?
my antivirus.exim file
Code:
# Exim filter
## Version: 0.17
#$Id: system_filter.exim,v 1.11 2001/09/19 11:27:56 nigel Exp $
## Exim system filter to refuse potentially harmful payloads in
## mail messages
## (c) 2000-2001 Nigel Metheringham <nigel@exim.org>
##
## This program is free software; you can redistribute it and/or modify
## it under the terms of the GNU General Public License as published by
## the Free Software Foundation; either version 2 of the License, or
## (at your option) any later version.
##
## This program is distributed in the hope that it will be useful,
## but WITHOUT ANY WARRANTY; without even the implied warranty of
## MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
## GNU General Public License for more details.
##
## You should have received a copy of the GNU General Public License
## along with this program; if not, write to the Free Software
## Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
## -A copy of the GNU General Public License is distributed with exim itself
## -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
## If you haven't worked with exim filters before, read
## the install notes at the end of this file.
## The install notes are not a replacement for the exim documentation
## -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
## -----------------------------------------------------------------------
# Only run any of this stuff on the first pass through the
# filter - this is an optomisation for messages that get
# queued and have several delivery attempts
#
# we express this in reverse so we can just bail out
# on inappropriate messages
#
if not first_delivery
then
finish
endif
## -----------------------------------------------------------------------
# Check for MS buffer overruns as per BUGTRAQ.
# [url]
# This could happen in error messages, hence its placing
# here...
# We substract the first n characters of the date header
# and test if its the same as the date header... which
# is a lousy way of checking if the date is longer than
# n chars long
if ${length_80:$header_date:} is not $header_date:
then
fail text "This message has been rejected because it has
an overlength date field which can be used
to subvert Microsoft mail programs
The following URL has further information
[url]
seen finish
endif
## -----------------------------------------------------------------------
# These messages are now being sent with a <> envelope sender, but
# blocking all error messages that pattern match prevents
# bounces getting back.... so we fudge it somewhat and check for known
# header signatures. Other bounces are allowed through.
if $header_from: contains "@sexyfun.net"
then
fail text "This message has been rejected since it has
the signature of a known virus in the header."
seen finish
endif
if error_message and $header_from: contains "Mailer-Daemon@"
then
# looks like a real error message - just ignore it
finish
endif
## -----------------------------------------------------------------------
# Look for single part MIME messages with suspicious name extensions
# Check Content-Type header using quoted filename [content_type_quoted_fn_match]
if $header_content-type: matches "(?:file)?name=("[^"]+\.(?:ad[ep]|ba[st]|chm|cmd|com|cpl|crt|eml|exe|hlp|hta|in[fs]|isp|jse?|lnk|md[be]|ms[cipt]|pcd|pif|reg|scr|sct|shs|url|vb[se]|ws[fhc])")"
then
fail text "This message has been rejected because it has
potentially executable content $1
This form of attachment has been used by
recent viruses or other malware.
If you meant to send this file then please
package it up as a zip file and resend it."
seen finish
endif
# same again using unquoted filename [content_type_unquoted_fn_match]
if $header_content-type: matches "(?:file)?name=(\S+\.(?:ad[ep]|ba[st]|chm|cmd|com|cpl|crt|eml|exe|hlp|hta|in[fs]|isp|jse?|lnk|md[be]|ms[cipt]|pcd|pif|reg|scr|sct|shs|url|vb[se]|ws[fhc]))"
then
fail text "This message has been rejected because it has
potentially executable content $1
This form of attachment has been used by
recent viruses or other malware.
If you meant to send this file then please
package it up as a zip file and resend it."
seen finish
endif
## -----------------------------------------------------------------------
# Attempt to catch embedded VBS attachments
# in emails. These were used as the basis for
# the ILOVEYOU virus and its variants - many many varients
# Quoted filename - [body_quoted_fn_match]
if $message_body matches "(?:Content-(?:Type:(?>\s*)[\w-]+/[\w-]+|Disposition:(?>\s*)attachment);(?>\s*)(?:file)?name=|begin(?>\s+)[0-7]{3,4}(?>\s+))("[^"]+\.(?:ad[ep]|ba[st]|chm|cmd|com|cpl|crt|eml|exe|hlp|hta|in[fs]|isp|jse?|lnk|md[be]|ms[cipt]|pcd|pif|reg|scr|sct|shs|url|vb[se]|ws[fhc])")[\s;]"
then
fail text "This message has been rejected because it has
a potentially executable attachment $1
This form of attachment has been used by
recent viruses or other malware.
If you meant to send this file then please
package it up as a zip file and resend it."
seen finish
endif
# same again using unquoted filename [body_unquoted_fn_match]
if $message_body matches "(?:Content-(?:Type:(?>\s*)[\w-]+/[\w-]+|Disposition:(?>\s*)attachment);(?>\s*)(?:file)?name=|begin(?>\s+)[0-7]{3,4}(?>\s+))(\S+\.(?:ad[ep]|ba[st]|chm|cmd|com|cpl|crt|eml|exe|hlp|hta|in[fs]|isp|jse?|lnk|md[be]|ms[cipt]|pcd|pif|reg|scr|sct|shs|url|vb[se]|ws[fhc]))[\s;]"
then
fail text "This message has been rejected because it has
a potentially executable attachment $1
This form of attachment has been used by
recent viruses or other malware.
If you meant to send this file then please
package it up as a zip file and resend it."
seen finish
endif
## -----------------------------------------------------------------------
#### Version history
#
# 0.01 5 May 2000
#Initial release
# 0.02 8 May 2000
#Widened list of content-types accepted, added WSF extension
# 0.03 8 May 2000
#Embedded the install notes in for those that don't do manuals
# 0.04 9 May 2000
#Check global content-type header. Efficiency mods to REs
# 0.05 9 May 2000
#More minor efficiency mods, doc changes
# 0.06 20 June 2000
#Added extension handling - thx to Douglas Gray Stephens & Jeff Carnahan
# 0.07 19 July 2000
#Latest MS Outhouse bug catching
# 0.08 19 July 2000
#Changed trigger length to 80 chars, fixed some spelling
# 0.09 29 September 2000
#More extensions... its getting so we should just allow 2 or 3 through
# 0.10 18 January 2001
#Removed exclusion for error messages - this is a little nasty
#since it has other side effects, hence we do still exclude
#on unix like error messages
# 0.11 20 March, 2001
#Added CMD extension, tidied docs slightly, added RCS tag
#** Missed changing version number at top of file :-(
# 0.12 10 May, 2001
#Added HTA extension
# 0.13 22 May, 2001
#Reformatted regexps and code to build them so that they are
#shorter than the limits on pre exim 3.20 filters. This will
#make them significantly less efficient, but I am getting so
#many queries about this that requiring 3.2x appears unsupportable.
# 0.14 15 August,2001
#Added .lnk extension - most requested item :-)
#Reformatted everything so its now built from a set of short
#library files, cutting down on manual duplication.
#Changed w in filename detection to . - dodges locale problems
#Explicit application of GPL after queries on license status
# 0.15 17 August, 2001
#Changed the . in filename detect to S (stops it going mad)
# 0.16 19 September, 2001
#Pile of new extensions including the eml in current use
# 0.17 19 September, 2001
#Syntax fix
#
#### Install Notes
View 1 Replies
View Related
May 28, 2009
We have identified that messages from your IP are being filtered based on the recommendations of the Symantec Brightmail filter as well as our internal Smart Screen Filters. The filter was initiated by Hotmail at (5/20/2009 12:00:00 AM) PST due to a large volume of emails that were sent prior to this time.
We will be happy to work directly with Symantec on your behalf to investigate and possibly resolve this problem. In order to move forward, we will need examples of the messages that were caught by the Brightmail filters.
View 3 Replies
View Related
Jun 30, 2008
I am running a small hosting operation and would like to know more about SPAM filtering and controlling. SpamAssasin is installed on my server but it does not do much. I was looking into different solution with a anti-spam device and would like to know which one is recommended the most: sonicwall, barracuda or symantec? (total email addresses is less than at thousand)
View 4 Replies
View Related
Jan 3, 2008
I have A/B power from my colo provider, each fed from separate xxxKVA UPSes. I have dual-cord servers powered from these two circuits.
Now I'm adding an EMC SAN, which comes with its own little APC UPS. EMC says the SAN's A-side power should come from my A-side circuit, and the SAN's B-side power should come from the included little APC UPS, which should be plugged into my B-side circuit.
Unfortnately, the little UPS puts out lots of garbage line noise on the upstream side. I know this because I plugged it in at home, and it was turning my X10 light fixtures on and off when the battery would hit a charging cycle. This is not the kind of dirty power I'd like to be feeding to the servers.
Does anyone have a suggestion for cleaning up the harmonic distortion that these UPSes send back upstream? Maybe a Tripp Lite IBAR12/20ULTRA? Or a Tripp Lite IS-1000 Full Isolation Transformer?
Alternatively, if there is a way to tell the EMC SAN to enable write caching without the APC UPS, that might work, too.
View 6 Replies
View Related
Jul 8, 2007
How much CPU & RAM load does your spam filter put on your VPS? It seems logical that the more mailboxes hosted, the more VPS resources the spam filter will consume - especially if any of the email addresses are targeted by spammers (or the user is careless and gives out their email address everywhere - as many do).
It's become so much with some of our subscribers that we have had to offer a hardware spam filter, to keep the load off the VPS. It's been great in that the VPS's protected by it have seen a dramatic performance increase....but are these subscribers unique in some way?
And so here is my question...how much CPU is everyone's spam assassin/spam filter using and how many mailboxes do you host on your VPS? very curious...
View 8 Replies
View Related
Apr 1, 2007
I am on a DirectAdmin server, with root access. Exim mail server.
Is there any way to filter outgoing messages? Let's say if IP of client is listed in blacklist, they can not send out emails
View 2 Replies
View Related
Mar 26, 2007
I have the same problem as many other users who try to send to Hotmail.... my e-mails are being put into the junk filter.
Could you please help me with this as I am going mad trying to figure out how to fix this? I am running a VPS which has 4 IP's and whenever I send an e-mail from the domain it moves into the junk filter. E-mails are received correctly by other e-mail providers (Yahoo, etc.), only Hotmail has the issue.
When I do a DNS Report, the only error I get besides the mail server having a single point of failure, is "Mail server host name in greeting". I have had the VPS provider add a RDNS entry, etc.
I have spoken with the support team at the VPS Host and they have tried everything they could and can't understand why there is an issue. I also contacted Hotmail and they redirected me to a Junk Filter troubleshooting page [url]
I don't know why this is being filtered as Junk and what I can do to fix it. Could you please help as I would be most grateful.
Is someone able to PM me with help as I would prefer not to post the network/website details on the forum please?
View 1 Replies
View Related
Apr 9, 2007
Anyone else have problems with Yahoo Spam filters being overzealous? I recently setup a new domain and a few email accounts. After sending a few test messages to my Yahoo email, each one of them was flagged as spam. I changed the text to a more regular email, with no controversial words and still the same result. I notice that Yahoo regularly traps some newsletters I subscribe to in my spam filter, but rarely, if ever legitimate emails. I checked with my web host who informed me they are not on any spam blacklists and that I would have to contact Yahoo. Yahoo's help is quite a maze (designed to make you lose interest and move on), so before I continue hunting for how to contact Y!, I thought I would come here to see if this is a common problem (with a common fix).
View 1 Replies
View Related
Jan 23, 2008
After obtaining a new allocation from ARIN, how long do you typically wait before your upstreams update their filters to allow the new announcement? Which providers are most proactive? Least?
View 8 Replies
View Related
Apr 11, 2008
i try to make a email filter using cpanel options but dont work this its the code
Code:
if first_delivery
and ( ("$header_from:" contains "id@dominio.com")
or ("$header_from:" contains "id1@dominio.com")
or ("$header_from:" contains "id2@dominio.com")
)
then
unseen deliver "id3@dominio.com"
endif
and Also
Code:
if
$header_from: contains "id@dominio.com"
then
deliver "id3@dominio.com"
endif
the main idea its all outgoing mail from id@dominio.com must be
send a copy (bcc) to id3@dominio.com
View 6 Replies
View Related
Jun 2, 2008
I want to filter some keywords showing on the website.
For example,some adult keywords or some force keywords,I want apache change it to * or close the webpage.
I use cPanel/WHM but I find no place to do the setting.
View 0 Replies
View Related
Dec 17, 2008
I've been reading about SpamAssassin's Bayes filter. Apparently I need to forward examples of Spam and Ham to a specific email address which SA then checks.
Unfortunately, I do not suspect my average user will do this.
Here's what I'd have in a perfect world: a Webmail program and plugins for Thunderbird/Outlook/Outlook Express that will forward emails to the appropriate place whenever the Junk/Not Junk buttons are pressed. So far, the only one I've been able to find that will do this is Zimbra. However, Zimbra requires an entire server which is overkill for our small company.
I realize Thunderbird already has Bayes filters built in but I would like to do all the spam filtering server-side for consistency.
Is what I want possible?
If not, can anyone link me to a good tutorial for writing Thunderbird extensions?
View 2 Replies
View Related
Aug 30, 2007
i have cpanel 11 with exim 4.6
i would like to be able to remove certain headers with the exim filter.
View 2 Replies
View Related
Aug 9, 2007
I'm wondering if theres anything I can install on the server that will either filter or track outgoing spam. I don't want to limit the number of emails sent per hour or anything, I just want to be able to maybe search through some flagged emails or something. Or if they send the exact same email more than x times it can disable their account... I'm not sure
View 1 Replies
View Related
Dec 28, 2007
I have a very strange error with mod_ext_filter (apache 2.2/php5.2.4)
Filter is defined with
ExtFilterDefine output_filter mode=output intype=text/html
outtype=text/html cmd="path_to_filter_file"
and called with
<Directory "/home/users/">
SetOutputFilter output_filter
filter file
------
use strict;
$_ = join('',<STDIN>);
$_=$_.'test';
print $_;
--------------
And now the problem. I want to automatically add extra html code with this
filter. On most sites working fine without problem. PhpBB3 working just fine, for example.
But on other php scripts like vBulletin, SMF, Nuke, IPB it brokes entire site.
IE6 have unterminable refreshes, IE7 displays an error page, Firefox retunrs a blank page if extra code is placed on header. With Opera errors are minimal but sometimes it displays some strange code.
These errors are only related to vBulletin, SMF, Nuke, IPB and few others.
Removing this line
$_=$_.'test';
everything is back to the normal without extra code
View 1 Replies
View Related
Aug 12, 2014
I am not able to add a new filter to fail2ban
If I go in plesk panel to: Home > Tools & Settings >IP Address Banning > Jails > managing Filters > add filter > type in name & filtercontent and save I get "Information: The jail filter was added". But i can not see the new added filter in the Plesk Filter List (still just the 12 Filters in the list).
On the filesystem > /etc/fail2ban/filter.d/ i can see the new file but with the extension .local - usulay the file is named like xyz.conf
The output of /usr/local/psa/admin/sbin/f2bmng --get-filters-list
[["apache-auth", "fail2ban"], ["apache-badbots", "fail2ban"], ["apache-common", "fail2ban"],
["common", "fail2ban"], ["plesk-courierlogin", "plesk-fail2ban-configurator"],
["plesk-dovecot", "plesk-fail2ban-configurator"], ["plesk-horde", "plesk-fail2ban-configurator"],
["plesk-panel", "plesk-fail2ban-configurator"], ["plesk-qmail", "plesk-fail2ban-configurator"],
["plesk-roundcube", "plesk-fail2ban-configurator"], ["postfix-sasl", "fail2ban"],
["proftpd", "fail2ban"], ["recidive", "fail2ban"], ["sshd", "fail2ban"], ["test", null]]
test is the name i choosed for the new filter and it seems the second field has "null" .....
View 1 Replies
View Related
May 14, 2009
we're using IIS 5.0 which doesn't have any flexible support for rewriting URLs which include parameters (AFAIK, anyway). We're currently testing an ISAPI Rewrite Plugin, which emulates the syntax and behaviour of mod_rewrite, which is why I felt this forum category to be the most appropriate for this post.
Our URLs at the moment look like:-
[url]where 'hg' is an instance of a key. In this particular example, we wish to rewrite our URL to read [url]homegarden' where 'homegarden' is a value of that respective key.
Since the value 'homegarden' isn't present in our current URL, we need some way to associate this value with it's relevant key, by mapping one to another. The ideal solution in this case would be to use RewriteMap, however the ISAPI plugin doesn't support this directive.
One other solution, that I've seen here is to use the 'homegarden' value as the key instead. However, due to the lack of flexibility of our system, this isn't possible.
Another method of achieving this is to hard-code a RewriteRule for every instance of a key, but this solution has obvious performance drawbacks (we have around 400 categories)
So, at the moment I'm out of ideas, which is why I thought I'd try and rack your brains for suggestions. An ideal scenario would be to use something like RewriteCond (which is supported by the plugin) to attach a file which would contain the key/value map.
The current supported directives are :-
RewriteRule
RedirectRule
RewriteHeader
RewriteCond
RewriteLog
RewriteLogLevel
RewriteFilterPriority
IterationLimit
MaxMatchCount
StrictParsing
Even if it's not possible using one of the above, maybe someone can point me to solution involving a method other than using this particular plugin?
View 6 Replies
View Related
Jun 16, 2008
whether it is possible for Procmail to filter based on the IP address of the host from which a message was received. In context, I want to reject any messages that come from hosts other than a specific list.
Is this possible? Search for the absence of particular IP addresses in received headers or something?
is it possible to have one "recipe" acting on behalf of all mail received for a domain name, or must it be per-user?
View 0 Replies
View Related
Aug 13, 2008
I am having problem loading filter rules in my iptables.
I am following this tutorial but am getting this error.
Code:
'ptables-restore v1.3.5: iptables-restore: unable to initializetable 'filter
Error occurred at line: 6
Line#6 says *filter
I am on CentOS and am using iptables 1.3.5
View 13 Replies
View Related
Nov 30, 2007
I just found out all my mails sent to Gmail are marked as spam. However, if I check my IP address in independant grass-root spam databases, my IP address isn't blacklisted. I checked 7 of them. Do you know how to solve this problem
View 6 Replies
View Related
Nov 20, 2007
i was wondering if there is any other free anti-spam filter other than Spam Assassin that can be used on a VPS account?
View 13 Replies
View Related
