Topic: How to Determine Data being sent and recieved.

[pssl]

Hi.

I noticed from my vnStat reports that my internet data usage jumped from about 10gig to 30gig for both Jan and Feb.  The jump was due to an increase in outgoing data, which is odd because I don't do a lot uploading, and the jumps took place on 4 specific days and very similar amounts of data being shipped out...which seems suspicious.  My wife's systems have SugarSync installed so that may the reason which I'm looking into as well.  I'm looking through the SME logs to see if I can find a reason for it but so far no luck.  I'm hoping to find out which particular machine on my network is the guilty party and it would be great to find out what IP addresses were involved.  I'm just not sure of the best way to go about tracing it.  Does anyone have an suggestions how to trace down what on my network is shipping out that much data?

Supplementary: Ok, so people are reading this but no one is responding.  So I assume it's because I haven't asked the right question.  When I look at the SME logs there are no column headings so I can't tell what the information is.  What I'm looking for is someway to determine the IP address/URL connected to, the amount of data being sent and received and the date of the action.  I can prick up on the URL's and date easy enough, they're pretty obvious, I'm just not sure how to get the number of bytes in/out.  Which column do I look at and which log is should I use?  Is there a document that explains the logs and their layout?  Once I know this I can awk through the log(s) to get what I need.

Peter

[janet]

pssl

At command prompt do
iptraf

[pssl]

Mary.

Thanks.  I'll give it a try.

[chris burnat]

Mary.

Thanks.  I'll give it a try.

it is not going to help you much. IPTRAF is a tool meant to monitor traffic alive, it does not store past activity AFAIK. All the same, is a very handy utility to see what is happening now. I am not aware of a log showing detail of all traffic in and out of a SME box.  I would be worried about SugarSync, check how it is setup, and see if it has logs, it may show something...  The user may also recall adding and deleting large amount of files, and if they are synced, it may create serious traffic. 
Best of luck.

[pssl]

Thanks Chris.

I was just poking around with iptraf.  Nice little tool.  Good to know about.  You can get it to log data, but for how long I don't know.  But you are correct, it doesn't help analyze the logs to see what traffic flowed out in the past.  It's a mystery. 

Not a lot of data is stored in Sugarsync, less than a gig, but who knows what SS does up in the cloud.  I'll take look to see if I can get the IP address of SS and see if I can find in the logs.  Maybe that'll show something.

Thanks
P

[janet]

pssl

Then maybe try
http://wiki.contribs.org/Sarg
I think there is another contrib or app that will help also, cannot remember it's name at the moment, anyone else ?

[pssl]

Thanks Mary.  It looks interesting, might help.

[janet]

pssl

I think this is the other one I was remembering
http://wiki.contribs.org/Vnstat

[pssl]

Mary,

Thanks for the info on vnstat.  I already have this installed.  That's how I found out about the spike in data usage.  I checked the man page for config parms last night to see if there was additional info I could get it to report but I didn't see anything useful.

Since vnstat can report historical data usage and since the logs show lan/web connections, you'd think there would be a relatively easy was to put the two bits of info together...but no so far.  It looks like a job for grep/awk/sed/python somehow.  Not being expert in any of these tools this may take some time.

Thanks again Mary.
P

[CharlieBrady]

Since vnstat can report historical data usage and since the logs show lan/web connections, you'd think there would be a relatively easy was to put the two bits of info together...but no so far.  It looks like a job for grep/awk/sed/python somehow.  Not being expert in any of these tools this may take some time.

Or an infinite amount of time, if the information you are seeking is not recorded anywhere - which I suspect is the case here. Most traffic which passes out to the Internet through the server doesn't generate any logs, and only contributes to undifferentiated packet counts and byte counts associated with the network interface.

[pssl]

Ah...oh well.  I'll have to look for other solutions then.  Possibly load monitors on the various machines on the lan.

Thanks Charlie.

[janet]

pssl

Sarg (as mentioned earlier) see sample report
http://sarg.sourceforge.net/squid-reports/2004Aug06-2004Sep13/index.html
from
http://sarg.sourceforge.net/sarg.php
Sarg - Squid Analysis Report Generator is a tool that allow you to view "where" your users are going to on the Internet.
Sarg provides many informations about Squid users activities: times, bytes, sites, etc...

[pssl]

Mary,

That's right, you mentioned it before and I was starting to look at it...then work got in the way.  Thanks for the reminder.

[CharlieBrady]

Sarg - Squid Analysis Report Generator is a tool that allow you to view "where" your users are going to on the Internet.

Only if the "where" is using ftp/http/https, i.e. something handled by squid. Sarg won't know anything about, e.g., file sharing protocols.

[janet]

pssl

As Charlie points out Sarg is limited in its "ability".

For a lot of good reading I suggest googling
analyse file sharing traffic on Linux

It is not a simple task, as file sharing systems try to disguise their activity & mutiple methods of detection may be needed.

Firstly you need to analyse where or what the problem is, the following are just various suggestions to consider, nothing to do with your existing logs.

Maybe look at Wireshark which has been mentioned in these forums before, so search the forums also on wireshark as well as google, doing so may also lead you to some interesting answers re blocking, here is one thread of interest but there are plenty more
http://forums.contribs.org/index.php/topic,46036.0/all.html


Personally I think you are better off to stop the usage of or installation of such apps on your network.

Re blocking P2P you might also look here (which refers to a defunct method) but it then refers you on to
Refer http://ipp2p.org/ which then refers to http://opendpi.org/
for ways to possibly block P2P

I think you are better off to stop the use of these apps, or otherwise live with their consequences.

Maybe setup your own cloud server to keep the wife happy instead of using SugarSync, see
http://wiki.contribs.org/OwnCloud

[chris burnat]

pssl,

I had a similar problem a while back on my own network, 8 workstations in the house.  Ended up loading IPTRAF at the ready set for "detailed interface stats" - looking at the WAN interface. My ADSL router is adjacent to my monitor, easy to keep an eye on it as I do other work.  It has taken me a few days to identify the guilty workstation by correlating sustained activity on the router and IPTRAF outgoing rate stats - Ip addresses can be seen in the traffic monitor (select WAN).  Admittedly, this method would be a nightmare on a large network,  but I imagine that your setup is similar to mine.  Hope it helps.

[janet]

chris burnat

Ended up loading IPTRAF at the ready set for "detailed interface stats" - looking at the WAN interface. ..... It has taken me a few days to identify the guilty workstation by correlating sustained activity on the router and IPTRAF outgoing rate stats - Ip addresses can be seen in the traffic monitor (select WAN).

That is what I meant for pssl to do when I first suggested iptraf.
No one else had answered after nearly two weeks so it was time to give some advice, albeit brief.

I am busy with a big personal project at the moment so do not have too much time for details.

[pssl]

Thanks folks for the help.  This is going to very hard to trace because it is so intermittent.  I had a spike on 14 Jan then 3 consecutive spikes on 21, 22 and 23 Feb and haven't had anything since.  I watch vnstat closely see if I can observe a spike in progress.  It's clear this is going to take some work.

Mary, thanks for the advise on cloud file sharing.  My own cloud, interesting.  I'm going to take a look at that.  Might be an interesting project.

[piran]

It's clear this is going to take some work.

In the server-manager's collaboration area there is an
option to implement quota... Apparently it will even
email you when that user nears its limit and goes into
limit-with-grace. Set the limits appropriately and then
let SME take the strain while awaiting the email or just
simply eyeballing the reported quota high tide mark.
Never used this feature myself so YMMV.

[janet]

pssl

You could also look at your ISP's data usage reports, which for me show usage for every connection
It should be easy to see when large amounts of data flow out.
Using Vnstat, Sarg or other tools, you should then be able to correlate the ISP 's report with local reports ie date, time & user, and that then should steer you in the right direction of where on your network this outgoing data is being generated.

piran

pssl is referring to outgoing data sent to the Internet, not how much data is stored on the server per user (quotas).

[piran]

pssl is referring to outgoing data sent to the Internet, not how much data is stored on the server per user (quotas).

Granted. There 'may' be a correlation.
Looks easy to implement and monitor.

[piran]

The sme8admin contrib's network use area shows a
network load graph over time and records that Mb/s.
Surges are easy to spot and there may be some
intelligence to be derived from knowing the instant
or period. It does not show which IP did the deed.
http://wiki.contribs.org/Sme8admin
(granularity can be 5mins AOT the usual hour)

PostEdit: added granularity spec

[mmccarn]

If your network switch provides data statistics by port, you could at least identify the offending computer by looking there.

If your switch doesn't provide stats, you can buy a small managed netgear switch that does for not that much:
http://www.netgear.com/business/products/switches/smart-switches/smart-switches/GS108T.aspx#one

* Setup a managed or web-managed switch
* Reset the port stats each night or morning, wait for an event, then check the port stats to ID the culprit.

[pssl]

Hi folks.  Thanks a bunch for all the help.  I can't keep up with the suggestions.

I installed SARG last night.  Nice little tool and will be very useful once I observe a spike in progress. 

My ISP is rather lame and doesn't have a tracking function for their 3g service, only the sat and 4g services; this is because there's no data cap on 3g, so no need to have a monitor service.  I'm on 3g for now.  In fact that's reason I'm going through all this, because I want to upgrade my service, but I needed to know my data usage in order to know what level of service (i.e., data cap) I need.  My average, ignoring the spikes, is 12 gig a month, so a 20 gig service should be plenty.  But at $3.50 a gig overage charge, an overage of 15 gig would hurt.  To jump from 12 gig to 35 gig is concerning.  That's a lot of data, half of which is outbound...we don't upload a lot of data.  Sugarsync data is maybe .5 gig and it only uploads changes (as far as I'm aware).

So why not dump my ISP?  I live in a rural area and the availability of ISPs is very limited.  Ah, such fun to live with a monopoly.

My net switch has a stats capability showing data in/out for IP address and mac address.  Thanks for the suggestion mmccarn, I never thought of that.  Now all I need to do is get it working...looks like a call to manufacturer's help line.

Here's a sample of the output from vnstat.  As you can see on the 21-23 there's quite a jump, especially in the outbound data.

Code: [Select]

In Out Total
28 February 251.02 MB 31.92 MB 282.94 MB
27 February 215.26 MB 42.25 MB 257.51 MB
26 February 2.78 GB 57.91 MB 2.84 GB
25 February 351.05 MB 38.22 MB 389.26 MB
24 February 242.93 MB 31.13 MB 274.06 MB
23 February 1.98 GB 3.94 GB 5.92 GB
22 February 1.70 GB 3.92 GB 5.62 GB
21 February 1.76 GB 3.90 GB 5.66 GB
20 February 405.70 MB 32.08 MB 437.78 MB
19 February 564.21 MB 53.90 MB 618.11 MB
18 February 152.71 MB 15.63 MB 168.34 MB
17 February 65.52 MB 7.93 MB 73.46 MB
16 February 291.12 MB 20.92 MB 312.04 MB


[piran]

Here's a same of the output from vnstat.  As you can see on the 21-23 there's quite a jump, especially in the outbound data.

Over the three days the aberrations look quite steady.
One avenue you could consider is determining whether it's
a slow continuous bleed of data, a series of large spikes or
a single daily overload. The sme8admin net load graph would
probably be able to illustrate which of those three situations
occurred. Driving its accuracy down to a granularity of 5mins
would give you far fewer logs through which to wade:-) No
help as to the IP/s involved... hopefully that's for your switch.

[pssl]

piran,

I'm running sme7 right now.  I know I should upgrade...just haven't gotten around to it.  But thanks for the suggestion.  And your right, those three days are very consistent, uncannily so.  It's very suspicious.  I've been scanning the squid logs on those dates to see if I can see anything but so far nothing.

[janet]

pssl

Depending how your network & user logins are configured, you should be able to see which workstations & users are connected on those days, by looking in the messages log file on sme server for matching date (& maybe time).
That should give clues as to where to look for the source of the data surge if not already obvious.
Have you done recent virus scans on workstations ?

[piran]

I'm running sme7 right now.

I used to run with SME7 and sme7admin.
It's still available I believe. Though if you
have a switch which provides stats then
that should be the best vector as it ought
to identify busy periods *and* their IPs.

[piran]

No outgoing torrents? Don't use them here but they
'could' take up your outgoing capacity in those sorts
of volumes. A proper feed ought to have a rate limiter
for its own good and that of the capacity of its host.
Might explain those consistent heavy traffic days.

[pssl]

Mary,

My systems are set to scan once a week.  However, now that you mention it I haven't check that they are in fact doing so recently.  I could be wrong but I would have thought a virus would have been more active more regularly...but you never know.

I'm looking at the squid access log.  It shows the lan ip addresses, which tells me which machine is connected to the associated website.  Where can I find out what the different columns are in the log.  It looks like there are two columns showing numbers that could be data transmitted (packets? bits? bytes? In? Out?)  Here's a sample with the data in question highlighted.

"Sun Feb 17 07:52:35 2013    142 192.168.0.235 TCP_MISS/200 1777 GET http://..."

I don't know what these numbers are.  If they are data transmission info that would be useful I think.

piran,
I don't use torrents, at least not that I'm aware of; I've never set one up and I know my wife wouldn't; there's no one else here so unless it has been done subversively, it not torrents.  I don't watch movies online (don't have the service speed).  I youtube a fair bit but that would account for a spike in download data, not upload data and besides, even at my heaviest I never broke 13.5 gig in a month.

I'll keep digging.  I'd really like to know what caused the spike before I sign up for a data capped service.  However, at some point I'll have to just make a choice and go.  I'd keep on monitoring and eat the cost if need be.  Hopefully I'd find the cause eventually.

Thanks folks.
P

[janet]

pssl

Use command formatting syntax like shown here to generate reports from log files
http://wiki.contribs.org/Virus:Email_Attachment_Blocking#Checking_logs

Charlie already advised the logs do not contain data quantities, so I think you are searching for something that is not there

Install wondershaper script (which works) or the rpm (not sure if that is fixed) to limit outgoing bandwidth & that may help for next time.
http://wiki.contribs.org/Wondershaper

[pssl]

Hi Mary.

I must have missed that note from Charlie...or I'm having senior's moment...my 61 year old brain ain't what it used to be.  I'll take a look at wondershaper to see if I can wrap my head around it.

Thanks Mary.

[piran]

That wondershaper thing isn't particularly easy...

With only two machines the situation ought just
to be eany meany myny mo... Be mindful of the
activity lights or noisier fans. When Immunet
kicks in here on our W8 box with its daily run the
fans start running slightly harder. It 'may' just be
easier to install a monitor on your boxes (are they
windows?) and keep an eye on things at source.

Alternative thoughts: WiFi. Could you be running
bandwidth out to your neighbour(s). An intermittent
drive-by (literally) stealing bandwidth from their car?
My mobile's data account can share too by its own
WiFi. When Microsoft get their corporate act together
here in the UK my intention is to get a Surface Pro
to use that otherwise unused tethering capability.
Could you be unexpectedly tethering your bandwidth?

I still think you should run up sme7admin and get some
graphs showing 'when' and 'by how much' the (mis)use.

[pssl]

Hi piran.

Thanks for the info on wondershaper.

I have seven machines on the net, 3 usually through Ethernet, the others almost always wifi.  I have a Macbook, an IPod, an android tablet an XP machine and 3 Windows 7 machines (2 laptops and 1 desktop).

My wifi is password protected with wpa2/AES and the distance between me and my neighbour is about 200 feet.  I do not detect any neighbourhood wifi signals and I doubt they can detect me.  It is 100 feet to road so drive-by may be possible if they have sensitive equipment.  It is a very quiet road so I would probably have noticed a car parked near my house.  They'd still have to get past the security. 

As for tethering, I'm not familiar with it.  I did a quick scan on the net and as I now understand it (connecting to the net via my cell phone), tethering isn't an issue since the issue is data flowing out through SME server with is connected to my ISP, not my cell provider.  Besides, I don't have a cell data plan; I don't connect to the net via my cell.

I'm going to install sme7admin and see what that gets me.

Thanks.
P

[piran]

I've never used or had MACs or iPods. My mobile uses
Windows Phone 7.5 and my ISP's data account allows
me to hook up other stuff to the internet via that phone's
account and connection. It's known as tethering. I've
only tested it with a Dell laptap of my neighbour when
he asked me to fix it (would not complete any boot).
It just WiFi's in to the phone, gets acknowledged etc
and then it's on the internet much the phone's own
browser. Mobile plan is Unlimited (monthly PAYG).

Sheer guess: WiFi probably. Possibly that Android
tablet or the XP box. Sheer guess YMMV.

sme7admin will give you the 'when' and 'how badly'
on the network use family of graphs. The ones further
down the page give internal and external networ loads.
You should be able to infer a lot of pointing information.

If it turns out to be a windows box then something
called Wireshark (open source) will allow you to squirrel
down through a complete morass of really technical stuff
on your network to give you a closer hunch of where to
search next. Wireshark is quite complex and not very
intuitive but it delivers, though it's up to you exactly
how you interprete what it delivers.
http://www.wireshark.org/

[pssl]

Thanks piran.

[janet]

piran

That wondershaper thing isn't particularly easy...

I disagree with that, but with a proviso.
At it's basic level there are usually only two settings to change, and two more to check.

You can do advanced customisation if you want (it's not strictly necessary), but that may take a bit of working out. There are other helpful sites with information & suggestions for advanced configuration, so initially just copy a suggested advanced configuration (only if you need it).

For the basics just test your Internet speed online and you have the answers you need

Follow the wiki to install the script & adjust the settings, it's easy to get it running in basic mode.

There is/was an rpm available, but there were reports of it not working correctly. At a quick look I cannot find it now.

Wondershaper does not solve pssl's problem, it just manages or controls the use of outgoing bandwidth, so may assist in sharing of the outgoing bandwidth so all devices get their "fair share", and limit or slow down whatever is gobbling up a few Gb every day (on the days in question).
 

[pssl]

I'm not seriously considering wondershaper because I don't want/need to manage the bandwidth via the server.  That'll have to be a mandraulic process.

[janet]

pssl

I'm not seriously considering wondershaper because I don't want/need to manage the bandwidth via the server

Actually all servers should really use some form of outgoing bandwidth shaping, so that heavy & sustained email loads, or large file uploads etc, do not hog all the bandwidth & disrupt web browsing or skype usage or other sorts of net access activity that people are doing etc etc etc. qmail is a big bandwidth hog, it will use it all if it can.

[johnp]

I agree with Mary. Coming from voice centric approach where priority needs to be maintained, the ability to set priority or reserve bandwidth is essential.

[pssl]

It seems like overkill for my situation.  It's just a home network with just my wife and I.  But if you think it's worth it maybe I'll take a closer look at it.

[piran]

I agree with Mary... however that doesn't alter the fact
that I found it difficult to impossible to implement that
wondershaping thing the last time I tried (a couple of
years back). That the OP should try and attempt to
get this going during a possible bandwidth abuse
period is unwise. Find the issue first, weigh up the
available resolutions, implement the fix. Then consider
that wondershaping thing when everything settles...

My router does some crude traffic shaping (Qos), 
it's sufficient for our server. May have another go
at wondershaper on a quiet day or week...

[johnp]

Agreed

[piran]

It seems like overkill for my situation.

Agreed. Our posts crossed.

[pssl]

I agree.  And my router has limited QoS as well.  However, once I upgrade my service and maybe start using it for more bandwidth intensive activities such as streaming movies and TV programs, then the issue of shaping may become more important.  In the meantime, the search for the cause of the data spikes goes on.  I checked the update logs on my XP machine I and noted that the spikes happened to coincide with updates.  But I've updated it a few times since and have not seen any spikes of similar magnitude.  Plus it is hard to imagine even Microsoft sending almost 4 gig of data to themselves 3 days in a row.

So far virus scans have found nothing.

[piran]

Virus and malware are the criminals' bread and butter.
The marmalade on top are the rootkits...
See if GMER can root [!] one such out...
http://www.downloadcrew.com/article/26354-gmer

[CharlieBrady]

With a few changes to iptables setup, you could count the number of packets in and out for each of your two workstations (create new in and out chains). I don't know of a simple way to count the number of bytes in and out.

[pssl]

I'm not knowledgeable enough to mess with IP tables.  But I have some people I know who could help me.

I didn't consider rootkit problems.  Perhaps I should run some checks just to be sure

Thanks folks.