Topic: How to Determine Data being sent and recieved.

[pssl]

Hi.

I noticed from my vnStat reports that my internet data usage jumped from about 10gig to 30gig for both Jan and Feb.  The jump was due to an increase in outgoing data, which is odd because I don't do a lot uploading, and the jumps took place on 4 specific days and very similar amounts of data being shipped out...which seems suspicious.  My wife's systems have SugarSync installed so that may the reason which I'm looking into as well.  I'm looking through the SME logs to see if I can find a reason for it but so far no luck.  I'm hoping to find out which particular machine on my network is the guilty party and it would be great to find out what IP addresses were involved.  I'm just not sure of the best way to go about tracing it.  Does anyone have an suggestions how to trace down what on my network is shipping out that much data?

Supplementary: Ok, so people are reading this but no one is responding.  So I assume it's because I haven't asked the right question.  When I look at the SME logs there are no column headings so I can't tell what the information is.  What I'm looking for is someway to determine the IP address/URL connected to, the amount of data being sent and received and the date of the action.  I can prick up on the URL's and date easy enough, they're pretty obvious, I'm just not sure how to get the number of bytes in/out.  Which column do I look at and which log is should I use?  Is there a document that explains the logs and their layout?  Once I know this I can awk through the log(s) to get what I need.

Peter

[janet]

pssl

At command prompt do
iptraf

[pssl]

Mary.

Thanks.  I'll give it a try.

[chris burnat]

Mary.

Thanks.  I'll give it a try.

it is not going to help you much. IPTRAF is a tool meant to monitor traffic alive, it does not store past activity AFAIK. All the same, is a very handy utility to see what is happening now. I am not aware of a log showing detail of all traffic in and out of a SME box.  I would be worried about SugarSync, check how it is setup, and see if it has logs, it may show something...  The user may also recall adding and deleting large amount of files, and if they are synced, it may create serious traffic. 
Best of luck.

[pssl]

Thanks Chris.

I was just poking around with iptraf.  Nice little tool.  Good to know about.  You can get it to log data, but for how long I don't know.  But you are correct, it doesn't help analyze the logs to see what traffic flowed out in the past.  It's a mystery. 

Not a lot of data is stored in Sugarsync, less than a gig, but who knows what SS does up in the cloud.  I'll take look to see if I can get the IP address of SS and see if I can find in the logs.  Maybe that'll show something.

Thanks
P

[janet]

pssl

Then maybe try
http://wiki.contribs.org/Sarg
I think there is another contrib or app that will help also, cannot remember it's name at the moment, anyone else ?

[pssl]

Thanks Mary.  It looks interesting, might help.

[janet]

pssl

I think this is the other one I was remembering
http://wiki.contribs.org/Vnstat

[pssl]

Mary,

Thanks for the info on vnstat.  I already have this installed.  That's how I found out about the spike in data usage.  I checked the man page for config parms last night to see if there was additional info I could get it to report but I didn't see anything useful.

Since vnstat can report historical data usage and since the logs show lan/web connections, you'd think there would be a relatively easy was to put the two bits of info together...but no so far.  It looks like a job for grep/awk/sed/python somehow.  Not being expert in any of these tools this may take some time.

Thanks again Mary.
P

[CharlieBrady]

Since vnstat can report historical data usage and since the logs show lan/web connections, you'd think there would be a relatively easy was to put the two bits of info together...but no so far.  It looks like a job for grep/awk/sed/python somehow.  Not being expert in any of these tools this may take some time.

Or an infinite amount of time, if the information you are seeking is not recorded anywhere - which I suspect is the case here. Most traffic which passes out to the Internet through the server doesn't generate any logs, and only contributes to undifferentiated packet counts and byte counts associated with the network interface.

[pssl]

Ah...oh well.  I'll have to look for other solutions then.  Possibly load monitors on the various machines on the lan.

Thanks Charlie.

[janet]

pssl

Sarg (as mentioned earlier) see sample report
http://sarg.sourceforge.net/squid-reports/2004Aug06-2004Sep13/index.html
from
http://sarg.sourceforge.net/sarg.php
Sarg - Squid Analysis Report Generator is a tool that allow you to view "where" your users are going to on the Internet.
Sarg provides many informations about Squid users activities: times, bytes, sites, etc...

[pssl]

Mary,

That's right, you mentioned it before and I was starting to look at it...then work got in the way.  Thanks for the reminder.

[CharlieBrady]

Sarg - Squid Analysis Report Generator is a tool that allow you to view "where" your users are going to on the Internet.

Only if the "where" is using ftp/http/https, i.e. something handled by squid. Sarg won't know anything about, e.g., file sharing protocols.

[janet]

pssl

As Charlie points out Sarg is limited in its "ability".

For a lot of good reading I suggest googling
analyse file sharing traffic on Linux

It is not a simple task, as file sharing systems try to disguise their activity & mutiple methods of detection may be needed.

Firstly you need to analyse where or what the problem is, the following are just various suggestions to consider, nothing to do with your existing logs.

Maybe look at Wireshark which has been mentioned in these forums before, so search the forums also on wireshark as well as google, doing so may also lead you to some interesting answers re blocking, here is one thread of interest but there are plenty more
http://forums.contribs.org/index.php/topic,46036.0/all.html


Personally I think you are better off to stop the usage of or installation of such apps on your network.

Re blocking P2P you might also look here (which refers to a defunct method) but it then refers you on to
Refer http://ipp2p.org/ which then refers to http://opendpi.org/
for ways to possibly block P2P

I think you are better off to stop the use of these apps, or otherwise live with their consequences.

Maybe setup your own cloud server to keep the wife happy instead of using SugarSync, see
http://wiki.contribs.org/OwnCloud