Koozali.org: home of the SME Server

Traffic Flooding, can't find the cause.

Offline Smitro

  • *
  • 349
  • +0/-0
Traffic Flooding, can't find the cause.
« on: December 07, 2018, 12:10:19 PM »
Hi,
I need help!
Recently my internet usage went crazy. I don't use a lot of traffic, about 100 - 300mb on an average day. Then one day 17gb up and 18 gb down, same the next, then the next until all my data allowance was used up.
I went through my network trying to work out what had happened. Even my router that shows the amount of traffic being used, seemed to miss this traffic.
I disconnected everything until I found that it was my SME server that caused this. The SME server is only working as a backup, using Affa. It downloads backups from another SME Server across the internet, and the router does not have any port forwards to it.
I've tried troubleshooting this, but I'm lost. I've decided to start over...

How can I detect what is flooding in and out of my server?

The amount of traffic that is going in and out seems almost symmetrical in size. I'm not sure if it's been compromised or if something has gone rogue?
.........

Offline mmccarn

  • *
  • 2,626
  • +10/-0
Re: Traffic Flooding, can't find the cause.
« Reply #1 on: December 07, 2018, 12:52:32 PM »
iptraf should be available on your sme, and lets you monitor your network traffic in ip address pairs in real time.  Here's a screenshot after using all the default selections:
- IP traffic monitor
- All Interfaces

Code: [Select]
IPTraf
┌ TCP Connections (Source Host:Port) ────────────────────────────────────────── Packets ─────────── Bytes ─── Flags ───── Iface ──────┐
│┌192.168.200.2:2202                                                          >    1194            293512     -PA-        eth0        │
│└192.168.200.110:50462                                                       >    1189             62360     --A-        eth0        │
│┌192.168.200.167:56634                                                       =       9              1572     --A-        eth0        │
│└192.168.200.2:9001                                                          =       8              3464     CLOSED      eth0        │
│┌192.168.200.2:9001                                                          >       6              1176     --A-        eth0        │
│└192.168.200.167:56508                                                       >       3              2262     -PA-        eth0        │
│┌192.168.200.17:64386                                                        >       1                46     --A-        eth0        │
│└192.168.200.2:3128                                                          =       0                 0     ----        eth0        │
│┌192.168.200.2:47328                                                         >       1                52     --A-        eth0        │
│└40.69.221.239:443                                                           =       0                 0     ----        eth0        │
│┌192.168.200.167:56636                                                       =       9              1572     --A-        eth0        │
│└192.168.200.2:9001                                                          =       8              3464     CLOSED      eth0        │
│┌192.168.200.167:56638                                                       =       8              1520     --A-        eth0        │
│└192.168.200.2:9001                                                          =       8              3464     CLOSED      eth0        │
│                                                                                                                                     │
│                                                                                                                                     │
│                                                                                                                                     │
│                                                                                                                                     │
│                                                                                                                                     │
└ TCP:      7 entries ────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
┌─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ UDP (100 bytes) from 127.0.0.2:45238 to 127.0.0.2:53 on lo                                                                          │
│ UDP (100 bytes) from 127.0.0.2:45238 to 127.0.0.2:53 on lo                                                                          │
│ UDP (100 bytes) from 192.168.200.2:42743 to 205.251.199.54:53 on eth0                                                               │
│ UDP (72 bytes) from 192.168.200.18:42262 to 192.168.200.2:53 on eth0                                                                │
│ UDP (269 bytes) from 205.251.199.54:53 to 192.168.200.2:42743 on eth0                                                               │
│ UDP (132 bytes) from 127.0.0.2:53 to 127.0.0.2:45238 on lo                                                                          │
│ UDP (132 bytes) from 127.0.0.2:53 to 127.0.0.2:45238 on lo                                                                          │
│ UDP (169 bytes) from 192.168.200.2:53 to 192.168.200.18:42262 on eth0                                                               │
│ UDP (169 bytes) from 192.168.200.2:53 to 192.168.200.18:42262 on eth0                                                               │
│ ICMP dest unrch (port) (197 bytes) from 192.168.200.18 to 192.168.200.2 on eth0                                                     │
 Pkts captured (all interfaces):                                2887       │   TCP flow rate:       32.40 kbits/s                     
 Up/Dn/PgUp/PgDn-scroll  Lft/Rt-vtcl scrl  W-chg actv win  S-sort TCP  X-exit

Offline ReetP

  • *
  • 3,722
  • +5/-0
Re: Traffic Flooding, can't find the cause.
« Reply #2 on: December 08, 2018, 02:18:52 AM »
Check it on both ends...
...
1. Read the Manual
2. Read the Wiki
3. Don't ask for support on Unsupported versions of software
4. I have a job, wife, and kids and do this in my spare time. If you want something fixed, please help.

Bugs are easier than you think: http://wiki.contribs.org/Bugzilla_Help

If you love SME and don't want to lose it, join in: http://wiki.contribs.org/Koozali_Foundation

Offline Smitro

  • *
  • 349
  • +0/-0
Re: Traffic Flooding, can't find the cause.
« Reply #3 on: December 09, 2018, 09:19:59 AM »
It seems like nothing obvious, but the bottom part seems to keep scrolling through with traffic to port 53, which should be DNS. The top 2 addresses are the server and the workstation connected via SSH.

Code: [Select]
IPTraf
l TCP Connections (Source Host:Port) qqqqqqqqqq Packets qqq Bytes Flags  Iface k
xl192.168.1.253:22                            >    3556   1034320 -PA-   eth0  x
xm192.168.1.151:52813                         >    3455    170918 --A-   eth0  x
x                                                                              x
x                                                                              x
x                                                                              x
x                                                                              x
x                                                                              x
x                                                                              x
x                                                                              x
x                                                                              x
x                                                                              x
x                                                                              x
m TCP:      1 entries qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq Active qj
lqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqk
x UDP (65 bytes) from 192.168.1.253:65227 to 192.5.5.241:53 on eth0            x
x UDP (65 bytes) from 192.168.1.253:19548 to 192.36.148.17:53 on eth0          x
x UDP (68 bytes) from 192.168.1.253:33203 to 193.0.14.129:53 on eth0           x
x UDP (68 bytes) from 192.168.1.253:53474 to 192.33.4.12:53 on eth0            x
x UDP (78 bytes) from 192.168.1.151:137 to 192.168.1.255:137 on eth0           x
m Bottom qqqqqq Elapsed time:   0:03 qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqj
 Pkts captured (all interfaces):       12681 x TCP flow rate:     38.60 kbits/s
 Up/Dn/PgUp/PgDn-scroll  M-more TCP info   W-chg actv win  S-sort TCP  X-exit
.........

Offline mmccarn

  • *
  • 2,626
  • +10/-0
Re: Traffic Flooding, can't find the cause.
« Reply #4 on: December 09, 2018, 03:40:17 PM »
By default the top window is showing only TCP traffic by IP pair.

You can see traffic by TCP/UDP and port by starting iptraf using "-s <network device>" -- but then you lose the IP pair information:
Code: [Select]
iptraf -s eth0
The UDP traffic in your output (at least the top two entries) are DNS root servers (http://ipinfo.io/192.5.5.241 http://ipinfo.io/192.36.148.17 ) - I don't think this traffic is the problem.

I missed the detail in your first post about "Even my router that shows the amount of traffic being used, seemed to miss this traffic." -- can you provide more details on this?  Is this router between the SME server and the internet, beside it, or behind it?  If the router is the last device in your network before the ISP you'd have to have a device on the WAN side of the router or the ISPs network has been compromised somewhere else. 

Can the ISP give you the IP and mac address of the device that is using all the bandwidth?
  If not, perhaps you can dispute their quota system.

What kind of router is it?  Is it up-to-date and does it have any unpatched security problems?  Does it include any VPN or proxy services that could have been compromised by a remote attacker? Does it provide wifi, and is the wifi traffic included in the stats you're looking at?

Does the router reset its stats when rebooted, and has it been rebooted lately?
  If so, perhaps someone is stealing your bandwidth and then rebooting the router to reset the network stats.

How big are the affa backups (how much total data)?  Could there be an affa problem that is causing it to execute a full backup on every run instead of a hardlinked differential backup?
  I had a problem like this when first testing Affa and using an SMB share as the backup destination...

If the router questions are a dead end -- what is your internet bandwidth supposed to be? Assuming 10mbit, it would only take about 5 hours to use 17GB of traffic.  You would need to be running iptraf when the rogue process is running to see the traffic.

Try running iptraf for an extended period, checking the progress from time to time.

If you have a physical monitor attached, you could use that.

If not, you can do this using "screen" to make sure it keeps running if your ssh session is disconnected.

To start 'screen':
Code: [Select]
screen
Start 'iptraf' and set it to monitor traffic, set it to sort its display by bytes transferred (press "s" then "b") then exit from the screen session using <Ctrl>-A D (that is, press <Ctrl>-A, then press the "d" key)

Re-attach to screen to see how it's going using "screen -r"
Code: [Select]
screen -r
Don't forget to come back and stop iptraf at some point...