Koozali.org: home of the SME Server

[SOLVED] Docker container can not ping outside world

guest22

[SOLVED] Docker container can not ping outside world
« on: August 04, 2017, 12:11:16 PM »
Hi,


I am doing some testing with SME10 (Server-gateway) and Docker.


A default install of docker and a test container (centos:centos7) will install correctly. But from within the container I am not able to ping the outside world. I can ping the internal external ethernet of the SME10 host tho.
Obviously the container receives a private IP address from docker (172.x.x.x) and a bridge is automatically created on the SME10 host. Also new iptables docker chainesmasqueraiding rules are added to allow 172 traffic.


I am above my head on this, does somebody have a clue to conceptually make his work with SME10 please?


TIA

Offline mmccarn

  • *
  • 2,626
  • +10/-0
Re: Docker container can not ping outside world
« Reply #1 on: August 04, 2017, 01:28:44 PM »
Can you add the docker subnet as a 'local network' in server-manager?

guest22

Re: Docker container can not ping outside world
« Reply #2 on: August 04, 2017, 01:50:48 PM »
I tried that, but that didn't work for the router field in 'adding local network' should be filled with the bridge IP which is 172.x.x.x


Thanks. Breaking my brains over this, but I'm sure the solution lays within SME firewall stuff.

Offline Daniel B.

  • *
  • 1,699
  • +0/-0
    • Firewall Services, la sécurité des réseaux
Re: Docker container can not ping outside world
« Reply #3 on: August 04, 2017, 01:54:00 PM »
Check in /var/log/iptables/current as it might be dropped by iptables on SME. You'd better install smeserver-bridge-interface, and then use pipework to attach your docker the br0 so that it'll be on the local network. (this is where Docker networking is lacking: you can't attach a Docker to a manually created bridge AFAIK)
C'est la fin du monde !!! :lol:

guest22

Re: Docker container can not ping outside world
« Reply #4 on: August 04, 2017, 02:08:21 PM »
You'd better install smeserver-bridge-interface


Thanks, my head is spinning. the bridge contrib is not available for SME10 or is it?

guest22

Re: Docker container can not ping outside world
« Reply #5 on: August 04, 2017, 02:30:37 PM »
Check in /var/log/iptables/current as it might be dropped by iptables on SME.

Docker version 17.06.0-ce, build 02c1d87


It seems that the container sends it's packets to the internal NIC (192....) and it stops there...

denylog: IN=docker0 OUT= MAC=02:42:c0:13:1f:a9:02:42:ac:11:00:02:08:00  SRC=172.17.0.2 DST=192.168.147.2 LEN=82 TOS=00 PREC=0x00 TTL=64 ID=37635 CE DF PROTO=UDP SPT=45006 DPT=53 LEN=62

and this the auto created bridge on SME by docker at docker service start:

  docker0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 172.17.0.1  netmask 255.255.0.0  broadcast 0.0.0.0
        ether 02:42:c0:13:1f:a9  txqueuelen 0  (Ethernet)
        RX packets 122  bytes 33405 (32.6 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 106  bytes 25973 (25.3 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0


and this is the container's ethernet:


eth0      Link encap:Ethernet  HWaddr 02:42:ac:11:00:02 
          inet addr:172.17.0.2  Bcast:0.0.0.0  Mask:255.255.0.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:109 errors:0 dropped:0 overruns:0 frame:0
          TX packets:127 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
 
          RX bytes:26253 (26.2 KB)  TX bytes:35525 (35.5 KB)


I can ping from the container to SME on both internal and external NIC's

and this are the auto added iptables rules by docker at start of the docker service:


Chain DOCKER (1 references)
target     prot opt source               destination         


Chain DOCKER-ISOLATION (1 references)
target     prot opt source               destination         
RETURN     all  --  anywhere             anywhere           


Chain DOCKER-USER (1 references)
target     prot opt source               destination         
 
RETURN     all  --  anywhere             anywhere           

 


And these are the details of the auto created bridge:


  [
    {
        "Name": "bridge",
        "Id": "7f5f9cdc4f7ec26d71ca5c21c203472bbb94ab20648ebb4ee717a03bf308a67a",
        "Created": "2017-08-04T13:47:02.564051444+02:00",
        "Scope": "local",
        "Driver": "bridge",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "172.17.0.0/16",
                    "Gateway": "172.17.0.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {},
        "Options": {
            "com.docker.network.bridge.default_bridge": "true",
            "com.docker.network.bridge.enable_icc": "true",
            "com.docker.network.bridge.enable_ip_masquerade": "true",
            "com.docker.network.bridge.host_binding_ipv4": "0.0.0.0",
            "com.docker.network.bridge.name": "docker0",
            "com.docker.network.driver.mtu": "1500"
        },
        "Labels": {}
    }
]

Offline mmccarn

  • *
  • 2,626
  • +10/-0
Re: Docker container can not ping outside world
« Reply #6 on: August 04, 2017, 11:52:54 PM »
I tried that, but that didn't work for the router field in 'adding local network' should be filled with the bridge IP which is 172.x.x.x

I think you need to add the bridged network with the SME LAN IP as the 'router'. 

My theory is that traffic from the docker container gets out to the SME, which then sends the responses to its own default gateway since it doesn't have a specific route for that network.

So - create a local network for '172.17.0.0/16' with the router set to '192.168.147.2'

guest22

Re: Docker container can not ping outside world
« Reply #7 on: August 05, 2017, 10:14:12 AM »
I can add that network and gateway, but still the same issue...

guest22

Re: Docker container can not ping outside world
« Reply #8 on: August 05, 2017, 11:48:44 AM »
Ok, we may have a game changer here. I CAN ping outside world IP addresses, but no domain names, so it seems to be DNS related....

The container /etc/resolv.conf does have 192.168.147.2. So why can't the container resolve domain names...

I guess I lack the conceptual understanding and skills about docker, networking and IT in general :wink:

guest22

Re: Docker container can not ping outside world
« Reply #9 on: August 05, 2017, 01:30:14 PM »
This is in the iptables log when I try to do an (yum) update from within the container:

denylog: IN=docker0 OUT= MAC=02:42:c0:13:1f:a9:02:42:ac:11:00:02:08:00  SRC=172.17.0.2 DST=192.168.147.2 LEN=64 TOS=00 PREC=0x00 TTL=64 ID=58288 CE DF PROTO=UDP SPT=34873 DPT=53 LEN=44

But I can ping from 172.17.0.2 (container) to 192.168.147.2 (SME10)

So if I read Daniels comments correctly then the default docker0 bridge is to bridge between the local ethernet adapter of SME10 and 1 or more containers. If so, is there an additional bridge required or a firewall change to allow 172.17.0.0 traffic to the outside world?


As a reference: https://docs.docker.com/engine/userguide/networking/default_network/container-communication/#communicating-to-the-outside-world

guest22

Re: [SOLVED] Docker container can not ping outside world
« Reply #10 on: August 06, 2017, 09:42:20 AM »

Solved: The docker container DNS requests to SME Server would not be allowed by SME Server. Adding the --dns x.x.x.x option to the docker run command overcomes this issue. In my case I added --dns 8.8.8.8 --dns 8.8.4.4 (google's DNS)


Not the most elegant way, but the root cause and a workaround is at least known.


Thanks for your time.

Offline Daniel B.

  • *
  • 1,699
  • +0/-0
    • Firewall Services, la sécurité des réseaux
Re: [SOLVED] Docker container can not ping outside world
« Reply #11 on: August 06, 2017, 10:29:07 AM »
What about setting --dns <Local IP of your SME Server> ? The problem is that by default, Docker will try to query your SME Server on the docker0 bridge IP, on which dnscache is not binding (and iptables not allowing it anyway). If the docker0 network has been added as a local network, and you query the Local IP of SME, it "should" work
C'est la fin du monde !!! :lol:

guest22

Re: [SOLVED] Docker container can not ping outside world
« Reply #12 on: August 06, 2017, 10:34:44 AM »
What about setting --dns <Local IP of your SME Server> ? The problem is that by default, Docker will try to query your SME Server on the docker0 bridge IP, on which dnscache is not binding (and iptables not allowing it anyway). If the docker0 network has been added as a local network, and you query the Local IP of SME, it "should" work


Unfortunately I tried that, but did not 'work'. Stil the same deny on port 53 messages in the logs


Offline Daniel B.

  • *
  • 1,699
  • +0/-0
    • Firewall Services, la sécurité des réseaux
Re: [SOLVED] Docker container can not ping outside world
« Reply #13 on: August 06, 2017, 10:37:57 AM »
I'd need to see the full iptables rules to debug this. You should also check what's happening after you run a signal-event remoteaccess-update. As Dockerd injects rules in iptables, I'm not sure how this will interract with SME Server's rules modifications.
C'est la fin du monde !!! :lol:

guest22

Re: [SOLVED] Docker container can not ping outside world
« Reply #14 on: August 06, 2017, 10:40:03 AM »
I'd need to see the full iptables rules to debug this. You should also check what's happening after you run a signal-event remoteaccess-update. As Dockerd injects rules in iptables, I'm not sure how this will interract with SME Server's rules modifications.

Dockers rules will be gone thus a docker engine restart is required after a masq update on SME. Not ideal, but good to know.


These are the rules:
  Chain DOCKER (1 references)
target     prot opt source               destination         
ACCEPT     tcp  --  anywhere             172.17.0.2           tcp dpt:webcache
ACCEPT     tcp  --  anywhere             172.17.0.2           tcp dpt:8000
ACCEPT     tcp  --  anywhere             172.17.0.2           tcp dpt:https


Chain DOCKER-ISOLATION (1 references)
target     prot opt source               destination         
RETURN     all  --  anywhere             anywhere           


Chain DOCKER-USER (1 references)
target     prot opt source               destination         
RETURN     all  --  anywhere             anywhere