Koozali.org: home of the SME Server

[SOLVED] Docker container can not ping outside world

guest22

[SOLVED] Docker container can not ping outside world
« on: August 04, 2017, 12:11:16 PM »
Hi,


I am doing some testing with SME10 (Server-gateway) and Docker.


A default install of docker and a test container (centos:centos7) will install correctly. But from within the container I am not able to ping the outside world. I can ping the internal external ethernet of the SME10 host tho.
Obviously the container receives a private IP address from docker (172.x.x.x) and a bridge is automatically created on the SME10 host. Also new iptables docker chainesmasqueraiding rules are added to allow 172 traffic.


I am above my head on this, does somebody have a clue to conceptually make his work with SME10 please?


TIA

Offline mmccarn

  • *
  • 2,626
  • +10/-0
Re: Docker container can not ping outside world
« Reply #1 on: August 04, 2017, 01:28:44 PM »
Can you add the docker subnet as a 'local network' in server-manager?

guest22

Re: Docker container can not ping outside world
« Reply #2 on: August 04, 2017, 01:50:48 PM »
I tried that, but that didn't work for the router field in 'adding local network' should be filled with the bridge IP which is 172.x.x.x


Thanks. Breaking my brains over this, but I'm sure the solution lays within SME firewall stuff.

Offline Daniel B.

  • *
  • 1,699
  • +0/-0
    • Firewall Services, la sécurité des réseaux
Re: Docker container can not ping outside world
« Reply #3 on: August 04, 2017, 01:54:00 PM »
Check in /var/log/iptables/current as it might be dropped by iptables on SME. You'd better install smeserver-bridge-interface, and then use pipework to attach your docker the br0 so that it'll be on the local network. (this is where Docker networking is lacking: you can't attach a Docker to a manually created bridge AFAIK)
C'est la fin du monde !!! :lol:

guest22

Re: Docker container can not ping outside world
« Reply #4 on: August 04, 2017, 02:08:21 PM »
You'd better install smeserver-bridge-interface


Thanks, my head is spinning. the bridge contrib is not available for SME10 or is it?

guest22

Re: Docker container can not ping outside world
« Reply #5 on: August 04, 2017, 02:30:37 PM »
Check in /var/log/iptables/current as it might be dropped by iptables on SME.

Docker version 17.06.0-ce, build 02c1d87


It seems that the container sends it's packets to the internal NIC (192....) and it stops there...

denylog: IN=docker0 OUT= MAC=02:42:c0:13:1f:a9:02:42:ac:11:00:02:08:00  SRC=172.17.0.2 DST=192.168.147.2 LEN=82 TOS=00 PREC=0x00 TTL=64 ID=37635 CE DF PROTO=UDP SPT=45006 DPT=53 LEN=62

and this the auto created bridge on SME by docker at docker service start:

  docker0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 172.17.0.1  netmask 255.255.0.0  broadcast 0.0.0.0
        ether 02:42:c0:13:1f:a9  txqueuelen 0  (Ethernet)
        RX packets 122  bytes 33405 (32.6 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 106  bytes 25973 (25.3 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0


and this is the container's ethernet:


eth0      Link encap:Ethernet  HWaddr 02:42:ac:11:00:02 
          inet addr:172.17.0.2  Bcast:0.0.0.0  Mask:255.255.0.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:109 errors:0 dropped:0 overruns:0 frame:0
          TX packets:127 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
 
          RX bytes:26253 (26.2 KB)  TX bytes:35525 (35.5 KB)


I can ping from the container to SME on both internal and external NIC's

and this are the auto added iptables rules by docker at start of the docker service:


Chain DOCKER (1 references)
target     prot opt source               destination         


Chain DOCKER-ISOLATION (1 references)
target     prot opt source               destination         
RETURN     all  --  anywhere             anywhere           


Chain DOCKER-USER (1 references)
target     prot opt source               destination         
 
RETURN     all  --  anywhere             anywhere           

 


And these are the details of the auto created bridge:


  [
    {
        "Name": "bridge",
        "Id": "7f5f9cdc4f7ec26d71ca5c21c203472bbb94ab20648ebb4ee717a03bf308a67a",
        "Created": "2017-08-04T13:47:02.564051444+02:00",
        "Scope": "local",
        "Driver": "bridge",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "172.17.0.0/16",
                    "Gateway": "172.17.0.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {},
        "Options": {
            "com.docker.network.bridge.default_bridge": "true",
            "com.docker.network.bridge.enable_icc": "true",
            "com.docker.network.bridge.enable_ip_masquerade": "true",
            "com.docker.network.bridge.host_binding_ipv4": "0.0.0.0",
            "com.docker.network.bridge.name": "docker0",
            "com.docker.network.driver.mtu": "1500"
        },
        "Labels": {}
    }
]

Offline mmccarn

  • *
  • 2,626
  • +10/-0
Re: Docker container can not ping outside world
« Reply #6 on: August 04, 2017, 11:52:54 PM »
I tried that, but that didn't work for the router field in 'adding local network' should be filled with the bridge IP which is 172.x.x.x

I think you need to add the bridged network with the SME LAN IP as the 'router'. 

My theory is that traffic from the docker container gets out to the SME, which then sends the responses to its own default gateway since it doesn't have a specific route for that network.

So - create a local network for '172.17.0.0/16' with the router set to '192.168.147.2'

guest22

Re: Docker container can not ping outside world
« Reply #7 on: August 05, 2017, 10:14:12 AM »
I can add that network and gateway, but still the same issue...

guest22

Re: Docker container can not ping outside world
« Reply #8 on: August 05, 2017, 11:48:44 AM »
Ok, we may have a game changer here. I CAN ping outside world IP addresses, but no domain names, so it seems to be DNS related....

The container /etc/resolv.conf does have 192.168.147.2. So why can't the container resolve domain names...

I guess I lack the conceptual understanding and skills about docker, networking and IT in general :wink:

guest22

Re: Docker container can not ping outside world
« Reply #9 on: August 05, 2017, 01:30:14 PM »
This is in the iptables log when I try to do an (yum) update from within the container:

denylog: IN=docker0 OUT= MAC=02:42:c0:13:1f:a9:02:42:ac:11:00:02:08:00  SRC=172.17.0.2 DST=192.168.147.2 LEN=64 TOS=00 PREC=0x00 TTL=64 ID=58288 CE DF PROTO=UDP SPT=34873 DPT=53 LEN=44

But I can ping from 172.17.0.2 (container) to 192.168.147.2 (SME10)

So if I read Daniels comments correctly then the default docker0 bridge is to bridge between the local ethernet adapter of SME10 and 1 or more containers. If so, is there an additional bridge required or a firewall change to allow 172.17.0.0 traffic to the outside world?


As a reference: https://docs.docker.com/engine/userguide/networking/default_network/container-communication/#communicating-to-the-outside-world

guest22

Re: [SOLVED] Docker container can not ping outside world
« Reply #10 on: August 06, 2017, 09:42:20 AM »

Solved: The docker container DNS requests to SME Server would not be allowed by SME Server. Adding the --dns x.x.x.x option to the docker run command overcomes this issue. In my case I added --dns 8.8.8.8 --dns 8.8.4.4 (google's DNS)


Not the most elegant way, but the root cause and a workaround is at least known.


Thanks for your time.

Offline Daniel B.

  • *
  • 1,699
  • +0/-0
    • Firewall Services, la sécurité des réseaux
Re: [SOLVED] Docker container can not ping outside world
« Reply #11 on: August 06, 2017, 10:29:07 AM »
What about setting --dns <Local IP of your SME Server> ? The problem is that by default, Docker will try to query your SME Server on the docker0 bridge IP, on which dnscache is not binding (and iptables not allowing it anyway). If the docker0 network has been added as a local network, and you query the Local IP of SME, it "should" work
C'est la fin du monde !!! :lol:

guest22

Re: [SOLVED] Docker container can not ping outside world
« Reply #12 on: August 06, 2017, 10:34:44 AM »
What about setting --dns <Local IP of your SME Server> ? The problem is that by default, Docker will try to query your SME Server on the docker0 bridge IP, on which dnscache is not binding (and iptables not allowing it anyway). If the docker0 network has been added as a local network, and you query the Local IP of SME, it "should" work


Unfortunately I tried that, but did not 'work'. Stil the same deny on port 53 messages in the logs


Offline Daniel B.

  • *
  • 1,699
  • +0/-0
    • Firewall Services, la sécurité des réseaux
Re: [SOLVED] Docker container can not ping outside world
« Reply #13 on: August 06, 2017, 10:37:57 AM »
I'd need to see the full iptables rules to debug this. You should also check what's happening after you run a signal-event remoteaccess-update. As Dockerd injects rules in iptables, I'm not sure how this will interract with SME Server's rules modifications.
C'est la fin du monde !!! :lol:

guest22

Re: [SOLVED] Docker container can not ping outside world
« Reply #14 on: August 06, 2017, 10:40:03 AM »
I'd need to see the full iptables rules to debug this. You should also check what's happening after you run a signal-event remoteaccess-update. As Dockerd injects rules in iptables, I'm not sure how this will interract with SME Server's rules modifications.

Dockers rules will be gone thus a docker engine restart is required after a masq update on SME. Not ideal, but good to know.


These are the rules:
  Chain DOCKER (1 references)
target     prot opt source               destination         
ACCEPT     tcp  --  anywhere             172.17.0.2           tcp dpt:webcache
ACCEPT     tcp  --  anywhere             172.17.0.2           tcp dpt:8000
ACCEPT     tcp  --  anywhere             172.17.0.2           tcp dpt:https


Chain DOCKER-ISOLATION (1 references)
target     prot opt source               destination         
RETURN     all  --  anywhere             anywhere           


Chain DOCKER-USER (1 references)
target     prot opt source               destination         
RETURN     all  --  anywhere             anywhere           

guest22

Re: [SOLVED] Docker container can not ping outside world
« Reply #15 on: August 06, 2017, 10:47:50 AM »
Sorry, the 8000 and https ports were added by me. So only webcache in Chain Docker.

Offline Daniel B.

  • *
  • 1,699
  • +0/-0
    • Firewall Services, la sécurité des réseaux
Re: [SOLVED] Docker container can not ping outside world
« Reply #16 on: August 06, 2017, 10:59:32 AM »
Ideally, Dockerd should not mess directly with iptables but instead call a wrapper which register the needed rules in an SME DB, so they can be handled in templates and persist accross masq restart/adjust. That's what I did with fail2ban for example. But I don't know if it's feasable.
C'est la fin du monde !!! :lol:

guest22

Re: [SOLVED] Docker container can not ping outside world
« Reply #17 on: August 06, 2017, 11:21:35 AM »
Ideally, Dockerd should not mess directly with iptables but instead call a wrapper which register the needed rules in an SME DB, so they can be handled in templates and persist accross masq restart/adjust. That's what I did with fail2ban for example. But I don't know if it's feasable.


They did have the --no-iptables option, but it was removed from the latest version of docker.They also removed masquerading rules and introduced the chains.

Offline Daniel B.

  • *
  • 1,699
  • +0/-0
    • Firewall Services, la sécurité des réseaux
Re: [SOLVED] Docker container can not ping outside world
« Reply #18 on: August 06, 2017, 11:24:41 AM »
there's --iptables=false. But the goal would not be to disable it entirely, but to use a wrapper instead.
C'est la fin du monde !!! :lol:

guest22

Re: [SOLVED] Docker container can not ping outside world
« Reply #19 on: August 06, 2017, 11:32:44 AM »
there's --iptables=false. But the goal would not be to disable it entirely, but to use a wrapper instead.


That's what I meant, could not remember it by hard. In docker 17.06 that is no longer available.

Offline Daniel B.

  • *
  • 1,699
  • +0/-0
    • Firewall Services, la sécurité des réseaux
Re: [SOLVED] Docker container can not ping outside world
« Reply #20 on: August 06, 2017, 11:40:41 AM »
Wow, if that's true, docker just went from "good prototyping tool, but not meant for prod use" to "little useless toy" in my eyes.
C'est la fin du monde !!! :lol:

guest22

Re: [SOLVED] Docker container can not ping outside world
« Reply #21 on: August 07, 2017, 10:43:10 AM »
Wow, if that's true, docker just went from "good prototyping tool, but not meant for prod use" to "little useless toy" in my eyes.

Please don't get me wrong, I am not an expert on Docker and the latest version (17.06CE) and previous documentation and google results differ, and I don't have the details. There are a zillion things I may be mistaking in.

I came across this slide deck which may help to get a better picture https://www.slideshare.net/SreenivasMakam/docker-networking-common-issues-and-troubleshooting-techniques. But then again, it may not apply to the latest Community Edition 17.06....


Reference documentation https://docs.docker.com/engine/userguide/networking/default_network/container-communication/