[docker 网络][flannel] 配置安装测试

1. 前言

docker跨主机网络有很多种方式, 包括桥接, 路由 以及 用ovs实现跨主机通信等等. 这篇文章主要使用两台机器来用flannel实现docker容器跨主机通信.

环境:

Machine 1 : 172.21.0.16 主机名:master
Machine 2 : 172.21.0.12 主机名:worker

flannel
1. [docker 网络][flannel] 配置安装测试
2. [docker 网络][flannel] 背后操作
3. [docker 网络][flannel] 源码简单分析

2. etcd

由于flannel为了避免ip重复分配, 使用了etcd来解决冲突. 由于测试, 在master(172.21.0.16)只使用了一个单机的etcd. 安装可以参考etcd的单节点手工安装.

etcdctl --endpoints http://172.21.0.16:2379 set /coreos.com/network/config '{"Network": "10.0.0.0/16", "SubnetLen": 24, "SubnetMin": "10.0.1.0","SubnetMax": "10.0.20.0", "Backend": {"Type": "vxlan"}}'

Network: 用于指定Flannel地址池, 整个overlay网络为10.0.0.0/16网段.
SubnetLen: 用于指定分配给单个宿主机的docker0的ip段的子网掩码的长度
SubnetMin: 用于指定最小能够分配的ip段
SudbnetMax: 用于指定最大能够分配的ip段，在上面的示例中，表示每个宿主机可以分配一个24位掩码长度的子网，可以分配的子网从10.0.1.0/24到10.0.20.0/24，也就意味着在这个网段中，最多只能有20台宿主机
Backend: 用于指定数据包以什么方式转发,默认为udp模式, 这里使用的是vxlan模式.

执行如下, 将配置信息放到etcd中保存.

[root@master ~]# etcdctl --endpoints http://172.21.0.16:2379 set /coreos.com/network/config '{"Network": "10.0.0.0/16", "SubnetLen": 24, "SubnetMin": "10.0.1.0","SubnetMax": "10.0.20.0", "Backend": {"Type": "vxlan"}}'
{"Network": "10.0.0.0/16", "SubnetLen": 24, "SubnetMin": "10.0.1.0","SubnetMax": "10.0.20.0", "Backend": {"Type": "vxlan"}}
[root@master ~]# etcdctl get /coreos.com/network/config
{"Network": "10.0.0.0/16", "SubnetLen": 24, "SubnetMin": "10.0.1.0","SubnetMax": "10.0.20.0", "Backend": {"Type": "vxlan"}}

3. 安装flannel

3.1 关闭docker

因为该宿主机的docker使用的子网网络是从flannel中获得, 而不是docker默认的172.17.0.1/16, 所以flannel需要在docker前启动. 以master配置为例, worker配置基本一致.

[root@master ~]# systemctl stop docker

3.2 下载flannel

[root@master flannel]# pwd
/root/flannel
[root@master flannel]# wget https://github.com/coreos/flannel/releases/download/v0.11.0/flannel-v0.11.0-linux-amd64.tar.gz
[root@master flannel]# tar -zxvf flannel-v0.11.0-linux-amd64.tar.gz 
flanneld
mk-docker-opts.sh
README.md
[root@master flannel]# cp flanneld mk-docker-opts.sh /usr/local/bin/
[root@master flannel]#

3.3 启动flannel

[root@master flannel]# /usr/local/bin/flanneld --etcd-endpoints="http://172.21.0.16:2379"
I1102 16:38:51.015597   20734 main.go:514] Determining IP address of default interface
I1102 16:38:51.015795   20734 main.go:527] Using interface with name eth0 and address 172.21.0.16
I1102 16:38:51.015813   20734 main.go:544] Defaulting external address to interface address (172.21.0.16)
I1102 16:38:51.015887   20734 main.go:244] Created subnet manager: Etcd Local Manager with Previous Subnet: None
I1102 16:38:51.015892   20734 main.go:247] Installing signal handlers
I1102 16:38:51.016953   20734 main.go:386] Found network config - Backend type: vxlan
I1102 16:38:51.016988   20734 vxlan.go:120] VXLAN config: VNI=1 Port=0 GBP=false DirectRouting=false
I1102 16:38:51.060136   20734 local_manager.go:234] Picking subnet in range 10.0.1.0 ... 10.0.20.0
I1102 16:38:51.060882   20734 local_manager.go:220] Allocated lease (10.0.13.0/24) to current node (172.21.0.16) 
I1102 16:38:51.061160   20734 main.go:317] Wrote subnet file to /run/flannel/subnet.env
I1102 16:38:51.061169   20734 main.go:321] Running backend.
I1102 16:38:51.061420   20734 vxlan_network.go:60] watching for new subnet leases
I1102 16:38:51.063824   20734 iptables.go:145] Some iptables rules are missing; deleting and recreating rules
I1102 16:38:51.063840   20734 iptables.go:167] Deleting iptables rule: -s 10.0.0.0/16 -j ACCEPT
I1102 16:38:51.063904   20734 main.go:429] Waiting for 22h59m59.996699728s to renew lease
I1102 16:38:51.064971   20734 iptables.go:167] Deleting iptables rule: -d 10.0.0.0/16 -j ACCEPT
I1102 16:38:51.065938   20734 iptables.go:155] Adding iptables rule: -s 10.0.0.0/16 -j ACCEPT
I1102 16:38:51.067710   20734 iptables.go:155] Adding iptables rule: -d 10.0.0.0/16 -j ACCEPT

启动后查看相关变化:

[root@master ~]# etcdctl ls /coreos.com/network/subnets
/coreos.com/network/subnets/10.0.13.0-24
[root@master ~]# etcdctl get /coreos.com/network/subnets/10.0.13.0-24
{"PublicIP":"172.21.0.16","BackendType":"vxlan","BackendData":{"VtepMAC":"aa:52:69:c2:8a:ef"}}
[root@master ~]# ifconfig flannel.1
flannel.1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1450
        inet 10.0.13.0  netmask 255.255.255.255  broadcast 0.0.0.0
        inet6 fe80::a852:69ff:fec2:8aef  prefixlen 64  scopeid 0x20<link>
        ether aa:52:69:c2:8a:ef  txqueuelen 0  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 8 overruns 0  carrier 0  collisions 0

[root@master ~]#

1. 宿主机上多了一个设备flannel.1, 地址为10.0.13.0/32, 并且它的mac地址已经存到了etcd中.
2. etcd分配了一个子网10.0.13.0/24, 所以该宿主机上的docker网络就使用该网段. 所以需要去配置docker0的配置. flannel也提供了修改docker0的网络配置方法, 就是修改docker的启动命令指定网络.

[root@master ~]# cat /run/flannel/subnet.env 
FLANNEL_NETWORK=10.0.0.0/16
FLANNEL_SUBNET=10.0.13.1/24
FLANNEL_MTU=1450
FLANNEL_IPMASQ=false
[root@master ~]# /root/flannel/mk-docker-opts.sh -c
[root@master ~]# cat /run/docker_opts.env
DOCKER_OPTS=" --bip=10.0.13.1/24 --ip-masq=true --mtu=1450"
[root@master ~]#

3.4 修改docker 启动文件

就是把--bip=10.0.13.1/24 --ip-masq=true --mtu=1450放到启动命令后面.

[root@master flannel]# vim /lib/systemd/system/docker.service
...
EnvironmentFile=/run/docker_opts.env
ExecStart=/usr/bin/dockerd $DOCKER_OPTS
...
[root@master flannel]# systemctl daemon-reload
[root@master flannel]# systemctl restart docker
[root@master flannel]#

3.5 查看docker0

[root@master flannel]# ifconfig docker0
docker0: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
        inet 10.0.13.1  netmask 255.255.255.0  broadcast 0.0.0.0
        inet6 fe80::42:62ff:fe53:ac4b  prefixlen 64  scopeid 0x20<link>
        ether 02:42:62:53:ac:4b  txqueuelen 0  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 8  bytes 648 (648.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
flannel.1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1450
        inet 10.0.13.0  netmask 255.255.255.255  broadcast 0.0.0.0
        inet6 fe80::a852:69ff:fec2:8aef  prefixlen 64  scopeid 0x20<link>
        ether aa:52:69:c2:8a:ef  txqueuelen 0  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 8 overruns 0  carrier 0  collisions 0
...

可以看到docker0已经由172.17.0.1/16变化为10.0.13.1/24.

4. 验证

现在两台机器都已经启动了flannel, 并且配置如下:

[root@master flannel]# etcdctl ls /coreos.com/network/subnets
/coreos.com/network/subnets/10.0.13.0-24
/coreos.com/network/subnets/10.0.10.0-24
[root@master flannel]# etcdctl get /coreos.com/network/subnets/10.0.10.0-24
{"PublicIP":"172.21.0.12","BackendType":"vxlan","BackendData":{"VtepMAC":"5e:44:e9:fd:6a:61"}}
[root@master flannel]#

可以看到worker(172.21.0.12)节点的子网是10.0.10.0/24, 并且该机器上flannel.1的地址为5e:44:e9:fd:6a:61.

4.1 在master(172.21.0.16)和worker(172.21.0.12)节点中启动容器

// master(172.21.0.16)
[root@master flannel]# docker run -d --name con1 busybox top
[root@master flannel]# docker ps
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
b8242a6be998        busybox             "top"               25 seconds ago      Up 24 seconds                           con1
[root@master flannel]# docker exec -it con1 sh
/ # ifconfig
eth0      Link encap:Ethernet  HWaddr 02:42:0A:00:0D:02  
          inet addr:10.0.13.2  Bcast:0.0.0.0  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1450  Metric:1
          RX packets:8 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:648 (648.0 B)  TX bytes:0 (0.0 B)

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

/ # route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         10.0.13.1       0.0.0.0         UG    0      0        0 eth0
10.0.13.0       0.0.0.0         255.255.255.0   U     0      0        0 eth0

// worker(172.21.0.12)
[root@worker flannel]# docker run -d --name con1 busybox top
[root@worker flannel]# docker exec -it con1 sh 
/ # ifconfig
eth0      Link encap:Ethernet  HWaddr 02:42:0A:00:0A:02  
          inet addr:10.0.10.2  Bcast:0.0.0.0  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1450  Metric:1
          RX packets:8 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:648 (648.0 B)  TX bytes:0 (0.0 B)

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

/ # route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         10.0.10.1       0.0.0.0         UG    0      0        0 eth0
10.0.10.0       0.0.0.0         255.255.255.0   U     0      0        0 eth0
/ #

4.2 验证相互访问

example-1.png

master的容器con1分别访问worker的容器con1, docker0, flannel.1以及主机.

[root@master flannel]# docker exec -it con1 sh
===> 访问worker的容器con1
/ # ping -c 1 10.0.10.2
PING 10.0.10.2 (10.0.10.2): 56 data bytes
64 bytes from 10.0.10.2: seq=0 ttl=62 time=0.559 ms

--- 10.0.10.2 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 0.559/0.559/0.559 ms

===> 访问worker的容器docker0
/ # ping -c 1 10.0.10.1
PING 10.0.10.1 (10.0.10.1): 56 data bytes
64 bytes from 10.0.10.1: seq=0 ttl=63 time=0.454 ms

--- 10.0.10.1 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 0.454/0.454/0.454 ms

===> 访问worker的容器flannel.1
/ # ping -c 1 10.0.10.0
PING 10.0.10.0 (10.0.10.0): 56 data bytes
64 bytes from 10.0.10.0: seq=0 ttl=63 time=0.475 ms

--- 10.0.10.0 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 0.475/0.475/0.475 ms

===> 访问worker
/ # ping -c 1 172.21.0.12
PING 172.21.0.12 (172.21.0.12): 56 data bytes
64 bytes from 172.21.0.12: seq=0 ttl=63 time=0.384 ms

--- 172.21.0.12 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 0.384/0.384/0.384 ms

worker的容器con1分别访问master的容器con1, docker0, flannel.1以及主机.

[root@worker flannel]# docker exec -it con1 sh
===> 访问master的容器con1
/ # ping -c 1 10.0.13.2
PING 10.0.13.2 (10.0.13.2): 56 data bytes
64 bytes from 10.0.13.2: seq=0 ttl=62 time=0.522 ms

--- 10.0.13.2 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 0.522/0.522/0.522 ms

===> 访问master的容器docker0
/ # ping -c 1 10.0.13.1
PING 10.0.13.1 (10.0.13.1): 56 data bytes
64 bytes from 10.0.13.1: seq=0 ttl=63 time=0.376 ms

--- 10.0.13.1 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 0.376/0.376/0.376 ms

===> 访问master的容器flannel.1
/ # ping -c 1 10.0.13.0
PING 10.0.13.0 (10.0.13.0): 56 data bytes
64 bytes from 10.0.13.0: seq=0 ttl=63 time=0.447 ms

--- 10.0.13.0 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 0.447/0.447/0.447 ms

===> 访问master
/ # ping -c 1 172.21.0.16
PING 172.21.0.16 (172.21.0.16): 56 data bytes
64 bytes from 172.21.0.16: seq=0 ttl=63 time=0.403 ms

--- 172.21.0.16 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 0.403/0.403/0.403 ms

可以看到两个容器以及互相连通起来了.

4.3 在master再启动一个容器

在master再启动一个容器, 验证一下机器内部访问情况.

[root@master flannel]# docker run -d --name con2 busybox top
fb4c2e01f937489e836ae59a513ea5afdd06bd76d101d4543474ddf337a7902f
[root@master flannel]# 
[root@master flannel]# docker exec -it con2 sh
/ # ifconfig
eth0      Link encap:Ethernet  HWaddr 02:42:0A:00:0D:03  
          inet addr:10.0.13.3  Bcast:0.0.0.0  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1450  Metric:1
          RX packets:8 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:648 (648.0 B)  TX bytes:0 (0.0 B)
...
===> 访问同一机器内的容器con1
/ # ping -c 1 10.0.13.2
PING 10.0.13.2 (10.0.13.2): 56 data bytes
64 bytes from 10.0.13.2: seq=0 ttl=64 time=0.097 ms

--- 10.0.13.2 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 0.097/0.097/0.097 ms

===> 访问docker0
/ # ping -c 1 10.0.13.1
PING 10.0.13.1 (10.0.13.1): 56 data bytes
64 bytes from 10.0.13.1: seq=0 ttl=64 time=0.077 ms

--- 10.0.13.1 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 0.077/0.077/0.077 ms

===> 访问本机
/ # ping -c 1 172.21.0.16
PING 172.21.0.16 (172.21.0.16): 56 data bytes
64 bytes from 172.21.0.16: seq=0 ttl=64 time=0.084 ms

--- 172.21.0.16 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 0.084/0.084/0.084 ms

===> 访问外网
/ # ping -c 1 www.baidu.com
PING www.baidu.com (220.181.38.150): 56 data bytes
64 bytes from 220.181.38.150: seq=0 ttl=249 time=5.879 ms

--- www.baidu.com ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 5.879/5.879/5.879 ms
/ #
example-2.png
最后编辑于 ：
©著作权归作者所有,转载或内容合作请联系作者
  • 人面猴
    序言：七十年代末，一起剥皮案震惊了整个滨河市，随后出现的几起案子，更是在滨河造成了极大的恐慌，老刑警刘岩，带你破解...
    沈念sama阅读 227,702评论 6赞 531
  • 死咒
    序言：滨河连续发生了三起死亡事件，死亡现场离奇诡异，居然都是意外死亡，警方通过查阅死者的电脑和手机，发现死者居然都...
    沈念sama阅读 98,143评论 3赞 415
  • 救了他两次的神仙让他今天三更去死
    文/潘晓璐 我一进店门，熙熙楼的掌柜王于贵愁眉苦脸地迎上来，“玉大人，你说我怎么就摊上这事。” “怎么了？”我有些...
    开封第一讲书人阅读 175,553评论 0赞 373
  • 道士缉凶录：失踪的卖姜人
    文/不坏的土叔 我叫张陵，是天一观的道长。 经常有香客问我，道长，这世上最难降的妖魔是什么？ 我笑而不...
    开封第一讲书人阅读 62,620评论 1赞 307
  • 港岛之恋（遗憾婚礼）
    正文 为了忘掉前任，我火速办了婚礼，结果婚礼上，老公的妹妹穿的比我还像新娘。我一直安慰自己，他们只是感情好，可当我...
    茶点故事阅读 71,416评论 6赞 405
  • 恶毒庶女顶嫁案：这布局不是一般人想出来的
    文/花漫 我一把揭开白布。 她就那样静静地躺着，像睡着了一般。 火红的嫁衣衬着肌肤如雪。 梳的纹丝不乱的头发上，一...
    开封第一讲书人阅读 54,940评论 1赞 321
  • 城市分裂传说
    那天，我揣着相机与录音，去河边找鬼。 笑死，一个胖子当着我的面吹牛，可吹牛的内容都是我干的。 我是一名探鬼主播，决...
    沈念sama阅读 43,024评论 3赞 440
  • 双鸳鸯连环套：你想象不到人心有多黑
    文/苍兰香墨 我猛地睁开眼，长吁一口气：“原来是场噩梦啊……” “哼！你这毒妇竟也来了？” 一声冷哼从身侧响起，我...
    开封第一讲书人阅读 42,170评论 0赞 287
  • 万荣杀人案实录
    序言：老挝万荣一对情侣失踪，失踪者是张志新（化名）和其女友刘颖，没想到半个月后，有当地人在树林里发现了一具尸体，经...
    沈念sama阅读 48,709评论 1赞 333
  • 护林员之死
    正文 独居荒郊野岭守林人离奇死亡，尸身上长有42处带血的脓包…… 初始之章·张勋 以下内容为张勋视角 年9月15日...
    茶点故事阅读 40,597评论 3赞 354
  • 白月光启示录
    正文 我和宋清朗相恋三年，在试婚纱的时候发现自己被绿了。 大学时的朋友给我发了我未婚夫和他白月光在一起吃饭的照片。...
    茶点故事阅读 42,784评论 1赞 369
  • 活死人
    序言：一个原本活蹦乱跳的男人离奇死亡，死状恐怖，灵堂内的尸体忽然破棺而出，到底是诈尸还是另有隐情，我是刑警宁泽，带...
    沈念sama阅读 38,291评论 5赞 357
  • 日本核电站爆炸内幕
    正文 年R本政府宣布，位于F岛的核电站，受9级特大地震影响，放射性物质发生泄漏。R本人自食恶果不足惜，却给世界环境...
    茶点故事阅读 44,029评论 3赞 347
  • 男人毒药：我在死后第九天来索命
    文/蒙蒙 一、第九天 我趴在偏房一处隐蔽的房顶上张望。 院中可真热闹，春花似锦、人声如沸。这庄子的主人今日做“春日...
    开封第一讲书人阅读 34,407评论 0赞 25
  • 一桩弑父案，背后竟有这般阴谋
    文/苍兰香墨 我抬头看了看天上的太阳。三九已至，却和暖如春，着一层夹袄步出监牢的瞬间，已是汗流浃背。 一阵脚步声响...
    开封第一讲书人阅读 35,663评论 1赞 280
  • 情欲美人皮
    我被黑心中介骗来泰国打工， 没想到刚下飞机就差点儿被人妖公主榨干…… 1. 我叫王不留，地道东北人。 一个月前我还...
    沈念sama阅读 51,403评论 3赞 390
  • 代替公主和亲
    正文 我出身青楼，却偏偏与公主长得像，于是被迫代替她去往敌国和亲。 传闻我的和亲对象是个残疾皇子，可洞房花烛夜当晚...
    茶点故事阅读 47,746评论 2赞 370