docker容器之间的网络通信会被`ctstate INVALID`规则丢弃

[Forgo7ten]

我有一个cfwp容器，ip为172.20.0.2，提供一个socks5 0.0.0.0:1080
我另外有一个serv容器，ip为172.20.0.3，这两个容器加入了同一个docker网络（docker network create cf-warp-network）

安装ufw-docker命令为：

sudo wget -O /usr/local/bin/ufw-docker \
  https://github.com/chaifeng/ufw-docker/raw/master/ufw-docker
sudo chmod +x /usr/local/bin/ufw-docker
sudo ufw-docker check
sudo ufw-docker install --system
sudo systemctl restart ufw

安装的iptables规则为

< -A DOCKER-USER -j ufw-user-forward
<
< -A DOCKER-USER -m conntrack --ctstate RELATED,ESTABLISHED -j RETURN
< -A DOCKER-USER -m conntrack --ctstate INVALID -j DROP
< -A DOCKER-USER -i docker0 -o docker0 -j ACCEPT
<
< -A DOCKER-USER -j RETURN -s 172.17.0.0/16
< -A DOCKER-USER -j RETURN -s 172.18.0.0/16
< -A DOCKER-USER -j RETURN -s 172.19.0.0/16
< -A DOCKER-USER -j RETURN -s 172.20.0.0/16
< -A DOCKER-USER -j RETURN -s 172.21.0.0/16
< -A DOCKER-USER -j RETURN -s 172.22.0.0/16
< -A DOCKER-USER -j RETURN -s 172.23.0.0/16
< -A DOCKER-USER -j ufw-docker-logging-deny -m conntrack --ctstate NEW -d 172.17.0.0/16
< -A DOCKER-USER -j ufw-docker-logging-deny -m conntrack --ctstate NEW -d 172.18.0.0/16
< -A DOCKER-USER -j ufw-docker-logging-deny -m conntrack --ctstate NEW -d 172.19.0.0/16
< -A DOCKER-USER -j ufw-docker-logging-deny -m conntrack --ctstate NEW -d 172.20.0.0/16
< -A DOCKER-USER -j ufw-docker-logging-deny -m conntrack --ctstate NEW -d 172.21.0.0/16
< -A DOCKER-USER -j ufw-docker-logging-deny -m conntrack --ctstate NEW -d 172.22.0.0/16
< -A DOCKER-USER -j ufw-docker-logging-deny -m conntrack --ctstate NEW -d 172.23.0.0/16
<
< -A DOCKER-USER -j RETURN
<
< -A ufw-docker-logging-deny -m limit --limit 3/min --limit-burst 10 -j LOG --log-prefix "[UFW DOCKER BLOCK] "
< -A ufw-docker-logging-deny -j DROP

测试过程中172.20.0.3无法访问172.20.0.2:1080

❯  sudo tcpdump -ni br-3b9caf4a5628 'host 172.20.0.2 or host 172.20.0.3'
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on br-3b9caf4a5628, link-type EN10MB (Ethernet), snapshot length 262144 bytes
00:50:13.599971 IP 172.20.0.3.36568 > 172.20.0.2.1080: Flags [.], ack 4186176001, win 83, options [nop,nop,TS val 637116800 ecr 3045207152], length 0
00:50:13.599976 IP 172.20.0.2.56780 > 162.159.192.1.2408: UDP, length 32
00:50:13.600108 IP 172.20.0.2.1080 > 172.20.0.3.36568: Flags [.], ack 1, win 626, options [nop,nop,TS val 3045222256 ecr 637041157], length 0
00:50:13.600163 IP 172.20.0.2.1080 > 172.20.0.3.36568: Flags [.], ack 1, win 626, options [nop,nop,TS val 3045222256 ecr 637041157], length 0
00:50:27.959505 IP 172.20.0.3.36568 > 172.20.0.2.1080: Flags [P.], seq 1:25, ack 1, win 83, options [nop,nop,TS val 637131159 ecr 3045222256], length 24
00:50:27.959825 IP 172.20.0.2.56780 > 162.159.192.1.2408: UDP, length 112
00:50:27.960004 IP 172.20.0.3.36568 > 172.20.0.2.1080: Flags [F.], seq 25, ack 1, win 83, options [nop,nop,TS val 637131160 ecr 3045222256], length 0
00:50:27.960078 IP 172.20.0.2.56780 > 162.159.192.1.2408: UDP, length 96
00:50:27.960133 IP 172.20.0.2.1080 > 172.20.0.3.36568: Flags [F.], seq 1, ack 26, win 626, options [nop,nop,TS val 3045236616 ecr 637131159], length 0
00:50:27.960162 IP 172.20.0.2.1080 > 172.20.0.3.36568: Flags [F.], seq 1, ack 26, win 626, options [nop,nop,TS val 3045236616 ecr 637131159], length 0
00:50:27.960192 IP 172.20.0.3.36568 > 172.20.0.2.1080: Flags [.], ack 2, win 83, options [nop,nop,TS val 637131160 ecr 3045236616], length 0
00:50:27.974323 IP 162.159.192.1.2408 > 172.20.0.2.56780: UDP, length 84
00:50:27.974884 IP 162.159.192.1.2408 > 172.20.0.2.56780: UDP, length 84
00:50:27.975016 IP 172.20.0.2.56780 > 162.159.192.1.2408: UDP, length 96
00:50:28.792885 IP 172.20.0.3.42106 > 172.20.0.2.1080: Flags [S], seq 1318813491, win 42340, options [mss 1460,sackOK,TS val 637131993 ecr 0,nop,wscale 10], length 0
00:50:28.792959 IP 172.20.0.2.1080 > 172.20.0.3.42106: Flags [S.], seq 2082133992, ack 1318813492, win 43440, options [mss 1460,sackOK,TS val 3045237449 ecr 637131993,nop,wscale 10], length 0
00:50:29.824142 IP 172.20.0.3.42106 > 172.20.0.2.1080: Flags [S], seq 1318813491, win 42340, options [mss 1460,sackOK,TS val 637133024 ecr 0,nop,wscale 10], length 0
00:50:29.824159 IP 172.20.0.2.1080 > 172.20.0.3.42106: Flags [S.], seq 2082133992, ack 1318813492, win 43440, options [mss 1460,sackOK,TS val 3045238480 ecr 637131993,nop,wscale 10], length 0
00:50:29.824349 IP 172.20.0.2.1080 > 172.20.0.3.42106: Flags [S.], seq 2082133992, ack 1318813492, win 43440, options [mss 1460,sackOK,TS val 3045238480 ecr 637131993,nop,wscale 10], length 0
00:50:31.839908 IP 172.20.0.2.1080 > 172.20.0.3.42106: Flags [S.], seq 2082133992, ack 1318813492, win 43440, options [mss 1460,sackOK,TS val 3045240496 ecr 637131993,nop,wscale 10], length 0
00:50:31.839951 IP 172.20.0.3.42106 > 172.20.0.2.1080: Flags [S], seq 1318813491, win 42340, options [mss 1460,sackOK,TS val 637135040 ecr 0,nop,wscale 10], length 0
00:50:31.839976 IP 172.20.0.2.1080 > 172.20.0.3.42106: Flags [S.], seq 2082133992, ack 1318813492, win 43440, options [mss 1460,sackOK,TS val 3045240496 ecr 637131993,nop,wscale 10], length 0
00:50:33.055822 ARP, Request who-has 172.20.0.2 tell 172.20.0.1, length 28
00:50:33.055937 ARP, Request who-has 172.20.0.3 tell 172.20.0.1, length 28
00:50:33.056032 ARP, Reply 172.20.0.2 is-at ee:27:f2:e6:a3:ac, length 28
00:50:33.056037 ARP, Reply 172.20.0.3 is-at aa:dc:9b:bf:45:c4, length 28
00:50:35.872026 IP 172.20.0.3.42106 > 172.20.0.2.1080: Flags [S], seq 1318813491, win 42340, options [mss 1460,sackOK,TS val 637139072 ecr 0,nop,wscale 10], length 0
00:50:35.872029 IP 172.20.0.2.1080 > 172.20.0.3.42106: Flags [S.], seq 2082133992, ack 1318813492, win 43440, options [mss 1460,sackOK,TS val 3045244528 ecr 637131993,nop,wscale 10], length 0
00:50:35.872165 IP 172.20.0.2.1080 > 172.20.0.3.42106: Flags [S.], seq 2082133992, ack 1318813492, win 43440, options [mss 1460,sackOK,TS val 3045244528 ecr 637131993,nop,wscale 10], length 0
00:50:42.186990 IP 172.20.0.3.42220 > 172.20.0.2.1080: Flags [S], seq 462347753, win 42340, options [mss 1460,sackOK,TS val 637145387 ecr 0,nop,wscale 10], length 0
00:50:42.187089 IP 172.20.0.2.1080 > 172.20.0.3.42220: Flags [S.], seq 2153574991, ack 462347754, win 43440, options [mss 1460,sackOK,TS val 3045250843 ecr 637145387,nop,wscale 10], length 0
00:50:43.039927 IP 172.20.0.2.56780 > 162.159.192.1.2408: UDP, length 32
00:50:43.199833 IP 172.20.0.3.42220 > 172.20.0.2.1080: Flags [S], seq 462347753, win 42340, options [mss 1460,sackOK,TS val 637146400 ecr 0,nop,wscale 10], length 0
00:50:43.199838 IP 172.20.0.2.1080 > 172.20.0.3.42220: Flags [S.], seq 2153574991, ack 462347754, win 43440, options [mss 1460,sackOK,TS val 3045251856 ecr 637145387,nop,wscale 10], length 0
00:50:43.199898 IP 172.20.0.2.1080 > 172.20.0.3.42220: Flags [S.], seq 2153574991, ack 462347754, win 43440, options [mss 1460,sackOK,TS val 3045251856 ecr 637145387,nop,wscale 10], length 0
00:50:43.552165 IP 172.20.0.2.56780 > 162.159.192.1.2408: UDP, length 148
00:50:43.567713 IP 162.159.192.1.2408 > 172.20.0.2.56780: UDP, length 92
00:50:43.568080 IP 172.20.0.2.56780 > 162.159.192.1.2408: UDP, length 32
00:50:44.063813 IP 172.20.0.2.1080 > 172.20.0.3.42106: Flags [S.], seq 2082133992, ack 1318813492, win 43440, options [mss 1460,sackOK,TS val 3045252720 ecr 637131993,nop,wscale 10], length 0
00:50:45.215815 IP 172.20.0.2.1080 > 172.20.0.3.42220: Flags [S.], seq 2153574991, ack 462347754, win 43440, options [mss 1460,sackOK,TS val 3045253872 ecr 637145387,nop,wscale 10], length 0
00:50:45.215852 IP 172.20.0.3.42220 > 172.20.0.2.1080: Flags [S], seq 462347753, win 42340, options [mss 1460,sackOK,TS val 637148416 ecr 0,nop,wscale 10], length 0
00:50:45.215875 IP 172.20.0.2.1080 > 172.20.0.3.42220: Flags [S.], seq 2153574991, ack 462347754, win 43440, options [mss 1460,sackOK,TS val 3045253872 ecr 637145387,nop,wscale 10], length 0
00:50:47.391835 ARP, Request who-has 172.20.0.1 tell 172.20.0.2, length 28
00:50:47.391845 ARP, Reply 172.20.0.1 is-at be:97:b6:7b:6f:4b, length 28
00:50:49.439755 IP 172.20.0.2.1080 > 172.20.0.3.42220: Flags [S.], seq 2153574991, ack 462347754, win 43440, options [mss 1460,sackOK,TS val 3045258096 ecr 637145387,nop,wscale 10], length 0
^C

cfwp:1080 -> serv:临时端口 的 SYN-ACK 被丢弃

查看iptables计数，发现 ctstate INVALID 计数增长（两次之间执行的命令：docker exec -it serv curl -v -x socks5://cfwarp:1080 ipinfo.io/ip ）

# iptables -vnL DOCKER-USER --line-numbers
# Warning: iptables-legacy tables present, use iptables-legacy to see them
Chain DOCKER-USER (1 references)
num   pkts bytes target     prot opt in     out     source               destination
1     141K   82M ufw-user-forward  all  --  *      *       0.0.0.0/0            0.0.0.0/0
2     108K   63M RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0            ctstate RELATED,ESTABLISHED
3       71  4260 DROP       all  --  *      *       0.0.0.0/0            0.0.0.0/0            ctstate INVALID

# iptables -vnL DOCKER-USER --line-numbers
# Warning: iptables-legacy tables present, use iptables-legacy to see them
Chain DOCKER-USER (1 references)
num   pkts bytes target     prot opt in     out     source               destination
1     141K   82M ufw-user-forward  all  --  *      *       0.0.0.0/0            0.0.0.0/0
2     108K   63M RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0            ctstate RELATED,ESTABLISHED
3       77  4620 DROP       all  --  *      *       0.0.0.0/0            0.0.0.0/0            ctstate INVALID

这该如何解决呢？我目前只能通过如下命令缓解：

sudo ufw route allow from 172.20.0.0/16 to 172.20.0.0/16