tcp状态机
上图是 tcp 状态流转，建立连接时三次握手，断开时四次。搭建双臂 full-nat 测试时发现，断开也通常是三次握手。并且 dpvs 流表的 tcp 状态机和教科书的差别很大，具体表现为无论主动还是被动关闭，tcp 状态都只会走到 TIME_WAIT, 而不是 LAST_ACK, 为什么呢？
测试环境
VIP: 202.108.10.1:6379
LIP: 192.168.168.7
RSIP: 192.168.168.2
CLIENT: 202.108.10.2
root@dpvs:~/dpvs# dpip addr show
inet 202.108.10.1/32 scope global dpdk0
valid_lft forever preferred_lft forever
inet 192.168.168.3/32 scope global dpdk1
valid_lft forever preferred_lft forever
inet 192.168.168.7/32 scope global dpdk1
valid_lft forever preferred_lft forever sa_used 1 sa_free 903151 sa_miss 0
root@dpvs:~/dpvs# ipvsadm -ln
IP Virtual Server version 0.0.0 (size=0)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 202.108.10.1:6379 wrr
-> 192.168.168.2:6379 FullNat 100 0 1
抓包分析
dpvs 编绎时一定打开 DEBUG 模式，日志也要调成 DEBUG 级别。客户端发送 redis 命令并主动断开连接
redis-cli -h 202.108.10.1 -p 6379 get set testclienthhhhhhhhhh testclienthhhhhhhhhh
客户端抓包结果
22:11:13.362012 IP 202.108.10.2.33896 > 202.108.10.1.6379: Flags [S], seq 4164191644, win 29200, options [mss 1460,sackOK,TS val 125357496 ecr 0,nop,wscale 7], length 0
22:11:13.362834 IP 202.108.10.1.6379 > 202.108.10.2.33896: Flags [S.], seq 4072520983, ack 4164191645, win 29200, options [mss 1452,nop,nop,sackOK,nop,wscale 7], length 0
22:11:13.362877 IP 202.108.10.2.33896 > 202.108.10.1.6379: Flags [.], ack 1, win 229, length 0
22:11:13.362926 IP 202.108.10.2.33896 > 202.108.10.1.6379: Flags [P.], seq 1:77, ack 1, win 229, length 76
22:11:13.363255 IP 202.108.10.1.6379 > 202.108.10.2.33896: Flags [.], ack 77, win 229, length 0
22:11:13.363406 IP 202.108.10.1.6379 > 202.108.10.2.33896: Flags [P.], seq 1:51, ack 77, win 229, length 50
22:11:13.363437 IP 202.108.10.2.33896 > 202.108.10.1.6379: Flags [.], ack 51, win 229, length 0
22:11:13.363598 IP 202.108.10.2.33896 > 202.108.10.1.6379: Flags [F.], seq 77, ack 51, win 229, length 0
22:11:13.364140 IP 202.108.10.1.6379 > 202.108.10.2.33896: Flags [F.], seq 51, ack 78, win 229, length 0
22:11:13.364153 IP 202.108.10.2.33896 > 202.108.10.1.6379: Flags [.], ack 52, win 229, length 0
首先，202.108.10.2:33896 与 202.108.10.1:6379 进行三次握手，然后发送数据。最后断开时是重点，只有三次挥手。后端 RS 202.108.10.1:6379 返回一个 [F.] 包，也就是 fin + ack 包，相当于原来四次挥手的二三步合并成一个包了。
服务端抓包结果
22:11:26.081295 IP 192.168.168.7.1028 > 192.168.168.2.6379: Flags [S], seq 2976553321, win 29200, options [exp-8468,mss 1460,sackOK,nop,nop,nop,nop,nop,nop,nop,nop,nop,nop,nop,wscale 7], length 0
22:11:26.081325 IP 192.168.168.2.6379 > 192.168.168.7.1028: Flags [S.], seq 4072520983, ack 2976553322, win 29200, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0
22:11:26.081825 IP 192.168.168.7.1028 > 192.168.168.2.6379: Flags [.], ack 1, win 229, options [exp-8468], length 0
22:11:26.081858 IP 192.168.168.7.1028 > 192.168.168.2.6379: Flags [P.], seq 1:77, ack 1, win 229, options [exp-8468], length 76: RESP "get" "set" "testclienthhhhhhhhhh" "testclienthhhhhhhhhh"
22:11:26.081870 IP 192.168.168.2.6379 > 192.168.168.7.1028: Flags [.], ack 77, win 229, length 0
22:11:26.081990 IP 192.168.168.2.6379 > 192.168.168.7.1028: Flags [P.], seq 1:51, ack 77, win 229, length 50: RESP "ERR wrong number of arguments for 'get' command"
22:11:26.082308 IP 192.168.168.7.1028 > 192.168.168.2.6379: Flags [.], ack 51, win 229, length 0
22:11:26.082524 IP 192.168.168.7.1028 > 192.168.168.2.6379: Flags [F.], seq 77, ack 51, win 229, length 0
22:11:26.082624 IP 192.168.168.2.6379 > 192.168.168.7.1028: Flags [F.], seq 51, ack 78, win 229, length 0
22:11:26.083037 IP 192.168.168.7.1028 > 192.168.168.2.6379: Flags [.], ack 52, win 229, length 0
服务端也正常，三次握手，传输数据，最后同样三次挥手断开连接。
dpvs 日志输出
IPVS: conn lookup: [5] TCP 202.108.10.2:33896 -> 202.108.10.1:6379 miss
SAPOOL: sa_pool_fetch: 192.168.168.7:1028 fetched!
IPVS: new conn: [5] TCP 202.108.10.2:33896 202.108.10.1:6379 192.168.168.7:1028 192.168.168.2:6379 refs 2
IPVS: state trans: TCP in [S...] 202.108.10.2:33896->192.168.168.2:6379 state NONE->SYN_RECV conn.refcnt 2
IPVS: conn lookup: [5] TCP 192.168.168.2:6379 -> 192.168.168.7:1028 hit NEIGHBOUR: REACHABLE trans state to REACHABLE.
IPVS: conn lookup: [5] TCP 202.108.10.2:33896 -> 202.108.10.1:6379 hit
IPVS: state trans: TCP in [..A.] 202.108.10.2:33896->192.168.168.2:6379 state SYN_RECV->ESTABLISHED conn.refcnt 2
IPVS: conn lookup: [5] TCP 202.108.10.2:33896 -> 202.108.10.1:6379 hit
IPVS: conn lookup: [5] TCP 192.168.168.2:6379 -> 192.168.168.7:1028 hit
IPVS: conn lookup: [5] TCP 192.168.168.2:6379 -> 192.168.168.7:1028 hit
IPVS: conn lookup: [5] TCP 202.108.10.2:33896 -> 202.108.10.1:6379 hit
IPVS: conn lookup: [5] TCP 202.108.10.2:33896 -> 202.108.10.1:6379 hit
IPVS: state trans: TCP in [.FA.] 202.108.10.2:33896->192.168.168.2:6379 state ESTABLISHED->CLOSE_WAIT conn.refcnt 2
IPVS: conn lookup: [5] TCP 192.168.168.2:6379 -> 192.168.168.7:1028 hit
IPVS: state trans: TCP out [.FA.] 202.108.10.2:33896->192.168.168.2:6379 state CLOSE_WAIT->TIME_WAIT conn.refcnt 2
IPVS: conn lookup: [5] TCP 202.108.10.2:33896 -> 202.108.10.1:6379 hit
重点在 dpvs 日志，我们逐条分析。
- 客户端 202.108.10.2:33896 发送 syn 请求建连，但是此时
conn lookup查找流表肯定不存在的，也就是第一条的 miss - 由于是双臂模式，并且 fdir 的存在，必须在 LIP 上分配一个端口，那就是从 sa_pool 中分配 1028 lport
- new conn 真正的建立了连接五元组: proto, client, vip, lip, rs
- state trans 将连接由 state NONE 变成 SYN_RECV 状态
- conn lookup hit 由后端 6379 返回的包命中了流表，然后 NEIGHBOUR 更新邻居表。其实这次的包就是三次握手的第二个 sync + ack
6-7. 完成 tcp 三次握手，状态由 state SYN_RECV 变更成 ESTABLISHED
8-11. 这几行都在双向传输数据，命中了流表
12-13. 这两行由 client 202.108.10.2:33896 发送 [.FA.] 主动关闭连接。 流表状态由 ESTABLISHED 变成了 CLOSE_WAIT
14-15. 这两行由后端 rs 192.168.168.2:6379 发送了 [.FA.] 包，也就是四次挥手中的二三步合一起了。状态由 CLOSE_WAIT 变成 TIME_WAIT
- 最后客户端发送的 ack 包命中了流表
查看 ipvsadm
root@dpvs:~/dpvs# ipvsadm -ln -c
[5]tcp 7s TIME_WAIT 202.108.10.2:33896 202.108.10.1:6379 192.168.168.7:1028 192.168.168.2:6379
最后可以看到流表中连接一直处于 TIME_WAIT 状态，直到超时被 conn_expire 删除。上面的步聚是 client 主动关闭，其实测试 rs 主动关闭也是一样的。
查看源码
static struct tcp_state tcp_states[] = {
/* INPUT */
/* sNO, sES, sSS, sSR, sFW, sTW, sCL, sCW, sLA, sLI, sSA */
/*syn*/ {{sSR, sES, sES, sSR, sSR, sSR, sSR, sSR, sSR, sSR, sSR}},
/*fin*/ {{sCL, sCW, sSS, sTW, sTW, sTW, sCL, sCW, sLA, sLI, sTW}},
/*ack*/ {{sCL, sES, sSS, sES, sFW, sTW, sCL, sCW, sCL, sLI, sES}},
/*rst*/ {{sCL, sCL, sCL, sSR, sCL, sCL, sCL, sCL, sLA, sLI, sSR}},
/* OUTPUT */
/* sNO, sES, sSS, sSR, sFW, sTW, sCL, sCW, sLA, sLI, sSA */
/*syn*/ {{sSS, sES, sSS, sSR, sSS, sSS, sSS, sSS, sSS, sLI, sSR}},
/*fin*/ {{sTW, sFW, sSS, sTW, sFW, sTW, sCL, sTW, sLA, sLI, sTW}},
/*ack*/ {{sES, sES, sSS, sES, sFW, sTW, sCL, sCW, sLA, sES, sES}},
/*rst*/ {{sCL, sCL, sSS, sCL, sCL, sTW, sCL, sCL, sCL, sCL, sCL}},
/* INPUT-ONLY */
/* sNO, sES, sSS, sSR, sFW, sTW, sCL, sCW, sLA, sLI, sSA */
/*syn*/ {{sSR, sES, sES, sSR, sSR, sSR, sSR, sSR, sSR, sSR, sSR}},
/*fin*/ {{sCL, sFW, sSS, sTW, sFW, sTW, sCL, sCW, sLA, sLI, sTW}},
/*ack*/ {{sCL, sES, sSS, sES, sFW, sTW, sCL, sCW, sCL, sLI, sES}},
/*rst*/ {{sCL, sCL, sCL, sSR, sCL, sCL, sCL, sCL, sLA, sLI, sCL}},
};
ip_vs_proto_tcp.c 源码状态机的流转图，翻了下 lvs 代码，其实是一样的。INPUT 流，OUTPUT 流用于 nat fnat 相关的，INPUT-ONLY 服务于 dr tunnel 等等
小结
这么做其实是对的，无论 client 还是 rs 主动关闭，总有一方会处于 time_wait 状态，等待 2MSL 时间后再清除。做为 LB 只能等待时间更长，才能兼容原有逻辑，做到包重传。默认 dpvs time_wait 超时时间 7s，够用了。
