envoy 启动流程
数据面组件启动流程
- initContainer执行初始化脚本，为Pod添加iptables规则
- Pilot-agent根据启动参数和K8S API Server中的配置信息生成Envoy的初始配置文件envoy-rev0.json，该文件告诉Envoy从xDS server中获取动态配置信息，并配置了xDS server的地址信息，即控制面的Pilot。
- Pilot-agent使用envoy-rev0.json启动Envoy进程。
- Envoy根据初始配置获得Pilot地址，采用xDS接口从Pilot获取到Listener，Cluster，Route等d动态配置信息。
- Envoy根据获取到的动态配置启动Listener，并根据Listener的配置，结合Route和Cluster对拦截到的流量进行处理。
istio劫持流量总流程
进流量：
downstream -> iptables -> envoy Inbound -> app
出流量：
app -> iptables -> envoy Outbound -> upstream
image.png
1. IPTABLES 劫持转发流量的过程
- 启动流程中有讲到第一步是envoy执行初始化脚本，添加iptables规则
参考文章
istio sidecar中创建iptables流程
2. envoy Inbound 和 Outbound 流程
istio 的监听端口
$ nsenter -t 20533 -n netstat -nl
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 0.0.0.0:15090 0.0.0.0:* LISTEN
tcp 0 0 127.0.0.1:15000 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:15001 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:15006 0.0.0.0:* LISTEN
tcp6 0 0 :::15020 :::* LISTEN
tcp6 0 0 :::8889 :::* LISTEN
由上图可以看到envoy有多个端口在监听
15000 端口: 管理端口
15001 端口：Outbound流量重定向端口
15006 端口：Inbound流量重定向端口
15020 端口：健康检查端口
15090 端口: 普罗米修斯端口
应用端口：8899
envoy Inbound处理流量过程
Inbound 流程是将 iptables 拦截到的 downstream 的流量转交给 localhost，与 Pod 内的应用程序容器建立连接。
下面以进入calculator的流量为例子，来分析一下Inbound流程
calculator Inbound流程图
calculator
Inbound配置
- listener
- virtual listener
{
"version_info": "2020-02-11T01:52:11Z/5",
"listener": {
"name": "virtualInbound",
"address": {
"socket_address": {
"address": "0.0.0.0",
"port_value": 15006
}
},
"filter_chains": [{
"filter_chain_match": {
"prefix_ranges": [{
"address_prefix": "::0",
"prefix_len": 0
}]
},
"filters": [{
"name": "envoy.tcp_proxy",
"typed_config": {
"@type": "type.googleapis.com/envoy.config.filter.network.tcp_proxy.v2.TcpProxy",
"stat_prefix": "InboundPassthroughClusterIpv6",
"cluster": "InboundPassthroughClusterIpv6"
}
}],
"filter_chain_match": {
"prefix_ranges": [{
"address_prefix": "192.xxx.xx.163",
"prefix_len": 32
}],
"destination_port": 8888
},
"filters": [{
"name": "envoy.http_connection_manager",
"typed_config": {
"@type": "type.googleapis.com/envoy.config.filter.network.http_connection_manager.v2.HttpConnectionManager",
"stat_prefix": "inbound_192.xxx.xx.163_8888",
"http_filters": [{
"name": "istio_authn"
},
{
"name": "mixer"
},
{
"name": "envoy.router"
}
],
"route_config": {
"name": "inbound|8888|grpc|calculator.vm.svc.cluster.local",
"virtual_hosts": [{
"name": "inbound|http|8888",
"domains": [
"*"
],
"routes": [{
"match": {
"prefix": "/"
},
"decorator": {
"operation": "calculator.vm.svc.cluster.local:8888/*"
},
"name": "default",
"route": {
"timeout": "0s",
"max_grpc_timeout": "0s",
"cluster": "inbound|8888|grpc|calculator.vm.svc.cluster.local"
}
}]
}],
"validate_clusters": false
}
}
}]
}]
}
}
- 该Listener中第三个filterchain用于处理Calculator服务的入向请求。
- 该filterchain的匹配条件为Calculator服务的 IP和8888端口，配置了一个http_connection_manager filter
- http_connection_manager 中又嵌入了istio_auth，Mixer，envoy.router等http filter，经过这些filter进行处理后，请求最终将被转发给""cluster": "inbound|8888|grpc|calculator.vm.svc.cluster.local""
接下来我们看下inbound cluster的配置
- cluster
- inbound cluster
{
"version_info": "2020-02-10T08:16:25Z/3",
"cluster": {
"name": "inbound|8888|grpc|calculator.vm.svc.cluster.local",
"type": "STATIC",
"connect_timeout": "10s",
"circuit_breakers": {
"thresholds": [
{
"max_connections": 4294967295,
"max_pending_requests": 4294967295,
"max_requests": 4294967295,
"max_retries": 4294967295
}
]
},
"http2_protocol_options": {
"max_concurrent_streams": 1073741824
},
"load_assignment": {
"cluster_name": "inbound|8888|grpc|calculator.vm.svc.cluster.local",
"endpoints": [
{
"lb_endpoints": [
{
"endpoint": {
"address": {
"socket_address": {
"address": "127.0.0.1",
"port_value": 8888
}
}
}
}
]
}
]
}
}
}
- 这个Inbound Cluster，由于该Inbound Cluster中配置的Upstream为127.0.0.1:8888(?如何查看其upstream)
- 由于iptable设置中127.0.0.1不会被拦截,该请求将发送到Calculator服务的8888端口上进行业务处理。
Inbound流程代码
//todo
envoy Outbound处理流量过程
webapp outbound流程图
webapp
outbound配置
- virtual outbound istener
{
"listener": {
"name": "virtualOutbound",
"address": {
"socket_address": {
"address": "0.0.0.0",
"port_value": 15001
}
},
"filter_chains": [{
"filter_chain_match": {
"prefix_ranges": [{
"address_prefix": "10.244.2.73",
"prefix_len": 32
}]
},
"filters": [{
"name": "envoy.tcp_proxy",
"typed_config": {
"@type": "type.googleapis.com/envoy.config.filter.network.tcp_proxy.v2.TcpProxy",
"stat_prefix": "BlackHoleCluster",
"cluster": "BlackHoleCluster"
}
}]
},
{
"filters": [{
"name": "mixer"
},
{
"name": "envoy.tcp_proxy",
"typed_config": {
"@type": "type.googleapis.com/envoy.config.filter.network.tcp_proxy.v2.TcpProxy",
"stat_prefix": "PassthroughCluster",
"cluster": "PassthroughCluster"
}
}
]
}
],
"use_original_dst": true
}
}
改listener的最后一行有个配置 "use_original_dst": true，表示用原有地址，不进行路由，这样就会转发到0.0.0.0_8888这个监听器上
下面看一下0.0.0.0_8888这个监听器的配置
- outbound listener
{
"listener": {
"name": "0.0.0.0_8888",
"address": {
"socket_address": {
"address": "0.0.0.0",
"port_value": 8888
}
},
"filter_chains": [{
"filter_chain_match": {
"prefix_ranges": [{
"address_prefix": "10.244.2.73",
"prefix_len": 32
}]
},
"filters": [{
"name": "envoy.tcp_proxy",
"typed_config": {
"@type": "type.googleapis.com/envoy.config.filter.network.tcp_proxy.v2.TcpProxy",
"stat_prefix": "BlackHoleCluster",
"cluster": "BlackHoleCluster"
}
}]
},
{
"filters": [{
"name": "envoy.http_connection_manager",
"typed_config": {
"@type": "type.googleapis.com/envoy.config.filter.network.http_connection_manager.v2.HttpConnectionManager",
"stat_prefix": "outbound_0.0.0.0_8888",
"http_filters": [{
"name": "mixer"
},
{
"name": "envoy.cors"
},
{
"name": "envoy.fault"
},
{
"name": "envoy.router"
}
],
"use_remote_address": false,
"generate_request_id": true,
"upgrade_configs": [{
"upgrade_type": "websocket"
}],
"stream_idle_timeout": "0s",
"normalize_path": true,
"rds": {
"config_source": {
"ads": {}
},
"route_config_name": "8888"
}
}
}]
}
],
"deprecated_v1": {
"bind_to_port": false
},
"listener_filters_timeout": "0.100s",
"traffic_direction": "OUTBOUND",
"continue_on_listener_filters_timeout": true
}
}
同理最后有一个envoy.router filter，这个filter中route_config_name:"8888"，所以我们可以找到名为"8888"的Route配置
- outbound Route
{
"route_config": {
"name": "8888",
"virtual_hosts": [{
"name": "calculator.vm.svc.cluster.local:8888",
"domains": [
"calculator.vm.svc.cluster.local",
"calculator.vm.svc.cluster.local:8888",
"calculator.vm",
"calculator.vm:8888",
"calculator.vm.svc.cluster",
"calculator.vm.svc.cluster:8888",
"calculator.vm.svc",
"calculator.vm.svc:8888",
"10.106.45.142",
"10.106.45.142:8888"
],
"routes": [{
"match": {
"prefix": "/"
},
"route": {
"weighted_clusters": {
"clusters": [{
"name": "outbound|8888|v1|calculator.vm.svc.cluster.local",
"weight": 50
},
{
"name": "outbound|8888|v2|calculator.vm.svc.cluster.local",
"weight": 50
}
]
}
},
"decorator": {
"operation": "calculator:8888/*"
}
}]
},
{
"name": "allow_any",
"domains": [
"*"
],
"routes": [{
"match": {
"prefix": "/"
},
"route": {
"cluster": "PassthroughCluster"
}
}]
}
],
"validate_clusters": false
}
}
根据Route 8888的配置可以找到outbound cluster
- outbound cluster
{
"version_info": "2020-02-07T10:38:17Z/1847",
"cluster": {
"name": "outbound|8888|v2|calculator.vm.svc.cluster.local",
"type": "EDS",
"eds_cluster_config": {
"eds_config": {
"ads": {}
},
"service_name": "outbound|8888|v2|calculator.vm.svc.cluster.local"
},
"connect_timeout": "10s",
"circuit_breakers": {
"thresholds": [
{
"max_connections": 4294967295,
"max_pending_requests": 4294967295,
"max_requests": 4294967295,
"max_retries": 4294967295
}
]
},
"http2_protocol_options": {
"max_concurrent_streams": 1073741824
},
"metadata": {
"filter_metadata": {
"istio": {
"config": "/apis/networking/v1alpha3/namespaces/vm/destination-rule/calculator"
}
}
}
},
"last_updated": "2020-02-07T10:38:17.893Z"
}
我们会发现这个cluster没有直接的endpoint,因为这是动态资源，
可以通过Pilot的调试接口获取该Cluster的endpoint：
$ curl http://xxx.xxx.xxx.xxx.108:15014/debug/edsz > pilot_eds_dump
最后发现endpoint为192.xxx.xxx.163:8888
然后就把流量转发到calculator服务了
- BlackHoleCluster
这是一个特殊的Cluster，并没有配置后端处理请求的Host。如其名字所暗示的一样，请求进入后将被直接丢弃掉。如果一个请求没有找到其对的目的服务，则被发到cluste。
{
"cluster": {
"name": "BlackHoleCluster",
"type": "STATIC",
"connect_timeout": "1s"
}
}
*PassthroughCluster
和BlackHoleCluter相反，发向PassthroughCluster的请求会被直接发送到其请求中要求的原始目地的，Envoy不会对请求进行重新路由。
{
"cluster": {
"name": "PassthroughCluster",
"type": "ORIGINAL_DST",
"connect_timeout": "1s",
"lb_policy": "CLUSTER_PROVIDED",
"circuit_breakers": {
"thresholds": [
{
"max_connections": 4294967295,
"max_pending_requests": 4294967295,
"max_requests": 4294967295,
"max_retries": 4294967295
}
]
}
}
}
outbound流程代码
//todo
