部署ElasticSearch
步骤一：创建持久化存储PVC
在 K8s 中创建持久化存储 PVC。
步骤二：创建服务Service
在ACK集群控制台左侧菜单，进入网络>服务。
切换到空间，单击使用YAML创建资源。
输入下列内容，单击创建。
apiVersion: v1
kind: Service
metadata:
name: es-headless-svc
spec:
clusterIP: None
selector:
app: es-cluster
ports:
- name: transport
port: 9300
protocol: TCP
---
# cluster ip
apiVersion: v1
kind: Service
metadata:
name: es-external-svc
spec:
selector:
app: es-cluster
ports:
- name: http
port: 80
targetPort: 9200
type: ClusterIP
- 确认服务创建成功。
步骤三：创建配置ConfigMap
在ACK集群控制台左侧菜单，进入配置管理>配置项。
单击使用YAML创建资源，输入下面的内容。
apiVersion: v1
kind: ConfigMap
metadata:
name: es-config-map
data:
elasticsearch.yml: |
node.name: ${HOSTNAME}
cluster.name: es-cluster
http.port: 9200
transport.tcp.port: 9300
network.host: "0.0.0.0"
node.master: true
bootstrap.memory_lock: false
cluster.initial_master_nodes: ["es-cluster-0","es-cluster-1","es-cluster-2"]
discovery.seed_hosts: ["es-cluster-0.es-headless-svc.ops-share.svc.cluster.local", "es-cluster-1.es-headless-svc.ops-share.svc.cluster.local","es-cluster-2.es-headless-svc.ops-share.svc.cluster.local"]
discovery.zen.minimum_master_nodes: 2
discovery.zen.ping_timeout: 60s
discovery.zen.join_timeout: 60s
http.cors.enabled: true
http.cors.allow-origin: '*'
http.cors.allow-headers: Authorization,X-Requested-With,Content-Length,Content-Type
node.master: 当前节点是否可以作为master候选节点。
cluster.initial_master_nodes: 集群初始时的候选master节点。
discovery.seed_hosts: 集群内其他节点的地址。
discovery.zen.minimum_master_nodes: 最少的master节点数，计算公式=所有节点数 / 2 +1 = 2。
xpack.security.transport.ssl.keystore.path：证书所在路径。
步骤四：修改Nas存储卷子目录的权限
由于前面使用了Nas +子目录，默认创建的子目录是root用户，ACK容器无权限读取。
需要在nas页面，挂载nas到某个ecs上，然后通过下面的命令修改子目录es的属主和权限。
chown -R 1000:1000 eschmod -R 774 es
步骤五：创建负载StateFulSet
在ACK集群控制台左侧菜单，进入工作负载>有状态。
单击使用YAML创建资源，输入下面的内容。
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: es-cluster
labels:
app: es-cluster
spec:
serviceName: es-headless-svc
selector:
matchLabels:
app: es-cluster
replicas: 3
updateStrategy:
type: RollingUpdate
template:
metadata:
labels:
app: es-cluster
spec:
securityContext:
fsGroup: 1000
initContainers:
- name: init-sysctl
image: busybox
imagePullPolicy: IfNotPresent
securityContext:
privileged: true
# increase mmap limits
command: ["sysctl", "-w", "vm.max_map_count=262144"]
containers:
- name: elasticsearch
resources:
requests:
memory: 3Gi
limits:
memory: 3Gi
securityContext:
privileged: true
runAsUser: 1000
capabilities:
add:
- IPC_LOCK
- SYS_RESOURCE
image: docker.elastic.co/elasticsearch/elasticsearch:7.15.1
imagePullPolicy: IfNotPresent
env:
- name: ES_JAVA_OPTS
value: "-Xms2g -Xmx2g"
- name: POD_NAME
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.name
ports:
- containerPort: 9200
name: es-http
- containerPort: 9300
name: es-transport
volumeMounts:
- name: es-data
mountPath: /usr/share/elasticsearch/data
subPathExpr: data/$(POD_NAME)
- name: es-config
mountPath: /usr/share/elasticsearch/config/elasticsearch.yml
subPath: elasticsearch.yml
- name: volume-localtime
mountPath: /etc/localtime
- name: es-data
mountPath: /usr/share/elasticsearch/config/cert
subPath: cert
- name: es-data
mountPath: /usr/share/elasticsearch/logs
subPathExpr: logs/$(POD_NAME)
volumes:
- name: es-config
configMap:
name: es-config-map
items:
- key: elasticsearch.yml
path: elasticsearch.yml
- name: es-data
persistentVolumeClaim:
claimName: es-data-pvc
- name: volume-localtime
hostPath:
path: /etc/localtime
type: ''
- 确认创建成功，并查看状态。
步骤六：创建路由Ingress
进入网络>路由，页面。
选择创建，参考下图进行配置。这里选择的服务时对外暴露的服务，即es-external-svc。
添加完成后会得到ingress的公网ip地址，DNS解析到此IP上即可。
步骤七：开启ES的TLS及安全认证
进入ES的某个节点终端。
执行下面命令，创建CA证书，完成后生成一个文件：elastic-stack-ca.p12。
./bin/elasticsearch-certutil ca
为了便于集群扩缩容，这里认证就不加密码了。
- 生成秘钥文件。
./bin/elasticsearch-certutil cert --ca elastic-stack-ca.p12
将秘钥文件复制到{ES安装目录}/config/cert目录下。
修改ConfigMap里的值，增加下列几行。
xpack.security.enabled: true
xpack.security.authc.accept_default_password: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: cert/elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: cert/elastic-certificates.p12
重新部署ES，确认启动完成后，进入任一节点的终端。
执行下面的命令，为ES内置用户创建密码。
./bin/elasticsearch-setup-passwords interactive
结果验证
浏览器访问下面的地址，确认会弹出登录提示框。
http://es.xxxx.com/cluster/stats?pretty
并且可以看到节点数为3。
部署Kibana
步骤一：部署ConfigMap
apiVersion: v1
kind: ConfigMap
metadata:
namespace: *****
name: kibana-config
data:
kibana.yml: |-
server.host: 0.0.0.0
elasticsearch:
hosts: http://es-external-svc:9200
username: kibana_system
password: *****
i18n.locale: "zh-CN"
步骤二：部署Service
image.png
步骤三：部署Deployment
apiVersion: apps/v1
kind: Deployment
metadata:
namespace: *****
name: kibana
labels:
app: kibana
spec:
replicas: 1
selector:
matchLabels:
app: kibana
template:
metadata:
labels:
app: kibana
spec:
containers:
- name: kibana
image: docker.elastic.co/kibana/kibana:7.15.1
resources:
requests:
memory: 2Gi
limits:
memory: 2Gi
ports:
- containerPort: 5601
name: kibana-http
volumeMounts:
- name: config
mountPath: /usr/share/kibana/config/kibana.yml
readOnly: true
subPath: kibana.yml
volumes:
- name: config
configMap:
name: kibana-config
步骤四：部署Ingress
image.png
部署FileBeat
步骤一：部署ConfigMap
apiVersion: v1
kind: ConfigMap
metadata:
name: filebeat-config
namespace: *****
data:
filebeat.yml: |-
filebeat.autodiscover: # 使用自动发现
providers:
- type: kubernetes
templates:
- condition:
equals:
kubernetes.namespace: ops-share #仅收集此命名空间下的日志
config:
- type: container
paths:
- /var/log/containers/*-${data.kubernetes.container.id}.log
exclude_lines: ["^\\s+[\\-`('.|_]"]
scan_frequency: 10s # 扫描频率，默认10s
tail_files: false # 是否读取最新的日志，调试模式下建议改为true，生产环境使用false
fields_under_root: true # 将fields下面的字段调整到根目录下
multiline.pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}' # 多行匹配
multiline.negate: true
multiline.match: after
multiline.timeout: 30
- type: kubernetes
templates:
- condition:
equals:
kubernetes.namespace: *****
config:
- type: container
paths:
- /var/log/containers/*-${data.kubernetes.container.id}.log
exclude_lines: ["^\\s+[\\-`('.|_]"]
scan_frequency: 10s
tail_files: false
fields_under_root: true
multiline.pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}'
multiline.negate: true
multiline.match: after
multiline.timeout: 30
processors:
- add_kubernetes_metadata: # 添加k8s的元数据，例如pod name、node name等
default_indexers.enabled: true
default_matchers.enabled: true
host: ${NODE_NAME}
matchers:
- logs_path:
logs_path: "/var/log/containers/"
- drop_fields:
#删除的多余字段
fields: ["host", "tags", "ecs", "log", "prospector", "agent", "input", "beat", "container.id","container.runtime","kubernetes.labels","kubernetes.namespace_uid","kubernetes.node.labels","kubernetes.node.uid","kubernetes.pod.uid","kubernetes.node.hostname"]
ignore_missing: true # 忽略未匹配的
- rename: # 重命名字段
fields:
- from: "container.image.name"
to: "image"
- from: "kubernetes.container.name"
to: "container.name"
- from: "kubernetes.namespace"
to: "namespace"
- from: "kubernetes.node.name"
to: "node"
- from: "kubernetes.pod.ip"
to: "pod.ip"
- from: "kubernetes.pod.name"
to: "pod.name"
ignore_missing: false
fail_on_error: false # 失败时是否终止，建议false
output.elasticsearch:
hosts: ["http://es-external-svc:9200"]
username: "elastic"
password: "*****"
indices:
- index: "k8s-%{[namespace]}-%{+yyyy.MM.dd}"
步骤二：部署ServiceAccount
由于Filebeat在处理日志时，是采用自动发现以及添加K8S元数据的方式，所以需要创建ServiceAccount以授权访问K8S的资源。
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
name: filebeat
subjects:
- kind: ServiceAccount
name: filebeat
namespace: *****
roleRef:
kind: ClusterRole
name: filebeat
apiGroup: rbac.authorization.k8s.io
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
name: filebeat
labels:
app: filebeat
rules:
- apiGroups:
- rbac.authorization.k8s.io
resources:
- namespaces
- pods
- nodes
verbs:
- get
- watch
- list
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: filebeat
namespace: *****
labels:
app: filebeat
---
apiVersion: v1
kind: ConfigMap
metadata:
name: filebeat-inputs
namespace: *****
labels:
app: filebeat
data:
kubernetes.yml: |-
- type: docker
containers.ids:
- "*"
processors:
- add_kubernetes_metadata:
in_cluster: true
---
主要是为filebeat这个app授予namespace、pods、nodes的查询权限（get、watch、list）。
步骤三：创建持久化存储卷PVC
创建 PVC。
步骤四：部署DaemonSet
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: filebeat
namespace: *****
labels:
app: filebeat
spec:
selector:
matchLabels:
app: filebeat
template:
metadata:
labels:
app: filebeat
spec:
serviceAccountName: filebeat
terminationGracePeriodSeconds: 30
containers:
- name: filebeat
image: elastic/filebeat:7.15.1
args: [
"-c", "/etc/filebeat.yml",
"-e",
]
env:
- name: NODE_NAME
valueFrom:
fieldRef:
fieldPath: spec.nodeName
securityContext:
runAsUser: 0
# If using Red Hat OpenShift uncomment this:
#privileged: true
resources:
limits:
memory: 200Mi
requests:
memory: 200Mi
volumeMounts:
- name: config
mountPath: /etc/filebeat.yml
readOnly: true
subPath: filebeat.yml
- name: data
mountPath: /usr/share/filebeat/data
subPathExpr: data/$(NODE_NAME)
- name: varlog
mountPath: /var/log
readOnly: true
- name: containers
mountPath: /var/lib/docker/containers
readOnly: true
- name: volume-localtime
mountPath: /etc/localtime
volumes:
- name: config
configMap:
defaultMode: 0600
name: filebeat-config
- name: containers
hostPath:
path: /var/lib/docker/containers
# varlog 读取宿主机上的日志目录
- name: varlog
hostPath:
path: /var/log
# data folder 存储所有文件的读取状态注册表，因此不会在filebeat pod重启后再次发送所有的文件。
- name: data
persistentVolumeClaim:
claimName: filebeat-data-pvc
- name: volume-localtime
hostPath:
path: /etc/localtime
type: ''
结果验证
登录Kibana控制台，在创建索引模式时，确认可以看到以命名空间为名称的索引列表。
创建索引模式，然后在Kibana的发现页面，确认可以看到类似下面的日志信息。
可以通过k8s的namespace、node、pod ip等信息筛选，方便快速定位日志。
