本文介绍的组件搭建，均为docker环境，如需查看其他环境搭建，请移步官网进行查看
1.elasticsearch 组件列表
2. docker elasticsearch start
elasticsearch官方文档
elasticsearch简介
Elasticsearch is a highly scalable open-source full-text search and analytics engine. It allows you to store, search, and analyze big volumes of data quickly and in near real time. It is generally used as the underlying engine/technology that powers applications that have complex search features and requirements.
拉取elasticsearch镜像
docker pull docker.elastic.co/elasticsearch/elasticsearch:6.5.4
启动elasticsearch
docker run -it -d --name elasticsearch -p 9200:9200 -p 9300:9300 -e "discovery.type=single-node" docker.elastic.co/elasticsearch/elasticsearch:6.5.4
3. docker start logstash
logstash官方文档
logstash简介
Logstash is an open source data collection engine with real-time pipelining capabilities. Logstash can dynamically unify data from disparate sources and normalize the data into destinations of your choice. Cleanse and democratize all your data for diverse advanced downstream analytics and visualization use cases.
While Logstash originally drove innovation in log collection, its capabilities extend well beyond that use case. Any type of event can be enriched and transformed with a broad array of input, filter, and output plugins, with many native codecs further simplifying the ingestion process. Logstash accelerates your insights by harnessing a greater volume and variety of data
拉取logstash镜像
docker pull docker.elastic.co/logstash/logstash:6.5.4
启动logstash
docker run -it -d --name logstash -v /usr/local/logstash/config/logstash.yml:/usr/share/logstash/config/logstash.yml -v /usr/local/logstash/pipeline/:/usr/share/logstash/pipeline/ -v /usr/local/logstash/lib/:/usr/local/logstash/lib/ docker.elastic.co/logstash/logstash:6.5.4
参数说明
-v /usr/local/logstash/config/logstash.yml:/usr/share/logstash/config/logstash.yml将本地logstash的配置文件挂载到容器内部
-v /usr/local/logstash/pipeline/:/usr/share/logstash/pipeline/将logstash管道配置文件挂载到容器内部
-v /usr/local/logstash/lib/:/usr/local/logstash/lib/在使用logstash全量增量导入数据的时候，需要使用jdbc进行导入，所以将jar的目录挂载到容器内部
demo数据导入配置文件
input {
jdbc {
jdbc_connection_string => "jdbc:mysql://127.0.0.1:3306/test"
jdbc_user => "user"
jdbc_password => "password"
# The path to our downloaded jdbc driver
jdbc_driver_library => "/usr/local/logstash/lib/mysql/mysql-connector-java-5.1.47.jar"
jdbc_driver_class => "com.mysql.jdbc.Driver"
# our query
schedule => "* * * * *"
statement => "select* from test where id> :sql_last_value"
use_column_value => true
# not lowercase column name
lowercase_column_names => false
tracking_column => id
}
}
output {
stdout { codec => json_lines }
elasticsearch {
hosts => ["127.0.0.1:9200"]
index => "test"
document_type => "data"
document_id => "%{id}"
}
}
格式化后的sql，需开启logstash自动转义，默认为关闭
statement => "select
id,
name,
user_id as userId
from test
where id> :sql_last_value"
设置方法：修改logstash.yml配置文件，开启自动转义
config.support_escapes: true
4. docker start kibane
kibane官方文档
kibana简介
Kibana is an open source analytics and visualization platform designed to work with Elasticsearch. You use Kibana to search, view, and interact with data stored in Elasticsearch indices. You can easily perform advanced data analysis and visualize your data in a variety of charts, tables, and maps.
Kibana makes it easy to understand large volumes of data. Its simple, browser-based interface enables you to quickly create and share dynamic dashboards that display changes to Elasticsearch queries in real time.
Setting up Kibana is a snap. You can install Kibana and start exploring your Elasticsearch indices in minutes — no code, no additional infrastructure required.
拉取kibana
docker pull docker.elastic.co/kibana/kibana:6.5.4
启动kibana
docker run --name kibana -e ELASTICSEARCH_URL=http://127.0.0.1:9200 -p 5601:5601 -d docker.elastic.co/kibana/kibana:6.5.4
至此elk的基本组件就搭好了 ，尽情玩耍把!
