CentOS7.4部署ELK日志分析系统

官网地址:https://www.elastic.co/cn/

官网权威指南:https://www.elastic.co/guide/cn/elasticsearch/guide/current/index.html

安装指南：https://www.elastic.co/guide/en/elasticsearch/reference/5.x/rpm.html

ELK是Elasticsearch、Logstash、Kibana的简称，这三者是核心套件，但并非全部。

Elasticsearch是实时全文搜索和分析引擎，提供搜集、分析、存储数据三大功能；是一套开放REST和JAVA API等结构提供高效搜索功能，可扩展的分布式系统。它构建于Apache Lucene搜索引擎库之上。

Logstash是一个用来搜集、分析、过滤日志的工具。它支持几乎任何类型的日志，包括系统日志、错误日志和自定义应用程序日志。它可以从许多来源接收日志，这些来源包括syslog、消息传递（例如RabbitMQ）和JMX，它能够以多种方式输出数据，包括电子邮件、websockets和Elasticsearch。

一、java环境，安装jdk 1.8及以上的版本

[root@elk-node1 ~]# yum install -y java-1.8.0

[root@elk-node1 ~]# java -version

openjdk version "1.8.0_121"

OpenJDK Runtime Environment (build 1.8.0_121-b13)

OpenJDK 64-Bit Server VM (build 25.121-b13, mixed mode)

   二、elasticsearch安装

1、下载Elasticsearch最新版本

解压至指定目录(安装目录)/usr/local/下，并将其重命名为elasticsearch (完整的目录应该为/usr/local/elasticsearch)

# tar xzf elasticsearch-6.4.2.tar.gz  -C /usr/local

# cd /usr/local/

# mv elasticsearch-6.4.2/ elasticsearch

2、创建一个用于运行elasticsearch的普通用户，随后修改elasticsearch家目录的所属权限为该用户；创建elasticsearch数据存储目录/data/elasticsearch

# groupadd elasticsearch

# useradd -g elasticsearch elasticsearch -m

useradd  elasticsearch(用户名) -g elasticsearch(组名) -p elasticsearch(密码)

# useradd elasticsearch -g elasticsearch -p elasticsearch

更改Elasticsearch文件夹以及内部文件的所属用户以及组为elasticsearch

# chown -R elasticsearch:elasticsearch elasticsearch

# mkdir /data/elasticsearch

# chown -R elasticsearch. /data/elasticsearch

3、配置环境变量

# vim /etc/profile

ulimit -u 4096

# source /etc/profile

4、修改elasticsearch.yml配置文件

# vim config/elasticsearch.yml

cluster.name: my-application     #ELK集群名称

path.data: /data/elasticsearch    #elasticsearch 数据存储目录

path.logs: /usr/local/elasticsearch/logs       #elasticsearch 日志存储路径

network.host: 10.66.1.23       #elasticsearch 监听地址，默认为localhost

http.port: 9200          #elasticsearch 监听端口，默认问9200

5、修改相关的内核参数

[root@localhostlocal]# vim /etc/security/limits.conf

添加以下内容

* soft nproc 65536

* hard nproc 65536

* soft nofile 65536

* hard nofile 65536

#修改vm.max_map_count=262144

# echo "vm.max_map_count=262144" >> /etc/sysctl.conf

6、以下由elasticsearch用户操作，以elasticsearch用户登录服务器，运行elasticsearch (注意：要切换到普通用户运行)

# su - elasticsearch

运行elasticsearch

$ ./elasticsearch/bin//elasticsearch或

$cd /usr/local/elasticsearch/

$ ./bin/elasticsearch

一般情况我们要求elasticsearch在后台运行，使用命令如下：

$ ./bin/elasticsearch -d

7、检查elasticsearch状态，如下则表示正常运行

# curl http://10.66.1.23:9200

{

  "name" : "dlOHzTB",

  "cluster_name" : "elasticsearch",

  "cluster_uuid" : "WuNxFom3QUWZLqC61-FSCw",

  "version" : {

    "number" : "6.4.2",

    "build_flavor" : "default",

    "build_type" : "tar",

    "build_hash" : "04711c2",

    "build_date" : "2018-09-26T13:34:09.098244Z",

    "build_snapshot" : false,

    "lucene_version" : "7.4.0",

    "minimum_wire_compatibility_version" : "5.6.0",

    "minimum_index_compatibility_version" : "5.0.0"

  },

  "tagline" : "You Know, for Search"

}

跳转到Elasticsearch的config配置文件下，使用vim打开elasticsearch.yml，找到里面的"network.host",将其改为你的IP,保存。

#cd elasticsearch/config/

#vim elasticsearch.yml

重启ElasticSearch，然后使用http://10.66.1.123:9200/访问，访问结果所示：，则说明ElasticSearch安装成功。如果是使用命令./bin/elasticsearch来启动的Elasticsearch，如果想要停止Elasticsearch的执行，则直接按住键盘Ctrl+C则会停止，停止之后你在浏览器中再次测试发现已不能操作。

http://10.66.1.23:9200/

name"dlOHzTB"

cluster_name"elasticsearch"

cluster_uuid"WuNxFom3QUWZLqC61-FSCw"

version

number"6.4.2"

build_flavor"default"

build_type"tar"

build_hash"04711c2"

build_date"2018-09-26T13:34:09.098244Z"

build_snapshotfalse

lucene_version"7.4.0"

minimum_wire_compatibility_version"5.6.0"

minimum_index_compatibility_version"5.0.0"

tagline"You Know, for Search"

三、Logstash安装

1、下载logstash软件包

# wget https://artifacts.elastic.co/downloads/logstash/logstash-6.4.2.tar.gz

2、解压至指定安装目录

# tar -zxf logstash-6.4.2.tar.gz -C /usr/local

# cd /usr/local/

# mv logstash-5.5.2 logstash

3、运行logstash

# cd logstash/

# ./bin/logstash -e 'input { stdin { } } output { stdout {} }'

输入”hello world!  ”，验证是否正常输出

[root@elk-server logstash]# ./bin/logstash -e 'input { stdin { } } output { stdout {} }'ERROR StatusLogger No log4j2 configurationfilefound. Using default configuration: logging only errors to the console.Sending Logstash's logs to /usr/local/logstash/logswhichis now configured via log4j2.properties[2017-08-28T15:11:33,267][INFO][logstash.setting.writabledirectory]Creating directory{:setting=>"path.queue", :path=>"/usr/local/logstash/data/queue"}[2017-08-28T15:11:33,273][INFO][logstash.setting.writabledirectory]Creating directory{:setting=>"path.dead_letter_queue", :path=>"/usr/local/logstash/data/dead_letter_queue"}[2017-08-28T15:11:33,300][INFO][logstash.agent]No persistent UUIDfilefound. Generating new UUID{:uuid=>"2fb479ab-0ca5-4979-89b1-4246df9a7472", :path=>"/usr/local/logstash/data/uuid"}[2017-08-28T15:11:33,438][INFO][logstash.pipeline]Starting pipeline{"id"=>"main","pipeline.workers"=>8,"pipeline.batch.size"=>125,"pipeline.batch.delay"=>5,"pipeline.max_inflight"=>1000}[2017-08-28T15:11:33,455][INFO][logstash.pipeline]Pipeline main startedThe stdin plugin is now waitingforinput:[2017-08-28T15:11:33,497][INFO][logstash.agent]Successfully started Logstash API endpoint{:port=>9600}hello world!2017-08-28T07:11:42.724Z elk-server.huangming.org hello world!

四、Kibana安装

1、下载kibana

# wget https://artifacts.elastic.co/downloads/kibana/kibana-5.5.2-linux-x86_64.tar.gz

2、解压至安装目录

# tar -zxf kibana-6.4.2-linux-x86_64.tar.gz -C /usr/local

# cd /usr/local/

# mv kibana-6.4.2-linux-x86_64/ kibana

3、修改配置

# cd kibana/

# vim config/kibana.yml

server.port: 5601   # 监听端口

server.host:"10.66.1.23"    # 指定后端服务器

elasticsearch.url:"http://10.66.1.23:9200"      # 指定elasticsearch实例地址

4、运行kibana

# ./bin/kibana &

# ./bin/kibana &[1]3219

[root@Anwar01 kibana]# ./bin/kibana

  log  [06:48:27.127] [info][status][plugin:kibana@6.4.2] Status changed from uninitialized to green - Ready

  log  [06:48:27.222] [info][status][plugin:elasticsearch@6.4.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch

  log  [06:48:27.227] [info][status][plugin:xpack_main@6.4.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch

  log  [06:48:27.234] [info][status][plugin:searchprofiler@6.4.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch

  log  [06:48:27.240] [info][status][plugin:ml@6.4.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch

  log  [06:48:27.329] [info][status][plugin:tilemap@6.4.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch

  log  [06:48:27.332] [info][status][plugin:watcher@6.4.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch

  log  [06:48:27.352] [info][status][plugin:license_management@6.4.2] Status changed from uninitialized to green - Ready

  log  [06:48:27.355] [info][status][plugin:index_management@6.4.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch

  log  [06:48:28.747] [info][status][plugin:timelion@6.4.2] Status changed from uninitialized to green - Ready

  log  [06:48:28.751] [info][status][plugin:graph@6.4.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch

  log  [06:48:28.843] [info][status][plugin:monitoring@6.4.2] Status changed from uninitialized to green - Ready

  log  [06:48:28.849] [warning][security] Generating a random key for xpack.security.encryptionKey. To prevent sessions from being invalidated on restart, please set xpack.security.encryptionKey in kibana.yml

  log  [06:48:28.855] [warning][security] Session cookies will be transmitted over insecure connections. This is not recommended.

  log  [06:48:28.871] [info][status][plugin:security@6.4.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch

  log  [06:48:28.913] [info][status][plugin:grokdebugger@6.4.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch

  log  [06:48:28.924] [info][status][plugin:dashboard_mode@6.4.2] Status changed from uninitialized to green - Ready

  log  [06:48:28.928] [info][status][plugin:logstash@6.4.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch

  log  [06:48:28.965] [info][status][plugin:apm@6.4.2] Status changed from uninitialized to green - Ready

  log  [06:48:29.028] [info][status][plugin:console@6.4.2] Status changed from uninitialized to green - Ready

  log  [06:48:29.031] [info][status][plugin:console_extensions@6.4.2] Status changed from uninitialized to green - Ready

  log  [06:48:29.037] [info][status][plugin:notifications@6.4.2] Status changed from uninitialized to green - Ready

  log  [06:48:29.042] [info][status][plugin:metrics@6.4.2] Status changed from uninitialized to green - Ready

  log  [06:48:45.846] [warning][reporting] Generating a random key for xpack.reporting.encryptionKey. To prevent pending reports from failing on restart, please set xpack.reporting.encryptionKey in kibana.yml

  log  [06:48:45.850] [info][status][plugin:reporting@6.4.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch

  log  [06:48:45.868] [error][status][plugin:xpack_main@6.4.2] Status changed from yellow to red - Request Timeout after 3000ms

  log  [06:48:45.869] [error][status][plugin:searchprofiler@6.4.2] Status changed from yellow to red - Request Timeout after 3000ms

  log  [06:48:45.870] [error][status][plugin:ml@6.4.2] Status changed from yellow to red - Request Timeout after 3000ms

  log  [06:48:45.871] [error][status][plugin:tilemap@6.4.2] Status changed from yellow to red - Request Timeout after 3000ms

  log  [06:48:45.871] [error][status][plugin:watcher@6.4.2] Status changed from yellow to red - Request Timeout after 3000ms

  log  [06:48:45.872] [error][status][plugin:index_management@6.4.2] Status changed from yellow to red - Request Timeout after 3000ms

  log  [06:48:45.873] [error][status][plugin:graph@6.4.2] Status changed from yellow to red - Request Timeout after 3000ms

  log  [06:48:45.874] [error][status][plugin:security@6.4.2] Status changed from yellow to red - Request Timeout after 3000ms

  log  [06:48:45.875] [error][status][plugin:grokdebugger@6.4.2] Status changed from yellow to red - Request Timeout after 3000ms

  log  [06:48:45.875] [error][status][plugin:logstash@6.4.2] Status changed from yellow to red - Request Timeout after 3000ms

  log  [06:48:45.876] [error][status][plugin:reporting@6.4.2] Status changed from yellow to red - Request Timeout after 3000ms

  log  [06:48:45.877] [error][status][plugin:elasticsearch@6.4.2] Status changed from yellow to red - Request Timeout after 3000ms

  log  [06:48:47.418] [info][license][xpack] Imported license information from Elasticsearch for the [data] cluster: mode: basic | status: active

  log  [06:48:47.423] [info][status][plugin:xpack_main@6.4.2] Status changed from red to green - Ready

  log  [06:48:47.424] [info][status][plugin:searchprofiler@6.4.2] Status changed from red to green - Ready

  log  [06:48:47.425] [info][status][plugin:ml@6.4.2] Status changed from red to green - Ready

  log  [06:48:47.425] [info][status][plugin:tilemap@6.4.2] Status changed from red to green - Ready

  log  [06:48:47.426] [info][status][plugin:watcher@6.4.2] Status changed from red to green - Ready

  log  [06:48:47.427] [info][status][plugin:index_management@6.4.2] Status changed from red to green - Ready

  log  [06:48:47.428] [info][status][plugin:graph@6.4.2] Status changed from red to green - Ready

  log  [06:48:47.429] [info][status][plugin:grokdebugger@6.4.2] Status changed from red to green - Ready

  log  [06:48:47.430] [info][status][plugin:logstash@6.4.2] Status changed from red to green - Ready

  log  [06:48:47.430] [info][status][plugin:reporting@6.4.2] Status changed from red to green - Ready

  log  [06:48:47.431] [info][kibana-monitoring][monitoring-ui] Starting monitoring stats collection

  log  [06:48:47.436] [info][status][plugin:security@6.4.2] Status changed from red to green - Ready

  log  [06:48:47.564] [info][license][xpack] Imported license information from Elasticsearch for the [monitoring] cluster: mode: basic | status: active

  log  [06:48:50.527] [info][status][plugin:elasticsearch@6.4.2] Status changed from red to green - Ready

  log  [06:49:07.536] [info][listening][server][http] Server running at http://10.66.1.23:5601

5、验证kibana

在客户端浏览器打开http://10.66.1.23:5601

在该页面提示我们需要创建一个index

首先创建一个kinana默认的index(名称为.kibana),如果输入的index名不存在，则无法创建

查看运行状态及已安装的插件

至此ELK已经搭建完成了，下面来创建一个收集message系统日志的实例

五、收集syslog日志

1、创建配置文件

# cd logstash/

# vim config/logstash.conf

input{file{path=>["/var/log/messages"]type=>"syslog"}}filter{grok{match=>["message","%{SYSLOGBASE} %{GREEDYDATA:content}"]}}output{elasticsearch{hosts=>["10.17.83.42:9200"]index=>"syslog-%{+YYY.MM.dd}"}stdout{codec=>rubydebug}}

其中match => [ "message", "%{SYSLOGBASE} %{GREEDYDATA:content}" ]这一行也可以具体写成如下：


match => [ "message", "%{SYSLOGTIMESTAMP:timestamp} (?:%{SYSLOGFACILITY} )?%{SYSLOGHOST:logsource} %{SYSLOGPROG}: %{GREEDYDATA:content}" ]

参考官方文档logstash配置：

https://www.elastic.co/guide/en/logstash/current/plugins-filters-grok.html

2、指定配置文件运行logstash

# ./bin/logstash -f ./config/logstash.conf &

[root@elk-server logstash]# ./bin/logstash -f ./config/logstash.conf &[1]4479[root@elk-server logstash]# ERROR StatusLogger No log4j2 configuration file found. Using default configuration: logging only errors to the console.Sending Logstash's logs to /usr/local/logstash/logswhichis now configured via log4j2.properties[2017-08-28T16:29:05,950][INFO][logstash.outputs.elasticsearch]Elasticsearch pool URLs updated{:changes=>{:removed=>[], :added=>[http://10.17.83.42:9200/]}}[2017-08-28T16:29:05,955][INFO][logstash.outputs.elasticsearch]Running health check to seeifan Elasticsearch connection is working{:healthcheck_url=>http://10.17.83.42:9200/, :path=>"/"}[2017-08-28T16:29:06,059][WARN][logstash.outputs.elasticsearch]Restored connection to ES instance{:url=>"http://10.17.83.42:9200/"}[2017-08-28T16:29:06,060][INFO][logstash.outputs.elasticsearch]Using mapping template from{:path=>nil}[2017-08-28T16:29:06,110][INFO][logstash.outputs.elasticsearch]Attempting toinstalltemplate{:manage_template=>{"template"=>"logstash-*","version"=>50001,"settings"=>{"index.refresh_interval"=>"5s"},"mappings"=>{"_default_"=>{"_all"=>{"enabled"=>true,"norms"=>false},"dynamic_templates"=>[{"message_field"=>{"path_match"=>"message","match_mapping_type"=>"string","mapping"=>{"type"=>"text","norms"=>false}}},{"string_fields"=>{"match"=>"*","match_mapping_type"=>"string","mapping"=>{"type"=>"text","norms"=>false,"fields"=>{"keyword"=>{"type"=>"keyword","ignore_above"=>256}}}}}],"properties"=>{"@timestamp"=>{"type"=>"date","include_in_all"=>false},"@version"=>{"type"=>"keyword","include_in_all"=>false},"geoip"=>{"dynamic"=>true,"properties"=>{"ip"=>{"type"=>"ip"},"location"=>{"type"=>"geo_point"},"latitude"=>{"type"=>"half_float"},"longitude"=>{"type"=>"half_float"}}}}}}}}[2017-08-28T16:29:06,117][INFO][logstash.outputs.elasticsearch]New Elasticsearch output{:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//10.17.83.42:9200"]}[2017-08-28T16:29:06,187][INFO][logstash.pipeline]Starting pipeline{"id"=>"main","pipeline.workers"=>8,"pipeline.batch.size"=>125,"pipeline.batch.delay"=>5,"pipeline.max_inflight"=>1000}[2017-08-28T16:29:06,343][INFO][logstash.pipeline]Pipeline main started[2017-08-28T16:29:06,380][INFO][logstash.agent]Successfully started Logstash API endpoint{:port=>9600}

3、登录kibana页面


点击Management --> Index Patterns --> Create index

在Index name or pattern处输入在logstash中指定的index，后面的日期直接用*号代替即可

创建完成之后进入Discover页面

4、验证是否正常收集syslog日志，执行以下命令手动生成日志

[root@elk-server kibana]# logger "helloooooooo22"[root@elk-server kibana]# yum install httpd

查看源日志文件，最近生成的的日志

[root@elk-server ~]# tail -f -n 10 /var/log/messagesAug 28 16:46:15 elk-server root: helloooooooo33Aug 28 16:47:17 elk-server yum[5048]: Installed: apr-1.4.8-3.el7.x86_64Aug 28 16:47:17 elk-server yum[5048]: Installed: apr-util-1.5.2-6.el7.x86_64Aug 28 16:47:18 elk-server yum[5048]: Installed: httpd-tools-2.4.6-45.el7.centos.4.x86_64Aug 28 16:47:18 elk-server yum[5048]: Installed: mailcap-2.1.41-2.el7.noarchAug 28 16:47:19 elk-server systemd: Reloading.Aug 28 16:47:19 elk-server systemd: Configurationfile/usr/lib/systemd/system/auditd.service is marked world-inaccessible. This has no effect as configuration data is accessible via APIs without restrictions. Proceeding anyway.Aug 28 16:47:19 elk-server systemd: Configurationfile/usr/lib/systemd/system/ebtables.service is marked executable. Please remove executable permission bits. Proceeding anyway.Aug 28 16:47:19 elk-server systemd: Configurationfile/usr/lib/systemd/system/wpa_supplicant.service is marked executable. Please remove executable permission bits. Proceeding anyway.Aug 28 16:47:19 elk-server yum[5048]: Installed: httpd-2.4.6-45.el7.centos.4.x86_64

5、回到Kibana管理页面，重新刷新页面，可以看到新的日志已经展示出来了

展开最近的一条日志


[root@elk-node1 logstash]# grep -v "#" logstash.yml

path.data: /var/lib/logstash

path.config: /etc/logstash/conf.d

path.logs: /var/log/logstash

   2.pipeline文件

根据默认配置，pipeline实例文件默认应放置于/etc/logstash/conf.d目录，此时目录下无实例文件，可根据实际情况新建实例，以处理本机messages信息为例，如下：

[root@elk-node1 logstash]# cd /etc/logstash/conf.d/

[root@elk-node1 conf.d]# vim messages.conf

input {

    file {

        path => "/var/log/messages"

    }

}

output {

    elasticsearch {

        hosts => ["192.168.217.131:9200","192.168.217.132:9200"]

        index => "messages-%{+YYYY.MM.dd}"

    }

    stdout {

# codec => rubydebug

    }

}

[root@elk-node1 conf.d]# cd ..

[root@elk-node1 logstash]# chown -R logstash:logstash conf.d/

[root@elk-node1 logstash]# chmod 644 /var/log/messages

   3.启动验证

1)启动测试

    1

[root@elk-node1 logstash]# cd /usr/share/logstash/

[root@elk-node1 logstash]# bin/logstash -e 'input { stdin { } } output { stdout {} }'

WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults

Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs to console

logstash可以启动，但此种验证方式会有告警，可以提示方式处理，在“$LS_HOME”下建立“config”目录，并将”/etc/logstash/”下的文件建软链接到“config”目录，再次执行即可，如下：

[root@elk-node1 logstash]# mkdir -p /usr/share/logstash/config/

[root@elk-node1 logstash]#  ln -s /etc/logstash/* /usr/share/logstash/config

[root@elk-node1 logstash]# chown -R logstash:logstash /usr/share/logstash/config/

[root@elk-node1 logstash]# bin/logstash -e 'input { stdin { } } output { stdout {} }'

   2）启动logstash并验证

 [root@elk-node1 ~]#

systemctl start logstash

[root@elk-node1 ~]# systemctl enable logstash.service

Created symlink from /etc/systemd/system/multi-user.target.wants/logstash.service to /etc/systemd/system/logstash.service.

[root@elk-node1 ~]# netstat -anplut

//查看9600端口

  4.验证

访问http://192.168.217.131:9100/

这里写图片描述

4.安装kibana

1.yum安装kibana

[root@elk-node1 ~]# yum install -y kinaba

[root@elk-node1 ~]# cd /etc/kibana

[root@elk-node1 kibana]# grep -v ^# /etc/kibana/kibana.yml

server.port: 5601

server.host: "0.0.0.0"

elasticsearch.url: "http://192.168.217.131:9200"

[root@elk-node1 kibana]# systemctl start kibana

[root@elk-node1 kibana]# netstat -anplut |grep :5601

tcp        0      0 0.0.0.0:5601            0.0.0.0:*              LISTEN      4335/node

  2.浏览器访问验证

访问http://192.168.217.131:5601/

这里写图片描述

在红线部分添加messages-*，然后点击Discover，如果不能显示图形，请把时间选择一小时或更长。

这里写图片描述

elk，就简单介绍到这里，有什么问题欢迎提出来。

安装elasticsearch的yum源的密钥（这个需要在所有服务器上都配置）

# rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch

配置elasticsearch的yum源

# vim /etc/yum.repos.d/elasticsearch.repo

在elasticsearch.repo文件中添加如下内容

[elasticsearch-5.x]name=Elasticsearch repository for 5.x packagesbaseurl=https://artifacts.elastic.co/packages/5.x/yumgpgcheck=1gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearchenabled=1autorefresh=1type=rpm-md

安装elasticsearch# yum install -y elasticsearch安装java环境(java环境必须是1.8版本以上的)wget http://download.oracle.com/otn-pub/java/jdk/8u131-b11/d54c1d3a095b4ff2b6607d096fa80163/jdk-8u131-linux-x64.rpmrpm -ivh jdk-8u131-linux-x64.rpm 验证java安装成功java -versionjava version"1.8.0_131"Java(TM) SE Runtime Environment (build1.8.0_131-b11)Java HotSpot(TM)64-Bit Server VM (build25.131-b11, mixed mode)

创建elasticsearch data的存放目录，并修改该目录的属主属组

# mkdir -p /data/es-data  (自定义用于存放data数据的目录)# chown -R elasticsearch:elasticsearch /data/es-data

修改elasticsearch的日志属主属组

#chown -R elasticsearch:elasticsearch /var/log/elasticsearch/

修改elasticsearch的配置文件

# vim /etc/elasticsearch/elasticsearch.yml找到配置文件中的cluster.name，打开该配置并设置集群名称cluster.name: demon找到配置文件中的node.name，打开该配置并设置节点名称node.name: elk-1修改data存放的路径path.data: /data/es-data修改logs日志的路径path.logs: /var/log/elasticsearch/配置内存使用用交换分区bootstrap.memory_lock: true监听的网络地址network.host: 0.0.0.0开启监听的端口http.port: 9200增加新的参数，这样head插件可以访问es (5.x版本，如果没有可以自己手动加)http.cors.enabled: truehttp.cors.allow-origin: "*"启动elasticsearch服务

启动服务

/etc/init.d/elasticsearch startStarting elasticsearch: Java HotSpot(TM)64-BitServer VMwarning: INFO: os::commit_memory(0x0000000085330000,2060255232,0) failed;error='Cannotallocate memory' (errno=12)## Thereisinsufficient memoryforthe Java Runtime Environmenttocontinue.# Native memory allocation (mmap) failedtomap2060255232bytesforcommitting reserved memory.# Anerrorreportfilewithmore informationissaved as:# /tmp/hs_err_pid2616.log                                                          [FAILED]这个报错是因为默认使用的内存大小为2G，虚拟机没有那么多的空间修改参数：vim /etc/elasticsearch/jvm.options-Xms512m-Xmx512m再次启动/etc/init.d/elasticsearch start查看服务状态，如果有报错可以去看错误日志 less /var/log/elasticsearch/demon.log(日志的名称是以集群名称命名的)创建开机自启动服务# chkconfig elasticsearchon

注意事项

需要修改几个参数，不然启动会报错vim /etc/security/limits.conf在末尾追加以下内容（elk为启动用户，当然也可以指定为*）elk soft nofile 65536elk hard nofile 65536elk soft nproc 2048elk hard nproc 2048elk soft memlock unlimitedelk hard memlock unlimited继续再修改一个参数vim /etc/security/limits.d/90-nproc.conf将里面的1024改为2048（ES最少要求为2048）*          soft    nproc    2048另外还需注意一个问题（在日志发现如下内容，这样也会导致启动失败，这一问题困扰了很久）[2017-06-14T19:19:01,641][INFO ][o.e.b.BootstrapChecks    ] [elk-1] bound or publishing to a non-loopback or non-link-local address, enforcing bootstrap checks[2017-06-14T19:19:01,658][ERROR][o.e.b.Bootstrap          ] [elk-1] node validation exception[1] bootstrap checks failed[1]:system call filters failed to install; check the logs and fix your configuration or disable system call filters at your own risk    解决：修改配置文件，在配置文件添加一项参数（目前还没明白此参数的作用）vim /etc/elasticsearch/elasticsearch.yml bootstrap.system_call_filter: false

通过浏览器请求下9200的端口，看下是否成功

先检查9200端口是否起来netstat -antp |grep9200tcp00:::9200:::*                        LISTEN2934/java浏览器访问测试是否正常（以下为正常）# curl http://127.0.0.1:9200/{"name":"linux-node1","cluster_name":"demon","cluster_uuid":"kM0GMFrsQ8K_cl5Fn7BF-g","version": {"number":"5.4.0","build_hash":"780f8c4","build_date":"2017-04-28T17:43:27.229Z","build_snapshot":false,"lucene_version":"6.5.0"},"tagline":"You Know, for Search"}

如何和elasticsearch交互

JavaAPIRESTfulAPIJavascript,.Net,PHP,Perl,Python利用API查看状态# curl -i -XGET 'localhost:9200/_count?pretty' HTTP/1.1200OKcontent-type: application/json; charset=UTF-8content-length:95{"count":0,"_shards": {"total":0,"successful":0,"failed":0}    }

安装插件

安装elasticsearch-head插件安装docker镜像或者通过github下载elasticsearch-head项目都是可以的，1或者2两种方式选择一种安装使用即可1.使用docker的集成好的elasticsearch-head# docker run -p 9100:9100 mobz/elasticsearch-head:5docker容器下载成功并启动以后，运行浏览器打开http://localhost:9100/2.使用git安装elasticsearch-head# yum install -y npm# git clone git://github.com/mobz/elasticsearch-head.git# cd elasticsearch-head# npm install# npm run start检查端口是否起来    netstat -antp |grep9100浏览器访问测试是否正常    http://IP:9100/

LogStash的使用

安装Logstash环境：官方安装手册:https://www.elastic.co/guide/en/logstash/current/installing-logstash.html下载yum源的密钥认证：# rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch利用yum安装logstash# yum install -y logstash查看下logstash的安装目录# rpm -ql logstash创建一个软连接，每次执行命令的时候不用在写安装路劲（默认安装在/usr/share下）ln -s /usr/share/logstash/bin/logstash /bin/执行logstash的命令#logstash -e'input { stdin { } } output { stdout {} }'运行成功以后输入:nihaostdout返回的结果:

注:  -e          执行操作input标准输入  {input}  插件output标准输出  {stdout}  插件  通过rubydebug来输出下更详细的信息# logstash -e'input { stdin { } } output { stdout {codec => rubydebug} }'执行成功输入:nihaostdout输出的结果:

如果标准输出还有elasticsearch中都需要保留应该怎么玩，看下面# /usr/share/logstash/bin/logstash -e 'input{ stdin { } }output{ elasticsearch { hosts => ["192.168.1.202:9200"] } stdout { codec => rubydebug }}'运行成功以后输入:I am elk返回的结果(标准输出中的结果):

logstash使用配置文件

官方指南:https://www.elastic.co/guide/en/logstash/current/configuration.html创建配置文件01-logstash.conf# vim /etc/logstash/conf.d/elk.conf文件中添加以下内容input { stdin { } }output {  elasticsearch { hosts => ["192.168.1.202:9200"] }  stdout { codec => rubydebug }}使用配置文件运行logstash# logstash -f ./elk.conf运行成功以后输入以及标准输出结果

logstash的数据库类型

1.Input插件    权威指南:https://www.elastic.co/guide/en/logstash/current/input-plugins.htmlfile插件的使用# vim /etc/logstash/conf.d/elk.conf添加如下配置    input {        file {            path =>"/var/log/messages"type =>"system"start_position =>"beginning"}    }    output {                elasticsearch {                    hosts => ["192.168.1.202:9200"]                    index =>"system-%{+YYYY.MM.dd}"}    }运行logstash指定elk.conf配置文件，进行过滤匹配#logstash -f /etc/logstash/conf.d/elk.conf

来一发配置安全日志的并且把日志的索引按类型做存放，继续编辑elk.conf文件

# vim /etc/logstash/conf.d/elk.conf添加secure日志的路径input {    file {        path =>"/var/log/messages"type =>"system"start_position =>"beginning"}    file {        path =>"/var/log/secure"type =>"secure"start_position =>"beginning"}}output {if[type] =="system"{        elasticsearch {            hosts => ["192.168.1.202:9200"]            index =>"nagios-system-%{+YYYY.MM.dd}"}    }if[type] =="secure"{        elasticsearch {            hosts => ["192.168.1.202:9200"]            index =>"nagios-secure-%{+YYYY.MM.dd}"}    }}运行logstash指定elk.conf配置文件，进行过滤匹配# logstash -f ./elk.conf

这些设置都没有问题之后，接下来安装下kibana，可以让在前台展示

Kibana的安装及使用

安装kibana环境官方安装手册:https://www.elastic.co/guide/en/kibana/current/install.html下载kibana的tar.gz的软件包# wget https://artifacts.elastic.co/downloads/kibana/kibana-5.4.0-linux-x86_64.tar.gz解压kibana的tar包# tar -xzf kibana-5.4.0-linux-x86_64.tar.gz进入解压好的kibana#mv kibana-5.4.0-linux-x86_64 /usr/local创建kibana的软连接#ln -s /usr/local/kibana-5.4.0-linux-x86_64/ /usr/local/kibana编辑kibana的配置文件#vim /usr/local/kibana/config/kibana.yml修改配置文件如下，开启以下的配置server.port: 5601server.host: "0.0.0.0"elasticsearch.url: "http://192.168.1.202:9200"kibana.index: ".kibana" 安装screen,以便于kibana在后台运行（当然也可以不用安装，用其他方式进行后台启动）# yum -y install screen# screen#/usr/local/kibana/bin/kibananetstat -antp |grep 5601tcp        0      0 0.0.0.0:5601                0.0.0.0:*                  LISTEN      17007/node 打开浏览器并设置对应的indexhttp://IP:5601

二、ELK实战篇

好，现在索引也可以创建了，现在可以来输出nginx、apache、message、secrue的日志到前台展示（Nginx有的话直接修改，没有自行安装）

编辑nginx配置文件，修改以下内容（在http模块下添加）log_format json'{"@timestamp":"$time_iso8601",''"@version":"1",''"client":"$remote_addr",''"url":"$uri",''"status":"$status",''"domian":"$host",''"host":"$server_addr",''"size":"$body_bytes_sent",''"responsetime":"$request_time",''"referer":"$http_referer",''"ua":"$http_user_agent"''}';修改access_log的输出格式为刚才定义的json access_log  logs/elk.access.log  json;继续修改apache的配置文件LogFormat"{ \

        \"@timestamp\": \"%{%Y-%m-%dT%H:%M:%S%z}t\", \

        \"@version\": \"1\", \

        \"tags\":[\"apache\"], \

        \"message\": \"%h %l %u %t \\\"%r\\\"%>s %b\", \

        \"clientip\": \"%a\", \

        \"duration\": %D, \

        \"status\": %>s, \

        \"request\": \"%U%q\", \

        \"urlpath\": \"%U\", \

        \"urlquery\": \"%q\", \

        \"bytes\": %B, \

        \"method\": \"%m\", \

        \"site\": \"%{Host}i\", \

        \"referer\": \"%{Referer}i\", \

        \"useragent\": \"%{User-agent}i\" \

      }"ls_apache_json一样修改输出格式为上面定义的json格式CustomLog logs/access_log ls_apache_json编辑logstash配置文件，进行日志收集vim /etc/logstash/conf.d/full.conf    input {    file {        path =>"/var/log/messages"type =>"system"start_position =>"beginning"}      file {        path =>"/var/log/secure"type =>"secure"start_position =>"beginning"}      file {        path =>"/var/log/httpd/access_log"type =>"http"start_position =>"beginning"}      file {        path =>"/usr/local/nginx/logs/elk.access.log"type =>"nginx"start_position =>"beginning"}  }    output {if[type] =="system"{        elasticsearch {            hosts => ["192.168.1.202:9200"]            index =>"nagios-system-%{+YYYY.MM.dd}"}          }if[type] =="secure"{        elasticsearch {            hosts => ["192.168.1.202:9200"]            index =>"nagios-secure-%{+YYYY.MM.dd}"}    }if[type] =="http"{        elasticsearch {            hosts => ["192.168.1.202:9200"]            index =>"nagios-http-%{+YYYY.MM.dd}"}    }if[type] =="nginx"{        elasticsearch {            hosts => ["192.168.1.202:9200"]            index =>"nagios-nginx-%{+YYYY.MM.dd}"}    }}运行看看效果如何logstash -f /etc/logstash/conf.d/full.conf

可以发现所有创建日志的索引都已存在，接下来就去Kibana创建日志索引，进行展示（按照上面的方法进行创建索引即可），看下展示的效果

接下来再来一发MySQL慢日志的展示

由于MySQL的慢日志查询格式比较特殊，所以需要用正则进行匹配，并使用multiline能够进行多行匹配（看具体配置）input {    file {        path =>"/var/log/messages"type =>"system"start_position =>"beginning"}      file {        path =>"/var/log/secure"type =>"secure"start_position =>"beginning"}      file {        path =>"/var/log/httpd/access_log"type =>"http"start_position =>"beginning"}      file {        path =>"/usr/local/nginx/logs/elk.access.log"type =>"nginx"start_position =>"beginning"}          file {        path =>"/var/log/mysql/mysql.slow.log"type =>"mysql"start_position =>"beginning"codec => multiline {            pattern =>"^# User@Host:"negate =>truewhat =>"previous"}    }}filter {    grok {        match => {"message"=>"SELECT SLEEP"}            add_tag => ["sleep_drop"]            tag_on_failure => []    }if"sleep_drop"in[tags] {            drop {}    }        grok {        match => {"message"=>"(?m)^# User@Host: %{USER:User}\[[^\]]+\] @ (?:(?<clienthost>\S*) )?\[(?:%{IP:Client_IP})?\]\s.*# Query_time: %{NUMBER:Query_Time:float}\s+Lock_time: %{NUMBER:Lock_Time:float}\s+Rows_sent: %{NUMBER:Rows_Sent:int}\s+Rows_examined: %{NUMBER:Rows_Examined:int}\s*(?:use %{DATA:Database};\s*)?SET timestamp=%{NUMBER:timestamp};\s*(?<Query>(?<Action>\w+)\s+.*)\n# Time:.*$"}        }      date {            match => ["timestamp","UNIX"]            remove_field => ["timestamp"]    }  }  output {if[type] =="system"{        elasticsearch {            hosts => ["192.168.1.202:9200"]            index =>"nagios-system-%{+YYYY.MM.dd}"}          }if[type] =="secure"{        elasticsearch {            hosts => ["192.168.1.202:9200"]            index =>"nagios-secure-%{+YYYY.MM.dd}"}    }if[type] =="http"{        elasticsearch {            hosts => ["192.168.1.202:9200"]            index =>"nagios-http-%{+YYYY.MM.dd}"}    }if[type] =="nginx"{        elasticsearch {            hosts => ["192.168.1.202:9200"]            index =>"nagios-nginx-%{+YYYY.MM.dd}"}    }if[type] =="mysql"{        elasticsearch {            hosts => ["192.168.1.202:9200"]            index =>"nagios-mysql-slow-%{+YYYY.MM.dd}"}    }}

查看效果(一条慢日志查询会显示一条，如果不进行正则匹配，那么一行就会显示一条)

具体的日志输出需求，进行具体的分析

三：ELK终极篇

安装reids# yum install -y redis修改redis的配置文件# vim /etc/redis.conf修改内容如下daemonize yesbind192.168.1.202启动redis服务# /etc/init.d/redis restart测试redis的是否启用成功# redis-cli -h 192.168.1.202输入info如果有不报错即可redis192.168.1.202:6379> inforedis_version:2.4.10....编辑配置redis-out.conf配置文件，把标准输入的数据存储到redis中# vim /etc/logstash/conf.d/redis-out.conf添加如下内容input {            stdin {}}output {        redis {                host =>"192.168.1.202"port =>"6379"password =>'test'db =>'1'data_type =>"list"key =>'elk-test'}}  运行logstash指定redis-out.conf的配置文件# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/redis-out.conf

运行成功以后，在logstash中输入内容（查看下效果）

编辑配置redis-in.conf配置文件，把reids的存储的数据输出到elasticsearch中# vim /etc/logstash/conf.d/redis-out.conf添加如下内容input{    redis {                    host =>"192.168.1.202"port =>"6379"password =>'test'db =>'1'data_type =>"list"key =>'elk-test'batch_count =>1#这个值是指从队列中读取数据时，一次性取出多少条，默认125条（如果redis中没有125条，就会报错，所以在测试期间加上这个值）}}output {        elasticsearch {                hosts => ['192.168.1.202:9200']                index =>'redis-test-%{+YYYY.MM.dd}'}}运行logstash指定redis-in.conf的配置文件# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/redis-out.conf

把之前的配置文件修改一下，变成所有的日志监控的来源文件都存放到redis中，然后通过redis在输出到elasticsearch中更改为如下，编辑full.confinput {    file {            path =>"/var/log/httpd/access_log"type =>"http"start_position =>"beginning"}    file {            path =>"/usr/local/nginx/logs/elk.access.log"type =>"nginx"start_position =>"beginning"}    file {            path =>"/var/log/secure"type =>"secure"start_position =>"beginning"}    file {            path =>"/var/log/messages"type =>"system"start_position =>"beginning"}}output {if[type] =="http"{        redis {            host =>"192.168.1.202"password =>'test'port =>"6379"db =>"6"data_type =>"list"key =>'nagios_http'}    }if[type] =="nginx"{        redis {            host =>"192.168.1.202"password =>'test'port =>"6379"db =>"6"data_type =>"list"key =>'nagios_nginx'}    }if[type] =="secure"{        redis {            host =>"192.168.1.202"password =>'test'port =>"6379"db =>"6"data_type =>"list"key =>'nagios_secure'}    }if[type] =="system"{        redis {            host =>"192.168.1.202"password =>'test'port =>"6379"db =>"6"data_type =>"list"key =>'nagios_system'}    }} 运行logstash指定shipper.conf的配置文件# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/full.conf在redis中查看是否已经将数据写到里面(有时候输入的日志文件不产生日志，会导致redis里面也没有写入日志)

把redis中的数据读取出来，写入到elasticsearch中(需要另外一台主机做实验)编辑配置文件# vim /etc/logstash/conf.d/redis-out.conf添加如下内容input {    redis {        type =>"system"host =>"192.168.1.202"password =>'test'port =>"6379"db =>"6"data_type =>"list"key =>'nagios_system'batch_count =>1}        redis {        type =>"http"host =>"192.168.1.202"password =>'test'port =>"6379"db =>"6"data_type =>"list"key =>'nagios_http'batch_count =>1}    redis {        type =>"nginx"host =>"192.168.1.202"password =>'test'port =>"6379"db =>"6"data_type =>"list"key =>'nagios_nginx'batch_count =>1}        redis {        type =>"secure"host =>"192.168.1.202"password =>'test'port =>"6379"db =>"6"data_type =>"list"key =>'nagios_secure'batch_count =>1}}    output {if[type] =="system"{        elasticsearch {            hosts => ["192.168.1.202:9200"]            index =>"nagios-system-%{+YYYY.MM.dd}"}    }if[type] =="http"{        elasticsearch {            hosts => ["192.168.1.202:9200"]            index =>"nagios-http-%{+YYYY.MM.dd}"}      }if[type] =="nginx"{        elasticsearch {            hosts => ["192.168.1.202:9200"]            index =>"nagios-nginx-%{+YYYY.MM.dd}"}      }if[type] =="secure"{        elasticsearch {            hosts => ["192.168.1.202:9200"]            index =>"nagios-secure-%{+YYYY.MM.dd}"}      }  }注意:input是从客户端收集的output是同样也保存到192.168.1.202中的elasticsearch中，如果要保存到当前的主机上，可以把output中的hosts修改成localhost，如果还需要在kibana中显示，需要在本机上部署kabana，为何要这样做，起到一个松耦合的目的说白了，就是在客户端收集日志，写到服务端的redis里或是本地的redis里面，输出的时候对接ES服务器即可运行命令看看效果# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/redis-out.conf

效果是和直接往ES服务器输出一样的（这样是先将日志存到redis数据库，然后再从redis数据库里取出日志）

上线ELK

１.　日志分类        系统日志        rsyslog    logstash    syslog插件        访问日志        nginx      logstash    codec  json        错误日志filelogstash    mulitline        运行日志filelogstash    codec  json        设备日志        syslog      logstash    syslog插件        Debug日志filelogstash  json 或者 mulitline2.  日志标准化        路径      固定        格式      尽量json3.  系统个日志开始-->错误日志-->运行日志-->访问日志

因为ES保存日志是永久保存，所以需要定期删除一下日志，下面命令为删除指定时间前的日志

curl-X DELETE http://xx.xx.com:9200/logstash-*-`date +%Y-%m-%d -d"-$ndays"`

最后编辑于 ：
©著作权归作者所有,转载或内容合作请联系作者
  • 人面猴
    序言：七十年代末，一起剥皮案震惊了整个滨河市，随后出现的几起案子，更是在滨河造成了极大的恐慌，老刑警刘岩，带你破解...
    沈念sama阅读 195,719评论 5赞 462
  • 死咒
    序言：滨河连续发生了三起死亡事件，死亡现场离奇诡异，居然都是意外死亡，警方通过查阅死者的电脑和手机，发现死者居然都...
    沈念sama阅读 82,337评论 2赞 373
  • 救了他两次的神仙让他今天三更去死
    文/潘晓璐 我一进店门，熙熙楼的掌柜王于贵愁眉苦脸地迎上来，“玉大人，你说我怎么就摊上这事。” “怎么了？”我有些...
    开封第一讲书人阅读 142,887评论 0赞 324
  • 道士缉凶录：失踪的卖姜人
    文/不坏的土叔 我叫张陵，是天一观的道长。 经常有香客问我，道长，这世上最难降的妖魔是什么？ 我笑而不...
    开封第一讲书人阅读 52,488评论 1赞 266
  • 港岛之恋（遗憾婚礼）
    正文 为了忘掉前任，我火速办了婚礼，结果婚礼上，老公的妹妹穿的比我还像新娘。我一直安慰自己，他们只是感情好，可当我...
    茶点故事阅读 61,313评论 4赞 357
  • 恶毒庶女顶嫁案：这布局不是一般人想出来的
    文/花漫 我一把揭开白布。 她就那样静静地躺着，像睡着了一般。 火红的嫁衣衬着肌肤如雪。 梳的纹丝不乱的头发上，一...
    开封第一讲书人阅读 46,284评论 1赞 273
  • 城市分裂传说
    那天，我揣着相机与录音，去河边找鬼。 笑死，一个胖子当着我的面吹牛，可吹牛的内容都是我干的。 我是一名探鬼主播，决...
    沈念sama阅读 36,672评论 3赞 386
  • 双鸳鸯连环套：你想象不到人心有多黑
    文/苍兰香墨 我猛地睁开眼，长吁一口气：“原来是场噩梦啊……” “哼！你这毒妇竟也来了？” 一声冷哼从身侧响起，我...
    开封第一讲书人阅读 35,346评论 0赞 254
  • 万荣杀人案实录
    序言：老挝万荣一对情侣失踪，失踪者是张志新（化名）和其女友刘颖，没想到半个月后，有当地人在树林里发现了一具尸体，经...
    沈念sama阅读 39,644评论 1赞 293
  • 护林员之死
    正文 独居荒郊野岭守林人离奇死亡，尸身上长有42处带血的脓包…… 初始之章·张勋 以下内容为张勋视角 年9月15日...
    茶点故事阅读 34,700评论 2赞 312
  • 白月光启示录
    正文 我和宋清朗相恋三年，在试婚纱的时候发现自己被绿了。 大学时的朋友给我发了我未婚夫和他白月光在一起吃饭的照片。...
    茶点故事阅读 36,457评论 1赞 326
  • 活死人
    序言：一个原本活蹦乱跳的男人离奇死亡，死状恐怖，灵堂内的尸体忽然破棺而出，到底是诈尸还是另有隐情，我是刑警宁泽，带...
    沈念sama阅读 32,316评论 3赞 313
  • 日本核电站爆炸内幕
    正文 年R本政府宣布，位于F岛的核电站，受9级特大地震影响，放射性物质发生泄漏。R本人自食恶果不足惜，却给世界环境...
    茶点故事阅读 37,706评论 3赞 299
  • 男人毒药：我在死后第九天来索命
    文/蒙蒙 一、第九天 我趴在偏房一处隐蔽的房顶上张望。 院中可真热闹，春花似锦、人声如沸。这庄子的主人今日做“春日...
    开封第一讲书人阅读 28,990评论 0赞 19
  • 一桩弑父案，背后竟有这般阴谋
    文/苍兰香墨 我抬头看了看天上的太阳。三九已至，却和暖如春，着一层夹袄步出监牢的瞬间，已是汗流浃背。 一阵脚步声响...
    开封第一讲书人阅读 30,261评论 1赞 251
  • 情欲美人皮
    我被黑心中介骗来泰国打工， 没想到刚下飞机就差点儿被人妖公主榨干…… 1. 我叫王不留，地道东北人。 一个月前我还...
    沈念sama阅读 41,648评论 2赞 342
  • 代替公主和亲
    正文 我出身青楼，却偏偏与公主长得像，于是被迫代替她去往敌国和亲。 传闻我的和亲对象是个残疾皇子，可洞房花烛夜当晚...
    茶点故事阅读 40,859评论 2赞 335

推荐阅读更多精彩内容