ELK日志管理平台的搭建
作者：互联网
ELK 日志管理平台
1>Elasticsearch是个开源分布式搜索引擎，提供搜集、分析、存储数据三大功能。它是基于Lucene（一个全文检索引擎的架构）开发的分布式存储检索引擎，可用来存储各类日志，采用 Java 开发，可通过 RESTful Web 接口，通过浏览器来与 Elasticsearch 通信。它的特点有：分布式，零配置，自动发现，索引自动分片，索引副本机制，restful风格接口，多数据源，自动搜索负载等;负责存储最终数据、建立索引、提供搜索功能。
2>Logstash 主要是用来日志的搜集、分析、过滤日志的工具，负责采集日志，支持大量的数据获取方式。一般工作方式为c/s架构，client端安装在需要收集日志的主机上，server端负责将收到的各节点日志进行过滤、修改等操作在一并发往elasticsearch上去。
另外，它使用JRuby 语言编写，运行在Java虚拟机（JVM）上，是一款强大的数据处理工具，可以实现=数据传输、格式处理、格式化输出。 Logstash 还具有强大的插件功能，常用于日志处理。
3>Kibana 也是一个开源和免费的工具，它基于Node.js开发，负责提供可视化界面，Kibana可以为 Logstash 和 ElasticSearch 提供的日志分析友好的 Web 界面，可以帮助汇总、分析和搜索重要数据日志。
4>Filebeat隶属于Beats，其作为原logstash-forwarder的替代来完成。Filebeat轻量级的日志传输工具， ，也可称为：是一个轻量级的日志采集器。它可以读取系统、nignx、apache等logs文件，监控日志文件，传输数据到Elasticsearch或者Logstash，最后在Kibana中实现可视化。目前Beats包含四种工具：
Packetbeat（搜集网络流量数据）
Topbeat（搜集系统、进程和文件系统级别的 CPU 和内存使用情况等数据）
Filebeat（搜集文件数据）：FileBeat采集数据时是Json化的，这个日志采集工具相当轻量级，对系统资源的消耗很少。而LogStash的优点则是有丰富的Filter插件，用于对数据作粗处理，占用资源高。一般，我们使用FileBeat采集数据之后会传输给Kafka消息队列，然后LogStash采集消息队列中的数据，作过滤处理，最后将数据传输给ES。
Winlogbeat（搜集 Windows 事件日志数据）
Elastic Stack的优点
Elastic Stack 目前已成为机器数据分析，或者说实时日志处理领域，开源界的第一选择，和传统的日志处理方案相比，Elastic Stack 具有如下几个优点：
1）处理方式灵活。Elasticsearch 是实时全文索引，不需要像 storm 那样预先编程才能使用；
2）配置简易上手。Elasticsearch 全部采用 JSON 接口，Logstash 是 Ruby DSL 设计，都是目前业界最通用的配置语法设计；
3）检索性能高效。虽然每次查询都是实时计算，但是优秀的设计和实现基本可以达到全天数据查询的秒级响应；
4）集群线性扩展。不管是 Elasticsearch 集群还是 Logstash 集群都是可以线性扩展的；
5）前端操作炫丽。Kibana 界面上，只需要点击鼠标，就可以完成搜索、聚合功能，生成炫丽的仪表板。
一、介绍
1、组成
ELK由Elasticsearch、Logstash和Kibana三部分组件组成；
Elasticsearch是个开源分布式搜索引擎，它的特点有：分布式，零配置，自动发现，索引自动分片，索引副本机制，restful风格接口，多数据源，自动搜索负载等。
Logstash是一个完全开源的工具，它可以对你的日志进行收集、分析，并将其存储供以后使用
kibana 是一个开源和免费的工具，它可以为 Logstash 和 ElasticSearch 提供的日志分析友好的 Web 界面，可以帮助您汇总、分析和搜索重要数据日志。
2、四大组件
Logstash: logstash server端用来搜集日志；
Elasticsearch: 存储各类日志；
Kibana: web化接口用作查寻和可视化日志；
Logstash Forwarder: logstash client端用来通过lumberjack 网络协议发送日志到logstash server；
3、工作流程
在需要收集日志的所有服务上部署logstash，作为logstash agent（logstash shipper）用于监控并过滤收集日志，将过滤后的内容发送到Redis，然后logstash indexer将日志收集在一起交给全文搜索服务ElasticSearch，可以用ElasticSearch进行自定义搜索通过Kibana 来结合自定义搜索进行页面展示。
下面是在两台节点上都安装一下环境。
nfs-31
rsync-41
二、安装JDk（两个节点都要安装）
配置阿里源：wget -O /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-7.repo
yum clean all
yum makecache
Logstash的运行依赖于Java运行环境，Elasticsearch 要求至少 Java 7。
[root@controller ~]# yum install java-1.8.0-openjdk -y
[root@controller ~]# java -version
openjdk version "1.8.0_151"
OpenJDK Runtime Environment (build 1.8.0_151-b12)
OpenJDK 64-Bit Server VM (build 25.151-b12, mixed mode)
1、关闭防火墙
systemctl stop firewalld.service #停止firewall
systemctl disable firewalld.service #禁止firewall开机启动
2、关闭selinux
sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/selinux/config
setenforce 0
三、安装Elasticsearch
基础环境安装（两个节点都要安装，配置文件不一样）
1）下载并安装GPG Key
[root@elk-node1 ~]# rpm --import https://packages.elastic.co/GPG-KEY-elasticsearch
2）添加yum仓库
[root@elk-node1 ~]# vim /etc/yum.repos.d/elasticsearch.repo
[elasticsearch-2.x]
name=Elasticsearch repository for 2.x packages
baseurl=http://packages.elastic.co/elasticsearch/2.x/centos
gpgcheck=1
gpgkey=http://packages.elastic.co/GPG-KEY-elasticsearch
enabled=1
3）安装elasticsearch
[root@elk-node1 ~]# yum install -y elasticsearch
4）添加自启动
chkconfig --add elasticsearch
5）启动命令
systemctl daemon-reload
systemctl enable elasticsearch.service
6）修改配置
[root@nfs-31 ~]#cd /etc/elasticsearch/
[root@nfs-31 /etc/elasticsearch]#ls
elasticsearch.yml logging.yml scripts
[root@nfs-31 /etc/elasticsearch]#cp elasticsearch.yml{,.bak}
[root@nfs-31 /etc/elasticsearch]#mkdir -p /data/es-data
[root@nfs-31 /etc/elasticsearch]#vim elasticsearch.yml
[root@nfs-31 /etc/elasticsearch]# grep '^[a-z]' elasticsearch.yml
cluster.name: hejianlai //集群名称
node.name: elk-node1 //节点名称
path.data: /data/es-data //数据存放目录
path.logs: /var/log/elasticsearch/ //日志存放目录
bootstrap.memory_lock: true //打开内存
network.host: 0.0.0.0 //监听网络
http.port: 9200 //端口
discovery.zen.ping.multicast.enabled: false //改为单播
discovery.zen.ping.unicast.hosts: ["10.0.0.31", "10.0.0.41"]
#注意
#data/es-data没权限，赋权限即可
chown -R elasticsearch:elasticsearch /data/es-data/
#启动
systemctl start elasticsearch
[root@nfs-31 /etc/elasticsearch]#systemctl status elasticsearch
● elasticsearch.service - Elasticsearch
Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: disabled)
Active: active (running) since Sat 2022-08-20 20:46:06 CST; 5s ago
Docs: http://www.elastic.co
Process: 2912 ExecStartPre=/usr/share/elasticsearch/bin/elasticsearch-systemd-pre-exec (code=exited, status=0/SUCCESS)
Main PID: 2913 (java)
CGroup: /system.slice/elasticsearch.service
└─2913 /bin/java -Xms256m -Xmx1g -Djava.awt.headless=true -XX:+UseParNewGC -XX:+...
访问地址10.0.0.31:9200
安装ES插件
#统计索引数
[root@nfs-31 /etc/elasticsearch]#curl -i -XGET 'http://10.0.0.31:9200/_count?pretty' -d '
> "query":{
> "match_all":{}
> }'
HTTP/1.1 200 OK
Content-Type: application/json; charset=UTF-8
Content-Length: 95
{
"count" : 0,
"_shards" : {
"total" : 0,
"successful" : 0,
"failed" : 0
}
}
#es插件，收费的不建议使用(这个不安装)
[root@nfs-31 ~]#/usr/share/elasticsearch/bin/plugin install marvel-agent
#安装开源的elasticsearch-head插件
[root@nfs-31 ~]#/usr/share/elasticsearch/bin/plugin install mobz/elasticsearch-head
-> Installing mobz/elasticsearch-head...
Trying https://github.com/mobz/elasticsearch-head/archive/master.zip ...
Downloading ..........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................DONE
Verifying https://github.com/mobz/elasticsearch-head/archive/master.zip checksums if available ...
NOTE: Unable to verify checksum for downloaded plugin (unable to find .sha1 or .md5 file to verify)
Installed head into /usr/share/elasticsearch/plugins/head
访问：10.0.0.31:9200/_plugin/head/
使用POST方法创建查询
使用GET方法查询数据
基本查询
41机器配置
[root@rsync-41 /etc/elasticsearch]#cat elasticsearch.yml
cluster.name: hejianlai
node.name: elk-node2
path.data: /data/es-data
path.logs: /var/log/elasticsearch/
bootstrap.memory_lock: true
network.host: 0.0.0.0
http.port: 9200
discovery.zen.ping.multicast.enabled: false
discovery.zen.ping.unicast.hosts: ["10.0.0.31", "10.0.0.41"]
#data/es-data没权限，赋权限即可
chown -R elasticsearch:elasticsearch /data/es-data/
在构建Elasticsearch(ES)多节点集群的时候，通常情况下只需要将elasticsearch.yml中的cluster.name设置成相同即可，ES会自动匹配并构成集群。但是很多时候可能由于不同的节点在不同的网段下，导致无法自动获取集群。此时可以将启用单播，显式指定节点的发现。具体做法是在elasticsearch.yml文件中设置如下两个参数：
重启31机器 开启41机器
[root@nfs-31 ~]#systemctl restart elasticsearch
[root@rsync-41 /etc/elasticsearch]#systemctl start elasticsearch
访问 10.0.0.31:9200/_plugin/head/
安装监控kopf
[root@nfs-31 ~]#/usr/share/elasticsearch/bin/plugin install lmenezes/elasticsearch-kopf
-> Installing lmenezes/elasticsearch-kopf...
Trying https://github.com/lmenezes/elasticsearch-kopf/archive/master.zip ...
Downloading ......................................................................................................................................................................................................................................................................................................................................................................................................................................................DONE
Verifying https://github.com/lmenezes/elasticsearch-kopf/archive/master.zip checksums if available ...
NOTE: Unable to verify checksum for downloaded plugin (unable to find .sha1 or .md5 file to verify)
Installed kopf into /usr/share/elasticsearch/plugins/kopf
访问 10.0.0.31:9200/_plugin/kopf/#!/cluster
四、安装Logstash(客户端即需要收集日志的机子，ES节点上都要安装)
官方文档地址：https://www.elastic.co/guide/en/logstash/current/index.html
31机器
#1、下载安装GPG Key
rpm --import https://packages.elastic.co/GPG-KEY-elasticsearch
#2、添加yum仓库
vim /etc/yum.repos.d/logstash.repo
[logstash-2.1]
name=Logstash repository for 2.1.x packages
baseurl=http://packages.elastic.co/logstash/2.1/centos
gpgcheck=1
gpgkey=http://packages.elastic.co/GPG-KEY-elasticsearch
enabled=1
#3、安装logstash
yum install -y logstash
#4、测试数据
#简单的输入输出
[root@nfs-31 ~]# /opt/logstash/bin/logstash -e 'input { stdin{} } output { stdout{} }'
OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N
Settings: Default filter workers: 1
Logstash startup completed
hello world
2022-08-20T13:35:44.716Z nfs-31 hello world
hi afei
2022-08-20T13:35:57.028Z nfs-31 hi afei
你好啊 渣渣辉
Received an event that has a different character encoding than you configured. {:text=>"你好啊 渣渣\\xE7辉", :expected_charset=>"UTF-8", :level=>:warn}
2022-08-20T13:36:14.242Z nfs-31 你好啊 渣渣\xE7辉
2022-08-20T13:36:19.675Z nfs-31
logstash shutdown completed
#可以使用rubydebug详细输出
[root@nfs-31 ~]#/opt/logstash/bin/logstash -e 'input { stdin{} } output { stdout{ codec => rubydebug } }'
OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N
Settings: Default filter workers: 1
Logstash startup completed
{
"message" => "",
"@version" => "1",
"@timestamp" => "2022-08-20T13:37:34.060Z",
"host" => "nfs-31"
}
mimi
{
"message" => "mimi",
"@version" => "1",
"@timestamp" => "2022-08-20T13:37:40.516Z",
"host" => "nfs-31"
}
#内容写进elasticsearch中
[root@nfs-31 ~]#/opt/logstash/bin/logstash -e 'input { stdin{} } output { elasticsearch{hosts=>["10.0.0.31"]} }'
OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N
Settings: Default filter workers: 1
Logstash startup completed
hello afei
你好帅
啦啦啦啦
dasda
[root@nfs-31 ~]#/opt/logstash/bin/logstash -e 'input { stdin{} } output { elasticsearch { hosts => ["10.0.0.31:9200"]} stdout{ codec => rubydebug}}'
OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N
Settings: Default filter workers: 1
Logstash startup completed
北京
{
"message" => "北京",
"@version" => "1",
"@timestamp" => "2022-08-20T13:41:44.333Z",
"host" => "nfs-31"
}
afei
{
"message" => "afei",
"@version" => "1",
"@timestamp" => "2022-08-20T13:42:05.132Z",
"host" => "nfs-31"
}
Logstash shutdown completed
logstash日志收集配置文件编写
#1、交换式输入信息
[root@nfs-31 ~]#vim /etc/logstash/conf.d/logstash-01.conf
[root@nfs-31 ~]#cat /etc/logstash/conf.d/logstash-01.conf
input { stdin { } }
output {
elasticsearch { hosts => ["10.0.0.31:9200"]}
stdout { codec => rubydebug }
}
#执行命令
[root@nfs-31 ~]#/opt/logstash/bin/logstash -f /etc/logstash/conf.d/logstash-01.conf
OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N
Settings: Default filter workers: 1
Logstash startup completed
wuhan
{
"message" => "wuhan",
"@version" => "1",
"@timestamp" => "2022-08-20T13:50:19.731Z",
"host" => "nfs-31"
}
小阿飞
Received an event that has a different character encoding than you configured. {:text=>"\\xE9\\x98小阿飞", :expected_charset=>"UTF-8", :level=>:warn}
{
"message" => "\\xE9\\x98小阿飞",
"@version" => "1",
"@timestamp" => "2022-08-20T13:50:34.473Z",
"host" => "nfs-31"
}
#收集系统日志
[root@nfs-31 ~]#vim /etc/logstash/conf.d/systemlog.conf
[root@nfs-31 ~]#cat /etc/logstash/conf.d/systemlog.conf
input{
file {
path => "/var/log/messages"
type => "sysstem"
start_position => "beginning"
}
}
output{
elasticsearch{
hosts => ["10.0.0.31:9200"]
index => "systemlog-%{+YYYY.MM.dd}"
}
}
#放在后台执行
[root@nfs-31 ~]#/opt/logstash/bin/logstash -f /etc/logstash/conf.d/systemlog.conf &
[1] 4471
[root@nfs-31 ~]#OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N
收集elk错误日志配置文件编写
[root@nfs-31 ~]#vim /etc/logstash/conf.d/elk_log.conf
[root@nfs-31 ~]#cat /etc/logstash/conf.d/elk_log.conf
input {
file {
path => "/var/log/messages"
type => "system"
start_position => "beginning"
}
}
input {
file {
path => "/var/log/elasticsearch/hejianlai.log"
type => "es-error"
start_position => "beginning"
codec => multiline {
pattern => "^\[" //正则匹配[开头的为一个事件
negate => true
what => "previous"
}
}
}
output {
if [type] == "system"{
elasticsearch {
hosts => ["10.0.0.31:9200"]
index => "systemlog-%{+YYYY.MM.dd}"
}
}
if [type] == "es-error"{
elasticsearch {
hosts => ["10.0.0.31:9200"]
index => "es-error-%{+YYYY.MM.dd}"
}
}
}
#放入后台运行
[root@nfs-31 ~]#/opt/logstash/bin/logstash -f /etc/logstash/conf.d/elk_log.conf &
[2] 4523
五、安装Kibana
官方下载地址：https://www.elastic.co/downloads/kibana
官方最新的版本出来了6.3.1太新了，下载后出现很多坑后来就下了4.3.1的·先用着吧
#1、kibana的安装：
[root@nfs-31 ~]#cd /usr/local/
[root@nfs-31 /usr/local]#wget https://download.elastic.co/kibana/kibana/kibana-4.6.3-linux-x86_64.tar.gz
[root@nfs-31 /usr/local]# tar -xf kibana-4.6.3-linux-x86_64.tar.gz
[root@nfs-31 /usr/local]#ln -s /usr/local/kibana-4.6.3-linux-x86_64 /usr/local/kibana
[root@nfs-31 /usr/local]#cd kibana
[root@nfs-31 /usr/local/kibana]#ls
bin data LICENSE.txt node_modules package.json src
config installedPlugins node optimize README.txt webpackShims
#2、修改配置文件
[root@nfs-31 /usr/local/kibana]#cd config/
[root@nfs-31 /usr/local/kibana/config]#pwd
/usr/local/kibana/config
[root@nfs-31 /usr/local/kibana/config]#cp kibana.yml kibana.yml.bb
[root@nfs-31 /usr/local/kibana/config]#vim kibana.yml
[root@nfs-31 /usr/local/kibana/config]#grep -Ev "^#|^$" kibana.yml
server.port: 5601
server.host: "0.0.0.0"
elasticsearch.url: "http://10.0.0.31:9200"
kibana.index: ".kibana"
#3、screen是一个全屏窗口管理器，它在几个进程（通常是交互式shell）之间复用物理终端。每个虚拟终端提供DEC VT100的功能。
yum install -y screen
#4、启动screen命令后运行kibana最后按ctrl+a+d组合键让其在单独的窗口里运行。
[root@nfs-31 /usr/local/kibana/config]#/usr/local/kibana/bin/kibana
log [22:21:42.921] [info][status][plugin:kibana@1.0.0] Status changed from uninitialized to green - Ready
log [22:21:42.940] [info][status][plugin:elasticsearch@1.0.0] Status changed from uninitialized to yellow - Waiting for Elasticsearch
[root@nfs-31 /usr/local/kibana/config]#screen -ls
There is a screen on:
4834.pts-0.nfs-31 (Attached)
1 Socket in /var/run/screen/S-root.
[root@nfs-31 /usr/local/kibana/config]#/usr/local/kibana/bin/kibana
log [22:22:27.423] [info][status][plugin:kibana@1.0.0] Status changed from uninitialized to green - Ready
log [22:22:27.440] [info][status][plugin:elasticsearch@1.0.0] Status changed from uninitialized to yellow - Waiting for Elasticsearch
log [22:22:27.448] [info][status][plugin:kbn_vislib_vis_types@1.0.0] Status changed from uninitialized to green - Ready
log [22:22:27.456] [info][status][plugin:markdown_vis@1.0.0] Status changed from uninitialized to green - Ready
log [22:22:27.459] [info][status][plugin:metric_vis@1.0.0] Status changed from uninitialized to green - Ready
log [22:22:27.463] [info][status][plugin:spyModes@1.0.0] Status changed from uninitialized to green - Ready
log [22:22:27.465] [info][status][plugin:statusPage@1.0.0] Status changed from uninitialized to green - Ready
log [22:22:27.466] [info][status][plugin:table_vis@1.0.0] Status changed from uninitialized to green - Ready
log [22:22:27.469] [info][listening] Server running at http://0.0.0.0:5601
log [22:22:27.480] [info][status][plugin:elasticsearch@1.0.0] Status changed from yellow to green - Kibana index ready
六、kibana简单使用
访问kibana地址：10.0.0.31:5601
第一次登录我们创建一个elk的es-error索引
添加之前写的systemlog索引
*为正则匹配
添加message和path字段
运用搜索栏功能，我们搜soft关键字
ELK日志平台搭建基本搞掂
标签：ELK,31,kibana,elasticsearch,nfs,日志,root,logstash,搭建 来源： https://www.cnblogs.com/Xafei/p/16634895.html