elk通过kafka收集处理Tomcat日志
作者：互联网
环境说明
| server | 安装软件 |
| 10.16.0.15 | elasticsearch+logstash+kibana+kafka |
| 10.16.0.14 | tomcat+logstash+mysql |
实现原理
业务主机
部署jsp业务环境[jdk8、tomcat、mysql(mariadb)]
发布一个简单的程序测试
部署logstash收集tomcat日志推送到kafka
分析主机
部署kafka消息服务器
部署ELK服务
logstash拉取kafka中json格式日志到elasticserach
kibana展示elasticsearch收集到的数据
业务主机-安装配置tomcat
1、jdk安装
[root@mode-01-0004 ~]# tar fx jdk-8u161-linux-x64.tar.gz -C /usr/local/jdk1.8 #配置Java环境变量 [root@mode-01-0004 ~]# vim /etc/profile export JAVA_HOME=/usr/local/jdk1.8 export JRE_HOME=/usr/local/jdk1.8/jre export CLASSPATH=.:/lib:/lib: export PATH=/bin:/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin #使变量生效 [root@mode-01-0004 ~]# sourcr /etc/profile #查看Java版本 [root@mode-01-0004 ~]# java -version java version "1.8.0_161" Java(TM) SE Runtime Environment (build 1.8.0_161-b12) Java HotSpot(TM) 64-Bit Server VM (build 25.161-b12, mixed mode)
2、安装Tomcat
安装tomcat
[root@mode-01-0004 ~]# wget http://apache.mirrors.hoobly.com/tomcat/tomcat-9/v9.0.14/bin/apache-tomcat-9.0.14.tar.gz [root@mode-01-0004 ~]# tar fx apache-tomcat-9.0.14.tar.gz -C /opt [root@mode-01-0004 ~]# cd /opt [root@mode-01-0004 opt]# mv apache-tomcat-9.0.14 tomcat
启动tomcat
[root@mode-01-0004 opt]# cd tomcat/bin/ [root@mode-01-0004 bin]# ./startup.sh
端口是否启动
[root@mode-01-0004 bin]# netstat -ntlp | grep 8080 tcp6 0 0 :::8080 :::* LISTEN 2469/java
页面响应码访问测试
[root@mode-01-0004 bin]# curl -I http://127.0.0.1:8080 HTTP/1.1 200 Content-Type: text/html;charset=UTF-8 Transfer-Encoding: chunked Date: Mon, 07 Jan 2019 07:42:58 GMT
查看日志文件是否正确生成
[root@mode-01-0004 bin]# ls ../logs/ catalina.2019-01-07.log host-manager.2019-01-07.log manager.2019-01-07.log catalina.out localhost.2019-01-07.log
3、配置Tomcat
编辑server配置文件
[root@mode-01-0004 bin]# cd ../conf/ [root@mode-01-0004 conf]# vim server.xml
修改日志配置
注释掉原日志格式
<!-- Access log processes all example. Documentation at: /docs/config/valve.html Note: The pattern used is equivalent to using pattern="common" <Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs" prefix="localhost_access_log" suffix=".txt" pattern="%h %l %u %t "%r" %s %b" /> -->
添加json格式参数
<Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs" ##指定日志存放目录
prefix="tomcat_access_log" suffix=".log" ##指定日志文件名称及后缀
pattern="{"clientip":"%h","ClientUser":"%l","authenticated":" %u","AccessTime":"%t","method":"%r","status":"%s","SendBytes":" %b","Query?string":"%q","partner":"%{Referer}i","AgentVersion":"%{User-Agent}i"}" />测试日志格式是否正确
停止并删除启动的测试文件
[root@mode-01-0004 conf]# cd ../bin [root@mode-01-0004 bin]# ./shutdown.sh [root@mode-01-0004 bin]# rm -rf ../logs/*
按上边的步骤启动tomcat测试是否会有报错
[root@mode-01-0004 bin]# ./startup.sh
访问tomcat主页
[root@mode-01-0004 bin]# curl http://127.0.0.1:8080
查看是否生成tomcat访问日志
[root@mode-01-0004 bin]# ls ../logs/ catalina.2019-01-07.log host-manager.2019-01-07.log manager.2019-01-07.log catalina.out localhost.2019-01-07.log tomcat_access_log.2019-01-07.log
查看日志格式
[root@mode-01-0004 logs]# cat tomcat_access_log.2019-01-07.log
{"clientip":"127.0.0.1","ClientUser":"-","authenticated":" -","AccessTime":"[07/Jan/2019:16:10:59 +0800]","method":"GET / HTTP/1.1","status":"200","SendBytes":" 11286","Query?string":"","partner":"-","AgentVersion":"curl/7.29.0"}4、发布一个图书管理系统
上传bookmanage压缩包并解压
链接：https://pan.baidu.com/s/1c-T-U0hA0bvd0Q5Im43SNw
提取码：8wp7
[root@mode-01-0004 ~]# ll bookmanage.zip -rw-r--r-- 1 root root 6065967 Jan 1 18:41 bookmanage.zip [root@mode-01-0004 ~]# unzip bookmanage.zip [root@mode-01-0004 ~]# mv bookmanage /opt/tomcat
将web数据拷贝到根目录
[root@mode-01-0004 tomcat]# cp -r bookmanage/WebRoot webapps/book
安装数据库
[root@mode-01-0004 tomcat]# yum -y install mariadb-server [root@mode-01-0004 tomcat]# systemctl start mariadb [root@mode-01-0004 tomcat]# mysql -uroot -p123456 MariaDB [(none)]> create database bookmanage;
将数据导入数据库
[root@mode-01-0004 tomcat]# mysql -uroot -p123456 bookmanage < bookmanage/bookManage.sql [root@mode-01-0004 tomcat]# mysql -uroot -p123456 MariaDB [(none)]> use bookmanage MariaDB [bookmanage]> show tables; +----------------------+ | Tables_in_bookmanage | +----------------------+ | admin | | books | | borrow_book | | history | | return_book | | student | +----------------------+ 6 rows in set (0.00 sec)
配置数据库登陆方式
[root@mode-01-0004 classes]# cat jdbc.properties driver=com.mysql.jdbc.Driver url=jdbc:mysql://localhost:3306/bookmanage?useUnicode=true&characterEncoding=UTF-8 username=root password=123456 [root@mode-01-0004 classes]# pwd /opt/tomcat/webapps/book/WEB-INF/classes
web测试页面是否可以访问
5、安装配置logstash
logstash安装参考《使用logstash收集并json化MySQL慢日志》
配置logstash
[root@mode-01-0004 ~]# cd /etc/logstash/conf.d/ [root@mode-01-0004 conf.d]# cp ../logstash-sample.conf ./logstash_tomcat.conf
修改配置文件
[root@mode-01-0004 conf.d]# cat logstash_tomcat.conf
# Sample Logstash configuration for creating a simple
# Beats -> Logstash -> Elasticsearch pipeline.
input {
file {
type => "tomcat_log"
path => "/opt/tomcat/logs/tomcat_access_log.*.log"
start_position => "beginning"
codec => json
}
}
output {
kafka {
bootstrap_servers => "10.16.0.15:9092"
topic_id => "tomcat"
compression_type => "snappy"
codec => json
}
}logstash启动测试
[root@mode-01-0004 conf.d]# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/logstash_tomcat.conf -t
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console [WARN ] 2019-01-08 17:13:31.024 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified Configuration OK [INFO ] 2019-01-08 17:13:32.065 [LogStash::Runner] runner - Using config.test_and_exit mode. Config Validation Result: OK. Exiting Logstash
提示OK即可继续配置分析主机
分析主机配置
elasticsearch+logstsh+kibana安装参考《使用logstash收集并json化MySQL慢日志》
下载安装kafka
[root@mode-01-0005 ~]# wget http://mirrors.hust.edu.cn/apache/kafka/2.1.0/kafka_2.12-2.1.0.tgz [root@mode-01-0005 ~]# tar fx kafka_2.12-2.1.0.tgz [root@mode-01-0005 ~]# mv kafka-2.1.0-src /usr/local/kafka
配置kafka
[root@mode-01-0005 ~]# cd /usr/local/kafka [root@mode-01-0005 kafka]# vim config/server.properties
修改：
listeners=PLAINTEXT://10.16.0.15:9092 zookeeper.connect=10.16.0.15:2181
启动kafka
[root@mode-01-0005 bin]# jobs [1]- Running nohup ./zookeeper-server-start.sh ../config/zookeeper.properties > zookeeper.log 2>&1 & [2]+ Running nohup ./kafka-server-start.sh ../config/server.properties > kafka.log 2>&1 &
[root@mode-01-0005 bin]# netstat -ntlp ... tcp6 0 0 10.16.0.15:9092 :::* LISTEN 24212/java tcp6 0 0 :::2181 :::* LISTEN 23572/java ...
kafka测试
#定义producer写入数据到topic test [root@mode-01-0005 bin]# ./kafka-console-producer.sh --broker-list 10.16.0.15:9092 --topic test >hello #消费消息 [root@mode-01-0005 bin]# ./kafka-console-consumer.sh --bootstrap-server 10.16.0.15:9092 --topic test --from-beginning hello
topic管理
#创建topic [root@mode-01-0005 bin]# ./kafka-topics.sh --create --zookeeper 10.16.0.15:2181 --replication-factor 1 --partitions 1 --topic test Created topic "test".
#查看topic [root@mode-01-0005 bin]# ./kafka-topics.sh --list --zookeeper 10.16.0.15:2181 test
#查看某个topic得详细信息 [root@mode-01-0005 bin]# ./kafka-topics.sh --describe --zookeeper 10.16.0.15:2181 --topic test Topic:testPartitionCount:1ReplicationFactor:1Configs: Topic: testPartition: 0Leader: 0Replicas: 0Isr: 0
partition：partition id
leader：当前负责读写得lead broker id
replicas：当前partition得所有replication broker list
lsr：relicas得子集，只包含处于活动状态得broker
#删除topic [root@mode-01-0005 bin]# ./kafka-topics.sh --delete --zookeeper 10.16.0.15:2181 --topic test
配置logstash
[root@mode-01-0005 ~]# cat /etc/logstash/conf.d/logstash_tomcat.conf
# Sample Logstash configuration for creating a simple
# Beats -> Logstash -> Elasticsearch pipeline.
input {
#beats {
# port => 5044
#}
kafka {
bootstrap_servers => "10.16.0.15:9092"
topics => "tomcat"
codec => "json"
consumer_threads => 5
decorate_events => true
}
}
output {
elasticsearch {
hosts => ["http://10.16.0.15:9200"]
index => "tomcat-%{+YYYY.MM.dd}"
#user => "elastic"
#password => "changeme"
}
#stdout {
# codec => rubydebug
#}
}屏幕输出测试
{
"host" => "mode-01-0004.novalocal",
"method" => "GET /book/lib/ligerUI/skins/Aqua/images/win/taskbar-task.gif HTTP/1.1",
"Query?string" => "",
"status" => "200",
"AccessTime" => "[12/Jan/2019:23:02:23 +0800]",
"path" => "/opt/tomcat/logs/tomcat_access_log.2019-01-12.log",
"ClientUser" => "-",
"type" => "tomcat_log",
"authenticated" => " -",
"clientip" => "10.16.0.10",
"SendBytes" => " 277",
"@timestamp" => 2019-01-12T15:02:25.587Z,
"@version" => "1",
"partner" => "http://10.16.0.14:8080/book/lib/ligerUI/skins/Aqua/css/ligerui-dialog.css",
"AgentVersion" => "Mozilla/5.0 (Windows NT 6.3; Win64; x64; rv:64.0) Gecko/20100101 Firefox/64.0"
}启动elasticsearch+kibana配置相应的索引及图表
标签：01,root,mode,tomcat,0004,bin,kafka 来源： http://blog.51cto.com/13944252/2342080