ELK--03 收集docker日志
作者:互联网
目录
ELK--03 收集docker日志
1.filebeat收集docker类型日志 ( 普通版本)
1.安装dockder
[root@db02 ~]# yum install -y yum-utils device-mapper-persistent-data lvm2
[root@db02 ~]# wget -O /etc/yum.repos.d/docker-ce.repo https://download.docker.com/linux/centos/docker-ce.repo
[root@db02 ~]# sed -i 's+download.docker.com+mirrors.tuna.tsinghua.edu.cn/docker-ce+' /etc/yum.repos.d/docker-ce.repo
[root@db02 ~]# yum makecache fast
[root@db02 ~]# yum install docker-ce -y
[root@db02 ~]# mkdir -p /etc/docker
[root@db02 ~]# tee /etc/docker/daemon.json <<-'EOF'
{
"registry-mirrors": ["https://ig2l319y.mirror.aliyuncs.com"]
}
EOF
[root@db02 ~]# systemctl daemon-reload
[root@db02 ~]# systemctl restart docker
2.启动2个Nginx容器并访问测试
[root@db02 ~]# docker run -d -p 80:80 nginx
[root@db02 ~]# docker run -d -p 8080:80 nginx
3.测试数据是否能通
[root@db02 ~]# curl 10.0.0.52
[root@db02 ~]# curl 10.0.0.52:8080
4.配置filebeat
[root@db02 ~]# cat /etc/filebeat/filebeat.yml
filebeat.inputs:
- type: docker
containers.ids:
- '*'
output.elasticsearch:
hosts: ["10.0.0.51:9200"]
index: "docker-%{[beat.version]}-%{+yyyy.MM}"
setup.template.name: "docker"
setup.template.pattern: "docker-*"
setup.template.enabled: false
setup.template.overwrite: true
5.重启filebeat
[root@db02 ~]# systemctl restart filebeat
6.重启es
[root@db02 ~]# systemctl restart elasticsearch
7.访问生成测试数据
[root@db02 ~]# curl 10.0.0.52/1111111111
[root@db02 ~]# curl 10.0.0.52:8080/2222222222
8.登录es-head插件查询和kibana添加
2.filebeat收集docker日志使用docker-compose按服务拆分索引
1.假设的场景
nginx容器 80端口
toncat容器 8080端口
2.理想中的索引名称
docker-nginx-6.6.0-2020.02
docker-tomcat-6.6.0-2020.02
3.理想的日志记录格式
nginx容器日志:
{
"log": "xxxxxx",
"stream": "stdout",
"time": "xxxx",
"service": "nginx"
}
tomcat容器日志:
{
"log": "xxxxxx",
"stream": "stdout",
"time": "xxxx",
"service": "tomcat"
}
4.docker-compose配置
[root@db02 ~]# yum install docker-compose -y
[root@db02 ~]# cat >docker-compose.yml<<EOF
version: '3'
services:
nginx:
image: nginx:latest
labels:
service: nginx
logging:
options:
labels: "service"
ports:
- "80:80"
tomcat:
image: nginx:latest
labels:
service: tomcat
logging:
options:
labels: "service"
ports:
- "8080:80"
EOF
5.删除旧的容器
[root@db02 ~]# docker stop $(docker ps -q)
[root@db02 ~]# docker rm $(docker ps -qa)
6.启动容器
[root@db02 ~]# docker-compose up -d
7.配置filebeat
[root@db02 ~]# cat >/etc/filebeat/filebeat.yml <<EOF
filebeat.inputs:
- type: log
enabled: true
paths:
- /var/lib/docker/containers/*/*-json.log
json.keys_under_root: true
json.overwrite_keys: true
output.elasticsearch:
hosts: ["10.0.0.51:9200"]
indices:
- index: "docker-nginx-%{[beat.version]}-%{+yyyy.MM}"
when.contains:
attrs.service: "nginx"
- index: "docker-tomcat-%{[beat.version]}-%{+yyyy.MM}"
when.contains:
attrs.service: "tomcat"
setup.template.name: "docker"
setup.template.pattern: "docker-*"
setup.template.enabled: false
setup.template.overwrite: true
EOF
8.重启filebeat
[root@db02 ~]# systemctl restart filebeat
9.生成访问日志
[root@db02 ~]# curl 127.0.0.1/nginxxxxxxxxxxx
[root@db02 ~]# curl 127.0.0.1:8080/dbbbbbbbbb
10.es-head插件查看
3.filebeat收集docker日志 ,按照日志类型,access/error拆分
1.之前收集的docker日志目前不完善的地方
正常日志和报错日志放在一个索引里了
2.理想中的索引名称
docker-nginx-access-6.6.0-2020.02
docker-nginx-error-6.6.0-2020.02
docker-db-access-6.6.0-2020.02
docker-db-error-6.6.0-2020.02
3.filebeat配置文件
[root@db02 ~]# cat >/etc/filebeat/filebeat.yml <<EOF
filebeat.inputs:
- type: log
enabled: true
paths:
- /var/lib/docker/containers/*/*-json.log
json.keys_under_root: true
json.overwrite_keys: true
output.elasticsearch:
hosts: ["10.0.0.51:9200"]
indices:
- index: "docker-nginx-access-%{[beat.version]}-%{+yyyy.MM}"
when.contains:
attrs.service: "nginx"
stream: "stdout"
- index: "docker-nginx-error-%{[beat.version]}-%{+yyyy.MM}"
when.contains:
attrs.service: "nginx"
stream: "stderr"
- index: "docker-tomcat-access-%{[beat.version]}-%{+yyyy.MM}"
when.contains:
attrs.service: "tomcat"
stream: "stdout"
- index: "docker-tomcat-error-%{[beat.version]}-%{+yyyy.MM}"
when.contains:
attrs.service: "tomcat"
stream: "stderr"
setup.template.name: "docker"
setup.template.pattern: "docker-*"
setup.template.enabled: false
setup.template.overwrite: true
EOF
4.重启filebeat
[root@db02 ~]# systemctl restart filebeat
5.生成测试数据
[root@db02 ~]# curl 127.0.0.1/nginxxxxxxxxxxx
[root@db02 ~]# curl 127.0.0.1:8080/dbbbbbbbbb
6.登录es-head插件查看
4.filebeat收集docker日志优化版
1.需求分析
json格式并且按照下列索引生成
docker-nginx-access-6.6.0-2020.02
docker-tomcat-access-6.6.0-2020.02
docker-tomcat-error-6.6.0-2020.02
docker-nginx-error-6.6.0-2020.02
2.停止并且删除以前的容器
[root@db02 ~]# docker stop $(docker ps -qa)
[root@db02 ~]# docker rm $(docker ps -qa)
3.创建新容器并将容器内的日志映射出来
[root@db02 ~]# docker run -d -p 80:80 -v /opt/nginx:/var/log/nginx/ nginx
[root@db02 ~]# docker run -d -p 8080:80 -v /opt/tomcat:/var/log/nginx/ nginx
[root@db02 ~]# ll /opt/
drwxr-xr-x 2 root root 41 Mar 1 10:24 nginx
drwxr-xr-x 2 root root 41 Mar 1 10:25 tomcat
4.准备json格式的nginx配置文件,将其他机器的nginx的配置文件复制到本台服务器上面
[root@db02 ~]# scp 10.0.0.51:/etc/nginx/nginx.conf /root/
[root@db02 ~]# ll
-rw-r--r-- 1 root root 1358 Mar 1 10:27 nginx.conf
#将日志格式个更改为json格式
[root@db02 ~]# grep "access_log" nginx.conf
access_log /var/log/nginx/access.log json;
5.拷贝到容器里并重启
#查看容器id
[root@db02 ~]# docker ps
[root@db02 ~]# docker cp nginx.conf Nginx容器的ID:/etc/nginx/
[root@db02 ~]# docker cp nginx.conf tomcat容器的ID:/etc/nginx/
[root@db02 ~]# docker stop $(docker ps -qa)
[root@db02 ~]# docker start Nginx容器的ID
[root@db02 ~]# docker start tomcat容器的ID
6.删除ES已经存在的索引( 在 es-head 插件中删除 )
7.配置filebeat配置文件
[root@db02 ~]# cat >/etc/filebeat/filebeat.yml <<EOF
filebeat.inputs:
- type: log
enabled: true
paths:
- /opt/nginx/access.log
json.keys_under_root: true
json.overwrite_keys: true
tags: ["nginx_access"]
- type: log
enabled: true
paths:
- /opt/nginx/error.log
tags: ["nginx_err"]
- type: log
enabled: true
paths:
- /opt/tomcat/access.log
json.keys_under_root: true
json.overwrite_keys: true
tags: ["tomcat_access"]
- type: log
enabled: true
paths:
- /opt/tomcat/error.log
tags: ["tomcat_err"]
output.elasticsearch:
hosts: ["10.0.0.51:9200"]
indices:
- index: "docker-nginx-access-%{[beat.version]}-%{+yyyy.MM}"
when.contains:
tags: "nginx_access"
- index: "docker-nginx-error-%{[beat.version]}-%{+yyyy.MM}"
when.contains:
tags: "nginx_err"
- index: "docker-tomcat-access-%{[beat.version]}-%{+yyyy.MM}"
when.contains:
tags: "tomcat_access"
- index: "docker-tomcat-error-%{[beat.version]}-%{+yyyy.MM}"
when.contains:
tags: "tomcat_err"
setup.template.name: "docker"
setup.template.pattern: "docker-*"
setup.template.enabled: false
setup.template.overwrite: true
EOF
8.重启filebeat
[root@db02 ~]# systemctl restart filebeat
9.访问并测试
[root@db02 ~]# curl 127.0.0.1/hahaha
[root@db02 ~]# curl 127.0.0.1:8080/hahaha
[root@db02 ~]# cat /opt/nginx/access.log
[root@db02 ~]# cat /opt/tomcat/access.log
9.es-head查看
标签:ELK,filebeat,03,nginx,db02,docker,root,日志 来源: https://www.cnblogs.com/gongjingyun123--/p/12490927.html