首页 > 其他分享> > 使用 filebeat原生处理日志时间，就是使用日志文件中message字段值开头的时间覆盖默认的@timestamp时间，使用filebeat processor配置，pipeline不符合要求
使用 filebeat原生处理日志时间，就是使用日志文件中message字段值开头的时间覆盖默认的@timestamp时间，使用filebeat processor配置，pipeline不符合要求
作者：互联网
情况说明
Filebeat 收集的日志发送到 ElasticSearch 后，会默认添加一个 @timestamp 字段做为时间戳用于检索，而日志中的信息会所有添加到 message 字段中，可是这个时间是 Filebeat 采集日志的时间，不是日志生成的实际时间，因此为了便于检索日志，须要将 @timestamp 替换为 message 字段中的时间。
首先日志格式以下：
# info文件的日志内容
2022-05-24 00:12:19.196 INFO com.jdd.parking.modules.chargefee.standard.rest.GetCarChargeFeeController Line:131 - 有牌车出口码计费,plateNo:鄂A98J51
2022-05-24 00:12:19.196 INFO com.jdd.parking.modules.chargefee.standard.service.impl.ChargeFeeV2ServiceImpl Line:64 - 计费服务类,出场码计费,传入参数:OutParkChargeFeeDTO(parkCode=58818, serialNo=c79e5114-7103aa77, plateNo=鄂A98J51, imgPath=https://jdd-parking.oss-cn-qingdao.aliyuncs.com/picture/1/c79e5114-7103aa77/20220524/00/20220524_001218_813.jpg)
2022-05-24 00:12:19.205 INFO com.jdd.parking.modules.chargefee.standard.utils.ChargeFeeUtils Line:444 - 计费服务类,计费开始,车牌号：鄂A98J51,openId:null
2022-05-24 00:12:19.208 INFO com.jdd.parking.modules.chargefee.standard.utils.ChargeFeeUtils Line:877 - 计费服务类,单次入场纪录计费,传入参数：计费区域Id：59a3ef3754d87c85b02c51e4ed0e03ae,计费开始时间：2022-05-23 22:44:04,计费结束时间：2022-05-24 00:12:19
2022-05-24 00:12:19.209 INFO com.jdd.parking.modules.chargefee.standard.utils.ChargeFeeUtils Line:897 - 计费服务类,单次入场纪录计费,按24小时计算一天,停车整天数：0天,不足整天的停车时间：88分钟
# error文件的日志内容
2022-05-24 00:12:19.210 ERROR com.jdd.parking.modules.chargefee.standard.service.impl.ChargeFeeV2ServiceImpl Line:129 - 计费服务类,异常信息
com.jdd.parking.common.exception.JddException: 计费规则模块不存在
at com.jdd.parking.modules.chargefee.standard.utils.ChargeFeeUtils.lessThanOneDayFee(ChargeFeeUtils.java:85)
at com.jdd.parking.modules.chargefee.standard.utils.ChargeFeeUtils.chargeEnterLogFee(ChargeFeeUtils.java:920)
at com.jdd.parking.modules.chargefee.standard.utils.ChargeFeeUtils.chargeTotalFeeMonthlyV3(ChargeFeeUtils.java:525)
at com.jdd.parking.modules.chargefee.standard.service.impl.ChargeFeeV2ServiceImpl.outParkChargeFeeV2(ChargeFeeV2ServiceImpl.java:97)
at com.jdd.parking.modules.chargefee.standard.rest.GetCarChargeFeeController.getOutParkChargeFeeV2(GetCarChargeFeeController.java:133)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:190)
at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:138)
at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:105)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:878)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:792)
at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87)
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1040)
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:943)
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006)
at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:909)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:517)
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:584)
at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:74)
at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:129)
at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.doFilterInternal(WebMvcMetricsFilter.java:93)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
at io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:84)
at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62)
at io.undertow.servlet.handlers.ServletChain$1.handleRequest(ServletChain.java:68)
at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36)
at io.undertow.servlet.handlers.RedirectDirHandler.handleRequest(RedirectDirHandler.java:68)
at io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:132)
at io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57)
at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46)
at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64)
at io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60)
at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77)
at io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43)
at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:269)
at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:78)
at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:133)
at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:130)
at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48)
at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)
at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:249)
at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:78)
at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:99)
at io.undertow.server.Connectors.executeRootHandler(Connectors.java:370)
at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830)
at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:2019)
at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1558)
at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1449)
at java.lang.Thread.run(Thread.java:748)
2022-05-24 00:12:19.211 ERROR com.jdd.parking.modules.chargefee.standard.rest.GetCarChargeFeeController Line:137 - 有牌车出口码计费失败,错误信息：{}
com.jdd.parking.common.exception.JddException: 计费服务类,出场码计费系统错误,计费规则模块不存在
at com.jdd.parking.modules.chargefee.standard.service.impl.ChargeFeeV2ServiceImpl.outParkChargeFeeV2(ChargeFeeV2ServiceImpl.java:130)
at com.jdd.parking.modules.chargefee.standard.rest.GetCarChargeFeeController.getOutParkChargeFeeV2(GetCarChargeFeeController.java:133)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:190)
at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:138)
at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:105)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:878)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:792)
at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87)
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1040)
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:943)
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006)
at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:909)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:517)
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:584)
at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:74)
at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:129)
at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.doFilterInternal(WebMvcMetricsFilter.java:93)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
at io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:84)
at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62)
at io.undertow.servlet.handlers.ServletChain$1.handleRequest(ServletChain.java:68)
at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36)
at io.undertow.servlet.handlers.RedirectDirHandler.handleRequest(RedirectDirHandler.java:68)
at io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:132)
at io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57)
at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46)
at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64)
at io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60)
at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77)
at io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43)
at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:269)
at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:78)
at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:133)
at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:130)
at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48)
at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)
at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:249)
at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:78)
at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:99)
at io.undertow.server.Connectors.executeRootHandler(Connectors.java:370)
at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830)
at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:2019)
at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1558)
at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1449)
at java.lang.Thread.run(Thread.java:748)
当前的日志传输路径是filebeat->es，没有使用logstash
解决办法
使用elasticsearch pipeline功能
参考文章：http://www.javashuo.com/article/p-ghefxmal-nu.html
索引文件名上的时间是当前时间
pipeline中没有加"timezone": "Asia/Shanghai", 可以看到，有的是当前系统时间，有的是UTC时间，但是真正想要的时间是日志message开头的时间
pipeline中加上"timezone": "Asia/Shanghai", 可以看到，有的是当前系统时间，有的是UTC时间，但是真正想要的时间是日志message开头的时间
结合以上实践，使用elasticsearch pipeline功能不符合要求，建议使用使用filebeat processor配置
使用filebeat processor配置
参考网址：https://blog.csdn.net/qq_34083066/article/details/115354511
索引文件名上的时间是日志message开头截取的时间
# cat /etc/filebeat/filebeat.yml
filebeat.inputs:
- type: log
enabled: true
paths:
- /opt/log_error.log
fields:
log_source: fee
log_type: error
fields_under_root: true
tags: ["application"]
multiline.pattern: '^\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}.\d* ' # 匹配日志开头的时间，划分为一行
multiline.negate: true
multiline.match: after
- type: log
enabled: true
paths:
- /opt/log_info.log
fields:
log_source: fee
log_type: info
fields_under_root: true
tags: ["application"]
multiline.pattern: '^\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}.\d* '
multiline.negate: true
multiline.match: after
filebeat.config.modules:
path: ${path.config}/modules.d/*.yml
reload.enabled: false
reload.period: 10s
setup.ilm.enabled: false
setup.template.settings:
index.number_of_shards: 1
index.number_of_replicas: 0
index.codec: best_compression
setup.dashboards.enabled: true
setup.kibana:
host: "192.168.20.250:5601"
output.elasticsearch:
hosts: ["localhost:9200"]
indices:
- index: "work-%{+yyyy.MM.dd}"
when.contains:
tags: "application"
processors:
- decode_json_fields:
fields: ["message"]
target: ""
overwrite_keys: false
process_array: false
max_depth: 1
# 这个脚本的内容是根据message字段值开始的俩空格截取值然后拼接在一起，也就是时间：2022-05-24 00:12:19.211，赋给新变量：start_time
- script:
lang: javascript
id: my_filter
tag: enable
source: >
function process(event) {
var str= event.Get("message");
var time =str.split(" ").slice(0,2).join(" ");
event.Put("start_time",time);
}
# 使用新变量的值start_time替换默认的@timestamp的值
- timestamp:
field: start_time
#timezone: Asia/Shanghai
target_field: "@timestamp"
layouts:
- '2006-01-02 15:04:05'
- '2006-01-02 15:04:05.999'
test:
- '2006-01-02 15:04:05'
- '2006-01-02 15:04:05.999'
# 移除无用的字段
- drop_fields:
fields: ["input","agent","ecs","start_time"]
效果显示
本文中timezone说明
@timestamp 这个字段是系统自带的，默认时间就是采集log的UTC时间 。例如 现在时间是2020-12-30 18:59:58.234 ，那么他的值为2020-12-30T10:59:58.234Z
现在我们手动替换@timestamp为日志时间（东八区），需要将时间字符串解析成UTC时间，赋值给@timestamp，那么我们必须显示的指定timezone 信息。
1.指定timezone后的表现
原先log时间字符串：2020-08-05 16:21:51.824。重新赋值的@timestamp值： 2020-08-05T08:21:51.824Z （正确）
2.未指定timezone后的表现
原先log时间字符串：2020-08-05 16:21:51.824。重新赋值的@timestamp值： 2020-08-05T21:51.824Z （默认使用了UTC零时区解析）
按上面配置是没有指定timezone的，这个一定要注意。若是指定了则会跟log时间相差8个小时
标签：filebeat,java,handlers,org,io,使用,undertow,日志,servlet 来源： https://www.cnblogs.com/sanduzxcvbnm/p/16321325.html