编程语言
首页 > 编程语言> > ES数据未同步(flood stage disk watermark [95%] exceeded on [ts_jvdneTHa3dgd36yACwg][node-1][/usr/data/node

ES数据未同步(flood stage disk watermark [95%] exceeded on [ts_jvdneTHa3dgd36yACwg][node-1][/usr/data/node

作者:互联网

Elasticsearch索引只读问题

背景

手上项目在线上正常运行,数据库中有数据更改后,数据未同步到es对应的索引中

原因

将elasticsearch和logstash的日志下载下来后发现es日志中有很多行提示:

2021-10-21T23:53:03.448528162Z {"type": "server", "timestamp": "2021-10-21T23:53:03,448Z", "level": "WARN", "component": "o.e.c.r.a.DiskThresholdMonitor", "cluster.name": "my-cluster", "node.name": "node-1", "message": "flood stage disk watermark [95%] exceeded on [ts_jvdneTHa3dgd36yACwg][node-1][/usr/data/nodes/0] free: 3.9gb[3.9%], all indices on this node will be marked read-only", "cluster.uuid": "a8BvfadOTr95b2CplpFh7Q", "node.id": "ts_jvdneTHa3dgd36yACwg"  }

logstash日志中也有相关错误

2021-10-22T08:10:02.813081623Z [2021-10-22T08:10:02,813][INFO ][logstash.outputs.elasticsearch][indexname][0d06b97eedc01234eb09f649512131204e592b676bb231139b4b62e92ab] retrying failed action with response code: 429 ({"type"=>"cluster_block_exception", "reason"=>"index [indexname] blocked by: [TOO_MANY_REQUESTS/12/index read-only / allow delete (api)];"})

翻看ES官网文档后发现 es在发现磁盘空间大于95%以后,将变成只读状态,此时将不会更新数据,只能查询和删除。查看es索引设置后发现

get localhost:9200/index_name/_settings
    "settings": {
            "index": {
                "number_of_shards": "1",
                "blocks": {
                    "read_only_allow_delete": "true"
                },
            }
    }

//因日志显示剩余3.9g空间,此时通过
put localhost:9200/index_name/_settings
{
  "index": {
    "blocks": {
      "read_only_allow_delete": "false"
    }
  }
}
//关闭只读状态

然后再次查看索引设置,此时只读状态已关闭,过一会发现数据仍然没有同步成功,无果

解决方法

当磁盘空间大于5%后,es日志不再显示警告,此时还是无法更新索引,需要重复上述关闭只读状态操作

标签:node,10,index,read,marked,only,gb,日志
来源: https://www.cnblogs.com/aluna/p/15441510.html