Error reported in Logstash: retrying failed action with response code:403.
The error message is shown in the figure below.
This is because the index in ES is set to read-only mode. The possible reason for this is that when the disk is full or the usage exceeds 95%, Elasticsearch prevents nodes from running out of disk space and protects the cluster from complete downtime . automatically sets the index to read-only mode,
At this time, new data can only be queried and cannot be stored in the database. And an error will also be reported in the ES log: nexpected error while indexing monitoring document org.elasticsearch.xpack.monitoring.exporter.ExportException: ClusterBlockException.
At this time, even if the hard disk is expanded and the available disk space is increased, ES still reports an error, the problem still exists, and the index still cannot be stored in the database. This is because the ES cluster has set read-only for the index in order to protect the cluster from complete downtime. At this time, you need to manually set the read-only status of the index to remove it.
There are two solutions at this time, but the effect is the same, both are to lift the read-only state of the index.
1. Execute the following command in the developer tools in kibana to set the index to allow writing. After running this command. We can query the logstash log to see if it is back to normal.
PUT _all/_settings
{
"index.blocks.read_only_allow_delete": null
}2. Execute api on the server where ES is located:
curl -XPUT -H "Content-Type: application/json" http://localhost:9200/_all/_settings -d '{"index.blocks.read_only_allow_delete": null}'At this time, we checked the Logstash log and found that the error retrying failed action with response code: 403. was no longer reported. The problem is solved.