问题描述;
服务器断电，es 直接挂掉，再次启动时，因以前数据量过大，大约500g，单机版es 启动恢复数据很是吃力，jvm 回收频繁。
[2019-06-17T16:24:14,303][INFO ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][20] overhead, spent [319ms] collecting in the last [1s]
[2019-06-17T16:26:05,781][INFO ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][131] overhead, spent [263ms] collecting in the last [1s]
[2019-06-17T16:28:39,668][INFO ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][284] overhead, spent [337ms] collecting in the last [1.1s]
[2019-06-17T16:29:38,902][INFO ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][343] overhead, spent [272ms] collecting in the last [1s]
[2019-06-17T16:30:31,004][INFO ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][395] overhead, spent [305ms] collecting in the last [1s]
[2019-06-17T16:32:14,218][INFO ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][498] overhead, spent [359ms] collecting in the last [1s]
[2019-06-17T16:33:16,289][INFO ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][560] overhead, spent [421ms] collecting in the last [1s]
[2019-06-17T16:35:25,630][INFO ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][689] overhead, spent [286ms] collecting in the last [1s]
[2019-06-17T16:36:08,858][INFO ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][old][727][3] duration [5.7s], collections [1]/[6.1s], total [5.7s]/[6.3s], memory [3.2gb]->[2.2gb]/[7.8gb], all_pools {[young] [839.5mb]->[8.7mb]/[865.3mb]}{[survivor] [108.1mb]->[0b]/[108.1mb]}{[old] [2.2gb]->[2.2gb]/[6.9gb]}
[2019-06-17T16:36:08,859][WARN ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][727] overhead, spent [5.7s] collecting in the last [6.1s]
[2019-06-17T16:36:09,146][WARN ][o.e.i.c.IndicesClusterStateService] [node-1] [[audit-net-udpflow-2019-05-24][0]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.indices.recovery.RecoveryFailedException: [audit-net-udpflow-2019-05-24][0]: Recovery failed on {node-1}{bxqdAUPETm6og34d591R-Q}{93eABCNYTVioLPSVZlfbYA}{172.16.211.130}{172.16.211.130:9310}{rack=r1}
at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$6(IndexShard.java:2019) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:568) [elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_151]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_151]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_151]
Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed to recover from gateway
at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:402) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromStore$0(StoreRecovery.java:94) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:292) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
at org.elasticsearch.index.shard.StoreRecovery.recoverFromStore(StoreRecovery.java:92) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
at org.elasticsearch.index.shard.IndexShard.recoverFromStore(IndexShard.java:1577) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$6(IndexShard.java:2015) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
... 4 more
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: failed to open reader on writer
at org.elasticsearch.index.engine.InternalEngine.createSearcherManager(InternalEngine.java:564) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:229) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:154) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:25) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:2140) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:2122) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
at org.elasticsearch.index.shard.IndexShard.internalPerformTranslogRecovery(IndexShard.java:1335) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
at org.elasticsearch.index.shard.IndexShard.performTranslogRecovery(IndexShard.java:1295) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:395) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromStore$0(StoreRecovery.java:94) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:292) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
at org.elasticsearch.index.shard.StoreRecovery.recoverFromStore(StoreRecovery.java:92) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
at org.elasticsearch.index.shard.IndexShard.recoverFromStore(IndexShard.java:1577) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$6(IndexShard.java:2015) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
... 4 more
Caused by: java.io.IOException: Map failed: MMapIndexInput(path="/TongYou/hbdata/data/nodes/0/indices/aNIh4sZLQ8WjmVCqeitB8w/0/index/_1in2.fdt") [this may be caused by lack of enough unfragmented virtual address space or too restrictive virtual memory limits enforced by the operating system, preventing us to map a chunk of 753581538 bytes. Please review 'ulimit -v', 'ulimit -m' (both should return 'unlimited'), and 'sysctl vm.max_map_count'. More information: http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html]
at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:940) ~[?:?]
at org.apache.lucene.store.MMapDirectory.map(MMapDirectory.java:267) ~[lucene-core-7.1.0.jar:7.1.0 84c90ad2c0218156c840e19a64d72b8a38550659 - ubuntu - 2017-10-13 16:12:42]
at org.apache.lucene.store.MMapDirectory.openInput(MMapDirectory.java:242) ~[lucene-core-7.1.0.jar:7.1.0 84c90ad2c0218156c840e19a64d72b8a38550659 - ubuntu - 2017-10-13 16:12:42]
at org.apache.lucene.store.FilterDirectory.openInput(FilterDirectory.java:99) ~[lucene-core-7.1.0.jar:7.1.0 84c90ad2c0218156c840e19a64d72b8a38550659 - ubuntu - 2017-10-13 16:12:42]
at org.apache.lucene.codecs.compressing.CompressingStoredFieldsReader.<init>(CompressingStoredFieldsReader.java:150) ~[lucene-core-7.1.0.jar:7.1.0 84c90ad2c0218156c840e19a64d72b8a38550659 - ubuntu - 2017-10-13 16:12:42]
at org.apache.lucene.codecs.compressing.CompressingStoredFieldsFormat.fieldsReader(CompressingStoredFieldsFormat.java:121) ~[lucene-core-7.1.0.jar:7.1.0 84c90ad2c0218156c840e19a64d72b8a38550659 - ubuntu - 2017-10-13 16:12:42]
at org.apache.lucene.codecs.lucene50.Lucene50StoredFieldsFormat.fieldsReader(Lucene50StoredFieldsFormat.java:173) ~[lucene-core-7.1.0.jar:7.1.0 84c90ad2c0218156c840e19a64d72b8a38550659 - ubuntu - 2017-10-13 16:12:42]
at org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentCoreReaders.java:125) ~[lucene-core-7.1.0.jar:7.1.0 84c90ad2c0218156c840e19a64d72b8a38550659 - ubuntu - 2017-10-13 16:12:42]
at org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:78) ~[lucene-core-7.1.0.jar:7.1.0 84c90ad2c0218156c840e19a64d72b8a38550659 - ubuntu - 2017-10-13 16:12:42]
at org.apache.lucene.index.ReadersAndUpdates.getReader(ReadersAndUpdates.java:208) ~[lucene-core-7.1.0.jar:7.1.0 84c90ad2c0218156c840e19a64d72b8a38550659 - ubuntu - 2017-10-13 16:12:42]
at org.apache.lucene.index.ReadersAndUpdates.getReadOnlyClone(ReadersAndUpdates.java:258) ~[lucene-core-7.1.0.jar:7.1.0 84c90ad2c0218156c840e19a64d72b8a38550659 - ubuntu - 2017-10-13 16:12:42]
at org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:105) ~[lucene-core-7.1.0.jar:7.1.0 84c90ad2c0218156c840e19a64d72b8a38550659 - ubuntu - 2017-10-13 16:12:42]
at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:490) ~[lucene-core-7.1.0.jar:7.1.0 84c90ad2c0218156c840e19a64d72b8a38550659 - ubuntu - 2017-10-13 16:12:42]
at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:103) ~[lucene-core-7.1.0.jar:7.1.0 84c90ad2c0218156c840e19a64d72b8a38550659 - ubuntu - 2017-10-13 16:12:42]
at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:79) ~[lucene-core-7.1.0.jar:7.1.0 84c90ad2c0218156c840e19a64d72b8a38550659 - ubuntu - 2017-10-13 16:12:42]
at org.elasticsearch.index.engine.InternalEngine.createSearcherManager(InternalEngine.java:550) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:229) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:154) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:25) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:2140) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:2122) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
at org.elasticsearch.index.shard.IndexShard.internalPerformTranslogRecovery(IndexShard.java:1335) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
at org.elasticsearch.index.shard.IndexShard.performTranslogRecovery(IndexShard.java:1295) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:395) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromStore$0(StoreRecovery.java:94) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:292) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
at org.elasticsearch.index.shard.StoreRecovery.recoverFromStore(StoreRecovery.java:92) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
at org.elasticsearch.index.shard.IndexShard.recoverFromStore(IndexShard.java:1577) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$6(IndexShard.java:2015) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
... 4 more
[2019-06-17T16:36:09,163][WARN ][o.e.c.a.s.ShardStateAction] [node-1] [audit-net-udpflow-2019-05-24][0] received shard failed for shard id [[audit-net-udpflow-2019-05-24][0]], allocation id [MP_rxUh6Slu1ZGwpoGFtog], primary term [0], message [failed recovery], failure [RecoveryFailedException[[audit-net-udpflow-2019-05-24][0]: Recovery failed on {node-1}{bxqdAUPETm6og34d591R-Q}{93eABCNYTVioLPSVZlfbYA}{172.16.211.130}{172.16.211.130:9310}{rack=r1}]; nested: IndexShardRecoveryException[failed to recover from gateway]; nested: EngineCreationFailureException[failed to open reader on writer]; nested: IOException[Map failed: MMapIndexInput(path="/TongYou/hbdata/data/nodes/0/indices/aNIh4sZLQ8WjmVCqeitB8w/0/index/_1in2.fdt") [this may be caused by lack of enough unfragmented virtual address space or too restrictive virtual memory limits enforced by the operating system, preventing us to map a chunk of 753581538 bytes. Please review 'ulimit -v', 'ulimit -m' (both should return 'unlimited'), and 'sysctl vm.max_map_count'. More information: http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html]]; ]
org.elasticsearch.indices.recovery.RecoveryFailedException: [audit-net-udpflow-2019-05-24][0]: Recovery failed on {node-1}{bxqdAUPETm6og34d591R-Q}{93eABCNYTVioLPSVZlfbYA}{172.16.211.130}{172.16.211.130:9310}{rack=r1}
at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$6(IndexShard.java:2019) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:568) [elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_151]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_151]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_151]
Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed to recover from gateway
at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:402) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromStore$0(StoreRecovery.java:94) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:292) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
at org.elasticsearch.index.shard.StoreRecovery.recoverFromStore(StoreRecovery.java:92) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
at org.elasticsearch.index.shard.IndexShard.recoverFromStore(IndexShard.java:1577) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$6(IndexShard.java:2015) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
... 4 more
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: failed to open reader on writer
at org.elasticsearch.index.engine.InternalEngine.createSearcherManager(InternalEngine.java:564) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:229) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:154) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:25) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:2140) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:2122) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
at org.elasticsearch.index.shard.IndexShard.internalPerformTranslogRecovery(IndexShard.java:1335) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
at org.elasticsearch.index.shard.IndexShard.performTranslogRecovery(IndexShard.java:1295) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:395) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromStore$0(StoreRecovery.java:94) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:292) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
at org.elasticsearch.index.shard.StoreRecovery.recoverFromStore(StoreRecovery.java:92) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
at org.elasticsearch.index.shard.IndexShard.recoverFromStore(IndexShard.java:1577) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$6(IndexShard.java:2015) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
... 4 more
Caused by: java.io.IOException: Map failed: MMapIndexInput(path="/TongYou/hbdata/data/nodes/0/indices/aNIh4sZLQ8WjmVCqeitB8w/0/index/_1in2.fdt") [this may be caused by lack of enough unfragmented virtual address space or too restrictive virtual memory limits enforced by the operating system, preventing us to map a chunk of 753581538 bytes. Please review 'ulimit -v', 'ulimit -m' (both should return 'unlimited'), and 'sysctl vm.max_map_count'. More information: http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html]
at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:940) ~[?:?]
at org.apache.lucene.store.MMapDirectory.map(MMapDirectory.java:267) ~[lucene-core-7.1.0.jar:7.1.0 84c90ad2c0218156c840e19a64d72b8a38550659 - ubuntu - 2017-10-13 16:12:42]
at org.apache.lucene.store.MMapDirectory.openInput(MMapDirectory.java:242) ~[lucene-core-7.1.0.jar:7.1.0 84c90ad2c0218156c840e19a64d72b8a38550659 - ubuntu - 2017-10-13 16:12:42]
at org.apache.lucene.store.FilterDirectory.openInput(FilterDirectory.java:99) ~[lucene-core-7.1.0.jar:7.1.0 84c90ad2c0218156c840e19a64d72b8a38550659 - ubuntu - 2017-10-13 16:12:42]
at org.apache.lucene.codecs.compressing.CompressingStoredFieldsReader.<init>(CompressingStoredFieldsReader.java:150) ~[lucene-core-7.1.0.jar:7.1.0 84c90ad2c0218156c840e19a64d72b8a38550659 - ubuntu - 2017-10-13 16:12:42]
at org.apache.lucene.codecs.compressing.CompressingStoredFieldsFormat.fieldsReader(CompressingStoredFieldsFormat.java:121) ~[lucene-core-7.1.0.jar:7.1.0 84c90ad2c0218156c840e19a64d72b8a38550659 - ubuntu - 2017-10-13 16:12:42]
at org.apache.lucene.codecs.lucene50.Lucene50StoredFieldsFormat.fieldsReader(Lucene50StoredFieldsFormat.java:173) ~[lucene-core-7.1.0.jar:7.1.0 84c90ad2c0218156c840e19a64d72b8a38550659 - ubuntu - 2017-10-13 16:12:42]
at org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentCoreReaders.java:125) ~[lucene-core-7.1.0.jar:7.1.0 84c90ad2c0218156c840e19a64d72b8a38550659 - ubuntu - 2017-10-13 16:12:42]
at org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:78) ~[lucene-core-7.1.0.jar:7.1.0 84c90ad2c0218156c840e19a64d72b8a38550659 - ubuntu - 2017-10-13 16:12:42]
at org.apache.lucene.index.ReadersAndUpdates.getReader(ReadersAndUpdates.java:208) ~[lucene-core-7.1.0.jar:7.1.0 84c90ad2c0218156c840e19a64d72b8a38550659 - ubuntu - 2017-10-13 16:12:42]
at org.apache.lucene.index.ReadersAndUpdates.getReadOnlyClone(ReadersAndUpdates.java:258) ~[lucene-core-7.1.0.jar:7.1.0 84c90ad2c0218156c840e19a64d72b8a38550659 - ubuntu - 2017-10-13 16:12:42]
at org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:105) ~[lucene-core-7.1.0.jar:7.1.0 84c90ad2c0218156c840e19a64d72b8a38550659 - ubuntu - 2017-10-13 16:12:42]
at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:490) ~[lucene-core-7.1.0.jar:7.1.0 84c90ad2c0218156c840e19a64d72b8a38550659 - ubuntu - 2017-10-13 16:12:42]
at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:103) ~[lucene-core-7.1.0.jar:7.1.0 84c90ad2c0218156c840e19a64d72b8a38550659 - ubuntu - 2017-10-13 16:12:42]
at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:79) ~[lucene-core-7.1.0.jar:7.1.0 84c90ad2c0218156c840e19a64d72b8a38550659 - ubuntu - 2017-10-13 16:12:42]
at org.elasticsearch.index.engine.InternalEngine.createSearcherManager(InternalEngine.java:550) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:229) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:154) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:25) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:2140) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:2122) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
at org.elasticsearch.index.shard.IndexShard.internalPerformTranslogRecovery(IndexShard.java:1335) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
at org.elasticsearch.index.shard.IndexShard.performTranslogRecovery(IndexShard.java:1295) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:395) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromStore$0(StoreRecovery.java:94) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:292) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
at org.elasticsearch.index.shard.StoreRecovery.recoverFromStore(StoreRecovery.java:92) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
at org.elasticsearch.index.shard.IndexShard.recoverFromStore(IndexShard.java:1577) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$6(IndexShard.java:2015) ~[elasticsearch-6.1.2-SNAPSHOT.jar:6.1.2-SNAPSHOT]
... 4 more解决如下：
1. ulimit -v 和 ulimit -m 返回必须为 unlimited。
2. vm.max_map_count 的 值，要比上面报错信息中大一点。
查看（root用户）：sysctl -a|grep vm.max_map_count
临时设置：sysctl -w vm.max_map_count=xxxxx
永久设置：
vi /etc/sysctl.conf
文件最后添加一行
vm.max_map_count=262144
3.jvm 调优。
参考：