ELK问题

基本信息
1. elasticsearch-5.2.2
2. 一天600G 存量30多T
3. 存储节点内存128G，分了32G给ES用
4. 配置明细
cluster.name: "prod-elasticsearch-cluster"

node.name: "node1"
node.master: false
node.data: true
node.ingest: true    # 是否为预处理节点

#index.number_of_shards: 5
#index.number_of_replicas: 1
path.data: /opt/C3982/data
path.logs: /opt/C3982/elasticsearch-5.2.2/logs

bootstrap.memory_lock: true
bootstrap.system_call_filter: false

network.host: 10.17.4.247
transport.tcp.port: 19300
http.port: 19200

discovery.zen.minimum_master_nodes: 2
discovery.zen.ping_timeout: 30s
#discovery.zen.ping.multicast.enabled: false
discovery.zen.ping.unicast.hosts: ["10.17.4.248:19300","10.17.4.249:19300","10.17.4.140:19300","10.17.4.141:19300","10.20.3.20:19300","10.20.3.21:19300","10.20.3.22:19300","10.20.3.23:19300","10.20.3.24:19300","10.20.3.25:19300"]
#discovery.zen.fd.connect_on_network_disconnect : true 
#discovery.zen.initial_ping_timeout : 10s 
#discovery.zen.fd.ping_interval : 2s 
#discovery.zen.fd.ping_retries  : 10 

cluster.routing.allocation.node_initial_primaries_recoveries: 8
cluster.routing.allocation.node_concurrent_recoveries: 4
cluster.routing.allocation.disk.threshold_enabled: true
indices.recovery.max_bytes_per_sec: 0

#index.cache.field.type: soft
#index.cache.field.max_size: 50000
#index.cache.field.expire: 10m

#cluster.routing.allocation.disk.watermark.low: 100GB
#cluster.routing.allocation.disk.watermark.high: 60GB

5.
问题：
1、现在主要就是因为各种原因，ES中老有存储节点会退出集群，然后整个就不能用了
报错日志：
[2017-08-02T11:55:00,425][WARN ][o.e.m.j.JvmGcMonitorService] [node1] [gc][old][17753][388] duration [11.4s], collections [1]/[11.5s], total [11.4s]/[3.5m], memory [31.8gb]->[31.8gb]/[31.8gb], all_pools {[young] [1.1gb]->[1.1gb]/[1.1gb]}{[survivor] [136.5mb]->[135.6mb]/[149.7mb]}{[old] [30.5gb]->[30.5gb]/[30.5gb]}