Today I troubleshooted an Elasticsearch-cluster-down issue.

Several lessons were learned:

1. When many elasticsearch cluster nodes are restarted, to avoid HEAP spike, better to temporarily stop all connection attempts;
2. Avoid setting allow_primary=true when reroute shards via API;
3. Don’t forget backup! It could save you some day!

View Details