The incident

Zookeeper instances failed to cluster during maintenance. Quorum Lost

Even though Zookeper tasks were restarted separately and 2 zookeeper tasks were kept up all the time, during maintenance the current setup only allows for the failure of 1 tasks without loosing the quorum. however a second task entered in an unhealthy while one was already down for update, the zookeeper quorum was lost and SOLR tasks stopped being able to communicate with zookeeper.

Even though the error log was completely new to us, the issue was quickly detected due to better logging and monitoring systems implemented during the maintenance.

Mitigation steps:

• Zookeeper tasks restarted.
• Monitored systems with errors, and scaled solrs to 0 tasks
• Slowly scaled them back up and monitored logs for errors.
• Implemented new log based alarms for newly detected error log
• Improved internal documentation to better act on this kind of errors and avoid loosing quorum.
• Re-index triggered on all systems

How to avoid this in the future:

• Scale Zookeeper to 7 Tasks, this will allow failure of 3 tasks without loosing the quorum (Ticket 208166).
• Update Zookeeper to newer version