The incident

OPS Engineer failed to notice one of zookeeper tasks was not healthy before conducting maintenance on Aug 15th

Even though Zookeper tasks were restarted separately and 2 zookeeper tasks were kept up all the time, the current setup only allows for the failure of 1 tasks without loosing the quorum. By restarting one of the healthy tasks the quorum was lost and SOLR tasks stopped being able to communicate with zookeeper.

There were no alarms related with this kind of issues which delayed its detection.

Mitigation steps:

• Zookeeper tasks restarted.
• Monitored systems with errors, and scaled solrs to 0 tasks
• Slowly scaled them back up and monitored logs for errors.
• Implemented new log based alarms

How to avoid this in the future:

• Scale Zookeeper to 7 Tasks, this will allow failure of 3 tasks without loosing the quorum (Ticket 208166).
• Update Zookeeper to newer version