Issue: VidiCore Starter Edition failing to create or resume.
Timeline:
- The issue was detected before it started affecting any customer systems.
- Investigation revealed that the issue was related to an incompatibility between the Terraform provider version and the AWS API when creating the OpenSearch backend.
- As we keep up to date with Terraform development and the latest Terraform provider version, we quickly tested reverting the provider version to an earlier one, but it didn't work.
- To fix this, we updated our Terraform modules to handle this particular case of the OpenSearch backend creation.
- We thoroughly tested the solution before deploying it to production.
- A bug report was opened on the AWS Terraform provider repository to obtain further evaluation from the module developers and implement the necessary fixes.
Root Cause: The root cause of the issue was an incompatibility between the Terraform provider version and the AWS API when creating the OpenSearch backend.
Resolution: To resolve the issue, we updated our Terraform modules to handle the particular case of the OpenSearch backend creation. We also opened a bug report on the AWS Terraform provider repository to obtain further evaluation from the module developers and implement the necessary fixes. Thorough testing was performed before deploying the fix to production.
Lessons Learned:
- Continuously improve monitoring systems to detect issues faster.
- Continuously improve pipelines to reduce testing and deployment times.
- Keep track of bug reports and contribute to open-source communities to help identify and fix issues.