Degraded performance of DRACOON Cloud

Incident Report for DRACOON Cloud

Postmortem

We experienced an issue with the DRACOON cloud on 2023-11-10 09:27 - 10:24. Our team has worked diligently to identify the root cause and implement a resolution. In this post-mortem, we want to share the details of what happened, why it happened, what we did to resolve it, and what we will do to prevent similar incidents in the future.

What happened?
In order to fix an unrelated problem and improve performance a change has been implemented to some of our components, which resulted in 5xx errors and the experienced cloud outage.

Why did this happen?
The implemented change caused additional services to perform retries in case of an error. This resulted in some of the main components becoming overloaded and therefore resulting in 5xx errors.

What did we do?
We reverted the changes as soon as we realized there were problems.

What can we do to improve?
To prevent this from happening in the future, retries will only be enabled for our main components.

We apologize for any inconvenience this incident may have caused. We are committed to ensuring the stability and reliability of our services and will continue to take proactive measures to prevent similar incidents from happening in the future.

If you have any questions or concerns, please don't hesitate to reach out to our support team for assistance.

Posted Nov 29, 2023 - 12:55 CET

Resolved

The issue with our DRACOON Cloud has been fully resolved. All systems are now operating normally. We apologize for any inconvenience this may have caused and appreciate your patience. If you continue to experience any issues, please don't hesitate to reach out to our support team for assistance.

Posted Nov 10, 2023 - 13:03 CET

Monitoring

The issue with our DRACOON Cloud has been resolved, and we are monitoring the situation to ensure it remains stable. We apologize for any inconvenience this may have caused and appreciate your patience.

Posted Nov 10, 2023 - 10:24 CET

Identified

We have identified the issue with our DRACOON Cloud services. Our engineers are working on a resolution and we are confident that we will have a fix in place soon. We will provide more information and updates as soon as they become available.

Posted Nov 10, 2023 - 09:41 CET

Investigating

We are currently investigating an issue with our DRACOON Cloud. Our team is working to gather more information and resolve the issue as quickly as possible. We apologize for any inconvenience this may cause and will provide updates as soon as we have them.

Posted Nov 10, 2023 - 09:27 CET

This incident affected: API (API (group 01), API (group 02), API (group 03), API (group 04), API (group 05), API (group 06), API (group 07), API (group 08), API (group 09)) and Web App.