We experienced an issue with the Branding Service on 2024-01-25 09:00 - 10:15. Our team has worked diligently to identify the root cause and implement a resolution. In this post-mortem, we want to share the details of what happened, why it happened, what we did to resolve it, and what we will do to prevent similar incidents in the future.

What happened?
During the time of the incident, the default branding was displayed on the login page for some requests instead of the configured custom branding.

Why did this happen?
The underlying issue with the incorrect branding being displayed was a timing issue introduced with a recent update of our authentication service. Because the issue was an edge case, not all running instances of the authentication service were affected and therefore not every response contained the incorrect branding.

What did we do?
We applied a temporary workaround to fix the issue immediately and will permanently fix the issue in the next version of our authentication service.

What can we do to improve?
We will further analyze the root cause of the underlying issue and improve our automated testing procedures to prevent similar incidents from happening in the future.

We apologize for any inconvenience this incident may have caused. We are committed to ensuring the stability and reliability of our services and will continue to take proactive measures to prevent similar incidents from happening in the future.
If you have any questions or concerns, please don't hesitate to reach out to our support team for assistance.