Post-Mortem: Rich Text Performance Incident

Incident Duration:

• Start: January 8, 2025, at 10:45 UTC.
• End: January 8, 2025, at 16:11 UTC.

Impact:
Our services experienced ~5 hours of degraded performance. Posts, course lessons, and other rich text-based content were affected during the incident. Mentions (@) and entity links (#) added during this period broke upon saving, and existing embeds were temporarily non-functional.

Root Cause:
The incident was caused by a compatibility issue introduced by a major update to core libraries, specifically impacting how our services handle rich text content.

Resolution:
Upon identifying the root cause, our team rolled back the conflicting library to a stable version, which resolved the immediate issues. A script was executed to correct mentions, entity links, embeds, and polls affected during the incident.

Next Steps:
To mitigate the risk of similar incidents in the future, we are adopting the following measures:

1. Fixed library version: We have fixed the library version to prevent unintended upgrades in the future.
2. Enhanced Testing: We will expand our pre-deployment testing to include more rigorous checks for third-party library updates.

We sincerely apologize for the disruption and thank you for your patience and cooperation while we worked diligently to resolve the issue. If you have further questions or concerns, please contact our support team at support@circle.so.