Services are expected to be operating normally again. We are closely monitoring the situation to confirm that Azure’s fix has been fully applied.

Main API recovered.

Sentinel Web App recovered.

The service is healthy again. As a remediation measure, we are initiating a migration to additional search nodes to further improve resilience and prevent similar issues going forward.
Cluster scaling is still running in the background, and we continue to monitor closely until the process is fully completed.

Media Service recovered.

Web App recovered.

Public Web Component Gate recovered.

Knowledge Bots User Docs and Knowledge Bots API Docs went down.
Our external Docs provider, Gitbook, is using Cloudflare and their Dashboard APIs.
Cloudflare is currently facing issues with said APIs.

For more info, see
- https://www.gitbookstatus.com/
- https://www.cloudflarestatus.com/

Post Mortem - Database Cluster Overload

What happened

Blockbrain experienced a temporary outage affecting our production database cluster.
The primary node of the cluster suffered a CPU overload, which led to partial unavailability and degraded performance for some customers.
The overload occurred while we were running a data ownership transfer script.
This script executed queries that read an entire collection with millions of records without proper indexing, causing excessive load on ...

Agent Service recovered.