Jobs are backed up

resolved

This incident has been resolved.

5/7/2026, 10:51:28 PM

monitoring

The fiber cut has been identified and the providers are still working on the repair. We have been able to secure additional capacity to support regular operations and will be removing the concurrency limits. There may be additional interruptions until we get the main capacity back online.

5/7/2026, 10:50:18 PM

monitoring

Upstream outage affecting our servers. Intermittent delays and job limits in place while we await recovery. The network cut location has been identified and repairs are underway.

5/7/2026, 8:38:11 PM

monitoring

Upstream outage affecting our servers. Intermittent delays and job limits in place while we await recovery. Some partial fixes for the network cut are in place by the upstream providers.

5/7/2026, 5:27:42 PM

monitoring

Upstream outage affecting our servers. Intermittent delays and job limits in place while we await recovery.

5/7/2026, 2:35:22 PM

monitoring

Upstream outage affecting our servers. Intermittent delays and job limits in place while we await recovery.

5/7/2026, 1:35:24 PM

monitoring

v8 and v8.1 capacity has been partially restored. Concurrency limits are temporarily reduced while we work through the backlog, and some relax queues are still catching up.

5/7/2026, 10:17:06 AM

identified

A weather-related networking outage with an upstream provider has caused a significant reduction in available v8 and v8.1 servers. ETA up to 24 hours.

5/7/2026, 8:51:19 AM

identified

The issue has been identified and we're working on a fix

5/7/2026, 8:50:03 AM

identified

The issue has been identified and we're working on a fix.

5/7/2026, 8:49:09 AM

investigating

We're investigating long queues for generating v8 images

5/7/2026, 8:02:45 AM