A100 hardware partial outage

Updates

• resolved

  We've been below capacity for A100 hardware for the past 30 minutes and all queues have been drained. Thanks for your patience 💖

  Aug 27, 2026, 12:25 AM UTC

• monitoring

  We are seeing pods start up again now and they are working through the backlog.

  Aug 26, 2026, 11:07 PM UTC

• investigating

  We're seeing most A100 hardware automatically cordoned due to possible hardware failure.

  Aug 26, 2026, 9:17 PM UTC