Degraded scale-out due to failed setups pulling from huggingface

Updates

• resolved

  We're looking stable now, and with caching back on. Thanks again for your patience!

  Jul 31, 2026, 11:39 PM UTC

• monitoring

  All affected hardware types have recovered and we are continuing to monitor and troubleshoot. Thanks for your patience!

  Jul 31, 2026, 10:27 PM UTC

• identified

  We have confirmed that turning off our caching layer for huggingface downloads allows models to successfully set up, so we are making that change for all models while we continue to troubleshoot.

  Jul 31, 2026, 9:26 PM UTC

• identified

  Models that pull from huggingface during setup have not been succeeding, which is resulting in scale-out delays and queue backups.

  Jul 31, 2026, 9:25 PM UTC