replicate · 60d ago
Delayed scaling due to node failure
ResolvedYes — replicate had a problem on August 10, 2026, and replicate has since marked it resolved. replicate's latest word: “All affected models have been scaling correctly for more than 30 minutes at this point, and we see no residual prediction queues. Thank you for your patience!”
Timeline
Started: 8/10/2026, 9:05:30 PM
Last update: 8/10/2026, 9:45:44 PM
Resolved: 8/10/2026, 9:45:44 PM
Updates
resolved
All affected models have been scaling correctly for more than 30 minutes at this point, and we see no residual prediction queues. Thank you for your patience!
8/10/2026, 9:45:44 PM
monitoring
Scaling decisions were delayed by nearly 1 hour after the controller responsible for emitting queue metrics failed to schedule on a soft-failed node. We have since cordoned and drained the node, and the controller is emitting queue metrics again.
8/10/2026, 9:05:30 PM
More replicate incidents
- A100 hardware partial outage — resolved, 44d ago
- Significant degradation — resolved, 66d ago
- API degraded for A100s — resolved, 68d ago
- Degraded scale-out due to failed setups pulling from huggingface — resolved, 70d ago
- Hitting GPU Capacity for H100s creating large queue times for some models — resolved, 80d ago
- HuggingFace download issues — resolved, 86d ago