“Unfortunately while I had tests for this, I never tested regenerating the server state after a machine is “re-owned” while having data stored”
One EC2 instance in us-east-1 died and Tarsnap was down for 26 hours and 16 minutes. The rebuild tripped over a 2014 bug in moving machines between accounts, then over a resume step that left a sequence number at zero. The recovery was set to 250 parallel S3 requests because that was the limit a decade ago. Disaster recovery code is the code nobody runs until the disaster.