SourceHut recently experienced a 170-hour outage due to a distributed denial-of-service (DDoS) attack. The attack targeted SourceHut’s primary datacenter in PHL and eventually led to an emergency migration to the AMS datacenter. The response from the upstream network in PHL caused more damage than the actual attack. The team worked diligently to restore services and bring SourceHut back online. They encountered various challenges, such as restoring individual services and migrating object storage. Despite the difficulties, SourceHut credits their paying users with seven days of free service and is focused on improving resilience and reliability for the future.
https://sourcehut.org/blog/2024-01-19-outage-post-mortem/