• nebeker@programming.dev
    link
    fedilink
    English
    arrow-up
    14
    ·
    20 days ago

    To be clear, that was during recovery, not a root cause.

    The closest to that is this:

    Neither outage was caused by a code or configuration change. Both incidents were capacity failures at their core. We failed to scale critical components before demand exceeded their capacity. Since April, monthly commits have grown from 1.4 billion to 2.9 billion. That growth explains the pressure on our systems, but it does not excuse these outages.

    One could argue they still DDOSd themselves by promoting tools that produce commits at a much higher rate.

      • nebeker@programming.dev
        link
        fedilink
        English
        arrow-up
        2
        ·
        20 days ago

        Exactly. Imagine the board meeting where somebody has to suggest “maybe we don’t want that many more users.” I was about to type “customers,” but realistically a significant portion of this new load is on free plans.