Incident report - May 2026

On Tuesday May 12, Pitch was unusable for about an hour. From around 14:30 UTC, the majority of our users opening the app — web or desktop — were greeted with a ”something went wrong” screen instead of their workspace. We restored access for most web users by 15:20 UTC after a hotfix. A small follow-on bug — an intermittent canvas rendering issue that was workable around with a window resize — was cleaned up by a second hotfix later that evening, along with a final release to remove the offending integration entirely.

The cause was unusual enough that it’s worth explaining in some detail, because the lessons aren’t just about a single library, they’re also about how we allow third-party code to behave in our app.

What broke

The short version: we still had integration code for a vendor whose account we’d cancelled. When that vendor processed the cancellation, it changed what it served us and the new payload was missing a function our code calls. That single missing function threw an uncaught error, which prevented the rest of the app from loading.

The vendor in question is a product-tour and in-app messaging tool that Pitch used some time ago. We stopped using it and cancelled the subscription, but the integration code that loads their library from their CDN was never removed.

Their CDN, on detecting an expired account, doesn’t simply stop serving the library - it serves a different one. The replacement is a stub that defines a handful of method names (identify, start, track, and so on) so the host app doesn’t immediately crash on a missing global. Each method just logs a warning that the account has expired.

Our code happened to call a method called group, which isn’t included in that stub. So the moment the app tried to identify the current workspace to the vendor’s SDK, JavaScript threw a TypeError: a.group is not a function and the rest of our initialization never ran. Users on the web and desktop apps hit the error screen.

This was a change from an external source on a load-bearing code path.

Timeline

All times UTC.

• 14:30 - PagerDuty fires. Customer reports start arriving in parallel from our Customer Success team.
• 14:32 - Engineers identify the console error and start bisecting recent frontend builds, expecting to find a regression.
• 14:33 - Bisection finds nothing - every recent build shows the same error. The cause is external.
• 14:36 - We post the first status update on Hyperping (”investigating”).
• 14:39 - Sentry surfaces the smoking gun: a warning from the vendor that the account has expired. The team agrees the integration is the cause.
• 14:40 - We attempt an emergency mitigation: patch the runtime config served from S3 to disable the integration, then invalidate the CDN. The patched file produces a new error (a content-type mismatch) and we abandon the S3 route in favour of a proper patch and hotfix through CI.
• 15:11 - Hyperping update: ”identified.”
• 15:17 - Web users can reload (hard refresh) and use Pitch again.
• 15:32 - Hyperping marks the incident resolved for web. Desktop users are advised to update and restart.
• 16:36 - A second hotfix goes out for a small follow-on bug: one of the vendor’s tracking calls was still firing and could intermittently leave the editor canvas mis-rendered until you resized the window. Not a blocker, but worth cleaning up the same day.
• 18:07 - Final hotfix to remove the integration references entirely is merged and deployed.

An engineering post-mortem is scheduled for Monday, May 18.

What we’re changing

The post-mortem on Monday will produce the full list, but several things are already either underway or decided:

Remove the integration entirely. The PRs to strip out the remaining references are ready. There is no reason for code from a vendor we no longer use to be in our bundle.

Wrap third-party calls to make them non-fatal. An exception from a third-party analytics library should not be able to bring down our app. Any call into a vendor SDK should fail quietly, log, and let the rest of the app load. We’re going to audit our third-party integrations against this rule.

Treat removing a vendor as a real piece of work. Cancelling the subscription is not the same thing as removing the integration, and we treated it as if it were. Vendor offboarding needs a checklist that includes the code, the build config, the CSP, and confirmation that nothing in the running app still reaches out to the vendor’s domain. We’ll add this to an offboarding process.

A word to our customers

If you were on a customer call during the outage, or you were mid-pitch when the screen went blank, we’re sorry. That’s exactly the moment our software is meant to be invisible. Through the actions described above, we’ll make sure it doesn’t happen again.

- Phil Jackson, Engineering