They contained the impact. Root causing or “understanding” should come after impact mitigation. If needed find a safe way to reproduce the bug without customer impact.
We reverted the refactoring, deployed, un-banned the CEO, and set about analysis.
Relevant XKCD
It’s one backed by a lot of data. One example is from the Android project.
The percent of vulnerabilities caused by memory safety issues continues to correlate closely with the development language that’s used for new code. Memory safety issues, which accounted for 76% of Android vulnerabilities in 2019, and are currently 24% in 2024, well below the 70% industry norm, and continuing to drop.
https://security.googleblog.com/2024/09/eliminating-memory-safety-vulnerabilities-Android.html
There’s an argument that critical infrastructure software vendors are already meeting standards for basic, non-memory related items. Yes, there are other categories, but memory safety is one that’s harder to verify. Moving to memory safe languages is an ensure a category of correctness. This excludes usage of unsafe escape hatches.
Sorry for not having a direct answer for your question.
First, are these videos using Dolby Vision? If so try a google search for “Dolby Vision green tint”.
Second, if this really is a corruption issue, are you sure you really want to write something custom to fix this? What else is wrong in those files that you haven’t noticed yet? It seems unlikely that they would all corrupt consistently. Did something transform them since they were last in a known good state? Do you have other means to revert that transformation?
Finally, if you do want to continue down your current path there’s a light discussion on the topic in this stack overflow thread. You’re going to need to contend with how videos are compressed, i.e. not every frame is actually stored. You’re likely going to loose some quality or bloat the file size through applying a correction process. If you find the right tools and want to minimize loss and bloat, you may need to use different solutions for different codecs.
I’m going to whoosh the joke for a moment.
Avoid allocating memory in sig segv handlers. I worked at a company once that had written their own handlers that tried to print a stack trace. I was fortunate enough to find a machine in a test environment that segfaulted while allocating memory. The handler then caused a deadlock in a call to malloc (through new). I ended up rewriting the handler to fix two sources of memory allocation. First I preallocate enough space to do string manipulations in the handler. Second I made eager calls to a few libc functions used in the handler to avoid memory allocation from lazy binding from ld.so.