Just about everything is modifiable in matplotlib… It may not be easy, but all plotting libraries are designed to make some things easy at the expense of making other tasks more difficult. For matplotlib you just have to think about things the way matlab thinks about things… which is more computer graphics based. It can get ugly until you understand it. But if you understand how any plotting library actually works it’s not that bad. All plotting libraries ultimately are built on graphical primitives like lines and fonts and triangles and patches computing where things belong by transforming coordinates and feeding them to a layout engine. It’s not as magical as the APIs make them seem. So if you’re willing to dig into their bowels (as OP mentions) there really aren’t any many limits. Sometimes it’s actually easiest to just declare a canvas in memory and draw it all by hand. Ultimately, things are either vector or raster formats (or some abstraction that supports both) and fed into some computer graphics engine (like postscript or some OS’s or GPU canvas).
Anyway, sometimes the easiest answer is you export and edit the labels in the final figure. One really nasty way if you don’t have PS or PDF tools is to sidetrack through Windows EMF and mess with fonts and positioning of text in PowerPoint.
I’m not familiar with code.golf but I wonder how whitespace is handled? I find python is very concise anyway, but I wonder how the white space is counted (single tab, four spaces for black, etc).