Well, it was a huge amount of fairly tedious work, but I finally have pretty exhaustive coverage of all the physics features with golden tests. All said and done, there are 65 tests. For each test, I created a really simple preset, and either a MIDI file is played into it, or an audio input file, or both. And then the outputs are verified as not having changed from the golden version. I tried to keep each test focused narrowly on a specific feature, so that when I break things it will be easier to understand what part of the physics is broken.

This would have gone faster, except that while creating the tests I kept finding small bugs (and fixing them). Which is, obviously, a great thing. I was pretty happy to find that the bugs that I found were mostly relatively minor things. I'm starting to feel like with all this testing, I've eliminated the most obvious bugs, and what remains are the smaller bugs that don't turn up except in particular edge cases.

Overall, I now have 249 overall combined unit + golden test cases, and I think that at least the backend-type stuff (physics engine, data model, serialization, etc) is well-covered (though there are still a few minor things to test). Even the GUI has a bit of unit testing, but there's more I'd like to do there.

I'm guessing for the GUI, the most valuable automated testing I can do next will be the Chaos Monkey, which will randomly generate mouse/keyboard events. This isn't really meant to verify that the GUI works correctly, but is strictly to verify that it's not possible to crash things via user inputs. I plan to run it for long periods, e.g. overnight, and then when I wake up check in to see if (a) it crashed, or (b) the GUI got into some weird, broken state that it shouldn't have.