Captain's Log: Stardate 79711.6

The 0.9.30 release is finally out (download, release notes), and the two biggest features are post-processing GPU shaders and 1-click screen recording.

Post-processing Shaders

The post-processing shaders have been on my TODO list since almost the very beginning. Originally the 3D renderer was a hand-rolled OpenGL implementation, and I wrote all the shader code by hand. Along the way I realized it would be cool if the shaders could be customized, especially if users could even provide their own shaders at runtime. But it was always a “nice-to-have” feature, and more important things pushed it down the TODO list.

Anukari is getting close enough to a 1.0 release that I am getting around to adding these kinds of features. And of course another big factor is the huge productivity gains I am getting from using agents (Claude).

One way that Claude is incredibly useful is that I can take an item from my huge TODO list, give a brief description of the feature to Claude, and just see what happens. If Claude fails miserably, I can simply move on to the next TODO. But sometimes it just knocks it out of the park, and with some cleanup and testing, Anukari has a new feature.

The post-processing shaders were pretty straightforward to implement, and Claude made things go really fast. Basically all of the pieces were already in place: the new Filament 3D renderer supports off-screen textures and custom shaders, and I already built the machinery for loading custom 3D assets for the skyboxes and 3D models. So it was basically just some work to glue everything together.

The biggest chunk of work was adding more tests. The 3D renderer was never super well-tested. Visual things like this are tricky to test; you can do “goldens” -style tests where you make sure that a screenshot matches an expected one, but unit tests are still a challenge in many cases.

My main concern was less about whether the graphics looked right (though that is important), and more about whether it was possible to crash Anukari with the new shader feature. It adds complexity, some new failure modes, etc.

I ended up writing a fuzz test suite for the renderer. It creates a real visible OS window, attaches the renderer, and then just goes absolutely nuts on it. It does thousands of iterations of randomly mutating the Anukari preset, tearing down and restarting the renderer, changing the window size, changing the skin, the skybox, the shaders, hiding the window, reattaching the renderer to a new window and so on. It doesn’t validate anything except that the renderer continues to produce frames, and importantly, that it doesn’t crash.

This fuzz test has already paid off! It found not only crashes in the new shader code, but also multiple existing crashes in the underlying 3D renderer, which I helped to track down and fix (example1, example2, example3).

Built-in screen recording

On the other hand, the screen recording feature was absolutely not in the “Claude just goes off and implements it” category. Far from it. That’s not to say that Claude was not immensely helpful. But it required a huge amount of guidance.

All things said and done, the code for screen recording is about 6,500 lines of C++. Roughly speaking, it’s one-third Windows-specific code, one-third macOS-specific code, and one-third test code.

I had no idea how complex this feature was going to be! I always assumed it would be fairly annoying, due to the fact that screen recording is inherently platform-specific, but it turned out to be a gargantuan pain in the butt. Here is an incomplete sampling of some of the things that made the screen capture feature complicated:

• Separate Windows, macOS API code
• Windows 10 and macOS 13 have their own legacy screen capture bugs
• Standalone and plugin mode
• In-process and, out-of process plugins, sandboxing
• Screen capture permissions
• Dependent on DAW setup for native window (I test in ~15 DAWs)
• DAW output on windows might be ASIO
• Monitor DPI, which DAWs handle in various wacky ways
• MP4/AAC encoding is platform-specific (Anukari is statically linked so no LGPL libraries)
• Synchronizing audio and video
• Threading synchronization, OS capture threads

There are just so many dimensions to the problem that result in special cases or branches in the code, it is inherently very messy.

If I were writing all this code without Claude’s help, I would have gotten a couple days into the project and realized that the cost/value trade-off was not going to be worthwhile. I probably would have simplified the problem: screen recording would only be available for the standalone app, and only on the very newest OS versions. That would still be pretty useful, and probably 20% as complex.

But… I got into a groove with Claude where I’d be off testing the screen capture feature on one of my test machines (or test VMs) while Claude was investigating the last bug/crash I found. When Claude had results, I’d come back and review the code, give Claude feedback, and continue work on manual testing.

Not all of Claude’s fixes worked, but when they didn’t work I would explain what happened to Claude and have it try something else. Eventually I gave Claude a prompt like, “please add an insane amount of verbose logging about everything that we possibly might care about.” This sped up the feedback loop considerably, because when I would find an issue, there was potentially enough info in the logs that Claude could figure it out on the first try, without multiple rounds of testing.

Along the way, when I didn’t have any manual testing to do, and was waiting on Claude to debug the latest issue, I began work on automated testing. I had Claude write a sophisticated integration testing framework for the screen recorder.

The integration testing involves creating a bunch of real OS windows. The windows are pink, except the corners which have small boxes drawn in four very distinct colors. And the process owning the window plays back a sine wave. The integration tests exercise the screen recording machinery on those windows, open the resulting MP4 file, decode it, and check that it looks and sounds right.

These integration tests are the only way that I felt even remotely comfortable that maybe the whole insane project was going to work. I set up all the wacky scenarios that the code had to handle, like having the audio come from a different process than the one that owns the window, which is itself in a different process from the screen capture code. In this case the capture code has to walk the process hierarchy to the root and do a search back down the hierarchy using heuristics to determine which process owns the audio subsystem, the plugin window, etc. It is absolute madness, but with Claude’s help I was able to set up tests that actually create this process structure in various ways and prove that the capture code works!

Ultimately getting this feature to work took around 400-500 Claude chat prompts, and many hours of testing Anukari on my fleet of 10 or so test machines. Currently I am not aware of any situation where it doesn’t work. Knock on wood, though… and if you find a scenario with Anukari where the screen capture feature doesn’t work correctly, please let me know.

The test fleet

I’ve made a habit of whenever a user reports a crash that I deem hardware-specific, that I find the cheapest version of that hardware on eBay and buy it. This has resulted in a test fleet that gives me coverage across a bunch of graphics chips for Windows: Intel UHD, Intel Iris, AMD Radeon Mobile, AMD Radeon Desktop, NVIDIA. I also have macOS coverage for a 10 year old Intel Mac, an M1, and an M4. I use a VM on my Windows desktop to test against Windows 10.

I call this test fleet the “stack-o-laptops.”

My cat Marty helps with the testing.

In addition to compatibility testing, it’s also great for performance testing. I got most of these laptops used for $100-$200, so they are not exactly spring chickens. Most of them are the kind of machines where after opening the lid, you get to wait a while for them to wake up, rub their eyes, yawn, and remember that they’re supposed to show a login screen.

This is fairly annoying when I just want to quickly test compatibility, but overall it’s valuable to run Anukari on this kind of hardware to make sure that it still has reasonable performance. Anukari is solidly usable on even the worst of my test hardware, though things do get questionable with some of the most complex presets. Another challenge is that if the OS starts doing something like installing an update, everything grinds to a halt. But there’s not much I can do about that.

Captain's Log: Stardate 79584.2

Some NAMM Ramifications, or NAMMifications

I've been quite busy (in a good way!) the last couple of weeks, with some of the connections that I made at NAMM bearing fruit in various ways.

One thing I've been working on is some setup for collaborating on a demo video with a popular MPE controller that I won't yet name. This is super exciting since the video will have broad reach, so I am working hard to iron out any remaining wrinkles in Anukari's MPE implementation. And I'm really stoked that a sound designer whom I met at NAMM, Jake Siders is cooking up some amazing MPE presets for the demo.

Also I have engaged the help of a marketing expert, Chris Hayzel. We met at NAMM and immediately hit it off. Anukari is a kind of weird product, and I have some extremely strong opinions on how I want the marketing to look. A huge thing for me is doing marketing in a way that I feel good about, so no used car salesman stuff, no misleading hype, no lies, basically. But also, hey, I think Anukari is pretty cool and I'd like to tell more people about it, and explain it in a way that's fun to learn about and not a bunch of dry technical talk (my forte). So it was important to me to find someone who I felt I could trust with this vision. This is in the early stages, but it's likely that this website is going to undergo a significant metamorphosis in the coming months. I'm excited.

Polishing, polishing, polishing

A lot of people at NAMM were surprised that Anukari is still in Beta, given that it seems like a completely functional product. I think it's almost ready to be called 1.0, but there remain a few small things I want to polish first.

First, there were still a couple of crashes/hangs that I was aware of from user reports.

One hang only happened in Apple's AU validation, only sometimes. This turned out to be a weird issue where if you pass too long of a work cycle time to Apple's Core Audio workgroup feature, the thread won't be created. JUCE has some comments to this effect, and has a supposed workaround, but it didn't work for me. I implemented my own workaround and it fixed the hang. This issue appeared when the AU validation set the plugin up with a huge block size (4096 samples), so the estimated time to process the block went above 50ms, which appears to be the limit where Core Audio loses its mind.

For the crash report, I only really knew that it happened on AMD hardware. So of course I prayed that I could find a way to reproduce it on my AMD test machines, and thankfully I was able to. The biggest hint was that my user said that originally Anukari worked perfectly, but later it crashed on startup. I guessed that they had changed a graphics setting, which was persisted to the preferences file before causing a crash. So then it crashed forever. I had the user delete their preferences file and things worked again! So I went through all the graphics settings and toggled everything until I found that the Reflections feature crashed. I tried to find the root cause, but the Vulkan code involved is so complex that I gave up and filed a bug with the Google Filament folks. Here's hoping they can help.

One improvement I made based on this crash report is that Anukari now waits a couple seconds before persisting preferences changes to disk. This way, if a change crashes Anukari, hopefully it won't be persisted, and Anukari will work when it's restarted.

Some of the other polish work includes audio quality improvements, such as adding a bit of smoothing for LFO retriggers to avoid pops.

MTS-ESP Support

As of the 0.9.25 testing release, Anukari is fully compatible with the MTS-ESP protocol for microtuning using an MTS-ESP control plugin such as (MTS-ESP Mini](https://oddsound.com/mtsespmini.php). This protocol lets users tune all their MTS-ESP-compatible plugins to whatever scale they want, in sync, so they all play in tune. The control plugin can import .scl files, so there are thousands of tunings to play with. People have been asking me for this support for a long time.

Given that I've been busy with NAMM follow-ups and polishing Anukari into a 1.0-ready product one might wonder how I fit in the work for MTS-ESP. I am also surprised! Basically what happened is I ended up with a few hours free where I didn't have enough time to really start a new big task, and was finished with my last big task. And right at that moment I got another email asking for MTS-ESP support. So I just said screw it, let's just see if I can bang that out this afternoon and that way I can say "yes" to the email.

The MTS-ESP site says you can integrate the library in an hour. I was skeptical, but it's mostly true. I had things working at a basic level in less than an hour. I did spend several more hours after that making it work really well, ensuring some of the optional features worked, and writing tests and GUI stuff to make it all professional.

So far users are reporting back that it works great. I tested it myself, and despite not being a big microtuning geek, it was surprisingly fun to try out different scales. I'm glad to have been able to sneak in this feature!

Captain's Log: Stardate 79052

The problem

Anukari has long had a modulation system, with LFOs, host automation controllers, MIDI, etc. But adding modulation to a preset has always been a kind of labor-intensive process. And one big gaping hole in the UX was the lack of a way to bind a knob inside Anukari itself to allow modulation to be controlled via the mouse.

The lack of mouse control was a hassle, but the problems were a bit deeper than that. Because of this issue, interfacing with the DAW was always not quite the way users expected. For example, in FL Studio you can choose an automation via the learn feature by "wiggling" a knob inside a VST plugin. FL Studio watches and sees which knob was wiggled and binds it. But of course with no mouse-controlled knobs inside Anukari, this was not possible.

Furthermore, while it was possible to map host parameters to automations inside Anukari, they could only be controlled via the DAW, which is really inconvenient, and often is a really weird workflow. Users expect to be able to hit "record" in the DAW and then go play the VST, knobs and all, and have everything recorded.

Macros

The solution was to add some knobs inside Anukari that can be mapped to control the modulation system. Those are shown here in the lower right-hand corner:

(The icons and graphics are still provisional while I wait for my designer to improve them.)

There are eight knobs in total (in the screenshot only four are showing, and the other 4 are collapsed). Each knob can be renamed, and corresponds to a mapping that will automatically appear in the DAW. And each knob is connected to any number of 3D Macro objects, which it will control.

This is already really handy, but the killer feature is the little grabby-hand button icon next to each macro knob. When the user drags this, parameters in the right-hand editor panel that can be modulated will automatically be highlighted, and when the user drops onto one of them, a 3D Macro object will be created which is automatically linked to the given parameter on all selected entities. Here's an example:

This is a pretty transformative change. It is dramatically easier to create automations inside Anukari, play with them, and edit them. And then they can be performed with the knob movements recorded in the DAW.

Side benefits

The new macro system addressed a bunch of feedback I repeatedly got from users, and solved a bunch of problems. But in addition to that, there were a number of extra advantages to the new system that came very cheaply.

Having the drag-and-drop system for modulation naturally made it easy to do the same thing for other modulator types. So now, if a user drags in an LFO from the entity palette on the bottom of the screen, they can drag it straight to a highlighted parameter to create an LFO connected to that parameter on the selected entities. This can be done with any modulator and is hugely convenient.

Another big benefit is that now all the built-in automations for all the factory presets are discoverable. Previously with no knobs in the main UI, there was no easy way to see what parameters had been configured for modulation as you cycled through presets. Now you can see them all, and trivially play with them via the mouse. Even better, in the standalone app the 8 macro knobs map to MIDI continuous control parameters 1-8, so on most MIDI controllers you can just turn knobs and things will happen, with visual feedback.

Finally this opens the door for even more interesting drag-and-drop use cases. The first one I have in mind is for creating exciter objects, like Mallets. The idea is that the user will be able to select a bunch of Bodies (masses), and then drag the Mallet object from the palette onto the right-panel (which will be highlighted) and it will automatically create the Mallet and connect it to all the selected Bodies. This will be much more convenient than the workflow today.

Anukari Builder (preset API)

In the Anukari Discord server, the user 312ears is notable for providing extremely helpful feedback about Anukari, obviously borne out of using it in depth. When I first released the Beta, they were one of the people who suggested that it would be cool if it were possible to create presets programmatically, for example via a Python API.

I really wanted to help with this, but for the foreseeable future my time has to be focused on making the plugin itself work better. So I offered to release the Google Protocol Buffer definitions for the preset file format and provide a bit of support on using them, but could not commit to any kind of nice APIs.

Anyway, 312ears took the Protocol Buffer definitions and built an entire Python API for building Anukari presets. Their API can be found here: github.com/312ears/anukaribuilder.

This is an absolutely incredible contribution to the project and community. It allows users who can write Python to create presets that would otherwise be far too tedious to make. On some hardware Anukari supports up to 1,000 physics objects, and arranging them in a complex geometric pattern is difficult with the UI. But with Python all kinds of things become possible. For example, 312ears has shown demos in the Discord server of presets that can shift between two shapes, say a sphere and a pyramid, by turning a MIDI knob. Here's a quick example: