Since adding the Filament renderer, I noted that closing the editor GUI window when running as a VST3 plugin in Ableton caused Ableton to crash. I figured this was just due to my sloppy prototype code not cleaning up all the Filament resources correctly when the Editor was shut down. Today I decided to dig into that, and it turned out to be much trickier to diagnose and fix than I had expected.

After attaching a debugger to Ableton and reproducing the crash, I found that there was an invalid memory read deep in the bowels of the Filament Vulkan code, happening while it was attempting to create a new swap chain. This was immediately very confusing, because one might imagine that during shutdown you are not trying to create new swap chains.

After reading a bunch of Filament's code, I finally figured out that the new swap chain was being created in response to the window size changing. Eventually I added some logging and figured out that Ableton was closing the native OS window for the plugin before destroying the plugin Editor component. So Vulkan was exploding because it was not being informed about an impending window closure until after said window was closed.

The fix ended up being pretty simple: I just hooked into the native OS window handling and watched for WM_DESTROY, and changed it to call the renderer shutdown when that message appears. This allows the renderer to shut down before the native window is gone, and now everything works without crashing.