@reepblue pointed out this library for UV unwrapping:
https://github.com/jpcy/xatlas
I added these values to the AssetEditor class:
asseteditor.model: the open model, if it exists.
asseteditor:Invalidate(): Triggers a viewport redraw.
asseteditor:Modify(): Causes the file to be set as "modified".
asseteditor.modelmenu: Menu displayed when a model is loaded.
The event EVENT_INITASSETEDITOR is emitted when an asset editor window is initialized, so you can add a menu item to the modelmenu at this time.
Since Lua can call functions from a DLL, that should be everything needed to create an unwrapping extension.