Good question! It's been one of my pain points in Silvesrstripe for a while. As it is now, DOM is not ready to work with ModelAdmin, but it will be. There's a ton of stuff in the works that's really going to make some drastic improvements to ModelAdmin, including the inclusion of DOM. Stay tuned.
Woah I went to go use DOM in ModelAdmin today only to find that it doesn't work :( weird that it was the same day that this was posted... But yea I'm just posting to support your ModelAdmin efforts - would be great to have it working! :)