To get to my point I start with two examples:

EXAMPLE 1: I used DOM on a project where people wanted to insert internal links, links to documents and external links at the bottom of an article. I ended up making 3 different instances of the DOM for that. Even though this was right for the project (I could group different link types together), I still had the feeling this could also be done using 1 DOM + the option 'do you want to insert an internal link, an external one or a document link?'

EXAMPLE 2: I tried to get into TYPO3 a while ago (is pretty often used in Germany, dunno why) and there you have the option to add several content blocks to your content where you can choose between things like: 'Do you want an image to the left, with text to the right? Or do you want to insert an image to the right and the text flows around it? etc.'

I think these things could be done using the DOM if it would be possible to use different popup-layouts. And I think the only thing needed for that is the possibility to add your own CSS in the back-end (dom.css) + a way to choose the popup style. When choosing the popupstyle the only thing that needs to happen is that a css-class is set in the popup and the rest can be taken care of with your dom.css.

So - to use the first example - when you say 'I want to insert an internal link', the class .internalLink is added to the popup and in your dom.css you make sure only the SimpleTreeDropdownField is shown and the FileIFrameField or the text field where you insert an external url is hidden with display: none.

In the front end you could let Silverstripe decide on what to insert there (using <% if InternalLink %> etc), or you could use the same css-class there to style the output.

(Right now I am actually using 1 DOM to insert several items to a sidebar - thumbnail+image+lightbox, internal link+thumbnail+caption, external link+thumbnail+caption - but since this is my own project, I can just put all the fields in one popup and let SilverStripe sort out the rest in my page.ss. For a client this would be a big no-no, but YMMV.)

Why not just use a SelectionGroup field to expose the different choices? E.g. as in a FileIFrame field uses "Upload" or "Choose existing"

Sounds ok to me. But: can the output of a SelectionGroup field be saved as a class in a field in the table? Because I think it is important that someone who reopens a popup is back at the state it was saved with.

Yup. Look at the code for the form in /sapphire/form/ImageField.php.. it's got some tricky syntax, but you can probably just cut and paste.