I gave up on this pretty quickly and used the official tinyMCE image manager instead (it costs 30EUR but is pretty damn good). The links is more of a problem as the default tinyMCE links editor makes you type the link in, but it should work in a popup.
Just edit yout cms/javascript/tinyMCE.template.js and edit the second row of buttons: 'ssimage,ssflash,sslink' into 'image,media,link'. Downside is you then get this for the main editor too. I guess there must be a way to create a new HTMLEditorField type and have different settings but I never got that far.
Alernatively you could use the simple html editor that comes with unclecheeses DataObjectManager module, it has simple url inputs for images and links, but it's a little sparse on other options.