There's been a lot of work in creating custom link and image inserters based around TinyMCE. If you want to use another editor, then you will lose these.
But apart from that, it should be doable. However, to get to a satisfactory result, you'll have to muck with a bunch of code in the core packages, which will make it hard to upgrade.
The place where all this is controlled is HTMLEditorField. You should be able to edit that class to load your own editor javascript into the system.
If you stop including the TinyMCE javascript files, you will run into some javascript bugs in the CMS - particularly when you ajax-load a page. Your best bet is probably to use firebug to see where the code is tripping up, and commenting the TinyMCE-specific stuff out.
You should update LeftAndMain.php, to use Requirements::javascript() to include your editor's script files.
You'll also need to remove the WYSIWYG toolbar from the top panel, by editing the .ss templates for the CMS.
If you would like to make it easier to upgrade in the future, you might want to consider extending the system so that it has a configuration option to choose alternative editor implementations. Provided the code followed our coding standards, it would be a good candidate for merge into the mainline system - meaning you could continue to upgrade your site.