in that demo I didnt see a way to make headings etc. That might be a problem and seeing the 'size' field brought back horrible <font size="+2"> memory's

YUI components generally look pretty solid and well documented.
like lakshan said, integration makes more sense on a broader level - otherwise you're just introducing a massive new codebase for one feature. I could think of tabs, windows, tree, context-menu, lightbox, ...

pros:
- more solid CMSMain.js replacement
- consistent coding for most of our js-needs
- well documented & active community
- actively developed
- tested on multiple browsers
- consistent skinning support

cons:
- lots and lots of integration work & testing for silverstripe
- overlapping featureset with prototype/scriptaculous
- filesize-heavy
- customizations needed?

RTEs are a pain in the ass to write and maintain, so unless there's a compelling reason to switch, i'd stay with TinyMCE for now. i'm not saying we should switch to YUI right away either (too much work?), just getting the discussion going.