>I'm 90% done the project now so it's unlikely I can scrap SS and go with something like symfony.
Damn, I feel for ya' mate... I'd try to get in touch with Sam , Ingo , Will or one of the other technical guys at Silverstripe. I'm hoping they can provide you with a temporary workaround until this problem has been tackled (maybe the upcoming 2.4 fixes it , I'm not sure) ......
If I remember correctly , this had to do with the automatic refreshing/syncing of the assets directory and there is a patch floating around to disable this automatic indexing.
Well, our site is still running properly at this point within a 128MB memory limit , so no worries. The slowdown is noticeable but the system still functions.
AFAIK Silverstripe is well aware of this problem and the issue will be tackled in a future release. 2.4 alpha already outperforms the previous versions , but I haven't tested it with a sh*tload of assets yet. ;)