@MDrollette :
>snip
>I'm 90% done the project now so it's unlikely I can scrap SS and go with something like symfony.
>snip

Damn, I feel for ya' mate... I'd try to get in touch with Sam , Ingo , Will or one of the other technical guys at Silverstripe. I'm hoping they can provide you with a temporary workaround until this problem has been tackled (maybe the upcoming 2.4 fixes it , I'm not sure) ......

If I remember correctly , this had to do with the automatic refreshing/syncing of the assets directory and there is a patch floating around to disable this automatic indexing.

@yurigoul:
Well, our site is still running properly at this point within a 128MB memory limit , so no worries. The slowdown is noticeable but the system still functions.

AFAIK Silverstripe is well aware of this problem and the issue will be tackled in a future release. 2.4 alpha already outperforms the previous versions , but I haven't tested it with a sh*tload of assets yet. ;)

That 2.4 should be better sounds good!

I'm pretty sure that detection of new assets is now a manual process in recent trunk (2.4) versions. The previous autodetection and identification of "unlinked" assets were the main cause of slowdown when loading the files and images tab.

However from what I remember of the way the tree code works, you are still likely to encounter problems if you have large numbers of files/images/pages at a single point in the tree. Adding hierarchies (like organising your images into folders by category or simply hashing them on the first couple of letters) helps quite a bit.