I came across some interesting feedback on Silverstripe's backend/admin performance via the review comments on the Silverstripe book at Amazon:
I was most surprised to see the comment "The admin interface seems to slow down exponentially with site tree growth -the core of the CMS - and the book offers no text to address this issue.".
I wonder if you'd seen these? If you have, please accept my apologies. Anyhow, you may have already begun working on that issue, or even resolve it - after all, that comment was written late last year.