Skip to main content

This site requires you to update your browser. Your browsing experience maybe affected by not having the most up to date version.

General Questions /

General questions about getting started with SilverStripe that don't fit in any of the categories above.

Moderators: martimiz, Sean, biapar, Willr, Ingo, swaiba, simon_w

Asset Performance With Large Number Of Files


Go to End
Reply


11 Posts   2370 Views

Avatar
MDrollette

Community Member, 10 Posts

12 November 2009 at 1:58pm

I'm migrating a site to SS and I've used the CSV bulk uploader to get 6,000+ dataobjects loaded and that all works fine with no significant performance hit. The problem though is that each of those objects has 2-6 images associated with them. I've copied all the images to the assets folder and that automatically synced up fine and loaded all of the 16,000 File objects into the database but I cannot access the /assets section of admin and can not load any of the "select from file store" image selectors without hitting a PHP memory limit. I've bumped the memory limit in PHP up to 1gb and still no luck. I understand 16,000 files is a lot. But it shouldn't be impossible to implement this efficiently. Is there any workaround or known reason for this bottleneck?

As a note, the files are broken up into subdirectories something like this...

assets/CarParts/1/image_1.jpg
assets/CarParts/1/image_2.jpg
assets/CarParts/2/image_1.jpg
assets/CarParts/2/image_2.jpg

There are about 5,000 sub-directories inside CarParts, each containing a handful of images. Is it possible that having a certain ratio of sub-directories to files or a deeper nesting might at least let the asset manager load?

Avatar
Fuzz10

Community Member, 787 Posts

12 November 2009 at 11:15pm

This is a known problem, you can read up on :

http://groups.google.com/group/silverstripe-dev/browse_thread/thread/84833d523a3bfbf1

For me, the latest stable version improved the situation somewhat , but I can't imagine it working properly with 10.000+ files.

Avatar
yurigoul

Community Member, 202 Posts

12 November 2009 at 11:33pm

Edited: 12/11/2009 11:33pm

Good to know. So what this means is that you can not use Silverstripe for serious websites?

In the usenet discussion mentioned in the post above some people mention that you should be able to turn certain features off, for instance in the _config.php. Some of those things should be implemented in 2.3.x

Are there any recepies/code snippets/words of advice floating around?

Avatar
Fuzz10

Community Member, 787 Posts

13 November 2009 at 12:26am

>Good to know. So what this means is that you can not use Silverstripe for serious websites?

Well, I think that is a bit of an overstatement. Sure, there are some (performance) issues and no , if I'd be creating the new Facebook maybe Silverstripe would not be my weapon of choice, but all in all I think it is perfectly suited for large(r) websites.

IMHO, Having 10.000+ assets is not really an everyday use-case and I would love to see the problem fixed (since I have 1 site with the same problems as topic-starter).

Avatar
yurigoul

Community Member, 202 Posts

13 November 2009 at 12:41am

Sorry for being a so negative, but it does trouble me.

The question is when the problem kicks in: does it already start when you have 5.000+ images? Or maybe even 3.000+?

And how are the images counted? How about a site with 3.334 images, who have thumbnails + larger versions + originals = 10.002? Or are we really talking about 10.000 images and the thumbnails and resized-images come on top of that?

Avatar
Fuzz10

Community Member, 787 Posts

13 November 2009 at 1:22am

We are running 1 failry large site with > 3000 files in the assets directory (including resampled etc.) and the site is starting to show slowdowns when accessing assets (either through files and images or by selecting an image through the content-editor).

The situation improved greatly with the 2.3.3 though.

Avatar
MDrollette

Community Member, 10 Posts

13 November 2009 at 1:46am

The site is running on 2.3.3. Does anyone know the cause of the bottleneck, though? I only have 3 directories and 2 files in the top-level directory. It seems a little unnecessary to build the entire object tree in memory just to load this top-level directory. I'm 90% done the project now so it's unlikely I can scrap SS and go with something like symfony. Are there any ideas for a workaround? Maybe a way to exclude the asset manager from entering this directory?

Avatar
yurigoul

Community Member, 202 Posts

13 November 2009 at 2:15am

Edited: 13/11/2009 2:17am

Fuzz10: "We are running 1 failry large site with > 3000 files in the assets directory (including resampled etc.) and the site is starting to show slowdowns when accessing assets (either through files and images or by selecting an image through the content-editor)."

So does this mean that this problem becomes noticeable at 1.000 - 1.500 images?... Which translates to a site with 500 product-pages/article-pages with 3 images per page ( and then add thumbs + larger images + original images)? And what about the cms previews and downloads like pdfs?

Go to Top