Jump to:

23450 Posts in 18899 Topics by 2877 members

General Questions

SilverStripe Forums » General Questions » Asset Performance With Large Number Of Files

General questions about getting started with SilverStripe that don't fit in any of the categories above.

Moderators: martimiz, Sean, biapar, Willr, Ingo, swaiba, simon_w

Page: 1 2
Go to End
Author Topic: 2229 Views
  • MDrollette
    Avatar
    Community Member
    10 Posts

    Asset Performance With Large Number Of Files Link to this post

    I'm migrating a site to SS and I've used the CSV bulk uploader to get 6,000+ dataobjects loaded and that all works fine with no significant performance hit. The problem though is that each of those objects has 2-6 images associated with them. I've copied all the images to the assets folder and that automatically synced up fine and loaded all of the 16,000 File objects into the database but I cannot access the /assets section of admin and can not load any of the "select from file store" image selectors without hitting a PHP memory limit. I've bumped the memory limit in PHP up to 1gb and still no luck. I understand 16,000 files is a lot. But it shouldn't be impossible to implement this efficiently. Is there any workaround or known reason for this bottleneck?

    As a note, the files are broken up into subdirectories something like this...

    assets/CarParts/1/image_1.jpg
    assets/CarParts/1/image_2.jpg
    assets/CarParts/2/image_1.jpg
    assets/CarParts/2/image_2.jpg

    There are about 5,000 sub-directories inside CarParts, each containing a handful of images. Is it possible that having a certain ratio of sub-directories to files or a deeper nesting might at least let the asset manager load?

  • Fuzz10
    Avatar
    Community Member
    787 Posts

    Re: Asset Performance With Large Number Of Files Link to this post

    This is a known problem, you can read up on :

    http://groups.google.com/group/silverstripe-dev/browse_thread/thread/84833d523a3bfbf1

    For me, the latest stable version improved the situation somewhat , but I can't imagine it working properly with 10.000+ files.

  • yurigoul
    Avatar
    Community Member
    202 Posts

    Re: Asset Performance With Large Number Of Files Link to this post

    Good to know. So what this means is that you can not use Silverstripe for serious websites?

    In the usenet discussion mentioned in the post above some people mention that you should be able to turn certain features off, for instance in the _config.php. Some of those things should be implemented in 2.3.x

    Are there any recepies/code snippets/words of advice floating around?

  • Fuzz10
    Avatar
    Community Member
    787 Posts

    Re: Asset Performance With Large Number Of Files Link to this post

    >Good to know. So what this means is that you can not use Silverstripe for serious websites?

    Well, I think that is a bit of an overstatement. Sure, there are some (performance) issues and no , if I'd be creating the new Facebook maybe Silverstripe would not be my weapon of choice, but all in all I think it is perfectly suited for large(r) websites.

    IMHO, Having 10.000+ assets is not really an everyday use-case and I would love to see the problem fixed (since I have 1 site with the same problems as topic-starter).

  • yurigoul
    Avatar
    Community Member
    202 Posts

    Re: Asset Performance With Large Number Of Files Link to this post

    Sorry for being a so negative, but it does trouble me.

    The question is when the problem kicks in: does it already start when you have 5.000+ images? Or maybe even 3.000+?

    And how are the images counted? How about a site with 3.334 images, who have thumbnails + larger versions + originals = 10.002? Or are we really talking about 10.000 images and the thumbnails and resized-images come on top of that?

  • Fuzz10
    Avatar
    Community Member
    787 Posts

    Re: Asset Performance With Large Number Of Files Link to this post

    We are running 1 failry large site with > 3000 files in the assets directory (including resampled etc.) and the site is starting to show slowdowns when accessing assets (either through files and images or by selecting an image through the content-editor).

    The situation improved greatly with the 2.3.3 though.

  • MDrollette
    Avatar
    Community Member
    10 Posts

    Re: Asset Performance With Large Number Of Files Link to this post

    The site is running on 2.3.3. Does anyone know the cause of the bottleneck, though? I only have 3 directories and 2 files in the top-level directory. It seems a little unnecessary to build the entire object tree in memory just to load this top-level directory. I'm 90% done the project now so it's unlikely I can scrap SS and go with something like symfony. Are there any ideas for a workaround? Maybe a way to exclude the asset manager from entering this directory?

  • yurigoul
    Avatar
    Community Member
    202 Posts

    Re: Asset Performance With Large Number Of Files Link to this post

    Fuzz10: "We are running 1 failry large site with > 3000 files in the assets directory (including resampled etc.) and the site is starting to show slowdowns when accessing assets (either through files and images or by selecting an image through the content-editor)."

    So does this mean that this problem becomes noticeable at 1.000 - 1.500 images?... Which translates to a site with 500 product-pages/article-pages with 3 images per page ( and then add thumbs + larger images + original images)? And what about the cms previews and downloads like pdfs?

    2229 Views
Page: 1 2
Go to Top

Want to know more about the company that brought you SilverStripe? Then check out SilverStripe.com

Comments on this website? Please give feedback.