Jump to:

3447 Posts in 1061 Topics by 738 members

Data Model Questions

SilverStripe Forums » Data Model Questions » Dataobject insert about 4000 quearies

Moderators: martimiz, Sean, biapar, Willr, Ingo, swaiba, simon_w

Page: 1
Go to End
Author Topic: 2105 Views
  • imsas
    Avatar
    Community Member
    22 Posts

    Dataobject insert about 4000 quearies Link to this post

    I have a problem with inserting into the database. It should be at the same time to insert about 5000 records. I use ecommerce module. How to accelerate the insertion, as DataObjects takes a very long time ... For ongoing connection to the SOAP server, and information acquisition and storage of the ecommerce module. Received errors on the memory utilization, but hold it up to 128MB to avoid errors, execution time increases over the 300S.
    Maybe you have a recipe?

    code sample:

    function GetProductList($Name, $ID, $Parent){
          
          $results = $this->cacheToFile('getResultsProductList', 3600, false, array($Name, $ID));
          
          if (isset($results))
          foreach ($results->GetProductListResult->ProductList->Product as $key => $item){      
             
             $isNew = DataObject::get_one('AccProduct', '`SupplierCode` = "'. Convert::raw2sql($item->SupplierCode) .'"');
             if ($isNew){
                unset($this->GetProductList->GetProductListResult->ProductList->Product[$key]);
                continue;      
             }
             //$Parent = DataObject::get_one('AccProductGroup', "`ClassId` = '$item->ClassId'");
    //
             if (!$Parent){
                unset($this->GetProductList->GetProductListResult->ProductList->Product[$key]);
                continue;
             }
             
             $product = new AccProduct();
    //         
             $product->MenuTitle = $item->Name;
             $product->Title = $item->Name;
             $product->URLSegment = strtolower(trim(str_replace(' ', '-', $item->Name)));         
             $product->MetaTitle = $Parent->Title . " - " . $item->Name;
             $product->MetaDescription = $item->Name;
             $product->MetaKeywords = trim(str_replace(' ', ',', $item->Name));         
             $product->ProvideComments = 0;
             $product->Sort = $this->Sort++;         
             $product->ParentID = $Parent->ID;         
             $product->Content = '';            
             $product->Price = $item->Price;
             $product->Warranty = $item->Warranty;
             $product->Quantity = $item->Quantity;
             $product->FeaturedProduct = 0;
             $product->AllowPurchase = 1;
             $product->Weight = $item->GrossWeight;
             $product->DateExpected = $item->DateExpected;
             $product->RecommendedRetailPrice = $item->RecommendedRetailPrice;
             $product->IsNewProduct = $item->IsNewProduct;
             $product->SupplierCode = $item->SupplierCode;
             $product->PartNumber = $item->PartNumber;
             $product->Segment = $item->SegmentId;
             $product->Group = $item->GroupId;
             $product->Class = $item->ClassId;
             $product->Series = $item->SeriesId;
             
             
    //         print "<pre>";
    //         //var_dump($this->getProductClassification($item->ClassId));
    //         //Debug::show($this->getProductClassification($item->ClassId));
    //         print "</pre>";
             
             
             
             if (!$isNew){            
                $product->writeToStage('Stage');
                $product->publish('Stage', 'Live');            
             }else {
                $isNew->destroy();
             }
             unset($this->GetProductList->GetProductListResult->ProductList->Product[$key]);
             
             $product->destroy();
                      
             $Parent->destroy();
          }
          
          
          //Sunaikinam gauta sarasa produktu
          unset($results);
          return true;
       }

  • joshy
    Avatar
    Community Member
    57 Posts

    Re: Dataobject insert about 4000 quearies Link to this post

    Hiya,

    I've had the same problem and found that the easiest thing to do was to just run it three of four times - until all the products are entered. You are already checking if it's new - and if you output some debugging information then you can also see which ones have been done.

    I found it was quicker to just run it 3-4 times than actually re-write the code in different ways. You could, for instance, use MySQL's LOAD DATA INTO function to do it.

    If you code a quicker way - and I'm sure there are quicker ways - please do let me know.

    Cheers,

    Josh

  • imsas
    Avatar
    Community Member
    22 Posts

    Re: Dataobject insert about 4000 quearies Link to this post

    Throughout the day, trying to find out how to insert a large flow of records in the database. The decision is simple: insert without sitetree versions, and the stage version, and then the insertion is very fast ... But the versions of all hinders ...

    I use:

    $product->writeToStage('Stage');
    $product->publish('Stage', 'Live');


    When I use $product->writeToStage('Live'); and $product->writeWithoutVersion(); I get in sitetree table record with sitetree class and empty variables...
    Maybe is other magic way....

  • imsas
    Avatar
    Community Member
    22 Posts

    Re: Dataobject insert about 4000 quearies Link to this post

    I dont try, but found solution insert/update, less memory usage becouse its array dealing soliution check here http://doc.silverstripe.com/doku.php?id=manipulation-arrays

    2105 Views
Page: 1
Go to Top

Want to know more about the company that brought you SilverStripe? Then check out SilverStripe.com

Comments on this website? Please give feedback.