Skip to main content

This site requires you to update your browser. Your browsing experience maybe affected by not having the most up to date version.

Data Model Questions

Dataobject insert about 4000 quearies


4 Posts   2127 Views


20 August 2009 at 9:11pm (Last edited: 20 August 2009 9:17pm), Community Member, 22 Posts

I have a problem with inserting into the database. It should be at the same time to insert about 5000 records. I use ecommerce module. How to accelerate the insertion, as DataObjects takes a very long time ... For ongoing connection to the SOAP server, and information acquisition and storage of the ecommerce module. Received errors on the memory utilization, but hold it up to 128MB to avoid errors, execution time increases over the 300S.
Maybe you have a recipe?

code sample:

function GetProductList($Name, $ID, $Parent){
      $results = $this->cacheToFile('getResultsProductList', 3600, false, array($Name, $ID));
      if (isset($results))
      foreach ($results->GetProductListResult->ProductList->Product as $key => $item){      
         $isNew = DataObject::get_one('AccProduct', '`SupplierCode` = "'. Convert::raw2sql($item->SupplierCode) .'"');
         if ($isNew){
         //$Parent = DataObject::get_one('AccProductGroup', "`ClassId` = '$item->ClassId'");
         if (!$Parent){
         $product = new AccProduct();
         $product->MenuTitle = $item->Name;
         $product->Title = $item->Name;
         $product->URLSegment = strtolower(trim(str_replace(' ', '-', $item->Name)));         
         $product->MetaTitle = $Parent->Title . " - " . $item->Name;
         $product->MetaDescription = $item->Name;
         $product->MetaKeywords = trim(str_replace(' ', ',', $item->Name));         
         $product->ProvideComments = 0;
         $product->Sort = $this->Sort++;         
         $product->ParentID = $Parent->ID;         
         $product->Content = '';            
         $product->Price = $item->Price;
         $product->Warranty = $item->Warranty;
         $product->Quantity = $item->Quantity;
         $product->FeaturedProduct = 0;
         $product->AllowPurchase = 1;
         $product->Weight = $item->GrossWeight;
         $product->DateExpected = $item->DateExpected;
         $product->RecommendedRetailPrice = $item->RecommendedRetailPrice;
         $product->IsNewProduct = $item->IsNewProduct;
         $product->SupplierCode = $item->SupplierCode;
         $product->PartNumber = $item->PartNumber;
         $product->Segment = $item->SegmentId;
         $product->Group = $item->GroupId;
         $product->Class = $item->ClassId;
         $product->Series = $item->SeriesId;
//         print "<pre>";
//         //var_dump($this->getProductClassification($item->ClassId));
//         //Debug::show($this->getProductClassification($item->ClassId));
//         print "</pre>";
         if (!$isNew){            
            $product->publish('Stage', 'Live');            
         }else {
      //Sunaikinam gauta sarasa produktu
      return true;


20 August 2009 at 10:32pm Community Member, 57 Posts


I've had the same problem and found that the easiest thing to do was to just run it three of four times - until all the products are entered. You are already checking if it's new - and if you output some debugging information then you can also see which ones have been done.

I found it was quicker to just run it 3-4 times than actually re-write the code in different ways. You could, for instance, use MySQL's LOAD DATA INTO function to do it.

If you code a quicker way - and I'm sure there are quicker ways - please do let me know.




21 August 2009 at 8:13am (Last edited: 21 August 2009 8:20am), Community Member, 22 Posts

Throughout the day, trying to find out how to insert a large flow of records in the database. The decision is simple: insert without sitetree versions, and the stage version, and then the insertion is very fast ... But the versions of all hinders ...

I use:

$product->publish('Stage', 'Live');

When I use $product->writeToStage('Live'); and $product->writeWithoutVersion(); I get in sitetree table record with sitetree class and empty variables...
Maybe is other magic way....


28 August 2009 at 12:45am Community Member, 22 Posts

I dont try, but found solution insert/update, less memory usage becouse its array dealing soliution check here