Skip to main content

This site requires you to update your browser. Your browsing experience maybe affected by not having the most up to date version.

Data Model Questions /

Dataobject insert about 4000 quearies


Reply


4 Posts   2180 Views

Avatar
imsas

Community Member, 22 Posts

20 August 2009 at 9:11pm

Edited: 20/08/2009 9:17pm

I have a problem with inserting into the database. It should be at the same time to insert about 5000 records. I use ecommerce module. How to accelerate the insertion, as DataObjects takes a very long time ... For ongoing connection to the SOAP server, and information acquisition and storage of the ecommerce module. Received errors on the memory utilization, but hold it up to 128MB to avoid errors, execution time increases over the 300S.
Maybe you have a recipe?

code sample:

function GetProductList($Name, $ID, $Parent){
      
      $results = $this->cacheToFile('getResultsProductList', 3600, false, array($Name, $ID));
      
      if (isset($results))
      foreach ($results->GetProductListResult->ProductList->Product as $key => $item){      
         
         $isNew = DataObject::get_one('AccProduct', '`SupplierCode` = "'. Convert::raw2sql($item->SupplierCode) .'"');
         if ($isNew){
            unset($this->GetProductList->GetProductListResult->ProductList->Product[$key]);
            continue;      
         }
         //$Parent = DataObject::get_one('AccProductGroup', "`ClassId` = '$item->ClassId'");
//
         if (!$Parent){
            unset($this->GetProductList->GetProductListResult->ProductList->Product[$key]);
            continue;
         }
         
         $product = new AccProduct();
//         
         $product->MenuTitle = $item->Name;
         $product->Title = $item->Name;
         $product->URLSegment = strtolower(trim(str_replace(' ', '-', $item->Name)));         
         $product->MetaTitle = $Parent->Title . " - " . $item->Name;
         $product->MetaDescription = $item->Name;
         $product->MetaKeywords = trim(str_replace(' ', ',', $item->Name));         
         $product->ProvideComments = 0;
         $product->Sort = $this->Sort++;         
         $product->ParentID = $Parent->ID;         
         $product->Content = '';            
         $product->Price = $item->Price;
         $product->Warranty = $item->Warranty;
         $product->Quantity = $item->Quantity;
         $product->FeaturedProduct = 0;
         $product->AllowPurchase = 1;
         $product->Weight = $item->GrossWeight;
         $product->DateExpected = $item->DateExpected;
         $product->RecommendedRetailPrice = $item->RecommendedRetailPrice;
         $product->IsNewProduct = $item->IsNewProduct;
         $product->SupplierCode = $item->SupplierCode;
         $product->PartNumber = $item->PartNumber;
         $product->Segment = $item->SegmentId;
         $product->Group = $item->GroupId;
         $product->Class = $item->ClassId;
         $product->Series = $item->SeriesId;
         
         
//         print "<pre>";
//         //var_dump($this->getProductClassification($item->ClassId));
//         //Debug::show($this->getProductClassification($item->ClassId));
//         print "</pre>";
         
         
         
         if (!$isNew){            
            $product->writeToStage('Stage');
            $product->publish('Stage', 'Live');            
         }else {
            $isNew->destroy();
         }
         unset($this->GetProductList->GetProductListResult->ProductList->Product[$key]);
         
         $product->destroy();
                  
         $Parent->destroy();
      }
      
      
      //Sunaikinam gauta sarasa produktu
      unset($results);
      return true;
   }

Avatar
joshy

Community Member, 57 Posts

20 August 2009 at 10:32pm

Hiya,

I've had the same problem and found that the easiest thing to do was to just run it three of four times - until all the products are entered. You are already checking if it's new - and if you output some debugging information then you can also see which ones have been done.

I found it was quicker to just run it 3-4 times than actually re-write the code in different ways. You could, for instance, use MySQL's LOAD DATA INTO function to do it.

If you code a quicker way - and I'm sure there are quicker ways - please do let me know.

Cheers,

Josh

Avatar
imsas

Community Member, 22 Posts

21 August 2009 at 8:13am

Edited: 21/08/2009 8:20am

Throughout the day, trying to find out how to insert a large flow of records in the database. The decision is simple: insert without sitetree versions, and the stage version, and then the insertion is very fast ... But the versions of all hinders ...

I use:

$product->writeToStage('Stage');
$product->publish('Stage', 'Live');


When I use $product->writeToStage('Live'); and $product->writeWithoutVersion(); I get in sitetree table record with sitetree class and empty variables...
Maybe is other magic way....

Avatar
imsas

Community Member, 22 Posts

28 August 2009 at 12:45am

I dont try, but found solution insert/update, less memory usage becouse its array dealing soliution check here http://doc.silverstripe.com/doku.php?id=manipulation-arrays