Skip to main content

This site requires you to update your browser. Your browsing experience maybe affected by not having the most up to date version.

We've moved the forum!

Please use forum.silverstripe.org for any new questions (announcement).
The forum archive will stick around, but will be read only.

You can also use our Slack channel or StackOverflow to ask for help.
Check out our community overview for more options to contribute.

General Questions /

General questions about getting started with SilverStripe that don't fit in any of the categories above.

Moderators: martimiz, Sean, Ed, biapar, Willr, Ingo, swaiba

CsvBulkUploader splitting files


Go to End


2 Posts   749 Views

Avatar
jpmcc

Community Member, 20 Posts

10 March 2016 at 8:04am

Edited: 10/03/2016 8:05am

I've just been trying to figure out why my CSV data file was not importing correctly for certain records - they looked file, Both LibreOffice and Excel (to a certain degree - it really struggles with UTF8), opened the CSV file, with data expected to be where it was. Checking in plain text editors, I wasn't seeing extraneous commas or errant double quotes anywhere. I just couldn't see what the problem was with the data.

Well, eventually I figured it out - the CsvBulkUploader actually splits the original file into files of 1000 lines - I hadn't realised that before as I've never had to poke too deeply into the importing process beyond any custom importers I'd created. Well, usually I think this wouldn't be an issue, except one of the fields in the original file is an address field, containing multiple lines of addresses, yes, separated by newlines.

So, what is happening is the CsvBulkUploader is splitting the files in the middle of some address fields, so when the last record in the current file is imported, only the data up to the point in the address field that is included. Then the next file is started and the remaining data is loaded, being totally in the wrong place (so the created DataObjects don't have the required data). After a few rows, the data sorts itself out again until the next split.

A lot of background info there, probably more than required. Anyway, simple question - I see there is

private static $lines = 1000;
. Now other than setting that to a rather large value, is there any other way to just prevent the CsvBulkUploader from splitting the original file? I guess the other option is just to override the splitFile method to return an array of one file - the original one.

Just wondering if there was a better way (other than getting the client to remove all of the new lines in the address field).

Thanks,
Jason.

Avatar
jpmcc

Community Member, 20 Posts

10 March 2016 at 8:18am

Just an update - overriding splitFile and just returning an array containing the original file works for me, just wondering if that is the best way.