Hello,

I just created my first silverstripe website with my own template - and I like silverstripe!
Now I'm trying to migrate the content from my old joomla site. I exported the content into CSV-file and converted it to the structure of SiteTree-table. Then I imported the CSV into my silverstripe database. All fine ... but now I am in trouble with the charset.

- CSV-file is UTF-8
- mySQL database is UTF-8
- _config.php includes: ContentNegotiator::set_encoding('utf-8');
- my page.ss includes <meta http-equiv="Content-type" content="text/html; charset=utf-8" >
- I even checked the database content by SELECT HEX(TITLE) FROM SiteTree

But the german umlaut characters (i.e. ä = U+00E4 = UTF-8 c3 a4) is still not displayed, neither in CMS nor in in frontend.
The Content (HTML-field) is ok, but Title and MenuTitle is wrong.

Then I tried to enter an "ä" into CMS and I was very astonished that silverstripe coded it as hex C3 83 C2 A4 into database.
I don't think that this is UTF-8 !

Why doesn't Silverstripe use UTF-8 ? I'm very confused.

Thanks in advance
Fanta

The collation is "utf8_unicode_ci" and i also tried "utf8_general_ci".

Uhm, I really can't follow you. Just tried a run of the mill 2.3 installation with umlauts in Title, MenuTitle and Content (all entered through CMS), they all render fine in a standard blackcandy theme with content-type delivered as utf8.
Try it yourself on demo.silverstripe.com - if you're quick (next 30 min?), you can see my umlaut test page on there: http://demo.silverstripe.com/umlaut-test/.

This is how silverstripe sets the collation for SiteTree->Content: mediumtext character set utf8 collate utf8_general_ci

Here's the utf-8 setting which every site gets through the bootstrapper - admittedly its harder *not* to use utf8 than it is to have it working out of the box ;) http://open.silverstripe.com/browser/modules/sapphire/branches/2.3/main.php#L61

Hey Fanta, I've roughly traced through the execution of saving a dataobject through to the database column, can't find anything suspicious that would deviate from our UTF-8 defaults. We need to get better about using multibyte safe PHP5 functionality though, I've opened a new ticket about this: http://open.silverstripe.com/ticket/3746 - not sure if its related to your problem.
Any help in this area is appreciated (both in-depth testing and patches).

I found PHPWACT quite helpful to figure out UTF-8 in PHP: http://www.phpwact.org/php/i18n/utf-8
Unfortunately you can't expect PHP to do the "right thing" by default apparently... the list on this page is fairly frightening

Not sure why and how you got UTF-16 or even 32 characters in your db - can you give more details on your server setup, system locale, php locale, php.ini and mysql conf?