Skip to main content

This site requires you to update your browser. Your browsing experience maybe affected by not having the most up to date version.

Archive /

Our old forums are still available as a read-only archive.

Moderators: martimiz, Sean, biapar, Willr, Ingo, simon_w

Russian Unicode problem. What to do?


Reply


26 Posts   73078 Views

Avatar
Ingo

Forum Moderator, 801 Posts

31 January 2008 at 9:59pm

hm, you could try editing the fields directly in a utf8-savyy phpmyadmin, and see what comes out in the frontend. the urlencoding from tinymce (or our ajax-form-submission) might be a reason - does the same thing happen in normal varchar/TextFields?

Avatar
SiSt

Community Member, 5 Posts

31 January 2008 at 11:08pm

Edited: 31/01/2008 11:09pm

We have the same problems with one of our sites. On our local test system everything works fine, but on the live server some characters aren't displayed correctly.
We will try another mysql server version (hopefully this week)...

Please post if you find a solution or workaround

thx
sist

@Ingo the same happens for title, menutitle etc.

Avatar
Craneway

20 Posts

31 January 2008 at 11:16pm

Thanks for reply. I'm afraid I don't know how to edit the fields directly in phpmyadmin, tried though.
The incorrectly encoded strings are saved in normal varchar/TextFields too.

Avatar
Craneway

20 Posts

1 February 2008 at 12:27am

Edited: 01/02/2008 9:06am

Possible reason of the issue:
In one of the Russian forums a user reported the same problem with TinyMCE unrecognizing one cyrillic character, though the string was saved in db correctly.

tinyMCE.init({entity_encoding: 'raw'});

didn't help.

The user as well reported that he fixed the problem turning off the AutoEscaping in CGI.pm. AutoEscaping by CGI.pm is made with Latin encoding. AutoEscaping is no problem for all encodings, except for Unicode. In utf-8 it is "escaping" a certain part of the character, which leads to "unrecognized" characters in the output.

Could that be the reason? I've no idea how to deal with it. Where is CGI.pm on the server?

Avatar
Ingo

Forum Moderator, 801 Posts

1 February 2008 at 9:08am

they're referring to another script in another language, silverstripe doesn't use CGI.pm. both php5 and mysql4 should be safe to handle utf8, we encode all our content in it (and didn't have problems in spanish/french translations so far). the problem happening in title/menutitle pretty much rules out tinymce as a culprit. i think looking for help in russian forums is your best shot, as its only two out of a dozen russian special characters, so a very specific problem

Avatar
Craneway

20 Posts

1 February 2008 at 9:30am

Edited: 01/02/2008 9:37am

Ingo, they, actually, discussed another CMF. I just thought it coud be the reason. It's definitely not.
It looks like there's nothing in Russian forums about that. In Russia SS is not widely known yet. I couldn't find any Russian site running on SS.
Do you think the cause of the problem is somehow with db encoding? It's strange that the strings are not saved correctly, though the collation of the fields is set to utf8_general_ci.
Which script (file or module) in SS is responsible for saving the strings into DB?
Do you think it also might be some system templates that are not set to utf8 and causing the weird output?
The characters appear unrecognized after the edited content is saved.

And I tried to correct the strings in the db fields, after flushing everything, the whole content output in the CMS and on the site became unreadable.

Avatar
Ingo

Forum Moderator, 801 Posts

1 February 2008 at 9:34am

you don't need to find russian silverstripe developers, just russians with knowledge about character handling in php5/mysql :)

Avatar
Craneway

20 Posts

2 February 2008 at 1:26am

Edited: 02/02/2008 1:27am

Hi, Ingo!
I couldn't find the solution yet. If the site is on a shared hosting, there's not much that you can configure. I tried to create an empty db, then convert it to utf8 via phpmyadmin, then install SS. It doesn't help much. The server support guy said that I probably have to hack my php files (to insert mysql directives). Someth like:

mysql_query ("set character_set_client='utf8'");
mysql_query ("set character_set_results='utf8'");
mysql_query ("set collation_connection='utf8_general_ci'");


But to what file? I tried install.php before installing SS, works fine but no use.

Could you or anyone, please, look at my server variables after SS installation that are shown by phpmyadmin. This is a mess. Is there any ideas what I should do?

character set client    utf8
(Global value)    cp1251
character set connection   utf8
(Global value)    cp1251
character set database    cp1251
character set filesystem    binary
character set results    utf8
(Global value)    cp1251
character set server    cp1251
character set system    utf8
character sets dir    /usr/share/mysql/charsets/
collation connection    utf8_general_ci
(Global value)    cp1251_general_ci
collation database    cp1251_general_ci
collation server    cp1251_general_ci