UTF-8 mail function & special characters

Regular Joined: Jun 04, 2004 Posts: 84 Location: Germany

I have a question because i try a testinstalation with UTF-8 as Charset.

For example on the Feedback Module, I figgured out the Issue with special Character & German Umlauts in the subject & Body

for the body I found a workaround like:

Code:




$mailheaders .= "MIME-Version: 1.0\r\n";


$mailheaders .= "Content-Type: text/plain; charset=UTF-8\r\n";


$mailheaders .= "Content-Transfer-Encoding: 8bit\r\n";

but for the subject i have no idea in the moment.

Brujo

registered · Posted: Fri Apr 03, 2009 5:24 pm

Not my strong suit. Therefore I am <bumping> this post...

Posted: Sat Apr 04, 2009 12:22 am

@montego, thanks for <bumping>

as i wrote i try to move my website which is running under nuke/raven since 2002 ->utf-8. I think my major question should be: is it the time and possible to move over to UTF-8 under the aspect i have german umlauts and special characters on every place?

my expirince tells my <yes we can> the most of moving to the test domain works, and it is not too much work, i worekd with iconv for the DB and find & replace on sql level is very helpfull. For the forum i got a solution from Susann, so not much left and indeed some topics like Feeds works fine now. But of course the details like mail and so on needs time to get fixed..

Maybe one of the mod can open a Forum Topic relatred to UTF-8 issues / faq / howto ?

Brujo

Posted: Sat Apr 04, 2009 8:16 am

For new installations of RN it would be fairly straight forward, although potentially many string functions would have to be changed to their mb equivalents. Still quite a bit of work.

However, we have a quandary with existing sites and how to convert. Its not an easy problem statement, but I assure you, discussions have started on this topic, but I would be looking for a solution from us in the near term.

Posted: Sat Apr 04, 2009 9:52 am

thanks for the info sounds good, and i belive to convert existing sites it is not that easy, because i did a lot of testmigraations, but finaly i am done with all my Content, Articles, Surveys, Enceclopdia, FaQ anso so on to go over...

BTW, i just switched on tegomailer with the Mail function and voila it works also with german Umlauts

Brujo

Posted: Tue Apr 07, 2009 6:50 am

Brujo, any insight that you can provide us with regards to your migration efforts would be greatly appreciated. I'll be honest with you, this is not our strong suit.

Great news on the TegoNuke mailer. I/you have Swift Mailer to thank for that. Wink

Posted: Tue Apr 07, 2009 10:06 am

@montego
sure i can provide some insights of my test migration, personal I like to switch over to UTF-8 because I like to integrate the menalto G2 and to prevent issues as I had in the past with German Umlauts & Intl. Characters, but the reality is as you wrote with existing sites and to be honest I would say also for new sites if they have content like I have...

So my starting point is a copy of my existing site which runs since 2002 under nuke up to 7.6 and then i switched over to RN some years ago.
I use heavily German Umlauts and Intl. Characters, because my site is about Mexico in german.

MySQL char set: UTF-8 Unicode (utf8) & the table Collation & field Collation is utf8_general_ci

So after the change from iso -> utf-8 you can imagine I had tons of fancy Question marks, Diamonds etc. over all Core Modules.

So the First Module which I looked into was News & Content (Body not Title)
In majority the issue was with older Articles and Content pages which have been created before the Fckeditor was integrated, because the old form stores the Text as is in the Database. The same I faced on the FAQ, Survey, Weblink Modules etc…

New Articles or Content pages created with the Fckeditor are fine, because the Editor changes automatically Intl. Chars into htmlentities.

Resume - Intl Chars should be htmlentities or have to be converted. So this have to be fixed first, I did it for example via iconv (dumped the DB and converted it from iso –> utf-8, but this makes also some issues for example with the Forums) , but worked almost with all Modules including FAQ & Surveys, excepting some Chars like the ß & big Umlauts like Ü, why? No glue in the moment…

Another thing is: Titles - can’t have htmlentities, because the feeds will be not valid. Indeed Titles with intl. chars are one of the bigger issues becaus iconv do not convert all of them and i have no other solution in the moment.

Other issues which I found are:
mail function – Feedback Module (switch on Tegomailer helped here, but breaks Nukesentinal Mails – I have to check out why)
the Dateformat string: if a Article was created in März (march) – (Veröffentlicht von Brujo am Sonntag, 23. M�rz)
Forums: I got a link from Susann, which seems to work, but not if the DB was converted - I have to do more tests.

Feeds – utf-8 issue with intl. chars in title - I explained it on the nukeseo forum http://nukeseo.com/modules.php?name=Forums&file=viewtopic&t=339

At this point I would say, my test migration looks not so bad, I can do and adjust many things on my site before the migration to keep the effort small, but to be honest I feel it is to early to switch over an productive site

So if you have more Questions or need more details feel free to ask

Brujo

Posted: Tue Apr 07, 2009 6:14 pm

Brujo,

I cannot even properly express my gratitude for the excellent information you have provided. That took a lot of your time to write up let alone to come to all of these conclusions.

Thank you!

Posted: Fri Apr 10, 2009 9:28 am

@montego thanks for lively interest

therefore Part 2

I spent more time for the test migration ->utf-8 and did some small steps forward

My best results I reached are using iconv to convert the mysql dump -> utf-8. But iconv has some problems with (ß Ü – Ä & Ö) but this I fixed mostly on the DB table fields via:

Quote:

update [table_name] set [field_name] = replace([field_name],'[string_to_find]','[string_to_replace]');

about the date format string with the month March in german März, I changed in the _config the locale from de_DE -> de_DE.UTF8 this fixed for me this issue on all places like Articles, Weblinks & Forum, others Modules I didn’t tested.

Forums – that’s a bigger one, but I got from Susann a link and applied the changes as described, and it seems to work but not completely and it seems that we will have a mixture between Forum posts which looks ok and some fields like City, Interest in the Memberlist / Profile if there are intl. Chararcters.

But I found some interesting help on Chinese sites, it seems the Chinese folks have already done the convert of the phpbb2 -> utf8. Mostly they converted via shell script (which also use iconv) the files under forums/language/ and applied a change to the emailer.php to encode the subjects.

Therefore I did both, the changes from Susanns link & the Chinese changes and for me it seems to work, my test Forums are now completely converted ->UTF8 without gibberish and also the Memberlist / Userprofile looks now fine. Mails send from the Forums are in UTF8 ok and valid.

So for me left in the moment the issue with the feeds and the mail function….

Maybe I have next week more time to do more tests

Brujo

Posted: Sat Apr 11, 2009 8:22 am

I know the Persian Nuke folks have done this too, and I was intending on checking out their work some day. I had someone interested in producing a Persian version of HTML Newsletter, which he did, including the RTL direction changes. I don't know why I didn't think of it before, but I wonder if comparing the Only registered users can see links on this board! Get registered or login! could also give any hints. Sorry I didn't think of that before.

registered · Posted: Sat Apr 11, 2009 10:26 am

Actually this is critical discussion in that new phpBB3 and others are going to use UTF-8.

In my nuke site, I don't use ascii coding for langauges and contents (including DB) any longer. Why? ported and embedded phpBB3 inside of phpnuke and gallery2/ 3 (alpha) are using utf-8.

The mailing and news letter programs are the same. When you use ascii based character encoding in your php-nuke, and contents in different languages which use UTF-8 codes, your mail contents will be all shown in &#22438,... codes.

To convert all DB fields to UTF-8 is not difficult. However, you will change all language files as well with your UTF-8 editor.

In my opinion and php-nuke dev. projects, UTF-8 coding should be the major language and contents coding (including DB)