utf-8 characters problem.

Posted: Sun Jul 05, 2015 12:45 pm

Thanks anyway. Actually my issue is not the db thing, it's ONLY the preview of the post (new topic or a reply), so far if the message has any of those special chars in the title or body, then the whole post is removed from the preview page. If you just send the post without previewing, it'll appear correctly with special characters in the topic.
This won't happen using an earlier versión of Raven nuke on the same data base (mysql).
I do appreciate any hints around this.

registered · Site Admin Joined: Aug 22, 2007 Posts: 1775

which DB colation and which charset you are using?

Posted: Sun Jul 05, 2015 1:35 pm

chars: UTF-8 Unicode (utf8)
collation: latin1_swedish_ci

N.B I can't change the chars in phpMyAdmin, there is no key to do so.

thanks

Posted: Sun Jul 05, 2015 2:31 pm

Ok i reproduced the issue and it seems there is really a bug inside the posting.php since PHP 5.4.x the default charset for Only registered users can see links on this board! Get registered or login! was set to UTF-8!

So we have to add the used charset of the RN installation. The charset is stored in the definition _CHARSET in the language files. If you should get more issues like that, then push it - i will try a look into it.

... open modules/Forums/posting.php:

find:

php Code:

   $subject = ( !empty($HTTP_POST_VARS['subject']) ) ? htmlspecialchars(trim(stripslashes($HTTP_POST_VARS['subject']))) : '';


   $message = ( !empty($HTTP_POST_VARS['message']) ) ? htmlspecialchars(trim(stripslashes($HTTP_POST_VARS['message']))) : '';





   $poll_title = ( !empty($HTTP_POST_VARS['poll_title']) ) ? htmlspecialchars(trim(stripslashes($HTTP_POST_VARS['poll_title']))) : '';

change it to:

php Code:

   $subject = ( !empty($HTTP_POST_VARS['subject']) ) ? htmlspecialchars(trim(stripslashes($HTTP_POST_VARS['subject'])), ENT_QUOTES, _CHARSET) : '';


   $message = ( !empty($HTTP_POST_VARS['message']) ) ? htmlspecialchars(trim(stripslashes($HTTP_POST_VARS['message'])), ENT_QUOTES, _CHARSET) : '';





   $poll_title = ( !empty($HTTP_POST_VARS['poll_title']) ) ? htmlspecialchars(trim(stripslashes($HTTP_POST_VARS['poll_title'])), ENT_QUOTES, _CHARSET) : '';

find:

php Code:

$poll_options[intval($option_id)] = htmlspecialchars(trim(stripslashes($option_text)));

change it to:

php Code:

$poll_options[intval($option_id)] = htmlspecialchars(trim(stripslashes($option_text)), ENT_QUOTES, _CHARSET);

For all they don't get the issue - try to add a title and/or a message with the follwing chars: ÅÄÖ, press on the "Preview" button in the edit-form and you will get the issue that kenspa has noticed.

Posted: Sun Jul 05, 2015 3:45 pm

Thank you so much neralex!! That was it!! Spent some hours checking posting.php before your message, but I wasn't able to know to do it right!!!
I'll check all the functions on the fórum and see if something of this kind still missing and let you know!!
Very Happy

Posted: Sun Jul 05, 2015 4:52 pm

php Code:

// Mouse hover topic boxover MOD - BEGIN


//


		$topic_content = $topic_rowset[$i]['post_text'];


		$bbcode_uid = $topic_rowset[$i]['bbcode_uid'];


		$topic_content = ( $board_config['allow_bbcode'] ) ? bbencode_second_pass($topic_content, $bbcode_uid) : preg_replace('/\:[0-9a-z\:]+\]/si', ']', $topic_content);


		$topic_content = smilies_pass($topic_content);


		$topic_content = str_replace('\"', '"', substr(@preg_replace('#(\>(((?>([^><]+|(?R)))*)\<))#se', "@preg_replace(\$orig_word, \$replacement_word, '\\0')", '>' . $topic_content . '<'), 1, -1));


        $topic_content = str_replace("\n", "\n<br />\n", $topic_content);


		$topic_content = htmlspecialchars($topic_content);


		$topic_content = "cssbody=[boxbdy1] cssheader=[boxhdr1] header=[". $topic_title ."] body=[".$topic_content."] singleclickstop=[On] hideselects=[on] fade=[on]";


//


// Mouse hover topic boxover MOD - END

the code above is for mouse over topic which gives you a preview of last post, it also drops the content if any of the special characters is in the content!!
I tried to work around it but it doesn't work. Any clue??
thanks

Posted: Sun Jul 05, 2015 8:53 pm

find:

php Code:

$topic_content = htmlspecialchars($topic_content);

replace:

php Code:

$topic_content = htmlspecialchars($topic_content, ENT_QUOTES, _CHARSET);

Posted: Mon Jul 06, 2015 2:54 pm

Yes!!! That's fixed it! Smile

Now everything Works pretty good except for the quote and the [twitter] tags! I think the quote is not filtered because it uses java to expand and contract, whille the twitter tag I have no idea.

Thank you so much neralex for your help!

Posted: Mon Jul 06, 2015 3:46 pm

the twitter tag feature needs to be removed as the twitter api no longer allows anonymous access.

Client Joined: Jan 29, 2004 Posts: 624

Dunno where else to put this. I have an error in my Feeds module I discovered.

Code:




XML Parsing Error: XML or text declaration not at start of entity


Location: http://www.domain.net/feeds-1-rss20.xml


Line Number 5, Column 1:<?xml version="1.0" encoding="utf-8"?>

Thanks for ideas.

Posted: Thu Aug 13, 2015 6:31 am

hmm... check your PHP error log, please. Maybe you get a hint there.

Posted: Thu Aug 13, 2015 9:45 am

Okey doke.

Posted: Fri Aug 14, 2015 8:49 am

Without a link to the feed it's hard to tell but the most likely culprit is an empty character space where there shouldn't be one.

Posted: Fri Aug 14, 2015 11:49 am

Guardian2003 wrote:

Without a link to the feed it's hard to tell but the most likely culprit is an empty character space where there shouldn't be one.

I deactivated the Feeds module until this is resolved. It says an XML parsing error, and the XML file's location is in the root. Is this correct for RN 2.5?

Edited

No, it isn't correct. RN 2.5 has no ultramode.txt or backend.php. I still had links to those from earlier versions of RN which I've removed. This is resolved.

Posted: Sat Aug 15, 2015 2:42 am

RN25x has a backend.php! If you deleted this file, then play it back from the download-package.

Posted: Sat Aug 15, 2015 12:34 pm

neralex wrote:

RN25x has a backend.php! If you deleted this file, then play it back from the download-package.

I didn't delete the file, just removed the link to it.

New Member Joined: Feb 20, 2018 Posts: 14

I had the same problem.

Here's how I fixed it:

1. I set the database to utf8_general_ci
2. in the language file I changed define('_CHARSET','ISO-8859-1'); to define('_CHARSET','UTF-8');
3. in the mainfile.php, after:
require_once NUKE_BASE_DIR . 'db/db.php';
I added:
mysqli_set_charset($db,"utf8");

It works perfectly. However, I tested it only locally. (XAMPP, PHP Version: 7.2.1, phpMyAdmin: 4.7.4, MariaDB: 10.1.30)

Posted: Mon Oct 29, 2018 1:00 am

Nuke8 wrote:

I had the same problem.

Here's how I fixed it:

1. I set the database to utf8_general_ci
2. in the language file I changed define('_CHARSET','ISO-8859-1'); to define('_CHARSET','UTF-8');
3. in the mainfile.php, after:
require_once NUKE_BASE_DIR . 'db/db.php';
I added:
mysqli_set_charset($db,"utf8");

It works perfectly. However, I tested it only locally. (XAMPP, PHP Version: 7.2.1, phpMyAdmin: 4.7.4, MariaDB: 10.1.30)

This doesn't work perfectly, because RN25x isn't ready for utf-8 and you will run in many many other issues like that. Believe me, I made my radio website ready for utf-8 some years ago and I had to recode 95% of the core code. You have to change EACH filtering on each file of the CMS. To switch the database will run in a huge count of other issues and the change in language constant will create a bunch of more issues like that. RN25x isn't ready for utf-8 and it should stay on ISO to work correctly.

Posted: Mon Oct 29, 2018 3:09 am

I'm sorry, I should have written that "it works perfectly for me". This is the correct statement.

And I forgot to mention that I already re-written the lang-hungarian.php files (I'm hungarian) in each module I use.

With the modifications, at the moment the hungarian characters are correct everywhere:
- displayed correctly on the site
- put correctly in the database
- the content of the language files are also correct.

Yes, I changed the filtering in some modules. Since I don't use too many modules, it wasn't a problem for me.

Before using mysqli_set_charset($db,"utf8"); the hungarian characters were displayed correctly on the site, but not in the database. Which made impossible to manually read the database or the .sql file. That's why I needed this.

So, the reason why I wrote the previous post is to share this idea. And yes, you're right, there are further things to do. But it might be helpful for others.