Special characters in UI

marekzehra · 24 Jan 2018, 14:52

Hi,
I have now PanelDue v3 with 5" LCD. But, that is just for information what testing device I have.

What I have tried to do is adding my (Czech) language to PanelDueFirmware. Cloning and compiling firmware was without problem. Adding language was pretty easy at first sight. But when I uploaded firmware to device, there was only rectangles instead of special characters (in Czech language it is ě,š,č,ř,ž,ý,á,í,é,ú,ů).

I have looked on used fonts and discovered that Liberation Sans (even generated character map) contains some of those characters. But none of those special characters is displayed correctly on device. There are rectangles everywhere

So, my question is: What should I do to fix it?
Or, can you "point direction" where might be problem?
I think I can fix it by myself and then post pull request on github with Czech language, if you want.

dc42 · 24 Jan 2018, 16:28

The fonts used in PanelDue currently have Unicode code points 0x00 to 0xFF. Some of the characters you want are a little higher than that. So the first thing you need to do is extend the font range to include the extra characters. It looks to me that you need to go up to 0x17F.

After that, it should just be a case of ensuring that all the text strings are encoded in UTF8 format. If you are getting rectangles everywhere then I suspect the editor you are using isn't saving the file in UTF8 format.

HTH David

c310 · 24 Jan 2018, 19:43

I also have encoding issue:

I use russian file names (Cyrillic characters абвгдеёжз…) and in case i upload file directly from slic3r, in duet encoding is messed...

exporting same .gcode file from slic3r to PC file system works well

David, is it something that can be fixed in duet3D? i can email you file for test… thanks!

Also, if file name has brackets ---> USB штекер (испр.).gcode
duetWiFi is not able to find it after direct send from s3r....

dc42 · 24 Jan 2018, 20:09

How are you uploading the file directly from slic3r? If you are using a plugin, perhaps that plugin doesn't handle UTF8-encoded filenames.

marekzehra · 24 Jan 2018, 21:20

@dc42:

The fonts used in PanelDue currently have Unicode code points 0x00 to 0xFF. Some of the characters you want are a little higher than that. So the first thing you need to do is extend the font range to include the extra characters. It looks to me that you need to go up to 0x17F.

After that, it should just be a case of ensuring that all the text strings are encoded in UTF8 format. If you are getting rectangles everywhere then I suspect the editor you are using isn't saving the file in UTF8 format.

HTH David

Thanks.
For starters I checked files for encoding, fixed that and reflash. Some characters worked. Nice
Second I tried to expand font array, but it is limited by uint8_t which has max value 255. Now I have to dig a litle deeper and use uint16_t.

dc42 · 24 Jan 2018, 21:44

It looks to me that the following changes are needed:

members firstChar and lastChar of struct FontDescriptor need to be changed from uint8_t to uint16_t
parameter to writeNative needs to be changed to uint16_t
in UTFT::writeNative, translation should only be attempted if the character is <= 0xFF (but I don't think the translation mechanism is needed any more)
a few changes are also needed to UTFT::write.

I will make these changes in the next beta of PanelDueFirmware.

darookee · 24 Jan 2018, 22:47

@dc42:

How are you uploading the file directly from slic3r? If you are using a plugin, perhaps that plugin doesn't handle UTF8-encoded filenames.

Slic3r supports uploading to the Duet natively, it's in Printer Settings -> Print server upload. I'm not sure when the support was added but I'm using it for quite a while now.

c310 · 25 Jan 2018, 07:26

How are you uploading the file directly from slic3r? If you are using a plugin, perhaps that plugin doesn't handle UTF8-encoded

i upload directly from slic3r. there is a special option for duet in printer settings. no plug-ins at all… see below

marekzehra · 25 Jan 2018, 08:32

@dc42:

It looks to me that the following changes are needed:

members firstChar and lastChar of struct FontDescriptor need to be changed from uint8_t to uint16_t

parameter to writeNative needs to be changed to uint16_t

in UTFT::writeNative, translation should only be attempted if the character is <= 0xFF (but I don't think the translation mechanism is needed any more)

a few changes are also needed to UTFT::write.

I will make these changes in the next beta of PanelDueFirmware.

Looking forward to that

dc42 · 26 Jan 2018, 18:47

I have made those changes, tested them, and committed them to github. I will make further unrelated changes before I do a release.

marekzehra · 26 Jan 2018, 22:32

@dc42:

I have made those changes, tested them, and committed them to github. I will make further unrelated changes before I do a release.

Great. I've now tested build and getting ready generated font characters (GLCD Font Creator not cooperating as I expected, needs a little bit tuning) and translations. I'll hopefully finish it soon.

marekzehra · 31 Jan 2018, 15:14

Ok, now it's working. I've created pull request on github repo. Characters are generated and tested.

Interesting issue was, that GLCD Font Creator can't create characters after 0xFF. So I had to setup lcd-image-converter with some postprocessing scripts to get it done. But now it's perfect

Testing screen:

c310 · 3 Mar 2018, 08:04

file system is still an issue with unicode file names. 1.21 RC3

dc42 · 3 Mar 2018, 11:59

@c310:

file system is still an issue with unicode file names. 1.21 RC3

Can you give an example? I keep some files with UTF8 characters in them on my SD card, and they are still displayed correctly, with the following limitations:

PanelDueFirmware prior to 1.20RC3 only displays characters up to U+0x00FF
PanelDueFirmware 1.20RC3 (which I expect to release later today) on a 5" or 7" panel displays characters up to U+0x017F
PanelDueFirmware 1.20RC3 on a 4.3" panel doesn't display characters U+0x0100 to U_0x017F correctly because the font table is wrong