[BlueOnyx:25096] Re: 5210R i18n Bug?? - fixed -NG

[Michael Stauber]

Hi Sasaki,

Many thanks for your thorough analysis. This was quite helpful.

> 1) New user registration
> Japanese is entered in the full name, but if "テスト002" is entered,
> only "テスト" is registered.
> If you edit from the registered list, there is a "テスト", but there is
> still no "002"

I can confirm that. I created a user with the fullName "テスト002" and he
was created properly:

Sep 10 14:41:28 5210r cced(smd)[1030452]: client 19:[49:664734]: CREATE
User name = "test002" sortName = "" fullName = "テスト002" password = xxx
emailDisabled = "0" ftpDisabled = "0" localePreference = "browser"
stylePreference = "BlueOnyx" volume = "/home" description = "テスト002"
site = "site1" enabled = "1" capLevels = ""

The GUI page /usr/userMod.php then only shows the fullName as "テスト".

CODB does have the correct fullName (in Octal encoding), though:

[root at 5210r ~]# /usr/sausalito/bin/cceclient
100 CSCP/0.99
200 READY
find User name = test002
104 OBJECT 261
201 OK
get 261
[...]
102 DATA fullName = "\343\203\206\343\202\271\343\203\210002"
[...]
201 OK

But: On the lowest possible level of the GUI (CCE.php - which fetches
the data from CODB) with the function ccephp_get() the Array with the
returned User object only shows this:

Array
(
    [OID] => 261
    [CLASS] => User
    [NAMESPACE] =>
    [fullName] => テスト
    [...]
    [description] => テスト
    [...]
)

After a bit of more digging the problem seems to be in the function
unescape() in libraries/CCE.php, which you can find here:

https://devel.blueonyx.it/trac/browser/BlueOnyx/5210R/platform/alpine.mod/ci/application/libraries/CCE.php#L983

That tries to convert the octals of ...

\343\203\206\343\202\271\343\203\210002

... back into "テスト002", but it ignores the final octal \210002, which
should be translated back to '002'.

That code is somewhat hard to read and I have problems myself making
sense of it. That will take a moment. But apparently we convert from
Octal to Decimal, from Decimal to Hexadecimal and somehow are supposed
to end up with proper UTF-8.

For example:

I can see that it correctly translates \343 to 'E3 83' (HEX).

But the octal \210002 is only translated to '88 __' (HEX) with the
second Hexadecimal byte being empty.

In fact it took the leading '210', which is in fact 88 (HEX) and then
ignored the rest.

The problem clearly is that I only took 3-digit octets into account, but
\210002 is a six digit octet. :-/

While I now have isolated the problem, the fix will take me a moment to
figure out. I'll get back to you when I have something.

> 2) 5209R-> 5210R
> /usr/sausalito/sbin/easy-migrate.pl --source IP -p 22 --ip IP --vsites
> all --nodns
> There is no problem with the log after migrating with the migrate
> command, and as I contacted last time, it seems that the Japanese
> processing is not done correctly for 4 people.
> Also, the user data is migrated with the migrate command, but the name
> (Full Name) is not migrated, so the problem seems to remain.

I'll run a test migration from 5209R to 5210R with Japanese "fullName"
configured and will see what I can find out. It might just be a display
problem with the data in CODB being correct, but the GUI being unable to
correctly convert it to UTF-8 like in the problem above.

But I'll check that once the general display problem with conversion of
six digit octals to UTF-8 is fixed.

-- 
With best regards

Michael Stauber