Make WordPress Core

Opened 17 years ago

Closed 16 years ago

Last modified 2 weeks ago

#3780 closed defect (bug) (fixed)

gettext fails to determine byteorder on 64bit systems with php5.2.1

Reported by: abtime's profile abtime Owned by:
Milestone: 2.3.3 Priority: normal
Severity: critical Version: 2.2
Component: I18N Keywords: i18n has-patch commit 2.3.3-candidate
Focuses: Cc:

Description

After upgrading to php5.2.1 on FreeBSD6.2-amd64 the function in wp-includes/gettext.php fails to determine the correct byteorder, so the language (.mo) file is not loaded.

The problem possibly is around line 114 of gettext.php

Attachments (2)

3780.diff (761 bytes) - added by rob1n 17 years ago.
gettext-64-without-bitwise-ops.diff (587 bytes) - added by nbachiyski 17 years ago.

Download all attachments as: .zip

Change History (34)

#1 in reply to: ↑ description @abtime
17 years ago

Replying to abtime:

After upgrading to php5.2.1 on FreeBSD6.2-amd64 the function in wp-includes/gettext.php fails to determine the correct byteorder, so the language (.mo) file is not loaded.

The problem possibly is around line 114 of gettext.php

This seems to be a working fix (works for me): http://wordpress.org/support/topic/63038

@rob1n
17 years ago

#2 @rob1n
17 years ago

  • Keywords dev-feedback needs-testing added
  • Milestone changed from 2.0.10 to 2.1.1
  • Owner changed from anonymous to rob1n
  • Severity changed from blocker to critical
  • Status changed from new to assigned
  • Version changed from 2.0.7 to 2.1

Added a patch from the fix detailed in http://wordpress.org/support/topic/63038

Testing needed, and a dev to check over the code, as all I did was adapt the fix's code.

#3 @rob1n
17 years ago

  • Keywords has-patch added

#4 @Nazgul
17 years ago

  • Milestone changed from 2.1.1 to 2.1.2

#5 @ryan
17 years ago

  • Resolution set to fixed
  • Status changed from assigned to closed

(In [4917]) gettext byteorder fix for 64 bit archs. fixes #3780

#6 @JanRei
17 years ago

  • Resolution fixed deleted
  • Status changed from closed to reopened
  • Version changed from 2.1 to 2.2

This fix should be in Wordpress 2.2 now, but I have still the same problem. However, it works if I apply the changes suggested in http://wordpress.org/support/topic/63038.

#7 @rob1n
17 years ago

  • Keywords has-patch dev-feedback needs-testing removed

#8 @rob1n
17 years ago

  • Milestone changed from 2.1.2 to 2.2.1

#9 @rob1n
17 years ago

  • Milestone changed from 2.2.1 to 2.2.2
  • Owner changed from rob1n to anonymous
  • Status changed from reopened to new

#10 @marcinkk
17 years ago

I have the same problem. FreeBSD 6.2 on AMD64 with PHP5.2.1, WordPress localization with 2.2 original code doesn't work :(

Known code from gettext.php in WordPress 2.2:

// $MAGIC1 = (int)0x950412de; //bug in PHP 5.0.2, see https://savannah.nongnu.org/bugs/?func=detailitem&item_id=10565
$MAGIC1 = (int) - 1794895138;
// $MAGIC2 = (int)0xde120495; //bug
$MAGIC2 = (int) - 569244523;
// 64-bit fix
$MAGIC3 = (int) 2500072158;

$magic = $this->readint();

Results on my system:

echo $magic;				Result:	-1794895138
echo $MAGIC1;					-1794895138
echo $MAGIC2;					-569244523
echo $MAGIC3;					2500072158
echo $MAGIC1 & 0xFFFFFFFF;			2500072158
echo $MAGIC2 & 0xFFFFFFFF;			3725722773
echo $MAGIC3 & 0xFFFFFFFF;			2500072158
echo (int)0x950412de;				2500072158
echo (int)0xde120495;				3725722773

So the results on all comparisons was 0, so I modified functions:

if ($magic == ($MAGIC1 & 0xFFFFFFFF) || ($magic == ($MAGIC3 & 0xFFFFFFFF)) { // to make sure it works for 64-bit platforms
    $this->BYTEORDER = 0;
} elseif ($magic == ($MAGIC2 & 0xFFFFFFFF)) {
    $this->BYTEORDER = 1;
} else {
    $this->BYTEORDER = 0;
    //$this->error = 1; // not MO file
    //return false;
}

And localization worked properly. In above code magic value is not checked but I was sure that BYTEORDER should be 0. So next I made:

if (($magic & 0xFFFFFFFF) == ($MAGIC1 & 0xFFFFFFFF) || ($magic & 0xFFFFFFFF) == ($MAGIC3 & 0xFFFFFFFF)) { // to make sure it works for 64-bit platforms
    $this->BYTEORDER = 0;
} elseif (($magic & 0xFFFFFFFF) == ($MAGIC2 & 0xFFFFFFFF)) {
    $this->BYTEORDER = 1;
} else {
    $this->error = 1; // not MO file
    return false;
}

It works too and something is checked ... but I don'y know what this magic numbers do, so somebody better than me should check the results...

#11 @foolswisdom
17 years ago

  • Milestone changed from 2.2.2 to 2.2.3

#12 @hakre
17 years ago

Just two questions:

  1. How did you guys compiled PHP, does it support 64 Bit integers?
  2. Maybe this helps? http://www.gerd-riesselmann.net/archives/2006/01/a-64-bit-integer-for-php

#13 follow-up: @nbachiyski
17 years ago

  • Keywords i18n has-patch 2nd-opition test added

Do we really need the bitwise operators? All the numbers are 4-byte ints anyway.

I have tested the patch gettext-64-without-bitwise-ops.diff and it works on:

  • Linux x86_64, PHP 5.2.3
  • Linux i686, PHP 5.2.3
  • Linux i686, PHP 4.3.10
  • FreeBSD i386, PHP 4.4.0

#14 @foolswisdom
17 years ago

  • Milestone changed from 2.2.3 to 2.3

#15 in reply to: ↑ 13 @westi
17 years ago

Replying to nbachiyski:

Do we really need the bitwise operators? All the numbers are 4-byte ints anyway.

The problem is that on a 64bit platform with a PHP compiled for 64bit (not a 32bit PHP running on a 64bit platform) the php int is 8 bytes instead on 4 bytes.

The bitwise operators are there to mask off the top of the 8-byte int on 64bit platforms.

I have tested the patch gettext-64-without-bitwise-ops.diff and it works on:

  • Linux x86_64, PHP 5.2.3
  • Linux i686, PHP 5.2.3
  • Linux i686, PHP 4.3.10
  • FreeBSD i386, PHP 4.4.0

what it the value of PHP_INT_SIZE on the x86_64 PHP install - I expect it is 4 which why all your tests worked.

#16 @nbachiyski
17 years ago

Nope, westi, PHP_INT_SIZE is 8. And in that case $magic is equal to $MAGIC1, whihc is strange. However because of the 64-bit zero offsets we we have $MAGIC3, so we shouldn't care about removing the zeroes.

Maybe we should find more testers.

#17 @foolswisdom
17 years ago

  • Milestone changed from 2.3 to 2.4

#18 @ophy
16 years ago

  • Cc lapo@… added

(on the behalf of my friend lapo@…)

On my 64 bit server, with release 2.3:

magic=ffffffff950412de
MAGIC1=ffffffff950412de
MAGIC2=ffffffffde120495
MAGIC3=950412de

So the fix with MAGIC3 doesn't work and MAGIC1 would work if it was not truncated to 32bit itself.
I've made it working with the following patch:

--- wp-includes/gettext.php.orig        Mon Oct 22 22:35:56 2007
+++ wp-includes/gettext.php     Mon Oct 22 22:50:38 2007
@@ -113,7 +113,7 @@
                $MAGIC3 = (int) 2500072158;

                $this->STREAM = $Reader;
-               $magic = $this->readint();
+               $magic = $this->readint() & 0xFFFFFFFF;

                if ($magic == ($MAGIC1 & 0xFFFFFFFF) || $magic == ($MAGIC3 & 0xFFFFFFFF)) { // to make sure it works for 64-bit platforms
                        $this->BYTEORDER = 0;

#19 @thenlich
16 years ago

This bug is still present in Wordpress 2.3 on Linux/amd64/PHP 5.2.3

It worked for me to remove the & 0xFFFFFFFF's. Solution is described here: http://faq.wordpress-deutschland.org/wieso-zeigt-wordpress-trotz-sprachdatei-englischen-text-an/

#20 @thenlich
16 years ago

Addition: Bug still present in 2.3.1 also.

#21 follow-ups: @nbachiyski
16 years ago

Could everybody, please, try the both patches and report if any is working.

#22 in reply to: ↑ 21 @qb
16 years ago

Replying to nbachiyski:

Could everybody, please, try the both patches and report if any is working.

Both patches not working for me on Wordpress 2.3.1 / Linux 64 / PHP 5.2.4

#23 in reply to: ↑ 21 ; follow-up: @nyuwec
16 years ago

Replying to nbachiyski:

Could everybody, please, try the both patches and report if any is working.

I have a gentoo linux on a 64 bit AMD processor, described in: #5356, fighting with the above described problem. PHP using PHP_INT_SIZE=8.

Apply 3780.diff: FAILS.
Apply gettext-64-without-bitwise-ops.diff: WORKS!

As I can see the gettext-64-without-bitwise-ops.diff patches the gettext to the format described in: http://wordpress.org/support/topic/63038 That patch also works on both 64bit and on 32 bit systems I can test!

If only the 3780.diff applied to a 32 bit system it still works, applying the gettext-64-without-bitwise-ops.diff: still works!

So this code in the gettext.php (starting at line 105) works for me on both 32 and 64 bit systems:

		// Caching can be turned off
		$this->enable_cache = $enable_cache;

		// $MAGIC1 = (int)0x950412de; //bug in PHP 5.0.2, see https://savannah.nongnu.org/bugs/?func=detailitem&item_id=10565
		$MAGIC1 = (int) - 1794895138;
		// $MAGIC2 = (int)0xde120495; //bug
		$MAGIC2 = (int) - 569244523;
		// 64-bit fix
		$MAGIC3 = (int) 2500072158;

		$this->STREAM = $Reader;
		$magic = $this->readint();
		if ($magic == $MAGIC1 || $magic == $MAGIC3) { // to make sure it works for 64-bit platforms
			$this->BYTEORDER = 0;
		} elseif ($magic == ($MAGIC2 & 0xFFFFFFFF)) {
			$this->BYTEORDER = 1;
		} else {
			$this->error = 1; // not MO file
			return false;
		}

#24 @nyuwec
16 years ago

To my previous comment, here is a working version of the patched gettext.php on a 64bit AMD processor powered Gentoo Linux (jam 2.6.18-gentoo-r6) server with Apache2 and PHP 5.2.4:
http://koles.hu/turakraft/

It uses the hungarian language file, so it is mostly unreadable to the most of the world population. :)

#25 in reply to: ↑ 23 @madsjensen
16 years ago

This works !!! thanks ... I have tested many other hacks but only this seems to works.

Apache version 2.0.61
PHP version 5.2.4
MySQL version 5.0.45
Architecture amd64
Operating system: FreeBSD

Danish localization

cheers mads

Replying to nyuwec:

Replying to nbachiyski:

Could everybody, please, try the both patches and report if any is working.

I have a gentoo linux on a 64 bit AMD processor, described in: #5356, fighting with the above described problem. PHP using PHP_INT_SIZE=8.

Apply 3780.diff: FAILS.
Apply gettext-64-without-bitwise-ops.diff: WORKS!

As I can see the gettext-64-without-bitwise-ops.diff patches the gettext to the format described in: http://wordpress.org/support/topic/63038 That patch also works on both 64bit and on 32 bit systems I can test!

If only the 3780.diff applied to a 32 bit system it still works, applying the gettext-64-without-bitwise-ops.diff: still works!

So this code in the gettext.php (starting at line 105) works for me on both 32 and 64 bit systems:

		// Caching can be turned off
		$this->enable_cache = $enable_cache;

		// $MAGIC1 = (int)0x950412de; //bug in PHP 5.0.2, see https://savannah.nongnu.org/bugs/?func=detailitem&item_id=10565
		$MAGIC1 = (int) - 1794895138;
		// $MAGIC2 = (int)0xde120495; //bug
		$MAGIC2 = (int) - 569244523;
		// 64-bit fix
		$MAGIC3 = (int) 2500072158;

		$this->STREAM = $Reader;
		$magic = $this->readint();
		if ($magic == $MAGIC1 || $magic == $MAGIC3) { // to make sure it works for 64-bit platforms
			$this->BYTEORDER = 0;
		} elseif ($magic == ($MAGIC2 & 0xFFFFFFFF)) {
			$this->BYTEORDER = 1;
		} else {
			$this->error = 1; // not MO file
			return false;
		}

#26 @nbachiyski
16 years ago

  • Keywords commit added; 2nd-opition test removed

I got feedback from more than 10 Bulgarian users on 64-bit servers (Linux and various BSDs), experiencing the same problem. For all of them the second patch worked well, so I guess it is quite safe to be committed.

#27 @westi
16 years ago

  • Resolution set to fixed
  • Status changed from new to closed

(In [6488]) Avoid the 64bit dragons. Fixes #3780 props nbachiyski.

#28 @MichaelH
16 years ago

Confirming that [6488] fixes Version 2.3.2 installations on Bluehost.com servers when using a .mo language file.

If there is a 2.3.3, might consider this for that release.

From the phpinfo:

`System Linux box288.bluehost.com 2.6.22-9_1.BHsmp #1 SMP Fri Sep 28 23:36:16 MDT 2007 x86_64

Build Date Jan 10 2008 03:11:57
`

#29 @Nazgul
16 years ago

  • Keywords 2.3.3-candidate added

#30 @ryan
16 years ago

(In [6708]) Avoid the 64bit dragons. Fixes #3780 for 2.3 props nbachiyski.

#31 @ryan
16 years ago

  • Milestone changed from 2.5 to 2.3.3

#32 @swissspidy
2 weeks ago

In 57763:

I18N: Cast magic MO marker number to integer.

In gettext, 0x950412de is used to signal GNU MO files. In WP_Translation_File_MO this magic marker is used to detect whether a file uses little endian or big endian.

On 32 bit systems, this number will be interpreted by PHP as a float rather than an integer. This change adds extra casting to force an integer.

A similar change was done in the pomo library in the past, see #3780.

Props tmatsuur, swissspidy.
Fixes #60678.

Note: See TracTickets for help on using tickets.