[php-maint] Fixes for the PHP unit tests

[Thomas Goirand]

On 05/14/2012 07:45 PM, Ondřej Surý wrote:
> On Mon, May 14, 2012 at 1:10 PM, Thomas Goirand <zigo at debian.org> wrote:
>> On 05/14/2012 05:11 PM, Ondřej Surý wrote:
>>> Thomas,
>>>
>>> just to be sure. Have you tried your patches against vanilla PHP?
>>>
>>> O.
>>
>> I did, and they apply cleanly. Only one of the hunks has a line offset
>> thing...
> 
> Not speaking about applying, but about results of the tests.

As I wrote previously, out of about 703 tests in ./tests, only about 20
are failing, and 683 are working well after my patch. Now, while doing a
build of php5 from the sources in the git on Alioth, I've noticed that
these tests ARE executed, they do fail (of course, as they are totally
outdated), but the build continues... That's not really what I expected. :/

But I also found out that there's in total more than 11000 tests in
php5. For sure it'd be great to have:
- test results used for something (eg: fail the build if they fail)
- tests patched so they actually work with current PHP

Otherwise, we should just skip the tests, as currently, it's only a
waste of build time...

>> BTW, why is the php package still using format 1.0? Would it be ok to
>> switch to format 3.0 (quilt)? That'd be more easy to manage patches
>> (with dpkg-source --commit). Is it because the clean target doesn't
>> clean properly?
> 
> It would and patches would be welcomed.

Ok, I'll try to do it then.

>> I'm currently trying to add the tests in the build, but one thing I'm
>> not sure how to manage is that PHP might then build-depends on itself.
>> Indeed, if we need phpunit to build, then phpunit depends on php. That
>> might be very annoying for porters. Any thoughts on this?
> 
> Could you maybe elaborate more on what phpunit is and why it would be
> needed to build php itself?

phpunit is a unit test framework. You give it some .phpt files, and then
it tests, and compare what the script should output, and what it
actually writes on stdout. Here's the first test in
./tests/basic/001.phpt in the php sources:

--TEST--
Trivial "Hello World" test
--FILE--
<?php echo "Hello World"?>
--EXPECT--
Hello World

You would invoke it with:

phpunit tests/basic/001.phpt

If it succeeds, you have the following output:
PHPUnit 3.6.10 by Sebastian Bergmann.

.

Time: 0 seconds, Memory: 3.25Mb

OK (1 test, 1 assertion)

Now, let's say I replace "Hello World" by just "Hello", here's the output:
PHPUnit 3.6.10 by Sebastian Bergmann.

F

Time: 1 second, Memory: 3.25Mb

There was 1 failure:

1) 001.phpt
--- Expected
+++ Actual
@@ @@
-Hello World
+Hello


FAILURES!
Tests: 1, Assertions: 1, Failures: 1.

But it can be more complex. For example, you can define some php.ini
values, skip the test in some conditions, add values in the $_POST or in
the cookies, simulate a file upload, etc. For example (not a valid test,
I just wrote it to show you what's possible, with many directives...):

--TEST--
Comments (or title of the test if you want...)
--SKIPIF--
<?php
if (substr(PHP_OS, 0, 3) == 'WIN') {
    die('This test doesn't work on windows');
}
?>
--INI--
open_basedir=.
error_reporting=E_ALL | E_STRICT
--POST--
a=1&b=3
--FILE--
<?php
var_dump($_POST['a']);
?>
--EXPECT--
string(1) "1"

Many PEAR package embed some of these phpunit tests, which are then
testing some functionality. For example, in the PEAR package Mail
(package php-mail in Debian), there is Mail-1.2.0/tests/rfc822.phpt. You
can see the file in there:
http://anonscm.debian.org/gitweb/?p=pkg-php/php-mail.git;a=blob;f=Mail-1.2.0/tests/rfc822.phpt;h=71261ffb7d9d255cf979837608c3148800678943;hb=105b53dd3e215d21f9ede52a375a533ed12fcd3d

As you see, it's testing if the Mail package can properly parse RFC822
email address lists, just as you would expect.

Cheers,

Thomas