Testing PHP extensions - what makes a good test

March 22, 2015 at 3:04 PM - by Freek Lijten - 1 comment

Tags: PHP, PHP internals, open source

As an example I will use the enchant extension again. This extension basically helps you to find if words are in a dictionary. It can also offer suggestions for misspelled words, it offers one to start a personal list of words (useful to add new words to a dictionary, perhaps you're writing in your native tongue but do not have replacements for certain English words.) and one can also setup a list of words for the duration of a session. This is an enchant session in this case.

Our target for today is going to be the enchant_dict_add_to_session function. This allows you to add words to a dictionary for the duration of this spell-checking session. First let us have a look at the coverage report of this function in C:

Looking at this code it is clear that nothing is hit by a test at all. We may assume the PHP_ENCHANT_GET_DICT macro is tested trough other functions but the fact that it is called inside this function is not asserted. There are several ways of looking at the concept of a well tested function. What I would like to convey today is that 100% code coverage is one of them and it is not a good one.

Code coverage is not a goal

Repeat it with me one more time "Code coverage is not a goal". Simply covering all lines of code in a function is not enough. Lets look at a test that covers almost all lines of our function (I am ignoring the RETURN_FALSE on line 858 for now) and have a look at why this is a good start, but not great:

--TEST--
enchant_dict_check() basic function test
--SKIPIF--
<?php
if(!extension_loaded('enchant')) die('skip, enchant not loaded.');

$tag = 'en_US';
$broker = enchant_broker_init();
if (!enchant_broker_dict_exists($broker, $tag))
    die('skip, no dictionary for ' . $tag . ' tag');
?>
--FILE--
<?php

$tag = 'en_US';
$broker = enchant_broker_init();
$dictionary = enchant_broker_request_dict($broker, $tag);

enchant_dict_add_to_session($dictionary, "soong");
var_dump(enchant_dict_is_in_session($dictionary, "soong"));

enchant_broker_free_dict($dictionary);
enchant_broker_free($broker);
?>
--EXPECTF--
bool(true)

Sorry for the format, I have no syntaxhighlighter for phpt files :D

The line below TEST is nothing more than a title. This test makes sure it only runs if the dictionary can be loaded in the first place, you see this in the SKIPIF section. Then the FILE section defines our actual test.

As you can see we simply bootstrap the dictionary and add a word to it. After checking to see if it was truly added the resources are freed and we enter the result part of our test. In the EXPECTF section we basically expect the output of the previous section. In this case the output of a var_dump on a boolean true.

This will result in a coverage report with all lines of the function called (except for the aforementioned RETURN_FALSE line). But is this a good test? I would argue for no. There are at least two uncertainties left after this test:

1. There is no way to tell if a word is in the dictionary because we added it (and thus no way of telling adding works in the first place) or if it was always there in the first place.
2. We do not check if this word is truely added for the session only.

So even though our coverage report may now look something like below this should not lead to the conclusion that our function is well tested.

Testing "the spirit" of the function

Basically to address the earlier mentioned problems we should:

1. start a spell-checking session
2. Check if a word that is non-existent is actually NOT in the dictionary
3. Add the non-existent word
4. Check if it does exist now
5. Close the spell-checking session
6. Check if the word is again NOT in the dictionary

Below is the accompanying test. For brevity we only show the PHP code part. The rest should be obvious based on the previous test example.

$tag = 'en_US';
$broker = enchant_broker_init();
$dictionary = enchant_broker_request_dict($broker, $tag);

// Word should not be in session yet.
var_dump(enchant_dict_is_in_session($dictionary, "soong"));

// After adding we should be able to find it.
enchant_dict_add_to_session($dictionary, "soong");
var_dump(enchant_dict_is_in_session($dictionary, "soong"));

// Close session. Then the added word should be gone again.
enchant_broker_free_dict($dictionary);
enchant_broker_free($broker);

$broker = enchant_broker_init();
$dictionary = enchant_broker_request_dict($broker, $tag);
var_dump(enchant_dict_is_in_session($dictionary, "soong"));

// Final close
enchant_broker_free_dict($dictionary);
enchant_broker_free($broker);

This test tackles both problems mentioned earlier and does for more for actually testing the function than just calling the function once. So does this mean code coverage is useless? Not at all. It is very usefull to see what is most certainly not tested at all. Just remember that on top of that you have to think on how a function is meant to be used to test it properly.

I hope this helps some of you to understanding test even better and triggers you to write more and better tests. As always do not forget to drop by in #gophp7-ext on freenode and have a look at the gophp7-ext projects website.