Using the WordPress Test Suite

Writing automated tests for your WordPress project is a must in order to verify that your code works as expected. Of course you should always do severe manual testing for your plugin or theme, but as always, humans aren’t as precise and thorough as computers can be with that. Furthermore having sufficient automated tests (i.e. solid test coverage for your code) also indicates whether a subsequent change, as in a later release, unexpectedly breaks something you wouldn’t have detected otherwise. This post gives you an introduction on the test suite that WordPress core includes, which you can also use to test your plugin for example, but of course too if you’re contributing to WordPress core.

If you haven’t learned the basics of automated testing yet, please be aware that this is not a general introduction into writing tests, but a more specific introduction into writing PHP tests with the WordPress core test suite (WordPress also has some QUnit tests for JavaScript, but that is another topic that goes beyond this post). The term you’ve most probably at least heard about is “unit tests”, however this is just one type of automated tests that you can write. In order to test your project thoroughly, you also need to take the other types into account, such as “integration tests”, “acceptance tests”, “system tests” and “behavior tests”. If you wanna learn more about these different kinds of automated software tests, TestLodge have a good quick overview about which ones there are. For great introductory resources on the most basic kind of tests, “unit tests”, I recommend the following:

If you aren’t familiar with the different kind of tests and their basics, I recommend you read at least one of the above articles before proceeding with this one.

Before we get into the WordPress test suite, I’d like to highlight that, while WordPress uses PHPUnit for its tests, most of them aren’t actually unit tests, but rather integration tests. They require the entire WordPress codebase to be loaded and test how one or more specific functions integrate with each other, in order to verify things work as expected. Writing a unit test shouldn’t require WordPress to be loaded in its entirety, but instead only require the function or class to be tested to be loaded, while mocking the behavior of its dependencies if there are any. Note that you can possibly write a unit test with the WordPress test suite if the function is a closed unit that doesn’t call any other dependency that requires testing, but that is rarely the case. The above articles look at these differences a bit more closely, if you are interested. That being said, integration tests are just as important, and the WordPress ~~unit~~ test suite provides a good toolset for writing those.

How to setup the WordPress test suite

If you contribute to WordPress core and use the development version of WordPress, its codebase already contains the test suite: The actual WordPress codebase resides in the src directory, while the PHP test suite is located in tests/phpunit. The tests themselves are located in an extra tests/phpunit/tests subdirectory, grouped by their respective WordPress component. To write a new test, you simply need to add a new method to one of the classes or, if your test covers an area previously untested, create a new file for an extra test class and add the method there. Note that all method names that should represent tests must be prefixed with test.

If you wanna test a plugin, the easiest way to do so is to use WP-CLI to scaffold the tests for you: There is a command wp scaffold plugin-tests, and the WP-CLI handbook has a great introduction on how to use it. After running the command, there is already a file with a sample test in place which you can tweak to get started quickly. You can add as many classes and methods as you like, again make sure to prefix testable methods with test. If your plugin needs to perform some special setup logic, such as install additional database tables, you need to add that logic to the tests/bootstrap.php file. In some cases, if your plugin uses an activation hook for the functionality, adding a call activate_plugin( 'my-plugin/my-plugin.php' ); can be sufficient.

There’s also a more elaborate tutorial by Pippin Williamson on setting up the test suite for plugins.

How the WordPress test suite works

Every test class should inherit from the WP_UnitTestCase class, which provides the base functionality for your tests. That class in turn inherits from PHPUnit’s PHPUnit_Framework_TestCase class, so that you also have all standard PHPUnit features, such as assertions, available. There are several key features of the WP_UnitTestCase class that you should be aware of when writing unit tests, in order to make them efficient and prevent unnecessary bloat:

Any database queries that are run during a test are reverted afterwards. This allows you to generate database objects such as posts or terms on the fly for a test without possibly polluting subsequent tests with that data. Even new tables you create (like when creating a new site in a multisite network) won’t persist after the test has completed. Technically this works by preventing the database from auto-committing changes before each test and running a rollback query after each test. What this essentially means for writing tests is that you don’t need to clean up after yourself when modifiying the database in any way.
Any hooks that you add or remove for your test are reverted back to the original state afterwards. Before running a test, the $wp_filters global (which stores all registered actions, filters and their hooks added) is temporarily stored elsewhere as a backup. After running each test, the original global is set to the backup value again, efficiently restoring the original state. So if you add an action or filter in your test, don’t bother removing it afterwards – again, you don’t need to clean up after yourself here.
Any $_GET or $_POST values you set in your test are also ignored in subsequent tests, as the two globals are always reset to an empty array after each test. Yet another thing you don’t need to worry about cleaning up.
Any global variables related to running a query and going through the loop for its results are unset after each test. So if you need to run a WordPress loop (with the common functions like have_posts() and the_post()), don’t worry about cleaning up the globals set by those functions either.
If you set any specific user as the current user (for example with wp_set_current_user()), don’t bother setting that one back to 0 at the end of your test – again, the WordPress test suite already handles it.
In case you’re writing tests for WordPress core itself, you also don’t need to bother cleaning up after you registered a new post type, taxonomy or post status, or unregistered one. These are reset back to the default core ones after each test has been run. However, keep in mind that this is only enabled if core tests are run. For your plugin’s tests, you need to clean up temporary object types and statuses you create (but that probably happens rarely in a plugin test anyway).
If the tests in your class require some data that is similar across multiple tests, it makes sense to create that data only once for the entire class instead of doing it per test. This improves performance of the tests and decreases the amount of code you need to (re-)write. Particularly expensive actions, such as creating sites in a multisite network, should preferably only happen per class and not per test (unless it really has to happen inside the test). You can add such data by adding a wpSetUpBeforeClass() method to the class containing the relevant tests, and then adding the logic to create your database objects there. Note that anything you do in here will be committed to the database, unlike the database transactions that happen from inside a test, as noted earlier. This needs to happen so that the data is available for each test in the class. The logical consequence of that is that you need to clean this data up after all tests of your class have been run. You can do that by adding a wpTearDownAfterClass() method where you restore the exact state that was in place before your wpSetUpBeforeClass() method was called. It’s therefore important to always have both methods in place if you need to setup reusable database objects, never just one of them.

As you can see, the WordPress test suite does a good job reducing your work when writing tests, as there are countless occasions where you don’t need to clean up after yourself although you commonly would need to. When looking at some of the existing core tests, you will certainly find cases where a temporarily added filter is removed again or similar – however this is the result of the person who wrote the test not being aware of the fact that the test suite already takes that work away from them. Things like these are one of the primary reasons why I started writing this post, because the features of the WordPress test suite haven’t been well documented so far. We can possibly also merge some of this information into a handbook page to provide a more central access point for this.

Assertions

In addition to the common PHPUnit assertions, the WordPress test suite provides a few additional assertions you can use:

assertWPError( $actual, $message = '' ): Reports an error identified by $message if $actual is not a WP_Error instance.
assertNotWPError( $actual, $message = '' ): Reports an error identified by $message if $actual is a WP_Error instance.
assertIXRError( $actual, $message = '' ): Reports an error identified by $message if $actual is not a IXR_Error instance (error class for XML-RPC requests).
assertNotIXRError( $actual, $message = '' ): Reports an error identified by $message if $actual is a IXR_Error instance (error class for XML-RPC requests).
assertEqualFields( $object, $fields ): Reports an error if $object does not have properties for all keys of the $fields array with the respective values of the $fields array.
assertDiscardWhitespace( $expected, $actual ): Reports an error if the two variables $expected and $actual are not equal after stripping all whitespace out of them.
assertEqualSets( $expected, $actual ): Reports an error if the two arrays $expected and $actual do not have the same key-value pairs regardless of their order, by sorting by value.
assertEqualSetsWithIndex( $expected, $actual ): Reports an error if the two arrays $expected and $actual do not have the same key-value pairs regardless of their order, by sorting by key.
assertNonEmptyMultidimensionalArray( $array ): Reports an error if $array is an empty multidimensional array, which means that each of its sub-arrays must not be empty.

Factories

When creating database object types for your tests, you should always use the built-in factories the WordPress test suite provides. For example, use self::factory()->post->create( $args ) instead of wp_insert_post( $args ). The test suite provides the following factory classes:

WP_UnitTest_Factory_For_Post (instance accessible by self::factory()->post)
WP_UnitTest_Factory_For_Attachment (instance accessible by self::factory()->attachment)
WP_UnitTest_Factory_For_Comment (instance accessible by self::factory()->comment)
WP_UnitTest_Factory_For_User (instance accessible by self::factory()->user)
WP_UnitTest_Factory_For_Term (instance accessible by self::factory()->term)
- WP_UnitTest_Factory_For_Term specific to categories (instance accessible by self::factory()->category)
- WP_UnitTest_Factory_For_Term specific to tags (instance accessible by self::factory()->tag)
WP_UnitTest_Factory_For_Bookmark for the old wp_links database table (instance accessible by self::factory()->bookmark)
WP_UnitTest_Factory_For_Blog, multisite-only (instance accessible by self::factory()->blog)
WP_UnitTest_Factory_For_Network, multisite-only (instance accessible by self::factory()->network)

Note that the factory container instance that you commonly access via self::factory() is passed to your wpSetUpBeforeClass() method as the first parameter if you have one, so from in there you can simply access it that way.

Every factory instance has the following methods available which you can use to create new objects of their respective type:

create( $args = array(), $generation_definitions = null ): Creates a new object and returns its ID from the database.
create_and_get( $args = array(), $generation_definitions = null ): Creates a new object and returns its full object (such as a WP_Post instance).
create_many( $count, $args = array(), $generation_definitions = null ): Creates multiple new objects and returns their IDs in an array.

Note that the optional $generation_definitions parameter is rarely useful to provide, since the most common basic settings for each object are defined by the respective factory class and are used by default. If your plugin registers its own post types or taxonomies, it can help to implement your own factory classes. For example, a factory class for your own post type could simply extend the WP_UnitTest_Factory_For_Post, and you could adjust the $default_generation_definitions property to set the post_type to your custom post type instead of the default 'post'. In order to easily access an instance of your factory, you could extend the factory container WP_UnitTest_Factory and add your own as a property, and then also create your own test base class inheriting from WP_UnitTestCase – actually, writing a custom test base class is always a good idea, even if you don’t have anything to add to it yet, as it will allow you to tweak things later without having to adjust all test classes (as they would need their parent class changed). Getting more into detail about this is however an additional topic that I might write another post about. For now, see the code of the existing factories for an overview of how to implement and use your own ones.

Miscellaneous

Last but not least, here are some more things about the test suite which are good to know, but do not fit into one of the above sections:

As you may know, you can use @group annotations in PHPUnit for your tests, so that you can run specific tests only if you like, instead of all ones. For example, if you have a group annotation @group my-group, you can limit the test suite to only run those tests by calling PHPUnit with an extra argument like phpunit --group my-group. The WordPress test suite uses this as well of course, so as soon as your plugin gets bigger than one or two classes, you should probably group your tests so that you can run specific ones only when needed.
A special case for a group where the WordPress test suite applies custom logic is the group ms-required. If you add an annotation @group ms-required to a test method, this test is only run if the tests are run in a multisite setup. Similarly, if you add @group ms-excluded to a test method, this test is only run if the tests are not run in a multisite setup.
To run the tests in a multisite setup, you need to use an extra argument when calling PHPUnit that indicates the multisite configuration should be used. When running tests for WordPress core, you can simply do phpunit -c tests/phpunit/multisite.xml in order to do so. If you are running tests for your own plugin, you first need to create the proper configuration file: Copy your original phpunit.xml file, for example into the tests directory and rename it to multisite.xml. Then add the following above the line that says <testsuites>:
```
<php>
    <const name="WP_TESTS_MULTISITE" value="1" />
</php>
```
Save the file, and then you can run phpunit -c tests/multisite.xml to run tests in a multisite setup.
You can also test functions that are deprecated. In order to do so, you should add an @expectedDeprecated annotation to the test method, followed by the function name that triggers the deprecation notice. If you for example test wp_get_sites(), add @expectedDeprecated wp_get_sites to the test method.
The same applies to testing functions in a way that a _doing_it_wrong() notice is commonly triggered. If you explicitly want to call the function in that wrong way, add an @expectedIncorrectUsage annotation to the test method, followed by the function name that triggers the notice. If you for example call load_plugin_textdomain( $domain, '/languages' ) (the second parameter is deprecated), add @expectedIncorrectUsage load_plugin_textdomain to the test method.

Takeaways

Assuming you are familiar with writing automated tests, you should hopefully be able to write solid tests based on the WordPress test suite now, using its features and benefits correctly. If you’re fairly new to writing these tests, looking at existing ones is always a good idea. While there are cases which haven’t been solved in an optimal way, reading the code will generally give you a better idea of what tests can look like.

There’s one last recommendation I have: Note that sometimes writing tests can be complicated, especially for WordPress core where some functions are very closely interrelated. This indicates that the code itself has probably not been written properly, as it’s not well testable – but as we all know, we can’t just tweak the codebase completely, so try to think out-of-the-box, be tricky, and sometimes you find a way to test even something that you thought before wasn’t testable.