An issue with block editor e2e test cross-version compatibility

While I was making minor updates to my Attachment Taxonomies plugin (GitHub repository) the other day, I noticed that the e2e tests were failing when run against WordPress version 6.1. Since the plugin’s minimum requirement is that version, I still run tests against it, to make sure it keeps working as expected.

After some investigation, I noticed the issue was happening due to the way the block editor used to work prior to WordPress 6.3, when it started to be iframed by default. So the failure most likely wasn’t happening due to an actual bug in the plugin, but due to an issue with the e2e tests.

A note on the importance of e2e tests

Before I get to the solution, let me just quickly highlight the importance of e2e tests. If you’re a plugin author and you haven’t looked into writing e2e tests for your plugin(s) yet, I strongly encourage you to start doing so. For what it’s worth, I am not at all an expert in this, and the clunkiness of writing tests and flakiness of running them used to scare me away. But a while ago WordPress and Gutenberg changed to use Playwright for e2e tests, and it’s a difference like night and day. So especially if your plugin has slightly complex UI interactions or if you integrate with the block editor, having e2e tests coverage and running them for different WordPress versions in CI (e.g. GitHub Actions) is going to be a game changer in avoiding surprises where a plugin release suddenly doesn’t work in one of those versions anymore.

To get started, I recommend you take a look at some of Gutenberg’s e2e tests and read the documentation of the useful @wordpress/e2e-tests-playwright package.

Anyway, let’s go back to the problem I had run into.

Supporting the iframed and non-iframed block editor in e2e tests

When you look at Gutenberg’s e2e tests, you will notice that it’s typically using editor.canvas to access the locator from which you can further locate specific block content. For example:

await admin.createNewPost();

// Insert a new image block and ensure it's added.
await editor.insertBlock( { name: 'core/image' } );
const imageBlock = editor.canvas.locator( 'role=document[name="Block: Image"i]' );
await expect( imageBlock ).toBeVisible();Code language: JavaScript (javascript)

This test would already fail in WordPress versions below 6.3 because editor.canvas looks for the iframe with the block editor content – which is not present in those older versions (and sometimes not even in current versions, depending on which third-party blocks are used on the site).

In order to adjust this test to work for the non-iframed editor, I could simply replace editor.canvas with page, so that it tries to find the new image block directly within the current page, rather than within the iframe. Like this:

await admin.createNewPost();

// Insert a new image block and ensure it's added.
await editor.insertBlock( { name: 'core/image' } );
const imageBlock = page.locator( 'role=document[name="Block: Image"i]' );
await expect( imageBlock ).toBeVisible();Code language: JavaScript (javascript)

The problem with that approach though is that then it no longer works with the iframed variant of the block editor. So I needed to find a way to conditionally use editor.canvas or page. After some searching the web, I found an article that had a nice and simple solution: Use a promise with both locators, where whichever locator resolves first is then used. I took that approach and put it into a little helper function that I called oneOfLocators(). Here’s what that could look like for the previous code example:

function waitForLocator( locator ) {
	return locator.waitFor().then( () => locator );
};

function oneOfLocators( ...locators ) {
	return Promise.race( locators.map( waitForLocator ) );
}

await admin.createNewPost();

// Insert a new image block and ensure it's added.
await editor.insertBlock( { name: 'core/image' } );
const imageBlock = await oneOfLocators(
	editor.canvas.locator( 'role=document[name="Block: Image"i]' ),
	page.locator( 'role=document[name="Block: Image"i]' )
);
await expect( imageBlock ).toBeVisible();Code language: JavaScript (javascript)

It worked like a charm! And my Attachment Taxonomies unit tests are now passing again across all supported WordPress versions.

Maybe you have run into this problem, so maybe this is helpful to you. That said, I’m also curious if you have suggestions for an alternative solution? Potentially we could make the condition more specific, e.g. use editor.canvas on WordPress versions 6.3 or later while using page on WordPress versions older than 6.3. If you have ideas in that regard, please share them in the comments.

Potentially we can even improve this upstream in the @wordpress/e2e-tests-playwright package where editor.canvas is defined. A new utility that conditionally returns the correct locator could be helpful.