EmbedPress: the day 38 green tests turned red

The problem

EmbedPress turns a pasted URL into an embed. It claims support for around ninety providers, and each one has to work in two editors, Gutenberg and Elementor. That is the whole product surface, and it breaks silently: a provider changes an endpoint, the plugin still renders a tidy-looking wrapper, and nothing throws an error. The page just quietly shows nothing where a video should be.

The first version of this suite had that failure mode built into it. Specs asserted that EmbedPress produced its wrapper element and that the source URL appeared somewhere on the page. Every spec passed. The suite was green and it was not telling the truth, because a wrapper with no iframe inside it is exactly what a broken integration looks like.

On 9 May 2026 I rewrote the assertions to require the actual embed. An iframe, an image, or a blockquote carrying a unique marker from the source URL, inside the EmbedPress figure or widget. The suite went from green to roughly 155 passing, 38 failing and 1 skipped.

Those 38 were not test defects. They were EmbedPress integrations that had been broken the entire time, behind assertions too loose to notice.

The constraints

Fixed before any design decision:

• The matrix is the product. 91 of the 92 sources in sources.json carry a URL, and each needs coverage in both editors. LottieFiles has no URL and is not seeded.
• Some sources are API-gated. YouTube Channel and YouTube Live (Channel) render an “enter your YouTube API key” placeholder without a key. OpenSea falls back to a built-in demo key that can be rate-limited.
• Pro is a private repo. The free plugin can come from wordpress.org, but EmbedPress Pro has to be supplied from a local path or a CI checkout with a token.
• It has to run on someone else’s machine. Not just mine, and not just in CI.
• I maintain three other suites. Anything requiring constant hand-holding was not going to survive.

The decisions, and what I rejected

Verify the output, do not drive the editor

The obvious way to test an editor plugin is to automate the editor: open Gutenberg, insert the block, paste the URL, publish, assert. I rejected that for every one of the 225 specs.

Instead the seed pipeline writes the pages directly, and specs visit the finished URL at /ep-<editor>-<slug>/ and assert on what rendered. No editor automation runs at test time at all.

The reasoning, from the generator skill in the repo: re-driving the editor on every CI run is slow and brittle, welcome modals, command-palette overlays and Elementor panel races, and it adds nothing. If the embed renders on the seeded page, the integration is healthy. Editor coverage belongs in plugin-level tests, not in this harness.

That single decision is why 225 specs are maintainable by one person alongside three other suites.

Seed by SQL, not by clicking

seed/index.ts reads sources.json and generates SQL, which is piped straight into MySQL inside the DB container. For Gutenberg it builds <!-- wp:embedpress/embedpress --> block markup; for Elementor it builds the _elementor_data JSON for the widget. Re-runs are idempotent, previously seeded pages are wiped before insertion, and you can reseed one source or one editor rather than the whole set.

There is one thing direct SQL cannot do. Several dynamic providers, Instagram, OpenSea, Wistia and Google Photos, have a block render callback that only honours the embedHTML attribute. Insert the row and the page renders nothing. So after seeding, scripts/resolve-gutenberg-embeds.php runs through wp eval-file, resolves the [embedpress] shortcodes and bakes the iframe HTML into both the block’s inner content and its embedHTML attribute, matching what the editor itself would have saved. The shortcut around the editor needed one deliberate step to stay faithful to it.

No retries

retries: 0. A failed spec is not attempted again, locally or in CI.

Retries are the standard answer to flake, and they are how a suite goes back to lying. This suite exists because its assertions were once too generous. Adding automatic retries on top of strict assertions would have re-introduced the same problem in a different place: a test that passes on the second attempt is a test whose result I cannot read.

The cost is real. It means flake has to be diagnosed rather than absorbed, and one genuinely resource-related flake is still open, below.

Let CI go green on failing tests

The Playwright step runs with continue-on-error: true. The job succeeds even when specs fail. That looks wrong and is deliberate: 38 known-red specs would otherwise mean a permanently red pipeline, which people stop reading within a week. The signal moves to the artefacts instead. The HTML report publishes to GitHub Pages, and a Slack message carries the result with a link to it. The workflow comment says plainly that the job will be green even on test failures and to check the report or Slack for status.

What went flaky, and what actually fixed it

Every one of these is a fix in the repo, not a workaround.

• The WordPress login form. WordPress 6.7 added a password visibility toggle whose autocomplete behaviour interfered with a normal fill(). global-setup.ts now sets el.value directly through evaluate, which native form submission reads at submit time. One UI login per run, stored to .auth/state.json and reused by every spec.
• Mojibake in seeded titles. The seed SQL was being interpreted as latin1. Fixed by forcing --default-character-set=utf8mb4 on the MySQL connection.
• Plugins that would not delete. WP-CLI runs as root so plugin files were root-owned, while Apache runs as www-data. Dashboard deletes failed with “Could not fully remove the plugin”. Fixed with a chown -R www-data:www-data on wp-content after setup, in both the local script and CI.
• Seed IDs colliding with WordPress. SEED_ID_START had to move from 1000 to 10000, because WordPress creates auto-drafts, wp_global_styles and wp_navigation posts at IDs from 1184 upward, which collided once the YouTube Channel variants pushed the seed range past 1183.
• Still open. In a run of 90 minutes or more, elementor/google-forms and elementor/nrk-radio have flaked while passing cleanly in isolation. The evidence points at Chromium and container resource pressure rather than a test defect. It is written down as unresolved rather than patched over with a retry.

What it cost, and what it caught

Cost. 225 spec files: 121 Gutenberg, 103 Elementor, 1 smoke. Four parallel workers, a 90-second per-test timeout for slow Elementor UI, and a 90-minute CI job ceiling. A full pass takes 20 to 25 minutes, and most of that is failing specs sitting on their 15-second strict timeout. Dropping that timeout to 5 seconds would cut the run to roughly 12 minutes, at the risk of false negatives on sources whose vendor JS genuinely needs a moment. That trade is written down and not yet taken.

Caught. The suite’s job is to name broken integrations precisely, and it does:

• GitHub Gist renders a wrapper and never emits the gist.github.com script tag the browser needs. The fix is one tag.
• CodeSandbox, BeautifulAI, Matterport, ActBlue, Gloria TV all produce a wrapper and no iframe, despite having well-known embed URLs the provider should be generating.
• OpenSea Collection and Single render the wrapper with no content, because the API fetch never completes.
• Audiomeans, SmugMug, 23hq, Gumroad, Byzart, Animoto do not even get a wrapper. The URL is left as plain text inside the figure.
• SocialExplorer emits <iframe marginmarginsrc="..."> where it should emit src. That is a string-replace bug in the provider. The spec asserts on raw HTML to work around it, with a note to switch to a normal selector once it is fixed upstream.

Each one is triaged by impact in the repo, so the list is a work queue rather than a complaint.

The same harness also drove the WordPress 7.0 RC compatibility pass, which produced 20 written reports on individual sources, including a YouTube Error 153 case and a PDF Gallery block bug.

What I deliberately did not automate

Written down in coverage.md as architecture, not backlog:

1. Editor flows. Block insertion, widget drop, control toggles, save round-trip. Belongs in plugin-level tests.
2. Most Pro controls. Autoplay, loop, colour, end time. The exception is the YouTube Channel layout matrix, which has real variant coverage.
3. Player interactivity. Whether a video actually plays. Asserting on third-party player JS buys unreliability, not confidence.
4. Cross-browser. Chromium only.
5. Mobile viewports. No responsive project configured.
6. Failure modes. Invalid URLs, expired oEmbed responses, network errors.
7. Authentication and capabilities. Admin storage state only.
8. Source-specific API content. OpenSea NFT cards, Instagram Pro feeds, YouTube Live channel content when nothing is live. API-gated and not deterministic.

The through line is that I did not automate anything whose result I could not trust. A test that fails because YouTube had no live stream that morning teaches nobody anything, and it trains the team to ignore red. Given the suite exists because its own assertions were once too weak to believe, an untrustworthy test is the one thing it cannot afford.