ThinkRank: an API-first suite that files its own bugs

The problem

ThinkRank is an SEO plugin, which means most of what it does is invisible on the page. It writes meta tags, schema, sitemaps, robots rules and canonical URLs, and it exposes a large REST surface behind a React admin. It also ships as two plugins, free and Pro, in two separate repositories, where a fix might land in either.

Testing it by clicking around the admin would verify the least important layer. What matters is what ends up in the document head, what the endpoints return, and whether the Pro features stay behind their guards.

The constraints

• Two repos, one product. A branch under test might exist in the free repo, the Pro repo, or only one of them.
• The value is in the output, not the UI. Head tags, schema blocks, sitemap and robots responses.
• The admin is React and chatty. Each screen fires several API calls on mount.
• Authorisation is a feature. Pro capabilities and license guards are part of what needs asserting, not incidental.

The decisions, and what I rejected

Test the API and the rendered output, not the admin UI

The 43 specs split by what they actually verify: 8 API specs, 18 free-feature specs, 12 Pro specs, 3 front-end output specs, plus a dashboard spec and a smoke test.

The front-end group is the point of the suite. head-output, seo-output and block-schema assert what a crawler would see. The API group covers settings management, security, the migration wizard and the free tooling endpoints directly, without going through a screen that might be broken for unrelated reasons.

Authorisation gets its own specs rather than being assumed: authz, ai-tools-guards, apply-guards, license-guards. On a plugin with a paid tier, “does the Pro feature refuse to run without a license” is a functional requirement.

Run against a real WordPress site, not a container

Unlike the other three suites, this one points at a real local WordPress install served over HTTPS with a self-signed certificate. Nothing about the target is hardcoded: the base URL comes from the environment, and the suite runs against any WordPress with ThinkRank Free and Pro active.

The trade is explicit. A container gives a clean, reproducible environment; a real site gives a realistic one, with a real theme and a real web server, which matters when the assertions are about what appears in the rendered head and whether /robots.txt actually serves. Finding number two below could not have surfaced any other way.

Let the worker count be evidence, not a default

Two workers. The reasoning is written into the config: a single local WordPress backend has limited request concurrency, the React admin screens each fire several API calls on mount, and more than two workers saturates the backend until those calls stall. At two, the suite runs clean without needing retries to paper over it.

There is one retry locally as well as two in CI, and that is also written down: the dashboard’s async data calls can lose a race against a busy local backend. Naming the reason keeps a retry setting from quietly becoming a place where real failures hide.

Sweep fixtures globally, after a bug taught me to

Cleanup runs once in globalTeardown, after every worker has exited. It started as a per-worker afterAll sweep, which raced live tests, because with fullyParallel on, Playwright splits a single spec file across workers, so one worker finishing a file does not mean the file is done. Moving the sweep to global teardown fixed it, and it doubles as recovery for fixtures left behind by a crashed run.

What it cost, and what it caught

Cost. 71 commits between May and September 2026, and the most actively maintained of the four suites.

Caught. Building the suite produced FINDINGS.md, a documented defect log: six issues, two of them High, each with the file and function, a reproduction, the expected behaviour and a severity, routed to the correct repo since free and Pro are separate codebases. The two worth naming:

• /robots.txt returned 404 while a physical robots.txt existed. ThinkRank had written a real file to the web root, and WordPress only registers its virtual robots.txt rewrite when no physical file is present, so WordPress did not serve it and the plugin’s robots_txt filter never ran either. Search engines read a 404 as “no rules” and ignore the directives entirely, so every configured crawl rule and the sitemap hint were silently inert. That is an SEO plugin defeating itself, on the one file that is pure SEO.
• A Global SEO title template that never reached the browser when a theme filters pre_get_document_title later in the chain. The setting saves, the admin shows it applied, and the page ships someone else’s title.

Both are failures that a passing admin screen would have hidden completely.

The CI workflow takes a free ref and a Pro ref and resolves each against its repo. When a fix branch exists in only one of the two, which is the normal case, the workflow warns in the run summary and falls back to that repo’s default branch instead of failing, and the Slack message records which build was actually tested. Testing a pre-release branch is a form input, not a checkout dance.

What I deliberately did not automate

• Search engine behaviour. The suite asserts what ThinkRank emits. Whether Google honours it is not a testable proposition inside a spec.
• Live Google integrations. Search Console and the URL Inspection API are exercised through their endpoints and guards, not by depending on a live third-party account during a run.
• Ranking outcomes. The rank tracker’s plumbing is covered; actual positions are external reality.
• Cross-browser and mobile. Chromium only, desktop only. The output being asserted is markup, which does not vary by browser.