When QA Can't Be Linear: Testing Across the Deployment, Not After It

I am not a QA person. I am a software engineer and architect, and I end up doing QA for one reason: I can't leave a problem alone once everyone around me has agreed it can't be solved. This article is about one of those problems, and about the way most companies think about testing that made it look unsolvable in the first place.

The short version: most teams treat QA as a linear process. Something is built, something is deployed, and then a test runs and returns a verdict. For a large class of systems that model works. For some systems it quietly doesn't, and the teams running them often don't realize that the model, not the system, is what's failing.

The linear model of QA

Look at how testing is drawn in almost any delivery process. There is development, there is deployment, and somewhere along that line there is a box labelled QA. The box can move: shift-left puts it earlier, smoke tests put it right after the deploy, synthetic monitoring keeps it running after go-live. But the shape stays the same. QA is a separate, independent step that looks at the system at one point in time and decides whether it is good.

Two assumptions are built into that model, and they are rarely said out loud:

1. Each test is self-contained. It runs, it finishes, it produces a result. Nothing from an earlier moment is needed to interpret it.
2. The result is conclusive. If the release went live and the tests are green, everything live is fine.

Both assumptions depend on a third, even quieter one: that the test knows what the correct result looks like. An assertion like expect(price).toBe(49.99) only works because someone knew, in advance, that the price should be 49.99. Fixtures, seeded databases and mocks all exist to make that third assumption true.

When you can't know the expected result in advance, the whole model falls over. And there are more systems like that than people expect.

The system everyone said was impossible to test

At a company I used to work for, the website was driven by a content management system. Editors changed content all day: pages, components, product stories, promotional blocks. Nothing about that content was static, and it wasn't meant to be.

The content did not go straight from the CMS to the browser. It went through a BFF (backend for frontend) that processed it heavily: it reshaped it, merged sources, resolved references, applied rules and produced the payloads the frontend rendered. The BFF was real code with real logic, and it was exactly the kind of code that breaks in subtle ways. A refactor of a mapping function, a changed default, a new rule for an edge case, and some fraction of pages would render slightly wrong.

The release flow was the standard linear one: deploy the BFF, deploy the frontend, then run automated tests to check that everything was correct. And the conclusion across the company was that this part could not really be tested. I agreed with them, up to a point. It was impossible to test the linear way.

Think about what a post-deployment test has to decide. It loads a page and sees a hero banner, three content blocks and a product carousel. Is that right? It might be. An editor might have changed the banner ten minutes ago. The carousel might be shorter because a campaign ended. There is no fixed checkpoint to compare against, because the content that defines "correct" is a moving target. Hard-coding expectations breaks the tests every time an editor does their job. Loosening them until they stop breaking leaves you asserting that the page has "some blocks", which catches nothing.

In testing research this has a name: the test oracle problem. An oracle is whatever tells you the expected output for a given input. Our system didn't have one. Or rather, it did, and we weren't using it.

The insight: you don't need to know what's correct, only what changed

A deployment changes the code. It does not change the content. So if you look at what the BFF produces from a given piece of content just before the deployment, and again just after it, the content is (nearly) the same in both observations. Any difference between them comes from the code.

That turns an unanswerable question into an answerable one. "Is this page correct?" can't be answered without an oracle. "Did this deploy change how we process this content?" can, because the previous version of the system is the oracle. It ran in production yesterday, editors and customers were looking at its output, and that output is the best definition of "correct" available.

One test, two phases

So instead of one test that runs after the deployment, I split each test into two halves that bracket it:

1. Phase one, before deploying: probe a set of test points. A test point is a specific request to the BFF (a page, a route, a component payload) whose output matters. Record every response.
2. Deploy the new version.
3. Phase two, after deploying: probe exactly the same test points again and record the responses.
4. Compare each pair of responses, before and after.

Only at step four is there a verdict. Phase one on its own proves nothing. Phase two on its own proves nothing either; it is the same unanswerable post-deploy check as before. The quality signal only exists once both halves are in hand and compared. That is what I mean by non-linear QA: the test is not a box placed at one point on the line. It is a process with state that spans the deployment.

In practice the BFF was the place where this mattered most, because content was the biggest variable in the whole system. We checked how the BFF processed content before the deployment, deployed, checked again, and compared. If nothing had changed that we didn't intend to change, the release was good. If something had, we knew exactly which test point, which field and which page, and we could look at it before customers did.

What is this actually called?

Internally we called it a regression test. The name fitted the situation, but it isn't really accurate: classic regression testing re-runs existing tests against expected results derived from requirements. Before writing this article I went looking for what the industry calls the technique. It turns out it has relatives, and one close match.

Name	What it compares	How it relates
Back-to-back testing	Two or more variants of a system run with the same inputs	The closest formal name. The ISTQB glossary defines it as testing in which two or more variants of a component or system are executed with the same inputs, and the outputs are compared and analyzed when they differ.
Differential testing	Two implementations given the same input	Described by William McKeeman in 1998 for testing C compilers. It solves the oracle problem by using one implementation as the oracle for another.
Golden master / characterization testing	Current output against a stored, approved output	Popularized by Michael Feathers for legacy code. Same idea, but the baseline is long-lived, which breaks down when content keeps changing.
Diffy-style comparison	Live traffic sent to old and new versions at once	Twitter open-sourced Diffy in 2015. It runs the candidate next to two copies of the known-good version, so it can tell real differences from noise.
Automated canary analysis	Metrics from a canary against a fresh baseline	Spinnaker and Kayenta compare a canary with a baseline started at the same time, rather than with long-running production, for exactly the same reason: the comparison must share conditions.

So the honest answer is that what I built was back-to-back testing (or differential testing) of two versions of the BFF, against live content. What is specific about it is the timing: the baseline is not stored for weeks, and it is not produced by a parallel copy of the old version. It is captured from production immediately before the deploy, and the comparison happens immediately after. If I had to give it a precise name today, I would call it deployment-bracketed back-to-back testing. "Two-phase QA" is shorter.

Making it work in practice

The idea is simple. Getting a quiet, trustworthy result out of it takes a few deliberate decisions.

Choose test points that represent the processing, not the pages

You don't need every URL. You need enough test points to exercise every path through the processing code: each content type, each component the BFF transforms, each rule with edge cases. A few hundred well-chosen test points will tell you more than ten thousand random pages.

Keep the window short

The whole method rests on the content being the same in both phases. The longer the gap between them, the more editorial changes slip in and show up as false differences. Run phase one as the last step before the deploy and phase two as the first step after it.

Measure your noise floor before you trust a diff

This is the trick Diffy uses, and it works without any extra infrastructure: probe the test points twice before deploying. Anything that differs between two runs of the same unchanged code is noise: timestamps, request IDs, rotating content, cache artifacts. Turn those into ignore rules, and the after-deploy diff only shows what the deployment actually changed.

Record the input as well as the output

If you can, capture the raw CMS content for each test point alongside the BFF's output. Then a difference is easy to attribute. If the input changed, it is an editor. If the input is identical and the output changed, it is the code.

Treat the verdict as a diff to triage, not a boolean

Some differences are the point of the release: a new field, a fixed bug, a changed rule. The comparison should show you each change by path and let someone confirm it was intended. Unexplained changes are your regressions. That review is usually a few minutes, and it replaces the far longer job of a person clicking through the site after every release.

Where else linear QA quietly fails

CMS content is the clearest example, but the same pattern shows up anywhere the expected result depends on data your tests don't control:

• Pricing and promotions, where rules and campaigns change independently of releases.
• Search and recommendations, where rankings shift as the catalog and behavior data change.
• Personalization and feature flags, where the same request legitimately returns different results for different segments.
• Third-party data, such as stock levels, delivery estimates or exchange rates.
• Data migrations, where the question is not "is this right?" but "is this the same as it was?"

In every one of these cases teams typically end up with one of two bad outcomes: brittle tests that fail every day for reasons nobody cares about, or loose tests that pass every day and catch nothing. Both are symptoms of applying a linear model to a system that needs a comparative one.

Testing as a process, not a stage

The lesson I took from this is not about a specific tool. It is that "where does QA sit in the pipeline?" is sometimes the wrong question. For systems whose correct output keeps moving, QA can't be a box at one point on the line. It has to remember what the system did before the change, so that it can judge what the system does after it.

That before-and-after comparison is the core of what we now build at APIpact: snapshot your API responses, deploy, compare, and assess the differences with noise rules and severity. If you want the mechanics of the comparison itself, our guide to API content regression testing goes deeper, and catching breaking API changes in CI/CD shows how to wire both phases into a pipeline so nobody has to remember to run phase one.

Frequently asked questions

Is two-phase testing the same as regression testing?

Not quite. Regression testing usually re-runs tests against expected results that were written down in advance. Two-phase testing doesn't write expected results down at all. It uses the previous version's output, captured just before the deployment, as the expected result. The formal name for that is back-to-back testing.

What happens if content changes between the two phases?

It shows up as a difference. Keep the gap between the phases as short as possible, record the raw input next to each output so you can see when the content itself changed, and filter known volatile fields with ignore rules you build by probing twice before the deploy.

Why not just run the old and new versions side by side?

That is what tools like Diffy do, and it removes the timing problem completely. It also needs both versions running at once and traffic routed to both, which many teams can't set up easily. The two-phase approach needs nothing but the ability to call your own API before and after a deploy.

Does this replace unit and integration tests?

No. Unit and integration tests check the logic you meant to write, with data you control. Two-phase testing catches what you didn't mean to change, on the real data you don't control. The second is the gap that linear QA leaves open.