Automated Tests

Tests are the project’s regression net. Every change made from here on risks breaking something that worked yesterday, and a growing system stops fitting in anyone’s head long before it stops growing. Tests are how that risk is checked automatically rather than remembered.

The Automated Tests page (under Technical in the Explorer) appears in your project from the Build stage, when there is a real system to test.

Coverage is per feature, not per file

Test coverage in most tools means “what percentage of lines ran” — a number that says nothing about whether your invoices actually work. Sapilon reports coverage against the features your project declares: which of them have tests, at which layers, and whether those tests passed on the last run. “Invoices — 12 passing” is a sentence you can act on.

There are three layers: unit tests for a feature’s logic, API tests for its endpoints, and end-to-end tests that drive real journeys through its pages.

Who writes them

The agent does, on request. Tests are not generated on every turn — that would be slow and expensive for a change to a label — so the page lets you ask for them where you want them: Generate tests for the whole project, or Generate on one feature’s row.

Generate asks before it starts. It lists what it will do, feature by feature: new tests where a layer has none yet, and updates where a feature’s code changed after its tests were written. It also says the most the job can cost, because writing tests is AI work billed like any other agent turn. Nothing is spent until you say yes.

Generate never deletes tests or starts over. Tests that still match their code are left alone, and tests whose feature has changed are updated in place, keeping the cases that still hold. It writes test files only, never your app, and commits them like any other change, so you can undo them.

A job stops after a fixed number of agent turns, however big the project is, so one press always has a ceiling. On a project with many features missing tests, one job does the first few and leaves the rest on the grid for the next press. The confirmation says so before you start. To choose which features come first, use Generate on their rows.

They can also come from a task’s plan. When a plan changes what a feature does, it asks whether to add automated tests. If you say yes, its last step writes tests that check the decisions you agreed, not just that the feature exists.

They are ordinary project source, kept in your repository alongside the code. A technical user can read and edit them through the normal code surfaces; there is no separate test editor.

Running them

Runs are started by you: Run all tests on the Automated Tests page, Run on one feature’s row, or Run the tests again after the Agent has worked on a failure. There is no schedule and no setting that runs them for you. They execute on your Build Server, so a run costs you nothing extra to host, and take from under a minute to a few minutes depending on how much there is.

Two other things run tests without being asked, and neither is this page’s full run:

• Every change the agent makes runs the unit tests of the part of the app it changed before the change is kept. A unit test that goes red goes back to the agent to fix.
• A checkup runs your tests only if you tell it to. It doesn’t run this page’s suite on its own. To keep a version with failing tests from going Live, add a custom check that runs your unit or API tests, and turn on Blocks going Live. End-to-end tests stay on this page, because they take too long for a checkup.

When a run finishes, failures are listed with what broke, said in words rather than in the test runner’s shorthand: “the API crashed: 500 Internal Server Error instead of 200 OK”, not “expected 500 to be 200”. Open one and the side panel has the rest:

• Overview — what that kind of failure usually means, what the test expected and what it got, and the line of the test that caught it.
• Error — the test runner’s own output and stack trace, exactly as reported, to copy.
• Code — the whole test, with the line where it failed highlighted.
• Server output — when an API test got a server error back, what the API itself logged while answering it, if the run recorded that. It is where the cause is — a stack trace, a failed query — because all the test ever sees is the status code.

Handing a failure to the Agent is usually the fastest way through it; the test itself is often the clearest description of what the code was supposed to do.

After the Agent has worked on a failure, the finished turn offers Run the tests again. It takes you back to this page with the run already starting, scoped to the feature and layer that went red — because the only thing that can say a failure is gone is another run, not a look at the app.

When a test doesn’t make sense

A red test means the app and the test disagree. It does not mean the app is wrong: the test may describe a journey nobody asked for. That happens most with generated end-to-end tests, which are written from what the code appears to do, and are named after the journey the agent inferred — a name you never chose, describing a promise you never made.

So every failure offers three things, not one:

• Explain and fix — the test is right; change the app until it passes. The one exception is a test that asks for something the app cannot do by design (say, a reply body on a response that by definition has none). The agent corrects that test instead, and tells you it did.
• What does this test check? — have the test read back to you in plain language, with where it came from and whether the app or the test is the thing that is wrong. Nothing is changed; you get a recommendation and decide.
• Remove test — the test describes something this project never promised. It is deleted as an ordinary change you can undo, and nothing else is touched.

Removing a test is a real answer, not a defeat. A test you cannot read and did not ask for costs you a red suite forever, and a red suite hides the next real regression. What it does cost is coverage: nothing watches that behaviour afterwards, and the feature’s cell on the grid can drop back to a gap.

When the tests are out of date

End-to-end tests follow a journey through your pages: open Topics, add a topic, pin it. When a page is redesigned, its journeys can go red with nothing broken. The journey still describes what a user does, but it looks for a button, a heading or a label that the new page doesn’t have.

The page recognises this case and marks the failure Out of date. It uses that label only when both of these are true: the test waited for something on the page that never appeared, and your pages changed after the test was written. Those tests don’t need fixing one by one. They need writing again, so the card at the top offers Rewrite tests instead of Fix.

Rewrite asks the agent to read your pages as they are now and write the journeys again. Each journey keeps its name and what it checks; only the way it finds things on the page changes. The agent runs the new journeys before it finishes, and commits them like any other change. It touches end-to-end tests only, never your app. If a journey describes something your app no longer does, it is dropped, and the job names it so you can tell whether that was intended.

Fix is still there for everything else, and for any out-of-date test you’d rather handle yourself. When a run has both kinds, the card says how many are left for Fix after the rewrite.

What the page tells you

The card at the top of the Automated Tests page says where your tests stand, in one word, and what to do next:

• Healthy — every feature is covered, and the last run passed on your code as it is now. Nothing to do.
• Waiting — your move: some features are missing tests, so generate them, or your tests haven’t run on your latest code, so run them.
• Creating — the agent is writing tests.
• Running — your tests are running on the Build Server.
• Failed — some tests failed, or the last job didn’t finish. Fix in Agent hands every failure to the Agent in one go, and the finished turn offers the run that checks it. When the failures come from pages that changed, the card offers Rewrite tests first.

Creating and Running happen in the background, so you can keep working. The card beside it has the numbers: how many features are covered, how many tests are still to write, and how many tests the last run passed, failed and skipped.

What tests are not

They are not a substitute for using the app yourself. A green suite means nothing regressed against what was written down, not that the product is right — which is why Build also asks you to run the app end to end, and why Rehearsal exists.

Next: Checkup · Security · Build