Prove a change before you publish it

A rule is live. Somebody wants it changed, and the conversation has become an argument about what will probably happen.

This is how that argument stops being a matter of opinion: you run both versions over the same set of cases, see exactly which ones come out differently, and publish knowing the list.

About thirty minutes, and the habit it builds is the point rather than the individual change.

The shape of it

Three surfaces, in order.

Build is where the change is made and where the records you will test against live. Operate is where the two versions are compared. Build again is where you publish.

You never edit the thing that is running. A published version keeps running while you work; what you are editing is a draft beside it.

1. Assemble the cases you want to be sure about (Build, Datasets)

Before touching the rule, decide what “did this change what I expected” means, and the honest answer is always a specific set of records.

Click BUILD at the top, then Datasets in the left menu, under Automation. Click New. A Dataset is a table of records a flow can be run against, and it belongs to the Test centre rather than to your live work.

Put the awkward ones in. The application that should be declined. The one on the boundary. The one with the field missing that broke something last year. Ten deliberately chosen records tell you more than a thousand ordinary ones, because the ordinary ones were never going to move.

This is the step that decides whether the rest is worth anything. A comparison is exactly as representative as the records you gave it, and nothing in the product assembles them for you. If the set is thin, the answer is confident and thin with it.

[screenshot: a Dataset with a handful of deliberately chosen rows]

2. Make the change as a draft (Build, Flows)

Click Flows in the left menu and click the flow’s row to open it. Then click Create Draft, at the top right.

That button is the whole safety of this step, and it is worth knowing what it does: the published version carries on serving live work untouched, and what you now edit is a draft beside it. Nothing you do next reaches a live case until you publish.

Change the rule. Do not publish yet.

3. Compare the two versions (Operate, Experiments)

Click OPERATE at the top, then Experiments. Click New.

An experiment takes two sides, and each side is a version of a flow. Set the current published version on one side and your candidate on the other, point both at the Dataset you assembled, and run it.

If you are not sure which version is currently live, open the flow and click Versions on its toolbar: the list names every published version, who published it and when, and marks the one serving live work. Each row also offers Restore, which puts that version back as your draft, and is the quickest way to set up a comparison against something you published a month ago.

What comes back is a result per record: what each side did, and where they differed. Not a summary, a list.

[screenshot: an experiment result, the differing records highlighted]

Read the ones that did not move as carefully as the ones that did. A change that moves nothing is either unnecessary or aimed at a case your Dataset does not contain, and the second is the one that bites.

The run is kept. So is the last one, and the one before that, which means the history of a decision lives on the experiment rather than in somebody’s memory of what was agreed on Tuesday.

4. Publish (Build, Flows)

Back in the flow, click Publish at the top right. From that moment new work uses it.

Publishing is recorded and attributed: which version went live, who published it, and when. So “which version of this process was running when that case was decided” is something you establish rather than reconstruct.

If the flow declares policies, meaning guarantees it must not break, publishing checks them. A version that can be shown to break one is refused, and you are told which policy and given the exact case that breaks it, with the path it took through the flow. That check finds what it can demonstrate rather than proving the guarantee holds in every case, which is worth knowing when you read a clean result.

5. Watch the first real ones (Operate, Runs)

Published is not finished. Open Runs and look at the first few live ones through the new version, against what the experiment told you to expect.

Click OPERATE, then Runs, and open the first few. This takes two minutes and it is where a discrepancy between your Dataset and real life shows up, while it is still five cases rather than five hundred.

Two things to plan for

You can go back, and going back is itself a change. The Versions list on a flow shows every published version, who published it and when, and every row has a Restore beside it.

Restoring puts that version’s content into your draft. It does not push it live. You look at it, run the same comparison over the same Dataset if the change was significant, and publish it like anything else. If you do not like the restore either, Cancel takes you back to where you were.

So the honest sequence for a change that turned out badly is the same one as for any other change: restore, look, compare, publish. That is slower than a button that undoes something, and it is the reason you can trust what is running.

Keep the Dataset. The instinct is to build one per change and discard it. The value compounds the other way: a Dataset that has accumulated every awkward case somebody found is the thing that makes next quarter’s change a thirty minute job instead of an argument. Add to it each time something surprises you in step 5.

Where to go next

A change that must never break a guarantee. Declare the guarantee as a policy on the flow, and publishing will hold you to it.

A change somebody else has to agree to. Cases carry their own approval, where the person who worked one is not the person who signs it off.