How to test a change, and read what it did.
Most stores change something every week: a new homepage banner, a price, an offer, a batch of ads, a new email flow. Very few can say a month later which of those changes helped. This guide sets out a simple habit that fixes that, and shows how That Other Dashboard's board and decision log do the sums for you.
Why most changes are never judged
Changes go live on a busy day, nobody notes the date, and by the time anyone looks at the numbers three other things have changed too. Sales went up, but was it the new banner, the email, or payday? Without a record, the answer becomes whoever argues best.
A test doesn't need special software. It needs a date, a prediction and a fair comparison.
Before you change anything
- Say what you expect it to move. A free delivery threshold should lift average order value. A new hero image should lift sales, not average order value. Pick the number first, so you can't choose a flattering one afterwards.
- Change one thing at a time where you can. If a sale, a new ad and a site change all go live on the same Monday, you'll never separate them.
- Pick a calm week. Avoid launching right before a planned sale, a bank holiday or a big email, unless that is the thing you're testing.
Log it on the board
In That Other Dashboard, every change is a card on the board. Press Log a change and fill in:
- What changed: a short title, such as "Free delivery over £40" or "New homepage hero".
- Type: site change, landing page, offer or price, creative or ad, email or SMS, launch, or other.
- Live on: the day it went live. This is where the marker goes.
- Owner: who made it, handy when several people work on the store.
- Note: what was done and what you expect it to move. This is your prediction.
Leave Draw a marker on the charts ticked, and the change appears as a marker on every chart on the day it went live. Move the card through Open, In progress and Blocked as you work on it. The board is free on every plan.
Read the week after
Once a change has been live for at least a full day, its card shows what net sales did: the full days after it went live, up to 7, against the same number of days just before. After a week that reads, for example, "Net sales, 7 days after vs 7 before: +12%".
A few details are deliberate:
- The go live day itself is left out. Most changes go live partway through a day, so that day is half before and half after.
- It compares like with like. Seven days after against seven days before covers every day of the week on both sides, so a strong Saturday doesn't tip the result.
- It needs history. If the days before the change are older than the history the app holds, there is nothing fair to compare with, so it shows no figure rather than a misleading one.
Net sales is the headline, but look at the number you predicted too. Open the same dates on the dashboard and check average order value, orders and units, plus spend and MER if you've connected your ads.
What can fool you
A before and after comparison is honest about what happened, not always about why. Before you give a change the credit, check:
- Seasonality. Sales rise into Christmas, payday weekends and warm spells whatever you do. Compare with the same weeks last year if you have them.
- Other changes. Look for other markers in the same fortnight, and check for emails sent, ad budgets raised or a sale on a marketplace.
- Ad spend. If spend went up 20% in the week after, a 10% sales lift may be the ads, not the change.
- Stock. A bestseller selling out (or coming back) moves sales more than most site changes.
- Small numbers. On 20 orders a week, two extra orders is a 10% swing. The fewer orders you take, the longer a change needs to run before the result means anything.
- A bad week before. Changes made in a panic after a poor week tend to look good, because sales often drift back to normal on their own.
If any of these apply, let the change run another week or two and look again, or note the doubt in the result. For site changes on a busy store, a proper split test (half your visitors see each version at the same time) removes most of these problems, and is worth doing for anything big.
Close it and keep the lesson
When you've made up your mind, write what happened in Outcome and move the card to Closed. It joins the decision log: what was decided, when it went live and what sales did either side.
Write the outcome for the person who'll read it in six months, maybe you. "Kept: AOV up from £38 to £44, orders flat, delivery cost per order up about £1" beats "worked well". Over a year the decision log becomes the most useful thing you own: a record of what works for your customers, not someone else's.
Common questions
How long should I run a test for?
At least one full week, so every day of the week is covered. Two to four weeks is better for small stores, or for changes like pricing where people take time to react.
Can I log changes in the past?
Yes. Set the Live on date to the day it actually went live and the marker and comparison work the same, as long as the app holds history from before that day.
What if the result is flat?
That is a result too. A change that made no difference can usually be kept or dropped on other grounds, such as cost or simplicity. Write it in the decision log so nobody tests the same idea again next year.