Deep Dive · Tracking Outcomes
Did That Fix Actually Work? How to Measure a Website Change Honestly
A practical, no-statistics-degree guide to proving whether a website change helped, with fair comparison windows, seasonality checks and the caveats worth saying out loud.
Pick one number tied to the change, record a baseline before you make it, annotate the launch date, then compare equal-length windows before and after that cover the same weekdays. Check the same weeks last year for seasonality and list anything else that changed. Report the result with its caveats, as evidence of what happened rather than proof of what caused it.
Key Takeaways
- Choose one number that the change should move, before you make the change.
- Compare equal-length windows with the same weekdays, and skip a short settling period.
- Check last year and list other changes, like ads, emails and Google updates.
- Say "this lines up with" rather than "this caused" unless you ran a real experiment.
You fixed the form. You rewrote the service page. You sped up the homepage. Now someone asks the big question: did it work?
It's a fair question, and a surprisingly easy one to answer badly. Compare the wrong weeks and a holiday makes you look like a genius. Ignore seasonality and a spring rush gets credited to your new button.
Here's how to measure a change in a way you'd be comfortable defending.
Step 1: Pick One Number Before You Change Anything
Decide what the change should move, and write it down first. Not after. If you wait until the results come in, it's very tempting to go hunting for whichever number went up.
| Change | Good number to watch |
|---|---|
| Shorter contact form | Completed form submissions |
| Rewritten service page | Clicks from search to that page, then leads from it |
| Faster homepage | Core Web Vitals, then bounce from that page |
| New phone button | Taps on the phone link |
One primary number, maybe one or two supporting ones. That's it.
Step 2: Record a Baseline
A baseline is what "normal" looked like before the change. Grab the last four weeks of your chosen number, plus the same four weeks from last year if you have them. Save a screenshot or export. Data has a way of getting reorganized later.
Step 3: Annotate the Launch
In GA4, right-click the line graph on the launch date and add an annotation. It shows up across every report with a line graph, for everyone on your team with access. Six months from now, when someone asks why October looks different, the answer is sitting right there on the chart.
Step 4: Compare Fair Windows
- Same length. Four weeks before versus four weeks after. Not two weeks versus six.
- Same weekdays. Most sites behave differently on weekends. Comparing full weeks keeps that fair.
- Skip the settling period. Search engines need time to recrawl, and caches need time to clear. Ignoring the first few days after launch avoids judging a change before it has fully landed.
Search Console and GA4 both let you compare two date ranges side by side, which makes this quick.
Step 5: Rule Out the Usual Suspects
Before you take credit, check what else happened:
- Seasonality. Did the same weeks jump last year too? If so, part of the lift is probably the season.
- Marketing. An email blast, a new ad campaign or a social post going viral can all move numbers.
- Google itself. Google's Search Status Dashboard lists ranking updates and incidents. If a core update landed the same week, say so.
- Tracking changes. If you also fixed a duplicate tag that week, your "before" and "after" numbers aren't measuring the same thing.
Step 6: Report It Honestly
Overclaiming
"Our new form increased leads by 40%."
Honest
"Completed forms rose from 22 to 31 in the four weeks after the change, compared with the four weeks before. Last year these weeks were flat, and no campaigns ran. This lines up with the shorter form helping."
The honest version is longer, but it's far more convincing to a skeptical boss or client. It also protects you when a result doesn't repeat.
When You Need Stronger Proof
Before-and-after comparisons show what happened. They can't fully rule out coincidence. If a decision is big enough, like a full redesign or a pricing change, consider an A/B test, where both versions run at the same time and differences in timing can't sneak in. Our guide to picking a first test covers when that's practical.
Keep a Simple Change Log
The single most useful habit is a running list of changes with dates. A spreadsheet works fine:
- Date
- What changed, and on which pages
- Why you changed it
- The number you expect to move
- What happened four weeks later
After a few months, that log becomes the best evidence you have of what actually works on your site. It also makes monthly reports much easier to write.
Frequently Asked Questions
How long should I wait before judging a change?
Long enough to collect several full weeks of data after the settling period. For low-traffic pages, that might be a month or more. If the number is tiny, wait longer rather than reading tea leaves.
What if the result is flat?
That's a real result. It tells you this change didn't move this number, which helps you decide where to spend effort next.
Can I measure several changes at once?
You can measure them, but you can't easily tell which one did what. If you must ship several together, report them as a bundle.
What if numbers went down?
Check for tracking problems and outside factors first. If the drop holds up, you've caught a regression early, which is exactly what measuring is for.
Sources
- About annotations, Google Analytics Help
- Using the Google Search Status Dashboard, Google Search Central
- Performance report overview, Search Console Help