Skip to content

How to spot unusual changes in your measurement

Bily Editorial Team

The Bily Editorial Team writes about advertising measurement, data quality, and the decisions they support.

Date

Read time

Read time

A stable sequence diverges at a detected node and stops at a containment line.

Attributed purchases fall sharply, but completed orders hold steady. Start with event capture, delivery, and matching. If both fall, investigate the storefront and campaign conditions as well. The same alert can require very different actions.

Compare the change with a relevant baseline

A fixed percentage alert is easy to create and hard to trust. Marketing data changes with weekday patterns, seasonality, promotions, campaign maturity, and spend. A useful baseline reflects those conditions and shows an expected range rather than a single rigid number.

Build separate expectations for metrics that behave differently. Conversion volume, cost per acquisition, event delivery, and reported revenue do not share the same rhythm. Review the baseline after material changes to pricing, channel mix, or campaign structure.

Rank anomalies by business consequence

Statistical surprise is only one part of severity. A small deviation on a high-spend campaign may matter more than a dramatic change in a low-volume segment. Rank an alert using its size, duration, affected spend or revenue, and the reversibility of the decision.

This keeps the team focused on anomalies that can change an outcome. It also reduces alert fatigue, which is itself a measurement failure. When every movement triggers a warning, important warnings become easier to ignore.

Separate measurement failure from performance change

Before pausing spend, compare the affected metric with independent business evidence. If attributed purchases fall but completed orders remain stable, investigate event capture, delivery, or matching. If both orders and attributed purchases fall, inspect the customer journey and campaign conditions.

Check the scope as well. A change isolated to one campaign suggests a different cause from a change across every channel. The purpose of this comparison is not to delay action. It is to choose the action that fits the failure.

Turn every alert into a runbook

Give each alert an owner, first checks, an action boundary, and the evidence needed to close it. Record the actual cause: measurement, media, site behavior, or a known business event. Use that history to reduce repeat false alarms.

Detection systems do not automatically improve whenever someone resolves an alert. Feedback must be captured and intentionally used to adjust baselines, rules, or models. Otherwise the same false positive will return with a different timestamp.

Start with one alert tied to material spend or revenue. Write and rehearse its diagnostic steps before adding more alerts. Measure whether the team reaches the right response sooner, not how many warnings the system sends.