A conversion optimization test begins with evidence of a specific user problem and a change designed to address it. Define the eligible population, primary metric, guardrails, assignment method, sample and duration plan, and decision rule before launch. Validate instrumentation and analyze assigned groups as designed. A higher conversion rate is not a win if quality, trust, accessibility, or revenue worsens.
Who this is for: Product marketers, analysts, and growth teams testing landing pages, forms, onboarding, or checkout experiences.
- Base the hypothesis on observed friction and name the mechanism the variation is expected to change.
- Predefine population, metrics, duration, exclusions, and stopping rules before viewing results.
- Check data quality, segment risks, and downstream guardrails before shipping a winner.
Diagnose before testing
Use analytics, usability studies, session evidence gathered with consent, customer interviews, form errors, support contacts, and sales feedback to find a decision obstacle. Separate symptoms from causes. A low form completion rate may reflect unclear eligibility, technical failure, excessive fields, or an offer that does not fit the traffic.
Write a mechanism hypothesis: 'Showing implementation requirements before the form will reduce uncertainty, increasing qualified submissions without lowering accepted-lead rate.' This is more informative than 'A shorter page will convert better.' Choose a change large enough to affect the proposed mechanism while keeping unrelated differences limited.
Design the experiment
Define eligible users, unit of randomization, control, variation, allocation, and handling of repeat visits. Use a trustworthy experimentation system so assignment is stable and exposure is recorded. Exclude bots, staff, or broken sessions through prewritten rules, not after results appear. Check whether concurrent campaigns or tests can contaminate the same decision.
Choose one primary metric close to the user and business outcome. Add guardrails for qualified rate, revenue, refunds, errors, page performance, accessibility, complaints, or support load. Estimate required sample and minimum duration using baseline rate, meaningful effect, traffic, and weekly cycles. Do not stop when a dashboard first turns green.
Validate and monitor
Before launch, test assignment, exposure, primary events, duplicate prevention, success and failure paths, mobile layouts, accessibility, consent, and rollback. Run an allocation check to detect unexpected group imbalance. Confirm that both variants load and that event definitions remain identical except where the hypothesis requires a difference.
During the test, monitor severe defects and guardrails using predefined thresholds. Avoid repeatedly inspecting significance and ending early. Record outages, promotions, tracking releases, and unusual traffic. If a defect invalidates one group, stop, document, repair, and restart under a clear rule rather than deleting inconvenient observations.
Analyze and decide
Analyze users according to assigned variation under the planned method. Report counts, rates, absolute and relative differences, uncertainty intervals, guardrails, and data-quality checks. Examine important prespecified segments for harm, but treat exploratory slicing as hypothesis generation rather than certain discovery.
A result can be ship, keep control, extend under the plan, or inconclusive. Consider implementation cost and downstream quality. Document the mechanism lesson even when the primary metric does not move. After shipping, monitor the same outcomes because novelty, traffic mix, and technical behavior can differ from the experiment period.
Test pricing-page qualification copy
Many visitors submit a sales form and later learn that the service has a six-month minimum engagement.
- Use sales rejection notes and user interviews to identify hidden commitment as the likely source of poor-fit submissions.
- Test a variation that explains minimum term and included work beside the form, keeping the offer and traffic stable.
- Set accepted opportunities as the primary metric, with form completion, complaint rate, and page performance as guardrails.
- Validate assignment, CRM qualification joins, repeat visits, mobile rendering, and event deduplication.
- Run for the planned sample and duration, then report both lower raw submissions and higher accepted-opportunity rate if observed.
CRO experiment protocol
Approve this protocol before exposing users to a variation.
- Evidence: observed friction, source, affected audience, alternative explanations, and proposed mechanism.
- Design: population, assignment unit, control, variation, allocation, exclusions, contamination, and rollback.
- Metrics: primary outcome, baseline, meaningful effect, guardrails, event definitions, and business join.
- Run plan: sample estimate, minimum duration, cycles, monitoring thresholds, stopping rule, and incident log.
- Decision: result method, uncertainty, segment review, implementation cost, ship rule, and post-launch check.
Common mistakes
- Testing a random visual preference without evidence of a user problem or a mechanism hypothesis.
- Stopping as soon as a dashboard shows a favorable result after repeated daily checks.
- Shipping higher form conversion while accepted leads, accessibility, or customer understanding declines.
Try one
A variation raises checkout completion but also raises refunds. Explain how the test should be judged.
The decision must include refund rate and net revenue or retained orders as guardrails, not checkout completion alone. A strong answer checks whether the variation obscured terms, changed traffic, or created a tracking issue, reports uncertainty, and rejects shipment if the downstream harm exceeds the intended gain. The user outcome matters with the immediate conversion.
Sources
- Google Analytics event definitionOfficial GA4 definition of event-based measurement.
- Google Analytics key events guideOfficial GA4 guidance for measuring and reporting business-important actions as key events.