False Positives in Click Fraud Detection: The Cost of Blocking a Real Customer

What is a false positive?

Define the unit and target class first: an ad interaction, website session or lead may be assessed for different reasons. A real human can generate invalid activity, and permitted automation can be mistaken for abuse. A finding about one stage should not silently become a verdict on the whole customer journey.

Why it matters financially

A mistaken block can remove a qualified enquiry, interrupt a returning customer or create support work. Not every rejected visitor would have purchased, so lost revenue cannot be inferred from traffic count alone. Use confirmed access problems, reviewed outcomes and a suitable comparison to estimate impact with stated uncertainty.

Shared IP problem

Carrier NAT, offices and shared exits can put unrelated visitors behind one public address. A rule derived from one abusive session may affect other people. Review range scope, duration and legitimate traffic before extending a block. An address match is not evidence that every session has the same operator.

Review IP exclusion scope and shared-network limitations

VPN problem

A VPN or proxy can be used for privacy, work or testing as well as abuse. Record the observed network context without automatically equating it with fraud. Ask whether the actual pattern, campaign and outcome evidence supports a stronger action than observation or proportionate verification.

Unusual-but-legitimate behaviour

Accessibility tools, interrupted connections, returning visitors and deliberate comparison shopping may look unusual. Short duration or limited scrolling does not establish abuse. Review ordinary cases alongside flagged ones, and document known exceptions instead of tuning a rule only on its most suspicious examples.

Thresholds and confidence

A threshold selects an operational response; it is not automatically a calibrated probability of fraud. Evaluate how the error trade-off changes by route, source and visitor cohort. Keep uncertain or unlabelled cases separate, and check whether an apparent improvement simply reflects a different population.

Holdout/control groups

Where safe, compare a limited treatment with a comparable holdout under a predeclared outcome rule. Do not leave known harmful activity untreated solely to create an experiment. Guard against spillover, campaign changes and unequal follow-up. A before/after comparison without these controls is useful context, not automatically a causal estimate of savings.

Observation mode

A no-enforcement assessment can show which visitors a rule would flag while preserving their later outcomes. Confirm whether the vendor actually offers this mode and what is observed. Shadow flags alone cannot prove the effect of a future block, and missing outcomes cannot be counted as confirmed legitimate or fraudulent.

Rollback criteria

Agree an owner, review date and stop conditions before enforcement. Include verified customer access failures, qualified-outcome loss and form availability, not just traffic reduction. Keep the previous rule configuration so a reversal is possible. Pause or narrow the control when evidence is insufficient to justify the customer cost.

Precision vs recall

For a defined set of reviewed cases, precision is the share of flagged cases confirmed to belong to the target class. Recall is the share of all confirmed target cases that were flagged. Unknown labels and selection bias limit both metrics. False-positive rate uses legitimate cases as its denominator and is not simply one minus precision.

What vendors should disclose

Ask for classification scope, reviewed-case coverage, uncertainty, exceptions and a response-review process. A detection rate without a dataset and denominator is not a procurement result. This guide does not claim that AdFraud.pl preserves every legitimate customer, supplies experimental holdouts or is MRC-accredited.

Turn error-rate questions into a software evaluation checklistScope an evidence-led review before enabling stronger controls