How to Test a Trading Scanner Without Fooling Yourself

A scanner is useful if it helps you find and review candidates that fit a defined process. An attractive chart of past signals is not enough to establish that it improves decisions or returns. To evaluate a scanner fairly, write the test rules before viewing the results and preserve every candidate—including the ones you reject.

This guide is a testing protocol, not a claim that a particular scanner has been tested or that a strategy is profitable.

Decide what success means

Choose a question the tool can actually answer. “Does it reduce the time needed to find candidates meeting my criteria?” is measurable. “Will it make me money?” depends on the strategy, execution, costs, risk, and future market conditions. Define a fixed observation window, a repeatable screening schedule, and the criteria for inclusion and rejection. Keep the same process for the comparison method, whether manual screening or another product.

Write down the tool version, data source and delay, market universe, settings, date, and time. A historical screen built with information that was unavailable at the original decision time creates hindsight bias. Test a stream of new ideas prospectively when possible.

Capture the whole funnel

For every signal, record whether it was available when you checked, whether it met the original criteria, whether earnings or another event ruled it out, whether liquidity was sufficient, and how long the review took. Include false starts and days with no useful result. Selecting only the strongest retrospective examples tells a misleading story.

If you conduct a paper exercise, write the proposed entry and exit rules before subsequent price movement is known. Record the spread, fees, and plausible execution assumptions. A hypothetical fill at the best visible price can be unrealistic, especially in a thin market. Paper trading cannot reproduce every aspect of live execution or decision-making.

Compare workflow, not just outcomes

Measure tasks separately: minutes to build a list, minutes to verify a candidate, ease of documenting a plan, and the number of unsuitable results requiring manual review. Record what information you had to fetch elsewhere. A paid tool may save time but still be a poor fit if its method differs from yours; a free workflow may cost more time but offer greater control.

Do not infer a durable advantage from a small sample. Market regimes change, and a favorable test window may be chance. If you revise a criterion midway, label it a new test instead of combining incompatible results.

A transparent reporting format

Publish the start and end dates, settings, data timing, sample size, all exclusions, execution assumptions, costs, tasks timed, and unresolved questions. Separate observed workflow from hypothetical performance. State whether the reviewer paid for access or could earn an affiliate commission. Readers should be able to see where evidence ends and interpretation begins.

Our vendor evaluation questions and PowerX Optimizer review provide context. The review describes an owner’s workflow; this protocol does not purport to be a new head-to-head product test. No scanner removes trading risk or replaces independent judgment.