Notes from the practice

A working survey can now be generated from a clean questionnaire in moments, by anyone, as many times as they like. Each of those scripts will run, and look right on the screens someone happens to click through. None of them comes with verification, and none comes with a person who answers for it.

Generated code will not notice that a filter contradicts the screener, that the last quota cell can never fill, or that a piped brand name is wrong on one path in twelve. Those were always the expensive errors, and it produces them with complete confidence. So how a script is drafted matters less than ever. What matters is that every line of it, however it was produced, is tested against a written plan, independently audited, and answered for by a named person.

A soft launch shows you what a few dozen respondents did. It cannot show you the paths they did not take: the rare screener combination, the final quota cell, the respondent who backs up and changes an answer. Those paths are where errors live. Soft launch confirms a survey that has already been verified; it should never be the first time anyone looks at the data.

A twelve-page questionnaire with a rotating concept test and nested quotas carries more risk than a forty-page attitudinal battery. Cost and risk follow logical complexity, not length. The questionnaire is reviewed first so that the quotation reflects what the study actually demands, and neither party is surprised later.

The person who wrote the logic is the person least able to see what it fails to do; they test what they intended, not what they built. So every build is audited by an external firm that did not write it. And because that firm is paid for every defect it finds, rather than a flat fee for signing off, its only incentive is to look harder. Every defect it finds is one your respondents never see.