Ten Questions to Better Pilot Programs is an obscure but interesting Fiscal Brief produced last month by the Fiscal Research Division of the North Carolina General Assembly. Study bills and pilot programs are often used to molify sponsors of legislation otherwise faced with certain defeat. When properly designed and implemented pilot programs can produce useful information. Many such programs are inadvertently doomed to preclude meaningful assessment, producing ambiguous results.
The Fiscal Brief has some general recommendations:
Policymakers should insist upon pilot programs that are designed as randomly controlled trials whenever possible.
Policymakers should avoid insisting that their district or districts be included in treatment groups.
A pilot program that generates actionable data is far more important than having a poorly designed program placed in a home district.
Policymakers should allow time for pilot programs to reach their full implementation and allow time to observe program effects. Acting too early might result in the abandonment of programs that are actually working.