60 likes | 65 Views
How do you know your product “works”?. And what does it mean for a product to “work”?. Purposes and Uses of Evidence. Common Uses of Evaluation Evidence In Process Product Design and Modification Gathering “proof” that the product works
E N D
How do you know your product “works”? And what does it mean for a product to “work”? www.SEGMeasurement.com
Purposes and Uses of Evidence • Common Uses of Evaluation Evidence • In Process Product Design and Modification • Gathering “proof” that the product works • Understanding how teachers and students use the product • Complying with grant or foundation giving requirements • Supporting product adoption decisions from piloting • Understand your purpose for collecting information at the beginning. • Design your evidence collection in a way that is consistent with those goals. SEG Measurement WWW.SEGMeasurement.com
Sources of Evidence and Ways to Collect It • Sources of Evidence • Students • Teachers • Administrators • Parents • Experts or other third parties • How to collect evidence • Pre/Post Testing (Cognitive and Affective • Anecdotes • Surveys • Interviews • Observation • Focus Groups • Product Usage Data • Amount its used • How its used SEG Measurement WWW.SEGMeasurement.com
Are you looking for “proof” that use of your product is the “cause” of improved learning outcomes? • “Proof” and “Causality” (The scientific model) • The “Medical model” • If we want to know if a drug works, we know you can’t just ask a bunch of people. • Random Assignment to treatment and control groups or matched groups designs • Pre and post testing • Sound statistical models to eliminate any potential initial differences or “interfering” variables SEG Measurement WWW.SEGMeasurement.com
Challenges • Getting results quickly • Desire for rapid cycle, iterative improvement • Costs (One good adoption won pays the bill) • Lack of Research Literacy (We offer pro bono assistance) SEG Measurement WWW.SEGMeasurement.com
How to prove your product is effective • Treatment and Control Group • Sufficient Sample Size • Statistical Power • Generalizability • Pre post assessment of desired outcomes (external, reliable and valid) • Fidelity of implementation • Qualitative and quantitative data • Third party conducted • Peer reviewed • Multiple studies SEG Measurement WWW.SEGMeasurement.com