Experimental Design How to organise a successful research project Including notes on sample size and reliability

1. Experimental DesignHow to organise a successful research projectIncluding notes on sample size and reliability Research Methods Lecture Bruce Lynn Director, MSc School of human Health and Performance March 09

2. Experimental DesignHow to organise a successful research project The initial idea Reading the literature Planning Doing Writing

3. Generating the initial idea Something you always wanted to investigate Look at past projects and shamelessly borrow ideas Read around a subject of interest looking for loose ends Talk to one of the labs associated with the MSc Programmes

4. Reading the literature Essential Make sure project has originality Find out about the methods used in the field of interest

5. PlanningThe key to a successful research project Start early Decide what measures to use Who are the test population? What is being compared to what? Statistical issues

6. What measure(s) to use? Questionnaires. Find suitable validated instruments. Try developing your own? Physiological measures Explore availability of equipment Investigate suitable analytical software Example Investigating heart rate variability. Find suitable heart rate monitoring equipment, e.g. Polar Find software package to analyse beat by beat data Decide which HRV parameter(s) are appropriate

7. Who are the test population? Biological investigations, e.g. animal studies, or studies of tissue properties, or cadaver anatomy studies Healthy volunteer studies Patient studies All are regulated re ethics. In particular patient studies need Central (NHS) ethical clearance (NRES) Recruitment for human studies Posters Letters or e-mails Will subjects be paid � if so who funds this? Subject information sheets Will enough subjects be available within the time period available for the project?

8. What is being compared with what?Experimental designs (1) Example A double blind random controlled trial (RCT) Prospective � subjects are checked out in advance of intervention Controlled. 2 (or more) groups run in parallel. One receiving an ineffective treatment that will appear equivalent to the real treatment, thus controlling for any placebo effect. This is often the hardest part of any design. Randomisation. Must not select subjects. More info in a minute. Subject blinds. Can be impossible. E.g. testing health education literature. Observer blinds. Nearly always possible. Need an assistant to organise this. May not be necessary if outcome measure is very objective, e.g. blood count. Dropout rates. Must be monitored. Differential dropout from control or treatment group can introduce bias.

9. What is being compared with what?Experimental designs Other designs Crossover trials. Control group subsequently receives treatment and vice versa. Each subject own control � good feature, but carry-over of effects is a problem. Non-experimental designs � surveys Longitudinal. Following same subjects for period of time. Cross-sectional. Looking at sample of population at one time point. Retrospective. Looking back at health records of patients.

10. More on randomisation So pure random sequences can cause problems. Need to remove long runs. Also need all numbers to appear within any given sequence the same number of times � for this use random permutations. Often useful to ensure same number of treatment (T) and control (non-treatment, NT) within blocks of subjects. So every 10, make sure you have equal T/NT even in a design for 30. Means if you have to stop short, still have roughly equal T and NT Often need to ensure balance by randomising separately for male/female, old/young. severe/less severe etc as appropriate. Make sure you have extra random allocations in sub-groups if not sure of numbers, e.g. for M/F

11. Statistical issues � how many subjects Key statistical issue in the planning phase is whether enough subjects can be studied to get a clear result. Involves the question of statistical power. Need to know 4 things ? The criterion for rejecting the null hypothesis, normally 5% The criterion for missing a possible result, normally 10% or 20% (NB 100-?, is the �Power� of the study) The standard deviation of the outcome measure. Or if using counted data or another non-continuous measure, then the appropriate error measure d The extent of change to be detected. For a comparison of 2 groups with same n, formulae for required n is n = 16?2/d2

12. Statistical issues � how many subjects Example (From Statistics at Square One, http://www.bmj.com/collections/statsbk/13.dtl) BP Trial, comparing 2 groups. Want to find 5mm Hg change if it occurs; standard deviation in population is 10mm Hg. n = 16*10*10/5*5 = 64 If we decide only a 10mm Hg change is of significance to us, then n = 16*10*10/10*10 = 16 So sample size needed depends very much on criterion we set. Obviously, the smaller the standard deviation in the measure/population, the smaller n will be. How do we find out s.d. in advance of the study!! Can find literature values. Or do a small pilot study. Also, if we set a tougher criterion for rejecting the null hypothesis, e.g. 1%, then n rises. A more detailed discussion of this issue, and the formulae for other types of design and measure, is in Kirkwood, Essentials of Medical Statistics, Chapt 26

13. Sample size for standardised difference at power 80 or 90% This is for comparing an average result against a fixed criterion or for paired data where SD is SD(diff); for comparing 2 groups need to double n in each group and use SD(error) [i.e. RMS(within) from ANOVA or SD(diff)/sqrt(2)]. Difference you wish to detect is given in �standardised� form, i.e. difference/standard deviation. e.g Diff to detect in actual units = 3, SD of repeated measurements in population being studied 5, then "standardised" diff to detect is 3/5 = 0.6

14. Sources of error Systematic � leading to bias Random Bias is the bigger problem. Can have apparently clear result that is wrong. Bias may be caused by inadequate randomisation, or not ensuring good match between T and NT samples, or inadequate technique (e.g. not maintaining calibration of equipment during study) Random error will not lead to false positives � but will of course lead to more false negatives. Important to spend time optimising methods. Also to consider using only a relatively homogeneous population, e.g. restrict to young males.

15. Reliability statistics Need to know how reliable a measurement is. Is it repeatable, e.g on test-retest. Also if comparing groups, how much variation within the group? Comparison with other methods � the calibration problem.

16. Reliability statistics Repeatability 1. Is it repeatable, e.g on test-retest. 2. Comparing groups, how much variation within the group? Need this information to compare different methods Importantly need it for sample size estimation Can just take a lot of measurements and calculate SD Often have pairs of measurements. The SD(diffs), the SD of the differences between the values in each pair, will be SD*21/2 See Reliability.doc and the example on Reliability.xls, sheet�Test-retest� on WebCT or at http://www.archway.ac.uk/Activities/Departments/SHHP/current/Res_Methods/resmet1.htm

17. Reliability statistics Comparison of new method with a standard method Sometimes referred to as validity, but this term also covers wider aspects of any test. Common approach is to measure same subjects with both methods Then can do SD(diffs) as for test-retest comparisons Often to methods are plotted as an XY plot, with correlation coefficient, r used to assess reliability This is bad practice, r depends on range of X values as well as errors. So more variable methods tested on a wide range of subjects can look better than a less variable methods tested over a more limited range. See Reliability.doc and the example on Reliability.xls, sheet �Comparison� on WebCT or at http://www.archway.ac.uk/Activities/Departments/SHHP/current/Res_Methods/resmet1.htm

18. Doing the studyThe fun bit! Usually need to run initial trials to learn methods, establish reliability etc Then plan the main data collection. Try to keep things standardised If unexpected observations occur, then DO FOLLOW THEM UP Back up data. Keep back ups in a separate place from the main data storage. Hard disc crashes do occur!!

19. Sorting out and analysing the data Excel is useful. Be sure to graph data before you analyse. Visualising the data usually indicates the real trends � your eyes are good at statistics! Look at average trends; normalise data where appropriate. For most regression, ANOVA or non-parametric tests you will need a statistical package. SPSS is on the cluster machines but is rather user unfriendly, so expect to spend some time finding your way around.

20. Resources Statistics at square one, last chapter. Good introduction to statistical issues. http://www.bmj.com/collections/statsbk/ Kirkwood, B.R. Essentials of Medical Statistics, Chapts 21-26 Bland, M. An Introduction to Medical Statistics, [Chapt 15, Reliability measures; Chapt 18, smaple size] Field, A. Discovering Statistics using SPSS [Detailed on regression; good explanation of SPSS computer package] Mackenzie, A. Mathematics and Statistics for Life Scientists, Bios Instant Notes, 2005 [mostly maths] Random permutations: http://calculators.stat.ucla.edu/perm.php http://www.webcalculator.co.uk/statistics/rpermute3.htm NRES (National Research Ethics Service) [http://www.nres.npsa.nhs.uk/ ]