Steve George HTACET Board member 2006-2010

1. Tips for getting Grants from HTA CET Steve George HTACET Board member 2006-2010

2. NIHR Research Programmes www.nihr.ac.uk/research/Pages/programmes_research_programmes.aspx CCF Managed Programmes Programme Grants for Applied Research (PGfAR) Research for Patient Benefit (RfPB) Invention for Innovation (i4i) Research for Innovation, Speculation and Creativity (RISC) NETSCC Managed Programmes (http://www.netscc.ac.uk/) Health Technology Assessment (HTA) Standard calls Clinical evaluation and trials calls Themed calls Technology Assessment Report (TAR) contracts Service Delivery and Organisation (SDO) Public Health Research (PHR) Health Services Research (HSR) Efficacy and Mechanism Evaluation (EME)

3. Research Innovation Pathway http://www.nihr.ac.uk/research/Pages/default.aspx

4. HTA research http://www.hta.ac.uk/funding/HTAremit.shtml Research that is immediately useful to clinical practice and policy/decision makers. Undertaken when there is evidence to show that technology is efficacious but there is uncertainty around its clinical and cost effectiveness in a real life NHS setting Standard calls - commissioned research proposals addressing specific topics Clinical evaluation and trials calls - research proposals for evaluation studies and clinical trials on topics proposed directly by researchers Themed calls - one-off invitations for proposals to fund research in areas where a particular need has been identified (eg diagnostic call in 2009) Technology Assessment Report (TAR) contract - specialist assessment centres contracted by the DoH to undertake technology assessment reports on behalf of the National Institute for Health and Clinical Excellence (NICE) and others

5. The HTA will not fund… Laboratory-based research or basic science Setting up or maintaining research units (except CTUs) Proposals which are solely service developments Proposals which are solely audits, surveys, needs assessment, technology developments Programmes of research in which (for instance) the development of the intervention forms the first stage, with a trial as the third or fourth stage Repetitions of definitive trials carried out in other countries

6. The HTACET Application Process Four annual outline application deadlines Anonymised extracts of outline applications scored by CETPG (Clinical Evaluations and Trials Prioritisation Group) on basis of need/NHS importance – typically 40-50 outlines received, and 15-20 selected). These scores go forward for ratification to Priorities and Strategy Group (PSG) then selected outlines go to: HTACET Board, which decides on basis of quality/methodological rigour to invite full applications from about 50% and to send for External peer and lay review from up to seven reviewers Then back to HTACET Board assessment Typically about 50% recommended for full funding (see later for scoring) Final ratification by PSG Notification of outcome

7. HTACET Board HTACET Board currently meets four times a year, each time for two whole days At the April 2010 meeting the Board considered 11 full proposals and 18 outlines The average price of a funded trial has risen over the last couple of years: it was around £1.5M but now exceeds £2M The price per subject is very important, although due consideration is given to number of centres, which is a major cost driver, and methodological quality is the final arbiter However, £2500 per subject is considered a fairly cost effective figure

8. HTACET Board 2009 Professor Jon Nicholl, University of Sheffield (chair) Professor Jenny Hewison, University of Leeds Professor Keith Abrams, University of Leicester Professor Martin Bland, University of York Professor Jane Blazeby, University of Bristol Professor Julia M Brown, University of Leeds Professor Alistair Burns, University of Manchester Professor Brian Cuthbertson, University of Aberdeen Professor David Dearnaley, Royal Marsden Hospital Professor Julia Fox-Rushby, Brunel University Dr Steve George, University of Southampton Professor Simon Gilbody- University of York Professor Alastair Gray, University of Oxford Professor Richard Gray, University of Birmigham Professor William Jeffcoate, Nottingham University Hospitals Trust Professor Paul Jones, St Georges Hospital Medical School Professor Khalid Khan, University of Birmingham Professor Michael King, Royal Free & University College Medical School Professor Catherine Sackley, University of Birmingham Professor Clare Wilkinson, University of Wales College of Medicine Dr Ian B Wilkinson, University of Cambridge Observers Ms Kate Law, Cancer Research UK Ms Morven Roberts, Medical Research Council

9. The CETPG Stage Extracts from outlines are scored anonymously by CETPG according to the importance of the subject of the trial to the NHS and the need for the trial. Only section C of the outline form is reviewed at this stage Pay attention to the individual sections within Section C: Specification of research question Importance of the health problem to the NHS Description of the technology and its possible effect on health status, including current and projected use in the NHS, with approximate costs Summary of the current evidence What outcomes will be measured Summary for the Non-Expert If you fail at this stage your proposal either does not meet the remit of the programme or you have failed to answer the SFW question (“S” and “W” stand for “So and “What”…)

10. Your outline at HTACET At the next stage your outline will be presented by a Designated Board Member (DBM) to the HTACTE Board. Each DBM is allocated a number of trials to present and will have gone through the proposal in detail (most Board members will have read all of the outlines but may not have gone as far as chasing up competing trials etc.) The Board considers the complete form, including those sections previously reviewed for importance, but considers particularly the methodological components of the study

11. Frequent comments about outlines What question are they trying to answer, precisely? Why haven’t they got a named statistician? They say they’ve got a CTU involved but they haven’t named it and there’s no CTU representative amongst the applicants I can’t replicate their sample size calculation… Do they really think they’re going to have their first subject recruited six months after receiving funding? Who’s going to recruit these patients? What’s a “semi-factorial” design when it’s at home? Who are this bunch? How much?!!!

12. Surviving the outline stage Many of these comments are richly deserved – if you send in a trial outline without having consulted a statistician, and without naming them amongst the applicants, you will not survive HTACET members will know about competing trials so there’s no point in trying to hide them At this stage you may be able to get away with a little lack of methodological detail, but not without a clear trial question and objective(s) – better if it’s accompanied by a brief discussion of methodological alternatives to solve the inevitable compromises between ideal design, timescale and cost - and beware…

13. Your Nemesis!

14. Your Nemesis! Martin Bland makes a point of recalculating each and every sample size calculation presented at each and every HTACET meeting If your calculation cannot be repeated from the information you give he will spot this and point it out to the Board He will do a similar job on your analysis plan… …and if he doesn’t one of the other statisticians will

15. Your full proposal at HTACET Your proposal will now be presented to HTACET by three DBMs, who have had prior access to the referees’ reports (up to seven, including a lay referee), and to Department of Health trial costing reports Officially 20 minutes is given for discussion of each full proposal but this can sometimes be extended by a factor of three or more, or curtailed… There are four possible decisions Straight fund, Fund with changes Re-submission at full proposal stage Reject.

16. Scoring of Full Proposals Proposals are scored 1 - 6 , using an electronic anonymised voting system, taking into account i) The methodological and scientific quality, ii) The likelihood of success, and iii) The value for money of the project.

17. Scoring of Full Proposals

18. Which full proposals succeed? Proposals that are recommended for funding achieve a minimum average score of 3.5 Until now, all proposals recommended for funding have been funded In future it is likely that funding constraints will mean that score is taken into account in deciding which recommended proposals receive funding

19. Frequent comments about full proposals We asked them to get rid of the third arm in this trial because it’s pointless, but it’s still there and they’ve come up with some poxy argument trying to justify it! We asked them to clarify the trial question but if anything it’s got worse! This is completely different from the outline! They still haven’t got a proper statistician/economist/CTU Why has the cost doubled? It’ still too expensive… …I still can’t repeat their sample size calculation…

20. Research design tips Some Board members do not like particular designs – factorial designs and non-inferiority designs to name but two If you insist on submitting a factorial design acknowledge that you are unlikely to be able to power for the “Treatment A + Treatment B” corner because of the unknown extent of interaction between the two treatments (unless there is no possible interaction between the treatments). State that information on degree of interaction gathered during the trial will be used to plan for future studies. Be prepared to justify, with evidence, any equivalence or non-inferiority margin specified

21. Research design tips Do as HTACET says, and don’t argue pointlessly Trial feasibility is the key: if HTACET doesn’t think a trial will “work”, it won’t be funded Make sure you explain fully the processes involved in the trial, particularly insofar as they include recruitment and ensuring the success of recruitment A feasibility study built into the trial is acceptable to HTACET, with second stage funding conditional on first stage success PPI is important so build in enough time and funding …and make sure that somebody who hasn’t seen your application before has a go at replicating the sample size calculation using the information you’ve given!

22. Give yourself plenty of time The process of getting HTACET money takes a year, so there’s no point in trying to rush in an application Get your team together a.s.a.p. Ensure that there’s going to be somebody in the finance department when you need to submit, and ask them how long they’re going to need to get the costing done As above for NHS Trust costs All deadlines are at 1pm, not 5pm!

23. Telephone: 023 8079 4778 Email: rds-sc@soton.ac.uk Web: www.rds-sc.org