Individual Patient Data IPD Reviews and Meta-analyses

Individual Patient Data IPD Reviews and Meta-analyses PowerPoint PPT Presentation


  • 234 Views
  • Uploaded on
  • Presentation posted in: General

IPD Systematic Review /Meta-analysis . Described as yardstick

Download Presentation

Individual Patient Data IPD Reviews and Meta-analyses

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


1. Individual Patient Data (IPD) Reviews and Meta-analyses Lesley Stewart, Jayne Tierney IPD Meta-analysis Methods Group

2. IPD Systematic Review /Meta-analysis Described as yardstick & gold standard of systematic review Central collection, validation & re-analysis of source data Less common than other types of review but becoming increasingly used Overall philosophy is same as for other Cochrane reviews, process differs only in terms of data collection & analysis

3. History Established history in cardiovascular disease Established history in cancer ~50 IPD meta-analyses of treatments or screening bladder, brain, breast, cervix, colorectal, head & neck, leukaemia, lymphoma, lung, myeloma, oesophagus, ovary, prostate, soft tissue sarcoma Becoming used in a wide range of fields hernia, epilepsy, Alzheimers, dyspepsia, AIDs, hepatitis B, leg ulcers, pain relief, pregancy & childbirth…

4. How IPD Meta-analyses are Organised Carried out by international collaborative group small local secretariat multi-disciplinary advisory group trialists who provide data Developing and maintaining group requires communication and careful management Publication in the name of collaborative group

6. Why IPD? Analyses based on published data can give different answers to an IPD meta-analysis Examples chemotherapy in advanced ovarian cancer radiotherapy in SCLC chemotherapy in NSCLC ovarian ablation in breast cancer immunisation for recurrent miscarriage chemotherapy for head and neck cancer

7. Example: Ovarian Cancer

8. Ovarian Cancer Conclusions Differences due to excluded trials, excluded patients, analysis timepoint, extra follow up, analysis method Published summary data gives a more statistically ‘convincing’ result Estimates of effect size are 7.5% and 2.5% improvement in survival at 30 months When balanced against other factors, clinical interpretation of results of the two approaches may be different

9. Limits of Relying on Published Data Unpublished trials not represented Papers may present inadequate data, they may selectively report particular outcomes measure or define outcomes in differently measure or define patient characteristics differently may report at different timepoints and/or with different periods of follow up exclusion of patients Potentially restricted to fraction of trials Need to collect additional data summary data or IPD from trialists

10. Limits of Relying on Published Data Unpublished trials not represented Papers may present inadequate data, they may selectively report particular outcomes measure or define outcomes in differently measure or define patient characteristics differently report at different timepoints and/or with different periods of follow up exclude patients from reported analyses Potentially restricted to fraction of trials Need to collect additional data summary data or IPD from trialists

11. Types of review/meta-analysis

12. Comparing types of review/meta-analysis The overall philosophy for a an IPD review is the same are for other types of well-conducted systematic review of published or other summary data e.g. those of the Cochrane Collaboration. In fact in the early stages of protocol development and trial identification the processes are identical. Where the IPD approach differs from those involving published or other summary data is maintaining the collaborative organisational structure which time-consuming especially if the groupmn is big. The data collection, analysis and presentation or results are similar in a broad sense, but differ at a more practical level, because of the need to negotiate the provision of and query data from source and the greater range and flexibility of analyses possible. The collaborators conference is unique to this type of review and can be costly. The structured report of the results will be very similar to any other type of review In terms of content, but because they are reviewed by the full collaborative group they can be time-consimungThe overall philosophy for a an IPD review is the same are for other types of well-conducted systematic review of published or other summary data e.g. those of the Cochrane Collaboration. In fact in the early stages of protocol development and trial identification the processes are identical. Where the IPD approach differs from those involving published or other summary data is maintaining the collaborative organisational structure which time-consuming especially if the groupmn is big. The data collection, analysis and presentation or results are similar in a broad sense, but differ at a more practical level, because of the need to negotiate the provision of and query data from source and the greater range and flexibility of analyses possible. The collaborators conference is unique to this type of review and can be costly. The structured report of the results will be very similar to any other type of review In terms of content, but because they are reviewed by the full collaborative group they can be time-consimung

13. Benefits of the IPD Approach Improve data quality Improve analysis quality Improve interpretation & dissemination via collaborative approach When carrying out an IPD meta-analysis, there are advantages to be gained both from the nature of the data itself, and from the processes involved in reviewing evidence as an international multi-disciplinary team. Groups taking the IPD approach have done so mainly for reasons relating to relate mostly to data manipulation and analysis. These include the ability to do time-to-event and subgroup analyses, to combine different scales of measurement, to combat poor reporting and to allow in-depth exploration of patient factors. Benefits from the nature of the data are of course restricted to IPD reviews and gains and losses of the approach will depend on the disease, treatment and therapeutic questions explored. However, some of the benefits that arise from the practical processes involved, may be transferable to other types of review. When carrying out an IPD meta-analysis, there are advantages to be gained both from the nature of the data itself, and from the processes involved in reviewing evidence as an international multi-disciplinary team. Groups taking the IPD approach have done so mainly for reasons relating to relate mostly to data manipulation and analysis. These include the ability to do time-to-event and subgroup analyses, to combine different scales of measurement, to combat poor reporting and to allow in-depth exploration of patient factors. Benefits from the nature of the data are of course restricted to IPD reviews and gains and losses of the approach will depend on the disease, treatment and therapeutic questions explored. However, some of the benefits that arise from the practical processes involved, may be transferable to other types of review.

14. Data Quality: Include all Trials ‘Positive’ trials more likely to be published than ‘negative’ trials Any review that uses only data from published reports at risk of publication bias Include all relevant trials published & unpublished Unpublished trials not peer reviewed, BUT trial protocol & IPD allows extensive ‘peer review’ publication of ‘quality’ manuscript does not guarantee quality of data It is widely known that trials with positive results are more likely to be published than those with negative results. So any meta-analysis that uses only data from published reports will be at risk of publication bias., so with IPD meta-analysis we include all trials published or unpublished. Some worry that by including unpublsihed trials we may be including poorer quality trials but. IPD allows…also…It is widely known that trials with positive results are more likely to be published than those with negative results. So any meta-analysis that uses only data from published reports will be at risk of publication bias., so with IPD meta-analysis we include all trials published or unpublished. Some worry that by including unpublsihed trials we may be including poorer quality trials but. IPD allows…also…

15. Data Quality: Include all Trials Including unpublished trials (& those published with insufficient data) can be done for both IPD and summary data approaches Should be a priority for all systematic reviews Proportion of trials published will vary disease, intervention, over time Extent of unpublished data can be considerable We believe that reviewing as much of the randomised evidence as possible is of the utmost importance in systematic reviews and that all types should attempt to identify unpublished trials. Even it is not possible to get data or review information from unpublished trials you should at least list as many as possible of such trials. This then gives an idea as to whether the unpublished data might influence the results. Obviously the proportion of trials that are published will vary from disease to disease and from question to question, but the extent of unpublished data can be quite considerable… We believe that reviewing as much of the randomised evidence as possible is of the utmost importance in systematic reviews and that all types should attempt to identify unpublished trials. Even it is not possible to get data or review information from unpublished trials you should at least list as many as possible of such trials. This then gives an idea as to whether the unpublished data might influence the results. Obviously the proportion of trials that are published will vary from disease to disease and from question to question, but the extent of unpublished data can be quite considerable…

16. Identification of Trials Cervix Meta-analysis (initiated 1999) For example, at the time of initiation of an IPD meta-analysis in cervix cancer, less that half the trials were published in full and approximately another quarter had been reported only as abstracts14. In this case relying on just published data would have meant using only a fraction of the evidence and would be potentially biased. For example, at the time of initiation of an IPD meta-analysis in cervix cancer, less that half the trials were published in full and approximately another quarter had been reported only as abstracts14. In this case relying on just published data would have meant using only a fraction of the evidence and would be potentially biased.

17. Data Quality: Include all Trials Systematic reviews do not need to be IPD based to seek unpublished trials, but direct contact with trialists is useful based on “raw” rather than summary data so no difference in data from published and unpublished trials inclusion requires no special provision Systematic reviews don’t need to be IPD-based to seek information from unpublished sources, but the international collaborative nature of IPD meta-analyses does help in identify trying to identify trials. Direct communication with trialists allows us to ask each of them whether they know of any missing trials, and may provide a means of discovering trials that would not have been picked up by any other method. For example a small single centre that was not entered on any trials registers Systematic reviews don’t need to be IPD-based to seek information from unpublished sources, but the international collaborative nature of IPD meta-analyses does help in identify trying to identify trials. Direct communication with trialists allows us to ask each of them whether they know of any missing trials, and may provide a means of discovering trials that would not have been picked up by any other method. For example a small single centre that was not entered on any trials registers

18. Identification of Trials Cervix Meta-analysis (initiated 1999) In the same meta-analysis in cervix cancer 14% of trials were identified by direct communication with trialists and other experts. These trials had not been identified by literature searches, trial registers or hand searching of publication bibliographies.In the same meta-analysis in cervix cancer 14% of trials were identified by direct communication with trialists and other experts. These trials had not been identified by literature searches, trial registers or hand searching of publication bibliographies.

19. Data Quality: Relevant Trials Clarify proper randomisation, eligibility Avoid duplicate trial inclusion trials often published more than once authors differ, trial description differs, different groups of patients direct contact with trialists helps identify duplicates Combat poor reporting It is just as important that only eligible trials are included in systematic reviews. Most reviews are limited to randomised controlled trials (RCTs), although, not all studies reported as such are in fact randomised. Few trial reports give adequate descriptions of the randomisation procedure, making it difficult to be sure that a publication does in fact report a RCT. It can transpire for example that so called that "randomisation" has been done by birthdate, date of clinic visit or by alternate allocation, all of which could be biased. Thus relying only on the information presented in published papers could result in inappropriate inclusion of non-randomised trials. Again, the direct communication with trialists involved in an IPD meta-analysis allows us to establish whether trials were in fact properly randomised. It also means that any other issues relating to uncertain eligibility can be clarified. As trials are often reported several times, it is important to ensure that each eligible trial is included only once. However, in practice it can sometimes be quite difficult to distinguish multiple publications of the same trial because authors change, different numbers of patients are reported or the trial design is described differently. Once again direct communication with trialists involved in an IPD meta-analysis can resolve these issues ensuring that the only appropriate trials are included in the systematic review. All of these issues relate to inadequate or ambiguous reporting of individual trials. Therefore if direct contact with trialists, as in the IPD approach, was adopted more widely, this could potentially improve the reliability and clarity of other forms of systematic review. It is just as important that only eligible trials are included in systematic reviews. Most reviews are limited to randomised controlled trials (RCTs), although, not all studies reported as such are in fact randomised. Few trial reports give adequate descriptions of the randomisation procedure, making it difficult to be sure that a publication does in fact report a RCT. It can transpire for example that so called that "randomisation" has been done by birthdate, date of clinic visit or by alternate allocation, all of which could be biased. Thus relying only on the information presented in published papers could result in inappropriate inclusion of non-randomised trials. Again, the direct communication with trialists involved in an IPD meta-analysis allows us to establish whether trials were in fact properly randomised. It also means that any other issues relating to uncertain eligibility can be clarified. As trials are often reported several times, it is important to ensure that each eligible trial is included only once. However, in practice it can sometimes be quite difficult to distinguish multiple publications of the same trial because authors change, different numbers of patients are reported or the trial design is described differently. Once again direct communication with trialists involved in an IPD meta-analysis can resolve these issues ensuring that the only appropriate trials are included in the systematic review. All of these issues relate to inadequate or ambiguous reporting of individual trials. Therefore if direct contact with trialists, as in the IPD approach, was adopted more widely, this could potentially improve the reliability and clarity of other forms of systematic review.

20. Data Quality: All Patients Aim to include maximum randomised evidence Reinstate excluded patients, because ad hoc exclusion of patients could introduce bias cannot do this with data extracted from publications can do this with summary data from trialists collect data summary data on all patients or supplementary for excluded patients can do this with IPD Whether information on excluded patients is retained may vary by disease and intervention All individuals who were randomised in a trial should be analysed and included in the meta-analysis and any patients who were not randomised should be excluded. Failure to do this could result in unpredictable bias. Unfortunately, not all trials follow this policy when reporting their results, and patients can be excluded (sometimes covertly) from published analysis for a number of reasons. Such patient exclusions, if related to treatment, could seriously bias the results. For example, if patients are excluded because they are unable to tolerate the allocated therapy or follow the treatment schedule. As meta-analyses of IPD, do not need to rely on the original analyses, information on excluded randomised patients can be obtained and reinstated in an intention-to-treat analysis. Also, any non-randomised patients can be removed from the dataset. Collecting data on excluded patients may also be possible is summery data is obtained directly from the trialist. In this case you could ask for summary stats from an analysis of all patients by intention to treat. Although not all trials will keep records of excluded patients, our experience in the cancer field has been good and we have been able to recover information from the majority of excluded patients. All individuals who were randomised in a trial should be analysed and included in the meta-analysis and any patients who were not randomised should be excluded. Failure to do this could result in unpredictable bias. Unfortunately, not all trials follow this policy when reporting their results, and patients can be excluded (sometimes covertly) from published analysis for a number of reasons. Such patient exclusions, if related to treatment, could seriously bias the results. For example, if patients are excluded because they are unable to tolerate the allocated therapy or follow the treatment schedule. As meta-analyses of IPD, do not need to rely on the original analyses, information on excluded randomised patients can be obtained and reinstated in an intention-to-treat analysis. Also, any non-randomised patients can be removed from the dataset. Collecting data on excluded patients may also be possible is summery data is obtained directly from the trialist. In this case you could ask for summary stats from an analysis of all patients by intention to treat. Although not all trials will keep records of excluded patients, our experience in the cancer field has been good and we have been able to recover information from the majority of excluded patients.

21. Data Quality: All Patients IPD we collect it on all patients which allows us to carry out an unbiased ITT analysis of all randomised patients. In the 14 IPD meta-analyses published by our group, there was a large range in the proportion of patients excluded from the published analyses. Some of these were for reasons of eligibility, but others were for reasons that might influence the treatment effect and so could potentially bias the meta-analysis result. We used the IPD to compared an analysis of juts those patienst included on the investigators analyses with and ITT analysis of all ramdomised patients. Your can see here that in fact sometimes the exclusion of patients led to the a meta-analysis result for just the included patients being more in favour of the treatment that that of all patients. IPD we collect it on all patients which allows us to carry out an unbiased ITT analysis of all randomised patients. In the 14 IPD meta-analyses published by our group, there was a large range in the proportion of patients excluded from the published analyses. Some of these were for reasons of eligibility, but others were for reasons that might influence the treatment effect and so could potentially bias the meta-analysis result. We used the IPD to compared an analysis of juts those patienst included on the investigators analyses with and ITT analysis of all ramdomised patients. Your can see here that in fact sometimes the exclusion of patients led to the a meta-analysis result for just the included patients being more in favour of the treatment that that of all patients.

22. Data Quality: All Patients Adjuvant chemotherapy for soft tissue sarcoma Obtained data for 14 trials, 1568 patients 341 (22%) of these patients excluded from the investigators’ analyses Focusing of one of the more extreme examples…a meta-analysis of CT for soft tissue sarcoma.This is a rare malignancy and the on generally randomised fewer than 100 patients, yet up to a third of patients were excluded from some of the trials Analysing survival for all 1568 patients gives a HR of 0.90 in favour of treatment that is not significant, whereas analysing just those 1227 included in the original trial analyses gives a HR of 0.85 which is of borderline statistical significance. The clinical interpretation of these 2 results could be quote different.Focusing of one of the more extreme examples…a meta-analysis of CT for soft tissue sarcoma.This is a rare malignancy and the on generally randomised fewer than 100 patients, yet up to a third of patients were excluded from some of the trials Analysing survival for all 1568 patients gives a HR of 0.90 in favour of treatment that is not significant, whereas analysing just those 1227 included in the original trial analyses gives a HR of 0.85 which is of borderline statistical significance. The clinical interpretation of these 2 results could be quote different.

23. Data Quality: All Patients Pre-specify in the protocol if any patients will be excluded from the analysis assess impact by sensitivity analyses probably only practical with IPD There might be good clinical reasons for excluding certain types of individuals from a meta-analysis. However, to be unbiased, any exclusions need be pre-specified and applied objectively and uniformly across trials and ideally, their impact should be assessed by sensitivity analyses (including and excluding patients to determine whether it influences the estimated treatment effect). This is probably only possible with IPD. There might be good clinical reasons for excluding certain types of individuals from a meta-analysis. However, to be unbiased, any exclusions need be pre-specified and applied objectively and uniformly across trials and ideally, their impact should be assessed by sensitivity analyses (including and excluding patients to determine whether it influences the estimated treatment effect). This is probably only possible with IPD.

24. Pre-specified Patient Exclusions In the soft tissue sarcoma IPD meta-analysis, such pre-specified sensitivity analyses made no appreciable impact on the estimated effect of chemotherapy, even although these were the same factors that had been used in varying combinations to exclude patients in individual trials (table 1). A further advantage of having done these analyses is that it re-assured us as to the robustness of the results. In the soft tissue sarcoma IPD meta-analysis, such pre-specified sensitivity analyses made no appreciable impact on the estimated effect of chemotherapy, even although these were the same factors that had been used in varying combinations to exclude patients in individual trials (table 1). A further advantage of having done these analyses is that it re-assured us as to the robustness of the results.

25. Data Quality: Include All Patients IPD may not be available from all trials If data available is high proportion of all evidence (>95%?), confident of results But need to be aware that bias could be associated with trial availability sensitivity analyses Availability of data in each format may be major factor in determining approach It is possible that in deciding to collect IPD despite you best efforts you are unable to obtain data for all the trials you have identified. Probably with a high proportion of the data you can be quite confident about the reliability of the results. However, with more missing trials you need to be aware that trials availability may be biased. If you have published data on the unavailable trials then you might be bale to carry out an analysis of this data and compare the results with those from IPD to see if they match up. In fact at the outset if you know that obtaining data might be a big problem, the this might be a factor in determining to use a summary data approach.It is possible that in deciding to collect IPD despite you best efforts you are unable to obtain data for all the trials you have identified. Probably with a high proportion of the data you can be quite confident about the reliability of the results. However, with more missing trials you need to be aware that trials availability may be biased. If you have published data on the unavailable trials then you might be bale to carry out an analysis of this data and compare the results with those from IPD to see if they match up. In fact at the outset if you know that obtaining data might be a big problem, the this might be a factor in determining to use a summary data approach.

26. Data Quality: Checking Data IPD enables detailed data checking,not easily achieved with any other approach Reasons for checking not to centrally police trials or to expose fraud improve accuracy of data improve follow-up ensure appropriate analysis ensure all randomised patients are included ensure no non-randomised patients are included Collecting IPD allows detailed data checking that is not available to any other approach. In the absence of IPD, only limited verification of data is possible and usually the data extracted from publications or summary tabular data provided by trialists has to be accepted at face value. The rationale for data checking is not to police trials, but rather to make sure that we represent data accurately, improve follow-up and can carry out an intention-to-treat analysis of all randomised patients. This ensures that we have the most unbiased, up-to-date estimate of the effect of the intervention. A by product of the data checking process is that you can get greater insight into the design and characteristics of a trial. This can come in handy in interpretation of both the individual trial and meta-analysis results. Collecting IPD allows detailed data checking that is not available to any other approach. In the absence of IPD, only limited verification of data is possible and usually the data extracted from publications or summary tabular data provided by trialists has to be accepted at face value. The rationale for data checking is not to police trials, but rather to make sure that we represent data accurately, improve follow-up and can carry out an intention-to-treat analysis of all randomised patients. This ensures that we have the most unbiased, up-to-date estimate of the effect of the intervention. A by product of the data checking process is that you can get greater insight into the design and characteristics of a trial. This can come in handy in interpretation of both the individual trial and meta-analysis results.

27. Data Quality: Checking Data Standard missing data, excluded patients internal consistency range checks Randomisation balance across arms, baseline factors pattern of randomisation Follow up up-to-date equal across arms The initial checks will be validity and range checks similar to those that would be carried out on an individual trial as it was performed and checks to make sure that we have all patients Then we want to be sure that the trial appears to have been randomised properly and this involves simple checks that the patients are balanced across arms of the trial and various baseline factors - another advantage of collecting baseline patients factors potential prognostic factors Finally we check whether follow-up is up-to-date and balanced across arms. We can then ask for updated information for all patients or just those that need to be followed up further. Any missing data, obvious errors, inconsistencies between variables or extreme values can be discussed with the trialist and where necessary corrected. The initial checks will be validity and range checks similar to those that would be carried out on an individual trial as it was performed and checks to make sure that we have all patients Then we want to be sure that the trial appears to have been randomised properly and this involves simple checks that the patients are balanced across arms of the trial and various baseline factors - another advantage of collecting baseline patients factors potential prognostic factors Finally we check whether follow-up is up-to-date and balanced across arms. We can then ask for updated information for all patients or just those that need to be followed up further. Any missing data, obvious errors, inconsistencies between variables or extreme values can be discussed with the trialist and where necessary corrected.

28. Checking Pattern of Randomisation: Radiotherapy vs Chemotherapy in Multiple Myeloma I am going to show just some examples of how we would check incoming data. This is plot of the cumulative pattern or randomisation - or simply put the no of patients randomised to each arm over time. This is a trial from a CTSU meta-analysis in multiple myeloma. As you can see up until this point in 1985 similar numbers of patients were being randomised to each arm. Then for a spell all patients were randomised to the chemotherapy arm. When the investigator was asked about this, it was discovered that it was because the RT machine had broken down and all patients were just put into the CT arm. This means these patients were not allocated at random. The Oxford group were thus able to remove the non-randomised patients and still use the trial I am going to show just some examples of how we would check incoming data. This is plot of the cumulative pattern or randomisation - or simply put the no of patients randomised to each arm over time. This is a trial from a CTSU meta-analysis in multiple myeloma. As you can see up until this point in 1985 similar numbers of patients were being randomised to each arm. Then for a spell all patients were randomised to the chemotherapy arm. When the investigator was asked about this, it was discovered that it was because the RT machine had broken down and all patients were just put into the CT arm. This means these patients were not allocated at random. The Oxford group were thus able to remove the non-randomised patients and still use the trial

29. Checking Weekday Randomised This is another simple check that we do looking at what day of the week patient were randomised - most stats and dbase packages will allow you to convert a dmy date into the actual day of the week.This check for example would allow you to check if patients were allocated to treatment depending on what clinic they attended. You can see that here everything seems OK and being a UK trial there are no weekend randomisations. This course might not be the case in trial form other countries This is another simple check that we do looking at what day of the week patient were randomised - most stats and dbase packages will allow you to convert a dmy date into the actual day of the week.This check for example would allow you to check if patients were allocated to treatment depending on what clinic they attended. You can see that here everything seems OK and being a UK trial there are no weekend randomisations. This course might not be the case in trial form other countries

30. Checking Follow Up ‘Reverse’ survival curve take patients event-free, use censoring as event This is s means that follow-up for survival is balanced by arm. To do this we take juts those patients who are alive and the data they were censored as the event. You can see here that the majority of living patients have been followed for about 10 years and the remainder are follwoed to the same degree on both arms. This is s means that follow-up for survival is balanced by arm. To do this we take juts those patients who are alive and the data they were censored as the event. You can see here that the majority of living patients have been followed for about 10 years and the remainder are follwoed to the same degree on both arms.

31. Analysis Quality Combining different scales of measurement outcome data patient characteristics Subgroup analyses (patient treatment interactions) Long-term outcomes Exploratory analyses (e.g. prognostic factors, baseline risk etc.) Time-to-event analysis … An area where IPD comes into its own is in improving the range, flexibility and complexity of analyses possible. This is a list of just some of them. In cancer where we often only anticipate prolongation of survival rather than cure it is vital that we analyse not just whether an event takes place, but also when it takes place. Other time to event analyses An area where IPD comes into its own is in improving the range, flexibility and complexity of analyses possible. This is a list of just some of them. In cancer where we often only anticipate prolongation of survival rather than cure it is vital that we analyse not just whether an event takes place, but also when it takes place. Other time to event analyses

32. Analysis Quality Individual patient data used Most commonly, 2-stage analysis summary statistics generated for each trial these statistics combined in meta-analysis i.e. stratified by trial Less commonly, 1 stage analysis regression/modelling approach all patients are combined into a single ‘mega’ trial (not appropriate)

33. Analysis Quality: Time-to-event Major benefit of IPD is that it allows time-to-event analysis which accounts for whether an event happens the time at which it happens For some diseases just the ability to do such an analysis justifies the IPD approach cure is not likely, prolongation of survival time to onset of disease, time free of symptoms

34. Analysis Quality: Time-to-event Published or Summary Data usually restricted to analysis at a fixed point in time or a series of fixed time points, different trials drop in/out of analyses over time usually estimates odds ratio (OR) can make better use of summary statistics to estimate HR* may not always be practical

35. Analysis Quality: Time-to-event Individual patient data uses individual times at which each event takes place & takes account of censoring uses log rank O-E & V summarises entire survival experience estimates hazard ratio (HR) allows survival curves

36. Subgroup analysis By patient level characteristics e.g. age, sex, tumour stage group by type of patient (subgroup analysis) difficult to do from publications which usually report trial-level summaries of patient-level information e.g. mean age can be done with summary data from trialists more easily done with IPD especially for multiple analyses allows for consistent definition of subgroups across trials By trial level characteristics …

37. NSCLC Subset Analysis

38. Analysis Quality: Subgroup Analysis Does not compare the prognosis of subgroups (not comparing survival of young with old patients) Does compare the size of any treatment effect in different subgroups of patients do young patients benefit more than old patients? looking for treatment interactions Analyses stratified by trial for TTE, calculate log rank O-E and V for males only in each trial, then combine in overall HR Currently not easy to input results via RevMan Rather subgroup analysis compares the size of treatment effect in different subgroups. So for example, we can ask ….. Rather subgroup analysis compares the size of treatment effect in different subgroups. So for example, we can ask …..

39. PORT Subgroup Analyses

40. PORT Subgroup Analyses

41. Analysis Quality: Long-term Outcomes Adjuvant tamoxifen for early breast cancer 5 years of follow-up absolute survival benefit of 5% 10 years of follow-up absolute survival benefit of 6% benefit irrespective of dose reduced risk of contralateral breast cancer 15 years of follow-up benefit applicable to broader range of women greater benefit with longer duration of tamoxifen

42. Analysis Quality: Different Scales Adjuvant chemotherapy for soft tissue sarcoma 3 grading systems used Translated into a common system

43. Benefits of Collaboration Trial identification Compliance in providing data Balance in interpretation multi-disciplinary group cross cultural group Integral part of research cycle further reviews new primary research As trialists are active participants in a formal collaboration, rather than passive providers of data, there are extra benefits of the IPD approach. They may be more likely to answer questions and provide additional or missing data. Interpretation of the results of a meta-analysis is likely to be influenced by nationality, culture, clinical speciality, as well as by patient characteristics, preferences, quality of life and cost0. The multi-disciplinary, cross-cultural membership of an IPD collaborative group may provide a more global and balanced interpretation of the meta-analysis results. An additional one spin-off from these collaborations that was recognised right from the first one of our group was the generation of new ideas and establishment of collaboration on new projects. Following the AOCTG and together with results from another IPD meta-analysis, two new trials were proposed that became of the largest trials ever conducted in ovarian cancer, ICON 1, ICON2 . This trend has continued with now ICON 5 just recently completedAs trialists are active participants in a formal collaboration, rather than passive providers of data, there are extra benefits of the IPD approach. They may be more likely to answer questions and provide additional or missing data. Interpretation of the results of a meta-analysis is likely to be influenced by nationality, culture, clinical speciality, as well as by patient characteristics, preferences, quality of life and cost0. The multi-disciplinary, cross-cultural membership of an IPD collaborative group may provide a more global and balanced interpretation of the meta-analysis results. An additional one spin-off from these collaborations that was recognised right from the first one of our group was the generation of new ideas and establishment of collaboration on new projects. Following the AOCTG and together with results from another IPD meta-analysis, two new trials were proposed that became of the largest trials ever conducted in ovarian cancer, ICON 1, ICON2 . This trend has continued with now ICON 5 just recently completed

44. Barriers: Negotiating Collaboration Practical (tracing people, language differences) e-mail, web-sites, directories, search engines Educational (unfamiliar with concepts) protocol, good communication Political (difficult people, powerful groups) protocol, good communication, intermediaries Financial (money for data) ??? A key but time-consuming aspect of IPD meta-analysis is contacting trialists and persuading them to participate and provide data. Establishing contact can be difficult, for example when researchers have moved or research groups have disbanded. Increasingly hospitals and research institutes have web-sites than can be searched for staff contact information. In addition, internet directories and the regular search engines can help track people throughout the world. Maintaining contact is now much easier using e-mailing lists and website bulletins. A key but time-consuming aspect of IPD meta-analysis is contacting trialists and persuading them to participate and provide data. Establishing contact can be difficult, for example when researchers have moved or research groups have disbanded. Increasingly hospitals and research institutes have web-sites than can be searched for staff contact information. In addition, internet directories and the regular search engines can help track people throughout the world. Maintaining contact is now much easier using e-mailing lists and website bulletins.

45. Barriers: Time to Assemble Data Time taken to transfer data Advanced Ovarian Cancer (initiated 1989) 44% on paper, 39% on disk, 17% by e-mail Soft tissue sarcoma (initiated 1994) 0% on paper, 29% on disk, 71% by e-mail Cervix cancer (initiated 1999) 0% on paper, 5% on disk, 95% by e-mail For example, in our early IPD meta-analyses in ovarian cancer15,20, a significant proportion of data was supplied on paper forms which had to be manually entered into the database and then checked thoroughly. In current projects18,26 the vast majority of data is sent in electronic format, often by e-mail. ….For example, in our early IPD meta-analyses in ovarian cancer15,20, a significant proportion of data was supplied on paper forms which had to be manually entered into the database and then checked thoroughly. In current projects18,26 the vast majority of data is sent in electronic format, often by e-mail. ….

46. Barriers: Time to Assemble Data Combining different data formats (ASCII, spreadsheet, database, stats packages) used to be time-consuming software packages more compatible improved software automates more tasks Re-coding offer suggested coding Checking improved software automates more tasks queries by e-mail Data collation and provision used to be time-consuming both for trialists providing data and for the team receiving it. Most organisations now store their trial data on computer and have access to e-mail. This reduces both the time taken to transfer data and the effort involved in assembling the meta-analysis database. Software advances have also meant that data is now much more easily transferred between different types of database package, and that the format in which data is supplied is seldom a problem. Little specialist programming expertise is now required to set up and run new IPD projects.. Given that most trial data is stored on computer in a database, spreadsheet or statistical packages, that afford easy data manipulation, we finding that we can get trialists or recode their data in the format required for the final database. This not only reduces the time taken to process data centrally, it also helps ensure consistency of definition of data across trials. It also means that responsibility for modifying variables is left with the investigator who knows the data best, so there is less chance of error or misinterpretation. Of course we do stipulate that this is not necessary and that we will accept data in any old format. Automated checking of the IPD and verification by the trialists might in fact be easier than the double data extraction and double data entry sometimes required for analyses based on publications. Time is also saved by being able to e-mail a trialist to clarify an issue, rather than having to re-read published papers many times to understand ambiguities. Data collation and provision used to be time-consuming both for trialists providing data and for the team receiving it. Most organisations now store their trial data on computer and have access to e-mail. This reduces both the time taken to transfer data and the effort involved in assembling the meta-analysis database. Software advances have also meant that data is now much more easily transferred between different types of database package, and that the format in which data is supplied is seldom a problem. Little specialist programming expertise is now required to set up and run new IPD projects.. Given that most trial data is stored on computer in a database, spreadsheet or statistical packages, that afford easy data manipulation, we finding that we can get trialists or recode their data in the format required for the final database. This not only reduces the time taken to process data centrally, it also helps ensure consistency of definition of data across trials. It also means that responsibility for modifying variables is left with the investigator who knows the data best, so there is less chance of error or misinterpretation. Of course we do stipulate that this is not necessary and that we will accept data in any old format. Automated checking of the IPD and verification by the trialists might in fact be easier than the double data extraction and double data entry sometimes required for analyses based on publications. Time is also saved by being able to e-mail a trialist to clarify an issue, rather than having to re-read published papers many times to understand ambiguities.

47. Barriers: Time to Assemble Data Neoadjuvant chemotherapy for locally advanced cervix cancer Protocol and searches May 98 - Jan 99 Invite to collaborate Mar 1999 Collaborators’ meeting Sep 2000 Neoadjuvant chemotherapy for locally advanced bladder cancer Protocol and searches Dec 00 - May 01 Invite to collaborate Jun 2001 Collaborators’ meeting Feb 2002 In addition to these technical advances that make the process of data collection and checking less time consuming, there are other ways to reduce the overall length of time taken to assemble data. This is by narrowing the deadline data for the provision of data at the outset and keeping more strictly to that. You can see here that in the cervix meta-analysis we allowed nearly 18 months between the initial invitations to collaborate and the collaborators’ meeting. In the bladder meta-analysis we set the data of the collaborators meeting at the outset and only allowed about 6 months for data collection. A more intensive and nervous 6 months granted, but we did managed to obtain most of the data by the collaborators’ meetingIn addition to these technical advances that make the process of data collection and checking less time consuming, there are other ways to reduce the overall length of time taken to assemble data. This is by narrowing the deadline data for the provision of data at the outset and keeping more strictly to that. You can see here that in the cervix meta-analysis we allowed nearly 18 months between the initial invitations to collaborate and the collaborators’ meeting. In the bladder meta-analysis we set the data of the collaborators meeting at the outset and only allowed about 6 months for data collection. A more intensive and nervous 6 months granted, but we did managed to obtain most of the data by the collaborators’ meeting

48. Barriers: Resources Time & resource? often reported as MUCH more expensive but technology advances to cut costs/ time need empirical data Cost of Collaborators’ Conference not encountered in other types of review It is often said that IPD reviews are much more time consuming and expensive than other forms of systematic review13. However, this assumption is often based on comparison with meta-analyses of the published data that have been done with minimum effort, rather than with high quality Cohrane reviews. Furthermore, since resource comparisons were first discussed, the technological progress that I have just talked about has considerably reduced enabled some of the labour-intensive aspects of IPD meta-analysis, allowing them to be done more easily, quickly and cheaply. It is often said that IPD reviews are much more time consuming and expensive than other forms of systematic review13. However, this assumption is often based on comparison with meta-analyses of the published data that have been done with minimum effort, rather than with high quality Cohrane reviews. Furthermore, since resource comparisons were first discussed, the technological progress that I have just talked about has considerably reduced enabled some of the labour-intensive aspects of IPD meta-analysis, allowing them to be done more easily, quickly and cheaply.

49. Collaborators’ Meeting Integral part of IPD MA IPD MA a collaborative project Incentive to collaborate Trialists have opportunity to discuss results to challenge the analysis to discuss interpretation and implication of results Sets a deadline to which secretariat and trialists have to work Most of the groups that we know working on IPD meta-analysis hold a closed meeting of collaborators meeting where the results are presented for the first time. This serves to re-emphasise the collaborative nature of the project and can act as an incentive to collaborate at the early stages. It give the whole group a chance to discuss the results and give valuable clinical input, challenge the analyses that we have done - and suggest further exploratory analyses. We usually try to to second guess them on these so that when they say ‘why didn’t you do it this way or why didn’t you do this analysis ‘ we say ta ta here is one I prepared earlier. The process is very useful for us to get a feel for how the discussion of the paper should be. Finally as with anything it gives us and the trialist a deadline to get their data in and us to get the complete data set well understood and well-described Most of the groups that we know working on IPD meta-analysis hold a closed meeting of collaborators meeting where the results are presented for the first time. This serves to re-emphasise the collaborative nature of the project and can act as an incentive to collaborate at the early stages. It give the whole group a chance to discuss the results and give valuable clinical input, challenge the analyses that we have done - and suggest further exploratory analyses. We usually try to to second guess them on these so that when they say ‘why didn’t you do it this way or why didn’t you do this analysis ‘ we say ta ta here is one I prepared earlier. The process is very useful for us to get a feel for how the discussion of the paper should be. Finally as with anything it gives us and the trialist a deadline to get their data in and us to get the complete data set well understood and well-described

52. Barriers: Software Primary analysis most IPD groups use own software carries out stratified analysis and produces graphical output Input into RevMan For time to event data use IPD or TTE outcome enter log rank summary statistics from primary analysis For TTE or other outcomes use generic inverse variance enter estimate and SE from primary analysis not easy to enter subgroup analyses There is no commercially available software that provides the full range of analyses and plotting that we need for IPD meta-analyses. Like ourselves, most groups have had to developed their own applications to analyse the results and produce the graphical output, usually by customising existing packages. Thus access to suitable software may pose a significant barrier to new groups contemplating the IPD approach. The Cochrane Collaboration software for producing systematic reviews (RevMan), which provides an excellent means of managing information about the trials and their associated references, has an individual patient data option. However, to use this requires pre-processing and analysis of the IPD to provide trial level summary statistics before it can be entered to RevMan. RevMan is not able to analyse the data on each individual patient. There is no commercially available software that provides the full range of analyses and plotting that we need for IPD meta-analyses. Like ourselves, most groups have had to developed their own applications to analyse the results and produce the graphical output, usually by customising existing packages. Thus access to suitable software may pose a significant barrier to new groups contemplating the IPD approach. The Cochrane Collaboration software for producing systematic reviews (RevMan), which provides an excellent means of managing information about the trials and their associated references, has an individual patient data option. However, to use this requires pre-processing and analysis of the IPD to provide trial level summary statistics before it can be entered to RevMan. RevMan is not able to analyse the data on each individual patient.

53. Summary: Benefits of IPD Carry out detailed data checking Ensure quality of randomisation & follow up Update follow up information* Carry out time-to-event analyses Only practical way to do subgroup analyses More detailed exploration of patient characteristics Ensure appropriateness of analysis* More flexible analysis of outcomes* These next 2 slides just summarise the benefits of IPD and which benefits might also be attainable if you approach a trialist for summary dataThese next 2 slides just summarise the benefits of IPD and which benefits might also be attainable if you approach a trialist for summary data

54. Summary: Other Benefits More complete identification of trials* Better compliance in providing missing data* More balanced interpretation of results* Wider endorsement & dissemination of results* Better clarification of further research* Collaboration on further research* *Also possible with summary data obtained from trialists And as discussed some of these derive from nature of IPD and other from nature of process (collaboration) And as discussed some of these derive from nature of IPD and other from nature of process (collaboration)

55. Summary: Barriers Likely to be more costly and time-consuming Range of expertise and skills required But differences between IPD and other types of systematic review may not be so great intense development phase longer data collection phase intense analysis and collaborators’ meeting phase IPD projects can be run concurrently Practical / political issues There are also the barriers, which is addition to those described it the requirement for organisational, data management, analysis and negotiation and communications skills. However may aspects are not unlike other types of systematic review in that they have an intense phase when the questions and development of the protocol take place. There is a longer data collection phase and certainly the scale of the analysis together with organisation of the collaborators meeting is another resource intensive phase. However, a number of different IPD projects at different stages could be run together…exactly what our groups does. There are also the barriers, which is addition to those described it the requirement for organisational, data management, analysis and negotiation and communications skills. However may aspects are not unlike other types of systematic review in that they have an intense phase when the questions and development of the protocol take place. There is a longer data collection phase and certainly the scale of the analysis together with organisation of the collaborators meeting is another resource intensive phase. However, a number of different IPD projects at different stages could be run together…exactly what our groups does.

56. To IPD or not to IPD? Some benefits may be potentially achieved by obtaining additional summary data Collecting summary data may entail too many tables,more work for trialists requires trialists to interpret and act on instruction eg. redo analyses, reclassify data So why not collect IPD ? While a number of the benefits may be obtained by collecting additional summary data….this may not be as trivial a task as it may seem for both reviewer and the trialist. Contacting and negotiating provision of data will still be required and the trialist may have to do more or the same level of data preparation as they would to provide IPD. So if you are prepared to go that far to improve the quality of your data and review that why not go the whole hog and collect IPD?!While a number of the benefits may be obtained by collecting additional summary data….this may not be as trivial a task as it may seem for both reviewer and the trialist. Contacting and negotiating provision of data will still be required and the trialist may have to do more or the same level of data preparation as they would to provide IPD. So if you are prepared to go that far to improve the quality of your data and review that why not go the whole hog and collect IPD?!

57. To IPD or not to IPD ? This final just gives you some pointers if you think that you have a review that could benefit from the collection of IPD.This final just gives you some pointers if you think that you have a review that could benefit from the collection of IPD.

  • Login