Challenges and opportunities of combining RWE and RCT data

Interview with Thomas Debray

I’ve been working on RWE most of my career and the clinical trial data for a while now. And 20 years ago, we would have never thought about merging them together but there’s a lot of opportunities if we do this.

The recent advances in analytical methods for combining evidence from RCTs and non-RCTs, and the development of new frameworks for the inclusion of RWE in HTA have provided a greater insight on how issues around RWE uncertainty can be dealt with when estimating treatment effects for new technologies.

Join our discussion today as Thomas and I talk about this.

Thomas Debray

Thomas Debray is an assistant professor specialized in prognosis research and evidence synthesis. He develops theoretical approaches, practical guidelines, training modules and statistical software to conduct meta-analysis of individual participant data (IPD) and published studies. Thomas currently leads various international projects involving the meta-analysis of IPD and the development of statistical methods for analyzing data from pooled cohort studies.

Key references:

  1. Debray TPA, Moons KGM, van Valkenhoef G, Efthimiou O, Hummel N, Groenwold RHH, et al. Get real in individual participant data (IPD) meta-analysis: a review of the methodology. Res Synth Methods. 2015 Aug 19;6:239–309.
  2. Debray TPA, Riley R, Rovers M, Reitsma JB, Moons K, on behalf of the Cochrane IPD Meta-analysis Methods group. Individual Participant Data (IPD) Meta- analyses of Diagnostic and Prognostic Modeling Studies: Guidance on Their Use. PLoS Med. 2015;12(10):e1001886.
  3. Debray TP, Schuit E, Efthimiou O, Reitsma JB, Ioannidis JP, Salanti G, et al. An overview of methods for network meta-analysis using individual participant data: when do benefits arise? Stat Methods Med Res. 2018;7(5):1351–64.
  4. Nguyen T-L, Debray TPA. The use of prognostic scores for causal inference with general treatment regimes. Stat Med. 2019 Jan 16;38:2013–29.
  5. Sarri G, Patorno E, Yuan H, Guo J (Jeff), Bennett D, Wen X, et al. Framework for the synthesis of non-randomised studies and randomised controlled trials: a guidance on conducting a systematic review and meta-analysis for healthcare decision making. BMJ EBM. 2020 Dec 9;bmjebm-2020-111493.

Listen to this episode and share this with your friends and colleagues!


Alexander: You’re listening to the Effective Statistician podcasts, a weekly podcast with Alexander, Benjamin Piske, and Sam Gardner that are designed to help you reach your potential, lead great science and serve patients without becoming overwhelmed by work.

Today, we are talking about challenges and opportunities of combining real-world evidence and randomized clinical trial data or other clinical trial data,  a really hot topic. Stay tuned for this one. 

I have been working on real-world evidence data for quite a long time. Most of my career, I worked on it. And of course, I also worked on clinical trial data.  Twenty years ago, we would have never thought about merging them together. But there’s a lot of opportunities if we actually do this. Today, I’m really happy to talk with our guest about this. Stay tuned and learn more about it. 

Speaking of learning, there’s a lot you can learn from PSI,which is the association I’m  running this podcast with. It’s a community dedicated to leading and promoting the use of statistics within the healthcare industry for the benefit of patients. You can learn why the video on demand content library, the free registration tool or PSI webinars, if you are PSI members . And many more activities, including the Flagship conference in 2022, which will happen, hopefully, live in Gothenburg. Head over to PSI at to learn more about PSI activities and become a PSI member today. 

Welcome to a new episode of the effective statistician. Today, I’m really happy to speak with Thomas. Thomas, how are you doing?

Thomas:  Hi, I’m doing well. Thank you.

Alexander: Very good. Today, we are talking about Real-World Evidence, Frameworks, HTA. How did you come into that area? 

Thomas: Yeah, I think that’s been a long road because I’ve been in Academia for quite a couple of years now. Initially, I did, of course, the PhD, like any other academic. My PhD was about prognosis and meta-analysis. That’s why I worked on methodology for integrating some kinds of concepts and methodologies. That is commonly being used in intervention research for summarizing evidence into a different area. But for me, it kind of opened a  new perspective because I started to understand what are the developments in this space of evidence synthesis. And how it can not only contribute to learning about treatment effect estimates, but also improving risk predictions.This kind of experience opened up new pathways for me to embark on new projects. And one of the projects I  started working on, some time after my PhD, was IMI GetReal. Basically, the focus was on combining evidence from both randomized and non-randomized  settings. This is the kind of project where, the knowledge or the experience that I gained and developed. The years before became really handy to explore how this kind of standard methods can be further extended and be leveraged for more advanced kinds of questions. I think that the key point is that when I started working with this kind of methodology, it was really IMI GetReal where we really made a lot of efforts to better understand this topic. 

Alexander: If you talk about combining clinical trials and real-world evidence, or observational data, what kind of observational data are you thinking about? 

Thomas: Yeah, this is a very broad question, because typically observational studies in the field of epidemiology. I mean, we think about observational studies about pre-designed prospective studies that, you know, are designed to answer intervention questions. And, of course in an ideal case, this would kind of be the ideal observational study, where you’ve designed your study, in a way to collect all the information that you need to address your effectiveness or efficacy questions where you collect all the informational confirms that you think are important where you collect information on the outcomes that you think are important where you build in enough. I don’t know how to, you know, collect follow-up information and so forth. This is kind of the ideal case for doing a kind of observational study to think about intervention effects. And, I mean, this is certainly one of the options and one of the possibilities of how you could use observational data in combination with non-randomized data. But the reality is that there are many other types of observational data out there that are maybe less idealists. Less complicated to use for the same kind of purposes. Nevertheless, they might still contain information that can help you to better understand how certain interventions, or drugs, work outside clinical trial settings, or also in settings, when you don’t have randomized evidence available so far. How do I see observational data or non-randomized data? So I see a much bigger type of bucket of evidence. Right? So it can also be that observational studies have not been designed for answering intervention questions, but for maybe some other type of purposes so they could still have been prospective but for a whole different, you know, they could have been set up for all different purposes. But nevertheless it might be information in those studies that could still allow you to investigate treatment effect estimates to a certain degree. Then also now are increasing new data available from Francis Registries 314 data. So it’s even more challenging to use these data to learn about treatment effects. But in some situations, it’s certainly possible. But of course, it comes with certain caveats that you could avoid. Of course, if you design a prospective study.

Alexander: A prospective study, if you know what you actually need always has much better quality because it’s fit for purpose as if you have something. Secondary always comes with problems. How about seeing one of these cases? If you would have data that comes only from one study, you have one clinical trial maybe, or one observational study. Would that also be part of your considerations? 

Thomas: Yes, of course,you can indeed use it like a trial and detailer. You only have like one arm and then you could indeed look for observational data to kind of supplement these data as a kind of control arm. It’s in the same kind of space at least from my point of view. I think it belongs in the same kind of area. It’s not necessarily a synthesis, right? Because you just have done two arms if you just have like one  comparator arm and one control arm. There isn’t really any synthesis going on.Conceptually at the same kind of problems that you’re struggling with. Maybe of course if you have access to multiple control arms or multiple observational data, maybe even multiple active comparator arms, then maybe of some that you generate, some possibilities to investigate to what extent, these groups that you’re using to what extent they are, I don’t know, what is generalizable to other populations, all the settings. You create some new opportunities to learn about, to some extent, the validity of the strategy that you’re pursuing. But also, to learn about the generalizability of the effects that you’re estimating and to investigate it. It creates an opportunity to investigate, you know, if there are discrepancies. What could be possible reasons for this? 

Alexander:  When you talk about validity, what do you understand by that?

Thomas: Yeah, that’s a challenging question, right? Because validity, I think in the technical term, relates to bias. Bias are the estimate that you’re trying to quantify or estimands that you’re going to quantify that represents. I mean the estimate that I get out of it. Is it accurate, right? This is the number that statistic that you get out. Is it the true number that we should be getting out of this analysis? That, of course, is kind of theoretical. So just to convert the equation to what I mean before it is. I usually refer to bias, but in practice, bias may sometimes be a complex terminology because now, actually, if all this discussion about the use of estimands, indeed, and depending on how you define your research question very  subtle, nuances may lead to very different their estimate sometimes, not always, but sometimes it may lead to a different estimates. So the question is, how do you interpret these estimates? In the past, I guess we haven’t always put up much attention to this kind of subtleties and maybe often it didn’t really make a big difference. But when you talk about bias, it can make quite a difference. So the way if you misinterpret certain estimates, either you might call it. Okay ‘my estimate is biased’  or he might just say that ‘Look, you’re just not interpreting your statistics in the right way’. 

Alexander:  Yeah. So that bias always depends on the estimand, yeah. Estimates for one estimand would be unbiased and for another estimand it would be biased.

Thomas:  Yes, indeed. 

Alexander: Yeah, so in terms of biases,where is this bias coming from? In this type of research where we combine clinical and observational data,  what are typical sources of bias? 

Thomas: Yeah, of course, for non randomized studies, I think one of the most common sources of biases is confounding where you have other variables or characteristics that are also affecting the article but also affecting the end of treatment allocation. So I think that’s one of the most common biases that has been widely studied and mentioned as a key source of concern. 

Alexander: So that is for example, the case, if let’s say you have two treatments one works better in females than the other. And then you have a set treatment and you also have an over  allocation of females. 

Thomas: Yes.

Alexander: The overall treatment effect looks much bigger than if you would have the same allocation for, you know, in terms of gender allocation for the two treatments. 

Thomas: Yeah, so this is a common problem that runs in my settings where treatment allocation is not random. Right? I mean, treatment allocation is driven by many factors, expectations about whether the treatment will work. But maybe also some convenience or reasons  that maybe could even relate to safety issues or other kinds of aspects. But I would be surprised ,I don’t know, if there are examples where somehow there are certain treatments that are kind of quasi randomized, allocated in clinical practice. Maybe for some interventions that I  don’t know, that don’t really have a strong effect, like  it’s paracetamol, I don’t know, just random. I really don’t have a clue yet.  

Alexander: What other sources of bias? 

Thomas: The other sources, yeah,  the quality of the measurements can also lead to bias.  I know in randomized trials they are heavily controlled, right? So there’s a lot of effort making sure that you measure the right thing in the way at the right time. So the result  of emphasis on quality of the measurements.  This is much less the case in observational research where data can be collected through questionnaires. But also, they can be collected, it’s like a part of clinical routine right? So you would ask a doctor or physician to participate in your study. And to just share your data and you trust them to collect the data in a meaningful way, but they don’t have much time to help you with all the different questions. So they might take a measurement, you have a bit not that exact moment where you would like it to be taken. They might sometimes see a patient and they think ‘I don’t need that information. I already know it’s problematic. Something like I don’t like smoking history or something like that. What do they look at this patient? I know he smokes. I’m not going to ask all these questions about, How much he smokes or when he smokes. Some information may be less accurate, or it may not even be collected like missing data is also much more common in the observational studies. The timing of the measurements are taken at a contact moment, right? So when you see the patient’s, their timing may not always be an issue. But sometimes the characteristics of the patients or the measurements may be not taken at the contact moment, but to retreat from a register or some other kind of source or maybe retrospectively to a questionnaire. And so, these are also some examples of the information that you intend to collect. That there is a mismatch between the actual data that you get. But sometimes, you know, there is a mismatch and sometimes you don’t have a clue. 

Alexander: It can be, for example, that in a clinical trial. You ask, after 12 weeks of treatment, every patient comes in. And the time window, you know, beyond and before this 12-week visit is very, very small. So in a clinical trial, maybe just just a couple of days, whereas in an observational study it could be much wider. So maybe you have patients like that coming in much earlier and other patients that were coming in much later. And  there by itself already drives much more kind of variability in an observational. How does it affect the treatment difference? And if you have more to say in one compared to the other. 

Thomas:  That’s really hard to say. I think, what’s common in the present of measurement error, it’s people or researchers often, you know, like they mention, in a discussion, they have like a short sentence where it would say something like, ah we know that there is measurement error and we expect that this will dilute whatever estimates that we’re trying to obtain. But several studies have shown that it can go either way, you know, you can, it leads to dilution  which can also go the other way. So it’s really hard to anticipate the presence of measurement error but also differential measurement errors, when the amount and magnitude and support of your error differs between groups and maybe treatment arms. It’s at least for my experience really hard to say how this will affect your results. But in general, it might lead to bias but also to, you know, problems with standard errors and precision and so forth. 

Alexander: Yeah, I can understand if you have, let’s say, within a clinical study, you have measurement error for certain covariate. It’s the biggest measurement error the smaller the effect of the covariate by design. So that’s always the case. But the problem is, if you have this differential thing, for once frequent time you have captured it, very precisely. And for the other treatment, we have more measurement errors. Then I think it’s kind of like CSC. It’s a really difficult thing because then you don’t truly know which direction goes, is that the case? 

Thomas: Yes, I don’t know exactly what the situations where you just lead to dilution. I know there are certain situations where you  can prove that it only dilutes the effect but we’ve mostly looked at more sophisticated scenarios. We actually recently did a study where we looked at a meta-analysis setting where we have data from multiple, like observational studies, mostly observational studies. But  the quality of those observational studies vary. So different observational studies have different, you know, quality of measurements and different variability measurement error, but also different amounts of bias in measurements. And so it was already too complex to reflect  a single answer on how it affects, you know, estimates. But of course, you have to realize for intervention studies, the main estimate of the variance treatment effect. And so, where a bias widens becoming a variable, the confounder adjustment, right? So when we have, when we have measurement, error in confounding variables. But I can imagine if you have a differential measurement error in certain confounder variables, you might observe effects that are not really there or the other way around. You might observe very strong effects that actually are in reality much weaker. But because there is this kind of differential error going on. Yeah, it could distort a lot of signals in the analysis that you’re trying to undertake. 

Alexander: Yeah, completely sees that. There was this reward evidence event earlier this year 2021.  Talk a little bit about this and your learnings from it. 

Thomas: You have to remind me which specific event. 

Alexander: You had said this event in July, you said, was a panel discussion and so on. 

Thomas: All right. Yes. Indeed, so, did so this is a panel discussion.  We set up as a kind of follow up of two papers that we wrote together with a special interest group on comparative effectiveness research. In this panel, we had a discussion about the use of non-randomized evidence for health technology assessment and decision-making. We kind of touched upon, you know, what is the current situation, what are the possibilities of leveraging real-world evidence. But also, how can we leverage real-world evidence in a way that’s not only analytically possible but also in a way that it would be acceptable for technology assessment bodies, and maybe other kinds of authorities, that require  evidence to make decisions on a more global level. So what exactly would you like to know about that? 

Alexander: What in terms of CHT acceptance? What’s your perception? This is the real world, you know, this is a more sophisticated methodology of combining observational studies and clinical trials. 

Thomas: My impressions are that I mean different stakeholders have an interest in, you know, making more use of real-world evidence. And the big question is, how can you do that in a way that is valid. But also in a way that you can make decisions that are consistent with each other. In order for the same type of evidence in the same type of quality of evidence, you make the same decisions. And that’s of course, much harder for the evidence that is partly based on non randomized data because they’re much more differences not only in the quality of data but the way how you can analyze this data. At this moment, as far as I know, there is not really any kind of formal guidance on how to do this analysis. We try to make a first step by publishing a framework for an analytical framework for obtaining real-world evidence and for deciding which kind of methods would be more appropriate for analyzing the evidence. We try to make a first step in this direction. But I understand that a lot more effort is needed to kind of harmonize the efforts that would be needed  to present an evidence package on certain interventions, that are both randomized and non randomized acceptance or even that are not even based on randomized evidence. My impression is that there are possibilities, there are demands but we’ve not reached the phase where we have a standardized procedure for dealing with these challenges. 

Alexander: Yeah.  My perception would be the first country that probably would be interested in England. That’s  nice and I’m saying specifically, England and not Great Britain, because of its couple of other stakeholders and the United Kingdom as well. Would you agree with that statement? 

Thomas: I don’t know how other countries are looking into this space. But I know they recently published a document where they highlight the possibilities of real-world evidence. And I think we made the kind of call for four methods. Although I don’t know the details, I would have to look that up. But I see that it is indeed nice. My impression is they’re interested in getting to look into these possibilities. But I might be wrong because I’m not a member of nice. 

Alexander: Yeah, it’s always great to check the latest technical support, documents, guidance and publications. From those people that said, work on the corresponding technical groups set to consult Nice. These are the coming people from these different universities, especially in the UK. 

Very good. So we talked quite a lot about different areas of how we can combine real world trials and clinical trials. There’s a lot of new advances and they are how we can combine them, how we can make best use of all the available evidence, not just rely on clinical trials and how that can enrich it. I think it’s also a really hot topic in the regulatory space because enriching clinical trials was observational data. Enriching clinical trials.  As a source of data both for efficacy and safety is a really hot topic. I think it’s currently a scenario where we are both in each area but also in the real world evidence and in the regulatory area. We are working on similar things. So thanks so much for this really great discussion. I think that helps a lot to understand the real problems coming from the observational data is not equal observational data.It really depends on what study you are looking into. But I still have quite a lot of things to do, but it’s an amazing field and a very, very hot topic. Thomas, is there anything that is keys that you would like to have as a takeaway for the listener. 

Thomas: Yeah, a key takeaway, my impression is that there are a lot of opportunities in the use of real-world data, but it also comes with certain challenges and certain dangers. Caution is warranted in this area, but regardless,  I’m optimistic about this, taking this direction and, I’m excited about this because it opens up so many more doors, especially in spaces where, you know, we need to consider other types of evidence. My thought is let’s embrace these opportunities and just be careful how we make use of these opportunities so I think that would be my takeaway. 

Alexander: Yeah, completely agree with it. Thanks so much Thomas. Have a good time. 

Thomas: Thank you.

Alexander: This show was created in association with PSI. Thanks to Reine and her team, who helped show in the background and thank you for listening. Head over to to find the show notes and learn more about this podcast to boost your career as statistician in the health area. Reach your potential, lead great signs and serve patients, Just Be an Effective statistician. 

Never miss an episode!

Join thousends of your peers and subscribe to get our latest updates by email!

Get the shownotes of our podcast episodes plus tips and tricks to increase your impact at work to boost your career!

We won't send you spam. Unsubscribe at any time. Powered by ConvertKit