What should you expect when outsourcing NMAs?

Transcript

What Should You Expect When Outsourcing NMAs?

[00:00:00] Alexander: Welcome to the third and last day of the conference of the effective statisticians. This is the second conference after we had [00:00:10] one in 2023 and last April already. And stay tuned for my presentation at the end of. [00:00:20] This three hour session, because then I will also speak about the next conference that will happen actually later this [00:00:30] year.

[00:00:30] Alexander: And that will include also some, some updates, some changes, some additions. Because I want to make it even more of [00:00:40] a community event. And so stay tuned for that. Our first speaker, Thomas, is an academic who [00:00:50] turned into Professional support for the pharmaceutical industry and beyond. He’s an expert in real world evidence, precision [00:01:00] medicine, indirect comparisons, evidence synthesis, all these kind of different things.

[00:01:05] Alexander: And he has his own company called Smart [00:01:10] Data Analysis and Statistics. And he will talk about what to expect when outsourcing NMAs. And just from my personal point of [00:01:20] view, I know that this can be a very, very painful process. I’ve worked with lots of different vendors.

[00:01:29] Alexander: [00:01:30] And usually it was quite a mixed back in terms of yeah, experiences. Let’s put it that way. So I’m super happy about Thomas [00:01:40] presentation. Just maybe one last thing. Animes is really, really an. important thing. And whenever you want to launch [00:01:50] or commercialize a product in a, in a crowded space where there’s more than, you know, standard of care, then you definitely will need to [00:02:00] work on this and not only for designing your phase three studies, but also later on.

[00:02:07] Alexander: Thanks so much. And the floor is [00:02:10] yours.

[00:02:11] Thomas: I would like to start with setting the scene a bit to, you know, imagine a bit of a situation where you might be thinking about an NMA. So for example, imagine that, you know, your [00:02:20] company developed a drug. So you went through all the different phases to, you know, to establish efficacy, safety and support.

[00:02:26] Thomas: And so you finally come at the stage where, you know, [00:02:30] you’ve demonstrated the value of the new drug to regulatory bodies and they have approved. your drug. So, you know, you’re, you’re ready to basically start [00:02:40] delivering drugs to patients. However to demonstrate that you know, this drug is really adding value to patients and to obtain reimbursement from, from local authorities.

[00:02:49] Thomas: [00:02:50] Some additional steps are typically needed. And so, you know, as a company, you decide, okay, let’s, let’s go for that. Let’s explore how can we convince now the [00:03:00] reimbursement agencies that they also should, you know, they get on board and you know, facilitate access of, of our drug to, to patients. So, you know, you start having conversations with them.[00:03:10] 

[00:03:10] Thomas:You start asking, okay, you know, this is the evidence we have so far. What do you need from us to, you know, to show that. To be convinced that you should prescribe and reimburse our drug [00:03:20] for individual patients. And so from having these discussions, you realize that there are different reimbursement agencies.

[00:03:26] Thomas:Each country has their own, you know, agency, their own requirements and so forth. [00:03:30] So you need to start digging a bit for the different evidence that they will meet. And so essentially you might decide, you know, let’s set a statistician to do this work. You know, let’s look in house. Who has some [00:03:40] time to do this because it’s quite urgent, it’s important who has some experience who can help us with that.

[00:03:45] Thomas: And so you free some resources, maybe you get a vendor on board to help with that. [00:03:50] And so they start working on this, right? So they start looking at all the trials that you have done. They start asking questions about, ah, okay, what’s what kind of evidence does the regulator sorry, the, the agency, what kind of [00:04:00] evidence do they need what’s the standard of care are there any other treatments available?

[00:04:04] Thomas:And so you might discover quite soon that different agencies, they will come with [00:04:10] different suggestions or recommendations on, on, you know, how to look at existing evidence to decide, you know, what are relevant comparative to, you know, to which you, you should [00:04:20] demonstrate added value. What is actually some of care?

[00:04:22] Thomas: What do we define us as providing added value? So each of these agencies might call me for a whole different set of requirements [00:04:30] and recommendations. And so, of course, you have your statistician. So, you know, they start working on all this putting getting all this piece of evidence. Maybe you realize that, ah, you know, we need [00:04:40] to conduct a systematic literature review to actually identify the relevant studies.

[00:04:44] Thomas:So, you know, there’s some additional people you get on board to get this type of evidence as well. And so after a long [00:04:50] process, you have all the studies that you think are relevant for the for the agencies you’re currently talking with. And you ask your statistician, you know, please, you know, summarize the evidence from all these studies.[00:05:00] 

[00:05:00] Thomas:And they start writing in SAP and, and start putting all the evidence, they start preparing, you know, slide and, and, and documents for you to review and, and prepare your [00:05:10] for the for the discussions. And at some point, you know, you, you submit this evidence and at the same time, maybe you already started talking to other agencies, you know, [00:05:20] from, I don’t know, maybe Germany or France and so forth.

[00:05:21] Thomas: And so, you know, these discussions keep coming back and forth. So, you know, as soon as your statistician has developed some outputs or your team has developed some outputs. [00:05:30] They’re back at work because the other countries are also asking for evidence that they need to prepare. And then you realize that, of course, the type of evidence that they want is [00:05:40] different, the type of output that they need is different.

[00:05:41] Thomas: So a lot of changes are needed, basically, along the way. And it’s possible that sometime during this process, Your statistician, you know, they, [00:05:50] they get ill or they, they, you know, they fall out. You no longer, they’re no longer available. So you have to look for a replacement at some point. And then of course once you start, [00:06:00] you know, you find a replacement or maybe you’re doing it yourself, you start looking into all the work that the statisticians don’t.

[00:06:06] Thomas: And maybe you don’t quite understand because there’s so many different [00:06:10] requests, right? Different. Agencies asking for different type of evidence. Maybe there’s different versions because, you know, over time some of the analysis have changed or some of the evidence has changed. [00:06:20] And so maybe you get lost right in all the different pieces of materials that are there and you don’t know where to start.

[00:06:25] Thomas:So there’s a chance that maybe you start from scratch. Maybe you don’t. Maybe you don’t [00:06:30] start from scratch, but maybe you have like a document and you copy the wrong numbers or you interpret them wrongly, right? So when you present the evidence at some point, maybe it’s [00:06:40] enough, maybe it’s sufficient, maybe you will get the approval.

[00:06:42] Thomas: And maybe it’s also a possibility that agencies are not convinced. Maybe they will conclude that, look, the quality of the evidence [00:06:50] is not up to date. It’s outdated by now. We have been talking about this one year and, you know, one year later, this new, new products have entered the market. Or maybe, you know, you have more follow up on your, [00:07:00] your trials and, and they want to, to, that you consider that evidence as well.

[00:07:04] Thomas: And so you have to get back right to the start and, and start again. And so this can be really [00:07:10] long and, and painful process and it’s hard to, as you can imagine, to kind of redefine how that process will look like and to kind of. You know, make it very specific in a, in [00:07:20] a scope of work that, that you would want to outsource and so this is the kind of the challenge I would like to touch upon today.

[00:07:25] Thomas: Like, how, how do you make this process easier and more manageable? Are you going to do it in house or [00:07:30] not? Either way, what are the kinds of the key issues you may want to look at to make sure that this process is successful? So, and I, I was looking a bit [00:07:40] into you know, some, some, some numbers, right.

[00:07:42] Thomas:About, you know, how does this process usually goes, you know, from, from direct development to post marketing you know studies and, and [00:07:50] market access. And apparently there have been some, some studies that I’ve looked into the time that, you know, that passes between these different stages. And so, you know, you may think that perhaps, you know, after a marketing authorization, you [00:08:00] know, we’re done, you know, that’s it.

[00:08:01] Thomas:We just have to have maybe a few, a few discussions to, to be, to get our product reimbursed, but it’s not that simple. It can take quite a bit of time. [00:08:10] And so apparently, at least in Europe, the average time that, that elapses between, you Is on average about [00:08:20] 504 days, but there’s a lot of variability within these different countries and to get actually reimbursed, you have to start negotiating with each of these different countries about, you know, to [00:08:30] demonstrate, you know, does our product offer something new or does it offer some value on top of what they are already reimbursing.

[00:08:37] Thomas:And so we can imagine it can be quite, [00:08:40] quite a painstaking procedure to, to interact with all these different countries. And at the same time to have this delay, right, of 510, four days or [00:08:50] more between the time that your product got approved and the time that it can actually be, you know, reimbursed on individual countries.

[00:08:58] Thomas:So it’s quite a [00:09:00] painstaking process. And so this brings me back to the point. So what is actually a network meta analysis? So, and why is it so relevant, right? In, in reimbursement negotiations, and the main [00:09:10] thing is that, of course, in clinical trials, especially for market authorization, the key focus is often on establishing efficacy and safety.

[00:09:17] Thomas:And often for this type of studies and trials, [00:09:20] the reference, product or the control treatment would be placebo or sometimes standard of care. And so that evidence is used for, you know, [00:09:30] for deciding on whether the drug can indeed, you know, whether it works as intended and whether it can be used effectively.

[00:09:36] Thomas: Now, of course, when you are talking to health technology agencies, [00:09:40] they are asking different types of questions. They want to know. Whether it makes sense for them to reimburse one additional product that is entering the market. And so they want, typically they want to know whether this [00:09:50] new drug is, you know, is adding value on top of the existing drugs that are already being reimbursed and already on the market.

[00:09:56] Thomas:So they are often interested in a question that [00:10:00] involves multiple treatments. They want, if there are multiple treatments available, they want to understand how does this new drug compare to all the existing drugs that are already on the market. And your [00:10:10] clinical trials, typically, they will not give you an answer to that question unless you have like run clinical trials where you are, you know, you’re randomizing patients, not only to placebo, but also lead to all this [00:10:20] alternative drugs, which of course, this will not.

[00:10:22] Thomas: I mean, this is not a common practice in, in in clinical trials. So what can you do? Like, how, how can you then, you know, establish [00:10:30] some evidence that your drug is adding value on top of these existing drugs. And this is what a network meta analysis is doing. Basically, it’s comparing multiple treatments against each other by [00:10:40] looking at available clinical trials that have been published.

[00:10:43] Thomas:So, for example, you may have your trials that compare your new drug to placebo. And in the literature in clinicaltrials. [00:10:50] gov and other sources. You will find older trials for, you know, the existing drugs that just like you that have compared their drug to placebo, or maybe in [00:11:00] some instances to another active comparator.

[00:11:03] Thomas: And so by putting all these trials together, you can basically make a network of all the existing evidence and you can start [00:11:10] comparing indirectly how different drugs perform against each other by, you know, by by connecting them to in this case, the placebo treatment. And so [00:11:20] the idea of a network meta analysis that you, all the existing evidence is, is being summarized.

[00:11:24] Thomas:So if you have two trials comparing your new drug to placebo, that evidence will be summarized, but also [00:11:30] by having these two trials and another trial comparing drug B to placebo, you also are getting indirect evidence between your drug and, you know, this drug B. And so this indirect evidence will [00:11:40] also be summarized.

[00:11:42] Thomas: Incorporated in your network meta analysis and essentially your network meta analysis will allow you to draw any comparative inferences between any two [00:11:50] drugs, as long as there is a connection one way or another between them. So that’s the main idea of a network meta analysis. Now, the implementation of a network meta analysis, of course requires quite a [00:12:00] lot of modeling and assumptions.

[00:12:01] Thomas: I will come back on that. So it’s, it’s not, it’s not so straightforward to get these, these estimates. And also. Even if you get this estimate, it’s also not always that fast [00:12:10] forward to interpret them in a, in a correct fashion. Regardless, the, the main advantages of, of the network meta analysis are that you can, first of all, increase Estimates [00:12:20] of of your treatment effect as well as, you know, increasing precision because you’re, you know, pulling evidence together so you can, you know, you can get more precision.

[00:12:28] Thomas: The 2nd issue is that [00:12:30] you can compare interventions for which maybe there are no head to head trials. And that’s that’s the main interest usually in in health technology assessments documents where you want to establish [00:12:40] like, look, we didn’t run a randomized trial where we compare our new drug to all these.

[00:12:45] Thomas:Existing competitors, but with a network meta analysis, we can show that our drug offers [00:12:50] maybe superiority for this particular sub populations. And then the final goal, for example, of a network meta analysis could be to provide a ranking of all these interventions [00:13:00] and to show that, look, you know, our new drug is at the top for these specific endpoints or perhaps for these specific subgroups.

[00:13:08] Thomas:So then the next question is like, [00:13:10] because I mentioned, so when, when do you typically run a network meta analysis and of course the most common scenario is in a post marketing setting, right? Once your drug has been approved [00:13:20] you want to negotiate reimbursement. So initially either when the drug just enters the market.

[00:13:28] Thomas: But it could also [00:13:30] be later on during the process when maybe your drug is already approved for reimbursement, but after some years, you know, new competitors are entering the market, or maybe new evidence has appeared about your [00:13:40] drug and HTA agencies may, you know, question the, the current price.

[00:13:45] Thomas: or the reimbursement rates, and you may have to, you know, run a new network meta analysis to show [00:13:50] that even after the new available evidence, your drug is still adding value as originally was the case. So network meta analysis, you know, [00:14:00] may happen multiple times. in a post marketing setting and also are used sometimes to develop clinical guidelines.

[00:14:06] Thomas: So my point is just that network made analysis are often used in a post [00:14:10] marketing setting, but they can also be used before that at different stages. So for example, you could consider a network made analysis when you are just, you know, very early [00:14:20] on. in a drug development phase. So maybe you’re just even thinking about phase one trials, but you could already use a network analysis and a systematic literature [00:14:30] review to explore what is actually out there.

[00:14:32] Thomas: You know, what, what is the landscape of the the, the, yeah, what is the landscape where we try to place our drug in the future? How does [00:14:40] it look like? What are the competitors? How big is the market? How many patients? are there that could benefit from taking this drug? What’s the efficacy of [00:14:50] the existing treatments and how could we differentiate against these existing treatments with our new drug that we are still investigating?

[00:14:58] Thomas: But it’s good informed, you know, [00:15:00] the pipeline, the future pipeline on what’s are potential strengths of the drug that you’re developing, as well as potential weaknesses of limitations. So, but this is of [00:15:10] course entirely optional. Now, where NMAs are sometimes used as well, is in much later on in the drug development phase.

[00:15:16] Thomas:So when you already have designing phase three trials, [00:15:20] so when you already have, like, very clear, you know, evidence that your drug might You know, be working very well and in specific populations as you want to establish [00:15:30] the efficacy and safety of your drug. And so in that case, an NMA could be run either to design your trial to inform like, okay, which endpoints are we going to focus on?

[00:15:39] Thomas:Like, what are [00:15:40] the primary endpoints where we think that we could differentiate on existing medications that are already in the market? Or what are the key populations we want to focus on in our [00:15:50] clinical trial? Which comparators? For example, in case of standards of care. So, and also in how that impacts sample size calculations.

[00:15:58] Thomas: On the other hand, you could [00:16:00] also do an NMA just as soon as you finish your phase three trial. So you’ll tend all your results from your clinical trial and they look promising, you know, you have significant results and so forth. [00:16:10] But still you might already, even before you, you start working towards your.

[00:16:15] Thomas: All the next steps, you might already want to investigate like, okay, but what does that mean? These [00:16:20] results like, how do these results compare to all these products that are on the market for which we will have to, you know have conversations about with reimbursement agencies. What does that mean? And how, you know, where [00:16:30] do we see some differentiation between our product and all these existing products?

[00:16:34] Thomas:And then the final area where meta analysis could be of interest is, is upon market authorization, but [00:16:40] it is very unconventional. But it can be used sometimes, meta analysis sometimes used to pool the, the evidence of the available clinical trials to either provide an overall summary of the [00:16:50] efficacy or to analyze less frequent outcomes.

[00:16:53] Thomas: perhaps to evaluate conflicting study results or study subgroups. So it’s not necessarily the focus of a network in that case, but [00:17:00] even then integrating evidence from, from alternative treatments could help to contextualize some of the results in a, in an approval process. But as I said, this is, I [00:17:10] mean, this is very unconventional and not, I’ve not seen examples like that.

[00:17:14] Thomas: So I’d like to move on and discuss a bit, what are the challenges to conducting an NMA because I, as I mentioned. There’s [00:17:20] quite a few things to, to think about beyond the methodology that can complicate the process of, of, you know running and, and getting results that are actually impacting [00:17:30] decisions.

[00:17:30] Thomas: And it’s the, the, the execution of a network meta analysis becomes already. challenging early on, because you have to define the research question, just like in a clinical [00:17:40] trial. But whereas in the clinical trial, you know, you can, you can shape the design of your trial to really you know, to be aligned with your research question.

[00:17:48] Thomas: This is much more difficult in the [00:17:50] network meta analysis because You don’t have control of, of, of the data, right? I mean, you’re pooling data that’s already been generated. So defining a research question, I mean, you can come up with [00:18:00] research questions that are like, that, you know kind of will give you like all the evidence you need, but it’s very well possible that you will never find the evidence to answer those questions.

[00:18:09] Thomas:So it’s [00:18:10] really a balancing act between defining a question that is relevant, but at the same time, defining a question that is, that can realistically be, be answered given the evidence that is, that is out [00:18:20] there. And also a question that once it’s been answered, it actually provides an answer to the, to the relevant people, right?

[00:18:27] Thomas: But, you know, the answer that your, [00:18:30] your NMA will provide actually informative, right? But it can inform the decisions that need to be made. And so this balancing act can be, can be quite a challenge in, in, in, in some [00:18:40] situations. The second issue, and this relates also a bit back to the first issue, is, is how, how are you going to get the relevant evidence to conduct your network meta analysis?

[00:18:48] Thomas: How are you going to find all [00:18:50] these studies that compare, you know, the, the efficacy of competitive treatments against against placebo or standard of care. How are you going to define you know, what is standard of care, which [00:19:00] is of course part of the research question. How are you going to decide whether it’s high quality which treatments are you going to include or not and so forth.

[00:19:07] Thomas:So sourcing this evidence [00:19:10] is also quite a painstaking process that can take quite a bit of time. The third item relates also again to the previous ones as well as to the steps that follow [00:19:20] after sourcing the evidence and that’s You know, finding the right people with the right expertise who are available at the right time.

[00:19:26] Thomas:So it’s really who’s going to run this analysis? Do you have access to these [00:19:30] people in house? Do you have, do they have enough bandwidth to run this analysis? Are they, you know, and are they available? And do they have the knowledge that is needed? Because I mean, you definitely need a [00:19:40] statistician, but only dealing with statisticians is not going to give you the answers that you need.

[00:19:45] Thomas:Typically this type of processes need a multidisciplinary approach to, you know, [00:19:50] to come up with all the evidence in a fashion that is needed for decision making, then afford challenges that. Methodology for network meta analysis has been evolving quite a bit [00:20:00] over time. So things that are possible today were not possible, I don’t know, a couple of months ago.

[00:20:05] Thomas: They were, they were not, the solutions didn’t exist. And at the same time, new [00:20:10] solutions are being developed that may not exist yet today. And this means that not only the, the technology is changing, but also software to run this analysis is changing and guidelines are [00:20:20] changing. So, different agencies may have different recommendations about how to conduct an NMA what should be part of your NMA what you should look at, how you should report, how you should look at [00:20:30] sensitivity analysis and so forth.

[00:20:31] Thomas:So, these things are changing over time and, and this will likely not be set in stone in the near future. And this means also that you need to have [00:20:40] some degree of flexibility to address all these, these changes to deal with change the methodology, but also deal with different requirements.

[00:20:47] Thomas: For example, when you’re [00:20:50] interacting with all these agencies. They may be asking about different methodologies, different statistical models, but they may also be asking about different for example, different treatments to [00:21:00] be incorporated in your network or different populations to be targeting for your evidence synthesis.

[00:21:05] Thomas:So you need to have a high degree of flexibility to deal with all these different requests. And to at the [00:21:10] same time to keep it manageable, then all this flexibility and changes with respect to requirements and resources [00:21:20] also introduces quite a bit of risk in terms of you know, complexity, but also in terms of there’s a risk of either producing results that are affected by errors.[00:21:30] 

[00:21:30]Thomas: Because of all this, you know, all these variations that are going on. Yeah, or, and there is a risk that maybe the evidence that you’re producing is not going to be to the satisfaction of the agency you’re [00:21:40] interacting with. And so identifying this risk early on is, is key to ensure that. There are no surprises at the end of this whole process.

[00:21:47] Thomas: And as a final item, I mentioned here [00:21:50] the need to generate necessary documentation and outputs, because a network based analysis, like it says, it’s an analysis. But an analysis is not a complete picture of an [00:22:00] interaction with a reimbursement agency about evidence. You need to be able to document all these different decisions that were made along the process.

[00:22:08]Thomas: As well as all the outputs, right, [00:22:10] to formulate a narrative around all these outputs that is coherent and consistent with the evidence that you’re presenting. And this also requires some academic skills to yeah, to [00:22:20] present it in an appropriate fashion. Yeah, there’s a few more technical challenges I just want to go over briefly.

[00:22:25] Thomas:Just to say, like, look, an NMA is, I mean, the items that I [00:22:30] mentioned is. Are quite generic and probably also apply to a lot of other type of analysis. And when it comes specifically to network meta analysis, there are some challenges that, you know, that are unique to [00:22:40] network meta analysis. And for network meta analysis, for example, some of the choices are related to how, what kind of model are you going to be using to conduct this network meta analysis, you know, what, what end points [00:22:50] are we talking about?

[00:22:51] Thomas: How are you going to incorporate, let’s say like, like time to events censoring or repeated measurements. What kind of link function and so forth. So all this [00:23:00] type of technical modeling choices, which may seem trivial, but you know, this might be discussion point later on when interacting with with agencies.

[00:23:09] Thomas:Now, a key [00:23:10] problem in network meta analysis relates to the assumptions that are needed to to put all this evidence together. And this evidence, I mean, if these assumptions, if they’re not met. You [00:23:20] can end up with networks that are, you know, either affected by network inconsistency, which means that if you compare the evidence from head to head trials, so for example, comparing a trial with your [00:23:30] drug to a placebo, when that evidence would be inconsistent with evidence that can be obtained indirectly for example, because there is a trial that compared your drug against some other active [00:23:40] drug.

[00:23:40] Thomas: And this, and the other thing is heterogeneity is when your similar treatment effects are not homogeneous. So it means that there is evidence that, you know, the, the treatment [00:23:50] effects differ across trials. So maybe trial one, you find a, an effect of, I don’t know, a knots ratio of 0. 8 and in another trial.

[00:23:58] Thomas: the odds [00:24:00] ratio for the treatment would be something very different. And so the, the, these two aspects they can, they can be problematic. They can, at least they can challenge the [00:24:10] interpretation of your summary estimates, and they can severely affect. you know, how to make decisions based on, on this on their presence.

[00:24:19] Thomas:So if your [00:24:20] network is affected by consistency, that means that your some assumptions are not valid. So that, that means that, okay, what, what, you know, can you take the, the summary estimates and at face value? Probably not. [00:24:30] Now, in the case of heterogeneity, I mean, it’s not necessarily a problem, but the point becomes problematic when the heterogeneity is extensive.

[00:24:37] Thomas: And there’s no clear answer. Okay, the average treatment [00:24:40] effect is of this magnitude, but we also see that there’s like a wide variation in treatment effect across trials. So what does that mean? So what, what can we expect enough of this treatment [00:24:50] in a specific population? So this can bring some additional questions in, in terms of decision making as well.

[00:24:55] Thomas: So I’m dealing, addressing, identifying these problems and dealing with it is, is again, it’s [00:25:00] quite a trivial challenge. And then there are other challenges as well, like this potential for bias on some trials, maybe of higher quality than others. So what do you do when, when you have to incorporate [00:25:10] some trials but they are not of great quality.

[00:25:13] Thomas:: How do you deal with that? How do you deal with effect modification when you think that there are certain. Patient level covariates [00:25:20] that do affect the efficacy of the treatment and for which a distribution varies across trials. How do you deal with that? What kind of sensitivity analysis will you conduct to explore [00:25:30] whether your results are robust and to what extent, you know, these treatment effect estimates can be generalized across different populations and sub populations?

[00:25:38] Thomas: And increasingly often also [00:25:40] one of the questions is is, should you incorporate patient level data from your own trials? Is there a benefit of trying to model your, you know, the patient level data from the trials that we have [00:25:50] at hand? Is there a, is it worth pursuing that and how does it affect, you know, the complexity of all these analysis that I mentioned before?

[00:25:57] Thomas: So these are a lot of questions, right, that you have to [00:26:00] consider internally, but also externally with the stakeholders that, that, that, you know, are requesting this type of analysis and it will have an impact. On the, on the network metanalysis process [00:26:10] of the whole, and just to make things a bit more complex.

[00:26:12] Thomas: So all these questions that I mentioned so all the answers to all these questions. May differ depending on which agency you’re talking about. So as [00:26:20] you can see, here is kind of an overview. It’s not super recent. It’s from a few years ago, 2019, but it gives an overview of requirements from different countries about how to run your NMA.

[00:26:29] Thomas: So what, what to [00:26:30] look at, what to report, what kind of checks to do and support. And so each, each country has different, different requirements. And this is on generic level, but you will find that also if you start [00:26:40] looking more specifically, you start talking about specific populations or, or comparator arms and support.

[00:26:45] Thomas: Also on that level, you will have, you know, depending on who you’re, what, what [00:26:50] agency you’re talking about. They might have different preferences on how to deal with these things. So all of these questions. That I, I mentioned, I mean, the conversation might look [00:27:00] quite different depending on with whom you’re having that conversation.

[00:27:03] Thomas: Now, of course, there is some, some light in the horizon because, I mean, there is a joint which is need to get to an end. [00:27:10] All right. Yeah. Let me just, then let me just make some comments on what I wanted to give away. It’s also like, okay, so let’s [00:27:20] assume that you want to work with a service provider.

[00:27:21] Thomas: So what are the elements you may want to look for to decide, you know, what vendor you want to work with? And so I’ve, I’ve identified at least [00:27:30] four different domains, which are capability, capacity, collaboration, and cost. And so essentially capabilities about do they have the skills in house? Do they have the right people with [00:27:40] the right expertise?

[00:27:41] Thomas: Do they have done this type of project before? Now capacity is about do they have the people that they even ask? Are they actually available? Do they have enough [00:27:50] bandwidth? Do you have any infrastructure in this vendor organization to, to run all this sophisticated analysis and to run, to interact with all these people that are [00:28:00] needed?

[00:28:00] Thomas: Now for collaboration, what’s really important is that you know, the expectations are clear and that vendor in this case would understand not only the project needs, but would actually, [00:28:10] you know, if they identify the, the, the, like uncertainties or lack of clarity of, or limitations that they, you know, proactively try to resolve them.

[00:28:19] Thomas: Instead of [00:28:20] waiting, you know, until the end to, you know, to ad hoc resolve them in a, in a, in a fashion that may not be appropriate for the timelines that you have to deal with. [00:28:30] And so here again, this flexibility and adaptability come into play. And of course, yeah, cost, I will not dive into that, but yeah, cost is always, always an issue to consider when, when dealing with vendors.

[00:28:39] Thomas: [00:28:40] And I would like to conclude that. So when dealing with an NMA. This, this projects are increasingly well common they’re not straightforward to implement and there are a lot of [00:28:50] steps that could benefit from a certain degree of automation to, you know, to avoid mistakes, but also to speed up the process.

[00:28:56] Thomas: And because of this rising demand for, for expertise in [00:29:00] NMA, there is a benefit of, of working with vendors, especially, I think at least if, if there is a possibility to develop capacity building with flexibility where, you know, there’s an [00:29:10] exchange of, of knowledge and information that’s, you know, that’s accelerates, right, the process of conducting the NMA, but also that facilitates The conduct of an NMA where you leverage the in house [00:29:20] resources that you have as much as possible and the vendor would only complement, you know, whatever resources that you do not have in house.

[00:29:28] Thomas: And this, this kind of [00:29:30] interaction well, may vary from, from company to company, right? Because different companies have different needs, different resources as well as different timelines to deal with, with [00:29:40] respect to this external agencies to deal with. So I will leave it at that for a year, but happy to discuss any questions.

[00:29:47] Alexander: Thanks so much.