From Automation to Insight: A Pragmatic Approach to AI in Clinical Development

Daniel Levine (00:00):

The Life Sciences DNA podcast is sponsored by AgilisiumLabs, a collaborative space where Agilisium works with its clients toco-develop and incubate POCs, products, and solutions. To learn how AgilisiumLabs can use the power of its generative AI for life sciences analytics, visitthem at labs.agilisium.com. Sri, we've got Dennis Salotti on the show today.Who is Dennis?

Nagaraja Srivatsan (00:30):

Dennis is the executive director and head of clinicaloutsourcing and innovation at Jazz Pharmaceuticals. He spent over two decadesdeep in the operational side of drug development. His background spans clinicalresearch, eClinical technology, CRO, and vendor management. He's becoming oneof the go-to voices on how sponsors should think about selecting, qualifying,and partnering with vendors to run complex global trials. What makes Dennisspecial? He doesn't look at innovation as a shiny tool. He looks at how processtechnology and partnerships come together to improve trial quality, speed, andultimately the experience of patients and science.

Daniel Levine (01:09):

For listeners not familiar with Jazz, what is JazzPharmaceuticals?

Nagaraja Srivatsan (01:13):

Jazz is a biopharmaceutical company that focuses onserious often underserved conditions with a particular emphasis on neuroscienceand oncology. They're known for bringing forward medicines in areas like sleepdisorders, difficult to treat epilepsies, and a range of hematology and solidtumor cancers, and for continuing to invest in new therapies where there's alot of unmet medical need. From an R&D perspective, Jazz runs complexglobal trials, increasing learning from data, digital tools, and AI to betterrun their studies, collect smarter endpoints, and make participation morefeasible and easier for both patients and investigative sites.

Daniel Levine (01:57):

And what are you hoping to learn from Dennis today?

Nagaraja Srivatsan (02:00):

As I mentioned, Dennis is a very practical and pragmaticleader. What he brings is a very thoughtful approach to how AI can beimplemented and deployed across various different use cases. And Dennis, havingthe experience both in the clinical technology side and the vendor managementside and the procurement side, brings a unique perspective not just around howtechnology can be used for technology's sake, but how do you implement it andmake value and ROI for the organization?

Daniel Levine (02:28):

Well, before we begin, I want to remind our audience thatthey can stay up on the latest episodes of Life Sciences DNA by hitting thesubscribe button. If you enjoyed the content, be sure to hit the like buttonand let us know your thoughts on the comments section. And don't forget tolisten to us on the go by downloading an audio-only version of the show fromyour preferred podcast platform. With that, let's welcome Dennis to the show.

Nagaraja Srivatsan (02:56):

Hi, Dennis. Welcome to Life Sciences DNA podcast. Reallyexcited to have you here. It'd be great to get your perspectives on yourjourney and what got you to Jazz. So I'd love to hear a little bit about yourbackground.

Dennis Salotti (03:11):

Thanks, Sri, and thanks for having me. I really appreciatethe opportunity to speak with you today. So what brought me to Jazz wasessentially a desire to get closer to making medicines. I spent a number ofyears in consultancies and in other different roles running technologycompanies and in other capacities on the service side of the business. And Ifound that I just wanted to get back to what brought me into the industryinitially, and that was really being close to the making of medicines and playingmy role and trying to bring those forward to patients that need them. Beenabout a six-year journey since then, and it's been wonderful.

Nagaraja Srivatsan (03:47):

Wonderful. Dennis, as you came to your calling in terms ofmedicine, you must have seen the impact of technology and digitization over thelast six years evolve. Share with me your journey on what you're seeing and howthat landscape has evolved and how that's evolved with the role and job youplay.

Dennis Salotti (04:06):

So it's interesting how much has changed across the lastsix years. Now, I came from prior positions working in the eCOA, ePRO space andlooking at watching technology evolve there, just seeing consumer gradetechnology move into the research space and just how the user experience forpatients and their caregivers had changed with the different modalities that wesaw evolve there. Coming into Jazz over the last six years, we're observing howwe've utilized some of the evolution of technology, particularly AI and otherpredictive analytic type technologies. Just it's been really exciting to seehow we can take things that were traditionally done in a very labor intensive,manual way to develop insights and how technology has really shrunk the amountof time from research to insight and really putting that insight into action inthe trials. That's been the most exciting part of the journey I've observed.

Nagaraja Srivatsan (05:05):

And that's fantastic because that's kind of the use casewe want to kind of explore. So why didn't you pick an area where there wasresearch done human-wise and what kind of insights were discerned before andthen how you've now evolved with AI from that insights to action? Can you pickone or two areas which we could talk through?

Dennis Salotti (05:28):

I'll pick a couple areas and I'm going to warn you, Sri.These are potentially boring areas, but I think that's where you find thegreatest opportunities to apply technology. I'll pick one that's near and dearto my heart being in a sourcing role at Jazz and overseeing a site contractingfunction as well. When we go out, we engage with investigators. One of thethings we seek to secure a contract is a critical path step to bringing a trialand trial access to patients. One of the steps that has to happen there is youhave to negotiate a contract, you have to negotiate a budget. There is anecessary business relationship with those investigative sites. And the idea isyou want to shrink that space, that time you've spent doing thoseadministrative items to really get into the research. And it's not an excitingarea. Even though it's a function I lead, I'm not going to say that it's alwaysexciting or that it's always something that people look forward to doing.

(06:24):

But in seeking really, how do we expedite that? How do weshrink that space so that we can get the innovations into the clinics and getthe trials running and the access to patients? One of the things I've observedtechnology being able to do is take that manual process of reviewing pastcontracts, reviewing past history, reviewing sort of what the budget structuresand the last negotiated budgets look like, and being able to ingest that as acorpus of knowledge and analyze that and summarize that very quickly forsomeone so that the starting point when we go to engage with an investigatorand say, "Hey, we'd like you to participate in this trial." We'recoming to them with something that's much more aligned to our past interactionsif we've had a past history with them. We're coming to them with something thatgets both parties a lot closer to saying yes, and being able to do theimportant work.

(07:14):

And I think from a customer experience standpoint, thatbrings a much more positive experience for the investigator and their staff,which right now, post- COVID, we know the constraints that are on sites, weknow the burden that's on sites. If we can alleviate a little bit of that andpresent a little bit more of a delightful experience and an otherwise probablynot exciting area, that delivers something different. I think thatdifferentiates us as a sponsor, but also for the industry as a whole, it achievesthe goal of really just lightening the burden on the sites and bringing trialaccess to more patients. And so that's been a really exciting development in arelatively unexciting space.

Nagaraja Srivatsan (07:52):

No, it is an exciting space because what you have done istake structure information from unstructured documents, which are pastcontracts, and then creating a structured infrastructure for you to then make adetermination on what could be the contract values. Let's deep down a littlebit more. So was there any particular tool you used? Was it an LLM, anyparticular aspects which you used in that journey to take unstructuredinformation, sorry, structured information from unstructured data and then collatethat together?

Dennis Salotti (08:25):

It's the coalition of multiple projects. And I'll behonest, they're at different stages of their life cycle right now. So we areworking with partners in those different areas with different solutions. Theyare LLMs, that's the underlying technology behind it. But one of the thingswe're looking at is agentic interfaces for those LLMs so that you could have avery conversational, you could ask questions in the way that an end user, acontract negotiator may want to ask the questions, thinking about how do theyprepare to engage with the clinical site. So that's how we approach it. I willsay it took ... In order to do that, one of the learnings I had was how muchyou have to think about your content and think about your underlying processand think about how you want to approach that problem so that you canappropriately engage with your partners to train the model, further train themodels or tune the models, further shape the way the agentic experience occurs.

Nagaraja Srivatsan (09:25):

Is this the user community's internal teams who are usingit, or are you rolling it out to other external stakeholders within theorganization?

Dennis Salotti (09:33):

So no, right now we work cross-functionally. So there's acouple of functions that engage in this activity, but it's predominantly aninternal user base. So we're working in a closed system within our content andwith our users. So it's a fairly, what I'll say, controlled experiment thatway. And we were careful in that the scope of it and the use case was verynarrow so that we could demonstrate success and then expand from there. Thatwas a deliberate choice in the beginning of this. So if you think about howthis process works, we segmented out sort of the contract negotiation where toyour point, there's a lot of unstructured data in a document, in a PDF. Andthen we separated that out from a separate AI initiative around the budgetanalysis and the budget information where it's a little bit more structured.It's in a grid, it's in a different format.

(10:27):

And to your question around the different user groups,there's different stakeholders that engage in both of those activities. And soby segmenting it out that way, we were able to engage deeply with the end usersin different use cases and in some cases, very different user groups.

Nagaraja Srivatsan (10:47):

And clearly the ROI, as you said, is the site experienceand the speed in which you're doing site contracts. Are you doing a before andafter? What was the before contracting time and experience? And then what is itafter? And are these quantitative or qualitative metrics you're collecting?

Dennis Salotti (11:03):

So we do have, we have quantitative, we do somebenchmarking with some external groups against industry, and then we have ourown internal benchmarks. So we do measure cycle time. We measure a coupledifferent KPIs around this. So it's your typical cycle time, how many days doesit take to negotiate a contract? That's the most intuitive one. But we alsolook at how many, what we call turns of the document or rounds of negotiations.How many times does a document or a contract or budget go to a clinical siteand leave our hands and then come back to us? We found that when we analyze ourdata, that is a correlative, sometimes predictive measure of how long anegotiation's going to take. The more turns of the document correlates with thelength of time you're going to ultimately negotiate. So the idea is if you canreduce the number of times you have to exchange, you can reduce the overallcycle time.

(11:51):

And then we measure the cycle time obviously as theoutcome. I can't speak to the after yet because we're in the middle of it. Ithink it's a little bit too premature to describe what the cycle time reductionis, but I can share what our goals are around it. So if we look at ourselvesagainst industry, we are in the middle of the pack in terms of how well we aredoing.That was our base case. We were approximately median sort of level ofperformance. What we really aspire to be is top quartile. We want to be in thetop quartile of performers around this in terms of cycle time. And so that'sthe goal for this pilot is really, can we, through this and other strategies,achieve that broader outcome. I don't know that just by virtue of what we'redoing with AI, it'll get us all the way there, but we think it's a materialdriver and all the underlying factors in terms of having a good starting point,having really good intelligence and understanding your history with thatparticular site or in that particular country.

(12:55):

All the building blocks that'll get us there, we think theAI has various use cases to enable that. I could speak to a few of the otherones that are surrounding that.

Nagaraja Srivatsan (13:04):

Just prompting up two more questions on this use case. Oneis they say AI is only as good as the data. How good is your datainfrastructure of past contracts and structure to get that recognizance to makesure that you're doing that site engagement better?

Dennis Salotti (13:21):

Yeah. So this is one of the most surprising things. Frommy perspective going in, our data was not good. This was an activity that wehad as a company had delegated, transferred that responsibility to externalpartners. So we didn't have the ownership over the data, over the activity tohave good process analytics, to have very good analytics. We only had what weneeded for oversight of that transfer responsibility. And I was concerned withthat. I'll be honest with you, I felt like that was going to be a realdetriment to us. And what I learned through this was that the capabilities hadadvanced so far in terms of AI that just by virtue of having a lot of thishistory of contracts, sometimes dead scans, sometimes just flat scans, paper,literally.

(14:06):

The models we were working with, you were able to ingestthat and still analyze that and use that. So what I thought was going to be amaterial weakness actually wasn't as material of a weakness in terms of ourdata. Now, would I have liked to have more data base cases around how wenegotiate and better cycle time? Sure. I think everybody would rather havethat. But I was pleasantly surprised at how far the technology had come thatyou could ingest all of that information and still get an analysis out of it,even though it's in PDFs and sometimes dead scan PDFs.

Nagaraja Srivatsan (14:39):

No, that's actually an amazing thing that you actuallythought the process was broken, but with the power of LLMs, it was gettingbetter. The last question is the change management effect, because people arenot used to using AI or other things to collate this thing out. I like the wayyou segmented so that the users are using what they're used to. The contractorsread the contracts, the budget guys read the deterministic piece. But outsideof that, what kind of challenges did you face? Because this was not an easyadoption. People are going to trust somebody else's output to make theirdecision. So how did you work through that?

Dennis Salotti (15:14):

I think the first thing is we're still living and learningas we get further into it. And it's important, I think, to listen to people andunderstand and have that safe space where you can have that dialogue. Thebiggest part of the change management, frankly, was lack of experience with AItools. Within our group, I think in this space, there was not proficiency insome of the things. I mean, I myself, I'm not an expert yet at promptengineering. It's one of the development areas I know I have for myself that'sjust getting better at that sort of thing. So the chain of management, I think,really comes around just basic core capabilities around AI fluency and thefoundational elements of if you're going to work with an agent, how do youprompt it? How do you ask it questions in a way that gets you as fast aspossible and as efficiently and as high quality responses as you want?

(16:09):

Just those sorts of core capabilities, I think aroundprompt engineering.

(16:16):

I think the other thing that's probably a little bit of acultural, but I think a very relevant consideration for organizations is whatdoes this mean? People want to know what things mean for themselves. So there'sa concern perhaps that ultimately, eventually that the bots or the agents arejust doing the negotiation and you don't need a negotiator, right? And we knowthat that's not the case, right? We know that the human elements, this AIenhances the human elements of some of these business processes as opposed totaking them or replacing them. So that I think there is some communication thathas to happen there and reassure people because in the popular media, you hearother things. You hear about layoffs, you hear about software engineers andsome of the impacts of Amazon or some of the other software as a servicecompanies, you hear some of the impacts they've had.

(17:05):

And I think that's a very relevant and human concern. Soyou just have to address that head on in your communication.

Nagaraja Srivatsan (17:12):

This is a fantastic use case. I don't think it's boding atall, Dennis. So it is a fascinating place of picking a process and thenautomating it out and then having some very clear autoize and changemanagement. So maybe we use the same motif for a couple of other use caseswhich you wanted to discuss.

Dennis Salotti (17:30):

So I'll speak to some of the other more basic ones thatwe're using right now. They're a little bit less splashy just within my group.I mean, we often have to do a lot of research and a lot of exploratory analysisinto what's changing in the environment, what's changing in the ecosystembecause my group focusing on external partners, focusing on our clinical sites,we have a very outward looking view and not everybody has that. And because Iwork in a sourcing group and we manage the contracts and the budgets, both forsites and our external service provider partners, a lot of times there's areality check sometimes on how the cost of running a clinical trial haschanged. So it becomes really important to keep abreast of what's happening inthe ecosystem, both from a macroeconomic and a microeconomic sort of industryperspective. And that in and of itself could be an entire job.

(18:26):

However, utilizing very basic tools, things like Copilot,we're able to ask the questions that we need to ask and get reputableinformation with sited sources to then go in and be able to develop that andshare that information back out with the organization. So to be able to tiewhat's happening in the external market to what some of our internal strategiesare in terms of how we position the partners we work with, sometimes the choiceof countries and sites we may go to, all kind of predicated upon some of theexternal intelligence. And we wouldn't be able to probably synthesize andsummarize that information as effectively if it weren't for some of thetechnologies we were able to use today.

Nagaraja Srivatsan (19:12):

No, then that's a fascinating use case. Basically, you'redoing almost equal into reg intelligence. This is much more marketintelligence. You're doing what's happening with sites, countries, staffs,vendors you work with and kind of collating information about them and thenscoring the sentiment on is good things happening, bad things happening,trending, and then giving a summary report to your organization. Is that a veryfair way to articulate what you're trying to do?

Dennis Salotti (19:40):

Yeah, it's very fair, Sri. I mean, we work, and this iswhere, I mean, ICH E6(R3) and some of the relevant regulatory guidance now isall about risk proportionality and doing good risk management. And in order todo that, to your point around having market intelligence, you have to assessthe external market. You have to understand what's happening, what's apotential threat or opportunity. But we live in a very dynamic world to say theleast right now. I mean, you called out a few of the things, right? So supplychain disruptions, geopolitical tension, even natural disasters and climatechange right now is a preeminent risk. And it's hard when you think about allthe different risk and opportunity domains to keep abreast of all of those. Andthen to further do that, to really synthesize that information and think about,okay, when I look internally, is my organization positioning appropriately?

(20:33):

Is my sourcing strategy appropriate? Do I have any threatsI need to mitigate? Those sorts of things, technology has been a wonderfulenabler for the teams to just be able to summarize some of that information andto go the step further and to now ask the AI a little bit about what they thinkthe optimal strategy is, giving it some prompts around certain conditions orcertain constraints. It becomes a thinking partner in a way, not to replace thehumans because it doesn't always have the answer and there's some tacit subjectmatter expertise that you really have to apply in these situations, especiallyfor our industry. But it becomes almost like a very rapid thinking partner thathelps bring insight into the organization and then process some of that insightinto something you can use strategically. That to me has been as a leader inthe organization, has been a game changer, the fact that my teams are enabledwith that capability now and they can leverage that and I can leverage thatpersonally.

Nagaraja Srivatsan (21:33):

That is what's fascinating us, isn't that what the futureof work should be where before you would spend orders of time going throughwebsites and stuff and distilling it. Right now, as you said, you're having athinking partner, you have to give your thoughts around what should bevaluable. Hey, climate change is valuable, but don't make it high risk, butsupply chain disruption based on water is much more higher. I'm just making itup, but you're giving inputs to this thought partner and you're evolving whatshould be that site engagement practice and what should be your shipment andsupply chain strategy and stuff like that where you're now elevating yourselfto asking those much more macro questions versus you would do 80% of thegroundwork and then you would have only 20% of the time for the macro. So you'dsimplify it and say, "I can't do all of this.

(22:23):

It's too complicated. Let me go to the simplestrategy." Right now, it's almost challenging you and your team to do morecomplex thinking. A, is your team ready for such this thing? Because when yougo from doing to thinking, thinking takes a lot of time and effort andexperience, and then that's part one. Part two is that synthesize of this data,sometimes we call it AI Slop. You start to rely on your thinking partner toomuch that you're not thinking and you're just being like, "Oh, AI must beright and go about doing the job in its way." So two parts to it, becausethis is really where I think the world is going, where you better be having athinking partner and you're going to be thinking a lot about what to do, but ithas two connects to it. One is you overtrust it or you under partner with it,right?

Dennis Salotti (23:16):

I want to talk about that second part first, Sri, becauseI think you hit upon a nerve there. I think that's a key thing, especiallyabout how we think about how we shape our organizations and how we shape thenext level of leaders. It's a real risk, what you called out around the AI slopand around the reliance on it without applying critical thinking. I think thatif I had to identify what's the most existential risk with AI, I'd probably sayit's that. I think there's a necessary discipline we have to instill in ourpeople and in ourselves to not take that shortcut, to not just rely on it andnot take it ... In this world where everybody wants things faster and with sucha productivity crush, it is a very real risk that we fall into that. And Ithink we have to resist that.

(24:05):

One of the things that I've observed, and I've observedthis really recently in a presentation I'm developing myself now and I'm usingCopilot, my partner, and doing the research and thinking about things. And I'lltell you, I had that delightful experience of it made me think differently. Itopened up ideas where then I applied that critical thinking and some thoughts.And the interesting thing is it actually can take longer. So I think peoplethink about AI as something that's going to speed everything up. And in the usecase we just described around developing strategy and synthesizing insight andexternal information into a strategy, I think it actually will make you takelonger because it can give you the ability to think about more and differentthings that you may not have considered. So it's almost that you have to kindof hold yourself accountable a bit to the discipline of saying, "Well, I'mnot just going to take what it says and think about thing.

(25:01):

I'm going to open up those citations. I'm going to look atsome of the primary research that's bringing in to me, some of that primaryinformation, and I'm going to be deliberate about slowing down and thinkingabout what this means for me as opposed to just copy pasting it right into yourPowerPoint deck or an email or whatever you're doing." And that'ssomething that I think when we bring up our next levels of leaders and theworkforce that sits within our organizations, that's a message that I think hasto get through as well is just not to think about it as an immediategratification tool, but think about how it can level you up in terms of yourability to think strategically and apply that human element that AI can ...

Nagaraja Srivatsan (25:44):

And actually, even though it physically takes more time,in the long term, it's going to save you time because you've talked throughdifferent scenarios. And so from a organizational proactivity is much better.But as you said, in the short term, it's not like, okay, I did this two minuteslater, I did that, cut and paste and put it together, and you need to applycritical thinking. Then let me explore this because this is an area which isreally fascinating for me is how do you build that skill? Because we all go tocollege, we do tests and exams, then we come to the workforce and then we loseall of that and we just do work. We grunt in email, then we finish our email,we have meetings, we finish our meetings, we get to ... And so that criticalthinking activity is actually less and less in our day-to-day jobs.

(26:34):

And so now that you've got to feel yourself up, how do youbring yourself up to doing more of that deep thinking? How do you build thatskillset and environment for that?

Dennis Salotti (26:44):

First of all, I think some of it is hard to build. Youalmost have to have intrinsically or you have to source for people in yourorganization that have sort of an intrinsic curiosity. I think that's somethingthat can be hard necessarily to teach, but if there's a little spark of itthere, you can nurture. So I think it starts with that curiosity and a desireto learn and to understand that the world is a dynamic changing place. And themore you become inquisitive about it, the more you'll learn and the more youlearn, the more you can apply that as additional value to the thinking and thestrategies that you put forward. What I tend to do is I try to ask morequestions. I try to ask, why? Why do we think that is? Why do you think thatis? To kind of nurture that curiosity so that people can develop that skill ofreally exploration and inquisitiveness and inquiry, critical inquiry, why do wethink that is?

(27:44):

To get to the root cause of things, to then bring that tothe table. It's hard because I think you're swimming upstream. I thinkeverybody wants the answer. I think there is a push to get things quickly andwithout a lot of effort sometimes. But I think slowing people down where it'sappropriate, well, why do you think that is? Let me understand your thinking.The challenge is that they have to develop that thought process.

Nagaraja Srivatsan (28:10):

And that is, you hit upon a very key part, which I want tojust highlight to everybody. They ask the question why in management, as theysay, the higher you go, the more wisely you have to ask. And me personally as aCEO, they say, have five whys before you give your opinion. So you ask thequestion exactly like, why is this happening? Okay, why did you make thisdecision? Why did you want to think that this is the right course of action?Why do you think this is going to benefit us? Why do you think [that] after thefive whys is when you're supposed to give your opinion? And as you said that,that just resonated with me that that's a critical underpinning of where weneed to build that critical thinking is not to go with the answer because theanswer is not going to be easy and cheap and you take that on, it's thatcritical thinking of asking those whys.

(29:00):

And maybe I'm going to use this like, you guys can't useany Copilot answer after without asking five whys. Can I ask why, why, why? Andyou get to the root of it, but you also learn from it before you can start tomake a decision, especially when it's a thought partner. I think there aredifferent use cases where it's deterministic like your previous use case, butthis one where it's a thought partner, this is where AI slop can come inbecause after some time you're going to say, "Man, everything which it saysis wildly wonderful. Why should I not trust it?" And that's the first whyyou should not trust it and you should keep asking the whys. What do you thinkof them? Does it make sense?

Dennis Salotti (29:40):

Yeah, I think it's wonderful. And I'm 100% with you on theidea that as a leader in organization, especially in your seat as a CEO, thatyou have to ask those questions because you want to understand your team'sthinking, you want to see the depth and the thinking. So you're not challengingthem because you're looking for the right answer. You're looking to understandthe thought process that sits behind the decision. And I think that's a pivotfor people to make. And it's funny because it's a balance. To your point, youwant to slow down and you want to be thoughtful on the thinking partner itemsand on the real strategic things. But then you want to use AI and some of thesetools like Copilot and others when there is things that'll just help you speedup. So I don't want to read it. I'm sure you end up in the situation as well.

(30:32):

Sometimes you get forwarded to 15 emails in a chainbecause they need to arbitrate something and they want a decision from you oryour input. You can use it to speed up by summarizing that. It's a verytactical use case, but it can summarize very eloquently what the key points ofthose 15 emails are. And now you're fully up to speed and you can have aninformed discussion with someone to help them further whatever they need tomove past. But in that case, it's a very fast ... It's a speed up sort of situation,not necessarily a slowdown. So I think there's also a level, just getting backto your original question, I think there's also some upskilling in theorganization and understanding where to apply which use case and where do youjust go very quick and very tactical and where do you slow down and be verythoughtful and be very cautious of the AI slop?

Nagaraja Srivatsan (31:20):

Why don't we pick one more kind of a critical use case?What's really exciting about this conversation is you're taking tangible areasand then deploying solutions to it. So is there any other area you would liketo explore and talk through?

Dennis Salotti (31:35):

I'll talk about something that's more future focused, notsomething that I could speak to the end game around and that we're even in themiddle of it now, but I'll tell you where I'm really interested in where AI andAI enabled tools are going to take us. And it's around risk. It's around justthinking about E6(R3) and thinking about the emphasis on risk proportionateapproaches and understanding where to apply rigor on what's critical to qualityand what you can take a lighter approach to or a less devolved approach becauseproportionally it's just not going to impact patient safety. It's not going tonecessarily have the same impact on data integrity. I'm really excited to seewhat comes out of the market in terms of identification of risk, predictive riskindicators, AI being able to synthesize across multiple data sources in morecomplex scenarios. And really, again, not making the deterministic decisionaround what it should be, but prompting as a thought partner to our riskmanagers, our central monitors, our CRAs, where, hey, you might want to look alittle bit deeper here, or there may be something here that you want to focuson.

(32:50):

Or maybe just thinking about systems like IRT or ECOAbills, just, "Hey, based on the complexity of this protocol, here'sprobably the areas where we really have to focus UAT or we really need to thinkabout how we design that system." Being that thought partner that caninterpret lots of different disparate pieces of information and synthesizethose and understand what that might mean where there could be a risk and wheremaybe that's been observed. And if you have the institutional assets, pastexperiences on trials and being able to prompt people to say, "Think aboutthis. Apply your subject matter expertise. Apply your knowledge. Look a littlebit further at this." That can really help us direct our efforts in theright way that's in alignment with, I think, what the regulators expect of usand what we should be doing as a business in terms of risk proportionateapproaches to quality.

Nagaraja Srivatsan (33:39):

That's a fascinating framework, right? Risk proportionateapproaches to quality. The FDA has said that there are a lot of regulators whosay that, but we still are doing many things pedantically. I mean, from atesting standpoint, I know that for sure. We just apply, okay, this has to betested and we do it even though ICH is just based: why testing thing which wehave already done before. As you said, how do you then look at complexity andthen saying which areas are at most risk, dosing and patient endpointcollection, and let's focus more on that versus check if the roles andresponsibility screen is working very well for the 30th time. But I think therewere three parts to what you said. The first is risk identification. So almostyou need a risk roster on what are risk parameters to play on. So that's just arisk catalog.

(34:34):

The second part is risk assessment, which is what are mydata sources to assess the risk? And then the third part, which you said, riskprediction, which is, okay, I've assessed this, I know my risk roster, theseare high risk, low risk, and this is my prediction of what's going on. And inall of that, you need to be a thought partner. It's not like it going away andassessing and doing that, but you coming in play, which is very fascinating,which is you're almost converting a role from a doer to a risk assessor. And Inever thought about that, but now every role will become something off of a"how do I assess the risk in doing this, believing the AI, of furtheringwhat I need to do," slowing down and critical thinking to make sure thatwe're getting to the right outcomes.

Dennis Salotti (35:23):

I would even add a fourth part to it, Sri. I would add,and this has been a pain point in the industry for a long time, and it's aboutknowledge management, and then it's about how do you treat that risk onceyou've assessed it, where you've made the decision, this is something we needto respond to. We all do lessons learned meetings, right? We finish a trial,the teams get together, they do lessons learned, it goes in a spreadsheet or adocument, it goes somewhere, right? Somebody makes a PowerPoint. And then it'salways the question of how do we make sure the teams that come after themactually benefit from those lessons learned? Well, if that becomes part of thecorpus of information you have that you can bounce off of your LLM or that youhave within your enterprise AI architecture, you just think about teams whenthey encounter one of these risks and they have to respond to them.

(36:12):

What if they were empowered with an agent that could thencrawl all of the knowledge from the past lessons learned around this specificdomain and be able to say, "Hey, these are some ideas. These are somethings that work. These are some things that didn't work. Here's maybe therelative matching to the problem you're encountering or the design of the trialmaybe you're working on." So maybe you may be encouraged to look a littlebit more closely at these solutions, but don't recreate the wheel. Don't recreatethe wheel. Think about leverage the knowledge, elaborate on that knowledge ofthe teams that came before you. I mean, it's kind of fundamental because itdifferentiates us from a lot of other animals in the animal kingdom. Humans areable to pass on knowledge from generation to generation and progressively buildit. Whereas other types of animals don't necessarily have that capability to dothat.

(37:02):

And so it is sort of a fundamental part of our evolutioneven to be able to do that. And that's maybe the evolution I think of AI inthis specific use case is even to take it to that fourth step. Not only is ithelping predict and helping focus our energies, but perhaps it's evensuggesting things that we've already built, solutions we've already developedthat may be either appropriate or be able to be elaborated on to be appropriatefor new situations.

Nagaraja Srivatsan (37:27):

And then spot on, Brett, everybody says the reason youlearn history is because that teaches you a lot of lessons. And what you'resaying is the same thing, organizational history and knowledge history. If youhad that, you can learn a lot.This has been just a fascinating conversation andwe can go on for some time. What would be your final message? As you lookforward in the next 12, 24 months, where do you see this thing going? I mean,you talked about the impact on the workforce, but as somebody who's been inthat transformational leadership, where do you see this going?

Dennis Salotti (38:00):

There's going to be a little bit of a reckoning. I thinkright now there's a lot of noise in the market. I think there's a lot ofexcitement, and I think that breeds a lot of maybe a little bit of oversellingwhat it could do. I think where I see it going is a rationalization. I thinkthere's going to be a reckoning that while there are some tremendous and veryexciting and interesting use cases for AI, I think what we're going to find asan industry is the greatest opportunities lie in some of the less excitingareas, the things that are repetitive, the things that are higher volume thatwe can automate. And probably the areas where traditionally as human beings,risk being one of them, we're just not wired, we're not built to be very goodat, but AI [can] enhance our capabilities around. So I think in the future,there's going to be a little bit of a tailoring to, as we become more fluent inthis, understanding where the highest value use cases are and maybe some of thethings that maybe are a little bit further off or a little bit next generation.

(39:01):

But in the next 12 to 24 months, I think we're reallygoing to be focusing on the value and the pragmatic value of where we can applysome of this technology. That's at least where I think the leadingorganizations are going to go, not really for the moonshots, but stacking upthose small W wins with AI to have big value in the end.

Nagaraja Srivatsan (39:20):

You're talking about the Moneyball concept. This is notgoing to be on giant home runs. It's going to come on singles and you're goingto make sure that you're going to have enough singles. And I think the firstnext 12 months, just removing mundane work and getting people to that criticalthinking is going to be a real good journey. But I think we're going to hit alittle bit of a plateau because some people will jump the S-curve to thecritical thinking, some people may not. And so to your point, how do you bringpeople who have to get up this S curve with that critical thinking knowledgewhere it's very appropriate? You talked about everybody wanting to learn aboutprompts, but I think there's a whole thing around how do you go from being adoer to a thinker? And I think that's the journey which everybody has to go.

(40:08):

But thank you so much. Really appreciate this. This hasbeen a fantastic conversation. I wish we had more time and we could continuethis for many, many more minutes or hours. Thank you.

Dennis Salotti (40:20):

Thank you, Sri. It's been a pleasure.

Daniel Levine (40:25):

Sri, you talked about Dennis being pragmatic. What did youthink of what he had to say?

(40:31):

I think it was a really fascinating discussion. And wetalked about two use cases. One was really removing grunt work, which wasreally looking at documents and contracts and really extracting data andinformation to be prepared for site and site engagement. And there, Dennisbrought a very thoughtful approach on how do you break a big problem intosmaller chunks and then really deploy AI to solve those problems well. But thesecond one was a very interesting use case where he really used AI as a criticalthinking partner. And we really explored how AI as a thinking partner not onlychallenges our own abilities, but also our ability to interact with AI.

(41:13):

In that first use case, there were two things that reallystruck me. I think there's a tendency to think that you simply plug in AI, flipthe switch, and it solves your problems. But you talked about the need toreally shape the agentic process and think deeply about how you're employing itand what you needed to do. What did you think of that approach?

Nagaraja Srivatsan (41:37):

Where Dennis as a pragmatist comes in is to break the bigproblems into smaller chunks, and then really to figure out what would berisk-mitigated efforts to make sure that you're trying to solve for that. Andso he broke a big problem into how do I first assess knowledge from pastcontracts, to second part of how do I then distill that knowledge into anarrative to where I can engage with the sites, to third, to really engagingand then really looking at cycle time improvement in contracting, but also in rework.So I think it was a very pragmatic way of thinking about this big problem andtaking it into small chunks and making sure that you're bringing in the rightscaffolding to ensure the success of those approaches.

Daniel Levine (42:22):

The other part of that segmentation that I thought was sointeresting was that the segmentation was done in a way that a lot ofunstructured data went into one part of the process, whereas the structureddata was in a separate part of the process. Does that help in other ways?

Nagaraja Srivatsan (42:38):

Yeah, I think it was actually the merging of these twotogether. So what he did was he knew that contracts and unstructured data andhe wanted to take structure components out of that contract, but that he waswanting to feed it was in study budgets which are much more deterministic innature. And therefore the output of one of those was an input into the othersto really making sure that you really looked at study budgets of the currentfor the past, what you're trying to give for the sites so that you can makesure that you're starting to drive it. So it was a very good way where I thinkthe future is going as you're bringing in more and more data from unstructured,which then can be structured through either summarization, sentiments andothers, and mix that with structured data, which are much more deterministic innature.

(43:25):

And bringing these two together really help you bettermake decisions and get to good results from an ROI standpoint.

Daniel Levine (43:35):

Dennis has a very outward facing role. He works withexternal partners and clinical trial sites. He talked about using AI as sort ofa market intelligence tool. Is it unusual to see a company do this? Is itsomething companies are doing broadly?

Nagaraja Srivatsan (43:53):

This is an area which actually companies are doingbroadly, whether it's regulatory intelligence, market intelligence, looking atwhat's going on in the marketplace. With AI right now, we're having access toinformation, but more importantly, distilling that access to information intoinsights and value either through summarizations or through critical inputs andfeedback. So I think this is a use case which we are going to see more andmore. And I call this use case knowledge democratization. And knowledge couldbe internal knowledge, which is stuck in documents or external knowledge, whichyou're discerning from different bodies of work, but bringing both those thingsand democratizing that to make it more and more accessible to people is goingto be a scale use case for AI.

Daniel Levine (44:38):

You also talked about a coming reckoning with AI that thegreatest opportunities are in the more mundane, repetitive, and high volumeareas. You agree?

Nagaraja Srivatsan (44:48):

I think so. I think the next 12 months, we could put a lotof effort in removing grunt work and training ourselves in doing more criticalwork. And that would lead us to be much more thought partners as AI starts todo more critical work. So I think it is a really good journey and a right wayto progress forward.

Daniel Levine (45:08):

Well, it was great to hear Dennis's insights today. Sri,thanks as always.

Nagaraja Srivatsan (45:13):

Thank you.

Daniel Levine (45:17):

Thanks again to our sponsor, Agilisium Labs. Life SciencesDNA is a bimonthly podcast produced by the Levine Media Group with productionsupport from FullView Media. Be sure to follow us on your preferred podcastplatform. Music for this podcast is provided courtesy of the Jonah LevineCollective. We'd love to hear from you. Pop us a note atdanny@levinemediagroup.com. For life sciences DNA, I'm Daniel Levine. Thanksfor joining us.

‍