Full Webinar Transcript
John Lyons:
We have an exciting topic today, we've gotten a great response, Improving the Patient Journey: Leveraging Novel Social Media Data to gain Insights into Patient, Caregiver, and Healthcare Provider Reported Outcomes. We have two speakers today, Dr. Nardin Farid, strategy lead, real world patient analytics at Real Life Sciences and Stephen Doogan, founder of Real Life Science. Stephen will be handling our Q&A session today. So with that, let me hand it over to Dr. Farid.
Dr. Nardin Farid:
Great. Thank you, John. So here at Real Life Sciences, we use NLP. That's the core of what powers both of our modules, and essentially we're using NLP to revolutionize outcomes research while protecting patient privacy.
So we have two modules. One of them is RLS Protect and the other is RLS Reveal. RLS Protect is a data anonymization, redaction and risk assessment platform that enables you to submit new drug applications by using risk assessments that are quantitative for Health Canada and to the EMA. Today's conversation will focus more around RLS Reveal. And with RLS Reveal, we process real world data and apply structure by using our NLP in order to have really rich data sets with the overall goal to understand and model the patient journey. And with that, I'll go over some of the presentation overview. So we're going to start off by going through some case studies in Alzheimer's disease, and then I'm going to focus a lot of time on the Alzheimer's disease and COVID case study and some insights we were able to gain there. And then I'll explain how we do what we do by giving a brief overview of how social media epidemiology using RLS Reveal works.
So to start off with our case studies, we really see the patient journey as a place where you can learn more about what patients are going through in the real world. And we start all the way in the beginning by looking at symptom recognition and when patients are seeking medical attention. So really in that first, I just had a symptom and I think I may have a disease phase. From there, patients go on to obtain a diagnosis and pick the treatments that they want to be on and it's optimization. And with picking treatments, of course, comes coverage, cost and access concerns and all the way to disease maintenance. So we're really trying to gain some insights across the entire patient journey in order to learn more about what patients are going through in the real world with these disease states.
We feel that social media is really good place to gain insights on the patient journey because people post on social media across all of those different phases of the patient journey. But some teams really struggle to hear the patient's voice in the patient journey. And some data is really noisy and accurate and therefore not useful. And without NLP, and without being able to structure your data, you may not have a scalable, repeatable methodology for structuring social data in order to gain some really insightful outputs from it. And then you may have a small sample size. So without social media you're really losing a lot of different patient voices. And so you may have a small subset that doesn't lack direct patient perspectives. With social media you don't have as picky of an inclusion criteria. So you really get to hear more from a wide variety of patients within the disease state that you're looking at.
And with that, I want to put an audience pull out. So the question is, what experience do you have using social media data to gain insights on patient perspectives? So do you have, A, is no experience, B is limited. So you do social media listening, using keyword searching, or see advanced experiences, spanning multiple diseases, treatments, or patient populations.
John Lyons:
So we have the results are rolling in, and it looks like there are responses for each. There are some folks that have no experience, some that have limited, some that have advanced, but the most responses are in no experience with the runner up being limited experience.
Dr. Nardin Farid:
Interesting. Okay. That's what I expected. So we'll go through what we've done with social media in order to learn more about the patient journey. And I'll also explain how we do it at the end. So it could give you some more insights, but we're able to look at a wide variety of disease states because people use social media for a lot of different things, and it's not really limited to any type of disease state. So we worked in a lot of different areas and different therapeutic areas where we're able to pull some insights from, but today I'm going to focus on Alzheimer's disease in the beginning of our conversation around some case studies, and then I'll go into more detail around a case study with Alzheimer's disease and COVID's impact on that population.
We work with a lot of sponsors that really want to publish their findings and the outputs of the insights that we found. And so we help support some medical writing efforts, but today's case studies are going to be around Alzheimer's like said, and from a few different posters and submissions that we've had. So the first one we had a sponsor wanted to learn more about the different stages of Alzheimer's disease and what people are reporting online and where they are against the different stages.
So we looked from mild cognitive impairment, Alzheimer's disease, all the way to moderate, severe. And we looked at patients and caregivers in this study, and we wanted to see what are the symptoms and impairments reported by these patients and caregivers across the different stages. And as expected with Alzheimer disease as it progresses, cognitive abilities decline. And so you would expect that people in moderate and severe aren't really using social media to discuss what's going on in their life, just because they're no longer able to, but it's interesting because it points out how important it is to have different stakeholders in your analysis. So, especially in a disease like Alzheimer's disease, where patients are no longer able to report themselves, sometimes caregivers here become the decision maker. And so it's really important to incorporate their voice in the patient journey.
Here we looked at the symptoms and impairments that are classified within social, physical, emotional, cognitive, and functional categories. And in a few minutes, I'll describe a little bit more on why we classify them in those five high level groupings, but that's what we found with this case study. And then we also wanted to look at genetic testing. So, in another case study, we wanted to see what are patients, caregivers, and healthcare providers talking about genetic testing on social media. And we pulled some high level topics and also some low level ones. But today we'll just look at the high level ones. And we looked at around 2,700 posts posted by over a thousand unique reporters. And we saw that stakeholders were really using social media to go and gather more information, to have some support, to discuss their experiences with genetic testing amongst each other.
And some of the key topics were unmet needs. So people looking for education and support, financial concerns, people asking each other how much is getting genetic testing going to cost? And also logistical challenges. So do I need a prescription to get genetic testing? Where can I get genetic testing? How long does it take? What would I need to go through logistically in order to get genetic testing? And then also the usefulness of it. So these findings really show that between the three different stakeholder types, people really perceive potential benefits of getting APOE4 testing, for the management of Alzheimer's disease, but we're clearly searching for more education and support around the topic. And for Alzheimer's disease, genetic testing is not part of the diagnosis, but can help aid in the diagnosis. It's been research that Alzheimer's disease, a good part of it is genetic. And so you can get APOE4 testing in order to find out if you may or may not have Alzheimer's disease later on in life.
And interestingly enough, a lot of the conversations were around asking each other, why would I get it? What happens if I get it? And there was a lot of conversations around the impact emotionally of getting genetic testing. And a lot of stakeholders really suggested to each other that if they got genetic testing done that they also should get genetic counseling and not even just for themselves, but also for their loved ones, because in thinking that you may have Alzheimer's disease later in life can really be emotionally impacting. And so having some counseling there was found useful by a lot of different stakeholders. So I mentioned a little bit earlier Spec-F, and I showed when we were talking about the different stages of Alzheimer's disease, the categories that we classify symptoms and impairments. And so I wanted to just give a little bit more background on that before going into the next case study.
So SPEC-F is our framework for how we classify impairments and symptoms that patients, caregivers and healthcare providers are reporting online. And this is our high level view of the different symptoms that people may have within a disease state. So SPEC-F is an acronym for social, physical, emotional, cognitive, and functional impairments. And we feel that sometimes diseases like Alzheimer's disease are really heavily given some cognitive attention, because those are the hallmark signs and symptoms within that disease, but a disease impacts an individual as a whole and impacts their entire life. And so it's important to gain insights on the impairments that people are facing socially and functionally that may not classically be reported, but people will report them on social media.
So the way that we are able to use the SPEC-F framework, is we look at verbatim narratives reported on social media. So we're able to classify them into the different stakeholders, like I mentioned earlier, and from the verbatim narratives, we're able to extract some sub concepts reported within them. So if we look at the emotional category here, the post is by a caregiver, and part of the post says, then severe anxiety at which point he could no longer focus enough to work. From here, the sub concept of anxiety was extracted and we use different dictionaries like MedDRA, or WHO-ICF in order to have a hierarchical approach at the different sub concepts that can be pulled and the different groups that they fall in. And then we classify them into SPEC-F concepts. So here anxiety is put under emotional versus driving impairment all the way on the right of the figure is under functional.
So another case study that we did within Alzheimer's disease using our SPEC-F framework was we wanted to see the difference between what stakeholders are reporting on social media, compared to what healthcare providers are asking patients or caregivers about within their clinical outcome assessments. So clinical outcome assessments are used sometimes to aid in diagnosis, especially in Alzheimer's disease, where a diagnosis isn't as structured as other diseases, but it also can be used in order to track progress within a disease state and see if a patient is progressing, and also to just gain some more insights in a structured way. So what we did was we pulled the concepts from social media that people were reporting, like I showed on that previous slide. And then we did the same methodology for the clinical outcome assessment. So we looked at each of the questions and then we were able to extract the concepts, asked within them. And we looked at 50 Alzheimer's related clinical outcome assessments in this case study.
And really we wanted to see the difference, what are people reporting on social media that healthcare providers aren't asking them about in their clinical outcome assessments? And the other way around, what are we asking about in clinical outcome assessments that people really aren't talking about on social media? And we found that there was a lot of things that people are talking about on social media that aren't really incorporated in clinical outcome assessments. And so we looked at our SPEC-F high level categories, so social, physical, emotional, cognitive, and functional challenges. And we saw that within Alzheimer's disease, cognitive is well covered because it's a cognitive disease. And so people were really using that in their clinical outcome assessments, but social, physical, emotional, and functional challenges were only partially represented, and actually within the 50 clinical outcome assessments, there wasn't one that incorporated all five SPEC-F categories.
So we looked at 50 clinical outcome assessments because we wanted to make sure we had a wide variety of clinical outcome assessments in order to have some depth to the case study, but also on the social media side of things, we want to take the same approach. So when you hear social media, you may be thinking the classic social media sites that you may use on a daily basis. So Twitter or Reddit, those are the classic social media sites that you and I may use, but within diseases there's disease focused forums. And so for example, an Alzheimer's forum, there's also general health forums like health boards or health, social networks, like DailyStrength. And then also treatment review websites like Ask a Patient. We really want to get a wide variety of sources on social media in order to be able to tap into a lot of different communities that are built on social media, and therefore get the most wide breadth of knowledge from social media.
So with that, I'm going to go into more depth on Alzheimer's disease and the impact of COVID on this population. So by understanding the impact of COVID, you can understand more about how this pandemic impacted this population. And by using social media, we aren't really troubled with having to do any face to face interactions, and we're still able to have social distancing and able to study what's happening in the disease state during a pandemic, without impacting anybody's health. So we wanted to see what are the impacts of COVID on this population. And our timeframe was between January 2020 and January 2021. So if we just look at the figure on the right, we see that we started off with around a 143,000 narratives, and these are all Alzheimer's specific. From there we further filtered it down to look at only patients and caregivers that are reporting a diagnosis within any stage of Alzheimer's disease. And that further narrowed down our group by 43,000.
And then finally, we looked at COVID within this data set. So now there's around 3,422 narratives that were related to COVID and were Alzheimer's disease patients or caregivers. So we use Reveals NLP and the platform to analyze the challenges and issues that are happening on 47 social media of sites. And we looked at, like I said, patients and caregivers during that timeframe, and there was around a 135 unique reporters with the data set between COVID and Alzheimer's disease. And with that was a 121 caregivers and 14 patients. When we aggregate narratives, we don't do anything that is password protected. So everything has to be open online, and we don't collect any personally identified information, and any quasi identifiers such as age or gender are aggregated before inclusion into the data set. And that's really just to protect the privacy of reporters. So we used our SPEC-F framework to look at what are people reporting. And we pulled some posts that were reported verbatim on social media, within each of the categories.
So we feel like pulling posts and actually reading about what people are posting is really important because it gives you more insights on what people are going through. And social in the social category, there was a lot of conversation around feeling isolated with the pandemic. That's definitely understandable. In physical, there were reports around long term insomnia, or sleep deprivation. And emotional, there were reports from caregivers and patients around the emotional impact of not being able to see each other and feeling guilty or worried about each other. Cognitive, of course, here, memory impairments is a big one with this disease, but interestingly enough, people were reporting their loved ones, the patients forgetting the arrows on the floor in a supermarket, or forgetting to do social distancing, or wash their hands, wear a mask. And then functional, we had reports around people losing their routine, just because your day to day routine definitely changed during the pandemic. And it's a lot harder for this patient population.
So with that, we had some few key takeaways. So we looked at, like I said, a little earlier, 3,422 narratives. And from them, we saw a lot of descriptions of the worsening of Alzheimer's disease in this population. People had difficulties in understanding and maintaining rules of masking and social distancing. There was limited opportunities for social interactions and reduced caregiver availability, which impacted the patients and then emotional disturbance, or exacerbation of fear, anxiety, and loneliness. Patient disease aggravation definitely resulted from restricted caregiver access, especially if they were in long term care facilities. And this was frequently communicated online by caregivers highlighting the increased caregiver, emotional stress. And both patients and caregivers really appeared to have suffered unique health, cognitive, social, emotional, interpersonal, and daily living burdens that resulted in worsening of disease and the declining of their wellbeing.
So those were our findings with our Alzheimer's disease and COVID study. I'm going to go into how we do what we do and how we're able to gain those insights by using social media and using our NLP. But before I do so, I want to ask the audience a poll. So our poll is, how would you rate your current understanding of natural language processing, NLP? So A is, I don't know anything about NLP. B is I have limited knowledge of NLP. And C is, I have an advanced knowledge of NLP.
John Lyons:
Okay. We've got some responses coming in. Let me give a couple more seconds because they're rolling in. And again, we have answers for three, but it's pretty much an even tie between, I don't know anything about NLP, and I have limited knowledge of NLP.
Dr. Nardin Farid:
Okay. Thank you for that, John. All right. So I'm going to briefly go over how our solution works, and I'll talk a little bit about NLP here, but I won't go into too much depth, so it's not overwhelming. And I know it's a lot on the slide, but I'll walk us through it. So we really categorize our process into three steps, and they're on the top of the slide. So aggregation classification and then insight generation. So, if we just focus on aggregation, here is where we use external data sources. So we aggregate from general social media sites, healthcare forums, healthcare provider forums, treatment forums, disease specific forums, all different sources that I mentioned a little bit earlier. And we aggregate all of the narratives from there and we run it through our noise elimination engine. So this is the NLP that we're using in our aggregation phase. From there, we are able to disqualify posts that are noisy and not going to be useful for the data set, but also we're able to qualify posts.
In here, we give our sponsors the ability to input any of the data sources that they might want to compare next to social media. So if they have any databases or medical information documents, then we can put them into here. And I'll explain a little bit more about different types of sources that can be used here, but really any structured data can be put alongside our social media because we structure our social media data. So we take that qualified bucket of post or, and in addition to internal data sources, if provided. And we move it to the next phase, which is the classification phase. In this phase, we run it through our data classification engine, which is also powered by our real NLP. Here we want to classify our data into demographics, diseases, and conditions, and then treatments and interventions. Again, we go through another qualification and disqualification phase in order to pull only really high quality data into our insight generation platform.
In our insight generation platform, we have five different dashboards and I'm going to explain them in more depth on the next slide, but they include population metrics, population characteristics, disease and lifestyle outcomes, treatment outcomes, and healthcare outcomes. And our insights are around learning more about gaps and care that are described by patients and caregivers, understanding the disease burden amongst those that are diagnosed, and then understanding what are the medication and disease perceptions that people are posting about on social media. So, as I mentioned earlier, in that aggregation phase where we were able to pull in some data sources from sponsors, here's a list in the gray boxes on the top of different data sources that can be included. So you can compare social media to other data sources that are structured, like patient surveys, or like things from EMRs or published literature. And this enables you on this to compare the two types of data, apples to apples almost, because we structure our social media, we can put it alongside the data that you may already be using today to pull some more insights from both.
Our dashboards are these five dashboards on the screen, and they're used in a lot of different ways, but our first dashboard is population metrics. This dashboard shows you anonymized patient profiles, some demographic segmentations, location mapping, and geographic profiling. And this dashboard can assist with patient recruitment learning where patients are understanding the patient population a little bit better. And then in order to learn even more about the patient population, our second dashboard is population characteristics. Here we learn more about comorbidities and the other treatments that patients are on. And if the disease has different stages, then you could also classify and subtype your patients into the different stages like we showed in that Alzheimer's case study with the mild cognitive impairment, early onset and moderate severe Alzheimer's disease.
Our third dashboard is our disease and lifestyle outcomes dashboard. Here is where we use our SPEC-F framework in order to learn more about the symptoms and impairments that stakeholders are reporting on social media. So our social, physical, emotional, cognitive, and functional impairments. And this teaches you more about what are the symptoms that people are reporting online. Treatment outcomes is around treatment efficacy, the tolerability of treatment. When people start, stop, or switch medications, what are the decisions that go alongside that? And then any adverse event monitoring or signal detection. And this one may be really important for pharmacovigilance, or people that want to look at the treatment efficacy in the real world. And then healthcare outcomes is our fifth dashboard. And here we really aim to learn more about what healthcare providers are saying online. So what is their care team mapping and modeling, what is their perception of the disease day experience within it and disease modeling decisions that are being made, and how can we better understand and support healthcare providers within the different diseases?
So we use those modules in order to learn more about the patient journey. So we use it to learn more about what symptoms are people going through, what happens when they're obtaining a diagnosis? Are they having any trouble with that? What are the decisions with the treatment selection and optimization? What are coverage cost, access concerns that the stakeholders may be reporting on social media? And then what's happening in the disease maintenance phase of the patient journey? So we really aim to use social media to better hear patients' voices within all stages of the patient journey from the beginning, all the way to steady state. And we find that social media is a really good source to get all of this information from because people use social media to find support, to gain education, to vent to others, and really to discuss their experience with the disease state.
With RLS Reveal, we're able to pair our technology with managed services if required. So technology is a SaaS solution offering. We use AI and NLP in order to structure data, categorize it and pull some really insightful insights from it. And we're able to do some novel multi-channel real world data sources, which may include social media, but enable you to combine internal and external sources in order to best understand the patient journey within your disease state. Alongside our technology, we compare managed services by having some data science and analysis support services, project management, and then also consulting services for medical publication efforts. So with that, I'm going to pass it onto Stephen for the Q&A.
Stephen Doogan:
Thank you Nardin. Hello, everyone. Just opening up some questions here. I see we've got a few ready. We'll give folks another minute if they want to ask a question here in the Zoom panel, but I'll, well, perhaps start with one of the easier ones, should we say, I have a question here about rare diseases. Have you been able to work in the rare disease space? So the quick answer to that is absolutely yes. I would say rare diseases in our estimation are a really big opportunity, particularly in social media. There's a rather large caveat to this though, as in not all rare diseases are created equal, not all have large active populations on social media. That said, for those that we do have data for and have active social media populations, we've been able to build some pretty impressive cohorts.
And by impressive cohorts of, I mean a couple of interesting things. One is very differentiated self-reported outcomes data, which doesn't really exist at a certain scale in certain rare diseases. And with social media, whilst it can be a bit messy, if you can extract the right concepts out of the data, you can build some really rich, quite decent sample sizes across those rare diseases. So, it is variable. There are certain rare diseases that people have asked us about for which they're very private communities where they don't actively report too much online. So I just caution that if you do work in rare diseases, some rare diseases can be better than others, but just a couple of examples, we worked in Huntington's disease, myasthenia gravis, neuromyelitis optica, the value propositions of the data we've generated, we've worked in rescue recruitment where we're able to find both potential investigators and caregivers as well, and actual communities that weren't working with sponsors or their CROs, and be able to turn them into recruitment channels. That was particularly valuable case study.
We've also been able to work in rare diseases, for example, where in early discovery and development, you've maybe in-licensed to compound, you've got potential autoimmune, or an immunology compound, which could serve potentially 10 indications. Many of which are rare. We are able to actually generate a range of data across a number of rare disease sources and compare and contrast. So, a couple of interesting use cases on rare diseases there. Main thing is ask us if you have a question about a given disease where we really are more than happy to evaluate for no charge what we can get access to. Okay. Sorry, my mouse was just ran out of battery there.
So another question is, how many disease states do you have representative data for? That's a big question. So what I would say is that we have wide coverage across most diseases, but not all diseases are created equal. I would say that we prioritize diseases that are longterm and chronic in nature, and diseases that typically have a patient or caregiver reported outcome element, or the diseases in, requires patient or caregiver reported outcomes to evaluate disease or treatment status. So, those are, particularly think of diseases like neuropsychiatric, anxiety, depression, generally drives more discussion than a congestive heart failure, but that isn't always the case. So again, I would always ask you to share indications you care about, or a high priority and we'll at no charge tell you what we can get. So, we have north of a hundred million profiles, they're real world profiles, wide variability, but if you're interested in understanding, just let us know. We've really worked on very rare diseases and very large ones and a number of things in between. Okay.
Another question here is, can you expand on any case studies in oncology that have shown a lot of value? We've worked on 50, 60 projects that have included oncology as well. For the persons who asked the question, be happy follow-up on this. But one key topic that comes to mind as I'm working in lymphoma DLBCL, which is diffused large cell, B-cell lymphoma, can't remember the exact acronym, but that's one example that stands out where the particular use case was in using a generalist patient reported outcome model to evaluate the impact of lymphoma. And when we were looking at very specific lymphomas, that was specific questions that weren't being asked as part of the traditional patient reported outcome measures. And what we learned was that there were additional concepts, symptoms, and concerns that were not captured by the traditional instruments, but were actively reported on social media.
We know who reported the question. So we'll happily follow-up with you on the specific insights from there, but I think in oncology, just a wide comment there is, we've got, again, same principle, wide variability of oncology data. The majority of the work that we've done has been in looking at not progression free survival metrics. We're looking at quality of life. We're looking at impact of disease, burden of disease. And we've been doing a lot of work into evaluating the usefulness and suitability of existing patient reported outcome instruments to actually capture the 360 of the patient experience, or caregiver experience for that matter. So we'll happily follow up on that question. Sorry, my mouse isn't working here.
Next question. Since you were able to work on a variety of datasets, what have you been able to do with other datasets next to social media? So I think the question is, what have you done outside of social media, as we've mentioned, those other datasets. So, we've worked on wide range of unstructured, real world data as Nardin mentioned, we've looked at the unstructured data that sits within medical information channels, for example, we've also looked into transcriptions of patient and medical staff conversations that have been held within a CRM or a customer relationship management platform. Just what I would say is that often the data that we get that is not social media data that comes from internal sponsors. So from pharma, it's typically that unstructured data that you collect that is inbound.
So drug safety databases, we've also looked at medical information channels. We've also looked at CRMs. We've also looked at spontaneous reporting channels, medical complaints. We've also looked at that. And these are sources of data for which you've already got people looking at today. There's already a vendor, or internal staff, internal medics that review this data. They take data and they perform a certain task like register an adverse event in a safety database. What we do with those sources of data is try to add a layer of structuring on top of that data that doesn't have the human intervention. And that allows you to basically turn those databases into structured metadata for further investigation. So, for example, we've worked in the antibiotic space, we've looked at adverse events reported on antibiotics on social media. We've been able to compare it to an internal safety database of a very large sponsor, big antibiotic producer. And we've been able to look at the differences in reporting rates of adverse events.
Do you get an earlier signal on social media versus your internal safety database? Do you find different events on social media that you've never found in your drug safety database? So I think the recurring theme of what we've been able to do outside of social media is largely comparing social media to other channels of data. And it's really using our NLP and our things like SPEC-F, things like our treatment outcome models to represent any channel of spontaneous data with the same structure. Once you can achieve that, you can then compare apples to apples. And we have various analytics that we put on top of it, which we've not gone into today that harden the insights, or perhaps account for biases, for example, when it comes to insight generation.
So, yeah, I think that's got us... So did I read out all of the questions here? I think that has all of our questions summarized here. So, I'd like to just make a closing remark here that we've started in social media, focusing on that type of verbatim language for a very specific reason. It's very chaotic data, but there's a tremendous amount of value in it if you can extract it. And we've focused on building NLP and we've started this seven plus years ago, because we know there's value to be extracted and we see an expansion of this social data happening. And the pandemic really, I think increased that trend of spontaneous data being reported online. And so we need structured methodologies of looking at this data. And the byproduct of being able to structure social media data is that you can actually structure other sources of data.
And our goal, and one thought to leave you with is, just even if you haven't worked with social media before, you've probably worked with other datasets that have given similar insights than what you can get out of social media, with social media you don't need to recruit. It's not going to give you every perfect patient population, but give us some suggestions around diseases, or questions that you have, and you might get really lucky with this kind of data. And we'd be happy to have a conversation with that about you at any time. So that sums up our questions for now, John. Oh, sorry, there's just another one come in. If you don't mind, I'll just answer that question and then I'll pass it back over to John.
So how are insights gathered through your process valued by regulatory authorities and researchers? That's a great question. So, first and foremost, we have actually been into regulators, FDA specifically. FDA have unstructured data problems, as I'm sure you are aware if you think about VAERS or FAERS, for example, the FDA adverse event reporting systems, or the vaccine adverse event reporting systems, they intake, they receive adverse event reporting data in unstructured formats. So, the FDA is interested in automated approaches to reducing the level of manual review of these intake forms. So my, again, I don't have lots of regulator anecdotes here, but my one regulator anecdote based on our experience is that there's a lot of methodological interest in deploying NLP and on unstructured sources of data.
And I think there's a lot of methodological considerations, accuracy rates, what are accuracy thresholds that are regulators willing to take? And we are just really starting to implement hard automation in large organizations like the FDA in this unstructured data arena and more to come. For researchers, we actually have a very active researcher collaboration as network, if you like. Part of it is brought about by sponsors. So, a lot of our work, we partner up with different universities. We have a very strong partnership, just for example, with Emory University. So, the researchers' general interest in this is, I think they appreciate the methodology. I think they appreciate the source of the data. So social data, and being able to, in collaborating with researchers, essentially what we're able to do is take our opinionated view of the methodology, we're able to work with additional experts in NLP, in the academic community, and are able to refine our approaches to get higher accuracy, higher throughput, but we're also able to provide academics with a very rich data set, that they frankly don't have great access to today.
So we can give them not only raw unstructured social data, that they can test their models out on. But what we're also able to do is provide the structured metadata that comes as a result of the NLP, and we're able to give them large datasets to do insight interrogation and evaluation. So, my comments and response to the regulatory authority and research of questions there are really, we have experience working with these groups early stage in some respects, very interesting in most respects. And I think I would say the core theme is novelty and wanting to make the novelty a real reality is a recurring theme across these two groups. And yeah, it's going to be an interesting future with those two stakeholders as well. So thanks for that question. And I think that sums up all our questions.
By examining traditional (e.g. patient reported outcomes) and non-traditional (e.g. social media) approaches to collecting patient insights that were mapped into a framework known as SPEC-F and modeled around Development Decision Points (DDPs) and milestones. Several approaches are highlighted based on the relative prevalence of clinical, economic, functional, behavioral and perceptional data and the efficiency for insight generation. Published retrospective analysis were conducted across 17 case studies from early-development, peri-launch and post-market programs and the impact of traditional and non-traditional approaches on patient access.
