Entity Resolution at Scale: Graphs, LLMs, and Real-World Data Messiness
59 minAudio
Summary In this episode, I spoke with Steven Renwick, CEO and co-founder of Tilores, about the realities of entity-oriented data processing and the practical role of LLMs in entity resolution. We covered the confusing terminology around entity resolution, identity resolution, record linkage, named entity recognition, and disambiguation, and why these problems show up almost everywhere once organizations reach a certain scale. Steven explained why entity resolution is especially critical in domains like financial services, public sector systems, logistics, and any environment where messy, duplicated, or inconsistent records can create compliance, operational, or analytical risk. We also dug into the question of whether LLMs can replace traditional entity resolution systems. Steven shared why the answer is only partly yes: LLMs can be very useful for discovery, entity extraction, weak-match review, onboarding, and configuration, but they struggle as the core production matching engine because of cost, scale, consistency, explainability, and state management requirements. We explored why graph structures are so useful for representing entities, how TeloRes approaches scale without relying on graph databases directly, and why teams should think carefully about build-versus-buy when tackling entity resolution. The conversation closed with lessons from real-world data quality, the growing importance of trustworthy identity layers for AI and analytics, and how clean entity data can strengthen everything from knowledge graphs to AI applications. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Today’s episode is sponsored by Parallel - where agents find answers. Most engineers today closely follow new model releases, but don’t pay attention to their agent’s most important tool: web search. Parallel develops enterprise-grade infrastructure for agents to retrieve high-quality context from the web. Their core products are a suite of APIs for retrieving high quality information from the web with Pareto-optimal quality, cost, and speed. Whether you work on voice agents that need 200 millisecond latency, chat bots that balance speed, depth, and quality, or long-horizon agents to do thorough, overnight research for you, Parallel is a single platform for all your agentic research. Get started for free at dataengineeringpodcast.com/parallel Your host is Tobias Macey and today I'm interviewing Steven Renwick about whether and where to use LLMs in entity-oriented data processing Interview Introduction How did you get involved in the area of data management? Can you start by giving your definition of what constitutes an "entity"? What are some examples of the types of data work that rely heavily on entity resolution/disambiguation? How do LLMs alter or augment the functional capabilities of entity-oriented data processing? As with many areas of knowledge work, LLMs can easily create a compelling demo, but quickly degrade when trying to go from demo to production. What are the traps and pitfalls that become evident when trying to do entity resolution in production? What are your heuristics for what constitutes the threshold beyond which naive approaches become untenable? Can you describe what a production grade pipeline for entity resolution looks like? What role can/should LLMs play in that process? As LLMs become more powerful and faster, how/does that change the equation of their role in entity-oriented data processing? (particularly with diffusion-based models) What are the most interesting, innovative, or unexpected ways that you have seen LLMs used in entity extraction/resolution/disambiguation? What are the most interesting, unexpected, or challenging lessons that you have learned while working on entity resolution at scale? When is entity resolution overkill or unwarranted? What are some predictions or wishes that you have for the future directions/applications of entity resolution? Extra - have we looked at the new "System One" AI decision models,like Jev? Contact Info LinkedIn Parting Question From your perspective, what is the biggest gap in the tooling or technology for data management today? Closing Announcements Thank you for listening! Don't forget to check out our other shows. Podcast.__init__ covers the Python language, its community, and the innovative ways it is being used. The AI Engineering Podcast is your guide to the fast-moving world of building AI systems. Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes. If you've learned something or tried out a project from the show then tell us about it! Email hosts@dataengineeringpodcast.com with your story. Links Tilores Named Entity Recognition (NER) Entity Resolution Neo4J AWS Neptune Apache Spark Graph Traversal Algorithms Entity Resolution Blocking GraphQL Clique-Based Graph Compression Splink Levenshtein Distance Cosine Similarity GraphRAG Entity-Resolved Knowledge Graph JEV Decision Model Laya The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA
Audio and video play straight from the show's own hosting. PodSubs stores and serves no media.
Transcript
From the show. Tap a time to play from there.
Tobias MaceyHello, and welcome to the Data Engineering Podcast, the show about modern data management. Today's episode is sponsored by Parallel, where agents find answers. Most engineers today closely follow new model releases, but don't pay attention to their agent's most important tool, web search. Parallel develops enterprise grade infrastructure for agents to retrieve high quality context from the web. Their core products are a suite of APIs for retrieving high quality information from the web with Pareto optimal quality, cost, and speed. Whether you work on voice agents that need two hundred millisecond latency, chatbots that balance speed, depth, and quality, or long horizon agents to do thorough overnight research for you, Parallel is a single platform for all of your agentic research.
Get started for free today at dataengineeringpodcast.com/parallel. Your host is Tobias Maci, and today I'm interviewing Steven Renwick about whether and where to use LLMs in entity oriented data processing. So, Steven, can you start by introducing yourself? Yeah. Thanks very much for having me on the podcast. Yeah. My name is Steven Renick. I'm the CEO and one of the co founders of TeloRes.
Steven RenwickWe are a entity resolution company based in Germany. I'm originally from Scotland, but have been based over here for the last fourteen, fifteen years.
Tobias MaceyAnd do you remember how you first got started working in data?
Steven RenwickOh, well, my background's quite eclectic to say the least. So I'm actually a biologist by training. I've got a PhD in genetics and that wasn't even bioinformatics. I was doing proper wet lab stuff, working on microscopes, looking at cell division in yeast. I decided to try and get out of biological sciences. I didn't find it was super interesting. So went back to university to do an MBA. And after that, I started what I thought would be a kind of data based company. I started
in 2012, a company that was connecting into cloud based accounting software run by small businesses. This was when cloud based accounting software like QuickBooks and Xero was just getting started. And I had this kind of realization that all this amazing data that was previously just locked away in small businesses was now available via an API. And if we could source that from all these small businesses, we could build some pretty cool products. And I built a company that was doing that, building real time credit reports, building a real time credit control system, then ultimately
building invoice finance services. And I stuck with that. I exited the business about ten years ago, ingloriously, I would always say, because I didn't make any money. So I had to still work. I joined then a company in Germany, which is a consumer credit bureau. And as a consumer credit bureau, you're basically dealing with data. And that's kind of where the Tilores story started. We were working with vast volumes of data in this credit bureau, and we needed to come up with a smarter way of doing entity resolution.
Tobias MaceyAnd so before we get too deep into the specifics of whether and where and how LLMs fit into this overall stack of dealing with entities, there are a number of terms that come along with territory that not everybody is going to be familiar with. So I'm wondering if you can talk through some of the different ways that people talk about entities in the context of data processing.
Steven RenwickYeah. As an entity resolution company, this is kind of the bane of our life, is entity resolution, it turns out that nine times out of 10, even if you're speaking to data scientists, they haven't actually heard of it because the field has got so many disparate terms. So we would call it entity resolution. If you're a bit more working in the marketing field, you would call it identity resolution. You might also call it record linkage. Other people might kind of simply call it deduplication. And we have people coming to us asking for all these things, but ultimately what they're meaning is entity resolution.
And then where you potentially get the confusion is often with named entity recognition. So kind of classic NLP stuff where you have a passage of unstructured data and you are trying to identify what is this passage of data about, like who is the person that was referred to in this passage and what are the relationships they have with other things which are in this unstructured
data. So ultimately it all comes down to an entity. What is an entity? Even that word itself seems to confuse people. But for us, an entity is a real world thing and it can be anything. Most of the time we're dealing with either natural persons or a company, but it could also be a product part. It could be a bit of DNA. It can be pretty much anything that which is a real world thing that you are trying to refer to.
Tobias MaceySo effectively anything that you can qualify as a noun.
Steven RenwickYeah. That sounds about fair.
Tobias MaceyBecause I've also seen entities be something that is more of an idea. So for instance, I've been experimenting with doing some entity extraction for some of my podcast transcripts, and one of the values or one of the concepts that I want to extract for being able to provide as a linkage across different episodes is what is the topic or what is the idea that was discussed in the context of this episode, and where else was that discussed in a similar way?
Steven RenwickYeah. I mean, certainly almost consider any way of classifying something could be an entity. So that could be the categories, and then you're trying to link everything which is related to that category into one entity. As it turns out, there's a million different ways to kind of skin the data cats.
Tobias MaceyAnd as I've been digging into this field over the past years, there are also other neighboring terms that come up. So you mentioned named entity recognition, which is also sometimes referred to as entity extraction. There is entity resolution, which is what we're focusing broadly on today. And then there's entity disambiguation, which is, in some ways, I've seen disjoint from resolution. Although, I I imagine that there's a decent amount of overlap there, and I'm wondering if you can provide some of the color that you've had to address in terms of customer education over the years.
Steven RenwickYeah. I mean, to be honest, I guess entities disambiguation isn't something which comes to us so often as a phrase, but it's effectively part of entity resolution because just as important as linking data together is making sure that you're not linking together the incorrect data. So it all effectively comes in together into one process. I mean, even the named entity recognition is something that we need to be, if not directly doing ourselves, at least able to advise our customers on how do that. So we will normally be working with customers that have got
structured record data already. So we need data where we can get hold of the attributes that we're actually using for the matching. But very often we have people coming to us saying, well, I've got these legal documents or something. So can you help us with the named entity recognition as well? So named entity recognition, entity resolution, entity disambiguation, in the end, they're all pretty much rolling together into one field and, yeah, confusing people when they come to speak to us already.
Tobias MaceyAnd you already alluded to unstructured data as one of the areas that this often comes up, but I'm wondering if you can give some sense of what are some of the types of data work or types of other industries or situations where this problem of entity entity oriented data processing. I'll I'll sort of use that as the mouthful phrase to try and be the all encompassing term, but what are the cases where you see that most heavily relied upon?
Steven RenwickYeah. I guess the crazy thing is that pretty much any reasonable sized business with any reasonable sized volume of data is going to have entity resolution problems, whether or not they actually know it and whether or not they've actually named it as an entity resolution problem. Of course, there are certain fields where it is much more critical. So I mentioned that I was working at a credit bureau, which is actually where we developed our technology. We couldn't find anything on the market which was able to do this at the scale and speed that we wanted. So we built this brand new entity resolution engine from scratch
because credit bureaus have this data on steroids. It's the same problem in every single country, is that credit bureaus are sourcing data about people from dozens of different sources. And that data is messy, it's inconsistent, it's formatted differently. People change their names, people change their addresses, people change email addresses. And working with that data is pretty challenging. So credit bureaus is where we came from. The kind of the space where it's probably the most critical
is financial services. So everyone's aware of kind of KYC, know your customer type things, which is basically knowing about your customers. So if you're a big bank and you've got millions of customers, there's a pretty good chance that you have the same customers in your systems multiple times without realising it. If you've got multiple banking units, if you've got personal banking, commercial banking, mortgage banking, stock trading, there's a very good chance you've got the same customer over those different units and you don't realise it.
Matching those customers across business units is entity resolution, as is sourcing external data and screening that against your customers. So if you're pulling in sanctions list data, if you're pulling in adverse media or politically exposed persons data, that's again just entity resolution. You're trying to screen mister whatever Vladimir Putin to make sure that you don't have Vladimir Putin or one of his many 30 or 40 aliases in your customer base. In the public sector, it's a massive issue.
So, obviously in The US, you've got some elections coming up soon. Voter rolls. Each county and each state is pretty much managing their own voter rolls. When people move house, you have to... They have to deregister for one place, register to a new place, then and you've to make sure they're not double registered. If somebody has died, then you can be comparing the death registers against the voting registers. You could be comparing the voting registers to the driving and social security registers. So, so many different spaces where
the only question really is, is it actually super critical that you get the entity resolution right? In banking, if you get it wrong, you can be putting yourself in for some pretty big compliance problems. And obviously, with things like voter data, it's critical for kind of, the integrity of democracy, that we trust the the data that people are actually working within the voting registries.
Tobias MaceyAnother place that this idea of entity extraction and resolution comes up is whenever you're discussing things like knowledge graphs for being able to do things like... I mean, GraphRAG has, I think, brought it back into vogue, but it's something that has been an area of study for a long time and is typically very easy to do a prototype and a demo that looks fun, but very hard to do in production at scale over longer time horizons. And I'm wondering as LLMs have been brought into
the set of capabilities, how that's changing some of the landscape or at least the expectations of the ability to do entity extraction and resolution.
Steven RenwickYeah. We get asked that a lot recently. You know, the sort of question investors will ask is, well, can't LLMs do everything that you're doing? And there's this wonderful German word, which is jain, which is a portmandeau of jat with nine together, which means, kinda. And that's the answer. Because if you put in a thousand records into any large language model and say, ask it, which of these records belong together? Then you're gonna get a reasonably good output.
The problem, as you said, comes from the difference between running a nice demo, which works on your desktop with a small volume of data, and actually doing something which is reliable in production and at scale. So we do believe that kind of large language models do have a place in entity resolution, but it's not in production. It's not in production for the actual matching. And I'll give you a few reasons why.
One of the challenges with entity resolution that not many people kind of realize is that it's a quadratically scaling problem. So a thousand records isn't too bad, but in theory, with a thousand records, you have to compare every single record with each other. So if you've got even just a million records, that's about 500,000,000,000 possible pairwise combinations that you need to check. So if you try and do that on a large language model, it's just gonna take absolutely ages and cost
way, way, way too much money. So a million records isn't really very much data. We are working with people that have got hundreds of millions of records that they might ingest every day. So you can see just on the scale of the data, a large language model is not really the right tool for doing that kind of scale of entity resolution. There's also other problems such as consistency. If you ingest the same data multiple times into an LLM, you're gonna end up getting a slightly different answer each time. And
explainability. We're usually working with regulated companies where being able to very accurately explain why A is linked to B is super important. And then there's things like the statelessness nature of it. Entity resolution needs to be state based. Whereas if you do it in an LLM, you've kind of done it once and then you'd have to do it again and again and again. When we're doing entity resolution, we structure the data as graphs. So you can see that A is linked to B, B is linked to C, C is linked to D. D and A are not directly linked, but you know that they are related because they are part of the same entity. I could ingest a new record which is related to B and C, it will link to both of them. And I know that the structure of that data makes sense. I can see
the reason for the edges between those nodes in the graph. And that for a regulated business makes a lot more sense. It makes a lot more sense for kind of managing the data later. If I need to delete the data, it's very... With a graph based system, it's relatively straightforward for us to remove one record and the logic of that entity still makes sense. That could mean that you actually have to split an entity into two. All those sorts of things are not suitable for large language models. But where we see the large language models actually working quite well is, for example,
the named entity recognition. So we see other... We don't do it ourselves directly, but we've seen others have relatively good success with the LLMs in the entity extraction. I can't speak to the costs of it, but that seems to work quite well there. Where we use it, like in our actual product, is as what you might call the agent in the loop. So you're always gonna have pairs matched where you're not a 100% certain as a weak match.
In that case, you wanna put it into a review queue. And traditionally you would have a human analyst look at that manually and make a judgment call. Okay, based on what I can see here, it's not met the threshold for an automatic connection. It's not met the threshold for an automatic rejection. It's in that gray area. Should A here link to B, a human can make that decision and write a justification for it, but an LLM can also do that. And then you're really only talking about a very small proportion of the linkages. And then you're basically still putting that record into the entity graph. So you're still maintaining the
best way to actually manage the data. So that's where we see LLMs working well in entity resolution day to day. And we also use LLMs for the onboarding onto TeloRes because entity resolution is as much of an art as it is a science. I mean, it is definitely a data science, but there's definitely an art to it. And we've managed over time to kind of LLN models to be able to configure
the entity resolution based on your data with a very good degree of accuracy. And it can do that very, very quickly and go through refinement cycles on its own without you really needing to intervene. And that for us works works very nicely.
Tobias MaceySo you mentioned the utility of graphs in the entity resolution functionality, which is something that I've seen in a lot of other contexts as well, which is also, I think, where the idea of knowledge graphs gains a lot of its benefit is because of the way that you can do that linkage and being able to traverse the edges and nodes to be able to have a confidence score or at least some recommendations for... We think these two things might be the same because they showed up in these similar contexts, and they share this other attribute that links them together.
And one of the challenges that I often see when talking to people about graph databases or graph structural representations is in terms of how how much they want to invest in the graph for purposes of things like graph based machine learning or being able to do long and complex node traversals because of the fact that you end up with super nodes, which complicates your ability to be able to shard across multiple machines, which then causes you to have to
spend a lot of time and energy trying to be very clever about how you shard the graphs if you have to go to that point. And so I'm wondering how you're approaching that and what are some of the limitations that you put on the graph and its traversals to be able to make that scalable?
Steven RenwickYeah. So like I mentioned, we built TLR S when we were inside this credit bureau. And we, first of all, tried the bunch of approaches you would first of all think of for an entity resolution engine. We tried Apache Spark, we tried Elasticsearch, and we tried graph databases. So we tried Neo4j and we tried Neptune from AWS because it makes sense to think of a graph database. It seems like it could work for entity resolution as well. But we found those problems that it didn't scale very well for exactly the reasons you're mentioning.
And particularly when you were working with any sort of entity that could be vaguely complex, those multi node transversals just became super, super slow. And we were trying to provide data to, or still are trying to provide data to people that are using these data in decisions for banking, for credit decisions, for fraud decisions. And they need to have that data back in less than a few 100 milliseconds. So we decided not to use the graph databases,
but we recognised that graph is at least the correct structure for the entities themselves, but the entities themselves need to be distinct. So the approach we took was to build it based on serverless technology, so specifically AWS, where we would index the data as it was ingested. So we're constant, as any record is getting ingested into Telaraz, like in density resolution in general, one of the biggest problems you have is this whole quadratic scaling problem. Which means that the matching isn't necessarily the super complex part, it's the scaling which makes it super difficult. So what we came up with was this kind of...
Or what you would do in entity resolution in general is the so called blocking technique. So you're trying to limit your universe of data so that you're only comparing this new record you've got with a universe of... Or a world of data which is likely to be related to that one. So rather than comparing a person record with every single person in The United States, you would say, Well, I know this person's in New York, therefore only compare them to other people in New York, which is arguably still a large volume of data. So what we did was when
we ingest any individual records, we index immediately in DynamoDB and then link it into the entity immediately. And then that means that the entities that we have in TLRS are already all pre computed and assembled. So if you ever need to find and access an entity, then you're literally just doing the search in DynamoDB and then pulling in an S3 file where the entire entity
exists. Now, we use GraphQL instead of a RESTful API. And the reason for that is that these entities can get pretty big. If you've got an entity which has got a thousand records in it, that could be a lot of edges there. You don't necessarily need to see an individual person's, an individual's name a thousand times or their address a thousand times. You want to be able to summarise that data on the API. So that's one reason that we use a GraphQL API, because for every individual query, you can change
exactly what you're asking for. You can do some kind of analytics on the actual API. But the result is that we kind of solve the graph scaling problem. So whenever you query an entity from us, on kind of the cloud version, you will always get the entire entity back in a consistent time. Doesn't matter if it's one record or 100,000 records in the entity. Once you've found it, then it's just a kind of a timeline to transfer that across the network
to your computer. We do still run into some pretty big entity clusters sometimes. So humans, like natural persons, is not so difficult to do the entity resolution on because a person is a person, right, at the end of the day. They don't get super complicated. You maybe do households, you maybe do families, but a person is a person. Where it does get quite complicated is companies, for example. So depending on how you want to structure companies and depending on what data you're working with, you can end up with entities that have got tens of thousands or hundreds of thousands
of records in them. And that did start becoming a challenge for us. And so we developed something which we'd seen mentioned in the academic literature, which was click based graph compression. So if you have a bunch of sub clusters in a graph, which you think are all related, you can cluster them down into an individual clique rather than pulling out all the records and all the edges together. And then it simplifies and kind of compresses your graph together. So that and a few other things, those sorts of things we've come up with to kind of overcome the graph database problem. And
because we're running on serverless, we never have to think about sharding or scaling any databases. We don't even think about it. We are a very, very small team and we've run hundreds of instances of tRelayers in parallel with billions of records and hundreds of millions of entities at any one time. And we've never had a second of downtime in five years, and we've never had to think about scaling anything. I know serverless gets a little bit of criticism in some quarters for being unnecessarily expensive, but for us, it's been
beautiful. Like the nature of our business means you can't predict the spikes of the demand, but serverless just handles it and it works nicely for us.
Tobias MaceyYeah, there's always going to be critique no matter what the solution is. There are also still people who are debating the utility of the cloud and all of the conversations about moving to cloud and then cloud repatriation.
Steven RenwickYeah, especially over in Europe, that's kind of going the other direction. I mean, now you have to start talking a lot about cloud sovereignty and stuff like that. So yeah, that's definitely an interesting discussion as well. Absolutely. And then that also brings up conversations around LLM sovereignty and
Tobias Maceywhether and where the data can traverse different network topologies, etcetera.
Steven RenwickYep. Yeah. Yeah. Like I said at the beginning, we mostly run on AWS and we always localise for customers. So if you're a US customer, only run-in The US. If you're in Europe, only run-in Europe. And so far for most people that seems to work, but we definitely are having more people asking about running on private infrastructure.
Tobias MaceyAs you mentioned, most organizations do have some problem of entity resolution, whether they acknowledge it or not. And so by that virtue, most data teams are going to have to address this problem in some capacity even if it's a manual operation of somebody saying, oh, this data looks funny. Are these two people the same one, or this person is linked to two different things that shows that maybe it's not the same person.
And I'm wondering what are some of the ways that you think about the gradations of necessary investment for dealing with entity resolution and entity extraction for teams of varying sizes or varying problem domains?
Steven RenwickYeah. I guess one of the first things you've obviously got to think of is the sheer volume of data. We sometimes do have people come to us where actually they've only got maybe 10,000 entities or something. And you're like, well, for that volume of data, you're not gonna have scaling problems. You can probably maybe even do it in the LLM. It'll be fine. There are open source libraries, which for pairwise matching are pretty good. Like Splink is
a pretty good one. But it more comes down to everything around the entity resolution. So where's the data coming from? Is it batch based? In which case, batch based can be pretty challenging as well if you're suddenly ingesting another 100,000,000 records. That's difficult to deal with. But real time as well. Real time doesn't mean doing batches very quickly, right? It means being able to stream data in, more or less the order that it actually comes in. So if you have real time requirements,
that's again something where it's gonna be pretty difficult to do that yourself. The diversity of data sources. So we've built a whole kind of ETL tool into our solution because I think people often don't realize just how bad real world data can actually be until you've worked on these data projects and seen big and big real data coming from the real world where maybe some of it's manually entered or
something went wrong with some database some time ago. Like, the actual transformation and normalization of the data is as important as the strategy you're using for the matching the data as well. But that itself is a massive project. Do you wanna really build a whole ETL tool for transforming and normalizing data? That's not a wheel that most people should really be looking at reinventing. And then there's like, what's the consequences of
the merging? It's like, is this actually something which could cost you money if it gets you wrong? Is this something that could get you wrong, get you in trouble with the authorities? And then what if you need to delete or reverse matching? When people talk about sometimes deduplication instead of entity resolution, the assumption is that you're pretty much merging everything into one record. Once you've got everything merged into one record, you can't reverse that. That's
a one way door. Doing it properly means that you can reverse those things, or you can delete an individual record from one source. You might have something like five different sources, and data from one source needs to be deleted one year after each individual record is acquired, data from another source, you have to retain it for ten years. Having those sorts of things happening automatically on the data as it's coming in, that's pretty nontrivial as well. So if you just look at it as a data matching challenge, then it's
not super, super, super complicated. A good data scientist on their computer can come up with a reasonable set of matching rules, which will probably work pretty well on a relatively small volume of data. Once you start scaling that up, once you start doing that in real time with all those explainability and reversibility features, then you're running into a world of hurt that you shouldn't really be putting yourself through. And if you speak to people that have built entity resolution systems once or twice, they very often say, God, I would never want to do that again. It's just that the long tail of things that you have to take care of is super annoying. And once you've got over the kind of the fun part at the start, you're just onto boring stuff you don't want to deal with.
Tobias MaceyYeah. One of the other avenues that I've discussed this challenge of entity resolution is, I think a few years ago, I was speaking with a company that does this for real estate of having to be able to identify, is this building lot the same thing as this tax record? Is it owned by the same person? And this... Is that the same thing as this company over here? Just being able to bring all of those things together so that you could accurately report on real estate? I think they were focused largely on commercial properties.
So it gets very complicated very quickly.
Steven RenwickYeah. It does. And that's the of the cool thing about this space is we position ourselves just now as very broad general entity resolution, which from a startup point of view is probably a bad idea, but it means you get lots of cool use cases coming to you. And we have people coming to us with use cases that we would never have thought of. And that does make it certainly more interesting.
Tobias MaceyOne of the other challenges of something like entity resolution is that as your use cases mature or as your business changes, you might need to expand or change what constitutes an entity. And maybe you have a decade worth of data, and now all of a sudden you have a new entity type that you care about. And so you have to then go and backfill all of those records through that processing again. And I'm wondering what are some of the
ways that you have found effective for addressing the evolution of the data model or of the problem domain so that you can be a little bit proactive about maybe identifying some potential new entity types that aren't explicitly designated at the start?
Steven RenwickYeah. So what we want to try to avoid is having to re ingest your data. Because as I mentioned, we are creating the entities on ingest, which means the entity is kind of defined at the time that you ingest the data. Which means if you wanted to change a rule in Telarize, so Telarize is a rules based system, then a new rule would only apply to data which is ingested
after that point. And that works fine for us. That means that we're effectively fine tuning tLorez constantly as you go along. You can do that as a human or you can use the MCP server to use an LLM to do the fine tuning on the rules. But that will only then apply to the data going forwards. We do have people that will sometimes say, Well, I want to have that apply to all the data retrospectively, in which case we will need to do a re ingestion.
Which as long as you're not doing a full re ingestion once a month or something like that, that's definitely something which is possible. Again, that's kind of like the beauty of serverless, is you can say, right, I've got a billion records here. I need to just re ingest it. And just depending on how much resources you apply to it, how many Lambdas you've got in parallel doing the ingestion and the matching, it'll happen in a few hours. So that works pretty well for us. If somebody says they want to look at a new entity type, then in our case, it's more likely that they might run multiple different versions of TeloRes and each
version's got a different kind of setup to detect different type of entities. Because in that case, it's more likely they're working with different data sets. We're usually working with a kind of homogen ish data set. So it'll be customer data it'll be company data or it'll be product data. That's kind of separate datasets. It's not like kind of like a graph database where you've got all those sort of data in one place and you're looking for the relationships between it.
Tobias MaceyThough I imagine that LLMs could be valuable in the discovery phase of identifying what are some of the different types of entities that I have in this collection of data, particularly if you're dealing with any sort of unstructured or semi structured documents to be able to say, this is my business. Here's a sample of my data. I want to start doing entity extraction and resolution. Help me identify what are the prime candidates for entity types that I might want to deal with.
Steven RenwickYeah. Exactly. That's that's where we think the LLMs work really well is in this kind of preconfiguration and discovery phase where the LLM is the over enthusiastic data analyst that has the time to go through all those documents and suggest to you what makes sense to look at. So yeah, exactly. That works really well. And then either using an LLM or a kind of maybe a more regular NLP approach to do the entity extraction works perfectly.
And then you can use the LLMs to kind of sense check things. You've done some entity extraction from the data. Does this match up with what you were actually expecting to see? And then just configure the whole entity resolution system to all match it together. So LLMs can jump in at every stage as a kind of data analyst checker of everything you're doing. Also works pretty well for sampling the data. One of the challenges is measuring precision and recall. So entity resolution systems,
you wanna be able to measure precision and recall to see how well is it actually doing. The challenge with that is most of the time people come to us, they don't have labeled data. So it's pretty difficult to actually measure precision and recall. So arguably, you can use at your expense an LLM to do some entity resolution, to label some data, and then keep that as your consistent dataset, which you then put into your entity resolution engine to refine as you go along, then use the LLM to check the results, that can work quite nicely as well.
Tobias MaceyBecause of the fact that LLMs make a lot of those early prototypes faster and easier and cheaper than trying to get a human dedicated to it for a week or a month to come up with an early proof of concept, how are you seeing that change the marketplace for the work that you're doing and the demand or recognition of the necessity for entity resolution where maybe five years ago, somebody would have said, yeah. It would be really great if I had this, but I don't have the time or money to invest in proving it out. But now that the experimentation
phase can be sped up and compressed, maybe it brings it to the fore as something that people actually want to bring on board as a capability?
Steven RenwickYeah. Good question. There's two or three kind of bits there which are relevant. First of all, of course, like we said a couple of times, you can use the actual LLM for direct entity resolution. What makes a bit more sense is using an LLM to build a basic entity resolution system, right? So having it build you a kind of Levenshtein distance, cosine similarity, metaphor and phonetic matching mini entity resolution system in Python and running that on a dataset.
And what we've seen is that people are building entity resolution prototypes using LLMs, and then using that on modest datasets to say, okay, we actually have data here where if we had a proper entity resolution system, we'd actually get some business value out of it. So like I said before, almost every business has an entity resolution problem, whether or not they realise it exists.
Where the kind of the LLM approaches or the LLM basic entity resolution systems help is helping the data teams do internal kind of POCs to prove it to management that they're gonna get some value if they invest in actually cleaning up the data properly. What's also really interesting for us is, of course, everyone is wanting to run LLMs on their own data. Everyone wants to build agentic tools. And that's why GraphRAG has become,
one of the reasons graph databases have become so popular is because of GraphRAG. But we think there's also a big opportunity for entity resolution systems to be part of the data infrastructure for AI as well. So we tried to coin that identity rag, didn't quite take off to the same degree as graph rag because we're not Microsoft. But if you're, for example, you're trying to run an AI agent on your customer data, then it's not gonna work very well if the customer data is duplicated
or if you can't link customer data from multiple different data sources, and then that customer data has the usual real world problem, that is that it's not easily linked together. So that's another reason that we think entity resolution is about to have a bit of a renaissance, is we've seen real demands from people that want to maybe do something which is a bit simpler than GraphRAG because they're using a one type of data, but they want to have clean data for LLMs and an entity resolution system, especially a real time entity resolution system, makes a pretty perfect
data source for AI.
Tobias MaceyAnother way that the general ecosystem investment that has been driven by LLMs can factor into this is the prevalence of vector embeddings and embedding, vector databases for being able to do some of that discovery of semantically relevant material where maybe you have a large backlog of data that you don't know whether it has any of the entities that you're already reporting on,
and maybe doing the investment in chunking and embedding those helps you surface some of those additional documents to say, hey. Now that we have identified this particular named entity or this entity type, here are all of the other locations that might hold that and then being able to surface that as candidates for additional extraction and enrichment of the information that you already hold.
Steven RenwickYeah. I mean, we quite often work with people who are working on vector embeddings at the same time. So we haven't used any vector embeddings in the matchers for TeloRes and we haven't seen people have success. People often try to use vector databases for entity resolution, or we've seen people come to us after trying to use a vector database for entity resolution, and it doesn't really work. There's too much chance of this kind of
fuzzy overlap between different entities where they're similar, but they should actually be distinct. Of course, the vector databases work well for kind of unstructured data where you're maybe doing the entity extraction, first of all, to say that this document is about this person and then the rest of the document is split into embeddings and used in a vector database. That makes a lot of sense. So we like to think that we're pretty complementary to the vector embeddings, but it's not something that we have to kind of compete with head on.
Tobias MaceySo in your work of building TLRAS and focusing on this overall space of entity resolution? What are some of the most interesting or innovative or unexpected ways that you've seen people bring that capability to bear?
Steven RenwickSo we worked on a project relatively recently, like a pro bono project after the Venezuela earthquake. So after that earthquake, there were tens of thousands of people who were missing. And there was very quickly a bunch of kind of citizen initiatives to source data about who was missing. What you inevitably end up with is multiple missing persons lists. And you have these poor relatives of people who are trying to search through this list to find if their loved one is on one of these missing lists already. Or people will submit the name multiple times or they'll submit it to different lists,
and you end up with a big data problem. And my wife is Venezuelan, so we were following the news very closely after the earthquake. And she told me, Hey, there's people talking about big data problems with this. There's like so much data duplication that the data is almost unusable. So we kind of spun up a TeloRes instance and started de duplicating the data. Where the LLM became interesting was that was very shortly after we'd released our MCP server. And I'm
not a computer scientist, so I was not the best person in the company at setting up a TeloRes instance. But because we had this MCP server, I was able on my own over the weekend to kind of get this set up running, get it connected to all these public data sets that were being produced by citizens in Venezuela. And use it to de duplicate the data and then basically build like a person search engine. Then we found that at the same time, there was obviously a lot of people in hospitals
and a lot of these people were just brought in. They couldn't... You know, they didn't have relatives with them. They might have been unconscious. They might have been badly injured. And there was literally people taking photos of lists of names that were scribbled on bits of paper on hospital wards, and there was loads of these. So other people were then sourcing that data, kind of structuring that data into databases, and I would imagine they were using LLMs for that. But we then had these kind of two datasets. We had the
multiple missing persons datasets, and we had the people who were in the hospital dataset. And we were able to combine the two. That's a kind of classic entity resolution problem. We were able to combine the two to help people identify when somebody who was listed as missing was actually in a hospital bed. So we're used to working with banks and supply chain risk companies and logistics companies, but this was a lot more rewarding, if I'm honest, and made me realize that these data problems, they
affect corporations, but they also affect humanitarian programmes as well. And I was pretty pleased we were able to help with that.
Tobias MaceyYeah. It's definitely a very interesting way of using some of these technologies that are built and funded by corporate needs, but are generalizable enough that they can be applied to a lot more of these, as you said, more humanitarian efforts.
Steven RenwickExactly.
Tobias MaceyAnd as you have been working in this space and building TeloRes, what are the most interesting or unexpected or challenging lessons that you've learned personally while dealing with entity resolution at scale?
Steven RenwickI think people have no idea just how bad real data is when you're working at scale. I think many data scientists will have a sort of idealistic view of the data that they're expecting to be working with. And then they come to do a project and they realise that 80% of data science is data cleaning. And a lot of that data cleaning is basically entity resolution. So when we're doing POCs with customers, when we're basically testing their data to see if it'll work in Telarize,
we're constantly amazed by just how bad the data can be. And it's surprisingly simple things like you might end up with certain organisations always send data in a particular way. So we were working with one of the world's biggest logistics companies and they're getting data from all the big e commerce companies. And you would just be amazed at how bad the data is from these supposedly tech savvy e commerce companies. And they do these crazy random things with the data that you think, why would anyone do that?
One provider, one big provider will always merge the title into the surname. So why would they do that in the first place? And then how would you deal with that once they've done that? So we constantly have these crazy, crazy things, which just these crazy data scenarios that we would never have imagined. And luckily the software is pretty good at dealing with them, But, you know, there's always another kind of curveball waiting with us when we start working with a new customer.
Tobias MaceyAbsolutely. The other interesting piece of any particular technology or type of solution is that once you become well versed in it, then you start to think that it's applicable to everything. The whole... If you have a hammer, everything looks like a nail problem. So what are the cases where something that may seem as though entity resolution is the right approach may not actually be the right either way to solve the problem or not worth the time and investment required?
Steven RenwickYeah. That's a good question. So where you shouldn't be using us is, we quite often get people coming to us for like CRM deduplication. So it works, right? You can use TLS for that. But when someone wants to do CRM deduplication, they're usually coming to you thinking like, Oh, maybe that's 50 or $100 a month. You know, they're expecting like this HubSpot level pricing. There are tools which are built for CRM deduplication.
You don't need a kind of super fancy large scale entity resolution system for that. If it becomes more complex and you're trying to connect up multiple internal data systems, sure. But for the simple kind of CRM stuff, no, that doesn't make sense. If you've got two datasets and you wanna see how many records in this dataset are also in this dataset, and it's like a one off, then no, that's not again something you should be investing the time in an entity resolution system. That sort of use case,
just tell you as an LLM for that, or I'll use Splink. Splink is great for that kind of really fast pairwise records linkage. But yeah, you don't need to be going into the investment of a full scale ER system for that.
Tobias MaceyWhat are some of the ways that you see the landscape changing or some of the capabilities evolving as people do start to bring on board more LLMs for entity extraction or invest more in entity resolution as a core capability. I'm just curious what are some of the evolutions of the ecosystem or any particular investments that you are looking to make to make this either more scalable, more cost efficient,
more accurate, etcetera?
Steven RenwickYeah. So the whole kind of use of LLMs for entity extraction means there's a lot more entity resolution that's needed just because the entity extraction, you know, you end up with these entities being described in different ways. So you inevitably have more entities from more documents that need to be matched. So effectively, entities that were locked away previously in unstructured documents are suddenly potentially accessible. I also think the growth in graph databases
is also very complementary to entity resolution. So arguably, what you should be doing is running an entity resolution engine as part of your pipeline before you put data into a graph database. So a so called entity resolved knowledge graph. There's no point in putting in a 100 records which refer to the same thing into a graph database, because then you'll end up with a whole bunch of edges which are not necessary. And that's one of the things which is gonna restrict your scaling. And a graph database
makes far more sense to make entity resolution part of the pipeline, resolve all the data in the entity resolution, then push a single entity, which is already kind of linked together into the knowledge graph. So we see that there's definitely a kind of a rising tide here, which is pushing up the knowledge graphs that's helping the entity resolution systems as well. At the same time, new, the large language models are just giving us much more ease with which we can do the entity extraction to
give us more entities to resolve.
Tobias MaceyAre there any other aspects of the work that you're doing at TeloRes, the overall space of entity resolution, or just some of the ways that peal... People should be thinking about whether and where and how to apply this type of technology that we didn't discuss yet that you'd like to cover before we close out the show, or just any parting thoughts or parting advice that you'd like to leave people with?
Steven RenwickI mean, is gonna... This always sounds a little bit salesy, but as commercial entity resolution company, our biggest competitor is always build versus buy. Entity resolution is a pretty interesting problem, right? People like to think it's intellectually challenging, the scaling is intellectually challenging, the whole kind of blocking and that sort of thing is very intellectually interesting. But the economics of build versus buy usually favor buy, right? Now, obviously I'm gonna say that I'm biased. I'm trying to sell the entity resolution system. But I would say
people need to really consider the long tail of things that they need to actually fix. Not just the matching, the matching is kind of the relatively easy part. You really need to think about everything else that's around it. So, you know, be cautious about where you tread. A lot of people have tried this before and run into problems. Far better to get someone that's dealt with the problems already. In terms of some of the new stuff, we've talked a lot about the large language models, but it's maybe also worth mentioning
some of the new kind of AI models that are coming out. And it's only like in the past, was it past two weeks that JEV has been released, which seems to be the hottest thing that I've seen at least across my social feeds. And, you know, JEV is taking a different approach to AI. It's not giving you a kind of a text based output, it's giving you a probabilistic output. It's much cheaper and it's much faster than running a large language model. So it's interesting to see whether that's something that can do entity resolution, whether we could use it. At the time of speaking, we've not managed to use JEV because there's such high demand on their services that we can't sign up anymore. So we've been working with LIA, which is the, I think, open source model that came out a few days after JEV was announced. And LIA is a bit cheaper,
possibly at the same sort of speeds. And we don't see it as being... We see those new models as potentially being interesting matchers in our system. So we use different matchers, which are usually based on certain algorithms, cosine similarity, Levenshtein distance, blah, blah, You could potentially use Liar or JEV as an individual matcher within an entity resolution engine. But from what we've seen already, it doesn't look like it would be something that would replace a proper entity resolution engine. We ran
some numbers and JEV is pretty cheap compared to large language models. But if you were to use JEV to do entity resolution on a 100,000,000 records, which is something we do quite regularly, it would cost about $800,000 which is a little bit out of the price range of most people doing entity resolution.
Tobias MaceyYeah. It's definitely interesting to see how the overall space of AI is starting to move beyond the transformer architecture. Diffusion models, I think, caught a little bit of attention for a little while, and I'm sure there's still a decent amount of investment there. But these decision models that people talk about world models, which is I think a very vague term. So it'll be curious to see how the various approaches start to move beyond the current vogue of transformer and attention based systems.
Steven RenwickYep, I mean, we're certainly always trying to keep an eye on it. We consider ourselves AI adjacent, so the more people are using AI, the more they need to have clean structured data, so that works nicely for us.
Tobias MaceyWell, for anybody who wants to get in touch with you and follow along with the work that you're doing, I'll have you add your preferred contact information to the show notes. And as the final question, I'd like to get your perspective on what you see as being the biggest gaps in the tooling or technology that's available for building data systems today.
Steven RenwickYeah, so one of the biggest gap we see is this trustworthy identity layer. There's always been this idea that if you have all your data in a data warehouse, then the data warehouse becomes your source of truth. But all you've really done is taken disparate data from multiple different places and put it into one place. It still doesn't become data that you can trust. That I think is a big gap.
It's obviously part of what we are trying to solve. So the next thing is then how do you measure how good your data is? Measuring the quality of data is pretty challenging. Measuring the precision and recall of data at scale when you're doing entity resolution is pretty challenging. And that's where we see some gaps that we want to be trying to help fill.
Tobias MaceyWell, thank you very much for taking the time today to join me and share the work that you've been doing on TeloRes and just the overall space of entity resolution and all of the complexities that come along with it. It's definitely a very interesting problem space and one that, as we've pointed out, is becoming increasingly necessary or at least increasingly visible. So I appreciate all the time and effort that you're putting into that, and I hope you enjoy the rest of your day.
Steven RenwickThank you, Tobias. It's been a good chat.
Tobias MaceyThank you for listening, and don't forget to check out our other shows. Podcast.net covers the Python language, its community, and the innovative ways it is being used. And the AI engineering podcast is your guide to the fast moving world of building AI systems. Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes. And if you've learned something or tried out a project from the show, then tell us about it. Email hosts at data engineering podcast dot com with your story. Just to help other people find the show, please leave a review on Apple Podcasts and tell your friends and coworkers.
Summary In this episode, I spoke with Keerti Kulkarni about the data and AI systems behind physical logistics automation at Amazon. She explained how her team supports dock-door operations across more than 100 North...
Summary In this episode, I spoke with Christian Bremeau and Simon Dynin about their decades-long work in metadata management and how that experience shaped the evolution of Meta Integration Technology and its...
Summary In this episode Christopher Doidge talks about his Agile Ledger Architecture (ALA) approach to data warehousing and how it aims to reduce data debt while shortening the path from raw data to trustworthy business...
Summary In this episode Soham Mazumdar, co-founder and CEO of Wisdom.ai, talks about what “context” really means in data engineering and AI systems. He explores why context has become such an overloaded term, spanning...
Summary In this episode Yetunde Dada discusses Otto, Astronomer’s AI agent for Airflow, and the broader challenge of making agentic tooling actually useful for data engineers. She explored why generic coding assistants...