Skip to content
PodSubs
ShowsPeopleGuestsJobsNewsEvents
More
BlogPodCred
Menu
ShowsPeopleGuestsJobsNewsEventsBlogPodCredNetworksSign in
Join nowSign in
  • Shows
  • People
  • Jobs
  • News
  • More
    • Guests
    • Events
    • Blog
    • PodCred
    • Networks
    Join nowSign in

Retailgentic | Consumer Behavior & Retail Trend

Retailgentic | Consumer Behavior & Retail Trend artwork

A Conversation with Luca Fiaschi of PyMC Labs: Synthetic Consumers & the Future of Product Testing

Dec 11, 202554 minAudio
ShareXLinkedInFacebookWhatsAppEmail

In this episode of Retailgentic, Scot sits down with Luca to unpack a research paper that blends academic rigor with real-world implications: “LLMs Reproduce Human Purchase Intent via Semantic Similarity licitation of Likert Ratings.” The paper, co-authored with Colgate-Palmolive researchers, explores whether AI can accurately simulate human reactions to product concepts, enough to replace or accelerate traditional consumer panels, which are slow, expensive, and hard to scale. This one goes deep, but in ways that any retail or AI leader should care about. Scot and Luca discuss: Colgate’s Challenge: How to test product concepts faster and at scale. Synthetic Consumers: AI models that react to products like human panels. Accuracy Breakthrough: Reaching ~73-74% agreement with real consumers. Fixing LLM Failure Modes: Why naive prompts don’t work, and what does. Bayesian Reasoning: Adding uncertainty so AI stops being confidently wrong. Smarter A/B Testing: Using AI to pre-screen ideas before running live experiments. Digital Clones: Future consumers earning money by sharing preference data safely. Simulated Populations: Matching real audiences for testing and predictions. AI isn’t just helping brands write copy or generate images, it’s beginning to think like their customers. If synthetic consumers continue to evolve at this pace, product development, A/B testing, and personalization may look completely different in just a few years. Timestamps: 03:00 - Luca’s background: Rocket Internet, HelloFresh, Lazada, Stitch Fix 08:00 - How PyMC Labs was founded & why Bayesian modeling matters 16:00 - Bayesian thinking explained in simple terms 19:00 - High-stakes decisions & why probabilistic reasoning matters 24:00 - Colgate’s challenge: testing product concepts at scale 26:00 - How synthetic consumer panels work 30:00 - Accuracy results: humans vs. AI (~73-74%) 32:00 - Why naive LLM prompting fails (“mode collapse”) 35:00 - How reasoning → scoring solves accuracy issues 38:00 - Example: synthetic consumers evaluating PyMC’s own website redesign 44:00 - How AI can pre-screen ideas for smarter A/B testing 48:00 - Where AI cannot replace causal testing 49:00 - Digital clones & monetizable consumer preferences 52:00 - Future benchmarks, new LLMs & evaluation methods 👉 Connect with Luca: https://www.linkedin.com/in/lfiaschi/ 👉 Learn more about PyMC Labs: https://www.pymc-labs.com 👉 Check out the paper: https://arxiv.org/pdf/2510.08338 🧠 Want to stay ahead in AI commerce? Subscribe and follow along: 📰 Subscribe to the free Substack: retailgentic.ai 📺 Watch episodes on YouTube & Subscribe for updates 🎧 Listen wherever you get your podcasts: Spotify, Apple, etc: https://retailgentic.transistor.fm #Retailgentic #AgenticCommerce #SyntheticConsumers #AIShopping #ProductTesting #CPGInnovation #GenerativeAI #BayesianModeling #Ecommerce #FutureOfRetail #DataScience #LLMs #AIResearch

Audio and video play straight from the show's own hosting. PodSubs stores and serves no media.

Transcript

From the show. Tap a time to play from there.

You know, I was one of the customers of PyMC Labs, uh, in, uh, uh, five to six years ago when, when they really started. And with EloFresh, we used them for one of our projects because we were using the open-source library PyMC internally in our, like, uh, the data science teams. And so I reached out to Thomas to help me with the practical implementation of a media mix model at that time. And so after that, we became friends, and I've always have been advising

him on how to grow the company. And at the beginning of the year, he asked me to join full-time as business partner and Welcome to the Retailgentic Podcast, where we explore the intersection of retail, e-commerce, and AI agents, which we call retailgentic. Our mission is simple. We are shining a bright light on the new agentic AI shopping megatrend, which will disrupt the trillion-dollar digital retail market. The stakes are high. Google, OpenAI, Microsoft, Perplexity, Amazon, Apple, Meta, and

Anthropic are all fighting to be the consumer AI shopping agent winner. There are gonna be some tough decisions for online brands and retailers to make. What happens to checkouts? How should payments work? Should you continue to invest in retail media networks? What does the world look like when over half of your traffic is not from humans, but from AI agents? These existential questions are the tip of the iceberg. On this podcast, we're talking to industry experts, analysts, and influencers to get answers to these questions, identify trends and best practices.

This is Retailgentic, your podcast partner in navigating our AI shopping agent future, and now here's your host, Scot Wingo. Welcome to this episode of Retailgentic. Some of our most popular shows are where we take a research paper that's centered around LLMs and/or agentic commerce that's a bit out there, kind of future thinking, because this is research, so it's not in production right now and not something we can use. But it's fun to think about.

We boil it down, and then we apply our digital retail lens and experience to this concept and see if, if it's interesting and what it means for the future of where this is all going. Uh, this is some of the ways that I, I get pretty good at predictions, is trying to, like, understand where, where academic is, uh, the academic world, because we're always a couple steps behind. In this series of interviews, we've had three guests kind of in this category. We've had Luca Chi- uh, Chian, uh, from UVA's Darden School of Business.

Uh, he talked about the psychology of agentic shopping. We had Amin from Columbia Custom AI, and he talked about simulated buying agents. And then we had Ronnie from ChatGPT/OpenAI on how people use ChatGPT. On today's show, I'm excited to welcome Luca Fiaschi. He is with a firm called PyMC Labs, which is a very big in the open source statistical world, and he co-wrote a paper called…

Let me pull it up for you to see. Uh, it is called LLMs reproduce human purchase intent via semantic similarity elicitation of Likert ratings. It's a mouthful. You can see his name is up there, uh, in the upper right. He's one of, uh, looks like nine different authors on this one. You'll also notice that some of the folks with attribution on here are from Colgate-Palmolive, um, which I thought was interesting. Um, what caught my attention about this paper, other than the Colgate part,

uh, was that it-- there's this f-very futuristic idea here that thanks to LLMs now and agents, um, you can create simulated humans that act close enough to real humans in how they think about products and how to buy them and what they like and don't like, that they could be… If this, if this paper be-becomes true, um, and if they prove their thesis, which we'll talk about, did they or did they not, um, then if they can, then that means we can use these simulated humans for things like potentially A/B

testing or product development in the future, or maybe a bunch of other things. Maybe you could show it some different ads, and it could tell you which ones it likes more. Um, uh, Luca has a really good example, uh, in the talk where they've used it kind of beyond what's in the paper that I thought was really interesting. So I, I learned a lot from this one. Um, I learned, uh, some new statistical things and different ways, weaknesses of LLMs, uh, and then how to counteract those, uh, and also about how this approach, uh, worked and in some ways doesn't work for certain things

that I thought maybe it could. So I hope you enjoy this as much as I do. Again, this is kind of gonna be like a two or three years out, we'll see things like this happening. Um, and, but I think it's interesting to keep track of this 'cause this world moves very quickly, and this is a little glimpse into the future. I hope you enjoy the conversation as much as I did. Luca Fiaschi, welcome to the Retail Gentic Podcast. Hello, Scott. Thank you for inviting me. Absolutely. Um, you put out a paper, uh, and that paper is one of my favorites.

It is called LLMs Reproduce Human Purchase Intent, which I totally understood, and then it gets a little, little kind of crazy for me, but we're gonna get into that. Uh, via semantic similarity, elicitation of Likert ratings. Um, so this is well thumbed through, so I, I really enjoyed the paper. And, uh, when I reached out, I was expecting an academic, but when I, uh, looked at your LinkedIn, I saw you've been, uh, in industry for quite a while. You, you do have a PhD of course, but you've, you're actually not an academic.

Uh, you, you know, I guess you're both an academic and you actually are, like, an operator, uh, you know, actual real, uh, internet person doing stuff. So excited. And you have a lot of retail experience, which was also exciting 'cause that's, that's our whole topic here is tr- figuring out the intersection of retail and agentic commerce, and I, I think, you know, you're sitting right at the, the middle of that. Um, so let's jump into your background. So, uh, let's start at education, uh, and then kinda go through some of your, your work history.

Sure. Thank you. And, you know, I've always been excited actually about combining the cutting edge of research with, uh, uh, applications in the industry. It's actually what really motivated me. And, you know, I started many years back, um, more than 15 years ago, uh, in Germany, where I got my PhD at the University of Heidelberg. At that time, uh, my PhD was an industry, uh, PhD. We were working with, uh, this pharma company, German pharma company called Bayer, in, uh, uh, basically developing a new type of microscopes

for, um, like, high throughput screens. And it, it was a PhD under Department of Physics, uh, but it was all computer science and machine learning. And I had the good fortune to hook up on this topic, which was, uh, uh, machine learning really early on, uh, really by meeting my doctoral father, uh, by chance during, uh, an exchange program. And, um, there I really picked up this idea of, uh, really working on, on the

bleeding edge of what research can do and transferring that really, really quickly into, uh, industry applications. And, you know, the academic world completely wasn't for me, so, like, when I got my PhD, I decided to move for the industry. I worked with a VC fund, uh, called Rocket Internet, which, uh, used to be, uh, one of the largest VC fund in Europe- Yeah. And that was these famous guys that would take, uh, it was two brothers or something, and they would take ideas from the US and they would, like, spin

up companies and incubate them there. So they, uh, they had many companies. And, and weren't they involved in Skype and just, like, all kinds of big internet companies in Europe? Yes. So I do think that they were, um, either investors or, like, they, they, uh, participated in the building of Skype. They started it and spun it out. Yeah. Yeah. So they were very famous because of, uh, um, their company was acquired by eBay, and then became eBay Europe at some point. So, um, the Samwer Brothers, um, uh, which, uh, I actually worked

quite closely with some of … We are one of the tour with Oliver. Um, they, they were, like, really pioneers at that time, and they had this strategy of, A, it's, uh, actually 30% about the ideas and more 70% about execution. So they would take these ideas, um, of the very early commerce days, and they would just, like, spin it up across the globe. And they usually worked in pretty difficult market where you have to do a lot of customizations. For example, this brought me to work with them on a venture which

then became part of Alibaba. Uh, the ventures was called Lazada, which was in Southeast Asia. And it's, it's … In, as an idea, it's a clone of Amazon. It's, uh, e-commerce. And they, uh … It was exciting as a very young, uh, you know, person at that time like me, because they would really literally tell you, "Hey, here you have hundreds of millions of dollars to play with," which comes with specific money, and go and conquer in like, uh, this new market.

And the internet penetration in those markets is actually pretty high, like in Southeast Asia, because these, uh, countries, they, um, they start digital first, and then, uh, you know, and it was, like, figuring out things like how do you do logistics and be super efficient with forecasting is actually pretty complicated, especially in these massive countries where you have very complicated, uh, logistics situations like shipping to islands and so on and so forth. So you get an idea that starts from the US, and then you add a lot of

complexity and execution on top, and that's why you actually needed to have very good data science across different domains, from operations to marketing optimization and so on, to make sure you succeed in, in those markets. And then, uh, the, uh, Rocket Internet gave me really the opportunity to work in many different business models and understand a lot of different problems that the enterprise have today, like from, uh, e-commerce retailers to food delivery systems, uh, to, uh, things like subscription systems and so on, so forth.

So it's just been really exciting throughout my career Yeah. And you worked at HelloFresh and Stitch Fix where there are two subscription businesses, and I imagine it's really interesting to think through the lifetime value of customers and churn modeling and, and I'm sure being a data scientist that you were-- you turned your attention to those things. Is that a fair guess? Uh, what I would-- what I'd say is that interestingly enough, HelloFresh is, uh, also a Rocket Internet company. It was, uh, actually created by Rocket Internet.

And, um, they, uh, that's why I say like I worked for like almost 10 years with Rocket Internet ventures, because it, it was actually, I started with HelloFresh at very early beginnings where it was 20 people team. Um, and, uh, uh, you know, mothership Rocket helping them develop, and then the founder asked me to build their data team in the US because the US is the largest market of HelloFresh. And so when I arrived, it was just one person. I left, it was 75 people with different sub-teams handling things from analytics,

experimentation, customer, uh, like understanding customer analytics is part of one of the things we did. Marketing analytics, to things like recommendation engine, forecasting, data infrastructure and platform and so on and so forth. So it was a, a very large scale set up at that time, uh, and, uh, one of the best in the industry, I would say, with that type of industry Cool. So it sounds like you were kinda like the go-to guy at Rocket Inter- Rocket Internet when they had a startup that was starting to get some momentum.

They're like, "Let's get Luca in here to run some analysis and build a team," to kinda like help solidify that, that business. Yeah, yeah. My role was, uh, um, basically to collect learnings and practices across the enterprise and across different batches of Rocket. I helped the founders build their data teams, uh, projects, mostly around growth. How do you, um, allocate budgets in marketings? How do you, um… We had our own ad tech infrastructure too, so I was partly responsible to build some of our ad tech infrastructure, our bidding engines, our marketing recommendation

systems, and so on, so forth. Going back to Heidelberg, uh, you don't seem to have a German accent, so, uh, where, where are you from before then? I'm or- originally Italian, from this area of Italy called Tuscany, and it's a beautiful area. I've lived there. I, I, unfortunately, I lived eight year in Germany, I never picked up German. As Mark Twain said, "Life is too short." We actually had another Luca on the podcast from Italy that, uh, is in the agentic world as well, so it's interesting. He's at the Darden University. Luca, Luca, uh, Cha- Chaian.

I don't know if you know him or not. I'll have to introduce you. You'll have a Luca party. Um, then, uh, and, and today, you know, uh, for this paper, you're at a company, and let's, let's see if I say this right. Pi MC? Is that how you say it? PyMC Labs. Correct. PyMC, PyMC Labs, yep, okay. Yeah, so, so tell us about that. You know, you're, you're one of the… Let's see, what are you? You're one of the founders, right? No, I'm, I'm a partner. Yeah. Oh. So the company was founded by- A partner. Okay, partners. Yeah. Yeah. So, but the company was, uh, uh, founded by, uh, my business partners,

Thomas Wiecki, about five years ago. Um, so Thomas is very visionary person, so his original idea is to build a company around, uh, the open source library that he developed that's called, uh, PyMC, and, uh, basically find a place for, uh, uh, academics or researcher, uh, that work with him to actually make money out of the work with it. And, um, you know, I was one of the customers of PyMC Labs, uh,

in, uh, five to six years ago when, when they really started. And with EloFresh, we used them for one of our projects because we were using the open source library PyMC internally in our, like, uh, the data science teams. And so I reached out to Thomas to help me with the practical implementation of a media mix model at that time. And so after that, we became friends and I've always been advising him on how to grow the company. And at the beginning of the year, he asked me to join full-time as business partner

and, um, we have complementary skill set. I'm more on the machine learning engineering side of the house, whereas, uh, like a great statistician. And so he comes from the Bayesian angle, I comes from the agents generative AI world of the house, and we have combined the two ideas together in several different projects. And what… The way we work with clients is a typical consulting approach. So even the, um, um, paper that we are gonna discuss later on came

from an engagement we had with one of our clients, which is Colgate. And, uh, you know, we developed these systems with them, which marries some ideas from Bayesian statistics with actual generative AI eng- agents techniques Yeah. So to help listeners, um, yeah, so, so tell us about what the PyMC open source library does. And, um, you know, I know you're gonna talk about Bayesian, so, so when you do that, like, let's, let's define, you know, uh, for, for layman's terms,

what a, what a Bayesian, um, model is. Yeah. So Bay- Bayesian it's, uh, it's very good exercise also for me because I, I traditionally I'm not, I'm not actually a Bayesian person myself. I, I actually came across the topics. Yeah. I have a feeling you have a good way to summarize it ver- If I asked a Bayesian person, they probably couldn't make it short, but I'm sure you can. Yeah. That's right. That's exactly right. So I would define Bayesian as a way of thinking about probability of statistics

that comes from first principles. So you start defining, uh, probability distributions about your beliefs, and then you define a model that basically represent the world. And out of this model, then you can use to s- sample predictions. Um, for example, this could be a forecasting models about what will happen in the future for, uh, the revenue of your companies. And then you can get, um… You can encode belief about, uh, the markets, about what competitors are

gonna do, and then you can get out probability distributions on what, uh, the world will look like, yeah, in the future and how the companies are gonna perform with confidence intervals. And so it, it's something that it's a little bit different than traditional machine learning engineering type of the house, a way, way of thinking. As machine learning engineers, you think always about the most accurate model, and you don't care what the model specification is really, as, uh, much as it performs on a test set very accurately.

In a Bayesian world, you actually want to think about the structure of the model because you want something that's interpretable, and you don't want the model just to be accurate, but you want the probability dis- uh, that it is outputting to be, uh, as tight as possible and to give you credible intervals that are both, um, representative of the coverage and tightness, which means that they are very, um- It, it represents your uncertainty about the world in the future and adapt to it. So

Got it. Okay. So when I see, uh, I see all these surveys, right? So there's these stated preference surveys where people are like 80% of, uh, you know, the survey respondents are using AI for, for buying things online. And if you look in the small print, it'll say N equals 4,000. So they've taken a… They've taken that and then applied a Bayesian framework to scale it up to the population of the US. Is that kind of like, is that Bayesian or not Bayesian? It, it depends. So it's a way, it's a way, it's a way of thinking, uh, it's a way of thinking

that allows you to generalize things as, as statistics, as statistics. You know, in … Th- there is two world in the statistics. There is the frequentist approach where I have a sample set, and then I extrapolate everything from a finite sample. The Bayesian way of the world, it's a little bit different. So they … You think about an abstract model of how the world will looks like, and then your data is just evidence that supports or falsifies the model. And, and therefore, you use the data to reconcile, uh, what

the world is gonna look like. So i- in, in more like technical terms, in the Baye- in the, um, like traditional statistics, frequentist one, you have P values, while in the Bayesian world, you have confidence intervals and probabilities, actually credible intervals and probabilities that allows you to define in a more nuanced way what different alternatives of the world will look like. Now, that has a lot of implications from the bus- from business terms, actually. When I, when I try to explain in abs- abstract, it sounds really weird, but

w- like in, uh, uh … It's, it's a technique that's used very much when you need to take a, a high stake decision. For example, uh, some of our clients are hedge funds. They do derivatives pricings, and they have billion dollars decisions to make based on that. So knowing the probability of something to happen, uh, is extremely important to them because they can value the level of risk of specificity. And so that's why we charge Okay, cool. And what's your role there at PyMC?

So I'm, I'm building the generative AI vertical, and the idea is, uh, Bayesian statistics is a bit of a Swiss Army knife for certain set of business problems. It's a technique that's very hard actually to, um, use in practice. So what we do is that, um, because the, the language is complicated and even the way that I try to explain it, simplifying it today, you can understand it's very complicated a technique to actually explain. While LLMs are universal translators, and so what we can do is that we

can embed some of these Bayesian models, uh, into LLMs to help them make decisions in a principled way. And, and these are, there's two benefits. So LLMs are basically general, uh, translators, so they can actually help you build these complex Bayesian models similar to the way they help you code things in software engineering. Uh, but with, uh, an understanding of the causal structure of your data. It's a principled way of thinking about the world.

And the other things is that LLMs can be, uh, often very, uh, confident about, uh, things, but be very wrong. And so Bayesian techniques allows you to, um, have LLMs reasons about uncertainty in a principled way. Uh, so like, uh, correcting for some of the biases that these LLMs have. And within this world, um, you know, uh, we also think about ourself in a more abstract way as a expert in making simulations, very

complex simulations, even while you cannot explicitly build the model. And so the papers we built is an example of a simulations in a way that you can do with AI, where you don't have an explicit model of human behavior. It would be too complex to actually build something like this. So you rely on LLM to learn from a, uh, training data what the heuristics of human decision-making are. And, and that's the way you are actually, um, physically you can do that.

You can build very-- these very complex simulations Okay, cool. So, so you guys have clients that come to you and say, "I'd, I, I'd really like to… You know, here's my business problem." And that could be a hedge fund. Colgate is obviously a brand. And, and then it's your job to figure out in this toolkit that your company offers, which range from these Bayesian, you know, statistical models that are very advanced. You've brought in kind of LLMs and AI as another tool in the toolbox to kind of help, help figure out the right way to approach this. C- Correct. So, and, uh, a, a lot of clients come for, from, because they've

seen our open source tools. We are very much, uh, you know, contributor to different open source libraries for different types of applications. And then they get curious about that. They read our white papers, they come on our website, and then we start collaborations with them. Yeah. Red Hat is who where I am, and that's kind of… They, they, they took the Linux distribution and added an enterprise layer, uh, to it, and that's where, you know. And then they have a licensing model on top of that, that, that kind of beefs it up and makes it more enterprise ready. And then they have-- But, like, the bulk of what they do is also support

and consulting around, around. So it's a, it's a way to monetize open source that's pretty clever. Yeah. Um, so let's talk about the paper. So what was the problem that Colgate came to you… Colgate-Palmolive, uh, came to you and said, "Hey, we, we need help figuring out this problem"? Yeah. So it's a very big problem for them and, uh, it… they invest tons of, uh, money every year. So they're talking about, you know, hundreds of millions of dollars, which is product development. And so the-- typically, the way that, that it's done is that when they

think about the new concept for a new product, before they go and produce that product, um, they need to-- they, they wanna evaluate whether it's gonna resonate to a certain target audience. There are a lot of different considerations that goes into this in terms of even, like, is-- can we produce it at the right costs? Um, what's the price point for these products? And so on and so forth. And so these are very complex decisions and workflows, and the idea was take parts of these very complex decisions and workflows and try to, um, automate them.

And one of the most time-consuming one is really to do these consumer panels where they give these products to, um, to real humans and they, they ask, uh, questions like, "Hey, would you buy this? How unique you think it is? What do you think when you-- about this new product compared to competitive product? At what price points would you buy it?" So these type of questions, um, they're typically, uh, conducted with re-research of, like, 30 to 40 humans.

Um, assembling these panels is time-consuming. Uh, running these interviews is expensive. It can cost you quite a bit amount of money to work with some of the consultancies. And right now, like even where they're using even a generative AI technique to produce new product concepts, it just doesn't scale anymore to run all these type of interviews. And so what Colgate wanted is a way to screen new product ideas really, really fast to prioritize them offline so that they can focus only on the, I guess, value

one, and only those run with maybe humans or just really bring it to productions. And so that's what we did. So we built basically a, a synthetic consumer panel that is able to answer, uh, questions about these product concepts in a way that's very similar to what a real human, uh, panel could have answered about that product Yeah. Uh, let me-- I was gonna pull it up, but I'll just show it on the screen. Um, so this was an example from the paper where you had a, a picture of a

product, um, and you know, the, this is kind of the, the kind of thing they would show people and say, "Here's a product. What do you think about it?" Um, and, uh, so for those that listen to the audio, it's called Orofoam Mood Infused Body Wash, and then it has, like, a little description, a picture, a product picture, and then four or five bullets talking about the product. Um, so the, their problem is that, uh, they spend all this money, you

know, hundreds of thousands of dollars getting, putting these things in front of people, and I guess their question is, "Can we do this with a, you know, with a simulated human?" Yeah. Or they just said, they said to you, "We need to figure out a cheaper way to do this," and you said, "Let's simulate humans." Like how, how did that, how did that kinda go? Um, it-- That's a very good question. So they, they're… Colgate is actually a bit of the forefront, uh, compared to other, uh, companies in doing this. Actually, they, um, had started already pilots with, uh, other, um,

firms on developing this technology. They also had an in- an internal team working on something like this. However, um, they, they had two problems. One problem is that the results that they, they were getting weren't that great, and they weren't really sure, um, why and what determines, what are the factors that determines, like, uh, a good outcome. Is it the type of LLMs? Is it the way you prompt them? Um, so there are different parameters in these algorithms that you can actually,

uh, tweak to, to make it actually perform. And the second thing is they weren't sure is that how even do you measure accuracy of something like this? What does it mean accurate in itself? And so when they came to us, they came with the research questions of, "Hey, can you help us figure it out within this problem domain? A technique that's actually good and actually works so that even within our enterprise, we can convince all the different business partners to invest more in this technology and then productize it to become part of, uh, our workflows."

And, and that's the reason why we came up with a, a research type of engagement with them, which then became an actual published paper. Um, the paper is, uh, it's gonna be submitted to the conference. It's currently in archive for everyone to check. And we also open-sourced part of the code base that we used to really make sure that people can reproduce these type of results, uh, by themselves. If I was gonna try to run this experiment, the baseline is humans, right? So, so we're trying to do simul- si- you know, simulations versus humans.

Is that kinda how you set up the experiment? You kinda said, "All right. Here's how humans do on these types of things, and, you know, we're gonna, we're gonna try to get as close to that as possible." Um, yeah. Is that, is that kind of the setup of the experiment? Yeah. That's right. So the question is how, um, how do you measure accuracy? And the idea is that Colgate has pretty large datasets of past consumer studies because they've been running this business for many, many years. And we use, uh, basically a dataset which I believe, uh- I don't remember the exact numbers, but over 8,000 unique users, uh, and, uh, hundreds of products.

And, uh, basically the idea for us is if we want the, um, LLMs to be accurate. So accurate means that if they have a new product, they would rank this product, um, similarly to what human… You would, would rank this product so that you can actually prioritize them on a scale. This is our high priority versus low priorities. And, and then if you ask them to rate these products on a scale from one to five, the distribution of the ratings of the LLMs look similar to the

distributions of, uh, the human panel. And notice that there is a nuance here that if you ask two panels of people, uh, to rate these same products, and these two panels are extracted exactly with the same demographics and, uh, you know, they are almost identical twins, they would anyway rate the products not 100% the same, but they will still correlate, uh, like with a certain correlation fact. Uh, but so even nuance, they have an intrinsic variabilities.

And so in the way we had to evaluate this was taking into account the variability from our clients and the variability of LLMs in providing the answers to make sure that the results are aligned as possible. And if we know that with, uh, these historical data, we can reproduce the results of humans, we have confidence that, of course, this can potentially generalize to new products, uh, and to future situations. You can never have 100% confidence in science. You can always falsify things, but that's the way you actually do is you infer

with data that you have, and that gives you confidence about future predictions. Cool. So that, that helps me understand the setup. Let's just go right to the punchline. Did you get close to this or you couldn't make it work? Yes, we, we could, we could prove that we could make it work. Um, we actually proved that it, uh, works. You know, we, uh, I do think the main insight of the, uh, paper is if you do the naive things, uh, of prompting an LLM as a human, you get

okay-ish results but not good results. Uh, and, uh, I think the insights of the paper is how can you get something that's better, and why the, if you prompt an LLM naively to answer as a human, it actually doesn't work. And so that insights is what kind of like help us, uh, and we discover by using the paper, and we can get into it a little bit if you like. Is there a, uh… So let's kind of talk about, so humans are here. Let's, let's say they're at 100.

Did you get halfway to human or like where did you get on the scale? And, and it's fine if you reject the way I'm, I'm d- I'm framing this. Yeah, yeah. Yeah, and so humans in itself are not at 100 because if you get two humans partners that are actually identical, they don't correlate with each other 100%. They are something around 90%, uh, like correlations. And- Okay. So the high water mark is 90%. Okay. The high water mark is 90%. And, uh, um, the … I believe our latest results published in the paper

is something like 73, 74%, which is, uh, n- close to 90% of that 90%. Is that enough where, uh, Colgate said you solved this, or did they feel like that it still wasn't close enough? Yeah. So I … It … The … To them, uh, the, the most important, um, outcome of this project was to get the confidence that they needed to convince, uh, their business stakeholders and counterparts that that's a technique that they can invest on to automate their internal processes.

And, uh, their short answer is yes, they continued investing into the platform, um, which is proprietary to them that they use to prioritizing products. So that's based on this technique and, uh, evolutions of these techniques. We … And we are, you know, still working with them. They, they continue to develop this algorithm and these ideas, uh, today. So. So you talked about naively asking the LLMs, um, i- is not the way to go. And why is that? Like, the LLMs have been trained on all of human knowledge, shouldn't

they just be basically people? That's the … I'll play the role of the naive, uh, the naive person in this and explain to me why, why that's not gonna work. Yeah. And so you, you get the, you get the thing that's called mode collapse. Uh, so basically if you ask the LLM, um, like, "Give me a rating on this product," and describe you the product, and you tell the LLM, "Answer as a person who lives in Brooklyn, uh, that's a female between this and that age." Or sorry, "Of this age, and she makes around this bracket of income,"

and so on and so forth, they tend to answer with the most One single value, which is the most probable value in, they've seen in the dataset. And the reason why they do that is that they haven't spent enough thinking tokens into reason about what makes that product a good fit for that person. So one simple thing you can do is actually to ask instead of asking, uh, just the rating number to, um, give you a reason for it, and you should ask for the reason before you ask the number so that the thinking token are spent upfront

to come up with, uh, uh, a good answer. And the, uh, other thing that you can do is to ask, uh, for just the reason and then have a technique to remap that reason onto the scale one to five that gives you a soft wording, voting score instead of out voting score. Because the problem you get is this mode collapse. Otherwise, the response of the LLMs will be very peaked. They will always answer three or they will always answer five, and you don't get the variability that you would get in a normal, you know, uh, distribution.

Can you, can you tell-- can you give the LLM more context? Like I've seen people kind of say they'll, they'll take… So I've seen retailers do this where they'll know that they have, and I'll just like, you know, they'll have like, let's say they have a really simple demographic, like one third soccer moms and one third, you know, tech bros and one third, uh, I don't know, whatever. And they'll kind of say, "You are a soccer mom. You like these things and you're very busy and, and…" Will that influence the LLM enough or no? It's still, it still will go home to the most statistical probable thing.

It, it does, uh, but it will still go home to the most probable things for that type of subset of prompts. And- So that's, that's not gonna work. Yeah. Yes. And, and the reason is, again, it's about the thinking tokens. So like LLMs, the output of LLMs, it depends on two things. Depends on your prompt that you give them, but also depends on how much time they spent reasoning about the answer. And it's the same reason why reasoning LLMs actually do work today, is because they spend significant amount of time considering different points of view on

the answer before they actually give it. So if you just ask for one answer, yes or no, it would probably give not thinking about this enough. And if you ask for to think about the answer and give you an extended one, then you get these variability. So I understand why LLMs won't work. How did you build a better synthetic, uh, representation of how people react? Like what, what was the unlock, uh, to do that? And does it involve Likerts? Well, let, let me ask what a Likert is.

Like, that was new to me So a L- Likert's, uh, s- scale is, uh, a scale that's used in, uh, um, basically consumer surveys. And this simply means that to the question you can answer on a scale from one to five. And, uh, instead of like, uh, um, giving you like, uh, you know, uh, what's supposed to scale from one to 10 or it's giving you a yes or no answer. So, uh, and, uh, the, um, and so you, you have a distribution of…

If you ask the panel about these questions, you have a distribution of responses. And so this distribution is typically a sh- a certain shape. And, um, the, the idea is that if you get the distributions of responses from a, uh, a synthetic consumer panel, you want the, the two histograms to look as, uh, similar as possible. And you can measure that in different ways. Um, the way we choose to measure this is to use, um, a, a scoring which is called the KS divergence.

And the KS divergence is simply a, a good score because it's normalized between zero and one. It tells you how much two histograms are far apart. And, uh, um, being normalized between zero and one, it's actually are easy to interpret. So that's the way we choose to, uh, to measure this. And so the, the technique that we developed, uh, instead of, uh, asking you for one single token and answers, again, ask, uh, the LLMs for, uh, an entire sentence of what they like about the products and what they don't like.

And if you do that, you can take that answers, uh, and then remap them on a score to one to five, uh, basically by clustering them in the embedding space. You can take a sentence and then you can send it back to ChatGPT, you can get an embedding of that sentence, which means a vector. And then you can look at the clusters in the embedding space of these answers called similar answers that have the similar reasoning will be, uh, close together. And then you can, uh, aggregate those, uh, answers instead of with the hard scoring,

you can use a soft scoring approach. So maybe something that a nuanced answer is something in between, give, uh, a score of, uh, 50% to a four and 30% to a five and 20% to a two. It's not that you have a hard score or you have just a five. And that way of aggregating responses, it's more representative of the nuanced world that, uh, humans think about. And so represent better the variability of humans in terms of this KS divergence

Okay. Let me just restate that to make sure I got it. The, um… So if you get a full sentence or, or more than, you know, a pretty good explanation from the LLM, that gives you enough sentiment in there that you can map it better onto, you know, a… You can, you can give it a better mapping and create more divergence of the responses that, that's closer, 90% to the 90% of humans. That, that's right. Actually, th- you, um… When we think that the similarity between this technique and sentiment analysis, I…

It's something that I haven't thought about, but it's very, uh, similar to sentiment analysis. So you basically get the sentence from the LLM, then you map that sentiment into this score of one to five, which is very similar to that technique there. If you watch real humans and, you know, you can imagine if you're just seeing them, you know, write one or five, it's not as good as them saying, "Oh, I really love that color." And you know. So that's gonna be very, that's a strong five versus maybe they would have scored it, yeah. Or, you know, it's okay. Like that, that's negative sentiment in there that you

only really get from a sentence. Sometimes you only get it from in- intonation of voice and body language. That's probably where that last 10% is, would be my guess. Uh, just kinda like those little things we get as being humans that, that are, you know, not in the written language. Um. Yeah. And, and then there's other advantage, by the way, as you point out too, that the hu- the LLMs or the humans, if they answer with an extended answer, they can tell you the reasons of why they like and don't like the product. Now, those reasons are valuable because then you can, uh, um,

basically analyze them and use them to improve the product Which is a buy, so it's, you get it for free usually Okay. So, so you've- And I, it's a funny story because, sorry, this I have to tell you because I got the results this morning. It's actually really funny. We are rebranding our website, and we are choosing two different variants of the colors of the website and the messaging that we're gonna use on our website. And so we did two things. We did an internal survey within our team. It's about 40-ish people. And then we use our synthetic consumer technology to evaluate

that, and the results are spot on. And both the complaints that our team had about some of the fonts of the new website being too small or some of the messaging not being great, uh, were exactly the same as our synthetic consumers' equivalent. And we also, in this case, instead of using a generic panels, we prompted the synthetic consumers to look like a group of PhD students and professors, which is more or less representative of our, like, uh, component.

And so they gave a very good answer. Does that freak you out? Does it sometimes… The statisticians are probably already there, like, uh, but sometimes it does freak me out that, like, we're not that far away from LLMs at the end of the day. Yeah. It did, it did freak me out a little bit. I wasn't expecting survey so close, Randall. Yeah. That's the fun part of this field we're in. It's kinda like you, you… I try not to think too much about it. Again, r- it'd kinda be a bummer Makes me wanna just go learn to play guitar or something. Do something very, very human that an LLM can't do.

Um, so, so that was one of my questions. So you've built this really interesting framework, and, uh, did I hear you say you've open sourced it? Is it out on a GitHub somewhere? Um, you, you get the code, uh, for the, uh, for the reproducing the results of the paper. Of course, there is part of the things we, we have to keep proprietary given the, uh, collaboration with Colgate, but they, they were really kind to let us open source most of it. Yeah. But you guys now have this black box of simulated humans, and you can have it… You looked at a, a consumer product.

Now you can have it look at your website. Um, and, uh, you know, presumably this is pretty interesting because it is one thing to go and, you know, hire a firm to go do a $300,000 consumer study and, and all the analysis and everything. That's, that's the really high end of this type of consumer research. In my digital world, we're frequently doing A/B test, right? So we'll have a, you know, an audience of 5% will get A and another five will get B, and then we look at some metric like conversion rate, and then we'll…

There'll, there'll be a, a hero and a loser, and then the hero takes over the last 90% or whatever it is. Um, so now we can actually… But in that example, you, you kind of messed up the people that, the humans that were the 5% that got the crappy, the one, the one that didn't win, right? The loser. Um, so what's interesting is there could be a world where you run it in a, in a… Also, it takes time, right? So this may be in the B2B context or even consumer, sometimes it may take a week or two to like figure out your A/B split. So you guys do it, uh, you could do it simulated instantaneously because you

can always scale up a simulation, right? Uh, and then because it's j- you can di- you can scale up a digital simulation. Um, and then you also don't have the problem of the testing always has a, a set of real possible customers that, that didn't see the best thing. Um, so it also eliminates that, whatever the cost is of those two things. Um, you know, so that's, that's pretty interesting. Have you guys thought about commercializing that or? Yeah. There, there are several companies that approached us with the idea

of doing exactly that, and we're working with a few of them. We also have, um, uh, uh, implementations that we may wanna release, um, a- about these. So we are considering releasing. Uh, probably it's gonna happen actually next week or so. So the, the, the idea is, um, it's fascinating- I do think that, um… So there is two things. Uh, there is a-- There are some caveats. If you want to, um, AB tests will ever remain the, the gold standard

for proving causality and the-- this technique of simulations, the best thing it can do, it can help you prioritize, uh, um, like assets for AB testing and the, uh, which is not a small thing. So if you think about the difference between the old world and the new world, is that with generative AIs, you can create thousands, millions of assets, actually pretty, pretty quickly. And so you can't AB test possibly all of them. Plus, as you said, you have constraints, it's expensive.

So even just prioritize a small subset of those, I think it's already extremely interesting. Now, can you, uh, reconstruct the exact effect size in the simulations that you would have when you run the AB test, meaning the delta, the incremental revenue or whatever? Can it be estimated exactly? That's probably not because, um, we already know that inferring causality from observational data, uh, there is tons of theory that

tells you that's super hard to do. But I do think that establishing these correlations and prioritizing out of a very large set of AB test, the top ten percent or five percent is actually probably possible to do, and then you would have to run the AB test downstream. So that's what we want to achieve. Now, there are difficulties because the question is, when you take an AB test and you expose your audience online to that AB test- The customer traffic that you collect is not distributed at random.

It's not generally the entire US populations. So even how you instrument your offline simulations to be representative of, um, the audience that you will see once you do the AB test online is complicated in itself. But actually here, we are thinking about using Bayesian method and techniques to up-sample and down-sample specifically, uh, subsets of the synthetic consumers to be as close as possible to, uh, the, uh, online population.

So there's a calibration step that's still we need to figure it out how to make it happen, I understand. Yeah. You know how, um, so, so, so a lot of the ad systems have a, uh, an audience lookalike function where you can upload your audience, and then they have enough data that they can create a, a, a map of, you know, that. They must be doing Bayesian stuff on their side for that. But yeah, um, within this black box piece, th- this simulation, there could be a way where you like have a map, you know, m- mimic my audience functionality down the road.

That's, that's a very good… That's a, that's a very interesting idea. Yeah. And the idea is that, um, how do you do that? We're still thinking about this, but how- If you're Facebook, you, you have all the data, right? That's, that's how they do it. Correct. And in- indeed there, there are startup that actually approached us. They had a very clever way of actually getting through this data. So, um, we, we are not under NDA with them, so I can, I can disclose this, but basically they're thinking about gamified systems where you, um, maybe as a consumer, you link your personal data, you get something reward out of

it, and then this personal data is used to create a synthetic representation of you that the brand can interface with to get recommend- uh, to recommend you personalized products and to ask you questions about new products. And so you will, in the future, I think you will have a world where I can, uh, get some money by sharing my personal data and have, have a digital avatar of myself interact with any brand in the world. And the brand can ask me in a privacy compliant, safe way, questions about my

preferences, my consumer preferences. I get money out of that, and I get great recommendations, which is the best of both worlds. Huh, that's crazy. So you could create a digital clone and have it go make money for you? I like that. Sign me up. I'm, I'm, uh… Sign me up to the beta. Sounds like I could make money while I sleep, which is always good. Um, cool. I wanna be… I, I know you're busy. Um, the… All right, so we talked about this. It sounds like you've got some cool stuff coming up next week.

Anything else about the paper you wanna share before we wrap up? Um, I, I would say that, uh, y- yeah, I mean, read, read the paper. So we still, again, we got submitted to, uh, one of the big conferences next. Uh, check the code, reach out to us. I mean, if you are interested in these topics, uh, um, we are always happy to understand new use cases of s- some of these technologies and keep developing these technologies. So reach out and, uh, share your ideas, thoughts, and we can probably

work on these problems together Cool. And if, uh, people want to find you online, is it, um, do you have a place you- so the website is, uh, pymc-labs.com, and I'll, I'll put that in the show notes. Um, so that's the company website. Um, but you look like you're pretty active on LinkedIn. Is that a good place to find you? Yeah, yeah. Feel, feel, feel, uh, uh, free to reach out and DM me on LinkedIn. I'll very responsive there. Yep, yep. And then, um, cool.

And that's, so when you post new papers, they'll show up kind of on your LinkedIn, and I think it was on, on your website in a research area, if I recall. You guys have a nice little library of different research you've done there as well. Correct. Uh, correct. We have a pretty extensive research blog, and we publish continuously any content or material. Uh, we have benchmarks as well, so we, uh, we publish the benchmark about LLMs and pricing. So which LLMs are better at predicting pricing, which LLMs are most strategic about pricing competitions, you know, like the Price Is Right and so on.

We have a little, uh, simulation of the setup of, for The Price Is Right, the TV show. And we, um, as new models come up, we make them compete into simulations to see which one is, we make it a leaderboard. So, so Price Is Right has, like, six different games. Are you just doing the closest without going over the basic Price Is Right or like the little, the Himalayan guy or like wh- which, which one of the- It's a, it's a good question. Actually, I should, I should ask Alan who has actually implemented this. I, um- It's probably closest without going over. That- I, I think it's- … that's what most people- I think it's the basic one

that they have implemented so far. But the, I, I mean, like I, I thought it was a very fun exercise. I think benchmark is generally a big topics in, uh, uh, LLMs. I do think if we want LLMs to become better representing humans, uh, then we also need benchmarks for something like this. And publishing this Price Is Right was a way of helping researcher of, uh, traditional models developing this type of competition. Yeah. Very relevant to our world of, a lot of, lot of people in retail are using,

thinking about using these for different pricing, you know, systems of, you know, how do you, how do you do dynamic pricing? How do you think about, uh, regional pricing? You know, like a sweater in, uh, Los Angeles may be worth less than one in New York or something like that. Interesting. So, uh, so when a new model comes out, like, y- you guys run it through your little, your battery of different things and… Okay. So Gemini 3.0 is rumored to be coming out this week, so that's gonna be a big week for you guys. You, you're, you ready? Gonna, gonna run it through the paces?

Yeah, we're gonna do that. We still need to run on GPT-5.1. We, we're gonna publish the results soon. So yeah, a lot of exciting stuff coming up. Yeah. This is what's fun about this LLM world, a lot of job security. Everything's changing basically every week, all the time. So, awesome. Luca, thanks for taking the time to come and, uh, uh, answer my novice questions about Likert's and, uh, your paper. I really enjoyed it, and look forward to my digital, uh,

clone making some extra money. For sure. Thank you very much, Scott, for inviting me. Have a good day. Bye-bye. Follow, rate, and share to stay up to date with Retailgenic, your guide to the future of AI shopping agents.

Open the transcript file

Hosts and Guests

Hosts

  • Scot Wingo Host Feed-stated

Guests

  • Luca Fiaschi Guest Feed-stated

Credit the people on this episode

Be the first to rate this episode

More episodes

All episodes
  1. "Every Ad Will Be an Agent," InMobi & Glance Founder & CEO Naveen Tewari

    Nov 16, 202647 minAudio

    This week we go international. Scot is joined live from India by the founder of one of the largest agentic shopping platforms you may not have heard of yet. This one is part founder story and part product deep dive: how...

    Credit the people on this episode

  2. How Brands are Leveraging AI in the Design Phase: Raspberry AI Founder & CEO Cheryl Liu

    Nov 12, 202645 minAudio

    About every six months on Retailgentic, we step away from the front end of agentic commerce to look at how AI is changing a different part of the retail stack. This week it’s product creation. Raspberry AI helps fashion...

    Credit the people on this episode

  3. AI Meets Fashion: Co-Founder & Chief Brands Officer Lisa Yamner on Daydream’s Big Bet

    Nov 5, 202648 minAudio

    When you think about vertical shopping agents, the AI agents built to solve shopping in one category, one name keeps coming up: Daydream. We’d wanted to get the Daydream team on Retailgentic for a while, and this week...

    Credit the people on this episode

  4. Shoppers Take the Agent’s Top Pick: Growth Memo Author Kevin Indig on How People Really Shop in ChatGPT and AI Mode

    Oct 29, 202645 minAudio

    This week is a research-heavy episode with a conversation of one of the sharpest observers of how real people search and shop now that AI sits between them and the products they want. Kevin’s user-behavior research...

    Credit the people on this episode

  5. The Missing Link Is Shopper Context: eComID CEO Oscar Rundqvist; Plus eComID’s Big Funding Announcement 📣

    Oct 22, 202644 minAudio

    This week is a product deep dive with a fresh funding story attached. eComID just closed Europe’s largest fashion tech seed round, and the company is building something agentic commerce needs: shopper context that...

    Credit the people on this episode