Data Engineering Podcast

Data Engineering Podcast artwork

How Amazon Uses Telemetry and AI to Automate Physical Logistics

50 minAudio

Summary In this episode, I spoke with Keerti Kulkarni about the data and AI systems behind physical logistics automation at Amazon. She explained how her team supports dock-door operations across more than 100 North American facilities by turning telemetry from physical workflows into actionable operational intelligence. We discussed the end-to-end data path from CloudWatch, Firehose, and S3 into Redshift and AI-powered analysis, along with the realities of handling petabyte-scale event streams, late and duplicate data, reconciliation, cost controls, and the need to treat physical telemetry as a first-class signal rather than noisy exhaust. Kirti also shared how these systems are designed to protect worker privacy while still surfacing the right information for safety and operational decisions. We also dug into the role of LLMs and agentic workflows in helping operations teams diagnose failures quickly through conversational interfaces. Kirti described how AI-generated summaries help identify root causes such as authorization problems, hardware issues, or connectivity failures, while human feedback loops, structured outputs, and strict vocabulary controls help manage hallucination risk. Beyond troubleshooting, we explored how the platform is evolving toward proactive maintenance, training insights, per-site cost attribution, and broader modular agent architectures that could be shared across logistics teams. It’s a detailed look at what it takes to combine data engineering, real-time operations, and AI in a large-scale physical environment. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Today’s episode is sponsored by Parallel - where agents find answers. Most engineers today closely follow new model releases, but don’t pay attention to their agent’s most important tool: web search. Parallel develops enterprise-grade infrastructure for agents to retrieve high-quality context from the web. Their core products are a suite of APIs for retrieving high quality information from the web with Pareto-optimal quality, cost, and speed. Whether you work on voice agents that need 200 millisecond latency, chat bots that balance speed, depth, and quality, or long-horizon agents to do thorough, overnight research for you, Parallel is a single platform for all your agentic research. Get started for free at dataengineeringpodcast.com/parallel Your host is Tobias Macey and today I'm interviewing Keerti Kulkarni about scaling insights for physical logistics automation with AI-enabled intelligence Interview Introduction How did you get involved in the area of data management? Can you start by unpacking what "physical logistics automation is"? What are the technical and organizational systems involved in the production and consumption of that data? What are some of the ways that inaccuracies or increased latency manifest in real-world impact? You recently worked on a substantial system for managing this data. Can you describe the challenges and limitations of the system that you started from? What was your process for identifying the weak points of the system and how to evolve it? What is the design of the system that you have running today? What are some of the design choices that had the biggest impact on the final result? What are the most complicated aspects of managing cost attribution to the different physical locations, data flows, and model usage? What is the role of AI in providing insights to the organizational users? How do you measure and manage accuracy of the models as the system evolves? What are the most interesting, innovative, or unexpected ways that you have seen your system used? What are the most interesting, unexpected, or challenging lessons that you have learned while working on physical logistics automation? What do you have planned for the future of your platform? Contact Info LinkedIn Parting Question From your perspective, what is the biggest gap in the tooling or technology for data management today? Links Amazon Key AWS Cloudwatch Analytics AWS Firehose AWS Kinesis AWS Quick OSHA == Occupational Safety and Health Administration OmniGraph Podcast Episode The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA

Audio and video play straight from the show's own hosting. PodSubs stores and serves no media.

Transcript

From the show. Tap a time to play from there.

Tobias MaceyHello, and welcome to the Data Engineering Podcast, the show about modern data management. Today's episode is sponsored by Parallel, where agents find answers. Most engineers today closely follow new model releases, but don't pay attention to their agent's most important tool, web search. Parallel develops enterprise grade infrastructure for agents to retrieve high quality context from the web. Their core products are a suite of APIs for retrieving high quality information from the web with Pareto optimal quality, cost, and speed. Whether you work on voice agents that need two hundred millisecond latency, chatbots that balance speed, depth, and quality, or long horizon agents to do thorough overnight research for you, Parallel is a single platform for all of your agentic research.

Get started for free today at dataengineeringpodcast.com/parallel. Your host is Tobias Macey, and today I'm interviewing Keerti Kulkarni about how she is working on physical logistics automation at AWS and some of the work that she's done on managing that at scale. So, Kirti, can you start by introducing yourself?

Keerti KulkarniYeah. Absolutely. Excited to be here. Thanks for giving me the opportunity. I lead the technical program for AI enabled operations automation at Amazon Key, a team in Sunnyvale. My day job is building the telemetry data and the AI layer that runs the physical logistics automation. This is a system that handles loading and loading stop across all of the North American logistics facilities owned by Amazon. And we handle about 10 k docs today at 100 plus facilities, and this has led to 77%

adoption of the automated workflow as of March 2026. So what I do specifically is build the data plumbing, which lets the physical operations be automated and run themselves. And I do it at the Sims where the telemetry warehouses and now AI have to actually cooperate. So I would like to give also a little bit background of how I got here and what inspired that journey. I spent about fifteen plus years as a TPM, technical program managers, working at the crossroads of data engineering,

basically distributed systems, and physical operations. Before physical automation, I led the BI platform at Amazon Key. I separated the KeyRedshift environment from the Ring's shared cluster, migrated in about six months with only two BI engineers at that time and no dedicated data engineers. So with this effort, we managed to cut our dashboard failures by 68%. And today, this environment supports about 20 plus BI and DE engineers and still

is performing at scale. Before Amazon, I led a 18,500,000 safety assessment program for Genentech. And before that, I was in Watson Health and Watson Drug Discovery as a TPM in IBM for four years and earlier in the... In IBM, but in a different team, which was the Asia Pacific master data management practice for about six years with building an eight person team across the continent. So that's my journey in data so far. And you mentioned that you're focused on physical logistics automation

Tobias Maceyand the ways that that is manifest in terms of Amazon's distribution facilities. And I'm wondering if you can just unpack a little bit more of what that term means and some of the technical and organizational systems that feed into it.

Keerti KulkarniAbsolutely. When I'm talking about physical automation, or physical logistics automation, it is usually paired with the software or warehouse management systems that basically tells the machines in the warehouses what to do. But the automation itself is the physical execution layer. That is the robots or the conveyor belts actually moving things, not just the data. So physical automation in Amazon is how we handle the workflow of basically the vehicles coming in to the warehouses. They getting secured

in the docks, then doing all of the processes which actually make it locked and safe for the loading and loading associates to go inside and start that workflow where the bell starts moving and the goods start coming in or going out. So at... On the surface, it seems very simple, but there are multiple steps that make this more safe and more automated. So this is what my team specifically has been doing, but there are other teams handling

various other parts that moving goods from one area to the other in terms of the category of the goods or the next destination of the goods and so on. So physical automation at this scale is happening at Amazon Logistics Center. So that's basically the details of what physical automation is. And then your other question was about, I think, the technical and organizational systems involved in the production of this data.

So technical, systems are pretty much in one line. I use AWS CloudWatch Analytics, then I use s three buckets to transfer this data to via Firehose mechanism. So as we are an Amazon house, we are part of Amazon. We use basically the AWS ecosystem. We have Kinesis Data Streams as well. And then for the AI part of it, we use Amazon Quick, which is the Amazon AI toolkit, the Spaces, knowledge bases, and AI agents.

In my case, it's also a chatbot conversational agent, which works with now the organizational systems. They are basically the site associates using the system, and then they are hierarchy for overriding any common issues, like multiple layers of management or their IT support or their maintenance folks, which have the authority or ability to take care of certain kinds of issues and basically canceling or pausing the automated workflow to take care of any safety or hardware issues.

Another organizational system is basically the role based control over all of this, using this hierarchy and controlling each of the objects in this physical layer. There is also an integration with the yard where the yard basically integrates saying, this is where this vehicle is going to go, whether it's inbound, outbound, or basically mixed docks, which is the source of the trailers and then which are sending logistics

workflows to the doors. So I'll stop here and see if there are any follow-up questions, Tobias.

Tobias MaceySo given the variety of physical systems that you're integrating with, what are some of the types of data and some of the relative volumes of data, both at the kind of individual event level as well as in the aggregate and some of the ways that that influences the ways that you think about the overarching solution that you're focused on?

Keerti KulkarniGreat question. So the scale of the data is pretty much in petabytes because every second there is some vehicle docking at some of the, one of the Amazon warehouse in The US or in the world. So currently we are live in The US, but we are planning to expand globally as well. So the data is definitely scale is petabytes, but the events are basically time streams, which are coming, as I said, every second.

And we are not just maintaining the events, but we are also maintaining the history, in which we are saying that say forty seconds back, this door was active and this event was happened. And based on that context, we are actually going to rate the next event, whether it had a potential of causing any failure or it is the regular expected event. So maintaining all of that history as well as the context makes

the scale even more complex and, more daunting.

Tobias MaceyAnd the other aspect of this is because you're dealing with real world physical systems and because the operational capabilities of Amazon relies information that you receive, and you're also dealing with a wide number of people at various locations whose jobs are also going to be impacted. What are some of the failure modes or some of the effects of inaccuracies or delays in terms of the processing of the data that you're working with?

Keerti KulkarniYeah, that's a good question. Because the jobs of the people at the warehouse who are plugging in to this workflows and doing the jobs which are needed there, even with the automated workflow, if there is any safety issue or if they do not follow any particular step guide, then this causes, basically a warning and then a future consequence for them. So this becomes a very high priority for the logistics leadership to track

and manage. So we are making sure that the data is as detailed as possible, but some of the data is obfuscated at the user ID level. So if there is, like, a bit of delay, which is not causing a safety concern, then we wouldn't want the user ID to be immediately circulated into various dashboards and into the logs to... The logs, it is already there, but we are not going to move it to s three and where we do the analysis.

So that there is a confirmation that this shift or this site needs some training, but all of the user IDs are not exposed. Only at the high level where there is a very safety targeted issue and which is already known to them and this associate or personnel has that issue, then that... The details of that can be surfaced using the system, but that will require many roles and many

clearances before people get to it. So we are making sure that none of the warehouse personnel are adversely affected because of this analysis and data being available. That's the first one. And then the weak points of the system, let me touch base upon that. We treat the physical layer as a first class telemetry, not as a noise, which is to be smoothed later. So each vehicle entry point reports state, and we tag events with side clock and ingest clock. So if there is any difference between this, then there are major effects seen downstream.

So any pipeline stage that assumes tight ordering, we do a reconciliation step on it to confirm that everything is up to that point has happened as per expectation. The lesson from about 1,000,000 cycles that we have completed in last year for 30 plus warehouses, which we were launched at that time, and now we are at 100 plus. On those 1,000,000 cycles, we saw that the late or duplicate data will happen at every layer. So we designed a ledger

with the event key, and we made a summarization layer on top of it, which is explicitly aware. As I said, that the last thing you heard from a specific door was for a specific event at a specific vehicle, so was ninety seconds ago. So based on that context, you are able to dictate further. So as we learn more lessons, we are trying to add more summarizations and synergies into the AI layer that we are building, but there might be some cases which we are still... Have still not encountered, we are looking forward to work with it and update it as we go. So

Tobias Maceydigging now into what you're building, you mentioned some of the technical components, but I'm wondering if you can talk through a bit more of the journey that the data takes and some of the types of analysis that you're doing, and then especially the role that the LLMs are playing in that overall system.

Keerti KulkarniYeah. Absolutely. Would love to. So first, for analysis, we are checking all of the events from the time stream as well as, from the CloudWatch, any errors and any messages that we are getting on each of the events. We tie those together to get a full picture of the whole network, but specifically one site, one door at that particular moment. So we go down to that level. Though... So the Firehose basically

takes the CloudWatch data and we ingest into s three. As you have previously pointed on one of your episodes, CloudWatch is basically expensive if we treat it as a primary source and start doing analysis on it. So our way to, what do you call, mitigate that risk is basically moving it directly to s three, which is a low cost storage. And then, we basically analyze on it, with the Amazon AI framework.

So once it is in s three, we have a way to aggregate it or analyze it further based on what operations team or org needs or our product needs to make decisions on the road map or where we are going next. Also, the leadership team to check the entire state of the network with new launches. We have that ability. So let me dig a little bit deeper into this. So we also move this data into Redshift for

consumption into the various dashboards that are pretty static, but that's another consumer from our system. So Orkfit. Amazon, basically, for CloudWatch data to migrate into s three and Firehose, we can also purge it from CloudWatch as per the retention policy. So we are controlling the cost there. Then I will talk about the AI summarization layer. So now AI summarization layer is basically AI automation along with a chatbot conversational agent specifically targeted for the operational team. So if they see that there is an issue docking

a vehicle at a door, then they only get from the associates who are pretty much at the job and doing their work. They do not have much time to explain what's happening, but they are just like the automated workflow is failing. So we have to, based on that, we have to make sure that we are giving, analyzing the right cause and giving them a way to fix it. Otherwise they are interested in basically completing the work with or without the automated workflow. So we have a a risk of our usage dropping as well. So for that, we have given the operations team a way to just ask questions to the agent

in a simple language saying, hey. Tell me the issue happening at, say, Door 42 at this site, in this warehouse, and at around this time between eleven and noon on this particular day. So all of this three, the logs which were transferred in terms of the messages are scanned and specifically a history on that door or on that site is also tied back to the question. And it said that, hey. This issue is happening because

an unauthorized user who doesn't have access to canceling the workflow is trying to do it. And this is similar to the same user accessing at various inbound doors. So they get a little bit more context. Also, can reach out to that user and say, you are not authorized. So please get authorization from this person. We also give them the links on how that authorization is requested. So in case that user is not supposed to have the authorization, somebody else will do and come come and complete the workflow. But if they are just missing that authorization

by a mistake, then they will be able to get that authorization and continue that workflow from that point on. So currently, it is about an hour of a delay in getting those CloudWatch and putting it into s three because our jobs run every hour. But we are trying to move it down to 10 so that we have close to real time data and we are... That will be more... The turnaround time is reduced for the people on the job and working on the doors.

Tobias MaceyIn terms of where the system was when you started, I'm wondering what are some of the evolutionary points that you used as checkpoints to where you are today to make sure that you're headed in the right direction and some of the types of feedback and some of the change management given the number of people who are involved.

Keerti KulkarniYeah. That's a great point. So adding a conversational AI agent versus what was there initially versus just using push notifications or dashboards made the platform more accessible for the operations teams, who are the people on the ground, and they are working with the actual associates, so who are giving real time feedback for our system. So making this most useful for them was one of the key design choices that we made. So if the, conversational AI agent we had not

added to this mix, then we would have basically push notifications and we would have dashboards, but we would not have something real time which can solve their problems. So that's one design choice we made that served as a data plumbing. That lets physical operations run themselves. And we did it basically in the crossroads or sims where telemetry warehouses and now the AI have to actually cooperate. So we don't have to get out and build something specific, but we managed to build it as an integrated part of this whole solution.

With the AI usage at Amazon skyrocketing, this was a natural design choice that we made. Also, as I was telling earlier, the CloudWatch only could have... And Firehose could have been also a choice, but we decided to use S3 and put a retention policy on CloudWatch because we are anyways having all of the logs for this particular analysis in s three. In that particular effort, we also managed to reduce the overall cost by

putting a more, what do you call, strict poll... Data retention policy on CloudWatch. So we saved a lot of cost for the, teams. So those were the two design choices I'm pretty proud of.

Tobias MaceyAdding an LLM to a high volume data system can add latency if you're not careful about where it's being applied and what responsibilities you're giving to it. So I'm wondering what was your process for figuring out what were the constraints that you wanted to place on how and where the LLMs were used and some of the ways that you work on doing the context engineering around what specific details are useful without just overwhelming the LLM and having it

produce a a bunch of useless or irrelevant details in its responses?

Keerti KulkarniSo, frankly speaking, we are still working on that. We are still experimenting which of the asks caused the AI agent to either cause a significant delay or hallucinate. So we do not have the option to cache because every request is coming from a different site and a different door and means either a hardware failure or something in the automated workflow, which is related to software. So we are still experimenting it, but a large...

From our knowledge, we have seen that analysis network wide or something related to the site with multiple dog doors saying, hey, what is the pattern I'm observing for this particular shift when there is a high inbound traffic and multiple dog doors having, like, authorization issues or something where the workflows are not synced. Those are the, some examples where we have seen a high degree of latency,

and we are looking at ways to reduce it into a threshold which is manageable. Like, if there are times when my leadership wanted basically all of the network health status. And then it has taken about thirty minutes to forty five minutes to pull that out over a period of time. And that is a lot of delay as per my understanding. So what we are doing is trying to

get those into batch jobs versus real time asks and run those pretty frequently before business review or a specific period of time and then send it proactively to not get the ask versus making it a batch process that runs in the background. But that said, there are some other also asks which are causing a little bit of delay because if I am from here asking for a Europe data center or AWS

setup to run a query on the doors in, say, Germany or somewhere, then that does cause more delay than I'm running something in, The US data centers. So those particular things we are trying to figure out whether there should be a backup of it locally or at multiple nodes, and that's all on the road map. So we have still working through it.

Tobias MaceyAnd a lot of what we've been discussing so far seems like it's focused predominantly on current state and recent state where you've mentioned some ability to do more longitudinal analysis. But I'm curious what are some of the potential capabilities that you see for being able to do some historical analysis of issues that have recurred and maybe doing some time series forward looking projections on being able to anticipate

potential problems, whether that's for maintenance requirements or being able to adjust some of the delivery scheduling because of... You know, maybe we're... Because we're approaching the holiday season, you know that there's going to be a lot more volume churning through and some of the ways that that factors into the design of how you're processing the telemetry, both mechanically and using the LLMs.

Keerti KulkarniYeah, absolutely. This is a great segue into the future plan because this agent... We have other agents also built by my team where we have sort of an agentic framework over the physical logistics automation, and we are trying to integrate that whole framework and create basically a portal sorts for the operations team where they can actually look at any door and the health status on that door.

So it can tell you if the hardware that gives you the automated workflow is basically optimal at that point. And if there are any previous occurrences which are possible to happen again, previous incidents, which have either caused a high severity issue on that particular door, an outage, a Wi Fi issue, or, some sensors failing because of which this automated workflow itself could not be accessed and the vehicle had to be moved to a different door.

So we are trying to exactly do that, that check it from the door, which is basically the object which we are tracking for the automated workflow and seeing if the failures in the past give us some context and pointers to see if a sensor is this particular is causing some concern, then how long do we have before that door starts giving an alarm or creates the...

An incident? In addition, we also have connectivity issues going on at the site because there are various access points, which are sensors and other equipment connects to. And if there is a central IT team, which manages all of that. So if there is any access point signals, which are fluctuating, then our doors don't get signals and then our devices are not able to really work in full capacity. So that too, we are trying to see if automated

tickets or automated incidents can be cut to the central support team where they can actually proactively fix any such issues where we are going to see problems. And we are currently figuring out what are the patterns that we see, which cause what issues exactly. So we are building an error framework with actual messages and the actions to be taken and the person or the role

which will be taking that action. So based on that, we are creating an automated framework. So that's in our roadmap.

Tobias MaceyOne of the perennial challenges of doing anything at scale and across a broad set of... And especially physical locations is that you can't guarantee consistency, particularly when you're talking about any physical elements where different sensors might be running different versions of firmware, or you might have different generations of sensors across the different locations based on when and how they were built, or different locations might have different physical or geological constraints that prevent you from having the exact same

estate everywhere. And then also because of the local differences between whether it's the plant management or just the seasonality of that location. People are going to be asking different questions of the system. So I'm wondering how you manage the scalability of the core platform while allowing for some of that more personalized access or personalized treatment for the different locations, given the degree of variance and the different axes along which they can vary?

Keerti KulkarniYeah. So I would like to go back a little to answer that question. With the certification that we have for this setup, it was tested on all of the variants, I would say, whether it's temperature or location specific in terms of camera, if some locations are not getting enough light to actually see a vehicle backing up properly or seeing if somebody else is in the field in the yard, which is causing...

Will be causing any harm or damage or fatality to that person. So this hole causes a little bit of difference, but when we were certified, the OSHA basically came to different locations where we are planning to install these devices and did a threshold analysis of what is the range we should be considering for each of our components in this hardware ecosystem. And we were able to either create the enclosures or add additional lenses

where we are supporting the extraneous ends of our spectrum. And we are still very much within that. So we have not seen anything which will be pushing the boundaries either ways. So that's one. But in usage, there is variance in the usage. So also, as you correctly said, in the holiday season,

all of the shifts are working extremely hard. So in some usages, we see people being a little bit too busy to take care of something which needs to be moved before a trailer gets out. So if there is like a fan component which has to be moved, then if they are forgetting to move that and it gets hit multiple times, then our camera basically won't capture the needed image. So to avoid that, we are making changes in the training plans to make sure

that they have a very good understanding of exactly this needs to be done. Otherwise, the automated workflow will be failing or will be compromised the next time. So we try to make that into the newsletter of Amazon Logistics or making sure the sites which are non compliant or we see more problems in those hardware at those sites have these training inputs sent to them proactively with our data collection and analysis. And that's one of the usages I had not seen my agent

being capable of doing, but this is a surprise to me as well that this insight was surfaced that we can do this training changes based on site level patterns that we see. So we are trying to address that as they come and send some proactive input to the learning teams, which sees... Takes over all of the Amazon logistics learning tasks. And that is one of the usage that are... We are looking to improve in the future.

Tobias MaceyOne of the other interesting aspects of running a platform that other people rely on is that sometimes you have to be responsible for identifying who are the heaviest users as well as whether or not it's being used at all. And so there's one aspect of that from the training and evangelizing perspective of just getting people to want to use the system. And then because of the fact that nothing in this world is free, you need to sometimes do cost attribution of who is causing the heaviest load on the system so that we can make sure that everybody is aware of the actual impacts of increased load or increased usage. And I'm wondering what are some of the elements of that training and onboarding as well as the cost attribution that needs to be done.

Keerti KulkarniYeah. So Amazon engineering, from the various teams doing this workflow, as well as the operations leadership, they wanted to see economics per site and rightly so, as you said. So the option was to take the fleet functionality in AWS, which rolls up the costs by teams or orgs, but that's what, excuse me, that was not giving us the enough granularity,

which was needed to see which were the power users or which is causing the most cost on the network wide usage. So we were looking as the objective to attribute every dollar of pipeline cost to sites specifically. So that was the task. And then what we did, to accommodate this ask was basically tag the Firehose streams by sites. So when the data comes in, we are tagging it by site. So every

query on the s three that we are generating with a conversational ask from our org team or some analysis request from other teams, then we roll up those costs daily into a cost ledger sort of for the sites. So we create alerts on every site cost per assignment, which is above threshold. So whether it's a day, whether it's a week, whether it's a month, we take care of all the periods and try to generate these thresholds

with which we can send out these alerts to whoever is the POC for that site, as well as the leadership for operations at logistics as well as our product leadership. With this, we make sure that the leaders see the cost per site and also the anomalies that we are seeing. So whether the, workflow is not functioning properly, that's why we are seeing this high cost, or there is some additional stuff going on there. So the anomalies are detected pretty easily.

Then we have to do some root cause analysis with the help of our data gathered. But the anomalies, I would say, are caught within a day almost because of our tagging the site to the rolled up cost. And this helps, tremendously figure out the power users, figure out any anomalies, figure out any issues in training or, in the hardware and so on.

Tobias MaceyAnd going back to the LLM question, you mentioned that you're still working on dialing in the accuracy and some of the specific use cases and investing in making it more of an agentic capability. And agents in particular can be challenging because of the fact that they're so open ended in terms of their capabilities. And so I'm wondering what your infrastructure is for managing the experimentation and being able to track and evaluate

the accuracy as far as when and how it improves or degrades so that you can continue to maintain and evolve the system with confidence.

Keerti KulkarniAs you correctly said, LLM outputs are probabilistic. So the operations humans will trust this summary if it looks confident, but there could be hallucinations. So we have, like, a structure created for the output, which are based on their ledger events and event IDs. So we attach that to every summary at the bottom. So even if that particular operations user are not aware what that really means, but any...

If we see their feedback and we collect regular feedback from every question asked. The operations personnel have a way to, at the end, rate it if the issue was resolved or it was basically not the correct root cause. And based on the last twelve months of the summarizer that I have created, there was no production incident traceable to the hallucination. So since this infra setup was completed.

But saying this, it's not true that we won't be facing this. So we are trying to basically limit the summary length. We are trying to, as I said, attach the ledger event and event IDs and the vocabulary. We are having very strict vocabulary control on the, agent take a framework as well. So this along with the human in the loop, like, enable the users to share feedback on the resolution recommendation, that actually has a power to

do a quick check before actually letting the associates know. So that is a multiple layer. First of all, the summary is pretty well structured. And then there is a human in the loop, which is our operations team who's communicating the root cause to the associate saying, do this. It will solve the problem. So if they think think something is seriously wrong, they will give us the feedback and stop giving that feedback ahead or result ahead to the actual associate.

So that's our current method. And once we get enough feedback and saying, okay, this is good enough as per the threshold, and we should be just creating an automated way for associates to ask without our operations team involved in every ask at every point. We will be automating this in terms of tickets coming in, answers going out, and so on. So, yeah, that's our plan. Another

Tobias Maceyinteresting element of the system that you're working with is because you're dealing with such a high volume and velocity of data, it opens the question as far as what is the utility of using embeddings or if embeddings are useful at all given the scale and shape of the data that you're dealing with. And if and when you do decide to bring some form of embeddings and vector search into that agentic harness, whether and how you're evaluating the viability of s three vectors as that storage layer since you've invested so heavily in s three as the core storage layer for the rest of the system?

Keerti KulkarniYeah. So we have not... That's a awesome next step, but I have not had enough deep dive into what could be the next better option for us to invest in. But as we... I was telling you, we are trying to create this agentic framework. And at that point, we will be looking at our infrastructure to see what we can make better in terms of search and the capability of reducing the delay or,

improving the cost element. So this is one of the topics we will be looking into. But currently, I do not have any great knowledge for that. I have been pretty busy making this agent useful, but that would definitely be the logical next step for us.

Tobias MaceyAnd as you have been building this system and training up the various stakeholders and end users on its capabilities, what are some of the most interesting or innovative or unexpected ways that you have either seen it used or seen it change some of outcomes that it is designed to feed into.

Keerti KulkarniSo let me think about this a little. Yeah. So our operations team is basically... Has a specific way of working with the associates. So sometimes they are on-site, sometimes they are at a different location, and they are still looking at their phones, seeing what is the issue on a video call with the associates saying, okay. This is what is happening at the door and what the issue could be.

So in that, they are unable to basically log on to a proper system and start any Amazon agentic framework on anything like that. So my system is basically compliant enough that operations teams have their own wrapper on top of it, whether it is a very, very conversational chatbot

made in Claude or there is a different kind of interface that they are currently using and this is an API call that coming to my agent or my framework, I have made those available to integrate with. So this request I got after I thought of a conversational AI chatbot agent, and they use it when they are at a site or Amazon corporate sites and working with the associates. But this was something I had not initially planned for. But based on their feedback, we were able to make this integration touch points available to them with which they find their best way to create that

for their wrapper to integrate with my framework. So that makes it more usable, to multiple contractors as well as in future associates as well at the site when the operation dependencies go away. So this is something I had not anticipated supporting. Also, the other thing as I was saying was training needs. So depending on the shift, we can actually trace the callouts or different behavior patterns

down to a shift site. We can also go to a user ID, but we try to avoid doing that. So up to a shift, if we do that, then the training teams get valuable feedback saying, this training seems to be missing because the workflow goes up to this point without a hitch. And here, many people are going back to doing the old way. In that case, they know exactly where the training doesn't seem to be working for those folks

Because based on cultural differences or any... Even regional differences, there could be different needs that the people have in those areas in terms of training. So they will be able to modify the training based on these patterns. So I'm giving right now training teams access directly to my framework instead of me sending out the summaries to them. This is one of the things I'm currently working on so that they get the insights by themselves as per the format they want, and they can make those decisions on training updates and stuff. So these are two of the things that I had not initially thought of. But after

the framework was used, these came to the notice. So I'm looking forward to even supporting more cross teams and more functionality.

Tobias MaceyAnd as you have been working in this space and building this system, what are some of the most interesting or unexpected or challenging lessons that you've learned personally?

Keerti KulkarniSo I would say the culture wise, right? Amazon's culture, as you had also hinted a little bit earlier, makes it easier to ship these, more than at a startup. But then, again, at Amazon scale, you are actually launching these tools to the network. So it does not give us the organizational patience, say, for a slow rollout. So we have been walking this thin line

with boring kind of small blast radius method where we are actually launching it to test doors at the sites where we have the test doors and making sure we test out the full functionality there and then go network wide. But we also have something planned for next year where we would be able to select the sites and the doors in advance and go at at least at a few doors per site in the areas which seem pretty diverse

and test it out without, like, compromising on basically just the test doors that we have and so on. So we are building this intrinsically into the software layer of slow rollout and targeted rollout for testing, and then we can go network wide. So that's one of the unexpected challenges that we saw, and we are still working on fixing it as what our need.

Tobias MaceyAnd you mentioned a little bit on some of the future work that you have planned for this system. I'm just curious, what are some of the ways that you anticipate the broader ecosystem encompassing and integrating with this physical logistics automation platform, particularly as you make it more agentic and as multi agent systems become part of the general substrate of how we build technical systems?

Keerti KulkarniThat's a great question. So I would up level this a little bit. What everybody is doing is building AI agent tech systems, and they are trying to build agents which actually do the work and then connect with some other agent. But modular agentic AI systems is something which I see a need at least for such physical logistics automation. So whatever we are doing in terms of connecting the vehicles to the doors, there are other players, other Amazon teams

doing a similar effort but for different part of the logistics workflows. So some part of the modular framework could be shared between these teams, like, specifically for authorization, authentication and authorization parts or basically rule based checks where there is an integration with a very specific hierarchy in logistics. So that that part could be a modular, and then we could have a cross team

integration, basically. And that would be very easy to even integrate across agents in that case. So trying to build a cross org council at the logistics level where it's more focused on automation and various teams in Amazon, which are trying to automate that workflow, whether it's the robots loading or, like, something like docking that I'm working on or even loading and loading the belts, automated belts. Those all work together.

So we are in... Very in the initial stages to figure out that what the mechanism will be and how the cross org council could make it more modular so that ongoing, there is less of a disposable work and more like, on top of each other we are building. So that's one of the, items I'm seriously looking to pursue along with other POCs from other teams.

Tobias MaceyAre there any other aspects of this overall space of Amazon's physical logistics automation or the systems that you're building to support that that we didn't discuss yet that you'd like to cover before we close out the show?

Keerti KulkarniI think we covered most of it.

Tobias MaceyAlright. Well, for anybody who wants to get in touch with you and follow along with the work that you're doing, I'll have you add your preferred contact information to the show notes. And as the final question, I'd like to get your perspective on what you see as being the biggest gap in the tooling or technology that's available for building data systems today.

Keerti KulkarniI think governance for the probabilistic writers is the biggest gap I see. So every governance framework I have used assumes basically deterministic producers. But AI agents that write to your operational store are probabilistic producers, and none are the mature tooling. So they don't know how to draw a boundary around them or track their provenance or roll back their effects cleanly.

So specifically the last one, rolling back the effects cleanly. So you have touched this in a previous episode as well, I think the Omnigraph episode. And I think that is the frontier for the next at least few years that we will have to concentrate on.

Tobias MaceyYeah. It's definitely a interesting time to be working in technology. So I appreciate you taking the time today to join me and share the work that you've been doing and some of the ways that you're integrating the physical requirements into these technical systems and bringing LLMs to bear to make the overall system easier to interact with and more powerful for the people who rely on it. So I appreciate all the time and energy you're putting into that, and I hope you enjoy the rest of your day. Thank you so much, Tobias. It was exciting to speak with you and your audience about this.

Thank you for listening, and don't forget to check out our other shows. Podcast.net covers the Python language, its community, and the innovative ways it is being used. And the AI engineering podcast is your guide to the fast moving world of building AI systems. Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes. And if you've learned something or tried out a project from the show, then tell us about Email hosts at data engineering podcast dot com with your story. Just to help other people find the show, please leave a review on Apple Podcasts and tell your friends and coworkers.

Open the transcript file

Hosts and Guests

Hosts

Guests

Credit the people on this episode

Be the first to rate this episode

More episodes

All episodes
  1. Entity Resolution at Scale: Graphs, LLMs, and Real-World Data Messiness

    59 minAudio

    Summary In this episode, I spoke with Steven Renwick, CEO and co-founder of Tilores, about the realities of entity-oriented data processing and the practical role of LLMs in entity resolution. We covered the confusing...

    Credit the people on this episode

  2. Why Metadata Still Matters: From Data Catalogs to AI-Ready Semantic Hubs

    1 hr 25 minAudio

    Summary In this episode, I spoke with Christian Bremeau and Simon Dynin about their decades-long work in metadata management and how that experience shaped the evolution of Meta Integration Technology and its...

    Credit the people on this episode

  3. Reducing Data Debt with Agile Ledger Architecture

    50 minAudio

    Summary In this episode Christopher Doidge talks about his Agile Ledger Architecture (ALA) approach to data warehousing and how it aims to reduce data debt while shortening the path from raw data to trustworthy business...

    Credit the people on this episode

  4. What Context Really Means in Data Engineering and AI

    53 minAudio

    Summary In this episode Soham Mazumdar, co-founder and CEO of Wisdom.ai, talks about what “context” really means in data engineering and AI systems. He explores why context has become such an overloaded term, spanning...

    Credit the people on this episode

  5. Specialized AI for Data Engineers: Inside Astronomer’s Otto

    46 minAudio

    Summary In this episode Yetunde Dada discusses Otto, Astronomer’s AI agent for Airflow, and the broader challenge of making agentic tooling actually useful for data engineers. She explored why generic coding assistants...

    Credit the people on this episode