New Jersey's department of labor, in collaboration with USDR and Google.org, has assembled a set of training materials designed to turn off-the-shelf language models into bilingual unemployment insurance experts. We sat down with two of the people behind the launch to break it down.
Audio and video play straight from the show's own hosting. PodSubs stores and serves no media.
Transcript
From the show. Tap a time to play from there.
Over the past two years, AI developers have been on a mission to achieve integration. Recently, that mission made its way to New Jersey's Labor Department, where unemployment insurance workers are using off-the-shelf models to make English-to-Spanish translations. And apparently, it's made them three times faster than they were before. Stick around, we're breaking it all down. But first, subscribe, hit the like button, drop a comment, you know, all that good stuff. The
New Jersey Department of Labor, in collaboration with Google.org and the non-profit U.S. Digital Response, today released a free, open-source set of LLM training materials that are specifically designed to improve access to unemployment insurance for Spanish speakers. The idea here, which is already in action in New Jersey, is to fine tune existing off-the-shelf generative AI models, turning them into reliable translation assistants. The core of these materials is an English to Spanish unemployment insurance glossary. This was developed in collaboration with bilingual staff and policy experts,
and it was tested and verified both for accuracy and efficacy. New Jersey's own implementation of a language translation assistant here has The idea here is that we have processes in place to try to make things more accessible. Unemployment, as an example of a benefit area in New Jersey, is very complicated. And professional translation is great, but it doesn't necessarily mean a professional translator
who understands UI regulations and UI language and UI standards All of our work related to AI has been to center the human, and I mean the human expert. On the call center, those people are always getting humans. They're never getting a chatbot sort of answer. When they're talking, they're asking for someone to speak Spanish with them, to ask a question about a claim, that's always a human. We don't don't have it fully figured out yet but playing around with what are different ways that we can present bilingual information in the same email. Part of the reason that we spend so much
time trying to figure out how do we test that this is as close caliber as we can to a human expert translator. How can we prove that this is efficient, right, is because we're trying to show governments that have different access to AI how they could build a similar thing and decrease their risk and increase their equity in this way. I believe we have over 80% of our staff have taken this training, and that's staff all the way, executive all the way to a call center agent to someone
The materials on offer here are developer agnostic, meaning a given agency or state can use ChatGPT or Cloud or LLAMA or any other chatbot that they prefer. The idea behind this is that it would erase the cost of specialized long-term contracts with tech firms. The focus is also, importantly, on First, there's the problem of language barriers. Nearly 20 million Americans say that they speak English less than well. Second problem these trained materials solve is that it's a common problem across the United States. Any
state can access them to create their own language assistant in the LLM of their choice. They can use these training materials and put them to work in Chet Chippy Tee or Claude or Gemini. This is one of those great examples of a public-private partnership that centered human beings at the center. This was not a collaboration about, how do we use this really cool technology, right? But rather, we took a human-first approach. And so what's key here is leveraging information that's publicly available, using that to develop
the training materials, We don't think of this as being one of those situations where a job would be replaced. Rather, we see it as giving tools to government workers to better serve residents in their communities. We really wanted to ensure that there were no risks in terms of privacy or security that were a part of this approach, and I think we've struck that balance Now, the ethical issues of something like this are very clear. The obvious two that come to mind involve hallucinations and job loss. On
that first point, hallucinations refer to the tendency of large language models to just produce false information. Since these models operate without a genuine understanding of any of the content that they produce. They tend to make these kinds of mistakes. The risk of hallucination in an environment like this seemed pretty severe. And there's also this fear of job loss that has spiraled alongside this rise we've seen in generative A.I. over the past couple of years. According to some early studies, that fear is not unfounded, though. So far, it's mainly targeting freelance industries at the moment. And there's evidence
that it is starting to create other jobs right here. We talk about the idea of prompt engineering, right? But on top of that, there's also risks of data privacy and security breaches, again, in what seems to be a pretty sensitive environment. I pose these questions to the New Jersey Labor Department and the USDR. And according to both organizations, the reality is much more focused and much more secure than it might seem. Let's get into it. Jillian, You know, the, the place I want to start, right, we've got the training, uh, set
coming out, um, and then going out being open sourced, uh, potentially across the country. Um, now I know New Jersey is the kind of first, uh, epicenter, I guess, that this is launching. And I just love to hear from you. how it's being used, how it's being used, what you're seeing on the ground so far in terms of how this is impacting the current processes, I
I mean, I think to start, the idea here is that we have processes in place to try to make things more accessible. Unemployment as a example of a benefit area in New Jersey is very complicated. There's a lot of legal standards, federal requirements that people have to make or meet in order to be deemed eligible. And those things are really difficult to be understood by someone who speaks English fluently, nevermind
someone who English isn't their first language. So when we try to think about how do we make equity and access to those benefits the priority, and whether it's unemployment or any other benefit that the government is trying to think about and make sure is reaching all the folks who are paying into it or who deserve it because it's a right in their workplace, One of the things we have to think about is language. And so for us, when we were rolling out our new unemployment application, an application that was built on modern technology, built in
a way that it could be connected to our legacy mainframe, we had to think about what message do we want to send about how we as an administration feel about equity. And so for us, the most important thing was to deliver it at the same time in English and Spanish. And normally for us, that would require us, you know, to pull through all of the words that we were going to use and get those out to a professional translation. And professional translation is great, but it doesn't necessarily mean a
professional translator who understands UI regulations and UI language and UI standards and what able and available means in UI speak and what being fired means versus being laid off means. in UI speak. So what this technology allowed us to do was pour in that knowledge of how do we, as bilingual call center agents, how do we explain that, those terms, those complicated ideas and requirements over
the phone to someone who's calling in when they speak Spanish or they prefer to understand those terms in Spanish? How do we pour all that knowledge and experience that our agents have into the application itself. So that's when someone is reading those questions, they are getting the knowledge of that call center agent in how the translation is appearing to them. And so that sort of explosion of the use of all of that knowledge of our bilingual agents
and that have that experience, being able to sort of magnify it across the different touch points that a human might have with her, with her unemployment application, right? The idea that you may, you're going to need it in the beginning when you're trying to understand, should I apply or not? You're going to need that knowledge again when you're doing the application. You're going to need that knowledge again when you get our communications of we need X and Y from you in order to figure this thing out. Or maybe you're going to
file an appeal. How do you understand your appeal rights and the steps of an appeal? So that magnification of the impact of that specialized knowledge is one of the things that I'm so excited about in terms of increasing equity Right, and so what you're talking about, right, it's a very technical and important process. And the
means, I guess, with which you're taking that specialized knowledge and turning it into Spanish, we know that these models have a tendency to confabulate or hallucinate. They kind of just output things that aren't necessarily grounded in truth. And so I'm wondering what procedures you have in place so that the person calling in looking for that information doesn't end up getting completely off-base information delivered and
It's really important. All of our work related to AI has been to center the human, and I mean the human expert. And so on the call center, those people are always getting humans. They're never getting a chatbot sort of answer. When they're talking, they're asking for someone to speak Spanish with them, to ask a question about a claim, that's always a human. But what this is allowing us to do is speed up the process to that final human review. And
I'll give you an example. So when we are updating the language in our UI application, like say we're adding a new use case that now people who served in the military can file through our online application. I am always looking to have parity with the English and the Spanish version, but how do I get that Spanish language that I can add in there so that both experiences are equal in English and Spanish? I'm
using this system, the set of probs and this glossary, to give me the suggestion. We think We, the AI, think that this would be a good translation of the English version. And then the human, me, or one of the bilingual call center agents can go in and review and be like, yes, that's right, or no, I would tweak it in this way. But the time that it takes to get something close to perfect to a human and
the amount of time that that human then has to review is much smaller because all of the complex terminology, all of the jargon that we have to work with, that's already been dealt with. That's already within the AI system. So the likelihood that I have to spend three hours approving the content, it's much more like now I have to spend 30 minutes, right? So the human is still there, both on the front end of determining what a applicable, what a realistic localized
Spanish translation of that term is, and the human is still at the other end approving the version that spit back to us. But the time it takes to do that circle is going down. And the risk to the government that the translation is poor is lower because the expertise of the UI is being brought into the translation at the beginning. And so the reviewer knows
what has gone in, what hasn't gone in, and can figure out, has the AI hallucinated? Is it not giving me back what I should have seen because I'm an expert? It just gives us a lot. a lot more confidence in the language that we're putting out there is the language that people are going to understand and reflects the reality of a very complicated Interesting. So at the end of the day, you still have not just human experts in UI, but bilingual human
experts in UI and in Spanish that are then approving at, you know, every stage of the process. And so this has just enabled them to do a lot more, much Yeah, and I don't have to take them off the phone, right? Like, what I really want is for those humans to be on the phone, if that's their job, right? And using the example as the bilingual call center agents, they're there to help people on the phone who are calling because they can't figure something out, or they feel much more confident speaking to a human. The more time that they can spend on the phone and less time reviewing documents,
The reviewing of documents has a value, but the impact to that human, if they're able to pick up the phone, is much more immediate and higher. So if I can take less time from them on the reviewing part because I'm giving them something much more accurate, then I feel like I'm getting And have you been seeing that play out, that kind of impact of enabling these experts to spend more time on the phone? Has Yeah, we see it in
the review time that it takes someone to add this language into the intake application. I'm trying to think of another relevant example. The amount of time that it takes us to We've been working on some email communications and thinking about how do we, we don't have it fully figured out yet, but playing around with what are different ways that we can present bilingual information in the same email, knowing that about 95% of the people who call
in asking for language support are asking for Spanish, prioritizing Spanish and putting those translations in some of the emails underneath the English and saying, you know, Español Abajo. the time now that it takes for us to do those translations and get them out in front of humans and see if they're working and see if they're not working is much quicker. Just because translations are typically an administrative process, right? You send them, you procure them, you go through a whole set of
steps to do that, and that has its place. But for the type of nuance that a UI applicant needs in order to have true equity and access to their benefits and truly understand their rights and responsibilities. We feel like having the bilingual call center agents or the bilingual adjudication agents or the bilingual staff be part of that process is helping us both increase in equity, increase our trust that we're presenting what
Right. And so I guess this approach, is this something that's being explored for areas beyond UI? And That's part of the reason that we spend so much time trying to figure out how do we test that this is of as close caliber as we can to a human expert translator. How can we prove that this is efficient, right, is because we're trying to show governments that have different access to AI how they could
build a similar thing and decrease their risk and increase their equity in this way. So some of the steps that we've taken so far is to start to train other areas within Department of Labor on how they could take these steps and what sort of glossaries they would want to build with our Paid Family Medical Leave program, for example, or workers' rights programs. How do you build the right glossary in English? You build the right glossary in plain language. You build the right glossary in Spanish so
that it could get you to a localized, acceptable, on like a risk and legal framework, translations faster. So we've been taking those steps and training some folks to see how they might be able to use it, what sort of work would be involved in developing a glossary that both the staff believe in, is legally sound, and then And so the last point I got for you, right, when we talk about the training
and integration of these systems, I wonder to what degree that also includes AI education, AI literacy education on the workforce to make sure people understand what they're dealing with, how Yeah, it's been a big effort. I don't own it. It's owned by the governor's office and the New Jersey Office of Innovation to train our public sector on the safe use of AI. And there's
extensive training, I think it's over two and a half hours, an extensive training on what is generative AI, how is it different than regular AI, what you should or shouldn't be putting into it, how to understand when and how it may hallucinate. And that training has been made available to all state staff. And in NJDOL, I believe we have over 80% of our staff have taken this training. And that's staff all the way, executive all the way to
a call center agent to someone who works in the mailroom. That's across the board. So we've taken it very seriously. I don't know that necessarily everyone is using the the AI instance that we have for the public sector in New Jersey. So after you've taken this training, you can access this sandbox that's specific for state employees. But that's where we're doing our practice and our training with other staff agency is once you've taken that training, then we
work with you to practice with Yeah. So, you know, we got this, I guess, pretty big launch today, right, with the new program, with the training materials. But where I want to start, right, tell me about the kind of motivation behind this. Why is this needed right
You know, there's a couple problems happening concurrently, and it's part of why I'm so excited about the free and open source training materials we're putting out. First, there's the problem of language barriers. Nearly 20 million Americans describe their English skills as less than they speak English. Let me say that again. Nearly 20 million Americans say that they speak English less than well. And so for those 20 million Americans, we need to make sure that, for example, when they experience job loss, They're
not left in the cold in terms of getting the help they need to get back on their feet, get back in the labor force, and re-engage with the economy. So those language barriers are the first sort of key problem we want to solve. Second problem these training materials solve is that it's a common problem across the United States. So, you know, in this case, we're just so excited to release these materials with New Jersey. I'm a former New Jersey resident, go Jersey. But the fact of the matter is every
state provides these services and these benefits. And we don't want each of them solving the problem themselves. So these training materials are replicable for all states. And so it means that there's sort of economies of scale with a free and open source solution. Last thing I'll mention, Ian, is that this solves the problem of lengthy and costly contracts with tech vendors. A lot of governments can get tied up when they
have to purchase technology. There's really complicated contracts, it locks them in for the long term. And I think what's really important when we think about, you know, return on taxpayer investment, is making sure that governments can purchase the technology they need, but not be locked in. Because it's constantly evolving and changing. So we think for those sort of set of reasons, those are some of the problems we And so that last point, I guess, is why this is a launch of
training materials, not a model itself. It's not just so no one is confused here. You're not launching a translation model. You're launching materials so that people can, That's right. These training materials can be used, any state can access them to create their own language assistant in the LLM of their choice. They can use these training materials and put them to work in CHAT GPT or CLAWD or Gemini. And so it's getting them
most of the way there and then a state can access these training materials and create a language assistant of their own to Gotcha. Now, the materials themselves, how were they gathered, curated, verified, collected, right? We know that in AI, the data is king, and making sure the data is clean and accurate and legitimate, right, is a process. So Yeah, it involved a lot of collaboration. And I'm excited. This
is one of those great examples of a public-private partnership that centered human beings at the center. This was not a collaboration about how do we use this really cool technology, right? But rather, we took a human-first approach. So in collaborating with the New Jersey Department of Labor and Google.org, we started by bringing together bilingual staff and policy experts to help make sense of the key information that individuals, residents in New Jersey would need to know to access unemployment insurance. So
what's the key information? What do they have to know? We then, sort of as we developed the training materials, humans were involved at every step of the way. I think what's so exciting for me is that with the approach we've taken with these training materials, We're leveraging Gen AI to streamline processing the information, but we're not leveraging it to make decisions. We're still keeping that in the hands of the humans so that they can be doing the more complex work and putting the technology to work to help streamline information.
I'm glad you brought up the decision-making side of things, right? Because I wanted to ask about that. We know issues of algorithmic discrimination are a documented problem, aside from, you know, when the algorithm just screws up, right? Hallucinations, confabulations, etc. These things are not necessarily completely reliable. So I wonder, you know, in terms of disseminating this open source kind of framework, right, to what degree does a sort of training go with that to make sure that whoever's using it knows, you know, verify on these points? Or is there some sort of
trust and transparency explainability layer that kind of is designed to go hand in hand with this? That's a great question. And I think I can have Brian and Kate follow up with some more of the details and the technical information. But what I'll share is, I think that for us, what we've been wanting to do is to ensure that there is a way that we're responsibly leveraging the technology, but not at the risk of the determinations and decisions that are being made. And so what's key here is leveraging
information that's publicly available, using that to develop the training materials, and then again, making it available for other states to access and customize on their end so that it works in their particular context. It works for them and the problem they're trying to solve. It's something I just appreciate. We're not trying to solve all the problems at once. But instead, we're doing some of the legwork to then empower folks in governments across the country to sort of take the
risk factors, right? And one is a little messy. We haven't seen details on how it's playing out yet, but it's something that people kind of associate with similar pushes, right? Is will a chatbot take my job type of thing? And in this environment, right? Is that to a degree what's going on or is it quite simply there Yeah. You know, I think there is there is a talent issue at play here in terms of the talent we have in government having
enough bilingual staff to be able to engage and translate the all of the government legalese that we have underpinning programs and services. Ultimately, what we're trying to do, though, is to think about how to, I guess, how to essentially give government workers a tool like a calculator. We don't see these training materials as replacing roles or getting rid of jobs. Rather, we think of these as
being a key tool that government workers can use, process the information, get it in a way that is easily understandable for the residents you're trying to serve, so that then those human beings can focus on the more complex tasks that they have at hand, like making determination decisions, and figuring out the right next step, both on the government side and the resident side. So we don't think of this as being one of those situations where a job would be replaced. Rather, we
see it as giving tools to government workers to better The other point, too, in talking about the risks, right, when we're talking about using off-the-shelf models, is there a risk of data privacy security issues by talking about potentially sensitive topics that get parsed through
I love that you're asking about it because it's something that's really top of mind at U.S. Digital Response. We sort of put forth an approach in all of our work on generative AI that centers responsibility and ethical use of the technology. These training materials that we're releasing today do not touch PII, and rather what we're saying is we're taking the publicly available data and we're making better use of that and translating it, making it more accessible. But it's entirely staying
on the government side in any state that chooses to use these materials, as New Jersey has done, to bring in any, you know, any private data. So we really wanted to ensure that there were no risks in terms of privacy or security that were a part of this approach. And I think we've struck that balance quite And the last point I want to leave off on, right, you mentioned that New Jersey is already using this. Yeah. I'm wondering what you can tell me about what that's looked like, responses so
far as New Jersey. I am a native New Jerseyan myself, right? How that integration is going, what that kind of In New Jersey, what they found is that by leveraging these training materials, they have tripled their translation speed, but not at the cost of quality. So we did evaluation, and the translations were produced quicker, but are nearly on par with the level of quality that
expert human translators would produce. I think that's really exciting. I think that that means that New Jersey residents who need these services so critically are going to get what they need, get it faster. And then on the government side, workers can be moving things along on their end, knowing that there's a high level of quality in terms of the information being given to residents. So, I think you're going to speak with our partner Jill at the State of New Jersey, and she'll be able to share more. I think
that impact, that result in terms of the quicker translation speed is really exciting, and what we want to see, not at Of course. Well, thanks so much for joining me, Tita. Thank Thank you. And Ian, can I just say one thing that I'm so excited about this for sort of, if I can, just at a high level. The training materials we're putting out, they're fast, again. Translation
speeds in New Jersey have tripled. They're free. Governments can use these resources without entering into lengthy or complex contracts with vendors. and doing so reduces risk and reduces the use of taxpayer dollars, and it benefits everyone. So constituents are getting better results, government workers are able to focus on complex tasks, and taxpayers know that their dollars aren't going to pay for some overly complex tool. I think it's just a really exciting
AI is showing up in nearly every new gadget. But what would make a device worth buying specifically for its AI? In this special episode of The Deep View Conversations, Jason Hiner and Sabrina Ortiz take stock of 2026's...
AI is making it easier for everyone to create software. The harder challenge is building the infrastructure, security and skills to turn that explosion of code into useful work. In this episode of The Deep View...
Coding agents can drastically accelerate how fast developers build, but defaulting to the most powerful models gets expensive fast. But what if your coding assistant could route to less expensive models when they don't...
The AI agents helping employees work faster also open up new risks for a company's most sensitive data. So how do businesses keep up with the opportunities and the risks? In this episode of The Deep View Conversations,...
Your next big AI win may come from someone your leadership team would never think to pick. In this episode of The Deep View Conversations, we sit down with Todd Earwood, CEO of Betra Works and an AI investor at...