This week we welcome Bobby Puglia, Chief Product Officer at Bloomberg Industry Group, to discuss Bloomberg Law’s latest generative AI-powered tools: Bloomberg Law Answers and Bloomberg Law AI Assistant. Bobby provides an insightful look into how these tools are designed to revolutionize legal research by making it faster, more intuitive, and seamlessly integrated into users’ workflows. The conversation begins with a discussion about the challenges of capturing the attention of legal professionals and the strategies that can bridge the gap between innovation and adoption.
Addressing “Tool Fatigue” and Driving Innovation Adoption
Bobby delves into the strategies behind overcoming “tool fatigue,” a common challenge in the legal industry. He highlights the importance of understanding specific user pain points and delivering targeted solutions. Marlene shares how peer advocates within law firms can champion the adoption of new tools, while Greg emphasizes the need for tailoring communication to align with the unique needs of different legal professionals. These insights reveal Bloomberg’s thoughtful approach to encouraging the adoption of its cutting-edge tools.
Inside Bloomberg Law Answers and AI Assistant
Bobby explains how Bloomberg Law Answers uses generative AI to provide concise, transparent answers directly in search results. With citations and detailed attributions, the tool seamlessly integrates into existing workflows. Meanwhile, the AI Assistant offers a groundbreaking way to interact with legal documents—allowing users to summarize, interrogate, and ask specific questions confined to the document’s “four corners.” These tools are designed to save time and enhance productivity for attorneys and legal researchers.
User-Centric Design and Quality Assurance
The discussion highlights Bloomberg’s user-focused development process, including its Innovation Studio, which uses pre-beta testing to refine tools based on user feedback. Bobby outlines how this iterative approach improves content sourcing, interface design, and overall functionality. Robust benchmarking, guardrails, and human-in-the-loop testing ensure that Bloomberg’s AI tools maintain the highest standards of accuracy and reliability.
The Future of AI in Legal Workflows
Looking ahead, Bobby shares his vision for generative AI’s role in reshaping the legal industry. He predicts a future of integrated, customizable workflows that reduce time spent on routine tasks, allowing legal professionals to focus on high-value work. The conversation also explores how Bloomberg plans to expand its AI Assistant to tackle broader research tasks, streamline navigation, and enable seamless integration with platforms like Microsoft.
Listen on mobile platforms: Apple Podcasts | Spotify | YouTube
Blue Sky: @glambertpod @marlgeb
Email: geekinreviewpodcast@gmail.com
Music: Jerry David DeCicca
TRANSCRIPT
Marlene Gebauer (00:07)
Welcome to The Geek and Review, the podcast focused on innovative and creative ideas in the legal industry. I’m Marlene Gabauer.
Greg Lambert (00:14)
And I’m Greg Lambert and this week we have with us Bobby Puglia who’s the Chief Product Officer with Bloomberg Industry Group. Hey Bobby, it’s great to have you on the Geek in Review. Welcome.
Bobby Puglia (00:27)
Thanks, Greg. Thanks, Marlene. It’s great to be here.
Marlene Gebauer (00:30)
So this month, Bloomberg is making waves with the launch of some exciting new tools powered by Generative AI, tools designed to streamline research and provide even greater value to their users. So before we jump into the announcement of the new tools, Bobby, we’ve been taking a bit of time with our guests lately to talk about something in the industry that we would like to get your opinion on.
Greg Lambert (00:50)
So we thought this week and this kind of fits, think, with the the launch of the two new tools that you have there at Bloomberg. And we wanted to see, you know, there’s a lot of. New products that come into the market, but I know that Marlene and I both struggle to kind of keep the curiosity going of the attorneys and others who these tools are designed for. So we wanted to kind of.
just chat a little bit about, you know, how do you approach attorneys that look at recent developments or look at new tools and just kind of, you know, kind of go meh, you know, I don’t have time for this. So Bobby, you want to start it off?
Bobby Puglia (01:30)
Yeah.
Yeah, so I think, yeah, there have been a lot of tools popping up. So I think the challenge in getting the attention for a user base that is extremely busy and their time is very demanding is a big challenge. So the way that we think about that is approaching it with really understanding what problem that we’re trying to solve for the user. So you could approach them.
with a lot more context of why they should try out that tool, knowing exactly where we’re trying to solve the problem in their workflow and give them some of that context. And if you can identify that problem and get a what’s in it for them to generate more of that interest and excitement is usually our starting point.
Greg Lambert (02:23)
Marlene, how do you get attorneys interested in, and it’s not just attorneys, it could be business professionals as well.
Marlene Gebauer (02:30)
Sure. Well, I mean, think there’s, I mean, there’s a number of things you can do, but a few I’d like to highlight. You know, first limit the amount of tools that you’re, throwing at your attorneys. Like we have said, there’s a number of things that, are out on the market and it just seems to be, you know, exploding with even more and more each day. So, you know, don’t, don’t sort of throw all those things at, at your attorneys, you know, and make sure that you keep it focused.
and develop those use cases as Bobby mentioned. The other thing I think you should do is have somebody who’s basically the cheerleader, somebody who’s a peer of the attorneys, one of the attorneys, leader in that group, get them on board, get them excited, and then you can get other people excited. If they hear what a peer is doing about it, they might try it.
they might be a little more willing to try it themselves. And then the last thing is make it easy for them. I know that we’ve had a lot of interest in tools, but we had a bit of a challenge and I think this was, well, I know this was a lot of firms had this challenge where you can’t put client information in. work with your risk groups to make sure that you can do something in terms of your policy.
that will allow your attorneys to use this more easily without having to have additional conversations to do that.
Greg Lambert (04:06)
Yeah, and I would say my contribution to this conversation would be, you know, this is a typical know your client kind of focus here. Understand when you’re talking to litigators that they’re going to speak a different language than your IP or your transactional attorneys and vice versa.
But yeah, I think it’s just a matter of make sure that you’re speaking their language and not just your language and trying to have them fit into what you’re doing, but rather have what you’re doing fit into what they’re doing. that’s how I see it. But yeah, there’s just so much going on that it is.
hard to capture their attention in the first place. It’s really hard to say, hey, we’ve got new stuff and we want to change how you’re doing your thing. So it’s an uphill battle or can be.
Bobby Puglia (04:53)
Yeah.
Yeah.
Marlene Gebauer (05:04)
Exactly.
Bobby Puglia (05:04)
Yeah, I
you’re spot on there.
Greg Lambert (05:06)
Ha
Marlene Gebauer (05:07)
So, Bobby, let’s get into the topic that we brought you on to talk about. And that’s the new Bloomberg Law Answers and Bloomberg Law AI Assistant. So can you give us an overview of these innovative products that Bloomberg’s rolling out and what makes them game changers for legal professionals?
Bobby Puglia (05:11)
Yeah.
Yeah, great. So, you know, while we’ve had previously rolled out some Gen. features, those were very focused on targeted problems. So we had previously released a complaint summaries feature within our docket alerts to, you know, quickly see what’s in a complaint or our clause advisor that’s very specific for contract drafting. Our latest features we’re releasing are much more broader functionality and more focused on the information retrieval problem.
So first one, Bloomberg Law answers. Bloomberg Law answers leverages generative AI and authoritative legal content to provide a brief but meaningful answer to user search queries right on top of the regular results with no need to learn a new tool or to learn any new prompting language in order to get answers. Some of the features where we really focused on providing value here is a couple areas. Number one is
being grounded and transparent in our answers. So each answer that we provide is gonna include citations and links back to the BLaw authorities and sources that used to generate it. I think that’s pretty expected at this point. But we also provide a lot of diverse content. So it’s gonna range from primary sources like your court opinions, statutes and regs among others, but also include some of BLaw specific content like our practical guidance, some of books and news.
Bloomberg Industry News, and we’re going to continue to add new content sets as we go along. Some other things that you can look for in this is very discrete attribution. So not only do we provide the sources that the answers were generated from, but we’re going to provide footnotes within the answer with specific parts of the answer to know where that piece very exactly was coming from. And finally, we’re going to provide extracts. So
you could read through the primary source that populated the answer without having to navigate all over the place. So we really have been focused on the user experience and the journey for the user throughout. So that is Bloomberg Law Answers. On the AI Assistant, it introduces a new way for users to extract information. And right now the first iteration is very focused on a document. So the way we think about it is it’s really like document interrogation.
So for example, when you go into a document, it allows the user to generate summaries of primary content, secondary content, right from within the source document. But even more so, you could ask very specific questions about the document quickly to identify the information they were looking for. So to give you an example, if I have a statute open, it’s very large statute, I could generate a summary and it’s gonna give me a good general overview. But if you have specific questions about maybe a certain subsection and some matter within there,
You could use the assistant to answer those questions and you’re only going to get answers that are confined to the four corners of that document. That’s also pretty broadly available throughout our content. So again, from court opinion, statutes and regs to some of our content, practical guidance, books and manuals. And one of the features we really been working on there is also to understand the context of what you’re asking and what document you’re in so we could suggest some follow-up prompts and questions to the user.
to help them figure out how to use the tool.
Marlene Gebauer (08:48)
For Bloomberg Law Answers, is that just querying Bloomberg vetted content, or does it go more broadly?
Greg Lambert (08:48)
Yeah.
Bobby Puglia (08:56)
It
is all of Bloomberg vetted content. So it only goes over content that’s on our platform that we’ve been able to test benchmark and tune for.
Greg Lambert (09:07)
Yeah. Now, one of the things that caught my attention in the press release that was sent out was you have worked a lot with what you’re calling your innovation studio participants, which gathering your feedback and refining the process before you launch the tool. So, you know, can you share some examples of how customer feedback has influenced the development of the products?
Bobby Puglia (09:36)
Yeah, absolutely. So the Innovation Studio really helped us out in being able to put out some very early functionality and incrementally to gain immediate user feedback to make sure we were on the right track. So what it really allowed us to do is get very specific feedback focused on functionality and usefulness while we continue to work on improving the quality and the accuracy of our answers. So the areas where it’s helped us out the most
is it helped us tune in on what the most pertinent content types that we want to use as source material to ground our answers. So really getting feedback on where are our clients looking for answers, what sources are they really looking to, and it helped us to tune that. But it also really spent a lot of time in helping us to design our user experience and the interface. Feedback on the functionality in there and some of the things I talked about, like inline citations,
the ability to copy certain functionality and the follow-up prompts. That was all feedback that came out from our innovation studio that we were able to quickly work back into the product.
Greg Lambert (10:42)
Now, just out curiosity, is the Innovation Studio, is that a physical place or is it more of a design concept?
Bobby Puglia (10:50)
It’s a design concept, so you could think of it as a pre-beta version of the product that you would have that is set to a limited set of users. So we treat it as more of a playground environment.
Marlene Gebauer (11:05)
one of the highlights of Bloomberg Law Answers, and you mentioned this earlier, is the integration into regular search results without requiring users to learn a new tool. And I had said the same thing, like it should be easy for users to adopt something. So why was this seamless integration a priority for you, and how do you see it impacting user adoption specifically for Bloomberg Law Answers?
Bobby Puglia (11:19)
Yeah.
Yeah, great question. So if you step back and look at where the latest rounds of generative AI is taking us, it does have the potential for us to build some tools that really change and could be very disruptive in how our clients go about their work and use their tools. And the way that we view it is anytime you’re introducing a new powerful technology, the way to get there is through really
being critical about the user experience and making it as seamless and as natural for our clients to learn how to use those tools without having to go in and really figure out where do I got to navigate to? How do I use this tool? And more seamlessly work it into a workflow that they’re already familiar with. So, you know, by taking the approach that we did and why it was important for us, it gave us the ability to
work within the existing BLaw search experience while continuing to demonstrate the value of the new features. It could drive adoption and do so without having to change the user behavior. So meeting them where they already were. And for the question answering feature, chat bots aren’t for everyone and AI assistants aren’t for everyone. We are putting one in our product because many people are getting value out of that.
The UX could be very difficult. It could be hard to think of follow-up questions. The chat interfaces could be very intimidating. you know, a focus with Bloomberg Law answers allows the users to get value from generative AI without the upfront costs and frustrations of having to adopt a completely new workflow.
Marlene Gebauer (13:16)
So this is going to be the name of my new band, Chatbots Aren’t for Everyone.
Greg Lambert (13:19)
So another thing in the press release mentioned that the Bloomberg law AI assistant will gain additional research skills in the coming months. So you mind talking or giving us a little preview of what new features or capabilities that Bloomberg users should expect in future updates?
Bobby Puglia (13:20)
That’s great.
Marlene Gebauer (13:21)
Ha ha ha ha.
Bobby Puglia (13:48)
Yeah, absolutely. So this initial release, as I said, is really focused on the document experience and document interrogation. What we will soon be releasing and what you could expect is we’ll be expanding that chat experience to handle more broad use cases and ones that aren’t just document specific. So a couple examples of that are our features like jurisdictional comparisons. So we currently have our Bloomberg Law chart builders within product, which are a very powerful tool.
So what is the experience for that within the AI assistant and being able to ask questions more directly and get those jurisdictional comparisons a lot easier. The ability to ask about citation and treatments pulling from our B site within BLaw and being able to interrogate and ask questions that way. And then moving on to more of a research task assistant where we could help you find
and guide you to the right materials on BLaw. BLaw is full of a lot of content and really useful features and tools. They’re not always the easiest to find to know in the moment when I need something. So really expanding our AI assistant to understand what you’re looking on and help guide you to the right tool or the right content. And just the overall chat experience, building in more features of chat history.
Getting to understand contextual recall so you could continue on the conversation. And these are some more common features and functionalities you would start to expect in AI assistance.
Marlene Gebauer (15:23)
So Bobby, with the introduction of these new tools, how do you see generative AI shaping the future of legal research and practice? I really feel that there’s a race to own the research space and combining that with other types of workflows. mean, you’ve mentioned document review. So are there specific workflow challenges in the legal industry that you’re looking to address next with
with the AI.
Bobby Puglia (15:54)
Yeah, I think there’s a short-term and a long-term view to it. Overall, what I think generative AI is really introduced is it’s bringing down the barrier to knowledge and the ability to access knowledge. It has the potential and where we are more focused on that short-term is, where can we reduce attorney time and those as a support attorneys on those more rote mundane tasks so they could be more focused on higher value work.
But where I think it’s headed when you think about workflows is the development of what’s happening now with more agentic workflows. And I think that’s where there’s a lot of potential where we could start to look at and build more customized workflow solutions that aren’t just a generic, know, everyone may have a general workflow, but in the way that they do it day to day, there may be some slight differences in how they approach it. So when you’re trying to develop a workflow tool,
is Agenta workflows now start to give us that ability where our end users could tailor their workflows and tweak it to make it really fit for them without a very high technical barrier.
Greg Lambert (17:04)
I know the last time that we talked to Bloomberg on the podcast, we were talking with Mat [Rotenberg] and when Dashboard Legal had come in, it sounds to me just as an outside observer that that’s had kind of a significant impact on how you’ve approached it. much more on the workflow now rather than just pure research. Am I interpreting that correctly?
Bobby Puglia (17:26)
Yes.
Yes,
absolutely. you know, as research is one part of the workflow and, you know, when you look across what’s available out there, I think good workflow tools and work management tools specifically in this space, there’s a lot of shortcoming to it. So what we see is getting more into this space and then combining that with our research and having a much more integrated solution. You could start to more seamlessly combine
you know, full workflows, whether that’s a, you know, your, your matter management workflow, or you’re starting to go from matter management and to have to draft something and research falls into that. We could start to provide a much more integrated, seamless experience. So you’re not hopping from one tool to another throughout your day and throughout your workflow. So that’s where, where we see the potential.
Greg Lambert (18:21)
are you finding that your customers are coming along for this? Are they also understanding kind of the transition more of a holistic workflow rather than I’m using Bloomberg for this one thing and then I’m jumping out? Are they catching on?
Bobby Puglia (18:36)
Yeah, I think they’re starting to and a lot of that that goes along with introducing the biggest things when you’re introducing new workflows is change management. How do you get your clients to understand what problem you’re trying to solve, how they should be using it and really, know, the hardest thing is to get someone to change their behavior on a day to day. So the answer to that is yes, they are getting it.
I fully expect that it’s going to take time for them to start to make that adjustment and adopt it and really get the full value out of how they could use these tools, very customized for them and not just in a generic way.
Marlene Gebauer (19:12)
So I want to address the elephant in the room, the elephant being Microsoft. How are you looking at Microsoft as sort of where attorneys work and how did the new Bloomberg tools and the Bloomberg roadmap fit into that?
Bobby Puglia (19:17)
Yes.
Yeah,
great question. You know, the way we really see it, it’s an enabler. You know, we’ve over the last, as an enabler, you know, the work that we’ve been doing to our platform over the last several years is really geared towards being in position where we could meet our clients where they already are. So, you know, when it comes to changing behavior, if you are drafting, you’re most likely drafting in Word.
Marlene Gebauer (19:38)
Microsoft the enabler.
Greg Lambert (19:40)
Hahaha
Bobby Puglia (19:58)
So we’ve been working on bringing our technology and being able to expose our technology much more in the Microsoft ecosystem. So whether that would be Word, things that we are looking at as a co-pilot becomes to become more more adopted. So that’s very much been part of our thought and strategy.
Greg Lambert (20:15)
like Microsoft the enabler sounds better than Microsoft the impaler. that’s. So, so Bobby, one of the things that especially when it came to legal research, legal research is probably one of the hardest things for generative AI to attack. And yet it was one of the first things that people immediately wanted and what providers are providing us.
Marlene Gebauer (20:20)
hahahaha
Bobby Puglia (20:20)
Yeah.
Greg Lambert (20:43)
But it also, you know, we’ve had issues of accuracy and reliability. you know, without having to get too deep into the weeds here, how is Bloomberg Law ensuring that the answers that these new tools are providing meet your high standards of accuracy for legal professionals that you have and we have?
Bobby Puglia (21:08)
Yeah, that’s probably the most important question that you ask there. You know, we took a very deliberate approach. We tried to be very patient in not putting these tools out there until we felt that they were ready for them. They passed all of our testing and our benchmark for that. So, you know, the way that we really approached that is it was a multi-prong approach. Number one is we rely very heavily on our in-house subject matter experts.
to define what good looks like. And more importantly, they’re able to identify the nitty gritty details and the gotchas as you get deeper into generative AI and answering questions that if you don’t have that domain expertise, you’re gonna overlook that. And that’s where you start generating some bad answers. So that is first and foremost. We took the approach of really building out a robust prompt shield and guard rails. So really our AI safety.
So, for example, we will monitor if you are asking questions or you’re asking to do something that is not within the domain of our product. We will politely tell you that we are not gonna answer that question for you, because it doesn’t fit in there. We also, within our guardrails, as we return answers, every answer we return is based on our content and our source materials. So we also are able to validate answers across those sources.
to make sure that they are grounded in truth. And again, we have no problem with not returning an answer if we are not confident that we’re giving you a good answer. We also have developed a robust internal benchmarking process. So every feature and tool that we use, we go back to those subject matter experts and figure out what is the accuracy levels we need, because it could be different for different types of problems. And then we set those benchmarks and have a regular process for going.
and testing the output of the models and over time making sure it doesn’t drift. And finally, we have extensive human review, testing, red teaming, and all of the other really best practices and standards that you would expect.
Marlene Gebauer (23:11)
it’s good to hear that basically, you you’re benchmarking your subject matter experts, you know, your human testers that you basically are keeping humans in the loop in terms of the final output.
Bobby Puglia (23:21)
Yes.
Yeah, absolutely. And it’s also being comfortable and being able to tell one of our clients that are using our product and giving the answer of, we can’t answer this, I’m sorry, instead of trying to force an answer.
Marlene Gebauer (23:35)
So, Bobby, you’ve been around the legal tech and information industry for a while now, so you definitely, I’m sure, have some visions here. So we’d like you to pull out your crystal ball and look into the future for us and see what you think are going to be some big changes or challenges for the legal industry over the next couple years.
Bobby Puglia (23:40)
Yeah.
Yeah.
Yeah.
Yeah, great question. So, you know, to answer that question, I’m going to put on more of my engineering hat. So I’m an engineering background and that’s how I started in this space. So, you know, the way I see it is accessibility to some of the core AI technologies evolving and, you know, the barrier to access it and start to build some of your own solutions has really dropped down. So…
You know, the way I would look into my crystal ball, I think you’re going to continue to see law firms invest and improve in their technical skills, whether that is very specific to, you know, upskilling and engineering and data capabilities, or just getting their, you know, current staff just more deeply trained in this area. And, you know, I think you could see it a bit already when you raise Microsoft there. I think there’s going to be additional demand for
different ways of accessing our content and accessing our technologies rather than just straight through our product and our web-based product. So those are the things that we’re thinking about and a trend that we’re following closely.
Greg Lambert (25:09)
Well, Bobby Puglia, Chief Product Officer with Bloomberg Industry Group. We want to thank you very much for joining us. I know this is kind of a last minute thing to get you onto the show, so we appreciate you making time for us.
Bobby Puglia (25:24)
Thanks for having me. Pleasure.
Marlene Gebauer (25:27)
And of course, thanks to all of you, our listeners, for taking the time to listen to the Geek in Review podcast. If you enjoy the show, share it with a colleague. We’d love to hear from you, so feel free to reach out to us on LinkedIn.
Greg Lambert (25:38)
And Bobby, I’ll make sure to put links on the show notes for all of this. But if listeners want to learn more or reach out, what’s the best way for them to do that?
Bobby Puglia (25:49)
Yeah, to learn more, I’d say the best way is you go to pro.bloombergloa.com and you can always find me on LinkedIn.
Marlene Gebauer (25:57)
And as always, the music you hear is from Jerry David DeSicca. Thank you, Jerry.
Greg Lambert (26:01)
Thanks, Jerry. All right, Marlene, I’ll talk to you later.
Marlene Gebauer (26:03)
Okay, bye bye.