5 Lessons for Responsible AI in Quality and Knowledge Management

00:00:00:21 – 00:00:38:11
Adam Pasch
Laura. So we’ve been rolling out a lot more AI programs, features in client work, in internal things in our personal lives. It’s ubiquitous. You’ve been focused both on knowledge management and now more in quality control. I’m just curious from your perspective, how how do these things relate? How does AI in its various forms improve or possibly hurt quality?

00:00:38:12 – 00:00:42:22
Adam Pasch
Help! Help me understand how I can kind of merge both these ideas together.

00:00:43:00 – 00:01:12:23
Laura Dunning
Yeah, this is how I see it. So I think AI will be a great tool for quality management because as the quality manager knowledge Manager at UNCOMN, I do a lot of data analysis, reporting, process review, internal audits, things like that. And AI is a good administrative tool to kind of help do some of those little tasks that I don’t have, you know, takes a lot of time so shortens up my time.

00:01:12:23 – 00:01:34:10
Laura Dunning
So I can do continue improvement and like risk management, things like that. In fact, I’ve used AI recently to help produce my audit plans. So it’s kind of shortened up the time to where I’m not spending all this time creating these documents. So that’s worked out really well.

00:01:34:12 – 00:01:53:09
Adam Pasch
So in a workflow like that, what AI can be mean so many different things, right? In what ways? It sounds like you have a standard workflow in some areas that you’ve cut a lot of time or complexity out of. Walked me through a little bit of it.

00:01:53:10 – 00:02:21:11
Laura Dunning
So actually one of the things that came out of our last audit was the external auditor was like, you know, you do really good audits and everything, but a lot of your questions aren’t open ended questions. I kind of stick to the whole what the requirement says. And he and so I was able to take some of that, put it in AI and come back with some more laid back, open ended questions.

00:02:21:11 – 00:02:33:06
Laura Dunning
So now I recreated my audit plans. So it still follows the requirement, but it’s more of a conversation than it is, you know, are you marking these boxes? You know, it kind of puts the two together.

00:02:33:07 – 00:03:08:23
Adam Pasch
Yeah. No, I really like that because there’s a whole science behind asking interview questions, removing bias from questions, keeping them open ended versus closed close a closed question. You’re you’re introducing a lot of. Like guided outcome where you end up based on how you frame the question is the answer that you get. And so it’s not as informative, but not we don’t all have time to go learn that skill of turning into open ended questions.

00:03:08:23 – 00:03:29:07
Adam Pasch
So I love that like very tactical use of and it sounds like you’re using generative AI like a large language model to do that, because also that means you’re feeding it enough context where it’s not going off wildly doing something super generic. You’re probably getting answers or well defined and specific.

00:03:29:08 – 00:03:52:01
Laura Dunning
Correct. Because I can go in there and I can tell it, okay, I need an audit plan for configuration management related to CMI level three. You know, and it’ll kind of split out. So then I can look at it and see. Yeah it is it is following the standard. So it is giving me questions that aren’t totally off the rail.

00:03:52:03 – 00:03:52:13
Laura Dunning
Yeah.

00:03:52:14 – 00:04:13:01
Adam Pasch
So along the way it couldn’t couldn’t possibly have been perfect the first time out. What were some of the mistakes that you saw that either you made it came back with bad results. What were some of those kind of early errors that you’ve been able to apply some learnings to?

00:04:13:03 – 00:04:55:20
Laura Dunning
So one of the things so for CMI related, it keeps coming back with level two or version two I’m sorry, version two of CMI. We’re actually in version three of CMMi. Some of it I can still use because none of it’s like version to the other didn’t change. But that’s one of the things like AI will never replace like a quality manager or manager in general, because you’re it actually is going to help elevate that person’s skill, because they’re going to have to be able to look at AI and analyze the results that come out and know, yes, I can use this.

00:04:55:20 – 00:05:05:06
Laura Dunning
This is good information. This is not good information. So and another way, AI will kind of help skill somebody up because they’ll be able to have to do that.

00:05:05:10 – 00:05:18:07
Adam Pasch
So and you mentioned kind of analyzing the results. Can you walk me through a little bit of what does it mean as a quality manager to be analyzing some of the results of these audits?

00:05:18:09 – 00:05:37:14
Laura Dunning
Well, like if you’re going to use AI, you need to make sure that what it’s telling you is matching reality. And it’s not just, you know, taking information and, you know, spitting it out willy nilly, I guess you can say sure. So yeah.

00:05:37:16 – 00:06:00:10
Adam Pasch
And what do you end up doing with the outputs of this? How does it get implemented into kind of a positive feedback loop of, of quality improvements or quality sustainment? What’s the overall goal of after you go through a quality audit that you’re trying to impact on the project?

00:06:00:12 – 00:06:24:16
Laura Dunning
Well, just the what you’re supposed to do with these audits is making sure that the team members are following the process on how UNCOMN does things. So you ask the questions you and as you’re asking these questions, you’re you’re feeling whether, yeah, they’re following the process. But are they understanding what they’re doing. Do they understand why they’re doing it.

00:06:24:16 – 00:06:25:08
Laura Dunning
Things like that.

00:06:25:08 – 00:06:35:15
Adam Pasch
So understood. What are some of the things I should have asked you?

00:06:35:17 – 00:07:04:07
Laura Dunning
Well, one of the things I have been concerned with, with the whole AI thing is, on my knowledge, manager hat, is to have governance around AI. I think we need governance around it. Absolutely. We need to make sure we get the right tools. We get policies in place, we get training in place so that our team members use it responsibly and don’t just take the information out of there without looking at it.

00:07:04:08 – 00:07:05:23
Laura Dunning
You know what?

00:07:06:01 – 00:07:47:17
Adam Pasch
I can only imagine from a knowledge management standpoint, right? Anytime in SharePoint, there are multiple versions of multiple documents that can be wildly different. Right. Are you citing the right material and are you cleaning up SharePoint? Not you. But are we all, as an enterprise, keeping SharePoint cleaned and optimized? So knowing that it is getting every bit of it is getting read regularly, maybe not by the human, but by the machine that the human is using to generate answers to a whole array of questions.

00:07:47:18 – 00:08:04:15
Adam Pasch
It’s reading all of those documents regularly, right? Correct. And so, yeah, that’s absolutely an area that I have. I’m not going to say concern, but a focused attention for ongoing improvement for sure.

00:08:04:17 – 00:08:31:22
Laura Dunning
Now Dustin Clifton is working on setting up something him and Brad Bowling. We’re working on creating something where, you know, you type in was the PTO policy, and it goes out to the team member handbook and it pulls the answer out for you. Things like that. So I talked to Dustin about one of the things we need to start doing is all our processes need to be reviewed at least once a year.

00:08:31:22 – 00:08:47:09
Laura Dunning
And so absolutely. Can you build an AI to like trigger these, you know, where minors hey, go out there and review this, you know, update the revision history, things like that. So that’s a lot of stuff that’s being worked on right now.

00:08:47:10 – 00:09:07:22
Adam Pasch
Yeah I mean it was really basic policy controls and workflow timing to ensure that those things, those things happen especially because all the versioning is dated. Right. So that’s fantastic, right. Did you have questions you wanted to jump in with?

00:09:08:00 – 00:09:31:22
Royce Lippert
I love that as a knowledge manager you’re like that. You hit on it like the the key for the data to be accurate. But the AI is not going to replace anyone, right? It’s a tool to use. It’s not a tool to replace. And some of it is because of the accuracy of this knowledge manager I can see I could believe the the concern you could have because you want the same answer, right?

00:09:31:23 – 00:09:51:02
Royce Lippert
Like if I asked the bot or anything a question, I’m sure as a knowledge manager, like if Adam asked the same question, you should get the same answer, correct. What’s the policy? We shouldn’t get two different answers. And so trying to have the policy in place like you talked about. So I think that’s that’s key to have folks, folks like you in place to make sure that the company is following the policy setting.

00:09:51:02 – 00:09:58:19
Royce Lippert
And it’s the AI is I think a lot of people fear it as a replacement. I think folks are quickly finding it as models.

00:09:58:21 – 00:10:12:11
Laura Dunning
And I have to admit, I was the whole AI thing in the beginning. I was like, oh my gosh, this is going to be a disaster, you know? Because that’s what I was thinking. It’s like people are going to be putting all this information in. It’s going to be spitting out all this other information. It’s like, is it good information?

00:10:12:11 – 00:10:13:05
Laura Dunning
Is it not?

00:10:13:06 – 00:10:35:20
Royce Lippert
And I love you mentioned it gives you the time to focus on the important stuff rather than the administrative, if you will. There are some things that that just take time, but it’s not necessarily hard. It just takes time and it has to be done. You get to now actually focus on the risky parts of it, which gives you time to now we get to grow or reduce or at least find a way to mitigate the risk.

00:10:35:22 – 00:10:36:18
Royce Lippert
All right. Yeah.

00:10:36:23 – 00:11:24:19
Adam Pasch
Great. Well, and I think in a part of what you had also just said, I got to pull it back up in my brain. Here was about this specific answers that need to be highly attuned, that this gets to the point of we don’t need to have one large language model, one bot to rule them all that we can have, especially when when there’s zero failure, really narrow, focused, small models that are highly trained on a small selection of source material that is designed to do one thing, be asked questions about stated policies, and that’s it.

00:11:24:19 – 00:11:43:03
Adam Pasch
That same bot does not need to be doing all of these other things. We can have a series of these specialized tools as well as having some general tools. So that’s fantastic. Yeah. So that’s all I got. All right.