Shadow AI Is Already in Your Enterprise: Turn Grassroots Into Governed Adoption

00:00:00:01 – 00:00:25:10
Adam Pasch
Before we get into a lot of the kind of AI project details. We’ve been talking a lot recently about greenfield and brownfield projects, the idea of like brand new projects or new ideas versus legacy systems, or like all these different systems that are integrated together. How how should I be thinking about greenfield versus brownfield?

00:00:25:12 – 00:00:47:00
Bryce Schumacher
So, I mean, a greenfield project you’re starting with with a blank page. And so if if you’re vibe coding, basically your AI gets to design that from the ground up and you’re going to have no tech debt.

00:00:47:02 – 00:00:59:00
Bryce Schumacher
It’s probably going to be able to come up with a coherent plan for what you ask for. It’s going to build it. It’s probably going to work.

00:00:59:02 – 00:01:16:09
Bryce Schumacher
But that’s that’s the idea of Greenfield is it’s it’s a brand new project. You’re not coming in with any tech debt. So your your business processes for that should be more design based.

00:01:16:11 – 00:01:19:02
Adam Pasch
What do you mean by design-based?

00:01:19:04 – 00:01:21:21
Bryce Schumacher
So.

00:01:21:23 – 00:01:53:15
Bryce Schumacher
Kind of like product or product selection. Problem selection. If if you have a blank page you can you can build anything. But it also always comes down to what should you build. So you need to be thinking about what new tech debt am I creating? What new complexities am I creating? Am I creating value? Will this fit into my business processes?

00:01:53:15 – 00:02:21:09
Bryce Schumacher
Will this fit into my teammates’ workflows? Because, I mean, you can build almost anything. It’ll look cool. We’ve seen all the examples and it’s a lot of fun. Whereas a brownfield project is usually going to be something that’s already in production. So it’s we’re coming in with all of our tech debt. We’re coming in with all of our process debt.

00:02:21:11 – 00:02:27:20
Bryce Schumacher
Sometimes we have customers relying on this in production.

00:02:27:22 – 00:02:35:10
Bryce Schumacher
So it’s going to be more of a change management issue.

00:02:35:12 – 00:03:04:03
Bryce Schumacher
So a lot of your research at the beginning is going to be about what do I have? What is the change? What are the second-order effects? What are the third-order effects? How can I verify that this is going to work? How do I verify that I’m doing the change that was asked? How do I get through my CI/CD pipelines and stuff like that?

00:03:04:03 – 00:03:32:02
Bryce Schumacher
So you’re typically not going to knock out all of your tech debt every time a change request comes in. So you’re going to have to figure out how to work with the systems already in place, you know, make almost the smallest change possible to create that value, to have limited second- and third-order effects, limited risk for your deploy.

00:03:32:04 – 00:03:51:18
Bryce Schumacher
So in the sense of you’re defining a box and engineering inside of it, greenfield and brownfield are the same. But it’s it’s really your brownfield project’s going to have more interfaces on that box and more risk involved usually.

00:03:51:20 – 00:04:38:23
Adam Pasch
Right. Especially because this is already something that is doing highly important business-critical, mission-critical things that if we roll out a new feature, new capability, and it doesn’t work well. Greenfield okay, we’ll go back into production. You didn’t break anything. Brownfield. You may have you may have sustained substantial risk and consequences. So what are we doing? Maybe this is jumping a little bit too far, but in in brownfield projects then where there are these real-world consequences to rolling out a new capability and not having factored in all of these other dependencies, what are we doing there to to mitigate that risk?

00:04:39:01 – 00:05:12:07
Bryce Schumacher
So I mean, typically you’re doing doing some research at the beginning. So if if a change has been asked for, you need to go look at the system itself, what all the system is connected to, you know, who’s relying on it, how large the change is going to be. And it and it varies wildly. Right. So anything in production is just something somebody is using basically.

00:05:12:07 – 00:05:30:08
Bryce Schumacher
In the software world, there’s a huge difference between an internal dashboard versus a banking system versus an aircraft that’s in flight when you’re making a change.

00:05:30:10 – 00:05:43:06
Bryce Schumacher
So it varies quite a bit. But like these should already be documented business processes.

00:05:43:08 – 00:06:24:09
Bryce Schumacher
In the CI/CD pipeline, we have very explicit processes where you’re you’re doing a code review, you’re running tests and passing them. You’re building images. You’re doing a database migration, moving everything forward, putting that in a in a staging environment. And again, like how much testing you do kind of comes down to the risks. And those those pipelines look very, very different depending on what what type of system you’re working on.

00:06:24:11 – 00:06:57:02
Bryce Schumacher
But you know, before all this AI coding, a lot of that was documented anyway. And the more risks you have, the more documentation and the more strict the process should have been. So we’re not changing a lot there. You’re still going to do the initial planning, the initial research, the verification, the validation, your staging environments, your your trial deploys and stuff.

00:06:57:04 – 00:07:08:14
Bryce Schumacher
Sometimes you have human users testing that out for you, sometimes you don’t. It just depends on the criticality.

00:07:08:16 – 00:07:44:07
Adam Pasch
Awesome. So okay, let’s think of a scenario then where what we’re seeing happening everywhere right now is I’m a leader in a government organization, and I’m looking to gain either efficiency or effectiveness by bringing more AI into my organization. In reality, I have a lot of executive leadership coming to me to say, you need to AI-ify everything, and they’re probably thinking of LLMs only when they say AI, right?

00:07:44:08 – 00:08:14:15
Adam Pasch
We obviously know that it’s a much broader term, but help me understand how we should help somebody that’s, you know, getting the onslaught of this tasking of go, go AI-ify everything. I can’t AI-ify everything all at once. How should that government leader be thinking through how to chunk and start making progress?

00:08:14:17 – 00:08:49:18
Bryce Schumacher
Yeah. So typically either C you’re doing an inventory of your organizational processes. And you know we see both sometimes you want to inventory all of your processes. So you can see which ones don’t make sense in an AI world. Sometimes people really want to want to throw it all out and start over. But if you don’t know what you’re throwing out, it’s going to be very hard to architect that new world.

00:08:49:20 – 00:08:54:11
Bryce Schumacher
And then on the other side of that is.

00:08:54:12 – 00:09:10:23
Bryce Schumacher
What if we just want to stick our toe in what are our least risky business processes? Which ones are well documented? Low variance? Which ones have repetitive tool usage?

00:09:11:01 – 00:09:33:11
Bryce Schumacher
And you know, at this point we might as well call them technical processes. So think of these being done more as a technician as opposed to kind of open-ended engineering or even like design or art.

00:09:33:13 – 00:09:49:03
Bryce Schumacher
The technician might need a lot of general knowledge, but they’re following, you know, a well-documented plan. They’re lining their tools up with that plan once they see the situation.

00:09:49:05 – 00:10:28:15
Adam Pasch
Would the example there be something like there’s a lot of government governmental departments that have very prescriptive laws and compliance that must report daily, weekly, monthly, quarterly, annually, depending on what it is. Standard reports, deliverables, outputs, assessments, etc. but there is a very specific legal compliant guideline to follow with standard inputs and outputs that forgive the term. But like cranking of the monkey grinder, that’s the kind of thing that you’re talking about.

00:10:28:16 – 00:10:37:02
Adam Pasch
Least risk. Most repeatable, most common tool use. Is that the kind of thing that I should be visioning, or maybe it’s something a little bit more nuanced?

00:10:37:04 – 00:11:14:08
Bryce Schumacher
I mean, it it could be. So you’ve you’ve hit on, you know, well documented repeated process. Now the, the criticality of those reports is important. So you could be looking more as like a second or third check to whether those reports got done. In the middle, you’d be more compiling the information that a human needs to do

00:11:14:08 – 00:11:27:03
Bryce Schumacher
those reports. Or if they are of lower criticality, your AI could be writing a lot of those reports for you.

00:11:27:05 – 00:11:40:03
Bryce Schumacher
So it again, I think it depends on what all is going to go wrong if that report’s not perfect.

00:11:40:05 – 00:11:53:01
Bryce Schumacher
But it I mean that should be part of the business process, documenting the risks that are associated with that report. Yeah.

00:11:53:03 – 00:12:07:11
Adam Pasch
Yeah okay. Yeah. I derailed you a little bit because before my tangent there, you were starting to go down the process a little bit I think of.

00:12:07:13 – 00:12:36:15
Adam Pasch
When you’re taking an assessment of or an inventory, I’m sorry of your organizational processes. Did you, did you have more kind of thought processes there of, of what’s going through? Because I think you were also thinking about like software development processes as well, as opposed to where I took you onto this like report deliverable example.

00:12:36:17 – 00:12:49:10
Bryce Schumacher
No, I mean, I think as long or I think we want to bias towards technical processes and so it doesn’t have to be software. Software is just the easy example.

00:12:49:12 – 00:13:27:10
Bryce Schumacher
Your organization has lots of different processes. And so you have planning and management, different administrative stuff. And you also have a lot of technical processes. I think that we like to bias towards the technical processes for AI because they’re easier to document. More of the parameters are known, and we can work that into kind of the AI workflow.

00:13:27:12 – 00:13:34:05
Bryce Schumacher
So do we want the AI?

00:13:34:07 – 00:13:42:11
Bryce Schumacher
Doing our annual strategy? Do we want the AI doing our quarterly planning? Probably not as a larger organization.

00:13:42:12 – 00:13:43:02
Adam Pasch
I’d hope not.

00:13:43:03 – 00:13:51:15
Bryce Schumacher
Yeah, you can imagine kind of in a solo entrepreneur scenario.

00:13:51:17 – 00:14:20:17
Bryce Schumacher
There could be some lower-risk strategy and planning that could be done. Sure. Especially if you can if you know, if it’s a single business with a single objective that’s selling one product, you know that strategy might be easier for the AI to figure out. Sure. But once we start moving a lot of people around, the market we’re in is changing, our product is changing.

00:14:20:19 – 00:14:25:20
Bryce Schumacher
We probably just want to leave that as a human process. Yeah.

00:14:25:22 – 00:15:13:01
Adam Pasch
And would you say that that is because when I’m a single entrepreneur, I think you’re kind of describing a little bit of this, like B2C product sale, right? That if I had a software product, for example, it is well documented in many, many places in all of the foundational models of this, like basic SaaS go-to-market. And if it was a dropshipping, like there’s just so much content out and it doesn’t matter if you were selling HR software or finance software, there’s just so much out there that you’re just running the same basic strategy, or you’re choosing between three is not going to be it’s not going to derail that solo entrepreneur as

00:15:13:01 – 00:15:45:22
Adam Pasch
opposed to now I’m a governmental organization or a large enterprise that comprises tens of thousands of staff and a highly nuanced, specific mission. There’s not a lot of other organizations exactly like me in my situation, and so that’s where it does need to be bespoke. And so it should still be. Those higher-level pieces are done inherently by humans, and that there’s going to be other, more repetitive work.

00:15:45:22 – 00:15:51:04
Adam Pasch
That is what we’re prioritizing for AI enablement.

00:15:51:06 – 00:16:12:23
Bryce Schumacher
Yeah, I mean, a lot of your higher-level work is deciding what the business processes should be, right, and how to define them and stuff like that. So if we’ve said the AI needs to follow your documented business process,

00:16:13:01 – 00:16:18:19
Bryce Schumacher
It’s going to have trouble doing that if you ask it to come up with a business process itself.

00:16:18:20 – 00:16:55:05
Adam Pasch
I don’t want my specifically we you know, if an organization spent so much time on its architecture in integrating dozens, hundreds of different systems together, I don’t want it inventing its own processes. Right. We spent a lot of time nailing these down and gaining agreement across departments and especially for global enterprises. Yeah. So how how do you ensure that it doesn’t go wildly off and start inventing its own approaches and processes and rules?

00:16:55:07 – 00:17:23:08
Bryce Schumacher
So one way to keep the AI from inventing a new business process is to give it your business process. Right? So it’s a really popular thing to do in an AI harness is to let it plan. And that’s that’s fine. But you can give it a checklist, you can give it the business process, give it the tools.

00:17:23:10 – 00:17:43:04
Bryce Schumacher
And that plan could be it aligning its available tools with the plan or business process that you gave it. Or you could give it nothing and let it invent the business process and how to apply the tools to the business process that it just made up on the fly.

00:17:43:06 – 00:18:09:13
Bryce Schumacher
So it’s it’s still, you know, there’s still some risk there. You know, there will still be risk if you had a human do that. But you’re giving it a much smaller task now. Yeah. Instead of asking it to figure out the whole problem, you’re telling it how you normally work the problem, and then you’re just letting it align the tools that it has with that workflow that’s already documented.

00:18:09:14 – 00:18:28:00
Adam Pasch
So and maybe this is a broad audience here. And so maybe we take a little bit of a step back. You just went through LLM, tools, and harness. Could you, let’s let’s parse this out a little bit.

00:18:28:02 – 00:18:58:21
Bryce Schumacher
Sure. Yeah I mean I think we mix the terms together all the time. I mean, you hear AI and LLM and and agent and and bot. So and I think it is super important to have a little bit of understanding of what those actually are. So the LLM, the large language model, is the neural network that’s at the heart of almost everything we’re doing with AI right now.

00:18:58:23 – 00:19:25:11
Bryce Schumacher
And the LLM is trained on all of our language. It’s trained on all of our books, our internet. Right. A lot of our expert writing. But it’s not doing a whole lot of the things that we associate with AI. It’s taking an input text sequence, and it’s just predicting the next character.

00:19:25:13 – 00:19:56:16
Adam Pasch
Which I think is a thing that gets glossed over in the common vernacular discussions of this is that it’s it’s statistically predicting. It doesn’t understand what it’s saying, but it is constantly giving us words that make it appear that it did understand. And it’s just making statistical inferences of strings of words and characters from these tokenization processes.

00:19:56:18 – 00:20:22:22
Bryce Schumacher
Yeah, it it is spitting out a string of characters. Yeah. And so a lot of the stuff that we see people doing with AI, that’s not necessarily a string of characters. So we see it doing web searches, we see it doing the plan that we were talking about. We see it using our computers. We see it using other AIs.

00:20:23:00 – 00:20:28:15
Bryce Schumacher
So these are tools, whether they’re on a server or on our laptop.

00:20:28:17 – 00:20:32:22
Adam Pasch
Yes. Okay. This is a great nuance. Yes.

00:20:33:00 – 00:20:38:22
Bryce Schumacher
The LLM is typically trained on the idea that it will have access to tools.

00:20:38:23 – 00:20:40:09
Adam Pasch
Right.

00:20:40:11 – 00:21:09:18
Bryce Schumacher
But we actually have something called a harness that tells it which tools it has available to it. And the LLM does not use the tools. The LLM can only spit out a string of characters. So essentially the harness is going to tell it: You have these tools available, and here is the string of characters you need to spit out if you want us to use this tool for you.

00:21:09:19 – 00:21:31:08
Bryce Schumacher
So basically, the LLM is going to ask nicely for a tool to be used, right? And the harness is going to pick that up and use the tool. So even even thinking models you can think of that as a tool. So it’s essentially the LLM asking the harness to let it use itself.

00:21:31:09 – 00:21:33:18
Adam Pasch
Yes.

00:21:33:20 – 00:21:37:01
Bryce Schumacher
And.

00:21:37:03 – 00:22:01:01
Bryce Schumacher
The same with the web search. So if it wants to search the web, it’s it’s basically just spitting out a string of characters that says, hey, please go search the web for me, send me back what you find. So the harness is every time it talks to the LLM, it is giving it all sorts of information in the input sequence.

00:22:01:03 – 00:22:02:20
Adam Pasch
Beyond what I have prompted it with.

00:22:02:21 – 00:22:06:22
Bryce Schumacher
Right. Absolutely. So.

00:22:07:00 – 00:22:35:12
Bryce Schumacher
You ask it a simple question, maybe 20 words, and then it’s going to go put another 2,000 words on there. It’s going to say, here’s the harness you’re sitting in. Here’s all the tools available to you. Here’s how you use them. And then also you can configure your harness to have lots of rules and hints. And so it’s going to append all those rules on each turn as well.

00:22:35:14 – 00:22:44:20
Bryce Schumacher
So essentially we’ve already you’re already doing most of what we’ve talked about.

00:22:44:22 – 00:23:15:18
Bryce Schumacher
So it’s it’s no stretch at all to just go ahead and append your business process, all of your team rules and everything you want done in a repeated way. So in our harness, we can put all of our business processes, all of our rules. It’s going to add in there all the tools that it has access to. And so it’s it’s already compiling it all together.

00:23:15:19 – 00:23:27:18
Bryce Schumacher
This is how all of the popular tools we use already work. We just need to align them with our business. Got it.

00:23:27:20 – 00:23:33:00
Adam Pasch
So how do we go about doing that?

00:23:33:02 – 00:23:54:22
Bryce Schumacher
I mean it can depend quite a bit on your business. So one is just documenting and selecting the right processes. So knowing this is the task I’m doing right now and this is the process that I follow.

00:23:55:00 – 00:24:34:05
Bryce Schumacher
So if we document that, put it into the harness in writing. And then we also kind of need to follow our own business processes as well. I mean absolutely be creative. You can try new things, but when you start repeating stuff that your business is running on, document that and provide that documentation to that LLM. There’s additional testing and validation and stuff that you can add to it, but that’s just part of the business process.

00:24:34:05 – 00:25:03:15
Bryce Schumacher
So when you tell it, this is the process you need to follow. You can also tell it these are the tests and the validation that we’re going to do when you’re done. Yeah it’s just an extension of that. Yeah. And they are super creative. They’ll find ways to get around the tests and the validation. If you’re in a if you’re in a critical role or you’re doing a critical task, you’re going to have to take some of that away from the AI and and do that verification independently.

00:25:03:17 – 00:25:30:22
Adam Pasch
You’ve mentioned that before around like one of the things that I had always been doing is, you know, having it cite sources used and giving like a confidence of how much hallucination may have come into this thing. But then I get this sense that it’s just giving me the inverse of the answer that it gave me to prove that it was right and what it did.

00:25:30:22 – 00:26:04:03
Adam Pasch
But it’s like it wrote the answers to the test. It read the questions to the test, and then it just inferred the reasons of why it was right, regardless of the actual rightness of the answer. And I think you had been describing separately of a way of separating those things out a little bit so that the evaluation criteria can actually still remain independent, or maybe not evaluation criteria, but like, help me with the I’m missing the right phrase here.

00:26:04:05 – 00:26:41:16
Bryce Schumacher
Yeah, we see hallucinated citations. We see misaligned citations. Yes. The the way that the AIs are trained, they’re they’re rewarded basically for tricking the trainer into thinking that they’re correct. Right. So they’re often going to sound super confident. We’d love to talk to people who sound confident. And and we’ve seen it several times where it’ll answer a question, we’ll ask it, where did you get that question from?

00:26:41:18 – 00:27:06:00
Bryce Schumacher
And it can make up a document. It can cite the wrong document. So often we will ask it, you know, what what page or what document did you get that answer from. And then we will route the page of that document through a completely different system that the AI has no access to, and show that to the human.

00:27:06:02 – 00:27:16:08
Bryce Schumacher
Got it. So essentially we’ve taken that verification step away from the AI so that it is harder for it to trick us.

00:27:16:08 – 00:27:31:16
Adam Pasch
So it’s not only important to make sure that, again, within that harness, that the AI knows what tools it has access to. It is also important to specifically deny it access to other tools and bring it in separately as a part of this overall process.

00:27:31:22 – 00:27:45:12
Bryce Schumacher
Yes. If you’re if you’re in a critical system, you’re making a critical decision. There’s a high risk. You absolutely are going to block the AI entirely from certain steps in the process.

00:27:45:13 – 00:28:11:08
Adam Pasch
Yeah. I mean, it feels like this is one of those unknown risks that many organizations have baked in right now, not realizing that this is sitting right under the surface, that if they haven’t separated out, this aspect of it, that they’re they’re operating with a certain level of risk that they don’t realize right now.

00:28:11:10 – 00:28:45:01
Bryce Schumacher
Yeah, I mean, it’s certainly possible. One of the one of the ways that it’s sneakier is what we just said. So if you leave a critical decision to a human, you know, they can get it wrong. Sure. They didn’t necessarily spend their whole lives training to convince you that they’re right all the time, though. So, yeah, it absolutely can sneak up on you, but it’s the same risk.

00:28:45:03 – 00:28:55:00
Bryce Schumacher
It doesn’t matter if you had the human make the decision, or if you had the AI make the decision, it doesn’t change the risk of a of a bad decision.

00:28:55:02 – 00:29:19:22
Adam Pasch
So kind of reminds me a little bit of on on the the two different sides of the argument around, well, you can’t hold the AI to the standard of it has to always be perfect because no human being is ever perfect and and that’s fair. But the flip side of like but when the human makes the error, they learn from it, right?

00:29:19:23 – 00:29:59:09
Adam Pasch
Like I won’t the all the things I know are only because I screwed it up before is really where I learned 90% of anything that I know, but that means that’s immediately training my actual intelligence. It’s getting into my foundational model every time I make a mistake. But that’s not necessarily true for the AI where it’s operating. This is the piece that I don’t fully understand personally of the difference between the foundational model and the chat response corrections that I’m doing on the front end, and some of these other kind of operationalized middle layers that might be running.

00:29:59:14 – 00:30:26:23
Adam Pasch
Can can you help tease out a little bit for me then, where? Yes, we’re accepting, you know, error risk, whether it’s the human or the AI. And where is this difference of it’s getting reinforced where, where our systems have now actually learned and will continue to apply that learning versus we’re just going to continue to have this risk of that same error happening over and over and over again, because if that happened with a human.

00:30:27:00 – 00:30:44:11
Adam Pasch
Sorry to belabor the point, but if that happened with a human that would refuse to learn from a lesson like we’re going to we’re going to we’re going to we’re on a path here of a PIP, and I’m not going to put I can’t put the AI on a PIP. So yeah, there’s there’s a lot baked into that question.

00:30:44:13 – 00:30:47:14
Adam Pasch
Over to you to where you’re going to take it.

00:30:47:16 – 00:31:14:01
Bryce Schumacher
I mean, I think we do see a lot of people that are ready to fire their AIs, maybe, maybe prematurely. Yeah. I think people often overestimate how quickly the AIs are learning. Really? You’re only getting a new model every couple of months. Sure it is. It is not learning in real time off of how you’re talking to it right now.

00:31:14:01 – 00:31:37:18
Bryce Schumacher
There’s an illusion of it learning quickly. And that’s that harness we were talking about where when you when you go to ChatGPT.com, you’re not really talking directly to the LLM. You’re talking to a harness. And that harness is providing all of the chat history, all the documents.

00:31:37:20 – 00:32:05:17
Bryce Schumacher
And so it’s changing its answers based off of your chat history. So that input sequence to the model is changing. But it’s it’s not learning in real time. It’s just as dumb as it was 20 seconds ago. You’re right. So yeah, I see it all the time. It it looks like it’s learning, but really you’re only getting a new model every few months.

00:32:05:19 – 00:32:49:05
Adam Pasch
Understood. So going back a little bit where we were looking at this example of this government leader that is getting this direction that they’ve got to AI-ify everything. Right. And we talked through a little bit about the differences between or the definitions around LLM, tools, harness. We started going down through this process of understanding what the right with the right and wrong kind of processes or types of work are best to start bringing to it, right, and kind of establish the like.

00:32:49:06 – 00:33:08:14
Adam Pasch
Do my overall strategic planning? Probably not the place to start unless you’re a solopreneur, that’s starting with the most the most repeatable, most documented or least mission-critical risk areas are.

00:33:08:16 – 00:33:30:01
Adam Pasch
What would be next. Once you’ve, if I understand it right, you’ve identified highly standard, highly repeatable common tool use. Less critical. Where where are we going from here?

00:33:30:03 – 00:33:43:05
Bryce Schumacher
I mean, I think we said one of the first things to do is to look at your, your process documentation. Okay. So inventory your processes.

00:33:43:07 – 00:34:20:05
Bryce Schumacher
If you want to take a low risk approach, you will be looking at your lower risk processes. If you want repeatable outcomes, you’re looking at your lower variance processes. I think it’s really up to each enterprise which processes they go after. You know, I do I do think you can bring AI into certain parts of the planning process or something that’s not super technical.

00:34:20:07 – 00:34:30:21
Bryce Schumacher
It’s just the technical stuff is an easy example, because a lot of times you have explicit routines through code.

00:34:30:23 – 00:34:41:14
Bryce Schumacher
You’re doing a lot of the same stuff over and over again. You’re training to that process. So if you’re training to it, it has to be documented.

00:34:41:16 – 00:35:06:12
Bryce Schumacher
Now the the other argument is, well, if we’re in this new AI world, why would we go automate all of these human processes? Maybe we need totally new processes. And that does not change what we just said. You still need to inventory your processes to know which ones you’re getting rid of.

00:35:06:13 – 00:35:30:18
Bryce Schumacher
And there’s a bit of an argument. Well, if that if that process is so easy and so low risk, why are we doing it anyway? Right. So I mean, I think just take it as an opportunity to look at everything. Which process can we automate? Which ones can we get rid of? What new processes can we come up with?

00:35:30:20 – 00:35:43:03
Bryce Schumacher
Which ones are we missing? How can we change the enterprise, make the enterprise more efficient? It’s probably homework we should have been doing anyway, so now now’s our chance to do it.

00:35:43:04 – 00:36:13:08
Adam Pasch
This sounds like some of the architecture and engineering technical debt that most organizations have known, and have continued to deprive while they attack different parts of the technical debt. But I think what I’m hearing from you is that with some of the AI tools or some of that technical debt, that absolutely gets easier at getting after, which now maybe gives us more space to do some of this other process documentation and clarification.

00:36:13:08 – 00:36:38:02
Adam Pasch
Because the immediate question that came into my mind that you were describing, that was, what if the real process is not the process that we have documented? I think we’ve seen that frequently of of we wrote that process document five years ago, but six different teams and six different regions all do something wildly different because they’re getting after the outcomes that were needed.

00:36:38:04 – 00:36:58:00
Adam Pasch
But inherently they ended up doing it in different ways with different inputs in order to get the outputs that are expected of them. So in that example, I think, what would you say is the approach for that? I mean, I guess maybe that’s the reason why we got to document it.

00:36:58:02 – 00:37:05:06
Bryce Schumacher
Yeah. I mean, I think at that point you’re you’re moving up the architecture stack.

00:37:05:08 – 00:37:13:16
Bryce Schumacher
So all our architectures have stakeholders and and motivations.

00:37:13:18 – 00:37:46:05
Bryce Schumacher
So we I mean, we absolutely can move up from operational into, you know, the enterprise level, like what are the capabilities we need, what are our stakeholder needs? What opportunities and challenges do we have. Like like you said, the the business process is trying to either.

00:37:46:07 – 00:37:55:11
Bryce Schumacher
Fulfill the need for a capability or some other measurable outcome.

00:37:55:13 – 00:38:08:23
Bryce Schumacher
And if five different groups are doing it differently, they probably are still chasing the same outcome, right? So if you document all five, you’ve you’ve got a chance to actually compare them to each other.

00:38:09:00 – 00:38:11:07
Adam Pasch
That’s great. Take the best of all.

00:38:11:09 – 00:38:32:01
Bryce Schumacher
And if we’re just talking, you know, expensive tech systems and stuff, you know, we we have customers with hundreds of technical systems. And the only way to understand which ones are duplicates,

00:38:32:03 – 00:38:40:20
Bryce Schumacher
which ones are similar but not exactly the same, is basically to document them.

00:38:40:22 – 00:38:50:15
Bryce Schumacher
And one of the things that the AI actually is really good at is searching it.

00:38:50:16 – 00:38:51:21
Adam Pasch
Like massive searching.

00:38:51:21 – 00:39:05:03
Bryce Schumacher
Right? It can read way faster than we can. Yeah. And it doesn’t necessarily read it one character at a time. Like it can read a whole book series in one go.

00:39:05:04 – 00:39:06:14
Adam Pasch
Right?

00:39:06:16 – 00:39:17:23
Bryce Schumacher
And we have AIs where you you could take an entire book series, ask it eight facts, and it just nails it first try.

00:39:18:01 – 00:39:25:06
Bryce Schumacher
So now our documentation is easier to use.

00:39:25:08 – 00:39:35:02
Bryce Schumacher
And so one of the reasons why we wouldn’t want to be drowning in documentation is now gone. It’s fair. Kind of don’t have to read it anymore.

00:39:35:04 – 00:40:04:18
Adam Pasch
We just need to be confident in what we made. Yes. And as something changes, we need to update it. But now this whole idea of why did all this architecture. But then nobody looked at it. The shelfware. It’s like, well, no. Yeah, the human may not be looking at it, but the AI is going to look at it every single time because it is the only entity that is going to keep referencing the architecture of the way that we actually wanted people to have been referencing the architecture.

00:40:05:00 – 00:40:22:09
Adam Pasch
So. So this is a great thing of where now, like that is inherently work that a human doesn’t want to do right, but that the enterprise benefits from. To me, that sounds like the perfect type of thing that we outsource to the AI, the stuff that the human doesn’t want to do, but that does need to get done.

00:40:22:11 – 00:40:56:05
Bryce Schumacher
Yeah, and I don’t love writing documentation either. As much as I love reading it. But if if the AI is going to start doing the doing, the human can move up that architecture stack as well. Okay, so the human can be working more on system definition, process definition, kind of the what should the AI be doing. So once they’re freed up from the doing, they can move up the architecture stack as well.

00:40:56:07 – 00:41:19:00
Adam Pasch
So can I building more on that, how the AI enables the human to move up that architecture stack or the engineering stack. Can you kind of lay out a little bit more then of what the AI is really good at versus what the AI is not so good at? And when we say AI right now, are we only talking about with a large language model?

00:41:19:00 – 00:41:25:07
Adam Pasch
Is good or bad at? Or are we talking AI more broadly?

00:41:25:09 – 00:41:46:15
Bryce Schumacher
We can cheat a little bit and talk about the AI and the harness, the large language model in the harness, but it’s going to be a similar answer, so that LLM is really good at search and reading quickly.

00:41:46:17 – 00:41:53:07
Bryce Schumacher
It’s as crazy as this sounds. It’s actually really good at spelling and syntax.

00:41:53:07 – 00:41:55:16
Adam Pasch
Certainly better than me.

00:41:55:18 – 00:42:17:02
Bryce Schumacher
I mean, I there’s been times where I’ve coded for a month without it making a single spelling mistake, and then it makes one. And I’m so excited that I call somebody to tell them that it made a mistake. And then during the call, I realized I made the spelling mistake and it was just copying me.

00:42:17:04 – 00:42:29:09
Bryce Schumacher
So, I mean, there may be a few other things it’s good at. Like general knowledge.

00:42:29:11 – 00:42:43:10
Bryce Schumacher
But that list is fairly short. Things that it’s bad at. It is really bad at time estimation.

00:42:43:12 – 00:42:54:09
Adam Pasch
For plans. And how can I study for this thing in 12 weeks? I’m like, you’re out of your mind. That’s how long that’s going to take it. That’s going to take me 12 months, not 12 weeks.

00:42:54:10 – 00:43:24:09
Bryce Schumacher
Yeah. And so it’s interacting with your team, which lives on a time scale. Yeah. It’s planning for itself, which lives on a time scale. It’s planning systems that rely on timeouts. And so we we see those issues. We’re asking AI agents to plan for other AI agents and to get them to work together. That requires time scales.

00:43:24:11 – 00:43:56:23
Bryce Schumacher
We see the LLMs being really bad at predicting human behavior and desires. And where that comes up a lot is the brownfield projects. So if you if you dump it into a code base and ask it to make a change, it is very bad at predicting what the original engineer or architect did when they built that code base. Now it has all the tooling it needs to go figure it out, right?

00:43:57:00 – 00:44:36:00
Bryce Schumacher
But you got to make sure it goes and does that. And that can be part of the business process around debugging. But if you ask it to just guess what the previous engineer did on that project, like good luck. We see the AI being pretty bad at interconnected systems. So there’s, there’s a benchmark that they do where they give an AI 25 puzzle pieces that are scrambled and basically ask it to put the jigsaw puzzle together.

00:44:36:00 – 00:45:07:20
Bryce Schumacher
And our best AIs can’t do it. Our best AIs can’t do a five-by-five jigsaw puzzle. So where do we see this? I mean, you see in a lot of places we see it on the government side where you just have 5 to 5,000 organizations interacting. We see it with every piece of software we write. So once you have front end, API, back end, database, and auth,

00:45:07:22 – 00:45:29:20
Adam Pasch
And I’m sitting here over thinking about like the 17 different systems that are needed to plan a mission. Yeah. And you’re actually talking about just like even within one thing that I’m thinking of, the system is where there’s the front end, the back end, there’s these other layers in between, like from that we’re already at.

00:45:29:21 – 00:45:30:15
Bryce Schumacher
Yeah.

00:45:30:17 – 00:45:32:01
Adam Pasch
Five plus level.

00:45:32:03 – 00:45:42:12
Bryce Schumacher
Exactly. You’re 20 minutes into coding a greenfield project and it loses itself. Right.

00:45:42:14 – 00:46:00:04
Bryce Schumacher
Now, you absolutely can have a business process around making a change to the front end or the database. You can architect in ways where fewer things are talking to fewer things. And hopefully you were doing that before AI.

00:46:00:06 – 00:46:14:02
Bryce Schumacher
Absolutely, yeah. 20 minutes into a software project, you you could be past that K5 graph, the five software systems talking to each other. Right.

00:46:14:04 – 00:46:21:07
Adam Pasch
I can see now how that also scales. Does that scale up with the number of different connections between. So it’s not.

00:46:21:09 – 00:46:22:22
Bryce Schumacher
It’s the arithmetic.

00:46:22:22 – 00:46:27:18
Adam Pasch
It’s logarithmic because it’s because of the number of joins and the complexities there.

00:46:27:18 – 00:46:29:16
Bryce Schumacher
Or graphs are combinatorial.

00:46:29:19 – 00:46:30:08
Adam Pasch
Which.

00:46:30:10 – 00:46:32:07
Bryce Schumacher
Much much worse than exponential.

00:46:32:08 – 00:46:35:04
Adam Pasch
Oh no. Okay. Yeah.

00:46:35:06 – 00:47:07:14
Bryce Schumacher
And, you know, software architecture, software design. They’ve known that since the ’50s and ’60s. And so like, we can build out our APIs and our database tables. They don’t all connect to each other. Okay. But you can get there very quickly. Got it at K4 or K5. Understood. So yeah I mean we can have an AI that can search across 200 skills and use all 200 of them.

00:47:07:15 – 00:47:15:20
Bryce Schumacher
Then they don’t depend on each other. But if they all depended on each other they would not be able to to predict that at all.

00:47:16:02 – 00:47:21:16
Adam Pasch
Got it. Okay. No that’s good. Pro tip.

00:47:21:18 – 00:47:57:13
Adam Pasch
Okay. So given that those are some of the things that the AI is good at versus what it’s not so good at within that context, then can you help me understand what when that human now is moving up that stack. Then what’s the inherently human role then. And how are they using the AI to free up their time of what they were doing today, to give them more time and space to do what they’re going to, what the human’s going to do tomorrow?

00:47:57:15 – 00:48:24:13
Bryce Schumacher
Yeah. So, I mean, you start with, with what it’s good at. So we said it’s good at search. Okay. Received a bug ticket. I have this massive production system. I now have to go search through all the code. So, you know, the AI absolutely can help you with that. So find which piece of code is causing the bug.

00:48:24:16 – 00:48:40:10
Bryce Schumacher
What are all the the interconnected systems that we just said? It doesn’t understand all the interconnections, but we can start looking for them in absolutely massive code bases.

00:48:40:12 – 00:48:57:12
Bryce Schumacher
It’s good at spelling, so it’s good at writing code without syntactical errors. So if we tell it exactly what code to write, that is no longer a human task.

00:48:57:14 – 00:49:05:05
Bryce Schumacher
It’s fairly well accepted that coding syntax is just not a human thing anymore.

00:49:05:07 – 00:49:08:06
Adam Pasch
Right?

00:49:08:08 – 00:49:26:16
Bryce Schumacher
Now, should we let it write whatever it wants to write? No, no, we should define what it should be writing and why, but then let it write it. Yeah.

00:49:26:18 – 00:49:59:23
Bryce Schumacher
But and then in places where, you know, we don’t think the AI is good at that point, you know, we’re putting in the guardrails, we’re putting in the business process, we’re guiding it through it. So we could in a super critical system, we could take that away from the AI, but in a non-critical system, basically, we would just want to get away from the headache of having it do it incorrectly over and over and over again.

00:50:00:03 – 00:50:19:03
Bryce Schumacher
Yeah. So you’re debugging it, tells you it fixed the bug, you go run it. That’s completely wrong. And you tell it, hey, you did that wrong. It’s like, oh, I did do that wrong. It goes and fixes it again. You go and test it. It’s still completely wrong. Yes.

00:50:19:05 – 00:50:20:12
Adam Pasch
I have been there.

00:50:20:17 – 00:50:48:19
Bryce Schumacher
So all of a sudden you, you know, you blow through 4 to 6 hours. Yes. So that’s on us. Like we knew that was going to happen. We knew there was a possibility that that was going to happen. Yeah. So we should have prevented that by basically having a process on how to debug each of our systems, how to plan or how to follow our plan in line with our plan.

00:50:48:20 – 00:51:13:02
Bryce Schumacher
You know, present the solution to us, all the information that we need in order to decide if it’s a if it’s a good fix. And so basically those, those guardrails or that process to keep it out of the things we don’t think it’s good at, or to involve us more in those things being more collaborative.

00:51:13:06 – 00:51:52:09
Adam Pasch
Yeah. And so those guardrails and those processes, the documentation, you’ve kind of hit a few different times of, of where those become really critical. Again, guardrails. And you brought up a great point of that. Like that’s a lot of work that we may need to do right now to catch up on that documentation and architecture debt, or is there kind of an easier place to start now, kind of MVP or V0 that gains us some standard benefit?

00:51:52:14 – 00:52:27:13
Adam Pasch
While we then now take a step back and do that critical documentation, something something that gives us like an interim benefit because the reality is we’ve got AI in all of our different organizations. We’re accepting the inherent risk right now, whether we know it or not. There should be something that I could do today that is a quick change that gains me some benefit, that now buys me time to go do the bigger, harder documentation effort.

00:52:27:15 – 00:52:50:08
Bryce Schumacher
I think it depends a little bit on your team and your workflow. So we have teammates that are essentially configuring their harness. So, you know, fool me once, fool me twice kind of thing. Yeah. A lot of times, you know, we do just let it rip, but it would make some mistake. You know, we can document those mistakes.

00:52:50:09 – 00:52:57:21
Bryce Schumacher
We can add it to the to the harness configuration as an MD file or a rule or a template.

00:52:57:22 – 00:53:10:15
Adam Pasch
I guess this is kind of like see it, kill it. And the kind of guardrails organically grow through this kind of like just-in-time quality feedback kind of process.

00:53:10:17 – 00:53:41:22
Bryce Schumacher
Yeah, I mean very quick and dirty. But why, you know, why make the same mistake twice or at least try not to, you know, some of it could could be education. Can we talked about what is the tool you’re using. How does it work. What are the other tools that it uses. You know, there’s definitely a conversation around creating more work for your teammates.

00:53:42:00 – 00:54:12:08
Bryce Schumacher
So before you send that email, before you send those documents, editing that down to as small of a product as possible, stuff like that. But really, like if you have a business, especially if it’s a profitable business, there’s probably things that you’re doing over and over again, just take a second and write them down.

00:54:12:10 – 00:54:31:19
Adam Pasch
All right Bryce. So we went through a lot of things to keep in mind as this government leader that is being asked to AI-ify everything, so then how how do we start to bring this into our enterprise.

00:54:31:20 – 00:54:39:00
Bryce Schumacher
I, I think what we see across the board is AI is already in your enterprise fair?

00:54:39:01 – 00:54:39:10
Adam Pasch
Yes.

00:54:39:11 – 00:54:45:11
Bryce Schumacher
So this has been the public conversation for four years right now.

00:54:45:12 – 00:54:45:21
Adam Pasch
Fair?

00:54:45:22 – 00:55:02:18
Bryce Schumacher
Yeah. I can’t go to a restaurant in the evening without overhearing the table next to me talking about AI. Yes. So all of your employees are using it. It has invaded your enterprise already.

00:55:02:20 – 00:55:35:05
Bryce Schumacher
So one of the things we’ve we’ve seen be successful at multiple organizations is just getting everyone in the room and asking them what they’re using AI on. You’re going to have shadow IT. You have shadow AI. Yes. If you’re not providing AI tools and education to your workforce, they’re using their own tools, whether they’re sneaking it in on their phone, whether they’re using it on their laptops, at home, at night.

00:55:35:07 – 00:56:05:03
Bryce Schumacher
It doesn’t matter how locked down your buildings are, your your employees are still using AI throughout their day. But with it, you know, have it. Since it’s all we’ve talked about for the last four years, and your employees have been sitting at their desks doing their jobs for the last four years, they probably already know which part of their jobs they want to do with AI.

00:56:05:04 – 00:56:18:10
Bryce Schumacher
So when we get people in a room together, we’ve had hour-long meetings where as many as 20 examples have come out of a single meeting.

00:56:18:12 – 00:56:50:23
Bryce Schumacher
And it’s been going on long enough that we’ve probably already aligned with what the AI is good at. So lots of document search, lots of report writing. So you can absolutely get your team together, have them talk it out, figure out where where you’re leaking data, where your risks are. You already have the risks. You might as well find out where they are and just start planning there.

00:56:51:01 – 00:57:16:14
Bryce Schumacher
If. If your workforce is using AI in certain areas, that’s probably where you need to go invest in it anyway. You still need to go inventory your business processes and see what the really big things you need to knock out are. But your team’s already using AI in the workplace, even if you don’t want them to, right?

00:57:16:16 – 00:57:25:00
Bryce Schumacher
So you might as well be the provider. Stop a lot of the data leaks there and use a lot of those grassroots ideas.

00:57:25:02 – 00:57:48:14
Adam Pasch
Well, you’ve got built-in from a change-management perspective, right? You’ve mentioned that several times going going with what’s already been prototyped. It’s just been prototyped in shadow IT. Shadow AI. That sounds like the perfect way. You’ve already got a built-in audience that wants to be making the bigger impact, instead of just making it up on your own from the ivory tower, get into the field.

00:57:48:15 – 00:58:01:04
Adam Pasch
Take the lessons that have already been learned. At the same time as you’re de-risking some of these other unintentional data leaks and other risks that may come along with that shadow piece.

00:58:01:10 – 00:58:33:08
Bryce Schumacher
Yeah. And I, I mean, one thing I’ve learned doing empathetic design interviews for ten years is managers are hilariously bad at predicting their direct reports’ daily activities. Yes. So when you when you get people in a room, you’re going to get all those different perspectives on all the different things they do during their day and where they want to use AI.

00:58:33:10 – 00:59:02:02
Bryce Schumacher
Again, you’re still going to want to take a look at it from the top down and decide where the enterprise wants to use AI, what business processes can be changed? Which ones can be eliminated, what new ones can pop up? But the business processes that are already running have been running for the last four years. Your team has already been daydreaming about where they would put AI in their day.

00:59:02:03 – 00:59:26:19
Adam Pasch
Totally. Yeah, impossible not to love it. That’s what I got, man. No, I appreciate you. I think you’ve shed a lot of clarity on where and how to get started. Some of the things that we’re not looking at now of where a lot of risk is probably coming in, but a lot of kind of structured ways to really get started at this.

00:59:26:21 – 00:59:29:03
Adam Pasch
This is great. Thanks for your time, man.

00:59:29:06 – 00:59:30:12
Bryce Schumacher
Well, thanks for having me.