2 Ways Functional Expertise Strengthens Predictive Modeling

00:00:00:03 – 00:00:10:06
Adam Pasch
Carter, I’m so pumped to talk to you. DataRobot. I have questions.

00:00:10:08 – 00:00:13:03
Carter Meseck
I have answers.

00:00:13:05 – 00:00:40:05
Adam Pasch
It was awesome. You were in the hackathon that we did, and it was awesome to see how many people came up with, like, actionable prototypes. Which, total side note, I need to follow up with everybody to figure out. It’s been like two or three months, and, like, how many of those prototypes ended up having, like, more versions and, like, impactful stuff come out of it.

00:00:40:07 – 00:00:58:11
Adam Pasch
But aside from that, I was lost once it got into the technical side. Yeah. And so I’m just curious from you, just, like, at a high level, what are you doing inside DataRobot that’s starting to have some impacts?

00:00:58:12 – 00:01:20:20
Carter Meseck
So I think we could take one of those. So, two of those use cases from that hackathon we’ve been developing, and one of them we’re working on just delivering overall. But most of that, we’re taking it from, you know, these people trying to work with this data and figure out, like you were, they get to the technical side and they’re like, “I’m not sure what’s happening.”

00:01:20:20 – 00:01:47:10
Carter Meseck
I know what’s happening, you know, functionally, but I’m looking at all these numbers, just, like, these different features. Like, I’m not sure how that works then within, you know, what they’re trying to do. So one of them, you said from the hackathon, the MAM mission aircraft use case, they’re taking those, like, repair jobs and they have all this data on it.

00:01:47:10 – 00:01:59:13
Carter Meseck
And it’s like they’re not really sure how to go from there in terms of predicting with DataRobot how long these jobs could take, which then could help them.

00:01:59:17 – 00:02:07:01
Adam Pasch
Oh, like how long the mission is going to last to, like, when that aircraft is going to become available to work again. Yeah, totally.

00:02:07:01 – 00:02:36:07
Carter Meseck
Because if you have a bunch of jobs that need to get done and you’re like, “I don’t know how long, realistically, these guys take,” it’s like, well, with all this data, if you put it together right and kind of do the technical side of it, then from there you can kind of extract some insights and have a much better guess so that you can prioritize these different, in this case, different aircraft, so that everything is working up to speed instead of getting bottlenecked.

00:02:36:07 – 00:02:36:21
Carter Meseck
And that kind of…

00:02:36:21 – 00:03:02:03
Adam Pasch
Because from a functional side, I totally see how, you know, you have a certain number of C-5s, C-17s, other mobility aircraft where you’ve got to run this global logistics network. Some of them are special assignment missions, some of them are channel predictable missions. But you’ve got to balance it amongst the total capacity of how many aircraft, how many aircrews.

00:03:02:05 – 00:03:34:22
Adam Pasch
After a certain amount of time, things are down for maintenance. So I’m with you on it. It’s much easier then for planners and people that are executing those missions to know when additional capacity is coming back online. Yeah. So what are some of the technical things that you’re doing, or that you’re able to use DataRobot for, that help get us from where we are today and create that kind of functional outcome?

00:03:35:00 – 00:03:43:08
Adam Pasch
Help unpeel some of the layers for me on the technical side, yeah, of how we get there.

00:03:43:09 – 00:03:50:10
Carter Meseck
So I think, just at a high level, if we step away from this one a little bit, because it gets into the weeds of all these details.

00:03:50:11 – 00:03:51:00
Adam Pasch
Totally.

00:03:51:01 – 00:04:18:17
Carter Meseck
So at a very high level, let’s just say that when you repair an aircraft, there are just many different processes, we’ll say, overall. And these have their own kind of rules with them, or, you know, this kind of job has these other things that happen with it. And so it’s kind of taking that functional knowledge that these things are happening in the data.

00:04:18:17 – 00:04:34:03
Carter Meseck
But they don’t know that. They just know the functional side of it. So taking that and doing some kind of formatting into a model that can kind of explain to you what’s happening. So an example would be…

00:04:34:05 – 00:04:50:13
Carter Meseck
It takes a long time for a… Or, so, let’s say you’re driving your car, and all of a sudden while you’re driving, your car has a check engine light that comes on and it starts blinking at you, and you start hearing some things that are, like, you know, going on, some…

00:04:50:17 – 00:04:52:22
Adam Pasch
Clink-clank. Yeah, absolutely.

00:04:52:23 – 00:05:05:11
Carter Meseck
Yeah. So that would tell you a lot about kind of the nature of the fix. It’s like, you need to pull over immediately. Yes. But then some could be like, “Oh, the light just comes on.”

00:05:05:13 – 00:05:14:18
Adam Pasch
Right. It might be minor. It might be major. All I do is see the light, and I don’t know until, exactly, either I do something about it or something catastrophic happens.

00:05:14:19 – 00:05:15:05
Carter Meseck
Exactly.

00:05:15:06 – 00:05:24:01
Adam Pasch
So are you, then—is what you’re doing then helping shed some light on more than just a single light? Here’s a more contextual signal.

00:05:24:03 – 00:05:38:02
Carter Meseck
Yeah, so it would be if your car has the check engine light start blinking while you’re driving after you hit something, that’s, like, a much bigger problem than you just turn the car on and, for some reason, the check engine light is on.

00:05:38:06 – 00:05:39:00
Adam Pasch
Understood.

00:05:39:01 – 00:06:01:01
Carter Meseck
So putting that context together, if you’re, let’s say, a mechanic working on cars and you’re like, “Yeah, this guy hit something while he was driving. He hit, like, a deer going down the highway, and then his car just fell apart,” rather than, you know, “Oh, his car got busted up a little bit. Maybe he went in the ditch a little bit.”

00:06:01:07 – 00:06:22:10
Carter Meseck
Something like that, or something just hit the car. Like, there’s all these different scenarios, but you can kind of put that context together to be: this car, the engine’s a little busted from a deer versus the engine’s busted because, I don’t know, someone just hit your car in a parking lot. Sure.

00:06:22:15 – 00:06:54:13
Adam Pasch
So let’s delve a little bit onto the technical side then. When you’re in DataRobot, how is this similar to, like, writing in Python or other languages? Versus, is the way that you’re using DataRobot inside of some, like, GUI database-style tool? What does it look like to use DataRobot on a daily basis?

00:06:54:15 – 00:07:20:15
Carter Meseck
So there is the option, if you want to get really in the weeds, with doing the code. But for DataRobot, just to get a basic overview and kind of, like, idea of what’s happening in your data set, you only need to use the GUI interface, which looks like, you know, you’re putting a data set in, kind of identifying what features you want to have.

00:07:20:16 – 00:07:44:03
Carter Meseck
So, like, with your car, what happened while you were driving your car? Like, did it just happen overnight or something? Those kind of things, and kind of curate that feature set of what kind of things am I looking for? Am I thinking about while the functional is maybe investigating, or the mechanic’s like, “Okay, let me see this, this, this, and this. Like, what’s happening here?”

00:07:44:04 – 00:08:03:18
Adam Pasch
Yeah. So it sounds like it’s more approachable data analysis, especially for functionals, but that you, as a data scientist, can set up the data and other processes behind the scenes that really empower them to do faster analysis.

00:08:03:19 – 00:08:27:19
Carter Meseck
Yeah. So an example of most of our kind of, like, pipelines would be, you know, if you’re the functional, you’re the mechanic or stakeholder of this data, I interview you, get some of this context of, okay, what’s actually happening? What’s the functional meaning of some of these columns? Like, what’s going on that would cause this to happen?

00:08:27:19 – 00:08:53:04
Carter Meseck
Or I’m seeing this in your data set. Is that normal for you? And then doing modeling, then there’s another round of, you know, going back with that and saying, “Hey, we’re noticing that, you know, a lot of repairs are taking longer when it’s a Saturday and your car has this happen to it. Does that make sense for what you’re seeing in your day to day?”

00:08:53:07 – 00:09:11:12
Carter Meseck
Yeah, because on the technical side, we’re not sure. You know, we can say this is what the model is showing, but we need to work with that functional input to be like, “Hey, I’m looking through the data on the technical side, seeing this. Yeah. Does that make sense with your day to day?”

00:09:11:14 – 00:09:49:07
Adam Pasch
Yeah, I like how that really helps. Then I like the collaborative nature of that, of how when you find these data norms or data extremes, in working with the functional teams you’re finding out either data quality problems, which is helpful, or you’re finding insightful signals that might lead to the functional teams then having these moments that lead to process improvements of probably a wide range of kinds, probably highly context-dependent.

00:09:49:07 – 00:10:12:09
Adam Pasch
But either way, you’re either improving the quality of the data that improves future analysis and change, or you’re improving the change now because the data was of high quality. But I get that on the technical data side, it’s all interrelated numbers at the end of the day without the functional context.

00:10:12:09 – 00:10:15:14
Adam Pasch
So it really takes that collaborative nature together.

00:10:15:14 – 00:10:44:08
Carter Meseck
Yeah, because I would say another thing that we do in these collaborations, it’s kind of: okay, you gave me your business rules, but then I’m looking at these business rules to validate. I’m seeing these things in the model that completely break that kind of rule that you’ve told me, or I’ve seen these kind of, what I would call, exceptions, that now we have to go through each of those and say, okay, what’s going on here that would cause that?

00:10:44:08 – 00:10:52:10
Carter Meseck
And how is that affecting what you’re seeing from a modeling standpoint?

00:10:52:12 – 00:10:59:05
Carter Meseck
Let’s say, like, you give me this business rule that…

00:10:59:07 – 00:11:00:20
Carter Meseck
Let me think real quick.

00:11:00:22 – 00:11:27:16
Adam Pasch
Let me ask you a different question. Okay. So you’ve got a ton of different data tools that you can use. And in your career as a data scientist, you’ve been exposed to even more. You’ve been using DataRobot a lot recently. Okay. So what have you seen that are some of the things that DataRobot does best and some of the things that it does…

00:11:27:16 – 00:11:37:16
Adam Pasch
Worse, where you end up having to do some workarounds that would have been easier in other tools? So I’m just generally curious, like, what’s the best and worst, okay, from your experience so far with DataRobot?

00:11:37:17 – 00:12:05:19
Carter Meseck
So with DataRobot, really what DataRobot does very well is, in data science and predictive modeling, machine learning, it takes a lot of time just to set up the skeleton of, like, running these things, because you have to set up, like, your training and testing set. And if you want to do, like, a cross-validation, et cetera, et cetera, these are all things you have to set up a whole new model for.

00:12:05:19 – 00:12:32:10
Carter Meseck
And then on top of that, determine, like, how many features you want in that, calculate all these metrics output from that to then evaluate the model, which is all done in DataRobot. So it’s basically just a plug-and-play of: get your data in there, make sure it’s correct, narrow down what kind of features you want, and then just model from there.

00:12:32:12 – 00:12:59:07
Carter Meseck
And so I would say, with DataRobot, I do most of my data cleaning, data wrangling, all that stuff mostly in Databricks or SQL, Python, PySpark, one of those three, whatever you’re most comfortable with. Because then from there you can just ingest it straight into DataRobot and then into a model, and then output those results back into any system that can connect to it.

00:12:59:07 – 00:13:01:15
Carter Meseck
But, got it. That’s cool.

00:13:01:17 – 00:13:33:19
Adam Pasch
Well, and I think from one of the earlier conversations we were having, you were describing how using DataRobot starts to help the onboarding of MLOps into an organization that maybe hasn’t done a lot of MLOps in the past. Yeah. What about DataRobot—and it may just be, like, DataRobot plus some other things—makes it advantageous for someone who’s in an organization that’s just starting out in getting serious about MLOps?

00:13:33:20 – 00:13:49:16
Carter Meseck
So I would say, what—so you’re trying to say, what makes it really easy for someone like me to come to you as a functional and show you and help you understand, instead of, like, showing you these metrics, scores, and acronyms that you have not exactly…

00:13:49:20 – 00:14:29:03
Adam Pasch
One hundred percent. Like, if I’m a functional-forward organization, we’ve got lots of robust systems, but we haven’t really made the leap yet into an enterprise-wide, machine-learning-forward, you know, where we have a rigorous MLOps culture, maybe, I’ll say. And so you’ve got to crawl, walk, run. And it sounds like DataRobot is making it a little bit easier for you, not just for you, but, like, for the functional teams in particular, to really start adopting and getting benefit out of machine learning.

00:14:29:05 – 00:14:57:04
Carter Meseck
Yeah. So I would say what makes it really easy about communicating to different functionals and helping them understand is that with kind of those output metrics, they also show a very easy-to-read graph of, “Hey, these are these top features in your model that are showing a lot of kind of impact.” And then from there they break it down even further to, like, so for…

00:14:57:06 – 00:15:21:01
Carter Meseck
The type of, let’s say, incident or whatever this feature is, you can see a graph or kind of a breakdown of, for this feature or this variable, these values have a much different impact on that model compared to other ones. So, like, a zero might be much less than a one in terms of whatever you’re trying to predict.

00:15:21:03 – 00:15:40:06
Carter Meseck
And so from that, it’s really easy to show someone, like, “Hey, the day of the week is very impactful.” And then after that it would be another variable and another variable. And then from there it’s like, okay, let’s break down the day of the week. Which days of the week are the most, right?

00:15:40:09 – 00:15:41:08
Adam Pasch
Maybe seasonality.

00:15:41:09 – 00:16:17:02
Carter Meseck
Exactly. Seasonality is a big one too. Yeah. And so being able to just quickly display that is very easy for communicating. As well as that, they’ll also do different prediction explanations with that model. So that could be, you can compute a few explanations and however many explanatory features you want and see exactly the value it predicted and the top features that contributed to that, and then how much and in which direction.

00:16:17:03 – 00:16:28:01
Carter Meseck
Awesome. So, like, a Saturday might be a—it’s a bad day to go outside or go to the mall because it’s super busy that day. But Monday’s great, right? No one’s outside, right? And so, yeah.

00:16:28:02 – 00:16:33:00
Adam Pasch
No, that’s great. I love it. Well, this is the first one, not the last one.

00:16:33:02 – 00:16:33:16
Carter Meseck
Hey.

00:16:33:18 – 00:16:35:14
Adam Pasch
This is great, man. That’s what we got.

00:16:35:16 – 00:16:36:09
Carter Meseck
Oh! Let’s go.