Melissa Perri | How I Test for the AI Build Trap
Melissa Perri - Bestselling Product Management Author & Board Member
In this episode, Melissa and I discuss how AI fundamentally transforms product operations and how companies can test these new found capabilities and actually build things customers need.
Have a Listen
Summary
In this episode, I’m joined by Melissa Perri. You may know her best as the author of Escaping the Build Trap and I believe she is one of the brightest minds in modern product management.
I’ve been privileged to know Melissa since the early days of the Lean Startup movement and I was thrilled to chat with her about what has changed in product and testing over the past 15 years.
We talk about the dangers of PM’s outsourcing their critical thinking to AI, and why experimentation with good data is maybe now more important than ever.
Melissa details out how AI fundamentally transforms product operations and how companies can test these new found capabilities and actually build things customers need.
If you want to learn how to make better product decisions, accelerate learning and avoid this latest AI version of the build trap, then you'll love this episode.
Takeaways
AI is accelerating the build trap. Companies can build software faster and cheaper than ever, but without clear strategy and customer understanding, they're simply building the wrong things at unprecedented speed.
Don't outsource judgment to AI. AI can generate specifications, analyze data, and accelerate execution, but experienced product leaders are still needed to challenge assumptions, interpret evidence, and determine what should actually be built.
Better data doesn't automatically mean better decisions. Organizations struggle with fragmented systems, conflicting metrics, and unreliable information. AI creates enormous opportunities to improve decision-making, but only when teams can access and trust the underlying data.
Product operations could be supercharged by AI. From synthesizing customer research to interrogating business performance, AI can dramatically reduce the time required to generate insights. Product Ops can help connect these capabilities to strategy, governance, and measurable business outcomes.
AI won't fix broken organizational systems. Functional silos, disconnected strategies, rigid funding cycles, and competing departmental goals remain major obstacles. Without addressing these systemic issues, organizations risk using AI to make dysfunctional processes run faster.
Guest Links
LinkedIn: https://www.linkedin.com/in/melissajeanperri/
Product Institute: https://productinstitute.com/
Transcript
David J Bland (00:00.812)
Welcome to the podcast, Melissa.
Melissa Perri (00:02.882)
Good to be here, David.
David J Bland (00:04.972)
You are one of my favorite people. I am so glad you're on here. I felt like I had to get a few episodes in before I invited you on. but I just wanted to pick your brain on, you know, AI and strategy and testing and the build trap and so many different things. We'll just gonna hang out and our listeners can just listen to whatever we have to say.
Melissa Perri (00:24.568)
Yeah, I'm so excited. I know we haven't like caught up in a long time.
David J Bland (00:28.364)
We haven't. I think my gosh, was it like twenty ten or twenty eleven I think we met? I don't even remember. It was early lean startup days.
Melissa Perri (00:33.89)
Yeah, it was it's early lean start of days. Yeah, it was like I think probably was it at the Lean UX twenty thirteen? May have been around then. Yeah. I think it was then. 'Cause I met a lot of people at that conference. So I'm like we're going back thirteen years now.
David J Bland (00:45.019)
maybe. I don't even know, yeah.
David J Bland (00:52.97)
my gosh. Yeah. I remember and we also did stuff with like Lean Start Machine, I think, back in the day. Like you were coach coaching that. Although we didn't cross paths though.
Melissa Perri (00:57.72)
We did. Yeah. And yes. Yeah, we did that too. that was like twenty I wanna say that was twenty twelve, twenty no, two thousand yeah, twenty twelve maybe or two thousand eleven. And yeah, I was coaching and I would do the New York ones and then I started running a while after that I started running the Agile UX meetup in New York City too. We would come to a lot.
David J Bland (01:22.06)
It's fun. It's fun to see like I don't want to say like us grow up in the community, but you know, spending that much time trying to figure this stuff out and seeing it evolve. And I just wanted to kind of dive into a bit and just jump right in. You know, so much has changed since we both got involved in all this stuff. And everything's all AI now, which I understand. But I'm wondering from your point of view, you know, you have this amazing book, you know, The Bill Trap. And
I feel like are we just recreating this with AI? Like please talk me back from the ledge here.
Melissa Perri (01:57.261)
Yeah, that's what I feel like. it's been it's been interesting because it's like I've been talking about this. I think the first the first talk I ever did on the build trap, like as a concept, was 2015. And of course, you know, it was Agile all at that time. It was just let's build faster with agile. That was the premise that people were like sold, let's say. Not necessarily the premise of what it was supposed to be. But you know, I I wrote from that point of view, but I feel like in the
In my book, it was very much about how product management needed to step back and really figure out what we should be building. And at the time it was expensive to build software. It's certainly getting cheaper to build software now, but the core premise was always more about wasting time to building the wrong things. And now I see the same exact thing with AI. It's just another shiny tool. It's way cheaper to build with software now.
But I see corporations spending so many hundreds of millions and billions of dollars on AI everything now, thinking that it's gonna get them like a hundred X leverage, but they're not spending the time and the dedication to actually figure out what we should be building with the AI. So the budgets are all there and it's huge. but it's it's still wasteful because the direction isn't there. It's still like exploratory of what we're doing.
so I find myself repeating exactly what I was writing about in twenty fifteen and what I was speaking about in twenty fifteen. And instead of the agile context, it's still more about, hey, how do we how do we learn? And I think the tools have evolved a lot, which is really cool about how we can experiment, how we can learn. and I'm really excited about that. Like I I love being able to like dive into code and I clog code all day like build my own little apps and stuff and like test them out with people and things. But
It's still I I still see so much direction missing from companies and from people because instead we're focusing on, hey, all these tools are able to do these things that we never were able to do before. Let's just hit the ground like running. And I go, run where? And just nowhere. In many, many cases. So I yeah, it's it's so interesting. I think the way we do our work has evolved, but so many of the principles.
Melissa Perri (04:26.22)
have not, and then there's so many like frameworks and tactics that are still necessary to be able to make use of all these amazing tools we now have the capabilities to use.
David J Bland (04:38.7)
Yeah, I think for our listeners, the your book was the first time any of my memes made it into like a major publication. So I was very excited by that. So thank you again, for including the product death cycle. And I know we just updated recently and said, Okay, it's the same thing but now with AI and I saw all the kind of fallout from that on on LinkedIn and it just seems like
Melissa Perri (04:47.669)
yeah.
Melissa Perri (04:51.608)
Thanks for letting me put it in there.
Melissa Perri (04:58.498)
Yep.
David J Bland (05:05.592)
That's where we are again. You know, it's it's all the where are we going? You know, we can go really fast in a direction, but I think we can go really fast in the wrong direction too now. And I'm wondering, how are you trying to grapple with that inside companies, you know, because they're like, we go like it's almost like they're prioritizing speed over everything at the moment, but you can go really fast in the wrong direction too. And I think in the end you kind of end up in the same spot where, we don't have something that fits with our customer. I don't know what what are your takes on that?
Melissa Perri (05:34.265)
Yeah, I it it's a hard one, right? 'Cause I think everybody's so excited about AI and I'm excited about it. Like I said, I I so much fun. It's like really cool to be able to work with these tools and and explore it. But at the same time, like the message I keep pushing in organizations is we have to stop, take a step back, figure out like where we should be heading. And that takes, you know, a lot of user research. It takes understanding your market. It takes market sizing, it takes segmentation, it takes data.
it takes having a vision of where do we want to go. And then it's like, okay, let's go take these tools and apply it and see if we're going in the right direction. And that those concepts have never changed, right? We we've been talking about that, both of us, like, you know, from the lean startup days. and what is happening, I think, is I see a better conversation happening now than in 2015 about people knowing that they should do that.
But organizations not creating the space where people are allowed to do that. So like I used to fight for the concept of product management and for product managers to just exist in organizations. And now they exist. And I feel like I don't have to fight that fight. The fight I do have to fight is for chief product officers, not seeing this just as still like somebody who works with developers and just keeps them busy. Some organizations are really good at it, some organizations are not. I
I'm having better conversations, I feel like, about how product management is also a business discipline and not just an engineering discipline. It's something that sits in the middle and it basically translates business strategy into what we do with our products. And because we all live in the age of software, those things are heavily intertwined. But in many organizations, I think the the mechanism of how it's set up, and a lot of times a funding cycle is a big part of this too, it creates
This chamber where it's like we have these really lofty, awesome visions. Some of them are just so grand and so nebulous that nobody can latch onto it, right? Everything you do can fall into a vision. And then you say, okay, cool, we prioritized it. But when it gets translated down into the organization, it's not it's not getting translated appropriately, let's put it that way. it just becomes build these things faster in a solution, but not, hey.
Melissa Perri (07:59.641)
Here's like our bets. Here's what we're thinking about doing. Like, let's give these people space to go out, talk to the customers, figure out what's going on. And because AI allows you to go so much faster, I think a lot of organizations kind of look at the people and go, we could just like replace a lot of this with AI and outsource these things to AI. And I see a couple of huge mistakes with that. Like one, a lot of organizations outsourcing judgment to AI.
And AI has extremely poor judgment on many things without the right guidelines, right? So you're like, okay, if you're gonna outsource these types of roles, right? Let's, for example, let's talk about outsourcing the product owner role and documentation, right? I love that Claude can help write a spec that can now be translated into something that you code. We do it every day. Like we just rebuilt our entire website and we did a bunch of specs into Claude, trained our AI to go build those things.
But who's reading this spec to make sure it's actually good? Right? Like who's reading this back to no like Claude was putting a bunch of random crap in it. And we were like, what are you like, what are you doing? No, right? It's just like going crazy with some stuff and it would go in some different directions. So I was like, I didn't I don't mean that. And then I'll go, yeah, you're right. I shouldn't have done that. Like, come on. So that's judgment, right? So if we think about it, it's like.
How we we still need to steer the AI. And to steer the AI, you need to have people who have good judgment, who understand the customers, who have this body of knowledge to basically be able to say, here's the, here's the guidelines around it. And now we can go faster now that I've trained it and now that I've built the guidelines. And I feel like we're we're skipping over that step. Like I think a lot of corporations and companies
We're just like, hey, with AI, like we can just replace a bunch of people, but we're not thinking about like what's the most strategic thing to replace. Like somebody just told me at an extremely large bank, they laid off all of their mid-level senior product people, but they kept all the product donors. And I was like, to me, I would have expected
Melissa Perri (10:15.362)
Differently, the opposite. I can understand why they did it, probably because of monetary reasons. They think it could do that. I said, you know, why would they make the decision? They said, well, they figured they didn't need mid-level senior product managers because they could get the cheaper people to just steer the AI and the AI would build things. And this is like a prime example of who is doing your strategy, who's steering your AI. Most product owners I work with do not have the strategy capabilities to steer an AI over a large
product or project, right? That you're building on there. And you need somebody who's going to be able to really help set that strategy and discern it and then be able to translate it. And I think AI makes us so much faster at certain things. Like yes, you don't have to sit there and write a spec word for word, right? But you have to be able to judge if it's correct or not. And that to me is like that's scary that we're going in that direction. I think like
For many startups, it's wonderful because the founders who have the direction and the strategy can now use AI tools to spin up their products very quickly. And I think corporations are going kind of in the opposite direction where they're thinking about like how do I cut costs with AI? And yet they're spending more on compute and losing a lot on judgment and building the right things. And that's a cost there too. Right. And when I wrote Escaping the Mill Trap, like back in 2015, again.
Software was expensive to build and it took a lot longer than it does today, but it was also about time to market. If we're stuck building the wrong things, when do we ever go back and find out what's the right thing? And we're still not measuring if it worked or not. And that's still a huge problem in corporations today, too, and most companies. We have so much better data tools now than we've ever had in our lives. And I'm so excited about that. Like I get really nerded out by data. I love to go through companies' data.
So many of them are not taking the time to instrument it properly. And it does take a lot of time. Like I've I've cleaned data myself. I've had I had a whole team when we did Products Labs that was literally going through court like company data when we worked with Insight, cleaning their data, trying to figure it out and trying to put it into BI systems so that we could pull the relevant information to set product strategy. Like that's a huge part of what we did. So I don't take it for granted how much it takes to clean data, but a lot of times.
Melissa Perri (12:41.822)
One thing that's good is a lot of companies realize you need clean data to use AI appropriately. So they have good pushes to do that, but they have nobody over the instrumentation of the AI to understand it and use it inside the company strategically to help set that strategy. So we're out there kind of doing these huge projects of cleaning all the data and then not really thinking about like how do we segment this, or we're deprioritizing those pieces that are needed for product strategy. And that's the part I think that's still
is hard and hits home. that I see there. So it's it's like a systemic thing. It's always been a systemic thing. I've always said it's a systemic thing, like the bill trap. AI is just exacerbating it. Right. It's just like we we expect to be able to go a hundred times faster tomorrow. But if you don't get your building blocks in order and your structure in order around an organization, like the way you fund things, the way you do your governance, the way you
set up your organizational structure, the way you have intelligence built in to make decisions, that's when things like the strategy cadences I talk about and the strategy levels and how they translate. If you don't have any of that, that's when this all breaks. So AI just makes it broken faster. And then you're just spending a lot of money to Amazon and Amazon and NVIDIA are getting very rich, but you're not shipping things that really matter.
without those pieces in there. And that's the part that I think is still true today. Like whether you want to say Scrum or Agile or we're getting hung up on the semantics, it's just like we replaced AI with Scrum. Like we thought Scrum was a silver bullet. I never said it was. now everybody's like AI is a silver bullet. It's just gonna be something else next. and while I do think it's powerful and I do think it's changing the way we can do so many things, I still think
We're missing the bigger picture in so many places.
David J Bland (14:44.671)
It's funny you mentioned Scrum. I still have I was going through my old photos the other day and I saw the photo you shared with me where someone quoted me at a meetup. I think it was in New York where they said inspect and adapt on everything except Scrum. And I was like, yeah, that's kind of like how I how I talk about it sometimes. But I mean the community, I feel as if you know, agile and scrum and all the stuff that we've tried to
Melissa Perri (14:53.186)
Yes.
Melissa Perri (14:59.138)
Yeah. It was true. Yeah.
Melissa Perri (15:05.4)
Mm-hmm.
David J Bland (15:12.511)
apply over the years, right? You know, it was all about a lot of it was about making delivery faster, right? It's like, we need trust in delivery. And then yeah, we'll figure out what we're delivering or why and all that. And I was always torn by that approach. But I understand why people took it. You know, I would talk to all these executives and they're like, look, we can't even trust our development teams to to to deliver anything on time. And it's like, okay, yeah, I get it. Okay. But I think I think my soapbox recently has been I feel as if
Melissa Perri (15:33.294)
I hear that all the time. Yeah.
David J Bland (15:42.583)
We're going faster, which isn't necessarily a bad thing, but then when we had to make the case for why to build something, there was some inherent friction there of we have to use our critical thinking skills, we have to kind of reason through this stuff, because if we make a big bet and we're wrong, it's gonna be very costly. And I feel as if with AI, we're and maybe this is the product ops conversation that we can dive into as well after this.
It feels as if we're streamlining stuff and so now some of those conversations that used to happen are not happening. And it's almost like we almost have to intentionally reimagine what that friction looks like in our company because a frictionless company, in my opinion, is just gonna officially d deliver stuff that nobody cares about. So there is value in the friction. I'm wondering with like produps and all that stuff that you've helped pioneer over the years, you know, where do you see that playing now or where you see that is it
complementary to AI? Is there some conflict there? Like how how are you approaching it?
Melissa Perri (16:41.922)
Product ops is extremely super powered by AI. And that's what makes me very excited about it. Because at its fundamental, we always saw product operations as a way to systemize and scale the value of product management and help to enable the teams. And you used to have to do that with a lot of people. So some of these product ops teams were huge in in some of these organizations. Like when I was at Athena Health, we had people who were
You know, data analysts embedded at every director level on the teams, supporting like a couple, you know, six squads, let's say teams to help pull the data out of the systems. You can interrog if you if you and this is where I get really hung up on the data stuff, like if you actually put your data together in a BI tool, now we have ways to actually interrogate the data ourselves. And anybody could start asking it questions and start to be like, All right, I have
You know, I have these three different customer personas. I want to understand what's my t total revenue of each of the persona segments. If you systemize your data correctly and plug all your tools into each other, anybody in your company s can start asking those questions, right? And that to me is really exciting. So like the integrations of tools and the capabilities that we have now to interrogate data without doing SQL is amazing. Like when I started in product management, one of the our
of my early companies I worked at was OpenSky. And OpenSky was a very early user of MongoDB. And I had to take a class on MongoDB to be able to get the data out of our database because I only knew SQL. And we didn't dump it into SQL until much later so that people could interrogate it with, you know, an actual tool. So I had to go take like a seven-week course on MongoDB just to get data out to be able to say, should we do this versus that?
Because our engineers were busy, right? Like it's my responsibility to go out there and start to be, I can't just ask them questions all day. We were like a small team scaling like crazy. And that's still happening to product managers today, right? Is if you don't do that, they're still gonna have things stuck in these systems. But like you can plug it into all these BI tools now and just be like, I wanna know what these types of customers do, what's the percentages, and it will spit it out. Like that is so cool. We have never been able to do that before. Like this is
Melissa Perri (19:02.574)
To me, like that's so amazing. But we have to take the time to actually instrument it, right? So like that's part of product ops. So one, like product ops is supposed to be that governing body that works. Now it's not replacing the data team, right? It's working with the data team so that when they're cleaning all that data for AI, they're also advocating for the product management data that we need and the systems that we need to plug in together so that the product managers can make decisions as well. And that goes all the way up to leadership, right? I always say like,
Product operations doesn't just enable product management, it enables that translation layer of what are our business outcomes doing with our products, right? Like, how are we actually achieving them? So if you instrument all of that, it's not just for your product managers, it's also for your leaders. And many of the leaders that I work with, they're like, I honestly just have no transparency into whether these bets or these initiatives are working. And that's their biggest frustration. They're saying, My product managers can't come back to me. And like even mid like directors, VPs, everybody.
They can't come back to me and tell me that for this thing that I budgeted, you know, a $50 million resources for for this initiative that's going to take a year and a half. This is how we're progressing towards those OKRs that we actually put on there. And if you can't tell that story because you don't have data, you can't connect back to your business outcomes. And that's that is all possible with the stuff that we have, but you need somebody to oversee it, right? You need somebody to oversee instrumenting all of that.
And thinking about it in a product way, not just an AI way. And right now, all of the resources are going into instrumenting our AI so we can learn from it intelligently and start to program our own AI, but they're not thinking about how do we decide like what that AI should do? And I've seen so many strategies where like corporations will I'm like, what's your AI strategy? And they're like to reduce the cost of you know customer service by 200%. How?
How would you like to do that? Like, right? Like, what does that mean? I've seen ones too where like the strategy is just to spend a certain amount of money on AI. Like, we're gonna invest a billion dollars in AI in our company in the next three years. And we celebrate it. Like you see it in the news, it gets celebrated. It's like, wow, this company's doing amazing. Look at how much money they're spending on this. But I'm like, spending it on what is always my question. And then many times I'm lucky enough to go into these companies and I can ask that question.
Melissa Perri (21:30.402)
And that's where we start to get confusing stories about like we're not actually sure. And that's where that experimentation comes from, or that's where that strategy building comes from. That's where that deployment process comes from. It's how do you take that money and justify what you get from it? Right. It's not just about spending a billion dollars with AI. It's about what do I get for spending a billion dollars with AI? What's gonna become better? And that's a story that we have to get better at telling. And that's where product really comes in. So
Product ops should enable that and it enables it with the helping with the data part, but also like the strategy part. So, what is our governance and what are our cadences for setting strategy? Who's responsible for what? How do we communicate it clearly? How do we make sure it's very intentional across the organization? And how do we make sure that we're not just setting strategies that are so big there's a pot that anything that you want to build just falls into it? And that's super common, right? Like if I make a strategy that's
I want to, you know, decrease the cost of X by, you know, 30% or whatever. So many things can go in there, but what's the right one for you? And at what cost? That that was like a I talk about mutually destructive pairs when it comes to KPIs and OKRs. So I'm saying like if you are basically saying I want to decrease the cost of customer service by 200%, what about your MPS score? What about your customer satisfaction? What about your retention?
Right. Like what's the guardrail that's there? And that's that's the pieces I think when we think about how much AI can do that get lost again. And it got lost, it's been getting lost for years. It's just getting lost faster now. Cause we're just thinking about how do we reduce costs? How do we go faster? How do we how do we do these things that we've never been able to do before?
David J Bland (23:17.836)
Yeah, a lot to unpack there. I I appreciate your answer. I feel as if one of the quotes that always stuck in my head was, Yeah, measure what you want to go up, but also measure what you don't want to go down or you know, or flip of that. And I think that's what you're hinting at a bit there. And you know, one of the things I was talking about recently w was something I was called assumption laundering. I don't know if that would stick or not. But it's this idea of these assumptions go into this the strategy and and creating our strategy, but it's almost like
Melissa Perri (23:28.812)
Yeah, exactly.
David J Bland (23:48.353)
We we refine our business artifacts and polish them and we and we do it in all these pr kind of predictable ways. And it comes out the other side of that planning process as looking like something that's a fact. And there's something weird that happens there. And I'm not sure what your take is on this, but I feel as if we didn't inject anything as far as evidence goes and making sure that thing was actually true. But the way we presented it, we're much more confident about it because of the way it was presented. Yeah.
Melissa Perri (24:20.046)
It was like a fact. Yes. That happens all the time. I see that all the time. So I ask like, I go into a lot of organizations and I do assessments on their product operating model. And I evaluate it from a couple of different lenses. I look at their strategy. So I want to look at like how do you set strategy today? What's the governance around it? What are the layers of it? And then what does it look like? I so I'm like, share your roadmap, share your strategy with me. And I look at tons and tons of corporate strategies all the time.
And I always ask them, like, how did you arrive to like this framework? I'm just curious. like I have my own framework, but I'm flexible. I've always been very like not dogmatic about frameworks. I'm like, if your framework works for you and we can zhuzh it into the right thing, I'm okay with it. But as long as it's gonna get the desired outcomes. I'm like, all right, so how do we get there? How'd you get these numbers on here? It's like I see your increase, you know, net retention by 200%. Like, what is that pulling into? And who's net retention, right? Like who is your
Customers that are churning and why? Right. And like, where is that on the strategy? And usually it'll be like, we just saw retention's going down. So we want to increase it. And I'm like, great. So we need the next step, right? We need a next layer of okay, why is it going down? What's happening here? How do we dig into it? How do we actually ask those questions? What are your assumptions? And a lot of times it'll just say, there will be these assumptions that are just listed there that says like,
You know, people don't want our product, or these people do that, but it wasn't followed up. Like there was, I asked, what's the user research on here? Do you have the notes? So I'm always asking for like show me what it is to just prod it. And we get into sometimes it was backed by loose facts, right? Like something was going around that indicated that that could be the problem, but we never followed up deeply enough to say, yes, 100% that is a problem. And we've tested it in a way where we can prove that.
If we solve this problem, it will increase retention, you know, halfway there to our 200% goal or whatever it is, right? And that's the piece I think that gets that's missing, right? Like it's kind of like we look at a piece of data and we interpret it and say, that's that's why it is, and then we just put it down and we don't follow up. And this is where, like, so like I know this is all like super negative, but I get excited about AI. Let me let me talk about my ideal.
Melissa Perri (26:43.448)
How I wish people would do this because I feel like like I want to say like how I I'm gonna talk about how I used to do this and why I'm excited that AI exists because now I think companies are not harnessing it correctly to be able to do these things too and set the strategy. When I'd go in as an interim CPO, and I've done this dozens and dozens and dozens of times, I would go in and we would usually have to figure out is this strategy that's there.
the right strategy to help this company get to where it needs to go. And a lot of this I did with Inside Venture Partners, but I've done it with a lot of other private equity firms. I've done it with a lot of corporations that have board goals as well. And we'd go in and I'd go in and I'd be like, okay, so what are the goals? What, you know, who set it? Did the board set it? Why do you have that? And there's some financial mumbo jumbo of why they say the goals. Okay, great. But we know what the goals are that have been set for the organization. Okay, let's look at your strategy. How did you get
To these things to hit that goal, how do you know? And that's where we start to look for facts or data. And if they had facts or data, like we'd pull them up, they would be able to show us market sizing reports that were done. Fantastic. Let me dig into that. I'd ask for user research. Now, when anybody comes into, ooh, did I just disappear? My Elgato, just like, no, I could see you, but I just went the other way. Sorry. My like moniker.
David J Bland (28:03.273)
Yeah, you kinda went out for a second. I'll just put a note in here for that. Yeah.
Melissa Perri (28:06.68)
Can you see me now? Okay.
So when we go into these organizations and we're trying to figure out what is, you know, what is your strategy or what should your strategy do be to get there, we try to figure out like what are the main problems that are standing in the way of getting there. And you look at the business goals and you'd be like, all right, how's our retention? How's our churn? How's our growth in certain segments? if there was a new product idea floating around, like how is it performing? Are we targeting the right people? Is that the right product to go after? We'd look at budgeting. How were you budgeted around these different organizations? Where are we spending our money to get
the actual goals of that. And we pull all that data out. And then we'd have to dive dive in and say, are these real legitimate problems that we are solving that we do believe with firm facts are going to get us there? And that was a mixture of getting the data out of the systems to see what we're currently doing and what our current state is. And then going out and pressure testing our assumptions around those through user research, market research, and further data analysis, and then through experimentation. So we do this on like a very high level.
But we'd pull all this data out of the systems. And I'd had like two two business analysts, type people who were from like McKinsey and Deloitte, and they'd go through all the data, clean it up, dump it all in there, put it all together into like the charts that I needed to see, all the things to actually visualize it so we could sit down with the leadership team and say, Hey, this is where we currently are. And that would take, if they didn't have a good data infrastructure, that could take three months, right? Just to get an Excel spreadsheet that you couldn't update later.
Because now we need to go get software engineers to plug this into BI tools to do X, Y, and Z, right? So now we're talking about like, and this was gross stage companies, like corporates wait longer to do this. Now we can look at that and we start to say the assumptions, like let's segment it correctly. How are we making our money? What are we doing? So we'd segment it and we start to understand and categorize how do we have personas. You need to do that from user research. Okay, what user research do you currently have? If you don't have user research, we'd go out and make a user research plan and we go and get the information to pressure test that.
Melissa Perri (30:06.914)
We do a bunch of you know, deep dives on it. And then we come back in and reconfigure the personas, look at some of the usage data if we could get that and try to figure out how people are using your products to get where you need to go. And that would form the basis of the strategy. And this whole process could take anywhere from three months if we were lucky and we had good data. And we were pretty fast and pretty good at it in certain like in some companies, but like if everything was a mess.
We're talking six months, like sometimes a year. If there was a lot of political stuff, it would take even longer, right? And it was always to get to like a first roadmap of what can we be doing to help scale. With tools now that we have with AI, that user research, like I I I pulled stuff out of Google Drives. We pulled stuff out of like people's literally like their notes apps on iPhones. We had to go interview everybody around the organization to know what they knew. Like
There are tools out there that aggregate all of this qualitative information that you could start to interrogate and just get trends. And that doesn't mean that it's a fact, but it's a trend that you can then see and go back and say this these people said it in this context. And now you have so much more information to start working off of. And now you can go and plan your user interviews so much faster. So I'm like, ugh, that blows my mind that that's available and that not everybody uses that today. Cause I'm like, that's such a gem.
be able to do that. And then two with the in the the data pieces, like if we could go in and start to pull that information up on it, like leaders could be interrogating their data, questioning like what people are saying every day if they wanted to. Like I don't suggest it changed strategy every day because that's like way too fast. But like they could be keeping a pulse on those things all the time. So this is where I get really excited about what AI can do for product management and for especially product leaders like chief product officers and leadership teams, like you can get
some of this information out of your own systems to keep a pulse on what's happening and tie it back into your business outcomes. But you have to take the time to actually instrument it. You have to take the time to build these models and this governance and the framework and the cadence. And I think a lot goes into cadences that we do too. Like when we set strategy, it shouldn't just be a one time a year thing of what should we build next year? It's like it's a free flowing thing where we have lofty goals that will take us
Melissa Perri (32:30.87)
A long time to hit, and then we have smaller targets that are broken down. But we can monitor that with real-time data and change it up as we go. But to do that as well, that's where like the funding cycles come in and the project funding and everything like that. And that's where everybody starts to get a little bit broken. But there are ways that you can massage it into your funding cycles if you could get ahead of it. Because you could put something into a funding cycle if you don't have to go research everything and try to figure it all out, like the
Four days before your budget's due, which is not enough time. Like if we're doing this continuously, we'll have enough stuff in the pipeline to keep looking forward and to keep getting it into a budget cycle. And that's the part that I get really passionate about that I could see corporations like using with AI, just like to get that transparency, to get that that information. But at the same time, that brings it back to if you instrument it all properly, you still need the people with the judgment to say.
Hey, AI gave me all this information. AI will never set your strategy for you. It does not understand humans. It's really bad at like just getting into the nitty-gritty of whys. Like, like we had one company that we worked with that we set strategy with. And if you looked at all the data, it was like with therapists, the they just basically were like, therapists don't want to, you know, use this product. They're just stuck in their ways. They're, you know, they're at these small companies or whatever.
And if you looked at the data, it was like, nobody's using this. This product is trash. Like, I guess we should just like sunset this product. If you go talk to the therapists, it was like, no, I want to use your product. It's just so freaking hard to use because it breaks on me all the time that I'd rather go to pen and paper because it's faster. So it's like if you only looked at the data, it would just tell you nobody wanted to use it and nobody was using it. But if you go talk to the people, they're like, I would use this because I would like it to integrate with this part and it would make my life so much easier.
But your UX is so bad, I can't use it. Right. And that's a nuance that an AI will never get if it's just looking at your data, right? And then there's those that body language and that understanding of your customers that really comes through the human interactions of it. And then that's the part that you have to pull together and that's what creates strategy. So it's the with the market sizing and like the business side of it too. So like like pulling all those things together, AI can do a lot of those pieces faster for you. Like it can size a
Melissa Perri (34:52.93)
can size a market for you now which is really cool like I love using it for that product managers ask me all the time before like chat GPT how do I do market research like I don't understand I like I I can't do a competitor analysis like plug it in chat GPT it does it for you now right like I suggest you follow up and like check out the stuff it tells you but it is so much easier to do some of that stuff and you do have to double check it that it's sizing it correctly with the right assumptions but again you have to have the right assumptions to be able to put into it
to get it to do those things. But it's been easier now than it ever has been to get some of that information. But you need the people to be able to go through it and evaluate it and say, is it right? Is this a like right call? And AI is not gonna do that for you.
David J Bland (35:35.485)
No, nor nor should it. I mean, I think that's been a theme last few convers conversations we've had, episodes, we were talking about decision making with Martin and with Barry and with Teresa and I it's interesting to me with AI, we think, okay, so we're gonna add AI to the mix and then we're gonna make better decisions. But I think we keep coming back to this data layer that you keep stressing. And you've had a lot of experience in that in that I think it's too easy for us or maybe leaders inside a company to be like, why why do they make that decision?
Melissa Perri (35:37.388)
No, it's a robot.
David J Bland (36:05.588)
Like, why did you possibly come to that conclusion? Because in hindsight, that looks like a terrible decision. But usually people are making decisions off of the data they have available to them. And so if you don't have the same source of truth with data inside your company, yes, you're going to have weird behaviors inside your company with or without AI, because you're making decisions based off of vastly different sets of data and just coming to different conclusions.
Melissa Perri (36:16.334)
They have exactly.
Melissa Perri (36:32.526)
That comes up in a lot of assessments I do. Cause I asked, like when I do assessments, I don't just talk to product teams, I talk to sales, I talk to operations, I talk to like I I want to talk to the entire C-suite, I talk to CEOs, I talk to head of technology. And I always ask them, I said, Do you feel like you have the data and the tools capable to make the decisions that you're responsible for making? And they must always say no. and then I'll ask them, like, do you believe that your data is
Like your decisions are data driven. And they go, My decisions personally are data driven with the data I have available to me. But anytime I try to bring them to the table, other people will disagree with me because whatever software they're using, for example, in finance versus sales, or like your Salesforce versus your financial systems, they don't agree. So there is no single source of truth. And then they'll argue about which one of those platforms was correct that you got that number from instead of.
reconciling it and trying to bring it together. So sometimes there is like people are trying to do the right thing and make the data back decisions. It's just that they can't because there's overwhelming amounts of data and none of the data makes sense. So nobody trusts it. And then they all disagree. So they just kind of like pick whatever's the least worst option.
David J Bland (37:55.371)
Yeah, I remember some Silicon Valley companies I advise, we would say, how do we know this like new customer came in through this specific channel? And we had no answer because we had so many different channels and they had to trace that kind of user across different fiefdoms inside the company. It it's
I mean, you you and I were were I know for a while anyway, we were talking about like pirate metrics, right? Like the A R R R and all that. And I remember giving a talk, I think it was somewhere in San Francisco, and I I I laid it out like here's acquisition, activation, retention, and then depending on your company, referral revenue or revenue referral, and I laid it all out for everybody. And then I took these giant red marker and I just kinda like started building walls between it's like, who owns this part of that cycle? And they're like, that's marketing, this is finance, this is and they don't talk to each other.
Melissa Perri (38:38.925)
Yeah.
David J Bland (38:43.34)
And they're different systems, right? And so while you draw it out, it looks amazing in reality if you don't have the data infrastructure to actually trace something through and say, Hey, we made this change and this impacted our business in this way. It it kinda like that whole idea of build measure learn just unravels because you you can't really trace it through the whole cycle.
Melissa Perri (39:02.626)
And it's so interesting. Like I I watch this a lot with so many companies. And I've I've I've I've worked with a lot of companies that have scaled quite a bit over the years. And the like when you're in a gross stage company or a smaller company, all the leaders work together as if they're in the same team because you're small and you're making decisions that impact each other very quickly. As soon as you start scaling and getting your own teams, that's where I see, and especially when you get to be a corporation, like everything's silos.
But we silo around functional departments, right? And we're not siloing necessarily around like business lines or anything like that. It's it those are siloed as well. But it's like marketing is so siloed from product, and the marketing leader has like their marketing strategy, and the product leader has their product strategy and the salesperson has their product strategy, but none of them built that strategy together. And that is when all those things start crumbling. And there's no usually there are like company level OKRs.
But I've also seen them just make an OKR for each department and call them a company level OKR. And I was like, that's not, that's not what that's supposed to be, right? It's supposed to be the joint OKRs that if we do, if we do our technology well, our product well, our marketing well, and our sales well, we reach these goals that everybody owns as a leader, right? And it's a, it's a, it's an interesting concept. Like I the reason why I feel like people
fall into that, like leaders fall into that with CEOs is because they want to be able to hold somebody accountable for their part when something goes wrong. So they're like, if I have a marketing OKR and I can say that that's that person's thing and they don't hit it, like I know to blame marketing. But if you have joint OKRs and you're a joint team, who do you blame when something goes wrong? Like you can diagnose that I've diagnosed it before. You could tell when like marketing's not getting the people and then but then you have to go back and say, is it our product offering or is it marketing's fault? Right? Like are we targeting the right people or are we not targeting it?
So then you have more nuanced conversations with it. But at the end of the day, what that also does is it starts segregating the leaders from each other and then they start fighting. And I'm like, you should never, you should have, like you said, healthy friction in an organization to like get to the best outcomes. But it should never be where like marketing and product are pitted against each other, or technology and product are pitted against each other, or sales and product are pitted against each other. Like everybody's on the same team.
Melissa Perri (41:25.218)
Everybody's trying to hit the, you know, hit their goals. And when you create strategies that divide everybody, that just divides the team. Right? Like we're all selling the same thing here. We should be. Like we should all be like, we should be celebrating when sales make wit makes their win. We should be helping sales make their wins, right? We should be celebrating when product finds like a great idea that's gonna help sales make more money. But that's not the the environment that a lot of corporations foster.
Because it's come from like legacy systems where everybody had their own little line and it all, you know, bubbled up from there. But that to me also is what like breaks that whole strategy cascade. And again, AI is not gonna solve that problem. It will never solve that problem. Like that to me is one of the most fundamental broken things in so many organizations. And if you dump a bunch of money on AI, it's not gonna fix it.
David J Bland (42:25.362)
In some ways I don't think AI is gonna solve that problem. I'm also kind of okay with AI blowing it up in a way where we're not clinging to industrial era ideals for managing our business and organizing. So I'm kind of okay with that part.
Melissa Perri (42:32.302)
That'd be great.
Melissa Perri (42:40.545)
Exactly.
And I I think that would be great. Like I'm excited. I'm excited for that. Like I think you're seeing a lot of disruption on smaller companies right now that makes me super excited about like how they scale, how they're working, the fact that they can make all this money or stay like a smaller team for longer. And they don't have to reach some of those scaling issues that you would by having to hire 150 people to do these jobs, right? You can start with like good groundwork. I think that's really cool to watch them do that. I think it's gonna be much more challenging for
organizations have been working a certain way for the last eighty, ninety years to start pivoting. And it always was. Like it was hard for them to pivot into software. It's harder for them. Like I have a lot of empathy for it too because it's like it's the environment. It's it's a huge ship to turn. It's not a startup. As they all say when I come in, it's like, no, you're not a startup. I know you're not a startup. But but it's also like we do have to take the baby steps to get there if we want the whole thing to turn. And we do have to learn from what other organizations are doing well.
even if we might not be a software native organization, like we can learn from them and harness some of those things that they bring together to collaborate around product to build wonderful software and start to incorporate those things into it. And that's the part that I think AI is gonna help with and like blow up, which is gonna be cool. Cause now everybody wants to use it everywhere and you start to see the power of it, but it it's fundamentally software at the end of the day. So once again, like we're bringing everything back into software.
We do have to learn how to operate like software companies where we collaborate around products to sell, right? Those could be services as well. But that kind of mentality of like this is our product and this is all a team and we're all an executive team around this, like that I think is a really big thing that we need to nail in the age of AI. And I do think AI is gonna disrupt that in corporations. And it's gonna be interesting. I've already seen them like kind of silo off AI.
Melissa Perri (44:43.19)
into its own, you know, style of chief AI officer. And I'm like, for what? Like, but like what are you using the AI for? Right. And it it I've seen it like seven times now. So they just got they have a department that does AI and data. And I'm like, what are you building the AI for? And it's for the other products. And I'm like, you can't just you can treat it like a platform. It is a platform team, sure. But like what's the strategy that connects it into your products there? And instead it's like, let's budget first.
Create the team first and then we'll go looking for problems to solve with AI rather than let's start from the strategy, figure out where can we harness AI to do things we've never been able to do before to help serve our customers better, to help serve our business better. And then how do we get the right team and the right people in there to build the AI and train it so that we can do those things? Like it we're going from the bottom up instead of the strategy down, and that's where I think we get stuck right now.
David J Bland (45:40.072)
Yeah, yeah, I see that too in my clients. You know, I've seen the new AI, head of AI, chief AI officer rules c popping up. It's it's almost like it needs to be instead of you versus me, it's us versus the problem, you know. I'm curious as we wrap up, i is there any I I I appreciate you being excited about where AI could could help and I can tell in your voice, like you there's some stuff that used to do manually, like, this is really cool. It
Melissa Perri (46:07.298)
Yeah, it's really cool.
David J Bland (46:08.955)
Is there like a core assumption for you looking for that you'd want to test over the next year or so? Like is there something that you feel like, there's I don't know how this is gonna play out, but it'd be really fun to test this and see if there's any evidence to support, like any leading indicators that we're headed in a certain direction. I'm just curious, like cause you cause you're on the forefront of all this, and I I'm just really curious if you have any like big assumptions you want to test going forward.
Melissa Perri (46:34.946)
I think like my assumption is that I think like with AI, like if I look at from a product ops perspective, I think in a strategy perspective, like I think my assumption is that for the the teams that take the time to instrument their data and like harness these tools correctly, I think they're going to blow it out of the water compared to those that don't.
I have always like, even when I was at HBS, I was like, I want to do a research like project. And they wouldn't let me because I wasn't like faculty research staff. I was like teaching staff. But I was like, I really want to like test a core assumption about like good product management and doing these things that we just talked about helps companies succeed more. Like I want to compare them. it's a hard, that's a hard research statistical problem to do, but I'd love to do it. So if anybody's doing that, I'd love to work with them on that.
But like that, that is my fundamental assumption is that if you have strategic thinkers in your company that have the right data available and you're making it accessible to them, I think, and you have like that that plan, like I do think they are going to go so much further than the companies that struggle with some of those basics there. And I'm really interested to see like what gets disrupted along the way. Right. Like when it's when it's faster to build these things more than ever.
I do think we also have like a population of people who are getting more and more comfortable with software. I think they're scared of AI still. I think a lot of people generally, like I'm excited about it because I work in this, but I think there are still people who are generally scared of it. Like I'm curious to see how people react to everybody going full AI. And like what are they willing to give up, like give up in service of ease for software? And like where are those lines? And I think that's gonna be a really interesting human problem as well.
David J Bland (48:25.125)
I agree. I agree. Thank you for sharing that with us. If folks are listening and they just want to reach out to you, maybe help to do your research or partner with you on that or anything else you're working on, what's the best way for them to get in touch?
Melissa Perri (48:37.24)
Yeah, you can reach out to me. my website is at Product Institute. You can drop us a line or hit me up at Melissa Product Institute dot com. But yeah, we help organizations navigate these problems. I work with a lot of executives to explain it. And we have an online school at productinstitute.com that addresses all the stuff about getting out of the bill trap.
David J Bland (48:58.117)
Awesome. So check out Product Institute. We will put those links in the description. I just want to thank you so much for hanging out with me. It was great to reminisce back in like early lean startup days. And then fast forward to everything we're dealing with now. I just appreciate you being open and honest and transparent with where we are and what the challenges are. And also being optimistic about it. It's refreshing. So I want to thank you for hanging out with us.
Melissa Perri (49:17.496)
Yeah, it was great. Thanks for having me, David.
