Time to read: 10 minutes

I hate this question.

“What is your favorite product?”


“How would you improve that product?”


I’ve been asked this question and it always strikes me as the wrong question to ask. Before unpacking further we should understand what the general point of an interview question is. In the most literal sense an interview is an exercise to explore how you think.

For some professions, what you know as a base layer is also important and that can be deducted through some form of questioning (often something along the lines of sort this array in descending order while splicing out all prime numbers)

But, as a heuristic for professionals it is reasonable to say you will be paid by a company to think. So deducting how you might think in an interview about a challenge is somewhat useful to the interviewer. I specifically said somewhat and the following paragraphs will elaborate on why I don’t believe this question is the tell all of product management roles.

In a hierarchical relationship the boss makes a statement and the individual contributor completes the work the boss asked for. This can be anything from take the trash out, fix the deployment script or do this sales call. Whatever it is, it is done. You can argue this should not be the case but more often than not it is.

In interview settings the interviewer often fills the role of “Boss”. He or she is directing where the interview should go (most of the time) and the person being interviewed has less power to dissent from that direction even if he or she believes the direction to be wrong.

When I studied game theory economics I would spend weeks digesting this with complex theory but I will keep it simple for our purposes:

If this is a one shot game the interviewee is incentivized to comply hoping to make it a repeated game.

If this is already a repeated game the interviewee is incentivized to do what he or she would actually do.

I find this very interesting. If you have ever tried to run/grow/scale a business before, you know speculating on improving a product is dangerous. This leads to navigating with your eyes closed. Yet, most product interviews ask this to deduct some “innate talent” that the gods have blessed you with. The reality of growth means a lot less speculation and a lot more data driven focus.

In the early days of a product you won’t have data models and excel backlogs of data to play with. In this case my determination of what to do with the product has very little likelihood to result in success.

If I owned the company in this setting I would categorize all customers with the following data:

Customer Name When Acquired How Acquired Contract Value Usage Value Derived Job Title Company Size Field
Dicks Sporting Goods January 9th Direct Sales 20,000 <10% of plan very little VP product 30,000 Sporting Goods/Retail
Amazon January 11th Direct Sales 500,000 75% of plan typical for usage Director of Alexa 620,000 Everything/Home Speaker
Walmart December 10th Trade Booth 1,200,000 98% of plan very high CTO 2.2 million Everything/Retail

Do this for all of your customers (hopefully you have some clean data to look at). Depending on how early it is you could do this in as little as 15 minutes. Once you have this categorized you should figure out NPS for your customers based on this categorical data and talk to them. Yes, that means actually pick up the phone.

If you are scaled like the iPhone is you could work a shift at the genius bar doing iPhone support and get a lot of learnings from there. In this scenario your individual insights without data are not very likely to be helpful.

To put this into a little more context, lets imagine you are running a small software startup. You have a few engineers, a designer and a founder who is mostly serving as a PM and sales person. This team costs you $50,000 a month.

You’ve just raised $600,000 on a great valuation because everyone on your team is incredible and the world is your oyster. You have 12 months to make this company work (yes, I know you should raise >18 months so you start raising again ahead of month 11-12 once you’ve hit your targets, don’t troll me please).

You are selling a solution with a ACV of $10,000. So you will (in theory) need 7ish (taxes, random expenses, etc) a month on an annual basis to hit a breakeven without any absurd financial engineering. Your product isn’t really selling and the customers that do buy aren’t getting a ton of value when they do (so probably won’t renew). This kind of question is often the reality of early stage products and a lot more worthy of figuring out how to think.

Let’s make a few assumptions here and then dive deep into how I might digest this problem.

Let’s assume:

  • You are decent to good at sales
  • Your product works
  • You are trying to improve product to increase sales

With this context, and the data above you can think through what are your options.

It looks like on the data above (again, heuristic, you would want a lot more data before making a sweeping conclusion)

trade booths occur with:

  • higher contract values
  • with more value delivered
  • with more senior champions in the organization

There are quite a bit of things to digest here (and test with statistical significance) but lets just assume trade booths coincide with more senior individuals whom have more say in the organization, and have access to a larger budget.

Do we believe this to be a fluke? What does the champion say about the product? Where is he or she not quite getting all of the value out of the product?

Maybe you have a download feature but Walmart isn’t using it, and the feature would be really good if it automatically put the data into salesforce because that would let this CTO help the VP of sales with tracking leads and thus, might even be able to pull budget from sales.

I created a hypothetical situation above that you would only figure out based on talking to your customers. So yes, pick up the phone. Call them and ask a lot of questions.

Wrapping this full circle, the question of how would you improve x product is often the wrong thing to ask.

The better question might be: what would you want to know before feeling comfortable trying to improve the product?

With the first you are relying on my gut and “innate talent” (of which I am not likely to be world class at). With the second you are taking something subjective (my opinion vs yours vs the developer team vs the CEO vs etc.) and quantifying it to a point where you can make an informed decision. Will this decision be right? It might be, or it might not be. At the end of this you’re still going to have to make a gut call. And watch the results.

But, by slowing down and digesting the problem you’ve done a few things in addition to taking opinions out of it. In programming you would never start changing a bunch of things in a program and testing a week later to see if it works. You’d change one thing, watch what happens and go from there.

If you’re being smart about your product you should do the same. Isolate as many variables as you can. For this, we make a hypothesis that higher value contracts and higher usage can be obtained through trade booths. And those who come from a trade booth have enough say in the organization.

By isolating we could run a variety of different tests to improve the product without “betting the company” on any individual test. Walmart finds a ton of value in the product and we’re trying to do two things:

  • get more companies like Walmart from trade booths
  • help Walmart get more value by making the download feature work better

Dicks might not have found value in the product because:

  • VP product doesn’t have internal sway to push developers onto platform
  • those who did get on platform were not active 30 days later

Dicks needs a faster way for users to find value to be retained. Walmart was able to force users to find value via their boss.

This situation could be solved by enabling the sign up/set up flow to become better and empowering internal champions to make it easier to onboard team members. There are quite a few ways we could hypothesize doing this:

  • email onboarding
  • webinar trainings
  • Oauth with Google

These above are ranked in what I’d guess (just a guess) would work best. As you can see solving this problem is radically different than the Walmart problem. But they both are small enough feature wise we could probably implement in 2-4 weeks with a small team.

Flashing back to our hypothetical software startup, you’d be making a decision to spend $25-50,000 on this feature set to improve the likelihood of success of the company.

We can take it even further that this experiment not working also includes opportunity cost of another experiment working, your team working at (and owning equity in) another company whose experiment does work, and thus results in a massive, Google IPO like, pay off.

Once you start to think in dollar amounts confidence can be deduced! Especially when the numbers are more than your downpayment on your house. If you don’t feel confident you probably have not talked to enough users. Do not pass go if this is the case.

And you could further model out other cohorts of users, look at the data and talk to them to determine how we might move the needle. Maybe your VP of Engineering will allow you 5 developers to take 2 guesses at this, and you pick 2 tests to run simultaneously. Maybe you get more resources.

The key here is not the number of resources, its distancing yourself from making opinion based judgements that result in many thousands of dollars lost if wrong. By taking this approach, and by answering the question “what would I want to know before feeling comfortable trying to improve the product?” you’ve set yourself up to be more successful, and understand the variables at play.

Rant over.

Take this with a grain of salt. It is one persons opinion. But as a product and technical person I want to make sure I am informed and acting on the right axis before I take action.

Asking someone to improve a product without knowledge of what is going on behind the scenes of that product isn’t a good use of time. Asking someone to take you through how you might gather the data, gather the buy in and quantify the possible ROI is.

Business is risk in exchange for profit. If you own the business you’d want to minimize as much risk as possible while raising the likelihood of profit.

Using this approach helps you deduct an individuals ability to do just that.