Why is this still so difficult: Data Literacy in Modern Product & Design Teams

I had a coffee chat (tea for me) with a peer this week. They are a co-founder and head of product, and when I asked them what their biggest concern/focus for the coming 6ish months for their team was, they said “be more data-driven”.

An interesting piece they’ve shared as well was how we, as product leaders all assume other companies have their ducks in a row when it comes to collecting useful metrics and data points, and do this so much “better” than ourselves, but I can tell you, the truth looks different: of the companies I’ve worked for, every company is so different in their data literacy* scoring and some embark on the journey to improve quickly, some won’t because of different priorities and no buy-in from either their immediate team members or their managers, or even the executive team.

So, we shouldn’t ignore this problem and challenge the status quo!

Let’s change that but how?

Problems:

Here are some potential problems that you might have run into in the past or even are currently still present. (Feel free to share some other problems you might have encountered in the comments below)

• Deadlines & out of sight, out of mind mentality: Engineering wasn’t able to add in metrics because of release deadlines. These items fall off first when rushing to finish a feature.
• Accountability / Buy-in from above: Similar to any other bigger initiatives at a company, e.g. DEIB, or Agile transformations etc, you need the buy-in from the leadership team.
• Accessibility / Skills Set: PMs don’t feel comfortable deciding on data points to collect, sometimes they feel it’s too hard since they assume you need to have a data scientist degree.

Solution(s):

Start with either of these solutions below, why not, even just pick only 1 if you feel overwhelmed, the idea is to start somewhere in order to raise your data literacy score.

• Have no fear, start small/assume you don’t get it right at the beginning: Encourage your team it’s OK to fail, it’s a muscle they need to train, iterate and learn. I tell team members, once you’ve shipped feature x, I want you to link me to a dashboard/chart when I ask you in 6 months how this feature has been performing.
• Tie it to OKRs, build an outcome-driven roadmap: I know this might be a bit of a harder solution to accomplish, let me still get you excited to at least try it: Assume your company OKR includes something like “%20 of new ARR comes from SMB customers”. By creating a tree of data points, OKRs, opportunities etc, you can easily tie back that change in the product x and how it aligns back to the overall outcome and goal. Shoutout to Jason Doherty for giving me the idea and iterating on it in Miro. Feel free to view/use this template. You will get PMs excited to focus on the outcome and how they get there, with what output is up to them.
• Schedule and hold Pre-release feature reviews: I’ve started this at Lever and the teams really liked it, similar to your first OKR review, people might feel stressed and embarrassed at the end of the quarter when it’s review time, because they might have forgotten on what they set out to be a success metrics for their feature release or the way they thought they could measure it, they weren’t able to. Have each PM (and EM) present a recap/review on their feature that they’ve released in the last quarter, that way they have enough time to collect adoption data, bug data and general trends (usage by segments etc.). I can promise you there is always at least 1 interesting data point that either they or leadership or peers have not thought of. That’s a win! Keep iterating, the team will get better over time, I promise you. Here is a suggested agenda:

Agenda Template for Review deck/meeting

• Establish a mindset of ProdOps as a role or function; it can vary a lot from company to company if/how people collect product data, but one thing that always blows my mind is how well organized the sales and revenue teams are when it comes to data. Normally one of the first hires on a sales/revenue team in a startup is to find a SalesOps person. I mean of course, for a SaaS company revenue and customer data is immensely important and therefore an ARR or segment dashboard (from SDFC) is set up quite quickly, including forecasting metrics. Why can’t we get this serious in product quicker? ProdOps related tasks can either be split up amongst the PM and EM, or even by the head of product until there is a real need/budget to hire a ProdOps person who will own the tracking of product (launch) features, velocity and outcome.

Data Literacy Assessment

There are data literacy assessment tools available online that you could leverage or alternate based to your liking. Besides your knowledge of how to use and analyse data, of course you need the buy-in from (mostly) Engineering to help make the data accessible in the first place.

I also encourage you to read this Reforge article providing an insightful solid framework to evaluate your data maturity, and where you want/don’t want to go.

Stay tuned, I’m thinking of writing another article about specific data points by doing a deep dive into qualitative and quantitative data points for SaaS products.

*Data Literacy: Gartner defines data literacy as the ability to read, write and communicate data in context, including an understanding of data sources and constructs, analytical methods and techniques applied, and the ability to describe the use case, application and resulting value.