Throughout my career in product, I’ve seen severeal colleagues and companies struggle with product reporting. For one reason or another, they feel they are not getting the information they need in order to make smart decisions. Most of the time, folks will try to make do with what they have and they’ll take a targetted approach to filling in what they think are the biggest and most importent gaps. This might sound familiar to you, especially if you are working with an already existing product and need to stay nimble. Still, you have to keep in mind that sooner rather than later, you’ll be back in the same position, wondering about all the data that you’re missing.
What if you’re creating a completely new product, re-architecting your existing one, or have finally decided to throw down and do reporting right for a change? This is an awesome opportunity, but of course a daunting challenge. Where do you begin? As with any large undertaking, before getting started, take a step back to contemplate what you really want to get out of the overall process. Build a guiding philosophy that will help you make the right decisions for you, whatever they are. Specifically for reporting, I peronally start out with the following list:
This guiding philosophy should be established well before you’ve started defining a taxonomy, picked a reporting system, or even started defining your end reports. You’ll find that with the proper start, the subsequent tasks actually become easier as you move forward.
For a moment, consider some other approaches. If you go in with the “requirement” that you need a funnel report for your registration process, that’s the only thing you’ll come out with. Likewise, there are many reporting products with variety of features that can help you visualise your product engagement, but concentrating on their features is not the same as concentrating on your goals. Finally, if you just dive right into defining a list of a few reports, you’ll probably miss out on something that is very important to you. When you design your reporting in the right order, everything builds upon the previous step.
One activity that may help you understand the scope of your reporting is to break down the types of data that you will be reporting on. Most places have at least the following two buckets for their reporting:
Many companies will have additional data that they’d like to combine with their reports. This may be data from 3rd party vendors that may align well with the event or object data. Alternatively, it may be information like the weather, stock prices, or other external factors that may want to correlate with certain events that are being tracked. Again, like object data, this is very application specific. You’ll have to have a good understanding of the data, where it comes from, and how it relates to your event and object data in order to combine into a cohesive reporting system.
Let’s continue by working through a hypothetical, yet common situation. We want reporting on a product that is represented over the following different environments:
As the first step of our philosophy, we want to list out some goals for our reporting. As always, goals should be straightforward and measurable, just like your reporting system! A simple list in the form of questions to help understand the key drivers of the business may be as follows:
These may seem a bit broad, but consider not being able to answer these questions. How long do you think a business could survive? This is the difference between success and luck.
Moving on to our next two steps of making it easy to understand and easy to implements. Ideally, you'd want to be able to track everything using a consistent taxonomy and report on it all within the same system. That way you'd be able to do things like roll up and segment who saw a single item across all the different environments very easily. What you don’t want are different systems for each individual environment – you won’t be able to tell if engagement on one platform drove conversion on another platform. Furthermore, nobody wants to go to multiple reporting systems to get incomplete insights, and frankly no team wants to implement and maintain multiple reporting systems.
From our list of goals stated above, we find that it’s very action oriented. We ask what drives a user to do something. From this we can start a concentrated effort on event data. If you’ve seen or worked with some of the popular event tracking systems such as Mixpanel, Amplitude, and others then you’ll be familiar with their typical layout of events and event properties (some systems call them attributes). Every time a user performs action A, that gets logged in the system. All the available information about A and the user who performed the action will get logged as well as properties to action A.
This is where some people veer off course in keeping their reporting simple. There are many articles people can refer to when investigating their event data. Amplitude has a couple of great articles (How to Approach Your Event Taxonomy and 10 Steps to Behavioral Analytics). Still, folks may be making their event data too complicated by trying to track every single action individually. Even the Amplitude articles are a little too lax in stressing the importance of keeping it simple and small. For example, they suggest keeping your event list between 20 and 200 different events – even 20 may be too much for many products. In another example, they mention tracking a user flow from “Purchase Coins” to “Purchase Gems”. This is a situation that would still be better off tracking as a single event of “Purchase” and making sure that the funnel analysis can distinguish the different types of purchases from the properties.
Here are a couple of other articles regarding keeping your event tracking simple:
Using the hypothetical situation listed above, most events can be tracked with the following simple list:
Make sure these event names are human readable. Nobody wants to go look at a report that looks like code, even if they are an engineer. Make is easy for people to understand. If this list seems too restrictive, keep in mind that you want to make the list of event properties as wide as possible. You want to know just about everything that was related to the action that the user performed. You can categorize these event properties as follows:
Again, when constructing your event properties, make the names and values as human readable as possible. Another thing to consider when constructing your event properties, is to make sure that it aligns as closely as possible to your actual product. This is where you have to work with your engineering team. Remember, they are the ones who will typically implement and maintain this and they want to have a system that is simple. Get as much of their feedback and input on this step as possible. Everyone will be glad you did.
One note about user privacy here. If you’re using a 3rd party service to do your tracking (or even if you’re doing tracking yourself), do some thorough due diligence with your engineers and even your legel consul. The last you want is to expose your users’ personally identifiable information in any way. Remeber that they are people who are placing some level of trust in you. Keep that trust. Here’s a short to do list:
When all is said and done, you should have a simple and well documented list of events with a wide list of well defined event properties that tracks every action the user performs. You’ll find that you’ve actually covered the final item in the reporting philosophy – covering what you might not initially realize is important. When you create your first reports, you might not be using every single event you’ve ever tracked, but only a relatively small portion. But you’ll be glad you put the effort into tracking as much as possible (and as simply as possible), because in six months you’ll have a question about activity then and how it’s different from activity now. My hope is that when you get to that point, this article will have helped you be prepared for it.