Creating Simple and Extensible Product Reporting

Throughout my career in product, I’ve seen severeal colleagues and companies struggle with product reporting. For one reason or another, they feel they are not getting the information they need in order to make smart decisions. Most of the time, folks will try to make do with what they have and they’ll take a targetted approach to filling in what they think are the biggest and most importent gaps. This might sound familiar to you, especially if you are working with an already existing product and need to stay nimble. Still, you have to keep in mind that sooner rather than later, you’ll be back in the same position, wondering about all the data that you’re missing.

What if you’re creating a completely new product, re-architecting your existing one, or have finally decided to throw down and do reporting right for a change? This is an awesome opportunity, but of course a daunting challenge. Where do you begin? As with any large undertaking, before getting started, take a step back to contemplate what you really want to get out of the overall process. Build a guiding philosophy that will help you make the right decisions for you, whatever they are. Specifically for reporting, I peronally start out with the following list:

  • Covers the primary business needs: While it should go without further explanation, this is where you have to really do some critical thinking – the same level that you did when you defined your minimum viable product. You’ll probably have a sub-list here which states what those business needs are. The important thing is to be focused without being myopic.
  • Is easy for you AND OTHERS to understand: You know your product the best, don’t expect others to instantly get it. The same goes with reporting. Lots of companies have teams that are dedicated to churning through numbers and producing beautiful, insightful results. Chances are that if you’re reading this, you’re not working at one of those companies. You need to be able to make it broadly understandable without major work. This is also where you should reinforce the need that your analytics needs to be documented as much as the rest of your product.
  • Is easy to implement, maintain, and extend: Give your engineering team a break for once. Nobody likes a crazy convoluted system that requires a lot of custom work. Collaborate with them and keep in mind that the extra effort spent towards a structured, logical approach will make everyone’s lives easier.
  • Covers what you may not realize is important in the future: This is why we’re here in the first place – at some point we’ve all tried to get some insight about our products only to find that it hasn’t been tracked before. We’re back at step one. This is why many experts will advise you to track everything. If it moves, track it.

 

This guiding philosophy should be established well before you’ve started defining a taxonomy, picked a reporting system, or even started defining your end reports. You’ll find that with the proper start, the subsequent tasks actually become easier as you move forward.

For a moment, consider some other approaches. If you go in with the “requirement” that you need a funnel report for your registration process, that’s the only thing you’ll come out with. Likewise, there are many reporting products with variety of features that can help you visualise your product engagement, but concentrating on their features is not the same as concentrating on your goals. Finally, if you just dive right into defining a list of a few reports, you’ll probably miss out on something that is very important to you. When you design your reporting in the right order, everything builds upon the previous step.

One activity that may help you understand the scope of your reporting is to break down the types of data that you will be reporting on. Most places have at least the following two buckets for their reporting:

  • Event Data: This is the user’s activity – analyzing the features used most widely or the steps users have taken to complete a task within an app. It is helpful in determining user activity, daily active users vs. monthly active users (general engagement breadth). Any time people saw something, interacted with something, converted or transaced, etc. is covered here.
  • Object Data: While event reporting will tell you what people did, it is not good on detailing the overall profile of your user base or other elements of your product. For example, you may have a product where people create documents. Event data can tell you how many documents people have created on average, but if you also give people the ability to delete a document, then you need to have additional information in order to see how many documents people still have. This is where your object data comes in. Object data can be information about each user (name, address, date of birth, number of transactions, revenue earned for the user, where they came from, what devices they've used, etc.). Object data may be other items that make up your product (items, articles, images, etc.). Much of this is application specific, but we’ll try to cover the basics.

 

Many companies will have additional data that they’d like to combine with their reports. This may be data from 3rd party vendors that may align well with the event or object data. Alternatively, it may be information like the weather, stock prices, or other external factors that may want to correlate with certain events that are being tracked. Again, like object data, this is very application specific. You’ll have to have a good understanding of the data, where it comes from, and how it relates to your event and object data in order to combine into a cohesive reporting system.

Let’s continue by working through a hypothetical, yet common situation. We want reporting on a product that is represented over the following different environments:

  • On the Web:
    • - Desktop web and mobile web experiences
    • - Across different brands or different marketing ecosystems
  • Mobile Applications:
    • - Native experiences for iOS and Android
    • - Phone and Tablet experiences
    • - Different apps for different purposes
    • - Possibly even hybrid experiences within each app that allow you to reuse the mobile web pages that you've previously built within the app
  • E-mail:
    • - Mobile optimized e-mails
    • - Links in the e-mail direct people to a web experience for the most part
  • Marketing:
    • - Search engine optimization, Search engine marketing, display ads online, etc.
    • - Typically linking back to the web experience
  • Additionally, as with many products, it could also have both an anonymous state and a signed in state (and hence there could be a registration, login, and even purchase process).

 

As the first step of our philosophy, we want to list out some goals for our reporting. As always, goals should be straightforward and measurable, just like your reporting system! A simple list in the form of questions to help understand the key drivers of the business may be as follows:

  • What are the key things our users are looking to get out of our product?
  • Where are our users coming from?
  • How often do they come back and what drives them to come back?
  • What are the top things users do while engaging with the product?
  • What drives people to convert or transact?  How long did it take?
  • How much did we have to pay in order to make that revenue?

 

These may seem a bit broad, but consider not being able to answer these questions. How long do you think a business could survive? This is the difference between success and luck.

Moving on to our next two steps of making it easy to understand and easy to implements. Ideally, you'd want to be able to track everything using a consistent taxonomy and report on it all within the same system. That way you'd be able to do things like roll up and segment who saw a single item across all the different environments very easily. What you don’t want are different systems for each individual environment – you won’t be able to tell if engagement on one platform drove conversion on another platform. Furthermore, nobody wants to go to multiple reporting systems to get incomplete insights, and frankly no team wants to implement and maintain multiple reporting systems.

From our list of goals stated above, we find that it’s very action oriented. We ask what drives a user to do something. From this we can start a concentrated effort on event data. If you’ve seen or worked with some of the popular event tracking systems such as Mixpanel, Amplitude, and others then you’ll be familiar with their typical layout of events and event properties (some systems call them attributes). Every time a user performs action A, that gets logged in the system. All the available information about A and the user who performed the action will get logged as well as properties to action A.

This is where some people veer off course in keeping their reporting simple. There are many articles people can refer to when investigating their event data. Amplitude has a couple of great articles (How to Approach Your Event Taxonomy and 10 Steps to Behavioral Analytics). Still, folks may be making their event data too complicated by trying to track every single action individually. Even the Amplitude articles are a little too lax in stressing the importance of keeping it simple and small. For example, they suggest keeping your event list between 20 and 200 different events – even 20 may be too much for many products. In another example, they mention tracking a user flow from “Purchase Coins” to “Purchase Gems”. This is a situation that would still be better off tracking as a single event of “Purchase” and making sure that the funnel analysis can distinguish the different types of purchases from the properties.

Here are a couple of other articles regarding keeping your event tracking simple:

 

Using the hypothetical situation listed above, most events can be tracked with the following simple list:

  • View Page: Person viewed a page or screen in any of these environments
  • View Offer: This would be a view of a product or sale, etc. you could split out view offer and view product separately
  • Click: When the user clicked on anything
  • Completed: Something happened in the back-end or in the background that the user initiated, but doesn't necessarily constitute a click
  • Failed: Some kind of error occured
  • Registration / Login: These are events that were triggered by the user, but in a way complete on the back-end
  • Notification: Any kind of e-mail, alert, or other type of notification that gets sent to the user should be tracked.

 

Make sure these event names are human readable. Nobody wants to go look at a report that looks like code, even if they are an engineer. Make is easy for people to understand. If this list seems too restrictive, keep in mind that you want to make the list of event properties as wide as possible. You want to know just about everything that was related to the action that the user performed. You can categorize these event properties as follows:

  • Event Specific Properties:
    • - Environment: Was the user on your web page or within the mobile app? Is the user clicking on an e-mail?
    • - Page Name: Like home page, category page, search results page, item page, checkout page, purchase completed page, etc.
    • - Click Destination: If they clicked on something, where they are going or doing (like Buy)
    • - Site: These would be for the different brands or apps you might have
    • - Search query: If they searched for something what was it
    • - Page Responsiveness: Whether it was via a desktop web page, mobile web page, hybrid, app, e-mail, search, etc.
    • - Completion Type: If it is a completed event, the type, like e-mail sent, notification sent, registration completed, login successful, purchase successful, etc.
    • - Failure Type: If it is a failed event, the type, like login failed, password incorrect, Touch ID failed (like we discussed), purchase failed, credit card denied, etc.
  • Product Properties: This is another element that will be very specific to your own product.
    • - Product Name
    • - Product Category
    • - Product ID
    • - Product Manufacturer
    • - Product Price
    • - Product Size
  • Device Properties: While this is related to a user, it is well known that people use more than one device, so it makes sense to track it. Much of this information should get tracked by your event tracking system. You don’t want to be the one maintaining this list because it only continues to grow rapidly.
    • - Device Type
    • - Operating System
    • - OS Version
    • - Browser
    • - App Name
    • - App Version
  • User Properties: Although most of the time you want to keep data about your users separate, there are some things that you may want to track for all the events because they may change or they may be for a user that is not registered yet. In a way it’s like taking a snapshot of the user at the time of the event.
    • - Unique ID for the user
    • - Logged In: Whether or not the user is logged in at the time
    • - Marketing Source: Did the user come from google, an ad campaign, directly, etc.
    • - AB Test Variations: While AB testing is a whole separate topic, you’ll want to know how any of your tests are effecting the events that a user performs.

 

Again, when constructing your event properties, make the names and values as human readable as possible. Another thing to consider when constructing your event properties, is to make sure that it aligns as closely as possible to your actual product. This is where you have to work with your engineering team. Remember, they are the ones who will typically implement and maintain this and they want to have a system that is simple. Get as much of their feedback and input on this step as possible. Everyone will be glad you did.

One note about user privacy here. If you’re using a 3rd party service to do your tracking (or even if you’re doing tracking yourself), do some thorough due diligence with your engineers and even your legel consul. The last you want is to expose your users’ personally identifiable information in any way. Remeber that they are people who are placing some level of trust in you. Keep that trust.  Here’s a short to do list:

  • Be very picky about what absolutely needs to be tracked. Do you need the user’s address tracked? Or can a simple city/state combination work?
  • Look for ways to obfuscate information that is unique to the user. You probably don’t need to track the user’s name and e-mail address or their actual user ID in your database. It is better to create some kind of unique random hash that you can then relate back to your user only when needed. This will still enable you to report on unique users and their engagement with your product.
  • Many systems use a JavaScript snippet to track events. Consider a server to server connection to include the user information you want to track instead of passing it up to the user’s browser.
  • Read up on your privacy policy and talk to your legal consul about what you plan on tracking. Does your privacy policy cover it? Look to the EFF for best practices.
  • Let your users know what is being tracked and give them an option to opt-out of being tracked when possible. You can always measure the percentage of users who have opted out and include that with any standard deviation you calculate.
  • Make sure that your product and your event tracking is using a secure, encrypted connection.

 

When all is said and done, you should have a simple and well documented list of events with a wide list of well defined event properties that tracks every action the user performs. You’ll find that you’ve actually covered the final item in the reporting philosophy – covering what you might not initially realize is important. When you create your first reports, you might not be using every single event you’ve ever tracked, but only a relatively small portion. But you’ll be glad you put the effort into tracking as much as possible (and as simply as possible), because in six months you’ll have a question about activity then and how it’s different from activity now. My hope is that when you get to that point, this article will have helped you be prepared for it.

Further Research

Comments (0)

The content of this field is kept private and will not be shown publicly.
CAPTCHA
This question is for testing whether or not you are a human visitor and to prevent automated spam submissions.
1 + 6 =
Solve this simple math problem and enter the result. E.g. for 1+3, enter 4.