How to create and validate hypotheses for product improvement

[#1]

Imagine you're developing an amazing product, but you're not sure which features your users really want. Here's a smart approach: ask a question, then test the answer. This method is known as "using hypotheses."

What's a Hypothesis?

A hypothesis is an educated guess about how adding something new will impact your product. For example, you might think, "If we add this feature, people will complete their projects faster."

Let's explore how to create reasonable hypotheses for products.

Step 1: Define variables

Identify independent (cause) and dependent (effect) variables. For example, introducing a "feature for adding email templates" is an independent variable, while expecting an "increase in email API usage" is a dependent one.

Independent variables might be product updates, such as revising landing page text or adding search filters. Dependent variables are typically measurable metrics: trials, subscriptions, monthly active users, etc.

Avoid ambiguous terms in your hypotheses. Instead of saying, "Users churn because the setup is hard," be specific: "Providing clear setup steps will reduce user churn."

Step 2: Link variables

The relationship between variables should be clear and logical. If it's not, regardless of how well-articulated your variables sound, your test results will not be reliable.

Common pitfalls:

• Weak relationship: increased traffic doesn't necessarily result in more trials. A more effective hypothesis would focus on modifying the product page's call-to-action (CTA) to directly impact trials.

• Made-up relationship: the assumption that "Increasing social media views will enhance our app users" may be erroneous. Social media users may be more attracted to your content than your actual product.

• Keep variables separate. For example, removing the "Sign up with Google" option will likely reduce users with Google Workspace accounts because the two are directly connected.

Step 3: Determine verification criteria for hypotheses

Establish specific metrics or outcomes to evaluate if a hypothesis holds true.

• Ensure outcomes are tangible and trackable. Avoid ambiguous criteria like "improve user satisfaction." Instead, opt for "Increase the average session duration by 2 minutes."

• Understand your current metrics before testing your hypothesis. For instance, if you have a 5% click-through rate (CTR) on a feature, and your hypothesis expects to increase it, you need this baseline for comparison.

• Test hypotheses within a specific period. For example, "We expect a 10% increase in new app installations in the next 30 days."

Examples:

Prioritizing Hypotheses in Product Development

Prioritizing hypotheses ensures your development efforts yield the most significant returns. Rank hypotheses effectively to allocate resources more efficiently.

1. Assign a score (e.g., on a scale of 1 to 10) for each hypothesis based on criteria like potential impact, feasibility, resource requirement, and risk.

2. Rank hypotheses by their scores. Those with the highest scores should be prioritized.

3. Revisit and re-prioritize hypotheses based on user feedback or shifting business goals.

Example:

In this example, despite its challenges, the voice-command feature is the top priority due to its game-changing potential. However, teams might tackle the onboarding tutorial first, as it's more feasible and has fewer risks. The key is to balance impact and feasibility while always keeping the user's best interests at heart.

At SaaSJet, we develop apps for Jira and leverage the wealth of data available on the Atlassian Marketplace to make informed decisions. To help every team member - from marketers to product managers to developers - explore this data effectively, we created a GPT - Marketplace Insights.

For example, here is how I used Marketplace Insights to analyze user reviews and gather insights.

The Benefits of Using Hypotheses

1. By testing hypotheses before fully implementing changes, you lower the risk of investing in features that might not deliver the expected results.

2. Hypotheses promote a data-driven approach, where decisions are based on evidence rather than assumptions, leading to more reliable and effective product development.

3. Regularly testing and refining hypotheses fosters a cycle of continuous improvement, helping to keep your product aligned with user needs and market trends.

So, stop guessing about what your product needs. Instead, make predictions, test them with real users, and see what the data tells you. 

[Joanna Weber]

These kinds of experiments often work well in combination - for example, depth interviews with surveys with analytics with asking a tool like this to tell you about competitors in the market.

The books 'Playing to Win' (Roger Martin & AG Lafley) and 'The Mom Test' (Rob Fitzpatrick) give further ideas on how to test hypotheses.

[Halyna Kudlak _SaaSJet_]

@Joanna Weber , you're right — combining experiments can give a fuller picture.

Thanks for sharing the books; they are very valuable!