Evaluating Product Experience Via The Lens Of Data Driven Guidance

Pre-Intro

In this guide, you will learn firsthand how product experience can be assessed. To achieve that we will showcase real examples of product onboarding practices. We encourage you to put those suggestions to work. Always, by taking a concerted focus on the particularities your products’ have.

Introduction

Stop for a minute and think. How many digital experiences do you encounter daily? At work, on your mobile, when you read the daily news? Our life is quite full of them. Daily millions of interactions are taking place by the second.

No matter in which organization you belong, it is imperative for your product to deliver stellar product experiences. If you are a user, you experience first hand the frustration a broken experience carries and the satisfaction product delights deliver.

The Emerging Product-Led Growth Era

As Product-Led Growth emerges, businesses need to fall the weight of responsibility on internal teams’ shoulders. Silos abandonment, makes it very clear that customer experience does not have a single owner. On the contrary, it is compiled by the sum of touchpoints that originate and end up in the product itself.

Sales practitioners need to close deals reflecting on the organizations’ vision and product mission. Product leaders need to deliver calculated results. Customer Success needs to be proactive and educate at scale.

We may think that we are in a transitioning era. But the truth is that  Product-Led Growth has already sow its seeds. It spreads like wildfire and its profound influence passes a very clear message. SaaS organizations need to either evolve by delivering a customer-centric product approach or become obsolete.

What is Product Experience?

Product experience (PX) is the part of the customer journey executed in-app. It is the point where users get onboarded, learn about new features, and realize value. Today’s product managers must, at all times, understand and improve product experience to create products customers love and fight churn rates. As a matter of fact, Pragmatic Institute found that 52% of users said a bad product experience makes them less likely to engage with a company.

Why is Product Experience important?

Most organizations fail to understand how users derive value from their products. Studies have shown that 80% of SaaS features go virtually unused. Something that costs around $30 billion in wasted R&D annually.

At the same time, usability and UX are the points where most organizations invest. While those are serious investments they are not enough to meet customers’ needs. Product experience needs to educate, engage, and seamlessly adapt to users’ needs. Product-Led organizations, already deliver that kind of product experiences, the kind that makes customers keep coming back for more.

Product Experience Assessment: The Road So Far

The entire SaaS industry has been sold into a false dichotomy. A sales strategy can be self serve or human-assisted.  Τhere is no in-between- or so we’ve been told. The optimization of product experience is a discussion that does not seem to stop. On one hand, organizations cannot yet deliver products without creating an experience gap, while on the other, product engagements are usually overlooked.

Customer Onboarding

On customer onboarding, product experience comes secondary to customer-facing teams’ activations. Partly the disposal of human-assisted activations resonates.  Due to the plethora of stakeholders involved. Product Led Growth, however, flips the script and projects the product as the main growth lever.

Stellar customer experiences are not defined, anymore, by the buyer-vendor relationship dynamic. Product experience can now leverage context of usage. While at the same time, products must educate, engage, and adapt to their users’ needs.

Self-Serve Onboarding

On user onboarding, “the machines” prevail. Our product experience research showed that self-serve adopters invest in activation (61%).  But, retention and expansion fall behind. This fact alone leads us to the following conclusions:

• For one it indicates that onboarding prevalence ends upon the activation stage.
• The limited investment in retention makes accounts susceptible to churn.

Product Management Activations

Product Management’s ongoing feature releases, also impact product experience. Product leaders should invest in a JTBD framework and contextual guidance to leverage user onboarding activations.

That realization though may be far from true. Our research showed that Product Management is accountable for the onboarding process (81%).  But critical indicators like usage (5%) and product engagements assessment (5%) are neglected.

Product-Led Onboarding

So, how can onboarding assess product experience? For starters, Product-Led Growth practices radically change its nature. Product-Led Onboarding, driven by product data, can be evaluated on every stage of the customer journey.

Product managers can now reactivate the user onboarding process on every release and acknowledge how to deliver value. Following that logic, PMs can assess where onboarding leads to upgrades, and accounts’ expansion. Capitalization in product data derive insights on every move a user makes in-app.

This process abandons the traditional sales model archetype, ending onboarding prevalence during activation. The sales funnel has evolved into a circle where onboarding stands in its center waiting for the next release, to deliver value again.

Assessing product experience

It won’t be long now until new terms will describe the intimacy levels of the User-Product relationship. The most common term so far is the Product-Qualified lead (PQL). The PQL term refers to prospects that signed up and demonstrated buying intent. While the criteria following it are product usage and behavioral data.

Being limited to the point where a paid conversion is made, PQLs have as their benchmarks early adoption. PQLs as a metric is invaluable to Sales teams but, if organizations want to assess users’ behavior on every stage of the funnel they need to consider Product OQLs ™ (Product Onboarding Qualified Leads) in their day-to-day evaluations.

Product OQLs™ rely on POEs metrics to segment in-product behavior. As with PQLs again in OQLs, there is no absolute in regards to which metric should prevail. Balance among breadth, depth, frequency, and efficiency of use though, is a good foundation internal teams can consider.

Product Experience Variable: Breadth of Use

A form of (team) activation, breadth helps PMs realize the extent a product is being used on an account level. As a product metric,  it monitors account health and helps PMs manage churn.

Enterprise Customers

The internal buy-in from buyers and end-users discourages the onboarding process deployment. Heterogeneity on skills and heavy workflows decrease the willingness to adopt a new solution. Focus on breadth, usage and use case will help internal teams overcome this barrier.

Self Serve Customers

1. In the absence of Sales and Customer Success, in-app flows should double down on team activation to embrace activation rates. How team onboarding increases perceived value should be the number one consideration.
2. Product teams should define which roles cause various drop-offs on their first-week cohorts.
3. Training procedures need to focus on both users’ roles and team training.

Breadth of Use- Points to consider

1. How many users log in during onboarding and how many later on?
2. How many users activate upon new releases (Assuming that product experience targets specific user segments that realize value with those features)?

Use Case: Hubspot (Team Activation)

One of the key activation metrics predicting usage for Hubspot is if team activation within an account was realized successfully. Product teams track team activities to realize if the solution is driving value. Internally, team activation associates with Hubspot’s long term success and substitutes its north star metric as well.

Product Experience Variable: Depth of Use

Enterprise Customers

Points of consideration here are if end-users grow within the product itself. Internal teams should track daily usage on an account and individual level. On top of that, milestones per role should be set in conjunction with users’ progression over a 12 months period.

Self Serve Customers

On high trajectory customers, desired depth levels are usually met. The increased involvement of customer-facing teams makes this inevitable. Self Serve customers, however, may never get there since lack of personalization and low investment in tailored onboarding flows, usually lead to high churn rates.

Depending on each organization’s practices, adoption may be evaluated within the first 7 days or after the end of the trial period.

At this point milestones should be set considering:

1. Since SMBs have small teams, depth of use should be assessed in conjunction with breadth to assess engagement levels.
2. PMs should consider renewing milestones regularly since self-serve customers are not usually bound by contracts.

User Segmentation Use Case: Gainsight PX

FIRST TIME ACTIVATION

Gainsight PX, segments user behavior and usage on the first-time activation. It does that by tracking if PQLs use key features during trial. When usage overcomes the freemium plan’s levels, internal teams measure conversions and generated revenue from the trial source.

NEW PRODUCT RELEASE

On a new product release, users’ behavior is segmented by assessing adoption. The product team releases targeted in-app guides based on historical usage. The Query Builder feature, for example, is presented to users who have shown interest in other analytics areas in the last 30 days. If the released feature is a paid-only module, the revenue is measured by attaching a rate to it.

” width=”1024″ height=”576″ />

Product Experience Variable: Efficiency of Use

The difficulty to complete common tasks is critical when evaluating product experience and onboarding effectiveness. For the right measurements to be in place product teams need to know the number of users per account who begin a task vs. those completing it.

Enterprise Customers

1. Being a composite of human-assisted and product activations in this instance,  efficiency does not rely only on onboarding effectiveness. That being said, the emerging investment in product data, turns product experience into the growth lever following accounts’ long-term prosperity.
2. When onboarding many teams the focus should be on users’ role early on to assess usage and adoption. That realization by default indicates Sales involvement in the onboarding process. This how onboarding will be assessed from the very beginning and how harmonization between high touch & high tech activations will be achieved.

Keypoint One:

Sales teams should claim ownership of trials’ product data and observe product-qualified leads behavior. Capitalization on passive feedback enables sales teams to:

• Pinpoint where product experience is broken.
• Acknowledge how they should capitalize on end-users workflows (context).
• Enables Sales to onboard the first team(s) and pass the necessary feedback to Customer Success.

Keypoint Two:

Customer Success analyzes Sales feedback and suggests the next steps going forward, based on real insights. Capitalization on product data, help Success practitioners:

• Get the internal buy-in easier
• Be aware of the paths users take in-app
• To act proactively whenever flows onboarding downgrade product experience

Keypoint Three:

Product management monitors users’ usage and trends.  The criteria here are again their role, profession, and proficiency level. Points of consideration are when the desired adoption levels are realized with or without the help of customer-facing teams.

In case the levels of human-assisted activations are beyond the expected levels, Product Management should inject those learnings in product onboarding flows to increase engagement levels.

Self Serve Customers

Having the product replicating human-assisted activations requires continuous iterations and heavy experimentation. Being autonomous and subject to a faster sales process Self-Serve customers should realize initial value fast without neglecting the required learning curve period reliable to products’ complexity.

Internal teams need to define the number of steps leading to adoption by considering that users familiarize themselves with the product by following their own pace.

Use Case: Drift (First-time activation)

Drift’s former onboarding was a quick process constituted by three steps, aiming to get users to install its javascript code. That resulted in high levels of churn as free users did not feel invested in the product.

To deal with that, the onboarding team launched a ten steps long onboarding flow. Each step was motivating users to complete three different tasks. The end goal was to create many wow moments until the set-up is complete. The flow has proved to be a tremendous success as it tripled conversion rates.

Use Case: Trial period (First-time activation) optimization

When Yesware, decided to invest in a Product-Led Growth strategy, one of its first goals was to build an infrastructure allowing rapid testing. In this vein, the internal teams decided to run an experiment that halved the free trial length from 28 days to 14.

THE EXPERIMENT

The hypothesis was that while the conversion rates would remain steady, the product team would benefit by being able to run tests twice as fast. Additionally, there was the expectation that by shortening the trial period a sense of urgency would be provided.

Because of certain technical restrictions, the test was run longitudinally as opposed to an A/B. The product team changed the trial length and compared the 14-day trial cohort to the preceding 28-day. After a month of testing, the results were fantastic!

As expected, there was a slight increase in the percentage of users (0,5%) who uninstalled the product during the trial. Overall though, the core hypothesis was valid since early engagement rates increased. That stood particularly with key features, which are the leading indicators for the solution’s power users. An effect assumed to have happened due to the urgency created. Moreover, conversion rates did not only maintain but increased by roughly 18%.

SPECIAL NOTE:

After rapid experimentation, the product and engineering team have decided to extend the 14-day trial to 28 days. That would stand only for users who completed specific actions, based around sticky features, in the product onboarding guide.

Use Case: High-Touch vs. High-Tech Harmonization

Userlane, launched a while ago, a scalable onboarding process to complement its customer onboarding strategy. The “academy” as it is being called internally, self serve end users all the way through without eliminating the required decision making between the two parties.

Customer Success calls that learning process “getting to Basecamp”. The various teams build their first Userlane guide, publish it and in the end they get certified. The process is highly transparent since users are aware of the tasks they need to complete. In addition, buyers acknowledge end-users progress at all times.

THE INCEPTION PROJECT:

The launch of the “Inception project”, improved a lot the onboarding process delivery. The implementation of targeted in-app guides allowed a personalized, scalable onboarding process that didn’t need active participation from different units. Time to initial value decreased and post-launch buyers could go through the setup completely by themselves.

INCEPTION PROJECT DESIRED DELIVERABLES

• Increase of engagement and activation
• Reduce time-to-value
• Decrease time to key features
• Decrease onboarding completion time
• Increase trial-to-paid conversions by at least 30%
• Reduce the costs and efforts connected to high touch activities by 60%

Across the spectrum, Userlane managed to optimize product delivery without compromising onboarding execution.

INCREASED ADOPTION RATES (+45%) CONFIRM THAT THIS APPROACH:

1. Yields the expected ROI.
2. Keeps end users engaged and buyers aware of end results.
3. Reinforce stickiness by actions capitalizing on product engagements.

Userlane’s Inception project proved one thing. Organizations requiring tailored implementations can indeed harmonize scalability with human-assisted activations.

Product Experience Variable: Frequency of Use

Frequency of use estimates how often and to what degree users engage with features. Reminding users why a specific feature is there in the first place and how it may further optimize their workflow is also something reliant on onboarding activations.

Enterprise Customers

In regards to high trajectory customers, usage levels are being closely monitored by Customer Success. Product-Led Growth embraces PMs’ involvement in the process too in order to deliver quantitative measurements in regards to adoption or churn levels.

Both teams should create common benchmarks and track whenever features’ usage drops. Following this logic targeted product activations should be in place, to restore engagement levels when necessary.

At the same time internal teams should monitor :

1. When human-assisted activations prompt users to return to the product
2. And if those learnings can be injected into the product onboarding flows.

Self Serve Customers

Having Self Serve customers returning to the product is something reliant both on product and email activations. In this instance, email practices replicate human-assisted activations. So, it resonates that customer-facing teams should adopt the right tools to monitor this correlation. Solutions like Gainisht PX mapping that kind of activities, come handy in this case.

In addition, internal teams should invest in customer feedback early on:

• Sales should suggest what learnings to insert in the trial, per use case.
• Product Management should consider passive feedback
• Customer Success should release in-app surveys by focusing on the features paid accounts to exploit vs. those neglected.

Frequency of Use- Points to Consider

1. Which features are used frequently?
2. Which features correspond to each team’s use case?
3. How can customers’ teams be triggered in-app to increase secondary features usage levels?
4. When and why each feature should be used? ( to better estimate the revenues and losses following them)5. What made users return to the product or abandon it?

Product Experience Adoption Loop

All things being equal, Breadth, Depth, Frequency, and Efficiency of use form an adoption loop.

The loop’s implementation is viable when:

1. The sum of account’s users exploit a product
2. use its features end-to-end
3. And repeat those actions often enough.

Again, the decisive role initiating this circle of events is (team) activation. But for the loop to be consistent, at every stage of the customer journey, all four metrics should be considered. Depending on the onboarding strategy the loop is being supplemented by additional business KPIs and benchmarks.

Product-Led Key Takeaways

Product Management

The product team should be involved in the activation process, no matter the onboarding strategy at hand.

Sales Organization

The sooner Sales’ decode the messages withheld into product data the better the conclusions drawn, in regards to the harmonization of human-product activations.

Depth of use has many variables to consider:

1. If during activation users can realize initial and true value. This is most likely to occur on high-velocity customers where the onboarding may take place post-purchase.
2. Users’ skills need to be defined early on. An advanced user may explore more key features during trial. However, no matter users’ proficiency levels products’ complexity should always be considered.
3. On Self Serve onboarding, retention’s initiation point varies, thus Depth of Use may not be measured on the activation stage (during trial) but only post-purchase.

Product-Led Key Takeaways

• Customer Success & Product Management: Both departments are the rightful owners of accounts retention and customer journey’s optimization.
• Depth: The retention stage defines which features prevail over the others. It is optimal at this point, internal teams to define when accounts reach the desired levels of adoption and show indications of expansion.
• Frequency: Whenever frequency of use increases so do the indications of those accounts reaching long-term retention.

Product-Led Key Takeaways

• Expansion: Accounts reaching expansion meet all four POEs standards. This the point where the variables following the adoption loop take full effect.
• Frequency of Use: Despite the impact, frequency has on the customer lifecycle stages, renewals and expansion teams should also consider:

1. Exploring usage-based pricing when the majority of accounts have surpassed the expected levels.
2. Which existing and upcoming features increase upsell and cross-sell opportunities.

Conclusion

What should you do next? Whether you invest in delivering a stellar product experience or not, at the end of the day users still evaluate product delivery. A bad experience is still an experience, following customers’ growth or just preventing it. Transforming your product experience to a data-driven force is not optional anymore. It is the only way to reach the desired growth levels.

Product-Led Growth constitutes a challenge, as it dictates alignment across an entire organization. But at the same time, it leverages and maps product experiences to the very end.  Take a closer look at the examples provided above and try to go out of your way.

Experiment and iterate until you reach the point where you can predict the customer journey’s anomalies. Try to remember, while there is no panacea for evaluating users’ behavior, the sole consideration of business & marketing metrics will lead to miscalculations.

Keep the conversation going

We are convinced that Product-Led Growth is key to achieving cross alignment and product success. Something not possible if benchmarks driven by users’ actions are not in place. Revisit your product delivery, by working on practices like the ones discussed above, and feel free to reach out to discuss the results. We are always keen on taking your feedback, brainstorming, and helping you deliver product-led experiences leading to measurable growth outcomes.