Lean Experimentation – the future of retail UX?

Marek Szeszycki is a UXer from Gap with a wealth of experience in the industry. He currently leads a team consisting of data analysts, product managers, developers and designers.

In March, we welcomed Marek to inform us all how using the Lean Experimentation method for improving customer services for GAP’s outlets can revolutionize the way we innovate.

The goal of this talk was to give you a better understanding of what Lean Experimentation is, the practical methods of putting it into action, and to give you ideas about how you can add lean experimentation to everyday practices to mitigate risk and learn the most from the least amount of effort.

Lean Experimentation is a type of rapid experimentation based on the principles of Lean UX, conceived with the idea of quickly converting ideas for a product or service into iterative experiments. The idea is to eliminate the bad ideas quickly, in addition to discovering viable solutions as fast as possible.

Failed experiments are looked upon as opportunities for the company to learn. It can be thought of as a type of rapid trial and error. Practitioners of Lean Experimentation work quickly to obtain feedback on MVPs and prototypes, then develop further iterations of these MVPs based on the feedback received. It helps teams to validate the ideas that will work best as quickly and efficiently as possible.

To do this, a “fail-fast; learn-faster” approach is applied to deduce and prioritize the riskiest assumptions. Why? Because invalid assumptions have the potential to turn into bad ideas, costing the company time and money. Weeding out these bad ideas gets you on the right track.

Lean Experimentation is thus considered more efficient than simply taking an idea, running with it and building it out, – only to have it fail further down the line, having wasted time, money and effort. It’s about learning as quickly as possible what works and what doesn’t, with minimum resources spent.

When beginning his explanation on Lean Experimentation, Marek makes reference to the book Lean Experimentation in Action, by Maryam Aidinin and Kylie Castellaw which outlines the actionable framework for the model in three main points:

1. Uncovering the assumptions
2. Experimenting
3. Interpreting the results

In order to experiment with and test new ideas, we first have to make assumptions about our users’ needs.

When various staff in GAP come up with assumptions, they can be divided into two main categories: valid and invalid, the latter, of course, posing the higher risk.

So how do Marek and his team go about this ideation and validation? In his presentation he sets out the assumption validation process as follows:

1. Validate the problem: in this instance, we first need to ascertain whether or not the problem is even worth solving in the first place. After all, if the users don’t consider it to be a problem, then the solution isn’t going to appeal to them.
2. Validate the market: once the problem has been validated for certain users, it’s time to look at a wider segment of the market – if the proposed problem is a pain point for enough customers, then the problem is continued into the next stage.
3. Validate the product or solution: we then need to find out whether the solution proposed actually solves the problem for the customer, before developing the idea further.
4. Validate a willingness to pay: if the problem isn’t big enough for the customers to be willing to part money for the solution, then no more time is spent working on it.

After isolating and prioritizing the riskiest assumptions, you need to experiment with them in order to weed them out and get set on the proper heading. The next step is the experimentation phase.

The Lean Experimentation model is split into three stages:

1. Build
2. Measure
3. Learn

It’s the latter, the measure-learn stage, that Marek places the most emphasis on. Learning is highly important with Lean Experimentation. Even assumptions which turn out to be invalid provide opportunities for deeper learning about what actually works, and can even pave the way to new ideas or assumptions. Marek then draws on Benson Garner’s blog post “don’t build when you can build-measure-learn”.

In this post, Garner states that the real goal shouldn’t be on building a limited version of their future product, but should instead focus on achieving the maximum learning possible with the minimum amount of effort. In this sense, we can think of the typical “minimum viable product” or prototype as being built to achieve this goal.

Marek then listed the possible techniques that can be used to achieve the KPIs and input from the user:

• Data sheets: excel spreadsheets can be used to compile data about users such as input from surveys or interviews, as well as value propositions for stakeholders and managers.
• Brochures or flyers: this is a way to attract potential early adopters.
• Storyboards: often used as an ideation technique, they can describe the customer or user’s journey when they interact with a new product, feature or service and can take a similar format to a comic strip.
• Landing page: a landing page can be put together quite cheaply to catch people’s interest by outlining the core benefits a new product, feature or service will offer. It can also be used to gauge their interest in the solution and in participating in testing. Marek advocates it as a great way of netting early adopters.
• Using videos: he states the benefit of making a quick video that can be shared where, like the landing page – the user interest can be gauged to decide whether or not to progress with the testing.
• Create a prototype: or what he calls a “learning prototype” – a high-fidelity experiment that you can give to your users to understand how they go about using the intended product or service . He recommends being lean with the prototype and only focusing on the assumption you want to test out.

What’s the method, how are we going to experiment? Who are the people we’re going to learn from? How can we learn from them? All these questions are answered in the learning agenda.

The agenda specifies how you can know when an idea has been validated or invalidated, and how to know when you have all the information you need. It gives you something documented to reference when interpreting results.

Many times during and after the experimentation phase, Marek describes the feelings of doubt that can kick in but is quick to emphasize that “our feelings are not defensible…validation and determining results against the KPIs, that’s defensible…”.

Having the facts and the details of the original assumptions to hand helps avoid being misguided by your emotions. During his speech he admitted that “It’s scary, it’s fast, and we’re constantly checking ourselves and we’re constantly second-guessing ourselves. “And when this happens, we refer back to the assumption we’re validating”.

We say he has a point. Will what we’re second-guessing actually impact what we’re doing, what we’re trying to learn and validate? The answer is most likely a resounding “no”.

Using qualitative and quantitative metrics for KPIs is what carries the real weight in these experiments. The idea from all this is to foster enough confidence to move forward and to know what’s next.

In one example, Marek describes using Lean Experimentation with his colleagues to come up with a method for allowing customers to access helpful product reviews and ratings on the shop floor. They knew that, from previous research, customers were heavily drawn to reviews to make their decisions.

They started off with the riskiest assumption: that customers were unable to easily-access reviews while being in-store.

Assumptions are based on the if/then approach, whereby, if [there is a given context or scenario], then [something will happen]. The if/then answer they came back with was as follows:

“If customers are able to easily access product ratings and reviews when in store, then customers will be more confident in the product and more likely to purchase.”

From this hypothesis, they form a how might we clause to address the assumption. After listing all the ideas in the how might we stage, they use a method called “lean limbo” where they determine the quickest, cheapest way of arriving at the solution.

The main question, then, to ask during the experiment stage was how they could use information from the website across different brands in GAP Inc. and bring it into the stores.

Marek and the staff came up with different solutions like creating an app where people can read customer reviews in the store, adding reviews to mannequins, as well as many others, though they eventually settled on adding pillar product (main products on the floor) ratings to repurpose signs for pillar products.

They determined the results of the experiment by conducting user intercepts with customers. They found that the original assumption about reviews and ratings not only turned out to be valid, but they also prevented a lot of potential losses in time and money developing apps and other forms of communication for the reviews.

Rapid experimentation to test out these invalidated assumptions can then convert them into valid or invalid assumptions, improving confidence in business decisions to build new features into an app or, in this case – in the retail sector – to invest in new, innovative methods to improve the customer or user experience.

Some of the tips he went on to give for how to operate while carrying out lean experimentation are to challenge yourself to be as lean as possible. Ask yourself: are you and your team achieving the maximum amount of learning possible in terms of cost, time, and effort?

He also encourages practitioners of this model to insist on setting the most “realistic-unrealistic deadlines” possible. Why? Because this keeps the team on their toes and encourages them to keep things moving as fast as possible.

Another important piece of advice he gives in this talk is to persist when experiments fail – there’s never a wasted experiment as long as it’s lean, and always provides some sort of lesson or insight that will be useful in the future.

Lastly, he places much emphasis on the importance of collaboration. Not everyone has the benefit of being able to work in a cross-functional team, and so he recommends at least requesting to have someone, from any team, to participate in the experimentation with you as it always helps to have a backup and a second pair of ears.

He places a need on good-quality empirical data and gathering information from the ground – in other words – roll your sleeves up and get your hands dirty kind of work. As Lean Experimentation is based on both qualitative and quantitative data, he actively encourages getting out of the office to research, learn, and collaborate with your potential users.

Design is a “team sport” according to Marek. In this talk, his main emphasis revolved around the power of quickly validating and invalidating assumptions using lean experimentation.

When highlighting the possible transformational benefit that Lean Experimentation bestows on companies and design teams to innovate, he referred to renowned Stanford University Psychologist, Carol Dweck’s book Mindset, which talks about growth and fixed mindsets. Individuals and groups with growth mindsets believe that abilities aren’t fixed and can be developed; they outperform those with the latter mindset.

In Marek’s opinion, it’s the contrast between the two that helps illustrate what separates companies that adopt lean experimentation and those that don’t: less innovation and less potential for growth. He maintained that Lean Experimentation is about constant learning – constant growth.

As Marek puts it: “Why do we do it? [lean experimentation]…it helps us act on truths. It give us confidence in the direction and execution and it’s defensible, we have a reason why and an answer for why we’re doing what we’re doing.”