Context

As one of two people in the company that looked at customer reviews on a daily basis, my most consistent touchpoint to our customer was through review management. On a daily basis, I was reading flagged ones, evaluating if it needs a comment, and managing the person who writes the merchant responses. We know from heatmap data (Hotjar) and usability testing that reviews are one of the first things our customer looks at, particularly when evaluating the fit of the garment, when on our product pages in the act of purchasing.

We also knew that Title Nine’s review standards are well above average in terms of breadth of coverage, depth of coverage, % of reviews from email, and fewer 1-2 starred products. For example, we had an average of 29 reviews per active product vs. the industry standard of 15.

We also knew there is a direct relationship between reviews and conversion rate -- the more reviews a page has, the higher the conversion rate (3.6% conversion rate with product that does not have a review vs. 4.6% conversion rate with product that does have reviews).

Feature #1: Review Filter Functionality

Tactic: Worked with vendor to plan, customize, implement and test new filter functionality that allows her to filter the list of reviews on every product page based on key attributes particularly regarding size and fit.

Role & Process: I worked directly with our Director of Development to provide project requirements and a style guide. The account manager of our vendor and I worked hand-in-hand as liaisons between their technical team and my developer team to ensure all details were communicated in regular calls and project management tools.

I provided direction to my team’s developers on customizing the new feature with CSS styling format. I tested the feature on our review site across all breakpoints and softwares.

After launching, I presented this new feature to a meeting full of all directors of the company.

Before

Impact: Through usability testing, we found qualitative data that customers enjoyed the filtered option as they were using the reviews to evaluate if the product would fit their specific body or not and the filters cut down the time to get that information.

However, to evaluate the feature’s impact even further, I turned to our A/B testing program.

We tested showing the reviews filters vs. not showing them. Results showed in a ten day period, a -5% revenue loss ($40K) when showing the filters.

These results were surprising, but upon deliberation, my hypothesis changed: when customers interact with review filters, the filtering presents unfavorable results (i.e. potentially more negative reviews, or perhaps no results at all).

We also were careful to take into account the fact that without the filters, she is potentially making a less informed decision and potentially would return product at a higher rate.

Given the revenue loss, we decided to temporarily change the variation to include 100% of users in the hide filter variation, ensuring no loss of revenue for the time being. The next phase of the roadmap took a look at return rates and strategized a new way forward given both quantitative and qualitative conflicting results. At this point, I had shifted to a role at another company.

Feature #2: “Best For” added to review form

After seeing success from our Bottoms Fit Finder tool that highlighted type of activity as a product benefit, we wanted to further leverage incorporating activity into the customer shopping experience.

We added an additional question on our write-a-review form called “Best For” -- an option to select what activity she thinks the product she bought is best for. From hiking to snow sports, customers categorize products into activities to help the next customer evaluate the product’s function, as well as speaking to the brand mission to provide functional clothing for the outdoors.

Before

Tactic: I worked with the merchant team to strategize best activity categories to include based on their purchasing strategies, got the sign off from our Founder, and communicated to our review vendor account manager the details of the project.

One takeaway from this project was learning how to streamline a process when communication channels were not optimal. I came across multiple scenarios where the vendor’s technical team implemented this particular feature pre-mature to full sign off from my team, resulting in the un-styled feature going live before our developers could adjust the CSS. These roadblocks allowed for the opportunity to shore up the process, which I did by laying out a three step sign-off process the vendor would need to abide by before setting anything live.

Impact: Qualitative data from the reviews themselves show that customers find the ‘Best For’ feature helpful, citing other women before them who mentioned activity in their review helped them with purchasing decisions. For example, they mention the ‘Best For’ callout helped them purchase an item for a specific activity they will do on an upcoming trip they are purchasing for.

Feature #3: Direct customer channel

We implemented new technology from our vendor that sends a direct email to the customer who left a review if we add a merchant response to that review.

Problem: When a customer posted a review, we had the option to write a pubic merchant response to help the next shopper. However, the customer we were responding to never got notified her review was responded to, unless we took an additional step to contact her privately and directly via email.

Tactic: Worked directly with vendor account manager to scope the project. We implemented a new feature that meant in addition to posting publicly, the merchant response is sent directly to the customer via an email, assuring them they were heard and beginning a direct conversation to address their needs further.

This feature in particular involved a lot of testing. We came across roadblocks where the email that was coming through didn’t show the product name or image. At each challenge, we tested until we found success.

Impact: We also created a new process in our Customer Service department to ensure that if a customer responded to the merchant response email, the response email would be flagged and automatically sent to the Merchant Responser on the Customer Service team directly. This ensured the customer could continue the conversation, if needed, with someone already aware of her questions and concerns, as opposed to getting randomly assigned to a Customer Service rep who had little context on the situation.