Etsy was founded in Brooklyn in 2005. The company found a devoted following among crafters and flea markets and defied the odds and naysayers who doubted whether anyone wanted to buy tchotchkes and needlepoint made by grandmothers around the world. The company became one of New York's most successful venture-backed success stories (backed by Fred Wilson of Union Square Ventures), with a valuation of around $3B at the time of its IPO in 2015.
When I joined Etsy in 2013, it was well known around New York City tech circles but was not yet part of mainstream culture. The company and the product were evolving from a quirky handmade community into a global digital marketplace, and it was striving to mature its infrastructure and policies in preparation for its IPO a couple of years down the road.
I was supporting the operations department as their first and at that time only data analyst, helping half a dozen different departments with a mix of business operations, process improvement, and ad-hoc analysis into experiments and evaluating new markets. The VP of Operations at the time posed very vague and audacious question. What are the most significant pain points facing our buyers and sellers, and what can we do to address them?
The challenge
What is the most significant pain point facing buyers and sellers and what can we do about it?
The approach
The first step in this process was a review and analysis of existing buyer and seller data to understand their experience in the marketplace. The company had run both an annual seller and buyer survey at the time.
The seller survey indicated a very long tail of feedback. There were a dozen different themes that emerged, and sellers themselves were not a united group; they covered a vast spectrum from weekend hobbyists to professional full-time businesses.
Buyers, however, had a very short list of complaints and issues. When buyers weren't promoters, they had just one of two complaints: the item they ordered didn't arrive, or it wasn't what they expected. This was really pronounced when the buyer was a first-time purchaser. First-time purchasers who experienced one of these two issues didn't just avoid a second purchase. They had a deeply negative net promoter score (-60). This means they were active detractors telling friends and family not to purchase on Etsy.
We focused on better understanding and solving the complaints among first-time purchasers. The biggest opportunity was clear, gain a better understanding of why first-time purchasers experienced issues and figure out how we could solve them. Being first-time purchasers, buyers rarely had much account activity or history with Etsy, so there wasn't data to be able to examine or analyze. Analyzing the seller accounts from which first-time buyers purchased became the best option.
We examined the account activity of these shops that sold to first time buyers. We looked at the history of the shops, their tenure on the Etsy platform, the products they sold, their price point, their account and profile information. We looked back at the actions of the shops in their first few days on the platform up until they started having order issues.
There were several different scenarios that emerged among these sellers. A small percentage were clearly malicious and accumulated a high volume of orders soon after setting up their shop and all indications were that they never intended to provide a product. Others, had a history of good selling and then a spike in orders followed by a number of issues and yet others still had no clear sign of a change in activity they just started to go unresponsive and fail to fulfill orders.
The result
Out of this analysis we crafted a series of metrics designed to detect these different types of sellers and quickly remediate them or cut them off.
We created a series of seller service level standards for the first time in Etsy's history, requiring shops to meet certain requirements for responding to buyers and limiting the percentage of their orders that could be in dispute.
When sellers breached these thresholds, they either received outreach from Etsy or their shop was immediately paused, depending on the extent of their breach.
These service levels were even taken into account in Etsy search ranking. Fast forward a decade and Etsy still hold shops accountable against this rubric of service level standards. Shops have insight into their response rate, on-time shipping rate, average review rating and case (disputed order) rate.
Tracking and managing shops against these metrics helps ensure all buyers and particularly first time buyers have a positive experience shopping on Etsy.
