Product Management Question Corner: Sean O’Neill, uTANGO
Posted by Ivan Chalif in Best Practices, Communication, Interviews, Marketing, Professional DevelopmentPsst…Hey kid. Wanna learn a little somethin’ about Product Management? I know a guy who knows this guy who can help. He went to one o’ dem fancy bidness schools, ya know, the one in Chicago. No, not that one, the other one. Then he got hisself a job at some big company in Seattle. No, not that one, the other one. He moved ’round a bit at dat place, learnin’ tons of’ stuff ’bout how the retail bidness works. If you want, I could introduce youz. Be careful though…he talks a lot.
Q: How did you become involved in Product Management and was it planned?
A: Originally I was a financial analyst, before business school. This gave a good grounding in attention to details and focus on the cash flow as the ultimate measure of success: how a business makes money.
I went back to b-school to move into technology product management. Kellogg gave me a great grounding in the cutting edge issues facing businesses. I was recruited by Amazon.com and joined them in 1999. I started off as a Product Manager in the Books retail team at a time when the goal was Get Big Fast. I spent time in several other teams in a variety of Product roles, managing a range of features and services from Shipment Pricing to Transaction Feedback (satisfaction/reputation systems) to Platform Seller Tools to Fraud Investigations Operations.
After 7 years at Amazon, I left in 2006 to join the exec team of uTANGO as SVP of Product Development and Operations. We developed and operate the world’s richest consumer credit-card rewards platform, driving lifetime loyalty while rewarding members up to $1 million over the long term.
Q: What are the biggest challenges that Product Managers face?
A: Product Managers work on shaping the features and services of the company to execute the corporate strategy. All too often it is a challenge at the end of the year to demonstrate the true impact of what was delivered. Either analytics/data instrumentation is left out completely of the feature set, or it is one of the first features triaged if the schedule is at risk. However, having a clear sense from the start as to what the specific measure of success is for a project can help clarify decisions all along the way.
If the team knows that the key metric to evaluate the impact of a project is Revenue per Member Month, or Revenue within 7 Days of New Acquisition, then it enables them to make better trade off decisions for the scope of features and how to optimize them. And at the end of a project cycle or at annual review time, you finally have some concrete metrics to point to for the impact on the business.
Q: What is your greatest Product Management achievement?
A: I have worked on well over 60+ release cycles over the years, but one of my favorite projects was the Amazon Transaction Feedback System. This is a key measure of buyer satisfaction with their transaction from Amazon’s 3rd party marketplace, where buyers rate their satisfaction from 1-5 stars and an optional short note. It is a critical data point for Amazon, as it is a strong predictor of repeat purchases by that buyer, a strong indicator of the quality of performance from the 3rd party seller, and an important consumer review for future customers in the marketplace.
In 2003 I took over ownership of the Feedback system. It had originally been launched as part of the Amazon Auctions platform in 1999, and mostly was a reflection of the eBay feedback system at that time. Buyers rated sellers and sellers could rate buyers. All the feedback from either role was included in the Reputation of the seller, which included all feedback ever received in the life of the account. Over time the Amazon Marketplace (fixed-price retailers) grew exponentially, where the vast majority of the transactions were from established sellers. The feedback system needed to be overhauled to serve the strategy of a high-trust marketplace where buyers know up front what kind of experience to expect from the seller. Similar to the eBay world, the vast majority of feedback Sellers left for Buyers was Positive.
In the Amazon world, the transaction payment was immediate, so there really was no meaningful criteria for evaluating the buyer. But if the buyer account later wanted to sell an item, their account feedback was overwhelmingly positive, when the reality is that they had no reputation yet as a seller. And from the Buyer side, they often were afraid to leave negative feedback for the Seller for fear of retaliatory negative feedback on their own account. So we were getting an overly biased view of the transaction from both sides.
The last problem was that the mathematics of the 1-5 stars made it hard for prospective buyers to clearly distinguish a mediocre from a great seller. Most feedback is at the extremes, not a normal distribution. So while a 4 star movie may be a good movie, a 4 star seller most likely has 80% 5 star and 20% 1 star. Any business disappointing 20% of their buyers is not worth using.
To tackle this challenge, we had some great designers work with ways to change the display of information. We ultimately retained the 5-star rating system, as Buyers were already educated in how to use it to rate their experience. We needed to tweak how this translated into the seller reputation.
Every seller has their Feedback appear in line with their Offer for each product on Amazon. We added the % Positive (4 or 5 star) as well as the count of Feedback, so that buyers can compare 100% positive 1 feedback versus 96% positive 123,512 feedback.
We also changed the duration of the primary feedback calculation to be past 365 days instead of lifetime (what have you done for me lately?). On the seller’s profile page, we broke out their relative positive, neutral, and negative performance for a range of periods (30 days, 90 days, 365 days, and lifetime) so that they buyer who wants more info can gain a sense of recency.
These changes initially launched in the US and eventually rolled out to all Amazon marketplaces around the world with billions of page views to date. The end result was a strong incentive for merchants to take care of every customer to improve their seller reputation, as good sellers attract even more buyers through their positive feedback. And poor sellers either quickly identify their problems and fix them, or fade away. Either way the buyer satisfaction rate (and likelihood of return visits) for the marketplace rises.
I must say that I was happy to see the validation that eBay announced this year (5 years after we launched this) that they were changing their feedback system in very similar ways. Their previous system was not necessarily bad, it was designed to force peer-to-peer auction transactions to work through their problems rather than evaluate unbiased buyer satisfaction with their transaction. And as eBay’s strategy has evolved to take on more fixed-price transactions, their feedback system was not serving this goal.
Q: What was your worst Product Management mistake and how did you recover?
A: Ironically, my worst product management mistake was from this same Feedback project. Sellers on any marketplace are extremely sensitive to how their business and reputation is displayed on the marketplace. And we implemented material changes to the feedback system.
We had conducted seller and buyer focus groups along the way, and had discussed some of these changes, but once sellers saw the final release product they were extremely upset by the change of the primary feedback rating from Lifetime to Past 365 Days. Amazon’s long-time sellers had amassed a significant count of feedback, which they viewed as part of their ‘brand’ on Amazon to differentiate themselves from newer sellers.
In fact, a few of these older sellers were the prime reason why we changed the duration, in that they built up a great reputation years back, but since have been sliding on their performance and have not been very good in the past year or two. I had misjudged how important this longevity display was to the sellers. To address this, we were able to make some minor tweaks to the display on the Offer pages to add the Lifetime count of ratings. The feedback score was still based on Past 365, but we added the information that allows the consumer to gain a sense of the size of the business beyond just the past year.
Q: What Product Management tool(s) would you consider most effective and why?
A: Feature Prioritization Matrix. This is a tool from my Six Sigma background, but it extremely useful in Product Management to ensure that the strategically most important features with the biggest impact are delivered first.
Many times I have seen development teams get sidetracked by the lack of a clear set of strategic priorities. They deliver the features they believe to be the most important, but their unstated criteria may not match up with what the executive team is expecting. Or one key stakeholder is pushing for a feature that is important on one aspect, but may not be the most important one to deliver at that time. The Feature Prioritization Matrix gets everyone on the same page as to relative importance and relevance to the overall strategy.
We develop features using the Agile Development approach of small, iterative development sprints that deliver functional code every cycle (usually 4-6 weeks). To ensure what we are delivering is aligned with our strategy, we have a backlog of features and services that are classified in our Prioritization Matrix. Identify the top 4-5 aspects of your corporate strategy. For our business we focus on Member Acquisition, Drives Revenue, Supports Members, and Platform Scalability. There can be many more, but the key is to focus on the most important elements of your strategy, not all possible elements. We then assign a relative % weighting to each strategic area that sums to 1.
We use excel to track the feature description and the high-level classification of the feature (member marketing, analytics, merchandising, transactional systems, infrastructure, etc). Our tech team estimates a rough Scope Of Work in terms of Developer Weeks.
For each project, we then rank from 0-5, with 5 the highest, what the relative impact is of that project in each strategic area. From all this we can compute a weighted average Value of the project and by dividing this by the Scope Of Work, we can compare a rough return on developer investment. Ranking these in descending order gives a great first pass at what projects would return the greatest return AND support your corporate strategy at the same time.
As these figures are just rough estimates, the computed rank is not expected to be a precise final word on the order of projects, but it is very effective for order of magnitude impact of a project in comparison to other potential features. When a key stakeholder is pushing hard to get their features moved to the top of the list, there is a defensible, objective ranking that was already agreed upon from top management.
Q: Where is the best place for the Product Management function in an organization and why?
A: At Amazon, there are a wide variety of roles that all may hold the title of Product Manager, ranging from close interaction with software development teams on feature specification, to ongoing management of operational business lines with profit and loss responsibility.
I believe that any firm developing products and services needs the Product Management role in the team that is prioritizing and executing the evolution of their value proposition against their strategy. So if their value prop derives from creating new products and services, then Product Management needs to be in the development team.
If their value prop comes from bundling together 3rd party content, applications, or services, then Product Management should sit in the supply chain or vendor management teams.
If the value prop emanates from the power of granular marketing and personalization, then the Product Management should sit in the Marketing team. But regardless of where the desk is, the role of a Product Manager is to connect all these areas to ensure that the strategy of the company is aligned with the features and services delivered to customers.
Q: How has your experience as a Product Manager influenced you as a Senior Exec, CEO or founder?
A: As a senior exec, I use my Product Management background to ensure that all features and services we develop include a clear measure of success, and a hard requirement for data instrumentation. Also, having an understanding of how the systems operate gives you insight to challenge responses of “it can’t be done.” It allows you to distinguish between “can’t” and “don’t think it is worth it.”
Q: If someone told you that they wanted to transition from a Product Manager role to CEO (or founder), what would you tell them?
A: Get as much exposure to all key stakeholders as you can. Go on sales calls with key vendors or clients. Get behind the glass for user focus groups. Understand the needs of the customer and ensure that the corporate strategy is in line with it. And ultimately understand the metrics of the firm. Know how you generate cash flow, not just how you book revenue (there is a significant difference).
—————-
Sean’s question for The Productologist:
Q: With the passage of a number of privacy laws around the world, including further clarification of CAN-SPAM, what should Product Managers and Marketers do to ensure that they can continue to communicate with their customers across any medium?
A: With or without the more stringent privacy laws and across all communication channels, there are two facets that are key to sucessful communication with customers (or members or subscribers): permission and relevance. If you only have one of those, you are doomed. And, it cannot be the lowest common-denominator, either.
Anyone who has been involved in customer communication in the past 5 years is probably familiar with these, but in my experience working many companies that use email to reach their customers, most will go only so far as to fulfill the minimum requirements of one or the other (or even both). And then they are surprised when their response rates are low and their unsubscribe and/or complaint rates are high.
My friend and former co-worker at StrongMail Systems, Spencer Kollas, wrote a great article back in August 2008 about this exact problem and what marketers can do to fix it. The hard part is that fixing it often requires more time/effort/money than just dealing with the fallout. In the short-term. That looks fine on a spreadsheet when you have to request/justify your spend for marketing communications, but in the long run, the marketer is DAMAGING their brand and their reputation.
Online communications are so easy to metric that marketers can lose perspective on the fact that performance metrics do not tell the whole story. Nowhere in that click report does it show how frustrated a customer is when they receive an email encouraging them to buy a product they already own. It doesn’t show how many friends/co-workers/family members they complained to about volume of email or irrelevance of the content they receive from your company.
Regardless of the topic of your message, customer communication is a service-industry. The marketer who provides poor service to the customer is going to get a bad tip (i.e., fewer clicks/revenue), but even worse, they could get a bad review for their company. I doubt that looks good in a spreadsheet, no matter what type chart you use.
A little more about Sean:
Sean has been the SVP of Product Development and Operations at uTANGO since 2006. He has twelve years of business experience, seven of which were at Amazon.com in a variety of strategic ecommerce roles across software Product Management, retail management, operations, vendor relations, and consumer trust.
During his time at Amazon.com, Sean co-authored two patents (patent pending) in the areas of dynamic-pricing mechanisms and vendor-performance algorithms, and became a certified Operational Excellence Six Sigma Green Belt. Sean holds an MBA from the Kellogg School of Management at Northwestern University and a BA double major with honors in Business Economics and Political Science from the University of California, Santa Barbara.
Popularity: 22% [?]
Entries (RSS)