The Biggest Professional Mistake I Ever Made As A Product Manager, Part 2
Yesterday, we started talking about the biggest mistake I’d ever made as a Product Manager. Today–the thrilling conclusion!
To recap: In a desperate bid to escape total annihilation, I activated the Space-Time Energy Projection crystal — which ripped a hole in the fabric of time, sending my team and I back to prehistoric Earth.
Unfortunately, I didn’t notice the Rulon flagship latch onto us with a tractor beam, allowing those evil bastards to follow us back to the past, where they immediately enslaved the dinosaurs in an attempt to… to… oh. Wait. That’s Dino-Riders.
Here’s what really happened.
Many Will Enter, Everyone Will Win?
On midnight, December 28. Eights days after launch. Hours after the Lead Developer had departed for Brazil. Bad Things Went Down.
The product (a new sweepstakes platform) revealed its fatal flaw: Instead of selecting one winner from the eligible entries and notifying that lucky person of their win, the system in live service opted to email everyone who entered–thousands, upon thousands of people–and tell everyone they’d won.
The engineering manager caught the error, took the necessary steps to handle the technical side of things and select the one, true winner. But I was left with the rather dark task of telling one person he’d won… and thousands of other people that they hadn’t actually won.
And Then I Made Things Worse
Here’s the text of the email we sent out, breaking the bad news:
Dear [redacted] –
We would like to take this moment to apologize for the email you received from [redacted] on December 28, 2003, titled ‘You won the iPod sweepstakes!’. This email incorrectly identified you as the winner of the iPod in our weekly Rewards sweepstakes. Unfortunately, you are NOT a weekly winner at this time.
We could bore you with the technical reasons for why the sweepstakes was not capable of running as planned. Instead, please rest assured that the issue has been resolved and that the correct winner has been determined and notified.
To apologize for this technical error, we are going to deposit 10,000 bonus Rewards into your [redacted] account. Once these Rewards have been deposited into your account, they are yours to keep or redeem. These bonus Rewards will be deposited in your account by January 15, 2004.
As always, we very much appreciate your patronage … and your feedback. Feel free to contact us any time if you have questions or issues about Rewards or anything.
Sincerely,
The staff of [redacted]
Fortunately, we had a good community of people, and many wrote back, expressing sadness but understanding. Others were really angry–not just at what happened, but at my messaging.
“Please, Bore Us Tiny Folks With The Details!”
Pros of the above email: we addressed every message individually; proactively sent them out within hours of the bug’s discovery; and offered every affected member a form of compensation.
However–the phrase “We could bore you with the technical reasons” struck a strong, strong chord with some folks. My attempt to defuse, instead, incited people. Made them feel like they were being belittled, insulted.
“Please, bore us tiny folks with the details!” (and various variations) flooded our in-box.
I was horrified. But there was nothing further to do. The emails were out and could not be rescinded.
To a much lesser degree–people also took issue with the message coming from the amorphous “staff” rather than an individual. Upon reflection, that was really lame: My name should have been on it.
Lessons Learned
Within a few weeks, the issue had evaporated. No other customer blowback. No protracted legal or PR issues. No board of inquiry.
But the experience definitely changed how I viewed my job, my products, and my way of interacting with customers.
My own core set of rules started to develop…
- Never launch a major initiative a week prior to your lead developer going on vacation;
- When you apologize for a mistake, don’t try to lighten the mood;
- Being “online” is not a “Get out of jail free” card;
- Consider both the potential value of the release to your customers as well as the potential harm buggy software might inflict on them and your reputation;
- Don’t let internal politics over-rule your common sense.
In hindsight, these are all pretty obvious. But, at that time, there was no one to teach me such things–and the incredible pressure of internal politics, of the sand shifting beneath my feet, convinced me to leap when calmer heads should have prevailed.
New Around Here?
Subscribe to the feed to receive future updates; follow me on Twitter to keep the discussion going!
The Biggest Professional Mistake I Ever Made As A Product Manager, Part 2 – The thrilling conclusion #prodmgmt http://bit.ly/1CTMpj
This comment was originally posted on Twitter
The Biggest Professional Mistake I Ever Made As A Product Manager, Part 2 – The thrilling conclusion #prodmgmt http://bit.ly/1CTMpj
This comment was originally posted on Twitter
Thank you for sharing. Very educational. I will keep the link to this post close by, just in case.
Chris,
Good story. Thanks for sharing. Maybe I missed it, but what could the lead developer have done to stop the error? That doesn’t look like something anyone would have predicted, except maybe the QA team. 🙂
Saed-
A QA team was a luxury we didn’t have. As I recall, the lead developer made a small, last minute, “harmless” change and didn’t fully test afterwords. Just a small difference between equals and not equals.
Chris didn’t mention that not only was the lead developer in Brazil, but the company was closed the week between Christmas and New Year’s and everyone was on vacation. Getting in touch with the legal staff was not easy.
I’m not sure why I decided to check my work email late that Saturday night, but I still remember the horror I felt at seeing thousands of emails, each announcing a different winner.
While there were certainly errors made as Chris pointed out, there were things we got right as well. We quickly got in contact with the right people, made a decision as to how to respond, and fixed the broken software to prevent a recurrence. All within 12 hours with our staff dispersed around the globe.
Lessons from a Product Management disaster: http://bit.ly/3wMeZY
This comment was originally posted on Twitter
In high-stress situations like this, always, always have someone else read the e-mail. And then walk away from it for at least an hour.
Re. the “bore you with the details” flap, were they angry because they felt belittled, or because people wanted some idea what happened, even if they don’t really understand it?
But, honestly, at least one user deserves belittling. And he doesn’t seem to mind it:
http://www.theheretech.com/2009/06/the-heretech-podcast-episode-8.html
@Tom – That’s excellent advice. I remember at the time, the impulse was to FIX THINGS RIGHT NOW. And even though the email was reviewed, it wasn’t necessarily read by a calm, impartial third party.
@Saeed – One thing I definitely impress now upon my teams (which, I guess, is lesson #7, after Tom’s lesson above) is to tell me when I’m being too adventurous. Engineering Manager got particularly good at that, after this incident–and I appreciated that very much.
Thanks for the story – I agree emails need to be crafted ever-so-carefully when dealing with end users (or even partners!) of your software…you never know how they would interpret it!
Do you have any examples of emails that might have worked better?