This is a guest post by Rachel Kibler.
At a prior company where I worked, we were testing a money transfer application. For all of our testing, we used production data that was sanitized, particularly emails and PINs. The application we were testing would send out emails and texts about money transfers.
I used my own phone number for testing, so it showed up in confirmation emails. I also had my favorite accounts for testing, and I used them heavily.
One day, I got a call from a very angry man asking what I was doing with his wife’s money. Let’s call the man Dave and the woman Sheila. Sheila had been receiving emails for about a day about money transfers in and out of her account, and she and Dave had become increasingly suspicious and furious during that time. Dave called the phone number that showed up in some of those confirmation emails — mine — and the conversation went something like this:
Dave: Is this [product]?
Me: No, but my company is integrating that product.
Dave: Why do you have my wife’s account information?
Me: Sorry, how did you get this number?
Dave: My wife has been getting emails with your number in them for a day!
Me: What is your wife’s name?
Dave: Sheila Carlton.
Me (stomach plummeting): Oh, I use her account for test data. No money is moving in or out of her account.
Dave: Why do you have real information for testing?
At this point, I was questioning the same thing. My heart was racing as I was frantically looking through email records and at the account info. And there it was in the account settings: a new email address that was not one of our test email addresses. It was obviously someone’s personal email, and it had been added as the primary email address.
I stammered to Dave that we sanitize our data, so I didn’t know how this had happened, but I would do the investigation and make sure we fixed it. I assured him that Sheila’s info had not been compromised and that we were using it legitimately, and I ended up having to defend using any kind of production data in our test environments.
It was awful. I hung up the phone, stared at my cubicle wall in shock for about half a minute, then sprang into action. It was time for crisis management.
I had been testing full-time and working for this company for about two months when this happened. I emailed everyone on the project to have them stop testing immediately. I gave a brief synopsis of what was going on and said I would follow up with more information, but I insisted that, in the meantime, no one change any settings or make any transactions until they had heard from me. Stop the bleeding — check.
My manager was out. The other test managers in my building were out. I finally talked with the test director, and though I was crying a lot, I explained what was going on and how I would go about figuring it out. Inform management — check.
I arranged a call with the developer, the project manager, the other tester and myself. I explained that I had seen these email addresses come in and failed to say anything about it (which is a whole topic unto itself) and that we needed to understand the extent of the problem. The developer immediately had an idea of what could have changed, so he made plans to investigate that, while the other tester and I planned on seeing just how many accounts were affected. The project manager agreed to talk with the customer if the need arose, removing me from getting yelled at again. Make a plan — check.
The other tester and I checked our accounts and those of the rest of our team through an administrative tool, to be sure we didn’t make any additional changes inadvertently. All told, about 20 accounts had live emails in them. It appeared to be only accounts that we had logged into recently, which narrowed down the change. The developer reported back that the wrong database had been brought into the test environment. Determine the extent of the problem — check.
The developer fixed the problem, and the other tester and I went about removing all of the live emails and carefully checking the other accounts. The project manager reported back to the customer that the problem was resolved. I had to deal with the paperwork about the data leak, which, as it was unlike a “real” leak, confused the lawyers. It took a while to make it clear that people other than the customers did not gain access to any of the information, and that even the customers only received their own data. Wrap things up — check.
This was an uncommon situation. Testers usually get to stay behind the scenes. But when a crisis happens, we have to take steps to make sure the bleeding stops, the right people are informed, a plan is made, the problem is understood, and things get resolved.
Rachel Kibler is a test lead at Anonyome Labs, which makes MySudo. She can be found on Twitter @racheljoi or on her website at racheljoi.com.
