Did you just experience a data snafu? Did you just use a few other choice words for what happened?
You aren’t alone.
Every day, marketers, marketing operations staff, and business analysts hose their databases accidentally.
Since you are reading this post, your initial panic has subsided and you are preparing to figure out how to roll back the change that occurred. Great!
Here is my recommended process for analyzing and resolving a database snafu.
First, stop panicking.
Take a deep breath and understand that in almost every case, the data can be fixed if you act quickly, calmly, and clearly.
Panicking will lead you to think: “The entire system is broken, they will blame me, and I will lose my job.” This thinking leads you to consider poor options like, “If I do nothing, they won’t notice,” or “I can fix it before they notice,” or even “Let me try x, y, z and then blame it on someone else.” This slippery slope will lead you to justify terrible ways to fix the problem or even make it worse. Eventually, someone else will find out and you will be held responsible.
Take that deep breath and remember you can resolve the problem, keep your job, and even look good at the same time.
Second, admit the mistake.
Did you make it? Admit it to yourself. Then admit it to your boss with an explanation and possible resolutions.
This is hard for people to do. No one likes to admit mistakes. I don’t, and neither do you. Admitting you did something or failed to stop something bad from happening is very difficult for the ego. But hiding the mistake or blaming others unnecessarily is not productive either. I’ve done this both ways and I can tell you from experience that admitting the mistake quickly and owning the resolution is far better than trying to cover up mistakes. Just ask Nixon.
Even if this error was not personally caused by you, admit a mistake was made. Work to find the cause and then the resolution regardless of who may be responsible. Retraining is only possible in a resentment free atmosphere.
Admitting the mistake and offering solutions makes it clear that you are a mature adult who owns problems and comes forward with solutions. Whenever I have done this properly, the only reproach I received was from myself.
Third, analyze the situation.
Solutions designed without understanding the source of the data error are useless. There are various frameworks for analyzing errors. Use one or more to drill down.
As you walk through the analysis process, take copious notes.
Questions you should ask to understand marketing automation mistakes:
- What is the error?
- What is the Current State?
- What is the Desired State?
- Is the error still occurring?
- Can you turn off the workflow that is causing the error?
- If you can do so without compounding the error with more errors, shut it down.
- If you cannot safely turn it off because it may impact other processes, can you potentially slow it down or remove the action that causes the error?
- What Caused the Error?
- Human input
- Workflow (Process)
- Something else?
Be very careful to remain judgment free of anyone at this stage. Human error happens (maybe it was you!), and the goal is to solve the error or roll it back, not to point fingers. For example, if a salesperson is uploading unverified, personal lists constantly, thus ruining data, the answer is to design the system to prevent this, not to call up the offender and yell at them.
At this point you may need to go into the Marketing Automation Platform (MAP), as well as the CRM to track down potential causes. If you are having trouble understanding the root cause, pull in additional people to help.
Five Whys Framework [for complex systems]
Popularized by Toyota, the Five Whys is to ask why a process did not produce the desired result at least five times until you understand the root cause. Keep in mind there could be multiple root causes. If your marketing infrastructure has multiple systems (CRM, MAP, other databases), you should bring in the administrators of those databases as well.
I find this method is ideal for very complex situations and is often a bit more than most people need. Here are a few ways to map out the Five Whys for such a situation.
Walk Through the Process
As part of your investigation, walk through the workflows and steps you expect the system to do. When you reach a point of error, note where, when, and how it happened.
Look closely at example leads that seem to be affected. Open their Activity Log to walk through changes to their data and which processes changed that data.
Some MAPs are better than others in providing detailed change logs. For the Marketing Automation Admin or Marketing Ops person, these are vital to fixing any snafu or even testing.
Call in Support: Vendor, Internal, and Other Admins
Your support network includes colleagues inside and outside the firm. Some errors you encounter may be identifiable, but not fixable by you. The error could be a vendor bug, or in a system you cannot act on alone.
Remember that even if the Vendor or another System causes the error, you must own the problem until it is resolved. No one else is going to take ownership or responsibility.
When do you call Support?
- Immediately: as you panic, you may need someone not directly affected who can guide you and remain calm. This is perfectly good use of Support.
- During the Investigation: vendor Support or other teammates may have more access or understanding than you do. Enlist their help to track down the problem.
- Resolution: your support team can help craft a resolution or even implement it. Again, this depends on what is needed to resolve the error.
How do you Build a Solution Without Causing More Damage?
Once you understand the situation, you need to solve it. The resolution may be to roll back the error and fix the process involved.
The good news is most systems will help you roll back data errors. In this example, let’s imagine we had a process that is supposed to move Leads to various stages:
|Initial State: Lead Status||Desired State: Lead Status|
Somehow, this process instead, did this:
|Initial State: Lead Status||Current State: Lead Status [ERROR]||Desired State: Lead Status|
Notice that I used tables to help me organize the data here. Tables make life in Marketing Operations much better and help you understand the data in nearly every situation, including creating processes.
The next step is to understand the total impact. How many leads are affected at each point? Let’s add a column to understand the counts involved.
|Initial State: Lead Status||Current State: Lead Status [ERROR]||Desired State: Lead Status|
How do we get such counts? For sake of this example, the counts all match, but they may not in your situation.
To discover who was in the Initial State and now in the Current (Error) State, we can use a few filters, depending on how this was setup in the first place:
- Static List – if the erroneous process used one, but the lead may have been removed automatically.
- Member of Smart Campaign: If a process caused the error that lead will be a Member of the campaign, so this helps us narrow down the affected group.
- Data Value Was Changed: there are several similar filters available. Most only work up to 90 days in the past. Use the Old Value to New Value so you can create the Smart List that shows you the total people who went from Prospect to SQL that were also a Member of the Smart Campaign.
Alternative methods in this example might include asking your CRM Admin to do something similar. In Salesforce, some fields have their history tracked and it may function better to run the roll back through Salesforce. You should discuss this with your team before attempting the resolution.
Solving the Problem: Rollback Workflow
At this point you should clone your Smart List to match the table above:
- Previously Prospect
- Previously Nurturing
- Previously MQL
Since the SQLs were correct, we do not need to worry about them.
Now that we know which leads were affected, we can roll back the change with a new flow action. However, we should take two more steps to ensure that we do not further compound the error:
- Run each Smart List through a Smart Campaign so we can maintain the Count and Member of Smart Campaign, just in case.
- Add each lead to a separate list during this Smart campaign so we know exactly who was fixed.
Another triple back up is to download the data before making a roll back change. This will help you identify the affected leads later. This may also be necessary if you plan to enact the rollback in another system. Your CRM Admin may need a list of Email Addresses and CRM IDs to do this.
Let’s Go to the Tape, the Backup Tape
Sometimes the data is beyond basic workflow help. Perhaps your MAP and CRM did not retain Last Value data. Or the data flows cleared out everyone’s Addresses and Phone Numbers. In these situations, the most you can do is identify the causes and the affected records.
The next step will be to discuss how to restore the data from the backups.
You do have backups, right?
Most system vendors will have some sort of backup of your data. Usually this is stored in case of their system failure, not yours. Call Support and discuss the options for restoring the data and when the Last Saved State is.
When restoring the backup, you may end up overwriting or losing data that was added since the Last Saved State. You will have to weigh the consequences and impact. If you acted quickly, the additional loss will be minimal compared to the Error State.
Your IT department may also have backups of this data, so be sure to discuss this process with them before starting the rollback.
The aftermath is not usually that dramatic. If the impact was internal, it is a good idea to discuss the process errors that led to the error. Take the time to put in additional safeguards within the system and on your checklists to reduce the chances of this happening again.
It is possible this error caused an external impact. That’s a fancy way to say you sent a ton of incorrect emails to your CUSTOMERS. In this case, you need to take the lists of affected people and determine how to explain the situation to your audience. This post is not about crisis management, but I can tell you that if your human team is honest and open in helping your human customers resolve your automation errors, it can turn your firm from social media pariah to remarkable.
Remember, most data snafus can be fixed.
Take a deep breath…