How to Fix a Marketing Database Snafu

Did you just experience a data snafu? Did you just use a few other choice words for what happened?

You aren’t alone.

Every day, marketers, marketing operations staff, and business analysts hose their databases accidentally.

Since you are reading this post, your initial panic has subsided and you are preparing to figure out how to roll back the change that occurred. Great!

Here is my recommended process for analyzing and resolving a database snafu.

First, stop panicking.

Take a deep breath and understand that in almost every case, the data can be fixed if you act quickly, calmly, and clearly.

Panicking will lead you to think: “The entire system is broken, they will blame me, and I will lose my job.” This thinking leads you to consider poor options like, “If I do nothing, they won’t notice,” or “I can fix it before they notice,” or even “Let me try x, y, z and then blame it on someone else.” This slippery slope will lead you to justify terrible ways to fix the problem or even make it worse. Eventually, someone else will find out and you will be held responsible.

Take that deep breath and remember you can resolve the problem, keep your job, and even look good at the same time.

Second, admit the mistake.

Did you make it? Admit it to yourself. Then admit it to your boss with an explanation and possible resolutions.

This is hard for people to do. No one likes to admit mistakes. I don’t, and neither do you. Admitting you did something or failed to stop something bad from happening is very difficult for the ego. But hiding the mistake or blaming others unnecessarily is not productive either. I’ve done this both ways and I can tell you from experience that admitting the mistake quickly and owning the resolution is far better than trying to cover up mistakes. Just ask Nixon.

Even if this error was not personally caused by you, admit a mistake was made. Work to find the cause and then the resolution regardless of who may be responsible. Retraining is only possible in a resentment free atmosphere.

Admitting the mistake and offering solutions makes it clear that you are a mature adult who owns problems and comes forward with solutions. Whenever I have done this properly, the only reproach I received was from myself.

Third, analyze the situation.

Solutions designed without understanding the source of the data error are useless. There are various frameworks for analyzing errors. Use one or more to drill down.

As you walk through the analysis process, take copious notes.

Questions you should ask to understand marketing automation mistakes:

  1. What is the error?
    1. What is the Current State?
    2. What is the Desired State?
  2. Is the error still occurring?
  3. Can you turn off the workflow that is causing the error?
    1. If you can do so without compounding the error with more errors, shut it down.
    2. If you cannot safely turn it off because it may impact other processes, can you potentially slow it down or remove the action that causes the error?
  4. What Caused the Error?
    1. Human input
    2. Workflow (Process)
    3. Form
    4. Script?
    5. Bots?
    6. Something else?

Be very careful to remain judgment free of anyone at this stage. Human error happens (maybe it was you!), and the goal is to solve the error or roll it back, not to point fingers. For example, if a salesperson is uploading unverified, personal lists constantly, thus ruining data, the answer is to design the system to prevent this, not to call up the offender and yell at them.

At this point you may need to go into the Marketing Automation Platform (MAP), as well as the CRM to track down potential causes. If you are having trouble understanding the root cause, pull in additional people to help.

Five Whys Framework [for complex systems]

Popularized by Toyota, the Five Whys is to ask why a process did not produce the desired result at least five times until you understand the root cause. Keep in mind there could be multiple root causes. If your marketing infrastructure has multiple systems (CRM, MAP, other databases), you should bring in the administrators of those databases as well.

I find this method is ideal for very complex situations and is often a bit more than most people need. Here are a few ways to map out the Five Whys for such a situation.

Walk Through the Process

As part of your investigation, walk through the workflows and steps you expect the system to do. When you reach a point of error, note where, when, and how it happened.

Look closely at example leads that seem to be affected. Open their Activity Log to walk through changes to their data and which processes changed that data.

Some MAPs are better than others in providing detailed change logs. For the Marketing Automation Admin or Marketing Ops person, these are vital to fixing any snafu or even testing.

Call in Support: Vendor, Internal, and Other Admins

Your support network includes colleagues inside and outside the firm. Some errors you encounter may be identifiable, but not fixable by you. The error could be a vendor bug, or in a system you cannot act on alone.

Remember that even if the Vendor or another System causes the error, you must own the problem until it is resolved. No one else is going to take ownership or responsibility.

When do you call Support?

  • Immediately: as you panic, you may need someone not directly affected who can guide you and remain calm. This is perfectly good use of Support.
  • During the Investigation: vendor Support or other teammates may have more access or understanding than you do. Enlist their help to track down the problem.
  • Resolution: your support team can help craft a resolution or even implement it. Again, this depends on what is needed to resolve the error.

How do you Build a Solution Without Causing More Damage?

Once you understand the situation, you need to solve it. The resolution may be to roll back the error and fix the process involved.

The good news is most systems will help you roll back data errors. In this example, let’s imagine we had a process that is supposed to move Leads to various stages:

Initial State: Lead Status Desired State: Lead Status
Prospect MQL
Nurturing MQL
MQL SQL
SQL No change

Somehow, this process instead, did this:

Initial State: Lead Status Current State: Lead Status [ERROR] Desired State: Lead Status
Prospect SQL MQL
Nurturing SQL MQL
MQL SQL SQL
SQL SQL No change

Notice that I used tables to help me organize the data here. Tables make life in Marketing Operations much better and help you understand the data in nearly every situation, including creating processes.

The next step is to understand the total impact. How many leads are affected at each point? Let’s add a column to understand the counts involved.

Initial State: Lead Status Current State: Lead Status [ERROR] Desired State: Lead Status
Prospect 500 SQL 500 MQL 500
Nurturing 676 SQL 676 MQL 676
MQL 1000 SQL 1000 SQL 1000
SQL 750 SQL 750 No change 750

How do we get such counts? For sake of this example, the counts all match, but they may not in your situation.

To discover who was in the Initial State and now in the Current (Error) State, we can use a few filters, depending on how this was setup in the first place:

  • Static List – if the erroneous process used one, but the lead may have been removed automatically.
  • Member of Smart Campaign: If a process caused the error that lead will be a Member of the campaign, so this helps us narrow down the affected group.
  • Data Value Was Changed: there are several similar filters available. Most only work up to 90 days in the past. Use the Old Value to New Value so you can create the Smart List that shows you the total people who went from Prospect to SQL that were also a Member of the Smart Campaign.

smart-list-identify-bad-leads

Alternative methods in this example might include asking your CRM Admin to do something similar. In Salesforce, some fields have their history tracked and it may function better to run the roll back through Salesforce. You should discuss this with your team before attempting the resolution.

Solving the Problem: Rollback Workflow

At this point you should clone your Smart List to match the table above:

  • Previously Prospect
  • Previously Nurturing
  • Previously MQL

fix-error-program-map

Since the SQLs were correct, we do not need to worry about them.

Now that we know which leads were affected, we can roll back the change with a new flow action. However, we should take two more steps to ensure that we do not further compound the error:

  • Run each Smart List through a Smart Campaign so we can maintain the Count and Member of Smart Campaign, just in case.
  • Add each lead to a separate list during this Smart campaign so we know exactly who was fixed.

flow-step-corrector

Another triple back up is to download the data before making a roll back change. This will help you identify the affected leads later. This may also be necessary if you plan to enact the rollback in another system. Your CRM Admin may need a list of Email Addresses and CRM IDs to do this.

Let’s Go to the Tape, the Backup Tape

Sometimes the data is beyond basic workflow help. Perhaps your MAP and CRM did not retain Last Value data. Or the data flows cleared out everyone’s Addresses and Phone Numbers. In these situations, the most you can do is identify the causes and the affected records.

The next step will be to discuss how to restore the data from the backups.

You do have backups, right?

Most system vendors will have some sort of backup of your data. Usually this is stored in case of their system failure, not yours. Call Support and discuss the options for restoring the data and when the Last Saved State is.

When restoring the backup, you may end up overwriting or losing data that was added since the Last Saved State. You will have to weigh the consequences and impact. If you acted quickly, the additional loss will be minimal compared to the Error State.

Your IT department may also have backups of this data, so be sure to discuss this process with them before starting the rollback.

The Aftermath

The aftermath is not usually that dramatic. If the impact was internal, it is a good idea to discuss the process errors that led to the error. Take the time to put in additional safeguards within the system and on your checklists to reduce the chances of this happening again.

It is possible this error caused an external impact. That’s a fancy way to say you sent a ton of incorrect emails to your CUSTOMERS. In this case, you need to take the lists of affected people and determine how to explain the situation to your audience. This post is not about crisis management, but I can tell you that if your human team is honest and open in helping your human customers resolve your automation errors, it can turn your firm from social media pariah to remarkable.

Remember, most data snafus can be fixed.

Take a deep breath…

 

 

Marketing Automation Platforms Are Too Delicate

With the recent release of Marketo’s “Abort” button for email campaigns, we are finally taking a step to making Marketing Automation Platforms more robust against human error. But if you reached a point where you need to press abort, then the platform has already failed you.

Marketing Automation is on a Data Tight Rope

Marketing Automation is On a Data TightropeIn my opinion, many firms with MAPs are walking on a tightrope because there are so few safeguards against data errors in many platforms.

In the past few months, I have committed, seen, and corrected massive data errors in my MAP. All were caused by human error in some way. As I worked to fix these, I reflected on just how delicate these systems are. We rely on our MAP to process huge amounts of data each day. We rely on these systems to make the correct decisions based on a rigid view of previously human processes.

I saw this fragility in my first implementation. The process we established to implement Marketo with Salesforce was very solid, from the human side. We did everything in a sandbox and I heavily researched just how things should work. We checked in with various teams before acting on data. And everything went smoothly.

But as I built out the lead management flows and important autoresponders, I began to think of the tool as a delicately balanced system, akin to erector set with very few bolts. It worked really well, but just one thing off could break the entire flow, collapsing my structure.

Years later, I see lead management flows and data management flows in the same way. Why? Because nothing has changed in the robustness of the tools themselves. From what I can tell, this is true for many major vendors. Where are the safeguards against someone accidentally breaking the carefully built erector set?

And I don’t mean an “abort button,” however useful to prevent external firestorms of criticism and embarrassment. I mean, how do I stop someone, myself included, from turning on a workflow, or changing a name, or moving something that causes massive data errors?

Common Solutions that are not enough

  • Security by Role: most systems let you restrict untrained or those who don’t need to know from accessing key areas. What I have seen, however, is that core workflows like Lead Management are still vulnerable to people who have access to run email campaigns.
  • Training: yes, that could work, to a point. I’m considered well trained and “experienced”, yet without proper documentation and checks, I can still do a lot of damage without intending to do so. Most marketing automation managers came up through marketing, not technology or system administration. Thus, a lot of data governance training is needed, and it’s not getting done.
  • Workspaces and Lead Partitions: these do help. The impact on the system can be contained in many cases if a user is restricted to just their area. Depending on the setup, however, many users can still import data that runs amok across the system. They can also hose their own data!
  • Don’t let anyone in: not really feasible for most organizations.
  • CRM and Database Safeguards: this is often available to large organizations that can afford to add rules and logic to the system to prevent massive data errors. It can help avoid many errors, but not all.
  • Design the system to be less prone to human error: this is more for the vendor to do, as the MAP rules are geared toward moving data round, not letting you prevent errors.

Real Safeguards for Marketing Automation Platforms

So what could real safeguards look like without ending up with an Enterprise System or custom tool?

  • Lock Programs and Smart Campaigns after they are set to run.
  • Lock Smart List configurations. [Vote]
  • Change Logs with more detail that permit more filtering.
  • Changed Data History expansion. I know this is expensive.
  • “Are you sure?” dialogs with count information. (this usually appears on delete).
  • Alert Dialog – “Changing this campaign will affect XYZ”. While “Used by” exists in Marketo, you have to look at it first. Same for other asset changes.
  • Refined User Role Access: I’d like to see Role Security down to flow step permission and leads affected. [Vote]
  • Schedule Tabs that display: Total Qualified – Total Blocked from Email = Total Estimated Send. [Vote]
  • Order of Execution. This can be controlled with Campaign is Requested, however, this seems to be only something an advanced user would be able to setup. [Vote: idea 1 and idea 2].
  • Individual record view should not permit editing unless you have permission and press “Edit Mode”.
  • Hide Core Lead Management functions from other users. This would reduce the chances of “explorers” finding something and touching it. [Vote]
  • Smart Campaign View Only Mode – Let people look at a campaign and then press Edit to modify the rules or workflow. [Vote]

You can vote for the ideas in Marketo. If you are using another tool, consider these as ideas for the integrity and security of your dataset.

Human Safeguards

Of course, the vendor cannot prevent all human error. As a marketing leader, you need to establish rules and processes for the team as well. Here are a few of my tips:

  • Document everything and train people to read it first.
  • Use diagrams to help understand dependencies.
  • Four Eyes – two people check campaigns.
  • Campaigns over 50,000 leads must be approved going up the chain.
  • CRM Admins must be consulted and approve major changes.
  • IT team should review and approve changes that would affect major systems such as websites, email, and other databases.

Do you have other ideas to help increase MAP robustness? Let us know in the comments.

Image Credit: martintaylor