How to Test in Marketing Automation

As you gain experience with Marketo, you will build increasingly complex flows to manage leads as well as to nurture them.

Marketo, in essence, is a rules engine. You decide the rules for your system and your audience. As with all computers, the rules you decide on are executed faithfully and without question. Thus, if your rules are not properly setup, they will go ahead anyway (if they are logically correct). For example, if you set your Smart List to ANY instead of AND, you will likely bring in many more leads than you intended, possibly ruining data or worse, sending out 100,000 emails to the wrong people.

Fortunately, there are ways to build workflows and test processes to avoid disasters. If you follow these principles and any other policies your firm has, you can reduce the error rate greatly.

Technique 1: Pay Attention

When to Use: all the time

Time Involved: 1 minute

Level: All

Paying attention seems like an obvious way to avoid mistakes. It is also prone to many human biases such as “Glossing over work you just did,” and “I’ve done this a thousand times before.” Be careful and follow a few of my rules when I operate alone.

  • Carefully create the smart lists. It is easy to drag, drop, and dash only to see the batch campaign send to 10 times the number you intended. When you think you are done, stop and look at the AND vs. ANY rules as well as the Counts.
  • Watch your Flows – I always check these three times. Remember a Flow step will run once for every lead that goes through.
  • Watch for red squiggly lines in Flow Steps and Filters. Even if it looks right, it means Marketo did not like something.
  • Watch the Schedule Count – does this count match what you thought? Did you subtract the blocked email count from the total? If something seems off, STOP.
  • Qualification Rules – Every vs. Once vs. something else. One of the most frequent questions on the Nation are related to this feature.
  • Scheduled Time – I always schedule a run for 10 minutes in the future because it is very easy to realize that the Email Subject Line is missing 9 minutes after you press Run.

Technique 2: Review Thrice

When to Use: always

Time Involved: 1 minute

Level: Any 

I use this technique in combination with Technique 1, cycling through the steps three times…or maybe I’m a little OCD about sending emails to thousands of people.

  1. Smart List x 3
  2. Flow x 3
  3. Schedule x 3

Technique 3: Paired Campaign Managers

When to Use: always

Time Involved: 1 minute to 1 day

Level: Any

A technique the Marketo marketing team uses is paired campaign managers. One person builds the Program, while the other prepares the creative. Then they switch to review each other’s work.

If you have the staff, I highly recommend setting up this system as it helps to avoid the human ability to ignore errors and typos after working on something for 4 hours.

You can go further and setup an entire approval process, even with just 3 to 4 team members:

  1. Build Program
  2. Add Creative
  3. Review Creative
  4. Review Program
  5. Test Program
  6. Approve by Director
  7. Launch

The one challenge with a full blown approval process is Marketo does not have an “approval system.” It may be possible for you to break out Roles according to the process above. For example:

  • Approver: can access all Marketing Activities
  • Program Builder: Marketing activities, but cannot send or approve emails or Pages.
  • Creative: Design Studio, Build Emails or Pages only. No Approval rights.

Technique 4: One Email, Multiple Leads

When to Use: Any system

Time Involved: 1 minute

Level: Any

This trick works on any email platform, although I tend to only use it on Gmail. You can create as many individual Leads in Marketo as you want and have them all go to the same email box.

Replace “test1” with anything you want and even leverage the spreadsheet drag and populate to add more. In fact, on my email list, several people use this technique to filter my emails to a specific box:

Send an email and you will get all of them in the same inbox.

A similar trick can be used if you happen to be the main box on a Google Apps Account – just set all emails to go to the one box and you can make up any email address you want.

Technique 5: Reset the Test Lead

When to Use: Any system

Time Involved: 1 hour or more

Level: Intermediate or above

This technique can be used on its own or as part of other techniques. If you have a Trigger or Batch that is listening for a particular value, you can test it continually using the same Test Lead.

The Test Lead should have the following components:

  • An internal email address or one you own.
  • Marked somewhere as a Test – IGNORE.
  • Fields values as you want them to be for the Test.
  • Qualification Rules set to Every Time. (otherwise, the lead goes through once regardless of test changes).

Once ready, be sure to copy the email address to a Form or notepad so you can keep using it. Then, make the change of value using any of the following steps:

  • Direct Edit
  • Edit in CRM
  • Fill Out Form
  • Change Data Value in Flow Action

Once your test is done, adjust your flow (if needed) and keep testing. To re-set your Lead, just undo the Data Value Change you made using the Direct Edit or another flow action. 

Technique 6: One More Time…

When to Use: Basic Trigger workflows, Drip campaigns, Engagement Nurture, Lifecycle Testing, Lead Routing

Time Involved: 1 minute

Level: Any

This technique is just to modify the Schedule to use a Qualification Rule of Every Time. This way you can continually run the same set of Test leads.

Technique 7: The Brake

When to Use: Basic Trigger workflows, Drip campaigns, Engagement Nurture, Lifecycle Testing, Lead Routing

Time Involved: 1 hour to 2 days

Level: Any

This is by far the best technique and it is the most simple. At the end of your Smart List, add one of these two filters:

Member of Smart List IN “Internal Test”
Email Address CONTAINS “”

Or better yet, restrict the Email Address filter to use specific test leads or a special test domain like mine.

Email Address IS “”…

Marketo Email Brake

The Brake prevents the Trigger or Smart List from activating any other Lead. It retains the rest of the system so you can easily remove the Brake when you are done. This way you can test the full functionality and triggers without emailing or affecting real leads.

Technique 8: Brake and Clone and Wait

When to Use: Wait Steps are involved with Basic Trigger workflows, Drip campaigns, Engagement Nurture, Lifecycle Testing, Lead Routing

Time Involved: 1 hour to 2 days

Level: Any

In this version, you use the Brake and add some tweaks:

  • Clone the Program or System – instead of editing the entire system you built. You can then edit the Brakes and modify as needed without losing previous work. You may end up having to duplicate your work in the original Program, or just use this one as the final.
  • Edit Wait Steps to 1 second or 1 minute – when you Clone, Brake, and Wait, you need to reduce all Wait Steps to 1 minute Any Time. Otherwise, you will Wait 2 days until Tuesday for the next email to go out. It can be a bit time consuming, which is why Cloning and Testing work better.

Technique 9: Software Style Testing Process

When to Use: Lifecycle Testing, Lead Routing, Very Complex systems

Time Involved: 1-5 days

Level: Advanced

This process involves setting up leads that meet various criteria to flow through your workflow. Each time you run each lead, you should have an Expected Result and an Actual Result. Once complete, you will have a clear list of potential flaws in the workflow and possible ways to resolve them.

Do not let “software” intimidate you. The test cases you setup will likely be a bit short of what a full Engineer in Test might do, but it’s close. Here are some terms you may come across:

  • Edge Case: used by engineers to discuss unlikely scenarios that could happen, but may not be worth the effort to test or fix. Be very careful that edge case leads do not bring the system to a halt.
  • Test Case: this is a planned test and lead that meets certain criteria we expect to happen. For example, the Lead is entered in Form X with State=CA and Country=Canada. What do we expect will happen? Test Cases may be called “Use cases” if created before the build.
  • Test Plan: The combination of Test Cases and materials to run through the system with Expected Results vs. Actual Result.

Next week, we’ll go through a walkthrough of a real-life Lead Routing example from my friends at

Bonus – Join me and dozens of other amazing marketing automation nerds at the Marketo Summit in Vegas. Use this coupon by March 14: Hill200

The ROI of Marketing Automation

In recent posts, I have discussed that the ROI of marketing automation, and martech in general, boils down to time savings. Most vendors, however, rarely discuss the time saved through automation itself. Most vendors discuss the revenue increases, conversion rate increases, and opportunities you will obtain. Marketing automation (MAP) vendors particularly enjoy talking about the internal sales-marketing alignment improvements and “predictable revenue” reports.

In the Marketing Tech Maturity Model, I posited that it is rare for any firm to obtain the revenue and predictive ROI levels in less than 2 years. The expectation of executives, however, is that ROI can be obtained in under one year.

The other day, a reader asked me about the ROI issue and I decided to scour the internet for data and concepts. Surprisingly, vendor data and case studies supported my view that there are three main drivers of martech ROI and they do not come all at once to the buyer.

There are three purposes to this discussion:

  • First, understand the logic of ROI of martech for your business so you can have the right conversation with leadership as well as show the right ROI over time as planned.
  • Second, resist the “correlation is causation” claims of vendors; instead ask for better data or at least understand that the tool and your strategy are intertwined.
  • Third, understand what your need is and articulate it before searching for a solution. The need for “more leads” is different than the need for “transparency.”

The Three ROI Drivers of Marketing Technology

To me, there appears to be much confusion about what ROI on martech really is. I recall that in the 1950s and 1960s, IBM pushed mainframes on companies as a way to gain financial control over the scale of the back-office. In a similar way, martech lets you do more with less, or at least that is what can happen. The reality is more nuanced and many firms find that they need an entire Marketing Operations team to build and operate marketing technology so the campaign and content managers can deploy properly. The IT department grew in a similar fashion.

Thus, it helps if everyone understands where to leverage martech and how ROI works for martech:

Time Saved on process execution through moving manual processes to automatic rule based systems. Reduced time can come in several forms: rethinking of business processes, automation of processes, and faster deployment of campaigns (cloning, etc).

The Mindset Shift is the one that most vendors rely on for case studies and sales pitches. This is the idea that the software forces a change in marketing at the firm and therefore is responsible for tremendous ROI. The shift in marketing thinking occurs because the software’s model forces the buyer to more rigidly adhere to a way of marketing. And because the buyer is paying for this mode of marketing, they feel obligated to try it. Simply focusing effort on blogging and systematizing lead collection and email segmentation will improve results, regardless of the tool.

Transparency into the sales funnel and marketing activities that allow better decisions on allocation of resources. The tool simply permits better data collection and better reporting that marketers have wanted for years. Vendors press on this point when they discuss “revenue” and “predictive” concepts. What they really mean is, for the first time, you can see what impact you have and make a faster decision to incrementally improve or rapidly iterate.

Time Saved

When I first purchased Marketo back in 2010, I had a lot of goals for the system, but ultimately my highest desire was to stop spending 2-3 days a month (or more) routing leads, ranking leads, and deduping. And this wasn’t just for me, it was for my colleagues as well. Once I had Marketo fully operational, it was on track to save up to 1832 man-hours/year (1 FTE!) of time.

I do not buy because a MAP “increases conversion rates.” It just does not. What it does do is help me automate the AB testing and incremental improvements that will help increase conversion rates over time. An Email Service Provider (ESP) can offer similar AB testing tools, as can a web only provider like Optimizely. Each provides the tool to increase conversions through “simpler” testing. The ROI is initially in time, and then eventually in higher conversion rates from the tests you perform.

You do these things. The platform helps you do them faster.

The vendors are right, but their logic skips to the conclusion. The same goes for the claims that a MAP will provide 4x increase in leads generated or 400% increase in Sales Ready Leads. This logic skip results in leadership becoming unhappy when they realize the investment required to reach the Transparency ROI.

Marketo’s 2012 survey had several case studies illustrating this point.

“Using marketing automation, ShipServ is able to eliminate manual processes, automate demand generation and lead management…’A significant reduction in manual processes has allowed the marketing team to focus on more strategic and creative initiatives…’”

Yet, the call out quote for John Watton, VP of Marketing is “…[W]e now drive significantly more sales opportunities with the same budget…” When in fact, he is able to achieve that because his team saved huge amounts of time to focus on his real mission. Similarly, Cloud9 is “doing more with less” and “Codesion is able to fully automate its Free Trials Program.”

Thus, the first major ROI for marketing technology is time savings that allows you to focus on the high impact campaigns or customer focused touches to drive pipeline. Of course, you probably think I missed the point of marketing here! No one buys inherently because of cost or time savings. What CMOs wanted in 2008 and today is a way to prove Marketing drives revenue! In a classic sell the sizzle moment, the messaging moved away from simple, boring automation to phrases with “revenue” in them: revenue performance management, revenue engines, etc.

Selling Time isn’t sexy if you aren’t an airline.

There is nothing wrong with selling the tools on the vision of revenue enhancement (that is our real goal, right?), but many buyers of martech get confused because they expect the revenue and vision in the first six months because that is what they were sold.

Several industry surveys seem to support the view that the real ROI is in time savings:

  • Redeye reported 36% of respondents said the main benefit was “Taking repetitive tasks out of marketers hands, allowing focus on new/more exciting projects.”
  • Adestra reported that 74% of respondents said the biggest benefit was saving time vs. 58% “increased opportunities.”
  • B2bmarketing and Circle Research claim “Only 8% of companies see increased revenues within 6 months of adopting marketing automation. After one year of MA use, 32% claim to see increased revenue…two years the figure is 40%” (via EmailMonday).

Of course, there are many surveys indicating the benefits are in revenue or pipeline or lead generation. In your first year of MAP use, and even into the future, time savings is the main benefit and driver of any return.

The Mindset Shift

The concept here is that adopting a martech solution for any part of the funnel or for a particular channel (adtech, event tech), is that the tool itself does nothing until you tell it to do something. As Scott Brinker pointed out in 2010, our marketing is the software we use, in much the same way that Code is Law, martech is marketing strategy.

Tools like martech can be a catalyst for business change and even mindset changes in how you and your team approach marketing. Vendors have spent millions on shifting our mindset from marketing as a cost, to marketing as a revenue center because of Transparency. In reality, this propaganda serves the mindset return on investment.

Going back to my first Marketo implementation—and this is true for everyone—we had already made the mindset shift over the previous 3 years of work:

  • Obtained close marketing-sales alignment through regular meetings, read lead SLAs, and lengthy reviews of buyer personas. Was this perfect? No. Was it far enough along that the lead process was more about automation, yes!
  • Content marketing as the base. My team had led the charge, proven that content-based events and webinars worked, and established processes for demand generation programs. Once this went global, all we had to do was automate it.
  • Audience building and free-registration content. This occurred during the RFP process and the high response rate meant Marketo was vital. We had a pressing need and the team was prepared for automation.

When Marketo was deployed, did I see ROI because we implemented demand generation? No. We did that already. We saw ROI because the automation saved time for creative long-term campaigns, lead processing speed, and reporting.

This is really the question for many teams that are still at Stage 0 or Stage 1 of their martech maturity cycle. Purchasing a system becomes the catalyst for undertaking the internal process changes, sales alignment work, and marketing strategy changes that the software demands of your team.

Do you see an ROI on the investment in a MAP because you can automate your nurturing, or because you can finally do nurturing at scale? Will you even conduct the level of nurturing your MAP can enable? If HubSpot has an easy to use blogging tool and your small business has barely done anything like this, then it is very easy to show a 4x increase in leads simply through blogging on any platform.

In a 2014 survey, Marketo found that 84% of long-term Marketo users (4 or more years) were using multi-channel marketing, while only 54% of new users did so. There is a clear progression of skill with a MAP and the use of its features to implement a strategy. A MAP is a catalyst as well as a marketing structure to be learned and you must build your expectations around that learning curve.

Thus, if you can follow the Martech Maturity Model and undertake the Stage 0 work first, you will naturally find the best tool for you to automate and scale your strategy.


The final ROI is in transparency. I say this is “final” because it is most likely the last phase of return you can expect over time. Expect transparency to occur in Stages 4-6 of the Martech Maturity Model.

Sadly, this is the same ROI that your CMO and CEO expect happens three days after the initial CRM sync. Please do not let the vendor or your business case give this impression. If you do, your project will be cancelled before your contract renewal date, or worse, you will switch to a new vendor and have the same issues and have to learn an entirely new system.

Transparency is composed of two critical areas that may occur at different times during the Maturity Cycle.

  • First, transparency is about enabling better data collection on the Sales Funnel, Lead Behaviors, and Marketing Attribution. This data collection can occur even before the reporting tools are fully available.
  • Second, transparency is about collating that data into meaningful, easy to read reports where a decision can be made to re-allocate budget to activities with clearly higher performance for the Goal at hand.

That Goal can be higher CTR or more marketing sourced Opportunities. Ideally, these Goals are chosen ahead of time and the charts clearly explain previous history. At this point, a Predictive Tool can provide insight on the best leads before they reach Sales. Other predictive and reporting tools can then show the likely pipeline scenarios.

It is only at this final Stage 6 that the promise of the vendors is fulfilled with “predictable revenue.” Any attempt to reach this point by skipping stages or expecting this ROI too early will result in actual failure, or the perception of failure.

If we look at some of the surveys conducted in the past few years, we can see some of these issues play out.

  • In 2013, Gleanster reported that, “79% of top-performing companies have been using marketing automation for more than 2 years.” (via Email Monday) Are they top performing because of their skill with the system, or were they already ahead of everyone else?
  • VentureBeat reported that 29% of marketing automation users “had no issues or delays in getting results.” This statistic suggests to me that 71% of those respondents were unprepared to implement or had unrealistic expectations for their MAP.
  • VentureBeat conducted a survey showing a 27.8% return from a MAP. Of the respondents, 83% claimed a positive ROI, while only 48% thought it was worth it. While the survey was incomplete at the time, I suspect the respondents felt it wasn’t worth it because they had to purchase consulting time and hire new staff. If those respondents examined the total cost of implementation (people, time, consultants), their ROI might still be negative after two years.
  • In a 2013 survey, Salesforce claimed that marketing automation users had a 53% higher conversion rate to MQL and a 9.3% higher rate of quota-rep achievement vs. non-users. Is this because marketing automation forced the firms to put together a proper process? Is this because marketing automation enabled better processes?
  • Lenskold’s 2013 study with The Pedowitz Group indicated highly effective teams thought their MAPs improved revenue contribution (78% vs. 54%) and improved “reliable measurements” (44% vs. 25%), while those same teams thought their content marketing performed well. I’d be curious if the high performing teams were also those using their MAPs for an extended period.

Similarly, many surveys claim that automation, which enables Lead Nurturing at scale, causes things like “20% increase in sales opportunities” (SFDC), or “28% of respondents reported an increase in average deal size” (Lenskold via Autopilot HQ, 2012) thanks to nurturing. Is it possible to construct nurturing with a basic ESP instead, and achieve similar results?

Is automated nurturing a time saver or a revenue generator? Perhaps as your team builds enough content, nurturing will be a revenue generator. In my experience, few firms have a solid nurturing program setup in their marketing automation system.

It appears that firms that do not take advantage of nurturing are the ones unhappy with the ROI of marketing automation. Their teams lack the time or skillset to build out the content required for a massive nurturing journey that would impact pipeline or revenue. When it comes time to look at the ROI of the system, executives then immediately feel betrayed by the vendor, thinking they were overpromised. In fact, the executives failed to plan out the implementation and expectations. Is that the vendor’s fault, or the buyer’s?

One of the key transparency ROIs is to achieve visibility into marketing attribution and marketing influence on pipeline and revenue. This can and does happen, with investment. An excellent presentation from 2014 discusses how to justify reporting investment before you have all the pieces in place to prove marketing generates revenue.

The ROI of Conclusions

I suspect that most readers are not convinced to buy a tool because of a simplistic claim from a vendor. Rather, most teams do go through an RFP process or a bake-off between two vendors. The team discusses the merits of the tools as well as the feelings they get from the salespeople and technical salespeople. Then a choice is made.

From my work with over 35 teams, I can tell you that in-spite of the “account based purchasing” that is done, buyers still have outsized expectations of what martech will do for them in the first year, or even in the second year. Please take the time to think through the Martech Maturity Stages to make your business case as a long-term project with ROI over time. Show the ROI like your CFO would and do not let the vendor’s claims dominate your deck.

Vendor salespeople are supposed to sell you the Vision, that’s their job. Remember, sifting through the claims and building a business case is your job as the buyer.

Special Note – if you are a marketing expert, marketing operations expert, or otherwise amazing Marketo User, you are welcome to submit guest posts. Email me at hello [at]