Making Big Data Actionable for Paid Search Optimization

It goes by many names, but you're paying money for it and it’s driving a lot of value to your business. Pay-per-click (PPC), search engine marketing (SEM), paid search, search advertising, Google Ads, etc… it’s the marketing channel that has the biggest opportunity to truly optimize, and do so based on hard statistics. Why? Because the big data it generates enables very accurate statistical analysis and computation.

Paid search optimization. It’s the practice of making the most effective use of advertising budget towards your SEM goals. More specifically, it’s identifying trends and opportunities based on past performance data in your search engine marketing campaigns, and applying those learnings to drive better results. Whether by eliminating waste and cutting costs where they aren’t leading to revenue or conversions, or (of course) investing more on the keywords, segments, and topics that have better results.

But you’re not here for that high-level overview, and neither are we. PPC bid optimization is a rigorous and technical field, and the goal of this article is to discuss the approach to automation and optimization of SEM campaigns with big data.

Sophisticated PPC Campaigns are Generating Big Data Sets

You and your business are likely using Google Ads (et al) in a very sophisticated way. Large SEM programs not only pay a lot of dollars to search engines and gather a lot of dollars from ad-conversions, but they are generating massive amounts of data as a byproduct.

For example, every search query that leads to a click and subsequent conversions has numerous elements along that entire journey that are being tracked inherently by Google… and then by Google Analytics, tag managers, “3rd party” pixels and analytics tools, and conversion tracking tools… and then deeper funnel data points about latent conversions, offline sales, and lifetime value that may live in a CRM or data warehouse. Plus you can collect browsing behavior and seemingly unrelated data on the sidelines that ultimately may correlate to some future value.

The point here is that when your organization uses PPC as a primary channel to generate revenue, you are (or should be) collecting a very, very large–and very powerful–dataset.

Alright, so what?

I'm glad you asked!

Big Data may be a buzzword that gets thrown around a lot lately, but let’s take a look at it in two manners:

1. The definition from Wikipedia: "Big data" is a field that treats ways to analyze, systematically extract information from, or otherwise deal with data sets that are too large or complex to be dealt with by traditional data-processing application software.

2. I like to simplify it into a simple concept: When nearly everything is tracked, you do get a lot more noise, but you also end up with a much more complete picture of associations and correlations. With big data, correlations trump causation.

Big Data removes the necessity of understanding cause and effect (the "Why?”) and turns the problem into a game of correlations. You see that when “A” happens, then “B” happens very frequently. To make that information actionable, we needn't make a claim about cause nor reason. There is a correlation. And that correlation has power.

Nowhere in marketing does this type of correlation have more easily accessible power and immediate actionable application than large paid search programs. SEM is the “rocket science” of marketing. Bold claim? Yes. Accurate? Probably.

Big Data Becomes Actionable

Your data, much of which is simply a natural byproduct of running your program and having tracking set up in a healthy way, has enormous potential to optimize and automate results in your SEM campaigns. Of course when you add in the other tools, tracking and analytics providers, and your offline data (CRM, data warehouse, historical results, etc), then you've got yourself a gold mine of data.

But you need to activate it. You need to turn that potential and make it kinetic. (Thanks, Physics.)

How do you take that fuel, that tinder, that coal, and turn it into a humming machine driving optimal results and performance? It’s similar to how a car does it: you need an engine.

Big Data Can’t Be Manually Worked

Manually parsing, sorting, and analyzing data to make decisions is out of the question. You don't have to be an Excel pro to realize that hundreds of thousands of clicks on your copious keywords and all the accompanying data from publisher attributes, conversions and revenue data, and analytics data won’t be easy to draw insights from in a spreadsheet.

Maybe you’ve been there too, with 20 tabs in Excel, trying to stitch, pivot, and analyze. Hitting “Save” is precarious by itself.. you might waste 15 minutes waiting! Besides the speed and infrastructure constraints of spreadsheets, statistical models need to be run on all that data, and preferably on a very frequent basis to make sure that the insights drawn from all that data can be applied to your program via bidding decisions.

The point is, you need something more robust if you’re going to use that data for a meaningful approach at optimizing your paid search campaigns.

At this point, it’s noteworthy to mention that an SEM Agency could be the next step. Give the agency access, and ask for results. Here’s the issue: unless that Agency is employing some additional technical solution, the optimization efforts are still manual and simply can’t hit optimal performance.

Big data, as noted by the Wikipedia definition, requires different methods and tools because you’re dealing with “data sets that are too large or complex to be dealt with by traditional data-processing”.

So, what are your options?

• Building custom scripts
• Purchasing a tool
• Becoming a machine

All jokes aside, hopefully you’re here reading to learn about how to truly optimize PPC programs with big data, automating this revenue-driving channel for your business. (So, let’s count out the cyborg option.)

PPC Optimization Engine for Big Data

Building a script is taking the responsibility into your hands to figure out that relationship between all your data and the results you desire. It’s a big undertaking, but can be very fulfilling.

Frankly, building any script for any job feels good, and makes you proud. In fact, if you’ve been involved in building anything (from a coding or programming or even process-driven perspective), let’s take a minute to realize you’re driving innovation in the small and immediate ways that help to make humans more efficient; the virtuous cycle of ever evolving and improving! (Pause and smile!)

Okay, back to action. Scripts are definitely a good first step if you’ve mastered the programming methodology, bidding methodology, and the syntax and logic. Most often you’ll have to hire the resources to build and maintain that solution.

The benefits? You have total control. Your engineering resource will know all the inner workings, the inputs and outputs, and logic for making bidding decisions based on the data. At a minimum, you’ll need someone with a very strong data analytics background, or better yet a data science background. Needless to say, it’s a lot of work. It’s the classic control versus effort challenge.

There’s another option we noted: purchase a bidding optimization tool.

PPC bidding software automates and optimizes your SEM program through a few steps, starting by integrating all of that big data: online and offline data sources. It models the relationship between the data on keyword performance, predictive data from publishers, audience and attribute segments, and revenue and conversion outcomes. It uses those machine learning-based models and projects the impacts of bids versus revenue. The big data comes into play heavily when estimating the keyword value from a Revenue-per-Click standpoint. Throughout the bid calculation process, PPC optimization tools will crunch your big data sets into accurate, actionable bid decisions.

Taking a step further than just the keyword-level bids, the front-runner PPC tools are able to use a slightly different set of your big data to calculate and automate bid adjustments for attributes like device, audience, or geography. The correlations seen between audience segments + the keywords they clicked and converted on in your data sets provide this.

I’ll paint the picture with a bit more color. Your revenue funnel can be sliced and diced into incredibly specific niches based on your big data. The performance outcome of each slice informs the bid that should be set. Layer in all the keyword data with your contextual data, and audience segments, time-of-day, day-of-week, geography, device, seasonality, etc etc etc: the venn diagram overlap of each of those various attributes will each have a different “worth” to your business. Processing that data with a data science based approach in an SEM optimization tool will figure out which of those segments have value (or not) and adjust bids based on the expected return of each.

Further, tools like these can provide other functionality that is built off of your data–such as forecasting ROAS or ROI. The more data there is to ingest, the better.

Different PPC Optimization Tools

There are some tools and vendors designed for smaller programs without as much data, while some designed for bigger programs with the types of data and scale we’re discussing here. These must have the infrastructural underpinnings that enable clean integration, fast processing, and actionable outputs.

Ad management platforms that “sit on top of” Google Ads (Adwords), Microsoft Advertising (Bing Ads), and Yahoo Ad Manager have a few flavors. Some focus on campaign management and easing up workflow. Some help to cater to multiple channels beyond just paid search. And others are in-line with our current discussion: focused on big data ingestion and activation for peak performance. All of them should do a variation of what we’re talking about here: using performance data to make bidding decisions. However, there are multiple depths to traverse.

Let’s introduce another word: predictive. Predictive bidding optimization tools approach the problem from a slightly different angle. While fundamentally the same, there are nuances that stand out:

• Natural language processing (NLP) allows long-tail keywords with little to no data to still have accurate keyword values associated because NLP shares data from semantically similar keywords with more data, based on scoring algorithms.
• Machine learning is employed to make predictions based on these massive data troves, by using non-linear, supervised learning models that get more accurate with more data and more time. This separates “one-time improvement” from “continually optimizing”.
• While a good model will use a time-series approach to predict the dollar value for a keyword-based on past values, a predictive model literally ingests data and runs it through deep learning algorithms to have a flexible and iterative approach to data-driven optimization. It’s a predictive, outcome-based decision-making process, not a simple “repeat what’s worked in the past” approach.

With big data and optimizing PPC performance, the underlying problem is one of scale. The amount of data we’re talking about isn’t easy to process. There are two solutions that the best SEM bidding optimization software will employ: infrastructure and lower space machine learning.

From an infrastructure perspective, distributed cloud servers with in-memory processing and anomaly detection help to manage the load and manage the scale with speed. Reporting on millions of rows can take a couple of minutes instead of the better part of an hour. These infrastructural elements also prevent problematic data from interfering with bids through bias correction and anomaly detection.

Machine learning strikes again to manage the scale. Specific algorithms can allow the interactions between millions of interactions to be simplified in a different “lower space” to remove the load issue, and reduce the problem into one that’s more practical. This class of algorithms is similar to those used by companies like Netflix, who is also dealing a scale problem: millions of users interacting with thousands of shows and movies.

Big data is difficult to deal with and make work in your favor without the right tools. However, with the right methodology, technology, and incentives (ROI!), all that data your collecting about your customer journey is the perfect weapon for optimizing PPC performance.

Wrapping Up

Bid optimization is very serious and technical business. It gets deeper and deeper the more you look at it. It’s not easy to manage with only our human brains (until we become cyborgs), but the modern suite of automated bidding software for paid search is doing a great job at extracting peak performance from programs with lots of data.

When you’re losing potential revenue from campaigns that aren’t performing at expectations, it’s painful. Some of us work just to live our lives on nights and weekends. However, others are looking to really get ambitious about how to add more value at work, and turn our profession and career into just that: something we are professionals at!