Businesses new to search marketing and veterans alike can easily come to the wrong conclusions about their pay-per click campaigns. Imagine you pay $1 each for 200 clicks and you only complete one $50 sale. You’ve spent $200, and only earned $50. You calculate your Return on Ad Spend and you’re earning $.25 for every $1 spent on pay-per-click advertising. You might decide this channel of advertising just isn’t effective, but in fact it is far too soon to tell.
In the previous article, Paid Search: Bidding Based on ROI, we showed how you can use ROI data to determine optimal bids. The methods we described assume that you know the conversion rate of a keyword, ad group or campaign. But in fact you can’t ever really know a conversion rate, the best you can do is to estimate using historical data. This article explains how to use confidence intervals to ensure that your estimates are reasonably accurate.
For example, let’s say that a keyword has had 100 clicks and 2 conversions. Do you know what the conversion rate is for that keyword? The obvious answer is 2%, but the correct answer is “no”. From experience, you probably know that you could get fewer or more conversions in the next 100 clicks. Given a set of sample data, the best you can say is that you expect that the conversion rate falls between X% and Y%. In statistics, this is called a “confidence interval”. A confidence interval also has a “confidence level”. The confidence level describes how sure you are that the conversion rate falls between X% and Y%. For example, an 80% confidence level means you can be 80% sure that the actual value falls between the lower and upper bound of the interval. This means that there is a 10% chance that the actual value falls below x% and a 10% chance that it falls above Y%.
Below is a table of confidence intervals, with the column on the left indicating the number of conversions observed, and the row across the top indicating the number of clicks observed. The confidence level for these intervals has been set at 80%. The cells with white backgrounds show the confidence interval for each combination of conversions and clicks. The confidence intervals are expressed as X% – Y%, with X% being the lower bound, and Y% being the upper bound.
| 100 | 200 | 300 | 400 | 500 | |
| 0 | 0 % – 2.3% | 0% – 1.1% | 0% – 0.7% | 0% – 0.6% | 0% – 0.5% |
| 1 | 0.1% – 3.8% | 0.1% – 1.9% | 0% – 1.3% | 0% – 1% | 0% – 0.8% |
| 2 | 0.5% – 5.2% | 0.3% – 2.6% | 0.2% – 1.8% | 0.1% – 1.3% | 0.1% – 1.1% |
| 3 | 1.1% – 6.6% | 0.6% – 3.3% | 0.4% – 2.2% | 0.3% – 1.7% | 0.2% – 1.3% |
| 4 | 1.8% – 7.8% | 0.9% – 4% | 0.6% – 2.7% | 0.4% – 2% | 0.4% – 1.6% |
| 5 | 2.5% – 9.1% | 1.2% – 4.6% | 0.8% – 3.1% | 0.6% – 2.3% | 0.5% – 1.9% |
| 6 | 3.2% – 10.3% | 1.6% – 5.2% | 1.1% – 3.5% | 0.8% – 2.6% | 0.6% – 2.1% |
For example, if we observe 3 conversions over 400 clicks, the confidence interval ranges from 0.3% to 1.7% – I have shaded that cell pink in the table. Therefore, we are 80% sure that the actual conversion rate falls between these two values. Compare the ranges in the “500” column to the “100” column and you will notice an important fact about confidence intervals: the more data we have, the smaller the confidence interval. And the smaller the confidence interval, the better an idea we have of the actual value. But also note that there is always an interval – we never truly know the actual value from observed data. Here are a few scenarios demonstrating how this data might be applied:
- Jane buys 200 clicks and gets no conversions and decides that paid search is a waste of money. Lisa points out that there’s a good chance that Jane’s conversion rate is as much as 1% and that she should wait it out a bit.
- Lisa wants to bid based on a target CPA. After buying 300 clicks, she gets 5 leads. He assumes a conversion rate of 0.8%. Given that the confidence interval is 0.8%-3.1%, Lisa is being conservative by taking the lower bound of the range.
- Jane thinks her conversion rate is 10%, but she measures 4 conversions over 200 clicks for her AdWords campaign. She realizes the data is not supporting her assumption: based on the data she can be 90% confident that her conversion rate is not more than 4%.
The main takeaway from this table is that even 500 clicks is not enough to have a very accurate idea of conversion rate, which is why I say you shouldn’t be bidding on most individual keywords based on ROI goals–there generally isn’t enough keyword-level data to make good decisions.
If you want to test out some confidence intervals of your own, this site has an interactive calculator. Use the binomial confidence interval and change the confidence level at the bottom of the page if you want it to be different than 95%. The higher the confidence level, the larger the intervals you will get. The numerator (x) is the number of conversions you observe, and the denominator (N) is the number of clicks you bought.
The approach I like to take when bidding is to start with an idea of the conversion rate for the campaign as a whole, and set bids based on that, since I have the most data at the campaign level. Then I look at individual ad groups and change bids if the confidence interval tells me that the ad group conversion rate is likely more or less than the campaign conversion rate. Then I look at individual keywords, but only those that have enough data to produce a useful confidence interval.
At Two Octobers we use automated tools to manage bid setting according to these principals, but it’s important to understand the underlying concepts. And bids are just one lever in a campaign. Ad copy, keyword selection and the user experience are at least as important as tweaking bids. More on that to come.
This article gives a high-level overview of a very complex topic. I hope it is useful to you, but please contact us if you would like help optimizing your paid search campaigns.
