CXL Institute Growth Marketing Review [Week 11]

This week was very insightful, the following are a few things that I learned.

When you’re running tests, if the test is a winner, variation is a winner. You also want to make sure that you’re using that other tools to tell you that the sample size is also okay besides just Optimizely.

Should we treat more important tests with more scrutiny? If you have a test that maybe potentially influences the path that we’re taking with our company, maybe we’re considering a pivot or launching a new product, which these are very important business decisions. For these tests, we want to isolate variables — which means that nothing else is tested at the same time. So no concurrent tests. If you run more than one test, it might cause an overlap between the traffic, impacted here. The second type of test is running the test for at least 28 days, no less, — for this test you definitely need a sample size.

Is the volume dependent on one’s traffic? If there isn’t traffic-related problems, it will mainly be implementation problems. Usually you cannot start a testing program until you have thousands of transactions a month, unless you’re measuring some sort of micro conversions. Micro conversions like, clicks, but clicks don’t correlate with purchases. So it would be best to optimize on “Add to Cart” button clicks. The more that people click on “Add to Cart”, you would think that it’s a better product page. But the user might drop out of the funnel in the next step — therefore, you don’t know if it makes more money. Therefore, this is sill considered a testing program for luxury as an example, for the high volume websites.

Next, I learned about conversion research. Conversion research consists of the following categories:

• Experience based assessment
• Site walkthroughs
• Heuristic analysis
• Usability analysis
• Qualitative research
• Online surveys with recent customers
• On-site polls
• Phone interviews
• Live chat transcripts
• Customer support insight
• User testing
• Quantitative research
• Web analytics analysis
• Mouse tracking analysis

When it comes to the ResearchXL Model, you need to first establish that a problem is occurring. If there isn’t a problem, yet you still try to fix something, that’s how most people optimize their website. Most see the website and think this is a problem, there’s some sort of a symptom, let’s fix it, with out any data, with out any analysis. Putting this as an example, would you want a doctor to operate on you based on a gut feeling or rather do blood tests, MRI scans to determine if there’s actually a problem? The same applies to a website. If you’re just operating based on opinions and gut feeling, you’re not getting very far and you’re gonna be screwing up a lot of things up —that aren’t needed. You need to focus on gathering data.

What kind of data do you need? As an example, if you need to cross the road, there are things that you might considered, such as the angle of the sun in the sky and the wind speed and how many birds are singing at the same time and how many people are standing next to you and what’s their eye color and all that stuff. But is this data actually helpful? No. The data that matters most is knowing if there’s a car coming and how fast are they driving at — those are the only data points that matter. The same stuff applies to conversion optimization. We could have the perfect data in the world, all data about every possible thing, but that only creates analysis paralysis. It’s just too much data, we’re overwhelmed by this data, we don’t know what to do with it. So we need the right amount of data.

We don’t just need any data, but data which we can act upon. So if you know the color of the eye of the person standing next to you, this data is not very useful. Is there a car coming and how fast is it going — this data is very useful. Something can be done with that information. The same can be applied to conversion optimization. To figure out the conversion optimization we need to gather data that we can read, analyze, pull in other data — if needed, and then say, okay, I think I know what we should do. For any data that you gather, you should be able to answer this question: What will I do differently based on this data? Or what could I possibly be doing differently based on this data. If the answer to these questions is nothing, then you don’t need that piece of data.

The next thing to do is a technical analysis — keep in mind that even the most persuasive website isn’t going to make money if it doesn’t work on a specific device and browser. If someone is using a website and it’s broken and doesn’t work, most likely a person won’t buy from it. Therefore, an analysis to see if a certain browser or a certain device that converts much lower than other browsers or devices works best is needed. This could be an indication of a quality assurance problem — maybe there’s a technical bug. Small technical bugs can cause a company to lose a lot of money, so those problems need to be fixed first. Another thing to take care of is speed analysis. Site speed matters, if a website takes 10 seconds to load a page, it’s too slow, it will have some impact on the user. Therefore, the second step of a technical analysis is to check on how fast a website loads.

After the technical analysis, we want to understand where the problem is, so that we can dive into digital analytics. For digital analytics, you could use Google Analytics. With Google Analytics we want to understand key things. For example, we want to understand where’s the flow stack? Meaning people flowing through your website going from page to page, where are they dropping out? Where is the friction point? We want to understand where is the money leaking out, in which segments, what type of users are dropping out more than others, all kind of things like that. We also want to find out correlations. What types of behavior correlate with more purchases?

For example, maybe people who buy more stuff, are searching for more stuff, or they use these filters, or they go read a specific type of page and take whatever actions. We want to be able to measure and see if there are any correlations between behaviors and final outcomes. In order to do that, we need to make sure that everything a user can do on your website is being measured. We need to check if we are measuring everything that needs to be measured, and whether that data is actually legit, 90% of the analytics configurations that most people come across are actually broken, so this is a good thing to practice doing.

Next, we move into qualitative research, which consists of surveys and user testing. With surveys, there are two types of surveys you should do. We should survey people who are on your website, when we survey them with on-site polls. So these people might or might not be buying something. Most of them are not buying anything, and so we can put a poll on key pages. Pages maybe where in digital analytics saw there’s a large drop-off rate, so people get to your product page, but they’re not adding the product to the cart. Why is that? So we need to put a poll on the website and ask them if there is anything holding them back from buying this product right now? If they select yes, then you ask them for an explanation. I’ll focus on learning the next survey the next week.