SEO Split Test Result: ChatGPT-Generated FAQs Enriched with FAQPage Schema

Before you start: if you’re unfamiliar with the principles of statistical SEO split-testing and how SplitSignal works, we suggest you start here.

First, we asked our Twitter followers to vote:

This time the majority of our followers were RIGHT! Read further to find out why.

The Case Study

We have previously written about the benefits of using FAQPage structured data, including a case study on the impact of removing FAQ snippets in the search results, as w a step-by-step guide on implementing FAQPage schema and testing its effectiveness using the SEO A/B test analyzer.

FAQPage structured data, commonly referred to as “FAQ schema,” has gained popularity among SEOs since the introduction of its rich result feature by Google in 2019. This feature displays questions and answers directly within search results, enhancing your website’s SERP visibility. The rich result nearly doubles the vertical space occupied by your snippet in search results, effectively pushing competitors down and potentially leading to a higher click-through rate (CTR). Additionally, by providing users with supplementary information about the subject (such as a product category), they can make more informed decisions directly from the search results.

With this in mind, we recently undertook a project to enhance the SEO performance of an e-commerce website by integrating FAQ content into their listing pages and implementing FAQPage structured data. During this process, we encountered a minor challenge: a noticeable absence of FAQ content on the website’s listing pages.

To overcome this hurdle without exhausting significant content resources on writing hundreds of FAQs, we employed ChatGPT, an AI-powered language model, to generate relevant FAQs for the listing pages based on their existing content. This approach ensured that the content was tailored to the specific products and services provided by the e-commerce website.

In this experiment, our objective was to evaluate the effectiveness of the FAQ-rich result in driving increased organic traffic to the listing pages and to build a compelling business case for allocating the necessary resources to implement this change on the website eventually.

The Hypothesis

We hypothesized that the FAQ riFAQ-richt would have a positive impact on organic traffic to the listing pages of the website in question. 

Our hypothesis is based on the following assumptions:

• By adopting the FAQ-rich result, we would claim more SERP real estate, making the website more prominent in search results.
• The increased SERP presence would push competitors further down in the search results, enhancing the website’s visibility.
• By furnishing users with additional information about the product category and its products, we aim to facilitate their decision-making process and enhance their engagement with the search result, ultimately leading to a potential increase in the click-through rate.

The Test

One of the challenges we faced during the test setup was that the listing pages had no existing FAQ content to mark up with structured data. To resolve this issue, we utilized the GPT-3.5-turbo API to create the required content ourselves.

This script processed the existing category text from over 400 URLs, analyzed the content, and generated two relevant FAQs for each URL, ensuring that the generated questions and answers were tailored to the specific content of each listing page.

To better illustrate the test setup, consider a hypothetical scenario involving Best Buy, a well-known electronics retailer. Suppose Best Buy’s listing pages lack any FAQ content to mark up with structured data.

We would employ the GPT-3.5-turbo API to generate the necessary FAQ content in this case. The script scrapes the text below the listing page using the CSS class selector called ‘content’:

Once the text is scraped, it is fed into a ChatGPT prompt to generate the FAQs based on the actual text of the page. This approach ensures that the generated FAQs are highly relevant to the specific product or service offered by the e-commerce website:

The resulting FAQs were then compiled into a Pandas DataFrame for further use.
After generating the necessary FAQ content, we used SplitSignal to set up and conduct the split test. Over 400 pages were selected as either variant or control, with the variant pages having the newly added FAQ content marked up with FAQPage structured data:

The split test ran for 21 days, during which Googlebot crawled 100% of the pages.

The Results

We found that adding FAQ content marked up with FAQPage structured data resulted in a significant 4.3% increase in clicks to the tested pages.

When the blue shaded area falls below or above the x=0 axis in the cumulative view, the test is considered statistically significant at the 95% level. This means that we can be confident that the change will positively impact organic traffic to the listing pages of the website.

As the test continued, we observed a continued increase in clicks to the tested pages, ultimately resulting in a confidence level of 99%. This confidence level further reinforces the implemented change’s efficacy and ability to drive organic traffic to the website’s listing pages. Based on these results, we can confidently recommend incorporating FAQ content marked up with FAQPage structured data as part of an overall SEO strategy to increase organic traffic. 

Note that we are not comparing the actual control group pages to our tested pages but rather a forecast based on historical data. The model predicts the counterfactual response that would have occurred had no intervention taken place. We compare this with the actual data. We use a set of control pages to give the model context for trends and external influences. If something else changes during our test (e.g., seasonality), the model will detect and take it into account. By filtering these external factors, we gain insight into the true impact of an SEO change.

Why?

Structured data offers numerous use cases, but one of the primary motivations for SEOs to adopt it is the potential to obtain rich results. Our test results demonstrate that rich results can indeed have a significant impact on SEO performance.

Analysis of GSC data revealed that there was a substantial impact on the click-through rate (CTR) of the tested pages. This is likely due to the fact that the rich result presents a more comprehensive and visually appealing search result. In the SERPs, the tested website stood out and effectively pushed competitors down in the search results, particularly on mobile devices

Furthermore, this test helped build a strong business case for the website by showing that incorporating FAQ content marked up with FAQPage structured data can significantly increase organic traffic to the website’s listing pages. A solid business case is crucial for justifying the allocation of resources to the project, such as time, personnel, and budget. By demonstrating the potential return on investment (ROI), the business case can help secure the necessary resources for implementing this SEO change.

In the aftermath of our test, it’s important to note that Google has made some adjustments regarding displaying FAQ-rich results. As reported by Barry Schwartz, Google seems to be showing fewer FAQ snippets for many websites starting on April 5th. This is not the first time such fluctuations have occurred, and it could potentially be a temporary change or a bug on Google’s side. Although tracking tools did not display a decline in FAQ snippets, numerous SEOs reported drops in FAQ snippets for some of the sites they monitored, especially on mobile devices.

It’s important to keep remember what works for one website may not work for another. The best way to determine the most effective strategies for your website is to conduct tests and analyze the results. By experimenting with different approaches and measuring the outcomes, you can better understand what works best for your website and target audience. This will help you make informed decisions about optimizing your website and driving more organic traffic to your site.