As artificial intelligence (AI) upends search engine optimization (SEO), marketers are working to update their digital strategies to ensure their brands are visible not only in traditional search engine results, but also in the outputs of large language models, such as ChatGPT and Claude. However, for all this new technology, the fundamentals of pre-AI SEO are still crucial building blocks for performance in the era of AI-driven searches.
Page Indexing
The first step to improving search performance is ensuring that the pages you want search engines to crawl and index are visible to those search engines. Success here is typically tied to the behind-the-scenes technical setup of your website (e.g., metadata, language references, index tags, xml sitemaps, and canonical tags).
Page indexing has several components that businesses must consider. Start with the sitemap, which gives the clearest picture of your site to any crawler. Does your sitemap accurately represent all the pages on your site that you would like the site crawlers to index? If not, you must perform some clean-up work before you can move forward. Try breaking down the sitemap into logical categories, such as products and geographically unique pages (for sites with a global footprint), which can help pinpoint common site content that is trailing behind in visibility. This exercise can also help you discover pages that are missing from your site and should be included, as well as pages that are included but should be removed.
Now that you know how many pages site crawlers could index for your site, you can dig into the reasons why your site is falling short. Make sure that you are not blocking access through a robots.txt file or no-index, no-follow tags, then dig into pages that look like they have identical content. Crawlers are trying to find the best page for each unique expression of information, and if they determine that two or more pages are too similar (for example, if the pages have identical titles, descriptions, and content), they will index the page they deem best and apply what is known as a canonical tag to the others, marking these pages as excluded from indexing.
Marketers can guide crawlers by reviewing pages with similar content and either adding a canonical tag to one or more of the pages, effectively removing them from the competition to be the page indexed, or revising the content to differentiate the pages. In some cases, the changes can be as simple as an updated page title and description. In other cases, marketers may want to more thoroughly rework the page content. There may also be legitimate reasons for duplicate content, such as regional indexing; in this case, you will need to employ hreflang tags to have the site crawlers index both pages.
Although there are additional issues that can affect indexing, these steps will address the majority of the issues you are likely to encounter and will position your site for the next fundamental—page visibility.
Key Performance Indicator (KPI): Percentage of pages indexed
- Formula: Pages indexed divided by total pages in your sitemap, less canonical tags (which we discuss above).
- Best tool for the job: Google Search Console. This is your behind-the-scenes look at exactly what Google sees when it looks at your site.
PAGE VISIBILITY
Traditional SEO would categorize this as page rank, while our AI-era lens looks at appearance and usage in creating AI results. In both cases, the goal is the highest possible exposure for the content that is relevant to your site.
When considering exposure, it can be useful to analyze content on a spectrum from “broad topic” to “highly ownable.” The expectation for visibility increases as we move up the scale. Broad topics typically have more search volume but are also more crowded spaces. Competing in these spaces requires paid advertising and an aggressive and sustained SEO plan. At the other end of the spectrum, highly ownable content includes branded terms and unique keywords (e.g., company name, employees, product names, trademarks). This content will have lower search volume but few or no organic competitors. It is reasonable to expect that highly ownable content will rank high with little effort. Further, the closer you get to either end of the spectrum, the less fluctuation you’ll see with regard to exposure; it is just as unlikely that you will jump from position 50 to position 2 in a broad reach keyword as it is that your branded term will drop from position 2 to position 50. If your highly ownable content is not ranking highly, you should spend some energy discovering and fixing the causes of this unexpected result.
The middle of the spectrum contains both the greatest opportunities and the most volatility. In this space, you can identify keywords that have strong search volume and little competition. You can also identify a subset of focused keywords around which you can build content to move up the ranks and increase your exposure in AI outputs.
Start by identifying any pages on your site that are not attributed to any ranking keywords. Is the page too generic? Is the content too short? Is no one searching for this information? When considering these issues, be sure to weigh the effort required to fix the issue against the potential gain you may realize as a result, to help prioritize your efforts.
Next, review keywords on pages that are in the middle of your buyers’ journey–awareness and consideration. Typically, these are also the pages that fall in the middle of the content spectrum. Look for keywords that hit the high volume/low competition sweet spot.
Improvement in this area is all about content. By filling in gaps between the content in your pages and the content in the pages above you in the search rankings, you can increase your pages’ visibility.
KPI: Pages appearing in AI and organic search results. It can be useful to set a ranking threshold (e.g., top 50 or top 100).
Best tool for the job: SEO monitoring tools that connect pages to search appearance.
PAGE PERFORMANCE
The previous two KPIs will help you troubleshoot and improve exposure in both organic and AI-driven searches; however, it is important to ensure that these efforts ultimately support the desired outcome of your SEO program: content findability.
It can be useful to look at the total traffic from organic and AI-driven searches and the purchases/leads you obtain from this channel; however, if you want to maximize the effectiveness of your entire site, focus on the unique pages visited as a result of these searches.
KPI: Unique pages visited as a result of organic and AI-driven searches.
Tip: Divide this number by the number of indexed pages to calculate the width of your search net.
CONCLUSION
Although the specific strategies and approaches to building page content are changing in response to the rise of AI-driven searches, the fundamentals still hold true. Start with technical issues to ensure you are not hampering or stopping site crawlers from indexing your content. Then focus your improvement efforts on the middle of the search opportunity spectrum. Finally, track your KPIs closely to assess the effectiveness of your efforts.
Searches remain a crucial avenue by which a brand attracts buyer interest. Focusing on the fundamentals can still help brands stand out, even amid the fast transformation of search behaviors and channels.
