The evolution of NLP methods and what that means for AI search
Before AI, much of the Natural Language Processing (NLP) methods used in data science were quite basic and blunt. Take the NLP technique of Term Frequency – Inverse Document Frequency (TF-IDF). It scores how important a page is for a target keyword by counting how often the keyword appears on that page and comparing it against how often it appears elsewhere on the web.. Thus a high TF-IDF score for a keyword aligned web page would be considered highly optimized.
While there have been refinements such as Best Matching 25 (BM25), Large Language Models (LLMs) are leagues above. Information retrieval has moved from word overlaps to meaning. An LLM understands the difference between the words “river bank” and “bank loans.” Classic frequency-based methods never could.
Optimizing for AI goes beyond language itself. To help calibrate its interpretation of language on the web, AI is increasingly becoming reliant on external web standards such as schema and more recently the Open Knowledge Format (OKF) announced by Google. OKF is an open specification Google Cloud that represents knowledge as a directory of Markdown files with YAML frontmatter. Therefore AI search optimization will need to move beyond classic data science methods to help make content more meaningful for visibility.
One approach to identifying topics for AI search
The core problem to solve is understanding what AI search would consider to be a topic or in AI terms an entity.
It’s likely that any single content blog or guide will cover multiple entities. Some of those entities are the point of the article. Others are passengers. And a passenger entity that carries real substance is a candidate for an article of its own.
One four-step approach to use:
- Export your URLs from Oncrawl and filter for the indexable ones .
- Scrape the body content from each URL using Python’s Beautiful Soup.
- Use an LLM API to analyze it and determine topics which will be verified by external sources.
- Compare lead entities against secondary entities to find topics that no page currently owns.
Step 1: Getting URLs from the Oncrawl API
The starting point is to launch a crawl on the website you’re working on as normal. In this use case we’re mostly interested in the body content and the URL. We’ll also assume that we’re only interested in content made available for crawling and indexing.
Once Oncrawl has run its audit, export the data by going to Data Explorer > Export data.
Step 2: Python’s Beautiful Soup for Content Extraction
With the obtained URLs from Oncrawl, we read these in and filter for the indexable ones before using Python’s Beautiful Soup module to scrape the entire content of each indexable URL (redacted):
url indexable 0 ... true 1 ... true 2 ... true 3 ... true 4 ... true
The code output for the column containing the scraped content will look like:
[{'tag': 'p', 'text': 'Innovation for Xyzs'},
{'tag': 'h1', 'text': 'Xyzs Innovation'},
{'tag': 'p',
'text': 'We do this for your company.'},
{'tag': 'h2', 'text': 'Our promises'},
{'tag': 'h2', 'text': 'We're here for you'},
{'tag': 'p',
'text': 'We do this and that to make you look good.'},...
Keeping the tags is worth the extra column. An entity that only ever appears in body paragraphs sits in a different position from one that has its own H2, and that distinction is useful later when you decide which candidates to act on first.
Step 3: Using a LLM API to identify topical entities
For each URL, the scraped text will be fed as a prompt using the Claude API to identify the topical entities including the lead entity and any other secondary entities. Claude has chosen (because it works and everyone trusts Claude).
This will help us identify any potential duplicate topics conceptually, but also any other topics that deserve an article of their own.
The prompt settings are set to a low temperature as we don’t want the LLM to get too creative, but instead be a bit more factual.
Also set a high token input limit so some effort is expended to verify that the topical entity is valid and real according to the knowledge graph and not hallucinated. The Wikipedia verification isn’t perfect, so we apply filters through additional LLM querying to drop wiki references that are incorrect.
The output is stored back into the DataFrame. For each URL you get a set of lead topics and a set of secondary topics, each with a proportion score and a verification link.
An extract is shown below:
top_topics \
0 [{'topic': 'Startup Ecosystems', 'prop': 25, 'wikipedia_url': 'https://en.wikipedia.org/wiki/Startup_ecosystem' }, {'topic': 'Business Strategy', 'prop': 20, 'wikipedia_url': 'https://en.wikipedia.org/wiki/Strategic_management'}]
1 [{'topic': 'Renewable Energy', 'prop': 35, 'wikipedia_url': 'https://en.wikipedia.org/wiki/Renewable_energy'}, {'topic': 'Smart Grid', 'prop': 25, 'wikipedia_url': 'https://en.wikipedia.org/wiki/Smart_grid'}]
2 [{'topic': 'Startup ecosystem', 'prop': 30, 'wikipedia_url': 'https://en.wikipedia.org/wiki/Startup_ecosystem'}, {'topic': 'Venture capital', 'prop': 25, 'wikipedia_url': 'https://en.wikipedia.org/wiki/Venture_capital'}]
other_topics
0 [{'topic': 'Entrepreneurship', 'prop': 12, 'wikipedia_url': 'https://en.wikipedia.org/wiki/Entrepreneurship'}]
1 [{'topic': 'Energy Storage', 'prop': 20, 'wikipedia_url': 'https://en.wikipedia.org/wiki/Energy_storage'}, {'topic': 'Carbon Capture', 'prop': 12, 'wikipedia_url': 'https://en.wikipedia.org/wiki/Carbon_capture_and_storage'}, {'topic': 'Electric Vehicles', 'prop': 10, 'wikipedia_url': 'https://en.wikipedia.org/wiki/Electric_vehicle'}, {'topic': 'Solar Energy', 'prop': 10, 'wikipedia_url': 'https://en.wikipedia.org/wiki/Solar_energy'}, {'topic': 'Wind Energy', 'prop': 10, 'wikipedia_url': 'https://en.wikipedia.org/wiki/Wind_power'}, {'topic': 'Artificial Intelligence', 'prop': 8, 'wikipedia_url': 'https://en.wikipedia.org/wiki/Artificial_intelligence'}, {'topic': 'Biofuel', 'prop': 8, 'wikipedia_url': 'https://en.wikipedia.org/wiki/Biofuel'}, {'topic': 'Nuclear Energy', 'prop': 5, 'wikipedia_url': 'https://en.wikipedia.org/wiki/Nuclear_power'}, {'topic': 'Geothermal Energy', 'prop': 5, 'wikipedia_url': 'https://en.wikipedia.org/wiki/Geothermal_energy'}, {'topic': 'Circular Economy', 'prop': 5, 'wikipedia_url': 'https://en.wikipedia.org/wiki/Circular_economy'}]
2 [{'topic': 'Artificial intelligence', 'prop': 8, 'wikipedia_url': 'https://en.wikipedia.org/wiki/Artificial_intelligence'}, {'topic': 'Biotechnology', 'prop': 5, 'wikipedia_url': 'https://en.wikipedia.org/wiki/Biotechnology'}]
The key information you should be able to extract from the above:
- URL 1 lead topics: Startup ecosystems (25), Business strategy (20). Secondary: Entrepreneurship (12).
- URL 2 lead topics: Renewable energy (35), Smart grid (25). Secondary: Energy storage (20), Carbon capture (12), Electric vehicles (10), Solar energy (10), Wind energy (10).
- URL 3 lead topics: Startup ecosystem (30), Venture capital (25). Secondary: Artificial intelligence (8), Biotechnology (5).
The proportion score is the share of the page the model attributes to that entity, which gives you a rough sense of whether a topic is central to the page or incidental to it.
Step 4: Find the topics that no page owns
We now have our lead and secondary topical entities which are scored. However what we really want to know are the topics that don’t have their own dedicated articles.
We’ll do this by analyzing the output to see which top two topics are not in the top_topics column.
The comparison is straightforward: take the highest-scoring secondary topics for each URL and check whether they appear anywhere in the lead topics column. Anything that does not is a gap:
new_topic
0 Startup Ecosystems Entrepreneurship 1 Renewable Energy Energy Storage 2 Startup ecosystem Artificial intelligence
In this case, the above which don’t exist as pages or content in their own right, are candidates for creating new content.
Applying this to your GEO content strategy
We’ve covered a basic procedure to audit the topics covered by your content and unearth any potential topics that deserve an article of their own that would be recognized by AI as a topic in their own right.
While the method is not exhaustive, it benefits from first party data that does not require any exotic or expensive paid data subscriptions. However, with additional dimensions from first party data, the analysis could be enhanced by using:
- GA4 traffic: The overall traffic data on URLs from all traffic sources could act as an index of what “normal” content demand and engagement looks like. I say normal because I’m assuming search traffic doesn’t make up any more than say 20% of overall site traffic. This could be used to help prioritize topics on your site.
- Google Search Console: Useful for seeing how many URL impressions a topic receives and therefore the search demand. However, caution must be exercised as impressions are largely a function of rank position and therefore some normalization needs to be applied to control for position i.e. 100 impressions for a URL ranked #2 is not the same for another URL with 100 impressions ranked #12.
- Social media shares: Another alternative proxy for prioritizing topics found by the URL’s popularity on social media.
In addition to topic discovery, the scraping alone could be used to create markdown files for Open Knowledge Formats (OKFs).
Wrapping up…
Topic research for AI search is not the same exercise as keyword research. What we are really trying to identify is which entities AI systems recognize and which of those entities your site can legitimately claim.
A crawl export, a scraper, and a well-constrained LLM prompt are enough to answer that question on your own content, without buying a single new data source. The output is a short list of topics your site already discusses well but has never given a home.
Start with the gap list. Prioritize with your own traffic data. Then go and write the article that entity deserves.
If you would like to explore more advanced methods for topic optimization and discovery, including the use of website analytics and data from social media, you can read more about it here: “Generative Engine Optimization with Python: Data-Driven Methods for LLM Retrieval and Citation”.
