You can automate keyword research in Python by sending a keyword list to a provider’s API, then storing each result with its source, market, metric definition, and retrieval time. That context matters: search volume and difficulty are estimates, not interchangeable measurements, and an AI Overview flag records a search-result feature in a provider’s snapshot—not a prediction of clicks or a guarantee your site appears in the Overview.
What to decide before you request keyword data
Start with the keywords and the market you want to investigate. For example, a list might contain python tutorial, learn python online, and python for data analysis. Decide which country, language, and search engine apply before fetching metrics. A volume figure without its market is difficult to interpret or reproduce.
As an Amazon Associate I earn from qualifying purchases.
Keep those settings attached to every result, rather than in a note that can be separated from the data. Also record the provider and endpoint: two vendors may use different sources, time windows, and calculations for fields with similar names.
Free tools Windows power users keep installed
One-click scans. No signup required.
Choose an API by the data and workflow you need
Compare documented capabilities, not headline scores. The available documentation does not establish which provider is more accurate; that would require a controlled comparison using the same terms, markets, and dates.
#1 Best Overall
| Provider and interface | Documented fields and definitions | Bulk capacity and qualifications | Useful caveats |
|---|---|---|---|
| Ahrefs API Overview | Country-scoped estimated search volume, latest-month volume, keyword difficulty, SERP features, device shares, and SERP last-update date. SERP feature values include ai_overview. Ahrefs describes volume as an average monthly estimate over the latest known 12 months, with a separate latest-month field. (Ahrefs API Overview) |
The API Overview page documents its request fields; the 10,000-keyword figure applies to the Keywords Explorer interface, not necessarily to this API endpoint. Ahrefs Help says Keywords Explorer accepts up to 10,000 keywords in one search via pasted terms or a TXT or CSV file. (Ahrefs bulk keyword analysis) | Advanced metrics in the UI consume one credit per keyword. Ahrefs says keywords with missing or gray KD can receive a SERP update. These are UI details, not API batch guarantees. (Ahrefs bulk keyword analysis) |
| Semrush Keyword Reports v4 | Can return volume, keyword difficulty, intent, CPC, competition, trends, and SERP features, including AI Overview. (Semrush Keyword Reports v4) | No batch capacity is established here for v4. | Semrush marks v4 as Early Access and warns endpoints, response formats, and pricing can change before general availability. (Semrush Keyword Reports v4) |
| Semrush v3 Batch Keyword Overview | Returns volume, CPC, competition, and number of results for a selected regional database; difficulty is documented separately as an estimate of ranking in Google’s top ten. (Semrush v3 Batch Keyword Overview) | Up to 100 keywords per request, per Semrush’s v3 documentation last updated September 1, 2026. (Semrush v3 Batch Keyword Overview) | Semrush says older v3 methods are deprecated and are not recommended for new integrations, though existing use continues temporarily. Prefer checking v4’s current status before building a new integration. (Semrush Keyword Reports v4) |
| DataForSEO Labs Keyword Overview | Returns CPC, paid competition, search volume, intent, SERP, backlink, and clickstream data. Its Bulk Keyword Difficulty endpoint returns a proprietary 0–100 score relative to Google’s current top ten. (Keyword Overview; Bulk Keyword Difficulty) | Keyword Overview accepts up to 700 keywords; Bulk Keyword Difficulty accepts up to 1,000, according to DataForSEO’s documentation. (Keyword Overview; Bulk Keyword Difficulty) | Other DataForSEO bulk endpoints, including Google Ads Search Volume, Bulk Clickstream Search Volume, and Search Intent, document up to 1,000 keywords per request. Limits are endpoint-specific. (DataForSEO bulk keyword data retrieval) |
| DataForSEO Historical Keyword Data | Provides historical keyword data as a separate endpoint from Keyword Overview, with data reaching back to the beginning of 2019, according to DataForSEO. (Historical Keyword Data) | Up to 700 keywords per request. (Historical Keyword Data) | Use a historical request when you need a series; do not assume the current overview contains the same history fields. |
DataForSEO distinguishes Google Ads data from proprietary metrics derived from its keyword and SERP databases. Its Google Keyword Database documentation says updates happen gradually in the latter part of each month in line with Google Ads’ update cycle; that is a stated pattern, not a guarantee that every keyword refreshes at once. (Google Keyword Database)
Fetch a batch with Python
Use the provider’s current endpoint documentation for the exact route, request fields, authentication method, and response schema. Ahrefs documents a Python requests sample for its API; DataForSEO’s bulk workflow guide also demonstrates a Python-oriented request flow. The example below shows the integration pattern, not a drop-in provider request: set the URL and payload to match the endpoint you choose.
import os
import requests
from datetime import datetime, timezone
keywords = [
"python tutorial",
"learn python online",
"python for data analysis",
]
# Use the exact endpoint and payload documented by your provider.
endpoint = os.environ["KEYWORD_API_ENDPOINT"]
headers = {
"Authorization": os.environ["KEYWORD_API_AUTHORIZATION"],
"Content-Type": "application/json",
}
payload = {
"keywords": keywords,
"country": "us",
"language": "en",
}
response = requests.post(endpoint, headers=headers, json=payload, timeout=30)
response.raise_for_status()
raw_result = response.json()
retrieved_at = datetime.now(timezone.utc).isoformat()Endpoint schemas differ, so do not assume that every service accepts the same parameter names, authentication header, or JSON shape. Keep credentials in environment variables or a secrets manager rather than committing them to source control. In a production job, add bounded retries for transient failures, respect provider rate limits, log status codes and request identifiers, and retain the raw response alongside normalized rows so you can audit or reprocess a result if a schema changes.
Do these 3 things before closing this tab:
1Repair Windows errors before they cause bigger problems2Scan for outdated or missing drivers - takes under a minute3Clear out junk files and repair common Windows errorsStore enough context to make each metric interpretable
Normalize responses into rows for analysis, but preserve the untouched response as well. A practical record includes:
- Keyword and market: keyword text, provider, country or location, language, and search engine or database where applicable.
- Volume: numeric value, provider’s field name, source or method, and measurement window. For Ahrefs, distinguish its latest-known-12-month average from its latest-month field.
- Difficulty: numeric value, provider, scale, and definition. Keep the provider’s terminology; do not merge scores as if they shared a formula.
- Other returned fields: intent, CPC, competition, and SERP features when the endpoint supplies them.
- Provenance: retrieval timestamp, endpoint and API version, response status, and a copy or reference to the raw response.
For example, an AI Overview value should be stored as a feature observation from a particular provider’s response and retrieval time. That makes it possible to distinguish a missing field from an explicit absence and to track changes in what the provider reports.
Interpret volume and difficulty before sorting keywords
Search volume is an estimate, not a universal count
Volume figures can differ because providers use different data sources, markets, update schedules, and time windows. Ahrefs’ API separates an average monthly estimate over the latest known 12 months from a latest-month value. Those answer different questions: the average smooths a year’s variation, while the latest-month field is more recent but can be less representative of a longer pattern. Always retain the provider, market, and window with the number.
DataForSEO describes multiple sources, including Google Ads and Google SERPs, and separates certain current metrics from historical series. Its stated gradual update pattern means a database-wide update window should not be treated as proof that every term has refreshed simultaneously.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
Keyword difficulty is provider-specific
Ahrefs defines KD on a 0–100 scale as an estimate of the difficulty of ranking in Google’s top ten. Its documented method is based on referring domains of top-ten organic pages and omits on-page SEO factors. (Ahrefs: What does KD stand for?)
DataForSEO also describes a proprietary 0–100 score, but frames it relative to the current Google top ten. Similar ranges do not make the scores equivalent: compare keywords within a provider’s methodology, and do not average or directly rank different vendors’ values as one scale. Neither score alone accounts for whether a page matches the searcher’s intent or whether your site can produce a competitive result.
Prioritize with a decision, not one column
Use volume and difficulty as inputs, then inspect intent, the actual search results, and the resources you can bring to a page. A lower-volume term may be more relevant to a specific audience; a high-volume term may be unsuitable if its intent or competition does not match your site. Treat vendor metrics as triage signals for deciding what to investigate, not as a guaranteed traffic forecast.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Detect AI Overviews as a SERP feature
Where the selected endpoint returns SERP features, check its documented feature field for ai_overview or the provider’s equivalent. Ahrefs’ Overview schema lists ai_overview; Semrush Keyword Reports v4 includes AI Overview in its feature list. (Ahrefs API Overview; Semrush Keyword Reports v4)
Recommended Free Tools
Record the field with provider, market, and retrieval time. Feature presence is a snapshot observation from that provider’s data. It does not establish click-through impact, predict future SERPs, or say that your site is cited or shown in the Overview. If you need to know whether your own site appears, use a data source or workflow that reports the actual displayed results, rather than inferring it from a keyword-level feature flag.
Turn the workflow into a repeatable pipeline
- Prepare the input. Deduplicate and validate keyword strings, then define country, language, and engine/database settings.
- Select fields and endpoints. Request only the metrics needed for the task. Use a separate historical endpoint if you need a time series, and check each endpoint’s documented batch cap.
- Submit in batches. Split the list to fit the specific endpoint limit, handle provider throttling and failed items, and retain each request’s settings and response.
- Normalize without discarding provenance. Convert results to a consistent row structure while preserving original field names, definitions, and raw payloads.
- Analyze within comparable groups. Filter or sort by provider, market, and metric definition before comparing values. Treat AI Overview presence as a SERP feature, not an outcome metric.
- Refresh deliberately. Store retrieval dates and SERP update dates where supplied. Re-run data when its age no longer fits the decision, rather than assuming every provider updates all keywords on one schedule.
Limits and API status to check before implementation
Published batch sizes are endpoint capabilities, not evidence of data quality or performance. Ahrefs’ up-to-10,000 figure is for its Keywords Explorer UI; DataForSEO’s documented 1,000- and 700-keyword limits differ by endpoint; Semrush v3’s 100-keyword figure applies to its Batch Keyword Overview method. Do not transfer a limit from one interface or endpoint to another.
Semrush v4 is explicitly Early Access in its documentation, with endpoints, formats, and pricing subject to change until general availability. Its v3 documentation says older methods are deprecated for new integrations. Confirm current availability, fields, limits, and pricing in the provider’s documentation before deploying a pipeline. The documentation establishes capabilities, but it does not provide an independent accuracy comparison across providers.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
