How to scrape Google AI Mode: citation pills + place cards

Google AI Mode is Google’s generative search interface, built around citation pills, multi-source synthesis, and streamed response generation that go well beyond what a standard LLM endpoint returns.

It also wasn’t built for programmatic access. The interface relies on async network calls, citation metadata buried in HTML comments, and dynamic response loading that off-the-shelf scrapers don’t handle.

After analyzing thousands of AI Mode interactions, we’ve worked out the full pipeline. This guide walks through how to scrape it and pull out the structured data worth keeping.

Table of contents

• Why scrape Google AI Mode responses?
• Understanding Google AI Mode’s architecture
• The citation pill parsing challenge
• Building the scraping infrastructure
• Parsing AI Mode responses and citations
• Extracting structured data from responses
• Handling network interception and async responses
• Using cloro’s managed Google AI Mode scraper

Why scrape Google AI Mode responses?

AI Mode produces content you can’t get through any other interface.

What’s in a response:

• The generated answer text, with formatting preserved
• Citation pills carrying embedded metadata and source links
• HTML-comment-based source attribution
• Streamed loading with mid-response updates
• Text, markdown, and HTML representations

Traditional search APIs don’t surface any of this. If you want to study how Google’s generative layer cites the web, you need the rendered page.

Use cases:

• AI research. Study citation patterns and source attribution.
• Content analysis. Examine the structure of generated answers.
• SEO intelligence. Track how AI Mode picks sources.
• Compliance monitoring. Watch response quality over time.

Compare this with scraping Google Gemini to understand the differences in Google’s AI implementations.

Understanding Google AI Mode’s architecture

A few moving parts make AI Mode harder to scrape than a normal SERP.

Request flow

1. Initial request. User searches with the udm=50 parameter.
2. Response routing. Google’s AI Mode pipeline takes over.
3. Async response loading. Content streams in from /async/folwr.
4. Citation pill generation. HTML comments carry the source metadata.

Response structure

A single response packs several data types:

• AI response text with inline citations
• Citation pills (buttons) wired to embedded source links
• HTML comments holding the structured citation metadata
• Different DOM selectors depending on page layout
• Text, markdown, and HTML representations of the same answer

Technical challenges

• Async loading via /async/folwr
• Citation metadata buried in HTML comments
• Selector drift between web-results and normal layouts
• Cookie-based session persistence
• Behavioral anti-bot detection

The citation pill parsing challenge

The citation system is the part that trips up most scrapers. Here’s what you’re working with.

Citation pill architecture

HTML comment embedding:

<!--Sv6Kpe[["uuid-12345",["label","description"],["https://example.com","source2"]]]-->
<button data-icl-uuid="uuid-12345" data-amic="true">[1]</button>

Multi-source citations:

• A single pill can reference multiple URLs
• UUIDs link pills to their metadata blocks
• Source extraction happens by parsing HTML comments
• Internal Google URLs need filtering out

Metadata extraction

UUID-based mapping:

# Extract citation pills with UUID mapping
pill_locators = page.locator('button[data-icl-uuid][data-amic="true"]')

# Parse HTML comments for citation metadata
pattern = r'<!--Sv6Kpe\[\["{uuid}".*?]]-->'
comment_blocks = re.findall(pattern, page_html, re.DOTALL)

Source URL processing:

• Filter out internal Google URLs
• Clean encoding and fragments
• Handle multiple sources per citation
• Pull out descriptions and labels

Building the scraping infrastructure

Here’s the infrastructure you need for a reliable AI Mode scraper.

Core components

import asyncio
from playwright.async_api import Page, Browser
from services.cookie_stash import cookie_stash
from services.page_interceptor import PlaywrightInterceptor
from services.captchas.solve import solve_captcha
from bs4 import BeautifulSoup
import html2text

AIMODE_URL = "https://www.google.com/search"

Request configuration

class AIModeRequest(TypedDict):
    prompt: str  # AI Mode query
    country: str  # Country code
    include: Dict[str, bool]  # Content options (markdown, html)

URL construction with AI Mode parameters

# AI Mode requires specific URL parameters
search_url = build_url_with_params(
    AIMODE_URL,
    {
        "udm": 50,  # Enable AI Mode
        "aep": 11,  # Additional AI Mode parameter
        "q": prompt,  # Search query
        "hl": google_params["hl"],  # Language
        "gl": google_params["gl"],  # Country
    },
)

Network interception setup

# AI Mode uses async response loading
page_interceptor = PlaywrightInterceptor(do_not_block_resources=True)
page_interceptor.add_capture_urls(["https://www.google.com/async/folwr"])

# Wait for async response (up to 60 seconds)
for _ in range(120):
    if len(page_interceptor.captured_responses):
        break
    await sleep(500)
else:
    raise Exception("Never received AI Mode response after 60 seconds")

Parsing AI Mode responses and citations

The citation system forces a parsing approach that doesn’t match typical SERP scraping.

Text extraction

async def extract_aimode_text(page: Page) -> str:
    """Extract text content from AI Mode response."""
    try:
        # Find element with data-session-thread-id
        thread_element = page.locator("[data-session-thread-id]")

        # Get parent div's text content
        parent_div = thread_element.locator("..")
        text = await parent_div.text_content() or ""
        return text.strip()
    except Exception as e:
        logger.warning(f"Could not extract text: {e}")
        return ""

Citation pill extraction

async def extract_aimode_citation_pills(page: Page) -> Dict[str, List[LinkData]]:
    """Extract citation pills with embedded metadata."""
    citation_pills: Dict[str, List[LinkData]] = {}

    # Find all citation buttons
    pill_locators = page.locator('button[data-icl-uuid][data-amic="true"]')
    pill_count = await pill_locators.count()

    # Get page HTML for comment parsing
    page_html = await page.content()
    page_html = html.unescape(page_html)

    for i in range(pill_count):
        pill_button = pill_locators.nth(i)

        if not await pill_button.is_visible():
            continue

        uuid = await pill_button.get_attribute("data-icl-uuid")
        if not uuid:
            continue

        # Extract citation metadata from HTML comments
        pattern = rf'<!--Sv6Kpe\[\["{re.escape(uuid)}".*?]]-->'
        comment_blocks = re.findall(pattern, page_html, re.DOTALL)

        current_pill: List[LinkData] = []
        for content in comment_blocks:
            # Extract description
            desc_match = re.search(
                rf'"{re.escape(uuid)}"\s*,\s*\[\s*"[^"]+"\s*,\s*"([^"]+)"',
                content,
            )
            description = desc_match.group(1) if desc_match else None

            # Extract all URLs, filter out Google internal
            all_urls = re.findall(r'"(https://[^"]+)"', content)
            url = None
            for potential_url in all_urls:
                if not any(skip in potential_url
                          for skip in ["google.com", "gstatic.com", "encrypted-tbn"]):
                    url = potential_url
                    break

            if url:
                # Clean up URL
                if "#:~:text" in url:
                    url = url.split("#:~:text")[0]
                url = url.replace("\\u003d", "=").replace("\\u0026", "&")

                current_pill.append(LinkData(
                    position=len(current_pill) + 1,
                    label=f"Source {len(current_pill) + 1}",
                    url=url,
                    description=description,
                ))

        if current_pill:
            citation_pills[uuid] = current_pill

    return citation_pills

Source link extraction

async def extract_aimode_sources(
    page: Page, is_web_results_page: bool = False
) -> List[LinkData]:
    """Extract source links with different selectors for page types."""

    # Different selectors for different page layouts
    sources_selector = (
        '[data-container-id="rhs-col"] [role="dialog"] a'
        if not is_web_results_page
        else "a.ZbQNgf"
    )

    sources: List[LinkData] = []

    try:
        await page.wait_for_selector(sources_selector, timeout=10_000, state="attached")
        sources_locator = page.locator(sources_selector)
        source_elements = await sources_locator.all()

        for position, element in enumerate(source_elements, start=1):
            url = await element.get_attribute("href")
            label = await element.get_attribute("aria-label")

            if url and label:
                sources.append(LinkData(
                    position=position,
                    label=label,
                    url=url,
                    description=None,
                ))
    except Exception as e:
        logger.warning(f"Could not extract sources: {e}")

    return sources

Extracting structured data from responses

AI Mode supports several output formats, each useful for different downstream work.

HTML to markdown conversion

def convert_aimode_html_to_markdown(
    html_content: str, citation_pills: Dict[str, List[LinkData]]
) -> str:
    """Convert AI Mode HTML to markdown with proper citation links."""
    if not html_content:
        return ""

    soup = BeautifulSoup(html_content, "html.parser")

    # Find citation buttons
    buttons = soup.find_all("button", attrs={"data-icl-uuid": True, "data-amic": "true"})

    for button in buttons:
        if not isinstance(button, Tag):
            continue

        uuid = button.get("data-icl-uuid")
        if not isinstance(uuid, str):
            continue

        pill_links = citation_pills.get(uuid, [])

        # Replace citation buttons with actual links
        new_anchors: List = []
        for _, link_data in enumerate(pill_links):
            source_text = link_data.get("label")
            url = link_data.get("url")

            new_anchor = soup.new_tag("a", href=url)
            new_anchor.string = source_text
            new_anchors.append(new_anchor)

        # Insert links and remove button
        for anchor in reversed(new_anchors):
            button.insert_after(anchor)
        button.decompose()

    # Convert to markdown
    h = html2text.HTML2Text()
    h.ignore_links = False
    h.ignore_images = False
    h.body_width = 0
    h.unicode_snob = True

    markdown = h.handle(str(soup))
    return markdown.strip()

Response processing pipeline

async def parse_aimode_response(
    page: Page, request_data: ScrapeRequest
) -> ScrapeAiModeResult:
    """Complete response processing pipeline."""
    include_markdown = request_data.get("include", {}).get("markdown", False)
    include_html = request_data.get("include", {}).get("html", False)

    # Detect page type
    is_web_results_page = bool(await page.locator(".RbCUdc").count())

    # Extract core content
    text = await extract_aimode_text(page)
    sources = await extract_aimode_sources(page, is_web_results_page=is_web_results_page)

    if not len(sources):
        raise Exception("no sources")

    # Extract citation metadata
    citations = await extract_aimode_citation_pills(page)

    result: ScrapeAiModeResult = {
        "text": text,
        "sources": sources,
    }

    # Optional markdown conversion
    if include_markdown:
        ai_mode_html = await extract_aimode_html(page)
        markdown = convert_aimode_html_to_markdown(ai_mode_html, citations)
        result["markdown"] = markdown

    # Optional HTML upload
    if include_html:
        result["html"] = await upload_html(
            request_data["requestId"], await page.content()
        )

    return result

Handling network interception and async responses

The async loading model needs explicit handling. The page renders before the answer arrives.

Async response capture

# Set up network interception for async responses
page_interceptor = PlaywrightInterceptor(do_not_block_resources=True)
page_interceptor.add_capture_urls(["https://www.google.com/async/folwr"])

# Configure page interceptor
await page_interceptor.setup_page_interceptor(page)

Response timeout handling

# Wait for async response with timeout
async_response_received = False

for attempt in range(120):  # 60 seconds max wait
    if len(page_interceptor.captured_responses):
        async_response_received = True
        break

    await sleep(500)  # 500ms intervals
else:
    raise Exception("Never received AI Mode response after 60 seconds")

if async_response_received:
    logger.info("Async AI Mode response captured successfully")

Error detection and recovery

# HTTP error handling with CAPTCHA detection
response = await page.goto(search_url, timeout=20_000)

if response is None:
    raise Exception("Navigation failed - no response received")

if not is_http_success(response.status):
    # Handle potential CAPTCHA
    solved_captcha = await solve_captcha(page, page_interceptor)
    metadata["solved_captcha"] = solved_captcha

    if not solved_captcha:
        raise Exception(f"HTTP error: {response.status} (probably captcha)")

Using cloro’s managed Google AI Mode scraper

Building and maintaining a reliable AI Mode scraper takes real engineering investment.

Infrastructure requirements

AI Mode-specific work:

• Async response interception and parsing
• HTML comment metadata extraction
• Citation pill UUID mapping
• Multi-format output generation
• Session management

Anti-bot evasion:

• Browser fingerprint rotation
• CAPTCHA solving
• Proxy pool management
• Rate limiting and backoff
• Behavioral simulation

Performance:

• Async response handling
• Efficient HTML parsing
• Multi-format conversion pipelines
• Error handling and recovery
• Geographic distribution

Managed solution API

import requests

# Simple API call - no browser management needed
response = requests.post(
    "https://api.cloro.dev/v1/monitor/aimode",
    headers={
        "Authorization": "Bearer sk_live_your_api_key",
        "Content-Type": "application/json"
    },
    json={
        "prompt": "What do you know about Tesla's latest updates?",
        "country": "US",
        "include": {
            "markdown": True
        }
    }
)

result = response.json()
print(f"AI Response: {result['result']['text'][:100]}...")
print(f"Sources: {len(result['result']['sources'])} citations found")
print(f"Markdown: {'Yes' if result['result'].get('markdown') else 'No'}")

Response structure

{
  "success": true,
  "result": {
    "text": "Tesla's recent updates include significant improvements to their Full Self-Driving capability...",
    "sources": [
      {
        "position": 1,
        "label": "Tesla FSD Updates",
        "url": "https://tesla.com/updates/fsd",
        "description": "Latest Full Self-Driving improvements and capabilities"
      }
    ],
    "html": "https://storage.googleapis.com/ai-mode-response.html",
    "markdown": "**Tesla's recent updates** include significant improvements...",
    "searchQueries": ["Tesla updates 2024", "Full Self Driving improvements"]
  }
}

Key benefits

• P50 latency under 8s, versus minutes for a manual run
• No infrastructure to run. We handle browsers, proxies, and async interception.
• Structured data with citation pill parsing and metadata extraction
• Multi-format output: text, markdown, and HTML with citation links intact
• Rate limiting and ethical scraping practices baked in
• Scales to thousands of requests without tripping AI Mode

For most teams, cloro’s AI Mode scraper is the shorter path. You get reliable infrastructure, automatic citation pill parsing, async response handling, CAPTCHA solving, structured JSON, and multi-format output.

Building the same in-house typically runs $5,000-$10,000/month once you account for engineering time, browser instances, proxies, and async response work.

If you need a custom solution, the approach above is a working foundation. Expect ongoing maintenance: Google updates AI Mode response formats and citation systems frequently.

Ready to pull AI Mode data into your stack? Get started with cloro’s API.