How to build a web scraping AI agent using LangChain and Python

Most "AI web scraping agent" demos are really a small, boring Python function — fetch a URL, parse the HTML, pull out the text you want — wrapped so a tool-calling LLM can invoke it on demand. The AI's job isn't to parse HTML (it's bad at that, and it's slow and expensive); its job is to decide which page to look at, what to extract, and how to turn scattered fragments into a coherent answer. Get that division of labor right and the rest is standard Python.

Building the scraping tool

1. Write a plain fetch-and-parse function first. Use requests.get(url, timeout=10) to fetch the page and BeautifulSoup(html, "html.parser") to parse it. Test this function on its own, outside of any agent — if it can't reliably pull the data you want from a page, wrapping it in LangChain won't fix that.

2. Wrap it as a LangChain tool. LangChain's @tool decorator turns any typed, docstringed Python function into something a model can call — the docstring is what the model reads to decide when the tool is relevant, so write it like documentation, not a comment.

3. Bind the tool to a chat model and let it loop. LangChain's bind_tools() plus an agent executor (or a small LangGraph loop) handles the call-tool → read-result → decide-next-step cycle so you don't have to hand-write the control flow.

# pip install langchain requests beautifulsoup4
import requests
from bs4 import BeautifulSoup
from langchain_core.tools import tool

@tool
def fetch_page_text(url: str) -> str:
    """Fetch a public webpage and return its visible text.
    Use this when you need the current content of a specific URL."""
    resp = requests.get(url, timeout=10, headers={"User-Agent": "research-bot/1.0"})
    resp.raise_for_status()
    soup = BeautifulSoup(resp.text, "html.parser")
    for tag in soup(["script", "style"]):
        tag.decompose()
    return soup.get_text(separator=" ", strip=True)[:8000]

The pattern generalizes past scraping: whenever a task has a deterministic sub-step (parsing HTML, calling an API, running a calculation), do it in code and hand the LLM only the distilled result. Reserve the model's judgment for the parts that actually require judgment — picking which link to follow next, deciding whether the extracted data answers the question, or summarizing across several pages.