AI
Classic browser automation (Selenium, Playwright scripts written by hand) is brittle in a specific way: it encodes assumptions about a page's structure — "the login button has id #submit-btn" — at the moment the script is written, and those assumptions rot as the target site changes. A "self-healing" agent avoids this not through magic, but by never hardcoding the assumption in the first place: at each step, it looks at the current state of the page and decides what to click based on what's there now, not what was there when the script was recorded.
How browser-use structures the healing loop
1. Observe: before each action, the agent captures the current DOM/accessibility tree (and optionally a screenshot) and passes it to the LLM as fresh context — no cached selectors from a previous run.
2. Act and check: the model chooses an action (click, type, scroll) based on that observation. After the action executes, the loop re-observes the resulting page state to confirm the action had the intended effect.
3. Retry with a ceiling: if the expected outcome doesn't appear, the agent can try an alternative approach (a different element, a scroll-then-click) — but only up to a fixed retry limit, after which it should fail loudly rather than loop forever.
# pip install browser-use
import asyncio
from browser_use import Agent
from langchain_openai import ChatOpenAI
async def run_with_retry_ceiling(task: str, max_attempts: int = 3):
for attempt in range(1, max_attempts + 1):
agent = Agent(task=task, llm=ChatOpenAI(model="gpt-4o"))
try:
result = await agent.run()
return result
except Exception as e:
# Re-observes page state fresh on the next attempt
if attempt == max_attempts:
raise| Approach | Behavior on UI change | Maintenance cost |
|---|---|---|
| Hardcoded Selenium/Playwright selectors | Fails immediately with an element-not-found error | High (scripts need manual updates after every redesign) |
| LLM agent re-observing DOM each step | Adapts by re-locating the element based on current state | Low (no selector maintenance, but higher per-run token cost) |
This isn't free — re-observing the page and calling an LLM at every step is slower and costs more per run than a hardcoded script executing instantly. The trade is worth it specifically for tasks that touch frequently-changing UIs (competitor sites, third-party portals you don't control) where the maintenance cost of a brittle script would otherwise fall on you every time the target redesigns.
Practical Challenge
Install browser-use, give it a simple multi-step task on a real site (e.g., "search for a product and read its price"), then intentionally point it at a page after signing out to see how it recovers — or fails to — when the expected element isn't present.
Concept Check
Sources & Further Reading
- browser-use (GitHub) — open-source library that lets LLMs observe and control a real browser step by step, the basis for the pattern described here.
- browser-use Documentation — setup, agent configuration, and the observe-act loop internals.
- Reflexion (Shinn et al., arXiv:2303.11366) — the broader research on verbal self-reflection loops that this retry-with-ceiling pattern draws from.