Building a Browser Automation Agent

Choose a suitable browser task

Browser automation is useful for a stable, authorized workflow that lacks an appropriate API, such as copying approved data into an internal legacy form. It is fragile when interfaces change and risky when pages mix trusted controls with arbitrary content. Define the allowed sites, account, start state, completion evidence, and actions that are never permitted.

Check terms, organizational policy, and access authorization before automating a third-party service. Do not use browser control to bypass access controls, usage limits, or consent. Prefer a direct integration when it offers clearer schemas, permissions, idempotency, and error handling. The visual interface should not become an excuse to hide an unsupported system dependency.

Observe structured page state

Use accessibility roles, labels, stable attributes, and verified URLs where possible instead of relying only on screen coordinates. Capture enough state to distinguish similarly named controls and to detect a changed page. The agent should know whether it is viewing, editing, previewing, or confirming, and application code should reject navigation outside the allowlist.

Page text, search results, uploaded documents, and messages are untrusted. They may instruct an automated visitor to reveal secrets or ignore policy. Keep browser content inside a data boundary. Credentials should be supplied through a protected session mechanism, not pasted into model context, page fields selected by generated text, or diagnostic screenshots.

Control side effects

Separate preparation from submission. The agent can fill a draft form using validated source data, then show a preview containing destination, fields, attachments, and consequence. A person approves the exact payload for consequential writes. If the page changes after approval, stop and create a new preview instead of continuing from stale consent.

After clicking, inspect the actual confirmation state and capture a permitted reference identifier. Use task-level idempotency because browser retries can duplicate submissions even when the interface does not expose an idempotency key. On ambiguity, stop for review. Do not click through warnings, captchas, permission prompts, or unexpected downloads as if they were ordinary steps.

Test for interface and security failures

Test changed labels, delayed elements, validation errors, session expiration, modal dialogs, duplicate controls, and partial submission. Include hostile page content that asks the agent to navigate elsewhere or expose information. Verify that allowlists, download rules, upload limits, and approval checks remain effective independently of model instructions.

Monitor task completion, wrong-action rate, human interventions, unexpected domains, repeated steps, and interface-change failures. Keep screenshots or traces only under an appropriate privacy and retention policy. Browser and agent platform capabilities change, so consult current official documentation quarterly and rerun tests against supported interface versions.

Enter approved supplier data into a legacy portal

A procurement team has reviewed supplier records in its internal system, but an old partner portal accepts updates only through a web form.

1. Allowlist the exact portal origin and update path, use a dedicated least-privilege account, and prohibit navigation to billing or user administration.
2. Load approved fields from the internal record and map them to labeled form controls while treating all portal text as untrusted.
3. Stop at preview and show supplier identity, changed fields, attachments, destination URL, and any portal warnings to the procurement reviewer.
4. After version-bound approval, submit once, verify the portal confirmation and displayed values, and save the reference identifier.
5. On session expiry, changed fields, unexpected redirect, or uncertain confirmation, stop without resubmitting and hand the trace to an operator.

Browser task boundary sheet

Define the browser agent's permitted environment before it opens a page.

• Navigation: allowed origins and paths, redirect policy, start state, and forbidden areas.
• Session: account, least privilege, credential handling, timeout, and logout behavior.
• Interaction: stable locators, input source, uploads, downloads, warnings, and unexpected UI.
• Submission: exact preview, reviewer, protected payload, expiration, idempotency, and confirmation evidence.
• Recovery: trace, screenshot policy, retry boundary, operator handoff, and interface-change owner.

Common mistakes

• Using broad browsing access for a task that needs one known form on one authorized domain.
• Following instructions displayed on a page as though they can override system permissions or approval policy.
• Assuming a click succeeded and retrying submission without checking confirmation or duplicate effects.

Try one

Sources

• OpenAI tools guideOfficial overview of currently supported tools and tool-use patterns.
• OWASP Top 10 for Large Language Model ApplicationsOfficial OWASP project covering major security risks in applications that use language models.