Browser agents

How to automate a website that has no API

The short answer

First look for a back door: an existing integration, an email the site already sends, or an export. If there is none, use a recorder or a browser agent that keeps you signed in and asks before it submits anything.

4 min read

Which way in does your site offer?

Is the site in an automation catalog?
Automation

Use the existing integration

Even if the site's own docs mention no API, a catalog entry means someone built the connection. Search viaSocket's catalog of 2,300+ apps by the site's name.

Does it email you, or let you export?
Automation

Automate from the email or the export

A supplier emails each invoice; an automation reads that email instead of the portal. Far steadier than clicking.

Does the page layout change often?

Use a recorder

Record the clicks once and replay them; Chrome's Recorder can edit each step. Same result every run, until the site changes.

Browser agent

Use a browser agent

It reads the page and looks for what you described, so small layout changes rarely break it. It needs approvals for anything it submits.

Recorder or browser agent?

ComparedRecorder or scriptBrowser agent
Finds things byThe exact buttons it recordedReading the page
When the site changesBreaks at that stepOften copes, sometimes guesses wrong
Same result every runYesMostly
Best forStable tools, many runs a dayA few pages, shifting layouts

Either way, the site's rules apply. Check its terms, keep to what a person would do by hand, and stop at CAPTCHAs: they are the site saying no.

How to automate a website with no API

  1. Step 1: Write the job as a person does it

    "Sign in, open Reports, choose last month, download the CSV." Every step, every click. If you cannot write it, a tool cannot do it.

  2. Step 2: Split reading from doing

    Reading a page is low risk. Submitting a form, placing an order or changing a setting is not. Plan for every "doing" step to ask you first.

  3. Step 3: Sort out the sign-in

    Sign in yourself once in the tool's browser and let it keep the session. Expect to sign in again when the site logs you out.

    Watch out. Never paste a password into a chat or a shared sheet. A tool that needs your password typed in for it is the wrong tool.

    Do this step in 50agents

    Set up a browser agent that reads last week's invoices on my supplier portal every Monday and tells me the total

    Build it

    Captain sets up a browser agent with a browser of its own. When the portal wants a sign-in, it asks you to sign in there yourself, then carries on.

  4. Step 4: Run it once while you watch

    Most tools show the browser live or record what it did. Watch the first run end to end.

  5. Step 5: Add a check that it worked

    A file downloaded, a confirmation number on the page, a row count. Without a check, a failed run looks like a quiet one.

  6. Step 6: Schedule it no more than needed

    Daily or weekly is enough for most portals, and keeps you well inside what the site expects from a person.

What can go wrong

  • The site changed and the script broke

    Recorded clicks point at a button that moved. Keep scripts short, and prefer a browser agent for pages that change often.

  • Signed out mid-run

    Sessions expire and two-step sign-in comes back. Expect a sign-in request every few weeks.

  • A submit went through twice

    A retry repeated the last step. Have every submit ask first, and check for the confirmation before trying again.

  • Blocked by the site

    Too many visits, or the site does not want automation. Slow down; if it shows a CAPTCHA, stop.

Questions people ask

Can I automate a website without an API?

Yes, by driving a browser: a recorder replays clicks, a browser agent reads the page and decides where to click. Check first for an export, an email notification or an existing integration, which are more reliable.

Is this the same as RPA?

Close. RPA (robotic process automation) is the older name for software that clicks through screens like a person. Browser agents add AI to read pages that change.

Will it break when the site changes?

Recorded scripts often do. Browser agents cope with small changes and can still misread a redesigned page, so keep a check at the end of each run.

Is it allowed to automate a website?

Check the site's terms first. Keep to what a person would do by hand, at a pace a person would, and stop at CAPTCHAs: a CAPTCHA is the site saying no.

Can it download files for me?

Yes. "Sign in, open Reports, choose last month, download the CSV" is a typical job. Make the downloaded file the run's check, so a failed run never looks like a quiet one.

How often should it run?

Daily or weekly is enough for most portals. It keeps the job well inside what the site expects from a person, and fewer runs mean fewer sign-ins to fix.

Read next

Or let Captain build it

One sentence to Captain. Free to start.

Build it with Captain
How to automate a website that has no APIBuild it