Skip to content
Autowright

Scheduled Scraper

Autowright is built for exactly this: a scraper on a schedule, running unattended, against a site that will eventually change. The scraper keeps its schedule, keeps failing honestly when broken, and a fix PR shows up for you to review — no midnight debugging.

The script

Nothing schedule-specific is needed in the code — the same script from Basic Scraper works as-is:

rates-scraper.ts
import { chromium } from '@autowright/browser';


const browser = await chromium.launch({
  headless: true,
  autowright: {
    scriptId: 'rates-scraper',            // one stable ID across every run
    serviceUrl: 'http://localhost:4400',
    repoUrl: 'https://github.com/your-org/scrapers',
    scriptPath: 'src/rates-scraper.ts',
  },
});


const page = await browser.newPage();
await page.goto('https://lender.example.com/rates');
const rows = await page.locator('#rates-table tbody tr').all();
console.log(`captured ${rows.length} rates`);
await browser.close();

Schedule it

With cron:

Terminal window
# every 15 minutes
*/15 * * * * cd /opt/scrapers && npx tsx src/rates-scraper.ts >> /var/log/rates.log 2>&1

Or with GitHub Actions:

.github/workflows/rates.yml
on:
  schedule:
    - cron: '*/15 * * * *'
jobs:
  scrape:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: npm ci && npx playwright install chromium
      - run: npx tsx src/rates-scraper.ts

The only requirement: serviceUrl must be reachable from wherever the schedule runs.

Why dedup matters here

At every-15-minutes, a broken selector fails 96 times a day. Without dedup that would be 96 fix jobs and 96 notifications. Autowright hashes each failure (script ID + classification + failed step + normalized message), so the same incident produces one fix job and one notification — repeat failures within the dedupe window (default 24h) are counted, not re-queued. See Fix Lifecycle.

The healing loop

  1. 03:00 — site renames #rates-table. The run fails, evidence is captured, one fix job is queued. Cron keeps firing; runs keep failing honestly (non-zero exit, real errors in your logs — nothing is masked).
  2. 03:02 — the fixer opens a PR: fix: update rates table selector. Slack pings you with the diff and the cost.
  3. Morning — you review the PR against the live site and merge.
  4. Next run — green again. The scraper healed without a single manual reproduction step.

Next