Gavroche
Explore Gavroche
Find work
Search jobsCountries
Career advice
GuidesSalary negotiationCV help
For employers
Contact usFor website owners
About GavrocheGet in touch
For website owners

gavroche-direct

The crawler behind Gavroche, a search engine for job vacancies.

Gavroche collects publicly posted job vacancies so that people can search them in one place. Every vacancy links back to the employer's own posting, and applications always happen there.

What the crawler does

  • It identifies itself with the user agent gavroche-direct/0.1 (+https://gavroche.io/crawler; mailto:admin@gavroche.io). Its robots.txt token is gavroche-direct.
  • It reads public job listings only: the public job-board APIs and feeds of applicant tracking systems, and public career pages: a listing page and the job pages it links to, read for their schema.org JobPosting data. It never logs in, submits forms or applies to jobs.
  • It obeys robots.txt, re-reading it at least once a day, and honours Crawl-delay. A group naming gavroche-direct replaces the * group for us. If your robots.txt cannot be read (any answer other than a file or a 404/410, a timeout, or an HTML page), we treat the whole host as closed.
  • On a company's own site it sends at most one request every 5 seconds and at most 2,000 requests a day. On large job-board platforms it sends at most one request per second over one connection, slows down as soon as responses slow down or fail, and sends at most 50,000 requests a day. Across all customers of one platform it sends at most two requests per second.
  • A 403, a 429 or a bot challenge stops us: we do not retry, and we leave the host alone for 7 days. After three such refusals we stop crawling that source.
  • If your robots.txt disallows ClaudeBot or anthropic-ai, or your responses carry Content-Signal: ai-train=no or ai-input=no, nothing we fetch from your host is ever sent to an AI model. Your vacancy text is shown as you wrote it.
  • A board is typically revisited every few hours, to notice when vacancies close.

How to slow us down or exclude us

Use the gavroche-direct user agent in your robots.txt. To slow down our crawl:

User-agent: gavroche-direct
Crawl-delay: 10

To exclude a section, or your whole site:

User-agent: gavroche-direct
Disallow: /careers/internal/

User-agent: gavroche-direct
Disallow: /

Changes take effect at our next robots.txt check, within 24 hours. Vacancies we already hold from an excluded source stop being updated; write to us if you want them taken down.