demandscout-leadbot
About the crawler behind DemandScout
Who we are
demandscout-leadbot is the crawler of DemandScout, a B2B buying-signal service operated by
Devops Team SRL. It sends the User-Agent demandscout-leadbot/1.0 (+https://demandscout.ai/bot).
Requests to SEC EDGAR carry the contact address the SEC requires in their User-Agent.
What it fetches
- Public job-board APIs (Greenhouse, Lever, Ashby).
- Public job listings from RemoteOK, We Work Remotely and Hacker News (through the Algolia API).
- News feeds whose
robots.txtallows us and whose terms permit our use. - Public registries and datasets (listed on data sources), including those we are still testing.
- Published cloud IP ranges.
- The homepage of companies we track, and of companies a news item or notice names (see below).
Every source, with the licence or terms we use it under and its status, is listed on data sources.
robots.txt
When it reads company websites, published cloud IP ranges and news feeds, it
obeys the group for demandscout-leadbot, or * when there is none. We re-read
robots.txt at least every 24 hours. A robots.txt answered with HTTP 451 is read as
disallow-all.
One exception: when a news item or a public procurement notice names a company we do not track yet, we check the
website we believe is theirs with one request to its homepage. That check follows its redirects (up to five),
does not consult robots.txt and does not wait between requests. We may repeat it on a later run or
from another source, including for a name we could not match.
Rate
Each crawler job sends at most one request per second to a host (up to four per second to SEC EDGAR, whose published limit is ten). Jobs that run at the same time keep that pace separately, and the homepage check described under robots.txt does not wait between requests.
Opting out
Add these two lines to your robots.txt:
User-agent: demandscout-leadbot Disallow: /
To remove your company from our lead database, email [email protected] (see the privacy policy).