DemandScout

Data sources

Where DemandScout's evidence comes from

Evidence in DemandScout names its source whenever we know it, and links to the original article, filing, notice or job posting when there is one. This page lists the sources we read, what we take from each, and the licence or terms we use it under.

robots.txt

Our crawler, demandscout-leadbot, honours robots.txt when it reads company websites, published cloud IP ranges and news feeds.

One exception: when a news item or a public procurement notice names a company we do not track yet, we check the website we believe is theirs with one request to its homepage. That check follows its redirects (up to five), does not consult robots.txt and does not wait between requests. We may repeat it on a later run or from another source, including for a name we could not match.

In use

Evidence from these sources appears in DemandScout.

Greenhouse, Lever and Ashby job boards

In use

Job titles, locations, dates and links from the public job-board APIs of companies that use these providers.

Public job-board APIs that each provider documents for automated retrieval.

RemoteOK, We Work Remotely and Hacker News

In use

Public job listings: RemoteOK's listings, We Work Remotely's job feeds and Hacker News hiring threads (through the Algolia search API).

Listings these services publish as a public API or feed. We keep the job title, company, date and a link to the listing.

Company websites

In use

The homepage of a company we track: its description and size and location hints, to classify it.

Read with robots.txt honoured, at most one request per second per site from each crawler job. One exception: when a news item or a public procurement notice names a company we do not track yet, we check the website we believe is theirs with one request to its homepage. That check follows its redirects (up to five), does not consult robots.txt and does not wait between requests. We may repeat it on a later run or from another source, including for a name we could not match.

Published cloud IP ranges (AWS, Oracle Cloud, Cloudflare)

In use

The IP ranges each provider publishes, to tell whether a company's domains are cloud-hosted.

Files the providers publish for download, fetched at most once a day with robots.txt honoured.

Being tested (not shown in DemandScout yet)

We read these sources in a trial run.

The GDELT Project

Being tested

Headline, publisher, date and link of news articles that GDELT indexes. Never article text or images.

GDELT's datasets are available for any use, including commercial use, with a citation and a link to The GDELT Project on every use.

We read it to compare it with our current news sources; nothing from it appears in DemandScout until we switch it on.

SEC EDGAR

Being tested

Filing type, item numbers, company, filing date and a link to the filing (8-K current reports).

Public information from the U.S. Securities and Exchange Commission's EDGAR system. DemandScout is not affiliated with or approved by the SEC.

We read it to compare it with our current news sources; nothing from it appears in DemandScout until we switch it on.

TED — EU public procurement notices

Being tested

Award notices: buyer and winner organisations, procurement codes, date and a link to the notice.

Procurement notices © European Union, reused under the TED legal notice.

We read it to compare it with our current news sources; nothing from it appears in DemandScout until we switch it on.

Before 5 October 2026

Until 5 October 2026 some headlines came from Google News search feeds and trade-press RSS feeds. We no longer read those feeds; headlines collected from them before that date remain as stored evidence, labelled Google News or with the publisher's website.

Dated changes to these sources are in the changelog. Questions: [email protected].