We corrected what our pages said about robots.txt
Our pricing and privacy pages said that DemandScout respects robots.txt. That was not true for news: our news feeds were read without consulting robots.txt, and some of them come from publishers whose robots.txt or terms do not permit our use.
On 14 September 2026 we replaced that claim with a dated statement: by 5 October 2026 we will collect news only from sources whose robots.txt and terms permit automated use, and until then some headlines still come from feeds whose robots.txt or terms do not.
For company websites and news feeds, one exception remains after that date, and our crawler and data sources pages state it: when a news item or a public procurement notice names a company we do not track yet, we check the website we believe is theirs with one request to its homepage. That check follows its redirects (up to five), does not consult robots.txt and does not wait between requests. We may repeat it on a later run or from another source, including for a name we could not match.