HOW WE FETCH
How Worthabid fetches
If you run a tender portal or a business website and have seen Worthabid in your logs, this page explains what it was fetching, how often, and how to reach us or stop it.
USER AGENT, EVERY REQUEST
Mozilla/5.0 (compatible; Worthabid/1.0; +https://www.worthabid.com.au/fetch)
What we fetch
Public tender portals. Listing and notice pages for open tenders, and published award data. We store the notice metadata (title, buyer, category, value, dates) and a link back to the original. We never download tender attachments and never reproduce tender documents.
A customer’s own website. When a business signs up and gives us its website address, we read that site’s public pages once to draft the business’s capability profile. It fetches only the site the customer named, a few dozen pages at most, and stops there.
It does not fetch anything behind a login, does not submit forms, and does not follow links off the site it was asked to read.
How often, and whose data it is
- AusTender approaches to market. The feed hourly, and each new tender's page once. Licence: CC BY 3.0 AU.
- AusTender contract notices. Daily. Licence: CC BY 3.0 AU.
- Tenders WA. Daily, 2:40am Perth time. Licence: Public web pages, read under the portal's Conditions of Use.
- Tenders WA contract award details. Weekly. Licence: CC BY 4.0.
- VendorPanel Marketplace. Hourly. Licence: Public RSS feed.
- Council websites. Daily. Licence: Each council's public-notice page; CC BY 4.0 where the council licenses it (Brisbane).
Every notice we show names its source and links back to the original, with the licence’s attribution where it asks for one. The full list, and the portals we do not read, is on the sources page.
How it behaves
- Obeys robots.txt, including crawl-delay, on every host, with one exception. Tenders WA’s robots.txt disallows all robots; we read its open-notice listing and notice pages once a day under its Conditions of Use, which do not prohibit this. We take the notice facts and a link back, never the documents. The Department of Treasury and Finance can ask us to stop through the contact form and we will.
- At most one request every three seconds per host, one connection at a time.
- Backs off on a 403 or 429 from a tender portal. If the refusals continue, it stops and does not return until we have spoken to the operator. Retries a server error once.
- Keeps no copy of the pages it fetches; only the facts listed above are stored.
- Requests come from a small fixed set of addresses, currently 51.161.202.178.
How to opt out
Add this to your robots.txt and we will stop on our next visit.
User-agent: Worthabid
Disallow: /Or write to the contact form with the host name and we will add it to the exclusion list by hand within two business days and confirm by reply. Either is the place for questions about a specific request you have seen in your logs.
Councils and portals can ask us to stop reading their pages. Email us through the contact form and we will stop within 2 business days.
The full list of portals we read, and which ones we have excluded and why, is on the sources page.