The fetch: what Eiger Vault reads at the publisher
Technical page, for the team that operates the publisher's site or CMS.
A fetch is a read that Eiger Vault performs, at the source the publisher chooses: its website (feed, pages), or the API of its CMS, which returns data that is already structured. Nothing is installed at the publisher; at most, it hands us an access credential for its API. This page says what we read, at what pace, how we identify ourselves, and how the publisher can limit or cut us off.
The rules below are those of the fetch connector that we operate today, which reads websites and public APIs. They apply to each publisher, subject to the adjustments that this publisher and we have agreed, in particular for reading the API of a CMS.
1. What we read
A trigger, then the article. We first read what announces the articles: a feed (RSS or Atom), or a public call without a secret. This announcement is only a trigger: none of its content is kept. Each announced article is then read at its own address.
The article: three ways to read it. When the site offers the article without a secret through a programming interface, for example the WordPress REST API, we read the article through it. When it offers only the page, we read the HTML page. When the publisher gives us access to the API of its CMS, we read data that is already structured – text split into blocks, complete metadata –, which avoids interpreting a page; access is made with a credential that the publisher hands us, and which travels only in a request header. For each source, our configuration says which of these ways we use; the fields read from a CMS API are agreed with the publisher.
The article identifier. We read the identifier that the CMS gives to the article wherever it appears – in the feed, in the backlog list, in the article as read –, and we require that these reads give the same string: if they differ, the fetch stops and nothing is deposited. We derive an article's reference from its address only if the CMS has no stable identifier, and the site's configuration then says so expressly.
The backlog. To find the articles already published before our first read, we walk through a backlog list that the site offers without a secret – a sitemap or the paginated list of an API –, from the most recent to the oldest. We read only the references there, without opening the articles. We stop at a floor date agreed for the source, or as soon as an entire batch of references is already known to us beyond a minimum window of the last 48 hours.
What we deposit. The bytes of the response as received, after decoding any compression, are kept as the original envelope. From them we extract a structured text and a set of metadata (headline, standfirst, authors, dates, category, canonical address, language), copied without completing anything: a missing date stays missing. We deposit no image or video read from a page, because they often come from agencies, under rights other than those of the text. If the article unambiguously designates the original of which it is the translation, we declare it as a relation; otherwise, no relation is declared.
The proof of the fetch. Each fetched article is deposited with a proof: the final address read after redirects, the instant of the read, the digest of the bytes received, and the TLS certificate chain that your server presented. Our vault checks that the address is https, that the chain leads back to a recognised certificate authority, that the certificate covers the server name, and that it was valid at the instant of the read. This proof attests control of the domain name at that instant, nothing more: it is not a signature by the publisher.
2. At what pace
These values are starting values, adjusted after measurement.
| What | Pace |
|---|---|
| Reading each source's feed | every 15 minutes |
| Walking the backlog | at connector start-up, then every 24 hours; never two passes at once for a source |
| Minimum backlog window | the most recent 48 hours are always re-read |
| Rate towards a single host | at most 1 request per second, and less if the Crawl-delay in your robots.txt asks for a longer interval |
Reading your robots.txt |
kept for at most 24 hours; if unreachable, retried after 15 minutes |
| Number of reads of an article | at most 3 per pass, after a deposit that failed |
When a source announces an embargo, we will fetch the article only after it is lifted, at its address. This rule is not yet in service.
3. How we identify ourselves
Each request carries a User-Agent header that names the connector and gives a contact address. That of our trial environment has the form EigerVaultTrialConnector/1 (+https://eigervault.ch; contact@eigervault.ch); a publisher who authorises a fetch will receive from us the exact identifier to recognise.
We send only four headers: User-Agent, Accept, Accept-Encoding: identity, and, if the publisher has given us one for a source, a single access header carrying a credential that the publisher has handed us. We send no cookie and keep none from one request to the next.
We read only over https. We remove from an address what is added to it as parameters (campaign tracking, session identifiers), except the parameters that the source's configuration declares necessary. A redirect is followed only towards https and towards a domain that the publisher has admitted for the source, within a limit on redirects; a redirect to an address that adds an undeclared parameter stops the read. We refuse to open a connection to an IP address that is not public. The duration of each request and the size of the response are bounded.
4. How the publisher limits us, or cuts us off
Through robots.txt, for a website. We read it on each host before any other request to that host, and we apply the group of rules that names us (or, failing that, the general group), according to RFC 9309. A prohibition applies to the feed, the backlog list and the articles. If the file responds with a 4xx error, we treat it as absent and read freely; if it responds with a 5xx error, or is not reachable, we read nothing until it responds. A Crawl-delay lengthens the interval between two requests. A Disallow in our name is the simplest way to cut us off, without warning us.
Through HTTP codes. Only a final response with status 200, received in full, is deposited. Any other response – 403, 404, 429, 5xx, truncated response, unknown compression, missing media type – stops the read of that article without depositing anything. We retry on the next pass, three times at most per article and per pass. After ten consecutive deposit failures, the pass for the source stops; this concerns our own deposit failures, not those of your site.
Through a CMS API. The publisher cuts us off by revoking the access credential it handed us, or by limiting the rate of its API: any response other than a complete success stops the read without depositing anything, as above.
By writing to us. The publisher can ask Eiger Vault to stop the fetch: the operator deactivates the source, or revokes the connector key; any later deposit is refused. A deactivation acts only on the future: what is already sealed does not change. To withdraw content already received, see Eiger Vault for publishers: overview, section 3.
Through the network. We fetch only the domains that the publisher has admitted for the source. We do not, to date, publish a range of source IP addresses.