Firehose Usage Policy & Restrictions

Pre-fetching, Caching or Storage of Content

You can use the imin Firehose to maintain an index of all open data items within your own datastore. Therefore, the imin Firehose is exempt from the general caching restrictions (as stated in Section 3.7 of our standard Terms Of Use). However, Firehose-specific caching restrictions do apply. To remain compliant with the specific caching restrictions of the imin Firehose, you must ensure that:

• Items are not more than 24 hours out-of-date; and
• Items marked as “updated” are completely replaced the previous data for that item in your data store, and any items marked as “deleted” must be hard-deleted from your data store within 24 hours of the “modified” timestamp. This ensures that you remain compliant with both our Terms Of Service and with GDPR.

Caution

To achieve compliance with these restrictions, the Firehose feeds must be frequently harvested, ideally continuously or at least every 24 hours.

Resync limit

When an RPDE feed is harvested from the beginning of time, it is termed a “resync”. Normal consumption of the Firehose should not result in frequent resyncs, since RPDE provides continuous near-real-time data synchronisation if used correctly. Use of the imin Firehose is, therefore, subject to a fair-use limit of 1 resync per week.

Rate limits

Rate limits for the imin Firehose RPDE feeds are outlined as follows:

Live mode (content pages) - no artificial limit

There is no artificially imposed rate limit on the feeds for pages with more than zero items - a client can simply consume them as quickly as they are retrieved.

Sleep mode (last page) - poll a maximum of once every 8 seconds

When a last page (with zero items) is encountered, it signifies that no further updates are currently available in the feed. This is known as “sleep mode”.

Caution

When a zero-item page is encountered, the client MUST wait at least 8 seconds before attempting to poll the same page again for new items.

Termination

Should you cease to be an imin customer for any reason, you must purge all content retrieved from Firehose within 24 hours.

Orphaned scheduled sessions

Whilst the SessionSeries and ScheduledSessions feeds are related, they are updated in their independent RPDE feeds by the data provider. The result of this can be that one feed is updated before the other, which can lead to temporary “orphan” scheduled sessions.

Note

If you are taking a daily pull of the data in the Firehose, because it will be a snapshot of the data at a specific time, you are liable to see some scheduled sessions that are yet to be matched.