Firehose Usage Policy & Restrictions
You can use the imin Firehose to maintain an index of all open data items within your own datastore. Therefore, the imin Firehose is exempt from the general caching restrictions (as stated in Section 3.7 of our standard Terms Of Use). However, Firehose-specific caching restrictions do apply. To remain compliant with the specific caching restrictions of the imin Firehose, you must ensure that:
- Items are not more than 24 hours out-of-date; and
- Items marked as “updated” are completely replaced the previous data for that item in your data store, and any items marked as “deleted” must be hard-deleted from your data store within 24 hours of the “modified” timestamp. This ensures that you remain compliant with both our Terms Of Service and with GDPR.
To achieve compliance with these restrictions, the Firehose feeds must be frequently harvested, ideally continuously or at least every 24 hours.
Because the Firehose is compatible with the OpenActive Modelling Specification and published using the OpenActive Paging Specification (as noted in the Introduction to the Firehose), the work needed to harvest the data into an existing data repository is small. For more information, please watch the video on the OpenActive How an RPDE data feed works page.
When an RPDE feed is harvested from the beginning of time, it is termed a "resync". Normal consumption of the Firehose should not result in frequent resyncs, since RPDE provides continuous near-real-time data synchronisation if used correctly. Use of the imin Firehose is, therefore, subject to a fair-use limit of 1 resync per week.
The Firehose must be continuously polled and must not re-downloaded (“resynced”) from the beginning except for test purposes for which you will have our explicit permission.
There is no artificially imposed rate limit on the feeds for pages with more than zero items - a client can simply consume them as quickly as they are retrieved.
Sleep mode (last page) - poll a maximum of once every 8 seconds
When a last page (with zero items) is encountered, it signifies that no further updates are currently available in the feed. This is known as "sleep mode".
When a zero-item page is encountered, the client MUST wait at least 8 seconds before attempting to poll the same page again for new items.
Should you cease to be an imin customer for any reason, you must purge all content retrieved from Firehose within 24 hours.
Whilst the session-series and scheduled-sessions feeds are related, they are updated in their independent RPDE feeds by the data provider. The result of this can be that one feed is updated before the other, which can lead to temporary "orphan"
ScheduledSessions.If you are taking a daily pull of the data in the Firehose, because it will be a snapshot of the data at a specific time, you are liable to see some scheduled sessions that are yet to be matched.
Last modified 2yr ago