Getting Started
This guide will help you along with using the importers!
Description
The data agent is a service within the backstage ecosystem designed to deals with importing & exporting data from external systems. Most of the time customers will have an existing OVP or other feeds (like MRSS) in which their metadata is available.
How it works
Each importer job runs on a predefined schedule making use of laravel's queue system. Jobs can also be triggered and executed outside the queue which is useful for one-off/development tasks.
Importer Configuration and Sources
The data agent makes use of Sources; which are object(s) stored within a service that describe where data should be ingested from, how often data should be ingested and how imported data should be output. The data-agent runs a heartbeat job every 5 minutes that checks the sources in each service and schedules any imports accordingly. This is done by checking the last run and next run timestamps saved within the object. Sources can be created, edited and deleted via the CMS and a service can have multiple sources. Assets ingested via that source will retain a reference to it which allows the system to treat assets differently depending on where they were ingested from, for example 1 source could be added to the system with no stream signing required, and another requiring it could be added and only the protected assets would have their streams signed. This then allows situations where you could be ingesting from 2 different OVPs and still sign FAPI stream requests with the correct handling.
The source of truth setting will dictate how data can be edited, and how it will be updated from it's source. Backstage so far offers 3 options of sources of truth, when configuring your importer:
Importer- this option allows the importer to create and update assetsImporter create only- this option allows the importer to just create assets, never update themImporter update only origin- allows create and update unless backstage changes have been made
Reporting
The data agent creates a new ImportResult object within the corresponding service on completion of an import. This result object references the source that created it so the results can be easily collated for viewing in the front end. Successful, skipped and failed assets are included in the report with corresponding errors and object data where applicable. This information can be viewed in the CMS to help with bugs or the identification of bad data. Exceptions dealt with throughout an import are also caught and collected as to not block the rest of the import. These are then reported to bugsnag, so we get some better visibility on issues running within the jobs. This logic could probably be improved further to better show error details.
Additional Functions
Exporting
The data-agent has the ability to export data from a backstage service, to a defined location in a defined format. Backstage currently supports export to s3 locations, whether these are 24i's bucket or external.