Break down the different parts of an integration
Every integration
...
can be broken down into three parts:
Source
...
Two types of sources
Polling
File system (local, sftp, ftp, etc.)
Database
API endpoint
Listening (i.e. event triggered)
LLP feed
API requests
Filter (we should rebrand this to something better) - Processor?
...
Actual filtering
Mapping
...
Processor
Destination
Source
Responsible for gathering data from the source system. There are two broad categories for how data can be gathered:
Polling interfaces are set to run at regular intervals and query the source for data to be processed. Some examples:
Polling a source directory (local, SFTP, etc.) for files to be processed
Querying a database for records to be processed
Making API queries
Listening interfaces are event triggered and listen for data sent by a source system. Some examples:
LLP listeners receive data through LLP from an LLP sender
Hosting an API endpoint that receives HTTP requests
Processor
Once data is received, there is often some kind of processing that needs to be done in order for the data to be compatible with downstream systems. Some examples can include:
Filtering unwanted messages
Mapping from one field to another, or more generally, converting from one message type to another
To a lookup
Transformation
Date formatsTransforming data (e.g. data formatting, string manipulation,
removing special characterscode mapping, database lookup, etc.)
Destination
...
LLP listener
...
API endpoint
...
Database
...
With the data in the correct format, all that is left is to send the data downstream to the destination. Some examples of how this can be done:
Sending through TCP (i.e. LLP, HTTP, etc.)
Inserting into a database
Dropping a file with the data in a file system (local, SFTP, etc.)