The FTP Reader component connects with a FTP, FTPS, or SFTP server and retrieves files from a target directory for processing. These files are downloaded to a local folder and then either deleted from the server or moved to a designated folder on the server. The downloaded files are then read and pushed downstream for further processing in the same way the https://interfaceware.atlassian.net/wiki/x/AQDqsw processes files. By default, this component polls every 60 seconds.
Running the Component
The below steps assume an existing FTP/FTPS/SFTP server is available to connect to and the necessary user permissions and folder structure are in place on the FTP server and locally.
Fields marked with an asterisk * are required.
Connection Configurations:
Field | Description | Default Value |
|---|---|---|
Protocol * | Specifies the protocol you’ll be using. Supports: FTP, FTPS, or SFTP |
|
Server * | The server you will be connecting with | - |
Port * | The port to use | - |
Username * | The remote user to login as. This account will need permissions to read, delete, and/or move files to the specified FTPDirectory | - |
Password | The password to authenticate with. | - |
Certificate | The name of your certificate file, if in use. By default, this should be a PEM certificate. The default configurations can be changed by modifying the net.[protocol].init function directly in the script. See Adapting the Component. This configuration is only applicable for FTP and FTPS. | - |
PrivateKey | The name of your private key file, if in use. By default, this should be a PEM certificate and not be password-locked. The default configurations can be changed by modifying the net.[protocol].init function directly in the script. See Adapting the Component. | - |
RetryCount | The number of times to retry the connection if it drops. Default is set to 3 attempts. |
|
FTP Workflow Configurations:
Field | Description | Default Value |
|---|---|---|
FTPDirectory * | The source directory on the FTP server you’ll be retrieving files from |
|
FileType
| The types of files you wish to retrieve (e.g. txt). You can also specify multiple files (e.g. txt,hl7) or all file types (e.g. *). | - |
MoveFTPProcessedFiles | Set to true if you wish to move processed files (i.e. files that have already been downloaded to the local server) to a folder on the FTP server designated by FTPProcessedDirectory. If false, processed files will be deleted. |
|
FTPProcessedDirectory | The directory on the FTP server that processed files will be moved into. You will need to create this folder if you choose to use this workflow. | - |
Local File Workflow Configurations:
Field | Description | Default Value |
|---|---|---|
LocalDirectory * | The local directory that the files will be downloaded to from the FTP server. Files in this directory will then be picked up for processing. | - |
MoveProcessedFiles | Set to true if you wish to move processed files to a local folder designated by ProcessedDirectory. If false, processed files will be deleted. |
|
ProcessedDirectory | The local directory that processed files will be moved into. You will need to create this folder if you choose to use this workflow. | - |
MinimumFileAge * | The time to wait in seconds after last modified date before processing. |
|
IgnoreSegments * | Specifies whether to remove identified segments when processing file. Default segments to ignore include: FHS, BHS, FTS, BTS. This can be modified in config.lua. See Adapting the Component. |
|
InputFileEncoding * | The encoding used in the output file. If left blank, encoding will be updated to the default according to OS. See supported encodings below. |
|
Adapting the Component
There are a few ways you can adapt and expand this component for more complex workflows:
By default, the polling frequency is set to poll every 60 seconds. If files are dropped onto the server infrequently, this can be increased accordingly.
The frequency can be changed in the main function:
By default, the FTP/FTPS/SFTP connection is authenticated using username and password authentication and uses the default configurations. In some cases, additional configurations may be desired, such as:
Specifying a FTP(S) mode
Using public and private key authentication
Disabling verify_peer or verify_host
Increasing the timeout
and more. Full details can be found by accessing the net.ftp.init, net.ftps.init, or net.sftp.init function help in the Translator
These configurations can be customized by modifying the Input parameter table in the loadConfigs() function in FTP/FTPprocess.lua:
Currently, the component retrieves files from the FTP server based on two conditions:
The file is retrievable
The file extension matches the provided extension (e.g. txt)
This is typically sufficient for most cases, but if additional conditions are needed, they can be added to the FTPmatchRules function in FTP/FTPmatchRules.lua:
