File Retriever

The File Retriever component is a simple example component that connects with a FTP, FTPS, or SFTP server and retrieves files from a target directory for processing. As each file is read, its contents will be pushed downstream for further processing, and the original file will be either deleted or moved to a designated folder. By default, this component polls every 60 seconds.

Running the Component

The below steps assume an existing FTP/FTPS/SFTP server is available to connect to and the necessary user permissions and folder structure are in place.

STEP 1: Import the File Retriever component

Using +COMPONENT, import the File Retriever

STEP 2: Set up the component configurations

Connection Configurations

Protocol: specify the protocol you’ll be using (FTP, FTPS, SFTP)
Server: the server you’ll be connecting with
Port: the port to use
Username: the remote user to login as. This account will need permissions to read, delete, and/or move files to the specified SourceDir
Password: the password to authenticate with
RetryCount: the number of times to retry the connection if it drops. By default, it will retry 10 times.

Workflow Configurations

SourceDir: the source directory you’ll be retrieving files from
FileExtension: the types of files you wish to retrieve (e.g. txt)
DeleteProcessedFiles: set to true if you wish to delete processed files. If false, processed files will be moved to a folder designated by ProcessedFileDir. This is set to false by default.
ProcessedFileDir: the directory on the server that files will be moved to. You will need to create this folder if you choose to use this workflow.

STEP 3: Start the component and view the processed files' contents in the logs

Once the component has been started, it will pull files that match the provided parameters and push their contents to the component’s queue for further processing. Open the component’s logs to see the queued files.

Adapting the Component

There are a few ways you can adapt and expand this component for more complex workflows:

Adjust the polling frequency

By default, the polling frequency is set to poll every 60 seconds. If files are dropped onto the server infrequently, this can be increased accordingly.

The frequency can be changed in the main function:

Add more file retrieval conditions

Currently, the component retrieves files based on the these conditions:

The file is retrievable
The file extension matches the provided extension (e.g. txt)

This is typically sufficient for most cases, but if additional conditions are needed, they can be added to the MatchRules function as shown below: