Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 5 Next »

The FTP Reader component is a simple example component that connects with a FTP, FTPS, or SFTP server and retrieves files from a target directory for processing. As each file is read, its contents will be pushed downstream for further processing, and the original file will be either deleted or moved to a designated folder. By default, this component polls every 60 seconds.

Running the Component

The below steps assume an existing FTP/FTPS/SFTP server is available to connect to and the necessary user permissions and folder structure are in place.

 STEP 1: Import the File Retriever component

Using +COMPONENT, import the FTP Reader

 STEP 2: Set up the component configurations

Connection Configurations

  • Protocol: specify the protocol you’ll be using (FTP, FTPS, SFTP)

  • Server: the server you’ll be connecting with

  • Port: the port to use

  • Username: the remote user to login as. This account will need permissions to read, delete, and/or move files to the specified SourceDir

  • Password: the password to authenticate with

  • RetryCount: the number of times to retry the connection if it drops. By default, it will retry 10 times.

Workflow Configurations

  • SourceDir: the source directory you’ll be retrieving files from

  • FileExtension: the types of files you wish to retrieve (e.g. txt)

  • DeleteProcessedFiles: set to true if you wish to delete processed files. If false, processed files will be moved to a folder designated by ProcessedFileDir. This is set to false by default.

  • ProcessedFileDir: the directory on the server that files will be moved to. You will need to create this folder if you choose to use this workflow.

 STEP 3: Start the component and view the processed files' contents in the logs

Once the component has been started, it will pull files that match the provided parameters and push their contents to the component’s queue for further processing. Open the component’s logs to see the queued files.

Adapting the Component

There are a few ways you can adapt and expand this component for more complex workflows:

 Adjust the polling frequency

By default, the polling frequency is set to poll every 60 seconds. If files are dropped onto the server infrequently, this can be increased accordingly.

The frequency can be changed in the main function:

image-20240711-133502.png
 Customize the FTP/FTPS/SFTP connection

By default, the FTP/FTPS/SFTP connection is authenticated using username and password authentication and uses the default configurations. In some cases, additional configurations may be desired, such as:

  • Specifying a FTP(S) mode

  • Using public and private key authentication

  • Disabling verify_peer or verify_host

  • Increasing the timeout

  • and more. Full details can be found by accessing the net.ftp.init, net.ftps.init, or net.sftp.init function help in the Translator

These configurations can be customized by modifying the Input parameter table in the main function:

image-20240722-183057.png

 Add more file retrieval conditions

Currently, the component retrieves files based on two conditions:

  • The file is retrievable

  • The file extension matches the provided extension (e.g. txt)

This is typically sufficient for most cases, but if additional conditions are needed, they can be added to the MatchRules function as shown below:

image-20240711-134134.png

  • No labels