/
Databricks Adapter

Databricks Adapter

    Overview

    The Databricks adapter is designed to provide a template for getting started with Databricks API integrations. It leverages the DATABRICKS Library to authenticate with the Databricks Server and provides a few default example workflows.

    This component can be customized and adapted according to your workflow needs. Additional operations can be easily added to the adapter and library.

    Requirements

    • The base URL of the Databricks Server.

      • E.g., https://<identifier>.cloud.databricks.com/

    • The client ID of your registered service principal (i.e., the user)

    • The client secret associated with the service principal.

    Running the Component

    Using +COMPONENT, import the Databricks Adapter component.
    Bolded fields are required, otherwise default values defined in config.json are used: Field Description Default Value Base URL The Databricks workspace URL for API requests should be the base host without the trailing /api path. E.g., https://dbc-a1b2345c-d6e7.cloud.databricks.com or https://adb-1234567890123456.7.azuredatabricks.net   Client ID The OAuth client ID for the Databricks service principal authenticates API requests and identifies the service principal.   Client Secret The OAuth client secret for the Databricks service principal. Used with the Client ID to request an access token.  
    One call retrieves the timestamp, the other attempts to run a job.

    Adapting the Component

    By default the underlying library includes operations for:

    • Retrieving SQL warehouses.

    • Performing SQL statements.

    • Running notebook jobs.

    You can add additional functionality by modifying the library and adding additional .lua modules to the methods folder. You can follow the same format in the DATABRICKSrunSQL.lua module then simply add your new module to the metatable at the top of the DATABRICKSclient.lua module.