Skip to main content

​
About Databricks

Databricks is a data and AI platform built on Apache Spark. The Serval integration connects to a single Databricks workspace using a service principal’s OAuth client credentials (the Databricks machine-to-machine flow). Serval exchanges those credentials for an account-level access token using Databricks’ all-apis scope, which workflows use to call both the Databricks Workspace APIs (clusters, jobs, Unity Catalog, MLflow, SQL, secrets, and more) and the Databricks Account APIs (workspaces, metastores, networking, budgets, and more). Each connection is registered per workspace - the workspace URL identifies the connection, and the display name defaults to the workspace URL unless you provide an Instance Name. Authentication: OAuth 2.0 client credentials (Databricks service principal, machine-to-machine flow) Data sync: On-demand only - there is no background sync or entity ingestion. Serval calls Databricks only when a workflow step runs, exchanging your stored credentials for a fresh token on each request.

​
What the Databricks integration enables

Anything defined in the Databricks Workspace API or Databricks Account API can be accessed through Serval.

​
Get your credentials

Serval needs four things: your Databricks Account ID, the Workspace URL of the workspace to connect, and a service principal’s Client ID and Client Secret. You create the service principal and its OAuth secret in the Databricks Account Console - see the official Databricks OAuth machine-to-machine (M2M) authentication guide.
1

Log in to the Databricks Account Console

Use the console for your cloud: accounts.cloud.databricks.com for AWS, accounts.azuredatabricks.net for Azure, or accounts.gcp.databricks.com for GCP.
2

Create a service principal

Go to User management, open the Service principals tab, click Add service principal, name it (e.g. “Serval Integration”), and click Add.
3

Generate an OAuth secret

Open the service principal, select the Secrets tab, and under OAuth secrets click Generate secret. Set the lifetime (maximum 730 days). Copy both the Secret and the Client ID immediately.
4

Assign the service principal to your workspace

In the Account Console, go to Workspaces, select the target workspace, open the Permissions tab, click Add permissions, select the service principal, and assign a role (e.g. User or Admin).
5

Find your Account ID

In the Account Console, click your email in the top-right corner (the user menu). The Account ID is a UUID like xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx.
The OAuth secret is shown only once, at generation time. Copy it before closing the dialog - if you lose it, you’ll need to generate a new one.
Workspace assignment and account-level access are granted separately in Databricks. The service principal needs both: workspace assignment for Workspace API calls, and account-level access for Account API calls. Serval’s health checks test each tier independently.

​
Connect in Serval

Every field except Instance Name is required - leaving a required field blank stops submission with “This field is required”.
1

Enter your Account ID

Type the bare account ID - the UUID from the Account Console user menu, matching the placeholder “xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx”. The field only accepts letters, numbers, and hyphens, so anything URL-shaped (containing dots or slashes) is rejected with “Enter your Account ID, not a workspace URL. The Account ID is found in Account Console > User menu.”
2

Enter your Workspace URL

For example “your-workspace.cloud.databricks.com”. You can paste it with or without “https://” - Serval strips the prefix and any trailing slash on save. This URL also tells Serval which cloud’s Account API to use (AWS, Azure, or GCP).
3

Enter the Client ID

The Client ID (Application ID) of your Databricks service principal.
4

Enter the Client Secret

The OAuth secret you generated in Account Console under Service principals.
5

Optionally set an Instance Name

The field is labeled Instance Name (Optional) - a friendly name to identify this Databricks workspace, e.g. “Production Workspace”. If left blank, the connection is named after the workspace URL.
When editing an existing connection, any field you clear or leave blank keeps its stored value - so to rotate the Client Secret, you only need to fill in the new secret. The settings page shows the stored secret masked (bullets with the last 4 characters visible); the Client ID is shown in full.

​
Verifying the connection

Two built-in health checks run against the connection, each testing a different access tier:
  • List Databricks Workspaces - lists the workspaces in your Databricks account via the Account API, verifying that the token exchange works and the service principal has account-level access. On success it reports: “Successfully listed workspaces from Databricks account (count: [number])”. On failure: “Unable to list workspaces from Databricks account. Please verify the service principal has account-level access.”
  • List Databricks Clusters - lists the clusters in the connected workspace via the Workspace API, verifying the service principal has workspace-level access. On success it reports: “Successfully listed clusters from Databricks workspace (count: [number])”. On failure: “Unable to list clusters from Databricks workspace. Please verify the service principal has workspace-level access.”
The pass/fail pattern tells you which grant is missing. If List Databricks Workspaces passes but List Databricks Clusters fails, the service principal hasn’t been assigned to the target workspace (Account Console, Workspaces, Permissions tab). If clusters pass but workspaces fail, the service principal lacks account-level access.

​
Gotchas and troubleshooting

The connect form only accepts an account ID made of letters, numbers, and hyphens (such as the UUID from the Account Console). A value containing dots or slashes - like a workspace URL - fails validation with “Enter your Account ID, not a workspace URL. The Account ID is found in Account Console > User menu.” Find the ID in the Account Console by clicking your email in the top-right corner.
Serval requests a single account-level token, but Databricks grants account-level and workspace-level access separately. A service principal can pass one health check and fail the other - use the pattern above to identify the missing grant. Note also that each connection is scoped to one workspace: connect each workspace separately.
Serval derives the accounts host from the Workspace URL: Azure workspace URLs route to accounts.azuredatabricks.net, GCP workspace URLs to accounts.gcp.databricks.com, and everything else falls back to accounts.cloud.databricks.com (AWS). An incorrect workspace URL can route the token exchange and Account API calls to the wrong cloud’s control plane - double-check the URL if both health checks fail.
On save, trailing slashes and the “https://” or “http://” prefix are stripped, and the normalized URL becomes the connection’s instance ID and its default display name. Changing the Workspace URL when editing the connection changes the connection’s identity, not just a setting.
The update path keeps the stored value for every field you leave empty - Account ID, Workspace URL, Client ID, Client Secret, and Instance Name - so you only fill in what you’re rotating. One caveat for direct API callers: the settings read returns the Client Secret masked (bullets plus the last 4 characters), and any non-empty submitted value is stored verbatim. Never send the masked value back as the new secret, or it will be saved literally and break authentication. The standard Serval form handles this for you.
Databricks displays the service principal’s OAuth secret a single time at generation, with a maximum lifetime of 730 days. If it’s lost or expires, generate a new secret in the Account Console and update the connection’s Client Secret field - all other fields can stay blank.
If a workflow lists resources, be aware that different Databricks APIs page differently: SCIM user and group lists use startIndex/count and return totalResults, newer lists like clusters and jobs use page_token with page_size or limit and return next_page_token, and some account lists (like workspaces) return a single unpaginated array.
Need help? Contact support@serval.com for assistance with your Databricks integration.