> ## Documentation Index
> Fetch the complete documentation index at: https://docs.serval.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Databricks

> Databricks is a data and AI platform built on Apache Spark - connect a workspace to Serval so workflows can call the Databricks Workspace and Account APIs on demand.

## About Databricks

Databricks is a data and AI platform built on Apache Spark. The Serval integration connects to a single Databricks workspace using a service principal's OAuth client credentials (the Databricks machine-to-machine flow). Serval exchanges those credentials for an account-level access token using Databricks' all-apis scope, which workflows use to call both the Databricks Workspace APIs (clusters, jobs, Unity Catalog, MLflow, SQL, secrets, and more) and the Databricks Account APIs (workspaces, metastores, networking, budgets, and more). Each connection is registered per workspace - the workspace URL identifies the connection, and the display name defaults to the workspace URL unless you provide an Instance Name.

**Authentication:** OAuth 2.0 client credentials (Databricks service principal, machine-to-machine flow)

**Data sync:** On-demand only - there is no background sync or entity ingestion. Serval calls Databricks only when a workflow step runs, exchanging your stored credentials for a fresh token on each request.

## What the Databricks integration enables

| Capability                     | Description                                                                                                                                                                                                                                                                      |
| ------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Call any Workspace API         | Workflows can read and manage everything in the connected workspace - Unity Catalog, MLflow, permissions, clusters, jobs, SQL, secrets, policies, vector search, serving endpoints, DBFS, pipelines, instance pools, repos, and apps - across 380 typed Workspace API endpoints. |
| Call any Account API           | Workflows can manage account-level resources - workspaces, credentials, metastores and storage credentials, networking, customer-managed keys, log delivery, OAuth app integrations, usage downloads, and budgets - across 51 typed Account API endpoints.                       |
| Manage users and groups (SCIM) | Read and update workspace users and groups, as well as account-level users, groups, and service principals, subject to the service principal's permissions.                                                                                                                      |
| Connection health checks       | Two built-in checks verify account-level and workspace-level access separately, so you can tell exactly which access tier is missing.                                                                                                                                            |

Anything defined in the [Databricks Workspace API](https://docs.databricks.com/api/workspace/) or [Databricks Account API](https://docs.databricks.com/api/account/) can be accessed through Serval.

## Get your credentials

Serval needs four things: your Databricks **Account ID**, the **Workspace URL** of the workspace to connect, and a service principal's **Client ID** and **Client Secret**. You create the service principal and its OAuth secret in the Databricks Account Console - see the official [Databricks OAuth machine-to-machine (M2M) authentication](https://docs.databricks.com/aws/en/dev-tools/auth/oauth-m2m) guide.

<Steps>
  <Step title="Log in to the Databricks Account Console">
    Use the console for your cloud: [accounts.cloud.databricks.com](https://accounts.cloud.databricks.com) for AWS, accounts.azuredatabricks.net for Azure, or accounts.gcp.databricks.com for GCP.
  </Step>

  <Step title="Create a service principal">
    Go to **User management**, open the **Service principals** tab, click **Add service principal**, name it (e.g. "Serval Integration"), and click **Add**.
  </Step>

  <Step title="Generate an OAuth secret">
    Open the service principal, select the **Secrets** tab, and under **OAuth secrets** click **Generate secret**. Set the lifetime (maximum 730 days). Copy both the **Secret** and the **Client ID** immediately.
  </Step>

  <Step title="Assign the service principal to your workspace">
    In the Account Console, go to **Workspaces**, select the target workspace, open the **Permissions** tab, click **Add permissions**, select the service principal, and assign a role (e.g. User or Admin).
  </Step>

  <Step title="Find your Account ID">
    In the Account Console, click your email in the top-right corner (the user menu). The Account ID is a UUID like xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx.
  </Step>
</Steps>

<Warning>
  The OAuth secret is shown **only once**, at generation time. Copy it before closing the dialog - if you lose it, you'll need to generate a new one.
</Warning>

<Note>
  Workspace assignment and account-level access are granted separately in Databricks. The service principal needs **both**: workspace assignment for Workspace API calls, and account-level access for Account API calls. Serval's health checks test each tier independently.
</Note>

## Connect in Serval

Every field except Instance Name is required - leaving a required field blank stops submission with "This field is required".

<Steps>
  <Step title="Enter your Account ID">
    Type the bare account ID - the UUID from the Account Console user menu, matching the placeholder "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx". The field only accepts letters, numbers, and hyphens, so anything URL-shaped (containing dots or slashes) is rejected with "Enter your Account ID, not a workspace URL. The Account ID is found in Account Console > User menu."
  </Step>

  <Step title="Enter your Workspace URL">
    For example "your-workspace.cloud.databricks.com". You can paste it with or without "https\://" - Serval strips the prefix and any trailing slash on save. This URL also tells Serval which cloud's Account API to use (AWS, Azure, or GCP).
  </Step>

  <Step title="Enter the Client ID">
    The Client ID (Application ID) of your Databricks service principal.
  </Step>

  <Step title="Enter the Client Secret">
    The OAuth secret you generated in Account Console under Service principals.
  </Step>

  <Step title="Optionally set an Instance Name">
    The field is labeled **Instance Name (Optional)** - a friendly name to identify this Databricks workspace, e.g. "Production Workspace". If left blank, the connection is named after the workspace URL.
  </Step>
</Steps>

<Note>
  When editing an existing connection, any field you clear or leave blank keeps its stored value - so to rotate the Client Secret, you only need to fill in the new secret. The settings page shows the stored secret masked (bullets with the last 4 characters visible); the Client ID is shown in full.
</Note>

## Verifying the connection

Two built-in health checks run against the connection, each testing a different access tier:

* **List Databricks Workspaces** - lists the workspaces in your Databricks account via the Account API, verifying that the token exchange works and the service principal has account-level access. On success it reports: "Successfully listed workspaces from Databricks account (count: \[number])". On failure: "Unable to list workspaces from Databricks account. Please verify the service principal has account-level access."
* **List Databricks Clusters** - lists the clusters in the connected workspace via the Workspace API, verifying the service principal has workspace-level access. On success it reports: "Successfully listed clusters from Databricks workspace (count: \[number])". On failure: "Unable to list clusters from Databricks workspace. Please verify the service principal has workspace-level access."

<Tip>
  The pass/fail pattern tells you which grant is missing. If **List Databricks Workspaces** passes but **List Databricks Clusters** fails, the service principal hasn't been assigned to the target workspace (Account Console, Workspaces, Permissions tab). If clusters pass but workspaces fail, the service principal lacks account-level access.
</Tip>

## Gotchas and troubleshooting

<AccordionGroup>
  <Accordion title="The Account ID field rejects anything that looks like a URL">
    The connect form only accepts an account ID made of letters, numbers, and hyphens (such as the UUID from the Account Console). A value containing dots or slashes - like a workspace URL - fails validation with "Enter your Account ID, not a workspace URL. The Account ID is found in Account Console > User menu." Find the ID in the Account Console by clicking your email in the top-right corner.
  </Accordion>

  <Accordion title="One token, two access tiers - grant both">
    Serval requests a single account-level token, but Databricks grants account-level and workspace-level access separately. A service principal can pass one health check and fail the other - use the pattern above to identify the missing grant. Note also that each connection is scoped to one workspace: connect each workspace separately.
  </Accordion>

  <Accordion title="The Workspace URL determines which cloud's Account API is used">
    Serval derives the accounts host from the Workspace URL: Azure workspace URLs route to accounts.azuredatabricks.net, GCP workspace URLs to accounts.gcp.databricks.com, and everything else falls back to accounts.cloud.databricks.com (AWS). An incorrect workspace URL can route the token exchange and Account API calls to the wrong cloud's control plane - double-check the URL if both health checks fail.
  </Accordion>

  <Accordion title="The Workspace URL doubles as the connection's identity">
    On save, trailing slashes and the "https\://" or "http\://" prefix are stripped, and the normalized URL becomes the connection's instance ID and its default display name. Changing the Workspace URL when editing the connection changes the connection's identity, not just a setting.
  </Accordion>

  <Accordion title="When updating the connection, blank fields keep their stored values">
    The update path keeps the stored value for every field you leave empty - Account ID, Workspace URL, Client ID, Client Secret, and Instance Name - so you only fill in what you're rotating. One caveat for direct API callers: the settings read returns the Client Secret masked (bullets plus the last 4 characters), and any non-empty submitted value is stored verbatim. Never send the masked value back as the new secret, or it will be saved literally and break authentication. The standard Serval form handles this for you.
  </Accordion>

  <Accordion title="The OAuth secret is shown only once">
    Databricks displays the service principal's OAuth secret a single time at generation, with a maximum lifetime of 730 days. If it's lost or expires, generate a new secret in the Account Console and update the connection's Client Secret field - all other fields can stay blank.
  </Accordion>

  <Accordion title="Pagination differs by API surface">
    If a workflow lists resources, be aware that different Databricks APIs page differently: SCIM user and group lists use startIndex/count and return totalResults, newer lists like clusters and jobs use page\_token with page\_size or limit and return next\_page\_token, and some account lists (like workspaces) return a single unpaginated array.
  </Accordion>
</AccordionGroup>

***

Need help? Contact **[support@serval.com](mailto:support@serval.com)** for assistance with your Databricks integration.