Blog/Knowledge

Insurance on Databricks: from data model to generated code

Kevin MonkDirector Global Channel & Alliances
· · 10 min read
<span id="hs_cos_wrapper_name" class="hs_cos_wrapper hs_cos_wrapper_meta_field hs_cos_wrapper_type_text" style="" data-hs-cos-general-type="meta_field" data-hs-cos-type="text" >Insurance on Databricks: from data model to generated code</span>

Many insurers already run analytics on Databricks. This article shows how a data model can drive the code that loads that lakehouse and keep its logic in step with the source systems: what that looks like in a Databricks workspace, where it helps most, and where people stay in charge.

In short: Most insurance data platforms get built twice, once as a data model and again as the code that moves the data, and the two drift apart. With VaultSpeed, the definitions live in one data model in the enterprise context store, and the code that loads the Databricks lakehouse is generated from it. A change becomes one reviewed update in one place, and a reviewer can follow every figure in a return back to its source field. Only metadata reaches VaultSpeed, and the data stays in your workspace.

Why insurance data platforms get built twice

Most insurance data platforms get built twice: once as a data model that describes the business, and again as the code that moves the data. First the business agrees what a policy, a producer or a claim means. Then engineers turn that agreement into code.

The two drift apart from the first sprint. A definition changes in the data model, the code changes in a later release, and the gap shows up as reconciliation work. Tables on Databricks that were built on an old definition keep serving reports until someone finds them.

Two timelines: the data model changes a definition at v1, v2 and v3, while the code on Databricks picks up v2 and v3 a release later. The gap is drift, found as reconciliation work.
Figure 1. The data model and the code describe the same entities, but they are maintained separately, so they drift.

Insurance makes this worse. Agency codes, rating factors and reserves each turn up in more than one system, from policy administration and rating to claims and the finance ledger, and each system defines them its own way.

Insurers feel this most during change. A new product, a migrated policy administration system or a revised reporting requirement touches many definitions at once. Each one has to be found, updated and checked in every place it lives.

On a lakehouse the gap usually shows in the silver layer. Landing raw data in bronze is the easy part. Silver then gets defined per project: each team integrates the sources it needs on its own terms, and the same policy and claims data is integrated several times over.

What changes when the data model drives the code

The approach in this article rests on one principle: the definitions live in one data model, and the code that loads the lakehouse is generated from it. This is how VaultSpeed, an agentic data modeling platform, works. The business is modeled top down, the technical metadata of the source systems is harvested bottom up, and the match between the two is kept as metadata: structures, keys, relationships, mappings and lineage.

That metadata lives in the enterprise context store (ECS), a metadata graph that holds the data model in machine readable form. The graph links three contexts.

ContextWhat it holdsAt an insurer
Business contextThe definitions the organization has settled on, independent of any technologyPolicy, producer, claim and exposure, and how they relate. An industry data model such as ACORD can be the starting point.
Implementation contextThe physical objects, the loading logic and the lineage across bronze, silver and goldThe policy administration tables, the Data Vault built from them, and the gold tables that read from it
Data product contextThe data products, their contracts and their semanticsLoss ratio by line of business, with the definition of incurred it uses
The enterprise context store links business, implementation and data product context in one metadata graph. Conceptual data models, existing Erwin or PowerDesigner data models, source metadata and data product needs fill it; modeling agents, code generation, coding agents over MCP and governance tools read from it.
Figure 2. What fills the context store, and what reads from it. The store holds metadata only, and the data stays on Databricks.

Because the data model is machine readable, a change to it regenerates the code that depends on it. A change becomes one reviewed update in one place, and engineers can query the same definitions over MCP from the tools they already use. The same data model can also generate code for another platform, which turns a migration into a regeneration of the code.

How the bronze, silver and gold layers run on Databricks

Take an insurer whose policy administration, claims, billing and finance data already lands in a Databricks workspace. The data model adds an integrated silver layer and the gold tables on top of it, one layer at a time.

LayerWhat the insurer gets on DatabricksHow the code is produced
BronzeDelta tables with the policy, claims, billing and ledger data as it landed. Tables that are already in Databricks can be read as a source.Agents write the landing and staging code, and the team reviews it in Git.
SilverA Data Vault with full history: one policy, one producer and one claim across all systems.Rules and templates, with no LLM in the code path. The same data model gives the same code on every run.
GoldStar schemas and data products that read from silver, such as loss ratio by line of business and reserve movements.Agents generate it from the data product definitions, and the team reviews it in Git.

In the workspace, that comes down to four things.

  • Unity Catalog. Schemas are addressed as catalog.schema, and a common setup is one catalog per layer. Table and column comments from the sources are applied to the Delta tables, so the definitions show up in Catalog Explorer.
  • Code. The DDL and the loading code arrive as Databricks SQL notebooks or dbt models, deployed to a workspace folder or promoted through Git.
  • Compute. Everything runs on your own cluster or SQL warehouse. Airflow, Azure Data Factory or your own scheduler runs the loads.
  • Boundary. A connector in your network reads the structure of the sources, meaning table names, columns, data types and keys, and deploys the code. Only metadata reaches VaultSpeed, and the data stays in your workspace.
The data model and context store hold metadata only at VaultSpeed. A connector in your network reads source structure and deploys generated code. Data moves from policy admin, claims, billing and finance ledger into bronze, silver and gold in your Databricks workspace, governed in Unity Catalog.
Figure 3. The data moves from the sources straight into your Databricks workspace. The data model sits outside it as metadata, and only generated code comes in.

The generated code is yours: it runs on its own, independently of the subscription. You also choose the LLM, your own provider or inference through VaultSpeed, and the rule based silver code involves no LLM either way.

Where agents fit and where people decide

AI goes on the analysis and the modeling. Agents profile sources, propose the data model and the mappings, convert existing data models and make repetitive changes. They work on the data model and on the code generated from it, and they do not run business processes such as claims handling or payments.

Every agent session runs in an ephemeral sandbox, with no standing warehouse credentials and a spend cap set at platform level. Changes are versioned in Git, and teams and agents review each other's work before a change ships.

Five steps from a change request to Databricks: agents propose a change in a sandbox, lineage shows the impact, people review in Git and can send it back, then the code is generated and deployed.
Figure 4. A change to the data model, from request to Databricks. Lineage shows the impact before the review, and the code is generated after it.
Agents doPeople decide
Convert an existing data model, for example from Erwin or PowerDesigner, into the context store, and flag where its definitions differ from the source systems.Which definition of a policy, producer or claim is correct for the business.
Profile a new source, propose the data model and mappings for it, and show the lineage the generated code will create.Whether that proposal is accepted, changed or rejected.
Check a proposed change against lineage, and show which downstream objects it affects.When the change ships, through review in Git.
Answer engineers' questions about the data model over MCP.What those answers mean for a regulatory filing or a financial statement.

Regulatory interpretation, sign-off and compliance decisions stay with the insurer and its advisers. The data model makes the evidence easier to produce. It does not decide what the regulation requires.

Three insurance use cases on the lakehouse

Growing the book

Growth work starts with data that describes producers, policies, exposures and products. When those definitions live in several places, analysts spend their time reconciling them.

On Databricks, each feed lands in bronze and is mapped once to the data model. When a new agency or delegated authority partner starts sending bordereaux, the team adds one mapping, and the Data Vault in silver absorbs the new source without a redesign of what is already there. The gold tables for distribution and growth reporting keep reading the same producer and policy.

Agency systems, partner bordereaux, rating engine, product catalog and policy admin map into one data model of producers, policies, exposures and products. Its generated code loads every feed into Databricks for distribution, policy and exposure, product definitions, and new and lost business.
Figure 5. Each business feed maps into one data model. The code generated from it runs on Databricks and feeds four outcomes.

Staying audit-ready

In regulatory and audit work, traceability matters more than speed. The lineage is derived from the data model and its mappings, so nobody has to reconstruct it from the code afterwards. A reviewer follows a figure that feeds a return from the gold table back through silver and bronze to the source field.

The context store generates the code that loads bronze, silver and gold on Databricks. Lineage runs from each figure in a regulatory return back to the source field. Row filters and column masks are applied in Unity Catalog.
Figure 6. Each figure in a return keeps a route back to its source, because the lineage and the code come from the same metadata.

Lineage supports governance review. It does not certify compliance. Controls on access and data protection, such as Unity Catalog row filters and column masks, are applied on Databricks.

Running finance efficiently

Finance teams spend a large share of each close reconciling figures before any analysis starts. Reconciliation is easier when each measure is defined once. Incurred, paid and reserve each get one definition in the data model, and every variant the insurer keeps on purpose gets its own name.

On Databricks, the gold tables behind loss ratio by line of business and reserve movements all read those definitions, so the dashboards and Genie spaces built on gold start from the same numbers.

Claims system, finance ledger and an actuarial extract each define incurred differently. The data model defines incurred, paid and reserve once and names each variant, so the gold layer behind loss ratio, reserve movements and report feeds reads the same definitions.
Figure 7. The data model defines each incurred measure once, so every close reconciles against the same definitions. The gold code generated from it runs in your Databricks workspace.

Questions to take back to your team

  • Which regulatory figures can you trace back to source today?
  • Where does the same entity have two definitions across your platforms?
  • How many teams have built their own silver tables for policy or claims data?
  • How much reconciliation happens before each reporting cycle?
  • Which changes to your data model pass through review in Git, and which do not?
  • Which agent changes to your data model and code should need a human approval step, and how is that approval recorded?

Find where your data model stops matching your code

Start with one definition that the business already disputes. Trace it through the data model, the code and the reports. The gap you find is the place to begin.

Next step: see how VaultSpeed runs on Databricks on the VaultSpeed and Databricks page, or at docs.vaultspeed.com. To see how the agents work, take a look at the platform.

Frequently asked questions

Why do insurance data platforms get built twice?

First the business agrees what a policy, a producer or a claim means. Then engineers turn that agreement into code. The two drift apart from the first sprint, and the gap shows up as reconciliation work.

What is the enterprise context store?

The enterprise context store (ECS) is a metadata graph that holds the data model in machine readable form. It links three contexts: business context, implementation context and data product context.

Does insurance data leave the Databricks workspace?

No. A connector in your network reads the structure of the sources, meaning table names, columns, data types and keys, and deploys the code. Only metadata reaches VaultSpeed, and the data stays in your workspace.

Is an LLM involved in generating the silver layer?

No. The silver Data Vault is generated from rules and templates, with no LLM in the code path, so the same data model gives the same code on every run. For the agents, you choose the LLM: your own provider or inference through VaultSpeed.

What do agents do, and what do people decide?

Agents profile sources, propose the data model and the mappings, convert existing data models and make repetitive changes. People decide which definition of a policy, producer or claim is correct for the business, whether a proposal is accepted, and when a change ships, through review in Git.

Does lineage certify compliance?

No. Lineage supports governance review. It does not certify compliance. Regulatory interpretation, sign-off and compliance decisions stay with the insurer and its advisers.

Keep reading

More from the blog

All articles
See it on your own data

One session on your own sources says more than any article

Bring a source system and a question. We connect the context store to it and show you what the agents propose.

Talk to us