Connecting build analytics to Databricks

EngFlow build analytics records data about every remotely executed action (timings, cache hits, resource usage, and CI and SCM context) as rows in an Apache Iceberg table in your cloud storage. This guide describes how to read that table from Databricks.

This guide applies to EngFlow clusters whose build analytics warehouse is a plain s3:// or gs:// bucket.

Architecture: reading data without a catalog

Iceberg tables require a catalog

An Iceberg table consists of Parquet data files together with a chain of metadata files recording which data files constitute the table at a given point in time. This indirection provides consistent reads while EngFlow writes, and supports schema evolution and time travel.

A reader cannot determine table state by listing the storage prefix. It must first resolve which metadata file is current. The component performing that resolution is the catalog: it maps the table name observability.executions to the current metadata file location and updates that mapping atomically on each commit.

Query engines therefore connect to a catalog rather than to storage directly. Storage holds the data files; the catalog makes them addressable as a table.

Databricks cannot reach EngFlow's catalog

Clusters writing to s3:// or gs:// register their table in an Iceberg JDBC catalog hosted inside the cluster's invocation index database. That database is internal EngFlow infrastructure, and Databricks does not support Iceberg JDBC catalogs or accept an arbitrary Iceberg REST catalog as a federation source. There is no supported way to connect Databricks to this catalog directly.

Because EngFlow writes append-only (see below), you can instead read the Parquet data files directly, bypassing the catalog entirely.

Why direct access works without a catalog

EngFlow's Iceberg writer is append-only: each flush writes new Parquet data files and commits them with an append operation. It does not write Iceberg delete files, and it does not rewrite or overwrite existing data files.

This property makes the data files self-consistent as a plain Parquet dataset. Reading them directly yields correct rows, because no data file's contents are ever logically negated by a later delete file.

What direct reads give up is Iceberg's transactional layer: snapshot isolation, atomic visibility of each commit, schema evolution handling, and metadata-based file pruning. The constraints below all follow from that.

Locate the data files

Iceberg's default layout places data files under <warehouse>/<namespace>/<table>/data/, partitioned by day:

Text Only

s3://<bucket>/observability/executions/data/start_timestamp_day=2026-08-28/*.parquet

Confirm the actual layout for your cluster before relying on it by querying the catalog directly against the invocation index database:

SQL

SELECT metadata_location FROM public.iceberg_tables;

The metadata file at that location records the table's true data file paths.

Configure access and query

Create a Unity Catalog external location and storage credential covering the data prefix, then read the Parquet files by path:

SELECT
  action_mnemonic,
  COUNT(*) AS executions
FROM parquet.`s3://<bucket>/observability/executions/data/`
WHERE start_timestamp >= CURRENT_DATE - INTERVAL 7 DAYS
GROUP BY action_mnemonic
ORDER BY executions DESC;

Equivalently in PySpark:

df = spark.read.parquet("s3://<bucket>/observability/executions/data/")

For a stable, named table, register an external table over the same path. Unity Catalog external tables support PARQUET but not ICEBERG as a path-based format, so the table is a Parquet dataset, not an Iceberg table:

CREATE TABLE main.engflow.executions_raw
USING PARQUET
LOCATION 's3://<bucket>/observability/executions/data/';

Run MSCK REPAIR TABLE main.engflow.executions_raw after new partitions appear, or use spark.read with a path glob to avoid partition registration entirely.

Constraints on direct file access

Direct reads bypass the Iceberg metadata layer, so the following apply:

• No snapshot isolation. A query may observe a partially written flush. EngFlow flushes at most every five minutes by default, so for reproducible results exclude the current flush window by filtering start_timestamp to completed periods.
• Orphan files are visible. The writer aborts cleanly on shutdown, but a crashed process can leave uncommitted Parquet files that Iceberg would exclude and a direct read includes. This can double-count rows.
• No schema evolution handling. Iceberg resolves schema changes by field ID; Parquet reads resolve by name and position. Files written before a schema change may fail to merge or produce null columns. Use mergeSchema and validate after EngFlow upgrades.
• No time travel and no Iceberg metadata pruning. Partition directory pruning still works when you filter on the partition column.

EngFlow configuration reference

EngFlow applies these settings on your behalf. They are listed so you can confirm your configuration and correlate it with the behavior described above.

EngFlow-managed settings

All flags below carry the experimental_ prefix, are hidden, and are subject to change. Do not set them directly. Request configuration changes through your EngFlow contact.

Storage and catalog selection

Flag	Default	Purpose
--experimental_observability_warehouse_location	empty	Storage destination. An S3 URI (s3://<bucket>) or GCS bucket (gs://<bucket>) stores analytics data as described in this guide. Analytics is disabled when empty.
--experimental_observability_catalog_kind	ICEBERG	Destination type: ICEBERG, SNOWFLAKE, or CLICKHOUSE. Multiple values are permitted, so a cluster can write to several destinations simultaneously.
--experimental_observability_catalog_name	observability_catalog	Iceberg catalog name.
--experimental_observability_tables_namespace	observability	Namespace containing the analytics tables: the observability path component in the data file layout above.
--experimental_observability_executions_table_name	executions	Name of the executions table.

This requires the EngFlow invocation index to be enabled, because that database hosts the Iceberg catalog. Without it, no catalog exists and analytics data is not written.

Write batching and retention

These affect write latency and how quickly new data becomes visible, which is directly relevant to the snapshot isolation caveat above.

Flag	Default	Purpose
--experimental_observability_max_flush_interval	5m	Maximum delay between Iceberg flushes. Bounds how soon a completed action appears.
--experimental_observability_max_buffer_size	100000	Entries buffered before a write. Each consumes roughly 1040 bytes of memory.
--observability_queue_size	100000	Maximum in-memory write queue size. Entries are dropped if the queue fills.
--experimental_observability_snapshot_expiry_duration	1d	Age threshold for snapshot expiry. Values below one day are rejected.

Query guidance

The table is partitioned by day of start_timestamp, so filtering on that column enables partition pruning. Queries without a time filter scan the whole table, and their cost grows as history accumulates.

Rows are sorted by invocation_id, then action_mnemonic. The action_mnemonic, target_id, and pool_id columns are the most useful grouping keys, identifying action type, build target, and cluster pool. The timestamp columns partition each action's lifetime into queueing, input fetch, execution, and output upload phases.

See the Build analytics table schema for the complete field list.