Skip to content

Build analytics

Beta

This feature is currently in beta. Contact EngFlow if you'd like to use it.

EngFlow captures build performance data for every remotely executed invocation. While this data is surfaced in the Build and Test UI as the EngFlow Profile, build analytics makes this data available as a queryable data store for you to extract powerful build performance insights. You can query the data directly using tools like DuckDB, or stream it to Snowflake via Snowpipe Streaming.

Queryable build data deepens your visibility into:

  • Build health information
  • Action-level metrics
  • Build trends over a period of time that's significant to your organization.

Data schema

EngFlow build analytics records data from the Remote Execution API v2 (REAPI) as rows in an Apache Iceberg table. If your EngFlow RE cluster is deployed on AWS, this data is stored in AWS’s S3 Tables. If your cluster is deployed on GCP, this data is stored in a GCS bucket, leveraging the invocation index database as the Iceberg catalog. This data can also be streamed to your Snowflake account via Snowpipe Streaming.

See Iceberg table schema for the full schema.

Enabling build analytics for your cluster

Contact EngFlow to enable build analytics for your cluster.

Please note that enabling build analytics will result in a small increase in your cloud costs. We recommend monitoring the associated cost during the first 30 days of usage.

Querying build analytics data

This section describes how to query your cluster's build analytics data, including the access and tooling you need to get started. Note that these instructions use DuckDB to query the database. You may use a similar tool of your choice. If your organization uses Snowflake, see Streaming to Snowflake.

Step 1: Install prerequisite tooling and verify access

  1. Make sure you have the following core tools installed:

    • Python 3.14 or above
    • A Python package manager like uv or pip.

  2. Install required dependencies. The requirements.txt file below lists the required packages. Create a copy of this file in the directory where you'll be running your queries:

    requirements.txt
     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
    [project]
name = "analytics-example"
version = "0.1.0"
dependencies = [
    "duckdb>=1.4.2,<1.5.2",
    "ipykernel>=7.1.0",
    "jupyter>=1.1.1",
    "pandas>=2.3.3",
    "pip>=25.3",
    "plotly>=6.4.0",
    "pyarrow>=22.0.0",
]
requires-python = '>=3.14'

  3. Run uv sync to install the dependencies.

  4. Install the AWS CLI.

  5. Make sure you can access S3 tables.

Step 2: Create a Jupyter Notebook

Using Jupyter Notebook in an IDE

You can do this step in an IDE like Visual Studio Code. If using an IDE, select the Python interpreter from the virtual environment created by uv as your notebook kernel. This ensures the notebook has access to all installed dependencies.

Start Jupyter Notebook and create a notebook named engflow_analytics.ipynb. Then, create the following cells in your notebook:

Python
# Import required libraries
import duckdb
import json
Python
# Install and load the Iceberg extension for DuckDB
# This enables DuckDB to read Apache Iceberg table format
duckdb.sql("INSTALL iceberg; LOAD ICEBERG")
Python
# Authenticate with AWS using Single Sign-On
! aws sso login
Python
# Export temporary AWS credentials for the specified cluster profile
# Update cluster name and region
credentials = !aws configure export-credentials --profile <your-cluster-name> --region us-east-1 --format process
credentials = json.loads(''.join(credentials))
print(f'These credentials will expire on {credentials["Expiration"]} and need to be refreshed.')
Python
# Find the S3 Tables bucket ARN that contains observability data
aws_arn = !aws --profile <your-cluster-name> s3tables list-table-buckets
table_bucket_result = json.loads(''.join(aws_arn))['tableBuckets']
table_bucket = [x['arn'] for x in table_bucket_result if '-observability' in x['name']][0]
table_bucket
Python
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
# Update region

region = 'us-east-1'

# Create a secret in DuckDB containing AWS credentials
duckdb.sql(f"""
  CREATE OR REPLACE SECRET s3table_secret (
    TYPE s3,
    KEY_ID '{credentials['AccessKeyId']}',
    SECRET '{credentials['SecretAccessKey']}',
    SESSION_TOKEN '{credentials['SessionToken']}',
    REGION '{region}'
);                   
""")

# Attach the S3 Tables database to DuckDB for querying
duckdb.sql(f"""
ATTACH '{table_bucket}' AS s3_tables (
    TYPE iceberg,
    SECRET s3table_secret,
    ENDPOINT_TYPE s3_tables
)
""")
Python
# Create a local DuckDB table from the remote executions table
# This is optional but can improve query performance for repeated queries
# It can take a long time to cold start if scanning over large data sets
# We recommend filtering this data to applicable data ranges
duckdb.sql("""
   CREATE OR REPLACE TABLE executions AS
SELECT * FROM s3_tables.observability.executions;        
""")

Step 3: Run a test query

To verify your setup, run a query to get the number of test executions that ran yesterday:

Python
# Update the database name, schema, and table name
a = duckdb.sql("""
    SELECT COUNT(*) as test_that_run_yesterday
    FROM <database>.<schema>.<table>
    WHERE start_timestamp_day = CURRENT_DATE() - INTERVAL 1 DAY
        AND action_mnemonic = 'TestRunner'
""").to_df()

If you see a result showing the number of test executions, your setup is working as expected.

Step 1: Install prerequisite tooling and verify access

  1. Make sure you have the following core tools installed:

    • Python 3.14 or above
    • A Python package manager like uv or pip.

  2. Install required dependencies. The requirements.txt file below lists the required packages. Create a copy of this file in the directory where you'll be running your queries:

    requirements.txt
     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
    [project]
name = "analytics-example"
version = "0.1.0"
dependencies = [
    "duckdb>=1.4.2,<1.5.2",
    "google-cloud-bigquery[all]>=3.38.0",
    "ipykernel>=7.1.0",
    "jupyter>=1.1.1",
    "pandas>=2.3.3",
    "pip>=25.3",
    "plotly>=6.4.0",
    "pyarrow>=22.0.0",
]
requires-python = '>=3.14'

  3. Run uv sync to install the dependencies.

  4. Install the Google Cloud CLI.

  5. Make sure you can access BigQuery datasets.

Step 2: Create a Jupyter Notebook

Using Jupyter Notebook in an IDE

You can do this step in an IDE like Visual Studio Code. If using an IDE, select the Python interpreter from the virtual environment created by uv as your notebook kernel. This ensures the notebook has access to all installed dependencies.

Start Jupyter Notebook and create a notebook named engflow_analytics.ipynb. Then, create the following cells in your notebook:

Python
# Import required libraries
import duckdb
import pandas as pd
import plotly.express as px
import pyarrow
Python
# Authenticate with Google Cloud using application default credentials
! gcloud auth application-default login
Python
# Initialize BigQuery client to query the analytics dataset
from google.cloud import bigquery
bq = bigquery.Client()

# Update your project ID
client = bigquery.Client(project='my-project-id-123abc456')

Step 3: Run a test query

To verify your setup, run a query to get the number of test executions that ran yesterday:

Python
# Query BigQuery to count test executions from yesterday
# Update the project ID, dataset name, and table name
jb = client.query_and_wait("""
SELECT COUNT(*) as test_that_run_yesterday
FROM `<projectId>.<datasetName>.<tableName>`  
WHERE start_timestamp_day = CURRENT_DATE() - INTERVAL 1 DAY
    AND action_mnemonic = 'TestRunner'
""")

# Convert query results to a pandas DataFrame for analysis
jb.to_dataframe()

If you see a result showing the number of test executions, your setup is working as expected.

Streaming to Snowflake

If your organization uses Snowflake, you can stream EngFlow build analytics data directly to your Snowflake account via Snowpipe Streaming. This lets you query execution data using Snowflake SQL and integrate it with your existing Snowflake workflows.

Schema differences

The build analytics data streamed to Snowflake uses the same underlying schema as the Iceberg table, but the fields are structured differently. Instead of top-level fields, there is a single top-level column called execution that contains a semi-structured object. This execution object holds all of the remaining fields, including nested objects.

For example, to query target_id in Iceberg you'd run:

SQL
SELECT target_id FROM ...

In Snowflake SQL, it becomes:

SQL
SELECT execution:target_id FROM ...

Set up Snowflake streaming

Step 1: Set up Snowflake objects

Existing Snowflake account required

The following instructions assume that you have an existing Snowflake account with billing enabled.

Create the following objects in your Snowflake account. Names shown in the code sample are suggestions and you can customize them, except for the table's column, which must be called execution.

SQL
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
-- Step 1: Create database
CREATE DATABASE engflow_observability;

-- Step 2 (optional): Create a schema. By default there will be a `public` schema.
CREATE SCHEMA engflow_observability.observability;

-- Step 3: Create the main table that will hold execution logs
CREATE TABLE engflow_observability.observability.executions (execution OBJECT);

-- Step 4: Create the Pipe that will be used for Snowpipe to write to the table above.
CREATE PIPE engflow_observability.observability.executions_pipe
AS COPY INTO engflow_observability.observability.executions FROM TABLE(DATA_SOURCE(TYPE => 'STREAMING'))
MATCH_BY_COLUMN_NAME = CASE_INSENSITIVE;

-- Step 5: Create the role that will be authorized to write to Snowflake
CREATE ROLE engflow_observability_executions_writer COMMENT = 'Role with privileges to write EngFlow''s Execution logs';

-- Step 6: Authorize the role created above
GRANT USAGE ON DATABASE engflow_observability TO engflow_observability_executions_writer;
GRANT USAGE ON SCHEMA engflow_observability.observability TO engflow_observability_executions_writer;
GRANT INSERT ON TABLE engflow_observability.observability.executions TO engflow_observability_executions_writer;
GRANT OPERATE ON PIPE engflow_observability.observability.executions_pipe TO engflow_observability_executions_writer;

-- Step 7: Create the user that will belong to the role created above.
CREATE OR REPLACE USER engflow_observability_user
    -- See https://docs.snowflake.com/en/user-guide/key-pair-auth for how to generate an *unencrypted* key.
    RSA_PUBLIC_KEY='MII...'
    DEFAULT_ROLE = engflow_observability_executions_writer
    COMMENT = 'Service user for writing EngFlow data';

GRANT ROLE engflow_observability_executions_writer TO USER engflow_observability_user;

Step 2: Create secrets in your cloud provider

Create two secrets in the cloud provider console where you administer your EngFlow cluster:

These secrets can have any name.

Create Snowflake config secret

The Snowflake config secret is a JSON object. All fields are required. In this example, the secret is named snowflake-observability-config.

JSON
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
{
  "user": "engflow_observability_user",
  "uri": "https://AAA-BBB123.snowflakecomputing.com:443",
  "account": "AAA-BBB123",
  "client": "engflow_observability_client",
  "channel": "engflow_observability_channel",
  "database": "engflow_observability",
  "schema": "observability",
  "executions_pipe": "executions_pipe"
}

Note

  • The client and channel values are arbitrary names used to identify the connection. They don't need to be pre-configured in Snowflake.
  • The database, schema, and executions_pipe values should match the objects created in Step 1.
  • The table name does not appear in this config, since writes happen via the pipe.
  • The account value must match the account identifier in the uri (e.g., AAA-BBB123 in https://AAA-BBB123.snowflakecomputing.com:443).
Create a private key secret

Store the full private key as the second secret. The key must be unencrypted (no passphrase). See Snowflake's key pair auth docs for how to generate one. In this example, the secret is named snowflake-observability-auth-private-key.

-----BEGIN PRIVATE KEY-----
MII...
-----END PRIVATE KEY-----

Step 3: Share secret names with EngFlow

Once you've created the secrets, contact EngFlow and share the names of the Snowflake config and private key secrets so that we can configure your cluster to stream execution data to Snowflake. In this example, you'd share the following secret names: snowflake-observability-config and snowflake-observability-auth-private-key.

Next steps

Now that you've verified your setup, you can:

  • Explore the full Iceberg table schema.
  • Create custom queries to analyze your build performance.
  • Set up automated reports using the queried data.
  • Stream data to Snowflake if your organization uses Snowflake for analytics.