Skip to content

Metrics Reference

Description of all metrics available for monitoring.

AnalyzeInvocation metrics

com.engflow.invocationanalyzer/bazel_profile_count

no unit
The number of Bazel profiles that were attempted to be fetched

Tags

  • status: The status of retrieving the Bazel profile.
Details

The number of Bazel profiles that were attempted to be fetched.

com.engflow.invocationanalyzer/bazel_profile_size

bytes
The size of the uncompressed Bazel profile handled
Details

The size of the uncompressed Bazel profile handled.

com.engflow.invocationanalyzer/engflow_profile_count

no unit
The number of EngFlow profiles that were attempted to be fetched

Tags

  • status: The status of retrieving the EngFlow profile.
Details

The number of EngFlow profiles that were attempted to be fetched.

com.engflow.invocationanalyzer/engflow_profile_size

bytes
The size of the uncompressed EngFlow profile handled
Details

The size of the uncompressed EngFlow profile handled.

com.engflow.invocationanalyzer/time_needed

milliseconds
The time distribution of handing individual profile analysis requests

Tags

  • status: The status of the analysis performed
Details

The time distribution of handling individual Bazel profiles.

Opencensus OperationController metrics reporter

com.engflow.operationcontroller/active

no unit
The number of active operations (i.e., currently running).

Tags

  • name: The name of the OperationController
Details

The number of active operations (i.e., currently running).

com.engflow.operationcontroller/latency

milliseconds
The latency to start operations (i.e., how long operations are waiting to be executed).

Tags

  • name: The name of the OperationController
Details

The latency to start operations (i.e., how long operations are waiting to be executed).

com.engflow.operationcontroller/queued

no unit
The number of operations queued for execution.

Tags

  • name: The name of the OperationController
Details

The number of operations queued for execution.

com.engflow.operationcontroller/runtime

milliseconds
The runtime of operations (i.e., the duration operations are running for).

Tags

  • name: The name of the OperationController
Details

The runtime of operations (i.e., the duration operations are running for).

Metrics derived from raw BEP streams

com.engflow.bep/invocation_completed

no unit
Fired with the count of completed invocations reported to the BEP.

Tags

  • exit_code: The human readable exit code of the invocation.

com.engflow.bep/invocation_duration

milliseconds
Fired on invocation completed with the average duration of the invocation.

com.engflow.bep/invocation_started

no unit
Fired with the count of newly started invocations reported to the BEP.

Blob-storage implementation metrics

com.engflow.blobstore/latency

milliseconds
The duration each operation takes.

Tags

  • operation

  • status

com.engflow.blobstore/ops

no unit
Fires every time an operation takes place.

Tags

  • operation

Docker proxy

com.engflow.dockerproxy/blob_upload_bytes

bytes
The size of Docker blobs that the proxy successfully uploaded to the CAS.

Tags

  • status

com.engflow.dockerproxy/cache_hit_bytes

bytes
The size of Docker blobs that the proxy could find in the CAS.

com.engflow.dockerproxy/cache_miss_bytes

bytes
The size of Docker blobs that the proxy expected but could not find in the CAS.

com.engflow.dockerproxy/known_blobs_total

no unit
The number of Docker blobs that the proxy has metadata about.

HTTP clients for the Docker proxy

com.engflow.dockerproxy/http_received_bytes_total

bytes
Bytes received over the HTTP client

Tags

  • client: The name of the HTTP client (can be used to distinguish layers)

com.engflow.dockerproxy/http_request_latency_seconds_total

s
Time it's taken to serve HTTP requests

Tags

  • client: The name of the HTTP client (can be used to distinguish layers)

  • status: The status code, reduced to 1xx..5xx, or FAILED if an exception occurred

com.engflow.dockerproxy/http_requests_total

no unit
Number of HTTP requests started on the HTTP client

Tags

  • client: The name of the HTTP client (can be used to distinguish layers)

  • method: The HTTP method of the request

BEP Event Storage and Replay

com.engflow.eventstore/bep_event_ack_latency

milliseconds
This is a distribution. Tracks how much time passed between receiving a build event and sending an acknowledgement to the client.

Tags

  • status

com.engflow.eventstore/bes_upload_delay

milliseconds
This is a distribution. Tracks how much longer an invocation's BES upload took compared to the invocation's duration as reported by the BES.

Tags

  • status

com.engflow.eventstore/build_event_owners

no unit
The total number of build owners residing in memory. A build owner is an internal piece of state associated to a particular build reporting build events.

com.engflow.eventstore/flushing_batches_size

no unit
The total size of complete build event batches that are currently being uploaded to storage. Normally, batches are flushed quickly, so this value should stay near zero; if it doesn't, that could mean we are falling behind with batch uploads. Every instance reports its own stats; sum them up to get a cluster-wide metric.

com.engflow.eventstore/grpc_eventstore_ttfb

milliseconds
This is a distribution. Tracks how much time passed between requesting EventStore data via gRPC, and receiving the first byte.

Tags

  • type

com.engflow.eventstore/inbound_bep_events

no unit
Incremented whenever an event is received on an inbound stream.

Tags

  • type
Details

An inbound stream means a client (e.g. Bazel) sending BES events to a scheduler.

Every instance reports its own stats; sum them up to get a cluster-wide metric.

com.engflow.eventstore/incomplete_batches_size

no unit
The estimated size in bytes it would take to serialize all incomplete build event batches. These batches aren't yet written to storage. Actual JVM heap footprint is likely larger. Every instance reports its own stats; sum them up to get a cluster-wide metric.

com.engflow.eventstore/new_outbound_streams

no unit
Incremented whenever a new outbound BEP stream is created.
Details

An outbound stream means a scheduler replaying a BES stream in order to reduce it to a result stream. It makes no difference if the scheduler is replaying to itself or to another instance.

Every instance reports its own stats; sum them up to get a cluster-wide metric.

com.engflow.eventstore/ongoing_streams

no unit
The total number of streams that are inbound, outbound, or both.
Details

An inbound stream means a client (e.g. Bazel) sending BES events to a scheduler. An outbound stream means a scheduler replaying a BES stream in order to reduce it to a result stream. It makes no difference if the scheduler is replaying to itself or to another instance.

Every instance reports its own stats; sum them up to get a cluster-wide metric.

com.engflow.eventstore/outbound_bep_events

no unit
Incremented whenever an event is sent on an outbound stream.
Details

An outbound stream means a scheduler replaying a BES stream in order to reduce it to a result stream. It makes no difference if the scheduler is replaying to itself or to another instance.

Every instance reports its own stats; sum them up to get a cluster-wide metric.

Virtual Machine Instances

com.engflow.instance.new/gc_avg_duration

milliseconds
The average duration spent in garbage collection since the last reported metric.

Tags

  • gc_type: GC Old generation / GC Young Generation

  • instance_role: type of the instance (scheduler/worker/etc.)

com.engflow.instance.new/gc_count

no unit
The total number of garbage collections during the lifecycle of this process.

Tags

  • gc_type: GC Old generation / GC Young Generation

  • instance_role: type of the instance (scheduler/worker/etc.)

com.engflow.instance.new/gc_time

milliseconds
The total wall time in milliseconds spent blocked in garbage collection since the start of the process. This measures time when the application is not running due to a collector pause.

Tags

  • gc_type: G1 Old generation / G1 Young Generation

  • instance_role: type of the instance (scheduler/worker/etc.)

com.engflow.instance.new/open_file_descriptors

no unit
The number of file descriptors the process has currently open.

Tags

  • instance_role: type of the instance (scheduler/worker/etc.)

com.engflow.instance.new/total_disk_space

bytes
The size of the volume.

Tags

  • instance_role: type of the instance (scheduler/worker/etc.)

  • volume: the name of the disk volume

com.engflow.instance.new/total_system_memory

bytes
The total amount of system memory in bytes.

Tags

  • instance_role: type of the instance (scheduler/worker/etc.)

com.engflow.instance.new/used_disk_percentage

percentage
The percentage of the volume that is currently used.

Tags

  • instance_role: type of the instance (scheduler/worker/etc.)

  • volume: the name of the disk volume

com.engflow.instance.new/used_disk_space

bytes
The total number of bytes used on the volume.

Tags

  • instance_role: type of the instance (scheduler/worker/etc.)

  • volume: the name of the disk volume

com.engflow.instance.new/used_process_native_buffer_memory

bytes
The total amount of native buffer memory for this process in bytes.

Tags

  • instance_role: type of the instance (scheduler/worker/etc.)

com.engflow.instance.new/used_system_memory

bytes
The amount of used system memory in bytes.

Tags

  • instance_role: type of the instance (scheduler/worker/etc.)

com.engflow.instance.new/used_system_memory_percentage

percentage
The percentage of system memory used.

Tags

  • instance_role: type of the instance (scheduler/worker/etc.)

com.engflow.instance/gc_avg_duration

milliseconds
The average duration spent in garbage collection since the last reported metric.

Tags

  • gc_type

com.engflow.instance/gc_count

no unit
The total number of garbage collections during the lifecycle of this process.

Tags

  • gc_type

com.engflow.instance/gc_time

milliseconds
The total estimated time in milliseconds performing garbage collection.

Tags

  • gc_type

com.engflow.instance/total_disk_space

bytes
The size of the volume.

Tags

  • volume

com.engflow.instance/total_system_memory

bytes
The total amount of system memory in bytes.

com.engflow.instance/used_disk_percentage

percentage
The percentage of the volume that is currently used.

Tags

  • volume

com.engflow.instance/used_disk_space

bytes
The total number of bytes used on the volume.

Tags

  • volume

com.engflow.instance/used_system_memory

bytes
The amount of used system memory in bytes.

com.engflow.instance/used_system_memory_percentage

percentage
The percentage of system memory used.

Netty monitoring

com.engflow.thirdparty.netty/used_direct_memory

bytes
Direct (non-heap) memory use

Tags

  • buffer_name

com.engflow.thirdparty.netty/used_heap_memory

bytes
Heap memory use

Tags

  • buffer_name

io.netty.buffer/used_direct_memory

bytes
Direct (non-heap) memory use

Tags

  • buffer_name

io.netty.buffer/used_heap_memory

bytes
Heap memory use

Tags

  • buffer_name

Action scheduling

com.engflow.re.scheduler/autoscaler_set_size_operations

no unit
Number of desired executors, per pool

Tags

  • pool: name of pool ("default" for the default pool)

  • status: "succeeded" or "failed"

Details

Count of builtin autoscaler attempts to set desired size of a pool

com.engflow.re.scheduler/available_workers

no unit
Deprecated; number of idle executors, per pool

Tags

  • name: name of the pool ("default" for the default pool)
Details

Deprecated. Indicates the number of idle executors, per pool, according to this scheduler. Only schedulers report this metric. Every scheduler reports the same (or about the same) value. This metric is deprecated, because it may be imprecise: schedulers that are started while workers are busy may report a higher value than they should for several minutes. We recommend monitoring com.engflow.re.scheduler/existing_executors instead.

com.engflow.re.scheduler/coalesced_executions

no unit
Number of action requests of coalesced into an existing execution
Details

Number of action requests of coalesced into an existing execution

com.engflow.re.scheduler/desired_executors

no unit
Number of desired executors, per pool

Tags

  • name: name of the pool ("default" for the default pool)
Details

Indicates an estimate for the number of required executors per pool. Every scheduler reports its own estimate - they should be summed up to get the total desired pool size.

com.engflow.re.scheduler/estimated_action_time

milliseconds
Estimated action time

Tags

  • name: name of the pool ("default" for the default pool)
Details

Indicates an estimate for duration of an action. Every scheduler reports its own estimate.

com.engflow.re.scheduler/estimated_induced_load

no unit
Estimated induced wait time

Tags

  • name: name of the pool ("default" for the default pool)
Details

Indicates an estimate for future incoming work in ms per ms. Every scheduler reports its own estimate.

com.engflow.re.scheduler/existing_executors

no unit
Number of existing executors, per pool

Tags

  • name: name of the pool ("default" for the default pool)
Details

Number of existing executors, per pool, according to this scheduler. Only schedulers report this metric. Every scheduler reports the same (or about the same) value.

com.engflow.re.scheduler/existing_schedulers

no unit
Number of existing schedulers
Details

Only schedulers report this metric. Every scheduler reports a constant "1". This can be used to detect schedulers that are unable to send monitoring metrics.

com.engflow.re.scheduler/global_queue_size

no unit
Number of waiting actions, per pool

Tags

  • name: name of the pool ("default" for the default pool)
Details

Indicates the number of actions waiting for execution, per pool, in the cluster. Only schedulers report this metric. The schedulers coordinate to calculate this sum.

com.engflow.re.scheduler/global_used_executors

no unit
Number of used executors, per pool

Tags

  • name: name of the pool ("default" for the default pool)
Details

Indicates the number of executors that are in use, per pool, in the cluster. Only schedulers report this metric.

com.engflow.re.scheduler/pool_utilization

percentage
Current executor utilization, per pool

Tags

  • name: name of the pool ("default" for the default pool)
Details

Reports current executor utilization (used*100/total) per pool, as a percentage ([0..100]). Only schedulers report this metric. Every scheduler reports the same (or about the same) value.

To help making scale-up decisions when a pool is empty, utilization is reported as 100 if there are actions waiting and 0 if not. A pool may be empty if it was scaled down, or if it never existed (the client may request any pool name).

com.engflow.re.scheduler/queue_age

milliseconds
Min/max age of queued actions, per pool

Tags

  • name: name of the pool ("default" for the default pool)

  • statistic: "min" (youngest) or "max" (oldest) action in the pool's queue

Details

Reports minimum and maximum age in each executor pool, i.e. how long entries have been waiting. Only schedulers report this metric. Every scheduler reports its own queue lengths. Changes in these values indicate a change in the cluster's throughput.

com.engflow.re.scheduler/queue_size

no unit
Number of waiting actions, per pool

Tags

  • name: name of the pool ("default" for the default pool)
Details

Indicates the number of actions waiting for execution, per pool, on this scheduler. Only schedulers report this metric. Every scheduler reports its own queue lengths.

Observability UI metrics

com.engflow.observability.ui/app_load

milliseconds
The duration of the initial application load.

Tags

  • page

com.engflow.observability.ui/caught_error

no unit
An error in the web client and was manually caught and reported.

Tags

  • error_description

  • page

com.engflow.observability.ui/navigation

no unit
A single user navigation to a new page.

Tags

  • page

com.engflow.observability.ui/page_load

milliseconds
Measures the duration elapsed loading the current page.

Tags

  • load_type

  • page

com.engflow.observability.ui/page_load_with_data_requests

milliseconds
Measures the duration elapsed loading the current page, plus any page-specific data requests sent after it was initially loaded. This is currently only selectively enabled for some pages.

Tags

  • page

  • request_status

com.engflow.observability.ui/uncaught_error

no unit
An uncaught error was thrown in the web client.

Tags

  • error_name

  • page

BES Replay Metrics

com.engflow.bes.replay/cpu_time

milliseconds
The amount of CPU time spent replaying and reducing build event streams, measured in milliseconds.

Tags

  • replay_type

  • status

com.engflow.bes.replay/cpu_time_for_event

milliseconds
The amount of CPU time spent replaying and reducing a single event within a build event stream, measured in milliseconds.

Tags

  • event_type

  • replay_type

  • status

com.engflow.bes.replay/estimated_active_memory_usage

bytes
A very rough estimate of the amount of memory consumed by active replays.

Tags

  • replay_type

ResultStore metrics

com.engflow.resultstore/new_reduce_bes_count

no unit
The number of BES replays for reducing to InvocationState which are currently running or recently completed.

Tags

  • replay_type

com.engflow.resultstore/reduce_bes_completed_duration_since_finish_event

milliseconds
This is a distribution. Tracks how much time passed between receiving aninvocation's finish event, and completing the BES reduction.

Tags

  • replay_type

  • status: Whether the BES reduction finished successfully.

com.engflow.resultstore/reduce_bes_replay_removed_from_cache_count

no unit
The number of BES replays that were removed from the replay cache.

Tags

  • replay_type

  • status: The status of the replay when it was removed from the cache.

com.engflow.resultstore/reduce_bes_replay_source_count

no unit
The number of BES replays requested, tagged by where the data was fetched from, and the type of replay.

Tags

  • replay_type

  • source: Indicates where the data was fetched from.

com.engflow.resultstore/reduce_bes_target_metadata_count

no unit
The number of target metadatas in memory for reducing to InvocationState.

com.engflow.resultstore/reduce_bes_target_tree_node_count

no unit
The number of current target tree nodes in memory for reducing to InvocationState.

com.engflow.resultstore/reduce_bes_target_tree_outdated_node_count

no unit
The number of outdated target tree nodes that are cached in memory while reducing to InvocationState.

Storage implementation metrics

com.engflow.storage.gc/gc_window_seconds

s
The storage service's GC window.

Tags

  • name: name of the storage service

com.engflow.storage.read/size

bytes
Number of file bytes sent to the client for a read request. May be smaller than the file size in case of error or partial read. Only recorded if the file was found.

Tags

  • name: name of the storage service

  • status: op result

com.engflow.storage.read/time_per_gb

milliseconds
Time taken per 1 billion bytes (1 GB) to download a file from storage.

Tags

  • name: name of the storage service

  • status: op result

com.engflow.storage.read/time_to_first_byte

milliseconds
Time taken between initiating a download to receiving the first byte.

Tags

  • name: name of the storage service

  • status: op result

com.engflow.storage.read/time_to_next_chunk

milliseconds
Time taken between being notified that the client is ready and sending the next response. May be recorded 0 or multiples times for the same call, depending on control flow events.

Tags

  • name: name of the storage service

  • status: op result

com.engflow.storage.write/size

bytes
Number of file bytes received from the client for a write request. May be smaller than the file size in case of error.

Tags

  • name: name of the storage service

  • status: op result

com.engflow.storage.write/time_per_gb

milliseconds
Time taken per 1 billion bytes (1 GB) to upload a file to storage.

Tags

  • name: name of the storage service

  • status: op result

com.engflow.storage.write/time_to_commit

milliseconds
Time between being notified the write is complete and committing the write.

Tags

  • name: name of the storage service

  • status: op result

NotificationQueue metrics

com.engflow.notificationqueue/dequeue_latency

milliseconds
This is a distribution. Refers to the time passed between notification creation and dequeuing it.

Tags

  • expired: Whether the notification was expired and discarded.

  • name: The name of the queue.

com.engflow.notificationqueue/head_age

milliseconds
The age of the first notification in the queue, i.e. the time passed since the notification was created. Notably, it does NOT reflect how much time has passed since the notification was last published. For example, if a notification is not acknowledged and published anew, the head age may be disproportionately high compared to the age of the next notifications in the queue. This can lead to acceptable, occasional spikes.

Tags

  • name: The name of the queue.

com.engflow.notificationqueue/publish

milliseconds
This is a distribution. Refers to the time needed to publish a notification.

Tags

  • name: The name of the queue.

  • status: The status of the operation.

com.engflow.notificationqueue/size

no unit
The approximate size of the queue.

Tags

  • name: The name of the queue.

Action execution

com.engflow.re.exec/completed_actions

no unit
Number of actions that ran to completion, grouped by exit code

Tags

  • exit_code: the action's exit code
Details

This metric reflects the rate of change. Each measurement indicates how many actions completed on this worker, in all pools combined, since the last time this metric was reported.

Only workers report this metric. All workers report their own values. We recommend grouping by exit_code=0 and exit_code!=0, and summing up the time series in the groups. This yields the rate of successful and unsuccessful action completion across the cluster.

com.engflow.re.exec/completed_actions_per_pool

no unit
Number of executed actions (not cached), grouped by pool and status

Tags

  • pool: name of the pool (_default_ for the default pool)

  • status: the action's status (ExecutionStatus: SUCCESS, NON_ZERO_EXIT, CLIENT_ERROR, ERROR)

Details

This metric reflects the rate of change. Each measurement indicates how many actions completed on this worker, per pool, since the last time this metric was reported.

Only workers report this metric. All workers report their own values. We recommend grouping by status, and summing up the time series in the groups. This yields the rate of successful and unsuccessful action completion across the pool/cluster.

com.engflow.re.exec/execution_latency

milliseconds
Bucketed latency (ms), grouped by pool and execution stage

Tags

  • pool: name of the pool (_default_ for the default pool)

  • stage: the action's stage (ExecutionStage: QUEUED, DOWNLOAD_INPUTS, EXECUTE_ACTION, UPLOAD_OUTPUTS, EXECUTOR_TOTAL)

com.engflow.re.exec/executors_existing

no unit
Total number of executors on this worker, in all pools combined
Details

Only workers report this metric. All workers report their own values. You should sum up the time series to get the total number of executors in the cluster.

com.engflow.re.exec/executors_existing_per_pool

no unit
Total number of executors on this worker, per pool

Tags

  • pool: name of the pool (_default_ for the default pool)
Details

Only workers report this metric. All workers report their own values. You should sum up the time series to get the total number of executors in the pool/cluster.

com.engflow.re.exec/max_rss_kib

KiBy
Reported MaxRSS (maximal resident set size) of successfully executed actions (not cached), grouped by pool

Tags

  • pool: name of the pool (_default_ for the default pool)
Details

This metric is a distribution. Each measurement indicates approximately how much memory (in Kib) a successful action (ExecutionStatus: SUCCESS) on this worker reportedly used.

Only workers report this metric. All workers report their own values.

com.engflow.re.exec/used_executors

no unit
Number of busy executors, in all pools
Details

Only workers report this metric. All workers report their own values. You should sum up the time series to get the total number of busy executors in the cluster.

com.engflow.re.exec/used_executors_per_pool

no unit
Number of busy executors, per pool

Tags

  • pool: name of the pool (_default_ for the default pool)
Details

Only workers report this metric. All workers report their own values. You should sum up the time series to get the total number of busy executors in the pool/cluster.

Hazelcast monitoring

com.engflow.re.hazelcast.map/entries

no unit
The number of entries in Hazelcast maps map

Tags

  • cluster_name: name of the Hazelcast cluster

  • map_name: name of the Hazelcast map

com.engflow.re.hazelcast.map/memory_used

bytes
The amount of memory used for the map

Tags

  • cluster_name: name of the Hazelcast cluster

  • map_name: name of the Hazelcast map

com.engflow.re.hazelcast/is_master

no unit
Whether a machine is a cluster master; if this sums up to more than one (with the same name), then the cluster is unhealthy.

Tags

  • name: name of the Hazelcast cluster.

com.engflow.re.hazelcast/member_count

no unit
The number of members in the cluster; only the master reports this value

Tags

  • name: name of the Hazelcast cluster.

com.engflow.re.hazelcast/op_time

milliseconds
Distribution of operation time

Tags

  • name: name of the distributed hash map

  • status: op result

com.engflow.thirdparty.hazelcast/partition_migration_finished

no unit
The number of finished Hazelcast partition migrations.

Tags

  • name: name of the Hazelcast cluster

com.engflow.thirdparty.hazelcast/partition_migration_started

no unit
The number of started Hazelcast partition migrations.

Tags

  • name: name of the Hazelcast cluster

com.engflow.thirdparty.hazelcast/partition_migration_time

milliseconds
Time of Hazelcast partition migrations, per Hazelcast cluster.

Tags

  • name: name of the Hazelcast cluster
Details

Reports the time of Hazelcast partition migrations.

com.engflow.thirdparty.hazelcast/replica_migration

no unit
The number of Hazelcast replica migrations.

Tags

  • name: name of the Hazelcast cluster

  • status: status of the operation (OK or FAILURE)

Uncaught exceptions

com.engflow.re/uncaught_exceptions

no unit
Fires every time there is an uncaught exception

CAS server metrics

com.engflow.re.cas/missing_digests

no unit
The total number of missing digests seen by findMissingBlobs.

com.engflow.re.cas/requested_digests

no unit
The total number of digests requested by a findMissingBlob call

Remote Execution metrics

com.engflow.remoteexecution/queue_time

milliseconds
This is a distribution. Refers to the time actions are queued.

Tags

  • pool: The name of the pool.
Details

This is a distribution. Refers to the time actions are queued.

Invocation index monitoring

com.engflow.resultstore.index/sql_invocation_index_database_queue_size

no unit
All enqueued or in-progress invocation index database operations
Details

Reflects the number of incomplete operations (either queued or being worked on).

Every instance reports this metric. Every instance reports its own stats.

CAS usage

com.engflow.re.cas/check_blob_exists

milliseconds
Distribution of time needed to check whether a blob exists

Tags

  • status: operation result, e.g. OK, NOT_FOUND
Details

This is a distribution. Refers to the time needed to check whether a blob exists.

com.engflow.re.cas/fetch_call_time

milliseconds
Distribution of CAS fetch operation time

Tags

  • source: name of the CAS location, e.g. EXTERNAL_STORAGE, DISTRIBUTED_CAS_NEAR

  • status: op result, e.g. OK, UNAVAILABLE

Details

The time distribution of individual CAS download calls; each call is measured independently, including when falling back between different sources.

com.engflow.re.cas/fetch_retries

no unit
Count of retries needed when fetching a CAS blob.
Details

Count of retries needed when fetching a CAS blob. Incremented after each failure, so 0 indicates a fetch without errors.

com.engflow.re.cas/find_replicas_time

milliseconds
Distribution of time needed to find which instances have copies of a file
Details

Distribution of time needed to find which instances have copies of a file.

com.engflow.re.cas/load_shed_errors

no unit
Count of RESOURCE_EXHAUSTED errors returned by workers for CAS requests due to load shedding.

Tags

  • method: method returning the error, e.g., read
Details

Count of RESOURCE_EXHAUSTED errors returned by workers for CAS requests due to load shedding.

com.engflow.re.cas/remote_check_blob_exists

milliseconds
Distribution of time needed to check whether a blob exists

Tags

  • source: address used to check on cas blob.

  • status: operation result, e.g. OK, NOT_FOUND

Details

This is a distribution. Refers to the time needed to check whether a blob exists.

com.engflow.re.cas/requests_in_flight_incoming

no unit
Number of currently open incoming cache requests, by method and pool.

Tags

  • method: read or write

  • pool: name of the pool serving the request

Details

Number of currently open incoming cache requests, by method and pool.

com.engflow.re.cas/requests_in_flight_outgoing

no unit
Number of currently open outgoing cache requests, by method and pool.

Tags

  • method: read or write

  • pool: name of the pool originating the request

Details

Number of currently open outgoing cache requests, by method and pool. Includes both distributed CAS (ByteStream) and external storage.

com.engflow.re.cas/time_to_next_message

milliseconds
Estimated number of milliseconds to the next grpc message.

Tags

  • pool: name of the pool serving the request
Details

Estimated number of milliseconds to the next grpc message.

Local CAS usage

com.engflow.re.cas/available_replica_space

bytes
Available storage space in the CAS that can be used for replicas
Details

Only workers report this metric. All workers report their own values.

com.engflow.re.cas/available_space

bytes
Available storage space in the CAS
Details

Only workers report this metric. All workers report their own values.

com.engflow.re.cas/free_time

milliseconds
Distribution of time needed to free space in the CAS
Details

This is a distribution. It refers to the deletion of expired replicas.

Only workers report this metric. All workers report their own values.

com.engflow.re.cas/gc_time

milliseconds
Distribution of time needed for the GC
Details

This is a distribution. It refers to the collection of expired replicas.

Only workers report this metric. All workers report their own values.

com.engflow.re.cas/lost_files_count

no unit
The number of files that were lost from the CAS
Details

The number of files that were deleted by some other process or the CAS instance detected that they no longer matched the expected digest.

Only workers report this metric. All workers report their own values.

com.engflow.re.cas/max_total_replica_size

bytes
The max total replica size
Details

This is the maximum amount of storage space the CAS is allowed to use for replicas.

Only workers report this metric. All workers report their own values.

com.engflow.re.cas/max_total_size

bytes
The max total CAS size on the node
Details

This is the maximum amount of storage space the CAS is allowed to use.

Only workers report this metric. All workers report their own values.

Client authorization

com.engflow.re.auth.async/duration

milliseconds
Authentication call duration
Details

This is a distribution. Only schedulers report this metric. Every scheduler reports its own stats.

SecretStore metrics

com.engflow.secretstore/operation_duration_seconds

s
Time taken to perform an operation on the secret store

Tags

  • operation: The secret store operation reported ('read', etc.)

  • status: The gRPC status code string, in SCREAMING_SNAKE_CASE

  • store_type: The implementation of the instrumented secret store

External storage use

com.engflow.re.storage.existence_cache/evictions

no unit
Evictions from the ExternalStorage CAS existence cache

com.engflow.re.storage.existence_cache/hits

no unit
Hits on the ExternalStorage CAS existence cache

com.engflow.re.storage.existence_cache/misses

no unit
Misses on the ExternalStorage CAS existence cache

com.engflow.re.storage.readonly/gc_check

no unit
GC status updates

Tags

  • result

com.engflow.re.storage.readonly/ops

no unit
All completed external storage operations

Tags

  • operation

  • result

com.engflow.re.storage.readonly/ops_queue_size

no unit
All enqueued or in-progress external storage operations

Tags

  • operation

com.engflow.re.storage.readonly/traffic

bytes
All external storage traffic

Tags

  • operation

com.engflow.re.storage/gc_check

no unit
GC status updates

Tags

  • result

com.engflow.re.storage/gc_deleted_objects

no unit
count objects deleted for GC
Details

Logged when GC deletes an objects

com.engflow.re.storage/ops

no unit
All completed external storage operations

Tags

  • operation

  • result

com.engflow.re.storage/ops_queue_size

no unit
All enqueued or in-progress external storage operations

Tags

  • operation

com.engflow.re.storage/proxy_stall_time_ms

milliseconds
Total milliseconds reads are blocked by client flow control

com.engflow.re.storage/traffic

bytes
All external storage traffic

Tags

  • operation

Licensing metrics

com.engflow.licensing/license_server_fetch_result

no unit
The result of attempted license renewals using the MyEngFlow License Server.

Tags

  • status

Docker use

com.engflow.re.exec.docker/cas_fetch_seconds

s
The time spent pulling Docker images from the CAS. This includes fetching only

Tags

  • status: result of the operation, e.g. "OK", "FAILED"

com.engflow.re.exec.docker/cas_pull_bytes

bytes
The size of Docker images pulled from the CAS

Tags

  • status: result of the operation, e.g. "OK", "FAILED"

com.engflow.re.exec.docker/cas_pull_seconds

s
The time spent pulling Docker images from the CAS. This includes both fetching and loading

Tags

  • status: result of the operation, e.g. "OK", "FAILED"

com.engflow.re.exec.docker/cas_save_seconds

s
The time spent uploading Docker images to the CAS. This includes saving only

Tags

  • status: result of the operation, e.g. "OK", "FAILED"

com.engflow.re.exec.docker/cas_upload_bytes

bytes
The size of Docker images uploaded to the CAS

Tags

  • status: result of the operation, e.g. "OK", "FAILED"

com.engflow.re.exec.docker/cas_upload_seconds

s
The time spent uploading Docker images to the CAS. This includes both saving and uploading

Tags

  • status: result of the operation, e.g. "OK", "FAILED"

com.engflow.re.exec.docker/container_lifecycle_events_count

no unit
Count of various lifecycle events relating to containers

Tags

  • container_lifecycle_event_name

  • pool

com.engflow.re.exec.docker/container_shutdown_time

milliseconds
The time needed to shutdown a docker container

com.engflow.re.exec.docker/container_startup_time

milliseconds
The time needed to start a docker container

Tags

  • status: result of the operation, e.g. "OK", "FAILED"

com.engflow.re.exec.docker/containers_failed

no unit
The number of docker containers that failed

com.engflow.re.exec.docker/docker_proxy_failure_count

no unit
The number of times pulls through the docker proxy failed

com.engflow.re.exec.docker/image_pull_time

milliseconds
The time needed to pull a docker image

Tags

  • status: result of the operation, e.g. "OK", "FAILED"

com.engflow.re.exec.docker/network_create_time

milliseconds
The time needed to create a docker network

Tags

  • status: result of the operation, e.g. "OK", "FAILED"

com.engflow.re.exec.docker/network_destroy_time

milliseconds
The time needed to destroy a docker network

Tags

  • status: result of the operation, e.g. "OK", "FAILED"

com.engflow.re.exec.docker/sibling_container_enabled_total

no unit
Counts times that sibling container access was requested via platform properties

Tags

  • pool

Persistent worker use

com.engflow.re.exec.worker/actions

no unit
The number of persistent worker actions run

Tags

  • pool: name of the pool

  • reuse_status: new or reused

Details

The number of persistent worker actions run, aggregated by whether they reused a previous persistent worker process or not

Scheduler metrics

com.engflow.profiling/publish_invocation_event

milliseconds
Timing for publishing invocation events.

com.engflow.re.cas/entries_evicted

no unit
The number of CAS entries that were evicted due to memory size limitations

com.engflow.re.cas/entries_lost

no unit
The number of CAS entries that could not be recovered on CAS node shutdown events

com.engflow.re.profiler/events

no unit
The number of server-side profile events recorded.

com.engflow.re.profiler/live_handles

no unit
The number of profiles being streamed to the eventstore.

com.engflow.re/remaining_certificate_validity_days

days
The number of remaining days before certificates expire

Tags

  • issuer: The issuer (issuer distinguished name) value from the certificate

  • serial_number: The serial number assigned to the certificate by the issuer

Details

Reports the number of remaining validity days for each X509 certificate processed by schedulers. The issuer and serial number uniquely identify certificates.

com.engflow.re/remaining_license_time

days
The number of remaining days before the license expires

Java memory metrics

com.engflow.re/java_heap

bytes
The amount of heap memory used
Details

Every instance reports this metric. Every instance reports its own stats.

Meta metrics

com.engflow.meta/engflow_version

no unit
A heartbeat metric that reports the EngFlow build label if present and "missing_version" otherwise.

Tags

  • version

Docker daemon

com.engflow.docker.container/existing

no unit
The number of existing containers.

Tags

  • daemon: The docker daemon this metric is for (e.g., host).

  • pool: The pool this metric is for (e.g., default or macos).

  • state: The state of the container (e.g., running, exited, ...).

com.engflow.docker.container/size

bytes
The size distribution of existing docker containers.

Tags

  • daemon: The docker daemon this metric is for (e.g., host).

  • filesystem: The filesystem the value applies to (overlay if the value is the size of files modified by the container, and root if the value is the total size of the container including the image).

  • pool: The pool this metric is for (e.g., default or macos).

  • scope: The scope of the value (container if the value is the size of a single container, and daemon if the value is the sum of all containers known to the docker daemon).

com.engflow.docker.image/size

bytes
The size distribution of existing docker images.

Tags

  • daemon: The docker daemon this metric is for (e.g., host).

  • pool: The pool this metric is for (e.g., default or macos).

  • scope: The scope of the value (image if the value is the size of a single image, and daemon if the value is the sum of all images known to the docker daemon).

DB Connection Pool usage

com.engflow.resultstore.index/db_cp_active_connections

no unit
The number of active connections in the pool

Tags

  • db_connection_pool_name

com.engflow.resultstore.index/db_cp_connection_acquire_time

us
The time it takes for the connection pool to acquire a DB connection

Tags

  • db_connection_pool_name

com.engflow.resultstore.index/db_cp_connection_create_time

milliseconds
The time it takes for the connection pool to create a new DB connection

Tags

  • db_connection_pool_name

com.engflow.resultstore.index/db_cp_connection_timeout_count

no unit
The count of timed-out connections

Tags

  • db_connection_pool_name

com.engflow.resultstore.index/db_cp_connection_usage_time

milliseconds
The duration of a use of a connection given by the connection pool

Tags

  • db_connection_pool_name

com.engflow.resultstore.index/db_cp_idle_connections

no unit
The number of idle connections in the pool

Tags

  • db_connection_pool_name

com.engflow.resultstore.index/db_cp_max_connections

no unit
Maximum number of connections existing in the pool

Tags

  • db_connection_pool_name

com.engflow.resultstore.index/db_cp_min_connections

no unit
Minimum number of connections existing in the pool

Tags

  • db_connection_pool_name

com.engflow.resultstore.index/db_cp_pending_connections

no unit
The number of pending connections in the pool

Tags

  • db_connection_pool_name

com.engflow.resultstore.index/db_cp_total_connections

no unit
The number of all currently existing connections in the pool

Tags

  • db_connection_pool_name

Caffeine cache metrics

com.engflow.caching.inmemory/evictions

no unit
Number of cache evictions and the reason for eviction.

Tags

  • name: The name of the cache

  • reason: The reason for eviction

com.engflow.caching.inmemory/hits

no unit
Number of cache hits.

Tags

  • name: The name of the cache

com.engflow.caching.inmemory/loads

milliseconds
Timing and status information for cache loads.

Tags

  • name: The name of the cache

  • status: The status of the load

com.engflow.caching.inmemory/misses

no unit
Number of cache misses.

Tags

  • name: The name of the cache

DB Query stats

com.engflow.resultstore.index/duration

milliseconds
The duration of a query

Tags

  • query_name

  • query_outcome

com.engflow.resultstore.index/preparation

milliseconds
The duration of creating a preparedQuery

Tags

  • query_name

Outgoing CI API Calls

com.engflow.ci.http/api_call_latency

milliseconds
Time taken for an outgoing API call, indexed by hostname and status

Tags

  • hostname: hostname of the API service that was called

  • status: HTTP status of the call (1xx, 2xx, 3xx, 4xx, 5xx, or UNKNOWN)

Details

Indicates the number of API calls made, per service and their status, on this scheduler. The status only contains the abbreviated HTTP status code. Only schedulers report this metric. Every scheduler reports its own API calls.

CI runner

com.engflow.ci.basic/gh_error_propagation_jobs

no unit
Number of finished GitHub error propagation jobs

Tags

  • status: the status of the error propagation job
Details

Number of finished GitHub error propagation jobs.

com.engflow.ci.basic/gh_runner_no_jobs

no unit
Number of finished GitHub runs that timed out and did not pick up any jobs.
Details

Number of finished GitHub runs that timed out and did not pick up any jobs.

com.engflow.ci.basic/gh_runner_wrong_job

no unit
Number of finished GitHub runs that picked up the wrong job.
Details

Number of finished GitHub runs that picked up the wrong job.

com.engflow.ci.basic/jobs_completed

no unit
Number of jobs completed

Tags

  • status: job completion status
Details

The number of completed CI jobs, by status. This includes jobs for which we failed before starting the remote runner.

com.engflow.ci.basic/jobs_queue_age

milliseconds
Age of queued jobs

Tags

  • ci_family: name of the CI of the job

  • statistic: min/max queue age

Details

The age of queued jobs on GitHub/BuildKite. This metric is only reported if polling is enabled, and only for jobs that are configured for EngFlow CI runners. The value is how long ago was the job created relative to the current poll time.

com.engflow.ci.basic/jobs_queued

no unit
Number of queued jobs

Tags

  • ci_family: name of the CI of the job
Details

The number of queued jobs on GitHub/BuildKite. This metric is only reported if polling is enabled.

com.engflow.ci.basic/poll_duration_millis

milliseconds
Number of polls started

Tags

  • ci_family: name of the CI system to poll

  • status: the status of the polling job

Details

Measures the duration of polls against the remote CI system

com.engflow.ci.full/builtin_step_duration

milliseconds
The duration of individual built-in steps in a CI job

Tags

  • ci_family: name of the CI of the job

  • git_repository: name of the repository of the job

  • runner_architecture: architecture of the requested runner for a given job

  • runner_os: OS of the requested runner for a given job

  • step: job step name

Details

Measures the duration of a job's individual built-in (non-user-defined) steps' execution.

com.engflow.ci.full/git_command_duration

milliseconds
The duration of individual git commands during a CI job

Tags

  • ci_family: name of the CI of the job

  • git_command: git command: checkout/fetch/index-pack

  • git_repository: name of the repository of the job

  • job: job name

  • pipeline: name of the pipeline that triggered the job

  • runner_architecture: architecture of the requested runner for a given job

  • runner_os: OS of the requested runner for a given job

Details

Measures the duration of various git subcommands during a job's execution

com.engflow.ci.full/job_duration

milliseconds
The duration of CI jobs

Tags

  • ci_family: name of the CI of the job

  • git_repository: name of the repository of the job

  • job: job name

  • pipeline: name of the pipeline that triggered the job

  • runner_architecture: architecture of the requested runner for a given job

  • runner_os: OS of the requested runner for a given job

Details

Measures the duration of a job's execution.

com.engflow.ci.full/jobs_completed

no unit
Number of jobs completed

Tags

  • ci_family: name of the CI of the job

  • git_repository: name of the repository of the job

  • job: job name

  • pipeline: name of the pipeline that triggered the job

  • runner_architecture: architecture of the requested runner for a given job

  • runner_os: OS of the requested runner for a given job

  • status: job completion status

Details

The number of completed CI jobs, by status. This includes jobs for which we failed before starting the remote runner.

com.engflow.ci.full/jobs_started

no unit
Number of jobs started

Tags

  • ci_family: name of the CI of the job

  • git_repository: name of the repository of the job

  • job: job name

  • pipeline: name of the pipeline that triggered the job

  • runner_architecture: architecture of the requested runner for a given job

  • runner_os: OS of the requested runner for a given job

Details

The number of jobs started on this scheduler. Every scheduler reports its own jobs.

com.engflow.ci.full/step_duration

milliseconds
The duration of individual user-defined steps in a CI job

Tags

  • ci_family: name of the CI of the job

  • git_repository: name of the repository of the job

  • job: job name

  • pipeline: name of the pipeline that triggered the job

  • runner_architecture: architecture of the requested runner for a given job

  • runner_os: OS of the requested runner for a given job

  • step: job step name

Details

Measures the duration of a job's individual steps' execution.

com.engflow.ci.full/time_to_start_job

milliseconds
How long it takes for CI runners to start a job

Tags

  • ci_family: name of the CI of the job

  • git_repository: name of the repository of the job

  • job: job name

  • pipeline: name of the pipeline that triggered the job

  • runner_architecture: architecture of the requested runner for a given job

  • runner_os: OS of the requested runner for a given job

Details

Measures the time between a given CI requesting a job's execution and an action actually starting it on CI runners.

gRPC factory metrics

com.engflow.grpc.factory/channels

no unit
The number of open gRPC channels in the channel factory

CI runner - Github Action metrics

com.engflow.ci.github.metrics/api_primary_rate_limit_max

no unit
Maximum number of requests available in the current API rate limiter slot

Tags

  • API_RESOURCE_NAME

com.engflow.ci.github.metrics/api_primary_rate_limit_used

no unit
Number of requests used in the current API rate limiter slot

Tags

  • API_RESOURCE_NAME

RPC metrics

com.engflow.rpc/duration

s
Time distribution of RPCs

Tags

  • method: The RPC method

  • protocol: The protocol of the RPC (e.g., rest)

  • status: The status of the operation (e.g., OK)

  • type: The type of the RPC (e.g., client or server)