Metrics Reference¶

Description of all metrics available for monitoring.

AnalyzeInvocation metrics¶

com.engflow.invocationanalyzer/bazel_profile_count¶

no unit: The number of Bazel profiles that were attempted to be fetched

Tags

status: The status of retrieving the Bazel profile.

Details: The number of Bazel profiles that were attempted to be fetched.

com.engflow.invocationanalyzer/bazel_profile_size¶

bytes: The size of the uncompressed Bazel profile handled
Details: The size of the uncompressed Bazel profile handled.

com.engflow.invocationanalyzer/engflow_profile_count¶

no unit: The number of EngFlow profiles that were attempted to be fetched

Tags

status: The status of retrieving the EngFlow profile.

Details: The number of EngFlow profiles that were attempted to be fetched.

com.engflow.invocationanalyzer/engflow_profile_size¶

bytes: The size of the uncompressed EngFlow profile handled
Details: The size of the uncompressed EngFlow profile handled.

com.engflow.invocationanalyzer/time_needed¶

milliseconds: The time distribution of handing individual profile analysis requests

Tags

status: The status of the analysis performed

Details: The time distribution of handling individual Bazel profiles.

Opencensus OperationController metrics reporter¶

com.engflow.operationcontroller/active¶

no unit: The number of active operations (i.e., currently running).

Tags

name: The name of the OperationController

Details: The number of active operations (i.e., currently running).

com.engflow.operationcontroller/latency¶

milliseconds: The latency to start operations (i.e., how long operations are waiting to be executed).

Tags

name: The name of the OperationController

Details: The latency to start operations (i.e., how long operations are waiting to be executed).

com.engflow.operationcontroller/queued¶

no unit: The number of operations queued for execution.

Tags

name: The name of the OperationController

Details: The number of operations queued for execution.

com.engflow.operationcontroller/runtime¶

milliseconds: The runtime of operations (i.e., the duration operations are running for).

Tags

name: The name of the OperationController

Details: The runtime of operations (i.e., the duration operations are running for).

Metrics derived from raw BEP streams¶

com.engflow.bep/invocation_completed¶

no unit: Fired with the count of completed invocations reported to the BEP.

Tags

exit_code: The human readable exit code of the invocation.

com.engflow.bep/invocation_duration¶

milliseconds: Fired on invocation completed with the average duration of the invocation.

com.engflow.bep/invocation_started¶

no unit: Fired with the count of newly started invocations reported to the BEP.

Blob-storage implementation metrics¶

com.engflow.blobstore/ops¶

no unit: Fires every time an operation takes place.

Tags

operation

Docker proxy¶

com.engflow.dockerproxy/blob_upload_bytes¶

bytes: The size of Docker blobs that the proxy successfully uploaded to the CAS.

Tags

status

com.engflow.dockerproxy/cache_hit_bytes¶

bytes: The size of Docker blobs that the proxy could find in the CAS.

com.engflow.dockerproxy/cache_miss_bytes¶

bytes: The size of Docker blobs that the proxy expected but could not find in the CAS.

com.engflow.dockerproxy/known_blobs_total¶

no unit: The number of Docker blobs that the proxy has metadata about.

HTTP clients for the Docker proxy¶

com.engflow.dockerproxy/http_received_bytes_total¶

bytes: Bytes received over the HTTP client

Tags

client: The name of the HTTP client (can be used to distinguish layers)

com.engflow.dockerproxy/http_request_latency_seconds_total¶

s: Time it's taken to serve HTTP requests

Tags

client: The name of the HTTP client (can be used to distinguish layers)
status: The status code, reduced to 1xx..5xx, or FAILED if an exception occurred

com.engflow.dockerproxy/http_requests_total¶

no unit: Number of HTTP requests started on the HTTP client

Tags

client: The name of the HTTP client (can be used to distinguish layers)
method: The HTTP method of the request

BEP Event Storage and Replay¶

com.engflow.eventstore/build_event_owners¶

no unit: The total number of build owners residing in memory. A build owner is an internal piece of state associated to a particular build reporting build events.

com.engflow.eventstore/inbound_bep_events¶

no unit: Fired whenever an event is received on an inbound stream.

Tags

type

com.engflow.eventstore/new_outbound_streams¶

no unit: Fired whenever a new outbound BEP stream is read.

com.engflow.eventstore/ongoing_streams¶

no unit: The total number of streams that are inbound, outbound, or both.

com.engflow.eventstore/outbound_bep_events¶

no unit: Fired whenever an event is sent on an outbound stream.

Virtual Machine Instances¶

com.engflow.instance.new/gc_avg_duration¶

milliseconds: The average duration spent in garbage collection since the last reported metric.

Tags

gc_type: GC Old generation / GC Young Generation
instance_role: type of the instance (scheduler/worker/etc.)

com.engflow.instance.new/gc_count¶

no unit: The total number of garbage collections during the lifecycle of this process.

Tags

gc_type: GC Old generation / GC Young Generation
instance_role: type of the instance (scheduler/worker/etc.)

com.engflow.instance.new/gc_time¶

milliseconds: The total wall time in milliseconds spent blocked in garbage collection since the start of the process. This measures time when the application is not running due to a collector pause.

Tags

gc_type: G1 Old generation / G1 Young Generation
instance_role: type of the instance (scheduler/worker/etc.)

com.engflow.instance.new/open_file_descriptors¶

no unit: The number of file descriptors the process has currently open.

Tags

instance_role: type of the instance (scheduler/worker/etc.)

com.engflow.instance.new/total_disk_space¶

bytes: The size of the volume.

Tags

instance_role: type of the instance (scheduler/worker/etc.)
volume: the name of the disk volume

com.engflow.instance.new/total_system_memory¶

bytes: The total amount of system memory in bytes.

Tags

instance_role: type of the instance (scheduler/worker/etc.)

com.engflow.instance.new/used_disk_percentage¶

percentage: The percentage of the volume that is currently used.

Tags

instance_role: type of the instance (scheduler/worker/etc.)
volume: the name of the disk volume

com.engflow.instance.new/used_disk_space¶

bytes: The total number of bytes used on the volume.

Tags

instance_role: type of the instance (scheduler/worker/etc.)
volume: the name of the disk volume

com.engflow.instance.new/used_process_native_buffer_memory¶

bytes: The total amount of native buffer memory for this process in bytes.

Tags

instance_role: type of the instance (scheduler/worker/etc.)

com.engflow.instance.new/used_system_memory¶

bytes: The amount of used system memory in bytes.

Tags

instance_role: type of the instance (scheduler/worker/etc.)

com.engflow.instance.new/used_system_memory_percentage¶

percentage: The percentage of system memory used.

Tags

instance_role: type of the instance (scheduler/worker/etc.)

com.engflow.instance/gc_avg_duration¶

milliseconds: The average duration spent in garbage collection since the last reported metric.

Tags

gc_type

com.engflow.instance/gc_count¶

no unit: The total number of garbage collections during the lifecycle of this process.

Tags

gc_type

com.engflow.instance/gc_time¶

milliseconds: The total estimated time in milliseconds performing garbage collection.

Tags

gc_type

com.engflow.instance/total_disk_space¶

bytes: The size of the volume.

Tags

volume

com.engflow.instance/total_system_memory¶

bytes: The total amount of system memory in bytes.

com.engflow.instance/used_disk_percentage¶

percentage: The percentage of the volume that is currently used.

Tags

volume

com.engflow.instance/used_disk_space¶

bytes: The total number of bytes used on the volume.

Tags

volume

com.engflow.instance/used_system_memory¶

bytes: The amount of used system memory in bytes.

com.engflow.instance/used_system_memory_percentage¶

percentage: The percentage of system memory used.

Netty monitoring¶

com.engflow.thirdparty.netty/used_direct_memory¶

bytes: Direct (non-heap) memory use

Tags

buffer_name

com.engflow.thirdparty.netty/used_heap_memory¶

bytes: Heap memory use

Tags

buffer_name

io.netty.buffer/used_direct_memory¶

bytes: Direct (non-heap) memory use

Tags

buffer_name

io.netty.buffer/used_heap_memory¶

bytes: Heap memory use

Tags

buffer_name

Action scheduling¶

com.engflow.re.scheduler/autoscaler_set_size_operations¶

no unit: Number of desired executors, per pool

Tags

pool: name of pool ("default" for the default pool)
status: "succeeded" or "failed"

Details: Count of builtin autoscaler attempts to set desired size of a pool

com.engflow.re.scheduler/available_workers¶

no unit: Deprecated; number of idle executors, per pool

Tags

name: name of the pool ("default" for the default pool)

Details: Deprecated. Indicates the number of idle executors, per pool, according to this scheduler. Only schedulers report this metric. Every scheduler reports the same (or about the same) value. This metric is deprecated, because it may be imprecise: schedulers that are started while workers are busy may report a higher value than they should for several minutes. We recommend monitoring com.engflow.re.scheduler/existing_executors instead.

com.engflow.re.scheduler/coalesced_executions¶

no unit: Number of action requests of coalesced into an existing execution
Details: Number of action requests of coalesced into an existing execution

com.engflow.re.scheduler/desired_executors¶

no unit: Number of desired executors, per pool

Tags

name: name of the pool ("default" for the default pool)

Details: Indicates an estimate for the number of required executors per pool. Every scheduler reports its own estimate - they should be summed up to get the total desired pool size.

com.engflow.re.scheduler/existing_executors¶

no unit: Number of existing executors, per pool

Tags

name: name of the pool ("default" for the default pool)

Details: Number of existing executors, per pool, according to this scheduler. Only schedulers report this metric. Every scheduler reports the same (or about the same) value.

com.engflow.re.scheduler/existing_schedulers¶

no unit: Number of existing schedulers
Details: Only schedulers report this metric. Every scheduler reports a constant "1". This can be used to detect schedulers that are unable to send monitoring metrics.

com.engflow.re.scheduler/pool_utilization¶

percentage: Current executor utilization, per pool

Tags

name: name of the pool ("default" for the default pool)

Details: Reports current executor utilization (used*100/total) per pool, as a percentage ([0..100]). Only schedulers report this metric. Every scheduler reports the same (or about the same) value.

To help making scale-up decisions when a pool is empty, utilization is reported as 100 if there are actions waiting and 0 if not. A pool may be empty if it was scaled down, or if it never existed (the client may request any pool name).

com.engflow.re.scheduler/queue_age¶

milliseconds: Min/max age of queued actions, per pool

Tags

name: name of the pool ("default" for the default pool)
statistic: "min" (youngest) or "max" (oldest) action in the pool's queue

Details: Reports minimum and maximum age in each executor pool, i.e. how long entries have been waiting. Only schedulers report this metric. Every scheduler reports its own queue lengths. Changes in these values indicate a change in the cluster's throughput.

com.engflow.re.scheduler/queue_size¶

no unit: Number of waiting actions, per pool

Tags

name: name of the pool ("default" for the default pool)

Details: Indicates the number of actions waiting for execution, per pool, on this scheduler. Only schedulers report this metric. Every scheduler reports its own queue lengths.

Observability UI metrics¶

com.engflow.observability.ui/app_load¶

milliseconds: The duration of the initial application load.

Tags

page

com.engflow.observability.ui/caught_error¶

no unit: An error in the web client and was manually caught and reported.

Tags

error_description
page

com.engflow.observability.ui/navigation¶

no unit: A single user navigation to a new page.

Tags

page

com.engflow.observability.ui/uncaught_error¶

no unit: An uncaught error was thrown in the web client.

Tags

error_name
page

ResultStore metrics¶

com.engflow.resultstore/reduce_bes_count¶

no unit: The number of BES replays for reducing to InvocationState which are currently running or recently completed.

com.engflow.resultstore/reduce_bes_target_metadata_count¶

no unit: The number of target metadatas in memory for reducing to InvocationState.

com.engflow.resultstore/reduce_bes_target_tree_node_count¶

no unit: The number of current target tree nodes in memory for reducing to InvocationState.

com.engflow.resultstore/reduce_bes_target_tree_outdated_node_count¶

no unit: The number of outdated target tree nodes that are cached in memory while reducing to InvocationState.

Storage implementation metrics¶

com.engflow.storage.read/size¶

bytes: Number of file bytes sent to the client for a read request. May be smaller than the file size in case of error or partial read. Only recorded if the file was found.

Tags

name: name of the storage service
status: op result

com.engflow.storage.read/time_per_gb¶

milliseconds: Time taken per 1 billion bytes (1 GB) to download a file from storage.

Tags

name: name of the storage service
status: op result

com.engflow.storage.read/time_to_first_byte¶

milliseconds: Time taken between initiating a download to receiving the first byte.

Tags

name: name of the storage service
status: op result

com.engflow.storage.read/time_to_next_chunk¶

milliseconds: Time taken between being notified that the client is ready and sending the next response. May be recorded 0 or multiples times for the same call, depending on control flow events.

Tags

name: name of the storage service
status: op result

com.engflow.storage.write/size¶

bytes: Number of file bytes received from the client for a write request. May be smaller than the file size in case of error.

Tags

name: name of the storage service
status: op result

com.engflow.storage.write/time_per_gb¶

milliseconds: Time taken per 1 billion bytes (1 GB) to upload a file to storage.

Tags

name: name of the storage service
status: op result

com.engflow.storage.write/time_to_commit¶

milliseconds: Time between being notified the write is complete and committing the write.

Tags

name: name of the storage service
status: op result

NotificationQueue metrics¶

com.engflow.notificationqueue/dequeue_latency¶

milliseconds: This is a distribution. Refers to the time passed between notification creation and dequeuing it.

Tags

expired: Whether the notification was expired and discarded.
name: The name of the queue.

com.engflow.notificationqueue/head_age¶

milliseconds: The age of the first notification in the queue, i.e. the time passed since the notification was created. Notably, it does NOT reflect how much time has passed since the notification was last published. For example, if a notification is not acknowledged and published anew, the head age may be disproportionately high compared to the age of the next notifications in the queue. This can lead to acceptable, occasional spikes.

Tags

name: The name of the queue.

com.engflow.notificationqueue/publish¶

milliseconds: This is a distribution. Refers to the time needed to publish a notification.

Tags

name: The name of the queue.
status: The status of the operation.

com.engflow.notificationqueue/size¶

no unit: The approximate size of the queue.

Tags

name: The name of the queue.

Action execution¶

com.engflow.re.exec/completed_actions¶

no unit: Number of actions that ran to completion, grouped by exit code

Tags

exit_code: the action's exit code

Details: This metric reflects the rate of change. Each measurement indicates how many actions completed on this worker, in all pools combined, since the last time this metric was reported.

Only workers report this metric. All workers report their own values. We recommend grouping by exit_code=0 and exit_code!=0, and summing up the time series in the groups. This yields the rate of successful and unsuccessful action completion across the cluster.

com.engflow.re.exec/completed_actions_per_pool¶

no unit: Number of executed actions (not cached), grouped by pool and status

Tags

pool: name of the pool (_default_ for the default pool)
status: the action's status (ExecutionStatus: SUCCESS, NON_ZERO_EXIT, ERROR)

Details: This metric reflects the rate of change. Each measurement indicates how many actions completed on this worker, per pool, since the last time this metric was reported.

Only workers report this metric. All workers report their own values. We recommend grouping by status, and summing up the time series in the groups. This yields the rate of successful and unsuccessful action completion across the pool/cluster.

com.engflow.re.exec/execution_latency¶

milliseconds: Bucketed latency (ms), grouped by pool and execution stage

Tags

pool: name of the pool (_default_ for the default pool)
stage: the action's stage (ExecutionStage: QUEUED, DOWNLOAD_INPUTS, EXECUTE_ACTION, UPLOAD_OUTPUTS, EXECUTOR_TOTAL)

com.engflow.re.exec/executors_existing¶

no unit: Total number of executors on this worker, in all pools combined
Details: Only workers report this metric. All workers report their own values. You should sum up the time series to get the total number of executors in the cluster.

com.engflow.re.exec/executors_existing_per_pool¶

no unit: Total number of executors on this worker, per pool

Tags

pool: name of the pool (_default_ for the default pool)

Details: Only workers report this metric. All workers report their own values. You should sum up the time series to get the total number of executors in the pool/cluster.

com.engflow.re.exec/used_executors¶

no unit: Number of busy executors, in all pools
Details: Only workers report this metric. All workers report their own values. You should sum up the time series to get the total number of busy executors in the cluster.

com.engflow.re.exec/used_executors_per_pool¶

no unit: Number of busy executors, per pool

Tags

pool: name of the pool (_default_ for the default pool)

Details: Only workers report this metric. All workers report their own values. You should sum up the time series to get the total number of busy executors in the pool/cluster.

Hazelcast monitoring¶

com.engflow.re.hazelcast.map/entries¶

no unit: The number of entries in Hazelcast maps map

Tags

cluster_name: name of the Hazelcast cluster
map_name: name of the Hazelcast map

com.engflow.re.hazelcast.map/memory_used¶

bytes: The amount of memory used for the map

Tags

cluster_name: name of the Hazelcast cluster
map_name: name of the Hazelcast map

com.engflow.re.hazelcast/is_master¶

no unit: Whether a machine is a cluster master; if this sums up to more than one (with the same name), then the cluster is unhealthy.

Tags

name: name of the Hazelcast cluster.

com.engflow.re.hazelcast/member_count¶

no unit: The number of members in the cluster; only the master reports this value

Tags

name: name of the Hazelcast cluster.

com.engflow.re.hazelcast/op_time¶

milliseconds: Distribution of operation time

Tags

name: name of the distributed hash map
status: op result

com.engflow.thirdparty.hazelcast/partition_migration_finished¶

no unit: The number of finished Hazelcast partition migrations.

Tags

name: name of the Hazelcast cluster

com.engflow.thirdparty.hazelcast/partition_migration_started¶

no unit: The number of started Hazelcast partition migrations.

Tags

name: name of the Hazelcast cluster

com.engflow.thirdparty.hazelcast/partition_migration_time¶

milliseconds: Time of Hazelcast partition migrations, per Hazelcast cluster.

Tags

name: name of the Hazelcast cluster

Details: Reports the time of Hazelcast partition migrations.

com.engflow.thirdparty.hazelcast/replica_migration¶

no unit: The number of Hazelcast replica migrations.

Tags

name: name of the Hazelcast cluster
status: status of the operation (OK or FAILURE)

Uncaught exceptions¶

com.engflow.re/uncaught_exceptions¶

no unit: Fires every time there is an uncaught exception

CAS server metrics¶

com.engflow.re.cas/missing_digests¶

no unit: The total number of missing digests seen by findMissingBlobs.

com.engflow.re.cas/requested_digests¶

no unit: The total number of digests requested by a findMissingBlob call

Remote Execution metrics¶

com.engflow.remoteexecution/queue_time¶

milliseconds: This is a distribution. Refers to the time actions are queued.

Tags

pool: The name of the pool.

Details: This is a distribution. Refers to the time actions are queued.

Invocation index monitoring¶

com.engflow.resultstore.index/sql_invocation_index_database_queue_size¶

no unit: All enqueued or in-progress invocation index database operations
Details: Reflects the number of incomplete operations (either queued or being worked on).

Every instance reports this metric. Every instance reports its own stats.

CAS usage¶

com.engflow.re.cas/check_blob_exists¶

milliseconds: Distribution of time needed to check whether a blob exists

Tags

status: operation result, e.g. OK, NOT_FOUND

Details: This is a distribution. Refers to the time needed to check whether a blob exists.

com.engflow.re.cas/fetch_call_time¶

milliseconds: Distribution of CAS fetch operation time

Tags

source: name of the CAS location, e.g. EXTERNAL_STORAGE, DISTRIBUTED_CAS
status: op result, e.g. OK, UNAVAILABLE

Details: The time distribution of individual CAS download calls; each call is measured independently, including when falling back between different sources.

Local CAS usage¶

com.engflow.re.cas/available_replica_space¶

bytes: Available storage space in the CAS that can be used for replicas
Details: Only workers report this metric. All workers report their own values.