Version 1.43 of the documentation is no longer actively maintained. The site that you are currently viewing is an archived snapshot. For up-to-date documentation, see the latest version.

Metrics Reference

Description of all metrics available for monitoring

Action scheduling

com.engflow.re.scheduler/availability_map_size (no unit)

Number of busy executors.

Details: Indicates the total number of busy executors, across all pools. Only schedulers report this metric. Every scheduler reports the same (or about the same) value. Do not sum up values from multiple schedulers!

com.engflow.re.scheduler/available_workers (no unit)

Number of idle executors, per pool.

Tags

name: name of the pool ("_default_" for the default pool)

Details: Indicates the number of idle executors, per pool, according to this scheduler. Only schedulers report this metric. Every scheduler reports the same (or about the same) value.

com.engflow.re.scheduler/existing_schedulers (no unit)

Number of existing schedulers.

Details: Only schedulers report this metric. Every scheduler reports a constant "1". This can be used to detect schedulers that are unable to send monitoring metrics.

com.engflow.re.scheduler/owner_map_size (no unit)

Number of entries.

Details: Deprecated. This is rarely useful and measures an internal data structure that is subject to change. Schedulers use the owner map to keep track of actions. The size of this map indicates how many actions are being executed.

com.engflow.re.scheduler/pool_utilization (percentage)

Current executor utilization, per pool.

Tags

name: name of the pool ("_default_" for the default pool)

Details

Reports current executor utilization (used*100/total) per pool, as a percentage ([0..100]). Only schedulers report this metric. Every scheduler reports the same (or about the same) value.

To help making scale-up decisions when a pool is empty, utilization is reported as 100 if there are actions waiting and 0 if not. A pool may be empty if it was scaled down, or if it never existed (the client may request any pool name).

com.engflow.re.scheduler/queue_age (milliseconds)

Min/max age of queued actions, per pool.

Tags

name: name of the pool ("_default_" for the default pool)
statistic: "min" (youngest) or "max" (oldest) action in the pool's queue

Details: Reports minimum and maximum age in each executor pool, i.e. how long entries have been waiting. Only schedulers report this metric. Every scheduler reports its own queue lengths. Changes in these values indicate a change in the cluster's throughput.

com.engflow.re.scheduler/queue_size (no unit)

Number of waiting actions, per pool.

Tags

name: name of the pool ("_default_" for the default pool)

Details: Indicates the number of actions waiting for execution, per pool, on this scheduler. Only schedulers report this metric. Every scheduler reports its own queue lengths.

Action execution

com.engflow.re.exec/completed_actions (no unit)

Number of actions that ran to completion, grouped by exit code.

Tags

exit_code: the action's exit code

Details

This metric reflects the rate of change. Each measurement indicates how many actions completed on this worker, in all pools combined, since the last time this metric was reported.

Only workers report this metric. All workers report their own values. We recommend grouping by exit_code=0 and exit_code!=0, and summing up the time series in the groups. This yields the rate of successful and unsuccessful action completion across the cluster.

com.engflow.re.exec/executors_existing (no unit)

Total number of executors on this worker, in all pools combined.

Details: Only workers report this metric. All workers report their own values. You should sum up the time series to get the total number of executors in the cluster.

com.engflow.re.exec/running_actions (no unit)

Number of running actions.

Details: Deprecated. This metric measured something between executor usage and actual action runtime, but what it actually measured was not well-defined. Use com.engflow.re.exec/used_executors to measure executor usage.

com.engflow.re.exec/used_executors (no unit)

Number of busy executors, in all pools.

Details: Only workers report this metric. All workers report their own values. You should sum up the time series to get the total number of busy executors in the cluster.

CAS usage

com.engflow.re.cas/available_replica_space (bytes)

Available storage space in the CAS that can be used for replicas.

Details: Only workers report this metric. All workers report their own values.

com.engflow.re.cas/available_space (bytes)

Available storage space in the CAS.

Details: Only workers report this metric. All workers report their own values.

com.engflow.re.cas/free_time (milliseconds)

Distribution of time needed to free space in the CAS.

Details

This is a distribution. It refers to the deletion of expired replicas.

Only workers report this metric. All workers report their own values.

CloudWatch note: as of 2021-02-04 this metric is not reported to CloudWatch.

com.engflow.re.cas/gc_time (milliseconds)

Distribution of time needed for the GC.

Details

This is a distribution. It refers to the collection of expired replicas.

Only workers report this metric. All workers report their own values.

CloudWatch note: as of 2021-02-04 this metric is not reported to CloudWatch.

com.engflow.re.cas/lost_files_count (no unit)

The number of files that were lost from the CAS.

Details

The number of files that were deleted by some other process or the CAS instance detected that they no longer matched the expected digest.

Only workers report this metric. All workers report their own values.

com.engflow.re.cas/max_total_replica_size (bytes)

The max total replica size.

Details

This is the maximum amount of storage space the CAS is allowed to use for replicas.

Only workers report this metric. All workers report their own values.

com.engflow.re.cas/max_total_size (bytes)

The max total CAS size on the node.

Details

This is the maximum amount of storage space the CAS is allowed to use.

Only workers report this metric. All workers report their own values.

com.engflow.re.cas/total_replica_size (bytes)

The total replica size.

Details: Deprecated, please use com.engflow.re.cas/available_replica_space instead. This is the total replica size.

com.engflow.re.cas/total_size (bytes)

Total CAS size.

Details: Deprecated, please use com.engflow.re.cas/available_space instead. Combined size of all files stored in the CAS.

Client authorization

com.engflow.re.auth.async/call_count (no unit)

Number of calls made.

Details

Deprecated. Though it may seem so, this metric doesn't actually track client connection attempts accurately.

Use com.engflow.re.auth.async/duration aggregated by count instead.

com.engflow.re.auth.async/duration (milliseconds)

Authentication call duration.

Details

This is a distribution. Only schedulers report this metric. Every scheduler reports its own stats.

CloudWatch note: as of 2021-02-04 this metric is not reported to CloudWatch.

External storage use

com.engflow.re.storage/ops (no unit)

All completed external storage operations.

Tags

operation: the type of operation, e.g. "cas_check", "ac_upload"
result: the result of the operation, e.g. "successful", "interrupted"

Details

This metric reflects the rate of change. Each measurement indicates how many operations completed on this instance since the last time this metric was reported.

Every instance reports this metric. Every instance reports its own stats.

com.engflow.re.storage/ops_queue_size (no unit)

All enqueued or in-progress external storage operations.

Tags

operation: the type of operation, e.g. "cas_check", "ac_upload"

Details

Reflects the number of incomplete operations (either queued or being worked on).

Every instance reports this metric. Every instance reports its own stats.

com.engflow.re.storage/traffic (bytes)

All external storage traffic.

Tags

operation

Details

This metric may be imprecise; the source of truth is the set of metrics published by the storage backend itself.

Every instance reports this metric. Every instance reports its own stats.