Calculate percentages with Flux

Calculating percentages from queried data is a common use case for time series data. To calculate a percentage in Flux, operands must be in each row. Use map() to re-map values in the row and calculate a percentage.

To calculate percentages

  1. Use from(), range() and filter() to query operands.
  2. Use pivot() or join() to align operand values into rows.
  3. Use map() to divide the numerator operand value by the denominator operand value and multiply by 100.

The following examples use pivot() to align operands into rows because pivot() works in most cases and is more performant than join(). See Pivot vs join.

  1. from(bucket: "example-bucket")
  2. |> range(start: -1h)
  3. |> filter(fn: (r) => r._measurement == "m1" and r._field =~ /field[1-2]/ )
  4. |> pivot(rowKey:["_time"], columnKey: ["_field"], valueColumn: "_value")
  5. |> map(fn: (r) => ({ r with _value: r.field1 / r.field2 * 100.0 }))

GPU monitoring example

The following example queries data from the gpu-monitor bucket and calculates the percentage of GPU memory used over time. Data includes the following:

  • gpu measurement
  • mem_used field: used GPU memory in bytes
  • mem_total field: total GPU memory in bytes

Query mem_used and mem_total fields

  1. from(bucket: "gpu-monitor")
  2. |> range(start: 2020-01-01T00:00:00Z)
  3. |> filter(fn: (r) => r._measurement == "gpu" and r._field =~ /mem_/)
Returns the following stream of tables:
_time_measurement_field_value
2020-01-01T00:00:00Zgpumem_used2517924577
2020-01-01T00:00:10Zgpumem_used2695091978
2020-01-01T00:00:20Zgpumem_used2576980377
2020-01-01T00:00:30Zgpumem_used3006477107
2020-01-01T00:00:40Zgpumem_used3543348019
2020-01-01T00:00:50Zgpumem_used4402341478
_time_measurement_field_value
2020-01-01T00:00:00Zgpumem_total8589934592
2020-01-01T00:00:10Zgpumem_total8589934592
2020-01-01T00:00:20Zgpumem_total8589934592
2020-01-01T00:00:30Zgpumem_total8589934592
2020-01-01T00:00:40Zgpumem_total8589934592
2020-01-01T00:00:50Zgpumem_total8589934592

Pivot fields into columns

Use pivot() to pivot the mem_used and mem_total fields into columns. Output includes mem_used and mem_total columns with values for each corresponding _time.

  1. // ...
  2. |> pivot(rowKey:["_time"], columnKey: ["_field"], valueColumn: "_value")
Returns the following:
_time_measurementmem_usedmem_total
2020-01-01T00:00:00Zgpu25179245778589934592
2020-01-01T00:00:10Zgpu26950919788589934592
2020-01-01T00:00:20Zgpu25769803778589934592
2020-01-01T00:00:30Zgpu30064771078589934592
2020-01-01T00:00:40Zgpu35433480198589934592
2020-01-01T00:00:50Zgpu44023414788589934592

Map new values

Each row now contains the values necessary to calculate a percentage. Use map() to re-map values in each row. Divide mem_used by mem_total and multiply by 100 to return the percentage.

To return a precise float percentage value that includes decimal points, the example below casts integer field values to floats and multiplies by a float value (100.0).

  1. // ...
  2. |> map(
  3. fn: (r) => ({
  4. _time: r._time,
  5. _measurement: r._measurement,
  6. _field: "mem_used_percent",
  7. _value: float(v: r.mem_used) / float(v: r.mem_total) * 100.0
  8. }),
  9. )
Query results:
_time_measurement_field_value
2020-01-01T00:00:00Zgpumem_used_percent29.31
2020-01-01T00:00:10Zgpumem_used_percent31.37
2020-01-01T00:00:20Zgpumem_used_percent30.00
2020-01-01T00:00:30Zgpumem_used_percent35.00
2020-01-01T00:00:40Zgpumem_used_percent41.25
2020-01-01T00:00:50Zgpumem_used_percent51.25

Full query

  1. from(bucket: "gpu-monitor")
  2. |> range(start: 2020-01-01T00:00:00Z)
  3. |> filter(fn: (r) => r._measurement == "gpu" and r._field =~ /mem_/ )
  4. |> pivot(rowKey:["_time"], columnKey: ["_field"], valueColumn: "_value")
  5. |> map(
  6. fn: (r) => ({
  7. _time: r._time,
  8. _measurement: r._measurement,
  9. _field: "mem_used_percent",
  10. _value: float(v: r.mem_used) / float(v: r.mem_total) * 100.0
  11. }),
  12. )

Examples

Calculate percentages using multiple fields

  1. from(bucket: "example-bucket")
  2. |> range(start: -1h)
  3. |> filter(fn: (r) => r._measurement == "example-measurement")
  4. |> filter(fn: (r) => r._field == "used_system" or r._field == "used_user" or r._field == "total")
  5. |> pivot(rowKey: ["_time"], columnKey: ["_field"], valueColumn: "_value")
  6. |> map(
  7. fn: (r) => ({
  8. r with _value: float(v: r.used_system + r.used_user) / float(v: r.total) * 100.0
  9. }),
  10. )

Calculate percentages using multiple measurements

  1. Ensure measurements are in the same bucket.
  2. Use filter() to include data from both measurements.
  3. Use group() to ungroup data and return a single table.
  4. Use pivot() to pivot fields into columns.
  5. Use map() to re-map rows and perform the percentage calculation.
  1. from(bucket: "example-bucket")
  2. |> range(start: -1h)
  3. |> filter(fn: (r) => (r._measurement == "m1" or r._measurement == "m2") and (r._field == "field1" or r._field == "field2"))
  4. |> group()
  5. |> pivot(rowKey: ["_time"], columnKey: ["_field"], valueColumn: "_value")
  6. |> map(fn: (r) => ({r with _value: r.field1 / r.field2 * 100.0}))

Calculate percentages using multiple data sources

  1. import "sql"
  2. import "influxdata/influxdb/secrets"
  3. pgUser = secrets.get(key: "POSTGRES_USER")
  4. pgPass = secrets.get(key: "POSTGRES_PASSWORD")
  5. pgHost = secrets.get(key: "POSTGRES_HOST")
  6. t1 = sql.from(
  7. driverName: "postgres",
  8. dataSourceName: "postgresql://${pgUser}:${pgPass}@${pgHost}",
  9. query: "SELECT id, name, available FROM example_table",
  10. )
  11. t2 = from(bucket: "example-bucket")
  12. |> range(start: -1h)
  13. |> filter(fn: (r) => r._measurement == "example-measurement" and r._field == "example-field")
  14. join(tables: {t1: t1, t2: t2}, on: ["id"])
  15. |> map(fn: (r) => ({r with _value: r._value_t2 / r.available_t1 * 100.0}))