Create custom aggregate functions

To aggregate your data, use the Flux aggregate functions or create custom aggregate functions using the reduce()function.

Aggregate function characteristics

Aggregate functions all have the same basic characteristics:

  • They operate on individual input tables and transform all records into a single record.
  • The output table has the same group key as the input table.

How reduce() works

The reduce() function operates on one row at a time using the function defined in the fn parameter. The fn function maps keys to specific values using two records specified by the following parameters:

ParameterDescription
rA record that represents the row or record.
accumulatorA record that contains values used in each row’s aggregate calculation.

The reduce() function’s identity parameter defines the initial accumulator record.

Example reduce() function

The following example reduce() function produces a sum and product of all values in an input table.

  1. |> reduce(
  2. fn: (r, accumulator) => ({
  3. sum: r._value + accumulator.sum,
  4. product: r._value * accumulator.product
  5. }),
  6. identity: {sum: 0.0, product: 1.0},
  7. )

To illustrate how this function works, take this simplified table for example:

_time_value
2019-04-23T16:10:49Z1.6
2019-04-23T16:10:59Z2.3
2019-04-23T16:11:09Z0.7
2019-04-23T16:11:19Z1.2
2019-04-23T16:11:29Z3.8
Input records

The fn function uses the data in the first row to define the r record. It defines the accumulator record using the identity parameter.

  1. r = { _time: 2019-04-23T16:10:49.00Z, _value: 1.6 }
  2. accumulator = { sum : 0.0, product : 1.0 }
Key mappings

It then uses the r and accumulator records to populate values in the key mappings:

  1. // sum: r._value + accumulator.sum
  2. sum: 1.6 + 0.0
  3. // product: r._value * accumulator.product
  4. product: 1.6 * 1.0
Output record

This produces an output record with the following key value pairs:

  1. { sum: 1.6, product: 1.6 }

The function then processes the next row using this output record as the accumulator.

Because reduce() uses the output record as the accumulator when processing the next row, keys mapped in the fn function must match keys in the identity and accumulator records.

Processing the next row
  1. // Input records for the second row
  2. r = { _time: 2019-04-23T16:10:59.00Z, _value: 2.3 }
  3. accumulator = { sum : 1.6, product : 1.6 }
  4. // Key mappings for the second row
  5. sum: 2.3 + 1.6
  6. product: 2.3 * 1.6
  7. // Output record of the second row
  8. { sum: 3.9, product: 3.68 }

It then uses the new output record as the accumulator for the next row. This cycle continues until all rows in the table are processed.

Final output record and table

After all records in the table are processed, reduce() uses the final output record to create a transformed table with one row and columns for each mapped key.

Final output record
  1. { sum: 9.6, product: 11.74656 }
Output table
sumproduct
9.611.74656

What happened to the _time column?

The reduce() function only keeps columns that are:

  1. Are part of the input table’s group key.
  2. Explicitly mapped in the fn function.

It drops all other columns. Because _time is not part of the group key and is not mapped in the fn function, it isn’t included in the output table.

Custom aggregate function examples

To create custom aggregate functions, use principles outlined in Creating custom functions and the reduce() function to aggregate rows in each input table.

Create a custom average function

This example illustrates how to create a function that averages values in a table. This is meant for demonstration purposes only. The built-in mean() function does the same thing and is much more performant.

Comments No Comments

  1. average = (tables=<-, outputField="average") => tables
  2. |> reduce(
  3. // Define the initial accumulator record
  4. identity: {count: 0.0, sum: 0.0, avg: 0.0},
  5. fn: (r, accumulator) => ({
  6. // Increment the counter on each reduce loop
  7. count: accumulator.count + 1.0,
  8. // Add the _value to the existing sum
  9. sum: accumulator.sum + r._value,
  10. // Divide the existing sum by the existing count for a new average
  11. avg: (accumulator.sum + r._value) / (accumulator.count + 1.0),
  12. }),
  13. )
  14. // Drop the sum and the count columns since they are no longer needed
  15. |> drop(columns: ["sum", "count"])
  16. // Set the _field column of the output table to to the value
  17. // provided in the outputField parameter
  18. |> set(key: "_field", value: outputField)
  19. // Rename avg column to _value
  20. |> rename(columns: {avg: "_value"})
  1. average = (tables=<-, outputField="average") => tables
  2. |> reduce(
  3. identity: {count: 0.0, sum: 0.0, avg: 0.0},
  4. fn: (r, accumulator) => ({
  5. count: accumulator.count + 1.0,
  6. sum: accumulator.sum + r._value,
  7. avg: (accumulator.sum + r._value) / (accumulator.count + 1.0),
  8. }),
  9. )
  10. |> drop(columns: ["sum", "count"])
  11. |> set(key: "_field", value: outputField)
  12. |> rename(columns: {avg: "_value"})

Aggregate multiple columns

Built-in aggregate functions only operate on one column. Use reduce() to create a custom aggregate function that aggregates multiple columns.

The following function expects input tables to have c1_value and c2_value columns and generates an average for each.

  1. multiAvg = (tables=<-) => tables
  2. |> reduce(
  3. identity: {
  4. count: 1.0,
  5. c1_sum: 0.0,
  6. c1_avg: 0.0,
  7. c2_sum: 0.0,
  8. c2_avg: 0.0,
  9. },
  10. fn: (r, accumulator) => ({
  11. count: accumulator.count + 1.0,
  12. c1_sum: accumulator.c1_sum + r.c1_value,
  13. c1_avg: accumulator.c1_sum / accumulator.count,
  14. c2_sum: accumulator.c2_sum + r.c2_value,
  15. c2_avg: accumulator.c2_sum / accumulator.count,
  16. }),
  17. )

Aggregate gross and net profit

Use reduce() to create a function that aggregates gross and net profit. This example expects profit and expenses columns in the input tables.

  1. profitSummary = (tables=<-) => tables
  2. |> reduce(
  3. identity: {gross: 0.0, net: 0.0},
  4. fn: (r, accumulator) => ({
  5. gross: accumulator.gross + r.profit,
  6. net: accumulator.net + r.profit - r.expenses
  7. }
  8. )
  9. )