Peer forwarder

Peer forwarder is an HTTP service that performs peer forwarding of an event between Data Prepper nodes for aggregation. This HTTP service uses a hash-ring approach to aggregate events and determine which Data Prepper node it should handle on a given trace before rerouting it to that node. Currently, peer forwarder is supported by the aggregate, service_map_stateful, and otel_trace_raw processors.

Peer forwarder groups events based on the identification keys provided by the supported processors. For service_map_stateful and otel_trace_raw, the identification key is traceId by default and cannot be configured. The aggregate processor is configured using the identification_keys configuration option. From here, you can specify which keys to use for peer forwarder. See Aggregate Processor page for more information about identification keys.

Peer discovery allows Data Prepper to find other nodes that it will communicate with. Currently, peer discovery is provided by a static list, a DNS record lookup, or AWS Cloud Map.

Discovery modes

The following sections provide information about discovery modes.

Static

Static discovery mode allows a Data Prepper node to discover nodes using a list of IP addresses or domain names. See the following YAML file for an example of static discovery mode:

peer_forwarder:4
  discovery_mode: static
  static_endpoints: ["data-prepper1", "data-prepper2"]

DNS lookup

DNS discovery is preferred over static discovery when scaling out a Data Prepper cluster. DNS discovery configures a DNS provider to return a list of Data Prepper hosts when given a single domain name. This list consists of a DNS A record, and a list of IP addresses of a given domain. See the following YAML file for an example of DNS lookup:

peer_forwarder:
  discovery_mode: dns
  domain_name: "data-prepper-cluster.my-domain.net"

AWS Cloud Map

AWS Cloud Map provides API-based service discovery as well as DNS-based service discovery.

Peer forwarder can use the API-based service discovery in AWS Cloud Map. To support this, you must have an existing namespace configured for API instance discovery. You can create a new one by following the instructions provided by the AWS Cloud Map documentation.

Your Data Prepper configuration needs to include the following:

• aws_cloud_map_namespace_name – Set to your AWS Cloud Map namespace name.
• aws_cloud_map_service_name – Set to the service name within your specified namespace.
• aws_region – Set to the AWS Region in which your namespace exists.
• discovery_mode – Set to aws_cloud_map.

Your Data Prepper configuration can optionally include the following:

• aws_cloud_map_query_parameters – Key-value pairs are used to filter the results based on the custom attributes attached to an instance. Results include only those instances that match all of the specified key-value pairs.

Example configuration

See the following YAML file example of AWS Cloud Map configuration:

peer_forwarder:
  discovery_mode: aws_cloud_map
  aws_cloud_map_namespace_name: "my-namespace"
  aws_cloud_map_service_name: "data-prepper-cluster"
  aws_cloud_map_query_parameters:
    instance_type: "r5.xlarge"
  aws_region: "us-east-1"

IAM policy with necessary permissions

Data Prepper must also be running with the necessary permissions. The following AWS Identity and Access Management (IAM) policy shows the necessary permissions:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "CloudMapPeerForwarder",
            "Effect": "Allow",
            "Action": "servicediscovery:DiscoverInstances",
            "Resource": "*"
        }
    ]
}

Configuration

The following table provides optional configuration values.

SSL configuration

The following table provides optional SSL configuration values that allow you to set up a trust manager for the peer forwarder client in order to connect to other Data Prepper instances.

Example configuration

The following YAML file provides an example configuration:

peer_forwarder:
  ssl: true
  ssl_certificate_file: "<cert-file-path>"
  ssl_key_file: "<private-key-file-path>"

Authentication

Authentication is optional and is a Map that enables mutual TLS (mTLS). It can either be mutual_tls or unauthenticated. The default value is unauthenticated. The following YAML file provides an example of authentication:

peer_forwarder:
  authentication:
    mutual_tls:

Metrics

Core peer forwarder introduces the following custom metrics. All the metrics are prefixed by core.peerForwarder.

Timer

Peer forwarder’s timer capability provides the following information:

• requestForwardingLatency: Measures latency of requests forwarded by the peer forwarder client.
• requestProcessingLatency: Measures latency of requests processed by the peer forwarder server.

Counter

The following table provides counter metric options.

Gauge

peerEndpoints Measures the number of dynamically discovered peer Data Prepper endpoints. For static mode, the size is fixed.