http_source

http_source is a source plugin that supports HTTP. Currently, http_source only supports the JSON UTF-8 codec for incoming requests, such as [{"key1": "value1"}, {"key2": "value2"}]. The following table describes options you can use to configure the http_source source.

OptionRequiredTypeDescription
portNoIntegerThe port that the source is running on. Default value is 2021. Valid options are between 0 and 65535.
health_check_serviceNoBooleanEnables the health check service on the /health endpoint on the defined port. Default value is false.
unauthenticated_health_checkNoBooleanDetermines whether or not authentication is required on the health check endpoint. Data Prepper ignores this option if no authentication is defined. Default value is false.
request_timeoutNoIntegerThe request timeout, in milliseconds. Default value is 10000.
thread_countNoIntegerThe number of threads to keep in the ScheduledThreadPool. Default value is 200.
max_connection_countNoIntegerThe maximum allowed number of open connections. Default value is 500.
max_pending_requestsNoIntegerThe maximum allowed number of tasks in the ScheduledThreadPool work queue. Default value is 1024.
authenticationNoObjectAn authentication configuration. By default, this creates an unauthenticated server for the pipeline. This uses pluggable authentication for HTTPS. To use basic authentication define the http_basic plugin with a username and password. To provide customer authentication, use or create a plugin that implements ArmeriaHttpAuthenticationProvider.
sslNoBooleanEnables TLS/SSL. Default value is false.
ssl_certificate_fileConditionallyStringSSL certificate chain file path or Amazon Simple Storage Service (Amazon S3) path. Amazon S3 path example s3://<bucketName>/<path>. Required if ssl is set to true and use_acm_certificate_for_ssl is set to false.
ssl_key_fileConditionallyStringSSL key file path or Amazon S3 path. Amazon S3 path example s3://<bucketName>/<path>. Required if ssl is set to true and use_acm_certificate_for_ssl is set to false.
use_acm_certificate_for_sslNoBooleanEnables a TLS/SSL using certificate and private key from AWS Certificate Manager (ACM). Default value is false.
acm_certificate_arnConditionallyStringThe ACM certificate Amazon Resource Name (ARN). The ACM certificate takes preference over Amazon S3 or a local file system certificate. Required if use_acm_certificate_for_ssl is set to true.
acm_private_key_passwordNoStringACM private key password that decrypts the private key. If not provided, Data Prepper generates a random password.
acm_certificate_timeout_millisNoIntegerTimeout, in milliseconds, that ACM takes to get certificates. Default value is 120000.
aws_regionConditionallyStringAWS region used by ACM or Amazon S3. Required if use_acm_certificate_for_ssl is set to true or ssl_certificate_file and ssl_key_file is the Amazon S3 path.

Metrics

The http_source source includes the following metrics.

Counters

  • requestsReceived: Measures the total number of requests received by the /log/ingest endpoint.
  • requestsRejected: Measures the total number of requests rejected (429 response status code) by the HTTP Source plugin.
  • successRequests: Measures the total number of requests successfully processed (200 response status code) the by HTTP Source plugin.
  • badRequests: Measures the total number of requests with either an invalid content type or format processed by the HTTP Source plugin (400 response status code).
  • requestTimeouts: Measures the total number of requests that time out in the HTTP source server (415 response status code).
  • requestsTooLarge: Measures the total number of requests where the size of the event is larger than the buffer capacity (413 response status code).
  • internalServerError: Measures the total number of requests processed by the HTTP Source with a custom exception type (500 response status code).

Timers

  • requestProcessDuration: Measures the latency of requests processed by the HTTP Source plugin in seconds.

Distribution summaries

  • payloadSize: Measures the incoming request payload size in bytes.