Access Logs

This topic describes configuration and usage for access logs. Consul can emit access logs to record application connections and requests that pass through proxies in a service mesh, including sidecar proxies and gateways. You can use the application traffic records in access to logs to help you performance the following operations:

• Diagnosing and Troubleshooting Issues: Operators and application owners can identify configuration issues in the service mesh or the application by analyzing failed connections and requests.
• Threat Detection: Operators can review details about unauthorized attempts to access the service mesh and their origins.
• Audit Compliance: Operators can use access less for security compliance requirements for traffic entering and exiting the service mesh through gateways.

Consul supports access logs capture through Envoy proxies started through the consul connect envoy CLI command and consul-dataplane. Other proxies are not supported.

Enable access logs

Access logs configurations are defined globally in the proxy-defaults configuration entry.

The following example is a minimal configuration for enabling access logs:

Kind      = "proxy-defaults"
Name      = "global"
AccessLogs {
  Enabled = true
}

apiVersion: consul.hashicorp.com/v1alpha1
kind: ProxyDefaults
metadata:
  name: global
spec:
  accessLogs:
    enabled: true

{
  "Kind": "proxy-defaults",
  "Name": "global",
  "AccessLogs": {
    "Enabled": true
  }
}

All proxies, including sidecars and gateways, emit access logs when the behavior is enabled. Both inbound and outbound traffic through the proxy are logged, including requests made directly to Envoy’s administration interface.

If you enable access logs after the Envoy proxy was started, access logs for the administration interface are not captured until you restart the proxy.

Default log format

Access logs use the following format when no additional customization is provided:

Security warning: The following log format contains IP addresses which may be a data compliance issue, depending on your regulatory environment. Operators should carefully inspect their chosen access log format to prevent leaking sensitive or personally identifiable information.

{
    "start_time":                        "%START_TIME%",
    "route_name":                        "%ROUTE_NAME%",
    "method":                            "%REQ(:METHOD)%",
    "path":                              "%REQ(X-ENVOY-ORIGINAL-PATH?:PATH)%",
    "protocol":                          "%PROTOCOL%",
    "response_code":                     "%RESPONSE_CODE%",
    "response_flags":                    "%RESPONSE_FLAGS%",
    "response_code_details":             "%RESPONSE_CODE_DETAILS%",
    "connection_termination_details":    "%CONNECTION_TERMINATION_DETAILS%",
    "bytes_received":                    "%BYTES_RECEIVED%",
    "bytes_sent":                        "%BYTES_SENT%",
    "duration":                          "%DURATION%",
    "upstream_service_time":             "%RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)%",
    "x_forwarded_for":                   "%REQ(X-FORWARDED-FOR)%",
    "user_agent":                        "%REQ(USER-AGENT)%",
    "request_id":                        "%REQ(X-REQUEST-ID)%",
    "authority":                         "%REQ(:AUTHORITY)%",
    "upstream_host":                     "%UPSTREAM_HOST%",
    "upstream_cluster":                  "%UPSTREAM_CLUSTER%",
    "upstream_local_address":            "%UPSTREAM_LOCAL_ADDRESS%",
    "downstream_local_address":          "%DOWNSTREAM_LOCAL_ADDRESS%",
    "downstream_remote_address":         "%DOWNSTREAM_REMOTE_ADDRESS%",
    "requested_server_name":             "%REQUESTED_SERVER_NAME%",
    "upstream_transport_failure_reason": "%UPSTREAM_TRANSPORT_FAILURE_REASON%"
}

Depending on the connection type, such TCP or HTTP, some of these fields may be empty.

Custom log format

Envoy uses command operators to expose information about application traffic. You can use these fields to customize the access logs that proxies emit.

Custom logs can be either JSON format or text format.

JSON format

You can format access logs in JSON so that you can parse them with Application Monitoring Platforms (APMs).

To use a custom access log, in the proxy-defaults configuration entry, set JSONFormat to the string representation of the desired JSON.

Nesting is supported.

Kind      = "proxy-defaults"
Name      = "global"
AccessLogs {
  Enabled = true
  JSONFormat = <<EOF
{
    "myCustomKey" : {
        "myStartTime" : "%START_TIME%",
        "myProtocol" : "%PROTOCOL%"
    }
}
EOF
}

apiVersion: consul.hashicorp.com/v1alpha1
kind: ProxyDefaults
metadata:
  name: global
spec:
  accessLogs:
    enabled: true
    jsonFormat: |
      {
        "myCustomKey" : {
          "myStartTime" : "%START_TIME%",
          "myProtocol" : "%PROTOCOL%"
        }
      }

{
  "Kind": "proxy-defaults",
  "Name": "global",
  "AccessLogs": {
    "Enabled": true,
    "JSONFormat": "{ \"myCustomKey\" : { \"myStartTime\" : \"%START_TIME%\", \"myProtocol\" : \"%PROTOCOL%\"} }"
  }
}

Text format

To use a custom access log formatted in plaintext, in the proxy-defaults configuration entry, set TextFormat to the desired customized string.

New lines are automatically added to the end of the log to keep each access log on its own line in the output.

Kind      = "proxy-defaults"
Name      = "global"
AccessLogs {
  Enabled = true
  TextFormat = "MY START TIME: %START_TIME%, THIS CONNECTIONS PROTOCOL IS %PROTOCOL%" 
}

apiVersion: consul.hashicorp.com/v1alpha1
kind: ProxyDefaults
metadata:
  name: global
spec:
  accessLogs:
    enabled: true
    textFormat: "MY START TIME: %START_TIME%, THIS CONNECTIONS PROTOCOL IS %PROTOCOL%"

{
  "Kind": "proxy-defaults",
  "Name": "global",
  "AccessLogs": {
    "Enabled": true,
    "JSONFormat": "MY START TIME: %START_TIME%, THIS CONNECTIONS PROTOCOL IS %PROTOCOL%" 
  }
}

Kubernetes

As part of its normal operation, the Envoy debugging logs for the consul-dataplane, envoy, or envoy-sidecar containers are written to stderr. The access log Type is set to stdout by default for access logs when enabled. Use a log aggregating solution to separate the machine-readable access logs from the Envoy process debug logs.

Write to a file

You can configure Consul to write access logs to a file on the host where Envoy runs.

Envoy does not rotate log files. A log rotation solution, such as logrotate, can prevent access logs from consuming too much of the host’s disk space when writing to a file.

Kind      = "proxy-defaults"
Name      = "global"
AccessLogs {
  Enabled = true
  Type = "file"
  Path = "/var/log/envoy/access-logs.txt" 
}

apiVersion: consul.hashicorp.com/v1alpha1
kind: ProxyDefaults
metadata:
  name: global
spec:
  accessLogs:
    enabled: true
    type: file
    path: "/var/log/envoy/access-logs.txt"

{
  "Kind": "proxy-defaults",
  "Name": "global",
  "AccessLogs": {
    "Enabled": true,
    "Type": "file",
    "Path": "/var/log/envoy/access-logs.txt"
  }
}