Splunk

Splunk

If your organization uses Splunk, you can configure Rancher to send it cluster or project logs. Afterwards logs are sent, you can use Splunk to view them.

Configuring Splunk Logging

You can configure Rancher to send Kubernetes logs to your instance of Splunk.

Prerequisites:
Configure HTTP event collection for your Splunk Server (Splunk Enterprise or Splunk Cloud).
Enable all tokens, and then create a new token.
For more information, see Splunk Documentation.

Browse to the cluster or project that you want to log.

To Configure Cluster Logging:

If you’re a cluster owner or member who works in operations or security, configure cluster logging.

From the Global view, open the cluster that you want to configure logging for.
From the main menu, select Tools > Logging.

To Configure Project Logging:

If you’re a project owner or member who works on an application, configure project logging.

From the Global view, open the project that you want to configure logging for.
From the main menu, select Resources > Logging.

Select Splunk.
Complete the Splunk HTTP Event Collector Configuration form.
- From the Endpoint field, enter the IP address and port for you Splunk instance (i.e. http://splunk-server:8088)

Splunk usually uses port 8088. If you’re using Splunk Cloud, you’ll need to work with Splunk support to get an endpoint URL.

Enter the Token you obtained while completing the prerequisites (i.e., when you created a token in Splunk).
From the Source field, enter the name of the token as entered in Splunk.
Optional: Enter one or more index that’s allowed for your token.

Complete the Additional Logging Configuration form.
- Optional: Use the Add Field button to add custom log fields to your logging configuration. These fields are key value pairs (such as foo=bar) that you can use to filter the logs from another system.
- Enter a Flush Interval. This value determines how often Fluentd flushes event data to the logging server. Intervals are measured in seconds.
Click Save.

Result: Rancher is now configured to send logs to Splunk. Log into your Spunk instance to view events for your cluster and containers.

Viewing Logs

Log into your Splunk server.
Click on Search & Reporting. The number of Indexed Events listed should be increasing.
Click on Data Summary and select the Sources tab.
To view the actual logs, click on the source that you declared earlier.

Troubleshooting

You can use curl to see if HEC is listening for HTTP event data.

$ curl http://splunk-server:8088/services/collector/event \
    -H 'Authorization: Splunk 8da70994-b1b0-4a79-b154-bfaae8f93432' \
    -d '{"event": "hello world"}'

If Splunk is configured correctly, you should receive json data returning success code 0. You should be ableto send logging data to HEC.

If you received an error, check your configuration in Splunk and Rancher.