Loki compared to other log systems

Loki / Promtail / Grafana vs EFK

The EFK (Elasticsearch, Fluentd, Kibana) stack is used to ingest, visualize, andquery for logs from various sources.

Data in Elasticsearch is stored on-disk as unstructured JSON objects. Both thekeys for each object and the contents of each key are indexed. Data can then bequeried using a JSON object to define a query (called the Query DSL) or throughthe Lucene query language.

In comparison, Loki in single-binary mode can store data on-disk, but inhorizontally-scalable mode data is stored in a cloud storage system such as S3,GCS, or Cassandra. Logs are stored in plaintext form tagged with a set of labelnames and values, where only the label pairs are indexed. This tradeoff makes itcheaper to operate than a full index and allows developers to aggressively logfrom their applications. Logs in Loki are queried using LogQL.However, because of this design tradeoff, LogQL queries that filter based oncontent (i.e., text within the log lines) require loading all chunks within thesearch window that match the labels defined in the query.

Fluentd is usually used to collect and forward logs to Elasticsearch. Fluentd iscalled a data collector which can ingest logs from many sources, process it, andforward it to one or more targets.

In comparison, Promtail’s use case is specifically tailored to Loki. Its main modeof operation is to discover log files stored on disk and forward them associatedwith a set of labels to Loki. Promtail can do service discovery for Kubernetespods running on the same node as Promtail, act as a container sidecar or aDocker logging driver, read logs from specified folders, and tail the systemdjournal.

The way Loki represents logs by a set of label pairs is similar to howPrometheus represents metrics. When deployed in anenvironment alongside Prometheus, logs from Promtail usually have the samelabels as your applications metrics thanks to using the same servicediscovery mechanisms. Having logs and metrics with the same levels enables usersto seamlessly context switch between metrics and logs, helping with root causeanalysis.

Kibana is used to visualize and search Elasticsearch data and is very powerfulfor doing analytics on that data. Kibana provides many visualization tools to dodata analysis, such as location maps, machine learning for anomaly detection,and graphs to discover relationships in data. Alerts can be configured to notifyusers when an unexpected condition occurs.

In comparison, Grafana is tailored specifically towards time series data fromsources like Prometheus and Loki. Dashboards can be set up to visualize metrics(log support coming soon) and an explore view can be used to make ad-hoc queriesagainst your data. Like Kibana, Grafana supports alerting based on your metrics.