Comparison

LoggieFilebeatFluentdLogstashFlume
Development LanguageGolangGolangRubyJRubyJava
Multiple Pipelinesupportsinglesinglesupportsupport
Multiple Output Sourcessupportsingle outputconfig copysupportsupport
Transfersupportnot supportsupportsupportsupport
Log Alarmsupportnot supportnot supportnot supportnot support
Container Log Collection in Kubernetessupport container stdout and container internal log filesonly container stdoutonly container stdoutnot supportnot support
Configuration DeliveryIn Kubernetes, it can be configured through CRD. And host configuration center is gradually supportedmanual configurationmanual configurationmanual configurationmanual configuration
Monitornatively support Prometheus metrics, and can be configured to output metrics log files separately, send metrics, etc.The API interface is exposed, and an additional exporter is required to access PrometheusSupport API and Prometheus metricsNeed additional exporterNeed additional exporter
Resource Occupancylowlowmediumhighhigh

Benchmarks and Comparisons

Test Environment:

  • Physical machine 48C, 256G
  • Kafka 3 Broker, mount SSD disk
  • Filebeat v7.8 version, no processor configured; Loggie includes cost, retry, metric interceptor by default;

Test Purposes:

Performance comparison between Filebeat and Loggie

Test Idea:

Both Filebeat and Loggie collect logs and send them to Kafka. Observe the corresponding resource usage and sending throughput.

Test Details:

Automatically generate 5,000,000 lines of logs in a single file. The content of each line is as follows:

  1. [13/May/2021:10:20:29 +0800] 0.015 10.200.170.107 "GET /static/3tJHS3Ubrf.html?activity_channel_id=22=1_00000&fromMiniapp=1&miniapp_uuid=uEd93lG2eG8Qj5fRXuiJwNt4bmiylkmg HTTP/1.1" 200 138957 "110.183.45.54, 10.200.151.37" act.you.163.com "" "Mozilla/5.0 (Linux; Android 8.1.0; PADM00Build/O11019; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/67.0.3396.87 XWEB/568 MMWEBSDK/190102 Mobile Safari/537.36 MMWEBID/6881 MicroMessenger/7.0.3.1400(0x2700033B) Process/appbrand0 NetType/WIFI Language/zh_CN miniProgram" "" [127.0.0.1:8990] [0.014] [] [] immsg={"st":1553307293614,"sb":138963,"rc":200,"cf":{"sr":1},"if":"default","ut":14,"sv":"static","pd":"activity","qb":764}

Configure Filebeat and Loggie to collect logs and send them to a topic in Kafka without client-side compression. The Kafka topic is configured with 3 partitions.

In the case of ensuring sufficient resources of the Agent, modify the number of files collected, the concurrency of the sending client (configure Filebeat worker and Loggie parallelism). Observe the sending rate of CPU, Memory, and Pod network.

The test got the following data:

AgentFize SizeNumber of Log FilesSending ConcurrencyCPUMEM (rss)NIC sending rate
Filebeat3.2G137.5~8.5c63.8MiB75.9MiB/s
Filebeat3.2G1810c65MiB70MiB/s
Filebeat3.2G10811c65MiB80MiB/s
Loggie3.2G132.1c60MiB120MiB/s
Loggie3.2G182.4c68.7MiB120MiB/s
Loggie3.2G1083.5c70MiB210MiB/s

Test Conclusion:

Under the same pressure test conditions and scenarios:

  • The CPU consumption of Loggie is only about ¼ of Filebeat, and the sending throughput is 1.6 to 2.6 times that of the Filebeat.

  • Loggie and Filebeat have nearly same memory consumption. Both low.

  • There is a bottleneck in the limit throughput of Filebeat, and it is difficult to increase after 80MB/s. Loggie can reach more than 200MiB/s.