Web cache server HTTP/2 performance benchmark: nuster vs nginx

This is a simple benchmark comparing the HTTP/2 cache performance of nuster and nginx.

I tested the RPS(Request per second) by h2load.

The test shown that nuster is almost three times faster than nginx.

Here is the result(RPS) when request /helloworld URL containing hello world data.

data sizeCONNnusternginx
12(hello world)1000338924110419

Testing environment

Server

Two linux servers are used, an origin web server on server129,and cache server nuster/nginx on server130.

Serverportapp
10.0.0.129wrk
10.0.0.1298080nginx, origin web server
10.0.0.1308081nuster, all cores
10.0.0.1308083nginx, all cores

origin web server: set server_tokens off; to make http header server same.

Hardware

  • Intel(R) Xeon(R) CPU X5650 @ 2.67GHz(12 cores)
  • RAM 32GB
  • 1Gbps ethernet card

Software

  • CentOS: 7.4.1708 (Core)
  • h2load: nghttp2/1.30.0
  • nginx: nginx/1.14.0
  • nuster: nuster/1.8.8.2

System settings

/etc/sysctl.conf

  1. fs.file-max =9999999
  2. fs.nr_open =9999999
  3. net.core.netdev_max_backlog =4096
  4. net.core.rmem_max =16777216
  5. net.core.somaxconn =65535
  6. net.core.wmem_max =16777216
  7. net.ipv4.ip_forward =0
  8. net.ipv4.ip_local_port_range =102565535
  9. net.ipv4.tcp_fin_timeout =30
  10. net.ipv4.tcp_keepalive_time =30
  11. net.ipv4.tcp_max_syn_backlog =20480
  12. net.ipv4.tcp_max_tw_buckets =400000
  13. net.ipv4.tcp_no_metrics_save =1
  14. net.ipv4.tcp_syn_retries =2
  15. net.ipv4.tcp_synack_retries =2
  16. net.ipv4.tcp_tw_recycle =1
  17. net.ipv4.tcp_tw_reuse =1
  18. net.ipv4.tcp_timestamps =1
  19. vm.min_free_kbytes =65536
  20. vm.overcommit_memory =1

/etc/security/limits.conf

  1. * soft nofile 1000000
  2. * hard nofile 1000000
  3. * soft nproc 1000000
  4. * hard nproc 1000000

Config files

nuster, all cores

  1. global
  2. maxconn 1000000
  3. nuster cache on data-size 1g
  4. daemon
  5. nbproc 12
  6. tune.maxaccept -1
  7. tune.ssl.default-dh-param 2048
  8. tune.h2.max-concurrent-streams 1000
  9. defaults
  10. retries 3
  11. maxconn 1000000
  12. option redispatch
  13. option dontlognull
  14. timeout client 300s
  15. timeout connect 300s
  16. timeout server 300s
  17. http-reuse always
  18. frontend web1
  19. bind *:8081 ssl crt asdf.pem alpn h2,http/1.1
  20. mode http
  21. default_backend app1
  22. backend app1
  23. balance roundrobin
  24. mode http
  25. nuster cache on
  26. nuster rule all ttl 0
  27. server a2 10.0.0.129:8080

nginx, all cores

  1. user nginx;
  2. worker_processes auto;
  3. worker_rlimit_nofile 1000000;
  4. error_log /var/log/nginx/errorall.log warn;
  5. pid /var/run/nginxall.pid;
  6. events {
  7. worker_connections 1000000;
  8. use epoll;
  9. multi_accept on;
  10. }
  11. http {
  12. include /etc/nginx/mime.types;
  13. default_type application/octet-stream;
  14. access_log off;
  15. sendfile on;
  16. server_tokens off;
  17. keepalive_timeout 300;
  18. keepalive_requests 100000;
  19. tcp_nopush on;
  20. tcp_nodelay on;
  21. client_body_buffer_size 128k;
  22. client_header_buffer_size 1m;
  23. large_client_header_buffers 44k;
  24. output_buffers 132k;
  25. postpone_output 1460;
  26. open_file_cache max=200000 inactive=20s;
  27. open_file_cache_valid 30s;
  28. open_file_cache_min_uses 2;
  29. open_file_cache_errors on;
  30. proxy_cache_path /tmp/cache_all levels=1:2 keys_zone=STATIC:10m inactive=24h max_size=1g;
  31. server {
  32. listen 8083 ssl http2;
  33. ssl on;
  34. ssl_session_cache shared:SSL:5m;
  35. ssl_session_timeout 1h;
  36. http2_max_concurrent_streams 1000;
  37. http2_max_requests 10000000;
  38. ssl_certificate a.pem;
  39. ssl_certificate_key a.key;
  40. location /{
  41. proxy_pass http://10.0.0.129:8080/;
  42. proxy_cache STATIC;
  43. proxy_cache_valid any 1d;
  44. }
  45. }
  46. }

Benchmark

nuster

  1. $ h2load -n 10000000-c 1000-m 1000-t 1000 https://10.0.0.130:8081/helloworld
  2. finished in29.51s,338924.15 req/s,48.81MB/s
  3. requests:10000000 total,10000000 started,10000000done,10000000 succeeded,0 failed,0 errored,0 timeout
  4. status codes:100000002xx,03xx,04xx,05xx
  5. traffic:1.41GB(1510024000) total,1.13GB(1210000000) headers (space savings 34.24%),114.44MB(120000000) data
  6. min max mean sd +/- sd
  7. time for request:14.51ms3.21s2.53s500.02ms74.50%
  8. time for connect:61.12ms1.26s672.73ms332.30ms63.50%
  9. time to 1stbyte:95.03ms3.74s2.68s777.51ms74.00%
  10. req/s :339.11459.68358.9217.3576.20%

nginx

  1. $ h2load -n 10000000-c 1000-m 1000-t 1000 https://10.0.0.130:8083/helloworld
  2. finished in90.56s,110419.16 req/s,15.62MB/s
  3. requests:10000000 total,10000000 started,10000000done,10000000 succeeded,0 failed,0 errored,0 timeout
  4. status codes:100000002xx,03xx,04xx,05xx
  5. traffic:1.38GB(1482955210) total,1.10GB(1182906210) headers (space savings 35.01%),114.44MB(120000000) data
  6. min max mean sd +/- sd
  7. time for request:3.98ms14.65s5.14s3.67s70.30%
  8. time for connect:69.96ms6.74s1.35s1.08s88.90%
  9. time to 1stbyte:114.92ms15.46s5.67s4.06s63.40%
  10. req/s :110.435143.28378.40590.4792.50%