Quickstart with Docker

Requirement

Start HStream requires an operating system kernel version greater than at least Linux 4.14

Installation

Install docker

TIP

If you have already installed docker, you can skip this step.

See Install Docker EngineQuickstart with Docker - 图1 (opens new window), and install it for your operating system. Please carefully check that you have met all prerequisites.

Confirm that the Docker daemon is running:

  1. docker version

TIP

On Linux, Docker needs root privileges. You can also run Docker as a non-root user, see Post-installation steps for LinuxQuickstart with Docker - 图2 (opens new window).

Install docker compose

TIP

If you have already installed docker compose, you can skip this step.

See Install Docker ComposeQuickstart with Docker - 图3 (opens new window), and install it for your operating system. Please carefully check that you met all prerequisites.

  1. docker-compose version

Start a local standalone HStream-Server in Docker

WARING

Do NOT use this configuration in your production environment!

Create a directory for storing db datas

  1. mkdir /data/store

TIP

If you are a non-root user, that you can not create directory under the root.

You can create it anywhere as you can.

  1. mkdir $HOME/data/store
  2. # make sure that you have set the environment variable DATA_DIR
  3. export DATA_DIR=$HOME/data/store

Start HStreamDB Server and Store

Create a docker-compose.yaml file for docker compose, you can downloadQuickstart with Docker - 图4 (opens new window) or paste the following contents:

  1. ## docker-compose.yaml
  2. version: "3.5"
  3. services:
  4. hserver:
  5. image: hstreamdb/hstream:v0.6.1
  6. depends_on:
  7. - zookeeper
  8. - hstore
  9. ports:
  10. - "127.0.0.1:6570:6570"
  11. expose:
  12. - 6570
  13. networks:
  14. - hstream-quickstart
  15. volumes:
  16. - ${DATA_DIR:-/data/store}:/data/store
  17. command: &start_hserver
  18. - bash
  19. - "-c"
  20. - |
  21. set -e
  22. /usr/local/script/wait-for-storage.sh hstore 6440 zookeeper 2181 600 \
  23. /usr/local/bin/hstream-server \
  24. --host 0.0.0.0 --port 6570 \
  25. --internal-port 6571 \
  26. --address $$(hostname -I | awk '{print $$1}') \
  27. --server-id 100 \
  28. --zkuri zookeeper:2181 \
  29. --store-config /data/store/logdevice.conf \
  30. --store-admin-host hstore --store-admin-port 6440 \
  31. --replicate-factor 3 \
  32. hstream-http-server:
  33. image: hstreamdb/hstream:v0.6.1
  34. depends_on:
  35. - hserver
  36. ports:
  37. - "127.0.0.1:6580:6580"
  38. expose:
  39. - 6580
  40. networks:
  41. - hstream-quickstart
  42. command:
  43. - bash
  44. - "-c"
  45. - |
  46. set -e
  47. /usr/local/bin/hstream-http-server \
  48. -gRPCServerHost hserver \
  49. -httpServerPort 6580 \
  50. -gRPCServerPort 6570
  51. hstore:
  52. image: hstreamdb/hstream:v0.6.1
  53. networks:
  54. - hstream-quickstart
  55. volumes:
  56. - ${DATA_DIR:-/data/store}:/data/store
  57. command:
  58. - bash
  59. - "-c"
  60. - |
  61. set -ex
  62. /usr/local/bin/ld-dev-cluster --root /data/store \
  63. --use-tcp --tcp-host $$(hostname -I | awk '{print $$1}') \
  64. --user-admin-port 6440 \
  65. --no-interactive
  66. zookeeper:
  67. image: zookeeper
  68. expose:
  69. - 2181
  70. networks:
  71. - hstream-quickstart
  72. networks:
  73. hstream-quickstart:
  74. name: hstream-quickstart

then run:

  1. docker-compose -f quick-start.yaml up

If you see some thing like this, then you have a running hstream:

  1. hserver_1 | [INFO][2021-11-22T09:15:18+0000][app/server.hs:137:3][thread#67]************************
  2. hserver_1 | [INFO][2021-11-22T09:15:18+0000][app/server.hs:145:3][thread#67]Server started on port 6570
  3. hserver_1 | [INFO][2021-11-22T09:15:18+0000][app/server.hs:146:3][thread#67]*************************

TIP

You can also run in background.

  1. docker-compose -f quick-start.yaml up -d

And if you want to show logs of server, run:

  1. docker-compose -f quick-start.yaml logs -f hserver

Start HStreamDB’s interactive SQL CLI

  1. docker run -it --rm --name some-hstream-cli --network host hstreamdb/hstream hstream-client --port 6570 --client-id 1

If everything works fine, you will enter an interactive CLI and see help information like

  1. __ _________________ _________ __ ___
  2. / / / / ___/_ __/ __ \/ ____/ | / |/ /
  3. / /_/ /\__ \ / / / /_/ / __/ / /| | / /|_/ /
  4. / __ /___/ // / / _, _/ /___/ ___ |/ / / /
  5. /_/ /_//____//_/ /_/ |_/_____/_/ |_/_/ /_/
  6. Command
  7. :h To show these help info
  8. :q To exit command line interface
  9. :help [sql_operation] To show full usage of sql statement
  10. SQL STATEMENTS:
  11. To create a simplest stream:
  12. CREATE STREAM stream_name;
  13. To create a query select all fields from a stream:
  14. SELECT * FROM stream_name EMIT CHANGES;
  15. To insert values to a stream:
  16. INSERT INTO stream_name (field1, field2) VALUES (1, 2);
  17. >

Create a stream

What we are going to do first is create a stream by CREATE STREAM statement.

  1. CREATE STREAM demo;

Run a continuous query over the stream

Now we can run a continuous query over the stream we just created by SELECT query.

The query will output all records from the demo stream whose humidity is above 70 percent.

  1. SELECT * FROM demo WHERE humidity > 70 EMIT CHANGES;

It seems that nothing happened. But do not worry because there is no data in the stream now. Next, we will fill the stream with some data so the query can produce output we want.

Start another CLI session

Start another CLI session, this CLI will be used for inserting data into the stream.

  1. docker exec -it some-hstream-cli hstream-client --port 6570 --client-id 2

Insert data into the stream

Run each of the given INSERT statement in the new CLI session and keep an eye on the CLI session created in (2).

  1. INSERT INTO demo (temperature, humidity) VALUES (22, 80);
  2. INSERT INTO demo (temperature, humidity) VALUES (15, 20);
  3. INSERT INTO demo (temperature, humidity) VALUES (31, 76);
  4. INSERT INTO demo (temperature, humidity) VALUES ( 5, 45);
  5. INSERT INTO demo (temperature, humidity) VALUES (27, 82);
  6. INSERT INTO demo (temperature, humidity) VALUES (28, 86);

If everything works fine, the continuous query will output matching records in real time:

  1. {"temperature":22,"humidity":80}
  2. {"temperature":31,"humidity":76}
  3. {"temperature":27,"humidity":82}
  4. {"temperature":28,"humidity":86}

Start a 3-node local HStream-Server Cluster in Docker

WARING

Do NOT use this configuration in your production environment!

If you did not follow quick start and have not yet got a running local HStream-Server, read this section instead.

If you already have a running standalone HStream-Server

You can manually start the other 2 servers and pass the same zkuri as the running server to create a hserver cluster. Note that every server needs the following options to be unique to work properly:

  • server-id : the id has to be an integer. This is the identifier of every server.
  • port : the port number that client connects to.
  • internal-port : the internal channel for server communication.

For example, run the following commands to start a cluster with 3 nodes, only if you followed quick start and did not change the configQuickstart with Docker - 图5 (opens new window) :

  1. docker run -it --rm --name some-hstream-server-1 -v $DATA_DIR:/data/store --network hstream-quickstart hstreamdb/hstream hstream-server --store-config /data/store/logdevice.conf --zkuri 127.0.0.1:2181 --port 6580 --internal-port 6581 --server-id 101
  1. docker run -it --rm --name some-hstream-server-2 -v $DATA_DIR:/data/store --network hstream-quickstart hstreamdb/hstream hstream-server --store-config /data/store/logdevice.conf --zkuri 127.0.0.1:2181 --port 6590 --internal-port 6591 --server-id 102

Start with docker compose

Download quick-start.yamlQuickstart with Docker - 图6 (opens new window) and copy the exact following contents under services section:

  1. hserver0:
  2. image: hstreamdb/hstream
  3. depends_on:
  4. - zookeeper
  5. - hstore
  6. ports:
  7. - "127.0.0.1:6580:6580"
  8. expose:
  9. - 6580
  10. networks:
  11. - hstream-quickstart
  12. volumes:
  13. - ${DATA_DIR:-/data/store}:/data/store
  14. command:
  15. - bash
  16. - "-c"
  17. - |
  18. set -e
  19. /usr/local/script/wait-for-storage.sh hstore 6440 zookeeper 2181 600 \
  20. /usr/local/bin/hstream-server \
  21. --host 0.0.0.0 --port 6590 \
  22. --internal-port 6591 \
  23. --address $$(hostname -I | awk '{print $$1}') \
  24. --server-id 101 \
  25. --zkuri zookeeper:2181 \
  26. --store-config /data/store/logdevice.conf \
  27. --store-admin-host hstore --store-admin-port 6440 \
  28. --replicate-factor 3 \
  29. hserver1:
  30. image: hstreamdb/hstream
  31. depends_on:
  32. - zookeeper
  33. - hstore
  34. ports:
  35. - "127.0.0.1:6590:6590"
  36. expose:
  37. - 6590
  38. networks:
  39. - hstream-quickstart
  40. volumes:
  41. - ${DATA_DIR:-/data/store}:/data/store
  42. command:
  43. - bash
  44. - "-c"
  45. - |
  46. set -e
  47. /usr/local/script/wait-for-storage.sh hstore 6440 zookeeper 2181 600 \
  48. /usr/local/bin/hstream-server \
  49. --host 0.0.0.0 --port 6590 \
  50. --internal-port 6591 \
  51. --address $$(hostname -I | awk '{print $$1}') \
  52. --server-id 102 \
  53. --zkuri zookeeper:2181 \
  54. --store-config /data/store/logdevice.conf \
  55. --store-admin-host hstore --store-admin-port 6440 \
  56. --replicate-factor 3 \

You can easily get a cluster with:

  1. docker-compose -f quick-start.yaml up -d
  2. docker-compose -f quick-start.yaml logs -f hserver hserver0 hserver1

Use HStreamDB’s interactive SQL CLI

Start a cli session in the similar way as when you have a standalone server. An HStream Server Cluster does not affect how you use CLI.

  1. docker run -it --rm --name some-hstream-cli --network host hstreamdb/hstream hstream-client --port 6570 --client-id 1