Starting and using Manticore in Docker

The image is based on current release of Manticore package.

The default configuration includes a sample Real-Time table and listens on the default ports:

  • 9306 for connections from a MySQL client
  • 9308 for connections via HTTP
  • 9312 for connections via a binary protocol (e.g. in case you run a cluster)

The image comes with libraries for easy indexing data from MySQL, PostgreSQL XML and CSV files.

How to run Manticore Search Docker image

Quick usage

The below is the simplest way to start Manticore in a container and log in to it via mysql client:

  1. docker run -e EXTRA=1 --name manticore --rm -d manticoresearch/manticore && echo "Waiting for Manticore docker to start. Consider mapping the data_dir to make it start faster next time" && until docker logs manticore 2>&1 | grep -q "accepting connections"; do sleep 1; echo -n .; done && echo && docker exec -it manticore mysql && docker stop manticore

Note that upon exiting the MySQL client, the Manticore container will be stopped and removed, resulting in no saved data. For information on using Manticore in a production environment, please see below.

The image comes with a sample table which can be loaded like this:

  1. mysql> source /sandbox.sql

Also the mysql client has in history several sample queries that you can run on the above table, just use Up/Down keys in the client to see and run them.

Production use

Ports and mounting points

For data persistence the folder /var/lib/manticore/ should be mounted to local storage or other desired storage engine.

Configuration file inside the instance is located at /etc/manticoresearch/manticore.conf. For custom settings, this file should be mounted to own configuration file.

The ports are 9306/9308/9312 for SQL/HTTP/Binary, expose them depending on how you are going to use Manticore. For example:

  1. docker run -e EXTRA=1 --name manticore -v $(pwd)/data:/var/lib/manticore -p 127.0.0.1:9306:9306 -p 127.0.0.1:9308:9308 -d manticoresearch/manticore

or

  1. docker run -e EXTRA=1 --name manticore -v $(pwd)/manticore.conf:/etc/manticoresearch/manticore.conf -v $(pwd)/data:/var/lib/manticore/ -p 127.0.0.1:9306:9306 -p 127.0.0.1:9308:9308 -d manticoresearch/manticore

Make sure to remove 127.0.0.1: if you want the ports to be available for external hosts.

Manticore Columnar Library and Manticore Buddy

The Manticore Search Docker image doesn’t come with the Manticore Columnar Library pre-installed, which is necessary if you require columnar storage and secondary indexes. However, it can easily be enabled during runtime by setting the environment variable EXTRA=1. For example, docker run -e EXTRA=1 ... manticoresearch/manticore. This will download and install the library in the data directory (which is typically mapped as a volume in production environments) and it won’t be re-downloaded unless the Manticore Search version is changed.

Using EXTRA=1 also activates Manticore Buddy, which is used for processing certain commands. For more information, refer to the changelog.

If you only need the MCL, you can use the environment variable MCL=1.

Docker-compose

In many cases you might want to use Manticore together with other images specified in a docker-compose YAML file. Here is the minimal recommended specification for Manticore Search in docker-compose.yml:

  1. version: '2.2'
  3. services:
  4.   manticore:
  5.     container_name: manticore
  6.     image: manticoresearch/manticore
  7.     environment:
  8.       - EXTRA=1
  9.     restart: always
  10.     ports:
  11.       - 127.0.0.1:9306:9306
  12.       - 127.0.0.1:9308:9308
  13.     ulimits:
  14.       nproc: 65535
  15.       nofile:
  16.          soft: 65535
  17.          hard: 65535
  18.       memlock:
  19.         soft: -1
  20.         hard: -1
  21.     volumes:
  22.       - ./data:/var/lib/manticore
  23. #      - ./manticore.conf:/etc/manticoresearch/manticore.conf # uncomment if you use a custom config

Besides using the exposed ports 9306 and 9308 you can log into the instance by running docker-compose exec manticore mysql.

HTTP protocol

HTTP protocol is exposed on port 9308. You can map the port locally and connect with curl:

  1. docker run -e EXTRA=1 --name manticore -p 9308:9308 -d manticoresearch/manticore

Create a table:

  • JSON

JSON

  1. POST /cli -d 'CREATE TABLE testrt ( title text, content text, gid integer)'

Insert a document:

  • JSON

JSON

  1. POST /insert
  2. -d'{"index":"testrt","id":1,"doc":{"title":"Hello","content":"world","gid":1}}'

Perform a simple search:

  • JSON

JSON

  1. POST /search -d '{"index":"testrt","query":{"match":{"*":"hello world"}}}'

Logging

By default, the server is set to send it’s logging to /dev/stdout, which can be viewed from the host with:

  1. docker logs manticore

The query log can be diverted to Docker log by passing variable QUERY_LOG_TO_STDOUT=true.

Multi-node cluster with replication

Here is a simple docker-compose.yml for defining a two node cluster:

  1. version: '2.2'
  3. services:
  5.   manticore-1:
  6.     image: manticoresearch/manticore
  7.     environment:
  8.       - EXTRA=1
  9.     restart: always
  10.     ulimits:
  11.       nproc: 65535
  12.       nofile:
  13.          soft: 65535
  14.          hard: 65535
  15.       memlock:
  16.         soft: -1
  17.         hard: -1
  18.     networks:
  19.       - manticore
  20.   manticore-2:
  21.     image: manticoresearch/manticore
  22.     environment:
  23.       - EXTRA=1
  24.     restart: always
  25.     ulimits:
  26.       nproc: 65535
  27.       nofile:
  28.         soft: 65535
  29.         hard: 65535
  30.       memlock:
  31.         soft: -1
  32.         hard: -1
  33.     networks:
  34.       - manticore
  35. networks:
  36.   manticore:
  37.     driver: bridge

  • Start it: docker-compose up

  • Create a cluster:

    1. $ docker-compose exec manticore-1 mysql
    3. mysql> CREATE TABLE testrt ( title text, content text, gid integer);
    5. mysql> CREATE CLUSTER posts;
    6. Query OK, 0 rows affected (0.24 sec)
    8. mysql> ALTER CLUSTER posts ADD testrt;
    9. Query OK, 0 rows affected (0.07 sec)
    11. MySQL [(none)]> exit
    12. Bye

  • Join to the the cluster on the 2nd instance

    1. $ docker-compose exec manticore-2 mysql
    3. mysql> JOIN CLUSTER posts AT 'manticore-1:9312';
    4. mysql> INSERT INTO posts:testrt(title,content,gid)  VALUES('hello','world',1);
    5. Query OK, 1 row affected (0.00 sec)
    7. MySQL [(none)]> exit
    8. Bye

  • If you now go back to the first instance you’ll see the new record:

    1. $ docker-compose exec manticore-1 mysql
    3. MySQL [(none)]> select * from testrt;
    4. +---------------------+------+-------+---------+
    5. | id                  | gid  | title | content |
    6. +---------------------+------+-------+---------+
    7. | 3891565839006040065 |    1 | hello | world   |
    8. +---------------------+------+-------+---------+
    9. 1 row in set (0.00 sec)
    11. MySQL [(none)]> exit
    12. Bye

Memory locking and limits

It’s recommended to overwrite the default ulimits of docker for the Manticore instance:

  1.  --ulimit nofile=65536:65536

For best performance, table components can be “mlocked” into memory. When Manticore is run under Docker, the instance requires additional privileges to allow memory locking. The following options must be added when running the instance:

  1.   --cap-add=IPC_LOCK --ulimit memlock=-1:-1

Configuring Manticore Search with Docker

If you want to run Manticore with your custom config containing tables definition you will need to mount the configuration to the instance:

  1. docker run -e EXTRA=1 --name manticore -v $(pwd)/manticore.conf:/etc/manticoresearch/manticore.conf -v $(pwd)/data/:/var/lib/manticore -p 127.0.0.1:9306:9306 -d manticoresearch/manticore

Take into account that Manticore search inside the container is run under user manticore. Performing operations with table files (like creating or rotating plain tables) should be also done under manticore. Otherwise the files will be created under root and the search daemon won’t have rights to open them. For example here is how you can rotate all tables:

  1. docker exec -it manticore gosu manticore indexer --all --rotate

You can also set individual searchd and common configuration settings using Docker environment variables.

The settings must be prefixed with their section name, example for in case of mysql_version_string the variable must be named searchd_mysql_version_string:

  1. docker run -e EXTRA=1 --name manticore  -p 127.0.0.1:9306:9306  -e searchd_mysql_version_string='5.5.0' -d manticoresearch/manticore

In case of listen directive, you can pass using Docker variable searchd_listen new listening interfaces in addition to the default ones. Multiple interfaces can be declared separated by semi-colon (“|”). For listening only on network address, the $ip (retrieved internally from hostname -i) can be used as address alias.

For example -e searchd_listen='9316:http|9307:mysql|$ip:5443:mysql_vip' will add an additional SQL interface on port 9307, an SQL VIP listener on port 5443 running only on the instance’s IP and an HTTP listener on port 9316, beside the defaults on 9306 and 9308, respectively.

  1. $ docker run -e EXTRA=1 --rm -p 1188:9307  -e searchd_mysql_version_string='5.5.0' -e searchd_listen='9316:http|9307:mysql|$ip:5443:mysql_vip'  manticore
  2. [Mon Aug 17 07:31:58.719 2020] [1] using config file '/etc/manticoresearch/manticore.conf' (9130 chars)...
  3. listening on all interfaces for http, port=9316
  4. listening on all interfaces for mysql, port=9307
  5. listening on 172.17.0.17:5443 for VIP mysql
  6. listening on all interfaces for mysql, port=9306
  7. listening on UNIX socket /var/run/mysqld/mysqld.sock
  8. listening on 172.17.0.17:9312 for sphinx
  9. listening on all interfaces for http, port=9308
  10. prereading 0 indexes
  11. prereaded 0 indexes in 0.000 sec
  12. accepting connections

Starting Manticore in Windows

On Windows if you want Manticore to start at boot, install Manticore as Windows Service.

Manticore can be started and stopped from the Control Panel or from the command line:

  1. sc.exe start Manticore
  1. sc.exe stop Manticore

If you don’t install Manticore as Windows service, you can start it from the command line:

  1. .\bin\searchd -c manticore.conf

Starting Manticore in MacOS

Starting Manticore via HomeBrew package manager

If Manticore is installed using HomeBrew, you can run it as a Brew service.

To start Manticore run the following command:

  1. brew services start manticoresearch

To stop Manticore run the following command:

  1. brew services stop manticoresearch