The uWSGI Legion subsystem

As of uWSGI 1.9-dev a new subsystem for clustering has been added: The Legionsubsystem. A Legion is a group of uWSGI nodes constantly fighting fordomination. Each node has a valor value (different from the others, ifpossible). The node with the highest valor is the Lord of the Legion (or ifyou like a less gaming nerd, more engineer-friendly term: the master). Thisconstant fight generates 7 kinds of events:

  • setup - when the legion subsystem is started on a node
  • join - the first time quorum is reached, only on the newly joined node
  • lord - when this node becomes the lord
  • unlord - when this node loses the lord title
  • death - when the legion subsystem is shutting down
  • node-joined - when any new node joins our legion
  • node-left - when any node leaves our legionYou can trigger actions every time such an event rises.

Note: openssl headers must be installed to build uWSGI with Legion support.

IP takeover

This is a very common configuration for clustered environments. The IP addressis a resource that must be owned by only one node. For this example, that nodeis our Lord. If we configure a Legion right (remember, a single uWSGIinstances can be a member of all of the legions you need) we could easilyimplement IP takeover.

  1. [uwsgi]
  2.  
  3. legion = clusterip 225.1.1.1:4242 98 bf-cbc:hello
  4. legion-node = clusterip 225.1.1.1:4242
  5.  
  6. legion-lord = clusterip cmd:ip addr add 192.168.173.111/24 dev eth0
  7. legion-lord = clusterip cmd:arping -c 3 -S 192.168.173.111 192.168.173.1
  8.  
  9. legion-setup = clusterip cmd:ip addr del 192.168.173.111/24 dev eth0
  10. legion-unlord = clusterip cmd:ip addr del 192.168.173.111/24 dev eth0
  11. legion-death = clusterip cmd:ip addr del 192.168.173.111/24 dev eth0

In this example we join a legion named clusterip. To receive messages fromthe other nodes we bind on the multicast address 225.1.1.1:4242. The valor ofthis node will be 98 and each message will be encrypted using Blowfish in CBCwith the shared secret hello. The legion-node option specifies thedestination of our announce messages. As we are using multicast we only need tospecify a single “node”. The last options are the actions to trigger on thevarious states of the cluster. For an IP takeover solution we simply rely onthe Linux iproute commands to set/unset ip addresses and to send an extraARP message to announce the change. Obviously this specific example requiresroot privileges or the CAP_NET_ADMIN Linux capability, so be sure to notrun untrusted applications on the same uWSGI instance managing IP takeover.

The Quorum

To choose a Lord each member of the legion has to cast a vote. When all of theactive members of a legion agree on a Lord, the Lord is elected and the oldLord is demoted. Every time a new node joins or leaves a legion the quorum isre-computed and logged to the whole cluster.

Choosing the Lord

Generally the node with the higher valor is chosen as the Lord, but there canbe cases where multiple nodes have the same valor. When a node is started aUUID is assigned to it. If two nodes with same valor are found the one with thelexicographically higher UUID wins.

Split brain

Even though each member of the Legion has to send a checksum of its internalcluster-membership, the system is still vulnerable to the split brain problem.If a node loses network connectivity with the cluster, it could believe it isthe only node available and starts going in Lord mode.

For many scenarios this is not optimal. If you have more than 2 nodes in alegion you may want to consider tuning the quorum level. The quorum level isthe amount of votes (as opposed to nodes) needed to elect a lord.legion-quorum is the option for the job. You can reduce the split brainproblem asking the Legion subsystem to check for at least 2 votes:

  1. [uwsgi]
  2.  
  3. legion = clusterip 225.1.1.1:4242 98 bf-cbc:hello
  4. legion-node = clusterip 225.1.1.1:4242
  5.  
  6. legion-quorum = clusterip 2
  7.  
  8. legion-lord = clusterip cmd:ip addr add 192.168.173.111/24 dev eth0
  9. legion-lord = clusterip cmd:arping -c 3 -S 192.168.173.111 192.168.173.1
  10.  
  11. legion-setup = clusterip cmd:ip addr del 192.168.173.111/24 dev eth0
  12. legion-unlord = clusterip cmd:ip addr del 192.168.173.111/24 dev eth0
  13. legion-death = clusterip cmd:ip addr del 192.168.173.111/24 dev eth0

As of 1.9.7 you can use nodes with valor 0 (concept similar to MongoDB’sArbiter Nodes), such nodes will be counted when checking for quorum but maynever become The Lord. This is useful when you only need a couple nodes whileprotecting against split-brain.

Actions

Each one of the four phases of a legion can trigger an action. The actionssystem is modular so you can add new kinds of actions. Currently the supportedactions are:

cmd:<command>

Run a shell command.

signal:<num>

Raise a uWSGI signal.

log:<msg>

Log a message. For example you could combine the log action with the alarm subsystem to have cluster monitoring for free.

Multicast, broadcast and unicast

Even if multicast is probably the easiest way to implement clustering it is notavailable in all networks. If multicast is not an option, you can rely onnormal IP addresses. Just bind to an address and add all of the legion-nodeoptions you need:

  1. [uwsgi]
  2.  
  3. legion = mycluster 192.168.173.17:4242 98 bf-cbc:hello
  4. legion-node = mycluster 192.168.173.22:4242
  5. legion-node = mycluster 192.168.173.30:4242
  6. legion-node = mycluster 192.168.173.5:4242

This is for a cluster of 4 nodes (this node + 3 other nodes)

Multiple Legions

You can join multiple legions in the same instance. Just remember to usedifferent addresses (ports in case of multicast) for each legion.

  1. [uwsgi]
  2.  
  3. legion = mycluster 192.168.173.17:4242 98 bf-cbc:hello
  4. legion-node = mycluster 192.168.173.22:4242
  5. legion-node = mycluster 192.168.173.30:4242
  6. legion-node = mycluster 192.168.173.5:4242
  7.  
  8. legion = mycluster2 225.1.1.1:4243 99 aes-128-cbc:secret
  9. legion-node = mycluster2 225.1.1.1:4243
  10.  
  11. legion = anothercluster 225.1.1.1:4244 91 aes-256-cbc:secret2
  12. legion-node = anothercluster 225.1.1.1:4244

Security

Each packet sent by the Legion subsystem is encrypted using a specified cipher,a preshared secret, and an optional IV (initialization vector). Depending oncipher, the IV may be a required parameter. To get the list of supportedciphers, run openssl enc -h.

Important

Each node of a Legion has to use the same encryption parameters.

To specify the IV just add another parameter to the legion option.

  1. [uwsgi]
  2.  
  3. legion = mycluster 192.168.173.17:4242 98 bf-cbc:hello thisistheiv
  4. legion-node = mycluster 192.168.173.22:4242
  5. legion-node = mycluster 192.168.173.30:4242
  6. legion-node = mycluster 192.168.173.5:4242

To reduce the impact of replay-based attacks, packets with a timestamp lowerthan 30 seconds are rejected. This is a tunable parameter. If you have nocontrol on the time of all of the nodes you can increase the clock skewtolerance.

Tuning and Clock Skew

Currently there are three parameters you can tune. These tuables affect allLegions in the system. The frequency (in seconds) at which each packet is sent(legion-freq <secs>), the amount of seconds after a node not sendingpackets is considered dead (legion-tolerance <secs>), and the amount ofclock skew between nodes (legion-skew-tolerance <secs>). The Legionsubsystem requires tight time synchronization, so the use of NTP or similar ishighly recommended. By default each packet is sent every 3 seconds, a node isconsidered dead after 15 seconds, and a clock skew of 30 seconds is tolerated.Decreasing skew tolerance should increase security against replay attacks.

Lord scroll (coming soon)

The Legion subsystem can be used for a variety of purposes ranging from masterelection to node autodiscovery or simple monitoring. One example is to assigna “blob of data” (a scroll) to every node, One use of this is to passreconfiguration parameters to your app, or to log specific messages. Currentlythe scroll system is being improved upon, so if you have ideas join our mailinglist or IRC channel.

Legion API

You can know if the instance is a lord of a Legion by simply calling

  1. int uwsgi_legion_i_am_the_lord(char *legion_name);

It returns 1 if the current instance is the lord for the specified Legion.

  • The Python plugin exposes it as uwsgi.i_am_the_lord(name)
  • The PSGI plugin exposes it as uwsgi::i_am_the_lord(name)
  • The Rack plugin exposes it as UWSGI::i_am_the_lord(name)

Obviously more API functions will be added in the future, feel free to expose your ideas.

Stats

The Legion information is available in the The uWSGI Stats Server. Be sure tounderstand the difference between “nodes” and “members”. Nodes are the peer youconfigure with the legion-node option while members are the effective nodesthat joined the cluster.

The old clustering subsystem

During 0.9 development cycle a clustering subsystem (based on multicast) wasadded. It was very raw, unreliable and very probably no-one used it seriously.The new method is transforming it in a general API that can use differentbackends. The Legion subsystem can be one of those backends, as well asprojects like corosync or the redhat cluster suite.