Network Segments

Enterprise

This feature requires version 0.9.3+ of self-managed Consul Enterprise. Refer to the enterprise feature matrix for additional information.

Consul requires full connectivity between all agents (servers and clients) in a datacenter within a given LAN gossip pool. By default, all Consul agents will be a part of one shared Serf LAN gossip pool known as the <default> network segment, thus requiring full mesh connectivity within the datacenter.

In some environments, full connectivity between all agents is not possible—known as a “segmented network”. This is usually the result of business policies enforced through network rules or firewalls. To use Consul in a segmented network, you must break up the LAN gossip pool along network communication boundaries into separate “network segments”. Network segments are isolated LAN gossip pools that only require full connectivity between agent members on the same segment.

To get started with network segments you can review the tutorial on HashiCorp Learn for Network Segments.

Info: Network segments enable you to operate a Consul datacenter without full mesh (LAN) connectivity between agents. To federate multiple Consul datacenters without full mesh (WAN) connectivity between all server agents in all datacenters, use Network Areas (Enterprise).

Consul Networking Models

To help set context for this feature, it is useful to understand the various Consul networking models and their capabilities.

Cluster: A set of Consul servers forming a Raft quorum along with a collection of Consul clients, all set to the same datacenter, and joined together to form what we will call a “local cluster”. Consul clients discover the Consul servers in their local cluster through the gossip mechanism and make RPC requests to them. LAN Gossip (OSS) is an open intra-cluster networking model, and Network Segments (Enterprise) creates multiple segments within one cluster.

Federated Cluster: A set of connected clusters, each representing a unique Consul “datacenter”. These Consul servers are federated together over the WAN. Consul clients make use of resources in federated clusters by forwarding RPCs through the Consul servers in their local cluster, but they never interact with remote Consul servers directly. There are currently two inter-cluster network models which can be viewed on HashiCorp Learn: WAN gossip (OSS) and Network Areas (Enterprise).

LAN Gossip Pool: A set of Consul agents that have full mesh connectivity among themselves, and use Serf to maintain a shared view of the members of the pool for different purposes, like finding a Consul server in a local cluster, or finding servers in a remote cluster. A segmented LAN Gossip Pool limits a group of agents to only connect with the agents in its segment.

Network Segments Configuration

Server Configuration

Server agents are members of all segments. The datacenter includes a <default> segment, as well as additional segments defined in the segments server agent configuration option. Each additional segment is defined by:

• a non-empty name
• a unique port
• optionally, a custom bind and advertise address for the additional segment’s Serf LAN listener on the server

Note: Prior to Consul 1.7.3, a Consul server agent configured with too many network segments may not be able to start due to limitations in Serf.

Example Server Configuration

The following server agent configuration will create two user-defined network segments: alpha and beta.

Example network segments configuration for server agents

Example network segments configuration for server agents

HCL

• HCL
• JSON

segments = [
  {
    name = "alpha"
    bind = "{{GetPrivateIP}}"
    advertise = "{{GetPrivateIP}}"
    port = 8303
  },
  {
    name = "beta"
    bind = "{{GetPrivateIP}}"
    advertise = "{{GetPrivateIP}}"
    port = 8304
  }
]

segments = [
  {
    name = "alpha"
    bind = "{{GetPrivateIP}}"
    advertise = "{{GetPrivateIP}}"
    port = 8303
  },
  {
    name = "beta"
    bind = "{{GetPrivateIP}}"
    advertise = "{{GetPrivateIP}}"
    port = 8304
  }
]

{
  "segments": [
    {
      "name": "alpha",
      "bind": "{{GetPrivateIP}}",
      "advertise": "{{GetPrivateIP}}",
      "port": 8303
    },
    {
      "name": "beta",
      "bind": "{{GetPrivateIP}}",
      "advertise": "{{GetPrivateIP}}",
      "port": 8304
    }
  ]
}

{
  "segments": [
    {
      "name": "alpha",
      "bind": "{{GetPrivateIP}}",
      "advertise": "{{GetPrivateIP}}",
      "port": 8303
    },
    {
      "name": "beta",
      "bind": "{{GetPrivateIP}}",
      "advertise": "{{GetPrivateIP}}",
      "port": 8304
    }
  ]
}

The server agent configuration options relevant to network segments are:

• ports.serf_lan: The Serf LAN port on this server for the <default> network segment’s gossip pool.
• segments: A list of user-defined network segments on this server, including their names and Serf LAN ports.

Client Configuration

Each client agent can only be a member of one segment at a time. This will be the <default> segment unless otherwise specified in the agent’s segment agent configuration option.

Join a Client to a Segment

For a client agent to join the Consul datacenter, it must connect to another agent (client or server) within its configured segment.

Clients A and B specify the same segment S. Client B is already joined to the segment S LAN gossip pool. Client A wants to join via Client B. In order to do so, Client A must connect to Client B’s configured Serf LAN port.

There are several methods to specify the port used by the join operation, listed in order of precedence:

1. Specify an explicit port in the join address. This can be done at the CLI when starting the agent (e.g., consul agent -retry-join "client-b-address:8303"), or in the agent’s configuration using the retry-join option. This method is not compatible with cloud auto-join.

2. Specify an alternate Serf LAN port for the agent. This can be done at the CLI when starting the agent (e.g., consul agent -retry-join "client-b-address" -serf-lan-port 8303), or in the agent’s configuration using the serf_lan option. When a Serf LAN port is not explicitly specified in the join address, the agent will attempt to join the target host at the Serf LAN port specified in CLI or agent configuration.

3. Use the default Serf LAN port (8301). The agent will attempt to join the target host on port 8301.

Agents within a segment can use different port numbers for their Serf LAN port. For example, on the <default> segment, Server S can use port 8301, Client A can use 8303, and Client B can use 8304. However, if an agent wishes to join a segment via an agent that uses a different port number, the target agent’s Serf LAN port must be specified in the join address (method 1 above).

Example Client Configuration

The following client agent configuration will cause the agent to:

• Open a Serf LAN listener port on 8303.
• Attempt to join the cluster via servers on port 8303 (since an alternate port is not specified in the retry_join addresses).

Example network segment configuration for client agents

Example network segment configuration for client agents

HCL

• HCL
• JSON

node_name = "consul-client1"
retry_join = ["consul-server1", "consul-server2", "consul-server3"]
segment = "alpha"
ports = {
  serf_lan = 8303
}

node_name = "consul-client1"
retry_join = ["consul-server1", "consul-server2", "consul-server3"]
segment = "alpha"
ports = {
  serf_lan = 8303
}

{
  "node_name": "consul-client1",
  "retry_join": ["consul-server1", "consul-server2", "consul-server3"],
  "segment": "alpha",
  "ports": {
    "serf_lan": 8303
  }
}

{
  "node_name": "consul-client1",
  "retry_join": ["consul-server1", "consul-server2", "consul-server3"],
  "segment": "alpha",
  "ports": {
    "serf_lan": 8303
  }
}

The client agent configuration options relevant to network segments are:

• segment: The name of the network segment this client agent belongs to.
• ports.serf_lan: Serf LAN port for the above segment on this client. This is not required to match the configured Serf LAN port for other agents on this segment.
• retry_join or start_join: A list of agent addresses to join when starting. Ensure the correct Serf LAN port for this segment is used when joining the LAN gossip pool using one of the available configuration methods.