HashiCorp Vault – Consul HA Architecture !

About Vault and using Consul as backend-

Vault is designed from the ground up as a secret management solution. As such, it protects secrets in transit and at rest. It provides multiple authentication and audit logging mechanisms. Dynamic secret generation allows Vault to avoid providing clients with root privileges to underlying systems and makes it possible to do key rolling and revocation.

The strength of Consul is that it is fault tolerant and highly scalable. By using Consul as a backend to Vault, you get the best of both. Consul is used for durable storage of encrypted data at rest and provides coordination so that Vault can be highly available and fault tolerant. Vault provides the higher level policy management, secret leasing, audit logging, and automatic revocation.

from Vault documentation.

In my previous article, I covered the steps to setup Vault and Consul in development mode for testing and poc purpose. In this article we will see how to setup Vault – Consul in High Availability mode for production setups.

Architecture

The examples that follow shows consul setup with 3 servers and 1 client –

Servers maintain state and data sync, while client is a light weight machine which acts as intermediary between calling application and the server.

Servername	IP Address	Role
server1.example.com	172.31.15.152	Consul Server
server2.example.com	172.31.166.226	Consul Server
server3.example.com	172.31.122.234	Consul Server
client.example.com	172.31.165.11	Consul Client

Consul Server Configuration

Create encryption key using below command and set it in encrypt configuration for all server and client nodes to enable encrypted communication between the nodes. This encryption works at network level.

consul keygen

Create config file for server agent and place it in /etc/consul.d/server

{
"bind_addr": "172.31.15.152", // replace with server ip address
"datacenter": "dc1",
"data_dir": "/var/consul",
"log_level": "INFO",
"encrypt": "FPn25hJDtF0qFRMkcnIWzQ==",//Generated key. Should be same on all servers and client
"enable_syslog": true,
"enable_debug": true,
"node_name": "Server1", //Server1, Server2, Server3
"server": true,
"bootstrap_expect": 3,
"leave_on_terminate": false,
"skip_leave_on_interrupt": true,
"rejoin_after_leave": true,
"retry_join": [
  "172.31.15.152:8301",
  "172.31.166.226:8301",
  "172.31.122.234:8301",
  "172.31.165.11:8301"
  ]
}

Start Consul Server Node –

sudo consul agent -config-dir /etc/consul.d/server

Consul Client Configuration

Create config file on client node and place it in /etc/consul.d/client

{
"bind_addr": "172.31.165.11", //replace with server ip address
"datacenter": "dc1",
"data_dir": "/var/consul",
"log_level": "INFO",
"encrypt": "FPn25hJDtF0qFRMkcnIWzQ==",
"enable_syslog": true,
"enable_debug": true,
"node_name": "Client",
"enable_script_checks" : true,
"server": false,
"service": {"name": "Apache", "tags": ["HTTP"], "port": 80,
"check": {"script": "curl localhost >/dev/null 2>&1", "interval": "10s"}},
"rejoin_after_leave": true,
"retry_join": [
"172.31.15.152:8301"
]
}

Start Consul Client Node-

sudo consul agent -config-dir /etc/consul.d/client -client 172.31.10.18

Vault Configuration with Consul Client

Primary Node- config.hcl

storage "consul" {
  address = "172.31.10.18:8500" //consul client
  path    = "vault"
}
listener "tcp" {
 address     = "0.0.0.0:8200"
 tls_disable = 0
// ssl cert for vault, exclude if running vault without ssl
 tls_cert_file = "/etc/vault/sslcert.cer"
 tls_key_file  = "/etc/vault/vault.key"
}
disable_mlock = "true"
api_addr = https://<public IP Address of Vault server>:8200

Start Vault server using this config –

vault server -config=config.hcl

Secondary Node- config.hcl

storage "consul" {
 address = "172.31.10.18:8500" //consul client
 path = "vault"
}
listener "tcp" {
 address = "0.0.0.0:8200"
 tls_disable = 0
// ssl cert for vault, exclude if running vault without ssl
tls_cert_file = "/etc/vault/sslcert.cer" 
 tls_key_file = "/etc/vault/vault.key"
}
disable_mlock = "true"
api_addr = https://<public IP of vault primary server>:8200

Start Vault server using this config –

vault server -config=config.hcl

When running Vault on 2 nodes (Primary and Secondary), first node runs in active mode and second one remains on standby. If the active node goes down, the secondary node becomes active and starts serving the requests and this is how it maintains High Availability.

Refer for setup details – https://imaginea.gitbooks.io/consul-devops-handbook/content/

Thanks for checking out.

Please let us know in comment section in case of any issue.

Author
Recent Posts

Abhimanyu

Cloud and DevOps Professional and previously a C# ASP.NET Developer with keen interest in system design and architecture. Currently involved in projects using AWS, Azure, Automation, Monitoring, REST APIs and DevOps. Email: [email protected] / [email protected]