About Vault and using Consul as backend-
Vault is designed from the ground up as a secret management solution. As such, it protects secrets in transit and at rest. It provides multiple authentication and audit logging mechanisms. Dynamic secret generation allows Vault to avoid providing clients with root privileges to underlying systems and makes it possible to do key rolling and revocation.
The strength of Consul is that it is fault tolerant and highly scalable. By using Consul as a backend to Vault, you get the best of both. Consul is used for durable storage of encrypted data at rest and provides coordination so that Vault can be highly available and fault tolerant. Vault provides the higher level policy management, secret leasing, audit logging, and automatic revocation.
- from Vault documentation.
In my previous article, I covered the steps to setup Vault and Consul in development mode for testing and poc purpose. In this article we will see how to setup Vault – Consul in High Availability mode for production setups.
Architecture
The examples that follow shows consul setup with 3 servers and 1 client –
Servers maintain state and data sync, while client is a light weight machine which acts as intermediary between calling application and the server.
Servername | IP Address | Role |
server1.example.com | 172.31.15.152 | Consul Server |
server2.example.com | 172.31.166.226 | Consul Server |
server3.example.com | 172.31.122.234 | Consul Server |
client.example.com | 172.31.165.11 | Consul Client |
Consul Server Configuration
Create encryption key using below command and set it in encrypt configuration for all server and client nodes to enable encrypted communication between the nodes. This encryption works at network level.
consul keygen
- Create config file for server agent and place it in /etc/consul.d/server
{ "bind_addr": "172.31.15.152", // replace with server ip address "datacenter": "dc1", "data_dir": "/var/consul", "log_level": "INFO", "encrypt": "FPn25hJDtF0qFRMkcnIWzQ==",//Generated key. Should be same on all servers and client "enable_syslog": true, "enable_debug": true, "node_name": "Server1", //Server1, Server2, Server3 "server": true, "bootstrap_expect": 3, "leave_on_terminate": false, "skip_leave_on_interrupt": true, "rejoin_after_leave": true, "retry_join": [ "172.31.15.152:8301", "172.31.166.226:8301", "172.31.122.234:8301", "172.31.165.11:8301" ] }
Start Consul Server Node –
sudo consul agent -config-dir /etc/consul.d/server
Consul Client Configuration
- Create config file on client node and place it in /etc/consul.d/client
{ "bind_addr": "172.31.165.11", //replace with server ip address "datacenter": "dc1", "data_dir": "/var/consul", "log_level": "INFO", "encrypt": "FPn25hJDtF0qFRMkcnIWzQ==", "enable_syslog": true, "enable_debug": true, "node_name": "Client", "enable_script_checks" : true, "server": false, "service": {"name": "Apache", "tags": ["HTTP"], "port": 80, "check": {"script": "curl localhost >/dev/null 2>&1", "interval": "10s"}}, "rejoin_after_leave": true, "retry_join": [ "172.31.15.152:8301" ] }
Start Consul Client Node-
sudo consul agent -config-dir /etc/consul.d/client -client 172.31.10.18
Vault Configuration with Consul Client
Primary Node- config.hcl
storage "consul" { address = "172.31.10.18:8500" //consul client path = "vault" } listener "tcp" { address = "0.0.0.0:8200" tls_disable = 0 // ssl cert for vault, exclude if running vault without ssl tls_cert_file = "/etc/vault/sslcert.cer" tls_key_file = "/etc/vault/vault.key" } disable_mlock = "true" api_addr = https://<public IP Address of Vault server>:8200
Start Vault server using this config –
vault server -config=config.hcl
Secondary Node- config.hcl
storage "consul" { address = "172.31.10.18:8500" //consul client path = "vault" } listener "tcp" { address = "0.0.0.0:8200" tls_disable = 0 // ssl cert for vault, exclude if running vault without ssl tls_cert_file = "/etc/vault/sslcert.cer" tls_key_file = "/etc/vault/vault.key" } disable_mlock = "true" api_addr = https://<public IP of vault primary server>:8200
Start Vault server using this config –
vault server -config=config.hcl
When running Vault on 2 nodes (Primary and Secondary), first node runs in active mode and second one remains on standby. If the active node goes down, the secondary node becomes active and starts serving the requests and this is how it maintains High Availability.
Refer for setup details – https://imaginea.gitbooks.io/consul-devops-handbook/content/
Thanks for checking out.
Please let us know in comment section in case of any issue.
- Monitor Kubernetes Control Plane Services Availability with Heartbeat [ELK] - December 14, 2020
- Setup and operate ELK Stack on Kubernetes cluster using Argo CD - October 26, 2020
- Auto clear notification using Watcher - June 10, 2020