This article describes Hashicorp Vault installation on Kubernetes cluster using Helm chart

Prerequisites:

- running kubernetes cluster;

- Helm installed and configured.

The Vault server can be started in 3 ways:
- dev (inmemory backend, no sealing, not for production);
- standalone (any supported backend, sealing, not recommended for production);
- HA mode (only certain list of backends described below, sealing, production usage).

Possible backends for HA mode: Consul, zookeeper, etcd, mysql. Official documentation recommends using consul as a backend in HA mode.
Mysql backend seems not to be the case for the deployment with maria_db galera cluster since it has a limitation of using GET_LOCK mysql function (https://mariadb.com/kb/en/library/mariadb-galera-cluster-known-limitations/)

Before usage newly-installed server should be initialized (once)– root token and unseal keys will be generated once at this stage. After that it should be unsealed using unseal keys (by default 3 out of 5 should be entered) stored somewhere. Moreover unseal action should take place after each restart of the server. In the HA cluster each instance of Vault server should be unsealed each time after startup.

Available Vault Helm charts/operators:

https://github.com/helm/charts/tree/master/incubator/vault

https://github.com/hashicorp/vault-helm (hashicorp official)
pros: official chart
cons: no custom unsealing mechanism at all (only officially supported auto-unsealing configurations using Google or AWS key management systems) hence it is needed to develop own mechanism to init server if needed, to store keys and root token, to export them to k8s object to make readable to 3rd-party applications like barbican

https://github.com/banzaicloud/bank-vaults/tree/master/charts/vault
pros: Unsealing mechanism is included as a separate container inside pod. Root token and unseal keys could be written to k8s secret thus information could be read by barbican to retrieve root token.
Unseal options:
- k8s secret
- file
- aws-kms-s3
- google-cloud-kms-gcs
cons: unsealing mechanism uses separate image with GO utility

Vault operator:
https://github.com/banzaicloud/bank-vaults/tree/master/operator

 Vault installation with etcd backend
Install etcd:
Chart : https://github.com/helm/charts/tree/master/incubator/etcd

helm repo add incubator http://storage.googleapis.com/kubernetes-charts-incubator
helm upgrade --install etcd-vault --namespace openstack incubator/etcd

Install Vault:
Chart: https://github.com/banzaicloud/bank-vaults/tree/master/charts/vault

helm repo add banzaicloud-stable http://kubernetes-charts.banzaicloud.com/branch/master

Contents of values.yaml:

image:
  repository: vault
  tag: 1.2.2
  pullPolicy: IfNotPresent
replicaCount: 3
vault:
  config:
    listener:
      tcp:
        address: '[::]:8200'
        tls_disable: true
        tls_cert_file: /vault/tls/server.crt
        tls_key_file: /vault/tls/server.key
    api_addr: http://vault.openstack.svc.kaas-kubernetes-da8f45c5b53311e98fc5fa163e5a4837:8200
    ui: false
    storage:
      etcd:
        address: http://etcd-vault.openstack.svc.kaas-kubernetes-da8f45c5b53311e98fc5fa163e5a4837:2379
        ha_enabled: "true"
unsealer:
  image:
    repository: banzaicloud/bank-vaults
    tag: 0.5.0
    pullPolicy: IfNotPresent
  args: [
      "--mode",
      "k8s",
      "--k8s-secret-namespace",
      "openstack",
      "--k8s-secret-name",
      "vault-keys"
  ]
  metrics:
    enabled: false
    debug: true
    name: metrics
    type: ClusterIP
    port: 9091
    serviceMonitor:
      enabled: false
      additionalLabels: {}
    annotations:
      prometheus.io/scrape: "true"
      prometheus.io/path: "/metrics"
      prometheus.io/port: "9091"

Important: make sure to use your own DNS names!

Install chart:

helm upgrade --install vault -f ./values_vault.yaml --namespace openstack banzaicloud-stable/vault

Unsealing:
This installation uses “k8s” unseal mode – the server is initialized and the k8s secret with a name specified in values will be created containing root token and unseal keys. Important: if the secret is deleted – the server with existing backend won`t be able to start after next restart/re-installation. If the secret is present – the server(s) will be unsealed automatically on restart.

Check k8s secret with root token and unseal keys:

kubectl get secret vault-config -n openstack -o yaml

Check vault installation:

apt install jq -y
export VAULT_ADDR=http://vault.openstack.svc.kaas-kubernetes-da8f45c5b53311e98fc5fa163e5a4837:8200
export VAULT_TOKEN=$(kubectl get secrets vault-keys -n openstack -o json | jq -r '.data["vault-root"]' | base64 –decode)
wget https://releases.hashicorp.com/vault/1.2.2/vault_1.2.2_linux_amd64.zip
unzip vault_1.2.2_linux_amd64.zip
mv vault /usr/bin/
vault status

Write test secret to Vault:

vault kv put secret/hello foo=bar