diff --git a/docs/PIPELINES.md b/docs/PIPELINES.md index 5d342176..55e9acff 100644 --- a/docs/PIPELINES.md +++ b/docs/PIPELINES.md @@ -1,6 +1,26 @@ Pipelines ========= +- [Pipelines](#pipelines) + * [Propagation](#propagation) + * [Configuration](#configuration) + + [Schema](#schema) + - [Pipeline YAML Field Guide](#pipeline-yaml-field-guide) + - [Secrets](#secrets) + - [Git Integration](#git-integration) + - [Locker Integration](#locker-integration) + - [Auto-Update Genesis Assets](#auto-update-genesis-assets) + - [Notifications](#notifications) + * [Notification: Slack](#notification--slack) + * [Notification: Email](#notification--email) + - [BOSH Associations](#bosh-associations) + - [Task Configuration](#task-configuration) + - [Groups](#groups) + - [Pipeline Override Configuration](#pipeline-override-configuration) + + [Useful commands](#useful-commands) + - [Retrieve **bosh** configuration](#retrieve---bosh---configuration) + - [Retrieve **vault** configuration](#retrieve---vault---configuration) + Concourse is an integral part of Genesis v2. But rather than force operators to define the full structure of a Concourse deployment pipeline (which can get rather complex), we want to provide them a domain-specific language for @@ -306,7 +326,7 @@ pipeline: #### Secrets - **pipeline.vault.url** - The URL of your Vault installation, i.e. - `https://vault.example.com`. This is **required**. + `https://vault.example.com` or `https://10.0.0.5`. This is **required**. - **pipeline.vault.role** - The AppRole GUID of a given Vault AppRole, used to generate temporary tokens for Vault-accessing during deploys. This field is @@ -543,19 +563,101 @@ being deployed, or vice versa. #### Pipeline Override Configuration If you need to edit and make changes to your deployment pipeline, simply add -the changes you need to the bottom of your ci.yml located in your -*-deployments repo. For example, if we have a pipeline we need to edit the -'check_every:' parameter of a resource named git we can add the block below -under our pipeline layouts: +the changes you need to the bottom of your `ci.yml` located in your +`*-deployments` repo to be merged by Spruce. + +For example, if we have a pipeline in which we need to enforce some +`check_every:` parameter on a resource named `git`, we can add the following +`resources:` block below under the pipeline generation settings specified in the +`pipeline:` section. **ci.yml** ``` -layouts: - test-sandbox-ops: |+ - auto *test-sandbox *test-staging - test-sandbox -> test-staging +pipeline: + + # (...) resources: -- name: git - check_every: 15m + - name: git # <-- required by default Spruce array merging, based on `name:` keys + check_every: 15m +``` + +Spruce will merge anything outside the `pipeline:` section on top of the +generated Concourse pipeline definition. + +### Useful commands + +#### Retrieve **Bosh** configuration + +With recent Genesis versions, the Bosh URL and credentials are now grabbed from +the exodus data. + +Furthermore, even in situations where the Bosh password and CA certificate are +required, they should come from the Concourse integrated Vault. Typical path +would be `concourse///` and default value +for secrets is to be set under a `value:` key in Vault, as advised +[in the Concourse documentation][vault_creds_lookup_rules]. + +[vault_creds_lookup_rules]: https://concourse-ci.org/vault-credential-manager.html#vault-credential-lookup-rules + +In other situations where tests needs to be made, Spruce `(( vault ))` +directives can be used. In order to fill out the `pipeline.boshes.*` section +first we need to know all of the informations. + +- get Bosh URL +- get Bosh username and password +- get Bosh CA certificate + +All of those params are available under: + +```bash +cd my-bosh-deployment #[ex. deployments/bosh] +genesis info [env] +``` + +Where `[env]` is the name of the env/yaml file. + +You should see that env `url`, `username`, `password` and `ca certificate`. Best +approach is to put them into the Vault vs plaintext in `ci.yml`. + +By convention, they should be stored under something similar to: + +```bash +secret/[env]/bosh/ssl/ca:certificate +secret/[env]/bosh/users/[user]:password ``` + +Make sure credentials are there before applying! + +#### Retrieve **Vault** configuration + +With recent Genesis versions, the genesis-pipeline app role should be +autodiscovered by the pipeline. + +To configure Vault access and: + +- get vault url +- get vault role and secret + +``` +cd my-vault-deployment #[ex. deployments/vault] +genesis info [env] +``` + +Where `[env]` is the name of the env/yaml file. + +If you don't have Vault installed via kit, try to get that information using +`safe` CLI or `vault` CLI directly: + +```bash +safe targets # shows all targets and `url` +safe status # shows all servers under current target +safe env # will get you url and `VAULT_TOKEN` +``` + +Once you get `VAULT_TOKEN` you can configure `approle` in it via API or `vault` +CLI. + +It is far more convenient to use the kit, but there is existing documentation +from Vault how to proceed from that point: +https://www.vaultproject.io/docs/auth/approle