What does it take to run Concourse? The tl;dr here is that in order to run Concourse for one or more teams, you will need:

  1. A secrets manager (e.g. Vault)
  2. A container registry (e.g. Harbor)

Let’s dig into why you need these two types of applications if you choose to run Concourse.

A Secrets Manager

Secrets managers, like Vault, securely store your secrets. Setup correctly, you can trust them to safely store and retrieve your secrets.

Concourse does securely store some stuff, but it’s not Concourse’s main job to securely store secrets, or give you any convenient way of retrieving them. You can get away with not using a secrets manager with Concourse, but it will make your life much hard.

For example, most Concourse users will probably have some pipeline that needs to track commits to some private git repository your team owns. They probably have a resource like this in their pipelines:

resources:
- name: source-code
  type: git
  source:
    uri: git@github.com:concourse/concourse.git
    branch: main
    private_key: |
      -----BEGIN RSA PRIVATE KEY-----
      MIIEowIBAAKCAQEAtCS10/f7W7lkQaSgD/mVea
      ...
      -----END RSA PRIVATE KEY-----

You could leave your private key stored in the pipeline like that, but you won’t want to commit your pipeline to a git repository with a secret stored in it. That’s a classic way for secrets to leak out; storing them in git repos!

Using a secrets manager allows you to clear out secrets from your pipeline configuration and leave only path references to where the secret is stored in your secrets manager. Your resources will look like this instead:

resources:
- name: source-code
  type: git
  source:
    uri: git@github.com:concourse/concourse.git
    branch: main
    private_key: ((git.private_key))

Using a secrets manager also allows you to DRY your pipeline configuration as well since you only need to reference values. References usually don’t change as often as their underlying values.

A secrets manager is a pretty useful thing to add to your Concourse deployment. Most teams will get a lot of mileage out of using one.

You can see all the secrets managers Concourse is compatible with here.

A Container Registry

Concourse is a container-based CI/CD tool. Those containers need to be stored somewhere. Back when Docker Hub was giving away storage for free, most people uploaded their container images there. Now that the free lunch is over, everyone has to figure out some private place to store their containers. Thankfully, running a container registry is such a common request in our Cloud Native world that all the major cloud providers offer a managed container registry service that you can easily use.

Okay, so far I’ve told you it’s super easy to run a container registry, just pick one of the managed ones! Now why do you need to use a container registry with Concourse? Because you will definitely, at some point, need to build your own container to run in your pipelines.

The first time teams run into this need, to build their own container image, is when they’re writing a task step. You’ll probably write some script that requires some CLI tool that just isn’t baked into any container image you can easily pull down, like ubuntu:22.04.

You can work around this temporarily by installing all your scripts dependencies at the start of the script, but that’ll burn a bunch of bandwidth, CPU cycles, and most importantly, time. You can easily get that time back by building your own container image that contains the tools needed in your task step.

The second time will be when you need to build your own resource-type. Resource-types are Concourse’s main extension point for when you want Concourse to interface with some external system. Just like the container image used in task steps, you need to store that container image somewhere, and the best place for that is a container registry.

So if you’re deploying Concourse, you’ll want a container registry within arms reach.

Is it Worth the Overhead?

I don’t know if there’s a way to avoid this kind of overhead!

  • Every CI/CD tool I know of has to either own the “secrets manager” problem themselves or offload it to a third-party (like Concourse)
  • If you’re doing anything with containers then you need to have a container registry. We haven’t figured out a way to torrent container images between dev machines (yet!)

If you’re deploying Concourse, be prepared to deploy more apps to handle 1) secrets management, and 2) container storage.