Containers - What Are They Good For? Running Our CI Builds

In the last post, I looked at creating a build environment, settling on a strategy where I used Docker Compose to create our build environment. From there, I need some way to actually run our build in an environment.

Typically, my Windows build run in some sort of hosted agent, whether it's AppVeyor or VSTS. The major advantage here is there's some pool of build agents that I don't really need to worry about, and it can run our build.

Things get a little modified in our containerized approach, as we still need something to kick off our build. There are containerized CI solutions out there, such as CircleCI, but as far as I can tell, they're Linux/macOS-only. Not helpful for us, as we want to target Windows build environments.

However, there are public build agents that do support Windows Docker containers, the first one being AppVeyor.

AppVeyor Docker Builds

We've already got our basic CI build (from the last post), so now all that's left is executing it in an AppVeyor build. AppVeyor does support Docker, with base images already installed in the cache. All we need to do is create our build, in my case with an appveyor.yml file:

version: 1.0.{build}
image: Visual Studio 2017
build_script:
- ps: .\ci.ps1

Nothing that exciting, as we've already built out our Docker-based build. Once we've got this ready, we can push up our code.

....and wait.

....and wait.

....

....and finally get a result! Failure :(

Build started
git clone -q --branch=master https://github.com/jbogard/contosouniversitydocker.git C:\projects\contosouniversitydocker
git checkout -qf eb996f64cbe61e3349bf901542e8037e17bc1b05
.\ci.ps1
Pulling test-db (microsoft/mssql-server-windows-developer:)...
latest: Pulling from microsoft/mssql-server-windows-developer
Digest: sha256:a3e77eb7ac136bf419269ab6a3f3387df5055d78b2b6ba2e930e1c6312b50e07
Status: Downloaded newer image for microsoft/mssql-server-windows-developer:latest
Building ci
Step 1/11 : FROM microsoft/powershell:nanoserver-1709 AS installer-env
nanoserver-1709: Pulling from microsoft/powershell
Service 'ci' failed to build: a Windows version 10.0.16299-based image is incompatible with a 10.0.14393 host
Exec: 

So we've got a problem here, our container image isn't exactly compatible with the AppVeyor host. Another problem we have is because we decided to use an off-the-shelf SQL Server image, it took 14 minutes to download that image.

Yikes!

There are some ways we can fix all this, but ultimately, it comes down to:

• Finding smaller images for our database
• Finding a supported image for our "starting point" images

I can't really do anything about a smaller database, other than switching to another database. As we saw in a previous post, this could mean SQL Local DB, but we run into the problem that it's quite hard to install from the command line without using Chocolatey. Which requires PowerShell. Which requires .NET 4.5, which requires an older nanoserver image.

As for the starting point image, one thing we can do to make our lives a little easier is to build our "build image" and push this to a container registry. That way our build doesn't have to build our CI image from scratch, instead it's all built, and ready to go.

Addressing our options

I don't want to walk through each (painful) step for me, but I tried a number of different options for running our build:

• AppVeyor, with Docker Compose
• VSTS, with Docker Compose
• VSTS, with Docker steps
• VSTS, with Docker steps and our images in Azure Container Registry (ACR)
• VSTS, with a custom agent hosted in Azure Kubernetes Service (AKS) and images in ACR

In the last example, I ditched the hosted agents and went purely with my own hosted agent. This meant that my CI build image was no longer a blank image, but the VSTS image so that it could connect with VSTS and build. My Kubernetes configuration then looked like:

apiVersion: v1
kind: ReplicationController
metadata:
  name: vsts-agent
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: vsts-agent
        version: "0.1"
    spec:
      containers:
      - name: vsts-agent
        image: jbogard.azurecr.io/vsts-agent:windows-10.0.14393
        env:
          - name: VSTS_ACCOUNT
            valueFrom:
              secretKeyRef:
                name: vsts
                key: VSTS_ACCOUNT
          - name: VSTS_TOKEN
            valueFrom:
              secretKeyRef:
                name: vsts
                key: VSTS_TOKEN
          - name: VSTS_POOL
            value: k8s-windowsservercore
        volumeMounts:
        - mountPath: /var/run/docker.sock
          name: docker-volume
      imagePullSecrets:
        - name: vsts-agent-login
      nodeSelector:
        beta.kubernetes.io/os: windows
      volumes:
      - name: docker-volume
        hostPath:
          path: /var/run/docker.sock

Yikes. At this point, I basically gave up. My build times did finally go down (from 30 minutes to ~5) on Kubernetes, but now I was managing my own k8s cluster just to run a build. It wasn't elastic either, I couldn't spin up new agents on demand without introducing Azure Functions, web hooks, and a bunch of other things that would take me forever to get set up.

Final conclusion

After all of this effort, I found that for the container experience on Windows, it works really well for local dependencies and building production-runnable instances.

But as a build/CI environment, it's still a poor experience. Either I completely change my targets (going from Windows containers to Linux), or have always-on container hosts that can cache my images effectively.

Where I still find hope for container images on Windows are for a better packaging and execution mechanism for my applications. Today, I largely use Octopus Deploy or VSTS and their artifact mechanisms to package my application for production:

Instead of using NuGet, a fancy zip file with my code and a manifest, what about a fully ready-to-run application? This is where I'd like to see my systems move towards - a better packaging mechanism that includes everything needed to run my application. Whether this is Docker or a better linker, I'm not sure yet, but something that packages my application and its runtime to ensure portability would reduce the amount of setup and headaches when I get to production.