Jason's Blog

Kubernetes Cluster Migrations with Velero and Tanzu Mission Control

Tue, 02 Feb 2021 00:00:00 +0000

Hey folks! I ran into an interesting situation with a customer the other day. They are using Tanzu Mission Control (TMC) with their kubernetes clusters and are trying to migrate from EKS to Tanzu Kubernetes (TKG) Clusters. I’m going to totally ignore the relative merits of EKS vs TKG and go straight to the how of the migration.

The rest of this article is going to talk about how to use kubectl, the velero cli, and TMC to migrate workloads from one cluster to another using Velero’s backup and restore functionality. I hope this is interesting and if you have anything else you’d like to hear about please ping me on twitter or Linkedin.

Thanks!

The Set Up

The TKG cli
- v1.2.0
2 TKG clusters
- Called mine workload1 and workload2 cause I’m super creative
- Joining the clusters to TMC is out of scope for this article
  - Look here if you’d like an article on that
Access to Tanzu Mission Control
- See your friendly neighborhood vmware salesperson
The Velero cli
- v1.4.2
kubectl
- v1.19.3
The helm cli
- v3.1.2
An app to migrate
- We’ll be deploying wordpress from the Bitnami helm repository

Getting things ready

First thing’s first, lets deploy a wordpress instance using the well built and well currated helm charts from the good folks over at Bitnami. You can dig into them in more detail here.

In the code block below you’ll see the steps to add the bitnami helm repo and install wordpress in it’s own namespace.

# Add the helm repo if needed
helm repo add bitnami https://charts.bitnami.com/bitnami

# Update to be sure you have the latest charts
helm repo update

# Lookup wordpress
helm search repo wordpress

# Checkout the chart values
helm show values bitnami/wordpress | less

## We're going to override the blog name and the service type which you can find in the values file as wordpressBlogName and service.type.

# Create our new namespace
kubectl create ns wordpress

# Deploy wordpress
## use -n to set the namespace and use --set to override the variables we identified in the values file.
helm install new-blog bitnami/wordpress -n wordpress --set wordpressBlogName=new-blog --set service.type=ClusterIP

Take a second here and browse to your wordpress page. It should look something like this:

If you’re having trouble getting access to the page here’s the port-forward command I used: kubectl port-forward svc/new-blog-wordpress 8080:80 -n wordpress. The site will now be available at localhost:8080.

Backing it up

Quick disclosure here: Everything I’m telling you to do in the TMC console can be done instead via kubectl and the Velero cli. I’m not covering any of that in this article.

Enroll in Data Protection

Head over to your cluster group and find your clusters.

Once again I’m going to save everyone a little time and point you at the vmware docs for enabling data protection on your clusters. It’s a fairly painless process and VMware gives you an easy to use cloud formation template to get the environment set up.

You want to ensure you have data protection enabled on both clusters.

Run a Backup

In the TMC console you need to select your cluster, in my case workload1 and head over the the data protection tab. From there you can schedule a backup and you have a few options. Feel free to do full or by namespace, I selected my wordpress namespace from before so I could simplify my restore.

Once you have that done validate that your backup.

With that we have our starting point for migrating our wordpress blog from one cluster to another! The next section will have us heading back to the cli.

Migrating

I start out doing a quick check that velero is installed on my cluster and validate my versions at the same time.

$ velero version
Client:
        Version: v1.4.2
        Git commit: 56a08a4d695d893f0863f697c2f926e27d70c0c5
Server:
        Version: v1.4.2

Seeing that tells me velero is good to go. Make sure you checkout each cluster before moving on.

Finding the Backup

Going to take a quick detour and talk about kubernetes custom resource definitions (CRDs) and how they relate to tools that integrate with the kubernetes api. CRDs are a handy way to extend kubernetes and allow apps like velero to build their own constructs right into the cluster. You’re going to see examples using both kubectl and the velero cli to perform various tasks.

With that in mind we’re going to explore the kubernetes API to find our backup and figure out how we can get our target cluster, workload2, to see the backup of the source cluster, workload1. When you look at the clusters, in either the TMC console or the cli, you’ll only see backups on the source cluster. We’ll dive into that in the next section.

Velero Contructs

Velero is going to create a bunch of new object types but we’re really only interested in 2 at this point. Backups, and Backup Storage Locations. You can see a bit more by running the following command: k get crd | grep velero.io

Backups

Neither construct is particularly complicated. Backups are just that, and they’re stored, and retrieved from, backup locations. We can see when we ask our clusters about backups workload1, the source cluster, sees a backup called wp. The backup wp represents the wordpress blog we want to move from workload1 to workload2. Unfortunately for us only our source cluster can see it for now. You can explore the clusters witht eh following commands.

# With the velero cli
velero backup get

# via kubectl
kubectl get backups -n velero
## Velero puts it's backups and other things in the velero namespace by default

You can see from the output that the backup is only visable from workload1. We’re going to checkout the backup locations and see what we can do about it.

Backup Locations

Looking at the screen grab you can see my storage location for wp is listed as jmo-dp. In fact when we look at both workload1 and workload2 they each have a backup location called jmo-dp but when you dig deeper you see that the values for each don’t line up.

# getting the backup location via velero
velero backup-location get

# getting the backup location via kubectl
kubectl get backupstoragelocations.velero.io -n velero

If you run the velero cli version of that command you’ll see something like the image below and you’ll note that while the s3 bucket name is the same the prefix the backup gets stored at are different.

Going to go ahead and spoil any remaining mystery we have at this point. In order to get our cross cluster restore working we need to copy over our backup location from workload1 to workload2. When you dig into the backup storage location on workload1 you’ll see an object that looks a lot like this:

# The below yaml has been modified to drop non required fields
apiVersion: velero.io/v1
kind: BackupStorageLocation
metadata:
  name: jmo-dp
  namespace: velero
spec:
  config:
    bucket: vmware-tmc-1234
    profile: jmo-dp
    s3ForcePathStyle: "false"
  objectStorage:
    bucket: vmware-tmc-1234
    prefix: 01ET5VEFZTES40MAJYZ98MC6G6/
  provider: aws

In order to successfully do a cross cluster restore we need to create a new backup location on workload2 and add new credentials to the velero cloud config secret.

Creating our Objects

Checkout the Velero docs here. This is a good place to start but in order to get this working I also needed to look at the aws backup plugin. After diving in there you see that the value under spec.config.profile refers to the aws credentials that velero will use to access the backups.

Let’s create a new backup location object on workload2. I’m going to show an example using my cluster config but you need to get the appropriate values from your own source and target clusters.

apiVersion: velero.io/v1
kind: BackupStorageLocation
metadata:
  name: source-cluster # renamed from jmo-dp, this is will represent the source for our cross cluster restore.
  namespace: velero
spec:
  config:
    bucket: vmware-tmc-1234
    profile: source-creds # renamed from jmo-dp, this will represent a new entry in the cloud-config secret.
    s3ForcePathStyle: "false"
  objectStorage:
    bucket: vmware-tmc-1234
    prefix: 01ET5VEFZTES40MAJYZ98MC6G6/
  provider: aws

Once you have your backup location object made and saved, I called mine backupLocation.yaml, you can apply it to your target cluster, workload2. So to be extra clear here, we are creating a backup location object on workload2 based on the jmo-dp backup location from workload1. You can create the new object any way you like, I did it by running `kubectl apply -f backupLocation.yaml. You can explore the new backup location in workload2 but you’ll quickly see in the velero logs that it doesn’t have permission to access backups. To get that working we need to migrate the backup credentials from workload1 over to workload2.

Migrating Credentials

In order to use workload1’s backup we need to update workload2’s cloud-credentials, the secret velero uses to backup and restore clusters. You can view your cloud-credentials secret like this kubectl get secret cloud-credentials -n velero -o json | jq -r .data.cloud | base64 -d && echo and you’ll see some output like:

# I replaced all my values, expect to see real information in your output
[jmo-dp] # this is the credential name you referenced in the backup location object
aws_access_key_id=ACCESS_KEY_ID
aws_secret_access_key=ACCESS_KEY_SECRET
aws_session_token=SUPER_LONG_TOKEN

[default]
aws_access_key_id=ACCESS_KEY_ID
aws_secret_access_key=ACCESS_KEY_SECRET
aws_session_token=SUPER_LONG_TOKEN

We’re going to add the values from workload1, our source cluster, under a new heading, source-creds. So here we’re going to do a little credential surgery and create a new entry in this file, base64 encode it, and then replace the cloud-credentials secret. It’s not exactly an easy or seemless process but I expect VMware is going to work on making cross cluster restores easier in a future version of TMC.

Updating Creds

In order to update the secret I did the following:

# Start by getting the cloud credentials for each cluster.
# Pipe the credentials from workload2 to a creds.txt file
kubectl get secret cloud-credentials -n velero -o json | jq -r .data.cloud | base64 -d > creds.txt

## Edit the file and add an entry to it under the name [source-creds] this should be the jmo-dp value from workload1
vim creds.txt

Example data:

[source-creds] # Our new entry
aws_access_key_id=ACCESS_KEY_ID_FROM_WORKLOAD1
aws_secret_access_key=ACCESS_KEY_SECRET_FROM_WORKLOAD1
aws_session_token=SUPER_LONG_TOKEN_FROM_WORKLOAD1

[jmo-dp] # this is the credential name you referenced in the backup location object
aws_access_key_id=ACCESS_KEY_ID_FROM_WORKLOAD2
aws_secret_access_key=ACCESS_KEY_SECRET_FROM_WORKLOAD2
aws_session_token=SUPER_LONG_TOKEN_FROM_WORKLOAD2

[default]
aws_access_key_id=ACCESS_KEY_ID_FROM_WORKLOAD2
aws_secret_access_key=ACCESS_KEY_SECRET_FROM_WORKLOAD2
aws_session_token=SUPER_LONG_TOKEN_FROM_WORKLOAD2

Then we’re going to re-encode it and update our secret on workload2.

# Re encode your credentials
## You can do this anyway you like as long as the data is good, adding my method in case you just want to follow along.
cat creds.txt | base64 | tr -d '\n'

## I have a clipboard utility so I really ran 
cat creds.txt | base64 | tr -d '\n' | clip

# On our workload2 cluster
kubectl edit secret -n velero cloud-credentials
## delete the string in the data.cloud section of the secret and replace it with the base64 string from above.

EDITORS NOTE Watch that your AWS token didn’t expire. In the time it took me to write and test this process I had to get a new token and re edit the cloud-credentials secret.

With that done let’s run some commands on workload2 to see if our new backup location is configured correctly and available.

# Show your backups
velero backup get

# via kubectl 
kubectl get backups -n velero

Running the Restore

Now that we’re happy with the velero config we’re heading back to the TMC console to handle the restore. Again please feel free to do the rest via the velero cli if you prefer.

Browse over to your cluster and go back to the data protection tab. You should be able to see the wp backup.

Select the backup and click restore, you’ll see a menu like this:

Restore either the entire cluster or the namespace. It doesn’t make any difference for me as I only backed up the namespace.

Now just wait for your restore to complete.

And that’s it! Go peak at your migrated wordpress blog.

Validating

Hop on to the terminal again and ensure you’re working with workload2. You can now run kubectl port-forward svc/new-blog-wordpress 8080:80 -n wordpress and dhe site will be available at localhost:8080.

Wrap Up

With that we’ve walked through the full process of doing a cross cluster restore with Velero and Tanzu Mission Control. I wouldn’t say it’s the easiest process in the world but ti gives you a chance to dive into a bit of how velero works under the hood and if you think this is something TMC should support the folks over at VMware would love to hear from you. Regardless you now have the steps required to do it on your own.

Thanks so much for reading and I’d love to hear any feedback you have, please hit me up on twitter or Linkedin.

Jason

Multi Cluster Service Mesh on TKG with Linkerd

Sun, 06 Dec 2020 00:00:00 +0000

Hey folks! Thanks for stopping by! Today I’m going to dive into using the linkerd service mesh to route traffic between 2 Tanzu Kubernetes Grid, or tkg, clusters.

We’re going to look at this because I see more and more folks looking at either spanning services between clusters or connecting an app in one cluster with a service in another. It could be for high availability reasons, in order to isolate workloads with more strict regulatory requirements, or even just to let stateful services run in their own clusters. Linkerd provides a secure, and relatively easy, way to do this and I’m going to set it up today. I hope you’re able to follow along and I’d love to hear if you have any thoughts on the process.

Thanks!

The Set Up

What we’re using:

The tkg cli
- to create, manage, and scale our k8s clusters
- made with version 1.2.0
The linkerd cli
- to do all our Linkerd work
- made with version stable-2.9.0
Our podinfo app
The step cli
- to generate our certificates
- made with version 0.15.3
Tanzu Mission Control (Optional component)
- To manage our tkg clusters
The tmc cli (Optional component)
- To connect our clusters to Tanzu Mission Control (TMC)
- You can download this from your Tanzu Mission Control portal

Creating and Managing our Clusters

I’ve previously set up a tkg management cluster in AWS and used it to provision two new clusters.

tkg create cluster -p dev -w 1 workload1
tkg create cluster -p dev -w 1 workload2

Once tkg finished with the cluster create I pulled down the relevant kubeconfig files and added them to Tanzu Mission Control. You have a couple options for adding them to TMC but I chose to use the cli.

# Get the kubeconfigs
tkg get credentials workload1 --export-file ~/configs/workload1
tkg get credentials workload1 --export-file ~/configs/workload2

Here you have the optional TMC commands, if you aren’t using TMC please skip ahead.

# login to TMC
tmc login

# Add your clusters
tmc cluster attach -n workload1 -g jmo -k ~/configs/workload1
tmc cluster attach -n workload2 -g jmo -k ~/configs/workload2
## I had already created a cluster group in TMC

Once the clusters appear in TMC you can move on to the next step.

Install Linkerd in Multi Cluster Mode

Setting up the CA

I’m following the Linkerd multi cluster docs you can find here. I start out creating a new root certificate authority with step.

# Generate the root ca

step certificate create identity.linkerd.cluster.local root.crt root.key \
  --profile root-ca --no-password --insecure --san identity.linkerd.cluster.local

# Generate an intermediary ca

step certificate create identity.linkerd.cluster.local issuer.crt issuer.key \
  --profile intermediate-ca --not-after 8760h --no-password --insecure \
  --ca root.crt --ca-key root.key --san identity.linkerd.cluster.local

We use the the ca to establish a common trust between the meshes in both clusters. You can find a lot more useful details about the certificates and how they’re used here and here.

Install the Mesh

Test your Clusters

First things first, lets ensure our clusters are good to go. Linkerd’s cli comes with a handy dandy check feature to see if your cluster is ready.

linkerd check --pre

Depending on your kubernetes api and Linkerd versions you may get some warnings about the crd api version is but that won’t impact the install.

If you are using TMC you’ll get additional warnings about Pod Security Policies (PSPs), you can safely ignore them provided you aren’t enforcing them, otherwise you’ll need to appropriately configure your PSPs which is beyond the scope of this article.

Running the Install

Provided you’re happy with the results of your pre checks it’s time to run the install. I like to set up two terminals, one for each cluster.

# Run this once per cluster

## Be sure your paths to the certs are valid, either by executing this in the same directory as the step command or by fixing the path.

linkerd install \
  --identity-trust-anchors-file root.crt \
  --identity-issuer-certificate-file issuer.crt \
  --identity-issuer-key-file issuer.key \
  | kubectl apply -f -

Once that’s wrapped wait a few minutes for the pods to boot up. You can watch the linkerd namespace or, if you’re using tmc you can take a peak at the workloads under the workloads tab or by looking at one of the worker nodes.

# Run this for each cluster
watch kubectl get pods -n linkerd

After the control plane is available you can check the status of Linkerd using the cli. Linkerd goes out of it’s way to be easy to use and debug. With that in mind they added the check option to the cli so you can get an at a glance health check. Once again be sure you’re running this for each workload cluster.

# Run this once per cluster
linkerd check

Adding Multi Cluster Support

Now that we’re happy with the per cluster install we need to extend it to handle multi cluster.

# Run this once per cluster
linkerd multicluster install | kubectl apply -f - 

You can check the status of your multicluster install a few different ways. We’re going to exercise the Linkerd cli a bit, then take a peak at our kubernetes objects. The big thing we’re looking for is to ensure our new gateway has a load balancer assigned to it.

# Run a check with linkerd
linkerd check --multicluster

# Check the new pod
kubectl get pods -n linkerd-multicluster

# Check for a loadbalancer attached to the linkerd-gateway service
kubectl get svc -n linkerd-multicluster

# Ensure the load balancer has been assigned

With our new gateways up and running we now want to “link” our clusters. You can get a more detailed explenations in the docs but the short version is that running the link commands will allow the clusters to talk and build/maintain the service mirrors.

This part can be a little tricky so I’ll include a screen grab after. What we need to do is run our Linkerd link command to generate the yaml from 1 cluster then use kubectl to apply it to the other cluster.

# Run for each cluster
## I accomplish this by running two terminals

### one with the $KUBECONFIG variable set to ~/configs/workload1
#### Then run:
linkerd multicluster link --cluster-name workload2 --kubeconfig ~/configs/workload2 | kubectl apply -f - 

### one with the $KUBECONFIG variable set to ~/configs/workload2
#### Then run:
linkerd multicluster link --cluster-name workload1 --kubeconfig ~/configs/workload1 | kubectl apply -f -

You can, hopefully, see that a little more clearly in the screen grab below. Note the red section of the prompt indicates the current kubernetes context.

Next you’ll want to validate the clusters are properly linked. Start by running linkerd multicluster gateways for each cluster. You should see something like this:

Note that each cluster sees the other’s gateway. On top of that we can rerun our Linkerd check to see more multicluster health check outputs.

linkerd check --multicluster

With that all your checks should be passing and your clusters are now ready for some workloads.

Getting our Test Service Installed

The default docs want you to use two podinfo configs mapped to the values east and west. I wanted to pair it up to my cluster names so I pulled the app config into it’s own repo. You can also use the east/west names from the docs by pulling app definitions from the Linkerd website.

When running the commands be sure you run a different version of the app manifest for each cluster.

# the podinfo repo at the path below has the config for 2 clusters, workload1 and workload2
kubectl apply -k github.com/jasonmorgan/podinfo/workload1/

With that done now is a good time to checkout our new podinfo web service and see what it looks like. Run the following command for each cluster, be sure to either use a different port or run one at a time then browse to your the page.

kubectl port-forward svc/frontend 8081:8080 -n test

Then open a browser to localhost:8081. You should see something that looks like this:

Later on when we get to Traffic Splitting you’ll be able to see the traffic shift from one cluster to the other with the port-forwarding.

Linking, or Exporting, a Service

When we decide we want to share a service between clusters we need to let Linkerd know which services to mirror. We do that with mirror.linkerd.io/exported=true label, alternatively if you’d like to modify the key you can find it on the links.multicluster.linkerd.io object in the linkerd-multicluster namespace.

# You'll only need to run this on the workloads in one cluster. I ran it on workload2
kubectl label svc -n test podinfo mirror.linkerd.io/exported=true

You can now check the other cluster for the new service. I got the following output on workload1:

kubectl get svc

NAME                TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)             AGE
frontend            ClusterIP   100.65.191.192   <none>        8080/TCP            38m
podinfo             ClusterIP   100.70.110.153   <none>        9898/TCP,9999/TCP   38m
podinfo-workload2   ClusterIP   100.68.52.124    <none>        9898/TCP,9999/TCP   7s

Technically you’ve now completed the task of sharing a service between clusters with Linkerd on Tanzu Kubernetes Grid. That being said we still don’t have any cool demos to show off so lets keep going.

You can move on to the next section where we split traffic. If you’d like some more detailed tests checkout this section of the docs or to see a walkthrough of validating TLS look here.

Splitting Traffic

This is where we start to see some of the power of a tool like Linkerd in combination with the Service Mesh Interface (SMI). SMI is aiming to provide for our mesh layer what CNI, or the Container Network Interface, provides for our pod networks. It is a standard interface that allows us to define common tasks like splitting traffic, surfacing metrics, or defining access controls. SMI is still early days but you’ll be able to see some of what it can do in this example. We’re going to leverage the Traffic Split spec in order to share requests between the new podinfo services on workload cluster 1 and 2.

Lets create a TrafficSplit object and hand it off to our kube cluster.

apiVersion: split.smi-spec.io/v1alpha1
kind: TrafficSplit
metadata:
  name: podinfo
  namespace: test
spec:
  service: podinfo
  backends:
  - service: podinfo
    weight: 50
  - service: podinfo-workload2
    weight: 50

The file above will tell Linkerd to split traffic for the podinfo service between the local podinfo on workload1, known locally as podinfo, with the podinfo service on workload2, know locally as podinfo-workload2. Save the yaml content from above into a file called split.yaml.

Before we talk anymore about it lets get it up and running and checkout the output. Start by doing that port forward operation we talked about when we first deployed the podinfo app, we need to ensure we’re running it on workload1.

kubectl port-forward svc/frontend 8081:8080 -n test

browse to localhost:8081 on your local browser and keep it open, we’re going to watch what happens when we set up traffic splitting.

Now apply that split.yaml we created earlier to your workload1 cluster.

kubectl apply -f split.yaml

At this point you should see your browser switching between the local and remote podinfo services. With that you’ve successfully split traffic between two kubernetes clusters with Linkerd! This is pretty neat as an example but think about some other ways we could apply this. We could isolate PCI workloads to a PCI cluster or run backing services in one cluster and front end apps in another.

Wrap Up

Well I hope y’all were able to follow along and I really hope you got to have a, “that’s pretty cool” moment when we split the traffic for podinfo between clusters. I certainly enjoyed it. If this is interesting I’d recommend you dig a little deeper into Linkerd and see how to expand on this example by connecting an app between clusters.

Once I got the hang of it I was able to run through the end to end example in about 30 minutes. I’m definitely looking forward to seeing one of my customers give this a shot and I’ll be eagerly waiting for multi cluster networking with Linkerd to support database connections.

Thanks so much for reading and I’d love to hear any feedback you have,

Jason

Installing code-server in Kubernetes

Mon, 27 Jul 2020 00:00:00 +0000

Hey folks! I wanted to write a quick article today on getting started using code-server on kubernetes. Code-server is an open source project from the folks at Coder that makes it easy to run and manage a cloud based vscode instance that you can connect to and operate remotely. Installing it in kubernetes has some neat benefits like allowing you to develop your app alongside any versions of other services you want. If you’re following a strong gitops flow, it’s pretty straightforward to build a dev cluster that closely mirrors one of your production environments; that and it allows you to log into the same vscode instance from any device and location you want.

Getting Started

Lately, I’ve gotten into the habbit of writing “getting started” guides and hosting them in Github. You can find my “getting started” guide for code-server here. Feel free to hop over there if you want to get up and running, the rest of this post will be all about what choices I made, and why I made them, while building the manifest and container.

The Dockerfile

FROM codercom/code-server
COPY .vscode /home/coder/.local/share/code-server
RUN curl -Lo shellcheck-v0.7.1.linux.aarch64.tar.xz https://github.com/koalaman/shellcheck/releases/download/v0.7.1/shellcheck-v0.7.1.linux.aarch64.tar.xz \
  && tar -xvf shellcheck-v0.7.1.linux.aarch64.tar.xz \
  && chmod +x shellcheck-v0.7.1/shellcheck && sudo mv shellcheck-v0.7.1/shellcheck /usr/local/bin/ \
  && rm -rf shellcheck* && sudo chown -R coder:coder /home/coder/.local/share/code-server \
  && curl -LO https://storage.googleapis.com/kubernetes-release/release/`curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt`/bin/linux/amd64/kubectl \
  && chmod +x kubectl && sudo mv kubectl /usr/local/bin/

If you aren’t really familiar with Dockerfile command words I dig through the FROM/RUN/COPY stuff a bit more in the git repo.

For my instance I decided I wanted my code-server to mirror my local vscode environment, and the best way to do that was to just copy over my extensions directly. I won’t dig into it now, but it turns out in spite of vscode being open source some components, like the extension store, have special access rules around them. Long story short, if you’re using a variant of vscode, like code-server, you don’t necessarily have access to install all the vscode extensions you might be using. You can get around any concern about extension stores by just copying over your current .vscode directory into the container. Just be aware the default extensions directory in code-server is ~/.local/share/code-server and once you move the extensions over you need to take ownership of the files. Other than that some extensions, like shellcheck and kubernetes, require you to have the binary available in your path so you’ll need to modify your dockerfile to pull down anything you need.

If you want to see a bit more detail about the dockerfile check out my comments in the getting started guide.

The Manifest

Before I go anywhere I want to point out that I got my initial template from the folks over at DigitalOcean. You can see their post about deploying code-server on kubernetes. I thought the whole thing was really well done and it’s probably worth your time to read through, especially if you want a second opinion.

A lot of this is going to be pretty standard, we have a deployment, a service, and a persistent volume claim. I want to make sure I run my code server, using the image I customized and pushed to dockerhub, I want to be able to access it reliably via a service, and I want to persist it’s data disk so I don’t lose my work if something happens to the pod.

When we get to persisting the data we have to watch out for a couple things. First and foremost, we have file permissions. I didn’t have any issues on docker desktop but mounting volumes on AWS set the user permissions on the file system to root, and code-server by default runs as coder, so I modified my prep script to chown the home directory. After that I built out a little script to set up a clean working directory and to try and clone whatever repo I told the init container to pull. You can find all this in the manifest file here.

The Ingress and TLS

This ended up being trickier than I anticipated as code-server uses websockets to get that fancy editor experience and project contour didn’t, maybe doesn’t, have great documentation around how to get that working. I swapped over to nginx’s ingress which supports web sockets by default and was up and running. If you aren’t using cert-manager I’d higly recommend you check it out. The ACME HTTP challenge works great and cert-manager and nginx, or contour, will auto generate certificates for you on demand. This blog has some articles about setting up cert-manager and contour has a great tutorial here. If you want to use nginx instead, like I did, you can checkout the how to on cert-manager’s docs page. The helm stuff is a bit dated but installing nginx’s ingress is pretty straightforward at this point.

I ended up using the following ingress:

---
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: code-server
  namespace: code-server
  annotations:
    kubernetes.io/ingress.class: nginx
    cert-manager.io/cluster-issuer: letsencrypt-prod
    nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
    kubernetes.io/tls-acme: "true"
spec:
  tls:
  - secretName: code-server
    hosts:
    - code-server.my-domain.com
  rules:
  - host: code-server.my-domain.com
    http:
      paths:
      - backend:
          serviceName: code-server
          servicePort: 8080

Be sure to add some annotations for forcing ssl so you don’t accidentally hit the unencrypted endpoint.

The Wrap Up

That’s all I have folks! Hope you enjoyed reading it and if there’s anything you’d like to read about in the future hit me up on the kubernetes slack, @jmo, or on twitter @rjasonmorgan.

Initial Experience with the Helm 3 Alpha

Fri, 31 May 2019 00:00:00 +0000

Hey folks, I’ve been working with Helm3 since the alpha came out (some amount of time ago that I’m unwilling to lookup). It’s been a pretty good experience so far but I wanted to sketch out my experience to date. I may come back and update this later as I run into new issues.

Goodbye Tiller

I’m super psyched that Tiller is gone. I, think I, get what tiller was intended to do but it represented both an operational and security failure plane. That ok if it provides significant value but I didn’t see that value. Even though most of my examples in this blog involve using helm I rarely used tiller myself. I would use helm templating to generate yaml manifests then just apply those directly. It was a decent system but it was definitely kind of a pain. Helm3 drops Tiller and once it matures it will probably be my go to.

Why Helm

Just wanted to write this down somewhere. I know there are a few different, and seemingly really good, alternatives to Helm. I don’t really use them. I don’t have anything against any particular tool or approach, I just believe that for package managers the top feature I care about is, is it a standard. Helm is, IMO, the closest thing to a standard for k8s “package” management. So I’ll use helm, write charts, and use tools that support helm packages. I’d rather see helm get better and evolve than look for the “best” tool. If the standard changes, or solidifies somewhere else, I’ll migrate to that.

It’s an Alpha

It’s an Alpha. Some stuff is broken, or doesn’t work, or kind of doesn’t work, or sometimes doesn’t work. So far I’ve noticed that it really doesn’t like dealing with resources that get namespace labels, particularly if they aren’t for the namespace you’re currently pinned to. Also helm upgrade seems to work great as long as you don’t actually want objects to update or change… But that being said it’s definitely worth trying out. A lot of what you want to do just works and you can get around the stuff that doesn’t work by adding the --dry-run switch to your install command or using the templating function to just create yaml files you can apply normally.

Things I’ve Noticed

helm upgrade doesn’t seem to do much
It really doesn’t like when you are creating objects in multiple namespaces
It doesn’t seem to like or necessarily respect it when you use the --namespace flag
The syntax has changed in what I think is a real positive way
The naming no longer appends some chart based name afterwords
- I love this cause for some reason I really care about the names of things

Check it Out

I’d suggest you check it out and start using it. For those of use that can order k8s clusters on demand, cause we use one of the *KS’s, it’s really easy to grab a cluster, try out some of our existing workflows with new tools, then blow it away. For you DIY k8s-ers out there Docker for Whatever OS or minikube make it pretty easy to test things out.

Issuing Certificates with Cert-Manager and Let's Encrypt

Fri, 26 Apr 2019 00:00:00 +0000

I go out of my way to secure all my sites with a valid https certificate. I’m also fairly cheap, this left me with a dilemma and AWS’s certificate manager used to be my only option. It was handy but I was effectively stuck on AWS and had to use an Elastic Load Balancer, ELB, to terminate my TLS connections. Before we go too deep here if any of the terms I’m using are a little ambiguous I’d recommend checking out this article from symantec on the meaning of SSL, TLS, and https. The first section covers the definitions well enough that hopefully you feel comfortable with the difference between the terms.

Concepts

Let’s Encrypt

Let’s Encrypt is a free service that allows you to programmatically generate TLS certificates. It’s not always super well documented or easy to use but once you get it in place you can generate, use, and renew certificates on a regular basis for free. On top of that Let’s Encrypt pioneered a new IETF standard for programmatically doing Domain Validation. I’ll get into how and why that works right now.

Cert-Manager

cert-manager is a kubernetes service that will interact with LetsEncrypt, or another CA, for you to programmatically request, generate, and renew certificates. We’re going to limit our scope today to working with Let’sEncrypt. Go here to ignore some of this guide and just install the latest cert-manager. They’re moving to a non default helm repo and have their own getting started instructions. I’ll also cover it in more detail in the Getting to the Code section.

ACME and ACMEv2

Let’s Encrypt originally came out with a protocol they called ACME, which was a mechanism for programmatically doing domain validation. This year they got ACME accepted as an IETF standard and they called it ACMEv2. From a practical standpoint ACME/ACMEv2 supports two mechanisms for domain validation.

For Let’sEncrypt we want to be careful to manage our interactions with the API. Let’sEncrypt provides a free service to anyone that wants certificates. Awesome right? The downside is they throttle requests against their production APIs. Let’sEncrypt has been changing their rate limits as they mature their infrastructure, you can keep up to date with their rate limits here. The impact of that is that you don’t want to test that your issuer and request are valid against the production API. In order to validate your issuer and request before hitting the production API Let’sEncrypt provides a staging API that allows you to happily mess up as much as you like. Clarification the staging API is also rate limited but it’s much more forgiving than the prod API, please don’t spam the staging API.

DNS

DNS Validation, covered in section 8.4 of the IETF RFC, is when you prove you’re authoritative for the domain by creating a custom DNS entry.

HTTP

HTTP Validation, covered in section 8.3 of the IETF RFC, involves putting a specific file on a web server at a specific URL.

Issuers

Issuers refer to the service that will act to issue a given certificate. Ultimately your CA issues the actual certificate. The issuer in the context of cert-manager refers to the broker between your kubernetes cluster and the CA.

In order to get your issuer up and running correctly you need to pick a domain validation method. Let’sEncrypt offers free Domain Validation (DV) certs which basically just check that the entity requesting the certificate has effective control of the domain in question. If that doesn’t sound like a whole ton of validation you’re right, it’s not. Troy Hunt among others actually has a lot of good talk tracks on what exactly SSL/TLS do, and more importantly don’t do, to secure a given site. The long and the short of it being a TLS connection makes it really hard to snoop on the traffic between a browser and a site. That’s it.

Back to DV: Let’sEncrypt has been pioneering programatic ways to do Domain Validation and they recently got ACMEv2 adopted as an IETF standard. With cert-manager you have two options for DV, HTTP and DNS.

Certificates

Certificates refer to the actual certificate you intend to generate. Basically what URL/Host are you trying to secure and where do you want to store your secret? I honestly prefer to think of these objects as requests as the actual certificate is stored in kubernetes as a kubernetes secret but that doesn’t really matter.

Code

This guide is working with cert-manager version 0.7.0, for the latest docs checkout cert-manager’s site.

Installing cert-manager

The helm chart has a good installation guide, which I wont spend a ton of time talking about here.

We need to first apply the cert-manager CRDs.

kubectl apply -f https://raw.githubusercontent.com/jetstack/cert-manager/release-0.7/deploy/manifests/00-crds.yaml

Next create the cert-manager namespace, kubectl create namespace cert-manager, although personally I create and maintain a yaml version of the namespace that I can apply in bulk as necessary. Once the namespace is up you need to apply a label to disable cert-manager validation. kubectl label namespace cert-manager certmanager.k8s.io/disable-validation="true"

For the namespace definition I use:

---
apiVersion: v1
kind: Namespace
metadata:
  name: cert-manager
  labels:
    certmanager.k8s.io/disable-validation: "true"

If you’re using helm you can complete the install with this:

## Add the Jetstack Helm repository
helm repo add jetstack https://charts.jetstack.io

# Update your local Helm chart repository cache
helm repo update

## Install the cert-manager helm chart
helm install \
  --name cert-manager \
  --namespace cert-manager \
  --version v0.7.0 \
  jetstack/cert-manager

Configuring a Cluster Issuer

I start from the assumption that I won’t run multi tenant clusters so I always build out cluster issuers as opposed to creating individual issuers by namespace.

In order to test out my issuer I create a ClusterIssuer that will connect to the staging API. I modify the spec.acme.server to use the staging API, https://acme-staging-v02.api.letsencrypt.org/directory. The full staging issuer is included below.

apiVersion: certmanager.k8s.io/v1alpha1
kind: ClusterIssuer
metadata:
  name: letsencrypt-stage # Arbitrary string, you'll reference this when you create a certificate
spec:
  acme:
    server: https://acme-staging-v02.api.letsencrypt.org/directory # LetsEncrypt API URL
    email: jason@59s.io # Your email address

    # Name of a secret used to store the ACME account private key
    privateKeySecretRef:
      name: letsencrypt-stage # Arbitrary string

    # ACME DNS-01 provider configurations
    dns01:

      # Here we define a list of DNS-01 providers that can solve DNS challenges
      providers:
        - name: dns # arbitrary string, you'll reference this later in your request.
          route53:
            region: us-east-1 # Region of the zone
            hostedZoneID: IMAVALIDHOSTEDZONEID
            # optional if ambient credentials are available; see ambient credentials documentation
            accessKeyID: IMAVALIDAWSACCESSKEY
            secretAccessKeySecretRef: # Because we're using a cluster issuer this secret, with the properties you see below, need to be placed in the cert-manager namespace.
              name: secret-name
              key: property-name

Using Staging

Use staging. Until you’re super comfortable that your certificate request and issuer work well stick with the staging API. The staging API will generate an invalid certificate and store it in your kubernetes cluster. You don’t actually want to use the staging certificate for anything other than to validate that your issuer and certificate request are valid.

Configuring a certificate request

Once your issuer is up and running you’ll request a certificate. Again, we start with the staging API then swap over to the prod API later.

apiVersion: certmanager.k8s.io/v1alpha1
kind: Certificate
metadata:
  name: 59s-io-stage # Arbitrary string
  namespace: n1
spec:
  secretName: 59s-io-tls-stage # Arbitrary string, name of the secret you want to create
  issuerRef:
    name: letsencrypt-stage
    kind: ClusterIssuer
  commonName: '*.59s.io'
  dnsNames:
  - 59s.io
  acme:
    config:
    - dns01:
        provider: dns # use the provider name from your issuer
      domains:
      - '*.59s.io' # Super sweet wildcard certificates let me use a single ingress for all my sites. LetsEncrypt started issuing wildcard certs back in 2018.
      - 59s.io

I like to watch the logs at this point to see what cert-manager is doing and ensure it’s able to issue my certificate. Once it’s issued you can begin migrating over to the production API.

Moving to the Production API

We’re going to effectively copy the issuer and certificate request from above. Swap out the names so that prod replaces staging. Also we need to update the URL.

Cluster Issuer

apiVersion: certmanager.k8s.io/v1alpha1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod # Arbitrary string, you'll reference this when you create a certificate
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory # LetsEncrypt API URL
    email: jason@59s.io # Your email address

    # Name of a secret used to store the ACME account private key
    privateKeySecretRef:
      name: letsencrypt-prod # Arbitrary string

    # ACME DNS-01 provider configurations
    dns01:

      # Here we define a list of DNS-01 providers that can solve DNS challenges
      providers:
        - name: dns # arbitrary string, you'll reference this later in your request.
          route53:
            region: us-east-1 # Region of the zone
            hostedZoneID: IMAVALIDHOSTEDZONEID
            # optional if ambient credentials are available; see ambient credentials documentation
            accessKeyID: IMAVALIDAWSACCESSKEY
            secretAccessKeySecretRef: # Because we're using a cluster issuer this secret, with the properties you see below, need to be placed in the cert-manager namespace.
              name: secret-name
              key: property-name

Certificate Request

apiVersion: certmanager.k8s.io/v1alpha1
kind: Certificate
metadata:
  name: 59s-io-prod # Arbitrary string
  namespace: n1
spec:
  secretName: 59s-io-tls-prod # Arbitrary string, name of the secret you want to create
  issuerRef:
    name: letsencrypt-prod
    kind: ClusterIssuer
  commonName: '*.59s.io'
  dnsNames:
  - 59s.io
  acme:
    config:
    - dns01:
        provider: dns # use the provider name from your issuer
      domains:
      - '*.59s.io' # Super sweet wildcard certificates let me use a single ingress for all my sites. LetsEncrypt started issuing wildcard certs back in 2018.
      - 59s.io

Conclusion

Use certificates to secure your sites. They’re free, the technical burden of requesting and maintaining them is relatively low, and the process can be fully automated using tools like cert-manager. Also Kubernetes has mechanisms to simplify the process of issuing, maintaining, and requesting certificates.

I’ll do a follow on post on using a wildcard certificate with an NGINX ingress and a wildcard DNS entry and wildcard certificate to automatically secure any sites you want to build on a given cluster.

Concourse in Kubernetes

Sun, 17 Mar 2019 00:00:00 +0000

Concourse is a handy tool for running build jobs or any other arbitrary code you want, otherwise known as a CI server. It has it’s own DSL, domain specific language, that you have to learn but it wasn’t too much of a burden for me to pick up. If you’re looking for some help getting started with concourse I recommend this delightful tutorial from the folks over at Starke & Wayne.

Deploying Concourse

You have a lot of options for deploying your own concourse instance. A few examples below.

a Bosh release
Pivotal provides a somewhat curated release over at network.pivotal.io
Engineer Better has Concourse-Up
- This will deploy
  - a Bosh Director, think kubernetes for VMs
  - Credhub, a secret store similar to Vault
    - it will even connect concourse to credhub so it can securely store and retrieve credentials
  - A prometheus instance with Grafana
- It’s pretty sweet overall and if you’re running in GCP or AWS it’ll work well for you
BUCC from Starke and Wayne
- Supposed to be similar to Concourse-up but more IaaS agnostic
- I personally haven’t gotten it working so take that for what it’s worth

Concourse in K8s

I think bosh is cool but I’m often impatient. Kubernetes lets me do a lot of what I do with bosh but it does it with containers which I can start and modify a lot faster than VMs. It also does a bunch of other really cool things that I like which we can get into later.

Helm Chart

I’m not going to get into what helm is and the rest of this article will assume you have some familiarity there. If you want me to write an article about that tweet me or something. Someone out there, I’m not sure who, is maintaining the stable/concourse chart. It works really well and they’ve been doing a pretty good job keeping it up to date.

Getting into the code

Kubernetes is effectively tons of YAML so we’re going to dive into that now.

Pre reqs

a Kubernetes Cluster
- I’m going to assume RBAC is enabled as that seems pretty standard
Helm
kubectl
Some kind of editor
A terminal
A clean working directory

Getting Started with Helm

Open your terminal and get your connection to your kube cluster squared away. You can validate that it’s up and running with this command.

kubectl --version

Now we’re going to set up our tiller account and initialize helm. If you have any questions while you’re doing this please refer to helm’s guide to installing tiller with RBAC enabled.

Tiller Account and Cluster Role Binding

I have a file laying around with my tiller service account and cluster role binding. Save this file as tiller.yaml in my current working directory.

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: tiller
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: tiller
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
  - kind: ServiceAccount
    name: tiller
    namespace: kube-system

The big thing here is making sure that the tiller account exists and has appropriate permissions to start launching new applications.

After saving the tiller file we’re going to apply it to our kube cluster.

kubectl apply -f tiller.yaml

you should see output that looks super similar to this

serviceaccount/tiller created
clusterrolebinding.rbac.authorization.k8s.io/tiller created

Helm Init

With our service account in place we’re clear to start tiller.

helm init --service-account tiller

You can test the install by running

helm version

You should get feedback about both the helm and tiller versions.

If you want to be done with the article now and just run the vanilla version of concourse that comes with the chart run helm install stable/concourse and follow the instructions in the output to access your instance. The rest of the article is just diving into the values file.

Getting the chart

To get started you can find the chart by running helm search concourse or just going to the stable/concourse folder in the stable helm chart repo.

I like to download a copy locally to get started. Also as a side note I’m going to avoid the most recent version of the chart as I’m not yet ready to dig into the concourse 5.0.0 release. You can view chart versions by running helm search stable/concourse -l

To download the chart locally

helm fetch stable/concourse --version 3.8.0 --untar

I like to download the charts locally and render them myself using helm’s templating functionality but that’s a topic for another day.

Keep in mind you have 2 version numbers in play here, the version of the app the chart is managing and the version of the chart. 3.8.0 refers to the chart version and 4.2.2 is the version of concourse it’s running. We’re actually going to fix that a little when we deploy the app as 4.2.2 still has the bad old version of runc so we’re going to fix that by going to 4.2.3.

I always maintain my own version of a chart’s values file even when I accept the defaults.

cp concourse/values.yaml values.yaml

Important Modifications

Just to get started the helm chart itself bundles a pretty good explanation of values right in the values file. The repo page also does a pretty good job getting you up and running. Make the modifications recommended below to the values.yaml file you have saved in your working directory.

Secrets

So the Chart ships with some secrets built into it to make it easier to use. Obviously if you plan on actually using concourse you ought to swap those out. Lucky for me the documentation has that subject pretty well covered so I’ll let you follow those steps yourself.

Persistence

Read this section carefully. They aren’t joking about the workers filling up your local disks, I managed to bring down 3 workers when I tried to save a little money by skipping the PVCs.

Deploying our Concourse Instance

Assuming you’ve gone ahead and followed the steps above we’re going to make one more modification to our values file then deploy. We need to swap out the imageTag value from 4.2.2 to 4.2.3. And again just for clarity we need to use 4.2.3 as 4.2.2 still bundles a bad version of runc that we don’t need to be putting out there.

With that done we can go ahead and set up our concourse instance.

helm install stable/concourse --version 3.8.0 --values values.yaml

Helm’s output will tell you how to access your instance and that will be enough to get up and running.

Conclusion

The Concourse helm chart is pretty functional and the maintainers seem to be doing a good job with it. I’d recommend it for getting started running concourse in k8s. My next post will go over using cert-manager to automatically generate certificates that you can use for apps like concourse.

Thanks for reading!