Jenkins is a popular and powerful automation server. You can run Jenkins in your Kubernetes cluster as part of a deployment infrastructure to build and push Docker images, and update the running applications in your cluster.
These exercises run a full local build setup in Kubernetes:
You don't typically run all this yourself, but it's very useful to see how it all fits together and have it as a backup option if your other services go down.
Using Declarative Jenkins Pipelines - Pluralsight course to learn more about Jenkins
The build components will all run as Deployments in a custom namespace:
infrastructure/gogs.yaml - defines the Git server, and a NodePort to access it
infrastructure/buildkitd.yaml - runs a BuildKit server, with a ClusterIP Service for internal access
infrastructure/jenkins.yaml - runs Jenkins with a NodePort to access on port 30008, plus setup scripts in a ConfigMap and RBAC rules for Jenkins to use the Kubernetes API server
Start by deploying all the infrastructure components:
kubectl apply -f labs/jenkins/specs/infrastructure
📋 Check on the Deployments. Not all the Pods are running - what's the problem?
Check the Deployments:
kubectl get deploy -n infra
The Gogs and BuildKit Deployments get up to scale, but the Jenkins Deployment stays at 0/1 Ready.
Check the Pod:
kubectl get po -n infra -l app=jenkins
It's stuck at the ContainerCreating status. Describe the Pod and you'll see it needs a Secret: secret "registry-creds" not found.
Jenkins will run a pipeline to build and push Docker images using BuildKit. For that it needs authentication to Docker Hub - or whichever image registry you use.
Just like the BuildKit lab, this is sensitive stuff so we'll use variables to hide your password.
Secrets are namespaced, and you can't mount a Secret from one namespace to a Pod in another namespace.
On Windows, use PowerShell to store your credentials:
$REGISTRY_SERVER='https://index.docker.io/v1/'
$REGISTRY_USER=Read-Host -Prompt 'Username'
$password = Read-Host -Prompt 'Password'-AsSecureString
$REGISTRY_PASSWORD = [System.Net.NetworkCredential]::new("", $password).Password
OR on MacOS or Linux:
REGISTRY_SERVER='https://index.docker.io/v1/'
read REGISTRY_USER
read -s REGISTRY_PASSWORD
📋 Now create a registry Secret in Kubernetes called registry-creds in the infra namespace, using the variables you've stored.
kubectl create secret docker-registry -n infra registry-creds --docker-server=$REGISTRY_SERVER --docker-username=$REGISTRY_USER --docker-password=$REGISTRY_PASSWORD
Your Jenkins Pod may be in a backoff state by now, so replace it with a new rollout:
kubectl rollout restart -n infra deploy/jenkins
The new Pod still won't start :)
📋 What's wrong with the Pod now?
Check the Pods and you'll see they're in CreateContainerConfigError status:
kubectl get po -n infra -l app=jenkins
Describe the latest Pod and you'll see there's one more dependency needed: Error: configmap "build-config" not found
We need a ConfigMap to store details of the image name to use for the build - the name includes the registry domain and your user (or group) name.
Set your registry domain and repository name in a ConfigMap, be sure to use your own registry ID:
kubectl create configmap -n infra build-config --from-literal=RELEASE_VERSION=21.09 --from-literal=REGISTRY_DOMAIN=docker.io --from-literal=REGISTRY_REPOSITORY=<your-registry-id>
Now you'll see the Jenkins Deployment come up to scale:
kubectl get deploy -n infra
The local build infrastructure is all running, and Jenkins is configured with a project which fetches a Git repo from the Gogs server.
To run a build we need to push our local code to Gogs. You can do this with the Git CLI - these commands adds the local server as a new remote and push a copy of the repo there:
# add the local Git server:
git remote add labs-jenkins http://localhost:30030/kiamol/kiamol.git
# and push:
git push labs-jenkins main
You'll need to authenticate with the server, use
kiamolas the username and password.
Now all the code from this repo is in your local Git server. You can browse here and see the build pipeline: http://localhost:30030/kiamol/kiamol/src/main/labs/jenkins/project/Jenkinsfile.
This is a Jenkins pipeline definition. If you're not familiar with the syntax, you can see we're mostly running shell scripts which use the BuildKit CLI to build and push images.
Browse to Jenkins at http://localhost:30008. Click log in at the top right of the screen, and use kiamol as the username and password.
Now browse to project at http://localhost:30008/job/kiamol/ (the project is created in the setup scripts in jenkins.yaml).
Click Enable and then click Build Now.
The build should complete sucessfully:
buildctl to build an image with the BuildKit serverIf it all goes well, you should see the push information in the logs, and you can browse to Docker Hub and see your image - e.g. https://hub.docker.com/r/courselabs/whoami-lab/tags.
If not check the Jenkins project logs - any problems are likely to be authentication using the Secret, or the image name compiled from the ConfigMap.
That's the CI part done - if the project had unit tests we could run them as another stage in the Dockerfile and be confident the built image was functionally correct.
Now we want to add Continuous Deployment. There's a Helm chart for the project which we can run manually to test the release:
Install a local release to verify the chart - using a public image:
helm upgrade --install whoami-dev --set serverImage=docker.io/courselabs/whoami-lab:21.09-1 labs/jenkins/project/helm/whoami
When the app is running, test it at http://localhost:30028
We'll add the CD stage to the build, deploying to a new namespace.
📋 Create a namespace called integration-test and label it with kubernetes.courselabs.co=jenkins so we can easily clean up at the end.
kubectl create ns integration-test
kubectl label ns integration-test kubernetes.courselabs.co=jenkins
We want to use the images Jenkins pushes to run the app in the test namespace - so that namespace will also need a registry authenticaion Secret to pull images.
Create a Secret using the same registry credentials:
kubectl create secret docker-registry -n integration-test registry-creds --docker-server=$REGISTRY_SERVER --docker-username=$REGISTRY_USER --docker-password=$REGISTRY_PASSWORD
Now open the Jenkinsfile, scroll to the end and uncomment the stage called Deploy to test namespace: delete the /* line at the start of the stage and the */ line at the end.
Push your changes to Gogs, and the pipeline will be updated in Jenkins:
# remove /* and */ comment lines from Jenkinsfile
git add labs/jenkins/project/Jenkinsfile
git commit -m 'Enable CD'
git push labs-jenkins
Browse back to your build at http://localhost:30008/job/kiamol/ - click Build Now a few times to push images with different version numbers.
Check the CI/CD deployment works and is running the latest version:
helm ls -n integration-test
curl localhost:30029
kubectl get rs -n integration-test -o wide
You should see the updated build versions in the image tags.
This lab gets you to make use of the CI/CD pipeline to update the whoami app. The Dockerfile for the app needs updating and optimizing:
FROM a minimal imageMake your changes to the Dockerfile and check the image builds locally. Then push your changes to Gogs and confirm the app updates to your new image version and works correctly.
Pull your latest image and the one before that - is there a size difference?
Remove the Helm releases:
helm uninstall whoami-dev
helm uninstall whoami-int -n integration-test
And all the other components:
kubectl delete ns -l kubernetes.courselabs.co=jenkins
Then remove your Git remote:
git remote rm labs-jenkins
👩🏫 For the instructor - remember to reset the Dockerfile and Jenkinsfile for the next class :)
cp labs/jenkins/project/src/Dockerfile.original labs/jenkins/project/src/Dockerfile
cp labs/jenkins/project/Jenkinsfile.original labs/jenkins/project/Jenkinsfile
git add labs/jenkins/project/src/Dockerfile
git add labs/jenkins/project/Jenkinsfile
git commit -m 'Reset Jenkins lab'
git push origin