GitOps makes configuration management seamless by creating a single source of truth for configuration changes to be transparent, validated, and low-risk. This article and this GitHub repository will show you how Traefik Proxy and Flux can work together to help you implement GitOps principles. But before we jump in, what are these tools, and how do they help?
Flux is a tool created by Weaveworks, the inventor of GitOps. The operator makes GitOps happen in your cluster, ensuring cluster configurations match the ones in Git. Flux exposes an interface to this tool that helps you build and apply YAML to your clusters so you can automate your deployments. It also uses Helm.
Traefik Proxy is an edge router that exposes your services. In the Kubernetes ecosystem, this is called an ingress controller. Traefik Proxy is continuously updated to work with the latest version of Kubernetes.
Traefik Proxy provides CRDs such as an ingress route, TCP/UDP ingress routes, and TraefikService (an abstraction layer running on top of Kubernetes Services and middleware). These objects help you expose Services to create a configuration. The middlewares are applied to routers and can tweak requests before hitting Services.
Traefik Proxy is a leading modern reverse proxy and load balancer that makes deploying microservices easier. Traefik Proxy also has a certificate resolver feature that lets you automatically obtain certificates from Let's Encrypt. Traefik Proxy's middlewares constitute small applications that you can use to tweak requests. They will help you, for example, add headers or circuit breaker rate limits. Traefik Proxy also supports canary deployments, and we recently added middlewares for TCP ingress routes.
We will be promoting code from staging to production using Kustomize, built into Flux. Before anything else, there are a few prerequisites you will need:
GITHUB_TOKEN, GITHUB_USER, GITHUB_REPOThis article aims to show you how to maintain Kubernetes environments with GitOps principles. We will highlight an everyday use case with two clusters: one staging environment and one production environment.
On both clusters, we will deploy Traefik Proxy and the sample applications that will be exposed externally thanks to Traefik and IngressRoute resources.
The configuration that we are preparing is based on the following structure:
All the changes that are created for a specific environment will be created as a patch, so we will only update the attributes that we are going to change. The remaining configuration is inherited from the base configuration. That entire process is managed using Kustomize, built into Flux.
The entire configuration will be prepared locally, without executing any imperative commands on those two clusters that have been provisioned as a prerequisite.
Once we create the configuration locally, we will push the entire code to the Git repository.
The Git repository is our source of truth. Any change we will apply to a cluster will go through our Git repository.
This is a core idea of GitOps. We configure the desired state of our infrastructure using commonly-known Kubernetes manifests. We then push the desired state to a Git repository. Flux then pulls the configuration to the cluster, so the configuration can be applied.
It is very beneficial to store our cluster configuration in a Git repository. That way, we can track all the changes and test pull requests before merging them into the main branch. We can ensure that the code committed to the Git repository runs on our clusters.
Flux will run our applications and infrastructures components: Traefik Proxy and our sample application on our cluster according to the code we prepared locally.
Let's get started!
Create a Git repository to store all configuration files. In our example, we will use a GitHub repo. Use the following command to create a GitHub repo:
gh repo create flux-traefik-demo --public --description "Flux and Traefik - demo" --clone
This command assumes you are in an empty directory with no Git repository. It will have the same effect as running git init. gh command refers to the GitHub CLI). If your home directory is a Git repository, you might want to run git init in an empty directory before running the above command. This will create the remote repository and set up the Git remote.
The following command will create the top directory structures needed to keep manifests that describe the entire infrastructure:
The apps contain custom manifests per cluster as well as Helm releases.
mkdir -pv ./apps/{base,staging,production}/traefik ./clusters/{production,staging} ./infrastructure/{sources,crds}
├── apps
│ ├── base
│ ├── production
│ └── staging
├── infrastructure
│ ├── crds
│ └── sources
└── clusters
├── production
└── staging
A typical way of installing Traefik is to create a service account, RBAC, deployment, and Service (of load balancer type) to expose Traefik.
First, create your RBAC resources.
We can also install Traefik using the official Helm Chart. Flux supports Helm releases. However, in this case we will install all the required resources manually.
cat > ./apps/base/traefik/rbac.yaml <<EOF
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: traefik-ingress-controller
rules:
- apiGroups:
- ""
resources:
- services
- endpoints
- secrets
verbs:
- get
- list
- watch
- apiGroups:
- extensions
- networking.k8s.io
resources:
- ingresses
- ingressclasses
verbs:
- get
- list
- watch
- apiGroups:
- extensions
resources:
- ingresses/status
verbs:
- update
- apiGroups:
- traefik.containo.us
resources:
- middlewares
- middlewaretcps
- ingressroutes
- traefikservices
- ingressroutetcps
- ingressrouteudps
- tlsoptions
- tlsstores
- serverstransports
verbs:
- get
- list
- watch
---
kind: ServiceAccount
apiVersion: v1
metadata:
name: traefik-ingress-controller
namespace: traefik
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: traefik-ingress-controller
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: traefik-ingress-controller
subjects:
- kind: ServiceAccount
name: traefik-ingress-controller
namespace: traefik
EOF
Then create your Traefik Proxy deployment resource.
cat > ./apps/base/traefik/traefik.yaml << EOF
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: traefik
labels:
app.kubernetes.io/instance: traefik
app.kubernetes.io/name: traefik
spec:
replicas: 1
selector:
matchLabels:
app.kubernetes.io/name: traefik
app.kubernetes.io/instance: traefik
template:
metadata:
labels:
app.kubernetes.io/name: traefik
app.kubernetes.io/instance: traefik
spec:
serviceAccountName: traefik-ingress-controller
terminationGracePeriodSeconds: 60
containers:
- name: traefik
image: traefik:2.5.4
args:
- "--entryPoints.web.address=:8000/tcp"
- "--entryPoints.websecure.address=:8443/tcp"
- "--entryPoints.traefik.address=:9000/tcp"
- "--api=true"
- "--api.dashboard=true"
- "--ping=true"
- "--providers.kubernetescrd"
- "--providers.kubernetescrd.allowCrossNamespace=true"
readinessProbe:
httpGet:
path: /ping
port: 9000
failureThreshold: 1
initialDelaySeconds: 5
periodSeconds: 5
successThreshold: 1
timeoutSeconds: 2
livenessProbe:
httpGet:
path: /ping
port: 9000
failureThreshold: 3
initialDelaySeconds: 5
periodSeconds: 5
successThreshold: 1
timeoutSeconds: 2
resources:
limits:
cpu: 1000m
memory: 1000Mi
requests:
cpu: 100m
memory: 50Mi
ports:
- name: web
containerPort: 8000
protocol: TCP
- name: websecure
containerPort: 8443
protocol: TCP
- name: traefik
containerPort: 9000
protocol: TCP
volumeMounts:
- mountPath: /data
name: storage-volume
volumes:
- name: storage-volume
emptyDir: {}
EOF
Next, create a Kubernetes load balancer type service that exposes Traefik Proxy.
cat > ./apps/base/traefik/svc.yaml << EOF
---
apiVersion: v1
kind: Service
metadata:
name: traefik
labels:
app.kubernetes.io/instance: traefik
app.kubernetes.io/name: traefik
spec:
selector:
app.kubernetes.io/instance: traefik
app.kubernetes.io/name: traefik
type: LoadBalancer
externalTrafficPolicy: Local
ports:
- port: 80
name: web
targetPort: web
protocol: TCP
- port: 443
name: websecure
targetPort: websecure
protocol: TCP
EOF
Create a Kustomization file that lists all the resources that have been created.
cat > ./apps/base/traefik/kustomization.yaml << EOF
---
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- rbac.yaml
- traefik.yaml
- svc.yaml
EOF
Create a namespace for your Traefik resources.
cat > ./apps/production/traefik/namespace.yaml << EOF
---
apiVersion: v1
kind: Namespace
metadata:
name: traefik-production
EOF
Create a patch for a Traefik Proxy deployment to happen in your cluster. You first created a base configuration, where you have a configuration that will be adapted to meet the requirements for a specific environment. Use a patch to update your base configuration. The patch will contain fewer lines than the base. It will only apply the lines that have been modified. This is what Flux does in the background after it has been applied to the cluster.
As previously mentioned, the patch contains only those attributes being changed explicitly for that environment. The patch is a feature coming from Kustomize. Once Flux deploys the code, the patch will inherit the base configuration and merge to obtain the final Kubernetes manifests. In the Traefik configuration, we are using one of the features that help issue TLS certificates from LetsEncrypt. It will connect with Let's Encrypt and obtain a certificate. Make sure you have updated your email address accordingly. There is a lot more to configure with Let's Encrypt, so I encourage you to visit the official documentation to learn more.
cat > ./apps/production/traefik/traefik-patch.yaml << EOF
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: traefik
spec:
template:
spec:
containers:
- name: traefik
args:
- "--entryPoints.web.address=:8000/tcp"
- "--entryPoints.websecure.address=:8443/tcp"
- "--entryPoints.traefik.address=:9000/tcp"
- "--api=true"
- "--api.dashboard=true"
- "--ping=true"
- "--providers.kubernetescrd"
- "--providers.kubernetescrd.allowCrossNamespace=true"
- "--certificatesresolvers.myresolver.acme.storage=/data/acme.json"
- "--certificatesresolvers.myresolver.acme.tlschallenge=true"
- "--certificatesresolvers.myresolver.acme.email=jakub.hajek+webinar@traefik.io"
EOF
Then, create Kustomization to add resources.
cat > ./apps/production/traefik/kustomization.yaml << EOF
---
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: traefik-production
resources:
- namespace.yaml
- ../../base/traefik
patchesStrategicMerge:
- traefik-patch.yaml
EOF
Create a namespace for your Traefik Proxy deployment on a staging cluster.
cat > ./apps/staging/traefik/namespace.yaml << EOF
---
apiVersion: v1
kind: Namespace
metadata:
name: traefik-staging
EOF
Create a Traefik Proxy patch for your staging cluster.
cat > ./apps/staging/traefik/traefik-patch.yaml << EOF
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: traefik
spec:
template:
spec:
containers:
- name: traefik
args:
- "--entryPoints.web.address=:8000/tcp"
- "--entryPoints.websecure.address=:8443/tcp"
- "--entryPoints.traefik.address=:9000/tcp"
- "--api=true"
- "--api.dashboard=true"
- "--ping=true"
- "--providers.kubernetescrd"
- "--providers.kubernetescrd.allowCrossNamespace=true"
- "--certificatesresolvers.myresolver.acme.storage=/data/acme.json"
- "--certificatesresolvers.myresolver.acme.tlschallenge=true"
- "--certificatesresolvers.myresolver.acme.email=jakub.hajek+webinar@traefik.io"
- "--certificatesresolvers.myresolver.acme.caserver=https://acme-staging-v02.api.letsencrypt.org/directory"
EOF
Create a Kustomization resource for deploying your Traefik Proxy-related resources.
cat > ./apps/staging/traefik/kustomization.yaml << EOF
---
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: traefik-staging
resources:
- namespace.yaml
- ../../base/traefik
patchesStrategicMerge:
- traefik-patch.yaml
EOF
Traefik Proxy requires you to have Custom Resources deployed on each cluster. The following command will create a common resource (including Treafik's CRD) that we will deploy on each cluster.
cat > ./infrastructure/crds/traefik-crds.yaml << EOF
---
apiVersion: source.toolkit.fluxcd.io/v1beta1
kind: GitRepository
metadata:
name: traefik-crds
namespace: flux-system
spec:
interval: 30m
url: https://github.com/traefik/traefik-helm-chart.git
ref:
tag: v10.3.0
ignore: |
# exclude all
/*
# path to crds
!/traefik/crds/
---
apiVersion: kustomize.toolkit.fluxcd.io/v1beta1
kind: Kustomization
metadata:
name: traefik-api-crds
namespace: flux-system
spec:
interval: 15m
prune: false
sourceRef:
kind: GitRepository
name: traefik-crds
namespace: flux-system
healthChecks:
- apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
name: ingressroutes.traefik.containo.us
- apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
name: ingressroutetcps.traefik.containo.us
- apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
name: ingressrouteudps.traefik.containo.us
- apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
name: middlewares.traefik.containo.us
- apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
name: middlewaretcps.traefik.containo.us
- apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
name: serverstransports.traefik.containo.us
- apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
name: tlsoptions.traefik.containo.us
- apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
name: tlsstores.traefik.containo.us
- apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
name: traefikservices.traefik.containo.us
EOF
Create Kustomization for your Traefik Proxy CRDS.
cat > ./infrastructure/crds/kustomization.yaml << EOF
---
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: flux-system
resources:
- traefik-crds.yaml
EOF
Create a Kustomization file that deploys the file contained in the CRDS directory.
cat > ./infrastructure/kustomization.yaml << EOF
---
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- crds
EOF
cat > ./infrastructure/kustomization.yaml << EOF
---
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- crds
EOF
cat > ./clusters/production/infrastructure.yaml << EOF
---
apiVersion: kustomize.toolkit.fluxcd.io/v1beta2
kind: Kustomization
metadata:
name: infrastructure
namespace: flux-system
spec:
interval: 10m0s
sourceRef:
kind: GitRepository
name: flux-system
path: ./infrastructure
prune: true
wait: true
EOF
cat > ./clusters/staging/apps.yaml << EOF
---
apiVersion: kustomize.toolkit.fluxcd.io/v1beta2
kind: Kustomization
metadata:
name: apps
namespace: flux-system
spec:
interval: 10m0s
dependsOn:
- name: infrastructure
sourceRef:
kind: GitRepository
name: flux-system
path: ./apps/staging
prune: true
wait: true
EOF
cat > ./clusters/staging/infrastructure.yaml << EOF
apiVersion: kustomize.toolkit.fluxcd.io/v1beta2
kind: Kustomization
metadata:
name: infrastructure
namespace: flux-system
spec:
interval: 10m0s
sourceRef:
kind: GitRepository
name: flux-system
path: ./infrastructure
prune: true
wait: true
EOF
Once we create all initial configurations, we can commit and push the code to your repository. The next step is to bootstrap clusters with the Flux CLI command.
├── apps
│ ├── base
│ │ └── traefik
│ │ ├── kustomization.yaml
│ │ ├── rbac.yaml
│ │ ├── svc.yaml
│ │ └── traefik.yaml
│ ├── production
│ │ └── traefik
│ │ ├── kustomization.yaml
│ │ ├── namespace.yaml
│ │ └── traefik-patch.yaml
│ └── staging
│ └── traefik
│ ├── kustomization.yaml
│ ├── namespace.yaml
│ └── traefik-patch.yaml
├── clusters
│ ├── production
│ │ ├── apps.yaml
│ │ └── infrastructure.yaml
│ └── staging
│ ├── apps.yaml
│ └── infrastructure.yaml
└── infrastructure
├── crds
│ ├── kustomization.yaml
│ └── traefik-crds.yaml
├── kustomization.yaml
└── sources
Once you have created the configuration files, you can bootstrap Flux on both clusters. Before running the bootstrap command, make sure to export the following environment variables.
export GITHUB_TOKEN
export GITHUB_USER
export GITHUB_REPO
GITHUB_USER refers to your username, and GITHUB_REPO refers to the repo you created earlier.
Change the Kubernetes context to the staging cluster and then execute the command that will bootstrap Flux on a staging cluster.
flux bootstrap github \
--branch=main \
--context=t1.aws.traefiklabs.tech \
--owner=${GITHUB_USER} \
--repository=${GITHUB_REPO} \
--path=clusters/staging \
--components-extra=image-reflector-controller,image-automation-controller \
--personal
Observe the following command that will present the status of the provisioned Flux resources. You can also manually explore other resources that have been declared in the created Kubernetes manifests.
Once the staging cluster has been correctly configured, switch the Kubernetes context to the production cluster and execute the following command.
flux bootstrap github \
--branch=main \
--context=t2.aws.traefiklabs.tech \
--owner=${GITHUB_USER} \
--repository=${GITHUB_REPO} \
--path=clusters/production \
--components-extra=image-reflector-controller,image-automation-controller \
--personal
Note
The two commands are very similar. The only difference is in the
pathparameter that's set to eitherclusters/stagingorclusters/production.
Explore the resources that have been configured on your cluster. You can also interact with the Flux CLI to explore the status of all your Flux resources.
whoami applicationCreate a base configuration for the whoami application. The whoami application is a simple demo application that displays headers from the containers. It is a kind of echo server that we use for testing purposes.
mkdir -pv ./apps/{base,staging,production}/whoami
cat > ./apps/base/whoami/deployment.yaml << EOF
---
kind: Deployment
apiVersion: apps/v1
metadata:
name: whoamiv1
labels:
name: whoamiv1
spec:
replicas: 1
selector:
matchLabels:
task: whoamiv1
template:
metadata:
labels:
task: whoamiv1
spec:
containers:
- name: whoamiv1
image: traefik/traefikee-webapp-demo:v2
args:
- -ascii
- -name=FOO
ports:
- containerPort: 80
readinessProbe:
httpGet:
path: /ping
port: 80
failureThreshold: 1
initialDelaySeconds: 2
periodSeconds: 3
successThreshold: 1
timeoutSeconds: 2
resources:
requests:
cpu: 10m
memory: 128Mi
limits:
cpu: 200m
memory: 256Mi
---
apiVersion: v1
kind: Service
metadata:
name: whoamiv1
namespace: app
spec:
ports:
- name: http
port: 80
selector:
task: whoamiv1
EOF
Create an IngressRoute object that exposes the whoami application. The Ingressroute is one of Traefik CRD's resources that create IngressRoute objects to create routers that expose applications externally. You can learn more about this in the official Traefik Proxy documentation on Kubernetes IngressRoutes.
cat > ./apps/base/whoami/ingressroute.yaml << EOF
---
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
name: whoami
spec:
entryPoints:
- websecure
routes:
- kind: Rule
match: Host(\`fix.me\`)
services:
- kind: Service
name: whoamiv1
port: 80
tls:
certResolver: myresolver
EOF
cat > ./apps/base/whoami/kustomization.yaml << EOF
---
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- deployment.yaml
- ingressroute.yaml
EOF
Create a namespace where the whoami application will be deployed.
cat > ./apps/staging/whoami/namespace.yaml <<EOF
---
apiVersion: v1
kind: Namespace
metadata:
name: whoami-staging
EOF
Create a patch for the whoami application. The changes we are adding here are strictly related to the staging environment. It is a tiny change. We are only changing the header, but you can add more environment-related updates such as mounting different secrets and adding Staging environment variables if you choose.
cat > ./apps/staging/whoami/whoami-patch.yaml << EOF
---
kind: Deployment
apiVersion: apps/v1
metadata:
name: whoamiv1
spec:
replicas: 4
template:
spec:
containers:
- name: whoamiv1
args:
- -ascii
- -name=STAGING
EOF
Create a patch that updates the Host rule for the whoami application. The patch is needed to update the host rule that will be used to access the whoami application.
cat > ./apps/staging/whoami/ingressroute-patch.yaml <<EOF
---
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
name: whoami
spec:
routes:
- kind: Rule
match: Host(\`whoami.t1.demo.traefiklabs.tech\`)
services:
- kind: Service
name: whoamiv1
port: 80
EOF
Create a Kustomization configuration to deploy the whoami application on your staging cluster.
cat > ./apps/staging/whoami/kustomization.yaml << EOF
---
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: whoami-staging
resources:
- namespace.yaml
- ../../base/whoami
patchesStrategicMerge:
- whoami-patch.yaml
- ingressroute-patch.yaml
EOF
Create a namespace for the whoami application to be deployed to production.
cat > ./apps/production/whoami/namespace.yaml << EOF
---
apiVersion: v1
kind: Namespace
metadata:
name: whoami-production
EOF
Create a patch for the whoami application. It is a tiny change to show the difference between staging and production examples. In an actual use case, you can add more changes to the production environment, such as mounting production ConfigMaps, Secrets, or environment variables.
cat > ./apps/production/whoami/whoami-patch.yaml << EOF
---
kind: Deployment
apiVersion: apps/v1
metadata:
name: whoamiv1
spec:
replicas: 8
template:
spec:
containers:
- name: whoamiv1
args:
- -ascii
- -name=PRODUCTION
EOF
Create a patch that updates the Host rule for the whoami application. The patch just changed the value of Host matching rule.
cat > ./apps/production/whoami/ingressroute-patch.yaml << EOF
---
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
name: whoami
spec:
routes:
- kind: Rule
match: Host(\`whoami.t2.demo.traefiklabs.tech\`)
services:
- kind: Service
name: whoamiv1
port: 80
EOF
Create a Kustomization configuration to deploy the whoami application to your production cluster.
cat > ./apps/production/whoami/kustomization.yaml << EOF
---
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: whoami-production
resources:
- namespace.yaml
- ../../base/whoami
patchesStrategicMerge:
- whoami-patch.yaml
- ingressroute-patch.yaml
EOF
And there you have it. You have successfully deployed Traefik Proxy in Flux using a GitOps approach if you have followed these steps. This makes it easier for you to maintain multiple configurations in Kubernetes.
Let's summarize what we have already achieved. We started with two built-from-scratch clusters where there were no resources created. Then we created our directory structure, and all the necessary Kubernetes manifests. We:
We created a base configuration and then used Kustomize to create overlays for our staging and production environments. We used a feature called patchStrategicMerge to update our configuration for these environments accordingly. Again, the base configuration was inherited, and the only changes were applied and merged to have the final manifests.
Our entire configuration was done locally, without imperative interactions with our clusters. Once we were happy with the configuration, we pushed the changes to the repo and then bootstrapped Flux on our two clusters.
Once Flux was correctly started on our cluster, it could pull the configuration from the repository and apply it to the appropriate clusters.
Every change we would like to make must be done through a repository, so you edit or update the files and commit them to the repo. Flux will take care of deploying on the clusters.
What are the next steps?
There are many things to explore in regards to Flux and designing scalable infrastructures. I encourage you to look at Traefik's more advanced load balancing techniques. Check out this webinar to learn more. To participate in our community, get involved in our forum. There you can ask questions, post answers, and engage in discussions. I hope to see you there!
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。