How to achieve a golden CI/CD pipeline
With consensus and decoupling
A golden CI/CD pipeline is the list of actions that every piece of software must take to move a new code to production.
In today’s issue, I want to share with you what a Continuous Integration and Continuous Delivery pipeline should look like and how to make it in a way that could be reused by most of your dev teams, no matter what coding language they are using for their applications.
☝🏼 The problem you already know; each dev team:
It is using a different pipeline file per service and repository, with a different list of stages.
It is implementing the applications in different languages.
The engineering organization comes with a new requirement: Upload the code coverage to a storage.
🤯 You have to refactor more than 100 repositories, to add the new stage required, finding that old legacy application that you haven’t touched since Thor was a kid.
That’s a lot of technical debt to solve.
In this essay, I will explain to you:
Stages of a golden pipeline.
How to make it flexible.
💮 Steps of a golden pipeline
You have to standardize.
Having different flavors of pipelines becomes unsustainable over time. Changes will inevitably arise, requiring updates to all pipelines for every application. This demands significant effort from development teams, delaying the implementation of new features and ultimately hindering the delivery of value to your company.
Remember: 👉🏼 Find a problem, seek a solution, and platformize that solution.
From my experience, these are the steps that a common CI/CD pipeline should have are:
Build: Whatever build means for your application.
Test: Run all tests that make sense on the Continuous Integration stage. Nice to have: Create a report that could be visible by the pipeline runner tool you are using.
Quality Gates: Ensuring that your code follows the right standards. Things like SonarQube checks and others are in this stage.
Publish: Here, you have to package your application and publish it into an artifact repository, like Artifactory or AWS Elastic Container Registry, so it’s available for shipping to production.
Deliver: This stage will update your GitOps repository or the configuration that rules the version of your applications, with the new version. That change of version should trigger another pipeline to deploy the application in production.
I’m pretty sure that list of stages sounds familiar to you.
This is good, we have a list of tasks to do but, how do we make them happen, taking into account that each application can be implemented in different languages. And no, using if…else statements in the pipeline is not the right way, as you already figure out.
⛓️💥 How to make it flexible
Decouple, decouple, and decouple.
If you are willing to have good code, that word is resonating in your brain.
☝🏼 We need to write every stage in a way that the user of the golden pipeline implements the actual “execution” of each step.
For this:
Use a tool that could run tasks and it’s language agnostic.
Use a pipeline runner that everybody can use.
My take nowadays is:
Makefile to implement each of the tasks to run for the application. A somewhat old technology that allows you to define a set of tasks to be executed. Each development team will implement the call of each of the steps that the GitHub Action defines.
For example, for the test stage:
For Java applications, it will mean to run: ./gradlew test.
For Python applications, it will mean to run: python -m pytest.
👉🏼 So, when the GitHub Action checks out each repository and calls make test, the real call will go to the implementation done by the development team.
Use GitHub Actions to run your CI/CD pipeline. I’ve been using Jenkins pipelines for more than 10 years and, while it’s a very good tool, I now find GitHub Actions to be much more user-friendly, faster to implement, and more intuitive.
The GitHub Action will be a workflow that defines the list of stages and will implement the call to make a task for each of them. Just that.
👉🏼 Remember, the real implementation will come from the repository on which the workflow made checkout.
With this approach, we are decoupling what is required for a CI/CD and the way to achieve it, since for each application each stage could mean something different.
✨ Takeaways
Let’s wrap up for the day:
Define your stages from the guide provided above, and include or remove the stages you need for the context of your company’s organization.
Get consensus from different parts of the engineering organization about the list of stages.
Find a tool that will help you to decouple the execution of each stage.
We are more than ✨709 subscribers✨, thank you all for your trust.
You rock! 🖖🏼
𝘐𝘧 𝘺𝘰𝘶 𝘦𝘯𝘫𝘰𝘺𝘦𝘥 𝘵𝘩𝘪𝘴 𝘱𝘰𝘴𝘵, 𝘵𝘩𝘦𝘯 𝘤𝘭𝘪𝘤𝘬 𝘵𝘩𝘦 💜. 𝘐𝘵 𝘩𝘦𝘭𝘱𝘴!
𝘐𝘧 𝘺𝘰𝘶 𝘬𝘯𝘰𝘸 𝘴𝘰𝘮𝘦𝘰𝘯𝘦 𝘦𝘭𝘴𝘦 𝘸𝘪𝘭𝘭 𝘣𝘦𝘯𝘦𝘧𝘪𝘵 𝘧𝘳𝘰𝘮 𝘵𝘩𝘪𝘴, ♻️ 𝘴𝘩𝘢𝘳𝘦 𝘵𝘩𝘪𝘴 𝘱𝘰𝘴𝘵.
Great summary on CI/CD pipelines, Marcos. I used Makefiles extensively at the start of my career, but for no reason (maybe influenced by other devs), I moved to shell scripts. What would you say the benefit is of using Makefiles over shell scripts?