Setting up a self-hosted CI/CD pipeline

March 23, 2023
ci/cd ctfreak devops self-hosted

As a reminder, a CI/CD pipeline (or Continuous Integration/Continuous Deployment pipeline) is a series of actions to be performed in order to consistently and reliably distribute a new version of a software.

In this article, we will see how to set up a minimalistic CI/CD pipeline (automatically triggering builds and deployments upon push events to a git repository) that is self-hosted and can be adapted according to your needs.

Prerequisites

We will need:

The sources

The sources of our application can be summarized in this Makefile:

build:
	@echo "Build my awesome app"
deploy: build
	@echo "Deploy my awesome app"

You can find them in this git repository, with the master branch corresponding to the latest deployed version and the dev branch to the version currently under development.

Objectives

Let’s review the objectives to be achieved by setting up our CI/CD pipeline:

To do this, Ctfreak will act as a gateway between the git forge and the build server.

That is, every time a push is received by the forge:

Build server configuration

Connect to your build server and run the following commands:

ctfreakdemo@buildsrv:~$ git clone https://github.com/jypsoftware/myawesomeapp.git myawesomeapp.master
> Cloning into 'myawesomeapp.master'...
> remote: Enumerating objects: 15, done.
> remote: Counting objects: 100% (15/15), done.
> remote: Compressing objects: 100% (13/13), done.
> remote: Total 15 (delta 2), reused 6 (delta 1), pack-reused 0
> Receiving objects: 100% (15/15), 14.38 KiB | 3.59 MiB/s, done.
> Resolving deltas: 100% (2/2), done.

ctfreakdemo@buildsrv:~$ git clone https://github.com/jypsoftware/myawesomeapp.git myawesomeapp.dev
> Cloning into 'myawesomeapp.dev'...
> remote: Enumerating objects: 15, done.
> remote: Counting objects: 100% (15/15), done.
> remote: Compressing objects: 100% (13/13), done.
> remote: Total 15 (delta 2), reused 6 (delta 1), pack-reused 0
> Receiving objects: 100% (15/15), 14.38 KiB | 7.19 MiB/s, done.
> Resolving deltas: 100% (2/2), done.

ctfreakdemo@buildsrv:~$ cd myawesomeapp.dev

ctfreakdemo@buildsrv:~/myawesomeapp.dev$ git checkout dev
> Branch 'dev' set up to track remote branch 'dev' from 'origin'.
> Switched to a new branch 'dev'

As you can see, we have cloned the git repo 2 times (1 per branch), which will allow us to avoid potential conflicts if the builds of the master and dev branches were to run concurrently.

Ctfreak Configuration

Log in to your Ctfreak instance with an admin account.

Adding the build server

Start by adding the SSH key to connect to the build server via SSH CredentialNew SSH Credential:

Adding the SSH key

Then add the server itself via NodesInternal NodesNew node:

Adding the server

Make sure that the SSH Key for buildsrv key is selected as the Credential (Ctfreak will use it to connect to the server), then validate to create the buildsrv node.

Creating the CI/CD project

We are going to create a dedicated project to gather all the tasks related to our CI/CD pipeline.

Go to ProjectsNew Project

Project creation

Validate to create the project.

Adding a notifier

Go to ProjectsCI/CDNotifiersNew notifier

Selecting the notifier type

Choose Discord as the notifier type (for example).

Editing the notifier

Specify the discord webhook URL to call and validate.

List of notifiers

A test notification can be sent by clicking on the send button (the one shaped like a paper airplane).

Creating the build task

Go to ProjectsCI/CDNew task

Selecting the task type

Choose Bash Script as the task type.

Continuous integration task

As you may have guessed, this task will compile the dev branch on the build server and send a notification in case of failure.

The smart chaining multiple execution policy selected here will avoid launching unnecessary builds.

Let’s take an example: you have a build that lasts half an hour and 3 pushes are sent to the git repository at 5-minute intervals.

At the first push, the build is launched immediately.

At the 2nd push, a 2nd build is put on hold since the first one is not yet finished.

At the 3rd push, we don’t plan to do a 3rd build because the 2nd one is still on hold.

Creating the deployment task

To save time, create this task from the build task via ProjectsCI/CDBuid myawesomeapp / devDuplicate.

Continuous deployment task

Making the following modifications:

Adding webhooks

Now that our build and deployment tasks are created, we will add an incoming webhook for each one that can be called by our git forge.

Let’s start with the deployment task by going to ProjectsCI/CDBuid myawesomeapp / devIncoming webhooksNew webhook.

Selecting webhook type

Choose GitHub as the webhook type.

When GitHub calls our webhook following a push, it will do so regardless of the branch involved, but our task should only be executed when this push concerns the dev branch.

To solve this issue, we add an execution condition that will apply to the JSON payload sent by GitHub. The task will only be executed if the condition is met (i.e. the payload does indeed refer to the dev branch).

Creating the dev webhook 1

Define a secret if necessary, which will allow Ctfreak to validate that calls to the webhook come from GitHub, and validate.

Once the webhook is created, we retrieve its URL for GitHub configuration.

Creating the dev webhook 2

In your GitHub repository settings, add the webhook we just created, specifying:

Configuring the dev webhook in github

Proceed in the same way (create the webhook, reference it in GitHub) for the deployment task (be sure to reference the master branch in the execution condition):

Creating the master webhook

Pipeline testing

Now that everything is set up, let’s see what happens when we do a git push on the dev branch:

After calling the GitPushDev webhook, the build task was executed successfully.

By clicking on the eye-shaped icon, we can see the execution logs.

Let’s do the same test with a git push on the master branch:

This time, the GitPushMaster webhook was called as expected.

And the added bonus here is the receipt of a Discord notification confirming that the deployment was successful.

Conclusion

You now have an operational CI/CD pipeline that is flexible enough to handle even the most complex build workflows.

By opting for this self-hosted solution and placing the core of the build process in a shell script rather than in a continuous integration software like Jenkins, GitHub Actions, etc., you will benefit from several advantages: