Update thousands of linux servers in no time with CTFreak

November 1, 2022
ctfreak scheduling sysadmin

Are you looking to automate the update of your linux servers?

By launching the updates concurrently?

Without having to use a sledgehammer to swat a fly?

Don’t look any further this tuto is made for you 😉.

Requirements

The script

On each server, this shell script will perform the update :

#!/bin/bash
export DEBIAN_FRONTEND=noninteractive
sudo apt update 
sudo apt upgrade -y -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold"

We find the classic apt update/upgrade enriched with some elements allowing to answer automatically any question likely to interrupt the script.

NB: If your servers are under Redhat (or any other distribution based on RPM), I invite you to adapt the script accordingly.

CTFreak

The next step will be to use ctfreak to deploy and run via SSH the script on our 1000 servers.

Login to ctfreak with your administrator account.

Connection

Adding the SSH Credential

Go to SSH Credential -> New SSH Credential button

SSH credential

Add the SSH private key MySSHKey which allows to connect to the servers and validate.

NB : You can also connect with a password instead of a private key.

Adding nodes

A node defines the parameters for connecting to a server via SSH, the nodes are grouped in source nodes.

As a reminder, there are 1000 servers, so 1000 nodes to add.

It is actually possible to add the nodes one by one via the web interface, but in this case, you will rather create a dedicated source node which will be fed by a yaml file describing our 1000 nodes.

Let the file /home/adminuser/ctfreak-nodes.yaml (readable for the user running the ctfreak instance) :

- name: server0001
  tagNames:
    - debian_server
    - linux_server
  username: adminuser
  hostname: server0001.local
  osFamily: UNIX
  sshPort: 22
  
- name: server0002
  tagNames:
    - debian_server
    - linux_server
  username: adminuser
  hostname: server0002.local
  osFamily: UNIX
  sshPort: 22

## ...  
## Complete with nodes 3 to 1000
## ...

Go to Nodes -> New External Node Source button

Connection

Check that the SSH key MySSHKey is selected as Credential (ctfreak will use it to connect to nodes via SSH), then validate to create the source node Linux Servers (this source node will resync its nodes from the yaml file every 2 hours).

If the yaml file has been successfully processed, you should see the following list :

Nodes

Adding the task

Go to Projects -> New Project button

Project

Validate to create the Sysadmin project.

Click on the New Task button.

Task

Validate to create the Upgrade debian servers task.

This task will deploy and execute the shell script :

Execution of the task

To avoid waiting for the 1st of the month to see what happens 😉 , execute the task by clicking the Execute button.

Execution

The execution is launched, then click on the execution identifier #T0ZJX to see its progress for each node.

Once the execution is finished, some nodes may fail (timeout on SSH connection opening, package repository unavailable at the launch of apt-update, DNS issues, …) :

Execution nodes

After checking the Failed only filter to only display failed nodes and check the logs to fix the issue (here a DNS problem), you can launch a new execution to process only failed nodes by clicking on Re-execute failed nodes.

Conclusion

Here you have a reliable process for updating your servers.

For those who would like to go further, here are some suggestions for improvement: