29 Oct

Industrializing the CI/CD environment (1): Jenkins

Software industrialization implies, amongst other things, that it must be buildable from a script which automates the whole process. In the case of a CI/CD environment, it is also interesting to industrialize it, as this allows a better understanding of the system configuration, the ability to introduce changes on a controlled environment, revert to a well-known state, etc. The alternative to this is what Martin Fowler denominates a “Snowflake Server“.

As for the concrete Jenkins’ case, not having a mechanism which allows to rebuild the whole system, can be somewhat risky, specially if you are working with weekly releases: the upgraded version might have problems, or there might be incompatible plugins or unwanted side effects… On these situations, it is complicated to rollback to a previous version. Furthermore, manually installing a previous version, with its plugins, jobs, users, nodes, etc. and leave it as it was un the first place can be quite a challenge; for instance, it is not completely unreasonable that, when upgrading a plugin, it decides to also migrate its configuration on every job that uses it. If, because of whatever reason, it is necessary to downgrade after a situation like this, well, that’s going to hurt. A lot.

Also, it doesn’t help that there isn’t a more or less standard approach to perform this task. As of today, the official documentation regarding this theme is basically null. It is true that there are some Jenkins plugins that help on trying to solve this task, namely: Backup Plugin, thinBackup and SCM Sync Backup, but they are either discontinuated, or they can’t be scheduled, or they also save workspace data, or they do not save all the configuration (f.ex., if you are a user of the promotions plugin, neither of these plugins will save your promotion data).

The way we have solved this at work is with a couple of scripts, the first one is meant to save all the configuration on the scm tool (in our case, Apache Subversion):

#!/bin/bash
echo '*****************************'
echo '0.- Change into $JENKINS_HOME'
echo '*****************************'
cd /path/to/your/jenkins_home


echo '************************'
echo '1.- update plugins list '
echo '************************'
curl -sSL "http://JENKINS_HOST:JENKINS_PORT/jenkins/pluginManager/api/xml?depth=1&xpath=/*/*/shortName|/*/*/version&wrapper=plugins" | perl -pe 's/.*?<shortName>([\w-]+).*?<version>([^<]+)()(<\/\w+>)+/\1 \2\n/g'|sed 's/ /:/' > plugins.txt


echo '*****************************************************************************'
echo '2.- Template saves wipe out promotions folder, so restore it with local data '
echo '*****************************************************************************'
# IFS weirdness to cd into directories whose names contain spaces, ref. http://www.cyberciti.biz/tips/handling-filenames-with-spaces-in-bash.html
cd jobs
SAVEIFS=$IFS
IFS=$(echo -en "\n\b")
for dir in `find ./* -type d -prune`;
do
  echo "svn update promotions on $dir" && cd $dir && svn update promotions --force --accept=mine-full && cd ..
done
IFS=$SAVEIFS
cd ..


echo '********************************************************************************************************************'
echo '3.- Add any new conf files, jobs with promotions, users, userContent, secrets, nodes and list of installed plugins. '
echo '********************************************************************************************************************'
svn add --force --parents *.xml jobs/*/config.xml jobs/*/promotions/*/config.xml users/*/config.xml userContent/* nodes/* secret.key secret.key.not-so-secret secrets/* plugins.txt


echo '***************************************************'
echo "4.- Ignore things in the root we don't care about. "
echo '***************************************************'
echo "warnlog\n*.log\n*.tmpn*.old\n*.bak\n*.jar\n*.json" > jenkins_ignores && svn propset svn:ignore -F jenkins_ignores . && rm jenkins_ignores 


echo '***************************************************'
echo "5.- Ignore things inside jobs we don't care about. "
echo '***************************************************'
echo "builds\nlast*\nnext*\n*.txt\n*.log\nworkspace*\ncobertura\njavadoc\nhtmlreports\nncover\ndoclinks" > jenkins_ignores && svn propset svn:ignore -F jenkins_ignores jobs/* && rm jenkins_ignores 


echo '***************************************************************'
echo '6.- Remove anything from SVN that no longer exists in Jenkins. '
echo '***************************************************************'
svn status | grep '!' | awk '{print $2;}' | xargs -r svn rm 


echo '**********************************************************************************'
echo '7.- And finally, check in of course, showing status before and after for logging. '
echo '**********************************************************************************'
svn st && svn ci --non-interactive --username=SVN_USER --password=SVN_PASSWORD -m "automated commit of Jenkins configuration" && svn st

exit 0

The second step won’t apply always, as it is meant to take into account interactions between EZ Templates and promotions plugins; if you are not using both of them this step won’t do anything.

We use Jenkins to run this script on a weekly basis, so we are able to have all the configuration in a safe place. To recover a Jenkins installation from version control we perform these steps:

  • check out the configuration on a directory, which will become the new JENKINS_HOME.
  • download, through wget or whatever, the appropiate Jenkins war.
  • execute the following script, adaptaded from the official Jenkins’ Docker image:
#! /bin/bash

# Parse a support-core plugin -style txt file as specification for jenkins plugins to be installed
# in the reference directory, so user can define a derived Docker image with just :
#
# FROM jenkins
# COPY plugins.txt /plugins.txt
# RUN /usr/local/bin/plugins.sh /plugins.txt
#
# Note: Plugins already installed are skipped
#

set -e

JENKINS_WAR=../jenkins.war
JENKINS_HOME=.
JENKINS_PLUGINS_DIR=$JENKINS_HOME/plugins
REF=$JENKINS_HOME/ref
# jenkins update center
JENKINS_UC=${JENKINS_UC:-https://updates.jenkins-ci.org}

if [ -z "$1" ]
then
    echo "
USAGE:
  Parse a support-core plugin -style txt file as specification for jenkins plugins to be installed
  in the reference directory, so user can define a derived Docker image with just :

  FROM jenkins
  COPY plugins.txt /plugins.txt
  RUN /usr/local/bin/plugins.sh /plugins.txt

  Note: Plugins already installed are skipped

"
    exit 1
else
    JENKINS_INPUT_JOB_LIST=$1
    if [ ! -f $JENKINS_INPUT_JOB_LIST ]
    then
        echo "ERROR File not found: $JENKINS_INPUT_JOB_LIST"
        exit 1
    fi
fi

# the war includes a # of plugins, to make the build efficient filter out
# the plugins so we dont install 2x - there about 17!
if [ -d $JENKINS_HOME ]
then
    TEMP_ALREADY_INSTALLED=$JENKINS_HOME/preinstalled.plugins.$$.txt
else
    echo "ERROR $JENKINS_HOME not found"
    exit 1
fi

if [ -d $JENKINS_PLUGINS_DIR ]
then
    echo "Analyzing: $JENKINS_PLUGINS_DIR"
    for i in `ls -pd1 $JENKINS_PLUGINS_DIR/*|egrep '\/$'`
    do
        JENKINS_PLUGIN=`basename $i`
        JENKINS_PLUGIN_VER=`egrep -i Plugin-Version "$i/META-INF/MANIFEST.MF"|cut -d\: -f2|sed 's/ //'`
        echo "$JENKINS_PLUGIN:$JENKINS_PLUGIN_VER"
    done > $TEMP_ALREADY_INSTALLED
else
    #JENKINS_WAR=/usr/share/jenkins/jenkins.war
    if [ -f $JENKINS_WAR ]
    then
        echo "Analyzing war: $JENKINS_WAR"
        TEMP_PLUGIN_DIR=/tmp/plugintemp.$$
        echo "TEMP_PLUGIN_DIR: $TEMP_PLUGIN_DIR"
        for i in `jar tf $JENKINS_WAR|egrep 'plugins'|egrep -v '\/$'|sort`
        do
            echo "TEMP_PLUGIN_DIR: $TEMP_PLUGIN_DIR"
            rm -fr $TEMP_PLUGIN_DIR
            mkdir -p $TEMP_PLUGIN_DIR
            PLUGIN=`basename $i|cut -f1 -d'.'`
            echo "$PLUGIN"
            (cd $TEMP_PLUGIN_DIR;jar xf $JENKINS_WAR "$i";jar xvf $TEMP_PLUGIN_DIR/$i META-INF/MANIFEST.MF >/dev/null 2>&1)
            VER=`egrep -i Plugin-Version "$TEMP_PLUGIN_DIR/META-INF/MANIFEST.MF"|cut -d\: -f2|sed 's/ //'`
            echo "$PLUGIN:$VER"
        done > $TEMP_ALREADY_INSTALLED
        rm -fr $TEMP_PLUGIN_DIR
    else
        rm -f $TEMP_ALREADY_INSTALLED
        echo "ERROR file not found: $JENKINS_WAR"
        exit 1
    fi
fi

#REF=/usr/share/jenkins/ref/plugins
mkdir -p $REF
COUNT_PLUGINS_INSTALLED=0
while read spec || [ -n "$spec" ]; do

    plugin=(${spec//:/ });
    [[ ${plugin[0]} =~ ^# ]] && continue
    [[ ${plugin[0]} =~ ^\s*$ ]] && continue
    [[ -z ${plugin[1]} ]] && plugin[1]="latest"

    if [ -z "$JENKINS_UC_DOWNLOAD" ]; then
      JENKINS_UC_DOWNLOAD=$JENKINS_UC/download
    fi

    if ! grep -q "${plugin[0]}:${plugin[1]}" $TEMP_ALREADY_INSTALLED
    then
        echo "Downloading ${plugin[0]}:${plugin[1]}"
        curl --retry 3 --retry-delay 5 -sSL -f ${JENKINS_UC_DOWNLOAD}/plugins/${plugin[0]}/${plugin[1]}/${plugin[0]}.hpi -o $REF/${plugin[0]}.jpi
        unzip -qqt $REF/${plugin[0]}.jpi
        COUNT_PLUGINS_INSTALLED=`expr $COUNT_PLUGINS_INSTALLED + 1`
    else
        echo "  ... skipping already installed:  ${plugin[0]}:${plugin[1]}"
    fi
done  < $JENKINS_INPUT_JOB_LIST

echo "---------------------------------------------------"
if [ $COUNT_PLUGINS_INSTALLED -gt 0 ]
then
    echo "INFO: Successfully installed $COUNT_PLUGINS_INSTALLED plugins."

    if [ -d $JENKINS_PLUGINS_DIR ]
    then
        echo "INFO: Please restart the container for changes to take effect!"
    fi
else
    echo "INFO: No changes, all plugins previously installed."

fi
echo "---------------------------------------------------"

#cleanup
rm $TEMP_ALREADY_INSTALLED
exit 0

and profit!

Leave a Reply

Your email address will not be published. Required fields are marked *