Processing modular and dynamic configuration information in shell

While engaged on a steady integration/steady growth (CI/CD) resolution for a buyer, one in all my first duties was to automate the bootstrapping of a CI/CD Jenkins server in OpenShift. Following DevOps greatest practices, I shortly created a configuration file that drove a script to finish the job. That shortly grew to become two configuration information after I realized I wanted a separate Jenkins server for manufacturing. After that got here the request that the client wanted multiple pair of engineering and manufacturing CI/CD servers for various teams, and every server had related however barely totally different configurations.

When the inevitable modifications needed to be made to the values widespread to 2 or extra of the servers, it was very tough and error-prone to propagate the modifications throughout two or 4 information. As CI/CD environments have been added for extra advanced testing and deployments, the variety of shared and particular values for every group and setting grew.

As the modifications grew to become extra frequent and the information extra advanced, making modifications throughout the configuration information grew to become increasingly unmanageable. I wanted a greater resolution to resolve this age-old drawback and handle modifications sooner and extra reliably. More importantly, I wanted an answer that might permit my shoppers to do the identical after turning my accomplished work over to them.

Defining the issue

On the floor, this appears like a really simple drawback. Given my-config-file.conf (or a *.ini or *.properties) file:

KEY_1=value-1
KEY_2=value-2

You simply should execute this line on the prime of your script:

#!/usr/bin/bash
set -o allexport
supply my-config-file.conf
set +o allexport

This code realizes all of the variables inside your configuration file within the setting, and set -o allexport robotically exports all of them. The unique file, being a typical key/worth properties file, can also be very normal and straightforward to parse into one other system. Where it will get extra sophisticated is within the following situations:

1. Some of the values are copied and pasted from variable to variable and are associated. Besides violating the DRY (“don’t repeat yourself”) precept, it is error-prone, particularly when values have to be modified. How can values throughout the file be reused?
2. Portions of the configuration file are reusable over a number of runs of the unique script, and others are helpful just for a selected run. How do you progress past copy and paste and modularize the information in order that some items will be reused elsewhere?
3. Once the information are modularized, how do you deal with conflicts and outline precedent? If a secret’s outlined twice in the identical file, which worth do you’re taking? If two configuration information outline the identical key, which will get priority? How can a selected set up override a shared worth?
4. The configuration information are initially meant for use by a shell script and are written for processing by shell scripts. If the configuration information have to be loaded or reused in one other setting, is there a approach to make them simply obtainable to different programs with out additional processing? I wished to maneuver among the key/worth pairs right into a single ConfigMap in Kubernetes. What’s one of the simplest ways to make the processed knowledge obtainable to make the import course of simple and straightforward in order that different programs do not have to grasp how the config information are structured?

This article will take you thru some easy code snippets and present how simple that is to implement.

Defining the configuration file content material

Sourcing a file means it should supply variables in addition to different shell statements like instructions. For this objective, configuration information ought to solely be about key/worth pairs and never about defining features or executing code. Therefore, I will outline these information equally to property and .ini information:

KEY_1=$
KEY_2=value-2
...
KEY_N=value-n

From this file, it’s best to anticipate the next habits:

$ supply my-config-file.conf
$ echo $KEY_1
value-2

I purposefully made this a bit of counterintuitive in that it refers to a price I have not even outlined but. Later on this article, I’ll present you the code to deal with this state of affairs.

Defining modularization and priority

To maintain the code easy and make defining the information intuitive, I carried out a left-to-right, top-to-bottom priority technique for information and variables, respectively. More particularly, given an inventory of configuration information:

1. Each file within the checklist can be processed first-to-last (left-to-right)
2. The first definition of a key would outline the worth, and subsequent values can be ignored

There are some ways to do that, however I discovered this technique simple, simple to code, and straightforward to elucidate to others. In different phrases, I’m not claiming that is the very best design choice, however it works, and it simplifies debugging.

Given this colon-delimited checklist of two configuration information:

first.conf:second.conf

with these contents:

# first.conf 
KEY_1=value-1
KEY_1=ignored-value

# first.conf 
KEY_1=ignored-value

you’ll anticipate:

The resolution

This perform will implement the outlined necessities:

_create_final_configuration_file() {

    # convert the checklist of information into an array

    native CONFIG_FILE_LIST=($(echo $1 | tr ':' ' '))

    native WORKING_DIR=$
    # removes any trailing whitespace from every file, if any
    # that is completely required when importing into ConfigMaps
    # put quotes round values if additional areas are obligatory
    sed -i -e 's/s*$//' -e '/^$/d' -e '/^#.*$/d' $
    # iterates over every file and prints (default awk habits)
    # every distinctive line; solely takes first worth and ignores duplicates
    awk -F= '!line[$1]++' $ > $
    # should export all the things, and supply it twice:
    # 1) first supply is to comprehend variables
    # 2) second time is to comprehend references
    set -o allexport
    supply $
    supply $
    set +o allexport
    # use envsubst command to comprehend worth references
    cat $ | envsubst > $
It performs the next steps:

It trims extraneous white house from every line.
It iterates by every file and writes out every line with a singular key (i.e. due to awk magic, it skips duplicate keys) to an intermediate configuration file.
It sources the intermediate file twice to comprehend all references in reminiscence.
The referenced values within the intermediate file are realized from the values now in reminiscence and written out to a last configuration file, which can be utilized for additional processing.

As the above notes, when the mixed configuration intermediate file is sourced, it have to be performed twice. This is in order that the referenced values which can be outlined after being referenced will be correctly realized in reminiscence. The envsubst substitutes the values of setting variables, and the output is redirected to the ultimate configuration file for attainable postprocessing. Per the earlier instance’s requirement, this may take the type of realizing the information in a ConfigMap:
kubectl create cm my-config-map --from-env-file=$ 
    -n my-namespace
Sample code
You can discover pattern code with particular.conf and shared.conf information demonstrating how one can mix information representing a selected configuration file and a normal, shared configuration file in my GitHub repository modular-config-file-sample. The configuration information are composed of:
# particular.conf
KEY_1=$
KEY_2='some worth'
KEY_1='this worth will likely be ignored'
# shared.conf
SHARED_KEY_1='some shared worth'
SHARED_KEY_2=$
SHARED_KEY_1='this worth won't ever see the sunshine of day'
KEY_1='this was overridden'
Note the only quotes across the values. I purposefully selected instance values with areas to make issues extra fascinating and in order that the values needed to be in quotes; in any other case, when the information are sourced, every phrase can be interpreted as a separate command. However, the variable references don’t have to be in quotes as soon as the values are set.
The repository accommodates a small shell script utility, pconfs.sh. Here’s what occurs once you run the next command from throughout the pattern code listing:
# NOTE: see the pattern code for the complete set of command line choices

$ ./pconfs.sh -f particular.conf:shared.conf
================== COMBINED CONFIGS BEFORE =================
KEY_1=$KEY_2
KEY_2='some worth'
SHARED_KEY_1='some shared worth'
SHARED_KEY_2=$SHARED_KEY_1
================ COMBINED CONFIGS BEFORE END ===============
================= PROOF OF SUBST IN MEMORY =================
KEY_1: some worth
SHARED_KEY_2: some shared worth
=============== PROOF OF SUBST IN MEMORY END ===============
================== PROOF OF SUBST IN FILE ==================
KEY_1=some worth
KEY_2='some worth'
SHARED_KEY_1='some shared worth'
SHARED_KEY_2=some shared worth
================ PROOF OF SUBST IN FILE END ================
This proves that even advanced values could also be referenced earlier than and after a price is outlined. It additionally exhibits that solely the primary definition of the worth is retained, whether or not inside or throughout information, and that priority is given left to proper in your checklist of information. This is why I specify parsing particular.conf first when operating this command; this permits a selected configuration to override any of the extra normal, shared values within the instance.
You ought to now have an easy-to-implement resolution for creating and utilizing modular configuration information within the shell. Also, the outcomes of processing the information ought to be simple sufficient to make use of or import with out requiring the opposite system to grasp the information’s unique format or group.