Shared Config Manager

Solves the problem of maintaining configuration volumes in the Docker world.

It allows to have a container (in the same POD for Kubernetes or as a sidekick in Rancher) maintaining a few configuration/resource directories from different sources:

• A GIT repository
• A S3 bucket
• Another volume or server through rsync

A source can be refreshed through a webhook and the status of all the containers can be queried through a simple API.

Configuration

First you need to configure the source for the master configuration using the MASTER_CONFIG environment variable with something like:

env:
  MASTER_CONFIG: |
    type: git
    repo: git@github.com:camptocamp/master_config.git

That will use the shared_config_manager.yaml file at the root of the repository to configure all the sources. This configuration file looks like that:

sources:
  test_git:
    type: git
    repo: git@github.com:camptocamp/test_git.git
    target_dir: /usr/local/tomcat/webapps/ROOT/print-apps

With this example, the config container will contain a /usr/local/tomcat/webapps/ROOT/print-apps directory that is a clone of the git@github.com:camptocamp/test_git.git repository and is identified as test_git.

You can configure more than one source.

There is a standalone mode where you can configure only one source through the MASTER_CONFIG environment variable. For that, set the standalone key to true in the MASTER_CONFIG and no shared_config_manager.yaml will be searched.

There is also a mode where you can put directly the sources in the MASTER_CONFIG by setting the sources section directly at the root of the MASTER_CONFIG.

Tunings

A few environment variables can be used to tune the containers:

• SCM__MASTER_CONFIG: The master configuration (string containing the YAML config)
• SCM__MASTER_TARGET: where to store the master config (defaults to /master_config)
• SCM__API_MASTER: if defined, this is a master with slaves (no template evaluation)
• SCM__SECRET: the secret used to authenticate the request between the client and the server

Slave-related variables:

• SCM__SLAVE__ENABLED: run in slave mode (set to true for the slave process)
• SCM__SLAVE__API_BASE_URL: how to reach the master for slaves
• SCM__SLAVE__TAG_FILTER: load only the sources having the given tag (the master config is always loaded)
• SCM__SLAVE__TARGET: default base directory for the target_dir configuration (defaults to /config)
• SCM__SLAVE__RETRY_NUMBER: retry attempts when fetching from master (defaults to 3)
• SCM__SLAVE__RETRY_DELAY: delay between retries in seconds (defaults to 1)
• SCM__SLAVE__REQUESTS_TIMEOUT: timeout in seconds for slave fetch requests (defaults to 30)
• SCM__SLAVE__INIT_SOURCES_CONCURRENCY: number of sources loaded in parallel while reading master config (defaults to 4)

SCM__SLAVE__API_BASE_URL should include the effective route prefix configured through C2C__ROUTE_PREFIX (for example http://api:8080/scm when C2C__ROUTE_PREFIX=/scm/).

See [https://github.com/camptocamp/c2casgiutils] for other parameters.

Slave fetch behavior

On the slaves, a source is fetched from the master in a temporary directory where the templates are evaluated, then the temporary directory is moved atomically in place of the source directory. That way, the source directory is never empty nor partially updated, and the fetches and refreshes of a source are serialized.

Sources

Common source configuration parameters

• type: the type of source
• target_dir: the location where the source will be copied (default is the value of id in /config)
• excludes: the list of files/directories to exclude
• template_engines: a list of template engine configurations
• tags: an optional list of tags. Slaves having SCM__SLAVE__TAG_FILTER defined will load only sources having the matching tag.

GIT source configuration parameters

• type: git
• ssh_key: the private SSH key to use as identity (optional)
• repo: the GIT repository URL
• branch: the GIT branch to use (defaults to master)
• sub_dir: if only a sub_directory of the repository needs to be copied (defaults to the root of the repository)
• sparse: if true (the default) and sub_dir is defined, will use a sparse clone. Disable that if you have multiple sources using the same repository (will avoid cloning it for each source).

Rsync source configuration parameters

• type: rsync
• source: the source for the rsync command
• ssh_key: the private SSH key to use as identity (optional)

Rclone source configuration parameters

• type: rclone
• config: The content of the rclone configuration section to use.
• sub_dir: An optional sub-directory inside the remote (bucket + path for the S3 remotes)

Template engines

• type: can be mako or shell
• data: a dictionary of key/value to pass as a parameter to the template engine
• environment_variables: If true, take into account the process's environment variables if not found in data. Only variables starting with a prefix listed in SCM_ENV_PREFIXES (list separated by :) are allowed.
• dest_sub_dir: If specified, all the files, including the ones not evaluated as templates will be copied into the given sub directory.

Slave only mode

By default the image starts a WSGI server listening on port 8080. In big deployments a full WSGI server could use a sizeable amount of RAM. So you could have only a couple of such containers and the rest running as slaves. For that, change the command run by the container to shared-config-slave.

Example docker-compose for Rancher

docker-compose.yaml:

version: '2'
services:
  scm_api:
    image: camptocamp/shared_config_manager:latest
    environment: &scm_env
      C2C_REDIS_URL: redis://redis:6379
      MASTER_CONFIG: &master_config |
        type: git
        repo: git@github.com:camptocamp/master_config.git
      SCM__SECRET: changeme
      SCM__SLAVE__TAG_FILTER: master
    links:
      - redis
    labels:
      io.rancher.container.hostname_override: container_name
      lb.routing_key: scm-${ENVIRONMENT_NAME}
      lb.haproxy_backend.timeout_server: 'timeout server 120s'
      lb.haproxy_backend.maxconn: default-server maxconn 1

  scm_slave:
    image: camptocamp/shared_config_manager:latest
    environment: *scm_env
    command: ['shared-config-slave']
    volumes:
      - /usr/local/tomcat/webapps/ROOT/print-apps
    links:
      - redis
    labels:
      io.rancher.container.hostname_override: container_name
      io.rancher.scheduler.global: 'true'

  print:
    image: camptocamp/mapfish_print:3.14.1
    labels:
      io.rancher.container.hostname_override: container_name
      io.rancher.scheduler.global: 'true'
      io.rancher.sidekicks: scm_slave
    volumes_from:
      - scm_slave

  redis:
    labels:
      io.rancher.container.hostname_override: container_name
    image: redis:4
    mem_limit: 64m
    command: redis-server --save "" --appendonly no
    user: www-data

API

Refresh

• GET {ROUTE_PREFIX}/1/refresh/{ID}

Refresh the given source {ID}. Returns 200 in case of success. The actual work is done asynchronously.

To refresh the master configuration (list of sources), use master as ID.

• POST {ROUTE_PREFIX}/1/refresh/{ID}

Same as the GET API, but to be used with a GutHub/GitLab webhook for push events. Will ignore events for other branches.

• GET {ROUTE_PREFIX}/1/refresh

Refresh all sources. Returns 200 in case of success. The actual work is done asynchronously.

The master configuration is not refreshed.

• POST {ROUTE_PREFIX}/1/refresh

Same as the GET API, but to be used with a GutHub/GitLab webhook for push events. Will ignore events for other branches.

Status

• GET {ROUTE_PREFIX}/1/status

Returns the glable status, looking like that:

When a source has no template engines configured, the template_engines field is an empty list.

{
  "slaves": {
    "api": {
      "sources": {
        "master": {
          "hash": "240930ea8580d8392544bc0f42bdd1720b772a46",
          "repo": "/repos/master",
          "type": "git"
        },
        "test_git": {
          "hash": "4e066840860d77b143cbecbb8d23db3b755980b2",
          "repo": "/repos/test_git",
          "template_engines": [
            {
              "environment_variables": { "TEST_ENV": "42" },
              "type": "shell"
            }
          ],
          "type": "git"
        }
      }
    },
    "slave": {
      "sources": {
        "master": {
          "hash": "240930ea8580d8392544bc0f42bdd1720b772a46",
          "repo": "/repos/master",
          "type": "git"
        },
        "test_git": {
          "hash": "4e066840860d77b143cbecbb8d23db3b755980b2",
          "repo": "/repos/test_git",
          "template_engines": [
            {
              "environment_variables": { "TEST_ENV": "42" },
              "type": "shell"
            }
          ],
          "type": "git"
        }
      }
    }
  }
}

• GET {ROUTE_PREFIX}/1/status/{ID}

Returns the status for the given source ID, looking like that:

{
  "statuses": [
    {
      "hash": "4e066840860d77b143cbecbb8d23db3b755980b2",
      "repo": "/repos/test_git",
      "template_engines": [
        {
          "environment_variables": { "TEST_ENV": "42" },
          "type": "shell"
        }
      ],
      "type": "git"
    }
  ]
}

Tarball

• GET {ROUTE_PREFIX}/1/tarball/{ID}

Returns a .tar.gz containing the current content for the given source.

Authentication and Permissions

The shared config manager supports GitHub OAuth authentication. User permissions are determined by their access level on the configured GitHub repository:

• Read access (pull): Users with only read access can view the status, download tarballs, and browse the web UI in read-only mode. They cannot trigger source refreshes.
• Write access (push): Users with write access can perform all read operations and additionally trigger source refreshes manually or via webhooks.
• Admin access: Users with admin access have full control over all sources and operations.

When a user without write access attempts to refresh a source (via API or webhook), they will receive an HTTP 403 Forbidden response. The web UI will display a "Read-only mode" indicator and hide the refresh button for users without write permissions.

Contributing

Install the pre-commit hooks:

pip install pre-commit
pre-commit install --allow-missing-config