Build and Import Puppet Modules into Katello

Posted on by

Building Puppet modules for Katello.

The Plan

We are going to:

  1. Install a few Puppet modules from Puppet Forge
  2. Generate a custom Puppet module
  3. Build Puppet modules
  4. Import Puppet modules into Katello
  5. Add Puppet modules to Git version control

Software

Software used in this article:

  1. CentOS 7
  2. Katello 3.5
  3. Puppet 4.x

Puppet Content Management on Katello

According to the latest stable Katello release documentation, there are two suggested ways to import Puppet modules into Katello:

  1. Importing from the Puppet Forge
  2. Importing Puppet modules from Git

Both have their use case, but aren’t something that fully meets our needs. We want to use Puppet Forge modules alongside the ones we will create ourselves, as well as have them all under version control. This would mean having a single Katello repository containing all our Puppet modules. A Katello repository may be a plain directory containing a Pulp manifest and packaged Puppet modules. According to the Pulp project documentation, the Pulp manifest is a file listing each Puppet module contained in the directory. Each module is listed on a separate line which has the following format: <name>,<checksum>,<size>. The name is the file name, the checksum is SHA256 digest of the file, and the size is the size of the file in bytes. The Pulp manifest must be named PULP_MANIFEST. Having all this information, we can build Puppet modules manually, generate a Pulp manifest and import everything into Katello.

Install Puppet Modules

We use the Katello server that we built some time ago.

Puppet Forge

For starters, we’re going to install several Puppet modules from the Puppet Forge:

# puppet module install -i /etc/puppet/modules puppetlabs-firewall ;\
 puppet module install -i /etc/puppet/modules puppetlabs-ntp ;\
 puppet module install -i /etc/puppet/modules erwbgy-limits ;\
 puppet module install -i /etc/puppet/modules thias-sysctl ;\
 puppet module install -i /etc/puppet/modules spiette-selinux ;\
 puppet module install -i /etc/puppet/modules Aethylred-postfix ;\
 puppet module install -i /etc/puppet/modules puppetlabs-apache ;\
 puppet module install -i /etc/puppet/modules puppetlabs-mysql

Let us verify:

# puppet module list
/etc/puppet/modules
├── Aethylred-postfix (v0.2.3)
├── erwbgy-limits (v0.3.1)
├── puppet-staging (v2.0.1)
├── puppetlabs-apache (v1.10.0)
├── puppetlabs-concat (v2.2.0)
├── puppetlabs-firewall (v1.8.1)
├── puppetlabs-mysql (v3.9.0)
├── puppetlabs-ntp (v4.2.0)
├── puppetlabs-stdlib (v4.12.0)
├── spiette-selinux (v0.5.4)
└── thias-sysctl (v1.0.6)
/usr/share/puppet/modules (no modules installed)

Generate a Custom Puppet Module

Change to the Puppet module directory:

# cd /etc/puppet/modules

We are going to create a new Puppet module that installs the rkhunter package. It’s a simple one. Generate a new Puppet module:

# puppet module generate lisenet-lsn_rkhunter
We need to create a metadata.json file for this module.  Please answer the
following questions; if the question is not applicable to this module, feel free
to leave it blank.

Puppet uses Semantic Versioning (semver.org) to version modules.
What version is this module?  [0.1.0]
--> 1.0.0

Who wrote this module?  [lisenet]
-->

What license does this module code fall under?  [Apache 2.0]
-->

How would you describe this module in a single sentence?
--> Installs rkhunter package

Where is this module's source code repository?
-->

Where can others go to learn more about this module?
-->

Where can others go to file issues about this module?
-->

----------------------------------------
{
  "name": "lisenet-lsn_rkhunter",
  "version": "1.0.0",
  "author": "lisenet",
  "summary": "Installs rkhunter package",
  "license": "Apache 2.0",
  "source": "",
  "project_page": null,
  "issues_url": null,
  "dependencies": [
    {
      "name": "puppetlabs-stdlib",
      "version_range": ">= 1.0.0"
    }
  ]
}
----------------------------------------

About to generate this metadata; continue? [n/Y]
-->

Notice: Generating module at /etc/puppet/modules/lisenet-lsn_rkhunter...
Notice: Populating ERB templates...
Finished; module generated in lisenet-lsn_rkhunter.
lisenet-lsn_rkhunter/README.md
lisenet-lsn_rkhunter/spec
lisenet-lsn_rkhunter/spec/spec_helper.rb
lisenet-lsn_rkhunter/spec/classes
lisenet-lsn_rkhunter/spec/classes/init_spec.rb
lisenet-lsn_rkhunter/manifests
lisenet-lsn_rkhunter/manifests/init.pp
lisenet-lsn_rkhunter/tests
lisenet-lsn_rkhunter/tests/init.pp
lisenet-lsn_rkhunter/Rakefile
lisenet-lsn_rkhunter/metadata.json

Open lisenet-lsn_rkhunter/metadata.json and remove the dependency bits so that it has the this:

"dependencies": []

Open lisenet-lsn_rkhunter/manifests/init.pp and put the following:

class lsn_rkhunter {
  package { "rkhunter": ensure => installed, }
}

Check the modules again, the new one should be listed:

# puppet module list
/etc/puppet/modules
├── Aethylred-postfix (v0.2.3)
├── erwbgy-limits (v0.3.1)
├── lisenet-lsn_rkhunter (v1.0.0)
├── puppet-staging (v2.0.1)
├── puppetlabs-apache (v1.10.0)
├── puppetlabs-concat (v2.2.0)
├── puppetlabs-firewall (v1.8.1)
├── puppetlabs-mysql (v3.9.0)
├── puppetlabs-ntp (v4.2.0)
├── puppetlabs-stdlib (v4.12.0)
├── spiette-selinux (v0.5.4)
└── thias-sysctl (v1.0.6)
/usr/share/puppet/modules (no modules installed)

We can now proceed forward and build them.

Build Puppet Modules for Katello

We have created a handy script to build local Puppet modules, which can then be imported into Katello. It’s available on https://github.com/lisenet/build-puppet-modules-for-katello, and is also shown below:

#!/bin/bash
# AUTHOR:   Tomas (www.lisenet.com)
# NAME:     puppet-module-build.sh
# VERSION:  1.0
# DATE:     15/09/2016 (dd/mm/yy)
# LICENCE:  Copyleft free software
#
# Where Puppet modules are located
MOD_DIR="/opt/puppet/modules";
MANIFEST="/opt/puppet/modules/PULP_MANIFEST";
MANIFEST_BACKUP="/opt/puppet/modules/PULP_MANIFEST.backup";

#
# Sanity checks
#

if [ ! -d "$MOD_DIR" ]; then
    echo "ERROR: directory "$MOD_DIR" does not exist.";
    exit 1;
fi
if ! type puppet >/dev/null 2>&1; then
    echo "ERROR: Puppet is not installed.";
    exit 1;
fi

# Backup the current manifest
mv -f "$MANIFEST" "$MANIFEST_BACKUP" 2>/dev/null;

# Puppet module array
MOD_ARRAY="$(ls -d "$MOD_DIR"/*/)";
for module in ${MOD_ARRAY[*]};do

    echo -e "\n$module";
    puppet module build "$module"

    if [ -d "$module"/pkg/ ]; then
        cd "$module"/pkg/;
        MOD_ARCHIVE="$(ls *tar.gz)";

        echo "Copying archive "$MOD_ARCHIVE".";
        cp -f "$MOD_ARCHIVE" ""$MOD_DIR"/";

        echo "Creating manifest "$MANIFEST" entry.";
        echo "$(sha256sum "$MOD_ARCHIVE"|awk '{print $2","$1}')","$(du -b "$MOD_ARCHIVE"|awk '{print $1}')" >>"$MANIFEST";
    else
        echo "ERROR: something went wrong while building '"$module"' module.";
    fi
done

Let us clone the repository:

# git clone https://github.com/lisenet/build-puppet-modules-for-katello.git ~/git

The below substitutes a pre-defined path of /opt/ to our path /etc/:

# sed -i 's/opt/etc/g' ~/git/puppet-module-build.sh

Time to build the modules:

# bash ~/git/puppet-module-build.sh

Verify:

# ls /etc/puppet/modules/
Aethylred-postfix-0.2.3.tar.gz     puppetlabs-concat-2.2.0.tar.gz
apache                             puppetlabs-firewall-1.8.1.tar.gz
concat                             puppetlabs-mysql-3.9.0.tar.gz
erwbgy-limits-0.3.1.tar.gz         puppetlabs-ntp-4.2.0.tar.gz
firewall                           puppetlabs-stdlib-4.12.0.tar.gz
limits                             puppet-staging-2.0.1.tar.gz
lisenet-lsn_rkhunter               selinux
lisenet-lsn_rkhunter-1.0.0.tar.gz  spiette-selinux-0.5.4.tar.gz
mysql                              staging
ntp                                stdlib
postfix                            sysctl
PULP_MANIFEST                      thias-sysctl-1.0.6.tar.gz
puppetlabs-apache-1.10.0.tar.gz

We see that the PULP_MANIFEST has been generated. We are now ready to import the modules into Katello.

Import Puppet Modules into Katello

We’re using Katello WebUI here, but feel free to use the Hammer CLI if you wish.

Katello Content

Katello can currently host two different types of content, RPMs and Puppet modules. We can have Products, and we can have Repositories. In short, a repository is a collection of content (either RPM or Puppet modules). A product is a collection of repositories. When we use content hosts, they subscribe to products.

Create a Katello Product

From the WebUI, navigate to: Content > Products > New Product (top right)

Name: PuppetModules
Label: PuppetModules

Save the new product.

Creating a Katello Repository

From the WebUI, navigate to: Content > Products > PuppetModules > Repositories > Create Repository

Name: puppet-modules
Label: puppet-modules
Type: puppet
URL: file:///etc/puppet/modules 
Publish via HTTP: yes

Save the new repository, and then sync it. We should see 12 Puppet modules available. That’s it, we have imported the modules into Katello.

Add Puppet Modules to Git Version Control

Our Puppet modules contain no private data, therefore we can use a public GitHub repository to store them. The repository URL is https://github.com/home-lab/katello-puppet, and we have an SSH key associated with the GitHub account. It’s a test account, not really used for anything else. Create an empty Git repository, add file contents to the index, record changes to the repository and update remote refs along with associated objects:

# cd /etc/puppet/modules
# echo "# katello-puppet" >> README.md
# git init
# git add README.md
# git commit -m "first commit"
# git remote add origin [email protected]:home-lab/katello-puppet.git
# git push -u origin master

Commit all Puppet modules to the remote repository:

# git add .
# git commit -m "adding Puppet modules
# git push origin master

Now we have Puppet modules under version control.

Leave a Reply

Your email address will not be published. Required fields are marked *