Ansible and third-party Python modules

A co-worker wanted a third-party Python modules installed onto the Ansible Tower servers that I maintain.  I don’t like installing any and all packages that people ask for since this is a shared system, but I had to bring myself up to speed on how he could get his module installed to use by his playbooks.

  • The machine that the Ansible playbook is executed from only needs to have Ansible and a set of pre-requisite Python modules installed on it.  For this document, we’ll refer to that system as the “Ansible Server” or just “the server”.
  • The machine(s) that the Ansible playbook make work on and perform changes to need ssh and a small sub-set of Python modules (usually the core Python packages).  We’ll refer to those systems as “the clients”.

To make things more confusing there are two types of “modules” that will be referenced:

  • An “Ansible module” is a package that Ansible uses on the server to execute steps on the clients.  These are usually written in Python and the “core” Ansible modules are included and maintained by the Ansible developers.
    • There are some Ansible modules that may be included with Ansible but they are maintained by the community.  These are usually specialized modules specific to a hardware or software vendor and are usually maintained by the vendor or others interested in automating that vendors tools.
  • The other “module” referenced in this document are add-ons to Python and are called “Python modules”.
    • A Python module may perform some low-level task (e.g. network connection, DNS lookup, etc) and are NOT Ansible specific.

Documentation for the Ansible modules are located here: https://docs.ansible.com/ansible/latest/modules/modules_by_category.html

The request mentioned the need for the “GitHub3.py” Python module so the “github_release” Ansible module would work.  The documentation for the “github_release” module has a requirements section and it notes that also.  The documentation page also notes that this is a preview module (“This module is not guaranteed to have a backwards compatible interface.”) and it is also maintained outside of the Ansible core develoers (“This module is maintained by the Ansible Community.”).

So, how do we add this module?  I’m glad you asked!

The first thing to understand is that all the requirements for this module have to be installed on the clients, not on the Ansible servers.  While this sounds like more work, it really isn’t and it keeps the Ansible servers free from conflicts that different users might have requiring different Python module versions.  The key to all this is the use of Python “Virtual Environments” (or “venvs”).  These virtual environments are walled-off areas that have their own Python executable and associated modules; it’s even possible to have different versions Python installed in different venvs for testing.

In the playbook that needs to use an Ansible module that has special Python module dependencies, there are a few steps to take that we’ll go over in detail below:

  1. Ensure pip is installed
  2. Install Python virtual environment packages into the venv
  3. Setup the virtual environment
  4. Install base Tower packages into venv
  5. Install the Python module specifically needed into venv
  6. Use the new venv to execute the Ansible module

Step 1 – Ensure pip is installed

This is a basic step and will vary by OS but the “pip” package is needed for the “pip:” modules later.

  - name: "Ensure pip is installed"
    yum:
      name: python2-pip
      state: installed

Step 2 – Install Python virtual environment packages

This is also OS dependent, but it installs the “python-virtualenv” package so Python can build venvs.


  - name: "Install Python virtual environment packages"
    yum:
      name:
        - python-virtualenv
      state: installed

Step 3 – Setup the virtual environment

This step does the initial work to build the virtual environment.  The venv is just a directory structure (in this case “/tmp/test01.venv”) that contains helper files and some wrapper scripts and configuration defaults.

  - name: "Setup the initial virtual environment"
    pip:
      name:
        - setuptools
      extra_args: --upgrade
      virtualenv: "/tmp/test01.venv"

Step 4 – Install base Tower packages into venv

Strictly speaking, if you’re not using this venv with Ansible Tower it is not necessary, but it will make the playbook usable in more places.


  - name: "Install base Tower packages"
    pip:
      name:
        - python-memcached
        - psutil
      umask: "0022"
      state: present
      virtualenv: "/tmp/test01.venv"
      virtualenv_site_packages: yes
      extra_args: --ignore-installed

Step 5 – Install the Python module specifically needed into venv

Finally we’re at the step where we’re installing the Python module we need.  This Python module (like the others earlier) are only installed into the venv directory structure.

  - name: "Install github3.py module"
    pip:
      name:
        - github3.py
      virtualenv: "/tmp/test01.venv"

Step 6 – Put the new venv with the Python module to work

The key at this step is the “vars:” section that tells the Ansible execution environment to use the “python” binary found in the “venv” on the remote system, “/tmp/test01.venv/bin/python” in this case.

  - name: "Download latest relase of ISOwithKS"
    # https://github.com/dglinder/ISOwithKS.git
    vars:
      ansible_python_interpreter: "/tmp/test01.venv/bin/python"
    github_release:
      user: "dglinder"
      repo: "ISOwithKS"
      action: latest_release

PLEASE NOTE: The “github_release:” example above does NOT work due to something un-related to the venv created.

How does this work?

When the playbook runs it connects to all of the clients and makes sure the “python2-pip” and “python-virtualevnv” packages are installed, it then builds the bare virtual environment into “/tmp/test01.venv/” and populates that venv with additional modules, then installs the Python modules necessary to execute the Ansible module.  The Ansible module is executed using the “python” executable in the newly built venv.

Note that ALL of these steps are preformed on the Ansible clients, no changes are made to the Ansible server.  In testing, the initial execution of these steps took about 40-50 seconds to get to the final step – most of that time was due to downloading packages from the Pip repository (on the Internet).  Subsequent runs that were able to re-use the venv directory took 20-25 seconds to get to the same location.

Caveats

One big shortcoming of this process is the necessity of the Ansible clients to have access to download the packages from an Internet location.  If the clients are shielded from the Internet, it may be necessary to setup a proxy server they can use (if permitted).

It might be necessary to perform the venv build on a single server with Internet access, then replicate that venv directory structure to each of the clients.  (These workarounds have not been validated, so test and report back any success or failures.)

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.