Automating SSH keys with Ansible

I was working with a co-worker to get his SSH key distributed to a large number of systems so we could start managing them with Ansible. For security and logging purposes, we login to the systems with our personal “elevated” account (different from our workstation account), and the elevated accounts are permitted to run “sudo” so their Ansible playbooks can make changes to the system.

Today we were fighting with getting the SSH key generated and distributed to all the systems. I’ve been working in Unix and using SSH for over 20 years, but today we kept making simple typos that kept the keys from working for us. Some of the issues were human error and typos on the command line, others were more subtle involving permissions issues on the files. To help remove this human error, I wrote a two-step playbook to generate an ssh keypair (if they don’t exist), then login to a set of systems and setup that key to permit password-less logins.

Here’s the playbook – named “copykeys.yml” – built to automate the distribution of SSH keys across your environment:

 #!/usr/bin/env ansible-playbook
 ---
 - name: "Create and upload SSH keys"
   hosts: all
   gather_facts: false

   tasks:
   - name: "Ensure ssh key exists"
     openssh_keypair:
       path: "{{ lookup('env','HOME') + '/.ssh/id_ed25519' }}"
       type: ed25519
     delegate_to: localhost
     run_once: true

   - name: "Copy user ssh keys"
     authorized_key:
       user: "{{ lookup('env', 'USER') }}"
       state: present
       key: "{{ lookup('file', lookup('env','HOME') + '/.ssh/id_ed25519.pub') }}" 

This playbook first creates an “ed25519” SSH key if it does not exist on the machine the script is run on (the “delegate_to” argument), and once the SSH key is created the public key is distributed to each of the systems the playbook runs against.

To use this, simply call the playbook with either an inventory file (‘-i inventory.ini’) or with a command-line inventory list (‘-i server1,server2,server3’). Here is the command to run:

ansible-playbook copykeys.yml --ask-pass -i server1,server2,server3 

Assuming your username and password are accepted by all the servers in the inventory, the public portion of your SSH key will be installed on each of the machines.

You can then test this by performing an Ansible “ping” to validate communication works without the “–ask-pass” option:

ansible all -m ping -i server1,server2,server3

The nice thing about Ansible is that it is easy to debug if you break down the communications and realize that all communications between nodes is using basic SSH. If you’re having communication problems, a simple “ssh servername” from the server you’re running Ansible from will often show if the error is communications based.

If this does not work, review the output that Ansible provides (you might have to add “-vvv” to increase the verbosity level for debugging). In nearly all cases the errors are common SSH error messages, not Ansible error messages. Any modern Unix administrator has probably encountered these errors before and is aware of what the underlying issue is that is breaking the SSH connectivity.

Windows 10, VMware Workstation 15.5, and “Device/Credential Guard”

So I’m trying to create some documentation around our VMware template build process and I’m starting from a freshly built Windows 10 system with all patches as of December 2019. Our process uses Hashicorp Packer to automate the build of our Red Hat VM templates using VMware Workstation 15.5 running a Bash script under Windows Subsystem for Linux (WSL). It is a bit more complex than I wanted, but ultimately we want this to run in a fully automated process on a Linux build server so WSL is a good bridge.

Today when I was going through our pre-requisite setup steps, my fresh VMware installation would throw an error pop-up message:

Googling for an answer, I found a lot of one-off solutions but none of them worked. After searching for most of the day, I came across this YouTube video by Britec09 : https://youtu.be/VIBdY-5zr58

I usually don’t sit through YouTube videos for issues such as these, but this time I was getting desperate for an answer.

And Mr. Britec09 came through!

To cut to the chase, the steps he provided were all similar or identical to others I had found, except his ran in this order and all at once seemed to be my solution.

To save time if this happens to me again, here are the steps as he included them in his YouTube notes.

NOTE: Please be careful – many of these tools used below will permit you to accidentally break your system.

Step 1. – In the search box type “gpedit” then Goto Computer Configuration Administrative Templates System Device Guard Turn on Virtualization Based Security.
Double click that option and choose “Disable”

Step 2. In search box, type Turn Windows features on or off, then uncheck Hyper-V and restart system.

Step 3. Open Registry Editor by typing regedit in the search box. Go to HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\DeviceGuard. Right-click in the right panel, add a new DWORD value named EnableVirtualizationBasedSecurity and set it to 0 to disable it.
Next Go to HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\LSA. Right-click and add a new DWORD value named LsaCfgFlags and set it to 0 to disable it.

Step 4. Open command prompt as a administrator and type the following commands

bcdedit /create {0cb3b571-2f2e-4343-a879-d86a476d7215} /d "DebugTool"  /application osloader

Then copy paste the rest below and press enter – there are four lines of commands, each beginning with “bcedit”.

bcdedit /set {0cb3b571-2f2e-4343-a879-d86a476d7215} path  "\EFI\Microsoft\Boot\SecConfig.efi"

bcdedit /set {bootmgr} bootsequence  {0cb3b571-2f2e-4343-a879-d86a476d7215}

bcdedit /set {0cb3b571-2f2e-4343-a879-d86a476d7215} loadoptions  DISABLE-LSA-ISO,DISABLE-VBS

bcdedit /set hypervisorlaunchtype off 

Step 5. If all of that worked without any errors, reboot your system and re-try VMware Workstation.

And so it ends…

No, I’m not giving up on my weekly updates. Have a little more faith in me…don’t expect me to give up until at least February. 🙂

What the title alludes to is the fact that this upcoming week is my last week at Intrado (previously West Corporation). It’s been a personally interesting ride to say the least.

I started as an architect, but soon found myself in an engineering role. I excelled in that role in part because I continued to apply the architecture mindset and applied that to the engineering designs I produced. I’m proud to say that many of the designs I contributed to are continuing to provide value to my co-workers.

Thinking back I feel that my personal growth while working in this position was equally half technical and half business/political. Early on, a lot of the discussions I had with directors and co-workers showed me that the human component of engineering was critical for projects to succeed. A solid technical design will never succeed if there is a well-placed individual who has their own plans. Teaming up with these individuals opened my eyes to their point of view and gave me a chance to advocate for my positions while finding a “happy medium”.

I’m hoping I’m leaving Intrado a better company. The past three years have been rough on all my co-workers, but I truly think that Intrado is finally turning the corner. Some have asked why I’m still leaving if I believe it is getting better. While I am comfortable that my position and employment is safe within Intrado, I feel that I’ve reached the end of the benefit I can provide. I’ve been the “automation advocate” and championing the “Digital Transformation” process, but there is only so much one person can do. I’m hoping my departure will rock the boat enough that multiple others will rise to the challenge and provider a larger group of experts.

And I realized that for the past 18 years I’ve had a 35 minute commute to work (either Intrado or Maryville) – excluding the ~3 years I worked from home and rounding to two 30-minute commute sessions each workday, I have spent a total of 150 days (over 3700 hours) sitting in my car for work days. I’m looking forward to my next employers office which is roughly 12 minutes from my home, even in inclement weather.

So, what else besides my job?

This week I started a number of conversations helping my parents sell their house and get completely moved into the assisted living apartment. We’re starting to work with the estate sales team while we work on some known things to fix up. We know we have some electrical that we’ll have to clean up before putting the house on the market, but some of that are things that Dad and I can do ourselves and we’ll leave the big work to the electrician.

As a late Christmas gift to my parents, we’re paying for a year of the Cox DVR service for them. Many of the shows that Mom loves are (in)conveniently timed at the same time or when they are at appointments. Now they can keep up with their shows as they have been used to when they had DirectTV.

Kris is still under the weather, but she did take herself to the Urgent Care office today and they have her on an antibiotic – I’m hoping she continues to improve, I’m not use to seeing her this sick.

Jilli and her friends headed back to college earlier this week. They left early to miss the snows coming in, but she ended up leaving some fairly important items (her winter coat!) so we have to get them shipped to her soon.

Lizzy took a placement exam for High School Saturday- she came out tired but in good spirits so I’m hoping she did well. To celebrate her “survival” we went out for lunch at Raising Caines – always a good choice IMHO too!

Finally, thanks to everyone I have worked with at Intrado. I wish you all the best and I hope you’ll keep in touch.

Dual use laptop…

Tails – The Amnesic Incognito Live System – is a live operating system that you can start on almost any computer from a USB stick or a DVD. Website: https://tails.boum.org/

A couple years ago my personal laptop was dying, so when I got to work early and sat at the local coffee shop I tried to use my companies laptop. Unfortunately, some of the sites I liked to read were considered “hacker” sites and my companies software blocked that. I had use bootable USB drives running various versions of Linux, but many of them would automatically try to use the local drive and possibly write data to the hard drive. I then came across Tails.

What got me interested in Tails was the fact that it fit on a small USB drive, and everything that you saved to that drive (and only that drive) was encrypted automatically. The Internet communications was also equally encrypted; it uses the TOR network (https://www.torproject.org/) along with a TOR-enabled Firefox browser to hide the traffic from prying eyes on the local network. With all this together, I could install Tails to a small (8GB or 16GB USB stick) and boot my companies laptop from there and not worry about my personal data being saved, or from my company laptop snooping on my private emails and communications.

The Tails system will let you connect to your coffee shop WiFi, but it won’t let you start browsing the web (or logging into your bank or email site) until it has fully connected into the TOR network. The TOR network encrypts all your communications through multiple computers on the Internet – many times the last computer (the “exit node”) is in a different country. (This makes for a wild multi-lingual experience trying to navigate a popular site if they attempt to use your “local” language.)

With Tails I could now use my work laptop but boot into Tails and securely take care of personal tasks (banking, email, medical, etc) without worrying about my companies computer or their filters blocking or breaking some of these sites.

Give Tails a shot if you want to have a simple to use and very secure web browsing experience that won’t mess with your regularly installed operating system.

My resolution

So I promised myself that I would start updating my blog on a weekly basis. Nothing major, no earth-shaking thoughts or deep introspective writings, just a basic update to get the habit started.

At least that is the plan…

For now, I’ll start out simply by noting that my **plan** is to add this every Sunday night, and of course this first one is being written Monday. 🙂 I won’t cop-out and schedule this for next Sunday, and be transparent with my failings, too.

So, we’re continuing to help clean up our parents house getting it ready to sell. Kate has been going out there nearly every weekend (quite often both days), taking Dad and Nick with her and occasionally getting a couple friends to help. To her I tip my hat with a big note of thanks. I’ve tried to keep up with her on those weekends, but have had to beg forgiveness and sit out a few times. And of course Amanda has been doing what she can remotely – she’s our backup for both Kate and I. Kate is helping navigate all the various health-related events with Mom and Dad, and I’m taking on keeping their bills paid and going through all the paperwork and either keeping the important pieces, or getting the remaining stuff to a shredder.

This past weekend we hit the garage attic and got it cleaned out. Many of those boxes had been packed and never opened when they moved from Columbus. Dad was proactive and put labels on each box, then noted in a document what was in each box. Unfortunately, the boxes and the document are both 20 years old, and we’re unable to find the list anymore. It was fun to open them and dive into the contents, but at the same time it was borderline overwhelming. Some of the findings did bring back some happy memories; Jilli found a small bottle of gemstones. They were (we presume) some natural garnets that her Great Grandfather Linder (William Linder, father of Gary) found when he, Dad, and I were hiking one summer in the Rocky Mountains (either near Estes Park, CO or in the Medicine Bow, Wyoming area). He suspected they were garnet, and since that is my gemstone he gave me the bottle of them to keep. Jilli might take them back to SDSM and see if one of the mineralogy professors might be able to determine what they are.

What this work has impressed upon me is to clean out my office a bit. I really doubt that anyone will ever crack open the “Windows 95 Unleashed” book, let alone many of the other related books. If I really get an urge to read it, I’ll find a PDF version but for now the paper version can go into the recycling. (I might donate it, but I don’t think Salvation Army nor Goodwill would have a need for them either.)

A good friend of ours gave us a DeeBot robotic vacuum cleaner for a family Christmas gift. It is amazing what volume of pet hair it picks up, as well as bits of paper, small stones, misc cat/dog food, etc. Our main floor is mostly wood, with only a small bit of carpet and it doesn’t have a problem with either location. What it does have a problem with is going under the couch and chairs. It just barely fits, but occasionally it gets stuck and can’t make it out. I’m brainstorming on how to add a small “guard rail” underneath the furniture, or the alternative is to add little 3/4″ risers under each leg. Since I’m the one with the long legs, I don’t think anyone else wants the chair raised so the guard rail is probably the best option.

Git branching basic workflow

To better handle multiple people working on a project in a Git repository, using branches and reviewing pull requests before committing to the master branch is strongly suggested.


Basic steps:

  1. Change to your home directory and clone the git repository
    1. cd $HOME
    2. git clone [email protected]/MyRepo.git
  2. Change into the new project directory
    1. cd ./MyRepo
  3. Create a branch to work on the new code
    1. git checkout -b MyNewBranch
  4. Verify you are working in the branch
    1. git branch
    2. Note: The branch will have a “*” to the left of the branch name denoting the active branch
  5. Update code, test, repeat
  6. Review and add any missing files
    1. git status
    2. git add <file_name>
  7. Push the code into the repository
    1. git push –set-upstream origin MyNewBranch
      1. This is only necessary for the first ‘git push’
    2. git commit -v {list of changed files}
      • Note the response from the system:
      • remote: Create a pull request for 'MyNewBranch' on GitHub by visiting:
      • remote: https://git.repo.example.org/MyRepo/pull/new/MyNewBranch
  8. Open the pull request (PR) in git.repo.example.org
    1. Add other repository contributors to request a code review before merging.
  9. Repeat the edit/test/PR cycle as necessary until merge is accepted
    1. edit code … test … git status … git add … git commit … git push
  10. When it has been accepted, clean-up your work area:
    1. cd $HOME/MyRepo/
    2. git checkout master
    3. git pull
    4. git branch –delete MyNewBranch
  11. Celebrate on a successful pull request!

IBM Model-M and the Windows Key

I really love my IBM Model-M keyboard, but one frustration is the more frequent need for the Windows key. Even some Linux desktops are using it. Oh I get it, it’s a handy “meta” key that helps differentiate keyboard tasks so it’s lack is annoying at times.

For the few times I am in Windows 10 using my Model-M keyboard, I found this answer on SuperUser.com to re-map the caps-lock key to the Windows key with a simple registry hack: https://superuser.com/a/1228990/101577

Just in case that link goes away, here’s the text:

Anyway, using SharpKeys I found the correct map for CAPS LOCK to Win is this:

https://superuser.com/a/1228990/101577
Windows Registry Editor Version 5.00
[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Keyboard Layout]
"Scancode Map"=hex:00,00,00,00,00,00,00,00,02,00,00,00,5b,e0,3a,00,00,00,00,00

Just save those lines as presented into a file, “c:\WinKeyRemap.reg”, then use File Manager to find it, then double-click to open it with the Registry Editor. (You’ll need to accept the warning.) Assuming the file is correct, Registry Editor will report that the values have been added to the registry.

Reboot to make the changes take effect.

Ansible and third-party Python modules

A co-worker wanted a third-party Python modules installed onto the Ansible Tower servers that I maintain.  I don’t like installing any and all packages that people ask for since this is a shared system, but I had to bring myself up to speed on how he could get his module installed to use by his playbooks.

  • The machine that the Ansible playbook is executed from only needs to have Ansible and a set of pre-requisite Python modules installed on it.  For this document, we’ll refer to that system as the “Ansible Server” or just “the server”.
  • The machine(s) that the Ansible playbook make work on and perform changes to need ssh and a small sub-set of Python modules (usually the core Python packages).  We’ll refer to those systems as “the clients”.

To make things more confusing there are two types of “modules” that will be referenced:

  • An “Ansible module” is a package that Ansible uses on the server to execute steps on the clients.  These are usually written in Python and the “core” Ansible modules are included and maintained by the Ansible developers.
    • There are some Ansible modules that may be included with Ansible but they are maintained by the community.  These are usually specialized modules specific to a hardware or software vendor and are usually maintained by the vendor or others interested in automating that vendors tools.
  • The other “module” referenced in this document are add-ons to Python and are called “Python modules”.
    • A Python module may perform some low-level task (e.g. network connection, DNS lookup, etc) and are NOT Ansible specific.

Documentation for the Ansible modules are located here: https://docs.ansible.com/ansible/latest/modules/modules_by_category.html

The request mentioned the need for the “GitHub3.py” Python module so the “github_release” Ansible module would work.  The documentation for the “github_release” module has a requirements section and it notes that also.  The documentation page also notes that this is a preview module (“This module is not guaranteed to have a backwards compatible interface.”) and it is also maintained outside of the Ansible core develoers (“This module is maintained by the Ansible Community.”).

So, how do we add this module?  I’m glad you asked!

The first thing to understand is that all the requirements for this module have to be installed on the clients, not on the Ansible servers.  While this sounds like more work, it really isn’t and it keeps the Ansible servers free from conflicts that different users might have requiring different Python module versions.  The key to all this is the use of Python “Virtual Environments” (or “venvs”).  These virtual environments are walled-off areas that have their own Python executable and associated modules; it’s even possible to have different versions Python installed in different venvs for testing.

In the playbook that needs to use an Ansible module that has special Python module dependencies, there are a few steps to take that we’ll go over in detail below:

  1. Ensure pip is installed
  2. Install Python virtual environment packages into the venv
  3. Setup the virtual environment
  4. Install base Tower packages into venv
  5. Install the Python module specifically needed into venv
  6. Use the new venv to execute the Ansible module

Step 1 – Ensure pip is installed

This is a basic step and will vary by OS but the “pip” package is needed for the “pip:” modules later.

  - name: "Ensure pip is installed"
    yum:
      name: python2-pip
      state: installed

Step 2 – Install Python virtual environment packages

This is also OS dependent, but it installs the “python-virtualenv” package so Python can build venvs.


  - name: "Install Python virtual environment packages"
    yum:
      name:
        - python-virtualenv
      state: installed

Step 3 – Setup the virtual environment

This step does the initial work to build the virtual environment.  The venv is just a directory structure (in this case “/tmp/test01.venv”) that contains helper files and some wrapper scripts and configuration defaults.

  - name: "Setup the initial virtual environment"
    pip:
      name:
        - setuptools
      extra_args: --upgrade
      virtualenv: "/tmp/test01.venv"

Step 4 – Install base Tower packages into venv

Strictly speaking, if you’re not using this venv with Ansible Tower it is not necessary, but it will make the playbook usable in more places.


  - name: "Install base Tower packages"
    pip:
      name:
        - python-memcached
        - psutil
      umask: "0022"
      state: present
      virtualenv: "/tmp/test01.venv"
      virtualenv_site_packages: yes
      extra_args: --ignore-installed

Step 5 – Install the Python module specifically needed into venv

Finally we’re at the step where we’re installing the Python module we need.  This Python module (like the others earlier) are only installed into the venv directory structure.

  - name: "Install github3.py module"
    pip:
      name:
        - github3.py
      virtualenv: "/tmp/test01.venv"

Step 6 – Put the new venv with the Python module to work

The key at this step is the “vars:” section that tells the Ansible execution environment to use the “python” binary found in the “venv” on the remote system, “/tmp/test01.venv/bin/python” in this case.

  - name: "Download latest relase of ISOwithKS"
    # https://github.com/dglinder/ISOwithKS.git
    vars:
      ansible_python_interpreter: "/tmp/test01.venv/bin/python"
    github_release:
      user: "dglinder"
      repo: "ISOwithKS"
      action: latest_release

PLEASE NOTE: The “github_release:” example above does NOT work due to something un-related to the venv created.

How does this work?

When the playbook runs it connects to all of the clients and makes sure the “python2-pip” and “python-virtualevnv” packages are installed, it then builds the bare virtual environment into “/tmp/test01.venv/” and populates that venv with additional modules, then installs the Python modules necessary to execute the Ansible module.  The Ansible module is executed using the “python” executable in the newly built venv.

Note that ALL of these steps are preformed on the Ansible clients, no changes are made to the Ansible server.  In testing, the initial execution of these steps took about 40-50 seconds to get to the final step – most of that time was due to downloading packages from the Pip repository (on the Internet).  Subsequent runs that were able to re-use the venv directory took 20-25 seconds to get to the same location.

Caveats

One big shortcoming of this process is the necessity of the Ansible clients to have access to download the packages from an Internet location.  If the clients are shielded from the Internet, it may be necessary to setup a proxy server they can use (if permitted).

It might be necessary to perform the venv build on a single server with Internet access, then replicate that venv directory structure to each of the clients.  (These workarounds have not been validated, so test and report back any success or failures.)

Docker on Windows Subsystem for Linux to play with RedHat 8

Ok, so this is kind of long but neat too!

A co-worker asked about using a Docker image for a project he’s working on and I suggested that he use the RedHat 7/8 based “Universal Base Image” that they announced at Summit. (Our company has a large installed base of RedHat, so there is a big advantage being to tap into that internal knowledge.)

–> https://www.redhat.com/en/blog/introducing-red-hat-universal-base-image

If you have a machine with Docker setup, then doing a pull of “registry.access.redhat.com/ubi8/ubi:latest” will pull down the RHEL-8 version.

–> $ docker run –rm -it registry.access.redhat.com/ubi8/ubi:latest /bin/bash

But I don’t have a Docker system, I only have Windows 10!” No fear, you can install Docker on Windows:

–> https://docs.docker.com/docker-for-windows/install/

From there you can kick off Docker from PowerShell or the command prompt with the exact same command prompt as shown above.

But I want to do this in a Linux environment on my Windows workstation!”  Use the “Windows Subsystem for Linux” feature of Windows 10:

–> https://medium.com/@sebagomez/installing-the-docker-client-on-ubuntus-windows-subsystem-for-linux-612b392a44c4

Here’s a screen shot of a RHEL-8 container running under WSL showing that “yum install …” works as expected:

And here it is running under PowerShell:

When is a disk space problem not a disk space problem?

A co-worker setup an Ansible playbook to update some packges but it kept erroring out. The error that Ansible reported from “yum” was “No space left on device“. He had jumped onto the system and saw that this partition had plenty of space left so asked if I could look into it.

I got on and confirmed that when I ran a simple “yum update” it showed this:

[[email protected] ~]# echo n | yum update

Loaded plugins: product-id, rhnplugin, search-disabled-repos, security, subscription-manager

[Errno 28] No space left on device: ‘/var/run/rhsm/cert.pid’

This system is receiving updates from RHN Classic or RHN Satellite.

Could not create lock at /var/run/yum.pid: [Errno 28] No space left on device: ‘/var/run/yum.pid’

Hmm, no disk space still. Looking at the “df /var” output looks good:

[[email protected] ~]# df /var

Filesystem           1K-blocks   Used Available Use% Mounted on

/dev/mapper/rootvg-varlv

                       2514736 914948   1468716  39% /var

Suspecting other resource issues I checked the inode availability using “df -i:

[[email protected] ~]# df -i /var

Filesystem           Inodes  IUsed IFree IUse% Mounted on

/dev/mapper/rootvg-varlv

                     163840 163840     0  100% /var

A ha! No inodes left. I’ll let you use your favorite search engine to look up details, but an easy way to think of “inodes” is as space on the first few pages of a book dedicated to being the “table of contents.” If you have a book with a few chapters, you only need a single page for the table of contents (the inodes). If you have a book with lots of chapters and sub-chapters, you might need a lot of pages (more inodes). By default Unix systems have a forumla on how much of the filesystem to dedicate to being “inodes” and how much is left for actual data storage. Usually this is fine for most systems.

To find them we want to look for directories which have chewed up the 163K files:

for i in /var/*; do echo $i; find $i |wc -l; done

This pointed to the “/var/spool/app01/” directory – it has over 160K small files.  The owner of the system was able to clean up some old files there and the “yum update” worked as expected.

It’s possible to override the inode settings when the filesystem is formatted, so if you know this ahead a time you can do this. If you run into this after the fact, the usual resolution is to backup the data, reformat the filesystem with more inodes allocated, then restore from backup.