Buildup

Thanks to “https://www.reddit.com/user/MaricxX/” for this photo – it demonstrates how small glitches over time can add up if they aren’t addressed rapidly – or better yet, not allowed to start in the first place.

Cross section of layers of paint showing deformation due to imperfections magnified with each layer.
Layers of paint – credit to MaricxX from Reddit – https://www.reddit.com/user/MaricxX/

At a previous job it was common to take our Windows virtual machine templates and power them on once a month to patch the OS and apply the latest security configurations. We had been doing this with our Red Hat Linux images, but a couple years ago I converted our process so each month we built those VM templates fresh from an ISO and a Hashicorp Packer script using VMware Workstation.

This monthly fresh build ensured that we always knew how to build the VM templates in the event of a disaster, and it ensured that our build process contained exactly what we planned and advertised (through our team Git repository). As new requirements were received from the InfoSec team or other sources with system concerns that could only be readily addressed during the initial build phase, we would add those steps to the Packer config file, then test and build new.

With the prevalence of new worms and other highly effective infection vectors, my fear was that we would get a piece of malware onto the templates and then that malware would be automatically replicated each time a new system was built. And there were many times when we started the patching process each month only to find that a couple of the Windows templates had been left running since the previous months patch effort. There is no telling what might have crawled onto these unmanaged systems in the intervening time, only waiting for us to start using them over time.

While the paint analogy doesn’t perfectly match with the IT world, there are sufficient correlations that it makes the possibility of replicating and amplifying a small defect all the more understandable. Still, I prefer to have my freshly-built template with it’s minimal layers of paint knowing that I am confident that it only contains the bits we wanted.

Slipping…

So my weekly blog post updates slipped a bit this week. So, like riding a bike I gotta keep getting back up and trying – soon I’ll get into this habit.

The past weekend was more cleaning up my parents house for an estate sale. We met with a couple of estate sales people, just waiting on their responses. I’m sure that one of them will say the house contents is worth well over $1M. 🙂

I’m trying to consolidate a lot of old computer hard drives so we don’t lose any important documents Mom or Dad might have worked on, but it’s getting pretty challenging to sort them on the fly. It would be great if there was a program that would scan the files and only copy them if they are not duplicates, and use the directory structure to inform the copy/backup process to sort them somewhat intelligently. If anyone has any ideas or software gems to point me to, pass it along in the comments.

Kris went up to visit Jilli over the weekend and had a good time. Saw Faith and John too, plus she got some pet time in with their pets. The flight from Omaha to Rapid City wasn’t cheap, but it wasn’t as bad as I was expecting when we bought it. A friend of mine says that Frontier Airlines has Omaha to Denver for less than $100 regularly. Now if we can find a Denver to Rapid City flight for about that, then it would make the trip pretty reasonable.

$RANDOM

Friday was my last day – and the weather was poor enough (snow with freezing rain) that the company sent an email the day before telling people to work from home if they could. I am glad I worked from home – I think I was able to get a lot of documentation wrapped up and some last-minute things completed and handed off. Even if I had another two weeks, I still wouldn’t have handed things off properly. There would always be one more thing to work on, one more thing to clean up, one more thing to polish. And the kicker was that I wasn’t truly handing things off as much as throwing documentation and notes into README.md files and Wiki pages and hoping someone at a future date would find them and keep the ball moving forward. But, all things come to and end – I’m looking forward to my new job starting this week and I wonder what sort of things I’ll get into next. 😀

Earlier this week Jilli sent Kris a text telling her that a Mountain Lion was roaming campus. Students were to call 911 immediately if they saw it. I was concerned that her first reaction would be to call “Here kitty, kitty!” and try to pet it. My next vision was Jilli and her friends running away from the lion, each with their phones in their hands Googling “How to escape a Mountain Lion”…

Liz had a normal week at school. She and Kris spent a lot of time together since I had a lot of late nights wrapping up work and helping with my parents. She’s continuing to use her weight training bag in the basement, plus she’s starting to cook more and more. Ready to bake Croissant Rolls are being made frequently, as are Chocolate Chip cookies. I’ve eaten way too many of both this week – my post-Christmas weight loss isn’t working.

The cold/crud that I brought home over Christmas has left me, but is continuing to annoy Kris. She was just starting to get over the worst of the coughing when she hurt a muscle in her back coughing so much. She was in a lot of pain after school on Friday – she says even sleeping is painful sleeping on her back and putting pressure on the muscle. I really need to talk to her mom about her bodies warranty coverage…

Mom and Dad both continue to kick around AV. It was so cold and icy Saturday morning that we decided not to go out to their house so I rescheduled the home inspection for another week. We met with a new financial advisor this week, but I keep hoping we stumble across some gold bars or a hand full of un-sold “Berkshire A” stock certificates. Probably not likely, but I can keep my fingers crossed.

Automating SSH keys with Ansible

I was working with a co-worker to get his SSH key distributed to a large number of systems so we could start managing them with Ansible. For security and logging purposes, we login to the systems with our personal “elevated” account (different from our workstation account), and the elevated accounts are permitted to run “sudo” so their Ansible playbooks can make changes to the system.

Today we were fighting with getting the SSH key generated and distributed to all the systems. I’ve been working in Unix and using SSH for over 20 years, but today we kept making simple typos that kept the keys from working for us. Some of the issues were human error and typos on the command line, others were more subtle involving permissions issues on the files. To help remove this human error, I wrote a two-step playbook to generate an ssh keypair (if they don’t exist), then login to a set of systems and setup that key to permit password-less logins.

Here’s the playbook – named “copykeys.yml” – built to automate the distribution of SSH keys across your environment:

 #!/usr/bin/env ansible-playbook
 ---
 - name: "Create and upload SSH keys"
   hosts: all
   gather_facts: false

   tasks:
   - name: "Ensure ssh key exists"
     openssh_keypair:
       path: "{{ lookup('env','HOME') + '/.ssh/id_ed25519' }}"
       type: ed25519
     delegate_to: localhost
     run_once: true

   - name: "Copy user ssh keys"
     authorized_key:
       user: "{{ lookup('env', 'USER') }}"
       state: present
       key: "{{ lookup('file', lookup('env','HOME') + '/.ssh/id_ed25519.pub') }}" 

This playbook first creates an “ed25519” SSH key if it does not exist on the machine the script is run on (the “delegate_to” argument), and once the SSH key is created the public key is distributed to each of the systems the playbook runs against.

To use this, simply call the playbook with either an inventory file (‘-i inventory.ini’) or with a command-line inventory list (‘-i server1,server2,server3’). Here is the command to run:

ansible-playbook copykeys.yml --ask-pass -i server1,server2,server3 

Assuming your username and password are accepted by all the servers in the inventory, the public portion of your SSH key will be installed on each of the machines.

You can then test this by performing an Ansible “ping” to validate communication works without the “–ask-pass” option:

ansible all -m ping -i server1,server2,server3

The nice thing about Ansible is that it is easy to debug if you break down the communications and realize that all communications between nodes is using basic SSH. If you’re having communication problems, a simple “ssh servername” from the server you’re running Ansible from will often show if the error is communications based.

If this does not work, review the output that Ansible provides (you might have to add “-vvv” to increase the verbosity level for debugging). In nearly all cases the errors are common SSH error messages, not Ansible error messages. Any modern Unix administrator has probably encountered these errors before and is aware of what the underlying issue is that is breaking the SSH connectivity.

Windows 10, VMware Workstation 15.5, and “Device/Credential Guard”

So I’m trying to create some documentation around our VMware template build process and I’m starting from a freshly built Windows 10 system with all patches as of December 2019. Our process uses Hashicorp Packer to automate the build of our Red Hat VM templates using VMware Workstation 15.5 running a Bash script under Windows Subsystem for Linux (WSL). It is a bit more complex than I wanted, but ultimately we want this to run in a fully automated process on a Linux build server so WSL is a good bridge.

Today when I was going through our pre-requisite setup steps, my fresh VMware installation would throw an error pop-up message:

Googling for an answer, I found a lot of one-off solutions but none of them worked. After searching for most of the day, I came across this YouTube video by Britec09 : https://youtu.be/VIBdY-5zr58

I usually don’t sit through YouTube videos for issues such as these, but this time I was getting desperate for an answer.

And Mr. Britec09 came through!

To cut to the chase, the steps he provided were all similar or identical to others I had found, except his ran in this order and all at once seemed to be my solution.

To save time if this happens to me again, here are the steps as he included them in his YouTube notes.

NOTE: Please be careful – many of these tools used below will permit you to accidentally break your system.

Step 1. – In the search box type “gpedit” then Goto Computer Configuration Administrative Templates System Device Guard Turn on Virtualization Based Security.
Double click that option and choose “Disable”

Step 2. In search box, type Turn Windows features on or off, then uncheck Hyper-V and restart system.

Step 3. Open Registry Editor by typing regedit in the search box. Go to HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\DeviceGuard. Right-click in the right panel, add a new DWORD value named EnableVirtualizationBasedSecurity and set it to 0 to disable it.
Next Go to HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\LSA. Right-click and add a new DWORD value named LsaCfgFlags and set it to 0 to disable it.

Step 4. Open command prompt as a administrator and type the following commands

bcdedit /create {0cb3b571-2f2e-4343-a879-d86a476d7215} /d "DebugTool"  /application osloader

Then copy paste the rest below and press enter – there are four lines of commands, each beginning with “bcedit”.

bcdedit /set {0cb3b571-2f2e-4343-a879-d86a476d7215} path  "\EFI\Microsoft\Boot\SecConfig.efi"

bcdedit /set {bootmgr} bootsequence  {0cb3b571-2f2e-4343-a879-d86a476d7215}

bcdedit /set {0cb3b571-2f2e-4343-a879-d86a476d7215} loadoptions  DISABLE-LSA-ISO,DISABLE-VBS

bcdedit /set hypervisorlaunchtype off 

Step 5. If all of that worked without any errors, reboot your system and re-try VMware Workstation.

And so it ends…

No, I’m not giving up on my weekly updates. Have a little more faith in me…don’t expect me to give up until at least February. 🙂

What the title alludes to is the fact that this upcoming week is my last week at Intrado (previously West Corporation). It’s been a personally interesting ride to say the least.

I started as an architect, but soon found myself in an engineering role. I excelled in that role in part because I continued to apply the architecture mindset and applied that to the engineering designs I produced. I’m proud to say that many of the designs I contributed to are continuing to provide value to my co-workers.

Thinking back I feel that my personal growth while working in this position was equally half technical and half business/political. Early on, a lot of the discussions I had with directors and co-workers showed me that the human component of engineering was critical for projects to succeed. A solid technical design will never succeed if there is a well-placed individual who has their own plans. Teaming up with these individuals opened my eyes to their point of view and gave me a chance to advocate for my positions while finding a “happy medium”.

I’m hoping I’m leaving Intrado a better company. The past three years have been rough on all my co-workers, but I truly think that Intrado is finally turning the corner. Some have asked why I’m still leaving if I believe it is getting better. While I am comfortable that my position and employment is safe within Intrado, I feel that I’ve reached the end of the benefit I can provide. I’ve been the “automation advocate” and championing the “Digital Transformation” process, but there is only so much one person can do. I’m hoping my departure will rock the boat enough that multiple others will rise to the challenge and provider a larger group of experts.

And I realized that for the past 18 years I’ve had a 35 minute commute to work (either Intrado or Maryville) – excluding the ~3 years I worked from home and rounding to two 30-minute commute sessions each workday, I have spent a total of 150 days (over 3700 hours) sitting in my car for work days. I’m looking forward to my next employers office which is roughly 12 minutes from my home, even in inclement weather.

So, what else besides my job?

This week I started a number of conversations helping my parents sell their house and get completely moved into the assisted living apartment. We’re starting to work with the estate sales team while we work on some known things to fix up. We know we have some electrical that we’ll have to clean up before putting the house on the market, but some of that are things that Dad and I can do ourselves and we’ll leave the big work to the electrician.

As a late Christmas gift to my parents, we’re paying for a year of the Cox DVR service for them. Many of the shows that Mom loves are (in)conveniently timed at the same time or when they are at appointments. Now they can keep up with their shows as they have been used to when they had DirectTV.

Kris is still under the weather, but she did take herself to the Urgent Care office today and they have her on an antibiotic – I’m hoping she continues to improve, I’m not use to seeing her this sick.

Jilli and her friends headed back to college earlier this week. They left early to miss the snows coming in, but she ended up leaving some fairly important items (her winter coat!) so we have to get them shipped to her soon.

Lizzy took a placement exam for High School Saturday- she came out tired but in good spirits so I’m hoping she did well. To celebrate her “survival” we went out for lunch at Raising Caines – always a good choice IMHO too!

Finally, thanks to everyone I have worked with at Intrado. I wish you all the best and I hope you’ll keep in touch.

Dual use laptop…

Tails – The Amnesic Incognito Live System – is a live operating system that you can start on almost any computer from a USB stick or a DVD. Website: https://tails.boum.org/

A couple years ago my personal laptop was dying, so when I got to work early and sat at the local coffee shop I tried to use my companies laptop. Unfortunately, some of the sites I liked to read were considered “hacker” sites and my companies software blocked that. I had use bootable USB drives running various versions of Linux, but many of them would automatically try to use the local drive and possibly write data to the hard drive. I then came across Tails.

What got me interested in Tails was the fact that it fit on a small USB drive, and everything that you saved to that drive (and only that drive) was encrypted automatically. The Internet communications was also equally encrypted; it uses the TOR network (https://www.torproject.org/) along with a TOR-enabled Firefox browser to hide the traffic from prying eyes on the local network. With all this together, I could install Tails to a small (8GB or 16GB USB stick) and boot my companies laptop from there and not worry about my personal data being saved, or from my company laptop snooping on my private emails and communications.

The Tails system will let you connect to your coffee shop WiFi, but it won’t let you start browsing the web (or logging into your bank or email site) until it has fully connected into the TOR network. The TOR network encrypts all your communications through multiple computers on the Internet – many times the last computer (the “exit node”) is in a different country. (This makes for a wild multi-lingual experience trying to navigate a popular site if they attempt to use your “local” language.)

With Tails I could now use my work laptop but boot into Tails and securely take care of personal tasks (banking, email, medical, etc) without worrying about my companies computer or their filters blocking or breaking some of these sites.

Give Tails a shot if you want to have a simple to use and very secure web browsing experience that won’t mess with your regularly installed operating system.

My resolution

So I promised myself that I would start updating my blog on a weekly basis. Nothing major, no earth-shaking thoughts or deep introspective writings, just a basic update to get the habit started.

At least that is the plan…

For now, I’ll start out simply by noting that my **plan** is to add this every Sunday night, and of course this first one is being written Monday. 🙂 I won’t cop-out and schedule this for next Sunday, and be transparent with my failings, too.

So, we’re continuing to help clean up our parents house getting it ready to sell. Kate has been going out there nearly every weekend (quite often both days), taking Dad and Nick with her and occasionally getting a couple friends to help. To her I tip my hat with a big note of thanks. I’ve tried to keep up with her on those weekends, but have had to beg forgiveness and sit out a few times. And of course Amanda has been doing what she can remotely – she’s our backup for both Kate and I. Kate is helping navigate all the various health-related events with Mom and Dad, and I’m taking on keeping their bills paid and going through all the paperwork and either keeping the important pieces, or getting the remaining stuff to a shredder.

This past weekend we hit the garage attic and got it cleaned out. Many of those boxes had been packed and never opened when they moved from Columbus. Dad was proactive and put labels on each box, then noted in a document what was in each box. Unfortunately, the boxes and the document are both 20 years old, and we’re unable to find the list anymore. It was fun to open them and dive into the contents, but at the same time it was borderline overwhelming. Some of the findings did bring back some happy memories; Jilli found a small bottle of gemstones. They were (we presume) some natural garnets that her Great Grandfather Linder (William Linder, father of Gary) found when he, Dad, and I were hiking one summer in the Rocky Mountains (either near Estes Park, CO or in the Medicine Bow, Wyoming area). He suspected they were garnet, and since that is my gemstone he gave me the bottle of them to keep. Jilli might take them back to SDSM and see if one of the mineralogy professors might be able to determine what they are.

What this work has impressed upon me is to clean out my office a bit. I really doubt that anyone will ever crack open the “Windows 95 Unleashed” book, let alone many of the other related books. If I really get an urge to read it, I’ll find a PDF version but for now the paper version can go into the recycling. (I might donate it, but I don’t think Salvation Army nor Goodwill would have a need for them either.)

A good friend of ours gave us a DeeBot robotic vacuum cleaner for a family Christmas gift. It is amazing what volume of pet hair it picks up, as well as bits of paper, small stones, misc cat/dog food, etc. Our main floor is mostly wood, with only a small bit of carpet and it doesn’t have a problem with either location. What it does have a problem with is going under the couch and chairs. It just barely fits, but occasionally it gets stuck and can’t make it out. I’m brainstorming on how to add a small “guard rail” underneath the furniture, or the alternative is to add little 3/4″ risers under each leg. Since I’m the one with the long legs, I don’t think anyone else wants the chair raised so the guard rail is probably the best option.

Git branching basic workflow

To better handle multiple people working on a project in a Git repository, using branches and reviewing pull requests before committing to the master branch is strongly suggested.


Basic steps:

  1. Change to your home directory and clone the git repository
    1. cd $HOME
    2. git clone [email protected]/MyRepo.git
  2. Change into the new project directory
    1. cd ./MyRepo
  3. Create a branch to work on the new code
    1. git checkout -b MyNewBranch
  4. Verify you are working in the branch
    1. git branch
    2. Note: The branch will have a “*” to the left of the branch name denoting the active branch
  5. Update code, test, repeat
  6. Review and add any missing files
    1. git status
    2. git add <file_name>
  7. Push the code into the repository
    1. git push –set-upstream origin MyNewBranch
      1. This is only necessary for the first ‘git push’
    2. git commit -v {list of changed files}
      • Note the response from the system:
      • remote: Create a pull request for 'MyNewBranch' on GitHub by visiting:
      • remote: https://git.repo.example.org/MyRepo/pull/new/MyNewBranch
  8. Open the pull request (PR) in git.repo.example.org
    1. Add other repository contributors to request a code review before merging.
  9. Repeat the edit/test/PR cycle as necessary until merge is accepted
    1. edit code … test … git status … git add … git commit … git push
  10. When it has been accepted, clean-up your work area:
    1. cd $HOME/MyRepo/
    2. git checkout master
    3. git pull
    4. git branch –delete MyNewBranch
  11. Celebrate on a successful pull request!

IBM Model-M and the Windows Key

I really love my IBM Model-M keyboard, but one frustration is the more frequent need for the Windows key. Even some Linux desktops are using it. Oh I get it, it’s a handy “meta” key that helps differentiate keyboard tasks so it’s lack is annoying at times.

For the few times I am in Windows 10 using my Model-M keyboard, I found this answer on SuperUser.com to re-map the caps-lock key to the Windows key with a simple registry hack: https://superuser.com/a/1228990/101577

Just in case that link goes away, here’s the text:

Anyway, using SharpKeys I found the correct map for CAPS LOCK to Win is this:

https://superuser.com/a/1228990/101577
Windows Registry Editor Version 5.00
[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Keyboard Layout]
"Scancode Map"=hex:00,00,00,00,00,00,00,00,02,00,00,00,5b,e0,3a,00,00,00,00,00

Just save those lines as presented into a file, “c:\WinKeyRemap.reg”, then use File Manager to find it, then double-click to open it with the Registry Editor. (You’ll need to accept the warning.) Assuming the file is correct, Registry Editor will report that the values have been added to the registry.

Reboot to make the changes take effect.