Buildup

Thanks to “https://www.reddit.com/user/MaricxX/” for this photo – it demonstrates how small glitches over time can add up if they aren’t addressed rapidly – or better yet, not allowed to start in the first place.

Cross section of layers of paint showing deformation due to imperfections magnified with each layer.
Layers of paint – credit to MaricxX from Reddit – https://www.reddit.com/user/MaricxX/

At a previous job it was common to take our Windows virtual machine templates and power them on once a month to patch the OS and apply the latest security configurations. We had been doing this with our Red Hat Linux images, but a couple years ago I converted our process so each month we built those VM templates fresh from an ISO and a Hashicorp Packer script using VMware Workstation.

This monthly fresh build ensured that we always knew how to build the VM templates in the event of a disaster, and it ensured that our build process contained exactly what we planned and advertised (through our team Git repository). As new requirements were received from the InfoSec team or other sources with system concerns that could only be readily addressed during the initial build phase, we would add those steps to the Packer config file, then test and build new.

With the prevalence of new worms and other highly effective infection vectors, my fear was that we would get a piece of malware onto the templates and then that malware would be automatically replicated each time a new system was built. And there were many times when we started the patching process each month only to find that a couple of the Windows templates had been left running since the previous months patch effort. There is no telling what might have crawled onto these unmanaged systems in the intervening time, only waiting for us to start using them over time.

While the paint analogy doesn’t perfectly match with the IT world, there are sufficient correlations that it makes the possibility of replicating and amplifying a small defect all the more understandable. Still, I prefer to have my freshly-built template with it’s minimal layers of paint knowing that I am confident that it only contains the bits we wanted.

$RANDOM

Friday was my last day – and the weather was poor enough (snow with freezing rain) that the company sent an email the day before telling people to work from home if they could. I am glad I worked from home – I think I was able to get a lot of documentation wrapped up and some last-minute things completed and handed off. Even if I had another two weeks, I still wouldn’t have handed things off properly. There would always be one more thing to work on, one more thing to clean up, one more thing to polish. And the kicker was that I wasn’t truly handing things off as much as throwing documentation and notes into README.md files and Wiki pages and hoping someone at a future date would find them and keep the ball moving forward. But, all things come to and end – I’m looking forward to my new job starting this week and I wonder what sort of things I’ll get into next. 😀

Earlier this week Jilli sent Kris a text telling her that a Mountain Lion was roaming campus. Students were to call 911 immediately if they saw it. I was concerned that her first reaction would be to call “Here kitty, kitty!” and try to pet it. My next vision was Jilli and her friends running away from the lion, each with their phones in their hands Googling “How to escape a Mountain Lion”…

Liz had a normal week at school. She and Kris spent a lot of time together since I had a lot of late nights wrapping up work and helping with my parents. She’s continuing to use her weight training bag in the basement, plus she’s starting to cook more and more. Ready to bake Croissant Rolls are being made frequently, as are Chocolate Chip cookies. I’ve eaten way too many of both this week – my post-Christmas weight loss isn’t working.

The cold/crud that I brought home over Christmas has left me, but is continuing to annoy Kris. She was just starting to get over the worst of the coughing when she hurt a muscle in her back coughing so much. She was in a lot of pain after school on Friday – she says even sleeping is painful sleeping on her back and putting pressure on the muscle. I really need to talk to her mom about her bodies warranty coverage…

Mom and Dad both continue to kick around AV. It was so cold and icy Saturday morning that we decided not to go out to their house so I rescheduled the home inspection for another week. We met with a new financial advisor this week, but I keep hoping we stumble across some gold bars or a hand full of un-sold “Berkshire A” stock certificates. Probably not likely, but I can keep my fingers crossed.

Docker on Windows Subsystem for Linux to play with RedHat 8

Ok, so this is kind of long but neat too!

A co-worker asked about using a Docker image for a project he’s working on and I suggested that he use the RedHat 7/8 based “Universal Base Image” that they announced at Summit. (Our company has a large installed base of RedHat, so there is a big advantage being to tap into that internal knowledge.)

–> https://www.redhat.com/en/blog/introducing-red-hat-universal-base-image

If you have a machine with Docker setup, then doing a pull of “registry.access.redhat.com/ubi8/ubi:latest” will pull down the RHEL-8 version.

–> $ docker run –rm -it registry.access.redhat.com/ubi8/ubi:latest /bin/bash

But I don’t have a Docker system, I only have Windows 10!” No fear, you can install Docker on Windows:

–> https://docs.docker.com/docker-for-windows/install/

From there you can kick off Docker from PowerShell or the command prompt with the exact same command prompt as shown above.

But I want to do this in a Linux environment on my Windows workstation!”  Use the “Windows Subsystem for Linux” feature of Windows 10:

–> https://medium.com/@sebagomez/installing-the-docker-client-on-ubuntus-windows-subsystem-for-linux-612b392a44c4

Here’s a screen shot of a RHEL-8 container running under WSL showing that “yum install …” works as expected:

And here it is running under PowerShell:

When is a disk space problem not a disk space problem?

A co-worker setup an Ansible playbook to update some packges but it kept erroring out. The error that Ansible reported from “yum” was “No space left on device“. He had jumped onto the system and saw that this partition had plenty of space left so asked if I could look into it.

I got on and confirmed that when I ran a simple “yum update” it showed this:

[[email protected] ~]# echo n | yum update

Loaded plugins: product-id, rhnplugin, search-disabled-repos, security, subscription-manager

[Errno 28] No space left on device: ‘/var/run/rhsm/cert.pid’

This system is receiving updates from RHN Classic or RHN Satellite.

Could not create lock at /var/run/yum.pid: [Errno 28] No space left on device: ‘/var/run/yum.pid’

Hmm, no disk space still. Looking at the “df /var” output looks good:

[[email protected] ~]# df /var

Filesystem           1K-blocks   Used Available Use% Mounted on

/dev/mapper/rootvg-varlv

                       2514736 914948   1468716  39% /var

Suspecting other resource issues I checked the inode availability using “df -i:

[[email protected] ~]# df -i /var

Filesystem           Inodes  IUsed IFree IUse% Mounted on

/dev/mapper/rootvg-varlv

                     163840 163840     0  100% /var

A ha! No inodes left. I’ll let you use your favorite search engine to look up details, but an easy way to think of “inodes” is as space on the first few pages of a book dedicated to being the “table of contents.” If you have a book with a few chapters, you only need a single page for the table of contents (the inodes). If you have a book with lots of chapters and sub-chapters, you might need a lot of pages (more inodes). By default Unix systems have a forumla on how much of the filesystem to dedicate to being “inodes” and how much is left for actual data storage. Usually this is fine for most systems.

To find them we want to look for directories which have chewed up the 163K files:

for i in /var/*; do echo $i; find $i |wc -l; done

This pointed to the “/var/spool/app01/” directory – it has over 160K small files.  The owner of the system was able to clean up some old files there and the “yum update” worked as expected.

It’s possible to override the inode settings when the filesystem is formatted, so if you know this ahead a time you can do this. If you run into this after the fact, the usual resolution is to backup the data, reformat the filesystem with more inodes allocated, then restore from backup.

SELinux and NFS $HOME directories

Recently we re-installed a common server with RHEL-7 and that went well.  But after a couple days I noticed that I was unable to login with my personal ssh key but I had before. It was a minor annoyance and didn’t pursue it … until today.

It turns out that the /home/ directory on this system is an NFS mount, and in RHEL-7 we have set SELinux to default to enforcing.  There is an SELinux boolean flag, “use_nfs_home_dirs” that needed to be set to “1” (true).  Running the “setsebool -P use_nfs_home_dirs 1” on this system was the fix and now we/I can resume logging in with the SSH key instead of typing in my passwordeach time.

Some were reluctant to fix this as they always typed in their password. While typing in your password over the SSH login connection is encrypted, but it does present the possibility that your password could get copied given a compromised endpoint, plus we are trying to use longer passwords so typing this in multiple times per day was frustrating and slowed workflow.  Using SSH keys eliminates this risk and provides for other features such as scheduled/scripted command execution and file transfers.

Delete yourself from Medium

I’ve been on the Internet for years. The openness of the whole environment was refreshing. Want to find out something obscure or highly technical, you could head over to a university website and find an article, or over to a vendors site to get details on something you were looking for..

Then came web-rings and blogs. A webring was a simple “next page” link on a site that would take users from one persons page to another, usually the pages had a common theme (a hobby, an interest, etc). Later there were blogs (like this WordPress site) that were more dynamic. You could write multiple pages on varying topics, or you could work with others to share access and create a source of information about your topics.

This wasn’t free, but many of us kept our sites up out of the love for the art we were discussing, or out of the feeling of providing back to the wider world. For times when sites got too big to support by one persons budget, there were advertisers who would trade some of the blog page real estate to host adds in return for a small bit of money back to the owner of the site. And for some sites, they turned to user supported options so that people who paid a small periodic fee could in turn get access to other articles earlier, or possibly in-depth that weren’t public. Many newspapers have turned to this – the general public gets the first paragraph of a story, but subscribers could get the entire article and additional features.

But over time the increase of the web as a “social medium” platform took root. After a while, the need to drive more and more eyeballs to a website took on a more “closed off” approach. I’m sure there are many more that I are out there, but the one that finally got to me and made me say “enough is enough” is the website “Medium.com“.

They are a hosted blogging site and that’s nice. They also have good search-engine-optimization features so a well written article gets found easily on Google and other search engines. Many times when I’m searching for some additional information on a news topic, I will come across something hosted on a Medium.com page and click the link. Or at least only up to a threshold of free posts per month then you have to wait until the next month. Or you can sign up for a monthly subscription to access more articles.

All that is good and I will not begrudge them for collecting a fee to offset their designers and staff who keep the website running smoothly, as well as the curation of articles they are performing. But I can’t justify spending another $5 a month on yet another blogging site.

And something about the whole “pay to see anything” mentality seems to be antithetical to what really makes the Internet ‘work’, and honestly what was the groundwork that permitted sites such as Medium, Facebook, Google, Amazon, etc. to thrive. It feels like all the good works that were poured into the initial Internet (open networking standards, operating systems, email, web servers and browsers) and kept free all this time are being clear-cut by these new digital locusts.

But they aren’t listening. Instead they are using the subscription fees to lock more and more content behind their closed doors. If this continues, we’ll have an entire Internet made up of toll roads and not the wide-open digital universe we have today.

If you agree, please take some time and remove yourself from the Medium website. It’s very easy – under your profile, the very bottom option is “delete from Medium”. I don’t need their constant barrage of articles that don’t provide back to the greater good, at least not without a fee and tracking to better serve up my time and attention to their article writers for another piece of fluff.

Do it – it felt good!

The humble check-list

Driving home last night I listened to the next episode of the NPR “Hidden Brain” podcast titled “Check Yourself”. The topic of this was around the humble “checklist” that we’ve all made but never gave much thought to.

Two parts of the story surprised me:

  • They came to be a requirement in the airline industry after a crash of a new and highly complex aircraft in 1939.
  • The very recent addition of checklists to hospital procedures.

Link to text of this podcast: https://www.npr.org/2018/08/27/642310810/you-2-0-check-yourself

General podcast link with option to listen to MP3 in browser or download: https://www.npr.org/podcasts/510308/hidden-brain

Come’on RedHat!

Come’on RedHat – give us the MSDN version for home labs and training.

I recently signed up for the RedHat Developer Program and have setup my lab system with a fully updated RedHat Enterprise Linux 7 operating system with all of the bells-and-whistles the OS provides.

The Developer Program not only gives you a fully licensed RHEL system, it also provides access to JBoss, and the RedHat Container Development Kit which contains a set of development tools and additional resources (Python, PHP, Ruby, OpenJDK, etc).  All great stuff, especially for developers looking to hone their OpenSource development practices.

Unfortunately, for those of us on the systems side of the world (not hard-core developers) the package omits a couple of great RedHat products: Cloudforms and RedHat Virtualization.

I know that I could attempt to knit together the upstream versions of each of these (ManageIQ and oVirt), but there is a reason that the Cloudforms and RHV exist to save us from the complexity of configuring all of the components together.  As a learning platform, the effort of researching numerous blog posts and reams of documentation for each project (and their pre-requisites) might be worthwhile, but for some one like me who is interested in learning more about Cloudforms/ManageIQ itself (automations, hooking into a lab VM environment, etc) it severely skews away from the objective I’m trying to educate myself on.

Microsoft has their MSDN – Microsoft Developer Network – that provides a license for a nominal fee (at the low end) that provides licensing for Windows OS installations (five licenses, many versions, both standard and Server), SQL, Exchange, SharePoint, Office and Office365, Azure, etc.  Windows Server has Hyper-V as an option – it’s answer to RHV/oVirt.  (I don’t believe they have a Cloudforms/ManageIQ alternative, though one might exist.)

I understand that MSDN is not free ($0.00 expense) like the RedHat Developer Program is, so it’s not a perfect comparison but Microsoft has a range of offerings price points.  If a small ($99/year) cost would expand the RedHat option to include Cloudforms, RHV, and multiple server subscriptions (not the single RHEL instance), I could greatly expand my experience with the options RedHat provides.

So RedHat, what do you say?  Expand the program for those of us who want to learn about the full breadth of RedHat products!

So, what you’re saying is…

Had Chinese with my family last night and I got this in my fortune-cookie:

 

Fortune: :) You are soon going to change your present line of work. :)
Interesting fortune…

I’m wondering if this is foreshadowing upcoming events?  What do the fates have in store for me?