We are continuing our series of Blog Entries to cover some MAS deployment scenarios either left out or not appropriately covered on the IBM Ansible scripts. We are going to also discuss considerations and tips and hints around these deployments. This time we are installing an offline OpenShift Cluster and Mirroring the required Image Registries. This is Part 1 of 3, to cover preparation and initial installation.
We wanted to have comprehensive step by step documentation on how to configure an offline or air-gapped OpenShift Cluster, or a cluster in an environment that is not permanently connected to the internet. In that regard, two situations may occur that require slightly different approaches:
We also want to test and state any required steps to update the catalogs and upgrade both the OpenShift Cluster and the MAS instances plus dependencies by periodically uploading/downloading new images to the separate Image Registry.
First, we need to ensure we have a separate workstation with internet access to use as the Image Registry for downloading the content of the Container Images from the primary repositories on the Internet. This may be the same Bastion Host in permanent installations that have intermittent Internet connectivity (case b) above) or some other workstation that will serve as a gathering point for downloading the initial package contents and any future updates later on (so it is not recommended to decommission it but it can be left turned off when it is not needed). Also, the right amount of storage will be needed for Images for both the RedHat OpenShift installation and the Maximo Application Suite images and dependencies. While size of images
In our setup, we set up a new Rocky Linux workstation with 4 vCPUs and 6GB of RAM and 1TB of primary disk space. The primary (OS/root fs) disk is going to serve as both the Operating System disk and as the Image Storage disk. The idea is to re-use this same VM both for the Staging VM where the images are downloaded to the secondary disk AND the final internal Image Registry where the packages will be loaded from a portable USB disk that was filled from the Staging VM. We are going to make it so those could be two separate VMs as needed and in our case, an external USB disk with the images will need to be moved from the Connected Image Registry workstation to the final Disconnected Image Registry workstation which will not happen in our case in order not to have to setup two separate VMs and be more efficient resource wise.
In that regard, we will have two separate disk and mount points:
To do so, we created a new VM with the intended size (4 vCPUs, 6GB of RAM and 40 + 1024 GB of disk) and named it “dcr.local02.mas.interloc.cloud” with IP address 192.168.8.122. In our case, DCR meaning “Disconnected Container Registry” which is a little misnomer as the VM will be connected to the Internet, but we are going to ensure the SNO RH OCP Cluster in the same will not have Internet access instead, more on that later.
We used the following configuration (the MAC Address was generated online randomly but using a custom reserved prefix that will never collision with any manufacturer assigned prefix). We also made sure the IP Address and MAC Address were defined in a DHCP reservation in our network and added an “A” DNS record in CloudFlare to map the name to the IP address.
Hostname: dcr.local02.mas.interloc.cloud
MAC address: 02:2c:29:4d:7d:f3
IP address: 192.168.8.133
We installed it using the latest “Rocky-10.1-x86_64-dvd1.iso” file and followed the steps as per below. In both cases this file needs to be downloaded first, then mounted on the virtual (or physical) DVD drive of the workstation to be set up.
Note: To install a FIPS compliant workstation instead (which is required for a FIPS enabled OpenShift), ensure to choose “Install Rocky Linux 10.1 in FIPS mode” from the Boot Menu.
In the Welcome to Rocky Linux 10.1 step, we chose “English” > “English (United States)” as language.
In the INSTALLATION SUMMARY step, we modified only:
Press <Done> then under “New mount points will use the following partitioning scheme > Standard Partition” and afterwards under “Manual Partitioning > Click here to create them automatically” then remove the “/home” (sda5) partition from the list and then expand the “/” (root) (sda4) partition to the whole disk by entering a number higher than existing free space on the “Desired Capacity” field such as “2048 GiB” and selecting <Done> then <Cancel & Return to Custom Partitioning> which should display the maximum allowable size on the “Desired Capacity” field. Finally press <Done> again and <Accept Changes> to submit.
Notice we are going to use the “root” user for everything given this is a test setup but otherwise, creating a separate user and using “sudo” should be the right approach.
To start the installation, we clicked on <Begin Installation>. Waited while the software installed as it did take a while.
Once the system has completely installed, we can login to it via SSH with the login and password configured above (in our case, just “root”). The next step was to run “dnf -y update” to install updated packages for everything, including reboot as a new kernel was installed and “dnf remove --oldinstallonly” to remove the old kernel once the workstation has rebooted with the new one. For this step to work, as expected, this workstation needs to have Internet access at that time. Otherwise, we can skip this step and leave with the version installed from the DVD without updating.
Also, the “nmstatectl” utility is required by the “openshift-install” binary but not present in the software selection made from the DVD. Therefore, we should install it using “dnf -y install nmstate”.
After that, we formatted and created the mount point for the external USB disk of 1TB to use as the Image Registry Export/Import path to transport the files between the connected and disconnected workstations (or Image Registries). In our case, we are going to mount it into “/mnt/images/transfer” using the following commands.
/dev/sdb1 3906973696 24576 3906949120 1% /mnt/images/transfer
Given that we already installed “Container Management” we should already have “podman” available. Otherwise “dnf -y install podman” should do.
At this point is also important to check that the root filesystem has all the available space for the primary hard disk by using “df” and not the “/home” partition which should not exist. If not, either we need to reinstall from scratch (follow instructions above) or to remove the “/home” partition, resize the “/” (root) partition and filesystem then remount.
Enable it as a service by executing “systemctl enable --now podman” plus “systemctl enable --now podman-restart”.
We wanted to execute some pre-requisites that are required both for “Direct Mirroring” and for the “2-phase Mirroring”, basically to install the “oc” client and “Docker registry” images we use:
On the public facing workstation:
export OCP_RELEASE="4.20"
mkdir -p ~/downloads
cd ~/downloads
wget https://mirror.openshift.com/pub/openshift-v4/x86_64/clients/ocp/stable-$OCP_RELEASE/openshift-client-linux.tar.gz
tar -xvzf openshift-client-linux.tar.gz
mv oc kubectl /usr/local/bin
To test, just issue “oc version”, and in our case, we got the following output:
Copy the resulting file “openshift-client-linux.tar.gz” into “/mnt/images” as well.
export LOCAL_TRANSFER_DIR="/mnt/images/transfer"
export MIRROR_SINGLE_ARCH="amd64"
cd ~/downloads
mkdir -p $LOCAL_TRANSFER_DIR
cp openshift-client-linux.tar.gz $LOCAL_TRANSFER_DIR
Note: If we want to mirror all available architectures instead of just “amd64” (restricted to save on storage space), we could use “.*” instead for MIRROR_SINGLE_ARCH.
Then, the “openshift-install” binaries as well, similar process to “oc”:
export OCP_RELEASE="4.20"
mkdir -p ~/downloads
cd ~/downloads
wget https://mirror.openshift.com/pub/openshift-v4/x86_64/clients/ocp/stable-$OCP_RELEASE/openshift-install-linux.tar.gz
tar -xvzf openshift-install-linux.tar.gz
mv openshift-install /usr/local/bin
wget https://mirror.openshift.com/pub/openshift-v4/x86_64/clients/ocp/stable-$OCP_RELEASE/openshift-install-rhel9-amd64.tar.gz
tar -xvzf openshift-install-rhel9-amd64.tar.gz
mv openshift-install-fips /usr/local/bin
To test, just issue “openshift-install version”, and in our case, we got the following output:
And the same for the “openshift-install-fips version”:
Copy the resulting file “openshift-install-linux.tar.gz” and “openshift-install-rhel9-amd64.tar.gz” into “/mnt/images” as well.
export LOCAL_TRANSFER_DIR="/mnt/images/transfer"
export MIRROR_SINGLE_ARCH="amd64"
cd ~/downloads
mkdir -p $LOCAL_TRANSFER_DIR
cp openshift-install-linux.tar.gz $LOCAL_TRANSFER_DIR
cp openshift-install-rhel9-amd64.tar.gz $LOCAL_TRANSFER_DIR
Next, we are going to download the “Registry” images as well to use in the disconnected environment:
mkdir -p $LOCAL_TRANSFER_DIR/registry
podman pull docker.io/library/registry:2
podman save -o $LOCAL_TRANSFER_DIR/registry/registry-2.tar docker.io/library/registry:2
Next, download the MAS CLI image as well:
oc image mirror --dir $LOCAL_TRANSFER_DIR/cli quay.io/ibmmas/cli:latest file://ibmmas/cli:latest --filter-by-os=$MIRROR_SINGLE_ARCH
This should create a folder structure inside the “/mnt/images/transfer” with the latest MAS CLI images.
After that, we are going to create a self-signed certificate for the Image Registry, to do so we used the following commands (the certificate is for 20 years):
export LOCAL_TRANSFER_DIR="/mnt/images/transfer"
export REGISTRY_HOST="dcr.local02.mas.interloc.cloud"
export REGISTRY_IP="192.168.8.133"
mkdir -p $LOCAL_TRANSFER_DIR/auth
openssl req -newkey rsa:4096 -nodes -sha256 \
-keyout "$LOCAL_TRANSFER_DIR/auth/registry.key" \
-x509 -days 7300 \
-out "$LOCAL_TRANSFER_DIR/auth/registry.crt" \
-subj "/CN=$REGISTRY_HOST" \
-addext "subjectAltName=DNS:$REGISTRY_HOST,IP:$REGISTRY_IP"
And to add authentication to it, we use:
export LOCAL_TRANSFER_DIR="/mnt/images/transfer"
export REGISTRY_USERNAME="registry"
export REGISTRY_PASSWORD="registry-password"
dnf -y install httpd-tools
htpasswd -Bbn "$REGISTRY_USERNAME" "$REGISTRY_PASSWORD" > "$LOCAL_TRANSFER_DIR/auth/htpasswd"
Having mounted the external hard drive on the same “/mnt/images” path, first, let’s install the “oc” client:
export LOCAL_TRANSFER_DIR="/mnt/images/transfer"
export MIRROR_SINGLE_ARCH="amd64"
tar -xvzf $LOCAL_TRANSFER_DIR/openshift-client-linux.tar.gz
mv oc kubectl /usr/local/bin
Next, load the “registry:2” image into podman using:
podman load -i $LOCAL_TRANSFER_DIR/registry/registry-2.tar
First, we are going to copy the required authentication, certificate, private key, etc. from the TRANSFER location to the STORAGE location due to the TRANSFER location being a removable drive whereas the STORAGE location is in a permanent mount. To do so, we used:
export LOCAL_TRANSFER_DIR="/mnt/images/transfer"
export LOCAL_STORAGE_DIR="/mnt/images/storage"
mkdir -p "${LOCAL_STORAGE_DIR}/data"
cp -r $LOCAL_TRANSFER_DIR/auth $LOCAL_STORAGE_DIR
Then, run the following command to execute the registry:
export LOCAL_STORAGE_DIR="/mnt/images/storage"
export REGISTRY_PORT="5000"
podman run -d \
-p ${REGISTRY_PORT}:5000 \
--name private-registry \
--restart always \
-v "${LOCAL_STORAGE_DIR}/data:/var/lib/registry:Z" \
-v "${LOCAL_STORAGE_DIR}/auth:/auth:Z" \
-e REGISTRY_HTTP_TLS_CERTIFICATE=/auth/registry.crt \
-e REGISTRY_HTTP_TLS_KEY=/auth/registry.key \
-e REGISTRY_AUTH=htpasswd \
-e REGISTRY_AUTH_HTPASSWD_REALM="Private Registry" \
-e REGISTRY_AUTH_HTPASSWD_PATH=/auth/htpasswd \
registry:2
Notice the above command runs a registry which we named “disconnected-registry” on TCP Port: 5000 of both the host and the container and mapping the host “/mnt/images/storage” folder to the container “/var/lib/containers” folder and using the “Z” option meaning private access to the path to avoid permission issues inside the container to “/var/lib/containers”. In this case, it will store the containers images outside of the Pod ephemeral filesystem so they will persist between reboots or other container restarts.
Next, to extract a Pull Secret from the above credentials that can be used to pull images from OpenShift, we used:
export LOCAL_TRANSFER_DIR="/mnt/images/transfer"
export REGISTRY_HOST="dcr.local02.mas.interloc.cloud"
export REGISTRY_PORT="5000"
export REGISTRY_USERNAME="registry"
export REGISTRY_PASSWORD="registry-password"
podman login --tls-verify=false -u $REGISTRY_USERNAME -p $REGISTRY_PASSWORD $REGISTRY_HOST:$REGISTRY_PORT --authfile $LOCAL_TRANSFER_DIR/auth/private-registry-pullsecret.json
To confirm the Pull Secret was generated correctly, we can use:
cat $LOCAL_TRANSFER_DIR/auth/private-registry-pullsecret.json
Which in our case, printed out:
{
"auths": {
"dcr.local02.mas.interloc.cloud:5000": {
"auth": "cmVnaXN0cnk6cmVnaXN0cnktcGFzc3dvcmQ="
}
}
}
Note: Additional useful commands may be (in case the mirroring fails to debug errors and clear space, etc.):
podman logs private-registry >> To see startup/current logs
podman ps >> To list containers running in the host, it will output the <container-id> that can be used later
podman exec -it <container-id> /bin/sh >> Execute a shell on the target container to issue commands inside (such as "df", etc.)
podman stop <container-id> >> To stop the target container
podman pod rm -a -f >> To clear all non-running containers
podman system prune -a --volumes >> To prune the filesystem and free space of cleared containers
To confirm, from another workstation with access to the IP and port and having “podman” (or “docker”), use:
export REGISTRY_HOST="dcr.local02.mas.interloc.cloud"
export REGISTRY_PORT="5000"
export REGISTRY_USERNAME="registry"
export REGISTRY_PASSWORD="registry-password"
podman login --tls-verify=false -u $REGISTRY_USERNAME -p $REGISTRY_PASSWORD $REGISTRY_HOST:$REGISTRY_PORT
The response we should expect is:
Login Succeeded!
To remove the saved credentials and log off, we can use:
podman logout $REGISTRY_HOST:$REGISTRY_PORT
To restore the MAS CLI images that were saved before on the Internet Connected workstation (once the registry is running there) we used the following commands:
export LOCAL_TRANSFER_DIR="/mnt/images/transfer"
export REGISTRY_HOST="dcr.local02.mas.interloc.cloud"
export REGISTRY_PORT="5000"
export REGISTRY_USERNAME="registry"
export REGISTRY_PASSWORD="registry-password"
export MIRROR_SINGLE_ARCH="amd64"
podman login --tls-verify=false $REGISTRY_HOST:$REGISTRY_PORT -u $REGISTRY_USERNAME -p $REGISTRY_PASSWORD
oc image mirror --insecure=true --dir $LOCAL_TRANSFER_DIR/cli file://ibmmas/cli:latest $REGISTRY_HOST:$REGISTRY_PORT/ibmmas/cli:latest --filter-by-os=$MIRROR_SINGLE_ARCH
The above command should restore the MAS CLI images in the local image registry. And therefore, we should be able to pull them when invoking the CLI container.
To test that it works, we could use:
podman run -ti --rm --tls-verify=false $REGISTRY_HOST:$REGISTRY_PORT/ibmmas/cli:latest
Just use “exit” to leave the container terminal and stop it.
This is just the beginning. Join us next week for Part 2 of our three-part series as we continue exploring offline OpenShift deployments and image registry management for IBM Maximo Application Suite environments.
To stay up to date on this series and future technical insights, tips, and best practices from our experts, complete the form below and subscribe to our blog mailing list.
References:
The related IBM documentation can be found under https://www.ibm.com/docs/en/masv-and-l/cd?topic=cli-in-disconnected-environments.
There are also important caveats and considerations contained in https://www.ibm.com/docs/en/masv-and-l/cd?topic=cli-prerequisites-installing.
Also, the following URLs contain some command references for the “mirror-images” and “mirror-redhat-images” commands when using the IBM MAS CLI: https://ibm-mas.github.io/cli/guides/image-mirroring/.
However, there are also related RedHat resources which are more general in nature to mirror the registries locally by using the “oc-mirror” tool for disconnected environment as discussed in https://docs.redhat.com/en/documentation/openshift_container_platform/4.9/html/installing/installing-mirroring-installation-images