MAS Deployment Series: Configuring S3 Object Storage as an OpenShift Storage Class

Abstract: We are continuing our series of Blog Entries to cover some MAS deployment scenarios either left out or not appropriately covered on the IBM Ansible scripts. We are going to also discuss considerations and tips and hints around these deployments. This time we are configuring and creating a Storage Class for the RedHat OpenShift Cluster so it can be used in some scenarios.

References:

The related IBM documentation to configure S3 for DOCLINKS and other uses inside Manage can be found under https://www.ibm.com/docs/en/masv-and-l/maximo-manage/continuous-delivery?topic=storage-configuring-s3.

The Storage CSI driver that we are using in this Blog post is located and being maintained at https://github.com/yandex-cloud/k8s-csi-s3.

Content:

We wanted to have an S3 (Cloud Object Storage) based filesystem Provider on RedHat OpenShift so we could use it as a target for Persistent Volume Claims (PVCs) and Persistent Volumes (PVs) which in turn could be used for functionalities such as Backups, DOCLINKS, Integration Files, etc. and be reused as appropriate.

The main benefits, drawbacks and considerations that we could see for the approach are:

• Resiliency and High Availability: For Backups, and specifically for IBM Cloud, it will allow us to have a “Global” S3/COS defined that is automatically replicated along multiple geographies and therefore should be available even in the instance of a whole datacenter getting damaged or destroyed in case of acts of God or events such as war.
• Usability: For the above and other uses, it simulates a filesystem-based approach as it provides a “ReadWriteMany” Storage Class that can be used to create and “mount” PVCs. Therefore, it provides an “agnostic” interface to workloads (the filesystem).
• Performance: Given that the actual back end is HTTP(s) based, advanced file operations will not be supported, and the type of usage should be carefully considered to operations such as writing a whole file at a time instead of doing multiple random accesses and writes to an existing file. Performance will not be good, and latency and bandwidth will be more limited than using Local Storage or Local Network Storage (such as NFS).
• Simplified administration: It will allow reusing the same S3/COS bucket instead of creating buckets for every single purpose on a Manage installation or for other uses such as backups. It could also create new buckets for each PVC dynamically instead of subfolders for a pre-existing bucket.
• Working around existing limitations: For DOCLINKS, current Manage implementation lacks flexibility as, at the time of writing, it puts all files inside the root of the S3/COS bucket and there is no support for subfolders even when using the “Manage Folders” functionality in Maximo.
• Cost: Typically, S3 buckets charge based on actual space consumed and additionally sometimes by the volume of data transferred. In contrast, File Share approaches are usually charged based on the provisioned space. Therefore, it is usually cheaper to use an S3 bucket for the same amount of space.
• Source: The Container Storage Interface (CSI) driver for S3 implementation is a GitHub project from the URL https://github.com/yandex-cloud/k8s-csi-s3 which means we will be installing code that was writing by external developers and OpenSource dependencies. This may require additional disclosure when using this driver for Federal or other customers requiring enhanced security disclosures.
• Downtime: There is a requirement for GeeseFS (the default implementation of the above driver) that needs to add a SELinux configuration to the Worker Nodes and therefore, a MachineConfig will be submitted to the Cluster to patch the Worker Nodes during startup. Which will cause the Machine Config Pool to render a new configuration and apply it to the Worker Nodes which in turn will cause a restart of the Worker Nodes. Therefore, applying during a maintenance window is recommended.

Before we start implementing the S3 CSI driver and connecting it to the S3/COS we need to gather a few configurations which are standard to S3/COS implementations, these are in the following Environment Variables:

BACKUPS_S3_ENDPOINT="<endpoint-url>"

Required. The S3 Endpoint URL. This should be provided by the S3 endpoint normally using either the 'http' or 'https' protocols. Plus, a hostname or IP address plus maybe some context afterwards. There is typically no default value for this setting which must be always provided unless the       BACKUPS_S3_AMAZON_REGION is set in which case this environment variable becomes derived from the value set for the Amazon Region.

BACKUPS_S3_AMAZON_REGION="<amazon-region>"

Optional. The Amazon Region Name to use when connecting to an Amazon AWS S3 Endpoint. Typically values such as 'eu-central-1' are used. If not specified, it is assumed we are not connecting to an Amazon AWS S3 Endpoint and therefore, BACKUPS_S3_ENDPOINT becomes mandatory. Otherwise, the BACKUPS_S3_ENDPOINT is overridden by using the following convention "https://s3.<amazon-region-name>.amazonaws.com".

BACKUPS_S3_ACCESS_KEY_ID=<s3-access-key-id>

Required. The required S3 Access Key ID. This should be provided by the S3 endpoint normally using a long string of characters like a login ID. There is no default value for this setting which must be always provided.

BACKUPS_S3_SECRET_ACCESS_KEY=<s3-secret-access-key>

Required. The required S3 Secret Access Key. This should be provided by the S3 endpoint normally using a long string of characters like a password. There is no default value for this setting which must be always provided.

BACKUPS_S3_STORAGE_CLASS_NAME=<storage-class-name>

The Storage Class Name to point to the above Endpoint, Access Key ID and Secret Access Key properties. Also see BACKUPS_S3_BUCKET_NAME immediately below. For this example, we are using our default value which is 'backups-csi-s3'.

BACKUPS_S3_BUCKET_NAME=<pre-existing-S3-bucket-name>

The pre-existing S3 bucket name to reuse, which will attempt to create a subfolder for each PVC/PV that gets created using the Storage Class as defined in the BACKUPS_S3_STORAGE_CLASS_NAME variable above. There is NO default value for this and setting it or not affects behavior significantly. We recommend setting the variable up and therefore provisioning a pre-existing bucket beforehand instead of allowing buckets to be created dynamically for each PVC. Notice that dynamic creation of buckets per each PVC requires the Credentials/Security Principal above as per BACKUPS_S3_ACCESS_KEY_ID and BACKUPS_S3_SECRET_ACCESS_KEY to have privilege to create new buckets.

BACKUPS_S3_ALLOW_INSECURE_CONNECTIONS=<true-or-false>

Whether to allow insecure (i.e. Self-Signed Certificates) when using "https" as the protocol stated on BACKUPS_S3_ENDPOINT. The default value is 'true' (so it allows Self-Signed Certificates by default).

We also have other variables to specify additional settings such as Target Namespace for the Provisioner, Provisioner version to use, Secret Names for the credentials, Image URLs for the Driver Provisioner, Registrar and CSI, plus Memory Limits and Number of Replicas for the Provisioner. However, for the purpose of this blog, we are going to hardcode them in our YAMLs. Notice that some of the YAMLs below are tested and should work for the specific versions we have hardcoded (v0.42.1 at the time of writing), and they are NOT guaranteed to work without modifications for future versions.

Notice that all environment variable reference in YAMLs below need to be substituted by actual values as per above before submitting (these are always in the ${envvar_name} format). Also, any values in YAML contents below in italics or bold are meant to be reviewed and adapted to the specific circumstances of the deployment or target environment.

Once we have all the environment variables above defined with appropriate values, we need to create the Project/Namespace. To do so, we submitted the following YAML:

kind: Project
apiVersion: project.openshift.io/v1
metadata:
  name: csi-s3-storage
  annotations:
    openshift.io/display-name: CSI Driver for S3 Storage backends

Next, we are going to create the Provisioner itself and dependencies, to do so, we submitted the following YAML (sourced with additional modifications from https://github.com/yandex-cloud/k8s-csi-s3/raw/master/deploy/kubernetes/provisioner.yaml): 

apiVersion: v1
kind: ServiceAccount
metadata:
  name: csi-s3-provisioner-sa
  namespace: csi-s3-storage
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: csi-s3-external-provisioner-runner
rules:
  - apiGroups: [""]
    resources: ["secrets"]
    verbs: ["get", "list"]
  - apiGroups: [""]
    resources: ["persistentvolumes"]
    verbs: ["get", "list", "watch", "create", "delete"]
  - apiGroups: [""]
    resources: ["persistentvolumeclaims"]
    verbs: ["get", "list", "watch", "update"]
  - apiGroups: ["storage.k8s.io"]
    resources: ["storageclasses"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["events"]
    verbs: ["list", "watch", "create", "update", "patch"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: csi-s3-provisioner-role
subjects:
  - kind: ServiceAccount
    name: csi-s3-provisioner-sa
    namespace: csi-s3-storage
roleRef:
  kind: ClusterRole
  name: csi-s3-external-provisioner-runner
  apiGroup: rbac.authorization.k8s.io
---
kind: Service
apiVersion: v1
metadata:
  name: csi-s3-provisioner
  namespace: csi-s3-storage
  labels:
    app: csi-s3-provisioner
spec:
  selector:
    app: csi-s3-provisioner
  ports:
    - name: csi-s3-dummy
      port: 65535
---
kind: StatefulSet
apiVersion: apps/v1
metadata:
  name: csi-s3-provisioner
  namespace: csi-s3-storage
spec:
  serviceName: "csi-provisioner-s3"
  replicas: 1
  selector:
    matchLabels:
      app: csi-s3-provisioner
  template:
    metadata:
      labels:
        app: csi-s3-provisioner
    spec:
      serviceAccount: csi-s3-provisioner-sa
      tolerations:
        - key: node-role.kubernetes.io/master
          operator: Exists
        - key: CriticalAddonsOnly
          operator: Exists
      containers:
        - name: csi-provisioner
          image: "quay.io/k8scsi/csi-provisioner:v2.1.0"
          args:
            - "--csi-address=$(ADDRESS)"
            - "--v=4"
          env:
            - name: ADDRESS
              value: /var/lib/kubelet/plugins/ru.yandex.s3.csi/csi.sock
          imagePullPolicy: "IfNotPresent"
          volumeMounts:
            - name: socket-dir
              mountPath: /var/lib/kubelet/plugins/ru.yandex.s3.csi
        - name: csi-s3
          image: "cr.yandex/crp9ftr22d26age3hulg/csi-s3:0.42.1"
          imagePullPolicy: IfNotPresent
          args:
            - "--endpoint=$(CSI_ENDPOINT)"
            - "--nodeid=$(NODE_ID)"
            - "--v=4"
          env:
            - name: CSI_ENDPOINT
              value: unix:///var/lib/kubelet/plugins/ru.yandex.s3.csi/csi.sock
            - name: NODE_ID
              valueFrom:
                fieldRef:
                  fieldPath: spec.nodeName
          volumeMounts:
            - name: socket-dir
              mountPath: /var/lib/kubelet/plugins/ru.yandex.s3.csi
      volumes:
        - name: socket-dir
          emptyDir: {}

As a result of executing the previous script, there should be a Pod created in the Project and after a little while the status should be “Ready” “2/2” for it.

Next, we are going to create the Driver itself and dependencies, to do so, we submitted the following YAML (sourced without additional modifications from https://github.com/yandex-cloud/k8s-csi-s3/raw/master/deploy/kubernetes/driver.yaml):

apiVersion: storage.k8s.io/v1
kind: CSIDriver
metadata:
  name: ru.yandex.s3.csi
spec:
  attachRequired: false
  podInfoOnMount: true

There are no obvious results at the Cluster for the above operation.

Next, we are going to create the CSI itself and dependencies, to do so, we submitted the following YAML (sourced with additional modifications from https://github.com/yandex-cloud/k8s-csi-s3/raw/master/deploy/kubernetes/csi-s3.yaml):

apiVersion: v1
kind: ServiceAccount
metadata:
  name: csi-s3
  namespace: csi-s3-storage
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: csi-s3
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: csi-s3
subjects:
  - kind: ServiceAccount
    name: csi-s3
    namespace: csi-s3-storage
roleRef:
  kind: ClusterRole
  name: csi-s3
  apiGroup: rbac.authorization.k8s.io
---
kind: DaemonSet
apiVersion: apps/v1
metadata:
  name: csi-s3
  namespace: csi-s3-storage
spec:
  selector:
    matchLabels:
      app: csi-s3
  template:
    metadata:
      labels:
        app: csi-s3
    spec:
      tolerations:
        - key: CriticalAddonsOnly
          operator: Exists
        - operator: Exists
          effect: NoExecute
          tolerationSeconds: 300
      serviceAccount: csi-s3
      containers:
        - name: driver-registrar
          image: "quay.io/k8scsi/csi-node-driver-registrar:v1.2.0"
          args:
            - "--kubelet-registration-path=$(DRIVER_REG_SOCK_PATH)"
            - "--v=4"
            - "--csi-address=$(ADDRESS)"
          env:
            - name: ADDRESS
              value: /csi/csi.sock
            - name: DRIVER_REG_SOCK_PATH
              value: /var/lib/kubelet/plugins/ru.yandex.s3.csi/csi.sock
            - name: KUBE_NODE_NAME
              valueFrom:
                fieldRef:
                  fieldPath: spec.nodeName
          volumeMounts:
            - name: plugin-dir
              mountPath: /csi
            - name: registration-dir
              mountPath: /registration/
        - name: csi-s3
          securityContext:
            privileged: true
            capabilities:
              add: ["SYS_ADMIN"]
            allowPrivilegeEscalation: true
          image: "cr.yandex/crp9ftr22d26age3hulg/csi-s3:0.42.1"
          imagePullPolicy: IfNotPresent
          args:
            - "--endpoint=$(CSI_ENDPOINT)"
            - "--nodeid=$(NODE_ID)"
            - "--v=4"
          env:
            - name: CSI_ENDPOINT
              value: unix:///csi/csi.sock
            - name: NODE_ID
              valueFrom:
                fieldRef:
                  fieldPath: spec.nodeName
          volumeMounts:
            - name: plugin-dir
              mountPath: /csi
            - name: stage-dir
              mountPath: /var/lib/kubelet/plugins/kubernetes.io/csi
              mountPropagation: "Bidirectional"
            - name: pods-mount-dir
              mountPath: /var/lib/kubelet/pods
              mountPropagation: "Bidirectional"
            - name: fuse-device
              mountPath: /dev/fuse
            - name: systemd-control
              mountPath: /run/systemd
      volumes:
        - name: registration-dir
          hostPath:
            path: /var/lib/kubelet/plugins_registry/
            type: DirectoryOrCreate
        - name: plugin-dir
          hostPath:
            path: /var/lib/kubelet/plugins/ru.yandex.s3.csi
            type: DirectoryOrCreate
        - name: stage-dir
          hostPath:
            path: /var/lib/kubelet/plugins/kubernetes.io/csi
            type: DirectoryOrCreate
        - name: pods-mount-dir
          hostPath:
            path: /var/lib/kubelet/pods
            type: Directory
        - name: fuse-device
          hostPath:
            path: /dev/fuse
        - name: systemd-control
          hostPath:
            path: /run/systemd
            type: DirectoryOrCreate

Next, we should grant some additional privileges required to the above Service Account to avoid errors like ‘forbidden: unable to validate against *** security context constraint’, to do so, we executed the following “oc” commands:

oc adm policy add-scc-to-user anyuid -z csi-s3 -n csi-s3-storage
oc adm policy add-scc-to-user privileged -z csi-s3 -n csi-s3-storage
oc adm policy add-scc-to-user hostmount-anyuid -z csi-s3 -n csi-s3-storage

We should receive responses like the below:

clusterrole.rbac.authorization.k8s.io/system:openshift:scc:anyuid added: "csi-s3"
clusterrole.rbac.authorization.k8s.io/system:openshift:scc:privileged added: "csi-s3"
clusterrole.rbac.authorization.k8s.io/system:openshift:scc:hostmount-anyuid added: "csi-s3"

As a result of executing the previous script, there should be a number (one per Worker Node) of additional Pods named like “csi-s3-<id>” created in the Project and after a little while the status for all of them should be “Ready” “2/2”.

The next step should be to create the Credentials to use to authenticate to the S3/COS Endpoint, for that, we submit the following YAML to the Cluster:

apiVersion: v1
kind: Secret
metadata:
  name: "csi-s3-backup-secret"
  namespace: "csi-s3-storage"
stringData:
  accessKeyID: "${BACKUPS_S3_ACCESS_KEY_ID}"
  secretAccessKey: "${BACKUPS_S3_SECRET_ACCESS_KEY}"
  endpoint: "${BACKUPS_S3_ENDPOINT}"
  insecure: "${BACKUPS_S3_ALLOW_INSECURE_CONNECTIONS}"
  region: "${BACKUPS_S3_AMAZON_REGION}"

Notice that the proper Namespace and all the above-mentioned environment variables should be substituted. As noted, both the “endpoint” and “region” values should not be filled at the same time. A sample configuration for guidance with the values substituted appears below:

apiVersion: v1
kind: Secret
metadata:
  name: "csi-s3-backup-secret"
  namespace: "csi-s3-storage"
stringData:
  accessKeyID: "4eb324cda65d99ab6fd534b89bef54d7"
  secretAccessKey: "907babadbb1995f452c33241668a73cf9791cc10a030c78b9c477"
  endpoint: "https://s3.us.cloud-object-storage.appdomain.cloud"
  insecure: "true"
  region: ""

Notice that in the above example we specified an Endpoint URL and no Region as it was not an Amazon S3 bucket.

The next step should be to create the Storage Class, by sending a YAML such as the below:

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: "${BACKUPS_S3_STORAGE_CLASS_NAME}"
provisioner: ru.yandex.s3.csi
parameters:
  mounter: geesefs
  options: "--memory-limit 1000 --dir-mode 0777 --file-mode 0666 --no-systemd"
  bucket: "${BACKUPS_S3_BUCKET_NAME}"
  csi.storage.k8s.io/provisioner-secret-name: csi-s3-backup-secret
  csi.storage.k8s.io/provisioner-secret-namespace: csi-s3-storage
  csi.storage.k8s.io/controller-publish-secret-name: csi-s3-backup-secret
  csi.storage.k8s.io/controller-publish-secret-namespace: csi-s3-storage
  csi.storage.k8s.io/node-stage-secret-name: csi-s3-backup-secret
  csi.storage.k8s.io/node-stage-secret-namespace: csi-s3-storage
  csi.storage.k8s.io/node-publish-secret-name: csi-s3-backup-secret
  csi.storage.k8s.io/node-publish-secret-namespace: csi-s3-storage
reclaimPolicy: Delete
volumeBindingMode: Immediate

Notice that the “bucket” value may be present if we want to reuse a single bucket and treat PVCs as subfolders (recommented) or omitted (the whole line removed from the YAML) if we wanted to create a bucket per PVC as per considerations at the beginning of the post. Also, review the “reclaimPolicy” and “volumeBindingMode” values as desired for the Storage Class.

Also, note the Memory Limit to configure on the Storage Class for the GeeseFS mounter. Normally there should be no need to increase the default value unless stated in a new version of the provisioner. The default value as above is “1000”.