MAS Deployment Series: Preparing and Executing Automated MongoDB Backups

Abstract: We are continuing our series of Blog Entries to cover some MAS deployment scenarios either left out or not appropriately covered on the IBM Ansible scripts. We are going to also discuss considerations and tips and hints around these deployments. This time we are configuring automated MongoDB backups against a pre-existing Storage Class and discuss limitations and considerations as well.

References:

The related IBM documentation for the Ansible Role to backup and restore the Suite Configurations (not the same as the MongoDB database) can be found at the URL https://github.com/ibm-mas/ansible-devops/tree/master/ibm/mas_devops/roles/suite_backup_restore.

The related IBM documentation for the Ansible Role to backup and restore Applications Configurations (such as Manage) (not the same as the Manage Database) can be found at the URL https://github.com/ibm-mas/ansible-devops/tree/master/ibm/mas_devops/roles/suite_app_backup_restore.

Content:

We wanted to periodically backup a MongoDB database running inside OpenShift using the default MongoDB Community Edition that can be installed as part of the IBM Ansible install or using manual or our own Interloc scripting (not publicly available).

While there may be more than one MongoDB CE (depending on deployment options and how many MAS Core instances are associated/connected to each MongoDB CE instance), the general case being that we want to periodically and automatically backup one instance identified by its Project/Namespace in OpenShift.

The main consideration being that we need a Backup Storage Class that can be used to store the backups, typically we recommend a Storage Class that is outside of the OpenShift Cluster in terms of actual storage, such as a Cloud Provisioned Storage (such as NFS based) or a separate Storage Server (NAS, SAN, etc.) for on-premises. Also, for Cloud Storage, the recommendation is to have the actual Storage outside of the same Data Center and if possible, even in a different Region that hosts the OpenShift Cluster. Naming such Storage Class as the Backup Storage Class for the purposes of this Blog.

We are going to discuss important Backup Storage Class considerations and architectural design on a future blog post. But just wanted to mention that in a previous Blog Entry with URL: MAS Deployment Series: Configuring S3 Object Storage as an OpenShift Storage Class, we discussed the creation of a Storage Class suited for backups that can be configured in IBM Cloud as a “Global” S3/COS and therefore is not tied to specific geography constraints.

Also, the Storage Class should support Dynamic Provisioning (i.e. automatically creating Persistent Volumes or PVs from Persistent Volume Claims or PVCs).

We are also going to automatically execute the Backups using an OpenShift Cron Job that periodically executes the backup using the “mongodump” utility which basically Dumps ALL the databases present in the local MongoDB database. Documentation for the “mongodump” utility can be found at https://www.mongodb.com/docs/database-tools/mongodump/.

The main dependencies and moving parts of the backup process are:

1. A Service Account and Cluster Role + Cluster Role Binding for authorizing the backups privileges required on the target MongoDB CE Project/Namespace.
2. The PVC contains the actual backup files. If the Storage Class supports dynamic provisioning as requested, then the PV will be dynamically created and depending on the strategy for the Storage Class a subfolder or other element will be created for the PV.
3. We always try to execute the backup on the Primary Replica (and therefore a command to find it is also executed).
4. Given that the MongoDB Pods are created by the Operator, and that we have no easy way to point the Backup Storage Class to a PVC that can be mounted directly on the Pod, then the actual Backup Pod issues commands to create the backup file on a temporary folder to the MongoDB Pod then to copies it to the PVC/PV Mounted Folder The Backup Pod where the “oc” Commands are being executed which is part of the OpenShift Cron Job execution.
5. All that sequence of commands is executed by the Backup Pod via Shell Chaining using the “&&” “logical AND” sequential operator where for a command to be executed, the previous one must have returned a zero-value return code (and have been therefore executed successfully).

While the above configurations rely on several settings, we are going to define environment variables for most of them so they can be configured appropriately and flexibly enough for future executions, these are:

MONGODB_BACKUPS_STORAGE_CLASS_NAME=<storage-class-name>

Will be the Storage Class Name to use to create the Persistent Volume Claims and Persistent Volumes as the Backup Destination. It needs to exist beforehand in the Cluster and ideally support Dynamic Provisioning and the “ReadWriteMany” Access Mode.

MONGODB_BACKUPS_STORAGE_CLASS_ACCESS_MODE=<access-mode>

As stated above, ideally “ReadWriteMany” (as supported by the Storage Class) although “ReadWriteOnce” should be usable as well. But if re-using the same Storage Class to backup other targets such as DB2 databases, then the “ReadWriteMany” Access Mode becomes more important.

MONGODB_NAMESPACE=<mongodb-namespace>

The Namespace (or Project) where the target MongoDB CE is installed.

MONGODB_ADMIN_PASSWORD=<password>

The password for the “admin” user for the target MongoDB instance. The assumption is that we are using the MongoDB CE included by default running in OpenShift and in that instance only the “admin” user is configured. Other users can be configured as well which is outside of the scope of this Blog entry.

MONGODB_BACKUPS_SCHEDULE=<backup-schedule>

Specifies when the backups (via OpenShift Cron Job) are run. The format of this field is intended for the OpenShift Cron Job Specification "schedule", which uses the standard Linux Cron "crontab" specification (see https://en.wikipedia.org/wiki/Cron). A god value for daily backups could be “15 5 * * *” which means every day at 5:15 am.

MONGODB_BACKUPS_RETENTION_PERIOD=<backups-retention-period>

The Backups Retention Period. The default value is "+10" (10 days). The value is meant to be passed to a "find" command line that deletes older Backup Files by using this value after the "mtime" parameter. See the "find" man page or documentation for possible values and meaning.

MONGODB_BACKUPS_STORAGE_PVC_NAME=<pvc-name>

The PVC name to use to mount in the MongoDB Pods to execute the backups. A good default name could be "mongodb-ce-backups-pvc".

MONGODB_BACKUPS_FOLDER_PATH=<filesystem-path>

The Mount Point used by the OpenShift Cron Job to store the backup files (and also associated to the PVC). This value is largely artificial and here for completeness only, a good default could be "/mnt/backup".

MONGODB_BACKUPS_POD_TEMPORARY_FOLDER_PATH=<filesystem-path>

The temporary path in the actual MongoDB Pod where the backup file is going to be created and copied from, then later removed. This needs to be in a writable file system for the Pod and therefore, modifying it from the default value of "/tmp/backup" is not recommended. This is largely because we cannot mount the Backup PVC directly on the MongoDB Pods.

MONGODB_BACKUPS_BACKUP_STORAGE_REQUEST_SIZE=<backup-storage-size>

The Persistent Volume Claim size to request when creating the Storage to attach for the Backups destination. Expressed in Units that a PVC Size can take (Ti, Gi, Mi, Ki, etc.) The value can be higher or lower depending on target database size, retention policy and backup frequency; but generally, considering MongoDB dumps are in the small-ish side of things there is no need to make it overly large. The default value is "50Gi" which should be enough for most cases with space to spare.

MONGODB_BACKUPS_SUCCESSFUL_JOB_HISTORY_LIMIT=<number>

The number of successful OpenShift Cron Jobs Pods to leave on the history so they can be seen and their logs also visualized. The default number we typically use is "3".

MONGODB_BACKUPS_FAILED_JOB_HISTORY_LIMIT=<number>

The number of failed OpenShift Cron Jobs Pods to leave on the history so they can be seen and their logs also visualized. Normally if an execution fails for any reason, it gets retried right away so there should be a good execution afterwards in case of temporary failures. Considering that, the default number we like to use is "1". However, in case of permanent failures (misconfigurations, lack of space, expired certs, etc.) then the Cron Job waits an increasing amount of time between successive executions.

Once defined the above environment variables, we create the following Service Account, Cluster Role and Cluster Role Binding to give privileges enough to the Backup Cron Job and Pods:

apiVersion: v1

kind: ServiceAccount

metadata:

  name: mongodb-ce-backups-service-account

  namespace: ${MONGODB_NAMESPACE}

kind: ClusterRole

apiVersion: rbac.authorization.k8s.io/v1

metadata:

  namespace: ${MONGODB_NAMESPACE}

  name: mongodb-ce-backups

rules:

- apiGroups: [""]

  resources: ["pods", "pods/log"]

  verbs: ["get", "list"]

- apiGroups: [""]

  resources: ["pods/exec"]

  verbs: ["get", "create"]

- apiGroups: ["mongodbcommunity.mongodb.com"]

  resources: ["*"]

  verbs: ["list", "get", "watch", "update", "patch"]

- apiGroups: ["mongodb.com"]

  resources: ["*"]

  verbs: ["list", "get", "watch", "update", "patch"]

apiVersion: rbac.authorization.k8s.io/v1

kind: ClusterRoleBinding

metadata:

  name: mongodb-ce-backups  

subjects:

  - kind: ServiceAccount

    name: mongodb-ce-backups-service-account

    namespace: ${MONGODB_NAMESPACE}

roleRef:

  kind: ClusterRole

  name: mongodb-ce-backups

  apiGroup: rbac.authorization.k8s.io

 Next, we submit the PVC for creation:

kind: PersistentVolumeClaim

apiVersion: v1

metadata:

  name: ${MONGODB_BACKUPS_STORAGE_PVC_NAME}

  namespace: ${MONGODB_NAMESPACE}

spec:

  accessModes:

    - ${MONGODB_BACKUPS_STORAGE_CLASS_ACCESS_MODE}

  resources:

    requests:

      storage: ${MONGODB_BACKUPS_BACKUP_STORAGE_REQUEST_SIZE}

  storageClassName: ${MONGODB_BACKUPS_STORAGE_CLASS_NAME}

  volumeMode: Filesystem

Once submitted, wait for the “Status.Phase” for the PVC to return “Bound” to continue.

Run the following shell commands to extract configuration settings we will need and set them into environment variables as well:

export MONGODB_CE_DB_NAME=$(oc get mongodbcommunity --ignore-not-found -o=jsonpath='{.items[0].metadata.name}')

export MONGODB_CE_DB_NAME_LOWER="${MONGODB_CE_DB_NAME,,}"

export MONGODB_ADMIN_PASSWORD=$(oc extract secret/${MONGODB_CE_DB_NAME}-admin-password --keys=password --to=-)

export MONGODB_CA_PATH=$(oc exec -it ${MONGODB_CE_DB_NAME}-0 -c mongod -- bash -c "cat /data/automation-mongod.conf" | grep CAFile | cut -d ":" -f 2 | xargs | tr -d '\r')

mongo_url_multi=$(oc get MongoDBCommunity -n ${MONGODB_NAMESPACE} -o=jsonpath="{.items[0].status.mongoUri}" | awk -F:// '{print $2}')

mongo_url_splitted=(${mongo_url_multi//,/ })

export MONGODB_URL_0="${mongo_url_splitted[0]}"

export MONGODB_URL_0_PORT=$(echo ${mongo_url_splitted[0]} | awk -F'[:/]' '{print $2}')

read MONGODB_CE_PRIMARY_HOSTNAME < <(oc exec -qt ${MONGODB_CE_DB_NAME}-0 -n ${MONGODB_NAMESPACE} -c mongod -- bash -c "mongosh 'mongodb://admin:${MONGODB_ADMIN_PASSWORD}@${MONGODB_URL_0}' --tls --tlsCAFile '${MONGODB_CA_PATH}' --quiet --eval 'rs.status().members.find(r=>r.state===1).name'")

export MONGODB_CE_PRIMARY_PORT=$(echo ${MONGODB_CE_PRIMARY_HOSTNAME} | cut -d ":" -f 2 | xargs)

export MONGODB_CE_PRIMARY_HOSTNAME=$(echo ${MONGODB_CE_PRIMARY_HOSTNAME} | cut -d ":" -f 1 | xargs)

export MONGODB_CE_PRIMARY_POD=$(echo ${MONGODB_CE_PRIMARY_HOSTNAME} | cut -d "." -f 1 | xargs)

Note: Watch the contents of these environment variables as none should be empty afterwards. If any is empty, there was an issue that needs to be corrected. The command “echo $<envvar-name>” can be used to check.

Note: The value of MONGODB_CE_PRIMARY_POD must conform to the concatenation of ${MONGODB_CE_DB_NAME} followed by dash then a number typically from zero to nine. Example: “mas-mongodb-ce-0”.

Next, copy and paste the following file into the filesystem for the workstation:

apiVersion: batch/v1

kind: CronJob

metadata:

  name: mongodb-ce-backup-${MONGODB_CE_DB_NAME_LOWER}

  namespace: ${MONGODB_NAMESPACE}

spec:

  schedule: "${MONGODB_BACKUPS_SCHEDULE}"

  concurrencyPolicy: Forbid

  startingDeadlineSeconds: 200

  suspend: false

  successfulJobsHistoryLimit: ${MONGODB_BACKUPS_SUCCESSFUL_JOB_HISTORY_LIMIT}

  failedJobsHistoryLimit: ${MONGODB_BACKUPS_FAILED_JOB_HISTORY_LIMIT}    

  jobTemplate:                  

    spec:

      template:

        metadata:

          name: mongodb-ce-backup-${MONGODB_CE_DB_NAME_LOWER}

          namespace: ${MONGODB_NAMESPACE}

          labels:

            type: "mongodb-ce-backup"

            dbname: "${MONGODB_CE_DB_NAME}"

        spec:

          serviceAccountName: mongodb-ce-backups-service-account

          containers:

          - name: mongodb-ce-backup

            image: registry.redhat.io/openshift4/ose-cli:latest

            imagePullPolicy: IfNotPresent

            command: ["/bin/bash", "-c" ,"read MONGODB_CE_PRIMARY_HOSTNAME < <(oc exec -q ${MONGODB_CE_DB_NAME}-0 -n ${MONGODB_NAMESPACE} -c mongod -- bash -c \"mongosh 'mongodb://admin:${MONGODB_ADMIN_PASSWORD}@${MONGODB_URL_0}' --tls --tlsCAFile '${MONGODB_CA_PATH}' --quiet --eval 'rs.status().members.find(r=>r.state===1).name'\") && export MONGODB_CE_PRIMARY_PORT=$(echo \"${MONGODB_CE_PRIMARY_HOSTNAME@E}\" | cut -d \":\" -f 2) && export MONGODB_CE_PRIMARY_HOSTNAME=$(echo \"${MONGODB_CE_PRIMARY_HOSTNAME@E}\" | cut -d \":\" -f 1) && export MONGODB_CE_PRIMARY_POD=$(echo \"${MONGODB_CE_PRIMARY_HOSTNAME@E}\" | cut -d \".\" -f 1) && export TIMESTAMP=$(date +\"%Y%m%d%H%M%S\") && export BACKUP_FILENAME=\"${TIMESTAMP@E}-${MONGODB_CE_DB_NAME}.archive.gz\" && export BACKUP_PATH_FILENAME=\"${MONGODB_BACKUPS_POD_TEMPORARY_FOLDER_PATH}/${BACKUP_FILENAME@E}\" && echo \"MONGODB_CE_PRIMARY_HOSTNAME: ${MONGODB_CE_PRIMARY_HOSTNAME@E}\" && echo \"MONGODB_CE_PRIMARY_PORT: ${MONGODB_CE_PRIMARY_PORT@E}\" && echo \"MONGODB_CE_PRIMARY_POD: ${MONGODB_CE_PRIMARY_POD@E}\" && echo \"TIMESTAMP: ${TIMESTAMP@E}\" && echo \"BACKUP_FILENAME: ${BACKUP_FILENAME@E}\" && echo \"BACKUP_PATH_FILENAME: ${BACKUP_PATH_FILENAME@E}\" && oc exec ${MONGODB_CE_PRIMARY_POD@E} -c mongod -n \"${MONGODB_NAMESPACE}\" -- bash -c \"mkdir -p '${MONGODB_BACKUPS_POD_TEMPORARY_FOLDER_PATH}' && rm -f '${MONGODB_BACKUPS_POD_TEMPORARY_FOLDER_PATH}/'* && mongodump --host=${MONGODB_CE_PRIMARY_HOSTNAME@E} --port=${MONGODB_CE_PRIMARY_PORT@E} --username=admin --password=${MONGODB_ADMIN_PASSWORD} --authenticationDatabase=admin --ssl --sslCAFile=${MONGODB_CA_PATH} --gzip --archive=${BACKUP_PATH_FILENAME@E}\" && oc cp -n \"${MONGODB_NAMESPACE}\" -c mongod ${MONGODB_CE_PRIMARY_POD@E}:${BACKUP_PATH_FILENAME@E} ${MONGODB_BACKUPS_FOLDER_PATH}/${BACKUP_FILENAME@E} && oc exec ${MONGODB_CE_PRIMARY_POD@E} -c mongod -n \"${MONGODB_NAMESPACE}\" -- bash -c \"rm -f '${BACKUP_PATH_FILENAME@E}'\" && /usr/bin/find '${MONGODB_BACKUPS_FOLDER_PATH}' -name *.archive.gz -type f -mtime ${MONGODB_BACKUPS_RETENTION_PERIOD} -print -exec rm -f {} \\;"]

            volumeMounts:

            - mountPath: "${MONGODB_BACKUPS_FOLDER_PATH}"

              name: "${MONGODB_BACKUPS_STORAGE_PVC_NAME}"

            env:

            - name: MONGODB_NAMESPACE

              value: "${MONGODB_NAMESPACE}"

            - name: MONGODB_CE_DB_NAME

              value: "${MONGODB_CE_DB_NAME}"

            - name: MONGODB_ADMIN_PASSWORD

              value: "${MONGODB_ADMIN_PASSWORD}"

            - name: MONGODB_BACKUPS_FOLDER_PATH

              value: "${MONGODB_BACKUPS_FOLDER_PATH}"

            - name: POD_EXECUTION_TIMEOUT

              value: "${MONGODB_BACKUPS_SCHEDULE}"

            - name: MONGODB_BACKUPS_STORAGE_PVC_NAME

              value: "${MONGODB_BACKUPS_STORAGE_PVC_NAME}"

            - name: MONGODB_BACKUPS_RETENTION_PERIOD

              value: "${MONGODB_BACKUPS_RETENTION_PERIOD}"

            - name: MONGODB_BACKUPS_SUCCESSFUL_JOB_HISTORY_LIMIT

              value: "${MONGODB_BACKUPS_SUCCESSFUL_JOB_HISTORY_LIMIT}"

            - name: MONGODB_BACKUPS_FAILED_JOB_HISTORY_LIMIT

              value: "${MONGODB_BACKUPS_FAILED_JOB_HISTORY_LIMIT}"

          restartPolicy: Never

          volumes:

            - name: "${MONGODB_BACKUPS_STORAGE_PVC_NAME}"

              persistentVolumeClaim:

                claimName: "${MONGODB_BACKUPS_STORAGE_PVC_NAME}"

Once saved, run the following command to substitute the environment variables on the file with their values and generate another file with the substituted values instead of the references:

envsubst < file-name-with-envvars.yaml > file-name-substituted.yaml

Where the file-name-with-envvars.yaml is the file that was saved from above and the file-name-substituted.yaml is the generated file with the environment variables substituted.

Note: Any environment variable ending with “@E” are meant to take effect while executing and will not be substituted by “envsubst” when generating the substituted file.

Review the file-name-substituted.yaml file for correctness then submit to the cluster using:

oc apply -f file-name-substituted.yaml

 Finally, we are going to try to explain what the very long “command” contents of the OpenShift Cron Job mean step by step (remember they are separated by “&&”):

1. Derives the MONGODB_CE_PRIMARY_HOSTNAME by extracting from a mongosh execution inside the first MongoDB CE Pod.
2. Derives MONGODB_CE_PRIMARY_PORT by parsing the previously extracted MONGODB_CE_PRIMARY_HOSTNAME.
3. Derives MONGODB_CE_PRIMARY_HOSTNAME by parsing the previously extracted MONGODB_CE_PRIMARY_HOSTNAME.
4. Derives MONGODB_CE_PRIMARY_POD by parsing the previously extracted MONGODB_CE_PRIMARY_HOSTNAME.
5. Derives TIMESTAMP as current full date and time to use for generated file names.
6. Derives BACKUP_FILENAME using the previous timestamp and other values.
7. Derives BACKUP_PATH_FILENAME using the previous BACKUP_FILENAME and path.
8. Print values to output to help understanding the results of the above derivation on the Pod output
9. Creates (if not exist) the temporary destination folder inside the MongoDB Primary Pod as specified by MONGODB_BACKUPS_POD_TEMPORARY_FOLDER_PATH
10. Deletes any pre-existing files on the destination temporary folder
11. Calls “mongodump” to dump all the databases to the file on the temporary destination folder (gzipped)
12. Using “oc cp”; copies the generated dump file from the temporary destination folder on the MongoDB Pod to the folder specified by the MONGODB_BACKUPS_FOLDER_PATH variable on the local Pod (the Cron Job Pod which has the PVC mounted in the intended path).
13. Deletes the generated dump file from the temporary destination folder on the MongoDB Pod
14. Searches MongoDB Backup files (names ending on “.archive.gz”) on the Backup Path (PVC) that are older than the specified MONGODB_BACKUPS_RETENTION_PERIOD value and deletes them.

Note: The “env” section with the environment variables values are just for guiding/reference purposes and have no effect on the actual execution of the command.