IBM | Interloc Solutions | Maximo Implementation | IBM Maximo | Resources | Performance | Interloc | IBM Maximo 7.6 | Maximo Application Suite | Interloc Commitment | Operational Efficiency | Maximo Mapping Solutions | IBM MAS | OpenShift | Maximo Applications Suite 9.1 | RedHat
Abstract: We are continuing our series of Blog Entries to cover some MAS deployment scenarios either left out or not appropriately covered on the IBM Ansible scripts. We are going to also discuss considerations and tips and hints around these deployments. This time we are configuring automated DB2 backups against a pre-existing Storage Class and discussing limitations and considerations as well.
Due to the length of the procedure and considerations being discussed, we are splitting this blog post into two parts. This is Part 2 of 2.
References:
The related IBM documentation for the Ansible Role to backup and restore the Suite Configurations (not the same as the DB2 database) can be found at the URL https://github.com/ibm-mas/ansible-devops/tree/master/ibm/mas_devops/roles/suite_backup_restore.
The related IBM documentation for the Ansible Role to backup and restore Applications Configurations (such as Manage) (not the same as the Manage Database) can be found at the URL https://github.com/ibm-mas/ansible-devops/tree/master/ibm/mas_devops/roles/suite_app_backup_restore.
Content:
This is the continuation of the previous Blog Post named “MAS deployment series: Preparing and executing automated DB2 backups – Part 1 of 2” and we are going to resume our procedure to execute periodic DB2 backups with the actual configurations that will be submitted to the OpenShift Cluster.
Please see the previous blog post for important considerations and an introduction to the problem and general approach.
While the configurations discussed previously rely on several settings, we are going to define environment variables for most of them so they can be configured appropriately and flexibly enough for future executions, some of these are:
DB2_BACKUPS_STORAGE_CLASS_NAME=<storage-class-name>
Will be the Storage Class Name to use to create the Persistent Volume Claims and Persistent Volumes as the Backup Destination. It needs to exist beforehand in the Cluster and ideally support Dynamic Provisioning and the “ReadWriteMany” Access Mode.
DB2_BACKUPS_STORAGE_CLASS_ACCESS_MODE=<access-mode>
As stated above, it should be “ReadWriteMany” (as supported by the Storage Class)
DB2_NAMESPACE=<db2-namespace>
The Namespace (or Project) where the target DB2 database is installed.
DB2_DB_NAME=<db2-database-name>
The Database Name for the DB2 instance where to execute backups against.
DB2_CLUSTER_NAME=<db2-database-name>
The DB2U Cluster Name for the DB2 instance where to execute backups against. This can be obtained by executing a command like the following:
export DB2_CLUSTER_NAME=$(oc get db2ucluster -n ${DB2_NAMESPACE} -o=jsonpath='{range .items[*]}{@.spec.environment.database.name} {@.metadata.name}{"\n"}{end}' | grep ${DB2_DB_NAME} | awk '{print $2}')
Which should directly assign the value to the variable as intended. Notice that it depends on having the DB2_NAMESPACE and DB2_DB_NAME configured first as a pre-requisite.
DB2_BACKUPS_SCHEDULE=<backup-schedule>
Specifies when the backups (via OpenShift Cron Job) are run. The format of this field is intended for the OpenShift Cron Job Specification "schedule", which uses the standard Linux Cron "crontab" specification (see https://en.wikipedia.org/wiki/Cron). A god value for daily backups could be “15 3 * * *” which means every day at 3:15 am.
DB2_BACKUPS_FOLDER_PATH="/mnt/backup"
The backup destination for the backups generated by the DB2 utility. This is basically where the “backup” target of /spec/storage is going to be specified for the “db2ucluster” Custom Resource. More details on this later. This value is NOT intended to be configurable.
DB2_BACKUPS_PARAM_NUM_DB_BACKUPS=<db2-number-of-backups>
This parameter is used to configure the underlying DB2 database NUM_DB_BACKUPS database configuration value. Which specifies the number of backups to retain for a database before marking the oldest backup as deleted.
DB2_BACKUPS_PARAM_REC_HIS_RETENTN=<db2-number-of-backups>
This parameter is used to configure the underlying DB2 database NUM_DB_BACKUPS database configuration value. Which specifies the number of backups to retain for a database before marking the oldest backup as deleted.
DB2_ARCHIVE_LOGS_FOLDER_PATH=" /mnt/logs/archive"
The backup destination for the backups generated by the DB2 utility. This is basically where the “archivelogs” target of /spec/storage is going to be specified for the “db2ucluster” Custom Resource. More details on this later. This value is NOT intended to be configurable.
DB2_ARCHIVE_LOGS_RETENTION_PERIOD=<backups-retention-period>
The Backups Retention Period. The default value is "+10" (10 days). The value is meant to be passed to a "find" command line that deletes older Backup Files by using this value after the "mtime" parameter. See the "find" man page or documentation for possible values and meaning.
DB2_BACKUPS_BACKUP_STORAGE_REQUEST_SIZE_GB=<backup-storage-size>
The Persistent Volume Claim size to request when creating the Storage to attach for the Backups destination. Expressed in Units that a PVC Size can take (Ti, Gi, Mi, Ki, etc.) The value can be higher or lower depending on target database size, retention policy and backup frequency. In our case, we are using a formula to calculate:
DB2_BACKUPS_BACKUP_STORAGE_REQUEST_SIZE_GB = DB2_CURRENT_SIZE_GB * FACTOR
Where FACTOR as discussed in the previous Part 1 of 2 of this Blog is usually “*20”. Therefore, we are using to estimate 20 times the current DB2 database size.
The current DB2 database size (DB2_CURRENT_SIZE_GB) can be obtained by using “db2 -x select db_size/1073741824 as DB2_CURRENT_SIZE_GB from systools.stmg_dbsize_info” using the DB2 command line utility from the DB2 Pod while connected to the correct database. For this number to be as accurate as possible, it is also recommended to run “db2 \"call get_dbsize_info(?,?,?,0)\"” first.
DB2_BACKUPS_ARCHIVE_LOGS_STORAGE_REQUEST_SIZE_GB=<archive-logs-storage-size>
The Persistent Volume Claim size to request when creating the Storage to attach for the Archive Logs destination. Expressed in Units that a PVC Size can take (Ti, Gi, Mi, Ki, etc.) The value can be higher or lower depending on target database size, retention policy and backup frequency. In our case, we are using a formula to calculate:
DB2_BACKUPS_ARCHIVE_LOGS_STORAGE_REQUEST_SIZE_GB = DB2_CURRENT_SIZE_GB * FACTOR
Where FACTOR as discussed in the previous Part 1 of 2 of this Blog is usually “/2”. Therefore, we are using to estimate as half the current DB2 database size.
The current DB2 database size (DB2_CURRENT_SIZE_GB) can be obtained as described in the above paragraph related to the previous size for backups.
DB2_BACKUPS_SUCCESSFUL_JOB_HISTORY_LIMIT=<number>
The number of successful OpenShift Cron Jobs Pods to leave on the history so they can be seen and their logs also visualized. The default number we typically use is "3".
DB2_BACKUPS_FAILED_JOB_HISTORY_LIMIT=<number>
The number of failed OpenShift Cron Jobs Pods to leave on the history so they can be seen and their logs also visualized. Normally if an execution fails for any reason, it gets retried right away so there should be a good execution afterwards in case of temporary failures. Considering that, the default number we like to use is "1". However, in case of permanent failures (misconfigurations, lack of space, expired certs, etc.) then the Cron Job waits an increasing amount of time between successive executions.
Note: What follows to configure the DB2 database will restart the DB2 Pods and shut down the database after reconfiguration so please consider an outage for the configurations that follow.
Once armed with all the sizes and variables, we can proceed to prepare the DB2 database for backups, to do so, the first step is to ensure there is a section of the “DB2UCluster” custom resource that we need to edit or add to ensure the Backup and Archive Logs destinations are correctly configured and mounted in the DB2 Pods.
Those are like what is being shown in the example below:
storage:
- name: backup
spec:
accessModes:
- ReadWriteMany
resources:
requests:
storage: 300Gi
storageClassName: backups-nfs-storage
type: create
- name: archivelogs
spec:
accessModes:
- ReadWriteMany
resources:
requests:
storage: 150Gi
storageClassName: backups-nfs-storage
type: create
Again, above is just an example, where the Sizes and Storage Class Name are meant to be replaced.
To create them if they do not exist, we can use the following commands:
For the Main Backups Destination:
oc patch db2ucluster ${DB2_CLUSTER_NAME} -n ${DB2_NAMESPACE} --type 'json' -p '[{\"op\": \"add\", \"path\": \"/spec/storage/-\", \"value\": {\"name\": \"backup\", \"spec\": {\"accessModes\": [\"${DB2_BACKUPS_STORAGE_CLASS_ACCESS_MODE}\"], \"resources\": {\"requests\": {\"storage\": \"${DB2_BACKUPS_BACKUP_STORAGE_REQUEST_SIZE_GB}Gi\"}},\"storageClassName\": \"${DB2_BACKUPS_STORAGE_CLASS_NAME}\"},\"type\": \"create\"} }]'
and for the Archive Logs:
oc patch db2ucluster ${DB2_CLUSTER_NAME} -n ${DB2_NAMESPACE} --type 'json' -p '[{\"op\": \"add\", \"path\": \"/spec/storage/-\", \"value\": {\"name\": \"archivelogs\", \"spec\": {\"accessModes\": [\"${DB2_BACKUPS_STORAGE_CLASS_ACCESS_MODE}\"], \"resources\": {\"requests\": {\"storage\": \"${DB2_BACKUPS_ARCHIVE_LOGS_STORAGE_REQUEST_SIZE_GB}Gi\"}},\"storageClassName\": \"${DB2_BACKUPS_STORAGE_CLASS_NAME}\"},\"type\": \"create\"} }]'
Both commands depend on variables that should have been defined with values as per above.
Once submitted the DB2U Operator should modify the database Pods accordingly and will mount
Next, we are going to configure the database itself and to do so we can execute the following commands:
db2 update database configuration for ${DB2_DB_NAME} using NUM_DB_BACKUPS ${DB2_BACKUPS_PARAM_NUM_DB_BACKUPS}
db2 update database configuration for ${DB2_DB_NAME} using REC_HIS_RETENTN ${DB2_BACKUPS_PARAM_REC_HIS_RETENTN}
db2 update database configuration for ${DB2_DB_NAME} using AUTO_DEL_REC_OBJ ${DB2_BACKUPS_PARAM_AUTO_DEL_REC_OBJ}
db2 update database configuration for ${DB2_DB_NAME} using LOGARCHMETH1 DISK:${DB2_ARCHIVE_LOGS_FOLDER_PATH}
Notice these should be executed in the DB2 Pod using the “db2inst1” user and the proper environment variables should be used for them as well. We could also use “oc” to execute directly but for that we need to resolve the DB2_POD_NAME which is typically in the form of “c-${DB2_CLUSTER_NAME}-db2u-0”
An example of the command using the DB2_POD_NAME follows:
oc exec ${DB2_POD_NAME} -n ${DB2_NAMESPACE} -c db2u --request-timeout=15m -- su - db2inst1 -c 'db2 update database configuration for ${DB2_DB_NAME} using NUM_DB_BACKUPS ${DB2_BACKUPS_PARAM_NUM_DB_BACKUPS}'
That would be for the first of the four above commands. The other three should be similar.
After that we should restart the database to take the new settings and copy the Archive Logs from the default location to the new in the Backups Storage Class, using:
oc exec ${DB2_POD_NAME} -n ${DB2_NAMESPACE} -c db2u --request-timeout=15m -- su - db2inst1 -c 'db2stop force && cp -rufv /mnt/bludata0/db2/archive_log/* ${DB2_ARCHIVE_LOGS_FOLDER_PATH}/'
It is recommended to manually review the above copy and delete the files that were copied over from the original Archive Logs source folder at “/mnt/bludata0/db2/archive_log/”.
Finally, we are ready to start defining the Backup Cron Job and to do so, we create the following Service Account, Cluster Role and Cluster Role Binding to give privileges enough to the Backup Cron Job and related Pods:
apiVersion: v1
kind: ServiceAccount
metadata:
name: db2-backups-service-account
namespace: ${DB2_NAMESPACE}
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
namespace: ${DB2_NAMESPACE}
name: db2-backups
rules:
- apiGroups: [""]
resources: ["pods", "pods/log"]
verbs: ["get", "list"]
- apiGroups: [""]
resources: ["pods/exec"]
verbs: ["get", "create"]
- apiGroups: ["db2u.databases.ibm.com"]
resources: ["*"]
verbs: ["list", "get", "watch", "update", "patch"]
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: db2-backups
subjects:
- kind: ServiceAccount
name: db2-backups-service-account
namespace: ${DB2_NAMESPACE}
roleRef:
kind: ClusterRole
name: db2-backups
apiGroup: rbac.authorization.k8s.io
Next, copy and paste the following file into the filesystem for the workstation:
apiVersion: batch/v1
kind: CronJob
metadata:
name: db2-backup-${DB2_DB_NAME}
namespace: ${DB2_NAMESPACE}
spec:
schedule: "${DB2_BACKUPS_SCHEDULE}"
concurrencyPolicy: Forbid
startingDeadlineSeconds: 200
suspend: false
successfulJobsHistoryLimit: ${DB2_BACKUPS_SUCCESSFUL_JOB_HISTORY_LIMIT}
failedJobsHistoryLimit: ${DB2_BACKUPS_FAILED_JOB_HISTORY_LIMIT}
jobTemplate:
spec:
template:
metadata:
name: db2-backup-${DB2_DB_NAME}
namespace: ${DB2_NAMESPACE}
labels:
type: "db2-backup"
dbname: "${DB2_DB_NAME}"
spec:
serviceAccountName: db2-backups-service-account
containers:
- name: db2-backup
image: registry.redhat.io/openshift4/ose-cli:latest
imagePullPolicy: IfNotPresent
command: ["/bin/bash", "-c" ,"oc exec c-$(oc get db2ucluster -o=jsonpath='{range .items[*]}{@.spec.environment.database.name} {@.metadata.name}{\"\\n\"}{end}' | grep ${DB2_DB_NAME} | awk '{print $2}')-db2u-0 -c db2u --request-timeout='15m' -- su - db2inst1 -c 'db2 -v \"backup database ${DB2_DB_NAME} on all dbpartitionnums online to ${DB2_BACKUPS_FOLDER_PATH} include logs without prompting\" && mkdir -p ${DB2_ARCHIVE_LOGS_FOLDER_PATH}/db2inst1/${DB2_DB_NAME} && /usr/bin/find ${DB2_ARCHIVE_LOGS_FOLDER_PATH}/db2inst1/${DB2_DB_NAME} -name *.LOG -type f -mtime ${DB2_ARCHIVE_LOGS_RETENTION_PERIOD} -print -exec rm -f {} \\;'"]
env:
- name: DB2_DB_NAME
value: ${DB2_DB_NAME}
- name: DB2_BACKUPS_FOLDER_PATH
value: "${DB2_BACKUPS_FOLDER_PATH}"
- name: POD_EXECUTION_TIMEOUT
value: "${POD_EXECUTION_TIMEOUT}"
- name: DB2_ARCHIVE_LOGS_FOLDER_PATH
value: "${DB2_ARCHIVE_LOGS_FOLDER_PATH}"
- name: DB2_ARCHIVE_LOGS_RETENTION_PERIOD
value: "${DB2_ARCHIVE_LOGS_RETENTION_PERIOD}"
restartPolicy: Never
Once saved, run the following command to substitute the environment variables on the file with their values and generate another file with the substituted values instead of the references:
envsubst < file-name-with-envvars.yaml > file-name-substituted.yaml
Where the file-name-with-envvars.yaml is the file that was saved from above and the file-name-substituted.yaml is the generated file with the environment variables substituted.
Note: Any environment variable ending with “@E” are meant to take effect while executing and will not be substituted by “envsubst” when generating the substituted file.
Review the file-name-substituted.yaml file for correctness then submit to the cluster using:
oc apply -f file-name-substituted.yaml
Finally, we are going to try to explain what the very long “command” contents of the OpenShift Cron Job mean step by step (remember they are separated by “&&”):
- Finds the actual name of the DB2 Pod and inside:
- Executes the backup using “backup database ${DB2_DB_NAME} on all dbpartitionnums online to ${DB2_BACKUPS_FOLDER_PATH} include logs without prompting”.
- Creates the Archive Logs folder if it does not exist so the next command does not fail due to folder not existing.
- Searches Archive Logs folder for files older than the specified DB2_ARCHIVE_LOGS_RETENTION_PERIOD value and deletes them.
Note: The “env” section with the environment variables values are just for guiding/reference purposes and have no effect on the actual execution of the command.
