While the IBM Ansible scripting covers the installation of the AI broker, at the time of writing, there are several dependencies and caveats that are not installed by default and the documentation is insufficient to completely do a successful installation.
Additionally, the script was developed with the “Open Data Hub” Operator version v2.11.1 in mind. However, time has passed, and the actual version being installed at the time of writing is v2.23.0 which causes issues with the script execution. Even if we could work around these issues, installing version v2.23.0 causes incompatibility with some of the installed CRDs that make the installation fail later. The root of the issue appears to be around the inability of one of the IBM Ansible python functions to create Operator Subscriptions supporting the “spec.startingCSV” tag which allows to install a specific version instead of the latest.
Also, there are important caveats around the plans to use on the WatsonX AI service as we found out when trying to train the model and after consulting with IBM support. Basically, if we use the “Lite” plan as stated on https://www.ibm.com/docs/en/masv-and-l/maximo-manage/continuous-delivery?topic=setup-create-watsonx-api-key then the model will refuse to train as a rate limit will be hit and cause errors. IBM support confirmed that the “Essentials” plan can be used and won’t have the issues that the “Lite” plan has.
First, we created a Project, Service ID, Deployment Space with a Plan (“Essentials” recommended, which is a paid plan), and setup the API Key as stated in the IBM link that appears in the previous paragraph. We are not documenting these steps in greater detail here as they worked without much tinkering although the navigation around the WatsonX.ai site was confusing at times. The output of this task should be the following settings that we store as Environment variables to be used by the IBM scripting:
export MAS_AIBROKER_WATSONXAI_APIKEY="<value-omitted>" ## The WatsonX API key
export MAS_AIBROKER_WATSONXAI_URL="https://us-south.ml.cloud.ibm.com" ## The WatsonX API URL
export MAS_AIBROKER_WATSONXAI_PROJECT_ID="5aba7552-9c38-43c0-9e86- 9ea730f0c323" ## The WatsonX Project ID
Notice that we chose to deploy our testing WatsonX instance in Dallas and the Endpoint URL reflects that choice.
First, there are several dependencies that are required to be installed which are not covered in the IBM Ansible scripts, basically these are:
A) A relational database (we are using a new MariaDB instance; but potentially another pre-existing instance could be used)
B) An S3 (Object Storage) set of buckets (we are using a new Minio instance; but potentially another pre-existing S3 service could be used).
The above is meant to populate several Environment Variables used by the Ansible Galaxy collection (more on it later).
Regarding the MariaDB and Minio instances, we intended to use specific instances of each for each MAS Core, so data and files don’t get mixed between instances. Therefore, we used project names containing our intended (testing) MAS Core Instance ID, in our case “p01-main”. Separate projects and installs could be used for different MAS Core Instances and the Project Names should follow suit, in our case:
The MariaDB project/namespace was named “mas-p01-main-aibroker-mariadb”.
The Minio project/namespace was named “mas-p01-main-aibroker-minio”.
The actual AI broker project generated for the mentioned MAS Core Instance ID is going to be “mas-p01-main-aibroker” so project naming should be all related.
Therefore, to accomplish that, we submitted the following YAMLs to the Cluster:
For the MariaDB instance:
---
apiVersion: v1
kind: Namespace
metadata:
name: mas-p01-main-aibroker-mariadb
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: mariadb-pvc
namespace: mas-p01-main-aibroker-mariadb
labels:
app: mariadb-instance
component: data-science-pipelines
spec:
accessModes:
- ReadWriteOnce
storageClassName: ocs-storagecluster-cephfs
resources:
requests:
storage: 20Gi
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: ds-pipelines-mariadb-sa-instance
namespace: mas-p01-main-aibroker-mariadb
labels:
app: mariadb-instance
component: data-science-pipelines
---
kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
name: mariadb-instance
namespace: mas-p01-main-aibroker-mariadb
spec:
podSelector:
matchLabels:
app: mariadb-instance
component: data-science-pipelines
ingress:
- ports:
- protocol: TCP
port: 3306
from:
- podSelector:
matchLabels:
app.kubernetes.io/name: data-science-pipelines-operator
namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: opendatahub
- podSelector:
matchLabels:
app: ds-pipeline-instance
component: data-science-pipelines
namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: mas-p01-main-aibroker
- podSelector:
matchLabels:
app: ds-pipeline-metadata-grpc-instance
component: data-science-pipelines
namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: mas-p01-main-aibroker
policyTypes:
- Ingress
---
kind: Secret
apiVersion: v1
metadata:
name: mariadb-instance
namespace: mas-p01-main-aibroker-mariadb
stringData:
password: <mariadb-password>
type: Opaque
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: mariadb-instance
namespace: mas-p01-main-aibroker-mariadb
labels:
app: mariadb-instance
component: data-science-pipelines
dspa: instance
spec:
strategy:
type: Recreate
selector:
matchLabels:
app: mariadb-instance
component: data-science-pipelines
dspa: instance
template:
metadata:
labels:
app: mariadb-instance
component: data-science-pipelines
dspa: instance
spec:
serviceAccountName: ds-pipelines-mariadb-sa-instance
containers:
- name: mariadb
image: registry.redhat.io/rhel8/mariadb-103:1-188
ports:
- containerPort: 3306
readinessProbe:
exec:
command:
- /bin/sh
- "-i"
- "-c"
- >-
MYSQL_PWD=$MYSQL_PASSWORD mysql -h 127.0.0.1 -u $MYSQL_USER -D
$MYSQL_DATABASE -e 'SELECT 1'
failureThreshold: 3
initialDelaySeconds: 5
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
livenessProbe:
failureThreshold: 3
initialDelaySeconds: 30
periodSeconds: 10
successThreshold: 1
tcpSocket:
port: 3306
timeoutSeconds: 1
env:
- name: MYSQL_USER
value: "mariadb"
- name: MYSQL_PASSWORD
valueFrom:
secretKeyRef:
key: password
name: "mariadb-instance"
- name: MYSQL_DATABASE
value: "kmpipeline"
- name: MYSQL_ALLOW_EMPTY_PASSWORD
value: "true"
resources:
requests:
cpu: 300m
memory: 800Mi
limits:
cpu: "1"
memory: 1Gi
volumeMounts:
- name: mariadb-persistent-storage
mountPath: /var/lib/mysql
volumes:
- name: mariadb-persistent-storage
persistentVolumeClaim:
claimName: mariadb-pvc
---
apiVersion: v1
kind: Service
metadata:
name: mariadb-instance
namespace: mas-p01-main-aibroker-mariadb
labels:
app: mariadb-instance
component: data-science-pipelines
spec:
ports:
- name: http
port: 3306
protocol: TCP
targetPort: 3306
selector:
app: mariadb-instance
component: data-science-pipelines
Notice that we have put some values above in bold, to signify that these will match and must follow the configuration for the target environment as mentioned. Also, the MariaDB password will have to be filled up with a value as well.
The Storage Class could be any Storage Class that offers ReadWriteOnce capabilities. ReadWriteMany could be used as well although it is not needed. Size may need to be adjusted as well.
As a result, we ended up with the following set of related environment variables that will make the input to the IBM Ansible script:
export MAS_AIBROKER_DB_HOST="mariadb-instance.mas-p01-main-aibroker- mariadb.svc.cluster.local" # Take it from the Service on the project
export MAS_AIBROKER_DB_PORT="3306" # Take it from the Service on the
project
export MAS_AIBROKER_DB_USER="mariadb" # Take it from the environment
variables on the Deployment
export MAS_AIBROKER_DB_DATABASE="kmpipeline" # Take it from the environment variables on the Deployment
export MAS_AIBROKER_DB_SECRET_NAME="mariadb-instance" # Take it from the environment variables on the Deployment
export MAS_AIBROKER_DB_SECRET_VALUE="<mariadb-password>" # Take it from the Secret as specified above
For the Minio instance:
---
apiVersion: v1
kind: Namespace
metadata:
name: mas-p01-main-aibroker-minio
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: minio-pvc
namespace: mas-p01-main-aibroker-minio
spec:
accessModes:
- ReadWriteOnce
volumeMode: Filesystem
storageClassName: ocs-storagecluster-cephfs
resources:
requests:
storage: 40Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: minio
namespace: mas-p01-main-aibroker-minio
spec:
selector:
matchLabels:
app: minio
template:
metadata:
labels:
app: minio
spec:
volumes:
- name: storage
persistentVolumeClaim:
claimName: minio-pvc
containers:
- name: minio
image: quay.io/minio/minio:latest
command:
- /bin/bash
- -c
args:
- minio server /data --console-address :9090
env:
- name: MINIO_ROOT_USER
value: minio
- name: MINIO_ROOT_PASSWORD
value: <minio-password>
volumeMounts:
- mountPath: /data
name: storage
---
apiVersion: v1
kind: Service
metadata:
name: minio-service
namespace: mas-p01-main-aibroker-minio
spec:
type: ClusterIP
ports:
- name: api
port: 9000
targetPort: 9000
protocol: TCP
- name: console
port: 9090
targetPort: 9090
protocol: TCP
selector:
app: minio
---
kind: Route
apiVersion: route.openshift.io/v1
metadata:
name: minio-route
namespace: mas-p01-main-aibroker-minio
spec:
to:
kind: Service
name: minio-service
weight: 100
port:
targetPort: 9090
tls:
termination: edge
insecureEdgeTerminationPolicy: None
Notice that we have put some values above in bold, to signify that these will match and must follow the configuration for the target environment as mentioned. Also, the Minio password will have to be filled up with a value as well.
The Storage Class could be any Storage Class that offers ReadWriteOnce capabilities. ReadWriteMany could be used as well although it is not needed. Size may need to be adjusted as well.
As a result, we ended up with the following set of related environment variables that will make the input to the IBM Ansible script:
export MAS_AIBROKER_STORAGE_ACCESSKEY="minio" # Take it from the environment variables on the Deployment
export MAS_AIBROKER_STORAGE_SECRETKEY="<minio-password>" # Take it from the environment variables on the Deployment
export MAS_AIBROKER_STORAGE_HOST="minio-service.mas-p01-main-aibroker- minio.svc.cluster.local" # Take it from the Service
export MAS_AIBROKER_STORAGE_PORT="9000" # Take it from the Service (the "api" port)
export MAS_AIBROKER_STORAGE_REGION="" # Leave blank
export MAS_AIBROKER_STORAGE_PROVIDER="minio"
export MAS_AIBROKER_STORAGE_SSL=false
export MAS_AIBROKER_STORAGE_PIPELINES_BUCKET="km-pipelines"
export MAS_AIBROKER_STORAGE_TENANTS_BUCKET="km-templates"
export MAS_AIBROKER_STORAGE_TEMPLATES_BUCKET="km-tenants"
Notice that Bucket names are recommended.
Once submitted the Route, go to it and note the URL, in our case:
https://minio-route-mas-p01-main-aibroker-minio.apps.demo05.mas.interloc.cloud/
Then we should be able to login using the credentials in the Deployment above, in our case:
Username: minio
Password: <minio-password>
Next, we went to Administrator -> Buckets and created the following buckets (using default options):
km-pipelines
km-templates
km-tenants
Ater going to the object browser, these buckets will appear with zero size. We are not sure if the Buckets will be created automatically if they don’t exist therefore, we resorted to create them manually.
Next, we already had an instance of the MAS Ansible Galaxy collection installed (or a container could be used to deploy the latest image) and we had to update it and install a dependency that was not specified initially, in our case:
ansible-galaxy collection install --upgrade ibm.mas_devops
python3 -m pip install boto3
To execute the Ansible collection we need also to finish setting up some additional environment variables, in our case:
export MAS_ENTITLEMENT_KEY="<ibm-entitlement-key>" # The IBM Entitlement
key to access the IBM Container Registry, from https://myibm.ibm.com/products-services/containerlibrary
export MAS_INSTANCE_ID="p01-main" # Declare the instance ID for the AI Broker install
But before executing the Ansible collection, we needed to make some changes to it in order for it to work as intended.
To work around the error “"namespaces "openshift-serverless" not found.” We submitted the following YAMLs first:
---
apiVersion: v1
kind: Namespace
metadata:
name: openshift-serverless
---
apiVersion: operators.coreos.com/v1
kind: OperatorGroup
metadata:
name: openshift-serverless-og
namespace: openshift-serverless
spec:
upgradeStrategy: Default
Then to avoid the errors around the Open Data Hub Operator mentioned above; we manually submitted the following Subscription:
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
name: opendatahub-operator
namespace: openshift-operators
spec:
channel: fast
installPlanApproval: Manual
name: opendatahub-operator
source: community-operators
sourceNamespace: openshift-marketplace
startingCSV: opendatahub-operator.v2.11.1
Optionally approve the Install Plan for the Operator (as it is Manual otherwise it automatically upgrades to the latest version which will cause issues as mentioned above).
Next, we proceeded to comment out the section that installs the Operator on the related IBM Ansible script, located in our case at:
~/.ansible/collections/ansible_collections/ibm/mas_devops/roles/odh/tasks/odh-operator.yml
The change looked like the below:
# Install Operator & create entitlement openshift-odh
# -----------------------------------------------------------------------------
#- name: "Install Openshift odh Operator"
# ibm.mas_devops.apply_subscription:
# namespace: ""
# package_name: "opendatahub-operator"
# package_channel: ""
# catalog_source: ""
# register: subscription
Next, we executed the actual Ansible Playbook from the collection, by using:
ansible-playbook ibm.mas_devops.oneclick_add_aibroker
We should also check that all Pods on the “aibroker” (in our case “mas-p01-main-aibroker”) project/namespace are Ready and execute without errors:
To configure Manage next, we need to retrieve some additional information, first, let’s get the AI Broker API Key from the secret named “aibroker-user----apikey-secret” on the AI Broker project/namespace. That value should be used into the “mxe.int.aibrokerapikey” Manage System Property.
Also, we should grab the value of the AI Broker Route, in the same project/namespace, our Route has the URL “https://aibroker.p01.demo05.mas.interloc.cloud/” and we used it to configure the “mxe.int.aibrokerapiurl” Manage System Property by adding “/ibm/aibroker/service/rest/api/v1/” to it, so it became “https://aibroker.p01.demo05.mas.interloc.cloud/ibm/aibroker/service/rest/api/v1/”.
Finally, we configured the “mxe.int.aibrokertenantid” Manage System Property. The value we used was “aibroker-user” which appears to be a hardcoded value that can be retrieved from the internal URL: https://km-controller.mas-p01-main-aibroker:8443/api/v1/tenants, values in our case we used (on the Terminal of the “*-aibroker-api-*” Pod):
wget https://km-controller.mas-p01-main-aibroker:8443/api/v1/tenants --no-check- certificate
And the response came back as “[aibroker-user]”.
Next, we went to the "Cron Task Setup" application and ensure the Crontask Instances for “AIINFJOB” and “AITRAINJOB” are enabled and set to run periodically. Wait for the “AIINFJOB” to run at least once.
Finally, we went to the "AI configuration" application and there should be only one row named "WOPROBLEMCODE". We selected it and from Actions choose "Activate" then "Train Model". Waited while the model is trained as it may take some time.
A row should appear on the Model Training log table stating that the model training was completed. Then going back into the list of AI Configuration Models, check for the "Check Model Status" option that appears when you hover over the Active column value and the Model ID should be populated as well as the Ready value should be "True".
References:
We mostly followed a great tutorial by Sai Manjunath that can be found at https://developer.ibm.com/tutorials/awb-ai-driven-work-order-intelligence-maximo/
Also, the related IBM documentation can be found under https://www.ibm.com/docs/en/masv-and-l/maximo-manage/continuous-delivery?topic=watsonx-maximo-manage-ai-overview