OpenShift / OKD Extra Configuration Steps

Notes

Main changes and tasks for OCP are:

  • On OCP / OKD, the Operating System is Managed by the Cluster
  • OCP Imposes Security Context Constraints
    • This requires everything to run with the least privilege possible. For the moment every component has been given access to run as higher privilege.
    • Something to circle back on is network polices and which components can have their privileges reduced without impacting functionality.
      • The UI probably can be for example.
  • openshift/oauth-proxy for authentication to the Longhorn Ui
    • ⚠️ Currently Scoped to Authenticated Users that can delete a longhorn settings object.
      • ⚠️ Since the UI it self is not protected, network policies will need to be created to prevent namespace <--> namespace communication against the pod or service object directly.
      • Anyone with access to the UI Deployment can remove the route restriction. (Namespace Scoped Admin)
  • Option to use separate disk in /var/mnt/longhorn & MachineConfig file to mount /var/mnt/longhorn
  • Adding finalizers for mount propagation

Known Issues

  • General Feature/Issue Thread
  • 4.10 / 1.23:
    • 4.10.0-0.okd-2022-03-07-131213 to 4.10.0-0.okd-2022-07-09-073606
      • Tested, No Known Issues
  • 4.11 / 1.24:
  • 4.12 / 1.25:
    • 4.12.0-0.okd-2022-12-05-210624 to 4.12.0-0.okd-2023-01-20-101927
      • Tested, No Known Issues
    • 4.12.0-0.okd-2023-01-21-055900 to 4.12.0-0.okd-2023-02-18-033438:
    • 4.12.0-0.okd-2023-03-05-022504 - 4.12.0-0.okd-2023-04-16-041331:
      • Tested, No Known Issues
  • 4.13 / 1.26:
    • 4.13.0-0.okd-2023-05-03-001308 - 4.13.0-0.okd-2023-08-18-135805:
      • Tested, No Known Issues
  • 4.14 / 1.27:
    • 4.14.0-0.okd-2023-08-12-022330 - 4.14.0-0.okd-2023-10-28-073550:
      • Tested, No Known Issues

Preparing Nodes (Optional)

Only required if you require additional customizations, such as storage-less nodes, or secondary disks.

Default /var/lib/longhorn setup

Label each node for storage with:

oc get nodes --no-headers | awk '{print $1}'

export NODE="worker-0"
oc label node "${NODE}" node.longhorn.io/create-default-disk=true

Separate /var/mnt/longhorn setup

Create Filesystem

On the storage nodes create a filesystem with the label longhorn:

oc get nodes --no-headers | awk '{print $1}'

export NODE="worker-0"
oc debug node/${NODE} -t -- chroot /host bash

# Validate Target Drive is Present
lsblk

export DRIVE="sdb" #vdb
sudo mkfs.ext4 -L longhorn /dev/${DRIVE}

⚠️ Note: If you add New Nodes After the below Machine Config is applied, you will need to also reboot the node.

Mounting Disk On Boot

The Secondary Drive needs to be mounted on every boot. Save the Concents and Apply the MachineConfig with oc apply -f:

⚠️ This will trigger an machine config profile update and reboot all worker nodes on the cluster

apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
  labels:
    machineconfiguration.openshift.io/role: worker
  name: 71-mount-storage-worker
spec:
  config:
    ignition:
      version: 3.2.0
    systemd:
      units:
        - name: var-mnt-longhorn.mount
          enabled: true
          contents: |
            [Unit]
            Before=local-fs.target
            [Mount]
            Where=/var/mnt/longhorn
            What=/dev/disk/by-label/longhorn
            Options=rw,relatime,discard
            [Install]
            WantedBy=local-fs.target

Label and Annotate Nodes

Label and annotate storage nodes like this:

oc get nodes --no-headers | awk '{print $1}'

export NODE="worker-0"
oc annotate node ${NODE} --overwrite node.longhorn.io/default-disks-config='[{"path":"/var/mnt/longhorn","allowScheduling":true}]'
oc label node ${NODE} node.longhorn.io/create-default-disk=config

Example values.yaml

Minimum Adjustments Required

openshift:
  oauthProxy:
    repository: quay.io/openshift/origin-oauth-proxy
    tag: 4.15  # Use Your OCP/OKD 4.X Version, Current Stable is 4.15

# defaultSettings: # Preparing nodes (Optional)
  # createDefaultDiskLabeledNodes: true

openshift:
  enabled: true
  ui:
    route: "longhorn-ui"
    port: 443
    proxy: 8443

Installation

# helm template ./chart/ --namespace longhorn-system --values ./chart/values.yaml --no-hooks > longhorn.yaml # Local Testing
helm template longhorn --namespace longhorn-system --values values.yaml --no-hooks  > longhorn.yaml
oc create namespace longhorn-system -o yaml --dry-run=client | oc apply -f -
oc apply -f longhorn.yaml -n longhorn-system

Refs