cosi-driver-ceph

Sample Driver that provides reference implementation for Container Object Storage Interface (COSI) API for Ceph Object Store aka RADOS Gateway (RGW)

Installing CRDs, COSI controller, Node adapter

kubectl create -k 'github.com/kubernetes-sigs/container-object-storage-interface//?ref=release-0.2'

Following pods will running in the container-object-storage-system namespace :

NAME                                                   READY   STATUS    RESTARTS   AGE
container-object-storage-controller-64ff5586fb-jl96b   1/1     Running   0          2d6h

Building, Installing, Setting Up

Code can be compiled using:

make build

Now build docker image and provide tag as ceph/ceph-cosi-driver:latest

make container
Sending build context to Docker daemon  41.95MB
Step 1/5 : FROM gcr.io/distroless/static:latest
 ---> 1d9948f921db
Step 2/5 : LABEL maintainers="Ceph COSI Authors"
 ---> Using cache
 ---> 8659e9813ec5
Step 3/5 : LABEL description="Ceph COSI driver"
 ---> Using cache
 ---> 0c55b21ff64f
Step 4/5 : COPY ./cmd/ceph-cosi-driver/ceph-cosi-driver ceph-cosi-driver
 ---> a21275402998
Step 5/5 : ENTRYPOINT ["/ceph-cosi-driver"]
 ---> Running in 620bfa992683
Removing intermediate container 620bfa992683
 ---> 09575229056e
Successfully built 09575229056e

docker tag ceph-cosi-driver:latest ceph/ceph-cosi-driver:latest

Now start the sidecar and cosi driver with:

kubectl apply -k .
kubectl -n ceph-cosi-driver get pods
NAME                                         READY   STATUS    RESTARTS   AGE
objectstorage-provisioner-6c8df56cc6-lqr26   2/2     Running   0          26h

Create Bucket Requests, Bucket Access Request and consuming it in App

kubectl create -f examples/bucketclass.yaml
kubectl create -f examples/bucketclaim.yaml
kubectl create -f examples/bucketaccessclass.yaml
kubectl create -f examples/bucketaccess.yaml

Need to provide access details for RGW server via secret and it needs to be referenced in BucketAccessClass and BucketClass.

parameters:
  objectStoreUserSecretName: <secret name>
  objectStoreUserSecretNamespace: <namespace>

In the app, credentials can be consumed as secret volume mount using the secret name specified in the BucketAccess:

spec:
  containers:
      volumeMounts:
        - name: cosi-secrets
          mountPath: /data/cosi
  volumes:
  - name: cosi-secrets
    secret:
      secretName: sample-access-secret

An example for awscli pods can be found at examples/awscliapppod.yaml. Credentials will be in json format in the file.

{
      apiVersion: "v1alpha1",
      kind: "BucketInfo",
      metadata: {
          name: "ba-$uuid"
      },
      spec: {
          bucketName: "ba-$uuid",
          authenticationType: "KEY",
          endpoint: "https://rook-ceph-my-store:443",
          accessKeyID: "AKIAIOSFODNN7EXAMPLE",
          accessSecretKey: "wJalrXUtnFEMI/K...",
          region: "us-east-1",
          protocols: [
            "s3"
          ]
      }
    }

Known limitations

1. Handle access policies for Bucket Access Request

Configuration Options

Option	Default value	Description
--driver-address	unix:///var/lib/cosi/cosi.sock	COSI driver address, must be a UNIX socket
--driver-prefix	empty	prefix added before name, e.g, <prefix>.ceph.objectstorage.k8s.io

RGW IAM accounts

With Ceph Squid or later, you can use an existing RGW IAM account. Configure the credentials Secret as you would for an RGW user without an account, using AccessKey, SecretKey, and Endpoint. Use account root credentials or credentials with bucket and IAM management permissions. Reference the Secret using objectStoreUserSecretName and objectStoreUserSecretNamespace.

Set these parameters identically on both BucketClass and BucketAccessClass. Use authenticationType: Key on BucketAccessClass:

parameters:
  objectStoreUserSecretName: <secret name>
  objectStoreUserSecretNamespace: <namespace>
  userProvisioningMode: iam
  rgwAccountID: <account ID>
  rgwTenant: <tenant> # omit for an account without a tenant

Buckets belong to the account; the credentials determine their tenant. Each BucketAccess gets an IAM user, key, and policy limited to its bucket. The driver stores the key in an internal Secret for retries and removes its IAM resources and Secret on revoke.

Omitting userProvisioningMode or setting it to adminops selects the default RGW Admin Ops mode.

Integration with Rook

The ceph cosi driver integrates with Rook from v1.12 onwards to provide object storage for Kubernetes applications. More details can be found here.

RGW IAM accounts managed by Rook

Follow Rook's Object Store Accounts guide to create a CephObjectStoreAccount. Once the account is ready, retrieve its ID and root credentials Secret name:

kubectl -n rook-ceph get cephobjectstoreaccount my-account \
  -o jsonpath='{.status.accountID}{"\n"}{.status.rootAccountSecretName}{"\n"}'

Use the reported accountID from the account status as rgwAccountID. Copy AccessKey and SecretKey from the reported Secret into a COSI provisioning Secret, adding Endpoint with the RGW S3 endpoint URL. Reference that provisioning Secret through objectStoreUserSecretName and objectStoreUserSecretNamespace on both classes, together with the parameters described under RGW IAM accounts.

Alternatively, use a CephObjectStoreUser associated with the account through spec.accountRef.name, as described in the Rook guide. Account users have no permissions by default; grant the user bucket and IAM management permissions before using its credentials for COSI provisioning.

Community, discussion, contribution, and support

You can reach the maintainers of this project at:

• Slack
• Mailing List

Code of conduct

Participation in the Kubernetes community is governed by the Kubernetes Code of Conduct.