Add Application Credential support

[Deydra71]

Jira: OSPRH-16628

Adds end-to-end support for consuming Keystone Application Credentials (AppCred) in the telemetry-operator, enabling Ceilometer, Aodh, and CloudKitty pods to use AppCred-based authentication when available.

API changes:

Adds an optional auth field to telemetry service CRs:

• spec.auth.applicationCredentialSecret — name of the Secret that contains the Keystone Application Credential ID and Secret (AC_ID and AC_SECRET).

This is added for:

• Ceilometer CRs
• Aodh CRs
• CloudKitty CRs (and the related API/Proc CRs where applicable)

Reconcile behavior:

• Reads spec.auth.applicationCredentialSecret
• Attempts to load AC_ID / AC_SECRET from the referenced Secret (via the Keystone helper)
• If the Secret is missing or incomplete, reconciliation falls back to password authentication (AppCred auth is optional and not treated as an error)

Once the AppCred Secret is ready with valid AC_ID / AC_SECRET:

• Templates AppCred credentials into the generated service configuration (e.g. ceilometer.conf, aodh.conf, cloudkitty.conf)

• The rendered config hash naturally includes AppCred values, triggering rolling updates when credentials rotate

• All controllers use the same service user’s AppCred Secret for their respective service (ceilometer/aodh/cloudkitty)

• Updates Keystone auth sections to support AppCred with a block-based if/else structure:

  • AppCred: auth_type = v3applicationcredential + application_credential_id + application_credential_secret
  • Else: existing password-based auth

Depends-on: openstack-k8s-operators/keystone-operator#567

[Deydra71]

Note: kuttl tests are yet to be added

[softwarefactory-project-zuul]

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/9f80e1823add462ea5c60df2672f72d6

✔️ openstack-k8s-operators-content-provider SUCCESS in 2h 21m 14s
⚠️ telemetry-operator-multinode-cloudkitty SKIPPED Skipped due to failed job telemetry-openstack-meta-content-provider-master
❌ telemetry-openstack-meta-content-provider-master FAILURE in 8m 49s
❌ telemetry-operator-multinode-default-telemetry FAILURE in 1h 57m 01s
⚠️ functional-tests-osp18 SKIPPED Skipped due to failed job telemetry-openstack-meta-content-provider-master

[softwarefactory-project-zuul]

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/c51230a124434844b7ac185afdb52021

✔️ openstack-k8s-operators-content-provider SUCCESS in 1h 53m 25s
⚠️ telemetry-operator-multinode-cloudkitty SKIPPED Skipped due to failed job telemetry-openstack-meta-content-provider-master
❌ telemetry-openstack-meta-content-provider-master FAILURE in 8m 24s
❌ telemetry-operator-multinode-default-telemetry FAILURE in 1h 26m 52s
⚠️ functional-tests-osp18 SKIPPED Skipped due to failed job telemetry-openstack-meta-content-provider-master

[mrunge]

Is this kind of replacement from an official to a non-official package name is intentional here?

[mrunge]

same here, this weird package name looks fishy.

[mrunge]

We should not pull from non openstack-k8s-operators

[Deydra71]

Hi @mrunge ! It's only temporary before the dependency in keystone-oeprator is merged

[Deydra71]

After that, it will be removed before final reviews and merge

[vyzigold]

What about moving the auth_url to the outside of the conditional block (e.g. under the "{{- end }}" line) and also removing the auth_url on L55, so that it isn't duplicated? Similar for the other service templates.

[vyzigold]

I tested with the PR and there is an issue with watching the appcred secret in CloudKitty. The CloudKitty uses 3 controllers (cloudkitty, cloudkittyproc, cloudkittyapi), each controller manages its own CR. The cloudkitty.conf is initially created in the cloudkitty controller and saved into cloudkitty-config-data secret. The cloudkittyproc and cloudkittyapi controllers then read that secret, extract the config from it and create their own secrets for the resources they manage (secrets named cloudkitty-proc-config-data, cloudkitty-api-config-data). The cloudkittyproc and cloudkittyapi don't really do any changes to the config, they're currently just directly copying it.

What that means, when a cloudkitty appcred secret changes, the "cloudkitty" controller needs to reconcile. It'll refresh the cloudkitty.conf in the shared cloudkitty-config-data secret and because the cloudkittyproc and cloudkittyapi controllers are watching all secrets owned by the cloudkitty controller, they'll automatically reconcile as well and propagate the changes into their pods. So I don't think in the end the cloudkittyapi and cloudkittyproc controllers will actually need to watch anything.

What happened in my environment right now: I had appcreds created with just the "service" role, which doesn't seem enough for CK. So I added the "admin" role. cloudkittyapi and cloudkittyproc controllers reconciled, but the cloudkitty controller didn't reconcile. This resulted in no change in the config used inside the pods and the old (insufficient) credentials stayed being used.

[vyzigold]

sorry for the wall of text. TLDR: watch the secrets in cloudkitty controller, not in cloudkittyapi and cloudkittyproc :D

[Deydra71]

@vyzigold thanks a lot for checking the reconciliation! I updated the controllers so that only the cloudkitty_controller is indexing and watching the app cred secret. Can you please recheck the changes.

[softwarefactory-project-zuul]

This change depends on a change that failed to merge.

Change openstack-k8s-operators/keystone-operator#567 is needed.

[softwarefactory-project-zuul]

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/adb5f869f7e44dd7a494b6aaab89cbd8

✔️ openstack-k8s-operators-content-provider SUCCESS in 2h 12m 09s
⚠️ telemetry-operator-multinode-cloudkitty SKIPPED Skipped due to failed job telemetry-openstack-meta-content-provider-master
❌ telemetry-openstack-meta-content-provider-master FAILURE in 8m 25s
✔️ telemetry-operator-multinode-default-telemetry SUCCESS in 1h 31m 36s
⚠️ functional-tests-osp18 SKIPPED Skipped due to failed job telemetry-openstack-meta-content-provider-master

[vyzigold]

I retested with the new changes. ✔️

Before (without the changes - bad behavior):

# see the credential ID we're starting with
$ oc rsh -c cloudkitty-proc cloudkitty-proc-0 cat /etc/cloudkitty/cloudkitty.conf.d/00-cloudkitty.conf | grep application_credential_id
application_credential_id = c89548003bc24a1b970a80d6d4730985
application_credential_id = c89548003bc24a1b970a80d6d4730985

# edit the app credentials (add a role) to trigger creation of new credentials
$ oc edit oscp
openstackcontrolplane.core.openstack.org/openstack-galera-network-isolation edited

# see which credentials are being used (it's still the old ones, which is wrong). In the logs I see CloudKittyProc and CloudKittyAPI controllers reconcile, but no CloudKitty
$ oc rsh -c cloudkitty-proc cloudkitty-proc-0 cat /etc/cloudkitty/cloudkitty.conf.d/00-cloudkitty.conf | grep application_credential_id
application_credential_id = c89548003bc24a1b970a80d6d4730985
application_credential_id = c89548003bc24a1b970a80d6d4730985

# manually restart the telemetry-operator to trigger reconciliation of all controllers (this way we end up with the correct credentials, but it's just a WA of a bad behavior)
$ oc rsh -c cloudkitty-proc cloudkitty-proc-0 cat /etc/cloudkitty/cloudkitty.conf.d/00-cloudkitty.conf | grep application_credential_id
application_credential_id = 5203a941b49d467bb28b5143a03a3432
application_credential_id = 5203a941b49d467bb28b5143a03a3432

After (with the current code):

# See the starting credentials
$ oc rsh -c cloudkitty-proc cloudkitty-proc-0 cat /etc/cloudkitty/cloudkitty.conf.d/00-cloudkitty.conf | grep application_credential_id
application_credential_id = 5203a941b49d467bb28b5143a03a3432
application_credential_id = 5203a941b49d467bb28b5143a03a3432

# Edit oscp to remove a role from the app creds to trigger creating new credentials
[stack@osp-cloudops-03 ~]$ oc edit oscp
openstackcontrolplane.core.openstack.org/openstack-galera-network-isolation edited

# See that the credentials successfuly changed.
[stack@osp-cloudops-03 ~]$ oc rsh -c cloudkitty-proc cloudkitty-proc-0 cat /etc/cloudkitty/cloudkitty.conf.d/00-cloudkitty.conf | grep application_credential_id
application_credential_id = 9983cc10c3a64dc3aad9f8c8dc9904ca
application_credential_id = 9983cc10c3a64dc3aad9f8c8dc9904ca

In the logs with the new code I can see:

Controllers.CloudKitty	input source ac-cloudkitty-secret changed, reconcile: cloudkitty - openstack
Controllers.CloudKittyAPI	Secret cloudkitty-config-data and CR cloudkitty-api marked with label: cloudkitty
Controllers.CloudKittyProc	Secret cloudkitty-config-data and CR cloudkitty-proc marked with label: cloudkitty

So I think the watching of the app credentials in cloudkitty now works as it should. Thanks @Deydra71 .

[vyzigold]

There is one more issue with the watching in the CloudKitty controller, which I didn't mention because I thought it's unrelated and would need a separate fix, but after investigating, I think it should be addressed here.

The findObjectForSrc function matches everything (so it matches every secret). So let's also copy a findObjectsForSrc from elsewhere (like https://github.com/openstack-k8s-operators/telemetry-operator/blob/main/internal/controller/cloudkittyapi_controller.go#L444) and use that for the MapFunc. This is following what we found to be a pattern in other operators (having findObjectForSrc to match everything and findObjectsForSrc to match based on fields in the CR) e.g.:
https://github.com/openstack-k8s-operators/nova-operator/blob/main/internal/controller/nova_controller.go#L2108
https://github.com/openstack-k8s-operators/neutron-operator/blob/main/internal/controller/neutronapi_controller.go#L353
https://github.com/openstack-k8s-operators/cinder-operator/blob/main/internal/controller/cinder_controller.go#L377

I'd personally name the findObjectForSrc a bit differently, but let's try to stay consistent with the other operators.

[softwarefactory-project-zuul]

Merge Failed.

This change or one of its cross-repo dependencies was unable to be automatically merged with the current state of its repository. Please rebase the change and upload a new patchset.
Warning:
Error merging github.com/openstack-k8s-operators/telemetry-operator for 829,7beeac6472f3b6b025df034138b0e49b0572f66e

[openshift-ci]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: Deydra71
Once this PR has been reviewed and has the lgtm label, please ask for approval from vyzigold. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[openshift-ci]

@Deydra71: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/telemetry-operator-build-deploy	65bb887	link	false	/test telemetry-operator-build-deploy
ci/prow/telemetry-operator-build-deploy-kuttl	65bb887	link	true	/test telemetry-operator-build-deploy-kuttl

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

[softwarefactory-project-zuul]

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/8739c6d40eeb41df85a50de10cce3917

❌ openstack-k8s-operators-content-provider FAILURE in 16m 37s
⚠️ telemetry-operator-multinode-cloudkitty SKIPPED Skipped due to failed job telemetry-openstack-meta-content-provider-master
❌ telemetry-openstack-meta-content-provider-master FAILURE in 13m 24s
⚠️ telemetry-operator-multinode-default-telemetry SKIPPED Skipped due to failed job openstack-k8s-operators-content-provider
⚠️ functional-tests-osp18 SKIPPED Skipped due to failed job telemetry-openstack-meta-content-provider-master

[vyzigold]

I retested with the latest code and the watching in cloudkitty seems to work correctly now. The controller reconciles for changes to the ac secrets, but I don't see the additional unnecessary reconciliations anymore 👍