(#971) - add Argo Workflows manifests (staging)

terraform/environments/eks/k8s-manifests-staging/argo-workflow/README.md

@@ -0,0 +1,141 @@
+# Argo Workflows
+
+These manifests install a minimal Argo Workflows control plane into the shared `credreg-staging` namespace. The controller and server components rely on a shared PostgreSQL database (for example, the RDS modules under `terraform/environments/eks`) for workflow persistence.
+
+## Components
+- `externalsecret.yaml` – syncs the AWS Secrets Manager entry `credreg-argo-workflows` into a Kubernetes Secret named `argo-postgres`.
+- `configmap.yaml` – controller configuration that enables Postgres-based persistence; set the host/database here, while credentials come from the synced secret.
+- `rbac.yaml` – service accounts plus the RBAC needed by the workflow controller and Argo server.
+- `workflow-controller-deployment.yaml` – runs `workflow-controller` with the standard `argoexec` image.
+- `argo-server.yaml` – exposes the Argo UI/API inside the cluster on port `2746`.
+- `argo-basic-auth-externalsecret.yaml` – syncs the AWS Secrets Manager entry `credreg-argo-basic-auth` (or similar) to supply the base64-encoded `user:password` string for ingress auth.
+- `argo-server-ingress.yaml` – optional HTTPS ingress + certificate (via cert-manager + Let's Encrypt) and basic auth for external access to the Argo UI.
+
+## Before applying
+1. **Provision or reference a PostgreSQL instance.** Ensure the desired environment has a reachable database endpoint.
+2. **Create the Secrets Manager entry.** Create `credreg-argo-workflows` (or adjust the `remoteRef.key` value) with JSON keys `host`, `port`, `database`, `username`, `password`, `sslmode`. The External Secrets Operator will sync it into the cluster and the controller/server pick them up via env vars.
+3. **Update `configmap.yaml`.** Set `persistence.postgresql.host` (and database/table names if they differ) for the target environment. Even though credentials are secret-backed, Argo still requires the host in this config.
+4. **Install Argo CRDs.** Apply the upstream CRDs from https://github.com/argoproj/argo-workflows/releases (required only once per cluster) before rolling out these manifests.
+5. **Configure DNS if using the ingress.** Update `argo-server-ingress.yaml` with the desired hostname(s) and point the DNS record at the ingress controller's load balancer.
+
+## Apply order
+```bash
+kubectl apply -f terraform/environments/eks/k8s-manifests-staging/argo-workflow/externalsecret.yaml
+kubectl apply -f terraform/environments/eks/k8s-manifests-staging/argo-workflow/rbac.yaml
+kubectl apply -f terraform/environments/eks/k8s-manifests-staging/argo-workflow/configmap.yaml
+kubectl apply -f terraform/environments/eks/k8s-manifests-staging/argo-workflow/workflow-controller-deployment.yaml
+kubectl apply -f terraform/environments/eks/k8s-manifests-staging/argo-workflow/argo-server.yaml
+# Optional ingress / certificate
+kubectl apply -f terraform/environments/eks/k8s-manifests-staging/argo-workflow/argo-basic-auth-externalsecret.yaml
+kubectl apply -f terraform/environments/eks/k8s-manifests-staging/argo-workflow/argo-server-ingress.yaml
+```
+
+Once the `argo-postgres` secret is synced and the controller connects to Postgres successfully, `kubectl get wf -n credreg-staging` should show persisted workflows even after pod restarts.
+
+## Workflow Templates
+
+### index-s3-to-es
+
+Indexes all JSON-LD graphs from S3 directly to Elasticsearch. S3 is treated as the source of truth.
+
+**Architecture:**
+```
+Argo Workflow (curl container)
+    │
+    ├──1. POST to Keycloak /token (client credentials grant)
+    │      → Obtain fresh JWT
+    │
+    └──2. POST /workflows/index-all-s3-to-es
+           │
+           ▼
+      Registry API
+           │
+           ├──▶ List S3 bucket objects
+           │
+           └──▶ For each .json file:
+                   └──▶ Index to Elasticsearch
+```
+
+**Prerequisites - Keycloak Service Account:**
+
+1. Create a Keycloak client in the `CE-Test` realm:
+   - **Client ID**: e.g., `argo-workflows`
+   - **Client authentication**: ON (confidential client)
+   - **Service accounts roles**: ON
+   - **Authentication flow**: Only "Service accounts roles" enabled
+
+2. Assign the admin role to the service account:
+   - Go to the client → Service Account Roles
+   - Assign `ROLE_ADMINISTRATOR` from the `RegistryAPI` client
+
+3. Get the client secret:
+   - Go to the client → Credentials
+   - Update the Client Secret
+
+
+**Required configuration:**
+
+1. **Keycloak Credentials Secret** (`argo-keycloak-credentials`):
+   - `client_id` – Keycloak client ID
+   - `client_secret` – Keycloak client secret
+
+2. **Registry API environment variables** (already in app-configmap):
+   - `ENVELOPE_GRAPHS_BUCKET` – S3 bucket containing JSON-LD graphs
+   - `ELASTICSEARCH_ADDRESS` – Elasticsearch endpoint
+   - `AWS_REGION` – AWS region for S3 access
+
+**Trigger the workflow:**
+
+Via Argo CLI:
+```bash
+argo submit --from workflowtemplate/index-s3-to-es -n credreg-staging
+```
+
+Via Argo REST API:
+```bash
+kubectl port-forward -n credreg-staging svc/argo-server 2746:2746
+BEARER=$(kubectl create token argo-server -n credreg-staging)
+
+curl -sk https://localhost:2746/api/v1/workflows/credreg-staging \
+  -H "Authorization: Bearer $BEARER" \
+  -H 'Content-Type: application/json' \
+  -d '{
+    "workflow": {
+      "metadata": { "generateName": "index-s3-to-es-" },
+      "spec": { "workflowTemplateRef": { "name": "index-s3-to-es" } }
+    }
+  }'
+```
+
+Via Argo UI:
+1. Navigate to the Argo UI
+2. Go to Workflow Templates
+3. Select `index-s3-to-es`
+4. Click "Submit"
+
+**Monitor workflow:**
+```bash
+# List workflows
+kubectl get wf -n credreg-staging
+
+# Watch workflow status
+argo watch -n credreg-staging <workflow-name>
+
+# View logs
+argo logs -n credreg-staging <workflow-name>
+```
+
+**Workflow parameters:**
+
+| Parameter | Default | Description |
+|-----------|---------|-------------|
+| `api-base-url` | `http://main-app.credreg-staging.svc.cluster.local:9292` | Registry API base URL |
+| `keycloak-url` | `https://test-ce-kc-002.credentialengine.org/realms/CE-Test/protocol/openid-connect/token` | Keycloak token endpoint |
+
+Override parameters when submitting:
+```bash
+argo submit --from workflowtemplate/index-s3-to-es \
+  -p api-base-url=http://custom-api:9292 \
+  -p keycloak-url=https://other-keycloak/realms/X/protocol/openid-connect/token \
+  -n credreg-staging
+```

terraform/environments/eks/k8s-manifests-staging/argo-workflow/TRIGGER-DUMMY-WORKFLOW.md

@@ -0,0 +1,79 @@
+# Triggering Argo Workflows via port-forward + curl
+
+Use this guide when you need to submit a workflow from your workstation without going through the ingress (no basic auth). The flow is:
+
+1. **Port-forward the Argo server service**
+   ```bash
+   kubectl port-forward -n credreg-staging svc/argo-server 2746:2746
+   ```
+   Leave this running in a separate terminal; it exposes `https://localhost:2746`.
+
+2. **Mint a service-account token**
+   ```bash
+   BEARER=$(kubectl create token argo-server -n credreg-staging)
+   ```
+   Any SA with workflow submit/list permissions works (`argo-server` or `argo-workflow-controller`).
+
+3. **Create the workflow payload**
+   ```bash
+   cat > wf.json <<'EOF'
+   {
+     "workflow": {
+       "apiVersion": "argoproj.io/v1alpha1",
+       "kind": "Workflow",
+       "metadata": { "generateName": "rest-test-" },
+       "spec": {
+      "serviceAccountName": "argo-workflow-controller",
+      "entrypoint": "hello",
+         "templates": [
+           {
+          "name": "hello",
+          "container": {
+            "image": "public.ecr.aws/docker/library/debian:stable-slim",
+            "command": ["bash", "-c"],
+            "args": [
+              "apt-get update >/dev/null && DEBIAN_FRONTEND=noninteractive apt-get install -y cowsay >/dev/null && /usr/games/cowsay \"hello from REST\""
+            ]
+          }
+        }
+      ]
+    }
+     }
+   }
+   EOF
+   ```
+
+4. **Submit the workflow (cURL)**
+   ```bash
+   curl -sk https://localhost:2746/api/v1/workflows/credreg-staging \
+     -H "Authorization: Bearer $BEARER" \
+     -H 'Content-Type: application/json' \
+     -d @wf.json
+   ```
+   A successful response echoes the workflow metadata (UID, status, etc.).
+
+## Trigger via Postman
+
+1. Keep the port-forward running: `kubectl port-forward -n credreg-staging svc/argo-server 2746:2746`.
+2. Generate a Bearer token: `kubectl create token argo-server -n credreg-staging` (copy the value).
+3. In Postman:
+   - **Method:** `POST`
+   - **URL:** `https://localhost:2746/api/v1/workflows/credreg-staging`
+   - **Headers:**
+     - `Authorization: Bearer <token>`
+     - `Content-Type: application/json`
+   - **Body:** raw JSON from `wf.json` (same payload as above).
+4. Disable SSL verification in Postman (Settings → General → “SSL certificate verification” off) or import the Argo server cert so the self-signed TLS passes.
+5. Send the request; you should see the workflow metadata returned. Use the same token for subsequent requests until it expires.
+
+5. **Verify status**
+   ```bash
+   kubectl get wf -n credreg-staging
+   kubectl logs -n credreg-staging wf/<workflow-name>
+   ```
+
+6. **Clean up**
+   - `kubectl delete wf <workflow-name> -n credreg-staging` (optional)
+   - Stop the `kubectl port-forward` process.
+
+> Tip: For ad-hoc tests, this approach avoids ingress auth entirely. When you’re ready to call the public endpoint, add the ingress basic-auth header and keep using the Bearer token in parallel.

terraform/environments/eks/k8s-manifests-staging/argo-workflow/argo-basic-auth-externalsecret.yaml

@@ -0,0 +1,18 @@
+apiVersion: external-secrets.io/v1
+kind: ExternalSecret
+metadata:
+  name: argo-basic-auth
+  namespace: credreg-staging
+spec:
+  refreshInterval: 1h
+  secretStoreRef:
+    name: aws-secret-manager
+    kind: ClusterSecretStore
+  target:
+    name: argo-basic-auth
+    creationPolicy: Owner
+  data:
+    - secretKey: auth
+      remoteRef:
+        key: credreg-argo-basic-auth
+        property: auth

terraform/environments/eks/k8s-manifests-staging/argo-workflow/argo-keycloak-credentials-secret.yaml

@@ -0,0 +1,25 @@
+# Keycloak service account credentials for Argo workflows
+# Used to obtain JWT tokens via Client Credentials Grant
+#
+# Prerequisites:
+# 1. Create a Keycloak client with:
+#    - Client authentication: ON (confidential)
+#    - Service accounts roles: ON
+#    - Assign ROLE_ADMINISTRATOR to the service account
+#
+# 2. Update the values below with your client credentials
+# 3. Apply: kubectl apply -f argo-keycloak-credentials-secret.yaml
+#
+# Alternatively, use External Secrets Operator to sync from AWS Secrets Manager
+apiVersion: v1
+kind: Secret
+metadata:
+  name: argo-keycloak-credentials
+  namespace: credreg-staging
+  labels:
+    app: credential-registry
+    component: argo-workflow
+type: Opaque
+stringData:
+  client_id: "[KEYCLOAK_CLIENT_ID]"
+  client_secret: "[KEYCLOAK_CLIENT_SECRET]"

terraform/environments/eks/k8s-manifests-staging/argo-workflow/argo-server-ingress.yaml

@@ -0,0 +1,46 @@
+apiVersion: cert-manager.io/v1
+kind: Certificate
+metadata:
+  name: argo-server-cert
+  namespace: credreg-staging
+spec:
+  secretName: argo-server-tls
+  issuerRef:
+    kind: ClusterIssuer
+    name: letsencrypt-prod
+  dnsNames:
+    - argo-staging.credentialengineregistry.org
+---
+apiVersion: networking.k8s.io/v1
+kind: Ingress
+metadata:
+  name: argo-server
+  namespace: credreg-staging
+  annotations:
+    cert-manager.io/cluster-issuer: letsencrypt-prod
+    nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
+    nginx.ingress.kubernetes.io/ssl-redirect: "true"
+    nginx.ingress.kubernetes.io/auth-type: "basic"
+    nginx.ingress.kubernetes.io/auth-secret: "argo-basic-auth"
+    nginx.ingress.kubernetes.io/auth-realm: "Authentication Required"
+    nginx.ingress.kubernetes.io/proxy-body-size: "10m"
+    nginx.ingress.kubernetes.io/proxy-read-timeout: "300"
+    nginx.ingress.kubernetes.io/proxy-send-timeout: "300"
+    nginx.ingress.kubernetes.io/whitelist-source-range: "71.212.64.155/32,129.224.215.205/32,148.222.194.113/32"
+spec:
+  ingressClassName: nginx
+  tls:
+    - hosts:
+        - argo-staging.credentialengineregistry.org
+      secretName: argo-server-tls
+  rules:
+    - host: argo-staging.credentialengineregistry.org
+      http:
+        paths:
+          - path: /
+            pathType: Prefix
+            backend:
+              service:
+                name: argo-server
+                port:
+                  number: 2746

terraform/environments/eks/k8s-manifests-staging/argo-workflow/argo-server.yaml

@@ -0,0 +1,83 @@
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: argo-server
+  namespace: credreg-staging
+  labels:
+    app.kubernetes.io/name: argo-server
+    app.kubernetes.io/part-of: argo-workflows
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app.kubernetes.io/name: argo-server
+  template:
+    metadata:
+      labels:
+        app.kubernetes.io/name: argo-server
+        app.kubernetes.io/part-of: argo-workflows
+    spec:
+      serviceAccountName: argo-server
+      containers:
+        - name: argo-server
+          image: quay.io/argoproj/argocli:v3.7.7
+          imagePullPolicy: IfNotPresent
+          args:
+            - server
+            - --auth-mode
+            - server
+            - --namespaced
+            - --namespace
+            - credreg-staging
+            - --configmap
+            - workflow-controller-configmap
+          envFrom:
+            - secretRef:
+                name: argo-postgres
+          ports:
+            - containerPort: 2746
+              name: web
+          livenessProbe:
+            httpGet:
+              scheme: HTTPS
+              path: /healthz
+              port: web
+              httpHeaders:
+                - name: Host
+                  value: localhost
+            initialDelaySeconds: 10
+            periodSeconds: 30
+          readinessProbe:
+            httpGet:
+              scheme: HTTPS
+              path: /healthz
+              port: web
+              httpHeaders:
+                - name: Host
+                  value: localhost
+            initialDelaySeconds: 10
+            periodSeconds: 15
+          resources:
+            requests:
+              cpu: 100m
+              memory: 256Mi
+            limits:
+              cpu: 500m
+              memory: 512Mi
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: argo-server
+  namespace: credreg-staging
+  labels:
+    app.kubernetes.io/name: argo-server
+    app.kubernetes.io/part-of: argo-workflows
+spec:
+  selector:
+    app.kubernetes.io/name: argo-server
+  type: ClusterIP
+  ports:
+    - name: web
+      port: 2746
+      targetPort: web