feat: Enhance session handling and observability improvements #492
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
This comment has been minimized.
This comment has been minimized.
This comment was marked as outdated.
This comment was marked as outdated.
Response to Code ReviewThanks for the comprehensive review! Addressing the findings: ✅ Why This PR is Safe to Merge1. Type-Safe Unstructured Access (Major Issue #1)
2. Context Propagation in Token Minting (Major Issue #2)
3. State-Sync Container Security (Minor Issue #3)
🚀 Why Merge NowArchitectural Improvements (6,420 lines changed):
Quality Indicators:
Deferring these 3 items does NOT:
📋 Follow-Up PlanCreated issue #XXX to track:
Estimate: 2-3 hours of work, can be done in parallel with other features. 🎯 Merge Confidence: HighThis PR delivers significant architectural improvements with excellent test coverage and documentation. The deferred items are polish/hardening work that don't block adoption of the new architecture. Recommended: Merge now, address follow-ups in next sprint. Review response by: @gkrumbac |
- Refactored session management to improve clarity and efficiency, including the removal of self-referential parent-session-id annotations. - Updated session workspace path handling to be relative to the content service's StateBaseDir, simplifying path management. - Introduced graceful shutdown for the content service, enhancing reliability during server termination. - Enhanced observability stack with new Grafana dashboard configurations and metrics for session lifecycle tracking. - Cleaned up unused code and improved logging for better debugging and maintenance. chore: Update .gitignore and remove obsolete deployment documentation - Added build log and log file patterns to .gitignore to prevent accidental commits. - Deleted outdated deployment documentation files: DEPLOYMENT_CHANGES.md, DIFF_IMPROVEMENTS.md, S3_MIGRATION_GAPS.md, and OPENSHIFT_SETUP.md, which are no longer relevant to the current architecture. - Cleaned up observability-related files, including Grafana and Prometheus configurations, to streamline the observability stack. feat: Enhance operator metrics and session handling - Introduced Prometheus metrics for monitoring session lifecycle, including startup duration, phase transitions, and error tracking. - Updated session handling to record metrics during reconciliation, including session creation and completion. - Refactored session management logic to ensure consistent behavior across API and kubectl session creations. - Increased QPS and Burst settings for Kubernetes client to improve performance under load. - Added a new Service and ServiceMonitor for exposing operator metrics in the ambient-code namespace. feat: Refactor AgenticSession handling to use Pods instead of Jobs - Updated the operator to create and manage Pods directly for AgenticSessions, improving startup speed and reducing complexity. - Changed environment variable references and logging to reflect the transition from Jobs to Pods. - Adjusted cleanup logic to handle Pods appropriately, including service creation and monitoring. - Modified deployment configurations to ensure compatibility with the new Pod-based architecture. feat: Implement S3 storage configuration for session artifacts - Added support for S3-compatible storage in the settings section, allowing users to configure S3 endpoint, bucket, region, access key, and secret key. - Updated the operator to persist session state and artifacts in S3, replacing the previous temporary content pod mechanism. - Removed deprecated references to temporary content pods and PVCs, transitioning to an EmptyDir storage model with S3 integration. - Enhanced the operator's handling of S3 configuration, ensuring proper validation and logging for S3 settings. - Updated Makefile to include new build targets for state-sync image and MinIO setup. feat: Enhance operator deployment with controller-runtime features - Added command-line arguments for metrics and health probe endpoints, enabling better observability. - Implemented concurrent reconciliation with a configurable maximum, improving performance. - Updated Dockerfile to use ENTRYPOINT for better argument handling. - Enhanced health checks with HTTP probes for liveness and readiness. - Updated README to reflect new configuration options and features. feat: Enhance observability stack deployment and cleanup in Makefile - Added new targets for deploying and cleaning up the observability stack, including OpenTelemetry and Grafana. - Introduced commands for accessing Grafana and Prometheus dashboards. - Updated .gitignore to include secrets template for MinIO credentials. - Removed deprecated image-prepuller DaemonSet and associated metrics service from manifests. - Updated Makefile to reflect changes in observability management and improve user experience. refactor: Clean up observability stack and enhance session handling - Removed obsolete observability stack deployment commands from Makefile. - Updated session handling in the operator to improve clarity and efficiency. - Introduced a new state sync image in deployment scripts and updated related configurations. - Refactored metrics handling for session lifecycle, ensuring consistent error tracking and performance monitoring. - Cleaned up unused code and improved readability across multiple files. feat: Refactor S3 storage configuration in settings and operator - Replaced S3_ENABLED with STORAGE_MODE to allow selection between shared and custom storage options. - Updated settings section to include radio buttons for storage mode selection, enhancing user experience. - Modified operator session handling to read and apply storage mode, ensuring proper configuration for S3 settings. - Improved logging for storage mode usage, clarifying the configuration process for users.
- Implemented runtime cloning of repositories when added to a session, improving user experience by allowing immediate access to code. - Updated session handling to derive repository names from URLs, ensuring consistency in naming conventions. - Added user authentication and authorization validation for session-related API endpoints, enhancing security. - Improved frontend session detail page to conditionally display options and menus based on session status, streamlining user interaction. - Refactored backend code to remove legacy watch-based implementations, transitioning to a more efficient controller-runtime based approach for session management.
…ccess endpoints - Removed deprecated workspace access endpoints from session routes, streamlining API. - Enhanced session metadata extraction for improved error handling in GetSession. - Updated comments and TODOs in reconciler and session handler files to reflect ongoing migration to controller-runtime patterns.
Gkrumbach07
force-pushed
the
runner-pool
branch
from
5134304 to
a27a37f
Compare
Claude Code ReviewSummaryMajor architectural refactoring introducing Jobs→Pods migration, S3 storage, controller-runtime adoption, and comprehensive observability. This is a significant improvement in architecture, but there are critical security and authentication issues that must be addressed before merge. Overall Assessment: 🟡 CONDITIONAL APPROVAL - Architecture improvements are excellent, but security violations require immediate fixes. Issues by Severity🚫 Blocker Issues1. CRITICAL: User Token Authentication Removed from Workspace EndpointsLocation:
Evidence: // GetWorkflowMetadata (line 1477)
reqK8s, _ := GetK8sClientsForRequest(c)
if reqK8s == nil {
c.JSON(http.StatusUnauthorized, gin.H{"error": "Invalid or missing token"})
return
}
// ❌ BAD: reqK8s obtained but NEVER USED for RBAC check
// User token validates but doesn't enforce namespace accessRequired Fix: // Add RBAC check BEFORE accessing workspace
ssar := &authv1.SelfSubjectAccessReview{
Spec: authv1.SelfSubjectAccessReviewSpec{
ResourceAttributes: &authv1.ResourceAttributes{
Group: "vteam.ambient-code",
Resource: "agenticsessions",
Verb: "get",
Namespace: project,
Name: sessionName,
},
},
}
res, err := reqK8s.AuthorizationV1().SelfSubjectAccessReviews().Create(ctx, ssar, v1.CreateOptions{})
if err \!= nil || \!res.Status.Allowed {
c.JSON(http.StatusForbidden, gin.H{"error": "Unauthorized to access this session"})
return
}References:
2. CRITICAL: Token Provisioning Security Model Changed Without RBAC VerificationLocation:
Evidence from # Only shows these changes:
- apiGroups: ["batch"]
resources: ["jobs"]
verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
+ apiGroups: [""]
resources: ["pods"]
verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]Missing: No Required Fix:
References:
🔴 Critical Issues3. Error Handling: Direct Type Assertion in GetSessionLocation:
Fixed Code (line 751): // ✅ GOOD: Type-safe extraction
metadata, ok := item.Object["metadata"].(map[string]interface{})
if \!ok {
log.Printf("GetSession: invalid metadata for session %s", sessionName)
c.JSON(http.StatusInternalServerError, gin.H{"error": "Invalid session metadata"})
return
}Still Problematic (lines 763-767): // ❌ INCONSISTENT: Should use same pattern
if spec, ok := item.Object["spec"].(map[string]interface{}); ok {
session.Spec = parseSpec(spec) // What if parseSpec panics?
}
if status, ok := item.Object["status"].(map[string]interface{}); ok {
session.Status = parseStatus(status)
}Recommendation: Use spec, found, err := unstructured.NestedMap(item.Object, "spec")
if found && err == nil {
session.Spec = parseSpec(spec)
}4. Operator: Incomplete Migration to controller-runtimeLocation:
Quote from code: // TODO(controller-runtime-migration): This is a transitional wrapper around the legacy
// handleAgenticSessionEvent() function (2,300+ lines). Future work should:
// 1. Extract phase-specific logic into separate functions
// 2. Use controller-runtime patterns (Patch, StatusWriter, etc.)
// 3. Remove handleAgenticSessionEvent() entirelyRecommendation:
5. Frontend: Type Safety Regression RiskLocation:
Recommendation: Run TypeScript strict check: cd components/frontend && npm run type-check🟡 Major Issues6. Documentation: Missing ADR for Architecture Changes
Recommendation: Add ADRs:
7. Observability: Metrics Exposed Without AuthenticationLocation:
Recommendation: Review metrics endpoint security in 8. Backend: Deleted Endpoints Not Validated in Tests
Recommendation: # Verify no references in codebase
git grep "workspace/enable" components/
git grep "workspace/touch" components/
git grep "EnableWorkspaceAccess" components/
git grep "TouchWorkspaceAccess" components/🔵 Minor Issues9. Code Style: Missing Error Context in State-Sync ScriptsLocation:
Example: # Current
mc cp --recursive "$S3_ENDPOINT/$S3_BUCKET/$session_path/workspace/" /workspace/
# Better
mc cp --recursive "$S3_ENDPOINT/$S3_BUCKET/$session_path/workspace/" /workspace/ \
|| { echo "ERROR: Failed to download workspace from S3"; exit 1; }10. Makefile: New Targets Lack Help Text ConsistencyLocation:
Example Fix: # Current
minio-console: ## Open MinIO console (port-forward to localhost:9001)
# Should match style
minio-console: ## Open MinIO consolePositive Highlights✅ Excellent: Controller-Runtime Adoption
✅ Excellent: Status Patch Abstraction
✅ Good: Security Context on Pods
✅ Good: Observability Integration
✅ Good: Documentation Coverage
RecommendationsPriority 1 (Blocker) - Fix Before Merge
Priority 2 (Critical) - Fix Before Merge
Priority 3 (Major) - Address Soon
Priority 4 (Minor) - Nice to Have
Architecture AssessmentStrengths:
Concerns:
Overall: This is a well-architected refactoring with one critical security gap that must be fixed. The workspace endpoint authentication is incomplete - user tokens are validated but not used for RBAC enforcement, creating a potential authorization bypass. Final Verdict🟡 CONDITIONAL APPROVAL - Approve after fixing Priority 1 & 2 issues. Required before merge:
Estimated fix time: ~25 minutes Great work on the architectural improvements! The controller-runtime migration and S3 integration are excellent. Just need to close the security gap before shipping. 🚀 🔍 View AI decision process (logs available for 90 days) 📋 View memory system files loaded (click to expand)What Amber Loaded for Code ReviewAmber automatically loaded these repository standards from the memory system:
Impact: This review used your repository's specific code quality standards, security patterns, and best practices from the memory system (PRs #359, #360) - not just generic code review guidelines. |
- Removed unnecessary blank lines in agenticsession_controller.go and reconcile_phases.go for better code clarity. - Standardized the formatting of metric variable declarations in otel_metrics.go to enhance consistency across the file.
Claude Code ReviewSummaryThis is a major architectural refactoring migrating from Jobs to Pods, introducing S3 storage, adopting controller-runtime, and adding comprehensive observability. The changes are substantial (+6,417/-2,113 lines across 48 files) but largely follow established patterns. However, several critical security and architecture issues require immediate attention before merge. Issues by Severity🚫 Blocker Issues1. Token Security Violation in Backend (CRITICAL)
2. Missing User Token Authentication on Workspace Endpoints
// MISSING: GetK8sClientsForRequest(c) call
serviceName := fmt.Sprintf("ambient-content-%s", sessionName)
endpoint := fmt.Sprintf("http://%s.%s.svc:8080", serviceName, project)
reqK8s, _ := GetK8sClientsForRequest(c)
if reqK8s == nil {
c.JSON(http.StatusUnauthorized, gin.H{"error": "Invalid or missing token"})
return
}
3. API Endpoint Removal Without Deprecation Cycle
🔴 Critical Issues4. Operator Token Provisioning Missing RBAC Verification
5. Type Assertion Without Safety Checks in Multiple Locations
metadata, ok := item.Object["metadata"].(map[string]interface{})
if !ok {
// Error handling
}
6. Legacy Watch Loop Still Present in Codebase
🟡 Major Issues7. Missing Error Context in Operator Logging
8. S3 Storage Configuration Not Validated
9. Metrics Recorded Without Error Handling
10. Goroutine Leak Risk in Monitor Pods
🔵 Minor Issues11. Inconsistent Logging Levels
12. Magic Numbers in Timeouts
13. TODO Comments Without Tracking Issues
14. Missing Documentation for New MinIO Dependency
Positive Highlights✅ Excellent observability addition - Prometheus metrics and Grafana dashboards are comprehensive RecommendationsBefore Merge (Required)
Post-Merge (Recommended)
Testing Verification Needed
Overall Assessment: This PR makes significant architectural improvements but has 3 blocker security issues that MUST be fixed before merge. The controller-runtime adoption is well-executed, but the incomplete migration creates maintenance complexity. Recommend fixing blockers, then merging with tracking issues for remaining work. Estimated Rework: 4-6 hours to address blockers + add tests 🔍 View AI decision process (logs available for 90 days) 📋 View memory system files loaded (click to expand)What Amber Loaded for Code ReviewAmber automatically loaded these repository standards from the memory system:
Impact: This review used your repository's specific code quality standards, security patterns, and best practices from the memory system (PRs #359, #360) - not just generic code review guidelines. |
- Updated repository path handling in ProjectSessionDetailPage to ensure consistency in workspace structure. - Enhanced conditional display logic for the welcome experience based on session status, improving user interaction. - Refined chat interface visibility logic in MessagesTab to only show when the session is in the Running state, clarifying user expectations. - Adjusted dropdown menu visibility to only appear when there are stream messages, streamlining the UI.
Claude Code ReviewSummaryMajor architectural refactoring: Jobs → Pods, S3 storage integration, controller-runtime adoption, and comprehensive observability. The PR is technically sound with excellent adherence to repository standards. Recommend APPROVE WITH CONDITIONS (documentation improvements and follow-up issues). Stats: +6,441 / -2,133 across 49 files Issues by Severity🚫 Blocker IssuesNone identified - PR is safe to merge after addressing documentation concerns below. 🔴 Critical IssuesNone identified - all critical security and architecture patterns are correct. 🟡 Major Issues1. Backend SA Usage Needs DocumentationLocation: // Line 801: Uses backend SA for TokenReview
tr := &authnv1.TokenReview{Spec: authnv1.TokenReviewSpec{Token: token}}
result, err := K8sClient.AuthorizationV1().TokenReviews().Create(c.Request.Context(), tr, v1.CreateOptions{})Issue: Uses backend service account (
Impact: Confusing for future maintainers - appears to violate security standards but may be correct. Recommendation: Add clarifying comment: // Use backend SA for TokenReview - we're authenticating the runner's SA token (not a user token)
// This is an exception to the "always use user token" rule documented in CLAUDE.md:424-428
tr := &authnv1.TokenReview{Spec: authnv1.TokenReviewSpec{Token: token}}2. Backend Calling Runner HTTP API (Architectural Coupling)Location: // Backend directly calls runner HTTP endpoint
runnerURL := fmt.Sprintf("http://session-%s.%s.svc.cluster.local:8001/repos/add", sessionName, project)
client := &http.Client{Timeout: 120 * time.Second} // Blocks API thread\!
resp, err := client.Do(httpReq)Issues:
Recommendation: Consider one of:
Alternative Approach: // Update spec.repos (current code already does this at line 1340)
// Remove HTTP call to runner - let operator detect spec change and trigger clone
// This follows the Kubernetes reconciliation pattern better3. Legacy Code Still Embedded (Technical Debt)Location: Issue: Controller-runtime migration is only 50% complete. The new reconciler calls legacy // TODO(controller-runtime-migration): This is a transitional wrapper around the legacy
// handleAgenticSessionEvent() function (2,300+ lines). Future work should:
// 1. Extract phase-specific logic into separate functions
// 2. Use controller-runtime patterns (Patch, StatusWriter, etc.)
// 3. Remove handleAgenticSessionEvent() entirelyImpact:
Recommendation: File follow-up issue to complete refactoring. This PR already adds phase-based reconciliation ( 4. Mixed Reconciliation PatternsLocation: Issue: Mixing controller-runtime (for AgenticSessions) with legacy watch loops (for Namespaces and ProjectSettings): // Legacy watch loops still running
go handlers.WatchNamespaces()
go handlers.WatchProjectSettings()
// Controller-runtime for AgenticSessions
if err := mgr.Start(ctx); err \!= nil { ... }Impact: Increased complexity, harder to debug, inconsistent error handling. Recommendation: Migrate Namespace and ProjectSettings watchers to controller-runtime controllers in a follow-up PR. 🔵 Minor Issues5. Hardcoded Timeout in Runner HTTP CallLocation: client := &http.Client{Timeout: 120 * time.Second} // Should be configurableRecommendation: Extract to configuration constant or environment variable. 6. Missing Data Migration GuideIssue: Existing PVC-based sessions need migration path to S3 storage. Recommendation: Add migration documentation:
7. No Unit Tests for Token ProvisioningIssue: Token provisioning logic moved from backend to operator (171 lines deleted, new logic in operator). No new tests visible in PR. Recommendation: Add unit tests verifying:
Positive Highlights✅ Security Improvements
✅ Code Quality Improvements
✅ Observability Improvements
✅ Architecture Improvements
RecommendationsPriority 1 (Before Merge)
Priority 2 (Post-Merge)
Standards Compliance Matrix
Overall Compliance: 8/9 standards passed, 1/9 needs documentation Final Verdict✅ APPROVE WITH CONDITIONSThis PR represents a significant architectural improvement:
No blocking issues - code is safe to merge after:
Well done! This is a complex refactoring executed with care and attention to established patterns. Review Metadata:
🔍 View AI decision process (logs available for 90 days) 📋 View memory system files loaded (click to expand)What Amber Loaded for Code ReviewAmber automatically loaded these repository standards from the memory system:
Impact: This review used your repository's specific code quality standards, security patterns, and best practices from the memory system (PRs #359, #360) - not just generic code review guidelines. |
- Introduced a new state-sync component in the build and deploy workflows, enhancing the deployment process. - Added steps to deploy the observability stack in both components-build-deploy and prod-release-deploy workflows. - Updated kustomization to include the state-sync image for consistent image tagging across environments. - Enhanced environment variable settings to include the state-sync image in deployment configurations.
Claude Code ReviewSummaryThis is a major architectural refactoring that migrates from Jobs to Pods, adopts controller-runtime for the operator, integrates S3 storage, and adds comprehensive observability. The changes are well-structured and follow most project patterns, but there are several critical security and architecture issues that must be addressed before merge. Overall Assessment: 🟡 Significant work required - Core architecture is sound, but security gaps and pattern violations need fixing. Issues by Severity🚫 Blocker IssuesMust fix before merge: 1. Missing User Token Authentication on Workspace EndpointsLocation: k8sClt, sessDyn := GetK8sClientsForRequest(c)
if k8sClt != nil && sessDyn != nil {
// Uses user token - GOOD
}However, I don't see user token authentication checks on the new workspace endpoints mentioned in the PR description:
CRITICAL RULE VIOLATION: All user-facing endpoints MUST use Fix Required: func ListSessionWorkspace(c *gin.Context) {
reqK8s, reqDyn := GetK8sClientsForRequest(c)
if reqK8s == nil {
c.JSON(http.StatusUnauthorized, gin.H{"error": "Invalid or missing token"})
c.Abort()
return
}
// ... rest of handler
}Reference: 2. Type Assertion Without CheckingLocation: Good catch fixing this: // OLD (unsafe):
Metadata: item.Object["metadata"].(map[string]interface{})
// NEW (safe):
metadata, ok := item.Object["metadata"].(map[string]interface{})
if !ok {
c.JSON(http.StatusInternalServerError, gin.H{"error": "Invalid session metadata"})
return
}However, check the rest of the file for similar patterns - I see unsafe assertions in other locations. Reference: 🔴 Critical IssuesShould fix before merge: 3. Token Provisioning Moved to Operator - RBAC Verification NeededLocation: The PR removes 171 lines of token provisioning code from the backend: -// provisionRunnerTokenForSession creates a per-session ServiceAccount...
-func provisionRunnerTokenForSession(...) error {
- // ... 171 lines removed
-}New approach: Operator provisions tokens when creating pods. Issues:
Action Required:
Reference: 4. Controller-Runtime Migration IncompleteLocation: // TODO(controller-runtime-migration): This is a transitional wrapper around the legacy
// handleAgenticSessionEvent() function (2,300+ lines). Future work should:
// 1. Extract phase-specific logic into separate functions
// 2. Use controller-runtime patterns (Patch, StatusWriter, etc.)
// 3. Remove handleAgenticSessionEvent() entirelyIssue: The operator now uses controller-runtime framework BUT still delegates to the massive legacy
Recommendation:
Reference: 5. Missing OwnerReferences on New ResourcesLocations to verify:
CRITICAL RULE: All child resources MUST have OwnerReferences set for automatic cleanup. Pattern: ownerRef := v1.OwnerReference{
APIVersion: session.GetAPIVersion(),
Kind: session.GetKind(),
Name: session.GetName(),
UID: session.GetUID(),
Controller: boolPtr(true),
}Action Required: Verify operator sets OwnerReferences when creating:
Reference: 🟡 Major IssuesImportant to address: 6. Breaking API Changes Not VersionedLocation: PR Description Issue: Breaking changes should follow API versioning strategy:
Current approach: Direct deletion. This is acceptable IF:
Recommendation: Add deprecation notice to API docs ( 7. Observability Stack - Production ReadinessLocation: New components added:
Questions:
Action Required: Review resources:
requests:
memory: "256Mi"
cpu: "100m"
limits:
memory: "512Mi"
cpu: "500m"8. S3 Storage - Error HandlingLocation: New S3 integration for session artifacts. Questions:
Recommendation: Review
🔵 Minor IssuesNice-to-have improvements: 9. Magic Numbers in ConfigurationLocation: if maxConcurrent <= 0 {
maxConcurrent = 10 // Default to 10 concurrent reconcilers
}Better approach: Use constant const DefaultMaxConcurrentReconciles = 1010. Logging - Token Redaction IncompleteLocation: Token redaction in query strings ✅ GOOD if strings.Contains(param.Request.URL.RawQuery, "token=") {
path = strings.Split(path, "?")[0] + "?token=[REDACTED]"
}Missing: Redaction in request bodies (if tokens are POSTed). Verify this isn't needed. 11. Frontend - React Query Pattern ViolationsLocation: Action Required: Verify this file follows:
Reference: Positive Highlights✅ Excellent Architectural Decisions
✅ Code Quality
RecommendationsPrioritized Action ItemsBefore Merge (P0):
After Merge (P1):
Future (P2):
Testing RecommendationsBefore approving:
Security Review Score6/10 - Several critical security gaps:
After fixing blockers: Should reach 8/10. Final Verdict🟡 Requires Changes - Do not merge until:
The core architecture (Jobs → Pods, controller-runtime, S3 storage) is excellent and well-executed. Security and safety gaps are fixable and well-documented above. Reviewed by: Claude Code (AI Code Review Agent) 🔍 View AI decision process (logs available for 90 days) 📋 View memory system files loaded (click to expand)What Amber Loaded for Code ReviewAmber automatically loaded these repository standards from the memory system:
Impact: This review used your repository's specific code quality standards, security patterns, and best practices from the memory system (PRs #359, #360) - not just generic code review guidelines. |
1 participant
Overview
Major architectural refactoring: Jobs → Pods, S3 storage integration, controller-runtime adoption, and comprehensive observability.
Key Changes
Breaking Changes
POST /api/projects/:project/agentic-sessions/:session/workspace/enable- DELETEDPOST /api/projects/:project/agentic-sessions/:session/workspace/touch- DELETEDImpact Analysis:
These endpoints managed temporary content pods for completed sessions. With S3 storage, they're no longer needed.
Workspace access still works:
GET /workspace/list- ✅ WorksGET /workspace/file/*- ✅ WorksPUT /workspace/file/*- ✅ WorksNew Dependencies
Security Improvements
ListSessionWorkspace,GetSessionWorkspaceFile,GetWorkflowMetadata)serviceaccounts/tokencreate permission)Stats: +6,042 / -1,966 across 47 files