Log diagnostics immediately after mutator ends (#3175)

## Changes
- New package: logdiag with
- logdiag.LogDiag(ctx, d): render and logs Diagnostic object on stderr
right away. Counts diagnostic type (error, warning, recommendation) on
the context variable.
- logdiag.HasError(ctx) — return true if there were any errors seen so
far.
- New functions bundle.ApplyContext/ApplySeqContext (to replace
bundle.Apply/ApplySeq). These functions do not return diagnostics, but
calls LogDiag on every diagnostic returned by the mutator.

bundle.Apply and bundle.ApplySeq are still available as wrappers around
Context counterparts. The main use case for these is unit tests.

Note, with this PR mutators may choose whether to use LogDiag to log
diagnostics right away or to return it as usual. This has exactly the
same effect (except if you return it it is delayed a bit). For new code,
LogDiag should be preferred, eventually mutator interface will be
modified not to return diagnostics. Note, even if you use LogDiag,
bundle.Apply will still return the diagnostics logged.

Other behavior changes:
- diagnostics are now always logged to stderr; previously "bundle
summary" non-json would print them to stdout.
- slashes in the filenames in diagnostics are normalized to have forward
slashes, to help with acceptance tests.
- summary command won't print recommendations (but it'll keep on
printing warnings)

## Why
Collection of diagnostics and printing them at the very end is poor user
and developer experience.

User experience:
 - Users will miss diagnostics if process crashes/panics; 
- For long running deployments, useful information is not printed right
away;
- For deploys with interactive approval, diagnostics may show issues
that should be fixed before the deployment. Since they not shown, users
cannot make informed choice.

Developer experience:
 - High accidental complexity to propagate the diagnostics properly.
- Various bugs over time where diagnostics were lost due to logic or
threading bugs (#2057, #2927,
#3123)

## Tests
Existing tests.

Note, we have no coverage for shell completion in bundle.

---------

Co-authored-by: Pieter Noordhuis <pieter.noordhuis@databricks.com>

Author: denik

NEXT_CHANGELOG.md

@@ -3,6 +3,10 @@
 ## Release v0.259.0
 
 ### Notable Changes
+* Diagnostics messages are no longer buffered to be printed at the end of command, flushed after every mutator ([#3175](https://github.com/databricks/cli/pull/3175))
+* Diagnostics are now always rendered with forward slashes in file paths, even on Windows ([#3175](https://github.com/databricks/cli/pull/3175))
+* "bundle summary" now prints diagnostics to stderr instead of stdout in text output mode ([#3175](https://github.com/databricks/cli/pull/3175))
+* "bundle summary" no longer prints recommendations, it will only prints warnings and errors ([#3175](https://github.com/databricks/cli/pull/3175))
 
 ### Dependency updates
 

acceptance/auth/bundle_and_profile/output.txt

@@ -21,6 +21,7 @@ Exit code: 1
 >>> errcode [CLI] current-user me -t dev -p profile_name
 Error: cannot resolve bundle auth configuration: the host in the profile (https://non-existing-subdomain.databricks.com) doesn’t match the host configured in the bundle ([DATABRICKS_TARGET]). The profile "DEFAULT" has host="[DATABRICKS_TARGET]" that matches host in the bundle. To select it, pass "-p DEFAULT"
 
+
 Exit code: 1
 
 === Bundle commands load bundle configuration when no flags, validation OK

acceptance/bundle/apps/config_section/output.txt

@@ -13,11 +13,11 @@ Workspace:
 Found 1 warning
 
 >>> [CLI] bundle deploy
-Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/test-bundle/default/files...
-Deploying resources...
-Updating deployment state...
-Deployment complete!
 Warning: App config section detected
 
 remove 'config' from app resource 'myapp' section and use app.yml file in the root of this app instead
 
+Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/test-bundle/default/files...
+Deploying resources...
+Updating deployment state...
+Deployment complete!

acceptance/bundle/deploy/jobs/shared-root-path/output.txt

@@ -1,15 +1,19 @@
 
 >>> [CLI] bundle deploy
+Warning: the bundle root path /Workspace/Shared/[USERNAME]/.bundle/[UNIQUE_NAME] is writable by all workspace users
+
+The bundle is configured to use /Workspace/Shared, which will give read/write access to all users. If this is intentional, add CAN_MANAGE for 'group_name: users' permission to your bundle configuration. If the deployment should be restricted, move it to a restricted folder such as /Workspace/Users/<username or principal name>.
+
 Uploading bundle files to /Workspace/Shared/[USERNAME]/.bundle/[UNIQUE_NAME]/files...
 Deploying resources...
 Updating deployment state...
 Deployment complete!
+
+>>> [CLI] bundle destroy --auto-approve
 Warning: the bundle root path /Workspace/Shared/[USERNAME]/.bundle/[UNIQUE_NAME] is writable by all workspace users
 
 The bundle is configured to use /Workspace/Shared, which will give read/write access to all users. If this is intentional, add CAN_MANAGE for 'group_name: users' permission to your bundle configuration. If the deployment should be restricted, move it to a restricted folder such as /Workspace/Users/<username or principal name>.
 
-
->>> [CLI] bundle destroy --auto-approve
 The following resources will be deleted:
   delete job foo
 

acceptance/bundle/deploy/mlops-stacks/output.txt

@@ -69,10 +69,6 @@ Workspace:
 Found 2 warnings
 
 >>> [CLI] bundle deploy
-Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/project_name_[UNIQUE_NAME]/dev/files...
-Deploying resources...
-Updating deployment state...
-Deployment complete!
 Warning: expected a string value, found null
   at targets.dev.workspace.host
   in databricks.yml:34:12
@@ -81,6 +77,10 @@ Warning: unknown field: description
   at resources.experiments.experiment
   in resources/ml-artifacts-resource.yml:21:7
 
+Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/project_name_[UNIQUE_NAME]/dev/files...
+Deploying resources...
+Updating deployment state...
+Deployment complete!
 
 >>> [CLI] bundle summary -o json
 Warning: expected a string value, found null
@@ -107,11 +107,27 @@ Warning: unknown field: description
   at resources.experiments.experiment
   in resources/ml-artifacts-resource.yml:21:7
 
+Warning: expected a string value, found null
+  at targets.dev.workspace.host
+  in databricks.yml:34:12
+
+Warning: unknown field: description
+  at resources.experiments.experiment
+  in resources/ml-artifacts-resource.yml:21:7
+
 {
   "name": "[dev [USERNAME]] dev-project_name_[UNIQUE_NAME]-batch-inference-job"
 }
 
 >>> [CLI] bundle destroy --auto-approve
+Warning: expected a string value, found null
+  at targets.dev.workspace.host
+  in databricks.yml:34:12
+
+Warning: unknown field: description
+  at resources.experiments.experiment
+  in resources/ml-artifacts-resource.yml:21:7
+
 The following resources will be deleted:
   delete job batch_inference_job
   delete job model_training_job

acceptance/bundle/deployment/bind/job/job-abort-bind/output.txt

@@ -3,7 +3,7 @@
 Created job with ID: [NUMID]
 
 === Expect binding to fail without an auto-approve flag:
-Error: failed to bind the resource, err: This bind operation requires user confirmation, but the current console does not support prompting. Please specify --auto-approve if you would like to skip prompts and proceed.
+Error: This bind operation requires user confirmation, but the current console does not support prompting. Please specify --auto-approve if you would like to skip prompts and proceed.
 
 === Deploy bundle:
 >>> [CLI] bundle deploy --force-lock

acceptance/bundle/telemetry/deploy-mode/output.txt

@@ -5,15 +5,15 @@ Deploying resources...
 Deployment complete!
 
 >>> [CLI] bundle deploy -t prod
-Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/test-bundle/prod/files...
-Deploying resources...
-Deployment complete!
 Recommendation: target with 'mode: production' should set 'workspace.root_path' to make sure only one copy is deployed
 
 A common practice is to use a username or principal name in this path, i.e. use
 
   root_path: /Workspace/Users/[USERNAME]/.bundle/${bundle.name}/${bundle.target}
 
+Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/test-bundle/prod/files...
+Deploying resources...
+Deployment complete!
 
 >>> cat out.requests.txt
 {

acceptance/bundle/variables/host/output.txt

@@ -55,7 +55,7 @@ Exit code: 1
 >>> errcode [CLI] bundle summary
 Warning: Variable interpolation is not supported for fields that configure authentication
   at workspace.host
-  in [TEST_TMP_DIR]/databricks.yml:10:9
+  in databricks.yml:10:9
 
 Interpolation is not supported for the field workspace.host. Please set
 the DATABRICKS_HOST environment variable if you wish to configure this field at runtime.

acceptance/pipelines/deploy/render-diagnostics-warning/output.txt

@@ -1,11 +1,11 @@
 
 >>> [PIPELINES] deploy
-Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/render-diagnostics-warning-pipeline/default/files...
-Deploying resources...
-Updating deployment state...
-Deployment complete!
 Warning: unknown field: unknown_property
   at resources.pipelines.test-pipeline
   in databricks.yml:8:7
 
+Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/render-diagnostics-warning-pipeline/default/files...
+Deploying resources...
+Updating deployment state...
+Deployment complete!
 <EOL>

bundle/artifacts/build.go

@@ -13,6 +13,7 @@ import (
 	"github.com/databricks/cli/libs/diag"
 	"github.com/databricks/cli/libs/exec"
 	"github.com/databricks/cli/libs/log"
+	"github.com/databricks/cli/libs/logdiag"
 	"github.com/databricks/cli/libs/patchwheel"
 	"github.com/databricks/cli/libs/utils"
 )
@@ -28,11 +29,9 @@ func (m *build) Name() string {
 }
 
 func (m *build) Apply(ctx context.Context, b *bundle.Bundle) diag.Diagnostics {
-	var diags diag.Diagnostics
-
 	cacheDir, err := b.CacheDir(ctx)
 	if err != nil {
-		diags = append(diags, diag.Diagnostic{
+		logdiag.LogDiag(ctx, diag.Diagnostic{
 			Severity: diag.Warning,
 			Summary:  "Failed to set up cache directory",
 		})
@@ -44,20 +43,20 @@ func (m *build) Apply(ctx context.Context, b *bundle.Bundle) diag.Diagnostics {
 		if a.BuildCommand != "" {
 			err := doBuild(ctx, artifactName, a)
 			if err != nil {
-				diags = diags.Extend(diag.FromErr(err))
+				logdiag.LogError(ctx, err)
 				break
 			}
 
 			// We need to expand glob reference after build mutator is applied because
 			// if we do it before, any files that are generated by build command will
 			// not be included into artifact.Files and thus will not be uploaded.
 			// We only do it if BuildCommand was specified because otherwise it should have been done already by artifacts.Prepare()
-			diags = diags.Extend(bundle.Apply(ctx, b, expandGlobs{name: artifactName}))
+			bundle.ApplyContext(ctx, b, expandGlobs{name: artifactName})
 
-			// After bundle.Apply is called, all of b.Config is recreated and all pointers are invalidated (!)
+			// After bundle.ApplyContext is called, all of b.Config is recreated and all pointers are invalidated (!)
 			a = b.Config.Artifacts[artifactName]
 
-			if diags.HasError() {
+			if logdiag.HasError(ctx) {
 				break
 			}
 
@@ -66,20 +65,19 @@ func (m *build) Apply(ctx context.Context, b *bundle.Bundle) diag.Diagnostics {
 		if a.Type == "whl" && a.DynamicVersion && cacheDir != "" {
 			b.Metrics.AddBoolValue(metrics.ArtifactDynamicVersionIsSet, true)
 			for ind, artifactFile := range a.Files {
-				patchedWheel, extraDiags := makePatchedWheel(ctx, cacheDir, artifactName, artifactFile.Source)
+				patchedWheel := makePatchedWheel(ctx, cacheDir, artifactName, artifactFile.Source)
 				log.Debugf(ctx, "Patching ind=%d artifactName=%s Source=%s patchedWheel=%s", ind, artifactName, artifactFile.Source, patchedWheel)
-				diags = append(diags, extraDiags...)
 				if patchedWheel != "" {
 					a.Files[ind].Patched = patchedWheel
 				}
-				if extraDiags.HasError() {
+				if logdiag.HasError(ctx) {
 					break
 				}
 			}
 		}
 	}
 
-	return diags
+	return nil
 }
 
 func doBuild(ctx context.Context, artifactName string, a *config.Artifact) error {
@@ -106,26 +104,29 @@ func doBuild(ctx context.Context, artifactName string, a *config.Artifact) error
 	return nil
 }
 
-func makePatchedWheel(ctx context.Context, cacheDir, artifactName, wheel string) (string, diag.Diagnostics) {
+func makePatchedWheel(ctx context.Context, cacheDir, artifactName, wheel string) string {
 	msg := "Failed to patch wheel with dynamic version"
 	info, err := patchwheel.ParseWheelFilename(wheel)
 	if err != nil {
-		return "", []diag.Diagnostic{{
+		logdiag.LogDiag(ctx, diag.Diagnostic{
 			Severity: diag.Warning,
 			Summary:  msg,
 			Detail:   fmt.Sprintf("When parsing filename \"%s\" encountered an error: %s", wheel, err),
-		}}
+		})
+		return ""
 	}
 
 	dir := filepath.Join(cacheDir, "patched_wheels", artifactName+"_"+info.Distribution)
 	createAndCleanupDirectory(ctx, dir)
 	patchedWheel, isBuilt, err := patchwheel.PatchWheel(wheel, dir)
 	if err != nil {
-		return "", []diag.Diagnostic{{
+		logdiag.LogDiag(ctx, diag.Diagnostic{
 			Severity: diag.Warning,
 			Summary:  msg,
 			Detail:   fmt.Sprintf("When patching %s encountered an error: %s", wheel, err),
-		}}
+		})
+		return ""
+
 	}
 
 	if isBuilt {
@@ -134,7 +135,7 @@ func makePatchedWheel(ctx context.Context, cacheDir, artifactName, wheel string)
 		log.Infof(ctx, "Patched wheel (cache) %s -> %s", wheel, patchedWheel)
 	}
 
-	return patchedWheel, nil
+	return patchedWheel
 }
 
 func createAndCleanupDirectory(ctx context.Context, dir string) {