Add option to programmatically customization declarative config components, add supplementary guidance

[jack-berg]

Resolves #4583.

We've had discussions in the declarative config SIG recently about how programmatic configuration interacts with declarative config. Questions of what should take priority, and whether there should be options to programmatically customize (#4583) to address concepts which are not expressible in declarative config.

The goal here is to avoid the type of spec ambiguity which resulted SDKs having such different behavior in environment variable <> programmatic configuration intersection. Declarative configuration is green field and for a short time longer is still experimental, and so we are not bound by historical constraints.

Reading through the relevant specs, I do not think there is any ambiguity about configuration interface priority. create returns resolved top-level SDK components. If a SDK supports updating components, then the components returned by create will be subject to whatever semantics the SDK already has decided for how updates occur to SDK components programmatically created to begin with. To make this clear, I opted to add a new supplementary guidelines document to give advice to implementers without cluttering or adding more normative language.

On a related note, I thought it was pertinent to address #4583 simultaneously, but adding explicit language to allow (i.e. normative MAY) implementations of create to allow programmatic customization of SDK components as they are initialized.

cc/ @open-telemetry/configuration-approvers, @JamieDanielson, @maryliag, @pellared

[pellared]

Loose idea, I have not spend to much time on analyzing it.

Because we want the declarative config to act in a WYSIWYG, then maybe instead of going forward with #4583 we should try adding some "configuration hooks" to the declarative config.

I mean something more or less like

file_format: 1.0
tracer_provider:
  processors:
    - batch:
        exporter:
          otlp_http:
hooks:
  - custom_otlp_executor_service

Hooks would be just another type of component that could be registered. Maybe each component type should accept a hook? I do not know...

This would make it visible in the file that something non-standard is happening.
It also forces to make the programmatical customization configurable (at least enabled/disabled, but they could have arguments as well).

[jack-berg]

Because we want the declarative config to act in a WYSIWYG, then maybe instead of going forward with #4583 we should try adding some "configuration hooks" to the declarative config.

@pellared I think this is an interesting idea, and like how it puts the onus on the user to select which hooks they want used, implying that hooks they have not selected will not be used. That provides users a good degree of control, preventing SDKs from being being initialized with unexpected / surprising options.

I'd like to find a way for standardization of programmatic customization to not be a blocker for stability. Programmatic customization is a nice feature, and will be important to some users, but declarative config will still be useful to most users without any programmatic customization options (whether they're standardized or not).

To accomplish this, I propose modifying the #4568 to mark this new programmatic configuration section of create as "in development". This would preserve the ability for us to prototype and potentially standardize.