Define an Entity Merge algorithm (from OTEP #264)

[jsuereth]

Changes

Adds the merge algorithm specified in OTEP#264 to the Resource + Entity data model specification (both still in development).

This merge algorithm has been used in Entity prototypes for the past year and has not seen any change. It's been adaptable enough to handle several re-architectures of SDKs through prototyping, so we're confident in pushing it forward into the specification.

A few caveats:

• See the note on Resource DataModel about conflicting merge algorithms. This one currently focused only on merging entities into resource when there is a known set of entities in the existing resource.
• I've split the merge algorithm into two: The Entity portion about eligibility of merging and descriptive attribute merging, and then the more robust resource section.

Link to prototypes:

• Java: https://github.com/jsuereth/opentelemetry-java/blob/wip-entity-and-providers/sdk/common/src/main/java/io/opentelemetry/sdk/entity/internal/EntityUtil.java#L289
• TypeScript: [WIP] entities prototype opentelemetry-js#5620

[dmitryax]

This step is unclear to me. Does it mean that any attributes of the new_entity can potentially drop attributes from other existing entities in E? Shouldn't we make resolving any cross-entity attribute conflicts a requirement for adding an entity to a resource?

[jsuereth]

It's more that they override them.

Remember, in the datamodel, attributes are a separate thing from entities vs. in OTLP. So what this says, is in the datamodel, you remove it from the "loose" attributes, because it exists in the new entity's attributes. The new entity will provide a new value that lands in OTLP resource.

[dashpole]

IMO this deserves a clarifying note, as I think most readers (for better or worse) are more familiar with OTLP than the data model document, and assume data model == OTLP. Consider:

Note: For cases where Entity Attributes are encoded as references to Resource Attributes, as is the case for OTLP, this implies Entity Attributes take precedence over any Resource Attributes which are not associated with an Entity.

Even though entities in the data model have their own set of attributes, I still agree that entity conflict resolution needs to be a part of the merge process. E.g. if two entities claim to have detected different values for the same key, it seems like that still needs to be resolved when merging entities into a resource.

[jsuereth]

@dashpole Ah, yes, we need to add a new mechanism to deconflict entities that have the same attributes. We don't expect this to be allowed in semconv, but it's possible if this is open.

Happy to add a clarifying note.

[jsuereth]

Discussion from SIG -

Principle - preserve as much information as possible

Actions

• Define "keep highest priority entity on key conflicts, drop entity ref of lower priority entity (if identity attribute), just the attribute (if descriptive)"
• Define dropped_attributes_count on EntityRef
• Define dropped_entities_count on Resource

[jsuereth]

So I've updated the specification here. I think dropped_*_count is a follow PR or set of actions.

[dashpole]

How does an SDK determine the relative priority of entities?

[jsuereth]

resource detection configuration

[dashpole]

if applicable

nit: maybe rephrase to "if they are mergeable"? If applicable confused me here.

otherwise ignore new_entity.

Do we consider this an error (e.g. surfaced using the SDK error handler or logged as a warning in the collector)? This makes it sound like these cases should be silently ignored.

[dyladan]

Yes it is considered to be an error case (or at least a misconfiguration). I think the term "ignore" just relates to what the SDK should construct in order to emit telemetry. Logging an error is also acceptable

[dashpole]

IMO this deserves a clarifying note, as I think most readers (for better or worse) are more familiar with OTLP than the data model document, and assume data model == OTLP. Consider:

Note: For cases where Entity Attributes are encoded as references to Resource Attributes, as is the case for OTLP, this implies Entity Attributes take precedence over any Resource Attributes which are not associated with an Entity.

Even though entities in the data model have their own set of attributes, I still agree that entity conflict resolution needs to be a part of the merge process. E.g. if two entities claim to have detected different values for the same key, it seems like that still needs to be resolved when merging entities into a resource.

[jsuereth]

From Entities SIG - Should this be a failure?

We could require attributes can only be owned by one entity and make this be a runtime failure in SDKs + Collector to prevent this type of modelling.