Spec: Add implementation note for determining manifest list format version

[yshcz]

Feature Request / Improvement

The current spec defines Avro file metadata requirements for manifest files in a clear table:

v1	v2	Key	Value
required	required	schema	JSON representation of the table schema at the time the manifest was written
optional	required	schema-id	ID of the schema used to write the manifest as a string
required	required	partition-spec	JSON representation of the partition spec used to write the manifest
optional	required	partition-spec-id	ID of the partition spec used to write the manifest as a string
optional	required	format-version	Table format version number of the manifest as a string
	required	content	Type of content files tracked by the manifest: "data" or "deletes"

But manifest list files have no equivalent specification for their Avro metadata, despite the Java implementation writing metadata such as format-version, snapshot-id, parent-snapshot-id, and sequence-number to manifest list files since 2020.

For manifests: #913 added format-version to code (2020-04), and #1499 added the spec (2020-10).
For manifest lists: #907 added format-version to code (2020-04), but there are no corresponding spec changes.

As a result, implementations have no standard way to detect the format version from a manifest list file alone. They are forced to either infer the version based on the presence of certain fields, or simply trust the table metadata version. The latter is unreliable in upgrade scenarios where a v2 table may contain v1 snapshots, introducing unnecessary complexity.

The following table might be a reasonable addition, though I'm not entirely certain about the requirements:

v1	v2	v3	Key	Value
required	required	required	snapshot-id	The snapshot ID for this manifest list as a string
required	required	required	parent-snapshot-id	The parent snapshot ID as a string
	required	required	sequence-number	The sequence number of the snapshot as a string
		required	first-row-id	The first row ID for row lineage as a string
optional	required	required	format-version	Table format version number as a string

Query engine

None

Willingness to contribute

• I can contribute this improvement/feature independently
• I would be willing to contribute this improvement/feature with guidance from the Iceberg community
• I cannot contribute this improvement/feature at this time

[RussellSpitzer]

We can't add in a requirement for avro metadata for any released version since that would be breaking for all existing manifest lists. We could have it be optional but I'm not sure thats really worth it.

I'm interested in why checking the schema isn't sufficient for the version check? I know it's a bit more complicated than reading a single value but is there a reason why it's not working in your use case? We consider adding this a as a requirement in v4 but we are also considering just dropping manifest_lists entiresly in V4 so it probably doesn't matter

[yshcz]

I was just exploring all the possible edge cases while implementing Iceberg and noticed this part of the spec felt inconsistent. Maybe we should at least specify the concrete format version detection mechanism for manifest list somewhere?

I'm fine with the avro schema based detection, but it's not documented anywhere, making it an implicit workaround rather than a standard method. The spec even includes a concrete algorithm for snapshot retention policy. Having the same level of specification for manifest lists would improve overall consistency. Even if v4 drops it, older tables will benefit from a clearer spec

[RussellSpitzer]

We could definitely add in an implementation note on how to determine format version, I don't think there is any problem with noting things we can already do that we want to codify as possible.

[yshcz]

Thanks for the feedback. I've updated the issue title and opened a PR.