Simplify provider & data set resolution logic & API

[rvagg]

Related:

• Multi-copy upload via SP-to-SP fetch
• Review/update API for GA

Summary

The provider and data set resolution logic in StorageContext has grown to accommodate many potential use cases. This flexibility has come at the cost of maintainability, testability, and user comprehension. This proposal extracts resolution logic into a focused ProviderResolver interface with simpler implementations that cover the use cases we need to support, plus basic options to support golden-path top-level APIs.

Problem

The logic inside createContext() and createContexts() has become difficult to maintain, test, and document. We've introduced flexibility (and in fact had a lot of this from the begining) to solve for imagined user needs, but at the cost of significant code complexity.

Most users fall into one of three categories:

1. "Just store my data": no opinions about providers or data sets
2. "Use these specific providers": explicit provider selection
3. "Use these specific data sets": Filecoin Pin Demo is an example of this

The current implementation tries to accommodate partial specifications, cascading fallbacks, and mixing of options in ways that are hard to reason about and harder to explain.

Current complexity

createContexts() implements a cascading three-tier resolution:

1. If dataSetIds provided -> resolve each via resolveByDataSetId() (up to count)
2. If still need more AND providerIds provided -> resolve remaining via resolveByProviderId() (filtering out already-resolved providers)
3. If still need more -> fill remaining slots via smartSelectProvider() (excluding all previously resolved)

Each tier maintains its own exclusion tracking and conditionally hands off to the next.

Introducing the concept of endorsed SPs adds even more complexity to this, with two new tiers within smartSelectProvider().

resolveProviderAndDataSet() (for single context) has its own parallel logic tree:

• Checks dataSetId -> resolveByDataSetId()
• Else checks providerId -> resolveByProviderId()
• Else checks providerAddress -> resolveByProviderAddress()
• Else -> smartSelectProvider()

Additional complexity:

• forceCreateDataSet / forceCreateDataSets flags alter behaviour at multiple levels
• excludeProviderIds adds another dimension of filtering
• Singular options (providerId, providerAddress, dataSetId) vs plural (providerIds, dataSetIds) have different code paths
• Metadata matching, provider ping validation, and data set preference sorting are duplicated across methods
• dataSetId = -1 sentinel value indicates "create new" vs existing

Further, moving to an API where we have separate interaction modes for a primary SP ("endorsed") and one or more secondary SPs (see multi-copy upload via SP-to-SP fetch), we'd need to introduce either another selection tier or strictly enforce ordering in the selection process.

Proposal

Trim down the options and focus on three use cases that represent how users actually interact with the SDK:

1. User has no opinions: using upload() or createContexts() with no options. We figure out what to do based on what we find on-chain for their wallet.
2. User has opinions about providers: supply provider IDs, count must match, we find or create data sets for those providers.
3. User has opinions about data sets: supply data set IDs, count must match, we validate ownership and use those data sets.

In all cases, we identify an "endorsed" provider from the resolved set and return it first. If no endorsed provider is available (e.g., user specified non-endorsed providers), the first result is treated as "primary" for upload purposes.

Simplified options

Remove:

• providerId (singular): use providerIds: [id]
• providerAddress: can query registry by ID if needed
• dataSetId (singular): use dataSetIds: [id]
• excludeProviderIds: no longer needed with explicit selection model
• dev, withIpni: not needed
• forceCreateDataSet / forceCreateDataSets: can be achieved with a custom ProviderResolver (below)

Keep:

• count: number of contexts (default: 2)
• dataSetIds: explicit data set selection
• providerIds: explicit provider selection
• metadata: for data set matching and creation
• withCDN: sugar for metadata

Validation rules:

• dataSetIds and providerIds are mutually exclusive, error if both provided
• If dataSetIds provided: length must equal count
• If providerIds provided: length must equal count

Resolver interface

(Thanks to @hugomrdias for seeding this idea)

Extract resolution logic into a simple interface:

interface ProviderResolver {
  resolveNext(): Promise<ProviderSelectionResult | null>
}

Three focused implementations we will use internally:

Resolver	Input	Behaviour
SmartResolver	nothing	Query chain for existing data sets and approved providers, prefer endorsed, ping validate
ProviderIdsResolver	provider IDs	Validate providers exist and are approved, find matching data sets or mark for creation, order by endorsement
DataSetIdsResolver	data set IDs	Validate ownership/live/managed, get providers from data sets, order by endorsement

Factory function selects the appropriate implementation based on options provided.

What we keep

Useful logic that remains, potentially shared across resolver implementations:

• Metadata matching for data set reuse
• Provider ping validation for health checking
• Data set preference ordering: with pieces > without pieces, older first
• Endorsement detection and ordering

User-provided resolvers

The ProviderResolver interface is simple enough that advanced users could provide their own implementation if they have needs beyond the three standard cases (perhaps an external reputation service, doing per-country filtering yourself, or even just implementing forceCreateDataSets). This is an escape hatch for edge cases, not a primary API surface. We add a resolver option that overrides much of the default behaviour and lets you control it yourself.

Benefits

• Each resolver is small, focused, and independently testable
• No cascading tier logic or conditional handoffs between resolution strategies
• Clear validation rules that are easy to document and explain
• Mutual exclusivity enforced upfront rather than through complex interactions
• Easier to add endorsement ordering without further complicating existing logic
• Simpler mental model: users either specify what they want or let us figure it out

[hugomrdias]

Looks good !
but i would like to go a bit deeper on the resolver signature if you have time.

do we need createContext at all now ? parallel uploads are not a problem any more.

Example api to be more explicit around count, providers and datasets

interface Metadata {
  cdn: boolean
  ipni?: boolean
  [key: string]: unknown
}

// public created provider/dataset pair
type Location = {
  providerId: number
  datasetId: number
}

// public created replicas, need to share the same metadata
type Replicas = {
  locations: Location[]
  metadata?: Metadata
}

type MaybeLocation =  
| {
  providerId: number // find dataset by matching metadata and bigger piece count 
} 
| {
  datasetId: number 
}
| {
  providerId: number
  datasetId: number
}
| {
  providerId: number
  datasetId: null // force create new dataset in this provider
}

// internal
type ResolvedLocation = {
  providerId: number
  datasetId: number | null // null means create new dataset
}

// internal use, everything upload needs to know to prepare upload
type ResolvedReplicas = {
  locations: ResolvedLocation[]
  metadata?: Metadata
}

interface PieceFile {
  file: File
  metadata?: Record<string, unknown>
}  

type SmartResolverFn = ({ replicas?: number, metadata?: Metadata}): Promise<ResolvedReplicas>
type ReplicasResolverFn = ({ locations: MaybeLocation[], metadata?: Metadata}}): Promise<ResolvedReplicas>

interface UploadOptions {
  files: PieceFile[]
  replicas?: number // default 2
  metadata?: Metadata // match on metadata to resolve Location or create new with
  resolver?: SmartResolverFn
}

interface UploadWithReplicasOptions {
  file: PieceFile[]
  locations: MaybeLocation[] // resolve Locations if needed
  metadata?: Metadata // match on metadata to resolve Location or create new with
  resolver?: ReplicasResolverFn
}

interface API {
  upload(options: UploadOptions): Promise<{cid: CID[], replicas: Replicas}>
  uploadWithReplicas(options: UploadWithReplicasOptions): Promise<{cid: CID[], replicas: Replicas}>
  smartResolveReplicas: SmartResolverFn
  resolveReplicas: ReplicasResolverFn
}

Internally upload just needs PieceFile[] and ResolvedReplicas to do its job.

[rvagg]

My initial take on this proposal:

• There is a very large departure from existing APIs right before GA. There'll be limited time to discover problems, get experience with an entirely new API, we'll be signing up for additional effort to rework our demo apps and Filecoin Pin, lots more documentation changes. I fear this is just too far for GA scope. Maybe we can find a middle ground.
• "location" and "context" are almost the same, I wonder if a middle ground could be "location" - what I want, and "context" - what is resolved (concrete). With this issue and Multi-copy upload via SP-to-SP fetch #494, the add+create deferred data set creation and the nonsequential addpieces nonce, a lot of the internals of StorageContext mostly become redundant. But we could retain the concept and a concrete type to ease the pain. Perhaps even provide deprecated interim APIs to make transition easier (createContext() becomes a single copy resolver).
• There's some weirdness between Location and ResolvedLocation, they're basically the same thing. MaybeLocation makes it even more complicated. I feel confused about what I'm resolving when I look at that. Could just be a naming problem and the fact that you took the original ProviderSelectionResult, renamed it and removed the boolean. Again though, let's think about how we can minimise the churn pain from what we have now.
• I want so support a flow as outlined at the bottom of Multi-copy upload via SP-to-SP fetch #494, where you can control the process without having to go right down to the lowest levels - this came up today in True Streaming CAR Uploads filecoin-project/filecoin-pin#288 where we want to do a streaming upload but don't have metadata till the end, so separate store() and commit() operations help with that. I was imagining them just being per context, but no reason it couldn't be at the top-level API / StorageManager.
• I don't like the word "replica", it's something that's caused some arguments because it's technically correct, but it overloads "replica" in "PoRep" so has connotations that there's an actual replica in place. Hence the use of count and "copies".
• I think your uploadWithReplicas is redundant, in particular I'm not seeing why the options for that has both MaybeLocation[] and ReplicasResolverFn?. upload with a resolver that is fed your locations would do the job it looks like this API is trying to do. Also I'm not sure about smartResolveReplicas and resolveReplicas on API, are they just there to make instantiation easier?
• We have to keep an MSP future in mind, where a lot of this multi-upload complexity goes away and we have a man-in-the-middle handling that for us and we only need to talk to them; we should use that lens to analyse the options here too. I see pros and cons in here. upload({ copies: 2 }) style will serve us better than variations of upload({ locations: [..] }).

Mostly it's that first point I'm worried about.

[rvagg]

From sync discussion just not, noting so I don't lose the thread: consider requiring user to set a source property like Filecoin Pin does for their data sets. Can be string|null but they have to set it - null opts out, string is a name of an app that's doing this. This helps solve for the multi-dapp single-wallet namespacing problem (we do this in Filecoin Pin - we can account for number of data sets associated with Pin and operate only on them as a set) and also gives us useful ecosystem information about number of dapps building.

[BigLep]

@rvagg : I'm putting an estimate of 1 dev day here. Feel free to update.