Allow `"$ref"` for LDOs

[handrews]

It's fairly common links to connect multiple source resources to the same target resource. This often involves a lot of duplication in the LDO, but only the schema keywords can be re-used through "$ref".

This can be annoying when the link is almost entirely the same. In particular, "href" and "hrefSchema" need to match, so they are best re-used in pairs, which is not currently possible.

Since "links" is an array, having a schema consisting of just a "links" array is one re-use option, but only if the desired effect is essentially concatenating the arrays. Tweaking an individual LDO (use case: same "href", "hrefSchema", "targetSchema" and "rel", but applying a different title at each point of use) is somewhere between challenging and impossible.

I propose that we allow "$ref" for an LDO (as elements of the "links" array).

Since keeping track of array positions is a pain, we should add a new "linkDefinitions" keyword, which is an object where each property value is an LDO instead of a schema.

---

I'm fine with waiting a while to get feedback on draft-06 and see how much this comes up in practice, but it has come up even in some fairly trivial examples I've tried to write recently.

[jdesrosiers]

I have no objection to allowing $ref for LDOs, but I'm not seeing how that solves the problem you describe.

same "href", "hrefSchema", "targetSchema" and "rel", but applying a different title at each point of use

$ref allows for reuse, not extension. LDOs would need to support some kind of extension feature as well.

[handrews]

I'm not trying to override fields, I'm trying to compose them, which is how we use "allOf" in general:

"links": [
  {
    "title": "All the Things",
    "rel": "collection",
    "allOf": [{"$ref": "#/definitions/thingQueryingLink"}]
  }
],
"definitions": {
  "thingQueryingLink": {
    "href": "/things{?a,b,c,d}",
    "hrefSchema": {
      "type": "object",
      "properties": {
        "a": {...},
        "b": {...},
        "c": {...}
      }
    }
  }
}

and that works just fine. And if I want to link from some other context resource to a pre-filtered instance of the target collection, I just do the same thing with the correct different rel, and also use the allOf to pre-pin one of the filters:

"links": [
  {
    "title": "My Related Things",
    "rel": "tag:example.com,2017:fancythings",
    "allOf": [
      {"$ref": "#/definitions/thingQueryingLink"},
      {
        "hrefSchema": {
          "properties": {"a": {"const": "fancy"}}
        }
      }
    ],
  }
]

As usual I'm coming up with this off the top of my head so if something doesn't make sense it's probably an error, just ask.

Anyway, "allOf" has the effect of "apply all of these things separately". In these examples, that works fine. Overlying a const property onto the "hrefSchema" is a very common sort of thing to do with `"allOf". There is no conflict with the other keywords.

If you try to use "$ref" something with a "rel" in your "allOf", and also supply a different "rel" , or "$ref" two different re-usable links which define conflicting "rel"s (or "href", etc.) we do need to declare what that means. Options include:

• You just can't. It's an error
• If there is a value for the field outside of the "allOf", then it overrides the field anywhere it appears inside a branch of the "allOf"
• Define an order within the "allOf" (either first or last wins)

We have the same problem in several other situations for documentation and UI work with "title" and "description", and several proposals to address it. That topic is one of the core topics for draft-07.

[jdesrosiers]

As I suspected there was something missing. Just allowing $ref to reference an LDO is not enough. We would also have to introduce allOf (or something like it) as an LDO keyword.

I'm thinking that using allOf might not be the right choice tho. It would have slightly different semantics than the validation allOf keyword. You've identified some of the problems already. I'd rather not have a keyword that means one thing for validation and another thing for links.

[handrews]

Yeah, looking back that's kind of a glaring omission, isn't it? I filed this really quickly when I ran across some annoying re-use issue and was like "this should be easier!" It was more a "don't forget this" than a carefully considered proposal :-P

Anyway, yeah, I agree that using something other than allOf would be preferred.

This might end up related to the Documentation project (which I really need to launch properly- we have the repo set up). The semantics needed for links are more like those needed to customize title and description at point of use. Not identical, but worth thinking about.

---

Alternatively, the only real problem that I have here is that I want to be able to keep href and hrefSchema pairs in some form of re-usable chunk. Because they can be complicated and need to be kept in synch. Everything else that I'd want to re-use is an independent schema already, or so simple that it's not a problem to just repeat it.

So that's the part I think is really worth solving. We don't necessarily have to solve that with general link re-use/merging/whatever. One option could even be:

"links": [
  {
    "rel": "self",
    "href": {
      "uri": "/foo/{id}/bar{?a,b,c}",
      "uriSchema": {
      }
    }
  }
]

and then allow a reference for the href keyword that references a uri with optional uriSchema. That's a more minimal approach that doesn't require an allOf. Since the goal is to re-use pairs, there's no point in adding a mechanism to split or selectively overwrite pairs (e.g. allOf). You can already do that.

[handrews]

Another use case that has come up is defining an LDO without a 'rel' and then using it multiple times with different values of rel (#350). This seems to add weight to a generic re-use for entire LDOs. I also like this because the Hyper-Schema spec already makes a point about LDOs being usable on their own, without validation. That means that both validation schemas and LDOs are first-class entities, so making them both re-usable makes sense.

Still not 100% sure what to do about "allOf". If we were to use it with LDOs, then extending its behavior into "hrefSchema", "submissionSchema" and "targetSchema" seems like it would produce the desired results. But perhaps it's better to use a new keyword ("linkTemplate"? although "template" may imply too much functionality) and to define that in some cases with schemas within the LDO, it simply behaves like "allOf"?

I'd like to sort this out enough to write a PR and get feedback on something more concrete soon.

[handrews]

Questions to resolve:

• Expand $ref to work with LDOs, or add something like $linkRef? (I prefer $ref)
• Use definitions for LDOs, or add a new keyword like linkDefinitions?
  • Allowing definitions to contain LDOs directly would mean that hyper-schemas are invalid validation schemas
  • Requiring a schema like {"links": [{...}]} in oder to put LDOs under definitions indirectly gets back to arrays being very confusing to re-use
  • I prefer linkDefinitions or some similar keyword to add a new re-use section for Hyper-Schema
• Use allOf or a new keyword specifically for re-using LDOs?
  • allOf is pretty close, and is correct of the LDO keywords that take a schema
  • however, resolving conflicts among the allOf branches is not a concept that exists in validation
  • A new keyword (useLink? linkBase? ??) could be given LDO-specific behaviors:
    • Acts like allOf for schema-valued LDO keywords
    • Only one href allowed among the partial LDOs being combined
    • Acts like anyOf for rel, submissionEncType, and mediaType
    • (I just made that anyOf bit up while typing it, it may not actually make sense)

Any other open issues? I am leaning towards a new keyword, but it clearly has some complexities.

I am also open to other suggestions on how to facilitate re-use. Nesting closely related keywords such as hrefand hrefSchema and defining simpler re-use behavior for them is also a possibility that I haven't spent much time with. I'm not thrilled about nesting keywords but it is an option.

[handrews]

As noted above, this is actually somewhat complex. I'm booting it out to the next draft. I may experiment with some implementations in doca if I really have clear use cases, and let that help determine what behavior is worth standardizing, if any.

[handrews]

Ultimately, allowing an array for rel (per #350 (comment)) and using regular $ref techniques for hrefSchema while simply copying and pasting href itself might be sufficient.

URI Templates can be pretty annoying to duplicate, but unless we get real feedback that it's a problem that needs solving, I'm inclined to close this without further action.

Any objections?