Collaboration on AI provider specification?

[gr2m]

Hi friends! I'm working on the JavaScript AI SDK. We have a spec that went through a lot of iterations, it's at the heart of our provider + model abstraction: https://github.com/vercel/ai/tree/main/packages/provider

Right now it's all TypeScript, but I could see we move the spec into it's own repository/organization in order to collaborate with similar projects across language barriers.

Just a thought. Hope the specs have some useful insights for your work! Good luck, go PHP!

[felixarntz]

@gr2m Thanks for sharing! I'd love to find ways to collaborate, or at least compare approaches. Obviously a few development paradigms between PHP and JS/TS don't necessarily translate well, but there's a lot of conceptual overlap where reasoning about a specification would be a great opportunity.

For reference, I'd say the best place to inspect our provider + model abstraction is:

• https://github.com/WordPress/php-ai-client/blob/trunk/src/Providers/Contracts/ProviderInterface.php
• https://github.com/WordPress/php-ai-client/blob/trunk/src/Providers/Models/Contracts/ModelInterface.php
• And the more specific model interfaces like https://github.com/WordPress/php-ai-client/blob/trunk/src/Providers/Models/TextGeneration/Contracts/TextGenerationModelInterface.php etc.

For a deep dive, there's also https://github.com/WordPress/php-ai-client/blob/trunk/docs/ARCHITECTURE.md.

[felixarntz]

Based on my understanding of the AI SDK, two initial key differences that come to mind are:

1. On our end, we're trying to abstract away modalities, with a generate() method at the core that receives one or more modalities. The idea behind this generic method name is to support multimodal output. Since the message/content shape already supports all kinds of modalities, in principle a generative method could also return all kinds of modalities.
   • There are still wrapper methods for convenience (like generateText and generateImage), but they all go through to generate(['text']) and generate(['image']) for example. The reality is that most developers today will still use those wrappers because they are more intuitive, but we're thinking the more abstract method makes sense from a multimodal point of view.
2. Our SDK includes a discovery engine which allows to find suitable models from different providers based on the prompt modalities and other provided configuration parameters.
   • One of the big focuses for our SDK is to allow developers to use AI without having any idea about specific models at all, and more importantly to go without requiring a specific model. The latter is crucial for a distributed platform like WordPress, because all the plugins that may implement AI based features won't know which provider the respective site the plugin runs on has configured.
   • For instance, if a site owner configures credentials only for OpenAI, and a distributed plugin specifies an Anthropic model, it just won't work.
   • Therefore WordPress plugin developers are encouraged to either not specify any model at all, or specify a model "preference list".
   • To power all of that, discovery mechanism tries to identify a list of suitable models for the job, and that is then filtered down to the subset of models that can be used on the site (based on which providers are configured). And then either the first model from the (optional) given preference list is used for the job, or alternatively the first one from the list (mostly just as a fallback).

I'm curious to hear your perspectives on these ideas. But also if you have other ideas or discussion points, I'd love to learn more.

[JasonTheAdams]

Excited to talk more about this as well! Thanks for reaching out, @gr2m!