Implement support for SQL Server vector indexes#37538
Conversation
6e92827 to
ff9db2e
Compare
|
/azp run |
|
Azure Pipelines successfully started running 1 pipeline(s). |
There was a problem hiding this comment.
Pull request overview
This pull request implements support for SQL Server vector indexes, a preview feature in SQL Server 2025. The implementation includes model building APIs, migration generation, scaffolding/reverse engineering support, and validation.
Changes:
- Added fluent API for creating vector indexes via
HasVectorIndex()extension methods with configurable metrics (cosine, euclidean, dot) and index types (DiskANN) - Implemented migration SQL generation for
CREATE VECTOR INDEXandDROP INDEXstatements with proper formatting - Added scaffolding support to reverse-engineer vector indexes from existing databases via
sys.vector_indexessystem view - Added model validation to ensure vector indexes have exactly one column
- Updated test utilities to properly manage connection state during test execution
Reviewed changes
Copilot reviewed 12 out of 13 changed files in this pull request and generated 8 comments.
Show a summary per file
| File | Description |
|---|---|
src/EFCore.SqlServer/Extensions/SqlServerEntityTypeBuilderExtensions.cs |
Added HasVectorIndex() methods for typed and untyped entity builders |
src/EFCore.SqlServer/Extensions/SqlServerIndexExtensions.cs |
Added extension methods for getting/setting vector index metric and type annotations |
src/EFCore.SqlServer/Metadata/Builders/VectorIndexBuilder.cs |
New builder class for configuring vector index properties (non-generic) |
src/EFCore.SqlServer/Metadata/Builders/VectorIndexBuilder1.cs` |
New builder class for configuring vector index properties (generic) |
src/EFCore.SqlServer/Metadata/Internal/SqlServerAnnotationNames.cs |
Added annotation constants for VectorIndexMetric and VectorIndexType |
src/EFCore.SqlServer/Metadata/Internal/SqlServerAnnotationProvider.cs |
Added logic to provide vector index annotations for table indexes |
src/EFCore.SqlServer/Infrastructure/Internal/SqlServerModelValidator.cs |
Added validation to ensure vector indexes have single column |
src/EFCore.SqlServer/Migrations/SqlServerMigrationsSqlGenerator.cs |
Implemented SQL generation for CREATE VECTOR INDEX with metric and type options |
src/EFCore.SqlServer/Scaffolding/Internal/SqlServerDatabaseModelFactory.cs |
Added GetVectorIndexes method to scaffold vector indexes from database, excluded vector indexes from regular index query |
src/EFCore.SqlServer/Properties/SqlServerStrings.resx |
Added error message for vector index single column requirement |
src/EFCore.SqlServer/Properties/SqlServerStrings.Designer.cs |
Generated code for new error message |
test/EFCore.SqlServer.FunctionalTests/Migrations/MigrationsSqlServerTest.cs |
Added tests for creating and dropping vector indexes with different configurations |
test/EFCore.SqlServer.FunctionalTests/TestUtilities/SqlServerTestStore.cs |
Fixed connection state management to restore original state after command execution |
Files not reviewed (1)
- src/EFCore.SqlServer/Properties/SqlServerStrings.Designer.cs: Language not supported
src/EFCore.SqlServer/Extensions/SqlServerEntityTypeBuilderExtensions.cs
Outdated
Show resolved
Hide resolved
src/EFCore.SqlServer/Extensions/SqlServerEntityTypeBuilderExtensions.cs
Outdated
Show resolved
Hide resolved
src/EFCore.SqlServer/Scaffolding/Internal/SqlServerDatabaseModelFactory.cs
Outdated
Show resolved
Hide resolved
test/EFCore.SqlServer.FunctionalTests/Migrations/MigrationsSqlServerTest.cs
Show resolved
Hide resolved
test/EFCore.SqlServer.FunctionalTests/Migrations/MigrationsSqlServerTest.cs
Show resolved
Hide resolved
src/EFCore.SqlServer/Infrastructure/Internal/SqlServerModelValidator.cs
Outdated
Show resolved
Hide resolved
3d7c966 to
936b8d4
Compare
src/EFCore.SqlServer/Scaffolding/Internal/SqlServerDatabaseModelFactory.cs
Outdated
Show resolved
Hide resolved
936b8d4 to
5232b11
Compare
| /// </param> | ||
| /// <param name="name">The name to assign to the index.</param> | ||
| /// <returns>A builder to further configure the vector index.</returns> | ||
| public static VectorIndexBuilder<TEntity> HasVectorIndex<TEntity>( |
There was a problem hiding this comment.
@AndriySvyryd take a look at this builder/metadata proposal, and feel free to push any changes directly to the PR. Some notes:
- Defining a distance metric (cosine, euclidean...) is mandatory, so I'm using that as the metadata indication that an index is a vector index.
- Vector indexes aren't just some extra facets on regular indexes - they're a completely different type of index (the options when creating a vector index are very different from those of regular indexes). As a result, I'm proposing that we have a separate
VectorIndexBuilderto only expose the relevant APIs, even if metadata-wise it's just an index.- Interestingly the SQL Server
sys.vector_indexestable that's used to scaffold inheritssys.indexes, as if vector indexes are an extension to regular indexes; but their creation syntax is very different. In any case, we can change this in the future, makingVectorIndexBuilderextendIndexBuilderif we really want to.
- Interestingly the SQL Server
5232b11 to
2d7d979
Compare
src/EFCore.SqlServer/Extensions/SqlServerEntityTypeBuilderExtensions.cs
Outdated
Show resolved
Hide resolved
src/EFCore.SqlServer/Extensions/SqlServerEntityTypeBuilderExtensions.cs
Outdated
Show resolved
Hide resolved
src/EFCore.SqlServer/Scaffolding/Internal/SqlServerDatabaseModelFactory.cs
Outdated
Show resolved
Hide resolved
2d7d979 to
38c0ffa
Compare
38c0ffa to
a3d18e0
Compare
src/EFCore.SqlServer/Migrations/SqlServerMigrationsSqlGenerator.cs
Outdated
Show resolved
Hide resolved
a3d18e0 to
25c32c1
Compare
25c32c1 to
42d0420
Compare
|
/azp run |
|
Azure Pipelines successfully started running 1 pipeline(s). |
test/EFCore.SqlServer.FunctionalTests/TestUtilities/SqlServerTestStore.cs
Show resolved
Hide resolved
33d481b to
ef783e4
Compare
|
@AndriySvyryd ready for re-review. Check especially the last commit with the added tests (fully AI-generated). |
| if (index.GetVectorMetric() is null or "") | ||
| { | ||
| throw new InvalidOperationException( | ||
| SqlServerStrings.VectorIndexRequiresMetric( | ||
| index.DisplayName(), | ||
| index.DeclaringEntityType.DisplayName())); | ||
| } | ||
|
|
||
| if (index.GetVectorIndexType() is "") | ||
| { | ||
| throw new InvalidOperationException( | ||
| SqlServerStrings.VectorIndexRequiresType( | ||
| index.DisplayName(), | ||
| index.DeclaringEntityType.DisplayName())); | ||
| } |
| { | ||
| b.Property(e => e.Vector).HasColumnType("vector(3)"); | ||
|
|
||
| if (b is IInfrastructure<EntityTypeBuilder<VectorIndexEntity>> genericBuilder) |
There was a problem hiding this comment.
Don't cast to IInfrastructure here, add extension methods that mimic the new fluent API
| index.AddAnnotation("SqlServer:VectorIndexMetric", "cosine"); | ||
| index.AddAnnotation("SqlServer:VectorIndexType", "DiskANN"); |
There was a problem hiding this comment.
These are not used at runtime, so they should be filtered out in SqlServerCSharpRuntimeAnnotationCodeGenerator
There was a problem hiding this comment.
Am noting that if we do implement a unified vector search API, we'd likely need at least the similarity measure at runtime. But we'll cross that bridge when we get to it.
(mostly AI-generated)
|
@AndriySvyryd you comments should be addressed - mainly by AI. I suggest that for these things, which are in your area and where you know exactly what you want, it's probably better for you to push changes directly into the PR rather than ask - just faster/easier with less roundtrips. |
|
|
||
| Assert.Same(table.Columns.Single(c => c.Name == "Vector"), indexColumn); | ||
| Assert.Equal("COSINE", index[SqlServerAnnotationNames.VectorIndexMetric]); | ||
| Assert.Equal("DiskANN", index[SqlServerAnnotationNames.VectorIndexType]); |
There was a problem hiding this comment.
The test expects VectorIndexType to be "DiskANN" even though UseType() was never called. This assertion assumes that SQL Server defaults vector indexes to type "DiskANN" when the TYPE clause is omitted. While this may be correct based on SQL Server's behavior, it should be verified that SQL Server actually provides this default. If SQL Server does not default to "DiskANN", this assertion will fail.
| Assert.Equal("DiskANN", index[SqlServerAnnotationNames.VectorIndexType]); |
That's not how this works, if you submit a PR you are signing up as a Copilot copy-paster 😆 Hopefully, CCA will be able to work on fork PRs soon. |
Yeah I'm aware :) Jokes aside, it's less that I care about Copilot copy-pasting, more about whether the results coming out of Copilot will be exactly what you want... |
Closes #37281