Refactored the codebase to replace raw SQL connections and Dapper with Entity Framework Core (EF Core). Modified `Program.cs` to configure EF Core services. Refactored `VectorSearchService` to use EF Core for all database operations. Updated project dependencies to remove Dapper and `Microsoft.Data.SqlClient`, and add EF Core packages. Added `ApplicationDbContext` for EF Core context and new `Document` and `DocumentChunk` classes for entity models.
Updated Microsoft.SemanticKernel from 1.22.0 to 1.24.1.
Updated MinimalHelpers.OpenApi from 2.0.16 to 2.0.17.
These updates may include bug fixes, performance improvements,
or new features.
Refactored code to utilize Dapper's `ExecuteAsync` and `ExecuteScalarAsync` methods, reducing boilerplate and simplifying SQL command execution. Updated `DeleteDocumentAsync` to a concise expression-bodied member. Replaced manual SQL parameter addition with anonymous objects for better readability and maintainability. Transaction handling remains unchanged to ensure consistent database operations.
Updated the following package dependencies:
- MinimalHelpers.OpenApi from 2.0.15 to 2.0.16
- Swashbuckle.AspNetCore from 6.8.1 to 6.9.0
- TinyHelpers.AspNetCore from 3.1.17 to 3.1.19
Updated the `AskQuestionAsync` method in `ChatService.cs` to include an additional example in the `ChatHistory` object. This example clarifies how the service should respond when the requested information is not available in the given context, such as responding with "This information isn't available in the given context" if a user asks "What is the capital of France?" and there is no information about France in the chat.
Removed the Microsoft.EntityFrameworkCore namespace from Program.cs.
In SqlDatabaseVectorSearch.csproj:
- Removed EntityFrameworkCore.Exceptions.SqlServer package.
- Updated Microsoft.AspNetCore.OpenApi from 8.0.8 to 8.0.10.
- Updated Microsoft.SemanticKernel from 1.21.1 to 1.22.0.
- Updated TinyHelpers from 3.1.16 to 3.1.18.
Updated README.md for clarity and additional setup instructions:
- Refined repository description to highlight native Vector type.
- Rephrased note on Vector Support feature for readability.
- Removed mention of EFCore.SqlServer.VectorSearch library.
- Added instructions for updating VECTOR column size and setting Dimension property.
Added comment in Scripts.sql to guide vector size setting in Embedding column.
Cleaned up VectorSearchService.cs by removing unused and commented-out SQL command execution code.
Updated primary key constraint names in Scripts.sql for Documents
and DocumentChunks tables. Enhanced document upload API endpoint
description in Program.cs to reflect the use of the new native
Vector type for embeddings. Added a comment in appsettings.json
to clarify the Dimensions setting in the Embedding section.
Refactored `VectorSearchService.cs` to use Dapper for DB operations, replacing raw ADO.NET commands. Updated methods for inserting, retrieving, and deleting documents and chunks. Modified vector search query to use Dapper's `QueryAsync`.
Updated `SqlDatabaseVectorSearch.csproj` to include Dapper package reference, version `2.1.35`.
Updated Program.cs to include dimensions parameter for AddAzureOpenAITextEmbeddingGeneration sourced from aiSettings.Embedding.Dimensions. Reformatted SQL command texts in VectorSearchService.cs for better readability. Introduced EmbeddingServiceSettings class in AzureOpenAISettings.cs to allow optional dimensions configuration. Updated appsettings.json to include new Dimensions property under Embedding section.
The custom stylesheet injection for Swagger UI has been removed.
Specific width constraints for checkbox inputs in the Swagger UI
have been removed from the CSS.
Updated the following package dependencies:
- Microsoft.SemanticKernel from 1.17.2 to 1.18.2
- MinimalHelpers.OpenApi from 2.0.11 to 2.0.12
- TinyHelpers from 3.1.11 to 3.1.16
- TinyHelpers.AspNetCore from 3.1.8 to 3.1.9
- Removed AddFormFile method from Swagger config in Program.cs
- Updated package versions in SqlDatabaseVectorSearch.csproj:
- Microsoft.SemanticKernel to 1.16.2
- MinimalHelpers.OpenApi to 2.0.11
- Swashbuckle.AspNetCore to 6.7.0
- TinyHelpers to 3.1.11
- TinyHelpers.AspNetCore to 3.1.8
- Removed old package references from SqlDatabaseVectorSearch.csproj
- Updated `Scripts.sql` to add a new `[Index]` column to `[dbo].[DocumentChunks]` for order tracking.
- Modified `DocumentChunk.cs` to include a new `Index` property, and introduced a new immutable record class for document chunks.
- Introduced new API endpoints in `Program.cs` for document and chunk retrieval, including embedding details, with OpenAPI documentation enhancements.
- Updated an API endpoint description in `Program.cs` to clarify document embedding handling.
- Updated `VectorSearchService.cs` to reflect schema changes in service logic, adding methods for fetching document chunks and specific embeddings.
- Updated `Microsoft.AspNetCore.OpenApi` from `8.0.6` to `8.0.7`
- Updated `Microsoft.EntityFrameworkCore.SqlServer` from `8.0.6` to `8.0.7`
- Updated `Microsoft.SemanticKernel` from `1.15.0` to `1.15.1`
- Updated `TinyHelpers` from `3.1.7` to `3.1.10`
Updated Swagger documentation title and endpoint description in Program.cs to reflect a shift towards general SQL Database usage instead of specific SQL Server implementation. Simplified the document upload endpoint by removing the `LinkGenerator` parameter and enhancing clarity in operation descriptions, emphasizing support for uploading documents to SQL databases for indexing and question answering purposes.
Refactored code to centralize configuration access through a single `AppSettings` instance in `ChatService` and `VectorSearchService`, improving maintainability and reducing verbosity. Introduced new configuration settings (`MaxTokensPerLine`, `MaxTokensPerParagraph`, `OverlapTokens`, `MaxChunksCount`) in `AppSettings.cs` and `appsettings.json` for enhanced flexibility in content processing. Adjusted existing settings usage (`MessageLimit`, `MessageExpiration`) to align with the new access method, and removed obsolete settings (`StoragePath`, `VectorDbPath`, `QueuePath`). These changes simplify the codebase, make the application more configurable and adaptable to different content characteristics, and allow for more controlled vector search operations.
Marco Minerva