278 Commits

Author SHA1 Message Date
Marco Minerva 62d596ea98 Refactor caching and OpenAPI integration
Updated Program.cs to replace Swagger with OpenApi and MemoryCache with HybridCache. Refactored ChatService.cs to use HybridCache asynchronously. Removed MessageLimit from AppSettings.cs and appsettings.json. Updated SqlDatabaseVectorSearch.csproj to include HybridCache package and update dependencies.
2024-12-10 11:53:58 +01:00
Marco Minerva 67c600e9d4 Remove MessageLimit and MessageExpiration properties
The `AppSettings` class in `AppSettings.cs` has been modified to
remove the `MessageLimit` and `MessageExpiration` properties.
Correspondingly, the `appsettings.json` file has been updated
to remove the `MessageLimit` and `MessageExpiration` settings
under the `AppSettings` section to maintain consistency.
2024-12-10 11:44:20 +01:00
Marco Minerva 7e632892c7 Refactor caching and update dependencies
Updated Program.cs to use HybridCache and improved Swagger setup.
Refactored ChatService to simplify chat history management.
Updated SqlDatabaseVectorSearch.csproj to target net9.0 and
updated package references.
2024-12-10 11:38:59 +01:00
Marco Minerva 0575482bff Add EnableRetryOnFailure to SQL Server config
Added EnableRetryOnFailure option to the AddSqlServer method in Program.cs. This change configures the SQL Server connection to automatically retry on failure, improving the application's resilience to transient faults.
2024-12-05 18:00:01 +01:00
Marco Minerva 7989a1570f Add EnableRetryOnFailure to SQL Server configuration
Enhanced ApplicationDbContext configuration by adding EnableRetryOnFailure to improve resilience against transient SQL Server failures. Existing configurations for UseVectorSearch and NoTracking remain unchanged.
2024-12-05 17:57:41 +01:00
Marco Minerva 810b25c233 Update TinyHelpers.AspNetCore to version 4.0.5
Updated the TinyHelpers.AspNetCore package from version 4.0.4 to 4.0.5 to incorporate the latest fixes and improvements.
2024-12-04 15:35:36 +01:00
Marco Minerva 3caae928ba Update TinyHelpers.AspNetCore to version 4.0.5
Updated the TinyHelpers.AspNetCore package from version 4.0.4 to 4.0.5 to incorporate the latest fixes and improvements.
2024-12-04 15:33:51 +01:00
Marco Minerva f5d5fe151f Update TinyHelpers.AspNetCore package versions
Updated TinyHelpers.AspNetCore from 4.0.2 to 4.0.4 and
TinyHelpers.AspNetCore.Swashbuckle from 4.0.3 to 4.0.5 in
SqlDatabaseVectorSearch.csproj.
2024-12-04 10:59:56 +01:00
Marco Minerva 32d7a4da9b Update version range formatting in csproj file
Adjusted the version range specification for the `Microsoft.EntityFrameworkCore.SqlServer` package in `SqlDatabaseVectorSearch.csproj` by removing the space between the version numbers and the comma. This change improves the readability and uniformity of the code without altering the functional behavior of the version constraint.
2024-12-04 10:59:27 +01:00
Marco Minerva f4362f1e92 Update package versions and add new package reference
Updated Microsoft.SemanticKernel from 1.30.0 to 1.31.0.
Updated Swashbuckle.AspNetCore from 7.0.0 to 7.1.0.
Updated TinyHelpers.AspNetCore from 3.1.19 to 4.0.4.
Added TinyHelpers.AspNetCore.Swashbuckle version 4.0.5.
2024-12-04 10:57:56 +01:00
Marco Minerva a358567c0e Update package versions and add new package reference
Updated Microsoft.SemanticKernel from 1.30.0 to 1.31.0.
Updated Swashbuckle.AspNetCore from 7.0.0 to 7.1.0.
Updated TinyHelpers.AspNetCore from 3.1.19 to 4.0.2.
Added TinyHelpers.AspNetCore.Swashbuckle version 4.0.3.
2024-12-04 10:55:14 +01:00
Marco Minerva 9a7ea2f5b0 Update .editorconfig, VectorSearchService, and .csproj
- Updated .editorconfig with new C# style preferences.
- Modified using directives in VectorSearchService.cs.
- Changed tuple element order in VectorSearchService.cs foreach loop.
- Updated SqlDatabaseVectorSearch.csproj to target .NET 9.0.
- Updated package references and removed TinyHelpers package.

Update .editorconfig, fix VectorSearchService, upgrade packages

Updated .editorconfig with new C# style preferences. Removed `TinyHelpers.Extensions` using directive and corrected variable order in `VectorSearchService.cs`. Upgraded target framework to `net9.0` and updated several package references in `SqlDatabaseVectorSearch.csproj`.
2024-11-21 17:57:10 +01:00
Marco Minerva 232be6f083 Update code style, prompt, and dependencies
.editorconfig: Add new code style preferences.
ChatService.cs: Add formatted question to prompt string.
VectorSearchService.cs: Remove TinyHelpers.Extensions using directive.
VectorSearchService.cs: Use paragraphs.Index() in foreach loop.
SqlDatabaseVectorSearch.csproj: Update target framework to net9.0.
SqlDatabaseVectorSearch.csproj: Update package references, remove TinyHelpers.
2024-11-21 17:46:50 +01:00
Marco Minerva bcd085e49d Update README, adjust prompt formatting, add new package
Updated README.md to include a note on Vector support in Azure SQL Database/Managed Instance (EAP) and corrected "Dimension" to "Dimensions". Adjusted prompt formatting in ChatService.cs by replacing "---" with "=====" and improved text chunk appending. Added new package reference for `Microsoft.ML.Tokenizers.Data.Cl100kBase` in SqlDatabaseVectorSearch.csproj.
2024-11-07 10:03:58 +01:00
Marco Minerva b9bcb5c9fd Update README and refine prompt logic in ChatService
Updated README.md to include a note on Vector support in Azure SQL Database and corrected "Dimension" to "Dimensions". Modified ChatService.cs to append separator "---" only once at the end of the prompt.
2024-11-07 09:40:20 +01:00
Marco Minerva 084177346f Change ChatService to singleton, update package version
- Changed ChatService registration in Program.cs from scoped to singleton.
- Modified ChatHistory initialization in ChatService.cs by removing unnecessary line breaks.
- Renamed loop variable in ChatService.cs from 'result' to 'text' for better readability.
- Updated Microsoft.SemanticKernel package version from 1.26.0 to 1.27.0 in SqlDatabaseVectorSearch.csproj.
2024-11-06 17:26:09 +01:00
Marco Minerva 5b43031251 Add TokenizerService and update settings configuration
Updated Program.cs to use ConfigureAndGet method for settings,
changed ChatService to singleton, and added TokenizerService
singleton. Modified ChatService to use TokenizerService for
token counting. Updated AppSettings and AzureOpenAISettings
with new properties. Added new package references in
SqlDatabaseVectorSearch.csproj. Updated appsettings.json with
new properties. Added TokenizerService class for token counting.
2024-11-06 17:20:05 +01:00
Marco Minerva c18a6b4e03 Reorder method calls for Embedding property
Swapped the order of `IsRequired` and `HasColumnType("vector(1536)")` method calls for the `Embedding` property in the `ApplicationDbContext` class. This change improves readability and adheres to a preferred coding convention without altering functionality.
2024-11-05 17:18:48 +01:00
Marco Minerva dc0b557010 Update Microsoft.SemanticKernel to v1.26.0
Upgraded the Microsoft.SemanticKernel package from version 1.25.0 to 1.26.0 to incorporate the latest features and bug fixes.
2024-11-05 17:07:15 +01:00
Marco Minerva bb3e794a29 Update README: clarify EFCore usage and SQL note
The README.md file has been updated to specify that Vectors are saved and retrieved using the `EFCore.SqlServer.VectorSearch` library with Entity Framework Core. Additionally, the note about using straight SQL has been moved to a separate section and rephrased for better clarity.
2024-11-05 11:25:45 +01:00
Marco Minerva 7e8d1245b1 Update README.md with vector handling details
Added information on using EFCore.SqlServer.VectorSearch for vector management with Entity Framework Core. Included a link to the `sql` branch for users preferring direct SQL usage.
2024-10-31 15:22:44 +01:00
Marco Minerva 0435f042f1 Refactor to use EF Core for database operations
Refactored the codebase to replace raw SQL connections and Dapper with Entity Framework Core (EF Core). Modified `Program.cs` to configure EF Core services. Refactored `VectorSearchService` to use EF Core for all database operations. Updated project dependencies to remove Dapper and `Microsoft.Data.SqlClient`, and add EF Core packages. Added `ApplicationDbContext` for EF Core context and new `Document` and `DocumentChunk` classes for entity models.
2024-10-31 15:16:38 +01:00
Marco Minerva 1d8fa7db12 Update Microsoft.SemanticKernel to 1.25.0
Updated the Microsoft.SemanticKernel package version from 1.24.1 to 1.25.0 in the SqlDatabaseVectorSearch.csproj file.
2024-10-29 10:23:08 +01:00
Marco Minerva bc26c03629 Update package versions in SqlDatabaseVectorSearch.csproj
Updated Microsoft.SemanticKernel from 1.22.0 to 1.24.1.
Updated MinimalHelpers.OpenApi from 2.0.16 to 2.0.17.
These updates may include bug fixes, performance improvements,
or new features.
2024-10-22 10:30:52 +02:00
Marco Minerva 1a5542f1d2 Refactor to use Dapper async methods and improve readability
Refactored code to utilize Dapper's `ExecuteAsync` and `ExecuteScalarAsync` methods, reducing boilerplate and simplifying SQL command execution. Updated `DeleteDocumentAsync` to a concise expression-bodied member. Replaced manual SQL parameter addition with anonymous objects for better readability and maintainability. Transaction handling remains unchanged to ensure consistent database operations.
2024-10-16 15:06:34 +02:00
Marco Minerva 31b2a79b9f Update package dependencies in SqlDatabaseVectorSearch.csproj
Updated the following package dependencies:
- MinimalHelpers.OpenApi from 2.0.15 to 2.0.16
- Swashbuckle.AspNetCore from 6.8.1 to 6.9.0
- TinyHelpers.AspNetCore from 3.1.17 to 3.1.19
2024-10-15 14:53:41 +02:00
Marco Minerva 3445059746 Enhance ChatService response instructions with example
Updated the `AskQuestionAsync` method in `ChatService.cs` to include an additional example in the `ChatHistory` object. This example clarifies how the service should respond when the requested information is not available in the given context, such as responding with "This information isn't available in the given context" if a user asks "What is the capital of France?" and there is no information about France in the chat.
2024-10-14 12:11:25 +02:00
Marco Minerva 5c0cf1cbd0 Update dependencies and remove unused EF Core
Removed the Microsoft.EntityFrameworkCore namespace from Program.cs.
In SqlDatabaseVectorSearch.csproj:
- Removed EntityFrameworkCore.Exceptions.SqlServer package.
- Updated Microsoft.AspNetCore.OpenApi from 8.0.8 to 8.0.10.
- Updated Microsoft.SemanticKernel from 1.21.1 to 1.22.0.
- Updated TinyHelpers from 3.1.16 to 3.1.18.
2024-10-14 09:58:59 +02:00
Marco Minerva c86bea6645 Merge branch 'master' of https://github.com/marcominerva/SqlDatabaseVectorSearch 2024-10-08 09:53:58 +02:00
Marco Minerva 8c1915f964 Update PdfPig package to version 0.1.9
The version of the PdfPig package has been updated from 0.1.8 to 0.1.9.
2024-10-08 09:53:53 +02:00
Marco Minerva a0f3679262 Update README.md 2024-10-01 17:57:25 +02:00
Marco Minerva a362083227 Merge pull request #1 from marcominerva/vector-type
Add support for native VECTOR type
2024-10-01 17:39:00 +02:00
Marco Minerva 8c6cc3c969 Improve README, add comments, and clean up VectorSearchService
Updated README.md for clarity and additional setup instructions:
- Refined repository description to highlight native Vector type.
- Rephrased note on Vector Support feature for readability.
- Removed mention of EFCore.SqlServer.VectorSearch library.
- Added instructions for updating VECTOR column size and setting Dimension property.

Added comment in Scripts.sql to guide vector size setting in Embedding column.

Cleaned up VectorSearchService.cs by removing unused and commented-out SQL command execution code.
2024-10-01 17:35:59 +02:00
Marco Minerva 1b2ebbd6a3 Update PK constraints, API description, and config comment
Updated primary key constraint names in Scripts.sql for Documents
and DocumentChunks tables. Enhanced document upload API endpoint
description in Program.cs to reflect the use of the new native
Vector type for embeddings. Added a comment in appsettings.json
to clarify the Dimensions setting in the Embedding section.
2024-10-01 14:37:01 +02:00
Marco Minerva 4355f72dab Refactor DB operations, rename tables, add Dapper
Refactored `VectorSearchService.cs` to use Dapper for DB operations, replacing raw ADO.NET commands. Updated methods for inserting, retrieving, and deleting documents and chunks. Modified vector search query to use Dapper's `QueryAsync`.

Updated `SqlDatabaseVectorSearch.csproj` to include Dapper package reference, version `2.1.35`.
2024-10-01 11:39:21 +02:00
Marco Minerva 2dff0aae55 Add dimensions parameter for embeddings; reformat SQL
Updated Program.cs to include dimensions parameter for AddAzureOpenAITextEmbeddingGeneration sourced from aiSettings.Embedding.Dimensions. Reformatted SQL command texts in VectorSearchService.cs for better readability. Introduced EmbeddingServiceSettings class in AzureOpenAISettings.cs to allow optional dimensions configuration. Updated appsettings.json to include new Dimensions property under Embedding section.
2024-09-30 17:53:59 +02:00
Marco Minerva 3e95251485 Refactor to use the native VECTOR type 2024-09-30 17:08:28 +02:00
Marco Minerva 1ef2076ba6 Update package dependencies in SqlDatabaseVectorSearch.csproj
Updated the following package dependencies:
- Microsoft.SemanticKernel: 1.18.2 -> 1.21.1
- MinimalHelpers.OpenApi: 2.0.12 -> 2.0.15
- Swashbuckle.AspNetCore: 6.7.3 -> 6.8.1
- TinyHelpers.AspNetCore: 3.1.9 -> 3.1.17
2024-09-30 14:55:57 +02:00
Marco Minerva 3c7b076659 Remove custom Swagger UI stylesheet and checkbox width
The custom stylesheet injection for Swagger UI has been removed.
Specific width constraints for checkbox inputs in the Swagger UI
have been removed from the CSS.
2024-09-05 14:39:44 +02:00
Marco Minerva 6757631511 Update package dependencies in SqlDatabaseVectorSearch.csproj
Updated the following package dependencies:
- Microsoft.SemanticKernel from 1.17.2 to 1.18.2
- MinimalHelpers.OpenApi from 2.0.11 to 2.0.12
- TinyHelpers from 3.1.11 to 3.1.16
- TinyHelpers.AspNetCore from 3.1.8 to 3.1.9
2024-09-04 11:12:08 +02:00
Marco Minerva 05d18c5f97 Fix typos 2024-09-02 10:48:23 +02:00
Marco Minerva b6a09d0926 Data access optimizations 2024-09-02 10:42:30 +02:00
Marco Minerva e4570008e0 Update package dependencies in SqlDatabaseVectorSearch.csproj
Updated the following package versions:
- EntityFrameworkCore.Exceptions.SqlServer: 8.1.2 -> 8.1.3
- Microsoft.AspNetCore.OpenApi: 8.0.7 -> 8.0.8
- Microsoft.EntityFrameworkCore.SqlServer: 8.0.7 -> 8.0.8
- Microsoft.SemanticKernel: 1.16.2 -> 1.17.2
- Swashbuckle.AspNetCore: 6.7.0 -> 6.7.3
2024-09-02 10:39:21 +02:00
Marco Minerva b628c8996e Fix a typo in the documentation 2024-08-02 15:33:51 +02:00
Marco Minerva 17eee5f775 Optimize query performance and memory usage 2024-08-02 14:56:19 +02:00
Marco Minerva 1264b7f502 Correct a typo 2024-08-02 12:24:34 +02:00
Marco Minerva 3c78af133b Merge branch 'master' of https://github.com/marcominerva/SqlDatabaseVectorSearch 2024-08-01 15:41:45 +02:00
Marco Minerva e54a091106 Update Swagger config and package versions
- Removed AddFormFile method from Swagger config in Program.cs
- Updated package versions in SqlDatabaseVectorSearch.csproj:
  - Microsoft.SemanticKernel to 1.16.2
  - MinimalHelpers.OpenApi to 2.0.11
  - Swashbuckle.AspNetCore to 6.7.0
  - TinyHelpers to 3.1.11
  - TinyHelpers.AspNetCore to 3.1.8
- Removed old package references from SqlDatabaseVectorSearch.csproj
2024-08-01 15:41:42 +02:00
Marco Minerva 468b6e7c5c Update README.md 2024-07-22 12:48:51 +02:00
Marco Minerva e59941cb0f Update documentation 2024-07-22 12:40:00 +02:00