Updated several package references in the `SqlDatabaseVectorSearch.csproj` file to newer versions, including:
- `Microsoft.AspNetCore.OpenApi` to `9.0.7`
- `Microsoft.EntityFrameworkCore.SqlServer` to `9.0.7`
- `Microsoft.EntityFrameworkCore.Tools` to `9.0.7`
- `Microsoft.Extensions.Caching.Hybrid` to `9.7.0`
- `Microsoft.Extensions.Http.Resilience` to `9.7.0`
- `Microsoft.SemanticKernel` to `1.60.0`
- `MinimalHelpers.FluentValidation` to `1.1.4`
- `Swashbuckle.AspNetCore.SwaggerUI` to `9.0.3`
Updated button styles in Documents.razor for better appearance and usability. Improved the document table layout for responsiveness, integrating checkboxes into rows. Corrected comments in VectorSearchService.cs for clarity. Added new styles for content-type badges in app.css.
Updated the return statement in the `VectorSearchService` class to return citations sorted by `FileName` and `PageNumber`, improving the organization of the output.
Updated `copilot-instructions.md` to include new guidelines for Markdown documentation and testing practices. Modified the `DocumentChunk` record in `DocumentChunk.cs` to add `PageNumber` and `IndexOnPage` properties. Adjusted `GetChunksAsync` and `GetChunkEmbeddingAsync` methods in `DocumentService.cs` to accommodate the new properties.
Enhance `copilot-instructions.md` to include updated best practices for code suggestions, style, architecture, documentation, and testing. Key changes emphasize high confidence suggestions, the latest C# features, specific formatting preferences, and comprehensive documentation requirements. Additionally, new testing practices are introduced to streamline unit test additions and ensure code quality.
Updated response guidelines in ChatService.cs to clarify
requirements for answer formatting, including citation rules
and examples of correct and incorrect responses.
Binary differences noted in SqlDatabaseVectorSearch_WebApp.png,
indicating modifications to the image file.
Updated the ChatService class to require that all responses end with a period and a space. Added a condition to include citations in a specified XML format when the answer is known, and to omit citations when the answer is unknown.
Updated `Home.razor` to provide a more detailed overview of the application's capabilities, including document loading, embedding generation, and semantic search. Improved clarity in the supported features section and added a new feature for citations. Included new paragraphs to encourage user interaction and referenced the README for API usage details.
Added a badge for "Minimal API" availability to the README.md file, enhancing the visibility of project features. Existing badges for .NET 9 and Blazor remain unchanged.
Enhanced the README.md file with the following changes:
- Added badges for .NET 9 and Blazor.
- Rephrased the application overview.
- Structured the Table of Contents for easier navigation.
- Reformatted setup instructions for better readability.
- Provided additional details for configuring the database and OpenAI settings.
- Expanded the supported features section.
- Included examples of API requests and responses.
- Clarified limitations and FAQ sections.
- Added a note about using straight SQL.
Updated the README.md file to enhance readability and grammatical structure. Key changes include improved phrasing, added missing commas, consistent formatting of section headers and bullet points, and correction of typographical errors. The description of the response streaming feature was also clarified to provide a better understanding of the application's functionality.
- Updated README.md to include a new **Citations** feature, detailing how users can access source information.
- Modified JSON response examples to include a `citations` field and updated token usage details.
- Enhanced streaming response section to clarify the end of the stream includes citations.
- Adjusted `VectorSearchService.cs` to return `StreamState.End` and improved citation handling in streaming.
- Updated `appsettings.json` with new model IDs for Azure OpenAI configuration.
- Updated `Microsoft.SemanticKernel` to version `1.57.0`.
- Updated `Swashbuckle.AspNetCore.SwaggerUI` to version `9.0.1`.
- Updated `TinyHelpers.AspNetCore` to version `4.0.29`.
Updated comments for clarity and streamlined logic for managing tokenUsageResponse. Removed explicit null checks in favor of a null-coalescing assignment. Ensured citations are always extracted and returned at the end of the streaming process.
- Updated `DocxContentDecoder` to process Word documents as chunks of text, removing page tracking and enhancing content handling.
- Modified `VectorSearchService.ImportAsync` to work with chunks, implementing batching for embedding generation.
- Added `EmbeddingBatchSize` property to `AppSettings` for configurable batch processing.
- Updated `appsettings.json` to include the new `EmbeddingBatchSize` setting for improved control over embedding processes.
- Wrapped checkbox input in a div for better alignment.
- Changed documents initialization from an empty array to a list.
- Updated document addition code for improved readability.
- Modified ConfirmDialogOptions and ToastMessage initializations to use object initializer syntax.
- Translated comment in DocxContentDecoder.cs from Italian to English.
- Modified `DocxContentDecoder` to use `IServiceProvider` for text chunking and improved paragraph processing with page break handling.
- Updated `PdfContentDecoder` and `TextContentDecoder` to trim whitespace from text before splitting into paragraphs.
- Reordered service registrations in `Program.cs` while retaining existing functionality.
- Updated `SqlDatabaseVectorSearch.csproj` with new package versions for several dependencies, including `Microsoft.AspNetCore.OpenApi` and `Microsoft.EntityFrameworkCore`.
Modified response examples and citation rules in ChatService.cs.
Changed context from France to Italy, added conditions for including citations, and emphasized formatting requirements. Updated examples to ensure compliance with new guidelines.
Updated `ChatService` to ensure citations are enclosed in XML tags with a consistent format. Modified `VectorSearchService` to implement a new regex pattern for citation matching and improved text cleaning by removing citation content more efficiently.
Updated citation management in the application by removing the `RemoveCitations` and `ExtractCitations` methods in `Ask.razor`, and directly processing citations from the `delta` object. The `Response` class now includes a `Citations` property for better data handling.
Modified `VectorSearchService.cs` to extract citations from the full answer in `AskQuestionAsync` and return them at the end of the streaming process in `AskStreamingAsync`.
Introduced a new `Citation` class in `Citation.cs` to encapsulate citation properties, ensuring structured management of citation data.
Updated citation formatting rules to enforce a specific XML format, ensuring citations are presented at the end of responses rather than within the answer text.
- Updated `Ask.razor` to change `PageNumber` to a nullable integer and added `IndexOnPage` to the `Citation` class. Adjusted regex for citation parsing.
- Introduced `PageNumber` and `IndexOnPage` properties in `DocumentChunk.cs`, marking `Content` as required.
- Modified migration files to reflect changes in `DocumentChunk` and `Document` entities.
- Updated citation format in `ChatService.cs` to include `index-on-page` and adjusted document chunk text formatting.
- Changed embedding generation method in `VectorSearchService.cs` and updated document chunk creation to include new properties.
- Changed `PageNumber` in `Chunk` to nullable `int?` in `IContentDecoder` and updated related logic in `TextContentDecoder`.
- Revised citation rules in `ChatService` for stricter placement and formatting.
- Introduced `QuestionValidator` class with validation rules for `Question` model's `Text` property.
Refactor regex pattern in `Ask.razor` to capture
`document-id` and `chunk-id`. Update `Citation` class
to include new properties and make `PageNumber` nullable.
Adjust citation addition logic and citation format rules.
Modify chunk text formatting in `ChatService.cs` to
include page number.
Updated the assistant message construction in `Ask.razor` to manage citations more effectively by introducing a `RawText` property and a new `RemoveCitations` method. The `ExtractCitations` method now processes raw input for citation extraction.
Removed outdated comments in `ChatService.cs` regarding citation formatting rules, indicating a potential shift in how citation handling is enforced.
Updated Ask.razor to include regex for citation extraction and display.
Introduced a new method to extract citations and updated the Message
class to store them. Added a Citation class for individual citation
representation. Enhanced app.css with styles for citation display.
Updated parameter types in ChatService and VectorSearchService from IEnumerable<string> to IEnumerable<Entities.DocumentChunk> for better structure. Enhanced citation formatting rules in ChatService. Increased MaxRelevantChunks and MaxInputTokens in appsettings.json to improve processing capabilities.
- Updated `TextContentDecoder` to use `ITextChunker` for paragraph splitting and return a list of `Chunk` objects.
- Changed return type of `Stream` method in `AskEndpoints.cs` from `IAsyncEnumerable<QuestionResponse>` to `IAsyncEnumerable<Response>`.
- Removed `QuestionResponse` class and introduced `Response` class to better handle streaming responses.
- Modified `AskQuestionAsync` and `AskStreamingAsync` methods in `VectorSearchService` to return `Response` instead of `QuestionResponse`, and adjusted token count calculation.
- Added namespace declaration in `Response.cs` and defined properties to align with new response structure.
Updated `DocxContentDecoder`, `PdfContentDecoder`, and `TextContentDecoder` to return `Task<IEnumerable<Chunk>>` instead of `Task<string>`, introducing a new `Chunk` record for structured output.
Restructured the `ApplicationDbContext`, `Document`, and `DocumentChunk` classes by moving them to the `SqlDatabaseVectorSearch.Data` namespace for better organization.
Updated database migration files to align with the new entity structure and modified references in `Program.cs`, `DocumentService.cs`, and `VectorSearchService.cs` to use the new namespace.
Updated the copy confirmation feature in `Ask.razor` by replacing the `isCopied` variable with `showCopyConfirmation`, improving user feedback with a longer display duration for the checkmark icon.
Also, updated the `Microsoft.SemanticKernel` package version from `1.52.1` to `1.53.1` for potential bug fixes and improvements.
Introduce a tooltip feature in `Ask.razor` that provides user feedback when copying text to the clipboard. Added `isCopied` to track the copy state and `toolTipText` to manage the tooltip message. The tooltip title updates from "Copy to Clipboard" to "Copied!" after the action, with a delay to reset the message for subsequent copies.
This update introduces a new feature that allows users to copy messages to the clipboard. A boolean variable `isCopied` tracks the copy action status, and the button now shows a check icon upon successful copy, reverting after 1.5 seconds. A null check is also added in the `CopyToClipboardAsync` method to enhance error handling.
- Updated `VectorSearchService.cs` to replace the text embedding generation service with an embedding generator and adjusted embedding generation and vector search logic.
- Changed `Microsoft.SemanticKernel` package version from 1.50.0 to 1.51.0 in `SqlDatabaseVectorSearch.csproj` and modified `NoWarn` property settings.
Updated several NuGet package versions in `SqlDatabaseVectorSearch.csproj` to improve functionality and compatibility. Key upgrades include:
- `Microsoft.AspNetCore.OpenApi` to version `9.0.5`
- `Microsoft.EntityFrameworkCore.SqlServer` to version `9.0.5`
- `Microsoft.EntityFrameworkCore.Tools` to version `9.0.5`
- `Microsoft.Extensions.Caching.Hybrid` to version `9.5.0`
- `Microsoft.Extensions.Http.Resilience` to version `9.5.0`
- `TinyHelpers.AspNetCore` to version `4.0.26`
Enhanced descriptions in `README.md` for features:
**Conversation History with Question Reformulation**,
**Information about Token Usage**, and
**Response Streaming**.
Reformatted feature list in `Home.razor` to include bold headings
and detailed explanations for improved clarity.
Marco Minerva