Updated comments for clarity and streamlined logic for managing tokenUsageResponse. Removed explicit null checks in favor of a null-coalescing assignment. Ensured citations are always extracted and returned at the end of the streaming process.
- Updated `DocxContentDecoder` to process Word documents as chunks of text, removing page tracking and enhancing content handling.
- Modified `VectorSearchService.ImportAsync` to work with chunks, implementing batching for embedding generation.
- Added `EmbeddingBatchSize` property to `AppSettings` for configurable batch processing.
- Updated `appsettings.json` to include the new `EmbeddingBatchSize` setting for improved control over embedding processes.
- Wrapped checkbox input in a div for better alignment.
- Changed documents initialization from an empty array to a list.
- Updated document addition code for improved readability.
- Modified ConfirmDialogOptions and ToastMessage initializations to use object initializer syntax.
- Translated comment in DocxContentDecoder.cs from Italian to English.
- Modified `DocxContentDecoder` to use `IServiceProvider` for text chunking and improved paragraph processing with page break handling.
- Updated `PdfContentDecoder` and `TextContentDecoder` to trim whitespace from text before splitting into paragraphs.
- Reordered service registrations in `Program.cs` while retaining existing functionality.
- Updated `SqlDatabaseVectorSearch.csproj` with new package versions for several dependencies, including `Microsoft.AspNetCore.OpenApi` and `Microsoft.EntityFrameworkCore`.
Modified response examples and citation rules in ChatService.cs.
Changed context from France to Italy, added conditions for including citations, and emphasized formatting requirements. Updated examples to ensure compliance with new guidelines.
Updated `ChatService` to ensure citations are enclosed in XML tags with a consistent format. Modified `VectorSearchService` to implement a new regex pattern for citation matching and improved text cleaning by removing citation content more efficiently.
Updated citation management in the application by removing the `RemoveCitations` and `ExtractCitations` methods in `Ask.razor`, and directly processing citations from the `delta` object. The `Response` class now includes a `Citations` property for better data handling.
Modified `VectorSearchService.cs` to extract citations from the full answer in `AskQuestionAsync` and return them at the end of the streaming process in `AskStreamingAsync`.
Introduced a new `Citation` class in `Citation.cs` to encapsulate citation properties, ensuring structured management of citation data.
Updated citation formatting rules to enforce a specific XML format, ensuring citations are presented at the end of responses rather than within the answer text.
- Updated `Ask.razor` to change `PageNumber` to a nullable integer and added `IndexOnPage` to the `Citation` class. Adjusted regex for citation parsing.
- Introduced `PageNumber` and `IndexOnPage` properties in `DocumentChunk.cs`, marking `Content` as required.
- Modified migration files to reflect changes in `DocumentChunk` and `Document` entities.
- Updated citation format in `ChatService.cs` to include `index-on-page` and adjusted document chunk text formatting.
- Changed embedding generation method in `VectorSearchService.cs` and updated document chunk creation to include new properties.
- Changed `PageNumber` in `Chunk` to nullable `int?` in `IContentDecoder` and updated related logic in `TextContentDecoder`.
- Revised citation rules in `ChatService` for stricter placement and formatting.
- Introduced `QuestionValidator` class with validation rules for `Question` model's `Text` property.
Refactor regex pattern in `Ask.razor` to capture
`document-id` and `chunk-id`. Update `Citation` class
to include new properties and make `PageNumber` nullable.
Adjust citation addition logic and citation format rules.
Modify chunk text formatting in `ChatService.cs` to
include page number.
Updated the assistant message construction in `Ask.razor` to manage citations more effectively by introducing a `RawText` property and a new `RemoveCitations` method. The `ExtractCitations` method now processes raw input for citation extraction.
Removed outdated comments in `ChatService.cs` regarding citation formatting rules, indicating a potential shift in how citation handling is enforced.
Updated Ask.razor to include regex for citation extraction and display.
Introduced a new method to extract citations and updated the Message
class to store them. Added a Citation class for individual citation
representation. Enhanced app.css with styles for citation display.
Updated parameter types in ChatService and VectorSearchService from IEnumerable<string> to IEnumerable<Entities.DocumentChunk> for better structure. Enhanced citation formatting rules in ChatService. Increased MaxRelevantChunks and MaxInputTokens in appsettings.json to improve processing capabilities.
- Updated `TextContentDecoder` to use `ITextChunker` for paragraph splitting and return a list of `Chunk` objects.
- Changed return type of `Stream` method in `AskEndpoints.cs` from `IAsyncEnumerable<QuestionResponse>` to `IAsyncEnumerable<Response>`.
- Removed `QuestionResponse` class and introduced `Response` class to better handle streaming responses.
- Modified `AskQuestionAsync` and `AskStreamingAsync` methods in `VectorSearchService` to return `Response` instead of `QuestionResponse`, and adjusted token count calculation.
- Added namespace declaration in `Response.cs` and defined properties to align with new response structure.
Updated `DocxContentDecoder`, `PdfContentDecoder`, and `TextContentDecoder` to return `Task<IEnumerable<Chunk>>` instead of `Task<string>`, introducing a new `Chunk` record for structured output.
Restructured the `ApplicationDbContext`, `Document`, and `DocumentChunk` classes by moving them to the `SqlDatabaseVectorSearch.Data` namespace for better organization.
Updated database migration files to align with the new entity structure and modified references in `Program.cs`, `DocumentService.cs`, and `VectorSearchService.cs` to use the new namespace.
Updated the copy confirmation feature in `Ask.razor` by replacing the `isCopied` variable with `showCopyConfirmation`, improving user feedback with a longer display duration for the checkmark icon.
Also, updated the `Microsoft.SemanticKernel` package version from `1.52.1` to `1.53.1` for potential bug fixes and improvements.
Introduce a tooltip feature in `Ask.razor` that provides user feedback when copying text to the clipboard. Added `isCopied` to track the copy state and `toolTipText` to manage the tooltip message. The tooltip title updates from "Copy to Clipboard" to "Copied!" after the action, with a delay to reset the message for subsequent copies.
This update introduces a new feature that allows users to copy messages to the clipboard. A boolean variable `isCopied` tracks the copy action status, and the button now shows a check icon upon successful copy, reverting after 1.5 seconds. A null check is also added in the `CopyToClipboardAsync` method to enhance error handling.
- Updated `VectorSearchService.cs` to replace the text embedding generation service with an embedding generator and adjusted embedding generation and vector search logic.
- Changed `Microsoft.SemanticKernel` package version from 1.50.0 to 1.51.0 in `SqlDatabaseVectorSearch.csproj` and modified `NoWarn` property settings.
Updated several NuGet package versions in `SqlDatabaseVectorSearch.csproj` to improve functionality and compatibility. Key upgrades include:
- `Microsoft.AspNetCore.OpenApi` to version `9.0.5`
- `Microsoft.EntityFrameworkCore.SqlServer` to version `9.0.5`
- `Microsoft.EntityFrameworkCore.Tools` to version `9.0.5`
- `Microsoft.Extensions.Caching.Hybrid` to version `9.5.0`
- `Microsoft.Extensions.Http.Resilience` to version `9.5.0`
- `TinyHelpers.AspNetCore` to version `4.0.26`
Enhanced descriptions in `README.md` for features:
**Conversation History with Question Reformulation**,
**Information about Token Usage**, and
**Response Streaming**.
Reformatted feature list in `Home.razor` to include bold headings
and detailed explanations for improved clarity.
- Updated `AskEndpoints.cs` to include `MinimalHelpers.FluentValidation` and standardize endpoint descriptions.
- Integrated FluentValidation in `Program.cs` and registered validators.
- Modified `SqlDatabaseVectorSearch.csproj` to add necessary package references for FluentValidation.
- Created `QuestionValidator` class to enforce validation rules on the `Question` model.
Updated `ChatService.cs` to improve user guidance for unavailable information responses. Added multiple phrases to ensure clarity and context relevance.
Modified `SqlDatabaseVectorSearch.csproj` to upgrade `Microsoft.SemanticKernel` from `1.45.0` to `1.47.0`, and incremented versions for `Swashbuckle.AspNetCore.SwaggerUI` and `TinyHelpers.AspNetCore` to `8.1.1` and `4.0.25`, respectively, for bug fixes and improvements.
Updated versions of several package references in the
`SqlDatabaseVectorSearch.csproj` file, including:
- `Microsoft.AspNetCore.OpenApi`, `Microsoft.EntityFrameworkCore.SqlServer`,
and `Microsoft.EntityFrameworkCore.Tools` from `9.0.3` to `9.0.4`.
- `Microsoft.Extensions.Caching.Hybrid` and
`Microsoft.Extensions.Http.Resilience` from `9.3.0` to `9.4.0`.
- `TinyHelpers.AspNetCore` from `4.0.22` to `4.0.23`.
These updates aim to incorporate bug fixes, new features,
and improvements from the latest versions.
Clarified the `Dimensions` property requirements for embedding models that support shortening. Added information about the maximum supported size for the `VECTOR` type (1998) and the need to update database migrations accordingly.
Updated `Swashbuckle.AspNetCore.SwaggerUI` from `8.0.0` to `8.1.0` and `TinyHelpers.AspNetCore` from `4.0.21` to `4.0.22`. Removed previous versions in favor of the latest updates.
Updated token count keys from `inputTokenCount`, `outputTokenCount`, and `totalTokenCount` to `promptTokens`, `completionTokens`, and `totalTokens` for improved clarity. Modified `question` and `answer` fields to align with the new structure, with some values set to `null` to indicate streaming state.
- Updated `TokenUsage` class properties to `PromptTokens` and `CompletionTokens`.
- Modified `Ask.razor` to display new token counts.
- Added logger to `ChatService` constructor for improved logging.
- Implemented logging for token usage in `ChatService` methods.
- Changed logging level to `Debug` for paragraph storage in `VectorSearchService`.
- Updated logging configuration in `appsettings.Development.json` for better visibility.
- Updated `Documents.razor` to change form submission handler to `UploadFile` and added `id` to `InputFile`.
- Implemented `UploadFile` method to handle file uploads and reset input after successful upload.
- Upgraded package references in `SqlDatabaseVectorSearch.csproj` for `Microsoft.SemanticKernel` and `Swashbuckle.AspNetCore.SwaggerUI`.
- Introduced `resetFileInput` function in `functions.js` to clear file input selections.
Incremented versions for several packages in `SqlDatabaseVectorSearch.csproj`, including:
- `Microsoft.AspNetCore.OpenApi` to `9.0.3`
- `Microsoft.EntityFrameworkCore.SqlServer` to `9.0.3`
- `Microsoft.EntityFrameworkCore.Tools` to `9.0.3`
- `Microsoft.Extensions.Caching.Hybrid` to `9.3.0` (removed preview version)
- `Microsoft.Extensions.Http.Resilience` to `9.3.0`
- `Microsoft.SemanticKernel` to `1.41.0`
- `TinyHelpers.AspNetCore` to `4.0.21`
Marco Minerva