- Changed `PageNumber` in `Chunk` to nullable `int?` in `IContentDecoder` and updated related logic in `TextContentDecoder`.
- Revised citation rules in `ChatService` for stricter placement and formatting.
- Introduced `QuestionValidator` class with validation rules for `Question` model's `Text` property.
Refactor regex pattern in `Ask.razor` to capture
`document-id` and `chunk-id`. Update `Citation` class
to include new properties and make `PageNumber` nullable.
Adjust citation addition logic and citation format rules.
Modify chunk text formatting in `ChatService.cs` to
include page number.
Updated the assistant message construction in `Ask.razor` to manage citations more effectively by introducing a `RawText` property and a new `RemoveCitations` method. The `ExtractCitations` method now processes raw input for citation extraction.
Removed outdated comments in `ChatService.cs` regarding citation formatting rules, indicating a potential shift in how citation handling is enforced.
Updated Ask.razor to include regex for citation extraction and display.
Introduced a new method to extract citations and updated the Message
class to store them. Added a Citation class for individual citation
representation. Enhanced app.css with styles for citation display.
Updated parameter types in ChatService and VectorSearchService from IEnumerable<string> to IEnumerable<Entities.DocumentChunk> for better structure. Enhanced citation formatting rules in ChatService. Increased MaxRelevantChunks and MaxInputTokens in appsettings.json to improve processing capabilities.
- Updated `TextContentDecoder` to use `ITextChunker` for paragraph splitting and return a list of `Chunk` objects.
- Changed return type of `Stream` method in `AskEndpoints.cs` from `IAsyncEnumerable<QuestionResponse>` to `IAsyncEnumerable<Response>`.
- Removed `QuestionResponse` class and introduced `Response` class to better handle streaming responses.
- Modified `AskQuestionAsync` and `AskStreamingAsync` methods in `VectorSearchService` to return `Response` instead of `QuestionResponse`, and adjusted token count calculation.
- Added namespace declaration in `Response.cs` and defined properties to align with new response structure.
Updated `DocxContentDecoder`, `PdfContentDecoder`, and `TextContentDecoder` to return `Task<IEnumerable<Chunk>>` instead of `Task<string>`, introducing a new `Chunk` record for structured output.
Restructured the `ApplicationDbContext`, `Document`, and `DocumentChunk` classes by moving them to the `SqlDatabaseVectorSearch.Data` namespace for better organization.
Updated database migration files to align with the new entity structure and modified references in `Program.cs`, `DocumentService.cs`, and `VectorSearchService.cs` to use the new namespace.
Updated the copy confirmation feature in `Ask.razor` by replacing the `isCopied` variable with `showCopyConfirmation`, improving user feedback with a longer display duration for the checkmark icon.
Also, updated the `Microsoft.SemanticKernel` package version from `1.52.1` to `1.53.1` for potential bug fixes and improvements.
Introduce a tooltip feature in `Ask.razor` that provides user feedback when copying text to the clipboard. Added `isCopied` to track the copy state and `toolTipText` to manage the tooltip message. The tooltip title updates from "Copy to Clipboard" to "Copied!" after the action, with a delay to reset the message for subsequent copies.
This update introduces a new feature that allows users to copy messages to the clipboard. A boolean variable `isCopied` tracks the copy action status, and the button now shows a check icon upon successful copy, reverting after 1.5 seconds. A null check is also added in the `CopyToClipboardAsync` method to enhance error handling.
- Updated `VectorSearchService.cs` to replace the text embedding generation service with an embedding generator and adjusted embedding generation and vector search logic.
- Changed `Microsoft.SemanticKernel` package version from 1.50.0 to 1.51.0 in `SqlDatabaseVectorSearch.csproj` and modified `NoWarn` property settings.
Updated several NuGet package versions in `SqlDatabaseVectorSearch.csproj` to improve functionality and compatibility. Key upgrades include:
- `Microsoft.AspNetCore.OpenApi` to version `9.0.5`
- `Microsoft.EntityFrameworkCore.SqlServer` to version `9.0.5`
- `Microsoft.EntityFrameworkCore.Tools` to version `9.0.5`
- `Microsoft.Extensions.Caching.Hybrid` to version `9.5.0`
- `Microsoft.Extensions.Http.Resilience` to version `9.5.0`
- `TinyHelpers.AspNetCore` to version `4.0.26`
Enhanced descriptions in `README.md` for features:
**Conversation History with Question Reformulation**,
**Information about Token Usage**, and
**Response Streaming**.
Reformatted feature list in `Home.razor` to include bold headings
and detailed explanations for improved clarity.
- Updated `AskEndpoints.cs` to include `MinimalHelpers.FluentValidation` and standardize endpoint descriptions.
- Integrated FluentValidation in `Program.cs` and registered validators.
- Modified `SqlDatabaseVectorSearch.csproj` to add necessary package references for FluentValidation.
- Created `QuestionValidator` class to enforce validation rules on the `Question` model.
Updated `ChatService.cs` to improve user guidance for unavailable information responses. Added multiple phrases to ensure clarity and context relevance.
Modified `SqlDatabaseVectorSearch.csproj` to upgrade `Microsoft.SemanticKernel` from `1.45.0` to `1.47.0`, and incremented versions for `Swashbuckle.AspNetCore.SwaggerUI` and `TinyHelpers.AspNetCore` to `8.1.1` and `4.0.25`, respectively, for bug fixes and improvements.
Updated versions of several package references in the
`SqlDatabaseVectorSearch.csproj` file, including:
- `Microsoft.AspNetCore.OpenApi`, `Microsoft.EntityFrameworkCore.SqlServer`,
and `Microsoft.EntityFrameworkCore.Tools` from `9.0.3` to `9.0.4`.
- `Microsoft.Extensions.Caching.Hybrid` and
`Microsoft.Extensions.Http.Resilience` from `9.3.0` to `9.4.0`.
- `TinyHelpers.AspNetCore` from `4.0.22` to `4.0.23`.
These updates aim to incorporate bug fixes, new features,
and improvements from the latest versions.
Clarified the `Dimensions` property requirements for embedding models that support shortening. Added information about the maximum supported size for the `VECTOR` type (1998) and the need to update database migrations accordingly.
Updated `Swashbuckle.AspNetCore.SwaggerUI` from `8.0.0` to `8.1.0` and `TinyHelpers.AspNetCore` from `4.0.21` to `4.0.22`. Removed previous versions in favor of the latest updates.
Updated token count keys from `inputTokenCount`, `outputTokenCount`, and `totalTokenCount` to `promptTokens`, `completionTokens`, and `totalTokens` for improved clarity. Modified `question` and `answer` fields to align with the new structure, with some values set to `null` to indicate streaming state.
- Updated `TokenUsage` class properties to `PromptTokens` and `CompletionTokens`.
- Modified `Ask.razor` to display new token counts.
- Added logger to `ChatService` constructor for improved logging.
- Implemented logging for token usage in `ChatService` methods.
- Changed logging level to `Debug` for paragraph storage in `VectorSearchService`.
- Updated logging configuration in `appsettings.Development.json` for better visibility.
- Updated `Documents.razor` to change form submission handler to `UploadFile` and added `id` to `InputFile`.
- Implemented `UploadFile` method to handle file uploads and reset input after successful upload.
- Upgraded package references in `SqlDatabaseVectorSearch.csproj` for `Microsoft.SemanticKernel` and `Swashbuckle.AspNetCore.SwaggerUI`.
- Introduced `resetFileInput` function in `functions.js` to clear file input selections.
Incremented versions for several packages in `SqlDatabaseVectorSearch.csproj`, including:
- `Microsoft.AspNetCore.OpenApi` to `9.0.3`
- `Microsoft.EntityFrameworkCore.SqlServer` to `9.0.3`
- `Microsoft.EntityFrameworkCore.Tools` to `9.0.3`
- `Microsoft.Extensions.Caching.Hybrid` to `9.3.0` (removed preview version)
- `Microsoft.Extensions.Http.Resilience` to `9.3.0`
- `Microsoft.SemanticKernel` to `1.41.0`
- `TinyHelpers.AspNetCore` to `4.0.21`
Added functionality to update userMessage.Text with the
reformulated question when the streaming response starts.
This improvement provides immediate feedback to users,
enhancing the overall user experience.
Replaced `UploadDocumentRequest` with `UploadDocument` class to streamline data management. Updated `EditForm` to bind directly to the new model, removing the `EditContext` and `HandleFileSelected` method. Adjusted validation and submission logic to work with the new structure, enhancing clarity and maintainability.
- Modify `AddHybridCache` to include local cache expiration.
- Change resilience handler addition to use configure method.
- Set total request timeout to 2 minutes.
Modified the `AddStandardResilienceHandler` method in
`Program.cs` to accept an `options` parameter. This allows
for setting a `TotalRequestTimeout` of 2 minutes, enhancing
the resilience handling of HTTP requests.
- Separated endpoint definitions into `AskEndpoints` and `DocumentEndpoints` classes for better organization and maintainability.
- Removed inline endpoint definitions in `Program.cs` and replaced with `app.MapEndpoints();`.
- Updated `Microsoft.SemanticKernel` package from version `1.40.0` to `1.40.1`.
- Replaced `MinimalHelpers.OpenApi` with `MinimalHelpers.Routing.Analyzers` in project dependencies.
- Defined new POST endpoints for asking questions and document management with detailed API documentation.
- Enhanced code structure for easier management and future extensibility.
Marco Minerva