- Updated `DocxContentDecoder` to process Word documents as chunks of text, removing page tracking and enhancing content handling.
- Modified `VectorSearchService.ImportAsync` to work with chunks, implementing batching for embedding generation.
- Added `EmbeddingBatchSize` property to `AppSettings` for configurable batch processing.
- Updated `appsettings.json` to include the new `EmbeddingBatchSize` setting for improved control over embedding processes.
- Wrapped checkbox input in a div for better alignment.
- Changed documents initialization from an empty array to a list.
- Updated document addition code for improved readability.
- Modified ConfirmDialogOptions and ToastMessage initializations to use object initializer syntax.
- Translated comment in DocxContentDecoder.cs from Italian to English.
- Modified `DocxContentDecoder` to use `IServiceProvider` for text chunking and improved paragraph processing with page break handling.
- Updated `PdfContentDecoder` and `TextContentDecoder` to trim whitespace from text before splitting into paragraphs.
- Reordered service registrations in `Program.cs` while retaining existing functionality.
- Updated `SqlDatabaseVectorSearch.csproj` with new package versions for several dependencies, including `Microsoft.AspNetCore.OpenApi` and `Microsoft.EntityFrameworkCore`.
- Changed `PageNumber` in `Chunk` to nullable `int?` in `IContentDecoder` and updated related logic in `TextContentDecoder`.
- Revised citation rules in `ChatService` for stricter placement and formatting.
- Introduced `QuestionValidator` class with validation rules for `Question` model's `Text` property.
- Updated `TextContentDecoder` to use `ITextChunker` for paragraph splitting and return a list of `Chunk` objects.
- Changed return type of `Stream` method in `AskEndpoints.cs` from `IAsyncEnumerable<QuestionResponse>` to `IAsyncEnumerable<Response>`.
- Removed `QuestionResponse` class and introduced `Response` class to better handle streaming responses.
- Modified `AskQuestionAsync` and `AskStreamingAsync` methods in `VectorSearchService` to return `Response` instead of `QuestionResponse`, and adjusted token count calculation.
- Added namespace declaration in `Response.cs` and defined properties to align with new response structure.
Updated `DocxContentDecoder`, `PdfContentDecoder`, and `TextContentDecoder` to return `Task<IEnumerable<Chunk>>` instead of `Task<string>`, introducing a new `Chunk` record for structured output.
Restructured the `ApplicationDbContext`, `Document`, and `DocumentChunk` classes by moving them to the `SqlDatabaseVectorSearch.Data` namespace for better organization.
Updated database migration files to align with the new entity structure and modified references in `Program.cs`, `DocumentService.cs`, and `VectorSearchService.cs` to use the new namespace.
Introduce support for `CancellationToken` across various methods to allow for task cancellation and improve responsiveness.
- Update `DecodeAsync` method in `DocxContentDecoder.cs`, `PdfContentDecoder.cs`, `TextContentDecoder.cs`, and `IContentDecoder.cs` to include an optional `CancellationToken` parameter.
- Modify endpoint handlers in `Program.cs` to accept and pass `CancellationToken` parameters.
- Update methods in `ChatService.cs` to include `CancellationToken` parameters.
- Update methods in `DocumentService.cs` to include `CancellationToken` parameters.
- Update methods in `VectorSearchService.cs` to include `CancellationToken` parameters.
These changes ensure that long-running operations can be canceled if needed, improving the application's ability to handle cancellation requests gracefully.
Updated README.md to reflect support for PDF, DOCX, and TXT files.
Removed commented-out code in DocxContentDecoder.cs.
Added TextContentDecoder service in Program.cs and updated exception handling middleware.
Updated document upload endpoint description in Program.cs.
Modified VectorSearchService to throw NotSupportedException for unsupported content types.
Added TextContentDecoder class in TextContentDecoder.cs.
- Added `using` statements in `Program.cs` for new content decoding.
- Registered new content decoder services in `builder.Services`.
- Modified `documentsApiGroup.MapPost` to pass `file.ContentType`.
- Refactored `VectorSearchService` to use `IServiceProvider` and handle content types.
- Added `DocumentFormat.OpenXml` package reference.
- Created `DocxContentDecoder` and `PdfContentDecoder` classes.
- Created `IContentDecoder` interface.
Marco Minerva