Origam unstable, many System.IndexOutOfRangeException at Origam.ResourceMonitor in the log

After deploying Origam version 2024.9.2.3552

I was expecting Origam to run well

Instead Origam started to be unstable from end user’s point of view (mainly interruptions of bigger workflows).

In the server log, there are new error messages having one in common - problem coming from Resource monitor

The problem didn’t start right after deployment, but after approx 16hrs. From that time on, it’s quite frequent.

2024-10-03 09:45:59,466 [73] FATAL IdentityServer4.Hosting.IdentityServerMiddleware - Unhandled exception: Index was outside the bounds of the array.
System.IndexOutOfRangeException: Index was outside the bounds of the array.
   at System.Collections.Generic.Dictionary`2.TryInsert(TKey key, TValue value, InsertionBehavior behavior)
   at Origam.ResourceMonitor.Transactions(String transactionId) in D:\a\1\s\backend\Origam.Common\ResourceMonitor.cs:line 216
   at Origam.DA.Service.AbstractSqlDataService.GetTransaction(String transactionId, IsolationLevel isolationLevel) in D:\a\1\s\backend\Origam.DA.Service\AbstractSqlDataService.cs:line 238
   at Origam.DA.Service.AbstractSqlDataService.UpdateData(DataStructureQuery query, IPrincipal userProfile, DataSet dataset, String transactionId, Boolean forceBulkInsert)
   at Origam.Workflow.DataServiceAgent.SaveData(DataStructureQuery query, DataSet data) in D:\a\1\s\backend\Origam.Workflow\Service Agents\DataServiceAgent.cs:line 241
   at Origam.Workflow.DataServiceAgent.Run() in D:\a\1\s\backend\Origam.Workflow\Service Agents\DataServiceAgent.cs:line 421
   at Origam.Workbench.Services.CoreServices.DataService.StoreData(DataStructureQuery dataStructureQuery, DataSet data, String transactionId) in D:\a\1\s\backend\Origam.Workbench.Services\CoreServices\DataService.cs:line 141
   at Origam.Server.Authorization.PersistedGrantStore.StoreAsync(PersistedGrant grant) in D:\a\1\s\backend\Origam.Server\Authorization\PersistedGrantStore.cs:line 81
   at IdentityServer4.Stores.DefaultGrantStore`1.StoreItemAsync(String key, T item, String clientId, String subjectId, String sessionId, String description, DateTime created, Nullable`1 expiration, Nullable`1 consumedTime)
   at IdentityServer4.Stores.DefaultGrantStore`1.CreateItemAsync(T item, String clientId, String subjectId, String sessionId, String description, DateTime created, Int32 lifetime)
   at IdentityServer4.ResponseHandling.AuthorizeResponseGenerator.CreateCodeFlowResponseAsync(ValidatedAuthorizeRequest request)
   at IdentityServer4.ResponseHandling.AuthorizeResponseGenerator.CreateResponseAsync(ValidatedAuthorizeRequest request)
   at IdentityServer4.Endpoints.AuthorizeEndpointBase.ProcessAuthorizeRequestAsync(NameValueCollection parameters, ClaimsPrincipal user, ConsentResponse consent)
   at IdentityServer4.Endpoints.AuthorizeCallbackEndpoint.ProcessAsync(HttpContext context)
   at IdentityServer4.Endpoints.AuthorizeCallbackEndpoint.ProcessAsync(HttpContext context)
   at IdentityServer4.Hosting.IdentityServerMiddleware.Invoke(HttpContext context, IEndpointRouter router, IUserSession session, IEventService events, IBackChannelLogoutService backChannelLogoutService)
2024-10-03 09:45:59,467 [73] ERROR Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware - An unhandled exception has occurred while executing the request.
System.IndexOutOfRangeException: Index was outside the bounds of the array.
   at System.Collections.Generic.Dictionary`2.TryInsert(TKey key, TValue value, InsertionBehavior behavior)
   at Origam.ResourceMonitor.Transactions(String transactionId) in D:\a\1\s\backend\Origam.Common\ResourceMonitor.cs:line 216
   at Origam.DA.Service.AbstractSqlDataService.GetTransaction(String transactionId, IsolationLevel isolationLevel) in D:\a\1\s\backend\Origam.DA.Service\AbstractSqlDataService.cs:line 238
   at Origam.DA.Service.AbstractSqlDataService.UpdateData(DataStructureQuery query, IPrincipal userProfile, DataSet dataset, String transactionId, Boolean forceBulkInsert)
   at Origam.Workflow.DataServiceAgent.SaveData(DataStructureQuery query, DataSet data) in D:\a\1\s\backend\Origam.Workflow\Service Agents\DataServiceAgent.cs:line 241
   at Origam.Workflow.DataServiceAgent.Run() in D:\a\1\s\backend\Origam.Workflow\Service Agents\DataServiceAgent.cs:line 421
   at Origam.Workbench.Services.CoreServices.DataService.StoreData(DataStructureQuery dataStructureQuery, DataSet data, String transactionId) in D:\a\1\s\backend\Origam.Workbench.Services\CoreServices\DataService.cs:line 141
   at Origam.Server.Authorization.PersistedGrantStore.StoreAsync(PersistedGrant grant) in D:\a\1\s\backend\Origam.Server\Authorization\PersistedGrantStore.cs:line 81
   at IdentityServer4.Stores.DefaultGrantStore`1.StoreItemAsync(String key, T item, String clientId, String subjectId, String sessionId, String description, DateTime created, Nullable`1 expiration, Nullable`1 consumedTime)
   at IdentityServer4.Stores.DefaultGrantStore`1.CreateItemAsync(T item, String clientId, String subjectId, String sessionId, String description, DateTime created, Int32 lifetime)
   at IdentityServer4.ResponseHandling.AuthorizeResponseGenerator.CreateCodeFlowResponseAsync(ValidatedAuthorizeRequest request)
   at IdentityServer4.ResponseHandling.AuthorizeResponseGenerator.CreateResponseAsync(ValidatedAuthorizeRequest request)
   at IdentityServer4.Endpoints.AuthorizeEndpointBase.ProcessAuthorizeRequestAsync(NameValueCollection parameters, ClaimsPrincipal user, ConsentResponse consent)
   at IdentityServer4.Endpoints.AuthorizeCallbackEndpoint.ProcessAsync(HttpContext context)
   at IdentityServer4.Endpoints.AuthorizeCallbackEndpoint.ProcessAsync(HttpContext context)
   at IdentityServer4.Hosting.IdentityServerMiddleware.Invoke(HttpContext context, IEndpointRouter router, IUserSession session, IEventService events, IBackChannelLogoutService backChannelLogoutService)
   at IdentityServer4.Hosting.IdentityServerMiddleware.Invoke(HttpContext context, IEndpointRouter router, IUserSession session, IEventService events, IBackChannelLogoutService backChannelLogoutService)
   at IdentityServer4.Hosting.MutualTlsEndpointMiddleware.Invoke(HttpContext context, IAuthenticationSchemeProvider schemes)
   at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)
   at IdentityServer4.Hosting.BaseUrlMiddleware.Invoke(HttpContext context)
   at Microsoft.AspNetCore.Localization.RequestLocalizationMiddleware.Invoke(HttpContext context)
   at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddlewareImpl.<Invoke>g__Awaited|10_0(ExceptionHandlerMiddlewareImpl middleware, HttpContext context, Task task)