Laura Hausmann
2506e42733
[backend/core] Remove no-duplicate request/job id from log messages
2024-08-13 20:57:06 +02:00
Laura Hausmann
dba3fc1a6a
[backend/core] Add request/job id to logs, improve log formatting
2024-08-13 20:47:06 +02:00
Laura Hausmann
e17cd0ac33
[backend/queue] Better explain the race condition that was recently resolved
2024-07-27 02:12:05 +02:00
Laura Hausmann
6d111e3e45
[backend] Fix typo
2024-07-27 02:06:39 +02:00
Laura Hausmann
7b415d4063
[backend/queue] Add a detailed explanation of the queue system algorithm
2024-07-27 01:54:07 +02:00
Laura Hausmann
e885f753a1
[backend/queue] Code cleanup
2024-07-25 01:35:17 +02:00
Laura Hausmann
6c3f78a18b
[backend/queue] Improve shutdown process
2024-07-25 01:29:50 +02:00
Laura Hausmann
a845405c45
[backend/queue] Fix QueueService race condition (properly this time)
...
While the previous fix was likely enough, there was still a razor-thin theoretical race condition remaining. This commit fixes said race condition, and simplifies some if statements across the file.
2024-07-25 01:29:32 +02:00
Laura Hausmann
09777fe6c4
[backend/queue] Code cleanup
2024-07-25 00:09:13 +02:00
Laura Hausmann
0f400e8023
[backend/queue] Remove cluster mode (ISH-141)
...
This is wontfix for now, though we might reevaluate this later.
2024-07-25 00:00:43 +02:00
Laura Hausmann
2d178e5a9a
[backend/queue] Allow configuring worker types in cluster mode (ISH-141)
2024-07-24 20:39:45 +02:00
Laura Hausmann
d7cfc24320
[backend/queue] Refactor cluster mode event methods to have less confusing method signatures
2024-07-24 18:01:26 +02:00
Laura Hausmann
7ea2f71abe
[backend/queue] Fix QueueService race condition causing transient queue stalls
2024-07-22 05:59:20 +02:00
Laura Hausmann
df3a7bdfe5
[backend] Apply formatting rules
2024-06-29 01:06:19 +02:00
Laura Hausmann
6743442ce8
[backend/queue] Code cleanup
2024-06-28 18:28:23 +02:00
Laura Hausmann
ca8ce9137d
[backend/queue] Refactor queue job count functions into DatabaseContext
2024-06-28 18:20:59 +02:00
Laura Hausmann
6ba374a681
[backend/queue] Only raise a job queued event for the first matching queue when retrying a job
2024-06-28 18:01:26 +02:00
Laura Hausmann
e971a781e6
[backend/queue] Don't reset RetryCount for manually retried jobs
2024-06-23 13:30:31 +02:00
Laura Hausmann
74b2f69b0c
[backend/razor] Add retry job functionality to queue dashboard
2024-06-22 02:17:35 +02:00
Laura Hausmann
3599150603
[backend/database] Add Exception column to Job table
2024-06-21 19:18:31 +02:00
Laura Hausmann
19c9b7bf90
[backend/database] Add StackTrace column to Job table
2024-06-21 14:05:34 +02:00
Laura Hausmann
47ab95cc1b
[backend/queue] Remove trace logging
2024-06-16 21:56:56 +02:00
Laura Hausmann
8ebef17938
[backend/queue] Fix race condition that could cause temporary queue stalls (until a new job gets queued)
2024-06-16 19:06:53 +02:00
Laura Hausmann
571ef8f027
[backend/queue] Improve logging
2024-06-16 18:55:54 +02:00
Laura Hausmann
07fd0260a8
[backend/queue] Add healthchecks that regularly check for stalled jobs (ISH-361)
2024-06-16 18:07:25 +02:00
Laura Hausmann
7ed39d0c47
[backend/queue] Fix AsyncAutoResetEvent handling, fix stalled queue workers causing no new workers to be spawned
2024-06-16 17:23:28 +02:00
Laura Hausmann
35148d7796
[backend/queue] Add queue processor timeout
...
For now this is hardcoded per queue, but this will be configurable down the line.
2024-06-13 22:41:41 +02:00
Laura Hausmann
aa21e4b05a
[backend/queue] Make queue concurrency configurable
2024-06-10 18:18:39 +02:00
Laura Hausmann
9949e24c8d
[backend] Miscellaneous refactors
2024-05-28 20:50:04 +02:00
Laura Hausmann
e77c768882
[backend/database] Switch job identifier generation scheme to from UUIDv4 to ULID
2024-05-27 22:11:18 +02:00
Laura Hausmann
35153a9080
[backend/queue] Improved AsyncAutoResetEvent implementation
...
This implementation improves the memory behavior of AsyncAutoResetEvents that fire in rapid succession.
2024-05-25 13:34:06 +02:00
Laura Hausmann
ba21cb2b0e
[backend/queue] Fix race condition that could cause PostgresJobQueue.ExecuteAsync to fail when actualParallelism is negative
2024-05-22 23:47:09 +02:00
Laura Hausmann
97532c2b22
[backend/queue] Clean completed jobs in a cron task instead of doing it at job completion
...
This prevents database deadlocks that can occur when many jobs are being processed simultaneously.
2024-05-14 15:05:01 +02:00
Laura Hausmann
363b0c930c
[backend/core] Add Guid.ToStringLower() extension method
2024-04-25 18:26:42 +02:00
Laura Hausmann
c25b613376
[backend/queue] Improve logging
2024-04-15 18:00:11 +02:00
Laura Hausmann
8fb7c8a93e
[backend/queue] Clear change tracker before saving the job state
...
This prevents a queue worker stall when a job fails to execute due to a database exception (leaving unsaved changes in the DbContext change tracker, preventing the job status from being set as failed)
2024-04-15 17:48:36 +02:00
Laura Hausmann
c5bd380344
[backend] Improved debug logging (ISH-238)
2024-04-10 00:15:46 +02:00
Laura Hausmann
60545e25ae
[backend/core] Fix sporadic "key not found" background task failures
2024-04-07 21:37:46 +02:00
Laura Hausmann
abf92b8ea5
[backend/queue] Increase heartbeat grace period from 15s to 60s
2024-03-28 13:21:51 +01:00
Laura Hausmann
d9ba6159f2
[backend/queue] Use C# events when not running multiple workers (ISH-234)
2024-03-26 17:54:31 +01:00
Laura Hausmann
f24c7dff08
[backend/database] Move queue job queries into DatabaseContext
2024-03-26 17:35:20 +01:00
Laura Hausmann
c235f1a586
[backend/queue] Check for correct heartbeat interval
2024-03-26 17:23:37 +01:00
Laura Hausmann
a4ae25d5f3
[backend/queue] Initial clustering support (ISH-141, ISH-220)
2024-03-26 17:08:43 +01:00
Laura Hausmann
9d5d3e935e
[backend/queue] Improved shutdown process (ISH-220)
2024-03-26 15:07:53 +01:00
Laura Hausmann
5556832de0
[backend/core] Fix queue system logging rendering UTC timestamps & reporting of incorrect duration for delayed jobs due to time zone differences
2024-03-26 12:11:29 +01:00
Laura Hausmann
7eba307d0a
[backend/core] Call RaiseJobQueuedEvent in QueueService.DelayedJobsHandler (ISH-229)
2024-03-26 12:10:16 +01:00
Laura Hausmann
a67fe3722d
[backend] Code cleanup
2024-03-24 14:17:02 +01:00
Laura Hausmann
c02b2bd34c
[backend/core] Improve handling of explicit service scopes & event logic
2024-03-24 14:13:51 +01:00
Laura Hausmann
5a53fd1698
[backend/core] Remove QueueService TODO item
2024-03-24 13:39:00 +01:00
Laura Hausmann
188a6f4251
[backend/core] Fix NullReferenceException
2024-03-24 00:25:27 +01:00
