Kopper
6b1d84801e
[backend] Implement infrastructure for outbox backfill
2025-01-07 01:42:15 +03:00
Laura Hausmann
7dec2514da
[sln] Apply code style
2024-11-20 00:48:29 +01:00
Laura Hausmann
effe3a7fef
[backend/queue] Fix jobs not being upserted with their correct default values
2024-10-26 04:41:06 +02:00
kopper
27ac36d747
[backend] Add config option to sign reply backfills with requesting user
...
Signed-off-by: Laura Hausmann <laura@hausmann.dev>
2024-10-24 21:18:58 +02:00
kopper
7173373330
[backend/federation] Improved reply backfill
...
Instead of backfilling every note we come across that has a reply
collection, only schedule a backfill job if someone wants to see the
replies (on GET MastoAPI /context, or Iceshrimp API /descendants)
Reply backfilling is also done on a ThreadIdOrId basis as opposed to the
previous way of backfilling individual notes. This allows us finer
grained control over the recursion and frees up the job queue, alongside
allowing for easier implementation of context collection backfill in the
future (by mapping each context collection to a thread)
---
Currently, note threads are implicit based on the "threadId" column of a
note, which can be null (where it's the same as the note's "id")
This commit turns note threads into an actual entity, and as a part of
that, makes "threadId" non-nullable (by specifically setting it to "id"
for those cases)
This is done to attach extra metadata to the entire thread, currently
just the time of when it was last backfilled, but more may be added in
the future (the context collection associated with this thread, for example)
---
The data format for backfill jobs have backwards-incompatibly changed
since the introduction of the feature. We can drop all old jobs without
causing too much trouble as they will be re-scheduled on demand
---
Signed-off-by: Laura Hausmann <laura@hausmann.dev>
2024-10-24 21:18:57 +02:00
Laura Hausmann
60b18f434f
[backend/queue] Make PostgresJobQueue<T> abstract
2024-10-18 04:32:34 +02:00
Laura Hausmann
85d9bc597f
[backend/queue] Improve logging for jobs that were queued for more than 10s
2024-10-14 21:46:56 +02:00
Laura Hausmann
7532d5c52a
[backend/queue] Add abandon button for delayed jobs to the queue dashboard
2024-09-27 03:43:17 +02:00
Laura Hausmann
7c93dcf79d
[backend/queue] Fix duplicate exception logging
2024-09-26 02:54:33 +02:00
Laura Hausmann
efd1878fc1
[backend/queue] Fix erroneous job timeout logs, improve error messages for job timeouts
2024-09-26 02:52:38 +02:00
Kopper
83e830b5df
[backend/configuration] Add configuration for backfill of note replies
2024-09-15 02:05:18 +02:00
Kopper
024de937d0
[backend/federation] Backfill replies
2024-09-15 01:47:41 +02:00
Laura Hausmann
2506e42733
[backend/core] Remove no-duplicate request/job id from log messages
2024-08-13 20:57:06 +02:00
Laura Hausmann
dba3fc1a6a
[backend/core] Add request/job id to logs, improve log formatting
2024-08-13 20:47:06 +02:00
Laura Hausmann
e17cd0ac33
[backend/queue] Better explain the race condition that was recently resolved
2024-07-27 02:12:05 +02:00
Laura Hausmann
6d111e3e45
[backend] Fix typo
2024-07-27 02:06:39 +02:00
Laura Hausmann
7b415d4063
[backend/queue] Add a detailed explanation of the queue system algorithm
2024-07-27 01:54:07 +02:00
Laura Hausmann
e885f753a1
[backend/queue] Code cleanup
2024-07-25 01:35:17 +02:00
Laura Hausmann
6c3f78a18b
[backend/queue] Improve shutdown process
2024-07-25 01:29:50 +02:00
Laura Hausmann
a845405c45
[backend/queue] Fix QueueService race condition (properly this time)
...
While the previous fix was likely enough, there was still a razor-thin theoretical race condition remaining. This commit fixes said race condition, and simplifies some if statements across the file.
2024-07-25 01:29:32 +02:00
Laura Hausmann
09777fe6c4
[backend/queue] Code cleanup
2024-07-25 00:09:13 +02:00
Laura Hausmann
0f400e8023
[backend/queue] Remove cluster mode (ISH-141)
...
This is wontfix for now, though we might reevaluate this later.
2024-07-25 00:00:43 +02:00
Laura Hausmann
2d178e5a9a
[backend/queue] Allow configuring worker types in cluster mode (ISH-141)
2024-07-24 20:39:45 +02:00
Laura Hausmann
d7cfc24320
[backend/queue] Refactor cluster mode event methods to have less confusing method signatures
2024-07-24 18:01:26 +02:00
Laura Hausmann
7ea2f71abe
[backend/queue] Fix QueueService race condition causing transient queue stalls
2024-07-22 05:59:20 +02:00
Laura Hausmann
df3a7bdfe5
[backend] Apply formatting rules
2024-06-29 01:06:19 +02:00
Laura Hausmann
6743442ce8
[backend/queue] Code cleanup
2024-06-28 18:28:23 +02:00
Laura Hausmann
ca8ce9137d
[backend/queue] Refactor queue job count functions into DatabaseContext
2024-06-28 18:20:59 +02:00
Laura Hausmann
6ba374a681
[backend/queue] Only raise a job queued event for the first matching queue when retrying a job
2024-06-28 18:01:26 +02:00
Laura Hausmann
e971a781e6
[backend/queue] Don't reset RetryCount for manually retried jobs
2024-06-23 13:30:31 +02:00
Laura Hausmann
74b2f69b0c
[backend/razor] Add retry job functionality to queue dashboard
2024-06-22 02:17:35 +02:00
Laura Hausmann
3599150603
[backend/database] Add Exception column to Job table
2024-06-21 19:18:31 +02:00
Laura Hausmann
19c9b7bf90
[backend/database] Add StackTrace column to Job table
2024-06-21 14:05:34 +02:00
Laura Hausmann
47ab95cc1b
[backend/queue] Remove trace logging
2024-06-16 21:56:56 +02:00
Laura Hausmann
8ebef17938
[backend/queue] Fix race condition that could cause temporary queue stalls (until a new job gets queued)
2024-06-16 19:06:53 +02:00
Laura Hausmann
571ef8f027
[backend/queue] Improve logging
2024-06-16 18:55:54 +02:00
Laura Hausmann
07fd0260a8
[backend/queue] Add healthchecks that regularly check for stalled jobs (ISH-361)
2024-06-16 18:07:25 +02:00
Laura Hausmann
7ed39d0c47
[backend/queue] Fix AsyncAutoResetEvent handling, fix stalled queue workers causing no new workers to be spawned
2024-06-16 17:23:28 +02:00
Laura Hausmann
35148d7796
[backend/queue] Add queue processor timeout
...
For now this is hardcoded per queue, but this will be configurable down the line.
2024-06-13 22:41:41 +02:00
Laura Hausmann
aa21e4b05a
[backend/queue] Make queue concurrency configurable
2024-06-10 18:18:39 +02:00
Laura Hausmann
9949e24c8d
[backend] Miscellaneous refactors
2024-05-28 20:50:04 +02:00
Laura Hausmann
e77c768882
[backend/database] Switch job identifier generation scheme to from UUIDv4 to ULID
2024-05-27 22:11:18 +02:00
Laura Hausmann
35153a9080
[backend/queue] Improved AsyncAutoResetEvent implementation
...
This implementation improves the memory behavior of AsyncAutoResetEvents that fire in rapid succession.
2024-05-25 13:34:06 +02:00
Laura Hausmann
ba21cb2b0e
[backend/queue] Fix race condition that could cause PostgresJobQueue.ExecuteAsync to fail when actualParallelism is negative
2024-05-22 23:47:09 +02:00
Laura Hausmann
97532c2b22
[backend/queue] Clean completed jobs in a cron task instead of doing it at job completion
...
This prevents database deadlocks that can occur when many jobs are being processed simultaneously.
2024-05-14 15:05:01 +02:00
Laura Hausmann
363b0c930c
[backend/core] Add Guid.ToStringLower() extension method
2024-04-25 18:26:42 +02:00
Laura Hausmann
c25b613376
[backend/queue] Improve logging
2024-04-15 18:00:11 +02:00
Laura Hausmann
8fb7c8a93e
[backend/queue] Clear change tracker before saving the job state
...
This prevents a queue worker stall when a job fails to execute due to a database exception (leaving unsaved changes in the DbContext change tracker, preventing the job status from being set as failed)
2024-04-15 17:48:36 +02:00
Laura Hausmann
c5bd380344
[backend] Improved debug logging (ISH-238)
2024-04-10 00:15:46 +02:00
Laura Hausmann
60545e25ae
[backend/core] Fix sporadic "key not found" background task failures
2024-04-07 21:37:46 +02:00
