Commit graph

94 commits

Author SHA1 Message Date
Laura Hausmann
a762a9201e
[backend/core] Revert IAsyncEnumerable naming 2024-11-20 01:12:32 +01:00
Laura Hausmann
24b412bd75
[sln] Reformat code 2024-11-20 00:48:29 +01:00
Laura Hausmann
7dec2514da
[sln] Apply code style 2024-11-20 00:48:29 +01:00
Laura Hausmann
6c76b2b2c5
[backend] Switch params methods to IEnumerable<T> 2024-11-18 19:02:41 +01:00
Kopper
94328a3eef [backend] Use a stack instead of queue when backfilling
This makes each reply chain load completely before loading the next
chain, instead of the current behavior that loads all replies of one
depth before loading the next "layer".

This won't make much of a difference *now*, but should result in more
intuitive behavior when live updating of newly loaded replies gets
implemented.
2024-11-06 20:02:29 +01:00
Laura Hausmann
1b885e6ed4
[backend/queue] Also deliver note updates to relays 2024-11-03 19:40:35 +01:00
Kopper
90f1e30ef2 [backend] Fix reply backfill limiting 2024-10-26 07:24:39 +02:00
kopper
27ac36d747
[backend] Add config option to sign reply backfills with requesting user
Signed-off-by: Laura Hausmann <laura@hausmann.dev>
2024-10-24 21:18:58 +02:00
kopper
7173373330
[backend/federation] Improved reply backfill
Instead of backfilling every note we come across that has a reply
collection, only schedule a backfill job if someone wants to see the
replies (on GET MastoAPI /context, or Iceshrimp API /descendants)

Reply backfilling is also done on a ThreadIdOrId basis as opposed to the
previous way of backfilling individual notes. This allows us finer
grained control over the recursion and frees up the job queue, alongside
allowing for easier implementation of context collection backfill in the
future (by mapping each context collection to a thread)

---

Currently, note threads are implicit based on the "threadId" column of a
note, which can be null (where it's the same as the note's "id")

This commit turns note threads into an actual entity, and as a part of
that, makes "threadId" non-nullable (by specifically setting it to "id"
for those cases)

This is done to attach extra metadata to the entire thread, currently
just the time of when it was last backfilled, but more may be added in
the future (the context collection associated with this thread, for example)

---

The data format for backfill jobs have backwards-incompatibly changed
since the introduction of the feature. We can drop all old jobs without
causing too much trouble as they will be re-scheduled on demand

---

Signed-off-by: Laura Hausmann <laura@hausmann.dev>
2024-10-24 21:18:57 +02:00
Laura Hausmann
9275e3b21a
[backend/drive] Add fixup-media startup command (ISH-520) 2024-10-18 22:43:25 +02:00
Laura Hausmann
5d01c11e35
[backend/queue] Increase inbox job timeout to 120 seconds 2024-10-17 22:54:40 +02:00
Laura Hausmann
300c7d661c
[backend/queue] Set background-task queue timeout to 15 minutes (ISH-532) 2024-10-15 17:13:17 +02:00
Laura Hausmann
c4cf6f5008
[backend/core] Only send activities to relays in state accepted 2024-10-15 15:47:37 +02:00
Laura Hausmann
60d112a6ad
[backend/core] Fix user purge job processor errors (ISH-525) 2024-10-15 02:41:01 +02:00
Laura Hausmann
bef7dd9855
[backend/core] Fix user purge failures (ISH-525) 2024-10-11 19:54:04 +02:00
Laura Hausmann
5f5a0c5c0f
[backend/api] Add moderation endpoints (ISH-116) 2024-10-09 19:11:54 +02:00
Laura Hausmann
99ea086731
[backend/federation] Only forward activities to relays that are addressed to the public audience 2024-09-28 04:21:31 +02:00
Laura Hausmann
df26db0585
[backend/federation] Add relay support (ISH-266) 2024-09-28 00:32:43 +02:00
Laura Hausmann
36f277e0d3
[backend/queue] Improve inbox queue retry mechanism (ISH-503)
This retries jobs for a cumulative maximum of 24 hours.
2024-09-26 20:43:45 +02:00
Laura Hausmann
9f39895142
[backend/queue] Improve inbox queue logging 2024-09-26 20:12:40 +02:00
Laura Hausmann
1c2079e1ee
[backend] Rework GracefulException throws without additional detail 2024-09-24 00:45:18 +02:00
Laura Hausmann
d1721d71c2
[backend/federation] Retry inbox jobs with exponential backoff (ISH-499)
This should cover most transient failures. If one is missing, we can add it later.
2024-09-24 00:36:18 +02:00
Kopper
48784690c1 [backend/federation] Do not queue backfill jobs that have reached their recursion limit 2024-09-18 18:27:14 +02:00
Laura Hausmann
9481e1d7fb
[backend/federation] Fix instances returning non-HTTP errors not getting marked as unresponsive
This fixes e.g. DNS errors from occurring perpetually when trying to federate with dead instances.
2024-09-15 16:49:21 +02:00
Kopper
d6a8282a18
[backend/federation] Try using the original collection object when possible 2024-09-15 01:49:42 +02:00
Kopper
024de937d0
[backend/federation] Backfill replies 2024-09-15 01:47:41 +02:00
Laura Hausmann
9af06c58a9
[backend] Code cleanup 2024-08-18 21:06:54 +02:00
Laura Hausmann
dea43be619
[backend/core] Remove userPrivateKey cache in deliver queue
There's no performance difference between fetching just the private key from the user_keypair table and the previous caching implementation, so there's no point in keeping it in the cache.
2024-08-18 21:06:33 +02:00
Laura Hausmann
c07bb35548
[backend/drive] Refactor ImageProcessor into a modular system
This commit lays the groundwork for a user-configurable image processing pipeline. It has exactly the same behavior as the old ImageProcessor, just modular & compartmentalized. It also adds support for AVIF & JXL encoding, though no code paths call it just yet.
2024-08-12 03:21:03 +02:00
Laura Hausmann
c67984c411
[backend] Manually reformat difficult-to-read switch expressions that got hard wrapped 2024-07-11 17:46:34 +02:00
Laura Hausmann
88bef87ef5
[backend] Improve client error logging in DeliverQueue 2024-07-10 20:11:39 +02:00
Laura Hausmann
70f822ab3d
[backend/core] Don't try to delete remote files from local/object storage 2024-07-02 23:34:16 +02:00
Laura Hausmann
df3a7bdfe5
[backend] Apply formatting rules 2024-06-29 01:06:19 +02:00
Laura Hausmann
d6c3fb028f
[backend/queue] Fix sporadic background task queue errors 2024-06-28 18:54:29 +02:00
Laura Hausmann
583e954106
[backend/federation] Don't mark client error deliver jobs as successful 2024-06-23 14:45:04 +02:00
Laura Hausmann
d5a5a3d464
[backend/federation] Don't retry deliver jobs that returned a client error (except for HTTP 429) 2024-06-23 14:38:45 +02:00
Laura Hausmann
3599150603
[backend/database] Add Exception column to Job table 2024-06-21 19:18:31 +02:00
Laura Hausmann
19c9b7bf90
[backend/database] Add StackTrace column to Job table 2024-06-21 14:05:34 +02:00
Laura Hausmann
eb167cf482
[backend/federation] Check for Instance.IsSuspended in PreDeliverQueue for ASFollow activities
We're not currently using that property, but we may in the future.
2024-06-19 18:37:43 +02:00
Laura Hausmann
9dbc7cfc14
[backend/federation] Filter skipped & blocked instances in PreDeliverQueue (ISH-366)
This:
- improves performance (because we can check them in batches)
- improves efficiency (because we don't create deliver jobs that get discarded immediately)
- allows us to check whether the activity is an ASFollow & let it through if it is (for skipped instances filter only)

We still check for blocked instances in the deliver queue in case a deliver job is created via another code path.
2024-06-19 17:37:57 +02:00
Laura Hausmann
24bde5ad29
[backend] Use User.IsLocalUser / .IsRemoteUser instead of null-checks where applicable (ISH-370) 2024-06-17 17:25:29 +02:00
Laura Hausmann
0a5e130783
[backend/federation] Log instance blocked failures in inbox queue as debug 2024-06-16 22:36:38 +02:00
Laura Hausmann
571ef8f027
[backend/queue] Improve logging 2024-06-16 18:55:54 +02:00
Laura Hausmann
35148d7796
[backend/queue] Add queue processor timeout
For now this is hardcoded per queue, but this will be configurable down the line.
2024-06-13 22:41:41 +02:00
Laura Hausmann
aa21e4b05a
[backend/queue] Make queue concurrency configurable 2024-06-10 18:18:39 +02:00
Laura Hausmann
adba960f8c
[backend/drive] Check for isLink attribute when handling deletion/expiry for deduplicated files (ISH-334) 2024-05-18 15:53:59 +02:00
Laura Hausmann
9dd34aa7e3
[backend/drive] Improve deletion/expiry logic for deduplicated files (ISH-334) 2024-05-18 15:48:25 +02:00
Laura Hausmann
363b0c930c
[backend/core] Add Guid.ToStringLower() extension method 2024-04-25 18:26:42 +02:00
Laura Hausmann
385f5843f7
[backend/helpers] Rename LDHelpers.Compact(ASActivity) to LDHelpers.CompactToPayload 2024-04-24 04:05:26 +02:00
Laura Hausmann
806bfca099
[backend/core] Improve user deletes (ISH-222) 2024-04-10 20:53:50 +02:00