Commit graph

95 commits

Author SHA1 Message Date
tobi 21557c92d9
[bugfix] Set refetch to true in iterateDescendants (#600) 2022-05-24 11:00:37 +02:00
tobi f0c9f4169b
[bugfix] Fix multiple dereferences of boosted status causing media duplication (#589)
* add some announces to test models

* start on announce test logic

* test federatingDB.Announce

* change signature of GetRemoteStatus

* remove 'refresh' logic and replace it with refetch

* go fmt

* remove timeline manager from processor test

* make zork created at determinate

* test get account statuses

* test get + serialize zork

* make account keys determinate

* make admin accountCreate time determinate

* test account to as

* init test config before test log

* test status to frontend

* remove daft Within check

* hack around a bit

* use index of slice
2022-05-23 16:40:03 +01:00
tobi 469da93678
[security] Check all involved IRIs during block checking (#593)
* tidy up context keys, add otherInvolvedIRIs

* add ReplyToable interface

* skip block check if we own the requesting domain

* add block check for other involved IRIs

* use cacheable status fetch

* remove unused ContextActivity

* remove unused ContextActivity

* add helper for unique URIs

* check through CCs and clean slice

* add GetAccountIDForStatusURI

* add GetAccountIDForAccountURI

* check blocks on involved account

* add statuses to tests

* add some blocked tests

* go fmt

* extract Tos as well as CCs

* test PostInboxRequestBodyHook

* add some more testActivities

* deduplicate involvedAccountIDs

* go fmt

* use cacheable db functions, remove new functions
2022-05-23 11:46:50 +02:00
kim 223025fc27
[security] transport.Controller{} and transport.Transport{} security and performance improvements (#564)
* cache transports in controller by privkey-generated pubkey, add retry logic to transport requests

Signed-off-by: kim <grufwub@gmail.com>

* update code comments, defer mutex unlocks

Signed-off-by: kim <grufwub@gmail.com>

* add count to 'performing request' log message

Signed-off-by: kim <grufwub@gmail.com>

* reduce repeated conversions of same url.URL object

Signed-off-by: kim <grufwub@gmail.com>

* move worker.Worker to concurrency subpackage, add WorkQueue type, limit transport http client use by WorkQueue

Signed-off-by: kim <grufwub@gmail.com>

* fix security advisories regarding max outgoing conns, max rsp body size

- implemented by a new httpclient.Client{} that wraps an underlying
  client with a queue to limit connections, and limit reader wrapping
  a response body with a configured maximum size
- update pub.HttpClient args passed around to be this new httpclient.Client{}

Signed-off-by: kim <grufwub@gmail.com>

* add httpclient tests, move ip validation to separate package + change mechanism

Signed-off-by: kim <grufwub@gmail.com>

* fix merge conflicts

Signed-off-by: kim <grufwub@gmail.com>

* use singular mutex in transport rather than separate signer mus

Signed-off-by: kim <grufwub@gmail.com>

* improved useragent string

Signed-off-by: kim <grufwub@gmail.com>

* add note regarding missing test

Signed-off-by: kim <grufwub@gmail.com>

* remove useragent field from transport (instead store in controller)

Signed-off-by: kim <grufwub@gmail.com>

* shutup linter

Signed-off-by: kim <grufwub@gmail.com>

* reset other signing headers on each loop iteration

Signed-off-by: kim <grufwub@gmail.com>

* respect request ctx during retry-backoff sleep period

Signed-off-by: kim <grufwub@gmail.com>

* use external pkg with docs explaining performance "hack"

Signed-off-by: kim <grufwub@gmail.com>

* use http package constants instead of string method literals

Signed-off-by: kim <grufwub@gmail.com>

* add license file headers

Signed-off-by: kim <grufwub@gmail.com>

* update code comment to match new func names

Signed-off-by: kim <grufwub@gmail.com>

* updates to user-agent string

Signed-off-by: kim <grufwub@gmail.com>

* update signed testrig models to fit with new transport logic (instead uses separate signer now)

Signed-off-by: kim <grufwub@gmail.com>

* fuck you linter

Signed-off-by: kim <grufwub@gmail.com>
2022-05-15 11:16:43 +02:00
tobi b28fd7433a
[chore] Fix inbox iris test to use 'contains' instead of index (#529) 2022-05-02 16:12:56 +02:00
tobi a5852fd7e4
[performance] Speed up some of the slower db queries (#523)
* remove unnecessary LOWER() db calls

* warn during slow db queries

* use bundb built-in exists function

* add db block test

* update account block query

* add domain block db test

* optimize domain block query

* fix implementing wrong test

* exclude most columns when checking block

* go fmt

* remote more unnecessary use of LOWER()
2022-05-02 11:53:46 +01:00
tobi d93e8ddf75
[chore] Add Federatingactor.Send() tests and log call (#509)
* expose testrig util functions

* add tests for federatingActor

* rename some suite vars
2022-04-29 15:53:04 +02:00
tobi 63f84da3e4
[chore] Inboxes for iri test (#508)
* tidy up some federation tests

* add missing license to following.go

* give zork some followers, as a treat

* test InboxesForIRI

* Go fmt

* update timeline tests
2022-04-29 15:05:13 +02:00
kim 420e2fb22b
replace async client API / federator msg processing with worker pools (#497)
* replace async client API / federator msg processing with worker pools
* appease our lord-and-saviour, the linter
2022-04-28 13:23:11 +01:00
kim cc5f2e98b7
[bugfix] Fix possible race condition in federatingdb (#490)
Signed-off-by: kim <grufwub@gmail.com>
2022-04-28 11:18:27 +02:00
tobi 9cf66bf298
[chore] Return more useful errors from auth failure (#494)
* try rsa_sha256 sig algo first

* return more informative errors from auth

* adapt to reworked auth function
2022-04-26 18:10:11 +02:00
tobi 8d34d5af3c
Revert "[chore] Tidy up federating db locks a tiny bit (#472)" (#479)
This reverts commit 31c1067a54.
2022-04-24 11:56:06 +02:00
tobi 31c1067a54
[chore] Tidy up federating db locks a tiny bit (#472) 2022-04-19 16:31:37 +02:00
tobi 707de094a0
[dependency] Update superseriousbusiness/activity dependency (#446)
* bump superseriousbusiness/activity version

* fetch list of follower inbox iris
2022-04-05 18:05:24 +02:00
tobi 25cab0e1f4
[bugfix] Fix images not being processed correctly sometimes (#437)
* bump exif-terminator to latest version

* add and test giant turnip from turnip.farm

* don't error if content property is nil
2022-03-21 19:46:51 +01:00
tobi 07727753b9
[feature] Clean up/uncache remote media (#407)
* Add whereNotEmptyAndNotNull

* Add GetRemoteOlderThanDays

* Add GetRemoteOlderThanDays

* Add PruneRemote to Manager interface

* Start implementing PruneRemote

* add new attachment + status to tests

* fix up and test GetRemoteOlderThan

* fix bad import

* PruneRemote: return number pruned

* add Cached column to mediaattachment

* update + test pruneRemote

* update mediaTest

* use Cached column

* upstep bun to latest version

* embed structs in mediaAttachment

* migrate mediaAttachment to new format

* don't default cached to true

* select only remote media

* update db dependencies

* step bun back to last working version

* update pruneRemote to use Cached field

* fix storage path of test attachments

* add recache logic to manager

* fix trimmed aspect ratio

* test prune and recache

* return errwithcode

* tidy up different paths for emoji vs attachment

* fix incorrect thumbnail type being stored

* expose TransportController to media processor

* implement tee-ing recached content

* add thoughts of dog to test fedi attachments

* test get remote files

* add comment on PruneRemote

* add postData cleanup to recache

* test thumbnail fetching

* add incredible diagram

* go mod tidy

* buffer pipes for recache streaming

* test for client stops reading after 1kb

* add media-remote-cache-days to config

* add cron package

* wrap logrus so it's available to cron

* start and stop cron jobs gracefully
2022-03-07 11:08:26 +01:00
tobi 55b83bea7c
[feature] Add postDataCallbackFunc to allow cleanup (#408) 2022-02-22 13:50:33 +01:00
tsmethurst dba9ad4348 hopefully fix potential race condition 2022-02-08 13:17:10 +01:00
tsmethurst 4c294a596a Merge branch 'main' into media_refactor 2022-02-08 12:17:18 +01:00
tsmethurst e811d03cc9 update outdated comment 2022-02-08 10:08:27 +01:00
Forest Johnson 6ed368cbeb
[feature] add authorization to the already-existing authentication (#365)
* add ensureUserIsAuthorizedOrRedirect to /oauth/authorize

* adding authorization (email confirm, account approve, etc) to TokenCheck

* revert un-needed changes to signin.go

* oops what happened here

* error css

* add account.SuspendedAt check

* remove redundant checks from oauth util Authed function

* wip tests

* tests passing

* stop stripping useful information from ErrAlreadyExists

* that feeling of scraping the dryer LINT off the screen

* oops I didn't mean to get rid of this NewTestRouter function

* make tests work with recorder

* re-add ConfigureTemplatesWithGin to handle template path err

Co-authored-by: tsmethurst <tobi.smethurst@protonmail.com>
2022-02-07 12:04:31 +01:00
tsmethurst c156602c66 ensure blocking calls to getRemoteAccount before showing stuff to client 2022-01-25 13:48:13 +01:00
tsmethurst eba66d3a88 only update account in db if changed 2022-01-25 11:21:22 +01:00
tsmethurst 926b37e271 change getaccount function signature 2022-01-24 18:12:42 +01:00
tsmethurst c12520167d use background context w/deadline 2022-01-24 18:12:04 +01:00
tsmethurst 667e7f112c update remote account get/deref logic 2022-01-24 13:12:17 +01:00
tsmethurst c157b1b20b rework data function to provide filesize 2022-01-23 14:41:58 +01:00
tsmethurst 589bb9df02 pass reader around instead of []byte 2022-01-16 18:52:55 +01:00
tsmethurst 113f9d9ab4 pass a function into the manager, start work on emoji 2022-01-11 17:49:14 +01:00
tsmethurst e0f9323b9a test the media manager a bit, add shutdown logic 2022-01-10 18:36:09 +01:00
tsmethurst 0ef478584c add async test 2022-01-09 18:52:28 +01:00
tsmethurst dccf21dd87 tests are passing, but there's still much to be done 2022-01-09 18:41:22 +01:00
tsmethurst f61c3ddcf7 compiling now 2022-01-08 17:17:01 +01:00
tsmethurst c4d63d125b more refactoring, media handler => manager 2021-12-28 16:36:00 +01:00
tobi ef5a9256a8
Extend license notices to 2022 (#354) 2021-12-20 18:42:19 +01:00
tobi cb8688f429
Remove unnecessary storage config variables (#344)
* rewire config to not use extraneous serve vars

* rename 'file' to 'local' for consistency

* use Type and Size again
2021-12-20 15:19:53 +01:00
tobi 0884f89431
Implement Cobra CLI tooling, Viper config tooling (#336)
* start pulling out + replacing urfave and config

* replace many many instances of config

* move more stuff => viper

* properly remove urfave

* move some flags to root command

* add testrig commands to root

* alias config file keys

* start adding cli parsing tests

* reorder viper init

* remove config path alias

* fmt

* change config file keys to non-nested

* we're more or less in business now

* tidy up the common func

* go fmt

* get tests passing again

* add note about the cliparsing tests

* reorganize

* update docs with changes

* structure cmd dir better

* rename + move some files around

* fix dangling comma
2021-12-07 13:31:39 +01:00
tobi f8630348b4
Enable stricter linting with golangci-lint (#316)
* update golangci-lint

* add golangci config file w/ more linters

* correct issues flagged by stricter linters

* add more generous timeout for golangci-lint

* add some style + formatting guidelines

* move timeout to config file

* go fmt
2021-11-22 08:46:19 +01:00
tobi 09ef9e639e
move to ssb gofed fork (#298) 2021-11-13 17:29:43 +01:00
tobi 829a934d23
update dependencies (#296) 2021-11-13 12:29:08 +01:00
tobi 4b1d9d3780
Serve outbox for Actor (#289)
* add statusesvisible convenience function

* add minID + onlyPublic to account statuses get

* move swagger collection stuff to common

* start working on Outbox GETting

* move functions into federationProcessor

* outboxToASCollection

* add statusesvisible convenience function

* add minID + onlyPublic to account statuses get

* move swagger collection stuff to common

* start working on Outbox GETting

* move functions into federationProcessor

* outboxToASCollection

* bit more work on outbox paging

* wrapNoteInCreate function

* test + hook up the processor functions

* don't do prev + next links on empty reply

* test get outbox through api

* don't fail on no status entries

* add outbox implementation doc

* typo
2021-10-24 11:57:39 +02:00
tobi 15621f5324
Follow request improvements (#282)
* tiny doc update

* add rejectfollowrequest to db

* add follow request reject to processor

* add reject handler

* tidy up follow request api

* tidy up federation call

* regenerate swagger docs

* api endpoint tests

* processor test

* add reject federatingdb handler

* start writing reject tests

* test reject follow request

* go fmt

* increase sleep for slow test setups

* more relaxed time.sleep
2021-10-16 13:27:43 +02:00
R. Aidan Campbell 083099a957
reference global logrus (#274)
* reference logrus' global logger instead of passing and storing a logger reference everywhere

* always directly use global logrus logger instead of referencing an instance

* test suites should also directly use the global logrus logger

* rename gin logging function to clarify that it's middleware

* correct comments which erroneously referenced removed logger parameter

* setting log level for tests now uses logrus' exported type instead of the string value, to guarantee error isn't possible
2021-10-11 14:37:33 +02:00
tobi 367bdca250
Handle forwarded messages (#273)
* correct path of foss_satan

* add APIri and notes

* test create forward note

* rename target => receiving account

* split up create into separate funcs

* update extractFromCtx

* tidy up from federator processing

* foss satan => http not https

* check if status in db

* mock dereference of status from IRI

* add forward message deref test

* update test with activities

* add remote_account_2 to test rig
2021-10-10 12:39:25 +02:00
tobi ff77bf1d11
fix logs not working properly (#264) 2021-10-04 16:00:35 +02:00
tobi e04b187702
Refactor/tidy (#261)
* tidy up streaming

* cut down code duplication

* test get followers/following

* test streaming processor

* fix some test models

* add TimeMustParse

* fix uri / url typo

* make trace logging less verbose

* make logging more consistent

* disable quote on logging

* remove context.Background

* remove many extraneous mastodon references

* regenerate swagger

* don't log query on no rows result

* log latency first for easier reading
2021-10-04 15:24:19 +02:00
tobi 0cd2bd2960
allow dereferencing of groups (#256) 2021-09-30 12:27:42 +02:00
tobi 36a09dd0df
handle remote account deletion more systematically (#254) 2021-09-30 10:56:02 +02:00
tobi 08cb8a3385
skip account on error instead of returning error (#251)
* skip account on error instead of returning error

* still return error on a real error
2021-09-28 16:19:13 +02:00
tobi b5a7e1ba32
Account update issue (#250)
* start poking around

* tests

* notes and fiddling
2021-09-28 15:21:59 +02:00