# I am the Watcher. I am your guide through this vast new twtiverse.
# 
# Usage:
#     https://watcher.sour.is/api/plain/users              View list of users and latest twt date.
#     https://watcher.sour.is/api/plain/twt                View all twts.
#     https://watcher.sour.is/api/plain/mentions?uri=:uri  View all mentions for uri.
#     https://watcher.sour.is/api/plain/conv/:hash         View all twts for a conversation subject.
# 
# Options:
#     uri     Filter to show a specific users twts.
#     offset  Start index for quey.
#     limit   Count of items to return (going back in time).
# 
# twt range = 1 46
# self = https://watcher.sour.is/conv/uqxxstq
Hack of the day: running watch -n 60 rm -rf /tmp/yarn-avatar-* in a tmux because all of a sudden, without warning, yarnd started throwing hundreds of gigabytes of files with names like yarn-avatar-62582554 into /tmp, which filled up the entire disk and started crashing other services.
@abucci that’s so weird! I wonder what in the world is causing it. Hopefully @prologic has an insight.
@bender I hope so too. I've never seen anything like this before. Whatever it is, it's strange.
@abucci Wer that's a bug!
@abucci Wer that's a bug!
@abucci What revision are you running?
@abucci What revision are you running?
@prologic 0.15.1, looks like.
https://anthony.buc.ci/info has the deets!
@abucci Please update!
@abucci Please update!
@prologic Sure, but why would this start happening all of a sudden today? Nothing like this has happened before. Is this a known bug?
@abucci I will have a look but I suspect it has something to do with the open nature of the external endpoint. I closed this loophole recently due to other reasons myself.
@abucci I will have a look but I suspect it has something to do with the open nature of the external endpoint. I closed this loophole recently due to other reasons myself.
At work right now so will have more concrete details in a few hours from now
At work right now so will have more concrete details in a few hours from now
I also think you may be running a version that had a bug and lacked cleanup of those temp files
I also think you may be running a version that had a bug and lacked cleanup of those temp files
For example this one that got fixed this year:


commit 4304ec7ea3c5df95e0ed82bfa292c9330e342f61
Author: James Mills <james@mills.io>
Date:   Mon Jan 24 00:10:33 2022 +0000

    Fix bug in DownloadImage() leaking termporary files for external avatar downloads (#746)
For example this one that got fixed this year:


commit 4304ec7ea3c5df95e0ed82bfa292c9330e342f61
Author: James Mills <james@mills.io>
Date:   Mon Jan 24 00:10:33 2022 +0000

    Fix bug in DownloadImage() leaking termporary files for external avatar downloads (#746)
@prologic Aha, got it. Thanks for looking into it. I'm updating now and we'll see if that stops it.
@prologic

abucci@buc:~/yarnd/yarn$ make preflight
Checking Go version ...                 [ ERR ]
Go 1.16+ is required, found go1.22.5
FATAL: πŸ™ preflight failed
make: *** [Makefile:33: preflight] Error 1


πŸ€”
@abucci Fuck that script 🀣 you're good! Just follow the Build from Source docs πŸ˜…
@abucci Fuck that script 🀣 you're good! Just follow the Build from Source docs πŸ˜…
@prologic Alright, running yarnd 0.15.1 now. I stopped my hack so we'll see if the VPS gets clogged with junk πŸ˜†
@abucci Hopefully it shouldn't 🀞
@abucci Hopefully it shouldn't 🀞
Hopefully you should see traffic die off a bit too as the /external endpoint is no longer externally abusable (_get it_) without being an authenticated user -- which became problematic πŸ€¦β€β™‚οΈ -- The web is so fucking hostile 🀬
Hopefully you should see traffic die off a bit too as the /external endpoint is no longer externally abusable (_get it_) without being an authenticated user -- which became problematic πŸ€¦β€β™‚οΈ -- The web is so fucking hostile 🀬
@prologic I'm still getting this crap:

abucci@buc:~/yarnd/yarn$ ls -lh /tmp/yarnd-avatar-*
-rw------- 1 abucci abucci 863M Jul 25 14:19 /tmp/yarnd-avatar-1594499680
-rw------- 1 abucci abucci 7.8G Jul 25 14:19 /tmp/yarnd-avatar-2144295337
-rw------- 1 abucci abucci 9.8G Jul 25 14:19 /tmp/yarnd-avatar-2334738193
-rw------- 1 abucci abucci  10G Jul 25 14:14 /tmp/yarnd-avatar-2494107777
-rw------- 1 abucci abucci 9.5G Jul 25 13:59 /tmp/yarnd-avatar-2619243454
-rw------- 1 abucci abucci  11G Jul 25 14:04 /tmp/yarnd-avatar-2922187513
-rw------- 1 abucci abucci 7.5G Jul 25 14:14 /tmp/yarnd-avatar-349775570
-rw------- 1 abucci abucci  10G Jul 25 14:09 /tmp/yarnd-avatar-3640724243
-rw------- 1 abucci abucci 901M Jul 25 14:19 /tmp/yarnd-avatar-3921595598
-rw------- 1 abucci abucci 9.5G Jul 25 13:59 /tmp/yarnd-avatar-609094539
-rw------- 1 abucci abucci 9.3G Jul 25 14:04 /tmp/yarnd-avatar-755173392
-rw------- 1 abucci abucci 7.9G Jul 25 14:09 /tmp/yarnd-avatar-984061000


Something like 100 Gbytes of this junk has accumulated since I updated and re-started the server. I'm now running the latest version of yarnd, so the update did not fix the problem. Something else is going wrong.

How are temporary files growing to 10 Gbytes in size? The name of the file is "yarn-avatar", but why would avatars be so large?
@prologic I'm still getting this crap:

abucci@buc:~/yarnd/yarn$ ls -lh /tmp/yarnd-avatar-*
-rw------- 1 abucci abucci 863M Jul 25 14:19 /tmp/yarnd-avatar-1594499680
-rw------- 1 abucci abucci 7.8G Jul 25 14:19 /tmp/yarnd-avatar-2144295337
-rw------- 1 abucci abucci 9.8G Jul 25 14:19 /tmp/yarnd-avatar-2334738193
-rw------- 1 abucci abucci  10G Jul 25 14:14 /tmp/yarnd-avatar-2494107777
-rw------- 1 abucci abucci 9.5G Jul 25 13:59 /tmp/yarnd-avatar-2619243454
-rw------- 1 abucci abucci  11G Jul 25 14:04 /tmp/yarnd-avatar-2922187513
-rw------- 1 abucci abucci 7.5G Jul 25 14:14 /tmp/yarnd-avatar-349775570
-rw------- 1 abucci abucci  10G Jul 25 14:09 /tmp/yarnd-avatar-3640724243
-rw------- 1 abucci abucci 901M Jul 25 14:19 /tmp/yarnd-avatar-3921595598
-rw------- 1 abucci abucci 9.5G Jul 25 13:59 /tmp/yarnd-avatar-609094539
-rw------- 1 abucci abucci 9.3G Jul 25 14:04 /tmp/yarnd-avatar-755173392
-rw------- 1 abucci abucci 7.9G Jul 25 14:09 /tmp/yarnd-avatar-984061000
@abucci Hmm that's a bit weird then. Lemme have a poke.
@abucci Hmm that's a bit weird then. Lemme have a poke.
Hah 😈


prologic@JamessMacStudio
Fri Jul 26 00:22:44
~/Projects/yarnsocial/yarn
 (main) 0
$ sift 'yarnd-avatar-*'
internal/utils.go:666:	tf, err := receiveFile(res.Body, "yarnd-avatar-*")


@abucci Don't suppose you can inspect one of those files could you? Kinda wondering if there's some other abuse going on here that I need to plug? πŸ”Œ
Hah 😈


prologic@JamessMacStudio
Fri Jul 26 00:22:44
~/Projects/yarnsocial/yarn
 (main) 0
$ sift 'yarnd-avatar-*'
internal/utils.go:666:\ttf, err := receiveFile(res.Body, "yarnd-avatar-*")


@abucci Don't suppose you can inspect one of those files could you? Kinda wondering if there's some other abuse going on here that I need to plug? πŸ”Œ
Hah 😈


prologic@JamessMacStudio
Fri Jul 26 00:22:44
~/Projects/yarnsocial/yarn
 (main) 0
$ sift 'yarnd-avatar-*'
internal/utils.go:666:	tf, err := receiveFile(res.Body, "yarnd-avatar-*")


@abucci Don't suppose you can inspect one of those files could you? Kinda wondering if there's some other abuse going on here that I need to plug? πŸ”Œ
These should be getting cleaned up, but I'm very concerned about the sizes of these πŸ€”

https://git.mills.io/yarnsocial/yarn/src/commit/983fa87d4ea17f76537e19714ad8a6d19ba9d904/internal/utils.go#L658-L670
These should be getting cleaned up, but I'm very concerned about the sizes of these πŸ€”

https://git.mills.io/yarnsocial/yarn/src/commit/983fa87d4ea17f76537e19714ad8a6d19ba9d904/internal/utils.go#L658-L670
@prologic 10 Gbytes has accumulated since I made that last post. It's coming in at a rate of 55 Mbits/second !
Do you happen to have the activitypub feature turned on btw? In fact could you just list out what features you have enabled please? πŸ™
Do you happen to have the activitypub feature turned on btw? In fact could you just list out what features you have enabled please? πŸ™
@prologic Inspect? What's sift? What would you like to know about the files?
@abucci sift is a tool I use for grep/find, etc.

> What would you like to know about the files?

Roughly what their contents are. I've been reviewing the code paths responsible and have found a flaw that needs to be fixed ASAP.

Here's the PR: https://git.mills.io/yarnsocial/yarn/pulls/1169
@abucci sift is a tool I use for grep/find, etc.

> What would you like to know about the files?

Roughly what their contents are. I've been reviewing the code paths responsible and have found a flaw that needs to be fixed ASAP.

Here's the PR: https://git.mills.io/yarnsocial/yarn/pulls/1169
@abucci So... The only way I see this happening at all is if your pod is fetching feeds which have multi-GB sized avatar(s) in their feed metadata. So the PR I linked earlier will plug that flaw. But now I want to confirm that theory. Can I get you to dump your cache to JSON for me and share it with me?
@abucci So... The only way I see this happening at all is if your pod is fetching feeds which have multi-GB sized avatar(s) in their feed metadata. So the PR I linked earlier will plug that flaw. But now I want to confirm that theory. Can I get you to dump your cache to JSON for me and share it with me?