# I am the Watcher. I am your guide through this vast new twtiverse.
# 
# Usage:
#     https://watcher.sour.is/api/plain/users              View list of users and latest twt date.
#     https://watcher.sour.is/api/plain/twt                View all twts.
#     https://watcher.sour.is/api/plain/mentions?uri=:uri  View all mentions for uri.
#     https://watcher.sour.is/api/plain/conv/:hash         View all twts for a conversation subject.
# 
# Options:
#     uri     Filter to show a specific users twts.
#     offset  Start index for quey.
#     limit   Count of items to return (going back in time).
# 
# twt range = 1 196296
# self = https://watcher.sour.is?offset=166856
# next = https://watcher.sour.is?offset=166956
# prev = https://watcher.sour.is?offset=166756
Just thinking out loud here... With that PR merged (_or if you built off that branch_), you _might_ hopefully see new errors popup and we might catch this problematic bad feed in the act? Hmmm 🧐
@slashdot I _thought_ Sunday was the hottest day on Earth πŸ€¦β€β™‚οΈ wtf is wrong with Slashdot these days?! 🀣
@slashdot I _thought_ Sunday was the hottest day on Earth πŸ€¦β€β™‚οΈ wtf is wrong with Slashdot these days?! 🀣
if we can figure out wtf is going on here and my theory is right, we can blacklist that feed, hell even add it to the codebase as an "asshole".
if we can figure out wtf is going on here and my theory is right, we can blacklist that feed, hell even add it to the codebase as an "asshole".
@stigatle The problem is it'll only cause the attack to stop and error out. It won't stop your pod from trying to do this over and over again. That's why I need some help inspecting both your pods for "bad feeds".
@stigatle The problem is it'll only cause the attack to stop and error out. It won't stop your pod from trying to do this over and over again. That's why I need some help inspecting both your pods for "bad feeds".
@prologic I'm running it now. I'll keep an eye out for the tmp folder now (I built the branch you have made). I'll let you know shortly if it helped on my end.
@prologic Ok, I'm running it now. I'll keep an eye out for the tmp folder now (I built the branch you have made). I'll let you know shortly if it helped on my end.
@abucci / @stigatle Please git pull, rebuild and redeploy.

There is also a shell script in ./tools called dump_cache.sh. Please run this, dump your cache and share it with me. πŸ™
@abucci / @stigatle Please git pull, rebuild and redeploy.

There is also a shell script in ./tools called dump_cache.sh. Please run this, dump your cache and share it with me. πŸ™
I'm going to merge this...
I'm going to merge this...
@abucci Yeah I've had to block entire ASN(s) recently myself from bad actors, mostly bad AI bots actually from Facebook and Caude AI
@abucci Yeah I've had to block entire ASN(s) recently myself from bad actors, mostly bad AI bots actually from Facebook and Caude AI
@stigatle I used the following hack to keep my VPS from running out of space: watch -n 60 rm -rf /tmp/yarn-avatar-*, run in tmux so it keeps running.
The vast majority of this traffic was coming from a single IP address. I blocked that IP on my VPS, and I sent an abuse report to the abuse email of the service provider. That ought to slow it down, but the vulnerability persists and I'm still getting traffic from other IPs that seem to be doing the same thing.
Or if y'all trust my monkey-ass coding skillz I'll just merge and you can do a git pull and rebuild πŸ˜…
Or if y'all trust my monkey-ass coding skillz I'll just merge and you can do a git pull and rebuild πŸ˜…
@stigatle / @abucci My current working theory is that there is an asshole out there that has a feed that both your pods are fetching with a multi-GB avatar URL advertised in their feed's preamble (metadata). I'd love for you both to review this PR, and once merged, re-roll your pods and dump your respective caches and share with me using https://gist.mills.io/
@stigatle / @abucci My current working theory is that there is an asshole out there that has a feed that both your pods are fetching with a multi-GB avatar URL advertised in their feed's preamble (metadata). I'd love for you both to review this PR, and once merged, re-roll your pods and dump your respective caches and share with me using https://gist.mills.io/
@prologic yeah I still do have that issue.
@prologic yeah I still do have that issue, I compiled latest main, did not apply any patches or anything like that.
@stigatle I'm wondering whether you're having the same issue as @abucci still? mulit-GB yarnd-avatar-*1 files piling up in /tmp/? πŸ€”
@stigatle I'm wondering whether you're having the same issue as @abucci still? mulit-GB yarnd-avatar-*1 files piling up in /tmp/? πŸ€”
@prologic yeah, I ran out of space again. also have the activitypub stuff turned off (just so you know).
@abucci So... The only way I see this happening at all is if your pod is fetching feeds which have multi-GB sized avatar(s) in their feed metadata. So the PR I linked earlier will plug that flaw. But now I want to confirm that theory. Can I get you to dump your cache to JSON for me and share it with me?
@abucci So... The only way I see this happening at all is if your pod is fetching feeds which have multi-GB sized avatar(s) in their feed metadata. So the PR I linked earlier will plug that flaw. But now I want to confirm that theory. Can I get you to dump your cache to JSON for me and share it with me?
@abucci Yeah that should be okay, you get so much crap on the web πŸ€¦β€β™‚οΈ
@abucci Yeah that should be okay, you get so much crap on the web πŸ€¦β€β™‚οΈ
@abucci sift is a tool I use for grep/find, etc.

> What would you like to know about the files?

Roughly what their contents are. I've been reviewing the code paths responsible and have found a flaw that needs to be fixed ASAP.

Here's the PR: https://git.mills.io/yarnsocial/yarn/pulls/1169
@abucci sift is a tool I use for grep/find, etc.

> What would you like to know about the files?

Roughly what their contents are. I've been reviewing the code paths responsible and have found a flaw that needs to be fixed ASAP.

Here's the PR: https://git.mills.io/yarnsocial/yarn/pulls/1169
@prologic There are *a lot* of logs being generated by yarnd, which is something I haven't seen before too:


Jul 25 14:32:42 buc yarnd[1911318]: [yarnd] 2024/07/25 14:32:42 (162.211.155.2) "GET /twt/ubhq33a HTTP/1.1" 404 29 643.251Β΅s
Jul 25 14:32:43 buc yarnd[1911318]: [yarnd] 2024/07/25 14:32:43 (162.211.155.2) "GET /twt/112073211746755451 HTTP/1.1" 400 12 505.333Β΅s
Jul 25 14:32:44 buc yarnd[1911318]: [yarnd] 2024/07/25 14:32:44 (111.119.213.103) "GET /twt/whau6pa HTTP/1.1" 200 37360 35.173255ms
Jul 25 14:32:44 buc yarnd[1911318]: [yarnd] 2024/07/25 14:32:44 (162.211.155.2) "GET /twt/112343305123858004 HTTP/1.1" 400 12 455.069Β΅s
Jul 25 14:32:44 buc yarnd[1911318]: [yarnd] 2024/07/25 14:32:44 (168.199.225.19) "GET /external?nick=lovetocode999&uri=http%3A%2F%2Fwww.palapa.pl%2Fbaners.php%3Flink%3Dhttps%3A%2F%2Fwww.dwnewstoday.com HTTP/1.1" 200 36167 19.582077ms
Jul 25 14:32:44 buc yarnd[1911318]: [yarnd] 2024/07/25 14:32:44 (162.211.155.2) "GET /twt/112503061785024494 HTTP/1.1" 400 12 619.152Β΅s
Jul 25 14:32:46 buc yarnd[1911318]: [yarnd] 2024/07/25 14:32:46 (162.211.155.2) "GET /twt/111863876118553837 HTTP/1.1" 400 12 817.678Β΅s
Jul 25 14:32:46 buc yarnd[1911318]: [yarnd] 2024/07/25 14:32:46 (162.211.155.2) "GET /twt/112749994821704400 HTTP/1.1" 400 12 540.616Β΅s
Jul 25 14:32:47 buc yarnd[1911318]: [yarnd] 2024/07/25 14:32:47 (103.204.109.150) "GET /external?nick=lovetocode999&uri=http%3A%2F%2Fampurify.com%2Fbbs%2Fboard.php%3Fbo_table%3Dfree%26wr_id%3D113858 HTTP/1.1" 200 36187 15.95329ms


I've seen that nick=lovetocode999 a bunch.
@prologic Inspect? What's sift? What would you like to know about the files?
@abucci I believe you are correct.
@abucci I believe you are correct.
@abucci That's fucking insane 😱 I know what code-paths is triggering this, but need to confirm a few other things... Some correlation with logs would also help...
@abucci That's fucking insane 😱 I know what code-paths is triggering this, but need to confirm a few other things... Some correlation with logs would also help...
Do you happen to have the activitypub feature turned on btw? In fact could you just list out what features you have enabled please? πŸ™
Do you happen to have the activitypub feature turned on btw? In fact could you just list out what features you have enabled please? πŸ™
@prologic 10 Gbytes has accumulated since I made that last post. It's coming in at a rate of 55 Mbits/second !
These should be getting cleaned up, but I'm very concerned about the sizes of these πŸ€”

https://git.mills.io/yarnsocial/yarn/src/commit/983fa87d4ea17f76537e19714ad8a6d19ba9d904/internal/utils.go#L658-L670
These should be getting cleaned up, but I'm very concerned about the sizes of these πŸ€”

https://git.mills.io/yarnsocial/yarn/src/commit/983fa87d4ea17f76537e19714ad8a6d19ba9d904/internal/utils.go#L658-L670
Hah 😈


prologic@JamessMacStudio
Fri Jul 26 00:22:44
~/Projects/yarnsocial/yarn
 (main) 0
$ sift 'yarnd-avatar-*'
internal/utils.go:666:	tf, err := receiveFile(res.Body, "yarnd-avatar-*")


@abucci Don't suppose you can inspect one of those files could you? Kinda wondering if there's some other abuse going on here that I need to plug? πŸ”Œ
Hah 😈


prologic@JamessMacStudio
Fri Jul 26 00:22:44
~/Projects/yarnsocial/yarn
 (main) 0
$ sift 'yarnd-avatar-*'
internal/utils.go:666:\ttf, err := receiveFile(res.Body, "yarnd-avatar-*")


@abucci Don't suppose you can inspect one of those files could you? Kinda wondering if there's some other abuse going on here that I need to plug? πŸ”Œ
Hah 😈


prologic@JamessMacStudio
Fri Jul 26 00:22:44
~/Projects/yarnsocial/yarn
 (main) 0
$ sift 'yarnd-avatar-*'
internal/utils.go:666:	tf, err := receiveFile(res.Body, "yarnd-avatar-*")


@abucci Don't suppose you can inspect one of those files could you? Kinda wondering if there's some other abuse going on here that I need to plug? πŸ”Œ
@prologic I think there's more to it than that. I've updated, yet hundreds of gigabytes of junk is still accumulating.
@abucci Hmm that's a bit weird then. Lemme have a poke.
@abucci Hmm that's a bit weird then. Lemme have a poke.
@prologic I'm still getting this crap:

abucci@buc:~/yarnd/yarn$ ls -lh /tmp/yarnd-avatar-*
-rw------- 1 abucci abucci 863M Jul 25 14:19 /tmp/yarnd-avatar-1594499680
-rw------- 1 abucci abucci 7.8G Jul 25 14:19 /tmp/yarnd-avatar-2144295337
-rw------- 1 abucci abucci 9.8G Jul 25 14:19 /tmp/yarnd-avatar-2334738193
-rw------- 1 abucci abucci  10G Jul 25 14:14 /tmp/yarnd-avatar-2494107777
-rw------- 1 abucci abucci 9.5G Jul 25 13:59 /tmp/yarnd-avatar-2619243454
-rw------- 1 abucci abucci  11G Jul 25 14:04 /tmp/yarnd-avatar-2922187513
-rw------- 1 abucci abucci 7.5G Jul 25 14:14 /tmp/yarnd-avatar-349775570
-rw------- 1 abucci abucci  10G Jul 25 14:09 /tmp/yarnd-avatar-3640724243
-rw------- 1 abucci abucci 901M Jul 25 14:19 /tmp/yarnd-avatar-3921595598
-rw------- 1 abucci abucci 9.5G Jul 25 13:59 /tmp/yarnd-avatar-609094539
-rw------- 1 abucci abucci 9.3G Jul 25 14:04 /tmp/yarnd-avatar-755173392
-rw------- 1 abucci abucci 7.9G Jul 25 14:09 /tmp/yarnd-avatar-984061000
@prologic I'm still getting this crap:

abucci@buc:~/yarnd/yarn$ ls -lh /tmp/yarnd-avatar-*
-rw------- 1 abucci abucci 863M Jul 25 14:19 /tmp/yarnd-avatar-1594499680
-rw------- 1 abucci abucci 7.8G Jul 25 14:19 /tmp/yarnd-avatar-2144295337
-rw------- 1 abucci abucci 9.8G Jul 25 14:19 /tmp/yarnd-avatar-2334738193
-rw------- 1 abucci abucci  10G Jul 25 14:14 /tmp/yarnd-avatar-2494107777
-rw------- 1 abucci abucci 9.5G Jul 25 13:59 /tmp/yarnd-avatar-2619243454
-rw------- 1 abucci abucci  11G Jul 25 14:04 /tmp/yarnd-avatar-2922187513
-rw------- 1 abucci abucci 7.5G Jul 25 14:14 /tmp/yarnd-avatar-349775570
-rw------- 1 abucci abucci  10G Jul 25 14:09 /tmp/yarnd-avatar-3640724243
-rw------- 1 abucci abucci 901M Jul 25 14:19 /tmp/yarnd-avatar-3921595598
-rw------- 1 abucci abucci 9.5G Jul 25 13:59 /tmp/yarnd-avatar-609094539
-rw------- 1 abucci abucci 9.3G Jul 25 14:04 /tmp/yarnd-avatar-755173392
-rw------- 1 abucci abucci 7.9G Jul 25 14:09 /tmp/yarnd-avatar-984061000


Something like 100 Gbytes of this junk has accumulated since I updated and re-started the server. I'm now running the latest version of yarnd, so the update did not fix the problem. Something else is going wrong.

How are temporary files growing to 10 Gbytes in size? The name of the file is "yarn-avatar", but why would avatars be so large?
Hmm remove the cpu limits on this pod, not even sure why I had 'em set tbh, we decided at my day job that setting cpu limits on containers is a bit of a silly idea too. Anyway, pod should be much snappier now πŸ˜…
Hmm remove the cpu limits on this pod, not even sure why I had 'em set tbh, we decided at my day job that setting cpu limits on containers is a bit of a silly idea too. Anyway, pod should be much snappier now πŸ˜…
@movq Oh nothing much 🀣 Just a bunch of folks running really old versions of yarnd that were susceptible to abuse on the open web 🀣
@movq Oh nothing much 🀣 Just a bunch of folks running really old versions of yarnd that were susceptible to abuse on the open web 🀣
What the heck is going on here today, so many messages. πŸ˜‚
What the heck is going on here today, so many messages. πŸ˜‚
What the heck is going on here today, so many messages. πŸ˜‚
What the heck is going on here today, so many messages. πŸ˜‚
Hopefully you should see traffic die off a bit too as the /external endpoint is no longer externally abusable (_get it_) without being an authenticated user -- which became problematic πŸ€¦β€β™‚οΈ -- The web is so fucking hostile 🀬
Hopefully you should see traffic die off a bit too as the /external endpoint is no longer externally abusable (_get it_) without being an authenticated user -- which became problematic πŸ€¦β€β™‚οΈ -- The web is so fucking hostile 🀬
@abucci Hopefully it shouldn't 🀞
@abucci Hopefully it shouldn't 🀞
@prologic Alright, running yarnd 0.15.1 now. I stopped my hack so we'll see if the VPS gets clogged with junk πŸ˜†
@abucci Fuck that script 🀣 you're good! Just follow the Build from Source docs πŸ˜…
@abucci Fuck that script 🀣 you're good! Just follow the Build from Source docs πŸ˜…
@prologic

abucci@buc:~/yarnd/yarn$ make preflight
Checking Go version ...                 [ ERR ]
Go 1.16+ is required, found go1.22.5
FATAL: πŸ™ preflight failed
make: *** [Makefile:33: preflight] Error 1


πŸ€”
@prologic Aha, got it. Thanks for looking into it. I'm updating now and we'll see if that stops it.
Thinking we need to adapt the UI a little bit to something like this
Thinking we need to adapt the UI a little bit to something like this
@bender I can see the same errors again hmmm 🧐 @stigatle Did you run out of disk again? πŸ˜…
@bender I can see the same errors again hmmm 🧐 @stigatle Did you run out of disk again? πŸ˜…
@prologic nope, I see no images. If you go to Stig pod, it has all images broken.
I had a play with LiveKit Agents Playground: KITT and I have to say it's pretty impressive. Not the ChatGPT part of course, but the speech recognition and text to speech synthesis.

> KITT is an AI voice assistant powered by LiveKit Agents, Deepgram, Eleven Labs, and ChatGPT. It is running on LiveKit Playground.

It's too bad it relies on three cloud services, none of which can be run locally (_with the exception of Ollama that you could replace the OpenAI component with_).
I had a play with LiveKit Agents Playground: KITT and I have to say it's pretty impressive. Not the ChatGPT part of course, but the speech recognition and text to speech synthesis.

> KITT is an AI voice assistant powered by LiveKit Agents, Deepgram, Eleven Labs, and ChatGPT. It is running on LiveKit Playground.

It's too bad it relies on three cloud services, none of which can be run locally (_with the exception of Ollama that you could replace the OpenAI component with_).
@lyse Man gotta love that sunset !!! So nice 😊
@lyse Man gotta love that sunset !!! So nice 😊
You should have the fancy new SPA-like UI too πŸ˜… (just checked!)
You should have the fancy new SPA-like UI too πŸ˜… (just checked!)
@stigatle No worries at all! πŸ‘Œ
@stigatle No worries at all! πŸ‘Œ
@prologic I got it working, installed go under home, and pointed to that, as well as the variables you mentioned, that enabled me to compile it. deleted the old yarnd , and made sure I run the new one.
Thanks for the help (as always :) ).
@prologic I got it working, I reinstalled go under home (instead of where go wiki tells me to install it), and pointed to that, as well as the variables you mentioned, that enabled me to compile it. deleted the old yarnd , and made sure I run the new one.
Thanks for the help (as always :) ).
@stigatle Note that "Building From Source" is covered in the docs
@stigatle Note that "Building From Source" is covered in the docs
@prologic Ok, thank you, I'll try that.
You are reminding me that I should cut a release soonβ„’ so there are binaires you can just "download" and use for the platform of choice πŸ˜…
You are reminding me that I should cut a release soonβ„’ so there are binaires you can just "download" and use for the platform of choice πŸ˜…
@stigatle So make deps would have installed some tools in either $GOPATH/bin or $GOBIN. See which with go env. Chuck that in your $PATH and you're good to run make server. Normally this would be something like:


GOBIN=$HOME/go/bin
GOPATH=$HOME/go
export GOPATH GOBIN
...
@stigatle So make deps would have installed some tools in either $GOPATH/bin or $GOBIN. See which with go env. Chuck that in your $PATH and you're good to run make server. Normally this would be something like:


GOBIN=$HOME/go/bin
GOPATH=$HOME/go
export GOPATH GOBIN
...
@prologic I did that, and it returns no error.

eps user@server:~/backup/yarn$ make server /bin/sh: 4: minify: not found /bin/sh: 5: minify: not found /bin/sh: 6: minify: not found make: *** [Makefile:84: generate] Error 127
@prologic I did that, and it returns no error.

eps user@server:~/backup/yarn$ make server /bin/sh: 4: minify: not found /bin/sh: 5: minify: not found /bin/sh: 6: minify: not found make: *** [Makefile:84: generate] Error 127
[47Β°09β€²30β€³S, 126Β°43β€²16β€³W] Wind speed: 83kph -- batteries low
@stigatle Run make deps. I use a non-standard (_written in Go_) minify tool
@stigatle Run make deps. I use a non-standard (_written in Go_) minify tool
@prologic hm, I installed latest go (vps did not have that intalled), I then did make deps, then make server, when I use the last command it said minify was not installed, I assumed minify package was the one to get, but it fails with that. (debian).
@stigatle Take a backup of the data dir in case I screwed something up πŸ™
@stigatle Take a backup of the data dir in case I screwed something up πŸ™