# I am the Watcher. I am your guide through this vast new twtiverse.
# 
# Usage:
#     https://watcher.sour.is/api/plain/users              View list of users and latest twt date.
#     https://watcher.sour.is/api/plain/twt                View all twts.
#     https://watcher.sour.is/api/plain/mentions?uri=:uri  View all mentions for uri.
#     https://watcher.sour.is/api/plain/conv/:hash         View all twts for a conversation subject.
# 
# Options:
#     uri     Filter to show a specific users twts.
#     offset  Start index for quey.
#     limit   Count of items to return (going back in time).
# 
# twt range = 1 196300
# self = https://watcher.sour.is?offset=166825
# next = https://watcher.sour.is?offset=166925
# prev = https://watcher.sour.is?offset=166725
@abucci sift is a tool I use for grep/find, etc.

> What would you like to know about the files?

Roughly what their contents are. I've been reviewing the code paths responsible and have found a flaw that needs to be fixed ASAP.

Here's the PR: https://git.mills.io/yarnsocial/yarn/pulls/1169
@prologic There are *a lot* of logs being generated by yarnd, which is something I haven't seen before too:


Jul 25 14:32:42 buc yarnd[1911318]: [yarnd] 2024/07/25 14:32:42 (162.211.155.2) "GET /twt/ubhq33a HTTP/1.1" 404 29 643.251µs
Jul 25 14:32:43 buc yarnd[1911318]: [yarnd] 2024/07/25 14:32:43 (162.211.155.2) "GET /twt/112073211746755451 HTTP/1.1" 400 12 505.333µs
Jul 25 14:32:44 buc yarnd[1911318]: [yarnd] 2024/07/25 14:32:44 (111.119.213.103) "GET /twt/whau6pa HTTP/1.1" 200 37360 35.173255ms
Jul 25 14:32:44 buc yarnd[1911318]: [yarnd] 2024/07/25 14:32:44 (162.211.155.2) "GET /twt/112343305123858004 HTTP/1.1" 400 12 455.069µs
Jul 25 14:32:44 buc yarnd[1911318]: [yarnd] 2024/07/25 14:32:44 (168.199.225.19) "GET /external?nick=lovetocode999&uri=http%3A%2F%2Fwww.palapa.pl%2Fbaners.php%3Flink%3Dhttps%3A%2F%2Fwww.dwnewstoday.com HTTP/1.1" 200 36167 19.582077ms
Jul 25 14:32:44 buc yarnd[1911318]: [yarnd] 2024/07/25 14:32:44 (162.211.155.2) "GET /twt/112503061785024494 HTTP/1.1" 400 12 619.152µs
Jul 25 14:32:46 buc yarnd[1911318]: [yarnd] 2024/07/25 14:32:46 (162.211.155.2) "GET /twt/111863876118553837 HTTP/1.1" 400 12 817.678µs
Jul 25 14:32:46 buc yarnd[1911318]: [yarnd] 2024/07/25 14:32:46 (162.211.155.2) "GET /twt/112749994821704400 HTTP/1.1" 400 12 540.616µs
Jul 25 14:32:47 buc yarnd[1911318]: [yarnd] 2024/07/25 14:32:47 (103.204.109.150) "GET /external?nick=lovetocode999&uri=http%3A%2F%2Fampurify.com%2Fbbs%2Fboard.php%3Fbo_table%3Dfree%26wr_id%3D113858 HTTP/1.1" 200 36187 15.95329ms


I've seen that nick=lovetocode999 a bunch.
@prologic Inspect? What's sift? What would you like to know about the files?
@abucci I believe you are correct.
@abucci I believe you are correct.
@abucci That's fucking insane 😱 I know what code-paths is triggering this, but need to confirm a few other things... Some correlation with logs would also help...
@abucci That's fucking insane 😱 I know what code-paths is triggering this, but need to confirm a few other things... Some correlation with logs would also help...
Do you happen to have the activitypub feature turned on btw? In fact could you just list out what features you have enabled please? 🙏
Do you happen to have the activitypub feature turned on btw? In fact could you just list out what features you have enabled please? 🙏
@prologic 10 Gbytes has accumulated since I made that last post. It's coming in at a rate of 55 Mbits/second !
These should be getting cleaned up, but I'm very concerned about the sizes of these 🤔

https://git.mills.io/yarnsocial/yarn/src/commit/983fa87d4ea17f76537e19714ad8a6d19ba9d904/internal/utils.go#L658-L670
These should be getting cleaned up, but I'm very concerned about the sizes of these 🤔

https://git.mills.io/yarnsocial/yarn/src/commit/983fa87d4ea17f76537e19714ad8a6d19ba9d904/internal/utils.go#L658-L670
Hah 😈


prologic@JamessMacStudio
Fri Jul 26 00:22:44
~/Projects/yarnsocial/yarn
 (main) 0
$ sift 'yarnd-avatar-*'
internal/utils.go:666:	tf, err := receiveFile(res.Body, "yarnd-avatar-*")


@abucci Don't suppose you can inspect one of those files could you? Kinda wondering if there's some other abuse going on here that I need to plug? 🔌
Hah 😈


prologic@JamessMacStudio
Fri Jul 26 00:22:44
~/Projects/yarnsocial/yarn
 (main) 0
$ sift 'yarnd-avatar-*'
internal/utils.go:666:\ttf, err := receiveFile(res.Body, "yarnd-avatar-*")


@abucci Don't suppose you can inspect one of those files could you? Kinda wondering if there's some other abuse going on here that I need to plug? 🔌
Hah 😈


prologic@JamessMacStudio
Fri Jul 26 00:22:44
~/Projects/yarnsocial/yarn
 (main) 0
$ sift 'yarnd-avatar-*'
internal/utils.go:666:	tf, err := receiveFile(res.Body, "yarnd-avatar-*")


@abucci Don't suppose you can inspect one of those files could you? Kinda wondering if there's some other abuse going on here that I need to plug? 🔌
@prologic I think there's more to it than that. I've updated, yet hundreds of gigabytes of junk is still accumulating.
@abucci Hmm that's a bit weird then. Lemme have a poke.
@abucci Hmm that's a bit weird then. Lemme have a poke.
@prologic I'm still getting this crap:

abucci@buc:~/yarnd/yarn$ ls -lh /tmp/yarnd-avatar-*
-rw------- 1 abucci abucci 863M Jul 25 14:19 /tmp/yarnd-avatar-1594499680
-rw------- 1 abucci abucci 7.8G Jul 25 14:19 /tmp/yarnd-avatar-2144295337
-rw------- 1 abucci abucci 9.8G Jul 25 14:19 /tmp/yarnd-avatar-2334738193
-rw------- 1 abucci abucci  10G Jul 25 14:14 /tmp/yarnd-avatar-2494107777
-rw------- 1 abucci abucci 9.5G Jul 25 13:59 /tmp/yarnd-avatar-2619243454
-rw------- 1 abucci abucci  11G Jul 25 14:04 /tmp/yarnd-avatar-2922187513
-rw------- 1 abucci abucci 7.5G Jul 25 14:14 /tmp/yarnd-avatar-349775570
-rw------- 1 abucci abucci  10G Jul 25 14:09 /tmp/yarnd-avatar-3640724243
-rw------- 1 abucci abucci 901M Jul 25 14:19 /tmp/yarnd-avatar-3921595598
-rw------- 1 abucci abucci 9.5G Jul 25 13:59 /tmp/yarnd-avatar-609094539
-rw------- 1 abucci abucci 9.3G Jul 25 14:04 /tmp/yarnd-avatar-755173392
-rw------- 1 abucci abucci 7.9G Jul 25 14:09 /tmp/yarnd-avatar-984061000
@prologic I'm still getting this crap:

abucci@buc:~/yarnd/yarn$ ls -lh /tmp/yarnd-avatar-*
-rw------- 1 abucci abucci 863M Jul 25 14:19 /tmp/yarnd-avatar-1594499680
-rw------- 1 abucci abucci 7.8G Jul 25 14:19 /tmp/yarnd-avatar-2144295337
-rw------- 1 abucci abucci 9.8G Jul 25 14:19 /tmp/yarnd-avatar-2334738193
-rw------- 1 abucci abucci  10G Jul 25 14:14 /tmp/yarnd-avatar-2494107777
-rw------- 1 abucci abucci 9.5G Jul 25 13:59 /tmp/yarnd-avatar-2619243454
-rw------- 1 abucci abucci  11G Jul 25 14:04 /tmp/yarnd-avatar-2922187513
-rw------- 1 abucci abucci 7.5G Jul 25 14:14 /tmp/yarnd-avatar-349775570
-rw------- 1 abucci abucci  10G Jul 25 14:09 /tmp/yarnd-avatar-3640724243
-rw------- 1 abucci abucci 901M Jul 25 14:19 /tmp/yarnd-avatar-3921595598
-rw------- 1 abucci abucci 9.5G Jul 25 13:59 /tmp/yarnd-avatar-609094539
-rw------- 1 abucci abucci 9.3G Jul 25 14:04 /tmp/yarnd-avatar-755173392
-rw------- 1 abucci abucci 7.9G Jul 25 14:09 /tmp/yarnd-avatar-984061000


Something like 100 Gbytes of this junk has accumulated since I updated and re-started the server. I'm now running the latest version of yarnd, so the update did not fix the problem. Something else is going wrong.

How are temporary files growing to 10 Gbytes in size? The name of the file is "yarn-avatar", but why would avatars be so large?
Hmm remove the cpu limits on this pod, not even sure why I had 'em set tbh, we decided at my day job that setting cpu limits on containers is a bit of a silly idea too. Anyway, pod should be much snappier now 😅
Hmm remove the cpu limits on this pod, not even sure why I had 'em set tbh, we decided at my day job that setting cpu limits on containers is a bit of a silly idea too. Anyway, pod should be much snappier now 😅
@movq Oh nothing much 🤣 Just a bunch of folks running really old versions of yarnd that were susceptible to abuse on the open web 🤣
@movq Oh nothing much 🤣 Just a bunch of folks running really old versions of yarnd that were susceptible to abuse on the open web 🤣
What the heck is going on here today, so many messages. 😂
What the heck is going on here today, so many messages. 😂
What the heck is going on here today, so many messages. 😂
What the heck is going on here today, so many messages. 😂
Hopefully you should see traffic die off a bit too as the /external endpoint is no longer externally abusable (_get it_) without being an authenticated user -- which became problematic 🤦‍♂️ -- The web is so fucking hostile 🤬
Hopefully you should see traffic die off a bit too as the /external endpoint is no longer externally abusable (_get it_) without being an authenticated user -- which became problematic 🤦‍♂️ -- The web is so fucking hostile 🤬
@abucci Hopefully it shouldn't 🤞
@abucci Hopefully it shouldn't 🤞
@prologic Alright, running yarnd 0.15.1 now. I stopped my hack so we'll see if the VPS gets clogged with junk 😆
@abucci Fuck that script 🤣 you're good! Just follow the Build from Source docs 😅
@abucci Fuck that script 🤣 you're good! Just follow the Build from Source docs 😅
@prologic

abucci@buc:~/yarnd/yarn$ make preflight
Checking Go version ...                 [ ERR ]
Go 1.16+ is required, found go1.22.5
FATAL: 🙁 preflight failed
make: *** [Makefile:33: preflight] Error 1


🤔
@prologic Aha, got it. Thanks for looking into it. I'm updating now and we'll see if that stops it.
Thinking we need to adapt the UI a little bit to something like this
Thinking we need to adapt the UI a little bit to something like this
@bender I can see the same errors again hmmm 🧐 @stigatle Did you run out of disk again? 😅
@bender I can see the same errors again hmmm 🧐 @stigatle Did you run out of disk again? 😅
@prologic nope, I see no images. If you go to Stig pod, it has all images broken.
I had a play with LiveKit Agents Playground: KITT and I have to say it's pretty impressive. Not the ChatGPT part of course, but the speech recognition and text to speech synthesis.

> KITT is an AI voice assistant powered by LiveKit Agents, Deepgram, Eleven Labs, and ChatGPT. It is running on LiveKit Playground.

It's too bad it relies on three cloud services, none of which can be run locally (_with the exception of Ollama that you could replace the OpenAI component with_).
I had a play with LiveKit Agents Playground: KITT and I have to say it's pretty impressive. Not the ChatGPT part of course, but the speech recognition and text to speech synthesis.

> KITT is an AI voice assistant powered by LiveKit Agents, Deepgram, Eleven Labs, and ChatGPT. It is running on LiveKit Playground.

It's too bad it relies on three cloud services, none of which can be run locally (_with the exception of Ollama that you could replace the OpenAI component with_).
@lyse Man gotta love that sunset !!! So nice 😊
@lyse Man gotta love that sunset !!! So nice 😊
You should have the fancy new SPA-like UI too 😅 (just checked!)
You should have the fancy new SPA-like UI too 😅 (just checked!)
@stigatle No worries at all! 👌
@stigatle No worries at all! 👌
@prologic I got it working, installed go under home, and pointed to that, as well as the variables you mentioned, that enabled me to compile it. deleted the old yarnd , and made sure I run the new one.
Thanks for the help (as always :) ).
@prologic I got it working, I reinstalled go under home (instead of where go wiki tells me to install it), and pointed to that, as well as the variables you mentioned, that enabled me to compile it. deleted the old yarnd , and made sure I run the new one.
Thanks for the help (as always :) ).
@stigatle Note that "Building From Source" is covered in the docs
@stigatle Note that "Building From Source" is covered in the docs
@prologic Ok, thank you, I'll try that.
You are reminding me that I should cut a release soon™ so there are binaires you can just "download" and use for the platform of choice 😅
You are reminding me that I should cut a release soon™ so there are binaires you can just "download" and use for the platform of choice 😅
@stigatle So make deps would have installed some tools in either $GOPATH/bin or $GOBIN. See which with go env. Chuck that in your $PATH and you're good to run make server. Normally this would be something like:


GOBIN=$HOME/go/bin
GOPATH=$HOME/go
export GOPATH GOBIN
...
@stigatle So make deps would have installed some tools in either $GOPATH/bin or $GOBIN. See which with go env. Chuck that in your $PATH and you're good to run make server. Normally this would be something like:


GOBIN=$HOME/go/bin
GOPATH=$HOME/go
export GOPATH GOBIN
...
@prologic I did that, and it returns no error.

eps user@server:~/backup/yarn$ make server /bin/sh: 4: minify: not found /bin/sh: 5: minify: not found /bin/sh: 6: minify: not found make: *** [Makefile:84: generate] Error 127
@prologic I did that, and it returns no error.

eps user@server:~/backup/yarn$ make server /bin/sh: 4: minify: not found /bin/sh: 5: minify: not found /bin/sh: 6: minify: not found make: *** [Makefile:84: generate] Error 127
[47°09′30″S, 126°43′16″W] Wind speed: 83kph -- batteries low
@stigatle Run make deps. I use a non-standard (_written in Go_) minify tool
@stigatle Run make deps. I use a non-standard (_written in Go_) minify tool
@prologic hm, I installed latest go (vps did not have that intalled), I then did make deps, then make server, when I use the last command it said minify was not installed, I assumed minify package was the one to get, but it fails with that. (debian).
@stigatle Take a backup of the data dir in case I screwed something up 🙏
@stigatle Take a backup of the data dir in case I screwed something up 🙏
@stigatle Sweet 👌
@stigatle Sweet 👌
@stigatle I think pods have become exploited over time so I've had to tighten up some feature like the external handler 😢
@stigatle I think pods have become exploited over time so I've had to tighten up some feature like the external handler 😢
@prologic Ok, I'll get it updated today.
@prologic Ok, good to know. So the issue is the same 'avatar in tmp' issue filling up the disk. I did not check the dates on the avatars there, but it worked fined earlier yesterday, and was full today, so it seems to be the same issue mentioned earlier on here. I'll keep an eye on it. I have not updated yarnd for a while, so I run v 0.15.1.
@stigatle Ahh! Please update to the latest main 🙏
@stigatle Ahh! Please update to the latest main 🙏
@stigatle Works now! 🥳
@stigatle Works now! 🥳
Threshold: 8.00 miles, 00:09:15 average pace, 01:14:00 duration
9:50 for warm-up and cool-down then 8:00 on and 11:30 off three times.
#running #treadmill
Threshold: 8.00 miles, 00:09:15 average pace, 01:14:00 duration
9:50 for warm-up and cool-down then 8:00 on and 11:30 off three times.
#running #treadmill
Threshold: 8.00 miles, 00:09:15 average pace, 01:14:00 duration
9:50 for warm-up and cool-down then 8:00 on and 11:30 off three times.
#running #treadmill
@lyse @bender try again please.
@prologic hm, it seems to be full disk that's the issue, same problem with the avatar in tmp it seems that's mentioned earlier here. I deleted them now. I regained 33% space (9GB).
@prologic hm, it seems to be full disk that's the issue, same problem with the avatar in tmp it seems that's mentioned earlier here
@prologic hm, okay, not sure what to do with it though. If I remember correct - I've just forwarded the subdomain to yarnd's port - and that's it.

This is the config:

r { server_name yarn.stigatle.no; location / { proxy_set_header Host $host; proxy_pass http://127.0.0.1:8000; proxy_redirect off; }
@stigatle It looks like your some kind of problem with the reverse proxy in front of yarnd? 🤔I 'm seeing this error: incomplete chunked encoding error(s) I don't know anything about this though, tbh I've never seen this before myself 🤔
@stigatle It looks like your some kind of problem with the reverse proxy in front of yarnd? 🤔I 'm seeing this error: incomplete chunked encoding error(s) I don't know anything about this though, tbh I've never seen this before myself 🤔
@stigatle It looks like your some kind of problem with the reverse proxy in front of yarnd? 🤔I 'm seeing this error: incomplete chunked encoding" error(s) ![](https://twtxt.net/media/CYtYtfTcvTzHXdxYFesUcX.png) I don't know anything about this though, tbh I've never seen this before myself 🤔
@stigatle I'm doing okay 👌 Busy with work as you can imagine, and still tinkering of course whenever I can spare a moment or two! 😅
@stigatle I'm doing okay 👌 Busy with work as you can imagine, and still tinkering of course whenever I can spare a moment or two! 😅
@prologic Thank you! How are you doing these days?
@lyse hm, that's weird, not sure what's going on there to be honest.
Jugando a ser mayor
/https://baldo.cat/media/photos/photo_20325-07-2024_09-39-29.jpg) #catsoftwtxt
Jugando a ser mayor
#catsoftwtxt
Jugando a ser mayor
#catsoftwtxt
For example this one that got fixed this year:


commit 4304ec7ea3c5df95e0ed82bfa292c9330e342f61
Author: James Mills <james@mills.io>
Date:   Mon Jan 24 00:10:33 2022 +0000

    Fix bug in DownloadImage() leaking termporary files for external avatar downloads (#746)
For example this one that got fixed this year:


commit 4304ec7ea3c5df95e0ed82bfa292c9330e342f61
Author: James Mills <james@mills.io>
Date:   Mon Jan 24 00:10:33 2022 +0000

    Fix bug in DownloadImage() leaking termporary files for external avatar downloads (#746)
[47°09′03″S, 126°43′31″W] Wind speed: 84kph -- batteries low
[47°09′10″S, 126°43′31″W] Wind speed: N/A -- Cannot comunicate