yarn after the latest upgrade. Like a good 60 seconds.
Jul 25 16:01:17 buc yarnd[1921547]: time="2024-07-25T16:01:17Z" level=error msg="https://yarn.stigatle.no/user/stigatle/twtxt.txt: client.Do fail: Get \\"https://yarn.stigatle.no/user/stigatle/twtxt.txt\\": dial tcp 185.97.32.18:443: i/o timeout (Client.Timeout exceeded while awaiting headers)" error="Get \\"https://yarn.stigatle.no/user/stigatle/twtxt.txt\\": dial tcp 185.97.32.18:443: i/o timeout (Client.Timeout exceeded while awaiting headers)"
I no longer see twts from @stigatle at all.
receieveFile()_)? ๐ค
receieveFile()_)? ๐ค
for url in $(jq -r '.Twters[].avatar' cache.json | sed '/^$/d' | grep -v -E '(twtxt.net|anthony.buc.ci|yarn.stigatle.no|yarn.mills.io)' | sort -u); do echo "$url $(curl -I -s -o /dev/null -w '%header{content-length}' "$url")"; done
...
๐ Let's see... ๐ค
for url in $(jq -r '.Twters[].avatar' cache.json | sed '/^$/d' | grep -v -E '(twtxt.net|anthony.buc.ci|yarn.stigatle.no|yarn.mills.io)' | sort -u); do echo "$url $(curl -I -s -o /dev/null -w '%header{content-length}' "$url")"; done
...
๐ Let's see... ๐ค
In today's economy, nobody optimizes something if it can be just called good enough with the next generation hardware. That's especially the mindset of big coorporations.
Anyway, getting sidetracked from the original post. :-)
$ jq '.Feeds | keys[]' cache.json | wc -l
4402
If you both don't mind dropping your caches. I would recommend it. Settings -> Poderator Settings -> Refresh cache.
$ jq '.Feeds | keys[]' cache.json | wc -l
4402
If you both don't mind dropping your caches. I would recommend it. Settings -> Poderator Settings -> Refresh cache.
$ jq '.Feeds | keys[]' cache.json | wc -l
4402
If you both don't mind dropping your caches. I would recommend it. Settings -> Poderator Settings -> Reset cache.
./tools/dump_cache.sh: line 8: bat: command not found
No Token Provided
I don't have
bat on my VPS and there is no package for installing it. Is cat a reasonable alternate?
https://twtxt.net/external?nick=nosuchuser&uri=https://foo.com
Change
nosuchuser to any phrase at all.If you hit https://twtxt.net/external?nick=nosuchuser , you're given an error. If you hit that URL above with the
uri parameter, you can a legitimate-looking page. I think that is a bug.
https://drive.proton.me/urls/XRKQQ632SG#LXWehEZMNQWF
lovetocode999 on my pod. I think it should 404, and maybe with a delay, to discourage whatever this abuse is. Basically this can be used to DDoS a pod by forcing it to generate a hunch of HTML just by doing a bogus GET like this.
yarnd server logs too. Any new interesting errors? ๐ค No more multi-GB tmp files? ๐ค
yarnd server logs too. Any new interesting errors? ๐ค No more multi-GB tmp files? ๐ค
backup_db.sh and dump_cache.sh They pipe JSON to stdout and prompt for your admin password. Example:
URL=<your_pod_url> ADMIN=<your_admin_user> ./tools/dump_cache.sh > cache.json
backup_db.sh and dump_cache.sh They pipe JSON to stdout and prompt for your admin password. Example:
URL=<your_pod_url> ADMIN=<your_admin_user> ./tools/dump_cache.sh > cache.json
"GET /external?nick=lovetocode999&uri=https://vuf.minagricultura.gov.co/Lists/Informacin%20Servicios%20Web/DispForm.aspx?ID=8375144 HTTP/1.1" 200 35861 17.077914ms
always to
nick=lovetocode999, but with different uris. What are these calls?
Mate, these are some really nice gems! What a stunning landscape. I love it. Holy cow, that wooden church looks really sick. Even though, I'm not a scroll guy and prefer simple, straight designs, I have to say, that the interior craftmanship is something to admire.
git pull, rebuild and redeploy.There is also a shell script in
./tools called dump_cache.sh. Please run this, dump your cache and share it with me. ๐
git pull, rebuild and redeploy.There is also a shell script in
./tools called dump_cache.sh. Please run this, dump your cache and share it with me. ๐
watch -n 60 rm -rf /tmp/yarn-avatar-*, run in tmux so it keeps running.
git pull and rebuild ๐
git pull and rebuild ๐
yarnd-avatar-*1 files piling up in /tmp/? ๐ค
yarnd-avatar-*1 files piling up in /tmp/? ๐ค
sift is a tool I use for grep/find, etc.> What would you like to know about the files?
Roughly what their contents are. I've been reviewing the code paths responsible and have found a flaw that needs to be fixed ASAP.
Here's the PR: https://git.mills.io/yarnsocial/yarn/pulls/1169
sift is a tool I use for grep/find, etc.> What would you like to know about the files?
Roughly what their contents are. I've been reviewing the code paths responsible and have found a flaw that needs to be fixed ASAP.
Here's the PR: https://git.mills.io/yarnsocial/yarn/pulls/1169
yarnd, which is something I haven't seen before too:
Jul 25 14:32:42 buc yarnd[1911318]: [yarnd] 2024/07/25 14:32:42 (162.211.155.2) "GET /twt/ubhq33a HTTP/1.1" 404 29 643.251ยตs
Jul 25 14:32:43 buc yarnd[1911318]: [yarnd] 2024/07/25 14:32:43 (162.211.155.2) "GET /twt/112073211746755451 HTTP/1.1" 400 12 505.333ยตs
Jul 25 14:32:44 buc yarnd[1911318]: [yarnd] 2024/07/25 14:32:44 (111.119.213.103) "GET /twt/whau6pa HTTP/1.1" 200 37360 35.173255ms
Jul 25 14:32:44 buc yarnd[1911318]: [yarnd] 2024/07/25 14:32:44 (162.211.155.2) "GET /twt/112343305123858004 HTTP/1.1" 400 12 455.069ยตs
Jul 25 14:32:44 buc yarnd[1911318]: [yarnd] 2024/07/25 14:32:44 (168.199.225.19) "GET /external?nick=lovetocode999&uri=http%3A%2F%2Fwww.palapa.pl%2Fbaners.php%3Flink%3Dhttps%3A%2F%2Fwww.dwnewstoday.com HTTP/1.1" 200 36167 19.582077ms
Jul 25 14:32:44 buc yarnd[1911318]: [yarnd] 2024/07/25 14:32:44 (162.211.155.2) "GET /twt/112503061785024494 HTTP/1.1" 400 12 619.152ยตs
Jul 25 14:32:46 buc yarnd[1911318]: [yarnd] 2024/07/25 14:32:46 (162.211.155.2) "GET /twt/111863876118553837 HTTP/1.1" 400 12 817.678ยตs
Jul 25 14:32:46 buc yarnd[1911318]: [yarnd] 2024/07/25 14:32:46 (162.211.155.2) "GET /twt/112749994821704400 HTTP/1.1" 400 12 540.616ยตs
Jul 25 14:32:47 buc yarnd[1911318]: [yarnd] 2024/07/25 14:32:47 (103.204.109.150) "GET /external?nick=lovetocode999&uri=http%3A%2F%2Fampurify.com%2Fbbs%2Fboard.php%3Fbo_table%3Dfree%26wr_id%3D113858 HTTP/1.1" 200 36187 15.95329ms
I've seen that
nick=lovetocode999 a bunch.
sift? What would you like to know about the files?
activitypub feature turned on btw? In fact could you just list out what features you have enabled please? ๐
activitypub feature turned on btw? In fact could you just list out what features you have enabled please? ๐
https://git.mills.io/yarnsocial/yarn/src/commit/983fa87d4ea17f76537e19714ad8a6d19ba9d904/internal/utils.go#L658-L670
https://git.mills.io/yarnsocial/yarn/src/commit/983fa87d4ea17f76537e19714ad8a6d19ba9d904/internal/utils.go#L658-L670
prologic@JamessMacStudio
Fri Jul 26 00:22:44
~/Projects/yarnsocial/yarn
(main) 0
$ sift 'yarnd-avatar-*'
internal/utils.go:666: tf, err := receiveFile(res.Body, "yarnd-avatar-*")
@abucci Don't suppose you can inspect one of those files could you? Kinda wondering if there's some other abuse going on here that I need to plug? ๐
prologic@JamessMacStudio
Fri Jul 26 00:22:44
~/Projects/yarnsocial/yarn
(main) 0
$ sift 'yarnd-avatar-*'
internal/utils.go:666:\ttf, err := receiveFile(res.Body, "yarnd-avatar-*")
@abucci Don't suppose you can inspect one of those files could you? Kinda wondering if there's some other abuse going on here that I need to plug? ๐
prologic@JamessMacStudio
Fri Jul 26 00:22:44
~/Projects/yarnsocial/yarn
(main) 0
$ sift 'yarnd-avatar-*'
internal/utils.go:666: tf, err := receiveFile(res.Body, "yarnd-avatar-*")
@abucci Don't suppose you can inspect one of those files could you? Kinda wondering if there's some other abuse going on here that I need to plug? ๐
abucci@buc:~/yarnd/yarn$ ls -lh /tmp/yarnd-avatar-*
-rw------- 1 abucci abucci 863M Jul 25 14:19 /tmp/yarnd-avatar-1594499680
-rw------- 1 abucci abucci 7.8G Jul 25 14:19 /tmp/yarnd-avatar-2144295337
-rw------- 1 abucci abucci 9.8G Jul 25 14:19 /tmp/yarnd-avatar-2334738193
-rw------- 1 abucci abucci 10G Jul 25 14:14 /tmp/yarnd-avatar-2494107777
-rw------- 1 abucci abucci 9.5G Jul 25 13:59 /tmp/yarnd-avatar-2619243454
-rw------- 1 abucci abucci 11G Jul 25 14:04 /tmp/yarnd-avatar-2922187513
-rw------- 1 abucci abucci 7.5G Jul 25 14:14 /tmp/yarnd-avatar-349775570
-rw------- 1 abucci abucci 10G Jul 25 14:09 /tmp/yarnd-avatar-3640724243
-rw------- 1 abucci abucci 901M Jul 25 14:19 /tmp/yarnd-avatar-3921595598
-rw------- 1 abucci abucci 9.5G Jul 25 13:59 /tmp/yarnd-avatar-609094539
-rw------- 1 abucci abucci 9.3G Jul 25 14:04 /tmp/yarnd-avatar-755173392
-rw------- 1 abucci abucci 7.9G Jul 25 14:09 /tmp/yarnd-avatar-984061000
abucci@buc:~/yarnd/yarn$ ls -lh /tmp/yarnd-avatar-*
-rw------- 1 abucci abucci 863M Jul 25 14:19 /tmp/yarnd-avatar-1594499680
-rw------- 1 abucci abucci 7.8G Jul 25 14:19 /tmp/yarnd-avatar-2144295337
-rw------- 1 abucci abucci 9.8G Jul 25 14:19 /tmp/yarnd-avatar-2334738193
-rw------- 1 abucci abucci 10G Jul 25 14:14 /tmp/yarnd-avatar-2494107777
-rw------- 1 abucci abucci 9.5G Jul 25 13:59 /tmp/yarnd-avatar-2619243454
-rw------- 1 abucci abucci 11G Jul 25 14:04 /tmp/yarnd-avatar-2922187513
-rw------- 1 abucci abucci 7.5G Jul 25 14:14 /tmp/yarnd-avatar-349775570
-rw------- 1 abucci abucci 10G Jul 25 14:09 /tmp/yarnd-avatar-3640724243
-rw------- 1 abucci abucci 901M Jul 25 14:19 /tmp/yarnd-avatar-3921595598
-rw------- 1 abucci abucci 9.5G Jul 25 13:59 /tmp/yarnd-avatar-609094539
-rw------- 1 abucci abucci 9.3G Jul 25 14:04 /tmp/yarnd-avatar-755173392
-rw------- 1 abucci abucci 7.9G Jul 25 14:09 /tmp/yarnd-avatar-984061000
Something like 100 Gbytes of this junk has accumulated since I updated and re-started the server. I'm now running the latest version of
yarnd, so the update did not fix the problem. Something else is going wrong.How are temporary files growing to 10 Gbytes in size? The name of the file is "yarn-avatar", but why would avatars be so large?
yarnd that were susceptible to abuse on the open web ๐คฃ
yarnd that were susceptible to abuse on the open web ๐คฃ