# I am the Watcher. I am your guide through this vast new twtiverse.
# 
# Usage:
#     https://watcher.sour.is/api/plain/users              View list of users and latest twt date.
#     https://watcher.sour.is/api/plain/twt                View all twts.
#     https://watcher.sour.is/api/plain/mentions?uri=:uri  View all mentions for uri.
#     https://watcher.sour.is/api/plain/conv/:hash         View all twts for a conversation subject.
# 
# Options:
#     uri     Filter to show a specific users twts.
#     offset  Start index for quey.
#     limit   Count of items to return (going back in time).
# 
# twt range = 1 196295
# self = https://watcher.sour.is?offset=169825
# next = https://watcher.sour.is?offset=169925
# prev = https://watcher.sour.is?offset=169725
[47°09′54″S, 126°43′24″W] Transponder jammed
@bender I was in bed 🤣
@bender I was in bed 🤣
🧮 USERS:1 FEEDS:2 TWTS:1084 ARCHIVED:78231 CACHE:2434 FOLLOWERS:17 FOLLOWING:14
maybe i'm overly restrained when it comes to making changes in a codebase, but i do a lot of the work in my head before committing to code.
running grey until after dark, it'll be around 40C until tuesday in the daytime so I can't really run the main server while I'm sat in my office. i'll have some pine quartz blades or my quattro arrays up some day to handle the lower power stuff so running grey doesn't mean the whole system is offline.
On my blog: Real Life in Star Trek, The Outcast https://john.colagioia.net/blog/2024/09/05/outcast.html #scifi #startrek #closereading
One last thing before I hit the hay. This endpoint could respond with the raw twt, when asking for text/plain (it serves HTML at the moment). Return the physical line from the feed. Maybe with a comment above for the feed URL. Or doesn't the registry format also include the URL separated with a tab somehow? I'm too lazy right now to look it up. Also, not sure how useful that would be. Anyway, good night.
@lyse Thanks
@prologic Perfect, thanks. For my own future reference: curl -H 'Accept: application/json' https://twtxt.net/twt/st3wsda
[47°09′22″S, 126°43′39″W] Transponder still failing
@quark Right, a little rain improves™ the situation so much… :-( Surprise, surprise, our rain has been delayed again.

Good hunting and bon appétit! :-) I never had Puerto Rico's national dish, but the photos look delicious. Yum!

I also tried ice cream, but I reckon I simply stick to your last tip instead. :-)
@prologic come on, provide us with a one liner curl foo that does just that, don't be lazy! :-P
I reckon, this is as raw as you can get, @falsifian: curl -sH "Accept: application/json" https://twtxt.net/twt/st3wsda | jq You can piece it together from created and text (and twter.uri).
@bender Haha I aggressively unfollow feds that are like this now 🤣
@bender Haha I aggressively unfollow feds that are like this now 🤣
@bender So far I've been following feeds fairly liberally. I'll check to see if we have anything in common and lean toward following, just because this is new to me and it feels like a small community. But I'm still figuring out what I want. Later I'll probably either trim my follower list or come up with some way to prioritize the feeds I'm more interested in.
@falsifian Yes hit a Twt permalink URI and ask for application/ json
@falsifian Yes hit a Twt permalink URI and ask for application/ json
@prologic Specifically, I could view yarnd's copy here, but only as rendered for a human to view: https://twtxt.net/twt/st3wsda
@lyse Uggh 🥵 That sounds awful and reminds me of our very odd little 3-day heat wave we had last week 😱
@lyse Uggh 🥵 That sounds awful and reminds me of our very odd little 3-day heat wave we had last week 😱
@bender Thanks! 🙇‍♂️
@bender Thanks! 🙇‍♂️
@movq thanks for getting to the bottom of it. @prologic is there a way to view yarnd's copy of the raw twt? The edit didn't result in a visible change; being able to see what yarnd originally downloaded would have helped me debug.
I see 🤔 Thanks!
I see 🤔 Thanks!
@quark Oh shit, second place! :-O


$ du -h .config/twtxt/cache*.db 
13M   .config/twtxt/cache2.db  # contains read status for each twt (very inefficient format)
7,0M  .config/twtxt/cache.db   # the actual cache by the original twtxt reference implementation


Yeah, wrong place for caches.
@prologic I don't. I will pay close attention on any oddity I see next time. Sorry!
@movq This ☝️
@movq This ☝️
@bender Do you recall what you were clicking through? 🤦‍♂️
@bender Do you recall what you were clicking through? 🤦‍♂️
@bender I follow feeds that are somewhat interesting to me. At least for the most part.
@lyse 31°C here, feels like 33°C, with a lovely 75% of humidity. It has been raining, on and off (to make matter "better") the whole day until now. No horses here, but if you go outside you will smell the same smell of farm animals (like goats, or pigs). That's because two or three kilometres from here there are private farms, and when the wind blows in such way, well, we are reminded of their existence.

I haven't left the house, so it feels well under air conditioning. In two more hours I will call it quits from the work day, and will have to dash to the grocery to get supplies for tonight's meal (arroz con gandules). I will let you know how it truly feels out there then. :-D

For those swollen fingers, nothing better than a mildly cold shower! Oh, and paws off the keyboard! :-P
@lyse 31°C here, feels like 33°C, with a lovely 75% of humidity. It has been raining, on and off (to make matter "better") the whole day until now. No horses here, but if you go outside you will smell the same smell of farm animals (like goats, or pigs). That's because two or three kilometres from here there are private farms, and when the wind blows in such way, well, we are reminded of their existence.

I haven't left the house, so it feels well under air conditioning. In two more hours I will call it quits from the work day, and will have to dash to the grocery to get supplies for tonight's meal (arroz con gandules). I will let you know how it truly feels out there then. :-D

For those swollen fingers, nothing better than a mildly cold shower! Oh, and paws off the keyboard! :-P
@prologic Thanks for the honor, but I'm not sure why I specifically should be part of a deciding committee here. :-D I get it, I just fear people might misunderstand your message here.

I have to read up on the twtxt registry documentation on the weekend (too tired at the moment), but it should probably be no real issue to integrate that API into yarnd.
@abucci well, those are top ten "twtxtrs" (as in, how many twtxts they have produced). @prologic sure is a conversational fellow. :-D
@abucci well, those are top ten "twtxtrs" (as in, how many twtxts they have produced). @prologic sure is a conversational fellow. :-D
When we passed a few horses in the forest, there was really strong soup odor in the air. It didn't smell like horse at all, but soup. Maybe they've been soup horses, chickens were out of stock.

29°C, zero wind, extremely humid, luckily the sun was behind the clouds. I'm soaking wet, sweat ran down in streams and dripped in my eyes, it burned a bit. The sky is getting a little dark, I hope the thunderstorm and rain are really arriving here later. Rain had always been finally cancelled the couple last days.

I'm gotta go cool off my fingers now, they're swollen from the heat.
@rrraksamam a few. :-) And thanks!
[47°09′11″S, 126°43′18″W] Transponder malfunction
@rrraksamam which one? 😅
@rrraksamam ya, entiendo. Asi nos pasa a todos, creo. Vamos, que nos gusta, nos gusta! 🙂
Err, Garner, rather.
Somebody likes Jennifer Gardner... 😊
@movq ha! Here are my top 10:


24056 "prologic"
5103 "lyse"
3932 "movq"
1984 "abucci"
1876 "adi"
1633 "fastidious"
1551 "jlj"
1455 "mckinley"
1413 "offgridliving
1280 "eaplmx"


Some of those I no longer follow, or do not exist, but their wisdom remains. LOL.
@movq ha! Here are my top 10:


24056 "prologic"
5103 "lyse"
3932 "movq"
1984 "abucci"
1876 "adi"
1633 "fastidious"
1551 "jlj"
1455 "mckinley"
1413 "offgridliving
1280 "eaplmx"


Some of those I no longer follow, or do not exist, but their wisdom remains. LOL.
@bender The size … it depends. 🥴

$ du -sh ~/Mail/twt
244M /home/user/Mail/twt

But:

$ du -sh --apparent-size ~/Mail/twt
33M /home/user/Mail/twt

There are about 60k twts in there.

Regarding one-way junk: True. Looks like I mostly unfollowed those, I don’t really have that in my inbox. 🤔

These are the Top 10, btw:

$ awk '/^From: / { user\n++ } END { for (u in user) { print user\n, u } }' * | sort -k1rn | head -n 10
24020 "prologic"
5269 "lyse"
3928 "movq"
2285 "adi"
1985 "abucci"
1713 "mckinley"
1415 "off_grid_living"
1352 "darch"
1280 "eaplmx"
956 "bender"*
@bender The size … it depends. 🥴

$ du -sh ~/Mail/twt
244M /home/user/Mail/twt

But:

$ du -sh --apparent-size ~/Mail/twt
33M /home/user/Mail/twt

There are about 60k twts in there.

Regarding one-way junk: True. Looks like I mostly unfollowed those, I don’t really have that in my inbox. 🤔

These are the Top 10, btw:

$ awk '/^From: / { user[$2]++ } END { for (u in user) { print user[u], u } }' * | sort -k1rn | head -n 10
24020 "prologic"
5269 "lyse"
3928 "movq"
2285 "adi"
1985 "abucci"
1713 "mckinley"
1415 "off_grid_living"
1352 "darch"
1280 "eaplmx"
956 "bender"*
@bender The size … it depends. 🥴

$ du -sh ~/Mail/twt
244M /home/user/Mail/twt

But:

$ du -sh --apparent-size ~/Mail/twt
33M /home/user/Mail/twt

There are about 60k twts in there.

Regarding one-way junk: True. Looks like I mostly unfollowed those, I don’t really have that in my inbox. 🤔

These are the Top 10, btw:

$ awk '/^From: / { user[$2]++ } END { for (u in user) { print user[u], u } }' * | sort -k1rn | head -n 10
24020 "prologic"
5269 "lyse"
3928 "movq"
2285 "adi"
1985 "abucci"
1713 "mckinley"
1415 "off_grid_living"
1352 "darch"
1280 "eaplmx"
956 "bender"*
@bender The size … it depends. 🥴

$ du -sh ~/Mail/twt
244M /home/user/Mail/twt

But:

$ du -sh --apparent-size ~/Mail/twt
33M /home/user/Mail/twt

There are about 60k twts in there.

Regarding one-way junk: True. Looks like I mostly unfollowed those, I don’t really have that in my inbox. 🤔

These are the Top 10, btw:

$ awk '/^From: / { user[$2]++ } END { for (u in user) { print user[u], u } }' * | sort -k1rn | head -n 10
24020 "prologic"
5269 "lyse"
3928 "movq"
2285 "adi"
1985 "abucci"
1713 "mckinley"
1415 "off_grid_living"
1352 "darch"
1280 "eaplmx"
956 "bender"*
@bender The size … it depends. 🥴

$ du -sh ~/Mail/twt
244M /home/user/Mail/twt

But:

$ du -sh --apparent-size ~/Mail/twt
33M /home/user/Mail/twt

There are about 60k twts in there.

Regarding one-way junk: True. Looks like I mostly unfollowed those, I don’t really have that in my inbox. 🤔

These are the Top 10, btw:

$ awk '/^From: / { user[$2]++ } END { for (u in user) { print user[u], u } }' * | sort -k1rn | head -n 10
24020 "prologic"
5269 "lyse"
3928 "movq"
2285 "adi"
1985 "abucci"
1713 "mckinley"
1415 "off_grid_living"
1352 "darch"
1280 "eaplmx"
956 "bender"*
@movq good idea, considering it might occasionally not work at all (because of edited twtxts).
@movq good idea, considering it might occasionally not work at all (because of edited twtxts).
@dbucklin very nice, thank you for sharing! I like that kind of retailers too, so those are on my list now. 🙂
@dbucklin very nice, thank you for sharing! I like that kind of retailers too, so those are on my list now. 🙂
Oh, and about the "There’s so little going on in general, it hardly matters. 😅", the issue *I* have with it is that it is mostly one-way junk. Why to bother following that which never engages, or spews seemingly non-sense into the air (or both!)? 😅
@movq this is @quark's follow list:


anth http://a.9srv.net/tw.txt
bender https://twtxt.net/user/bender/twtxt.txt
johanbove https://johanbove.info/twtxt.txt
ionores https://twtxt.net/user/ionores/twtxt.txt
lyse https://lyse.isobeef.org/twtxt.txt
mckinley https://twtxt.net/user/mckinley/twtxt.txt
movq https://www.uninformativ.de/twtxt.txt
prologic https://twtxt.net/user/prologic/twtxt.txt
stigatle https://yarn.stigatle.no/user/stigatle/twtxt.txt
thecanine https://twtxt.net/user/thecanine/twtxt.txt
xuu https://txt.sour.is/user/xuu/twtxt.txt
abucci https://anthony.buc.ci/user/abucci/twtxt.txt
soren https://darch.dk/twtxt.txt
aelaraji https://aelaraji.com/twtxt.txt
dbucklin https://www.davebucklin.com/twtxt.txt
falsifian https://www.falsifian.org/twtxt.txt
off_grid https://twtxt.net/user/off_grid_living/twtxt.txt
yarn_police https://twtxt.net/user/yarn_police/twtxt.txt


And the Maildir size is 212MB. What size (not followers list) is yours?
@movq right, there is no way to forbid anyone from editing their text files. There is no hand holding here. :-D Thank you very much for shedding light into this "mystery"!
@bender On twtxt, I follow all feeds that I can find (there are *some* exceptions, of course). There’s so little going on in general, it hardly matters. 😅

And I just realized: Mutt’s layout helps a lot. Skimming over new twts is really easy and it’s not a big loss if there are a couple of shitposts™ in my “timeline”. This is very different from Mastodon (both the default web UI and all clients I’ve tried), where the timeline is always *huge*. Posts take up a lot of space on screen. Makes me think twice if I want to follow someone or not. 😅

(I mostly only follow Hashtags on Mastodon anyway. It’s more interesting that way.)
@bender On twtxt, I follow all feeds that I can find (there are *some* exceptions, of course). There’s so little going on in general, it hardly matters. 😅

And I just realized: Mutt’s layout helps a lot. Skimming over new twts is really easy and it’s not a big loss if there are a couple of shitposts™ in my “timeline”. This is very different from Mastodon (both the default web UI and all clients I’ve tried), where the timeline is always *huge*. Posts take up a lot of space on screen. Makes me think twice if I want to follow someone or not. 😅

(I mostly only follow Hashtags on Mastodon anyway. It’s more interesting that way.)
@bender On twtxt, I follow all feeds that I can find (there are *some* exceptions, of course). There’s so little going on in general, it hardly matters. 😅

And I just realized: Mutt’s layout helps a lot. Skimming over new twts is really easy and it’s not a big loss if there are a couple of shitposts™ in my “timeline”. This is very different from Mastodon (both the default web UI and all clients I’ve tried), where the timeline is always *huge*. Posts take up a lot of space on screen. Makes me think twice if I want to follow someone or not. 😅

(I mostly only follow Hashtags on Mastodon anyway. It’s more interesting that way.)
@bender On twtxt, I follow all feeds that I can find (there are *some* exceptions, of course). There’s so little going on in general, it hardly matters. 😅

And I just realized: Mutt’s layout helps a lot. Skimming over new twts is really easy and it’s not a big loss if there are a couple of shitposts™ in my “timeline”. This is very different from Mastodon (both the default web UI and all clients I’ve tried), where the timeline is always *huge*. Posts take up a lot of space on screen. Makes me think twice if I want to follow someone or not. 😅

(I mostly only follow Hashtags on Mastodon anyway. It’s more interesting that way.)
@falsifian @prologic @bender The twt was edited. In my cache, it also has hash st3wsda and it started like this:

(#yqke7sq) I've been sketching out some …

When fetching the feed *now*, the twt starts like this and the current twt gets the hash 6mdqxrq:

(#yqke7sq) I've been sketching out some …

This can’t be avoided, really. Publishing twts and then editing them is like doing a git push --force after rewriting the commit history. Chaos will ensue. 😅
@falsifian @prologic @bender The twt was edited. In my cache, it also has hash st3wsda and it started like this:

(#yqke7sq) I've been sketching out some …

When fetching the feed *now*, the twt starts like this and the current twt gets the hash 6mdqxrq:

(#yqke7sq) I've been sketching out some …

This can’t be avoided, really. Publishing twts and then editing them is like doing a git push --force after rewriting the commit history. Chaos will ensue. 😅
@falsifian @prologic @bender The twt was edited. In my cache, it also has hash st3wsda and it started like this:

(#yqke7sq) I've been sketching out some …

When fetching the feed *now*, the twt starts like this and the current twt gets the hash 6mdqxrq:

(#yqke7sq) I've been sketching out some …

This can’t be avoided, really. Publishing twts and then editing them is like doing a git push --force after rewriting the commit history. Chaos will ensue. 😅
@falsifian @prologic @bender The twt was edited. In my cache, it also has hash st3wsda and it started like this:

(#yqke7sq) I've been sketching out some …

When fetching the feed *now*, the twt starts like this and the current twt gets the hash 6mdqxrq:

(#yqke7sq) I've been sketching out some …

This can’t be avoided, really. Publishing twts and then editing them is like doing a git push --force after rewriting the commit history. Chaos will ensue. 😅
Weird, this just happened. It went away on page reload.

Screenshot of Yarn.social showing duplicated headers.
@prologic it hasn't been solved, that's why we are here on this yarn, still. LOL. I believe the hash mismatch is happening because of an edited twtxt. I don't follow the OP, so I have no way to check (not that I am certain it could be possible), but I have seeing similar issues in the past as a result of an edit. That's one of the reasons I don't edit anymore. 😬
@prologic I tried hosting my own Matrix server once, I got wrecked! 🤣 and I believe that experience established a good base line for me to avoid self-hosting anything federated (except for a TWTX feed).

Skill issues aside (since I'm willing to learn), my internet speed is a huge limitation (I have less than 1Mb Up) 🫠 Then again, there's the Running stuff off of my RPI's SD Card 😅 that's just asking for trouble.

Seriously, I should get a proper Job, at least I'll be able to afford my curiosities 🙃
@prologic I tried hosting my own Matrix server once, I got wrecked! 🤣 and I believe that experience established a good base line for me to avoid self-hosting anything federated (except for a TWTX feed).

Skill issues aside (since I'm willing to learn), my internet speed is a huge limitation (I have less than 1Mb Up) 🫠 Then again, there's the Running stuff off of my RPI's SD Card 😅 that's just asking for trouble.

Seriously, I should get a proper Job, at least I'll be able to afford my curiosities 🙃
@prologic I tried hosting my own Matrix server once, I got wrecked! 🤣 and I believe that experience established a good base line for me to avoid self-hosting anything federated (except for a TWTX feed).

Skill issues aside (since I'm willing to learn), my internet speed is a huge limitation (I have less than 1Mb Up) 🫠 Then again, there's the Running stuff off of my RPI's SD Card 😅 that's just asking for trouble.

Seriously, I should get a proper Job, at least I'll be able to afford my curiosities 🙃
[47°09′17″S, 126°43′32″W] --bad checksum--
@falsifian You are totally right. The specs are at least "open enough" for us to consider that as an implementation detail. We, and by we I mean @movq @lyse @bender @xuu and others should discuss this in more detail I believe and try to see if we can agree on what we're trying to solve.

> Does yarnd provide an API for finding twts? Is it similar?

No, it doesn't. But yarns (_the search engine/crawler wrote_) seems more fitting here. It's been discussed before, the possibility of building a "Twtxt Register v1" compatible API for yarns. I _think_ a search engine + crawler + registry (_especially ones that can form a bit of a "distributed network_) are far more useful I _think_ in order to support the _actual_ decentralised Twtxt / Yarn ecosystem (_which is how I prefer to describe it_).
@falsifian You are totally right. The specs are at least "open enough" for us to consider that as an implementation detail. We, and by we I mean @movq @lyse @bender @xuu and others should discuss this in more detail I believe and try to see if we can agree on what we're trying to solve.

> Does yarnd provide an API for finding twts? Is it similar?

No, it doesn't. But yarns (_the search engine/crawler wrote_) seems more fitting here. It's been discussed before, the possibility of building a "Twtxt Register v1" compatible API for yarns. I _think_ a search engine + crawler + registry (_especially ones that can form a bit of a "distributed network_) are far more useful I _think_ in order to support the _actual_ decentralised Twtxt / Yarn ecosystem (_which is how I prefer to describe it_).
@falsifian Ahh but this is solved now with the new single shot fetch?
@falsifian Ahh but this is solved now with the new single shot fetch?
[47°09′34″S, 126°43′43″W] Re-taking samples
https://www.mattlayman.com/blog/2024/layman-guide-python-built-in-functions/
[47°09′13″S, 126°43′10″W] Transfer aborted
The actual end-user problem is that I can't see the thread properly when using neomutt+jenny.
@prologic One of your twts begins with (#st3wsda): https://twtxt.net/twt/bot5z4q

Based on the twtxt.net web UI, it seems to be in reply to a twt by @cuaxolotl which begins "I’ve been sketching out...".

But jenny thinks the hash of that twt is 6mdqxrq. At least, there's a very twt in their feed with that hash that has the same text as appears on yarn.social (except with ' instead of ’).

Based on this, it appears jenny and yarnd disagree about the hash of the twt, or perhaps the twt was edited (though I can't see any difference, assuming ' vs ’ is just a rendering choice).
@prologic I believe you when you say registries as designed today do not crawl. But when I first read the spec, it conjured in my mind a search engine. Now I don't know how things work out in practice, but just based on reading, I don't see why it can't be an API for a crawling search engine. (In fact I don't see anything in the spec indicating registry servers shouldn't crawl.)

(I also noticed that https://twtxt.readthedocs.io/en/latest/user/registry.html recommends "The registries should sync each others user list by using the users endpoint". If I understood that right, registering with one should be enough to appear on others, even if they don't crawl.)

Does yarnd provide an API for finding twts? Is it similar?
@bender I've sort of lost the plot here a bit 🤦‍♂️ What's the problem we're trying to figure out? 🤔
@bender I've sort of lost the plot here a bit 🤦‍♂️ What's the problem we're trying to figure out? 🤔
@prologic why would you think it is changed in jenny? Falsifian (I still can’t mention while on mobile) said jenny, and the manual calculation match. Yarn seems to be the one at odd.
@falsifian You are however right that registries always had a "search" capability, amost others.
@falsifian You are however right that registries always had a "search" capability, amost others.
@movq Jenny hasn't changed the way it computes hashes has it? (yarnd certainly hasn't).
@movq Jenny hasn't changed the way it computes hashes has it? (yarnd certainly hasn't).
@falsifian I _think_ I'm missing something in my description. When I say "search engine" I also mean "with a crawler" that is able to self-discover feeds. A registry (_as designed today, or as the spec described_) required users to add their feeds to one or more registries, putting the burden on the user(s). I for example do not bother adding my feed to a registry (_which one would I add it to anyway?_)_
@falsifian I _think_ I'm missing something in my description. When I say "search engine" I also mean "with a crawler" that is able to self-discover feeds. A registry (_as designed today, or as the spec described_) required users to add their feeds to one or more registries, putting the burden on the user(s). I for example do not bother adding my feed to a registry (_which one would I add it to anyway?_)_
@prologic I guess I thought they were search engines. Anyway, the registry API looks like a decent one for searching for tweets. Could/should yarn.social pods implement the same API?
I just manually followed the steps at https://dev.twtxt.net/doc/twthashextension.html and got 6mdqxrq. I wonder what happened. Did @cuaxolo edit the twt in some subtle way after twtxt.net downloaded it? I couldn't spot a diff, other than ' appearing as ’ on yarn.social, which I assume is a transformation done by twtxt.net.
@falsifian to my knowledge registries were never designed to crawl the Twtxt space. If they did, they would be considered a search engine 🤣
@falsifian to my knowledge registries were never designed to crawl the Twtxt space. If they did, they would be considered a search engine 🤣
@prologic What's the difference between search.twtxt.net and the /api/plain/tweets endpoint of a registry? In my mind, a registry is a twtxt search engine. Or are registries not supposed to do their own crawling to discover new feeds?
@falsifian So yes, you would ask a pod about the missing Twt by hash, or whatever. Pods do this already, even though there aren't that many now, so it maybe a bit less effective today. However it's more of a small/tiny "distributed" protocol, you ask _any_ pod.

On registries however, I think a registry is the wrong approach. I see far greater value in feed crawlers and search engines like the (_half baked one_) I built over at https://search.twtxt.net/
@falsifian So yes, you would ask a pod about the missing Twt by hash, or whatever. Pods do this already, even though there aren't that many now, so it maybe a bit less effective today. However it's more of a small/tiny "distributed" protocol, you ask _any_ pod.

On registries however, I think a registry is the wrong approach. I see far greater value in feed crawlers and search engines like the (_half baked one_) I built over at https://search.twtxt.net/