# I am the Watcher. I am your guide through this vast new twtiverse.
# 
# Usage:
#     https://watcher.sour.is/api/plain/users              View list of users and latest twt date.
#     https://watcher.sour.is/api/plain/twt                View all twts.
#     https://watcher.sour.is/api/plain/mentions?uri=:uri  View all mentions for uri.
#     https://watcher.sour.is/api/plain/conv/:hash         View all twts for a conversation subject.
# 
# Options:
#     uri     Filter to show a specific users twts.
#     offset  Start index for quey.
#     limit   Count of items to return (going back in time).
# 
# twt range = 1 196277
# self = https://watcher.sour.is?offset=171999
# next = https://watcher.sour.is?offset=172099
# prev = https://watcher.sour.is?offset=171899
Been trying to get acquainted with rsync(1) but, whenever I Tab for completion and get this:

> λ ~/ rsync --
> zsh: do you wish to see all 484 possibilities (162 lines)?

I'm like: Nope! a scp -rpCq ... or whatever option salad will do just fine. 😅 [Insert: "Ain't nobody got time fo'that!" Meme.]~
Been trying to get acquainted with rsync(1) but, whenever I Tab for completion and get this:

> λ ~/ rsync --
> zsh: do you wish to see all 484 possibilities (162 lines)?

I'm like: Nope! a scp -rpCq ... or whatever option salad will do just fine. 😅 [Insert: "Ain't nobody got time fo'that!" Meme.]~
experimenting with litefs has been really interesting. i'm still learning about consul, so nothing distributed is happening yet. so far i have a setup that shares a virtual filesystem with a set of nixos containers running ejabberd and redka. soon some ory services for auth and security which also support sqlite will join the party, but those require higher availability that i can manage with my current deployment. the big server needs to me migrated before security can come online.
Syncthing is also as good as everyone says it is.
@movq Interesting, it's always good to know how things work under the hood. But I'm very glad, that I do not have to deal with this low-level stuff. :-)
[47°09′34″S, 126°43′02″W] Wind speed: N/A -- Cannot comunicate
@prologic violent enough to be taken away by the police. 🤭😂
@prologic @movq Luckily, we were only touched by the thunderstorm cell. Even though the sky lit up a bunch and the thunder roared, there were no close thunderbolts. But it rained cats and dogs. The air smelled lovely.
@eapl.me All the best, see you next life around. :-) On Twtxt I only meet my online friends. I'm staying in touch with some of my real life mates on IRC or e-mail. But that's fine. That's just how it goes.

Thanks, @bender. :-)
@aelaraji Hahaha, brilliant! :-D
I know what keeps me coming back to twtxt. It is the little group of people with whom I interact. I don’t need a big audience. More often than not I have nothing interesting to write, but I enjoy the small interactions: bugging prologic, reading abucci, browsing Lyse’s clicks. I enjoy movq commentaries (I imagine him as a professor of some kind, don’t ask me why).

Anyway… cheers!
#catsoftwtxt
/https://baldo.cat/media/photos/IMG_2115.jpeg) #catsoftwtxt
#catsoftwtxt
#catsoftwtxt
/https://baldo.cat/media/photos/IMG_2113.jpeg) #catsoftwtxt
#catsoftwtxt
@eapl.me are you sure X will bring joy, and value? Will you have clear conscience knowing you are contributing to such despicable platform? It is your decision to make, sure.

Joy starts at you, not the platform you use. When you get bored, disgusted, offended, and leave X, come and let us know. I will be interested to read all about your experiment then. For now, “¡hasta pronto!”
[47°09′01″S, 126°43′17″W] Weather forecast alert -- storm from NE
@movq Yes, the tools are surprisingly fast. Still, magrep takes about 20 seconds to search through my archive of 140K emails, so to speed things up I would probably combine it with an indexer like mu, mairix or notmuch.
@eapl.me Aww. Well, I gave you a Follow on Mastodon (although that appears to be moastly Spanish 🤔).
@eapl.me Aww. Well, I gave you a Follow on Mastodon (although that appears to be moastly Spanish 🤔).
@eapl.me Aww. Well, I gave you a Follow on Mastodon (although that appears to be moastly Spanish 🤔).
@eapl.me Aww. Well, I gave you a Follow on Mastodon (although that appears to be moastly Spanish 🤔).
@falsifian You had me there for a second. 😅

I have to admit, even though I knew they existed, I never had a look at Leah’s mail tools. Just gave mthread a spin and this is crazy fast. 🤯 Tempting!
@falsifian You had me there for a second. 😅

I have to admit, even though I knew they existed, I never had a look at Leah’s mail tools. Just gave mthread a spin and this is crazy fast. 🤯 Tempting!
@falsifian You had me there for a second. 😅

I have to admit, even though I knew they existed, I never had a look at Leah’s mail tools. Just gave mthread a spin and this is crazy fast. 🤯 Tempting!
@falsifian You had me there for a second. 😅

I have to admit, even though I knew they existed, I never had a look at Leah’s mail tools. Just gave mthread a spin and this is crazy fast. 🤯 Tempting!
@eapl.me Sad to see you go, disappointed in your choice of X, but respect your decision and choice. I will never cave in myself, even if it means my "circle of friends" remains low. I guess we call 'em internet friends right? 😅
@eapl.me Sad to see you go, disappointed in your choice of X, but respect your decision and choice. I will never cave in myself, even if it means my "circle of friends" remains low. I guess we call 'em internet friends right? 😅
#fzf is the new emacs: a tool with a simple purpose that has evolved to include an #email client. https://sr.ht/~rakoo/omail/

I'm being a little silly, of course. fzf doesn't actually check your email, but it appears to be basically the whole user interface for that mail program, with #mblaze wrangling the emails.

I've been thinking about how I handle my email, and am tempted to make something similar. (When I originally saw this linked the author was presenting it as an example tweaked to their own needs, encouraging people to make their own.)

This approach could surely also be combined with #jenny, taking the place of (neo)mutt. For example mblaze's mthread tool presents a threaded discussion with indentation.
@lyse Gut festhalten!
@lyse Gut festhalten!
@lyse Gut festhalten!
@lyse Gut festhalten!
Someone recommended a nice (German) talk:

https://media.ccc.de/v/ds24-394-linux-hello-world-nur-mit-einem-hex-editor

Luckily, everything™ is easier™ on DOS with .COM files. A fun little time killer to make a HELLO.COM using only a hex editor, the Intel docs and the DOS interrupt list.

That ModR/M stuff is easy in the end, but it took me quite some time to understand it. 🥴

(I’m still new to DOS on this level and didn’t know that all segment registers are initialized to the same values, apparently, so copying CS to DS was not necessary. Too lazy to update the screenshot. File size shrinks by 4 bytes.)

https://movq.de/v/0139fbaabc/doshello.png
Someone recommended a nice (German) talk:

https://media.ccc.de/v/ds24-394-linux-hello-world-nur-mit-einem-hex-editor

Luckily, everything™ is easier™ on DOS with .COM files. A fun little time killer to make a HELLO.COM using only a hex editor, the Intel docs and the DOS interrupt list.

That ModR/M stuff is easy in the end, but it took me quite some time to understand it. 🥴

(I’m still new to DOS on this level and didn’t know that all segment registers are initialized to the same values, apparently, so copying CS to DS was not necessary. Too lazy to update the screenshot. File size shrinks by 4 bytes.)

https://movq.de/v/0139fbaabc/doshello.png
Someone recommended a nice (German) talk:

https://media.ccc.de/v/ds24-394-linux-hello-world-nur-mit-einem-hex-editor

Luckily, everything™ is easier™ on DOS with .COM files. A fun little time killer to make a HELLO.COM using only a hex editor, the Intel docs and the DOS interrupt list.

That ModR/M stuff is easy in the end, but it took me quite some time to understand it. 🥴

(I’m still new to DOS on this level and didn’t know that all segment registers are initialized to the same values, apparently, so copying CS to DS was not necessary. Too lazy to update the screenshot. File size shrinks by 4 bytes.)

https://movq.de/v/0139fbaabc/doshello.png
Someone recommended a nice (German) talk:

https://media.ccc.de/v/ds24-394-linux-hello-world-nur-mit-einem-hex-editor

Luckily, everything™ is easier™ on DOS with .COM files. A fun little time killer to make a HELLO.COM using only a hex editor, the Intel docs and the DOS interrupt list.

That ModR/M stuff is easy in the end, but it took me quite some time to understand it. 🥴

(I’m still new to DOS on this level and didn’t know that all segment registers are initialized to the same values, apparently, so copying CS to DS was not necessary. Too lazy to update the screenshot. File size shrinks by 4 bytes.)

https://movq.de/v/0139fbaabc/doshello.png
@lyse How violent is the thunderstorm? 🤔
@lyse How violent is the thunderstorm? 🤔
We're now having a thunderstorm with rain, lightning and thunder and the severe weather map shows all green. I'd expect it to be violet.
@aelaraji LOl 😂
@aelaraji LOl 😂
Okay, I figured out the cause of the broken output. I also replaced the first subject = '' for the existing conversation roots with subject > ''. Somehow, my brain must have read subject <> ''. That equality check should not have been touched at all. I just updated the updated archive for anyone who is interested to follow along: https://lyse.isobeef.org/tmp/tt2cache.tar.bz2 (151.1 KiB)
LMAO 🤣 ... I've been scrolling through mutt(1) man page and found this:

> BUGS
> None. Mutts have fleas, not bugs.
LMAO 🤣 ... I've been scrolling through mutt(1) man page and found this:

> BUGS
> None. Mutts have fleas, not bugs.
LMAO 🤣 ... I've been scrolling through mutt(1) man page and found this:

> BUGS
> None. Mutts have fleas, not bugs.
A new thing LLM(s) can't do well. Write patches 🤣
A new thing LLM(s) can't do well. Write patches 🤣
@lyse Yeah I _think_ it's one of the reasons why yarnd's cache became so complicated really. I mean it's a bunch of maps and lists that is recalculated every ~5m. I don't know of any better way to do this right now, but maybe one day I'll figure out a better way to represent the same information that is displayed today that works reasonably well.~
@lyse Yeah I _think_ it's one of the reasons why yarnd's cache became so complicated really. I mean it's a bunch of maps and lists that is recalculated every ~5m. I don't know of any better way to do this right now, but maybe one day I'll figure out a better way to represent the same information that is displayed today that works reasonably well.~
@prologic Yeah, relational databases are definitely not the perfect fit for trees, but I want to give it a shot anyway. :-)

Using EXPLAIN QUERY PLAN I was able to create two indices, to avoid some table scans:

CREATE INDEX parent ON messages (hash, subject);
CREATE INDEX subject_created_at ON messages (subject, created_at);

Also, since strings are sortable, instead of str_col <> '' I now use str_col > '' to allow the use of an index.

But somehow, my output seems to be broken at the end for some reason, I just noticed. :-? Hmm.

The read status still gives me headache. I think I either have to filter in the application or create more meta data structures in the database.

I'm wondering if anyone here already used certain storages for tree data.
My point is, this is not a small trade-off to make for the sake of simplicity 😅
My point is, this is not a small trade-off to make for the sake of simplicity 😅
@movq Maybe I misspoke. It's a factor of 5 in the size of the keyspace required. The impact is significantly less for on-disk storage of raw feeds and such, around ~1-1.5x depending on how many replies there are I suppose.

I wasn't very clear; my apologies. If we update the current hash truncation length from 7 to 11. But then still decide anyway to go down this location-based twt identity and threading model then yes, we're talking about twt subjects having a ~5x increase in size on average. Going from 14 characters (11 for the has, 2 for the parens, 1 for the #) to ~63 bytes (average I've worked out of length of URL + Timestamp) + 3 byte overhead for parents and space.~
@movq Maybe I misspoke. It's a factor of 5 in the size of the keyspace required. The impact is significantly less for on-disk storage of raw feeds and such, around ~1-1.5x depending on how many replies there are I suppose.

I wasn't very clear; my apologies. If we update the current hash truncation length from 7 to 11. But then still decide anyway to go down this location-based twt identity and threading model then yes, we're talking about twt subjects having a ~5x increase in size on average. Going from 14 characters (11 for the has, 2 for the parens, 1 for the #) to ~63 bytes (average I've worked out of length of URL + Timestamp) + 3 byte overhead for parents and space.~
@prologic A factor of 5 is hard to believe, to be honest. Especially disk usage. I know nothing about the internals of yarnd, but still.

If this constitutes a hard “no” to the proposal, then I think we don’t need to discuss it further.
@prologic A factor of 5 is hard to believe, to be honest. Especially disk usage. I know nothing about the internals of yarnd, but still.

If this constitutes a hard “no” to the proposal, then I think we don’t need to discuss it further.
@prologic A factor of 5 is hard to believe, to be honest. Especially disk usage. I know nothing about the internals of yarnd, but still.

If this constitutes a hard “no” to the proposal, then I think we don’t need to discuss it further.
@prologic A factor of 5 is hard to believe, to be honest. Especially disk usage. I know nothing about the internals of yarnd, but still.

If this constitutes a hard “no” to the proposal, then I think we don’t need to discuss it further.
@lyse Yes I think so.
@lyse Yes I think so.
@prologic I see. I reckon, it makes to combine 1 and 2, because if we change the hashing anyway, we don't break it twice.
Don't forget about the upcoming Yarn.social meetup coming up this Saturday! See #jjbnvgq for details! Hope to see some/all of y'all there 💪
Don't forget about the upcoming Yarn.social meetup coming up this Saturday! See # for details! Hope to see some/all of y'all there 💪
Don't forget about the upcoming Yarn.social meetup coming up this Saturday! See #jjbnvgq for details! Hope to see some/all of y'all there 💪
@lyse And your query to construct a tree? Can you share the full query (_screenshot looks scary 🤣_) -- On another note, SQL and relational databases aren't really that conduces to tree-like structures are they? 🤣_
@lyse And your query to construct a tree? Can you share the full query (_screenshot looks scary 🤣_) -- On another note, SQL and relational databases aren't really that conduces to tree-like structures are they? 🤣_
This organigram example got me started: https://www.sqlite.org/lang_with.html#controlling_depth_first_versus_breadth_first_search_of_a_tree_using_order_by

But I feel execution times get worse rather quickly with more data I add. Also, caching helps tremendously, executing it for the first time took over 600ms. From then on I'm down to 40ms.

I think, it's particularly bad that parents might be missing. Thus, I cannot use an index, because there is no parent to reference. But my database knowledge is fairly limited, so I have to read up on that.
In fact it depends on how many Twts there are that form part of a thread, if you take a much larger sample size of my own feed for example, it starts to approximate ~1.5x increase in size:


$ ./compare.sh https://twtxt.net/user/prologic/twtxt.txt 500
Original file size: 126842 bytes
Modified file size: 317029 bytes
Percentage increase in file size: 149.94%
...
~
In fact it depends on how many Twts there are that form part of a thread, if you take a much larger sample size of my own feed for example, it starts to approximate ~1.5x increase in size:


$ ./compare.sh https://twtxt.net/user/prologic/twtxt.txt 500
Original file size: 126842 bytes
Modified file size: 317029 bytes
Percentage increase in file size: 149.94%
...
~
In fact @falsifian you had quite a lot of good feedback, do you mind collecting them in a task list on the doc somewhere so I can get to em? 🤔
In fact @falsifian you had quite a lot of good feedback, do you mind collecting them in a task list on the doc somewhere so I can get to em? 🤔
Can someone make the edit?
Can someone make the edit?
[47°09′54″S, 126°43′08″W] Transfer 25% complete...
There you go, @prologic, the SQLite database (with a bit more data now) and the sqlitebrowser project file containing the query: https://lyse.isobeef.org/tmp/tt2cache.tar.bz2 (133.9 KiB)
@movq Tbis was just a representative sample. The real concrete cost here is a ~5x increase in memory consumption for yarnd and/or ~5x increase in disk storage.
@movq Tbis was just a representative sample. The real concrete cost here is a ~5x increase in memory consumption for yarnd and/or ~5x increase in disk storage.
@lyse Mind sharing your schema?
@lyse Mind sharing your schema?
@lyse Not sure I'll check
@lyse Not sure I'll check
@lyse My proposal is three steps:

- increase the hash length from 7 to 11

Then:

- Add support for changing your feed's location without breaking g threads

Then much later:

- Add formal support for edits
@lyse My proposal is three steps:

- increase the hash length from 7 to 11

Then:

- Add support for changing your feed's location without breaking g threads

Then much later:

- Add formal support for edits
@lyse No I don't either just say'n 😅
@lyse No I don't either just say'n 😅
@falsifian I agreee. It's an optional header.
@movq That's what I want to know 🤣
@movq That's what I want to know 🤣
@prologic What’s that in absolute numbers? My ~/Mail/twt is currently 26 MB in size. Increase that by 20% and we get 31.2 MB.

I don’t buy the argument with 2025 bytes. This worst case scenario is not relevant in practice.
@prologic What’s that in absolute numbers? My ~/Mail/twt is currently 26 MB in size. Increase that by 20% and we get 31.2 MB.

I don’t buy the argument with 2025 bytes. This worst case scenario is not relevant in practice.
@prologic What’s that in absolute numbers? My ~/Mail/twt is currently 26 MB in size. Increase that by 20% and we get 31.2 MB.

I don’t buy the argument with 2025 bytes. This worst case scenario is not relevant in practice.
@prologic What’s that in absolute numbers? My ~/Mail/twt is currently 26 MB in size. Increase that by 20% and we get 31.2 MB.

I don’t buy the argument with 2025 bytes. This worst case scenario is not relevant in practice.
@movq Oha! @bender Happy cooling off!
@prologic Well, mentions are also quite lengthy as they always include the feed URL. I know, that's not a good argument.

I just got a very, very wild idea that I have not put any brain power into, so it might be totally stupid: Since many replies also mention the original feed, maybe a mention and thread identifier could be compbined, something like: @<nick url timestamp>. But then we would also need another style if one does not want to mention the original author.

So, scratch that. But I put it out there anyway. Maybe this inspires someone else to come up with something neat.
It’s a different story when you just publish a twtxt file, I think. The question here is: When you publish a twt and don’t like it anymore and want to delete it, do you have the *right* to *force* others to delete it? (Not in a technical manner, but by sueing them.) What does the GDPR have to say about that? Not a clue. 😂
It’s a different story when you just publish a twtxt file, I think. The question here is: When you publish a twt and don’t like it anymore and want to delete it, do you have the *right* to *force* others to delete it? (Not in a technical manner, but by sueing them.) What does the GDPR have to say about that? Not a clue. 😂
It’s a different story when you just publish a twtxt file, I think. The question here is: When you publish a twt and don’t like it anymore and want to delete it, do you have the *right* to *force* others to delete it? (Not in a technical manner, but by sueing them.) What does the GDPR have to say about that? Not a clue. 😂