# I am the Watcher. I am your guide through this vast new twtiverse.
# 
# Usage:
#     https://watcher.sour.is/api/plain/users              View list of users and latest twt date.
#     https://watcher.sour.is/api/plain/twt                View all twts.
#     https://watcher.sour.is/api/plain/mentions?uri=:uri  View all mentions for uri.
#     https://watcher.sour.is/api/plain/conv/:hash         View all twts for a conversation subject.
# 
# Options:
#     uri     Filter to show a specific users twts.
#     offset  Start index for quey.
#     limit   Count of items to return (going back in time).
# 
# twt range = 1 196312
# self = https://watcher.sour.is?offset=171399
# next = https://watcher.sour.is?offset=171499
# prev = https://watcher.sour.is?offset=171299
Do you k ow what you clicked on before going back?
@eldersnake Sweet thank you! 🙇‍♂️ I'll merge this PR tonight I think.
@eldersnake Sweet thank you! 🙇‍♂️ I'll merge this PR tonight I think.
Seems to be working OK 🤔
🧮 USERS:1 FEEDS:2 TWTS:1098 ARCHIVED:79066 CACHE:2530 FOLLOWERS:17 FOLLOWING:14
On my blog: Real Life in Star Trek, The First Duty https://john.colagioia.net/blog/2024/09/19/first-duty.html #scifi #startrek #closereading
@david Well, I wouldn't recommend using my code for your main jenny use anyway. If you want to try it out, set XDG_CONFIG_HOME and XDG_CACHE_HOME to some sandbox directories and only run my code there. If @movq is interested in any of this getting upstreamed, I'd be happy to try rebasing the changes, but otherwise it's a proof of concept and fun exercise.
@falsifian hiya from the Sunshine State (also known as a never-ending hell, LOL)!
@falsifian hiya from the Sunshine State (also known as a never-ending hell, LOL)!
I forgot to git add a new test file. Added to the patch now at https://www.falsifian.org/a/oDtr/patch0.txt
@falsifian this one hits hard, as jenny was just updated today. :'-(
@falsifian this one hits hard, as jenny was just updated today. :'-(


its replacing the contents of body for some reason.


its replacing the contents of body for some reason.
@david Hello!
@prologic Hi. i have noticed sometimes when i hit the back button i lose all the surrounding layout and just have a list of twts.

@prologic Hi. i have noticed sometimes when i hit the back button i lose all the surrounding layout and just have a list of twts.

BTW this code doesn't incorporate existing twts into jenny's database. It's best used starting from scratch. I've been testing it using a custom XDG_CACHE_HOME and XDG_CONFIG_HOME to avoid messing with my "real" jenny data.
I wrote some code to try out non-hash reply subjects formatted as (replyto ), while keeping the ability to use the existing hash style.

I don't think we need to decide all at once. If clients add support for a new method then people can use it if they like. The downside of course is that this costs developer time, so I decided to invest a few hours of my own time into a proof of concept.

With apologies to @movq for corrupting jenny's beautiful code. I don't write this expecting you to incorporate the patch, because it does complicate things and might not be a direction you want to go in. But if you like any part of this approach feel free to use bits of it; I release the patch under jenny's current LICENCE.

Supporting both kinds of reply in jenny was complicated because each email can only have one Message-Id, and because it's possible the target twt will not be seen until after the twt referencing it. The following patch uses an sqlite database to keep track of known (url, timestamp) pairs, as well as a separate table of (url, timestamp) pairs that haven't been seen yet but are wanted. When one of those "wanted" twts is finally seen, the mail file gets rewritten to include the appropriate In-Reply-To header.

Patch based on jenny commit 73a5ea81.

https://www.falsifian.org/a/oDtr/patch0.txt

Not implemented:
- Composing twts using the (replyto ...) format.
- Probably other important things I'm forgetting.
@yarn_police I am noticing crimes around here, but I am too afraid to say anything (points at Lyse with mouth movements, no muttering a single word). Help us, officer!
Hahahahahaahaaaahaaaaaa, brilliant! I love it, @bender! :'-D
Hmm…
@movq Thanks for the summary!

So, what would happen if there is no original message anymore in the feed and you encounter an "edit" subject? Since you cannot verify that the feed contained it in the first place, would you obey it?

Some feed could just make a client update something from a different feed. In the cache, the client would need to store in a flag that this message was updated, so that when it later encounters the message from the real feed, it has a chance of reverting that bogus edit. Hmm. The devil is in the detail.

It's much easier with a delete subject. When it finds the message in its cache and the feeds match, remove it. Otherwise, just ignore it.
@lyse matter of fact, earlier you posted:


2024-09-19T20:20:00+02:00\tI don't like Australians!


And then deleted it, fearing the Australian Mafia (which, as we know, is very powerful in Bavaria). But I got the hash for it, p5zdahq, and that timestamp has tt written all over it. That's my proof! 😅😅😅
@lyse matter of fact, earlier you posted:


2024-09-19T20:20:00+02:00	I don't like Australians!


And then deleted it, fearing the Australian Mafia (which, as we know, is very powerful in Bavaria). But I got the hash for it, p5zdahq, and that timestamp has tt written all over it. That's my proof! 😅😅😅
@movq Right. That's why, I'd bite the bullet and go for huge URLs. :-)

I havent't looked at the code and I'm too lazy right now, does jenny also verify the fetched result against the hash?
@movq Yeah, but hashing also uses the main feed URL or whatever is written in the feed's first url metadata field. So, it's not a new problem, it's exactly the same.
@lyse indeed! There is no "central authority" acting as witness, and notary. The more I think of it... LOL.
@lyse indeed! There is no "central authority" acting as witness, and notary. The more I think of it... LOL.
@movq @david Yeah, he got a bit older but I could still easily recognize him.
@david Such a funny picture – we’ve been to Florida once some ~30 years ago and it looked almost *exactly* like that. 😅~
@david Such a funny picture – we’ve been to Florida once some ~30 years ago and it looked almost *exactly* like that. 😅~
@david Such a funny picture – we’ve been to Florida once some ~30 years ago and it looked almost *exactly* like that. 😅~
@david Such a funny picture – we’ve been to Florida once some ~30 years ago and it looked almost *exactly* like that. 😅~
@david Yeah, but it happened so fast with him. 😅 I remember watching some of his talks 1-3 years ago, looked completely different, I think. 😅

Luckily I can still recognize the voice, so I know it’s him, lol.
@david Yeah, but it happened so fast with him. 😅 I remember watching some of his talks 1-3 years ago, looked completely different, I think. 😅

Luckily I can still recognize the voice, so I know it’s him, lol.
@david Yeah, but it happened so fast with him. 😅 I remember watching some of his talks 1-3 years ago, looked completely different, I think. 😅

Luckily I can still recognize the voice, so I know it’s him, lol.
@david Yeah, but it happened so fast with him. 😅 I remember watching some of his talks 1-3 years ago, looked completely different, I think. 😅

Luckily I can still recognize the voice, so I know it’s him, lol.
@lyse The hash/thread-id would be shorter, but you’d lose two other benefits of (replyto:…):

1. You need a special client again to compute hashes.
2. The original feed URL is no longer visible, thus you might need to ask a Yarn pod occasionally for missing twts (I do that surprisingly often, now that I’ve implemented it) – but now you’ve lost the guarantee that Yarn gives you the correct information, because you can no longer verify it.
@lyse The hash/thread-id would be shorter, but you’d lose two other benefits of (replyto:…):

1. You need a special client again to compute hashes.
2. The original feed URL is no longer visible, thus you might need to ask a Yarn pod occasionally for missing twts (I do that surprisingly often, now that I’ve implemented it) – but now you’ve lost the guarantee that Yarn gives you the correct information, because you can no longer verify it.
@lyse The hash/thread-id would be shorter, but you’d lose two other benefits of (replyto:…):

1. You need a special client again to compute hashes.
2. The original feed URL is no longer visible, thus you might need to ask a Yarn pod occasionally for missing twts (I do that surprisingly often, now that I’ve implemented it) – but now you’ve lost the guarantee that Yarn gives you the correct information, because you can no longer verify it.
@lyse The hash/thread-id would be shorter, but you’d lose two other benefits of (replyto:…):

1. You need a special client again to compute hashes.
2. The original feed URL is no longer visible, thus you might need to ask a Yarn pod occasionally for missing twts (I do that surprisingly often, now that I’ve implemented it) – but now you’ve lost the guarantee that Yarn gives you the correct information, because you can no longer verify it.
Another thing: At the moment, anyone could claim that some feed contained a certain message which was then removed again by just creating the hash over the fake message in said feed and invented timestamp themselves. Nobody can ever verify that this was never the case in the first place and completely made up. So, our twt hashes have to be taken with a grain of salt.
@lyse Right, feed rotation gets ugly. We’d have (replyto:example.com/tw.txt,$timestamp) but maybe that feed doesn’t actually contain that stamp, so you have to got further back … but you should NOT reference an archived feed in your (replyto:…) thingy, it should still be the “main feed URL” (because the contents of archived feeds aren’t stable, see @prologic’s feeds for example). That’s not too great.

Man, I’m completely torn on this. I’d almost prefer not to decide anything. 😂
@lyse Right, feed rotation gets ugly. We’d have (replyto:example.com/tw.txt,$timestamp) but maybe that feed doesn’t actually contain that stamp, so you have to got further back … but you should NOT reference an archived feed in your (replyto:…) thingy, it should still be the “main feed URL” (because the contents of archived feeds aren’t stable, see @prologic’s feeds for example). That’s not too great.

Man, I’m completely torn on this. I’d almost prefer not to decide anything. 😂
@lyse Right, feed rotation gets ugly. We’d have (replyto:example.com/tw.txt,$timestamp) but maybe that feed doesn’t actually contain that stamp, so you have to got further back … but you should NOT reference an archived feed in your (replyto:…) thingy, it should still be the “main feed URL” (because the contents of archived feeds aren’t stable, see @prologic’s feeds for example). That’s not too great.

Man, I’m completely torn on this. I’d almost prefer not to decide anything. 😂
@lyse Right, feed rotation gets ugly. We’d have (replyto:example.com/tw.txt,$timestamp) but maybe that feed doesn’t actually contain that stamp, so you have to got further back … but you should NOT reference an archived feed in your (replyto:…) thingy, it should still be the “main feed URL” (because the contents of archived feeds aren’t stable, see @prologic’s feeds for example). That’s not too great.

Man, I’m completely torn on this. I’d almost prefer not to decide anything. 😂
… then, of course, I wouldn’t *need* to ask a Yarn pod for a certain twt if we used (replyto:…) instead of (#123467), because the original source of the twt is no longer obscured by a hash value and I can just pull the original feed. Asking a Yarn pod is only interesting at the moment because I have no idea where to get (#123467) from.

Only when the original feed has gone offline will querying a Yarn pod become relevant again.

I have to admit here that some of the goals/philosophy of Yarn simply don’t apply to my use cases. 😅 I don’t run a daemon that speaks a gossipping protocol with neighboring pods or stuff like that. I think I don’t have a hard time accepting that feeds might go offline in two months, so be it. Digging up ancient twts from some sort of globally distributed file system isn’t one of my goals. It’s a completely different thing for me. Hmmm. 🤔
… then, of course, I wouldn’t *need* to ask a Yarn pod for a certain twt if we used (replyto:…) instead of (#123467), because the original source of the twt is no longer obscured by a hash value and I can just pull the original feed. Asking a Yarn pod is only interesting at the moment because I have no idea where to get (#123467) from.

Only when the original feed has gone offline will querying a Yarn pod become relevant again.

I have to admit here that some of the goals/philosophy of Yarn simply don’t apply to my use cases. 😅 I don’t run a daemon that speaks a gossipping protocol with neighboring pods or stuff like that. I think I don’t have a hard time accepting that feeds might go offline in two months, so be it. Digging up ancient twts from some sort of globally distributed file system isn’t one of my goals. It’s a completely different thing for me. Hmmm. 🤔
… then, of course, I wouldn’t *need* to ask a Yarn pod for a certain twt if we used (replyto:…) instead of (#123467), because the original source of the twt is no longer obscured by a hash value and I can just pull the original feed. Asking a Yarn pod is only interesting at the moment because I have no idea where to get (#123467) from.

Only when the original feed has gone offline will querying a Yarn pod become relevant again.

I have to admit here that some of the goals/philosophy of Yarn simply don’t apply to my use cases. 😅 I don’t run a daemon that speaks a gossipping protocol with neighboring pods or stuff like that. I think I don’t have a hard time accepting that feeds might go offline in two months, so be it. Digging up ancient twts from some sort of globally distributed file system isn’t one of my goals. It’s a completely different thing for me. Hmmm. 🤔
… then, of course, I wouldn’t *need* to ask a Yarn pod for a certain twt if we used (replyto:…) instead of (#123467), because the original source of the twt is no longer obscured by a hash value and I can just pull the original feed. Asking a Yarn pod is only interesting at the moment because I have no idea where to get (#123467) from.

Only when the original feed has gone offline will querying a Yarn pod become relevant again.

I have to admit here that some of the goals/philosophy of Yarn simply don’t apply to my use cases. 😅 I don’t run a daemon that speaks a gossipping protocol with neighboring pods or stuff like that. I think I don’t have a hard time accepting that feeds might go offline in two months, so be it. Digging up ancient twts from some sort of globally distributed file system isn’t one of my goals. It’s a completely different thing for me. Hmmm. 🤔
[47°09′20″S, 126°43′29″W] Reading: 1.71000 PPM
@david Cool idea actually! The hash would also be shorter than the raw URL and timestamp.
@prologic I get where you're coming from. But is it really that bad in practice? If you follow any link somewhere in the web, you also don't know if its contents has been changed in the meantime. Is that a problem? Almost never in my experience.

Granted, it's a nice property when one can tell that it was not messed with since the author referenced it.
@movq The more I think about it, the more do I like the location-based addressing. That feels fairly in line with the spirit of twtxt, just like you stated somewhere else.

The big downside for me is that the subjects then become super long.

And if the feed relocates, we end up with broken conversation trees again. Just like nowadays. At least it's not getting worse. :-)

Using the feed URL in there might become a little challenging for new folks, when the twt rotates away into archive feeds. But I reckon, we already have a similar situation with the hashes. So, probably not too bad.
@quark Yeah, let's see what they reveal!
@movq I recognise him, but yes, he has aged quite a bit. I mean, I look at myself in the mirror and can't, often, recognise myself. Ageing is a bitch! 😅
@movq I recognise him, but yes, he has aged quite a bit. I mean, I look at myself in the mirror and can't, often, recognise myself. Ageing is a bitch! 😅
Nice, @david! The winter palms look nice. And the sky is full of snow.
I’m bad with faces, I know that. But I’m having a *really* hard time recognizing Linus in this video:

https://www.youtube.com/watch?v=4WCTGycBceg

Basically a different person to me. Is it just me or has he really changed that much? 😳
I’m bad with faces, I know that. But I’m having a *really* hard time recognizing Linus in this video:

https://www.youtube.com/watch?v=4WCTGycBceg

Basically a different person to me. Is it just me or has he really changed that much? 😳
I’m bad with faces, I know that. But I’m having a *really* hard time recognizing Linus in this video:

https://www.youtube.com/watch?v=4WCTGycBceg

Basically a different person to me. Is it just me or has he really changed that much? 😳
I’m bad with faces, I know that. But I’m having a *really* hard time recognizing Linus in this video:

https://www.youtube.com/watch?v=4WCTGycBceg

Basically a different person to me. Is it just me or has he really changed that much? 😳
Yesterday, both temperature and wind picked up. There was even wind in the night, which is rare over here. Today, we also got a lot of sunshine, around 22°C and heaps of wind. The leaves and twigs were blown at the house door, it reminded me of a snow drift, basically a leave bank. I should have taken a photo before I swept it, it looked quite bizarre.

But I photographed something else instead:

Possibly a large roof panel on a crane

My mate and I went out in the woods earlier and we came across 08 which broke off in roughly 6, 7 meters from 09. When it hit the ground, it made a 30 cm deep hole. Quite impressive. https://lyse.isobeef.org/waldspaziergang-2024-09-19/
@david Glad you like it. 😅
@david Glad you like it. 😅
@david Glad you like it. 😅
@david Glad you like it. 😅
I mean, really, it couldn't get any better. I love it!

Screenshot of Neomutt with twtxts populated by jenny.
I mean, really, it couldn't get any better. I love it!

Screenshot of Neomutt with twtxts populated by jenny.
@movq perfect in every way. Configurable too! Thank you!
@movq perfect in every way. Configurable too! Thank you!
@david Aye, I’ve pushed some commits. (And this is *really* going to be the last non-trivial change. 😂)
@david Aye, I’ve pushed some commits. (And this is *really* going to be the last non-trivial change. 😂)
@david Aye, I’ve pushed some commits. (And this is *really* going to be the last non-trivial change. 😂)
@david Aye, I’ve pushed some commits. (And this is *really* going to be the last non-trivial change. 😂)
@movq yes, that's perfect! <3
@movq yes, that's perfect! <3
@david Like that, right? https://movq.de/v/80f888d381/s.png
@david Like that, right? https://movq.de/v/80f888d381/s.png
@david Like that, right? https://movq.de/v/80f888d381/s.png
@david Like that, right? https://movq.de/v/80f888d381/s.png
[47°09′21″S, 126°43′56″W] Raw reading: 0x66EC4A81, offset +/-5
@eldersnake I wanted to ask you, are you running Headscale *and* WireGuard on the same VPS? I want to test Headscale, but currently run a small container with WireGuard, and I wonder if I need to stop (and eventually get rid of) the container to get Headscale going. Did you use the provided .deb to install Headscale, or some other method?
@eldersnake I wanted to ask you, are you running Headscale *and* WireGuard on the same VPS? I want to test Headscale, but currently run a small container with WireGuard, and I wonder if I need to stop (and eventually get rid of) the container to get Headscale going. Did you use the provided .deb to install Headscale, or some other method?
Okay, the recently implemented --fetch-context, which asks a Yarn pod for a twt, wouldn’t *break*, but jenny would not be able anymore to verify that it actually got the correct twt. That’s a concrete example where we would lose functionality.
Okay, the recently implemented --fetch-context, which asks a Yarn pod for a twt, wouldn’t *break*, but jenny would not be able anymore to verify that it actually got the correct twt. That’s a concrete example where we would lose functionality.
Okay, the recently implemented --fetch-context, which asks a Yarn pod for a twt, wouldn’t *break*, but jenny would not be able anymore to verify that it actually got the correct twt. That’s a concrete example where we would lose functionality.
Okay, the recently implemented --fetch-context, which asks a Yarn pod for a twt, wouldn’t *break*, but jenny would not be able anymore to verify that it actually got the correct twt. That’s a concrete example where we would lose functionality.
@david Yeah, I was annoyed by this myself lately. twts have become *so long* nowadays, it really gets in the way.
@david Yeah, I was annoyed by this myself lately. twts have become *so long* nowadays, it really gets in the way.
@david Yeah, I was annoyed by this myself lately. twts have become *so long* nowadays, it really gets in the way.
@david Yeah, I was annoyed by this myself lately. twts have become *so long* nowadays, it really gets in the way.
@prologic Can you come up with actual scenarios where it would break? Or is it more of a gut feeling?

The thing that keeps bugging me is this:

If we were to switch to location-based addressing and (replyto:…), the edit problem would resolve itself. Implementations could use that exact string (e.g., https://example.com/tw.txt,2024-09-18T12:45Z) as the internal identifier of a twt and that is pretty much the only change that you have to make. And then you could throw away all code and tests currently required for calculating hashes. (In jenny, I would also be able to and actually have to remove that code that skips over twts with a timestamp older than $last_fetch. This only got added as a workaround “to avoid broken threads all the time”.) The net result would be *less code*.

Implementing this whole (edit:#hash) thing means *more code*. (For jenny, specifically, *a lot* more code, if I want to allow users to create such twts.)

Do you see why I’m so reluctant to jump on this bandwagon? 😅

I haven’t come up yet with good, concrete examples where (replyto:…) would break. As soon as that happens, I’ll change my mind. 🤔
@prologic Can you come up with actual scenarios where it would break? Or is it more of a gut feeling?

The thing that keeps bugging me is this:

If we were to switch to location-based addressing and (replyto:…), the edit problem would resolve itself. Implementations could use that exact string (e.g., https://example.com/tw.txt,2024-09-18T12:45Z) as the internal identifier of a twt and that is pretty much the only change that you have to make. And then you could throw away all code and tests currently required for calculating hashes. (In jenny, I would also be able to and actually have to remove that code that skips over twts with a timestamp older than $last_fetch. This only got added as a workaround “to avoid broken threads all the time”.) The net result would be *less code*.

Implementing this whole (edit:#hash) thing means *more code*. (For jenny, specifically, *a lot* more code, if I want to allow users to create such twts.)

Do you see why I’m so reluctant to jump on this bandwagon? 😅

I haven’t come up yet with good, concrete examples where (replyto:…) would break. As soon as that happens, I’ll change my mind. 🤔
@prologic Can you come up with actual scenarios where it would break? Or is it more of a gut feeling?

The thing that keeps bugging me is this:

If we were to switch to location-based addressing and (replyto:…), the edit problem would resolve itself. Implementations could use that exact string (e.g., https://example.com/tw.txt,2024-09-18T12:45Z) as the internal identifier of a twt and that is pretty much the only change that you have to make. And then you could throw away all code and tests currently required for calculating hashes. (In jenny, I would also be able to and actually have to remove that code that skips over twts with a timestamp older than $last_fetch. This only got added as a workaround “to avoid broken threads all the time”.) The net result would be *less code*.

Implementing this whole (edit:#hash) thing means *more code*. (For jenny, specifically, *a lot* more code, if I want to allow users to create such twts.)

Do you see why I’m so reluctant to jump on this bandwagon? 😅

I haven’t come up yet with good, concrete examples where (replyto:…) would break. As soon as that happens, I’ll change my mind. 🤔
@prologic Can you come up with actual scenarios where it would break? Or is it more of a gut feeling?

The thing that keeps bugging me is this:

If we were to switch to location-based addressing and (replyto:…), the edit problem would resolve itself. Implementations could use that exact string (e.g., https://example.com/tw.txt,2024-09-18T12:45Z) as the internal identifier of a twt and that is pretty much the only change that you have to make. And then you could throw away all code and tests currently required for calculating hashes. (In jenny, I would also be able to and actually have to remove that code that skips over twts with a timestamp older than $last_fetch. This only got added as a workaround “to avoid broken threads all the time”.) The net result would be *less code*.

Implementing this whole (edit:#hash) thing means *more code*. (For jenny, specifically, *a lot* more code, if I want to allow users to create such twts.)

Do you see why I’m so reluctant to jump on this bandwagon? 😅

I haven’t come up yet with good, concrete examples where (replyto:…) would break. As soon as that happens, I’ll change my mind. 🤔
For implementations, it would be nice if “update twts” always came *after* the twt they are referring to. So I thought about using this opportunity to mandate append-style feeds. But that’s just me being lazy. Implementations will *have to* be able to cope with any order, because feeds cannot/should not be trusted. 🫤
For implementations, it would be nice if “update twts” always came *after* the twt they are referring to. So I thought about using this opportunity to mandate append-style feeds. But that’s just me being lazy. Implementations will *have to* be able to cope with any order, because feeds cannot/should not be trusted. 🫤
For implementations, it would be nice if “update twts” always came *after* the twt they are referring to. So I thought about using this opportunity to mandate append-style feeds. But that’s just me being lazy. Implementations will *have to* be able to cope with any order, because feeds cannot/should not be trusted. 🫤