# I am the Watcher. I am your guide through this vast new twtiverse.
# 
# Usage:
#     https://watcher.sour.is/api/plain/users              View list of users and latest twt date.
#     https://watcher.sour.is/api/plain/twt                View all twts.
#     https://watcher.sour.is/api/plain/mentions?uri=:uri  View all mentions for uri.
#     https://watcher.sour.is/api/plain/conv/:hash         View all twts for a conversation subject.
# 
# Options:
#     uri     Filter to show a specific users twts.
#     offset  Start index for quey.
#     limit   Count of items to return (going back in time).
# 
# twt range = 1 196266
# self = https://watcher.sour.is?offset=171456
# next = https://watcher.sour.is?offset=171556
# prev = https://watcher.sour.is?offset=171356
@prologic what time in UTC?
@prologic what time in UTC?
@prologic what time in UTC?
@prologic what time in UTC?
What about edits of edits? Do we want to “chain” edits or does the latest edit simply win?

Chained edits:

[#abcd111] [2024-09-20T12:00:00Z] [Hello!]
[#abcd222] [2024-09-20T12:10:00Z] [(edit:#abcd111) Hello World!]
[#abcd333] [2024-09-20T12:20:00Z] [(edit:#abcd222) Hello Birds!]

Latest edit wins:

[#abcd111] [2024-09-20T12:00:00Z] [Hello!]
[#abcd222] [2024-09-20T12:10:00Z] [(edit:#abcd111) Hello World!]
[#abcd333] [2024-09-20T12:20:00Z] [(edit:#abcd111) Hello Birds!]

Does the first version have any benefits? I don’t think so … ?
What about edits of edits? Do we want to “chain” edits or does the latest edit simply win?

Chained edits:

[#abcd111] [2024-09-20T12:00:00Z] [Hello!]
[#abcd222] [2024-09-20T12:10:00Z] [(edit:#abcd111) Hello World!]
[#abcd333] [2024-09-20T12:20:00Z] [(edit:#abcd222) Hello Birds!]

Latest edit wins:

[#abcd111] [2024-09-20T12:00:00Z] [Hello!]
[#abcd222] [2024-09-20T12:10:00Z] [(edit:#abcd111) Hello World!]
[#abcd333] [2024-09-20T12:20:00Z] [(edit:#abcd111) Hello Birds!]

Does the first version have any benefits? I don’t think so … ?
What about edits of edits? Do we want to “chain” edits or does the latest edit simply win?

Chained edits:

\n \n \n
\n \n [(edit:#abcd111) Hello World!]
\n \n [(edit:#abcd222) Hello Birds!]

Latest edit wins:

\n \n \n
\n \n [(edit:#abcd111) Hello World!]
\n \n [(edit:#abcd111) Hello Birds!]

Does the first version have any benefits? I don’t think so … ?
What about edits of edits? Do we want to “chain” edits or does the latest edit simply win?

Chained edits:

[#abcd111] [2024-09-20T12:00:00Z] [Hello!]
[#abcd222] [2024-09-20T12:10:00Z] [(edit:#abcd111) Hello World!]
[#abcd333] [2024-09-20T12:20:00Z] [(edit:#abcd222) Hello Birds!]

Latest edit wins:

[#abcd111] [2024-09-20T12:00:00Z] [Hello!]
[#abcd222] [2024-09-20T12:10:00Z] [(edit:#abcd111) Hello World!]
[#abcd333] [2024-09-20T12:20:00Z] [(edit:#abcd111) Hello Birds!]

Does the first version have any benefits? I don’t think so … ?
What about edits of edits? Do we want to “chain” edits or does the latest edit simply win?

Chained edits:

[#abcd111] [2024-09-20T12:00:00Z] [Hello!]
[#abcd222] [2024-09-20T12:10:00Z] [(edit:#abcd111) Hello World!]
[#abcd333] [2024-09-20T12:20:00Z] [(edit:#abcd222) Hello Birds!]

Latest edit wins:

[#abcd111] [2024-09-20T12:00:00Z] [Hello!]
[#abcd222] [2024-09-20T12:10:00Z] [(edit:#abcd111) Hello World!]
[#abcd333] [2024-09-20T12:20:00Z] [(edit:#abcd111) Hello Birds!]

Does the first version have any benefits? I don’t think so … ?
@prologic Yeah, you’re right. That’s an implementation detail of jenny. Right now, the order of twts doesn’t matter at all, because it’s only relevant at display time – and that’s the job of mutt. 😅
@prologic Yeah, you’re right. That’s an implementation detail of jenny. Right now, the order of twts doesn’t matter at all, because it’s only relevant at display time – and that’s the job of mutt. 😅
@prologic Yeah, you’re right. That’s an implementation detail of jenny. Right now, the order of twts doesn’t matter at all, because it’s only relevant at display time – and that’s the job of mutt. 😅
@prologic Yeah, you’re right. That’s an implementation detail of jenny. Right now, the order of twts doesn’t matter at all, because it’s only relevant at display time – and that’s the job of mutt. 😅
@falsifian Oof, yeah, I haven’t even started thinking about supporting two schemes at the same time. 😅 I’d be hoping for *not* having to use something like an sqlite database, if it can’t be avoided.

By the way: Since we have so few modern twtxt/Yarn clients, forking jenny might not be the worst idea. *If* you wanted to take it into a very different direction, then by all means, go for it. 👍
@falsifian Oof, yeah, I haven’t even started thinking about supporting two schemes at the same time. 😅 I’d be hoping for *not* having to use something like an sqlite database, if it can’t be avoided.

By the way: Since we have so few modern twtxt/Yarn clients, forking jenny might not be the worst idea. *If* you wanted to take it into a very different direction, then by all means, go for it. 👍
@falsifian Oof, yeah, I haven’t even started thinking about supporting two schemes at the same time. 😅 I’d be hoping for *not* having to use something like an sqlite database, if it can’t be avoided.

By the way: Since we have so few modern twtxt/Yarn clients, forking jenny might not be the worst idea. *If* you wanted to take it into a very different direction, then by all means, go for it. 👍
@falsifian Oof, yeah, I haven’t even started thinking about supporting two schemes at the same time. 😅 I’d be hoping for *not* having to use something like an sqlite database, if it can’t be avoided.

By the way: Since we have so few modern twtxt/Yarn clients, forking jenny might not be the worst idea. *If* you wanted to take it into a very different direction, then by all means, go for it. 👍
@lyse When it asks a Yarn pod, you mean? Yeah, it does so implicitly. It builds a tiny dummy feed from the JSON response and then looks for the specified twt hash in that feed.
@lyse When it asks a Yarn pod, you mean? Yeah, it does so implicitly. It builds a tiny dummy feed from the JSON response and then looks for the specified twt hash in that feed.
@lyse When it asks a Yarn pod, you mean? Yeah, it does so implicitly. It builds a tiny dummy feed from the JSON response and then looks for the specified twt hash in that feed.
@lyse When it asks a Yarn pod, you mean? Yeah, it does so implicitly. It builds a tiny dummy feed from the JSON response and then looks for the specified twt hash in that feed.
@prologic Wouldn’t work in what way? Could you elaborate? 🤔

Do you consider crawling archived feeds a problem/failure? 🤔
@prologic Wouldn’t work in what way? Could you elaborate? 🤔

Do you consider crawling archived feeds a problem/failure? 🤔
@prologic Wouldn’t work in what way? Could you elaborate? 🤔

Do you consider crawling archived feeds a problem/failure? 🤔
@prologic Wouldn’t work in what way? Could you elaborate? 🤔

Do you consider crawling archived feeds a problem/failure? 🤔
@prologic cool, I will be there! Are you going to post the regular banner notice? It will serve as a reminder, at least for me.
@prologic cool, I will be there! Are you going to post the regular banner notice? It will serve as a reminder, at least for me.
[47°09′46″S, 126°43′19″W] Raw reading: 0x66ED55B1, offset +/-5
I'm looking forward to a sunny weekend with mild temperatures.
AWESOME COOL ☻
👋 Reminder that next Saturday 28th September will be out monthly online meetup! Hope to see some/all of you there 👌
👋 Reminder that next Saturday 28th September will be out monthly online meetup! Hope to see some/all of you there 👌
[47°09′20″S, 126°43′08″W] --interrupted--
I've been building a new notebook app. Doesn't actually run any code yet, but take a look: https://akkartik.name/post/2024-09-19-devlog
I've been building a new notebook app. Doesn't actually run any code yet, but take a look: https://akkartik.name/post/2024-09-19-devlog
[47°09′57″S, 126°43′00″W] --no signal--
I'll try to reproduce locally later tonight
I'll try to reproduce locally later tonight
i kinda click a yarn then a fork and the back button. i have to do a few goes before it does it.
i kinda click a yarn then a fork and the back button. i have to do a few goes before it does it.
@lyse I don't think this is true.
@lyse I don't think this is true.
@lyse No that's never a problem because we really only want to "navigate" the web anyway not form threads of xonversation 🤣
@lyse No that's never a problem because we really only want to "navigate" the web anyway not form threads of xonversation 🤣
@movq this approach also wouldn't work and when that Feed gets archived so you'll be forced to crawl archived feeds at that point.
@movq this approach also wouldn't work and when that Feed gets archived so you'll be forced to crawl archived feeds at that point.
The important bits missing from this summary (_devil is in the details_) are two requirements:

- Clients _should_ order Twts by their timestamp.
- Clients *must* validate all edit and delete requests that the hash being indicated belongs to and came from that feed.
- Client _should_ honour delete requests and delete Twts from their cache/archive.
The important bits missing from this summary (_devil is in the details_) are two requirements:

- Clients _should_ order Twts by their timestamp.
- Clients *must* validate all edit and delete requests that the hash being indicated belongs to and came from that feed.
- Client _should_ honour delete requests and delete Twts from their cache/archive.
@lyse This is why hashes provide that level of integrity. The hash can be verified in the cache or archive as belonging to said feed.
@lyse This is why hashes provide that level of integrity. The hash can be verified in the cache or archive as belonging to said feed.
@movq I think the order of the lines in a feed don't matter as long as we can guarantee the order of Twts. Clients should already be ordering by Timestamp anyway.
@movq I think the order of the lines in a feed don't matter as long as we can guarantee the order of Twts. Clients should already be ordering by Timestamp anyway.
@movq Pretry much 👌
@movq Pretry much 👌
@lyse Sorry could you explain this sifferently?
@lyse Sorry could you explain this sifferently?
Do you k ow what you clicked on before going back?
Do you k ow what you clicked on before going back?
@eldersnake Sweet thank you! 🙇‍♂️ I'll merge this PR tonight I think.
@eldersnake Sweet thank you! 🙇‍♂️ I'll merge this PR tonight I think.
Seems to be working OK 🤔
🧮 USERS:1 FEEDS:2 TWTS:1098 ARCHIVED:79066 CACHE:2530 FOLLOWERS:17 FOLLOWING:14
On my blog: Real Life in Star Trek, The First Duty https://john.colagioia.net/blog/2024/09/19/first-duty.html #scifi #startrek #closereading
@david Well, I wouldn't recommend using my code for your main jenny use anyway. If you want to try it out, set XDG_CONFIG_HOME and XDG_CACHE_HOME to some sandbox directories and only run my code there. If @movq is interested in any of this getting upstreamed, I'd be happy to try rebasing the changes, but otherwise it's a proof of concept and fun exercise.
@falsifian hiya from the Sunshine State (also known as a never-ending hell, LOL)!
@falsifian hiya from the Sunshine State (also known as a never-ending hell, LOL)!
I forgot to git add a new test file. Added to the patch now at https://www.falsifian.org/a/oDtr/patch0.txt
@falsifian this one hits hard, as jenny was just updated today. :'-(
@falsifian this one hits hard, as jenny was just updated today. :'-(


its replacing the contents of body for some reason.


its replacing the contents of body for some reason.
@david Hello!
@prologic Hi. i have noticed sometimes when i hit the back button i lose all the surrounding layout and just have a list of twts.

@prologic Hi. i have noticed sometimes when i hit the back button i lose all the surrounding layout and just have a list of twts.

BTW this code doesn't incorporate existing twts into jenny's database. It's best used starting from scratch. I've been testing it using a custom XDG_CACHE_HOME and XDG_CONFIG_HOME to avoid messing with my "real" jenny data.
I wrote some code to try out non-hash reply subjects formatted as (replyto ), while keeping the ability to use the existing hash style.

I don't think we need to decide all at once. If clients add support for a new method then people can use it if they like. The downside of course is that this costs developer time, so I decided to invest a few hours of my own time into a proof of concept.

With apologies to @movq for corrupting jenny's beautiful code. I don't write this expecting you to incorporate the patch, because it does complicate things and might not be a direction you want to go in. But if you like any part of this approach feel free to use bits of it; I release the patch under jenny's current LICENCE.

Supporting both kinds of reply in jenny was complicated because each email can only have one Message-Id, and because it's possible the target twt will not be seen until after the twt referencing it. The following patch uses an sqlite database to keep track of known (url, timestamp) pairs, as well as a separate table of (url, timestamp) pairs that haven't been seen yet but are wanted. When one of those "wanted" twts is finally seen, the mail file gets rewritten to include the appropriate In-Reply-To header.

Patch based on jenny commit 73a5ea81.

https://www.falsifian.org/a/oDtr/patch0.txt

Not implemented:
- Composing twts using the (replyto ...) format.
- Probably other important things I'm forgetting.
@yarn_police I am noticing crimes around here, but I am too afraid to say anything (points at Lyse with mouth movements, no muttering a single word). Help us, officer!
Hahahahahaahaaaahaaaaaa, brilliant! I love it, @bender! :'-D
Hmm…
@movq Thanks for the summary!

So, what would happen if there is no original message anymore in the feed and you encounter an "edit" subject? Since you cannot verify that the feed contained it in the first place, would you obey it?

Some feed could just make a client update something from a different feed. In the cache, the client would need to store in a flag that this message was updated, so that when it later encounters the message from the real feed, it has a chance of reverting that bogus edit. Hmm. The devil is in the detail.

It's much easier with a delete subject. When it finds the message in its cache and the feeds match, remove it. Otherwise, just ignore it.
@lyse matter of fact, earlier you posted:


2024-09-19T20:20:00+02:00\tI don't like Australians!


And then deleted it, fearing the Australian Mafia (which, as we know, is very powerful in Bavaria). But I got the hash for it, p5zdahq, and that timestamp has tt written all over it. That's my proof! 😅😅😅
@lyse matter of fact, earlier you posted:


2024-09-19T20:20:00+02:00	I don't like Australians!


And then deleted it, fearing the Australian Mafia (which, as we know, is very powerful in Bavaria). But I got the hash for it, p5zdahq, and that timestamp has tt written all over it. That's my proof! 😅😅😅
@movq Right. That's why, I'd bite the bullet and go for huge URLs. :-)

I havent't looked at the code and I'm too lazy right now, does jenny also verify the fetched result against the hash?
@movq Yeah, but hashing also uses the main feed URL or whatever is written in the feed's first url metadata field. So, it's not a new problem, it's exactly the same.
@lyse indeed! There is no "central authority" acting as witness, and notary. The more I think of it... LOL.
@lyse indeed! There is no "central authority" acting as witness, and notary. The more I think of it... LOL.
@movq @david Yeah, he got a bit older but I could still easily recognize him.
@david Such a funny picture – we’ve been to Florida once some ~30 years ago and it looked almost *exactly* like that. 😅~
@david Such a funny picture – we’ve been to Florida once some ~30 years ago and it looked almost *exactly* like that. 😅~
@david Such a funny picture – we’ve been to Florida once some ~30 years ago and it looked almost *exactly* like that. 😅~
@david Such a funny picture – we’ve been to Florida once some ~30 years ago and it looked almost *exactly* like that. 😅~
@david Yeah, but it happened so fast with him. 😅 I remember watching some of his talks 1-3 years ago, looked completely different, I think. 😅

Luckily I can still recognize the voice, so I know it’s him, lol.
@david Yeah, but it happened so fast with him. 😅 I remember watching some of his talks 1-3 years ago, looked completely different, I think. 😅

Luckily I can still recognize the voice, so I know it’s him, lol.
@david Yeah, but it happened so fast with him. 😅 I remember watching some of his talks 1-3 years ago, looked completely different, I think. 😅

Luckily I can still recognize the voice, so I know it’s him, lol.
@david Yeah, but it happened so fast with him. 😅 I remember watching some of his talks 1-3 years ago, looked completely different, I think. 😅

Luckily I can still recognize the voice, so I know it’s him, lol.
@lyse The hash/thread-id would be shorter, but you’d lose two other benefits of (replyto:…):

1. You need a special client again to compute hashes.
2. The original feed URL is no longer visible, thus you might need to ask a Yarn pod occasionally for missing twts (I do that surprisingly often, now that I’ve implemented it) – but now you’ve lost the guarantee that Yarn gives you the correct information, because you can no longer verify it.
@lyse The hash/thread-id would be shorter, but you’d lose two other benefits of (replyto:…):

1. You need a special client again to compute hashes.
2. The original feed URL is no longer visible, thus you might need to ask a Yarn pod occasionally for missing twts (I do that surprisingly often, now that I’ve implemented it) – but now you’ve lost the guarantee that Yarn gives you the correct information, because you can no longer verify it.
@lyse The hash/thread-id would be shorter, but you’d lose two other benefits of (replyto:…):

1. You need a special client again to compute hashes.
2. The original feed URL is no longer visible, thus you might need to ask a Yarn pod occasionally for missing twts (I do that surprisingly often, now that I’ve implemented it) – but now you’ve lost the guarantee that Yarn gives you the correct information, because you can no longer verify it.
@lyse The hash/thread-id would be shorter, but you’d lose two other benefits of (replyto:…):

1. You need a special client again to compute hashes.
2. The original feed URL is no longer visible, thus you might need to ask a Yarn pod occasionally for missing twts (I do that surprisingly often, now that I’ve implemented it) – but now you’ve lost the guarantee that Yarn gives you the correct information, because you can no longer verify it.
Another thing: At the moment, anyone could claim that some feed contained a certain message which was then removed again by just creating the hash over the fake message in said feed and invented timestamp themselves. Nobody can ever verify that this was never the case in the first place and completely made up. So, our twt hashes have to be taken with a grain of salt.