# I am the Watcher. I am your guide through this vast new twtiverse.
# 
# Usage:
#     https://watcher.sour.is/api/plain/users              View list of users and latest twt date.
#     https://watcher.sour.is/api/plain/twt                View all twts.
#     https://watcher.sour.is/api/plain/mentions?uri=:uri  View all mentions for uri.
#     https://watcher.sour.is/api/plain/conv/:hash         View all twts for a conversation subject.
# 
# Options:
#     uri     Filter to show a specific users twts.
#     offset  Start index for quey.
#     limit   Count of items to return (going back in time).
# 
# twt range = 1 6505
# self = https://watcher.sour.is?uri=https://lyse.isobeef.org/twtxt.txt&offset=5434
# next = https://watcher.sour.is?uri=https://lyse.isobeef.org/twtxt.txt&offset=5534
# prev = https://watcher.sour.is?uri=https://lyse.isobeef.org/twtxt.txt&offset=5334
@movq Thanks for the summary!

So, what would happen if there is no original message anymore in the feed and you encounter an "edit" subject? Since you cannot verify that the feed contained it in the first place, would you obey it?

Some feed could just make a client update something from a different feed. In the cache, the client would need to store in a flag that this message was updated, so that when it later encounters the message from the real feed, it has a chance of reverting that bogus edit. Hmm. The devil is in the detail.

It's much easier with a delete subject. When it finds the message in its cache and the feeds match, remove it. Otherwise, just ignore it.
@movq Right. That's why, I'd bite the bullet and go for huge URLs. :-)

I havent't looked at the code and I'm too lazy right now, does jenny also verify the fetched result against the hash?
@movq Yeah, but hashing also uses the main feed URL or whatever is written in the feed's first url metadata field. So, it's not a new problem, it's exactly the same.
@movq @david Yeah, he got a bit older but I could still easily recognize him.
Another thing: At the moment, anyone could claim that some feed contained a certain message which was then removed again by just creating the hash over the fake message in said feed and invented timestamp themselves. Nobody can ever verify that this was never the case in the first place and completely made up. So, our twt hashes have to be taken with a grain of salt.
@david Cool idea actually! The hash would also be shorter than the raw URL and timestamp.
@prologic I get where you're coming from. But is it really that bad in practice? If you follow any link somewhere in the web, you also don't know if its contents has been changed in the meantime. Is that a problem? Almost never in my experience.

Granted, it's a nice property when one can tell that it was not messed with since the author referenced it.
@movq The more I think about it, the more do I like the location-based addressing. That feels fairly in line with the spirit of twtxt, just like you stated somewhere else.

The big downside for me is that the subjects then become super long.

And if the feed relocates, we end up with broken conversation trees again. Just like nowadays. At least it's not getting worse. :-)

Using the feed URL in there might become a little challenging for new folks, when the twt rotates away into archive feeds. But I reckon, we already have a similar situation with the hashes. So, probably not too bad.
@quark Yeah, let's see what they reveal!
Nice, @david! The winter palms look nice. And the sky is full of snow.
Yesterday, both temperature and wind picked up. There was even wind in the night, which is rare over here. Today, we also got a lot of sunshine, around 22°C and heaps of wind. The leaves and twigs were blown at the house door, it reminded me of a snow drift, basically a leave bank. I should have taken a photo before I swept it, it looked quite bizarre.

But I photographed something else instead:

Possibly a large roof panel on a crane

My mate and I went out in the woods earlier and we came across 08 which broke off in roughly 6, 7 meters from 09. When it hit the ground, it made a 30 cm deep hole. Quite impressive. https://lyse.isobeef.org/waldspaziergang-2024-09-19/
@falsifian Yeah, delete requests feel very odd.
@prologic I wish that was true! But I reckon there is still heaps of old stuff out there, that was created on a Windows machine. :-D And I wouldn't be surprised if even today in that environment a new file does not make use of UTF-8.
@quark I'm not convinced. :-D
@quark @movq Yep, they're all RFC3339. Obviously, +02:00 and +01:00 are best, because I use them! :-P In all seriousness, Z might be the best timezone, as it is shortest. And regarding privacy, it leaks the least information about the user's rough location. But of course, one can just look at the activity and narrow down plausible regions, so that's a weak argument.
@falsifian I can confirm, it's fixed. Thank you! Indeed, this is some wild quoting.

I still do not understand why the encoding suddenly broke, though. :-? Anyway. I concentrate on my rewrite and do things the right™ way. ;-) Still long ways to go.
@bender I know, I know… A relative time in a static HTML document is questionable at best. ;-)
Now WTF!? Suddenly, @falsifian's feed renders broken in my tt Python implementation. Exactly what I had with my Go rewrite. I haven't touched the Python stuff in ages, though. Also, tt and tt2 do not share any data at all.

By any chance, did you remove the ; charset=utf-8 from your Content-Type: text/plain header, falsifian?

interpreted in some crappy windows charset
@movq Non-ASCII characters were broken. Like U+2028, degrees (°), etc.

Turns out I used a silly library to detect the encoding and transform to UTF-8 if needed. When there is no Content-Type header, like for local files, it looks at the first 1024 bytes. Since it only saw ASCII in that region, the damn thing assumed the data to be in Windows-1252 (which for web pages kinda makes sense):

// TODO: change default depending on user's locale?
return charmap.Windows1252, "windows-1252", false

https://cs.opensource.google/go/x/net/+/master:html/charset/charset.go;l=102

This default is hardcoded and cannot be changed.

Trying to be smart and adding automatic support for other encodings turned out to be a bad move on my end. At least I can reduce my dependency list again. :-)

I now just reject everything that explicitly specifies something different than text/plain and an optional charset other than utf-8 (ignoring casing). Otherwise I assume it's in UTF-8 (just like the twtxt file format specification mandates) and hope for the best.
Hmmmm, I somehow run into an encoding problem where my inserted data end up mangled in the database. But, both SQLite and Go use UTF-8. What's happening here? :-?
@prologic Correct. :-D
@prologic I'm basically with @movq, but in contrast to him, I'm not looking forward to implement something like that. :-)

A feed URL is plenty good enough for me. Since I only fetch feeds that I explicity follow, there is some basic trust in those feeds already. Spoofing, impersonation and what not are no issues for me. If I were to find out otherwise, I just unsubscribe from the evil feed. Done.

To retrieve public feeds, I just rely on TLS. Most are served via HTTPS. If a feed is down, I'm not trying to fetch it from some other source, I just wait and try again later. So signed messages/feeds are not a use case I'm particularly benefitting from.

To me, it's just not worth at all adding this crypto complexity on top.
Found it: https://github.com/buckket/twtxt/issues/157
@prologic Yeah, but I reckon we can kill both birds with one stone. If we change it to support edits, it should be fairly easy to also tweak it to support feed URL changes. Like outlined in my first reply: https://twtxt.net/twt/n4omfvq The URL part sounds way easier to me. :-)
@sorenpeter There was or maybe still is a competing proposal for multiline twts that combines all twts with the same timestamp to one logical multiline twt. Not sure what happened to that, if it is used in the wild and whether anyone "here" follows a feed with that convention. "Our" solution for multiline twts is to use U+2028 Unicode LINE SEPARATOR as a newline: https://dev.twtxt.net/doc/multilineextension.html.
@movq What's you definition of "complete thread"? ;-) There might be feeds participating in the conversation that you have no idea of.

But yes, this has a nice discoverability bonus. And even simpler than a hash, that's right.
@movq Yeah, I think so.
Keys for identity are too much for me. This steps up the complexity by a lot. Simplicity is what made me join twtxt with its extensions. A feed URL is all I need.

Eventually, twt hashes have to change (lengthen at least), no doubt about that. But I'd like to keep it equally simple.
@prologic When the next hype train departs. :-)
@stigatle Yeah, the sudden drop makes it feel worse than it is. It made me wear a beanie and gloves on my bike ride on Friday evening. In a few weeks I consider the same temperatures not an issue anymore, maybe even nicely warm. ;-) The body is fairly quick to adopt, but not that fast.

I just saw that we're supposed to hit 19°C mid next week again. Let's see.
@off_grid_living Oh dear, what an epic adventure! Terrible at the time, but hilarious to tell later on. :-D

I do like this photo a lot. It brings up memories of cool scouting trips.
@off_grid_living Hahaha, this is really great, I love it! :-D
@off_grid_living Still a bit different, but this reminds me of the rusk boy on the Brandt boxes which is kinda iconic over here: https://cdn.idealo.com/folder/Product/2151/8/2151814/s1_produktbild_max/brandt-der-markenzwieback-225-g.jpg They should switch to this photo. :-)
@off_grid_living It's kinda cool to see how small cars were back in the days. Especially the left one looks really tiny.
Happy birthday @prologic! :-)
Ta, @bender! Correct, apart from resizing, no further processing on my end. That's just the Japanese sunset photo engineer's magic. :-) In all it's original glory (3.2 MiB): https://lyse.isobeef.org/abendhimmel-2024-09-13/02.JPG
@off_grid_living Looks like you're describing a captcha. They do not really work. Bots seem to solve them, too.
@movq Thanks! Yeah, one week for autumn and spring must be enough. Or so the weather thinks. Looks like there is only on or off.
@prologic Oh, that's a lovely campfire! Seeing them always makes me smile. Enjoy your time in nature with your loved ones.
Cool sunset when I went to the scouts: https://lyse.isobeef.org/abendhimmel-2024-09-13/

20° temperature drop in just a hand full of days. Ooof. We went on a stroll at 10°C today. I could have used a beanie, my ears were very cold. The sun was out, but hardly any people. Very nice. Also, no wind.

It was nice to finally hear a few birds singing again, although it was still fairly silent. The sun gave us a nice show. In hindsight, we should have stayed at the summit a bit longer. In the forest, we missed the very best, crazy red sky. We could only see parts shimmering through the tree lines.



https://lyse.isobeef.org/waldspaziergang-2024-09-12/
@movq Yeah, public transport is great if it works. All too often, it just doesn't, though. :-( Unfortunately, for my trips to the offices, it's always slower than a car.

That website looks like one I would build. :'-D I just always go to bahn.de. It even works alright if the train is operated by another company. At least it's good enough for my connections (VVS, Arverio, Ding & Co.). When GoAhead took over the line from DB, their delay/cancel information on their own website were just as bad as the one relayed by DB most of the time.
@movq @bender That was indeed a funny adventure. I really had to laugh about the mess on the floor I made. :-D
@movq Right!
The knowledge gain was still very limited, but it actually turned out a little better than I thought. Talking to the people face to face was really nice. And we also had a surprise barbie in the end, so it was worth coming. :-D

Also, the train connections worked out. Just on the way back, I made the error to use the toilet in the train. I've experienced way worse, but there was certainly a little Urine odor in the air. Second thing I noted was a large pile of toilet paper in the bowl.

When I wanted to wash my hands, I got the soap dispenser to work, but the tap just dripped extremely slowly. Not usable. Then it clicked why there was all this paper in the loo. I tried to wipe the soap off with toilet paper as best as I could and then used my water bottle to rinse my hands. Luckily, I had topped it off before I left the office. I only had to use my jumper to increase grip for actually getting the lid off. The sparkling water happily soaked my jumper and the floor in an instant. :-D

Tip for your next train ride: Bring your own water supply, preferably non-carbonated. Alternatively, just use the office toilet beforehand.

Turns out that at least this train model has two separate water tanks. One for the faucet and another for the loo. I flushed the paper without issues before I left.
Merci, @movq! I will keep you posted. :-)
@movq Same here for sure. :-D Great, I just saw the start was postponed by yet another half hour. I could have slept longer. Well, gonna catch the later train then.
@falsifian Regarding your last paragraph: Back in December 2020, we already once changed the hashing. I think that was my first contribution, breaking everything by switching to RFC 3339 for the timestamp format. ;-) I'm computing two hashes in my client, the old and current one. And then I just select whatever matching parent exists to build the thread tree.

I could do that again in my client, but you're right, it's a different story for jenny. If I'm not mistaken, In-Reply-To could contain several hashes, but the Message-ID header is the issue.

By increasing the hash length for a potential future change, clients could tell, which algorithm to use.

Maybe we could define a magic timestamp in the future that marks the cutoff point. Use the current implementation for messages authored before that magic date or the new algorithm for all messages after that.

But eventually, all clients have to be updated. There's no way around that, I believe. Simplicity is key and my magic time already adds complexity. :-/
@movq @aelaraji @bender Why not have both Jupiter and Venus together? https://www.youtube.com/watch?v=oXyCORLeLd0 I don't understand anything either, but it just sounds great to my ears.

> if not I am editing and breaking replies!

Bwahahahaaa! :'-D
Nice, thanks for the offer, @bender! You have to be a bit patient, though, it'll take a while until there is something to actually worth messing with. :-)
@movq @prologic :-D

It's like that for months. :-( And tomorrow I even gotta go into the office for some two day meeting, but I only attend a single day. On the positive side, I'm gonna see some workmates that I haven't ever met in the real world or for a very long time.
@movq Yep, that's very nice music. :-)

Can't help myself, but I have to include the Uranus song now. :-D https://www.youtube.com/watch?v=OSWszdSHkyE#t=7
Thank you, @bender! No, the rendering is based on https://github.com/rivo/tview.
https://github.com/spf13/afero looks better, but has a gazillion dependencies. So that's out.
I'm finally continuing with my tt rewrite in Go. So, I thought I use the shiny io/fs.FS. That's supposed to be a super cool new file system API. It allowed me to write tests more elegantly. I don't have to place actual test files on disk, but can keep everything nicely in RAM with testing/fstest.MapFS. That actually worked out great, I do like that.

However, os.DirFS("/") for production code is just a terrible solution. I noted that OS paths and io/fs.FS paths are fundamentally different. This new API does not allow leading slashes in the passed paths. This results in an error. So, I have to cut the leading slash off myself.

Also, the whole thing is totally useless on Windows, because of the drives. Simply does not work at all. Well, honestly, I don't care the slightest bit about that operating system, but it would be nice if this concept were cross-platform.

I haven't tested it, but I'm pretty sure relative paths or ~ do also not work. I have to first build absolute paths myself. Unfortunately, there is no builtin helper to translate an OS path into a io/fs.FS path.

Of course, others noted these shortcomings and surprising results, too: https://github.com/golang/go/issues/44279 There is no OSFileSystem implementation that would simply allow the easy transition from all the classical os.* functionality to io/fs.FS. And they also do not wanna add something like that either. Sigh.

I'm really wondering what they were thinking when introducing this. :-?

Even though, it's very silly, I'm gonna keep using it. At least for now. Tests have been written. I'm not keen on rewriting them. Sigh.
On second thought, the same rule with the last physically encountered URL when starting parsing from the top applies to prepend-style feeds as well. Much simpler and cleaner this way. Should also fit prepend-style feeds better I reckon.
@falsifian In my opinion it was a mistake that we defined the first url field in the feed to define the URL for hashing. It should have been the last encountered one. Then, assuming append-style feeds, you could override the old URL with a new one from a certain point on:

# url = https://example.com/alias/txtxt.txt
# url = https://example.com/initial/twtxt.txt


# url = https://example.com/new/twtxt.txt

# url = https://example.com/brand-new/twtxt.txt


In theory, the same could be done for prepend-style feeds. They do exist, I've come around them. The parser would just have to calculate the hashes afterwards and not immediately.
@prologic @bender That's exactly the case here with us as well. Maybe not 100% applicable to yarnd, but all other clients that only fetch from their user-controlled subscription list.
@movq @prologic Oh yeah, we have to take our time with that and craft it very carefully.

My theory about the descent of the original twtxt universe is that a) people just move on to other things and b) it was just not practical enough.
Nicely put, @movq! Exactly, reminding people to subscribe etc. is dodgy. To me it feels they think their audience is dumb (and they might be right, I don't know). Super annoying.
@aelaraji Just move to Mars to get an extra hour a day: https://spaceplace.nasa.gov/days/en/ If that's not enough, Mercury should have you covered for sure.
But I forgot to update the mentions. Oh well. I don't bother breaking hashes once more, so I just leave it at that. :-)
@prologic I simply have absolutely no interest in selling my thoughts to others. I don't like things behind paywalls. Each to their own, but I'm not gonna contribute to that.
@prologic https://lyse.isobeef.org/tmp/hashes.png Yep, broken hashes. I just fixed them after refollowing on the new URL (my client doesn't know metadata fields).
@prologic Yes, that's when I noticed. Luckily, I had another knive in my bagpack. But I cannot find the folding one at home either. Damn. I already lost my other knive at the flea market in May this year, now the next one. This sucks.
@movq It sounds complicated. After reading it only twice, I haven't gotten it. :-D

Yes, I'm all for dedicated message IDs. That would be a whole new format then. But I would be fine with it. The only thing is that all our clients have to be touched. At the moment, I do not worry about spoofing (however, I definitely should).
Fuck, I lost my pocket knive somewhere.
@cuaxolotl Yes, that's how it works. It never occurred to me that I just start changing things without a plan first.
@cuaxolotl Yes, that's how it works. It never occurred to me that I just start changing things without a plan first.
@cuaxolotl Hahahaha! :-D
@cuaxolotl Hahahaha! :-D
@prologic Not my cup of tea.
@bender Hahaha, that's what you get, suckers!
@prologic Tomorrow is getting hot again and then we might be lucky in that the summer is over.
It would be even funnier if @bender didn't have a Sendgrid account in the first place. Good catch!
One last thing before I hit the hay. This endpoint could respond with the raw twt, when asking for text/plain (it serves HTML at the moment). Return the physical line from the feed. Maybe with a comment above for the feed URL. Or doesn't the registry format also include the URL separated with a tab somehow? I'm too lazy right now to look it up. Also, not sure how useful that would be. Anyway, good night.
@quark Right, a little rain improves™ the situation so much… :-( Surprise, surprise, our rain has been delayed again.

Good hunting and bon appétit! :-) I never had Puerto Rico's national dish, but the photos look delicious. Yum!

I also tried ice cream, but I reckon I simply stick to your last tip instead. :-)
I reckon, this is as raw as you can get, @falsifian: curl -sH "Accept: application/json" https://twtxt.net/twt/st3wsda | jq You can piece it together from created and text (and twter.uri).
@quark Oh shit, second place! :-O


$ du -h .config/twtxt/cache*.db 
13M   .config/twtxt/cache2.db  # contains read status for each twt (very inefficient format)
7,0M  .config/twtxt/cache.db   # the actual cache by the original twtxt reference implementation


Yeah, wrong place for caches.
@bender I follow feeds that are somewhat interesting to me. At least for the most part.
@prologic Thanks for the honor, but I'm not sure why I specifically should be part of a deciding committee here. :-D I get it, I just fear people might misunderstand your message here.

I have to read up on the twtxt registry documentation on the weekend (too tired at the moment), but it should probably be no real issue to integrate that API into yarnd.
When we passed a few horses in the forest, there was really strong soup odor in the air. It didn't smell like horse at all, but soup. Maybe they've been soup horses, chickens were out of stock.

29°C, zero wind, extremely humid, luckily the sun was behind the clouds. I'm soaking wet, sweat ran down in streams and dripped in my eyes, it burned a bit. The sky is getting a little dark, I hope the thunderstorm and rain are really arriving here later. Rain had always been finally cancelled the couple last days.

I'm gotta go cool off my fingers now, they're swollen from the heat.
@movq Hahaha! :-D
@movq Right, not looking forward to disease-spreading mozzies and critters like that. We must become @stigatle's neighbors. :-)
My honey bread fell on the floor and would you believe which side was down? The clean, not the coated one. Witchcraft!
Suddenly, VLC crashes when I jump forward in videos. It's 100% reproducible. Reboot didn't fix it. Starting on the shell, I see:

Assertion @p->parent->stash_hwaccel failed at src/libavcodec/pthread_frame.c:649

Turns out, it's this: https://forum.mxlinux.org/viewtopic.php?t=81068 Before I even went online, I assumed that turning off hardware acceleration might help. And it does. Phew!
Turns out, there are seven species in Germany (two of them being venomous), but in my wider area there seem to be just the two: https://de.wikipedia.org/wiki/Liste_der_Schlangenarten_in_Deutschland Probably just even one, the common European adder is more to the south, just like I thought. But maybe with the climate getting hotter and hotter, they migrate north to me, too.
@movq That's a very good approach. I have the feeling that requirements engineering seems to be getting more and more a forgotten art these days.
@movq Haha, right. :-D
@prologic @movq Looking at the raw feed, there are a six mentions (even prologic) and a seven subjects. From a few years ago, though. :-)
@movq Zero wind here for days. Last year, we saw a long string of satellites passing by when we sat at the scout ground around the fire. That looked weird indeed.
@prologic We basically have no snakes around here. Just the grass snake and the common European adder. Or so I think. While it's super hard to come across the harmless first one, it's even rarer to find the little bit dangerous latter. Both are very shy. I never saw an adder. They probably do not live here in my area. A workmate told the other day that he accidentally run over one with this bicycle as a schoolboy, though.

@movq Yeah, I was happily suprised that the grass snake was still in this good shape.

I've no idea. Not sure how well frogs or toads see in the dark, but I would think if there is some noise coming towards you, you escape away and not towards it. Maybe they did not recognize me as a threat but were just curious?

This larger individual was also a frog or toad. :-) It didn't freak me out at all, it was super cool to watch. But I thought, what an idiot. :-D I mean, hopping against me once is alright, but then trying again is a bit silly. It then stopped and sat next to my boot. About half a centimeter away. I waited a few seconds and carefully moved on.
@movq Sure, but I would have expected that I would have read about it somewhere at least once. Anyway.
Very cool, fingers crossed, @off_grid_living! A simple wooden frame, like four posts and and a rectangle on top, to support the fabric/nets should suffice, wouldn't it? No need to build full sheds, I'd say. But I'm not a farmer at all.
@movq :-D
@movq Huh, this is the first time I heard of IBM WebExplorer.
I'm out of shape. I decided to walk up the local mountain to watch the sunset, but I arrived five minutes late, even though I sped up at the end. Should have started my journey ten or fifteen minutes earlier. I saw the setting sun at foot, but the photos were total disasters.

On the way there I picked two handful of blackberries in the forest. Delicious!

Today was the second time in my life that I saw a grass snake in the wild. They can easily be recognized by the yellow "ears". Unfortunately, this one was run over. :-( But I jumped at the opportunity to photograph it as it didn't escape in a fraction of a second like my first encounter three years ago. Still, poor fellow. :-(

Run over grass snake

On the way home, a deer jumped out of the brush in front of me and headed down the forest road before it went back in the other side. As always, that's nice.

I also had to slow down a bunch of times because of frogs or toads on the paths. Not sure which ones, it was already after dark. I guesstimate it must have been 60-70 amphibians in total, maybe more. Some of them did not move to the wayside but rather into the middle of the track, right in front of me. Crazy suicide frogs! There were four reeeeaaaallly close calls. I could just avoid stepping on them after they tried to hop right under my boot. Not a centimeter to spare. No toads were harmed during my trip. Phew!

Once I had to stop completely because of the large activity ahead of me. A larger (about the size of half a palm) individual surrounded my foot and then jumped against my heel. Twice! What the heck!? :-D But suuuper cool experience. I'm very glad I actually went out. Totally worth it. I met so many amazing animals. Don't care about the missed sunset a single bit.
@bender Nope, still don't have a mobile one. Works extremely well for me. :-)
Temperatures begin to drop a bit for the night, phew. https://lyse.isobeef.org/abendhimmel-2024-08-30/01.jpg
@movq @bender Haha, great! :-)

Just returned from an evening walk. We met a young slow worm, a bunch of small frogs, about the width of a pointing finger and a bucket load of sweat. It's bloody hot and humid. Also, heaps of ripped trash bags in the forest and lake. :-( No photos, it was too exhausting to even carry my pocket camera.