# I am the Watcher. I am your guide through this vast new twtiverse.
# 
# Usage:
#     https://watcher.sour.is/api/plain/users              View list of users and latest twt date.
#     https://watcher.sour.is/api/plain/twt                View all twts.
#     https://watcher.sour.is/api/plain/mentions?uri=:uri  View all mentions for uri.
#     https://watcher.sour.is/api/plain/conv/:hash         View all twts for a conversation subject.
# 
# Options:
#     uri     Filter to show a specific users twts.
#     offset  Start index for quey.
#     limit   Count of items to return (going back in time).
# 
# twt range = 1 6164
# self = https://watcher.sour.is?uri=https://lyse.isobeef.org/twtxt.txt&offset=5416
# next = https://watcher.sour.is?uri=https://lyse.isobeef.org/twtxt.txt&offset=5516
# prev = https://watcher.sour.is?uri=https://lyse.isobeef.org/twtxt.txt&offset=5316
@movq Non-ASCII characters were broken. Like U+2028, degrees (°), etc.

Turns out I used a silly library to detect the encoding and transform to UTF-8 if needed. When there is no Content-Type header, like for local files, it looks at the first 1024 bytes. Since it only saw ASCII in that region, the damn thing assumed the data to be in Windows-1252 (which for web pages kinda makes sense):

// TODO: change default depending on user's locale?
return charmap.Windows1252, "windows-1252", false

https://cs.opensource.google/go/x/net/+/master:html/charset/charset.go;l=102

This default is hardcoded and cannot be changed.

Trying to be smart and adding automatic support for other encodings turned out to be a bad move on my end. At least I can reduce my dependency list again. :-)

I now just reject everything that explicitly specifies something different than text/plain and an optional charset other than utf-8 (ignoring casing). Otherwise I assume it's in UTF-8 (just like the twtxt file format specification mandates) and hope for the best.
Hmmmm, I somehow run into an encoding problem where my inserted data end up mangled in the database. But, both SQLite and Go use UTF-8. What's happening here? :-?
@prologic Correct. :-D
@prologic I'm basically with @movq, but in contrast to him, I'm not looking forward to implement something like that. :-)

A feed URL is plenty good enough for me. Since I only fetch feeds that I explicity follow, there is some basic trust in those feeds already. Spoofing, impersonation and what not are no issues for me. If I were to find out otherwise, I just unsubscribe from the evil feed. Done.

To retrieve public feeds, I just rely on TLS. Most are served via HTTPS. If a feed is down, I'm not trying to fetch it from some other source, I just wait and try again later. So signed messages/feeds are not a use case I'm particularly benefitting from.

To me, it's just not worth at all adding this crypto complexity on top.
Found it: https://github.com/buckket/twtxt/issues/157
@prologic Yeah, but I reckon we can kill both birds with one stone. If we change it to support edits, it should be fairly easy to also tweak it to support feed URL changes. Like outlined in my first reply: https://twtxt.net/twt/n4omfvq The URL part sounds way easier to me. :-)
@sorenpeter There was or maybe still is a competing proposal for multiline twts that combines all twts with the same timestamp to one logical multiline twt. Not sure what happened to that, if it is used in the wild and whether anyone "here" follows a feed with that convention. "Our" solution for multiline twts is to use U+2028 Unicode LINE SEPARATOR as a newline: https://dev.twtxt.net/doc/multilineextension.html.
@movq What's you definition of "complete thread"? ;-) There might be feeds participating in the conversation that you have no idea of.

But yes, this has a nice discoverability bonus. And even simpler than a hash, that's right.
@movq Yeah, I think so.
Keys for identity are too much for me. This steps up the complexity by a lot. Simplicity is what made me join twtxt with its extensions. A feed URL is all I need.

Eventually, twt hashes have to change (lengthen at least), no doubt about that. But I'd like to keep it equally simple.
@prologic When the next hype train departs. :-)
@stigatle Yeah, the sudden drop makes it feel worse than it is. It made me wear a beanie and gloves on my bike ride on Friday evening. In a few weeks I consider the same temperatures not an issue anymore, maybe even nicely warm. ;-) The body is fairly quick to adopt, but not that fast.

I just saw that we're supposed to hit 19°C mid next week again. Let's see.
@off_grid_living Oh dear, what an epic adventure! Terrible at the time, but hilarious to tell later on. :-D

I do like this photo a lot. It brings up memories of cool scouting trips.
@off_grid_living Hahaha, this is really great, I love it! :-D
@off_grid_living Still a bit different, but this reminds me of the rusk boy on the Brandt boxes which is kinda iconic over here: https://cdn.idealo.com/folder/Product/2151/8/2151814/s1_produktbild_max/brandt-der-markenzwieback-225-g.jpg They should switch to this photo. :-)
@off_grid_living It's kinda cool to see how small cars were back in the days. Especially the left one looks really tiny.
Happy birthday @prologic! :-)
Ta, @bender! Correct, apart from resizing, no further processing on my end. That's just the Japanese sunset photo engineer's magic. :-) In all it's original glory (3.2 MiB): https://lyse.isobeef.org/abendhimmel-2024-09-13/02.JPG
@off_grid_living Looks like you're describing a captcha. They do not really work. Bots seem to solve them, too.
@movq Thanks! Yeah, one week for autumn and spring must be enough. Or so the weather thinks. Looks like there is only on or off.
@prologic Oh, that's a lovely campfire! Seeing them always makes me smile. Enjoy your time in nature with your loved ones.
Cool sunset when I went to the scouts: https://lyse.isobeef.org/abendhimmel-2024-09-13/

20° temperature drop in just a hand full of days. Ooof. We went on a stroll at 10°C today. I could have used a beanie, my ears were very cold. The sun was out, but hardly any people. Very nice. Also, no wind.

It was nice to finally hear a few birds singing again, although it was still fairly silent. The sun gave us a nice show. In hindsight, we should have stayed at the summit a bit longer. In the forest, we missed the very best, crazy red sky. We could only see parts shimmering through the tree lines.



https://lyse.isobeef.org/waldspaziergang-2024-09-12/
@movq Yeah, public transport is great if it works. All too often, it just doesn't, though. :-( Unfortunately, for my trips to the offices, it's always slower than a car.

That website looks like one I would build. :'-D I just always go to bahn.de. It even works alright if the train is operated by another company. At least it's good enough for my connections (VVS, Arverio, Ding & Co.). When GoAhead took over the line from DB, their delay/cancel information on their own website were just as bad as the one relayed by DB most of the time.
@movq @bender That was indeed a funny adventure. I really had to laugh about the mess on the floor I made. :-D
@movq Right!
The knowledge gain was still very limited, but it actually turned out a little better than I thought. Talking to the people face to face was really nice. And we also had a surprise barbie in the end, so it was worth coming. :-D

Also, the train connections worked out. Just on the way back, I made the error to use the toilet in the train. I've experienced way worse, but there was certainly a little Urine odor in the air. Second thing I noted was a large pile of toilet paper in the bowl.

When I wanted to wash my hands, I got the soap dispenser to work, but the tap just dripped extremely slowly. Not usable. Then it clicked why there was all this paper in the loo. I tried to wipe the soap off with toilet paper as best as I could and then used my water bottle to rinse my hands. Luckily, I had topped it off before I left the office. I only had to use my jumper to increase grip for actually getting the lid off. The sparkling water happily soaked my jumper and the floor in an instant. :-D

Tip for your next train ride: Bring your own water supply, preferably non-carbonated. Alternatively, just use the office toilet beforehand.

Turns out that at least this train model has two separate water tanks. One for the faucet and another for the loo. I flushed the paper without issues before I left.
Merci, @movq! I will keep you posted. :-)
@movq Same here for sure. :-D Great, I just saw the start was postponed by yet another half hour. I could have slept longer. Well, gonna catch the later train then.
@falsifian Regarding your last paragraph: Back in December 2020, we already once changed the hashing. I think that was my first contribution, breaking everything by switching to RFC 3339 for the timestamp format. ;-) I'm computing two hashes in my client, the old and current one. And then I just select whatever matching parent exists to build the thread tree.

I could do that again in my client, but you're right, it's a different story for jenny. If I'm not mistaken, In-Reply-To could contain several hashes, but the Message-ID header is the issue.

By increasing the hash length for a potential future change, clients could tell, which algorithm to use.

Maybe we could define a magic timestamp in the future that marks the cutoff point. Use the current implementation for messages authored before that magic date or the new algorithm for all messages after that.

But eventually, all clients have to be updated. There's no way around that, I believe. Simplicity is key and my magic time already adds complexity. :-/
@movq @aelaraji @bender Why not have both Jupiter and Venus together? https://www.youtube.com/watch?v=oXyCORLeLd0 I don't understand anything either, but it just sounds great to my ears.

> if not I am editing and breaking replies!

Bwahahahaaa! :'-D
Nice, thanks for the offer, @bender! You have to be a bit patient, though, it'll take a while until there is something to actually worth messing with. :-)
@movq @prologic :-D

It's like that for months. :-( And tomorrow I even gotta go into the office for some two day meeting, but I only attend a single day. On the positive side, I'm gonna see some workmates that I haven't ever met in the real world or for a very long time.
@movq Yep, that's very nice music. :-)

Can't help myself, but I have to include the Uranus song now. :-D https://www.youtube.com/watch?v=OSWszdSHkyE#t=7
Thank you, @bender! No, the rendering is based on https://github.com/rivo/tview.
https://github.com/spf13/afero looks better, but has a gazillion dependencies. So that's out.
I'm finally continuing with my tt rewrite in Go. So, I thought I use the shiny io/fs.FS. That's supposed to be a super cool new file system API. It allowed me to write tests more elegantly. I don't have to place actual test files on disk, but can keep everything nicely in RAM with testing/fstest.MapFS. That actually worked out great, I do like that.

However, os.DirFS("/") for production code is just a terrible solution. I noted that OS paths and io/fs.FS paths are fundamentally different. This new API does not allow leading slashes in the passed paths. This results in an error. So, I have to cut the leading slash off myself.

Also, the whole thing is totally useless on Windows, because of the drives. Simply does not work at all. Well, honestly, I don't care the slightest bit about that operating system, but it would be nice if this concept were cross-platform.

I haven't tested it, but I'm pretty sure relative paths or ~ do also not work. I have to first build absolute paths myself. Unfortunately, there is no builtin helper to translate an OS path into a io/fs.FS path.

Of course, others noted these shortcomings and surprising results, too: https://github.com/golang/go/issues/44279 There is no OSFileSystem implementation that would simply allow the easy transition from all the classical os.* functionality to io/fs.FS. And they also do not wanna add something like that either. Sigh.

I'm really wondering what they were thinking when introducing this. :-?

Even though, it's very silly, I'm gonna keep using it. At least for now. Tests have been written. I'm not keen on rewriting them. Sigh.
On second thought, the same rule with the last physically encountered URL when starting parsing from the top applies to prepend-style feeds as well. Much simpler and cleaner this way. Should also fit prepend-style feeds better I reckon.
@falsifian In my opinion it was a mistake that we defined the first url field in the feed to define the URL for hashing. It should have been the last encountered one. Then, assuming append-style feeds, you could override the old URL with a new one from a certain point on:

# url = https://example.com/alias/txtxt.txt
# url = https://example.com/initial/twtxt.txt


# url = https://example.com/new/twtxt.txt

# url = https://example.com/brand-new/twtxt.txt


In theory, the same could be done for prepend-style feeds. They do exist, I've come around them. The parser would just have to calculate the hashes afterwards and not immediately.
@prologic @bender That's exactly the case here with us as well. Maybe not 100% applicable to yarnd, but all other clients that only fetch from their user-controlled subscription list.
@movq @prologic Oh yeah, we have to take our time with that and craft it very carefully.

My theory about the descent of the original twtxt universe is that a) people just move on to other things and b) it was just not practical enough.
Nicely put, @movq! Exactly, reminding people to subscribe etc. is dodgy. To me it feels they think their audience is dumb (and they might be right, I don't know). Super annoying.
@aelaraji Just move to Mars to get an extra hour a day: https://spaceplace.nasa.gov/days/en/ If that's not enough, Mercury should have you covered for sure.
But I forgot to update the mentions. Oh well. I don't bother breaking hashes once more, so I just leave it at that. :-)
@prologic I simply have absolutely no interest in selling my thoughts to others. I don't like things behind paywalls. Each to their own, but I'm not gonna contribute to that.
@prologic https://lyse.isobeef.org/tmp/hashes.png Yep, broken hashes. I just fixed them after refollowing on the new URL (my client doesn't know metadata fields).
@prologic Yes, that's when I noticed. Luckily, I had another knive in my bagpack. But I cannot find the folding one at home either. Damn. I already lost my other knive at the flea market in May this year, now the next one. This sucks.
@movq It sounds complicated. After reading it only twice, I haven't gotten it. :-D

Yes, I'm all for dedicated message IDs. That would be a whole new format then. But I would be fine with it. The only thing is that all our clients have to be touched. At the moment, I do not worry about spoofing (however, I definitely should).
Fuck, I lost my pocket knive somewhere.
@cuaxolotl Yes, that's how it works. It never occurred to me that I just start changing things without a plan first.
@cuaxolotl Yes, that's how it works. It never occurred to me that I just start changing things without a plan first.
@cuaxolotl Hahahaha! :-D
@cuaxolotl Hahahaha! :-D
@prologic Not my cup of tea.
@bender Hahaha, that's what you get, suckers!
@prologic Tomorrow is getting hot again and then we might be lucky in that the summer is over.
It would be even funnier if @bender didn't have a Sendgrid account in the first place. Good catch!
One last thing before I hit the hay. This endpoint could respond with the raw twt, when asking for text/plain (it serves HTML at the moment). Return the physical line from the feed. Maybe with a comment above for the feed URL. Or doesn't the registry format also include the URL separated with a tab somehow? I'm too lazy right now to look it up. Also, not sure how useful that would be. Anyway, good night.
@quark Right, a little rain improves™ the situation so much… :-( Surprise, surprise, our rain has been delayed again.

Good hunting and bon appétit! :-) I never had Puerto Rico's national dish, but the photos look delicious. Yum!

I also tried ice cream, but I reckon I simply stick to your last tip instead. :-)
I reckon, this is as raw as you can get, @falsifian: curl -sH "Accept: application/json" https://twtxt.net/twt/st3wsda | jq You can piece it together from created and text (and twter.uri).
@quark Oh shit, second place! :-O


$ du -h .config/twtxt/cache*.db 
13M   .config/twtxt/cache2.db  # contains read status for each twt (very inefficient format)
7,0M  .config/twtxt/cache.db   # the actual cache by the original twtxt reference implementation


Yeah, wrong place for caches.
@bender I follow feeds that are somewhat interesting to me. At least for the most part.
@prologic Thanks for the honor, but I'm not sure why I specifically should be part of a deciding committee here. :-D I get it, I just fear people might misunderstand your message here.

I have to read up on the twtxt registry documentation on the weekend (too tired at the moment), but it should probably be no real issue to integrate that API into yarnd.
When we passed a few horses in the forest, there was really strong soup odor in the air. It didn't smell like horse at all, but soup. Maybe they've been soup horses, chickens were out of stock.

29°C, zero wind, extremely humid, luckily the sun was behind the clouds. I'm soaking wet, sweat ran down in streams and dripped in my eyes, it burned a bit. The sky is getting a little dark, I hope the thunderstorm and rain are really arriving here later. Rain had always been finally cancelled the couple last days.

I'm gotta go cool off my fingers now, they're swollen from the heat.
@movq Hahaha! :-D
@movq Right, not looking forward to disease-spreading mozzies and critters like that. We must become @stigatle's neighbors. :-)
My honey bread fell on the floor and would you believe which side was down? The clean, not the coated one. Witchcraft!
Suddenly, VLC crashes when I jump forward in videos. It's 100% reproducible. Reboot didn't fix it. Starting on the shell, I see:

Assertion @p->parent->stash_hwaccel failed at src/libavcodec/pthread_frame.c:649

Turns out, it's this: https://forum.mxlinux.org/viewtopic.php?t=81068 Before I even went online, I assumed that turning off hardware acceleration might help. And it does. Phew!
Turns out, there are seven species in Germany (two of them being venomous), but in my wider area there seem to be just the two: https://de.wikipedia.org/wiki/Liste_der_Schlangenarten_in_Deutschland Probably just even one, the common European adder is more to the south, just like I thought. But maybe with the climate getting hotter and hotter, they migrate north to me, too.
@movq That's a very good approach. I have the feeling that requirements engineering seems to be getting more and more a forgotten art these days.
@movq Haha, right. :-D
@prologic @movq Looking at the raw feed, there are a six mentions (even prologic) and a seven subjects. From a few years ago, though. :-)
@movq Zero wind here for days. Last year, we saw a long string of satellites passing by when we sat at the scout ground around the fire. That looked weird indeed.
@prologic We basically have no snakes around here. Just the grass snake and the common European adder. Or so I think. While it's super hard to come across the harmless first one, it's even rarer to find the little bit dangerous latter. Both are very shy. I never saw an adder. They probably do not live here in my area. A workmate told the other day that he accidentally run over one with this bicycle as a schoolboy, though.

@movq Yeah, I was happily suprised that the grass snake was still in this good shape.

I've no idea. Not sure how well frogs or toads see in the dark, but I would think if there is some noise coming towards you, you escape away and not towards it. Maybe they did not recognize me as a threat but were just curious?

This larger individual was also a frog or toad. :-) It didn't freak me out at all, it was super cool to watch. But I thought, what an idiot. :-D I mean, hopping against me once is alright, but then trying again is a bit silly. It then stopped and sat next to my boot. About half a centimeter away. I waited a few seconds and carefully moved on.
@movq Sure, but I would have expected that I would have read about it somewhere at least once. Anyway.
Very cool, fingers crossed, @off_grid_living! A simple wooden frame, like four posts and and a rectangle on top, to support the fabric/nets should suffice, wouldn't it? No need to build full sheds, I'd say. But I'm not a farmer at all.
@movq :-D
@movq Huh, this is the first time I heard of IBM WebExplorer.
I'm out of shape. I decided to walk up the local mountain to watch the sunset, but I arrived five minutes late, even though I sped up at the end. Should have started my journey ten or fifteen minutes earlier. I saw the setting sun at foot, but the photos were total disasters.

On the way there I picked two handful of blackberries in the forest. Delicious!

Today was the second time in my life that I saw a grass snake in the wild. They can easily be recognized by the yellow "ears". Unfortunately, this one was run over. :-( But I jumped at the opportunity to photograph it as it didn't escape in a fraction of a second like my first encounter three years ago. Still, poor fellow. :-(

Run over grass snake

On the way home, a deer jumped out of the brush in front of me and headed down the forest road before it went back in the other side. As always, that's nice.

I also had to slow down a bunch of times because of frogs or toads on the paths. Not sure which ones, it was already after dark. I guesstimate it must have been 60-70 amphibians in total, maybe more. Some of them did not move to the wayside but rather into the middle of the track, right in front of me. Crazy suicide frogs! There were four reeeeaaaallly close calls. I could just avoid stepping on them after they tried to hop right under my boot. Not a centimeter to spare. No toads were harmed during my trip. Phew!

Once I had to stop completely because of the large activity ahead of me. A larger (about the size of half a palm) individual surrounded my foot and then jumped against my heel. Twice! What the heck!? :-D But suuuper cool experience. I'm very glad I actually went out. Totally worth it. I met so many amazing animals. Don't care about the missed sunset a single bit.
@bender Nope, still don't have a mobile one. Works extremely well for me. :-)
Temperatures begin to drop a bit for the night, phew. https://lyse.isobeef.org/abendhimmel-2024-08-30/01.jpg
@movq @bender Haha, great! :-)

Just returned from an evening walk. We met a young slow worm, a bunch of small frogs, about the width of a pointing finger and a bucket load of sweat. It's bloody hot and humid. Also, heaps of ripped trash bags in the forest and lake. :-( No photos, it was too exhausting to even carry my pocket camera.
@movq Lol, someone just showed me this brilliant German (Saxon, actually) song. The title Hitzetodcheck translates to "heat death check". Sorry to all non-German speakers. https://www.youtube.com/watch?v=1qHH3RO_X34

I'm also ready absolutely ready for winter.
@abucci Yep, --open-registrations=false has to also use the equal sign, just like the short form, otherwise it's activated. At least that's consistent. But, is has no effect, if the settings.yaml claims something different.

I have the impression that command line flags only take effect the first time you start yarnd. Unless the option has no pendant in the config file, such as -A/--admin-user $user. Since -A is not a boolean, but takes a string, you are free to use a blank or an equal sign…

It's this package: https://github.com/spf13/pflag?tab=readme-ov-file#command-line-flag-syntax

I also noticed and fixed the typo. 8-)
Yeah, user error on my end, never mind. The persisted settings.yaml overrides the command line arguments. That's surprising to me. I expected the command line options to overrule the config file. Oh well.
@quark :-P
@aelaraji Hmmm, so it _is_ permanent damage. Damn! :-( I'm no electronics guy, but I'd suspect it to be broken, if unplugging it and plugging it in again doesn't fix it. Kinda doubt that a repair shop will get it going again. I have no idea where I would bring it over here. The best I can think of is to ask the volunteers at the repair café if they have any suggestions, I reckon they're not able to fix it either. But that place is only open once a month.

How much of your screen is gone by now? Looks like a lot.

Wow, crazy. A decade ago, I think I only experienced power outages three or maybe four times in my entire life. Since then, they became a bit more frequent. Probably five or six, maybe more. Not sure how many of these events are attributed to construction incidents, where an excavator ripped a power line apart. Last time, loggers threw a tree in an overhead power line, so the power company had to disconnect my area from the grid.
@abucci You can also use -R=false on the command line or leave it out entirely. When explicitly stating -R=false, there has to be an equal sign. With a space (-R false) it's somehow parsed as -R which is equivalent to -R=true. O_o Very weird. I'd really like to see an error instead.

I still have to figure out the precedence of the settings.yaml or command line arguments. I'm probably holding it wrong, but it seems to give me different results…
@quark @movq A general workaround in these cases is to wrap the command in a shell script and reference said script instead.
@aelaraji Oh, shoot! The broken rows are permanent? Looks like you're the king of power outages. :-( How frequent do you experience blackouts?
@movq Phew, that was a close call.
@quark Enjoy! :-)
@aelaraji Bwhahaha! :'-D
@mckinley Wow, I was not aware, that there are different kinds of blackberries. But of course there are. Everything has all sorts of different species, why would it be different with these tasty guys? :-)

I just read up on them and – surprise, surprise – it turns out, the Himalayans are not native to most of Europe either. Doh! It gets even more interesting, their origin is unclear. Maybe Armenia and the Caucasus region. Fascinating!
@movq @prologic Right, what can possibily go wrong!? ;-)

It's funny that you mention it, too. We also were quite surprised that it was incredibly quiet in nature. Not just no man-made noise (we obviously avoided the crowds), not even in the distance, but also hardly any birds. We joked they're still exhausted from the heat of the days before and still resting.
@abucci Thank you for using Lyse's Unofficial Yarnd Help Desk: https://lyse.isobeef.org/tmp/yarnd-disable-registrations.png
@prologic Or c) you click on a link somebody gave you. My brain is a bit dead now, but that might be a problem when you're logged in.
Today, I learned about vim "+normal $", how cool! :-) Thanks @quark!
I just learned from a German documentary that there is goldbeating. Never heard of that term before. Super interesting.
@prologic Great! Git knowledge is helpful in a lot of situations. What's the website about? You both had some fun writing HTML by hand? :-)