# I am the Watcher. I am your guide through this vast new twtiverse.
#
# Usage:
# https://watcher.sour.is/api/plain/users View list of users and latest twt date.
# https://watcher.sour.is/api/plain/twt View all twts.
# https://watcher.sour.is/api/plain/mentions?uri=:uri View all mentions for uri.
# https://watcher.sour.is/api/plain/conv/:hash View all twts for a conversation subject.
#
# Options:
# uri Filter to show a specific users twts.
# offset Start index for quey.
# limit Count of items to return (going back in time).
#
# twt range = 1 70
# self = https://watcher.sour.is?uri=https://twtxt.net/user/caesar/twtxt.txt&offset=70
@prologic Hmm, if I post a message and then it's gone when I reload the page, or if the message I replied to has gone, that's definitely a bug *from the UX point of view* ๐ โฆ but perhaps unavoidable in a distributed system. But since we're both on the same pod, I don't see why that's an issue? Or is this pod itself running on some kind of distributed architecture?
I guess I should go read the code before asking too many questions, but I'm a little puzzled why the same issues with a feed being huge don't present an issue *every time* you want to poll for updates? Particularly with the apparent convention of the newest posts being at the bottom of the file.
As for pagination, sure, it can be hard, but why would it be harder in this case than in the cases where Yarn already does it?
(As for infinite scroll, if you have pagination on the server side already, it's trivial on the client side. Yes you need JS of course, but not a lot)
I also totally get whet you're saying about a twtxt file *potentially* growing to be huge. I guess that, and the fact that it's necessary to work around it with a significant caching architecture, is a major downside to the model of twtxt itself which I hadn't considered.
I'm clearly going to have to take a proper look at the code and get a feeling for the data architecture to understand this! From the outside I have to say if something as simple as "display all of a user's posts" is impossible โ especially when a twtxt file *is* literally a list of all of a user's posts โ it feels like some *very* strange architectural choices must have been madeโฆ but I am also well aware that a lot of painstaking thought by very clever people has gone into this, and I haven't even looked at the code, so don't mind me ๐
@prologic
> philosophical reasons [...] design decision
That I can understand (though to the extent that I understand it, I think I disagree with it ๐). I was asking more about the technical barriers @mutefall mentioned.
> responses are provided from the cache
I see, so we're taking about an architectural limitation in Yarn, rather than twtxt. Still, I know cache invalidation is famously hard, but surely an intentional page load from a user trying to view a feed that isn't (fully) cached is about the best signal you could get to fetch that data from the origin? ๐ค
@prologic
> How do you display this in any reasonable way?
Pagination? Like Yarn uses elsewhere. Or infinite scroll, but from the server side that's still pagination.
> Which is what? To view the entire contents one someoneโs feed? ๐
Exactly. Every other social network has that feature; I've missed it here serveral times already and it looks like I'm not the only one.
I still don't get the difficulty from a technical point of view I'm afraid. ๐ค
@prologic so far as I can see both your and my most recent replies to #x6zqkha just disappeared - a bug?
@prologic Hmm but if you're self-hosting the bridges (the only option I think since they generally have to run on the same machine as the Matrix server) that man in the middle is yourself ๐
Of course you do have to trust the code, but it's all open source.
@mutefall @prologic I don't understand this answer at all from a technical perspective (leaving any philosophical arguments aside). A twtxt file is *literally* a flat file containing a list of all of a person's posts. Surely simply displaying all of that person's posts in Yarn should be the *easiest possible* thing to do, way easier than threading etc. Why would it require "investing heavily in infrastructure" or for the protocol to be "redesigned from the ground up"?
I'm guessing I've misunderstood what you're saying; can you help me understand?
@prologic but if you used those external services directly without bridging, you'd still have to trust all those things, right? Take, say, FB Messenger. Whether I 'bridge' it to Matrix or use Messenger directly, I have to 'trust' Facebook (ha ha, as if! ๐) Same for Signal, WhatsApp, IRC, or anything else you bridge to.
That said, I don't really use bridging much; for the services I tried it for it was too much hassle making the bridge work for it to be worthwhile.
@darch That's my approach, yep ๐ โ but I can also see @prologic's argument that Matrix is over-engineered and current servers are resource hogs and (arguably) hard to get set up...
@prologic I mean, sure, so long as it's fully e2ee and private (Yarn and feeds in general tend to be public...) ๐
@prologic oh it looks quite different on Android ๐
@prologic for real. Sounds like the whole meeting should have just consisted of them sharing that one piece of information, instead of buying it in vaguely reassuring filler text on a screen.
@prologic this is what it should look like (in Element / SchidliChat) 
@prologic what's that client? I don't recognise it
@prologic I think that's approximately what happens behind the scenes, it shouldn't be visible on that easy to the end user, so I guess something else is going wrong... (or bad UI in the client you're using?) ๐ค
@prologic I'm curious about this. Surely the implication of a twtxt file being self-hosted (unless you're using someone else's pod...) is that I control its content; I can delete/edit what I want. Sure, someone else might have saved/cached it, but the same would apply to any web page: if it's on my server, I can delete the canonical version. Doesn't mean every trace is immediately/permanently gone from the web, but any remaining cached versions are just outdated cache artifacts. Am I wrong?
@prologic that's definitely a feature I'd be interested in. Bookmarking to read the RFC later!
@carsten sometimes I seriously wonder whether the cons of the Internet outweigh the pros ๐ฃ
@movq that's exactly what it means ๐
As for clients, I prefer SchildiChat myself, it's an Element fork with a few improvements. I find FluffyChat too basic, but then I never liked WhatsApp either, which I guess it what it's trying to imitate UI-wise.
@prologic Oh for sure. I just would prefer if the twtxt file could be consumed raw inasmuch as possible; that seems to me to be one of the main points of a raw text-based format vs something more structured. But as you say, this doesn't really break that. As I say, a clever workaround to an annoying flaw in the original spec. ๐
@prologic Ah cheers. Pity the original spec doesn't allow real newlines, maybe with indentation or escaping a-la-Markdown to indicate continuation linesโฆ but using \\u2028 is a clever solution to working around that limitation.
@prologic Ah cheers. Pity the original spec doesn't allow real newlines, maybe with indentation or escaping a-la-Markdown to indicate continuation linesโฆ but using \u2028 is a clever solution to working around that limitation.
@prologic I love most of the modern Javascript syntax, including arrow functions (this doesn't include JSX, which *is not Javascript* and I *hate* it ๐) but I do agree that terseness can go too far to the point of getting in the way of readability โ definitely an issue with Python IMO. Honestly the only good thing about Python in my opinion is the ecosystem, particularly for data science.
I do like Go from my very limited experience with it; I will definitely be using it more.
@prologic how do newlines in twts work? I see they don't show up in the raw twtxt.txt (in my browser at least). The twtxt spec seems (?) to forbid actual newlines, so I'm guessing you are using some sort of workaround specific to Yarn?
@prologic Huh, supply chain problems, who'd'a thunk it ๐๐ซ Definitely not going to get better in the short (medium?๐ค) termโฆ
@mckinley I was lucky in a way: I was homeshooled and my studies were very much self-directed. My parents encouraged my interest in tech though they are complete muggles themselves and couldn't teach me anything about it, so I was entirely self-taught โ like many geeks, it seems. As for schools, I do think the situation is improving, at least from what I've heard from friends with school-age kids. @prologic's experience is reassuring. I'm sure it varies hugely from area to area though; it definitely needs to be a part of national curricula.
@prologic
> youโll love what my company is trying to do here
I'm intriguedโฆ look forward to hearing more!
@ullarah Ugh, please no! ๐ซ As a user I hate those interstitial pages, because they add an unwanted step between me and the page I'm trying to get to, and they obscure the real target of the link (also they're often used for user tracking). And as a web geek I hate the fact that they break the semantic model of a link pointing to its real target, turning external links into faux internal ones.
@prologic I *think* the "worst that could happen" is "it would be semantically wrong". I don't think it could ever be actively harmful, since it is correctly treated as a *claim*, not proof of ownership of the target โ it can be used to prove ownership of the *origin* page, but not of the *destination*.
That said, I would be in favour of making it explicit (ideally in the Metadata spec) that "User Links" SHOULD be "connected to the feed or author", not just "usually". This would make the link metadata more semantically useful.
@david I'm not sure what you mean. rel="me" is just an html attribute which makes a *claim* that the target of the link belongs to the same person. It doesn't *prove* it; unless it's reciprocal.
@prologic Hmm, short of "clarifying" the spec to specifically state that links *must* be to the user's own sites, it's hard to think of a universal solution. I think I'd still err towards *assuming* that links are to the user's own pages, since I think they will be in almost all cases, but obviously there's an argument to be made against making that assumption, tooโฆ
@prologic I think in those days "everyone" on the internet was a geek who loved doing things themselves. Now the internet is used by *literally* everyone, and most of them don't understand how it works any better than how their car works. It has to Just Work.
I guess whatโs needed is for self-hosting to *be* one of those things that Just Works, without the average person having to know how. (In addition to educating the public better about what the internet is, of course.)
@prologic Ah sorry, gotcha. ๐
Hmm, you make an interesting point. I would assume that *most* links a user would add would be to their own profiles, but maybe not all?
The Metadata spec says "A link to some other resource which is often connected to the feed or author".
I think my inclination would be yes, add it to all of them, but I can also see that a user *could* put links there that aren't their own. ๐ค
@prologic
> > People donโt want to run their own servers, and never will.
>
> ๐ ๐ ๐
On the face of it it's a generalisation, but s/People/99.99% of people/ and the statement becomes objectively true.
My opinion on decentralised communications protocols is basically: Being *able* to run your own instance - *easily* - is very, very important. But being *required* to run your own instance dooms the system to failure / being very niche at best. Mastodon is a great example which fails at both; it's hard to self-host and there's no obvious canonical instance to sign up to if you don't want to host your own.
@prologic another nice example is how Mastodon show's a user's external link to their own website as "verified" if the target site includes a rel="me" link back to their Mastodon profile, since this "proves" that the person has control of the linked site. I think I've seen other sites use it for verification in the same way.
@prologic If you check out someone's GitHub profile that has a link to that person's own website, for example you'll find rel="me" on the link.
@prologic The `me` value of the HTML `rel` attribute indicates that the linked page is the same user's profile on another site. It's useful for identity consolidation, enabling things like RelMeAuth on the IndieWeb. So it would be nice to be able to have that attribute on links to one's own website (and to one's profiles on other sites) from the "User Links" section of our profiles here.
@prologic Maybe it's harder to get Dendrite working properly (it's pre-release software after all I guess) but I never had any issue with Synapse, I had it up and running in ~15 minutes. The issue I did have was that it's a resource hog (mainly because of backfilling large federated rooms) but a few tweaks to caching settings improved that and I've been running for over a year with no issues at all.
I do agree it seems over-engineered in many ways, I must say. But overall I've been really happy with it and it's now one of my primary communication platforms.~
@prologic Hmm I don't recall any trouble getting federation working (with Synapse), what issue are you having? The federation tester is failing on your domain for me; it's getting a timeout:
> Get "https://104.21.85.152:8448/_matrix/key/v2/server": context deadline exceeded
@prologic I am on Matrix, @caesar:schinas.net.
I'm running Synapse; I'd be very interested to hear your experience with Dendrite as I've been considering switching over as soon as push notifications make it to a published release.
@screem Wow haha I'd never thought of Android as having much in the way of customisation options โ though I guess some vendors add a lot of that sort of stuff. I'm not one to customise my operating systems much โ I like good clean defaults and I don't really like to change them. I love macOS because I feel it hits the sweet spot there, but I loathe iOS because it's just too locked down. I don't care for UI customisation, but how can you forbid me to install my own choice of *web browser* FFS. Or anything else not blessed by the gods at Apple.
@prologic
> Decentralisation starts with decentering yourself.
๐ฏ Great quote. There's a chicken and egg problem, and those of us who understand the problem and are happy to be early adopters need to be the ones pushing forward hatching more and more eggs until that chicken appears (umm ok idk if that analogy makes sense but you get my drift ๐)
@mutefall
> decentralised and distributed (we likely are here now)
Nah. Unless by *we* you mean us here on Yarn. But as a society, I reckon we're *at least* one or two steps before that โ I'd go with your "netizens asking why?" at best. Much of society still hasn't even got to that point honestly.
I wonder how niche the market would actually be? I feel like half the people I know would love a phone that didn't mean either selling your soul to the world's biggest advertising company or being locked into an expensive walled garden by the world's most profitable brand.
@prologic I'm not so sure there'd be a lack of good apps, I don't think the situation on the desktop translates well to mobile. Google is pushing flutter pretty hard for mobile and it's getting pretty good adoption, and it's inherently cross-platform - Canonical are even using it for desktop apps.
@mutefall Cheers, that's good to know. Both of them do look like good options; I would have gone with Mailbox were it not for the hassle with multiple accounts at the same domain (they seem to be trying to force users who want multiple accounts onto their enterprise pricing). I've played with a free Mailfence account and it seems pretty good (better than Proton as it has real PGP support including automatic WKD lookups).
@screem The problem is this bit:
> All mailbox.org accounts have their own, individual security key. This makes it possible to associate external domains to several mailbox.org accounts, if desired, simply by adding their different security keys to the DNS configuration.
Meaning I can't just create accounts for each of my family at my domain from an admin panel; rather, they have to create their own accounts, find their "individual security key", give it to me, and I have to add a new TXT record to the DNS for every account. Way too much hassle ๐
@prologic That's very interesting that you were able to see the acquisition from the inside! And yes, I know WhatsApp now runs on the same infrastructure as the rest of Meta's properties. But then that's the whole thing about e2ee โ so long as you can trust the encryption, you don't need to trust the servers / network. (Metadata being the obvious exception).
But again, despite what I'm saying here, I am no WhatsApp fan. I avoid it as much as I can, and try to encourage my friends and family to switch to better alternatives.
@prologic Basically, my position is that of the security professionals cited in the articles I linked: yes, there are much, _much_ better choices than WhatsApp, but it's _actively harmful_ to overstate its flaws / tell people it's insecure without giving full context, because for the vast majority of cases (even for journalists in repressive regimes) it's _secure enough_, whereas often the alternatives people switch to are far _less_ secure.
Not to say that's the case here, obviously this community understands the context and alternatives in much more detail than most.
@prologic True, Signal LLC claim not to store metadata (and I believe them), but the protocol _requires_ that metadata to pass through their servers, and as a US-based company they could _theoretically_ be forced to store that metadata without being allowed to inform their users.
As for WhatsApp, much as I loathe Facebook and everything they touch, I do believe the Signal Protocol is correctly implemented (it's been audited by security professionals).
(This feels weird, normally I'm trying to persuade people not to use WhatsApp ๐
but accuracy is important here.)
@prologic Not true, they use the Signal protocol (well, admittedly you have to take their word for that, since the app is closed sourceโฆ). But FB does indeed have access to the metadata โ who you're talking to and when โ which is why I do my best to avoid it. (Signal have access to the same metadata if you use Signal, but I trust them better with itโฆ I'd still rather nobody had it, which is why I prefer Matrix.)
More info here and here.
@mutefall @carsten Any experience with Mailbox or Mailfence (or anything else good) with a custom domain? I'm looking to move my family's email away from Google Apps. Was quite excited about Mailbox but then I discovered it seems to be infeasible to have multiple accounts with them using your own domain, since they require each account to individually verify domain ownership through DNSโฆ Considering Mailfence now.
@screem Matrix whenever possible (ie, close friends and family who are willing to set up an account just to talk to meโฆ I host my own instance for my family), Signal as a second choice, but unfortunately WhatsApp is the de-facto way to communicate where I live. Still, could be worse, at least it's e2ee (in theory / if you believe them). The one I absolutely refuse to use is FB Messengerโฆ
What's the closest alternative to "retweeting" something on twtxt? Simply "replying" to a post?
@prologic I'll put some thought into what I think would be ideal for my use case (I'm not totally sure yet) and open an issue on Gitea.
@prologic send some of that my way, we've got a drought over here ๐ฌ
Yes I'd be happy to help build it out, I have next to no experience with Go, but that's a plus for me because it's an excuse to learn it ;-) Would you rather we discussed over on Gitea?
Hmm that's a good point about potentially wanting to customise the preamble. Does Yarn work as a frontend to a static twtxt.txt file or is it keeping everything in the DB (I see it uses one) and generating the twtxt files on-the-fly? In the former case, I imagine it wouldn't be hard to add a config option to use a different path? Or maybe I'm talking nonsense because I haven't looked into how this all works yet...
Very sad (though obviously nothing compared to the tragedy of the war itself).
Maybe something as simple as an HTTP redirect from example.com/twtxt.txt to yarn.example.com/users/caesar/twtxt.txt? ๐ค But I'm sure I'm not the first to want to do this so is there a 'canonical way' to do it?
@prologic Ah that's good to know. I have to look more into how discovery works etc, but is there any issue with using a root domain for my username (handle / whatever you call it here?) but hosting Yarn.social on a subdomain?
So my username might be @caesar@example.com but Yarn.social would be at something like yarn.example.com so that I could host a normal website at example.com. Does that make sense?
@prologic Roughly speakingโฆ western Europe ๐ But I'm all over the place ๐
@prologic Haha thanks. I'm not normally big on microblogging but thought I'd give it a try. If I get into it I'd hope to end up running my own instance / something else compatible (probably single-user so maybe my own instance of Yarn.social itself is overkill??)
@prologic Thanks! ๐ค I'm new to twtxt and Yarn.social, but I love the idea of it!