# I am the Watcher. I am your guide through this vast new twtiverse.
#
# Usage:
# https://watcher.sour.is/api/plain/users View list of users and latest twt date.
# https://watcher.sour.is/api/plain/twt View all twts.
# https://watcher.sour.is/api/plain/mentions?uri=:uri View all mentions for uri.
# https://watcher.sour.is/api/plain/conv/:hash View all twts for a conversation subject.
#
# Options:
# uri Filter to show a specific users twts.
# offset Start index for quey.
# limit Count of items to return (going back in time).
#
# twt range = 1 196278
# self = https://watcher.sour.is?offset=171599
# next = https://watcher.sour.is?offset=171699
# prev = https://watcher.sour.is?offset=171499
But this is no different to how jenny does things with storing every Twt in a Maildir I suppose? 🤔
This has specifically come up before in the form of "informal complaints" against yarnd because of the way it permanently stores and archives Twts, so even if you decide you changed your mind, or deleted that line out of your feed, if my pod or @xuu or @abucci or @eldersnake (_or any other handful of pods still around?_) saw the Twt, it'd be permanently archived._
This has specifically come up before in the form of "informal complaints" against yarnd because of the way it permanently stores and archives Twts, so even if you decide you changed your mind, or deleted that line out of your feed, if my pod or @xuu or @abucci or @eldersnake (_or any other handful of pods still around?_) saw the Twt, it'd be permanently archived._
Yeah I'm curious to find out too beyond just "here say". But regardless of whether we should or shouldn't care about this or should or shouldn't comply. We should IMO. I'd have to build something that horrendously violates someone's rights in another country.
Yeah I'm curious to find out too beyond just "here say". But regardless of whether we should or shouldn't care about this or should or shouldn't comply. We should IMO. I'd have to build something that horrendously violates someone's rights in another country.
@movq Care to explain how this explicit/attack works for me? 🤣
@movq Care to explain how this explicit/attack works for me? 🤣
Well that was bloody awful. This PR bokr my pod for some strange reason I can't figure out why or how 😱 The process just kept getting terminated from something, somewhere (_no panic_). weird. I've reverted this PR for now @xuu
Well that was bloody awful. This PR bokr my pod for some strange reason I can't figure out why or how 😱 The process just kept getting terminated from something, somewhere (_no panic_). weird. I've reverted this PR for now @xuu
@lyse Yeah, makes sense. You don’t even need hash collisions for that. 🤔 (I guess only individually signed twts would prevent that. 🙈 Yet another can of worms.)
@lyse Yeah, makes sense. You don’t even need hash collisions for that. 🤔 (I guess only individually signed twts would prevent that. 🙈 Yet another can of worms.)
@lyse Yeah, makes sense. You don’t even need hash collisions for that. 🤔 (I guess only individually signed twts would prevent that. 🙈 Yet another can of worms.)
@lyse Yeah, makes sense. You don’t even need hash collisions for that. 🤔 (I guess only individually signed twts would prevent that. 🙈 Yet another can of worms.)
@falsifian I’m curious myself now and might look it up (or even ask some of our legal guys/gals 😅).
I *think* none of this matters to people outside the EU anyway. These aren’t your laws. Even if you were to start a company in the US, it would only be a marketing instrument for you: “Hey, look, we follow GDPR!” EU people might then be more inclined to become your customers. But that’s it.
That said, I’m not sure anymore if there are any *other* treaties between the EU and the US which cover such things …
@falsifian I’m curious myself now and might look it up (or even ask some of our legal guys/gals 😅).
I *think* none of this matters to people outside the EU anyway. These aren’t your laws. Even if you were to start a company in the US, it would only be a marketing instrument for you: “Hey, look, we follow GDPR!” EU people might then be more inclined to become your customers. But that’s it.
That said, I’m not sure anymore if there are any *other* treaties between the EU and the US which cover such things …
@falsifian I’m curious myself now and might look it up (or even ask some of our legal guys/gals 😅).
I *think* none of this matters to people outside the EU anyway. These aren’t your laws. Even if you were to start a company in the US, it would only be a marketing instrument for you: “Hey, look, we follow GDPR!” EU people might then be more inclined to become your customers. But that’s it.
That said, I’m not sure anymore if there are any *other* treaties between the EU and the US which cover such things …
@falsifian I’m curious myself now and might look it up (or even ask some of our legal guys/gals 😅).
I *think* none of this matters to people outside the EU anyway. These aren’t your laws. Even if you were to start a company in the US, it would only be a marketing instrument for you: “Hey, look, we follow GDPR!” EU people might then be more inclined to become your customers. But that’s it.
That said, I’m not sure anymore if there are any *other* treaties between the EU and the US which cover such things …
@prologic I have no specifics, only hopes. (I have seen some articles explaining the GDPR doesn't apply to a "purely personal or household activity" but I don't really know what that means.)
I don't know if it's worth giving much thought to the issue unless either you expect to get big enough for the GDPR to matter a lot (I imagine making money is a prerequisite) or someone specifically brings it up. Unless you enjoy thinking through this sort of thing, of course.
Really though I only managed to save a few GB, but it's enough for now. 
Really though I only managed to save a few GB, but it's enough for now.
@bender Haha 😛 Faster? Maybe 🤔 But yeah it's good to have backups! (_that work_)
@bender Haha 😛 Faster? Maybe 🤔 But yeah it's good to have backups! (_that work_)
I've also put up this PR [Add compatible methods for Index to behave as the Archiver (transition) #1177
](https://git.mills.io/yarnsocial/yarn/pulls/1177) that will act as a transition from the old naive archiver to the new bluge-based search/index. I will switch my pod over to this soon to test it before anyone else does.
I've also put up this PR [Add compatible methods for Index to behave as the Archiver (transition) #1177
](https://git.mills.io/yarnsocial/yarn/pulls/1177) that will act as a transition from the old naive archiver to the new bluge-based search/index. I will switch my pod over to this soon to test it before anyone else does.
For those curious, the archive on this pod had reached around ~22GB in size. I had to suck it down to my more powerful Mac Studio to clean it up and remove a bunch of junk. Then copy all the data back. This is what my local network traffic looked like for the last few hours 😱 
~
For those curious, the archive on this pod had reached around ~22GB in size. I had to suck it down to my more powerful Mac Studio to clean it up and remove a bunch of junk. Then copy all the data back. This is what my local network traffic looked like for the last few hours 😱 ~
@prologic woot, woot! Glad everything went well. I feel it faster already!
And we're back. Sorry about that 😅
And we're back. Sorry about that 😅
🧮 USERS:1 FEEDS:2 TWTS:1099 ARCHIVED:79147 CACHE:2577 FOLLOWERS:17 FOLLOWING:14
Gotta unplug for a couple of minutes. I'm suspecting the extension cord to be the root of my monitor dead rows of pixels and flickering problems.
Gotta unplug for a couple of minutes. I'm suspecting the extension cord to be the root of my monitor dead rows of pixels and flickering problems.
Gotta unplug for a couple of minutes. I'm suspecting the extension cord to be the root of my monitor dead rows of pixels and flickering problems.
@lyse Hmmm I'm not sure sure I get what you're getting at here. In order for this to be true, yarnd would have to be maliciously fabricating a Twt with the Hash D.
@lyse Hmmm I'm not sure sure I get what you're getting at here. In order for this to be true, yarnd would have to be maliciously fabricating a Twt with the Hash D.
i.e: there must be two versions of the Twt in the feed.
i.e: there must be two versions of the Twt in the feed.
@lyse This is true. But the client MUST supply the original too! Or this doesn't work 😢
@lyse This is true. But the client MUST supply the original too! Or this doesn't work 😢
Have a nice weekend everyone
@prologic Let me try:
Invent anything you want, say feed A writes message text B at timestamp C. You simply create the hash D for it and reply to precisely that D as subject in your own feed E with your message text F at timestamp G. This gets hashed to H.
Now then, some a client J fetches your feed E. It sees your response from time G with text F where in the subject you reference hash D. Since client J does not know about hash D, it simply asks some peers about it. If it happens to query your yarnd for it, you could happily serve it your invention: "You wanna know about hash D? Oh, that's easy, feed A wrote B at time C."
The client J then verifies it and since everthing lines up, it looks legitimate and puts this record in its cache or displays it to the user or whatever. It does not even matter, if the client J follows feed A or not. The message text B at C with hash D could have just deleted or edited in the meantime.
Congrats, you successfully spread rumors. :-D
@prologic This does not hold if the edit happened before I even got the original.
If OTOH your client doesn't store individual Twts in a cache/archive or some kind of database, then verification becomes quite hard and tedious. However I think of this as an implementation details. The spec should just call out that clients must validate/verify the edit request and the matching hash actually exists in that feed, not how the client should implement that.
If OTOH your client doesn't store individual Twts in a cache/archive or some kind of database, then verification becomes quite hard and tedious. However I think of this as an implementation details. The spec should just call out that clients must validate/verify the edit request and the matching hash actually exists in that feed, not how the client should implement that.
@lyse Yes you do. You keep both versions in your cache. They have different hashes. So you have Twt A, a client indicates Twt B is an edit of A, your client has already seen A and cached and archived it, now your client fetches B which is indicated of editing A. You cache/archive B as well, but now indicate in your display that B replaces A (_maybe display, link both_) or just display B or whatever. But essentially you now have both, but an indicator of one being an edit of the other.
The right thing to do here of course is to keep A in the "thread" but display B. Why? So the thread/chain doesn't actually break or fork (_forking is a natural consequence of editing, or is it the other way around? 🤔_)._
@lyse Yes you do. You keep both versions in your cache. They have different hashes. So you have Twt A, a client indicates Twt B is an edit of A, your client has already seen A and cached and archived it, now your client fetches B which is indicated of editing A. You cache/archive B as well, but now indicate in your display that B replaces A (_maybe display, link both_) or just display B or whatever. But essentially you now have both, but an indicator of one being an edit of the other.
The right thing to do here of course is to keep A in the "thread" but display B. Why? So the thread/chain doesn't actually break or fork (_forking is a natural consequence of editing, or is it the other way around? 🤔_)._
@lyse I'm all for dropping delete btw, Or at least not making it mandatory, as-in "clients should" rather than "clients must". But yes I agree, let's explore all the possible ways this can be exploited (_if at all_).
@lyse I'm all for dropping delete btw, Or at least not making it mandatory, as-in "clients should" rather than "clients must". But yes I agree, let's explore all the possible ways this can be exploited (_if at all_).
@movq I think not.
> What about edits of edits? Do we want to “chain” edits or does the latest edit simply win?
This gets too complicated if we start to support this kind of nonsense 🤣
@movq I think not.
> What about edits of edits? Do we want to “chain” edits or does the latest edit simply win?
This gets too complicated if we start to support this kind of nonsense 🤣
@lyse Walk me through this? 🤔 I get what you're saying, but I'm too stupid to be a "hacker" 🤣
@lyse Walk me through this? 🤔 I get what you're saying, but I'm too stupid to be a "hacker" 🤣
But yes, at the end of the day if the edit request is invalid or cannot be verified, it should be ignored as treated as "malicious".
But yes, at the end of the day if the edit request is invalid or cannot be verified, it should be ignored as treated as "malicious".
@lyse @movq So a client that has the idea of a cache/archive wouldn't necessarily have to re-check that the Twt being marked as "edited" belongs to that feed or not, the client would already know that for sure. At least this is how yarnd works and I'm sure jenny can make similar assertions too.
@lyse @movq So a client that has the idea of a cache/archive wouldn't necessarily have to re-check that the Twt being marked as "edited" belongs to that feed or not, the client would already know that for sure. At least this is how yarnd works and I'm sure jenny can make similar assertions too.
@lyse @falsifian Contributions to search.twtxt.net, which runs yarns (_not to be confused with yarnd_) are always welcome 🤗 -- I don't have as much "spare time" as I used to due to the nature of my job (_Staff Engineer_); but I try to make improvements every now and again 💪
@lyse @falsifian Contributions to search.twtxt.net, which runs yarns (_not to be confused with yarnd_) are always welcome 🤗 -- I don't have as much "spare time" as I used to due to the nature of my job (_Staff Engineer_); but I try to make improvements every now and again 💪
@falsifian You make good points though, I made similar arguments about this too back in the day. Twtxt v2 / Yarn.social being at least ~4 years old now 😅~
@falsifian You make good points though, I made similar arguments about this too back in the day. Twtxt v2 / Yarn.social being at least ~4 years old now 😅~
@falsifian Do you have specifics about the GRPD law about this?
> Would the GDPR would apply to a one-person client like jenny? I seriously hope not. If someone asks me to delete an email they sent me, I don’t think I have to honour that request, no matter how European they are.
I'm not sure myself now. So let's find out whether parts of the GDPR actually apply to a truly decentralised system? 🤔
@falsifian Do you have specifics about the GRPD law about this?
> Would the GDPR would apply to a one-person client like jenny? I seriously hope not. If someone asks me to delete an email they sent me, I don’t think I have to honour that request, no matter how European they are.
I'm not sure myself now. So let's find out whether parts of the GDPR actually apply to a truly decentralised system? 🤔
LOL 😂 This:
> anyone could claim that some feed contained a certain message which was then removed again by just creating the hash over the fake message in said feed and invented timestamp themselves
I'd like to see a step-by-step reproduction of this. I don't buy it 🤣
Admittedly yarnd had a few implementation security bugs, but I'm not sure this is actually possible, unless I'm missing something? 🤔
LOL 😂 This:
> anyone could claim that some feed contained a certain message which was then removed again by just creating the hash over the fake message in said feed and invented timestamp themselves
I'd like to see a step-by-step reproduction of this. I don't buy it 🤣
Admittedly yarnd had a few implementation security bugs, but I'm not sure this is actually possible, unless I'm missing something? 🤔
And they have arrived (well, they did around 3 hours ago, LOL). Buttery smooth, my 16 Pro (one with dark cover). It took a bit over an hour to transfer all my data.
iPhones 16, and 16 Pro
And they have arrived (well, they did around 3 hours ago, LOL). Buttery smooth, my 16 Pro (one with dark cover). It took a bit over an hour to transfer all my data.
iPhones 16, and 16 Pro
[47°09′46″S, 126°43′09″W] Not enough data -- sampling finished
Ah, and now he is "conveniently" sleeping. How, well, convenient! LOL.
Ah, and now he is "conveniently" sleeping. How, well, convenient! LOL.
@lyse yeah, tell us, @prologic, what isn't true? 🤔 You can't just go around, "that's not true, and that's not true; and that, and that!" without spelling out exactly what isn't, and why? For the love of god, why?! 😂
@lyse yeah, tell us, @prologic, what isn't true? 🤔 You can't just go around, "that's not true, and that's not true; and that, and that!" without spelling out exactly what isn't, and why? For the love of god, why?! 😂
@falsifian Something similar exists over at https://search.twtxt.net/. But a usable search engine would be actually nice (to be fair, yarns improved a bit). :-) I don't care about feed changes over time. In fact, it would even feel creepy to me. Of course, anyone could still surveil, but I'm not looking forward to these stats.
@movq We could still let the client display a warning if it cannot verify it. But yeah.
@falsifian comments on the feeds as in nick, url, follow, that kind of thing? If that, then not interested at all. I envision an archive that would allow searching, and potentially browsing threads on a nice, neat interface. You will have to think, though, on other things. Like, what to do with images? Yarn allows users to upload images, but also embed it in twtxts from other sources (hotlinking, actually).
@falsifian comments on the feeds as in nick, url, follow, that kind of thing? If that, then not interested at all. I envision an archive that would allow searching, and potentially browsing threads on a nice, neat interface. You will have to think, though, on other things. Like, what to do with images? Yarn allows users to upload images, but also embed it in twtxts from other sources (hotlinking, actually).
@david Thanks, that's good feedback to have. I wonder to what extent this already exists in registry servers and yarn pods. I haven't really tried digging into the past in either one.
How interested would you be in changes in metadata and other comments in the feeds? I'm thinking of just permanently saving every version of each twtxt file that gets pulled, not just the twts. It wouldn't be hard to do (though presenting the information in a sensible way is another matter). Compression should make storage a non-issue unless someone does something weird with their feed like shuffle the comments around every time I fetch it.
I ended up installing Headscale on my little VPS. Just in case the collide, I turned off WireGuard. Turning that one off (which ran on a container) also frees some memory. Headscale is running quite well! Indeed, I have struggled getting any web management console to work, but it really isn't needed. Everything needed to commandeer the server is available through the CLI.
I ended up installing Headscale on my little VPS. Just in case the collide, I turned off WireGuard. Turning that one off (which ran on a container) also frees some memory. Headscale is running quite well! Indeed, I have struggled getting any web management console to work, but it really isn't needed. Everything needed to commandeer the server is available through the CLI.
@falsifian "*I was actually thinking about making an Internet Archive style twtxt archiver, letting you explore past twts*" --- that's an awesome idea for a project. Something I would certainly use!
@falsifian "*I was actually thinking about making an Internet Archive style twtxt archiver, letting you explore past twts*" --- that's an awesome idea for a project. Something I would certainly use!
@prologic Just what @bender did. :-D If he'd additionally serve the fake message from his yarnd twt endpoint, everybody querying that hash from him (or any other yarnd that synced it in the meantime) would believe, that I didn't like Australians.
In fact, I really don't. I love'em! 8-)
We would need to sign each message in a feed, so others could verify that this was actually part of that feed and not made up. But then we end up in the crypto debate for identities again, which I'm not a big fan of. :-)
I just want to highlight, one might get a false sense of message authenticity, if one just briefly looks at the hashes.
[47°09′51″S, 126°43′39″W] Taking samples
@lyse I think that’s what we would *have to* enforce – otherwise we’d run into the problem you’ve outlined. 😃
@lyse I think that’s what we would *have to* enforce – otherwise we’d run into the problem you’ve outlined. 😃
@lyse I think that’s what we would *have to* enforce – otherwise we’d run into the problem you’ve outlined. 😃
@lyse I think that’s what we would *have to* enforce – otherwise we’d run into the problem you’ve outlined. 😃
@falsifian I think we’re talking about different ideas here. 🤔
Maybe it’s time to draft all this into a spec or, rather, two different specs. I might do that over the weekend.
@falsifian I think we’re talking about different ideas here. 🤔
Maybe it’s time to draft all this into a spec or, rather, two different specs. I might do that over the weekend.
@falsifian I think we’re talking about different ideas here. 🤔
Maybe it’s time to draft all this into a spec or, rather, two different specs. I might do that over the weekend.
@falsifian I think we’re talking about different ideas here. 🤔
Maybe it’s time to draft all this into a spec or, rather, two different specs. I might do that over the weekend.