# I am the Watcher. I am your guide through this vast new twtiverse.
#
# Usage:
# https://watcher.sour.is/api/plain/users View list of users and latest twt date.
# https://watcher.sour.is/api/plain/twt View all twts.
# https://watcher.sour.is/api/plain/mentions?uri=:uri View all mentions for uri.
# https://watcher.sour.is/api/plain/conv/:hash View all twts for a conversation subject.
#
# Options:
# uri Filter to show a specific users twts.
# offset Start index for quey.
# limit Count of items to return (going back in time).
#
# twt range = 1 60515
# self = https://watcher.sour.is?uri=https://twtxt.net/user/prologic/twtxt.txt&offset=57015
# next = https://watcher.sour.is?uri=https://twtxt.net/user/prologic/twtxt.txt&offset=57115
# prev = https://watcher.sour.is?uri=https://twtxt.net/user/prologic/twtxt.txt&offset=56915
@movq Hmmm not sure what I was thinking sorry π€¦ββοΈbeen a long day π
@movq Hmmm not sure what I was thinking sorry π€¦ββοΈbeen a long day π
@movq Am I missing something? π
@movq Am I missing something? π
@movq Is t it? You read each Twt and compute its hash. It's a simple O(1) lookup of the hash in that feed or your cache/archive right?
@movq Is t it? You read each Twt and compute its hash. It's a simple O(1) lookup of the hash in that feed or your cache/archive right?
π Reminder that next Saturday 28th September will be out monthly online meetup! Hope to see some/all of you there π
π Reminder that next Saturday 28th September will be out monthly online meetup! Hope to see some/all of you there π
I'll try to reproduce locally later tonight
I'll try to reproduce locally later tonight
@lyse I don't think this is true.
@lyse I don't think this is true.
@lyse No that's never a problem because we really only want to "navigate" the web anyway not form threads of xonversation π€£
@lyse No that's never a problem because we really only want to "navigate" the web anyway not form threads of xonversation π€£
@movq this approach also wouldn't work and when that Feed gets archived so you'll be forced to crawl archived feeds at that point.
@movq this approach also wouldn't work and when that Feed gets archived so you'll be forced to crawl archived feeds at that point.
The important bits missing from this summary (_devil is in the details_) are two requirements:
- Clients _should_ order Twts by their timestamp.
- Clients *must* validate all edit and delete requests that the hash being indicated belongs to and came from that feed.
- Client _should_ honour delete requests and delete Twts from their cache/archive.
The important bits missing from this summary (_devil is in the details_) are two requirements:
- Clients _should_ order Twts by their timestamp.
- Clients *must* validate all edit and delete requests that the hash being indicated belongs to and came from that feed.
- Client _should_ honour delete requests and delete Twts from their cache/archive.
@lyse This is why hashes provide that level of integrity. The hash can be verified in the cache or archive as belonging to said feed.
@lyse This is why hashes provide that level of integrity. The hash can be verified in the cache or archive as belonging to said feed.
@movq I think the order of the lines in a feed don't matter as long as we can guarantee the order of Twts. Clients should already be ordering by Timestamp anyway.
@movq I think the order of the lines in a feed don't matter as long as we can guarantee the order of Twts. Clients should already be ordering by Timestamp anyway.
@lyse Sorry could you explain this sifferently?
@lyse Sorry could you explain this sifferently?
Do you k ow what you clicked on before going back?
Do you k ow what you clicked on before going back?
@eldersnake Sweet thank you! πββοΈ I'll merge this PR tonight I think.
@eldersnake Sweet thank you! πββοΈ I'll merge this PR tonight I think.
e.g: Shutdown yarnd and cp -a yarn.db yarn.db.bak before testing this PR/branch.
e.g: Shutdown yarnd and cp -a yarn.db yarn.db.bak before testing this PR/branch.
Can I get someone like maybe @xuu or @abucci or even @eldersnake -- If you have some spare time -- to test this yarnd PR that upgrades the Bitcask dependency for its internal database to v2? π
VERY IMPORTANT If you do; Please Please Please backup your yarn.db database first! π
Heaven knows I don't want to be responsible for fucking up a production database here or there π€£
Can I get someone like maybe @xuu or @abucci or even @eldersnake -- If you have some spare time -- to test this yarnd PR that upgrades the Bitcask dependency for its internal database to v2? π
VERY IMPORTANT If you do; Please Please Please backup your yarn.db database first! π
Heaven knows I don't want to be responsible for fucking up a production database here or there π€£
nevermind; I _think_ this might be some changes internally in Go 1.23 and a dependency I needed to update π€
nevermind; I _think_ this might be some changes internally in Go 1.23 and a dependency I needed to update π€
Location Addressing is fine in smaller or single systems. But when you're talking about large decentralised systems with no single point of control (_kind of the point_) things like independable variable integrity become quite important.
Location Addressing is fine in smaller or single systems. But when you're talking about large decentralised systems with no single point of control (_kind of the point_) things like independable variable integrity become quite important.
What is being proposed as a counter to content-addressing is called location-addressing. Two very different approaches, both with pros/cons of course. But a local cannot be verified, the content cannot be be guaranteed to be authenticate in any way, you just have to implicitly trust that the location points to the right thing.
What is being proposed as a counter to content-addressing is called location-addressing. Two very different approaches, both with pros/cons of course. But a local cannot be verified, the content cannot be be guaranteed to be authenticate in any way, you just have to implicitly trust that the location points to the right thing.
For example, without content-addressing, you'd never have been able to find let alone pull up that ~3yr old Twt of me (_my very first_), hell I'd even though I lost my first feed file or it became corrupted or something π€£ -- If that were the case, it would actually be possible to reconstruct the feed and verify every single Twt against the caches of all of you π€£~
For example, without content-addressing, you'd never have been able to find let alone pull up that ~3yr old Twt of me (_my very first_), hell I'd even though I lost my first feed file or it became corrupted or something π€£ -- If that were the case, it would actually be possible to reconstruct the feed and verify every single Twt against the caches of all of you π€£~
@david I _really_ thinks articles like this explain the benefits far better than I can.
@david I _really_ thinks articles like this explain the benefits far better than I can.
@david Witout including the content, it's no longer really "content addressing" now is it? You're essentially only addressing say nick+timestamp or url+timestamp.
@david Witout including the content, it's no longer really "content addressing" now is it? You're essentially only addressing say nick+timestamp or url+timestamp.
Speaking of AI tech (_sorry!_); Just came across this really cool tool built by some engineers at Googleβ’ (_currently completely free to use without any signup_) called NotebookLM π Looks really good for summarizing and talking to document π
Speaking of AI tech (_sorry!_); Just came across this really cool tool built by some engineers at Googleβ’ (_currently completely free to use without any signup_) called NotebookLM π Looks really good for summarizing and talking to document π
@eldersnake Yeah I'm looking forward to that myself π€£ It'll be great to see where technology grow to a level of maturity and efficiency where you can run the tools on your own PC or Device and use it for what, so far, I've found it to be somewhat decent at; Auto-Complete, Search and Q&A.
@eldersnake Yeah I'm looking forward to that myself π€£ It'll be great to see where technology grow to a level of maturity and efficiency where you can run the tools on your own PC or Device and use it for what, so far, I've found it to be somewhat decent at; Auto-Complete, Search and Q&A.
@sorenpeter I really don't think we can ignore the last ~3 years and a bit of this threading model working quite well for us as a community across a very diverse set of clients and platforms. We cannot just drop something that "mostly works just fine" for the sake of "simplicity". We have to weight up all the options. There are very real benefits to using content addressing here that really IMO shouldn't be disregarded so lightly that actually provide a lot of implicit value that users of various clients just don't get to see. I'd recommend reading up on the ideas behind content addressing before simply dismissing the Twt Hash spec entirely, it wasn't even written or formalised by me, but I understand how it works quite well π
The guy that wrote the spec was (is?) way smarter than I was back then, probably still is now π€£~
@sorenpeter I really don't think we can ignore the last ~3 years and a bit of this threading model working quite well for us as a community across a very diverse set of clients and platforms. We cannot just drop something that "mostly works just fine" for the sake of "simplicity". We have to weight up all the options. There are very real benefits to using content addressing here that really IMO shouldn't be disregarded so lightly that actually provide a lot of implicit value that users of various clients just don't get to see. I'd recommend reading up on the ideas behind content addressing before simply dismissing the Twt Hash spec entirely, it wasn't even written or formalised by me, but I understand how it works quite well π
The guy that wrote the spec was (is?) way smarter than I was back then, probably still is now π€£~
@falsifian Right I see. Yeah maybe we want to avoid that π€£ I do kind of tend to agree with @xuu in another thread that there isn't actually anything wrong with our use of Blake2 at all really, but we may want to consider all our options.
@falsifian Right I see. Yeah maybe we want to avoid that π€£ I do kind of tend to agree with @xuu in another thread that there isn't actually anything wrong with our use of Blake2 at all really, but we may want to consider all our options.
@xuu I don't think this is a lextwt problem tbh. Just the Markdown aprser that yarnd currently uses. twtxt2html uses Goldmark and appears to behave better π€£
@xuu I don't think this is a lextwt problem tbh. Just the Markdown aprser that yarnd currently uses. twtxt2html uses Goldmark and appears to behave better π€£
@xuu Long while back, I experimented with using similarity algorithms to detect if two Twts were similar enough to be considered an "Edit".
@xuu Long while back, I experimented with using similarity algorithms to detect if two Twts were similar enough to be considered an "Edit".
Right I see what you mean @xuu -- Can you maybe come up with a fully fleshed out proposal for this? π€ This will help solve the problem of hash collision that result from the Twt/hash space growing larger over time without us having to change anything about the way we construct hashes in the first place. We just assume spec compliant clients will just dynamically handle this as the space grows.
Right I see what you mean @xuu -- Can you maybe come up with a fully fleshed out proposal for this? π€ This will help solve the problem of hash collision that result from the Twt/hash space growing larger over time without us having to change anything about the way we construct hashes in the first place. We just assume spec compliant clients will just dynamically handle this as the space grows.
@xuu I _think_ we never progressed this idea further because we weren't sure how to tell if a hash collision would occur in the first place right? In other words, how does Client A know to expand a hash vs. Client B in a 100% decentralised way? π€
@xuu I _think_ we never progressed this idea further because we weren't sure how to tell if a hash collision would occur in the first place right? In other words, how does Client A know to expand a hash vs. Client B in a 100% decentralised way? π€
Plus these so-called "LLM"(s) have a pretty good grasp of the "shape" of language, so they _appear_ to be quite intelligent or produce intelligible response (_when they're actually quite stupid really_).
Plus these so-called "LLM"(s) have a pretty good grasp of the "shape" of language, so they _appear_ to be quite intelligent or produce intelligible response (_when they're actually quite stupid really_).
@eldersnake You don't get left behind at all π€£ It's hyped up so much, it's not even funny anymore. Basically at this point (_so far at least_) I've concluded that all this GenAI / LLM stuff is just a fancy auto-complete and indexing + search reinvented π€£
@eldersnake You don't get left behind at all π€£ It's hyped up so much, it's not even funny anymore. Basically at this point (_so far at least_) I've concluded that all this GenAI / LLM stuff is just a fancy auto-complete and indexing + search reinvented π€£
@bender This is the different Markdown parsers being used. Goldmark vs. gomarkdown. We need to switch to Goldmark π
@bender This is the different Markdown parsers being used. Goldmark vs. gomarkdown. We need to switch to Goldmark π
@quark i'm guessing the quotas text should've been emphasized?
@quark i'm guessing the quotas text should've been emphasized?
@slashdot NahahahahHa π€£ So glad I don't use LinkedIn π€¦ββοΈ
@slashdot NahahahahHa π€£ So glad I don't use LinkedIn π€¦ββοΈ
@falsifian No u don't sorry. But I tend to agree with you and I think if we continue to use hashes we should keep the remainder in mind as we choose truncation values of N
@falsifian No u don't sorry. But I tend to agree with you and I think if we continue to use hashes we should keep the remainder in mind as we choose truncation values of N
@falsifian Mostly because Git uses it π€£ Known attacks that would affect our use? π€
@falsifian Mostly because Git uses it π€£ Known attacks that would affect our use? π€
@xuu I don't recall where that discussion ended up being though?
@xuu I don't recall where that discussion ended up being though?
@xuu you mean my original idea of basically just automatically detecting Twt edits from the client side?
@xuu you mean my original idea of basically just automatically detecting Twt edits from the client side?
@xuu this is where you would need to prove that the editor delete request actually came from that feed author. Hence why integrity is much more important here.
@xuu this is where you would need to prove that the editor delete request actually came from that feed author. Hence why integrity is much more important here.
@falsifian without supporting dudes properly though you're running into GDP issues and the right to forget. π€£ we've had pretty lengthy discussions about this in the past years ago as well, but we never came to a conclusion. We're all happy with.
@falsifian without supporting dudes properly though you're running into GDP issues and the right to forget. π€£ we've had pretty lengthy discussions about this in the past years ago as well, but we never came to a conclusion. We're all happy with.
@movq it would work, you are right, however, it has drawbacks, and I think in the long term would create a new set of problems that we would also then have to solve.
@movq it would work, you are right, however, it has drawbacks, and I think in the long term would create a new set of problems that we would also then have to solve.