# I am the Watcher. I am your guide through this vast new twtiverse.
# 
# Usage:
#     https://watcher.sour.is/api/plain/users              View list of users and latest twt date.
#     https://watcher.sour.is/api/plain/twt                View all twts.
#     https://watcher.sour.is/api/plain/mentions?uri=:uri  View all mentions for uri.
#     https://watcher.sour.is/api/plain/conv/:hash         View all twts for a conversation subject.
# 
# Options:
#     uri     Filter to show a specific users twts.
#     offset  Start index for quey.
#     limit   Count of items to return (going back in time).
# 
# twt range = 1 51
# self = https://watcher.sour.is/conv/3ezvila
@lyse @movq I propose that we extend the Metadata spec once more and borrow from JSON API cursor pagination and for feed author's that wish to rotate or truncate their feed (feeds.twtxt.net already does at ~1MB, and soon yarnd will also at some point); Then one can _optionally_ provide # next = <link> and # prev = <link> Metdata in their feed to link to previous/older versions of their feed. I _think_ this is the easier way to implement this that works no matter how you host or the client used.~
@lyse @movq I propose that we extend the Metadata spec once more and borrow from JSON API cursor pagination and for feed author's that wish to rotate or truncate their feed (feeds.twtxt.net already does at ~1MB, and soon yarnd will also at some point); Then one can _optionally_ provide # next = <link> and # prev = <link> Metdata in their feed to link to previous/older versions of their feed. I _think_ this is the easier way to implement this that works no matter how you host or the client used.~
In addition, supporting "Range" requests _should_ be possible from a client's perspective with fallback options to do a full request. I don't think this needs to be part of the spec at all. The way I would implement this is to keep track of the last position, and re-fetch from that position minus a few hundred bytes just to be sure, throw away any initial garbage. I'd probably also try to detect if the feed is append-only or prepend and track this too (somewhere). If the Metadata preamble changes, seeking back a bit _should_ in theory work or fallback.
In addition, supporting "Range" requests _should_ be possible from a client's perspective with fallback options to do a full request. I don't think this needs to be part of the spec at all. The way I would implement this is to keep track of the last position, and re-fetch from that position minus a few hundred bytes just to be sure, throw away any initial garbage. I'd probably also try to detect if the feed is append-only or prepend and track this too (somewhere). If the Metadata preamble changes, seeking back a bit _should_ in theory work or fallback.
@prologic I'll have a look at the JSON API Cursor Pagination Spec later. Podlove hints to RFC 5005, Feed Paging and Archiving, for podcast feeds. I'll have a read of that later, too.
@lyse Yup all good specs to draw inspiration from šŸ‘Œ
@lyse Yup all good specs to draw inspiration from šŸ‘Œ
@prologic Exactly. :-)
@prologic Iā€™ll have a look at those specs, too. The gist of it really is just next and prev in metadata, IIUC. And that should work pretty well. šŸ¤”
@prologic Iā€™ll have a look at those specs, too. The gist of it really is just next and prev in metadata, IIUC. And that should work pretty well. šŸ¤”
@prologic Iā€™ll have a look at those specs, too. The gist of it really is just next and prev in metadata, IIUC. And that should work pretty well. šŸ¤”
@movq Yup šŸ‘Œ
@movq Yup šŸ‘Œ
@prologic @movq Basically yes. RFC 5005 makes a very interesting and subtle distinction between paged and archived feeds. Archive feeds are stable and navigating in them guarantees that no result will be duplicated or missed. On the other hand, paged feeds are not required to fulfil this property. The JSON Cursor Pagination is more like archived feeds. I reckon that we can use the archive approach, too. It suits us quite well, since we probably split off pages/archives by date ranges. Prologic might be the only one whose feed needs a max twts limit over a grouping by date. :-D So prev/next should do. Btw. I find it quite weird that RFC 5005 calls the link relations previous and next for paged but prev-archive and next-archive for archived feeds. Do we want to tie any paging/archiving semantics to that? Also use first and last? Too bad that link is already taken as titled hyperlinks in the metadata. What are your thoughts?
@lyse I think we should just go for the archival alpaorsch. next and prev seem fine to me as keys in the metadata.
@lyse I think we should just go for the archival alpaorsch. next and prev seem fine to me as keys in the metadata.
@lyse ā€¦ does that mean that archived feeds are supposed to be ā€œfrozen in timeā€? Nothing ever added to them, either? (Sorry, havenā€™t read the actual RFC, yet.)
@lyse ā€¦ does that mean that archived feeds are supposed to be ā€œfrozen in timeā€? Nothing ever added to them, either? (Sorry, havenā€™t read the actual RFC, yet.)
@lyse ā€¦ does that mean that archived feeds are supposed to be ā€œfrozen in timeā€? Nothing ever added to them, either? (Sorry, havenā€™t read the actual RFC, yet.)
@movq Yes, frozen and can only be found using that search tool I keep forgetting the name of. šŸ™ƒ
@movq @thecanine Correct. Fixing typos would be the only allowed thing, but that's it. Clients who have fetched a certain archive feed may not redownload this particular one again in the future since it's considered stable/fixed/frozen/unmodifiable. So people might not get any updates from this archive feed.
@movq like rotated Apache log files, yes I would think so šŸ‘Œ
@movq like rotated Apache log files, yes I would think so šŸ‘Œ
@thecanine https://search.twtxt.net/
@thecanine https://search.twtxt.net/
@lyse šŸ‘Œ
@lyse šŸ‘Œ
So itā€™d go like this?\n\n- twtxt-2021-10-25.txt: Feed of twts starting at 2021-10-25. For now, this is the ā€œcurrentā€ feed.\n- twtxt-2021-10-25.txt keeps growing for a while by appending new twts at the end.\n- Once twtxt-2021-10-25.txt is ā€œfullā€, we add a next field to its metadata which points to ā€¦\n- ā€¦ twtxt-2021-11-01.txt, which is initially empty except for a prev field that points back to twtxt-2021-10-25.txt. From this point on, twtxt-2021-11-01.txt is the ā€œcurrentā€ feed. It keeps growing in the same way. Eventually, itā€™s full and superseded by the next (partial) feed.\n- twtxt.txt could then be a symlink to the ā€œcurrentā€ feed file.\n- Non-current feeds could now indeed be considered as ā€œarchivedā€: Nothing ever changes in them anymore (except for metadata maybe? What if I change my nickname or feed URL?)\n\nWell, at least thatā€™s how Iā€™d go about implementing this in jenny. šŸ™‚
So itā€™d go like this?

- twtxt-2021-10-25.txt: Feed of twts starting at 2021-10-25. For now, this is the ā€œcurrentā€ feed.
- twtxt-2021-10-25.txt keeps growing for a while by appending new twts at the end.
- Once twtxt-2021-10-25.txt is ā€œfullā€, we add a next field to its metadata which points to ā€¦
- ā€¦ twtxt-2021-11-01.txt, which is initially empty except for a prev field that points back to twtxt-2021-10-25.txt. From this point on, twtxt-2021-11-01.txt is the ā€œcurrentā€ feed. It keeps growing in the same way. Eventually, itā€™s full and superseded by the next (partial) feed.
- twtxt.txt could then be a symlink to the ā€œcurrentā€ feed file.
- Non-current feeds could now indeed be considered as ā€œarchivedā€: Nothing ever changes in them anymore (except for metadata maybe? What if I change my nickname or feed URL?)

Well, at least thatā€™s how Iā€™d go about implementing this in jenny. šŸ™‚
So itā€™d go like this?

- twtxt-2021-10-25.txt: Feed of twts starting at 2021-10-25. For now, this is the ā€œcurrentā€ feed.
- twtxt-2021-10-25.txt keeps growing for a while by appending new twts at the end.
- Once twtxt-2021-10-25.txt is ā€œfullā€, we add a next field to its metadata which points to ā€¦
- ā€¦ twtxt-2021-11-01.txt, which is initially empty except for a prev field that points back to twtxt-2021-10-25.txt. From this point on, twtxt-2021-11-01.txt is the ā€œcurrentā€ feed. It keeps growing in the same way. Eventually, itā€™s full and superseded by the next (partial) feed.
- twtxt.txt could then be a symlink to the ā€œcurrentā€ feed file.
- Non-current feeds could now indeed be considered as ā€œarchivedā€: Nothing ever changes in them anymore (except for metadata maybe? What if I change my nickname or feed URL?)

Well, at least thatā€™s how Iā€™d go about implementing this in jenny. šŸ™‚
So itā€™d go like this?

- twtxt-2021-10-25.txt: Feed of twts starting at 2021-10-25. For now, this is the ā€œcurrentā€ feed.
- twtxt-2021-10-25.txt keeps growing for a while by appending new twts at the end.
- Once twtxt-2021-10-25.txt is ā€œfullā€, we add a next field to its metadata which points to ā€¦
- ā€¦ twtxt-2021-11-01.txt, which is initially empty except for a prev field that points back to twtxt-2021-10-25.txt. From this point on, twtxt-2021-11-01.txt is the ā€œcurrentā€ feed. It keeps growing in the same way. Eventually, itā€™s full and superseded by the next (partial) feed.
- twtxt.txt could then be a symlink to the ā€œcurrentā€ feed file.
- Non-current feeds could now indeed be considered as ā€œarchivedā€: Nothing ever changes in them anymore (except for metadata maybe? What if I change my nickname or feed URL?)

Well, at least thatā€™s how Iā€™d go about implementing this in jenny. šŸ™‚
@movq I _think_ this is how I'd implement it too šŸ‘Œ
@movq I _think_ this is how I'd implement it too šŸ‘Œ
@movq \nI would recommend a longer rotation, perhaps? The way I see it, you are proposing a monthly one. That can make metadata huge too. Maybe yearly, or every 6 months?
@movq
I would recommend a longer rotation, perhaps? The way I see it, you are proposing a monthly one. That can make metadata huge too. Maybe yearly, or every 6 months?
@quark I _think_ it's up to the feed author no? I mean the feeds service at feeds.twtxt.net (_loosely based on similar code_) rotated feeds at roughly ~1MB -- Soon yarnd will to as in practice this turns out to be a "good" value and approximately a year of Twts for socially active persons šŸ˜‚~
@quark I _think_ it's up to the feed author no? I mean the feeds service at feeds.twtxt.net (_loosely based on similar code_) rotated feeds at roughly ~1MB -- Soon yarnd will to as in practice this turns out to be a "good" value and approximately a year of Twts for socially active persons šŸ˜‚~
@quark Nah, that was just an example. šŸ™‚ Iā€™d probably go for ā€œ$n months or $m twts, whichever comes firstā€, both variables being customizable. I mean, thereā€™s no point in ā€œpaginatingā€ a feed if thereā€™s very little traffic.
@quark Nah, that was just an example. šŸ™‚ Iā€™d probably go for ā€œ$n months or $m twts, whichever comes firstā€, both variables being customizable. I mean, thereā€™s no point in ā€œpaginatingā€ a feed if thereā€™s very little traffic.
@quark Nah, that was just an example. šŸ™‚ Iā€™d probably go for ā€œ$n months or $m twts, whichever comes firstā€, both variables being customizable. I mean, thereā€™s no point in ā€œpaginatingā€ a feed if thereā€™s very little traffic.
@movq \n> both variables being customizable\n\nExcellent! And I agree on everything else, yes.\n\n@https://twtxt.net/user/prologic/twtxt.txt>\n> I think itā€™s up to the feed author no?\n\nNo if it is not made customisable. It is up to the client's author, in this case, @movq. ā˜ŗļø
@fastidious True šŸ¤£
@fastidious True šŸ¤£
@movq If I understood RFC 5005 correctly, the current and an archive feed are two separate things, there's no overlap between them. So an item is either in the current or in the archived, but not in both. So a symlink is not enough. But of course we could change that for our purpose. Regarding meta data changes, not sure how to go about them. Changing the nick is probably no problem, but changing the URL causes twt hashes to break again. The never ending story.
@lyse

> changing the URL causes twt hashes to break again. The never ending story.

It's actually not really as big of a problem as we've made it out to be in past conversations. If enough clients have seen versions of the feed then they'll have old copies of the Twts with the previous Hash and thus there will still be a valid chain. What changing the URL or the content of a individual Twt really does is fork the chain. Which isn't so bad IHMO. Just something to be aware of I guess? šŸ¤”
@lyse \n\n> changing the URL causes twt hashes to break again. The never ending story.\n\nIt's actually not really as big of a problem as we've made it out to be in past conversations. If enough clients have seen versions of the feed then they'll have old copies of the Twts with the previous Hash and thus there will still be a valid chain. What changing the URL or the content of a individual Twt really does is fork the chain. Which isn't so bad IHMO. Just something to be aware of I guess? šŸ¤”
@lyse

> changing the URL causes twt hashes to break again. The never ending story.

It's actually not really as big of a problem as we've made it out to be in past conversations. If enough clients have seen versions of the feed then they'll have old copies of the Twts with the previous Hash and thus there will still be a valid chain. What changing the URL or the content of a individual Twt really does is fork the chain. Which isn't so bad IHMO. Just something to be aware of I guess? šŸ¤”
@prologic Well, it depends on the client. The original twtxt client for example just caches a copy of the current feed, so on next update cycle it just replaces everything it had for this feed with the current contents. I'd imagine that's a common model.
@lyse Yeah not sure really. yarndā€™s cache originally came from twet in the early days and morphed over time to what it is now.

Gbe Twt Hash extension had the most influence.

I donā€™t think just sucking down the feed and storing the file is the most efficient nor useful thing to do given how weā€™ve extended the spec.
@lyse Yeah not sure really. yarndā€™s cache originally came from twet in the early days and morphed over time to what it is now.\n\nGbe Twt Hash extension had the most influence.\n\nI donā€™t think just sucking down the feed and storing the file is the most efficient nor useful thing to do given how weā€™ve extended the spec.
@lyse Yeah not sure really. yarndā€™s cache originally came from twet in the early days and morphed over time to what it is now.

Gbe Twt Hash extension had the most influence.

I donā€™t think just sucking down the feed and storing the file is the most efficient nor useful thing to do given how weā€™ve extended the spec.