# I am the Watcher. I am your guide through this vast new twtiverse.
#
# Usage:
# https://watcher.sour.is/api/plain/users View list of users and latest twt date.
# https://watcher.sour.is/api/plain/twt View all twts.
# https://watcher.sour.is/api/plain/mentions?uri=:uri View all mentions for uri.
# https://watcher.sour.is/api/plain/conv/:hash View all twts for a conversation subject.
#
# Options:
# uri Filter to show a specific users twts.
# offset Start index for quey.
# limit Count of items to return (going back in time).
#
# twt range = 1 51
# self = https://watcher.sour.is/conv/3ezvila
@lyse @movq I propose that we extend the Metadata spec once more and borrow from JSON API cursor pagination and for feed author's that wish to rotate or truncate their feed (feeds.twtxt.net already does at ~1MB, and soon yarnd will also at some point); Then one can _optionally_ provide # next = <link>
and # prev = <link>
Metdata in their feed to link to previous/older versions of their feed. I _think_ this is the easier way to implement this that works no matter how you host or the client used.~
@lyse @movq I propose that we extend the Metadata spec once more and borrow from JSON API cursor pagination and for feed author's that wish to rotate or truncate their feed (feeds.twtxt.net already does at ~1MB, and soon yarnd will also at some point); Then one can _optionally_ provide # next = <link>
and # prev = <link>
Metdata in their feed to link to previous/older versions of their feed. I _think_ this is the easier way to implement this that works no matter how you host or the client used.~
In addition, supporting "Range" requests _should_ be possible from a client's perspective with fallback options to do a full request. I don't think this needs to be part of the spec at all. The way I would implement this is to keep track of the last position, and re-fetch from that position minus a few hundred bytes just to be sure, throw away any initial garbage. I'd probably also try to detect if the feed is append-only or prepend and track this too (somewhere). If the Metadata preamble changes, seeking back a bit _should_ in theory work or fallback.
In addition, supporting "Range" requests _should_ be possible from a client's perspective with fallback options to do a full request. I don't think this needs to be part of the spec at all. The way I would implement this is to keep track of the last position, and re-fetch from that position minus a few hundred bytes just to be sure, throw away any initial garbage. I'd probably also try to detect if the feed is append-only or prepend and track this too (somewhere). If the Metadata preamble changes, seeking back a bit _should_ in theory work or fallback.
@lyse Yup all good specs to draw inspiration from š
@lyse Yup all good specs to draw inspiration from š
@prologic Iāll have a look at those specs, too. The gist of it really is just next
and prev
in metadata, IIUC. And that should work pretty well. š¤
@prologic Iāll have a look at those specs, too. The gist of it really is just next
and prev
in metadata, IIUC. And that should work pretty well. š¤
@prologic Iāll have a look at those specs, too. The gist of it really is just next
and prev
in metadata, IIUC. And that should work pretty well. š¤
@prologic @movq Basically yes. RFC 5005 makes a very interesting and subtle distinction between paged and archived feeds. Archive feeds are stable and navigating in them guarantees that no result will be duplicated or missed. On the other hand, paged feeds are not required to fulfil this property. The JSON Cursor Pagination is more like archived feeds. I reckon that we can use the archive approach, too. It suits us quite well, since we probably split off pages/archives by date ranges. Prologic might be the only one whose feed needs a max twts limit over a grouping by date. :-D So prev
/next
should do. Btw. I find it quite weird that RFC 5005 calls the link relations previous
and next
for paged but prev-archive
and next-archive
for archived feeds. Do we want to tie any paging/archiving semantics to that? Also use first
and last
? Too bad that link
is already taken as titled hyperlinks in the metadata. What are your thoughts?
@lyse I think we should just go for the archival alpaorsch. next and prev seem fine to me as keys in the metadata.
@lyse I think we should just go for the archival alpaorsch. next and prev seem fine to me as keys in the metadata.
@lyse ā¦ does that mean that archived feeds are supposed to be āfrozen in timeā? Nothing ever added to them, either? (Sorry, havenāt read the actual RFC, yet.)
@lyse ā¦ does that mean that archived feeds are supposed to be āfrozen in timeā? Nothing ever added to them, either? (Sorry, havenāt read the actual RFC, yet.)
@lyse ā¦ does that mean that archived feeds are supposed to be āfrozen in timeā? Nothing ever added to them, either? (Sorry, havenāt read the actual RFC, yet.)
@movq Yes, frozen and can only be found using that search tool I keep forgetting the name of. š
@movq @thecanine Correct. Fixing typos would be the only allowed thing, but that's it. Clients who have fetched a certain archive feed may not redownload this particular one again in the future since it's considered stable/fixed/frozen/unmodifiable. So people might not get any updates from this archive feed.
@movq like rotated Apache log files, yes I would think so š
@movq like rotated Apache log files, yes I would think so š
So itād go like this?\n\n- twtxt-2021-10-25.txt
: Feed of twts starting at 2021-10-25. For now, this is the ācurrentā feed.\n- twtxt-2021-10-25.txt
keeps growing for a while by appending new twts at the end.\n- Once twtxt-2021-10-25.txt
is āfullā, we add a next
field to its metadata which points to ā¦\n- ā¦ twtxt-2021-11-01.txt
, which is initially empty except for a prev
field that points back to twtxt-2021-10-25.txt
. From this point on, twtxt-2021-11-01.txt
is the ācurrentā feed. It keeps growing in the same way. Eventually, itās full and superseded by the next (partial) feed.\n- twtxt.txt
could then be a symlink to the ācurrentā feed file.\n- Non-current feeds could now indeed be considered as āarchivedā: Nothing ever changes in them anymore (except for metadata maybe? What if I change my nickname or feed URL?)\n\nWell, at least thatās how Iād go about implementing this in jenny. š
So itād go like this?
- twtxt-2021-10-25.txt
: Feed of twts starting at 2021-10-25. For now, this is the ācurrentā feed.
- twtxt-2021-10-25.txt
keeps growing for a while by appending new twts at the end.
- Once twtxt-2021-10-25.txt
is āfullā, we add a next
field to its metadata which points to ā¦
- ā¦ twtxt-2021-11-01.txt
, which is initially empty except for a prev
field that points back to twtxt-2021-10-25.txt
. From this point on, twtxt-2021-11-01.txt
is the ācurrentā feed. It keeps growing in the same way. Eventually, itās full and superseded by the next (partial) feed.
- twtxt.txt
could then be a symlink to the ācurrentā feed file.
- Non-current feeds could now indeed be considered as āarchivedā: Nothing ever changes in them anymore (except for metadata maybe? What if I change my nickname or feed URL?)
Well, at least thatās how Iād go about implementing this in jenny. š
So itād go like this?
- twtxt-2021-10-25.txt
: Feed of twts starting at 2021-10-25. For now, this is the ācurrentā feed.
- twtxt-2021-10-25.txt
keeps growing for a while by appending new twts at the end.
- Once twtxt-2021-10-25.txt
is āfullā, we add a next
field to its metadata which points to ā¦
- ā¦ twtxt-2021-11-01.txt
, which is initially empty except for a prev
field that points back to twtxt-2021-10-25.txt
. From this point on, twtxt-2021-11-01.txt
is the ācurrentā feed. It keeps growing in the same way. Eventually, itās full and superseded by the next (partial) feed.
- twtxt.txt
could then be a symlink to the ācurrentā feed file.
- Non-current feeds could now indeed be considered as āarchivedā: Nothing ever changes in them anymore (except for metadata maybe? What if I change my nickname or feed URL?)
Well, at least thatās how Iād go about implementing this in jenny. š
So itād go like this?
- twtxt-2021-10-25.txt
: Feed of twts starting at 2021-10-25. For now, this is the ācurrentā feed.
- twtxt-2021-10-25.txt
keeps growing for a while by appending new twts at the end.
- Once twtxt-2021-10-25.txt
is āfullā, we add a next
field to its metadata which points to ā¦
- ā¦ twtxt-2021-11-01.txt
, which is initially empty except for a prev
field that points back to twtxt-2021-10-25.txt
. From this point on, twtxt-2021-11-01.txt
is the ācurrentā feed. It keeps growing in the same way. Eventually, itās full and superseded by the next (partial) feed.
- twtxt.txt
could then be a symlink to the ācurrentā feed file.
- Non-current feeds could now indeed be considered as āarchivedā: Nothing ever changes in them anymore (except for metadata maybe? What if I change my nickname or feed URL?)
Well, at least thatās how Iād go about implementing this in jenny. š
@movq I _think_ this is how I'd implement it too š
@movq I _think_ this is how I'd implement it too š
@movq \nI would recommend a longer rotation, perhaps? The way I see it, you are proposing a monthly one. That can make metadata huge too. Maybe yearly, or every 6 months?
@movq
I would recommend a longer rotation, perhaps? The way I see it, you are proposing a monthly one. That can make metadata huge too. Maybe yearly, or every 6 months?
@quark I _think_ it's up to the feed author no? I mean the feeds service at feeds.twtxt.net (_loosely based on similar code_) rotated feeds at roughly ~1MB -- Soon yarnd
will to as in practice this turns out to be a "good" value and approximately a year of Twts for socially active persons š~
@quark I _think_ it's up to the feed author no? I mean the feeds service at feeds.twtxt.net (_loosely based on similar code_) rotated feeds at roughly ~1MB -- Soon yarnd
will to as in practice this turns out to be a "good" value and approximately a year of Twts for socially active persons š~
@quark Nah, that was just an example. š Iād probably go for ā$n
months or $m
twts, whichever comes firstā, both variables being customizable. I mean, thereās no point in āpaginatingā a feed if thereās very little traffic.
@quark Nah, that was just an example. š Iād probably go for ā$n
months or $m
twts, whichever comes firstā, both variables being customizable. I mean, thereās no point in āpaginatingā a feed if thereās very little traffic.
@quark Nah, that was just an example. š Iād probably go for ā$n
months or $m
twts, whichever comes firstā, both variables being customizable. I mean, thereās no point in āpaginatingā a feed if thereās very little traffic.
@movq \n> both variables being customizable\n\nExcellent! And I agree on everything else, yes.\n\n@https://twtxt.net/user/prologic/twtxt.txt>\n> I think itās up to the feed author no?\n\nNo if it is not made customisable. It is up to the client's author, in this case, @movq. āŗļø
@movq If I understood RFC 5005 correctly, the current and an archive feed are two separate things, there's no overlap between them. So an item is either in the current or in the archived, but not in both. So a symlink is not enough. But of course we could change that for our purpose. Regarding meta data changes, not sure how to go about them. Changing the nick is probably no problem, but changing the URL causes twt hashes to break again. The never ending story.
@lyse
> changing the URL causes twt hashes to break again. The never ending story.
It's actually not really as big of a problem as we've made it out to be in past conversations. If enough clients have seen versions of the feed then they'll have old copies of the Twts with the previous Hash and thus there will still be a valid chain. What changing the URL or the content of a individual Twt really does is fork the chain. Which isn't so bad IHMO. Just something to be aware of I guess? š¤
@lyse \n\n> changing the URL causes twt hashes to break again. The never ending story.\n\nIt's actually not really as big of a problem as we've made it out to be in past conversations. If enough clients have seen versions of the feed then they'll have old copies of the Twts with the previous Hash and thus there will still be a valid chain. What changing the URL or the content of a individual Twt really does is fork the chain. Which isn't so bad IHMO. Just something to be aware of I guess? š¤
@lyse
> changing the URL causes twt hashes to break again. The never ending story.
It's actually not really as big of a problem as we've made it out to be in past conversations. If enough clients have seen versions of the feed then they'll have old copies of the Twts with the previous Hash and thus there will still be a valid chain. What changing the URL or the content of a individual Twt really does is fork the chain. Which isn't so bad IHMO. Just something to be aware of I guess? š¤
@prologic Well, it depends on the client. The original twtxt
client for example just caches a copy of the current feed, so on next update cycle it just replaces everything it had for this feed with the current contents. I'd imagine that's a common model.
@lyse Yeah not sure really. yarnd
ās cache originally came from twet in the early days and morphed over time to what it is now.
Gbe Twt Hash extension had the most influence.
I donāt think just sucking down the feed and storing the file is the most efficient nor useful thing to do given how weāve extended the spec.
@lyse Yeah not sure really. yarnd
ās cache originally came from twet in the early days and morphed over time to what it is now.\n\nGbe Twt Hash extension had the most influence.\n\nI donāt think just sucking down the feed and storing the file is the most efficient nor useful thing to do given how weāve extended the spec.
@lyse Yeah not sure really. yarnd
ās cache originally came from twet in the early days and morphed over time to what it is now.
Gbe Twt Hash extension had the most influence.
I donāt think just sucking down the feed and storing the file is the most efficient nor useful thing to do given how weāve extended the spec.