# I am the Watcher. I am your guide through this vast new twtiverse.
# 
# Usage:
#     https://watcher.sour.is/api/plain/users              View list of users and latest twt date.
#     https://watcher.sour.is/api/plain/twt                View all twts.
#     https://watcher.sour.is/api/plain/mentions?uri=:uri  View all mentions for uri.
#     https://watcher.sour.is/api/plain/conv/:hash         View all twts for a conversation subject.
# 
# Options:
#     uri     Filter to show a specific users twts.
#     offset  Start index for quey.
#     limit   Count of items to return (going back in time).
# 
# twt range = 1 30
# self = https://watcher.sour.is/conv/u6uumbq
stackeffect
twtxt.stackeffect.de
View Thread
@movq How is deletion supposed to work? In mutt I deleted by D~d>1m and then fetched by !jenny -f. This brings back all deleted twts. Isn't lastmods used to skip older twts?
movq
www.uninformativ.de
View Thread
@stackeffect There is no support for deletion, really. lastmods only affects retrieval of feeds (i.e., it sets the if-modified-since HTTP header), but when a feed *is* retrieved, then it is always processed in full. So, eventually, you get all twts that are present in a feed.

I’m curious, what is your use case for deleting twts?
movq
www.uninformativ.de
View Thread
@stackeffect There is no support for deletion, really. lastmods only affects retrieval of feeds (i.e., it sets the if-modified-since HTTP header), but when a feed *is* retrieved, then it is always processed in full. So, eventually, you get all twts that are present in a feed.

I’m curious, what is your use case for deleting twts?
movq
www.uninformativ.de
View Thread
@stackeffect There is no support for deletion, really. lastmods only affects retrieval of feeds (i.e., it sets the if-modified-since HTTP header), but when a feed *is* retrieved, then it is always processed in full. So, eventually, you get all twts that are present in a feed.

I’m curious, what is your use case for deleting twts?
movq
www.uninformativ.de
View Thread
@stackeffect There is no support for deletion, really. lastmods only affects retrieval of feeds (i.e., it sets the if-modified-since HTTP header), but when a feed *is* retrieved, then it is always processed in full. So, eventually, you get all twts that are present in a feed.\n\nI’m curious, what is your use case for deleting twts?
stackeffect
twtxt.stackeffect.de
View Thread
@movq
> I’m curious, what is your use case for deleting twts?
Not just deleting, also sorting into other folders is impossible.
It also doesn't scale in the long term. When I cannot delete twts then I have a full copy of every twtxt I follow - forever. That's a waste of bandwidth and disk space.
stackeffect
twtxt.stackeffect.de
View Thread
@movq \n> I’m curious, what is your use case for deleting twts?\nNot just deleting, also sorting into other folders is impossible.\nIt also doesn't scale in the long term. When I cannot delete twts then I have a full copy of every twtxt I follow - forever. That's a waste of bandwidth and disk space.
movq
www.uninformativ.de
View Thread
@stackeffect Hmm. 🤔

At the moment, it’d be rather easy to implement this in jenny: We could simply store the time stamp of the newest twt we’ve seen (that’s different than lastmods) and then discard older twts. I think @quark asked about that a while ago, too. I *think* I didn’t implement this yet, because I simply didn’t need it. The twtxt world was so tiny that it didn’t matter. But you’re right, it’s going to be a problem in the long run.

Nobody ever really thought about “long term” in the twtxt world, if you ask me. I mean, it’s the same thing when *writing* twts. My feed keeps growing and growing … forever? Do I truncate at some point? Will all clients be able to handle that truncation? 🤔

I want to implement HTTP range requests next. That’s going to be interesting. Also, range requests and truncation probably don’t mix very well. 🤪 But I haven’t put much thought into it, yet.

tl;dr: I’ll put this on the TODO list after all. 👍
movq
www.uninformativ.de
View Thread
@stackeffect Hmm. 🤔\n\nAt the moment, it’d be rather easy to implement this in jenny: We could simply store the time stamp of the newest twt we’ve seen (that’s different than lastmods) and then discard older twts. I think @quark asked about that a while ago, too. I *think* I didn’t implement this yet, because I simply didn’t need it. The twtxt world was so tiny that it didn’t matter. But you’re right, it’s going to be a problem in the long run.\n\nNobody ever really thought about “long term” in the twtxt world, if you ask me. I mean, it’s the same thing when *writing* twts. My feed keeps growing and growing … forever? Do I truncate at some point? Will all clients be able to handle that truncation? 🤔\n\nI want to implement HTTP range requests next. That’s going to be interesting. Also, range requests and truncation probably don’t mix very well. 🤪 But I haven’t put much thought into it, yet.\n\ntl;dr: I’ll put this on the TODO list after all. 👍
movq
www.uninformativ.de
View Thread
@stackeffect Hmm. 🤔

At the moment, it’d be rather easy to implement this in jenny: We could simply store the time stamp of the newest twt we’ve seen (that’s different than lastmods) and then discard older twts. I think @quark asked about that a while ago, too. I *think* I didn’t implement this yet, because I simply didn’t need it. The twtxt world was so tiny that it didn’t matter. But you’re right, it’s going to be a problem in the long run.

Nobody ever really thought about “long term” in the twtxt world, if you ask me. I mean, it’s the same thing when *writing* twts. My feed keeps growing and growing … forever? Do I truncate at some point? Will all clients be able to handle that truncation? 🤔

I want to implement HTTP range requests next. That’s going to be interesting. Also, range requests and truncation probably don’t mix very well. 🤪 But I haven’t put much thought into it, yet.

tl;dr: I’ll put this on the TODO list after all. 👍
movq
www.uninformativ.de
View Thread
@stackeffect Hmm. 🤔

At the moment, it’d be rather easy to implement this in jenny: We could simply store the time stamp of the newest twt we’ve seen (that’s different than lastmods) and then discard older twts. I think @quark asked about that a while ago, too. I *think* I didn’t implement this yet, because I simply didn’t need it. The twtxt world was so tiny that it didn’t matter. But you’re right, it’s going to be a problem in the long run.

Nobody ever really thought about “long term” in the twtxt world, if you ask me. I mean, it’s the same thing when *writing* twts. My feed keeps growing and growing … forever? Do I truncate at some point? Will all clients be able to handle that truncation? 🤔

I want to implement HTTP range requests next. That’s going to be interesting. Also, range requests and truncation probably don’t mix very well. 🤪 But I haven’t put much thought into it, yet.

tl;dr: I’ll put this on the TODO list after all. 👍
quark
ferengi.one
View Thread
@movq
Yes, I did ask whether or not it was possible to move twts to an "archive" folder, but it will be the same at @stackeffect experienced (which I have, too), that is, twts will "come back".

There is no clear solution, I am afraid, right? It is the nature of the beast.
quark
ferengi.one
View Thread
@movq \nYes, I did ask whether or not it was possible to move twts to an "archive" folder, but it will be the same at @stackeffect experienced (which I have, too), that is, twts will "come back".\n\nThere is no clear solution, I am afraid, right? It is the nature of the beast.
lyse
lyse.isobeef.org
View Thread
@movq The original Twtxt File Format Specification says: "A specific ordering of the statuses is not mandatory." So range requests are getting problematic if feed authors don't append but prepend – or even worse, randomly sort – new twts. To be fair, most if not all (I don't know) feeds I found so far are in append style. Another thing: If the metadata header at the top is expanded, range requests might skip new twts. I just want to emphasize that this apparently simple problem might get really tricky quickly with its tons of edge cases.
lyse
lyse.isobeef.org
View Thread
@movq @prologic Maybe we need some kind of feed paging. E.g. next and prev metadata pointing to other URLs or something along those lines. Not sure if there are even some standard HTTP headers for that of thing. I think Podlove did something similar for podcast feeds if I'm not mistaken.
movq
www.uninformativ.de
View Thread
@lyse Ouch, thank you. That (“feeds don’t need to be ordered”) pretty much kills the idea of doing range requests. 💀 I also didn’t consider metadata changes. 🤦

FWIW, I know of at least one feed in the wild that *prepends* new twts: http://codemadness.org/twtxt.txt
movq
www.uninformativ.de
View Thread
@lyse Ouch, thank you. That (“feeds don’t need to be ordered”) pretty much kills the idea of doing range requests. 💀 I also didn’t consider metadata changes. 🤦\n\nFWIW, I know of at least one feed in the wild that *prepends* new twts: http://codemadness.org/twtxt.txt
movq
www.uninformativ.de
View Thread
@lyse Ouch, thank you. That (“feeds don’t need to be ordered”) pretty much kills the idea of doing range requests. 💀 I also didn’t consider metadata changes. 🤦

FWIW, I know of at least one feed in the wild that *prepends* new twts: http://codemadness.org/twtxt.txt
movq
www.uninformativ.de
View Thread
@lyse Ouch, thank you. That (“feeds don’t need to be ordered”) pretty much kills the idea of doing range requests. 💀 I also didn’t consider metadata changes. 🤦

FWIW, I know of at least one feed in the wild that *prepends* new twts: http://codemadness.org/twtxt.txt
fastidious
arrakis.netbros.com
View Thread
@movq I have seeing more (don't remember where now), and wondered too. With the explanation on the original twtxt file format specification provided by @lyse it all makes sense now.
lyse
lyse.isobeef.org
View Thread
@movq Oh, right, this feed. Well, if you detect some garbage (unparsable twts) with the range request, you can always fall back to a complete request. It probably works most of the time, but there are certainly plenty of pitfalls you can come across. You might detect the order of the feed and then formulate your queries accordingly. Maybe if you construct the range so that the last seen twt will be refetched, you can detect whether the request was off and a full sync needs to follow. Some metadata changes might be missed with the range request if the amount of bytes didn't chance, but fully querying the feed once a day/week/whatever might take care of that – if metadata are actually of any use to jenny. But it's getting complicated really quickly. So maybe some kind of pagination/archive would be easier.
movq
www.uninformativ.de
View Thread
@lyse Ya that gets out of hand real quick. I don’t think it’s worth it. There’s too much that can go wrong. 🤔
movq
www.uninformativ.de
View Thread
@lyse Ya that gets out of hand real quick. I don’t think it’s worth it. There’s too much that can go wrong. 🤔
movq
www.uninformativ.de
View Thread
@lyse Ya that gets out of hand real quick. I don’t think it’s worth it. There’s too much that can go wrong. 🤔
prologic
twtxt.net
View Thread
@quark yarnd archives old Twts based on time and no. of twts today
prologic
twtxt.net
View Thread
@quark yarnd archives old Twts based on time and no. of twts today
prologic
twtxt.net
View Thread
@lyse Actually this is what JSON-LD is all about. Linked Documents. We could adopt this pattern where we permit feed rotation or truncation but you can point to an old version 👌
prologic
twtxt.net
View Thread
@lyse Actually this is what JSON-LD is all about. Linked Documents. We could adopt this pattern where we permit feed rotation or truncation but you can point to an old version 👌
prologic
twtxt.net
View Thread
@movq You could probably detect whether a feed prepends or appends
prologic
twtxt.net
View Thread
@movq You could probably detect whether a feed prepends or appends