# I am the Watcher. I am your guide through this vast new twtiverse.
#
# Usage:
# https://watcher.sour.is/api/plain/users View list of users and latest twt date.
# https://watcher.sour.is/api/plain/twt View all twts.
# https://watcher.sour.is/api/plain/mentions?uri=:uri View all mentions for uri.
# https://watcher.sour.is/api/plain/conv/:hash View all twts for a conversation subject.
#
# Options:
# uri Filter to show a specific users twts.
# offset Start index for quey.
# limit Count of items to return (going back in time).
#
# twt range = 1 21
# self = https://watcher.sour.is/conv/3ll4fja
How often would you want your feed(s) re-crawled / re-scraped ? 🤔
How often would you want your feed(s) re-crawled / re-scraped ? 🤔
> How often would you want your feed(s) re-crawled / re-scraped ? 🤔
cc @anth @dbohdan as the most vocal Twtxt users.
\n\n> How often would you want your feed(s) re-crawled / re-scraped ? 🤔\n\ncc @anth @dbohdan as the most vocal Twtxt users.
> How often would you want your feed(s) re-crawled / re-scraped ? 🤔
cc @anth @dbohdan as the most vocal Twtxt users.
I feel like most feeds I would want to subscribe to are updated pretty infrequently so once a day would be sufficient. But I’ve seen other feeds that are updated more frequently. Perhaps some sort of heuristic based off the feed itself? (For example, average time between entries with a max of 1 day.) If you combined that with a feature to manually trigger a feed recrawl, that might be good enough.
@brasshopper This is a good suggestion! Right now the crawler is a "one-short" thing (_just to be nice while I develop/improve it_). From your suggestion it _sounds_ like I can possibly remove the "one-shot" restriction and just setup a daily job that re-crawls the entire Twtxt space from a seed feed. The _best_ seed so far is _probably_ my own followed by @jlj 's -- Then I can focus on the refetch/rescrape parts based in heuristics. What do you think?
@brasshopper This is a good suggestion! Right now the crawler is a "one-short" thing (_just to be nice while I develop/improve it_). From your suggestion it _sounds_ like I can possibly remove the "one-shot" restriction and just setup a daily job that re-crawls the entire Twtxt space from a seed feed. The _best_ seed so far is _probably_ my own followed by @jlj 's -- Then I can focus on the refetch/rescrape parts based in heuristics. What do you think?
As a user, I think 1-3 times a day would be fine.\nAs someone who pays to host a twtxt feed, I don't mind what you set it to as long as it's not unreasonably often. It won't really make a difference to me. As of about 6 months ago, 3 different yarn.social pods each request /twtxt.txt (with an If-Modified-Since header) every 5 minutes, 24 / 7. I think I also had a different twtxt client requesting it every 10 minutes.
As a user, I think 1-3 times a day would be fine.
As someone who pays to host a twtxt feed, I don't mind what you set it to as long as it's not unreasonably often. It won't really make a difference to me. As of about 6 months ago, 3 different yarn.social pods each request /twtxt.txt (with an If-Modified-Since header) every 5 minutes, 24 / 7. I think I also had a different twtxt client requesting it every 10 minutes.
@jlj There are some benefits to more frequent crawling / scraping 🤔 Auxh as filling in missing in a conversation where you don’t follow some participants, global timeline, etc
@jlj There are some benefits to more frequent crawling / scraping 🤔 Auxh as filling in missing in a conversation where you don’t follow some participants, global timeline, etc
Once a day.. though if it hasn't updated in n-months maybe once a week?
Once a day.. though if it hasn't updated in n-months maybe once a week?
Once a day.. though if it hasn't updated in n-months maybe once a week?
@xuu What about the utility of rescraping more active feeds so we can do things like fill in missing pieces in a conversation of participants you don’t follow?
@xuu What about the utility of rescraping more active feeds so we can do things like fill in missing pieces in a conversation of participants you don’t follow?