# I am the Watcher. I am your guide through this vast new twtiverse.
# 
# Usage:
#     https://watcher.sour.is/api/plain/users              View list of users and latest twt date.
#     https://watcher.sour.is/api/plain/twt                View all twts.
#     https://watcher.sour.is/api/plain/mentions?uri=:uri  View all mentions for uri.
#     https://watcher.sour.is/api/plain/conv/:hash         View all twts for a conversation subject.
# 
# Options:
#     uri     Filter to show a specific users twts.
#     offset  Start index for quey.
#     limit   Count of items to return (going back in time).
# 
# twt range = 1 60515
# self = https://watcher.sour.is?uri=https://twtxt.net/user/prologic/twtxt.txt&offset=56391
# next = https://watcher.sour.is?uri=https://twtxt.net/user/prologic/twtxt.txt&offset=56491
# prev = https://watcher.sour.is?uri=https://twtxt.net/user/prologic/twtxt.txt&offset=56291
@abucci appreciate it if you find the time to update again ๐Ÿ™
@abucci appreciate it if you find the time to update again ๐Ÿ™
Now that's rolling out, I _think_ that's it. The only final way I can improve that /external endpoint/view is to refactor how it works a bit and add some HTMX magicโ„ข so it has a nice snappy UX to it as it dynamically tries to validate the feed and provide useful feedback to the user, that way I can avoid injecting it into the cache unnecessarily in the first place!
Now that's rolling out, I _think_ that's it. The only final way I can improve that /external endpoint/view is to refactor how it works a bit and add some HTMX magicโ„ข so it has a nice snappy UX to it as it dynamically tries to validate the feed and provide useful feedback to the user, that way I can avoid injecting it into the cache unnecessarily in the first place!
@bender Yeah but I found another bug and just squished that. CD pipeline is gonna roll this pod soonโ„ข -- Basically wasn't handling feeds that redirect properly. e.g: https://google.com => https://www.google.com (_though it's not a feed ๐Ÿคฃ_)=_
@bender Yeah but I found another bug and just squished that. CD pipeline is gonna roll this pod soonโ„ข -- Basically wasn't handling feeds that redirect properly. e.g: https://google.com => https://www.google.com (_though it's not a feed ๐Ÿคฃ_)=_
Hmm I see this in the cache again ๐Ÿคฆโ€โ™‚๏ธ Not sure how tbh -- Job for me later.
Hmm I see this in the cache again ๐Ÿคฆโ€โ™‚๏ธ Not sure how tbh -- Job for me later.
@bender https://google.com has been removed from the cache (_without nuking the entire cache_) @abucci if you need to selectively do this for some reason, there's a script in the tools directory for this:


$ ./tools/cache_delete_feed.sh 'https://google.com'


Anyway, that's gone. This is much much harder to exploit now, even if you're an authenticated user.
@bender https://google.com has been removed from the cache (_without nuking the entire cache_) @abucci if you need to selectively do this for some reason, there's a script in the tools directory for this:


$ ./tools/cache_delete_feed.sh 'https://google.com'


Anyway, that's gone. This is much much harder to exploit now, even if you're an authenticated user.
Time for workโ„ข, But I quickly hacked together a bit of a better solution here. Rolling it out to my pod so we'll see how it _actually_ goes. Still possible to abuse if you're a logged in user, etc, but at least now we delete the invalid/bad feed afterwards if it a) was not even a text//plain content-type or b) it errored out and was a new fetch of a HTTP feed.
Time for workโ„ข, But I quickly hacked together a bit of a better solution here. Rolling it out to my pod so we'll see how it _actually_ goes. Still possible to abuse if you're a logged in user, etc, but at least now we delete the invalid/bad feed afterwards if it a) was not even a text//plain content-type or b) it errored out and was a new fetch of a HTTP feed.
Yeah okay.
Yeah okay.
@bender Not possible ๐Ÿคฃ
@bender Not possible ๐Ÿคฃ
@abucci I mean it's only suppose to do one thing really. What are you thinking here?
@abucci I mean it's only suppose to do one thing really. What are you thinking here?
Yeah I was afraid of this. Technically can still be abused by "logged in" users. Hmmm

So we really not trust ourselves? ๐Ÿคฃ๐Ÿ™„
Yeah I was afraid of this. Technically can still be abused by "logged in" users. Hmmm

So we really not trust ourselves? ๐Ÿคฃ๐Ÿ™„
@abucci No worries! ๐Ÿ˜…
@abucci No worries! ๐Ÿ˜…
But this is super weird, should behave the same as my pod ๐Ÿคฆโ€โ™‚๏ธ
But this is super weird, should behave the same as my pod ๐Ÿคฆโ€โ™‚๏ธ
@abucci nuke the cache file before starring
@abucci nuke the cache file before starring
How did you nuke your cache?
How did you nuke your cache?
@abucci Hmmm weird ๐Ÿค”
@abucci Hmmm weird ๐Ÿค”
@abucci You don't actually appear to be running that sha hmmm? ๐Ÿค”
@abucci You don't actually appear to be running that sha hmmm? ๐Ÿค”
Hmmm
Hmmm
It appears to be working to ๐Ÿ‘Œ silly bots ๐Ÿ™„
It appears to be working to ๐Ÿ‘Œ silly bots ๐Ÿ™„
It appears to be working to ๐Ÿ‘Œ silly bots ๐Ÿ™„
@abucci Bo worries! If you curl it too it'll return a proper 494 ๐Ÿ‘Œ Should make bots go away ๐Ÿคž
@abucci Bo worries! If you curl it too it'll return a proper 494 ๐Ÿ‘Œ Should make bots go away ๐Ÿคž
@abucci Blah my cache was poisoned ๐Ÿคฆโ€โ™‚๏ธ it's fine now! And this is no monger possible to do now.
@abucci Blah my cache was poisoned ๐Ÿคฆโ€โ™‚๏ธ it's fine now! And this is no monger possible to do now.
@bender Did they win though? Did they?! ๐Ÿคฃ
@bender Did they win though? Did they?! ๐Ÿคฃ
@abucci This is already in place. It will error, return 404 Feed Not Found for non-browsers and external feeds are never fetched (_unless you are an authenticated/valid user of the pod_) -- I patched that hole a while ago, because I already picked up it was being abused by bots ๐Ÿค–
@abucci This is already in place. It will error, return 404 Feed Not Found for non-browsers and external feeds are never fetched (_unless you are an authenticated/valid user of the pod_) -- I patched that hole a while ago, because I already picked up it was being abused by bots ๐Ÿค–
Support (_and thus abuse reports_) are now disabled on this pod. There's now a new setting in Settings -> Poderator Settings called "Disable Support".
Support (_and thus abuse reports_) are now disabled on this pod. There's now a new setting in Settings -> Poderator Settings called "Disable Support".
This happens again today. This is twice just today alone. Hmm I'm reconsidering this feature entirely, rarely used and if it's just going to be abused by spammer, I don't see the value in it. I'm certainly not going to try to build some kind of "anti-spam" filters or anything, sounds cool, I'd learn a lot, but smells of effort and time I simply don't have ๐Ÿ˜ข #spam #sucks
This happens again today. This is twice just today alone. Hmm I'm reconsidering this feature entirely, rarely used and if it's just going to be abused by spammer, I don't see the value in it. I'm certainly not going to try to build some kind of "anti-spam" filters or anything, sounds cool, I'd learn a lot, but smells of effort and time I simply don't have ๐Ÿ˜ข #spam #sucks
@slashdot OMG! ๐Ÿ˜ฆ What da hell is going on here?! I used to have a friend that came from North Carolina, this is terrible (attacking power grids) ๐Ÿคฌ wtf are these people smoking?! ๐Ÿšฌ
@slashdot OMG! ๐Ÿ˜ฆ What da hell is going on here?! I used to have a friend that came from North Carolina, this is terrible (attacking power grids) ๐Ÿคฌ wtf are these people smoking?! ๐Ÿšฌ
Like why does spammers even bother?! Don't they realize how fucking futile and useless it is to be abuse something like a support form? I mean clearly nothing is going to come of this, except it's going to be clearly ignored and toss in the bin. ๐Ÿคฃ
Like why does spammers even bother?! Don't they realize how fucking futile and useless it is to be abuse something like a support form? I mean clearly nothing is going to come of this, except it's going to be clearly ignored and toss in the bin. ๐Ÿคฃ
Wow! My god spammers really try hard song they? ๐Ÿคฃ Geez ๐Ÿคฆโ€โ™‚๏ธ Do we need to make the captcha harder? ๐Ÿ™„
Wow! My god spammers really try hard song they? ๐Ÿคฃ Geez ๐Ÿคฆโ€โ™‚๏ธ Do we need to make the captcha harder? ๐Ÿ™„
@lyse to be fair the settings that you can change in the user interface are persisted to the settings YAML file and yes override any environmental command online options. This is always made sense to me because there are subset of settings that can be changed dynamically at runtime without requiring any restart.
@lyse to be fair the settings that you can change in the user interface are persisted to the settings YAML file and yes override any environmental command online options. This is always made sense to me because there are subset of settings that can be changed dynamically at runtime without requiring any restart.
@abucci It's good enough IMO ๐Ÿคž
@abucci It's good enough IMO ๐Ÿคž
@aelaraji It is if he's turned off open profiles ๐Ÿ‘Œ
@aelaraji It is if he's turned off open profiles ๐Ÿ‘Œ
@lyse thank you! ๐Ÿ™‡โ€โ™‚๏ธ
@lyse thank you! ๐Ÿ™‡โ€โ™‚๏ธ
@quark Thanks! ๐Ÿ˜…
@quark Thanks! ๐Ÿ˜…
I'm happy with the current implementation though, because the only reason you should be hitting the external profile endpoint at all is a) you're logged in and happen to click on someone's profile that is external to the pod or b) you're anonymous and just clicking through the frontpage (see a)
I'm happy with the current implementation though, because the only reason you should be hitting the external profile endpoint at all is a) you're logged in and happen to click on someone's profile that is external to the pod or b) you're anonymous and just clicking through the frontpage (see a)
@bender The problem with this is we just don't know until we try. That's why if the external feed you're looking for isn't found in the cache, it'll try to fetch it in the background. It's a bit of a sucky UX really, but its better than the experience of "waiting, waiting waiting and then timeout".
@bender The problem with this is we just don't know until we try. That's why if the external feed you're looking for isn't found in the cache, it'll try to fetch it in the background. It's a bit of a sucky UX really, but its better than the experience of "waiting, waiting waiting and then timeout".
I've pushed a slightly improved version of this that will return a HTTP 404 Not Found if the UserAgent is determined to NOT be a Browser.
I've pushed a slightly improved version of this that will return a HTTP 404 Not Found if the UserAgent is determined to NOT be a Browser.
Now responds with an error page; but still a 200 OK which I'm not entirely sure I agree with or am happy with? Hmm
Now responds with an error page; but still a 200 OK which I'm not entirely sure I agree with or am happy with? Hmm
@abucci I've fixed this and pushed a commit to main. Will test it on my pod and see how it goes. Basically reporting an error if the feed isn't in the cache and you're unauthenticated (anonymous).
@abucci I've fixed this and pushed a commit to main. Will test it on my pod and see how it goes. Basically reporting an error if the feed isn't in the cache and you're unauthenticated (anonymous).
@movq Yup! ๐Ÿ‘Œ
@movq Yup! ๐Ÿ‘Œ
@abucci Hmmm I'll have a look at this today, hopefully. ๐Ÿคž Thanks for bringing this up! ๐Ÿ™‡โ€โ™‚๏ธ
@abucci Hmmm I'll have a look at this today, hopefully. ๐Ÿคž Thanks for bringing this up! ๐Ÿ™‡โ€โ™‚๏ธ
@xuu we really must invest more time and effort into salty im ๐Ÿคฃ
@xuu we really must invest more time and effort into salty im ๐Ÿคฃ
@aelaraji good morning! Are you wake up that early?! ๐Ÿคฃ
@aelaraji good morning! Are you wake up that early?! ๐Ÿคฃ
@movq it looks like it would kill you. ๐Ÿ˜…
@movq it looks like it would kill you. ๐Ÿ˜…
@lyse Definately! Next time I'll make sure I announce it a few days in advance. ๐Ÿ™„๐Ÿ˜…
@lyse Definately! Next time I'll make sure I announce it a few days in advance. ๐Ÿ™„๐Ÿ˜…
This โ˜๏ธ
This โ˜๏ธ
@movq I certainly am ๐Ÿคฃ I wonder whether she is one of the yontest? ๐Ÿค”
@movq I certainly am ๐Ÿคฃ I wonder whether she is one of the yontest? ๐Ÿค”
My 9yr old daughter just made her Git commit today, her first website, setup two-factor authentication and used several credentials (_which I helped her with_) ๐Ÿคฃ -- next lessons: password hygiene/management.
My 9yr old daughter just made her Git commit today, her first website, setup two-factor authentication and used several credentials (_which I helped her with_) ๐Ÿคฃ -- next lessons: password hygiene/management.
@movq It's all in this single [Converge()](https://git.mills.io/yarnsocial/yarn/src/commit/4f8a3570cf93dd02d1a4652131a6ac4f3ac22884/internal/cache.go#L1403-L1456) method. Let me see if I can decipher and document wtf is going on here...
@movq It's all in this single [Converge()](https://git.mills.io/yarnsocial/yarn/src/commit/4f8a3570cf93dd02d1a4652131a6ac4f3ac22884/internal/cache.go#L1403-L1456) method. Let me see if I can decipher and document wtf is going on here...
@movq Yeah, they should rotate though, but yes.
@movq Yeah, they should rotate though, but yes.
@hacker-news-newest Oh! Wow! ๐Ÿ˜ฎ Does anyone know or have any insight as to the arrest of the Telegram CEO (Pavel Durov)? ๐Ÿค” #Telegram
@hacker-news-newest Oh! Wow! ๐Ÿ˜ฎ Does anyone know or have any insight as to the arrest of the Telegram CEO (Pavel Durov)? ๐Ÿค” #Telegram
@aelaraji Please do! ๐Ÿ˜