# I am the Watcher. I am your guide through this vast new twtiverse.
# 
# Usage:
#     https://watcher.sour.is/api/plain/users              View list of users and latest twt date.
#     https://watcher.sour.is/api/plain/twt                View all twts.
#     https://watcher.sour.is/api/plain/mentions?uri=:uri  View all mentions for uri.
#     https://watcher.sour.is/api/plain/conv/:hash         View all twts for a conversation subject.
# 
# Options:
#     uri     Filter to show a specific users twts.
#     offset  Start index for quey.
#     limit   Count of items to return (going back in time).
# 
# twt range = 1 31
# self = https://watcher.sour.is/conv/puxvjcq
mckinley
twtxt.net
View Thread
@lyse As far as I know, they're still visible in the Web UI. Although, in the mobile app and youtube.com, I believe it tells you that the video isn't available without having to click on it. They don't tell you that in the RSS feed, and I agree; it gets annoying.

If we had a custom feed generator that hooks directly into the YouTube API, I'll bet we could find that information and put "[Scheduled][Scheduled=][Scheduled][Scheduled=][Scheduled][Scheduled=][Scheduled][Scheduled=]" in the title for premieres and remove it when the video is available.
lyse
lyse.isobeef.org
View Thread
Hmmm, what is going on here? I noticed this a couple of times in the recent past already. Very old twts appear in my client as unread. The pattern seems to be that there is always repeated text in square brackets and some of them contain equal signs. Is yarnd corrupting feeds somehow? I kind of doubt that people actually typed that themselves.

Last time, it was @bender's feed that showed me new weird twts in my client. I don't remember the details, but I'm pretty sure it was this week. Refetching his feed a couple of times (across multiple days) and I got new messages.

And it just happened again, this time with @mckinley's feed. This twt from 2023-01-09T22:42:37Z here newly popped up, it contains magic bracketed text:

> […] I'll bet we could find that information and put "[Scheduled][Scheduled=][Scheduled][Scheduled=][Scheduled][Scheduled=][Scheduled][Scheduled=]" in the title for premieres and remove it when the video is available.

Currently, its hash is puxvjcq. There is no sign of evidence that this twt existed ever before. Maybe I'm doing something wrong. But https://twtxt.net/twt/puxvjcq 404s and the search engine also just gives me "error loading twt from archive" (quite a generic error message): https://search.twtxt.net/twt/puxvjcq

Just open https://twtxt.net/user/mckinley/twtxt.txt and search for ][Scheduled or =][ to find even more. This also reveals some "[email protected][email protected=]" stuff. Is that maybe coming from Clownflare?

In https://twtxt.net/user/bender/twtxt.txt it felt to me that the bracketed text from 2024-03-28T18:34:36Z always got duplicated each time it changed for whatever reason: "[162.159.205.11][162.159.205.11=]" etc.

mckinley and bender, do you recall actually typing that out or somehow updating your feeds on yarnd? Or am I just doing something wrong here? But the fact, that my browser shows the same stuff, I'm pretty sure it's not my client, that's messing things up here.

Any idea, @prologic?
lyse
lyse.isobeef.org
View Thread
Hmmm, what is going on here? I noticed this a couple of times in the recent past already. Very old twts appear in my client as unread. The pattern seems to be that there is always repeated text in square brackets and some of them contain equal signs. Is yarnd corrupting feeds somehow? I kind of doubt that people actually typed that themselves.

Last time, it was @bender's feed that showed me new weird twts in my client. I don't remember the details, but I'm pretty sure it was this week. Refetching his feed a couple of times (across multiple days) and I got new messages.

And it just happened again, this time with @mckinley's feed. This twt from 2023-01-09T22:42:37Z here newly popped up, it contains magic bracketed text:

> […] I'll bet we could find that information and put "[Scheduled]\n[Scheduled]\n[Scheduled]\n[Scheduled]\n" in the title for premieres and remove it when the video is available.

Currently, its hash is puxvjcq. There is no sign of evidence that this twt existed ever before. Maybe I'm doing something wrong. But https://twtxt.net/twt/puxvjcq 404s and the search engine also just gives me "error loading twt from archive" (quite a generic error message): https://search.twtxt.net/twt/puxvjcq

Just open https://twtxt.net/user/mckinley/twtxt.txt and search for ][Scheduled or =][ to find even more. This also reveals some "[email protected]\n" stuff. Is that maybe coming from Clownflare?

In https://twtxt.net/user/bender/twtxt.txt it felt to me that the bracketed text from 2024-03-28T18:34:36Z always got duplicated each time it changed for whatever reason: "[162.159.205.11]\n" etc.

mckinley and bender, do you recall actually typing that out or somehow updating your feeds on yarnd? Or am I just doing something wrong here? But the fact, that my browser shows the same stuff, I'm pretty sure it's not my client, that's messing things up here.

Any idea, @prologic?
lyse
lyse.isobeef.org
View Thread
Hmmm, what is going on here? I noticed this a couple of times in the recent past already. Very old twts appear in my client as unread. The pattern seems to be that there is always repeated text in square brackets and some of them contain equal signs. Is yarnd corrupting feeds somehow? I kind of doubt that people actually typed that themselves.

Last time, it was @bender's feed that showed me new weird twts in my client. I don't remember the details, but I'm pretty sure it was this week. Refetching his feed a couple of times (across multiple days) and I got new messages.

And it just happened again, this time with @mckinley's feed. This twt from 2023-01-09T22:42:37Z here newly popped up, it contains magic bracketed text:

> […] I'll bet we could find that information and put "[Scheduled][Scheduled][Scheduled][Scheduled][Scheduled][Scheduled][Scheduled][Scheduled]" in the title for premieres and remove it when the video is available.

Currently, its hash is puxvjcq. There is no sign of evidence that this twt existed ever before. Maybe I'm doing something wrong. But https://twtxt.net/twt/puxvjcq 404s and the search engine also just gives me "error loading twt from archive" (quite a generic error message): https://search.twtxt.net/twt/puxvjcq

Just open https://twtxt.net/user/mckinley/twtxt.txt and search for ][Scheduled or =][ to find even more. This also reveals some "[email protected][email protected]" stuff. Is that maybe coming from Clownflare?

In https://twtxt.net/user/bender/twtxt.txt it felt to me that the bracketed text from 2024-03-28T18:34:36Z always got duplicated each time it changed for whatever reason: "[162.159.205.11][162.159.205.11]" etc.

mckinley and bender, do you recall actually typing that out or somehow updating your feeds on yarnd? Or am I just doing something wrong here? But the fact, that my browser shows the same stuff, I'm pretty sure it's not my client, that's messing things up here.

Any idea, @prologic?
lyse
lyse.isobeef.org
View Thread
Hmmm, what is going on here? I noticed this a couple of times in the recent past already. Very old twts appear in my client as unread. The pattern seems to be that there is always repeated text in square brackets and some of them contain equal signs. Is yarnd corrupting feeds somehow? I kind of doubt that people actually typed that themselves.

Last time, it was @bender's feed that showed me new weird twts in my client. I don't remember the details, but I'm pretty sure it was this week. Refetching his feed a couple of times (across multiple days) and I got new messages.

And it just happened again, this time with @mckinley's feed. This twt from 2023-01-09T22:42:37Z here newly popped up, it contains magic bracketed text:

> \n I'll bet we could find that information and put "\n\n\n\n\n\n\n\n" in the title for premieres and remove it when the video is available.

Currently, its hash is puxvjcq. There is no sign of evidence that this twt existed ever before. Maybe I'm doing something wrong. But https://twtxt.net/twt/puxvjcq 404s and the search engine also just gives me "error loading twt from archive" (quite a generic error message): https://search.twtxt.net/twt/puxvjcq

Just open https://twtxt.net/user/mckinley/twtxt.txt and search for ][Scheduled or =][ to find even more. This also reveals some "\n\n" stuff. Is that maybe coming from Clownflare?

In https://twtxt.net/user/bender/twtxt.txt it felt to me that the bracketed text from 2024-03-28T18:34:36Z always got duplicated each time it changed for whatever reason: "\n\n" etc.

mckinley and bender, do you recall actually typing that out or somehow updating your feeds on yarnd? Or am I just doing something wrong here? But the fact, that my browser shows the same stuff, I'm pretty sure it's not my client, that's messing things up here.

Any idea, @prologic?
movq
www.uninformativ.de
View Thread
@lyse puxvjcq certainly does not exist in my cache.

(I feel the need to implement jenny --debugfeed <url> now which just fetches a URL and prints <hash> <original_line> for the raw feed. Could have used this a couple of times already.)
movq
www.uninformativ.de
View Thread
@lyse puxvjcq certainly does not exist in my cache.

(I feel the need to implement jenny --debugfeed <url> now which just fetches a URL and prints <hash> <original_line> for the raw feed. Could have used this a couple of times already.)
movq
www.uninformativ.de
View Thread
@lyse puxvjcq certainly does not exist in my cache.

(I feel the need to implement jenny --debugfeed <url> now which just fetches a URL and prints <hash> <original_line> for the raw feed. Could have used this a couple of times already.)
movq
www.uninformativ.de
View Thread
@lyse puxvjcq certainly does not exist in my cache.

(I feel the need to implement jenny --debugfeed <url> now which just fetches a URL and prints <hash> <original_line> for the raw feed. Could have used this a couple of times already.)
movq
www.uninformativ.de
View Thread
You twt is truncated on twtxt.net, btw. 🤔

https://movq.de/v/7cb8a3bad4/huh2.png
movq
www.uninformativ.de
View Thread
You twt is truncated on twtxt.net, btw. 🤔

https://movq.de/v/7cb8a3bad4/huh2.png
movq
www.uninformativ.de
View Thread
You twt is truncated on twtxt.net, btw. 🤔

https://movq.de/v/7cb8a3bad4/huh2.png
movq
www.uninformativ.de
View Thread
You twt is truncated on twtxt.net, btw. 🤔

https://movq.de/v/7cb8a3bad4/huh2.png
lyse
lyse.isobeef.org
View Thread
Now, https://twtxt.net/twt/puxvjcq from my original post also works. Yarnd must have parsed it and indexed in the meantime. However, it renders the truncated version of mckinley's message. Notice that it directly ends at the beginning of the bracketed text.
mckinley
twtxt.net
View Thread
Definitely something going on here. Cloudflare is my main suspect.
prologic
twtxt.net
View Thread
Nope none that I can think of 🤔
prologic
twtxt.net
View Thread
Nope none that I can think of 🤔
lyse
lyse.isobeef.org
View Thread
And now, @bender's feed changed, too. Bracketed text got duplicated once again.

How do the feeds look on disk? Do they already contain this bracketed text?

For reference, I just placed a copy of the feed here: https://lyse.isobeef.org/tmp/bracketed-text/bender-2024-08-04-10-34.txt

I haven't marked the changed twts by @mckinley as read last time, so I don't know if something changed there as well. In any case, current snapshot: https://lyse.isobeef.org/tmp/bracketed-text/mckinley-2024-08-04-10-39.txt

Yarnd gossipping might be the reason for the truncated stuff, @prologic. Who are your peers? Any obvious broken yarnd version or even some kind of an attacker involved? But there must be something else broken in yarnd for the bracketed text to be duplicated.
prologic
twtxt.net
View Thread
@lyse This is really weird. Do you have an example of this bracketed text? Re peers, I'm aware of all the peers, nothing surprising there.
prologic
twtxt.net
View Thread
@lyse This is really weird. Do you have an example of this bracketed text? Re peers, I'm aware of all the peers, nothing surprising there.
prologic
twtxt.net
View Thread
Hmm I see it! It's so obvious 🤦‍♂️ I smell an attack of some kind.
prologic
twtxt.net
View Thread
Hmm I see it! It's so obvious 🤦‍♂️ I smell an attack of some kind.
lyse
lyse.isobeef.org
View Thread
@prologic Here's an attempt at an analysis: https://lyse.isobeef.org/tmp/bracketed-text/

I just set up a cronjob to fetch and analyze both feeds every six hours. I probably have to do some dedup, otherwise the list gets out of handy rather quickly.
prologic
twtxt.net
View Thread
@lyse Thank you! 🙏
prologic
twtxt.net
View Thread
@lyse Thank you! 🙏
lyse
lyse.isobeef.org
View Thread
Righto, @prologic, I just checkout out current main of yarnd (commit 5101ec240ddb0e5e39809bf8a7b847508b3ac298) and ran make dev. After registering a user and logging in, I then entered a twt with double bracketed text (without the equal sign on the second one, though) and it was expanded into eight brackets. So, this is clearly a bug. Let me dig deeper.

I hope I zoomed in enough, so you can read the stuff on my screenshot: https://lyse.isobeef.org/tmp/bracketed-text/bug.png
prologic
twtxt.net
View Thread
@lyse Holy fucking shit! You're right! You got me out of bed for this one, I spun my local dev instance and entered a Twt with [foo] and ended up with [foo][foo=][foo][foo=] wut da actual fuq?! 🤔
prologic
twtxt.net
View Thread
@lyse Holy fucking shit! You're right! You got me out of bed for this one, I spun my local dev instance and entered a Twt with [foo] and ended up with [foo][foo=][foo][foo=] wut da actual fuq?! 🤔
prologic
twtxt.net
View Thread
It's also (_expectedly_) in the feed file on disk:


2024-08-04T21:22:05+10:00	[foo][foo=][foo][foo=]

prologic
twtxt.net
View Thread
It's also (_expectedly_) in the feed file on disk:


2024-08-04T21:22:05+10:00	[foo][foo=][foo][foo=]

prologic
twtxt.net
View Thread
It's also (_expectedly_) in the feed file on disk:


2024-08-04T21:22:05+10:00\t[foo][foo=][foo][foo=]