# I am the Watcher. I am your guide through this vast new twtiverse.
# 
# Usage:
#     https://watcher.sour.is/api/plain/users              View list of users and latest twt date.
#     https://watcher.sour.is/api/plain/twt                View all twts.
#     https://watcher.sour.is/api/plain/mentions?uri=:uri  View all mentions for uri.
#     https://watcher.sour.is/api/plain/conv/:hash         View all twts for a conversation subject.
# 
# Options:
#     uri     Filter to show a specific users twts.
#     offset  Start index for quey.
#     limit   Count of items to return (going back in time).
# 
# twt range = 1 60515
# self = https://watcher.sour.is?uri=https://twtxt.net/user/prologic/twtxt.txt&offset=60115
# next = https://watcher.sour.is?uri=https://twtxt.net/user/prologic/twtxt.txt&offset=60215
# prev = https://watcher.sour.is?uri=https://twtxt.net/user/prologic/twtxt.txt&offset=60015
Fixed. Rolling out shortly...
@bender Thanks I'll fix that!
@bender LOL πŸ˜‚ Bitcask is still the user database bahahaha πŸ˜…
@bender Of course 🀣
@bender Hah, gonna have to go to bed though I think. I hope no-one minds too badly that there are going to be bugs for and weird stuff for a bit. I'll try to fix more things tomorrow (Sunday). πŸ‘Œ
Pod feels a bit snapper for sure....
@bender Dunno 🀷 so far uptime is 4m so maybe not? 🀣
@bender Answered :D
@bender I did, but wasn't sure so I asked 🀣
@bender Backend is SQLite now. I switched drivers and so far so good. 'm seeing very stable CPU utilization, and a stable memory consumption of under 100MB
@bender As soon as the drop the notion of "old" twts, drop the re-fetch all feeds. Yes.
_some_ things _may_ be broken or not quite right. LMK! πŸ™
@bender Sort of, still fixing bugs. I had so much trouble with this sqlite driver, that in the end I swithced it to another driver and so far so good 😊 fingers crossed 🀞
Is this thing on?! 🀣
@kat Did you build the software for it? πŸ€”
Okay. let's see how things work now.... (_sorry for all the fucking around!_)
Hmmm
Okay we've survived for ~6m now (_beating our 4m numerous OOM kills_) 🀣 Also massive improvements in performance so far πŸ₯³~
Hmm I spoke to soonβ„’ :( πŸ™ Another OOM kill :/ πŸ˜•


 time="2025-04-05T03:26:12Z" level=info msg="[mem] Feeds fetch started: HeapAlloc=34MB Goroutines=31"
 time="2025-04-05T03:27:17Z" level=info msg="[mem] Feeds fetch done: HeapAlloc=67MB Goroutines=92"
 time="2025-04-05T03:29:23Z" level=info msg="[mem] Feeds fetch started: HeapAlloc=56MB Goroutines=41"
 time="2025-04-05T03:29:23Z" level=info msg="[mem] Feeds fetch done: HeapAlloc=49MB Goroutines=39"
 time="2025-04-05T03:30:15Z" level=info msg="[mem] Feeds fetch started: HeapAlloc=36MB Goroutines=41"
 time="2025-04-05T03:30:15Z" level=info msg="[mem] Feeds fetch done: HeapAlloc=54MB Goroutines=40"
 time="2025-04-05T03:31:07Z" level=info msg="[mem] Feeds fetch started: HeapAlloc=53MB Goroutines=41"
 time="2025-04-05T03:31:21Z" level=info msg="[mem] Feeds fetch started: HeapAlloc=65MB Goroutines=95"
 time="2025-04-05T03:31:21Z" level=info msg="[mem] Feeds fetch done: HeapAlloc=69MB Goroutines=91"


Hmmm? πŸ€”
We're also running on the new SQLiteCache πŸ‘Œ
@xuu That last commit might have fixed the memory problem 🀞
Seem to be holding at 100-120MB RSS πŸ™
Trump is a fool !
@bender Holy moly! 🀯
@bender Haha 🀣
I'll fix this soon πŸ˜‚
@andros DRM is a solution to the wrong problem.
@bender So true πŸ˜₯
@javivf Welcome indeed πŸ€—
@eldersnake Same!
@andros That's a shame. But here's the thing about piracy. If I could download a PDF or MP3/OGG of a Book I'd happily pay for this. If I could download an MP3/OGG of some Music I would pay for this. If I could download an MP4 of a Movie I would pay for this.

The reason piracy exists at all is economics and greed. If you make something affordable and convenient, there's no need to steal.
Okay trying SqliteCache again with a slight tweak. I _might_ have to rethink the hard-coded LIMIT 50.
Going back to old MemoryCache for now and going to bed πŸ›Œ
Hmmm this needs a bit more work.
And we're back. On the new Sqlitecache πŸ₯³
Ooops I ran a docker build on one of my production nodes (_the ingress node_) 😱
@bender Okay πŸ‘Œ
πŸ‘‹ Folks that use this pod, I'll be rolling out an experimental new caching backend to this pod shortly. Stay tuned. Please provide feedback either here via Twtxt or on IRC. Thanks! πŸ™
There are however a few implications to note:

- It will be possibly to page through much larger quantities of twts per feed, this is potentially unlimited (_depends on disk space_)
- Automated Feeds (_bots_) like @tiktok will now behave/display very differently. You will see all it's historical Twts, whereas before you'd only see the one because the MemoryCache's behavior was to "replace" Twts.

I'm _hoping_ everything else remains the same and true to what we've collectively built and to spec. Replies work, Forks work, various views, filters and so on still work. I'm developing this new cache in a way that uses a "delegate" pattern and a double read / double write with metrics so I can over time see that none of the "old cache" is used anymore.
I _think_ my new SqliteCache is almost ready for prime time 🀞
This is why I blocked entire ASN(s) 🀣
@bender Soon 🀣
@bender Yeah me too hmm 🧐
@eapl.me@eapl.me Ahh I see πŸ˜…
@xuu Got the most basic view working πŸ˜…
@xuu Actually... I'm working on a new Sqlite backend/cache for yarnd 🀣 I _might_ revive yarns (_the crawler / search engine_) one day 🀞
@eapl.me@eapl.me What's with your client not using the proper syntax for mentions?


$ bat 'https://twtxt.net/twt/lnrgahq' | jq '.text'
"(#4xaabhq) thanks @prologic! 
@bender the idea of the RFC was to reach an agreement on a difficult problem, receiving proposals, and the voting is a simple count to gauge the sentiment of \"is this a problem worth to be fixed?, are we committed to implement a change in our clients?\" 

But that's a fair point. What do the community expect? What do y'all expect?"


πŸ€”
I wonder if I can do away with the followers follows mutes tables and expect the client to actually filter Twts appropriately before Display? This would simplify the SqliteCache considerably and also mean it would be agnostic of single-user or multi-user as that's delegated to another layer. Hmmm 🧐
Whoohoo! πŸ˜‡
@bender I never implemented it actually. That's why we have the # refresh = metadata field for those that yell loudly enough can add to their feeds. Otherwise yarnd uses WebSub between pods and is fairly dumb. I could never find an "intelligent" way to back-off without hurting freshness.
@eapl.me@eapl.me Done
@movq Hah I used to be a heavy mpd user myself once upon a time πŸ˜…
@xuu I had a lot of trouble figuring out how to do this too πŸ˜‚
Loved this so much I had to re-post it! 🀣 Thanks @xuu
@xuu LOL πŸ˜‚
LOL πŸ˜‚

:{:|:&};:
@bender I gave up after page 4 (DuckDuckGo) 🀣
@bender I tend to think of Twtxt like Email. It is _truely_ decentralised. So therefore by design there really is no way to easily discovers others except through social interactions and a sort of "word of mouth" of human exchanges of communications.
@xuu no disagreements from me!
Wait! What's going on?! 🧐
Yeah link please πŸ™
@lyse I _think_ the community should be allowed to experiment mate πŸ˜… Just make tt2 ignore such items in feeds and you're good πŸ‘
@eapl.me@eapl.me Yeah yarnd already filters/ignores them (_for now_)
It looks interesting but not enough for me to buy one πŸ˜…
@andros This is cool! 😍 Migut just have to add support for this to see how it might work in yarnd 🀣
@movq Is that the cwrsion that still ran on MS-DOS?
@andros Is this a book republished as a Markdown doc on GitHub or written that way?
@lyse I'm in the banking industry, so don't see this much🀣
@andros Cool! Can I joking ciwtuau? πŸ€”
@andros @lyse 's new client 😊
Unfortunately none of the content on the Wikipedia page for him really criticises much of his work. He must be really good at being a "businessman" and killing tech innovation 🀣
> After working 14 years at IBM, he became senior vice president and chief technology officer at enterprise search vendor Verity in 2004.[16][14][12] In July 2005, he was hired by Yahoo! to lead Yahoo! Research in Sunnyvale, California.[17] At Yahoo!, he worked on research projects including search and advertising.[15][18] In 2011, he was appointed as Yahoo!'s chief strategy officer.[19]
>
> In 2012, Prabhakar joined Google after severe funding cuts in Yahoo!'s research division.[19] In 2018, he was > put in charge of Ads and Commerce at Google and in 2020 his scope was expanded to include Search, Geo, and Assistant.[20] [21]
>
> In 2024, he transitioned to the role of Chief Technologist at Google.[2]
@thecanine I read this article and did some research. Wow! Amazing really how we can build really good tech, then kill it quietly all for the sake of more revenue and profits for shareholders 🀣
@movq It's stupid that this is geo blocked right? πŸ˜‚ Must be hmmm πŸ€”
@movq Hmm

> Video unavailable

πŸ₯²
So amazing! 🀩
@xuu Fuxk you're brilliant 😍
@xuu You're lucky πŸ€
@kat You are right! 🀣 That was a really lovely rendition of that song! 🎧 πŸ™Œ
@kat Sounds alright to me 😹
@kat That's thankfully not entirely true. There are some companies that are starting to realize that "Cloud" is _actually_ more expensive and so we're starting to see a bit of a migration back 🀞
@kat I _actually_ have experience building Caddy modules hmmm 🧐
@kat Interesting project πŸ€”
@andros Yes πŸ™Œ
Business trips / Off-Sites can be rather exhausting 😴
Yes
@xuu Nice! πŸ‘
@movq This is pretty good, love the music 🎢 Haha πŸ˜†
@xuu Got a sample to show/share? πŸ€”
@thecanine πŸ‘Œ
Kind of catchty πŸ˜„
@movq Bahahahahaha πŸ˜‚
@xuu Oh wow! Thank you ! πŸ™ And yeah even at 2x it sounds good enough to comprehend 🀣
@movq apparently very hard πŸ˜†
@movq I'd have to hear it to get it 🀣
(#uavwuqq) @eapl.me@eapl.me Nah, my pod seems to have the correct URL 🀣
@xuu Yeah looks like an edge case. Because of the way he announces his preferred nick in the feed the "Reply" button spits out @eapl.me@eapl.me@eapl.me for me, which then gets eaten as two mentions, probably matching twice against my following list?