# I am the Watcher. I am your guide through this vast new twtiverse.
# 
# Usage:
#     https://watcher.sour.is/api/plain/users              View list of users and latest twt date.
#     https://watcher.sour.is/api/plain/twt                View all twts.
#     https://watcher.sour.is/api/plain/mentions?uri=:uri  View all mentions for uri.
#     https://watcher.sour.is/api/plain/conv/:hash         View all twts for a conversation subject.
# 
# Options:
#     uri     Filter to show a specific users twts.
#     offset  Start index for quey.
#     limit   Count of items to return (going back in time).
# 
# twt range = 1 61083
# self = https://watcher.sour.is?uri=https://twtxt.net/user/prologic/twtxt.txt&offset=35391
# next = https://watcher.sour.is?uri=https://twtxt.net/user/prologic/twtxt.txt&offset=35491
# prev = https://watcher.sour.is?uri=https://twtxt.net/user/prologic/twtxt.txt&offset=35291
itamadev/golang-flutter-app-ffi: A Flutter app with Golang as a backend using FFI to bind and communicate between them

Working with the author of this little skeleton that is _attempting_ to try to demonstrate how to use FFI with Flutter on both Android and iOS -- Useful in in first-class Mobile App for Salty.im without having to rewrite all our client/crypto Go code in Dart (_uggh 🤮_) cc @eaplmx #Salty.im #Flutter #FFI_
itamadev/golang-flutter-app-ffi: A Flutter app with Golang as a backend using FFI to bind and communicate between them

Working with the author of this little skeleton that is _attempting_ to try to demonstrate how to use FFI with Flutter on both Android and iOS -- Useful in in first-class Mobile App for Salty.im without having to rewrite all our client/crypto Go code in Dart (_uggh 🤮_) cc @eaplmx #Salty.im #Flutter #FFI_
Totally my bad 🤦‍♂️
Totally my bad 🤦‍♂️
@darch Ahh! I _think_ know what's going on...

Also this link is a better view (ignoring whitespace ahgnes).

This function is used by two kinds of buttons that are nested differently and have different DOM structure. You'll have to write two different versions of this function, or find a better way to match the button(s).
@darch Ahh! I _think_ know what's going on...

Also this link is a better view (ignoring whitespace ahgnes).

This function is used by two kinds of buttons that are nested differently and have different DOM structure. You'll have to write two different versions of this function, or find a better way to match the button(s).
👋 Hey y'all Yarn.social pod operators... Looking for testing of this PR and the associated CLI tools there-in:

#1010 - Add support for indexing and searching an archive - yarn - Mills

cc @jlj @darch @xuu @eldersnake @abucci -- Let me know what you think of the interface, capabilities and so on. I plan to integrate this soon into the Web UI and API for yarnd 🤞
👋 Hey y'all Yarn.social pod operators... Looking for testing of this PR and the associated CLI tools there-in:

#1010 - Add support for indexing and searching an archive - yarn - Mills

cc @jlj @darch @xuu @eldersnake @abucci -- Let me know what you think of the interface, capabilities and so on. I plan to integrate this soon into the Web UI and API for yarnd 🤞
@justamoment That's just it...

> Another risk on a persistent comment system is to remember to have a way to moderate the posts, since you need to control what gets on your page and can potentially stay there forever.

This ☝️ There will be zero ways to "moderate" or "filter" such content if one were to use teh soon and upcoming search of any given Yarn.social pod. The only way for a user to filter content or hide things is to have an account on said pod where things like "hidden twts" and "muted feeds" can be taken into account.

It _might_ be possible to do things like "exclusion matches" and such, but now that's getting into some tricky territory. Oh well, let's see how it goes 😅 -- Google already indexes all pods anyway 🤣
@justamoment That's just it...

> Another risk on a persistent comment system is to remember to have a way to moderate the posts, since you need to control what gets on your page and can potentially stay there forever.

This ☝️ There will be zero ways to "moderate" or "filter" such content if one were to use teh soon and upcoming search of any given Yarn.social pod. The only way for a user to filter content or hide things is to have an account on said pod where things like "hidden twts" and "muted feeds" can be taken into account.

It _might_ be possible to do things like "exclusion matches" and such, but now that's getting into some tricky territory. Oh well, let's see how it goes 😅 -- Google already indexes all pods anyway 🤣
Oh 🤦‍♂️

> @tkanos dropped by to say hello and showed us his strange hotel room

Nevermind 😅
Oh 🤦‍♂️

> @tkanos dropped by to say hello and showed us his strange hotel room

Nevermind 😅
@mckinley @tkanos also joined us for a bit too and talked about his new Twtxt/Yarn client twx 👌
@mckinley @tkanos also joined us for a bit too and talked about his new Twtxt/Yarn client twx 👌
@darch Thanks for fixing this! 👌
@darch Thanks for fixing this! 👌
@darch Awesome! I'll jump on now then 👌

=> https://meet.jit.si/Yarn.social

cc @xuu if it's not too late for you 🤞=
@darch Awesome! I'll jump on now then 👌

=> https://meet.jit.si/Yarn.social

cc @xuu if it's not too late for you 🤞=
@xuu Looks like darch is awake 😅 We _could_ start now? 🤔
@xuu Looks like darch is awake 😅 We _could_ start now? 🤔
@xuu 5am UTC
@xuu 5am UTC
@xuu In one hour
@xuu In one hour
@mckinley True but we're talking about two different cases here 🤔
@mckinley True but we're talking about two different cases here 🤔
@mckinley Working on it 👌
@mckinley Working on it 👌
@xuu @lyse ?
@xuu @lyse ?
@mckinley Cool! 👌 @darch what about you? 🤔
@mckinley Cool! 👌 @darch what about you? 🤔
Success with Bluge 🥳
Success with Bluge 🥳
If y'all are still up for it 😅
If y'all are still up for it 😅
Actually I've just been informed, it _may_ still work out 😅
Actually I've just been informed, it _may_ still work out 😅
@ocdtrekkie Actually maybe we'll skip this week? My wife has double booked several things today, and that means I have a conflict at the same time 😅
@ocdtrekkie Actually maybe we'll skip this week? My wife has double booked several things today, and that means I have a conflict at the same time 😅
Who's coming to this weeks call? 👋
Who's coming to this weeks call? 👋
@justamoment Yup 👌
@justamoment Yup 👌
@justamoment Ahh I see 👌 -- Let's see what happens after I finish building an indexable and searchable archive 🤞
@justamoment Ahh I see 👌 -- Let's see what happens after I finish building an indexable and searchable archive 🤞
@justamoment Haha 😅 The way I see this working is, we present an interface that allows you to query the archive and load up older Twts (or replies).
@justamoment Haha 😅 The way I see this working is, we present an interface that allows you to query the archive and load up older Twts (or replies).
@justamoment This is already possible thanks to "Bookmarks". If you bookmark a twt, you always have access to it. There is a built-in feature of the interface. But you _could_ also use your Browser's own bookmarking facilities too.
@justamoment This is already possible thanks to "Bookmarks". If you bookmark a twt, you always have access to it. There is a built-in feature of the interface. But you _could_ also use your Browser's own bookmarking facilities too.
So... Indexing an old archive of this pod:


$ time ./index_archive ./data
...
ERRO[1046] error decoding archived twt data/archive/fd/bcbe3e.json  error="unexpected end of JSON input"
ERRO[1048] error decoding archived twt data/archive/fe/51cf5a.json  error="unexpected end of JSON input"

real    17m35.719s
user    16m42.758s
sys     3m48.809s


There were (sadly) 332 broken archived twts my archive 😢

Other interesting tidbits:


$ du -s -h data/archive/
5.0G    data/archive/

$ du -s -h data/index/
555M    data/index/


@xuu Feel like working on an archive compaction? 😅
So... Indexing an old archive of this pod:


$ time ./index_archive ./data
...
ERRO[1046] error decoding archived twt data/archive/fd/bcbe3e.json  error="unexpected end of JSON input"
ERRO[1048] error decoding archived twt data/archive/fe/51cf5a.json  error="unexpected end of JSON input"

real    17m35.719s
user    16m42.758s
sys     3m48.809s


There were (sadly) 332 broken archived twts my archive 😢

Other interesting tidbits:


$ du -s -h data/archive/
5.0G    data/archive/

$ du -s -h data/index/
555M    data/index/


@xuu Feel like working on an archive compaction? 😅
@justamoment I'm building an indexer now to indexnand query an archive. So let's see how that goes 🤞
@justamoment I'm building an indexer now to indexnand query an archive. So let's see how that goes 🤞
@justamoment I'm actually working on it now 😅 -- But philosophically I've always _believe_ (and still do) two things:

- We really don't care about old content so much or what people said in the past
- We really should get away from the notion of "oh buy you said blah blah"

Elaborating on my 2nd opinion, it is dangerous to make it easy to pull up what someone said or had an opinion on in the past. What if they have since changed their mind? What if they no longer believe in what they once said? What if it was something stupid and nefarious person(s) can/will hold that "post" accountable to them?

The _reason_ yarnd does this sort of thing in the first place is two-fold:

- It's an optimization and keeps Pods limited in "scale" (by design)
- It's a feature my old man talked to me about that I also believe in (see above).
@justamoment I'm actually working on it now 😅 -- But philosophically I've always _believe_ (and still do) two things:

- We really don't care about old content so much or what people said in the past
- We really should get away from the notion of "oh buy you said blah blah"

Elaborating on my 2nd opinion, it is dangerous to make it easy to pull up what someone said or had an opinion on in the past. What if they have since changed their mind? What if they no longer believe in what they once said? What if it was something stupid and nefarious person(s) can/will hold that "post" accountable to them?

The _reason_ yarnd does this sort of thing in the first place is two-fold:

- It's an optimization and keeps Pods limited in "scale" (by design)
- It's a feature my old man talked to me about that I also believe in (see above).
blugelabs/bluge: indexing library for Go Playing around with this library today to see if we can build an indexable archive for yarnd. First task, see how long it takes to walk the archive:


$ time ./index_archive ./data
...
INFO[0489] Twt 7776khq
INFO[0489] Twt vkkyqca

real    8m9.652s
user    7m35.754s
sys     2m31.565s
blugelabs/bluge: indexing library for Go Playing around with this library today to see if we can build an indexable archive for yarnd. First task, see how long it takes to walk the archive:


$ time ./index_archive ./data
...
INFO[0489] Twt 7776khq
INFO[0489] Twt vkkyqca

real    8m9.652s
user    7m35.754s
sys     2m31.565s
Hmmm ever noticed that most Twt hashes either end in a or q 🤔
Hmmm ever noticed that most Twt hashes either end in a or q 🤔
@slashdot Creepy 🙄
@slashdot Creepy 🙄
@abucci This is pretty interesting... Sadly I don't know enough enough about market dynamics to respond here, so instead I've shared with my economist neighbor for comments/thoughts 😅 -- It would be fun though to try to write a computer program that simulates some of these things? 🤔
@abucci This is pretty interesting... Sadly I don't know enough enough about market dynamics to respond here, so instead I've shared with my economist neighbor for comments/thoughts 😅 -- It would be fun though to try to write a computer program that simulates some of these things? 🤔
@justamoment This is sort of possible, kind of, by tweaking:


$ yarnd --help 2>&1 | grep 'max-cache'
      --max-cache-fetchers int        set maximum numnber of fetchers to use for feed cache updates (default 12)
  -I, --max-cache-items int           maximum cache items (per feed source) of cached twts in memory (default 150)
  -C, --max-cache-ttl duration        maximum cache ttl (time-to-live) of cached twts in memory (default 240h0m0s)


But you would have to supply insanely large values. Better solution would be to improve the search codebase and then incorporate those improvements (hopefully in a reusable way) into yarnd itself so we can index and search the archive.
@justamoment This is sort of possible, kind of, by tweaking:


$ yarnd --help 2>&1 | grep 'max-cache'
      --max-cache-fetchers int        set maximum numnber of fetchers to use for feed cache updates (default 12)
  -I, --max-cache-items int           maximum cache items (per feed source) of cached twts in memory (default 150)
  -C, --max-cache-ttl duration        maximum cache ttl (time-to-live) of cached twts in memory (default 240h0m0s)


But you would have to supply insanely large values. Better solution would be to improve the search codebase and then incorporate those improvements (hopefully in a reusable way) into yarnd itself so we can index and search the archive.
@justamoment @eaplmx Yes that's right. And this behaviour is configurable. The defaults are pretty good, 150 Twts per feed or 10 days (whichever is reached first) before they're archived. The only _real_ reason we can't display older posts in a thread/yarn is because the archive doesn't have an index. We _can_ fix that. For example this twt https://twtxt.net/twt/o6dsrga was from ~2 years ago now but is still accessible, just none of the replies are 😅~
@justamoment @eaplmx Yes that's right. And this behaviour is configurable. The defaults are pretty good, 150 Twts per feed or 10 days (whichever is reached first) before they're archived. The only _real_ reason we can't display older posts in a thread/yarn is because the archive doesn't have an index. We _can_ fix that. For example this twt https://twtxt.net/twt/o6dsrga was from ~2 years ago now but is still accessible, just none of the replies are 😅~
@movq No need to apologise for life getting in the way 🤗
@movq No need to apologise for life getting in the way 🤗
@darch Awesome! 👌
@darch Awesome! 👌
@taigrr Very good 👌 And many thanks! 🙏
@taigrr Very good 👌 And many thanks! 🙏
@eaplmx Looks like taigrr@ndl.taigrr.com beat you to it 😆
@eaplmx Looks like @taigrr beat you to it 😆
@eaplmx Looks like @taigrr beat you to it 😆
Hey y'all US friends 🤗 Anyone want $14.45 USD? I have a _virtual_ MasterCard here that apparently I can't even use (_can only use in the US 🙄_) that I got from a class action suit. First US Yarn.social netizen to respond gets it 😅 (_of course you'll have to figure out how to securely reach me too and all that, Email? Signal? Salty Chat? 😅)

NO THIS IS NOT A SCAM 🤣
Hey y'all US friends 🤗 Anyone want $14.45 USD? I have a _virtual_ MasterCard here that apparently I can't even use (_can only use in the US 🙄_) that I got from a class action suit. First US Yarn.social netizen to respond gets it 😅 (of course you'll have to figure out how to securely reach me too and all that, Email? Signal? Salty Chat? 😅)

NO THIS IS NOT A SCAM 🤣_
Hey y'all US friends 🤗 Anyone want $14.45 USD? I have a _virtual_ MasterCard here that apparently I can't even use (_can only use in the US 🙄_) that I got from a class action suit. First US Yarn.social netizen to respond gets it 😅 (of course you'll have to figure out how to securely reach me too and all that, Email? Signal? Salty Chat? 😅)

NO THIS IS NOT A SCAM 🤣_
@lyse 👌
@lyse 👌
@lyse True, but based on the analysis I've done, I _think_ I can fix the bad data in-place. That is, i can probably write some code that periodically checks the cache for bad data and deal with it, some examples:

- Check for dead feeds, and remove from cache
- Check for dead feeds and remove from User's Followers
- Check for redirects and rewrite urls
- etc...
@lyse True, but based on the analysis I've done, I _think_ I can fix the bad data in-place. That is, i can probably write some code that periodically checks the cache for bad data and deal with it, some examples:

- Check for dead feeds, and remove from cache
- Check for dead feeds and remove from User's Followers
- Check for redirects and rewrite urls
- etc...
@mckinley Good point 👌
@mckinley Good point 👌
@justamoment Oh wow! 😳
@justamoment Oh wow! 😳
@darch Ok! 👌
@darch Ok! 👌
@eaplmx Cute 😊
@eaplmx Cute 😊
@eaplmx They were good questions 👌
@eaplmx They were good questions 👌
Did some more analysis on my pod's cache and there's quite a few bad feeds in the cache's Twter list => https://gist.githubusercontent.com/prologic/7c1bf78a4134fc582abfd4fd7d2a1516/raw/ea7634071006f00c82d44ab6d7989ef420568ffe/gistfile1.txt=
Did some more analysis on my pod's cache and there's quite a few bad feeds in the cache's Twter list => https://gist.githubusercontent.com/prologic/7c1bf78a4134fc582abfd4fd7d2a1516/raw/ea7634071006f00c82d44ab6d7989ef420568ffe/gistfile1.txt=
No dupes or bad data for @movq 👌
No dupes or bad data for @movq 👌