# I am the Watcher. I am your guide through this vast new twtiverse.
#
# Usage:
# https://watcher.sour.is/api/plain/users View list of users and latest twt date.
# https://watcher.sour.is/api/plain/twt View all twts.
# https://watcher.sour.is/api/plain/mentions?uri=:uri View all mentions for uri.
# https://watcher.sour.is/api/plain/conv/:hash View all twts for a conversation subject.
#
# Options:
# uri Filter to show a specific users twts.
# offset Start index for quey.
# limit Count of items to return (going back in time).
#
# twt range = 1 12
# self = https://watcher.sour.is/conv/vkzs7ma
@prologic I'm taking a look at yarn.social search codebase. Just created a tiny PR to add some documentation around setup. I was thinking that I might start with "Document how to use the query language really well" since that would be a good learning exercise. It looks like maybe https://search.twtxt.net/help would be a good place to put that work. Were you wanting to put any search help text anywhere else in the UI?
@brasshopper Oh! π€¦ββοΈ That was you π
I _should_ have known π€£ -- The biggest thing I want to see improved as well is the memory usage of this thing, it keeps triggering "memory pressure" alarms and its kind of annoying, not really sure what to do about it yet...
@brasshopper Oh! π€¦ββοΈ That was you π
I _should_ have known π€£ -- The biggest thing I want to see improved as well is the memory usage of this thing, it keeps triggering "memory pressure" alarms and its kind of annoying, not really sure what to do about it yet...
Also note that whatever improvements you make here in the global search engine, I'd like to bring across to yarnd itself -- So let's build it like a library as well as a search engine and crawler (server) π
Also note that whatever improvements you make here in the global search engine, I'd like to bring across to yarnd itself -- So let's build it like a library as well as a search engine and crawler (server) π
@prologic Yeah, I figured Iβd finally dox myself (my git.mills.io profile). π
re memory usage: is there some memory target youβre trying to hit? I would suspect that it partly depends on the data set, but maybe that plus search stats could give me enough to figure out a good set of test data.
@brasshopper So right now the code uses bleve to index documents (feeds and twts here) and is also used as the query engine. It has thing thing called MemoryNeededForSearchResult that _might_ be useful here.
Alternatively there is another good Go library for indexing and searching called bluge that I've been following, it _might_ be worthwhile looking in to...
@brasshopper So right now the code uses bleve to index documents (feeds and twts here) and is also used as the query engine. It has thing thing called MemoryNeededForSearchResult that _might_ be useful here.
Alternatively there is another good Go library for indexing and searching called bluge that I've been following, it _might_ be worthwhile looking in to...
Also to give you an idea... This pile of crappy ass code I whacked together on a weekend I _think_ could be a lot more efficient π
The Twtxt search space is really not that large to warrant this kind of resource utilisation: 
-- I don't mind the daily CPU spikes -- But even that _could_ be improved I _think_.
Also to give you an idea... This pile of crappy ass code I whacked together on a weekend I _think_ could be a lot more efficient π
The Twtxt search space is really not that large to warrant this kind of resource utilisation: -- I don't mind the daily CPU spikes -- But even that _could_ be improved I _think_.