# I am the Watcher. I am your guide through this vast new twtiverse.
#
# Usage:
# https://watcher.sour.is/api/plain/users View list of users and latest twt date.
# https://watcher.sour.is/api/plain/twt View all twts.
# https://watcher.sour.is/api/plain/mentions?uri=:uri View all mentions for uri.
# https://watcher.sour.is/api/plain/conv/:hash View all twts for a conversation subject.
#
# Options:
# uri Filter to show a specific users twts.
# offset Start index for quey.
# limit Count of items to return (going back in time).
#
# twt range = 1 9
# self = https://watcher.sour.is/conv/ggv2t5q
Python's aiohttp.web.Response has absolutely no way to exclude the Content-Type header in responses, it will always send one by falling back to application/octet-stream, regardless of what you do. For testing purposes I'd like to omit this response header, though. So now I monkey-patched my test server handler in the unit tests like this. Let's see when this falls apart with a future aiohttp version:
async def http200_no_content_type(request):
original_write_headers = request._payload_writer.write_headers
async def write_headers(status_line, headers):
del headers["Content-Type"]
await original_write_headers(status_line, headers)
request._payload_writer.write_headers = write_headers
return aiohttp.web.Response(body="abcäöüß".encode("utf-8"))=
@lyse And now you’ve reminded me that jenny’s test suite is sorely lacking. 😬
Are you using async stuff on purpose? It’s the new hype, I guess. 🤔
@lyse And now you’ve reminded me that jenny’s test suite is sorely lacking. 😬
Are you using async stuff on purpose? It’s the new hype, I guess. 🤔
@lyse And now you’ve reminded me that jenny’s test suite is sorely lacking. 😬
Are you using async stuff on purpose? It’s the new hype, I guess. 🤔
@movq There are two reasons: First, I wanted to have a look at this async/await stuff for several years now. This is the first time I actually get after it. During this endeavor I came across this nice What color is your function article. Highly recommended. And so far I completely agree with the author, Go's model using goroutines is much nicer to work with compared to the explicit style declaring where the task can be switched. But I'm far from being used to async/await. So maybe once I wrap my head around it, it's getting comparably easy.
And lastly, it's supposed to integrate well with urwid's event/IO loop which I rely on in tt. I finally try to fetch the feeds on my own, so I can rip out the twtxt reference implementation. Still a very long way to go, but I have to start somewhere, and why not directly at the root. :-) So far I thought the interoperability between asyncio and urwid would be really elegant, but turns out it's a bit more complicated than I imagined (probably because of Python 2 and ancient Python 3 versions support). But maybe I just don't know the hidden tricks.
Anyways, to not block the user input, I believe I have to do it this way, there's not much choice. Threads in Python are a joke with the GIL, so I don't want to open that can of worms. If tt wasn't a user driven program, I'd just fetch the feeds synchronously. No need for this more involved stuff if it runs in a cronjob.
On a closing note, once I finally fetch and process the feeds myself, I can also support Gopher/Gemini feeds and implement parts of the metadata spec. Currently, there's absolutely nothing available in the cache, that is written by the reference implementation and read by tt. Looking forward to that.
@lyse I see! Hmm, maybe I need to take another look at async/await. I’m using urwid in another project. 🤔
> If tt wasn't a user driven program, I'd just fetch the feeds synchronously.
I’m using Python’s multi-threading in jenny to fetch feeds, which gives a huge speedup. Without multi-threading, it takes 1 minute and 10 seconds to fetch all my feeds. With 64 workers, it’s down to 15 seconds. The good thing is that fetching feeds is network-bound, so the GIL doesn’t really matter.
Not sure if it really matters when run as a cronjob … but it’s nice when a manual call of jenny -f is relatively fast. 🙃
@lyse I see! Hmm, maybe I need to take another look at async/await. I’m using urwid in another project. 🤔
> If tt wasn't a user driven program, I'd just fetch the feeds synchronously.
I’m using Python’s multi-threading in jenny to fetch feeds, which gives a huge speedup. Without multi-threading, it takes 1 minute and 10 seconds to fetch all my feeds. With 64 workers, it’s down to 15 seconds. The good thing is that fetching feeds is network-bound, so the GIL doesn’t really matter.
Not sure if it really matters when run as a cronjob … but it’s nice when a manual call of jenny -f is relatively fast. 🙃
@lyse I see! Hmm, maybe I need to take another look at async/await. I’m using urwid in another project. 🤔
> If tt wasn't a user driven program, I'd just fetch the feeds synchronously.
I’m using Python’s multi-threading in jenny to fetch feeds, which gives a huge speedup. Without multi-threading, it takes 1 minute and 10 seconds to fetch all my feeds. With 64 workers, it’s down to 15 seconds. The good thing is that fetching feeds is network-bound, so the GIL doesn’t really matter.
Not sure if it really matters when run as a cronjob … but it’s nice when a manual call of jenny -f is relatively fast. 🙃
@movq Absolutely, go for it. It's quite a brain-twist to some extent. Yesterday night I found some more code by searching GitHub for urwid and asyncio combinations and I might be able to draw some inspiration from the results. Perhaps I can simplify one or the other. Let's see. Haven't fully grasped it yet. But there's potentially something.
And yes, this is the first time I actually used GitHub as a code search engine to try figuring things out by looking at real code and attempting to map their course of action to my problems. Some code looks like complete garbage, however, other was definitely written by someone who is much more knowledgeable than I am. Another thing I didn't think of before is that I somehow need to find a way to ignore duplicates. A lot of search results just list the same code for all the fifty quadrillion forks or more. That's a bit annoying. Yeah, one piece of code showed up literally over 40 times already.
Oh nice, this reduction is quite an improvement I wouldn't have guessed. And yes, faster programs are always great. Fully agree here.