# I am the Watcher. I am your guide through this vast new twtiverse.
#
# Usage:
# https://watcher.sour.is/api/plain/users View list of users and latest twt date.
# https://watcher.sour.is/api/plain/twt View all twts.
# https://watcher.sour.is/api/plain/mentions?uri=:uri View all mentions for uri.
# https://watcher.sour.is/api/plain/conv/:hash View all twts for a conversation subject.
#
# Options:
# uri Filter to show a specific users twts.
# offset Start index for quey.
# limit Count of items to return (going back in time).
#
# twt range = 1 2
# self = https://watcher.sour.is/conv/ekpii7a
I have trouble with a web crawler using the TOR network. It's misusing the gopher proxy on my page. I don't want to disable/block tor (that would be the easy way out). It's permanently changing user agents and ignoring robots.txt. It ignores HTTP status codes. I'm currently serving it 4MB binary garbage in form of Link. It sucked in about 40GB of data now, but it doesn't explode and keeps crawling. Any other idea about what to do with it?=
I have trouble with a web crawler using the TOR network. It's misusing the gopher proxy on my page. I don't want to disable/block tor (that would be the easy way out). It's permanently changing user agents and ignoring robots.txt. It ignores HTTP status codes. I'm currently serving it 4MB binary garbage in form of Link. It sucked in about 40GB of data now, but it doesn't explode and keeps crawling. Any other idea about what to do with it?=