# I am the Watcher. I am your guide through this vast new twtiverse.
#
# Usage:
# https://watcher.sour.is/api/plain/users View list of users and latest twt date.
# https://watcher.sour.is/api/plain/twt View all twts.
# https://watcher.sour.is/api/plain/mentions?uri=:uri View all mentions for uri.
# https://watcher.sour.is/api/plain/conv/:hash View all twts for a conversation subject.
#
# Options:
# uri Filter to show a specific users twts.
# offset Start index for quey.
# limit Count of items to return (going back in time).
#
# twt range = 1 3
# self = https://watcher.sour.is/conv/2n7pifa
So... The last couple of days for me at my day job has been figuring out how to tier our various environments and setup some kind of tiered Prometheus.
Federation was never going to work out so well for me because you can't scrape 10-20k time-series of another Prometheus server, the scraping loop will just timeout consume that amount of data.
Turns out there is a new feature in Prometheus v2.34.0 called "Agent mode" that allows one to run a Prometheus server in such a way that it does two very important things:
- Sends/writes all time-series it locally scrapes via Remote Write to another Prometheus server
- Keeps a WAL (write-ahead-log) to ensure no metric is ever lost and,
- only deletes the WAL entries on a successful write to the remote Prometheus.
This setup allows (along with appropriate External Labels) to have a very nice tiered Prometheus setup where you effectively have a central Prometheus server (with no scraping configuration, except for Prometheus itself) that acts as the Remote receiver for Prometheus agents.
#SRE #DevOps #Prometheus #Monitoring
So... The last couple of days for me at my day job has been figuring out how to tier our various environments and setup some kind of tiered Prometheus.
Federation was never going to work out so well for me because you can't scrape 10-20k time-series of another Prometheus server, the scraping loop will just timeout consume that amount of data.
Turns out there is a new feature in Prometheus v2.34.0 called "Agent mode" that allows one to run a Prometheus server in such a way that it does two very important things:
- Sends/writes all time-series it locally scrapes via Remote Write to another Prometheus server
- Keeps a WAL (write-ahead-log) to ensure no metric is ever lost and,
- only deletes the WAL entries on a successful write to the remote Prometheus.
This setup allows (along with appropriate External Labels) to have a very nice tiered Prometheus setup where you effectively have a central Prometheus server (with no scraping configuration, except for Prometheus itself) that acts as the Remote receiver for Prometheus agents.
#SRE #DevOps #Prometheus #Monitoring
@prologic nice I will read more about it. thanks