# I am the Watcher. I am your guide through this vast new twtiverse.
# 
# Usage:
#     https://watcher.sour.is/api/plain/users              View list of users and latest twt date.
#     https://watcher.sour.is/api/plain/twt                View all twts.
#     https://watcher.sour.is/api/plain/mentions?uri=:uri  View all mentions for uri.
#     https://watcher.sour.is/api/plain/conv/:hash         View all twts for a conversation subject.
# 
# Options:
#     uri     Filter to show a specific users twts.
#     offset  Start index for quey.
#     limit   Count of items to return (going back in time).
# 
# twt range = 1 29
# self = https://watcher.sour.is/conv/pddmcjq
prologic
twtxt.net
View Thread
Hmmm πŸ€” _Somehow_ we went from ~276k archives Twts on this pod (twtxt.net) to 356k 😲

See o7yktgq and cykifgq

I guess someone followed a rather large feed containing 83k twts?! 😱 And we just archived them all?! 🀣~
prologic
twtxt.net
View Thread
Hmmm πŸ€” _Somehow_ we went from ~276k archives Twts on this pod (twtxt.net) to 356k 😲

See o7yktgq and cykifgq

I guess someone followed a rather large feed containing 83k twts?! 😱 And we just archived them all?! 🀣~
prologic
twtxt.net
View Thread
My god! Here's the graph of it!\n\n \n\nCan't tell whether this was malicious or benign πŸ€”\n\n
\n$ dki -t --rm -v twtxt_twtxt:/data alpine /bin/sh\n/ # cd /data/archive/\n/data/archive # du -s -h .\n1.7G\t.\n/data/archive #\n
\n\n😲
prologic@twtxt.net
twtxt.net
View Thread
My god! Here's the graph of it!\n\n \n\nCan't tell whether this was malicious or benign πŸ€”\n\n
\n$ dki -t --rm -v twtxt_twtxt:/data alpine /bin/sh\n/ # cd /data/archive/\n/data/archive # du -s -h .\n1.7G\t.\n/data/archive #\n
\n\n😲
prologic
twtxt.net
View Thread
My god! Here's the graph of it!



Can't tell whether this was malicious or benign πŸ€”


$ dki -t --rm -v twtxt_twtxt:/data alpine /bin/sh
/ # cd /data/archive/
/data/archive # du -s -h .
1.7G	.
/data/archive #


😲
prologic
twtxt.net
View Thread
My god! Here's the graph of it!



Can't tell whether this was malicious or benign πŸ€”


$ dki -t --rm -v twtxt_twtxt:/data alpine /bin/sh
/ # cd /data/archive/
/data/archive # du -s -h .
1.7G	.
/data/archive #


😲
prologic@twtxt.net
twtxt.net
View Thread
The interesting thing is that it's not really possible to do this as the cache limits what it fetches by config.MaxFetchlimit as seen [here](https://github.com/jointwt/twtxt/blob/2fe0d435a7ffadce412d82e23ae6e83739e9704e/internal/cache.go#) -- Unless (_assuming malicious here_) someone created a feed with ~80k old entries of nothing?~
prologic
twtxt.net
View Thread
The interesting thing is that it's not really possible to do this as the cache limits what it fetches by config.MaxFetchlimit as seen [here](https://github.com/jointwt/twtxt/blob/2fe0d435a7ffadce412d82e23ae6e83739e9704e/internal/cache.go#) -- Unless (_assuming malicious here_) someone created a feed with ~80k old entries of nothing?~
prologic
twtxt.net
View Thread
The interesting thing is that it's not really possible to do this as the cache limits what it fetches by config.MaxFetchlimit as seen [here](https://github.com/jointwt/twtxt/blob/2fe0d435a7ffadce412d82e23ae6e83739e9704e/internal/cache.go#) -- Unless (_assuming malicious here_) someone created a feed with ~80k old entries of nothing?~
prologic
twtxt.net
View Thread
Oh πŸ€¦β€β™‚οΈ I just worked out what that was! Ot was the new Twt hashing work you did @xuu which caused a en-mass re-archival to happen on my pod! 🀣
prologic
twtxt.net
View Thread
Oh πŸ€¦β€β™‚οΈ I just worked out what that was! Ot was the new Twt hashing work you did @xuu which caused a en-mass re-archival to happen on my pod! 🀣
prologic@twtxt.net
twtxt.net
View Thread
Oh πŸ€¦β€β™‚οΈ I just worked out what that was! Ot was the new Twt hashing work you did @xuu which caused a en-mass re-archival to happen on my pod! 🀣
prologic
twtxt.net
View Thread
I didn’t even notice until several days later so that means the pod performance works quite well πŸ‘Œ
prologic
twtxt.net
View Thread
I didn’t even notice until several days later so that means the pod performance works quite well πŸ‘Œ
prologic@twtxt.net
twtxt.net
View Thread
I didn’t even notice until several days later so that means the pod performance works quite well πŸ‘Œ
movq
www.uninformativ.de
View Thread
@prologic Sometimes the solution is simple, right? 😁 1.7 GB is a lot of stuff, though, is all that database overhead? Oof!
movq
www.uninformativ.de
View Thread
@prologic Sometimes the solution is simple, right? 😁 1.7 GB is a lot of stuff, though, is all that database overhead? Oof!
movq
www.uninformativ.de
View Thread
@prologic Sometimes the solution is simple, right? 😁 1.7 GB is a lot of stuff, though, is all that database overhead? Oof!
vain@www.uninformativ.de
www.uninformativ.de
View Thread
@prologic Sometimes the solution is simple, right? 😁 1.7 GB is a lot of stuff, though, is all that database overhead? Oof!
movq
www.uninformativ.de
View Thread
@prologic (Haha, my client has stored ~6100 twts at the moment and they use up 25 MB on disk – but only because each twt is an individual file, so the file system allocates 4k, even if it’s just 90 bytes. The real total size is 2.4 MB … πŸ™„)~
movq
www.uninformativ.de
View Thread
@prologic (Haha, my client has stored ~6100 twts at the moment and they use up 25 MB on disk – but only because each twt is an individual file, so the file system allocates 4k, even if it’s just 90 bytes. The real total size is 2.4 MB … πŸ™„)~
vain@www.uninformativ.de
www.uninformativ.de
View Thread
@prologic (Haha, my client has stored ~6100 twts at the moment and they use up 25 MB on disk – but only because each twt is an individual file, so the file system allocates 4k, even if it’s just 90 bytes. The real total size is 2.4 MB … πŸ™„)~
movq
www.uninformativ.de
View Thread
@prologic (Haha, my client has stored ~6100 twts at the moment and they use up 25 MB on disk – but only because each twt is an individual file, so the file system allocates 4k, even if it’s just 90 bytes. The real total size is 2.4 MB … πŸ™„)~
prologic@twtxt.net
twtxt.net
View Thread
@vain No, this is not database overhead at all 🀣 This is _quite literally_ ~300k Twts archived over a period of months on this pod (_I consume a lot of news via feeds.twtxt.net :P_)~
prologic
twtxt.net
View Thread
@vain No, this is not database overhead at all 🀣 This is _quite literally_ ~300k Twts archived over a period of months on this pod (_I consume a lot of news via feeds.twtxt.net :P_)~
prologic
twtxt.net
View Thread
@vain No, this is not database overhead at all 🀣 This is _quite literally_ ~300k Twts archived over a period of months on this pod (_I consume a lot of news via feeds.twtxt.net :P_)~
prologic@twtxt.net
twtxt.net
View Thread
There is.a _little_ overhead though as the archived twts are stored as .json blobs.
prologic
twtxt.net
View Thread
There is.a _little_ overhead though as the archived twts are stored as .json blobs.
prologic
twtxt.net
View Thread
There is.a _little_ overhead though as the archived twts are stored as .json blobs.