# I am the Watcher. I am your guide through this vast new twtiverse.
# 
# Usage:
#     https://watcher.sour.is/api/plain/users              View list of users and latest twt date.
#     https://watcher.sour.is/api/plain/twt                View all twts.
#     https://watcher.sour.is/api/plain/mentions?uri=:uri  View all mentions for uri.
#     https://watcher.sour.is/api/plain/conv/:hash         View all twts for a conversation subject.
# 
# Options:
#     uri     Filter to show a specific users twts.
#     offset  Start index for quey.
#     limit   Count of items to return (going back in time).
# 
# twt range = 1 196279
# self = https://watcher.sour.is?offset=170725
# next = https://watcher.sour.is?offset=170825
# prev = https://watcher.sour.is?offset=170625
@movq I'm glad you like it. A mention (@<movq https://www.uninformativ.de/twtxt.txt> ) is also long, but we live with it anyway. In a way a replyto: is just a mention of a twt instead of a feed/person. Maybe we chould even model the syntax for replies on mentions: (#<2024-09-17T08:39:18Z https://www.eksempel.dk/twtxt.txt>) ?!
@movq I'm glad you like it. A mention (@<movq https://www.uninformativ.de/twtxt.txt> ) is also long, but we live with it anyway. In a way a replyto: is just a mention of a twt instead of a feed/person. Maybe we chould even model the syntax for replies on mentions: (#<2024-09-17T08:39:18Z https://www.eksempel.dk/twtxt.txt>) ?!
@lyse and @movq and possibly @aelaraji and even @cuaxolotl -- I'm very curious to understand and hear thoughts, pros and cons or other feelings about introducing the notation of a feed's identify using cryptography? If we were to keep things simple, and use what's commonly available, for example SSH ED25519 keys? using the ssh-keygen -Y sign or ssh-keygen -Y verify tools already available? Maybe in combination with @xuu 's idea of generating a random unique ID for your feed, say # id = and signing it with your ED25519 key? 🔑
@lyse and @movq and possibly @aelaraji and even @cuaxolotl -- I'm very curious to understand and hear thoughts, pros and cons or other feelings about introducing the notation of a feed's identify using cryptography? If we were to keep things simple, and use what's commonly available, for example SSH ED25519 keys? using the ssh-keygen -Y sign or ssh-keygen -Y verify tools already available? Maybe in combination with @xuu 's idea of generating a random unique ID for your feed, say # id = and signing it with your ED25519 key? 🔑
@falsifian You mean the idea of being able to inline # url = changes in your feed?

> Whatever gets used, it would be nice to be able to rotate identities. I like @lyse’s idea for that.
@falsifian You mean the idea of being able to inline # url = changes in your feed?

> Whatever gets used, it would be nice to be able to rotate identities. I like @lyse’s idea for that.
[47°09′25″S, 126°43′13″W] Raw reading: 0x66E928F1, offset +/-5
[47°09′11″S, 126°43′57″W] Reading: 0.63000 PPM
@mckinley Yes, changing domains is be a problem if you tie your identity to an https url. But I also worry about being stuck with a key I can't rotate. Whatever gets used, it would be nice to be able to rotate identities. I like @lyse's idea for that.
This scheme also only support threading off a specific Twt of someone's feed. What if you're not replying to anyone in particular?
This scheme also only support threading off a specific Twt of someone's feed. What if you're not replying to anyone in particular?
Noting that this scheme cannot support disjoint threads that should be merged together once either party discovers each other 😅
Noting that this scheme cannot support disjoint threads that should be merged together once either party discovers each other 😅
@movq It does now as of several weeks ago or so 👌
@movq It does now as of several weeks ago or so 👌
@lyse Yea I think your idea of inclining url changes in the feed works perfectly as long as folks remember to do so I guess? 🤔
@lyse Yea I think your idea of inclining url changes in the feed works perfectly as long as folks remember to do so I guess? 🤔
@bender Seems to have used the hash correctly here 🤣
@bender Seems to have used the hash correctly here 🤣
Holy shot! what an old thread 🤣
Holy shot! what an old thread 🤣
@cuaxolotl Gitea is plenty fast for me 👌 Even with a SQLite database 🤣
@cuaxolotl Gitea is plenty fast for me 👌 Even with a SQLite database 🤣
I dunno whether any of this is actually true 🤷‍♂️ The LLM 🤖could have made (hallucinated) this shit up 🤣
I dunno whether any of this is actually true 🤷‍♂️ The LLM 🤖could have made (hallucinated) this shit up 🤣
WiscKey's approach to handling key-value pairs in SSDs offers several advantages:

1. It minimizes I/O amplification by separating keys and values, allowing for more efficient storage and retrieval.
2. The design leverages the SSD's performance characteristics, utilizing sequential writes and parallel random reads to enhance throughput and reduce latency.
3. WiscKey maintains excellent insert and lookup performance while improving SSD lifespan by reducing the number of erase cycles required.
WiscKey's approach to handling key-value pairs in SSDs offers several advantages:

1. It minimizes I/O amplification by separating keys and values, allowing for more efficient storage and retrieval.
2. The design leverages the SSD's performance characteristics, utilizing sequential writes and parallel random reads to enhance throughput and reduce latency.
3. WiscKey maintains excellent insert and lookup performance while improving SSD lifespan by reducing the number of erase cycles required.
WiscKey minimizes CPU usage compared to LevelDB by eliminating the log file, which reduces the CPU cost associated with encoding and writing key-value pairs. While LevelDB has higher CPU usage due to its single writer protocol and background compaction using one thread, WiscKey's architecture allows for garbage collection to be decoupled and performed later, minimizing its impact on foreground performance. Consequently, WiscKey generally exhibits lower CPU usage during workloads, except when utilizing multiple background threads for prefetching.
WiscKey minimizes CPU usage compared to LevelDB by eliminating the log file, which reduces the CPU cost associated with encoding and writing key-value pairs. While LevelDB has higher CPU usage due to its single writer protocol and background compaction using one thread, WiscKey's architecture allows for garbage collection to be decoupled and performed later, minimizing its impact on foreground performance. Consequently, WiscKey generally exhibits lower CPU usage during workloads, except when utilizing multiple background threads for prefetching.
The four critical ideas in the design of WiscKey are:

1. Separation of keys from values, with only keys stored in the LSM-tree and values in a separate log file.
2. Utilization of the parallel random-read characteristic of SSD devices to handle unsorted values during range queries.
3. Implementation of unique crash-consistency techniques to manage the value log efficiently.
4. Optimization of performance by removing the LSM-tree log without sacrificing consistency, reducing system-call overhead from small writes.
The four critical ideas in the design of WiscKey are:

1. Separation of keys from values, with only keys stored in the LSM-tree and values in a separate log file.
2. Utilization of the parallel random-read characteristic of SSD devices to handle unsorted values during range queries.
3. Implementation of unique crash-consistency techniques to manage the value log efficiently.
4. Optimization of performance by removing the LSM-tree log without sacrificing consistency, reducing system-call overhead from small writes.
# Summary of WiscKey: Separating Keys from Values in SSD-Conscious Storage

- Authors: Lanyue Lu, Thanumalayan Sankaranarayana Pillai, Andrea C. Arpaci-Dusseau, Remzi H. Arpaci-Dusseau
- Conference: 14th USENIX Conference on File and Storage Technologies (FAST '16)
- Key Concept: WiscKey is a key-value store that separates keys from values to minimize I/O amplification, optimizing performance for SSDs.
- Performance: WiscKey outperforms LevelDB and RocksDB in various workloads, achieving up to 111× faster loading and improved random lookup speeds.
- Design Goals: Focus on low write/read amplification, SSD optimization, and support for modern features like range queries.
# Summary of WiscKey: Separating Keys from Values in SSD-Conscious Storage

- Authors: Lanyue Lu, Thanumalayan Sankaranarayana Pillai, Andrea C. Arpaci-Dusseau, Remzi H. Arpaci-Dusseau
- Conference: 14th USENIX Conference on File and Storage Technologies (FAST '16)
- Key Concept: WiscKey is a key-value store that separates keys from values to minimize I/O amplification, optimizing performance for SSDs.
- Performance: WiscKey outperforms LevelDB and RocksDB in various workloads, achieving up to 111× faster loading and improved random lookup speeds.
- Design Goals: Focus on low write/read amplification, SSD optimization, and support for modern features like range queries.
🧮 USERS:1 FEEDS:2 TWTS:1095 ARCHIVED:78843 CACHE:2434 FOLLOWERS:17 FOLLOWING:14
i have been using grapheneos for a long time (maybe 7 years now?) and i would fully reccomend it to anyone who is OK with buying a new pixel device to run it on. the install instructions are really easy to follow and can be executed on any device that supports WebUSB https://grapheneos.org/install/web
afaik nobody has done this, but i really need some numbers that can indicate the relative performance of various git servers (cgit, gitea, gitlab) on comparable hardware. cgit claims to be hyperfast, but what does that mean in practice?
the first version of the minibase flake cdn is online as of today. more details on how this is populated and funded will be coming along soon. currently minibase includes a very small selection of packages, but you can install them using the experimental flakes feature today! https://src.cyb.red/
@movq the thing is, the twtxt is in Maildir. When I reply to it, it doesn’t use the existing hash.
so this path *has been* trod and its bad lmao https://web.archive.org/web/20230926192451/https://gemini.spam.works/~emery/sigil-report.gmi
so it looks like genode has taken some inspiration from nix.. that's a rabbit-hole for another time https://genode.org/documentation/developer-resources/package_management
@aelaraji 🥳
@aelaraji 🥳
@aelaraji 🥳
@aelaraji 🥳
tw.txt is fun because its a hackers distributed system. simple, with lots of sharp edges to keep things interesting. i hope we don't go chasing mass-appeal because it'll make writing my own crappy tw.txt services un-fun and pointless since i can barely keep up with the current somewhat glacial pace of development.
it would be interesting to build a new nixpkgs based on a totally different base system. like, genode (sel4+) that can take the interesting security properties of nix and apply them to a non-posix secure-by-construction type of OS
@bender Ah, haha, --fetch-context doesn’t go back into archived feeds … 🤦
@bender Ah, haha, --fetch-context doesn’t go back into archived feeds … 🤦
@bender Ah, haha, --fetch-context doesn’t go back into archived feeds … 🤦
@bender Ah, haha, --fetch-context doesn’t go back into archived feeds … 🤦
[47°09′36″S, 126°43′20″W] Transponder still failing -- switching to analog communication
Thank you @movq Things are working again!! 🙏
Thank you @movq Things are working again!! 🙏
Thank you @movq Things are working again!! 🙏
Trying to fetch the original (highlighting yours) with jenny renders this:

@movq the puzzle thickens:

@prologic

@quark This is what I get. 🤔
@prologic

@quark This is what I get. 🤔
@prologic

@quark This is what I get. 🤔
@prologic

@quark This is what I get. 🤔
@lyse Argh. Yeah. Well. 🤦
@lyse Argh. Yeah. Well. 🤦
@lyse Argh. Yeah. Well. 🤦
@lyse Argh. Yeah. Well. 🤦
@movq I'd guess the same goes for all twtxt.social feeds... I can't see bender's archived twts either, didn't check for the others.
@movq I'd guess the same goes for all twtxt.social feeds... I can't see bender's archived twts either, didn't check for the others.
@movq I'd guess the same goes for all twtxt.social feeds... I can't see bender's archived twts either, didn't check for the others.
Found it: https://github.com/buckket/twtxt/issues/157
@prologic Yeah, but I reckon we can kill both birds with one stone. If we change it to support edits, it should be fairly easy to also tweak it to support feed URL changes. Like outlined in my first reply: https://twtxt.net/twt/n4omfvq The URL part sounds way easier to me. :-)
This is how my original message shows up on jenny:


From: quark <quark>
Subject: (#o) @prologic this was your first twtxt. Cool! :-P
Date: Mon, 16 Sep 2024 12:42:27 -0400
Message-Id: <k7imvia@twtxt>
X-twtxt-feed-url: https://ferengi.one/twtxt.txt

(#o) @<prologic https://twtxt.net/user/prologic/twtxt.txt> this was your first twtxt. Cool! :-P
This is how my original message shows up on jenny:


From: quark <quark>
Subject: (#o) @prologic this was your first twtxt. Cool! :-P
Date: Mon, 16 Sep 2024 12:42:27 -0400
Message-Id: <k7imvia@twtxt>
X-twtxt-feed-url: https://ferengi.one/twtxt.txt

(#o) @<prologic https://twtxt.net/user/prologic/twtxt.txt> this was your first twtxt. Cool! :-P
@sorenpeter There was or maybe still is a competing proposal for multiline twts that combines all twts with the same timestamp to one logical multiline twt. Not sure what happened to that, if it is used in the wild and whether anyone "here" follows a feed with that convention. "Our" solution for multiline twts is to use U+2028 Unicode LINE SEPARATOR as a newline: https://dev.twtxt.net/doc/multilineextension.html.
Hmm... I replied to this message:


From: prologic <prologic>
Subject: Hello World! 😊
Date: Sat, 18 Jul 2020 08:39:52 -0400
Message-Id: <o6dsrga>
X-twtxt-feed-url: https://twtxt.net/user/prologic/twtxt.txt

Hello World! 😊


And see how the hash shows... Is it because that hash isn't longer used?
Hmm... I replied to this message:


From: prologic <prologic>
Subject: Hello World! 😊
Date: Sat, 18 Jul 2020 08:39:52 -0400
Message-Id: <o6dsrga>
X-twtxt-feed-url: https://twtxt.net/user/prologic/twtxt.txt

Hello World! 😊


And see how the hash shows... Is it because that hash isn't longer used?
@prologic this was your first twtxt. Cool! :-P
@prologic this was your first twtxt. Cool! :-P
The bug in jenny that @aelaraji found:

Jenny has to look for the metadata fields, it must find the # prev = ... line. To do so, I naively wrote something along these lines:


for line in content.splitlines():
    if line.startswith('# prev = '):
        ...


Problem is, we use \u2028 a lot in twtxt feeds and Python interprets those as line separators as well. That’s not what we want here. Jenny must only split at a \n.

Now @prologic had a quote/copy of some of his metadata fields in a twt. Like so:


# prev = foo bar


Perfectly legitimate, but now jenny found the # prev = *twice* (once in the actual header, once in a twt), didn’t know what to do, and thus did not fetch the archived feeds. 🤦

Should be fixed in this commit: https://www.uninformativ.de/git/jenny/commit/6e8ce5afdabd5eac22eae4275407b3bd2a167daf.html
The bug in jenny that @aelaraji found:

Jenny has to look for the metadata fields, it must find the # prev = ... line. To do so, I naively wrote something along these lines:


for line in content.splitlines():
    if line.startswith('# prev = '):
        ...


Problem is, we use \u2028 a lot in twtxt feeds and Python interprets those as line separators as well. That’s not what we want here. Jenny must only split at a \n.

Now @prologic had a quote/copy of some of his metadata fields in a twt. Like so:


# prev = foo bar


Perfectly legitimate, but now jenny found the # prev = *twice* (once in the actual header, once in a twt), didn’t know what to do, and thus did not fetch the archived feeds. 🤦

Should be fixed in this commit: https://www.uninformativ.de/git/jenny/commit/6e8ce5afdabd5eac22eae4275407b3bd2a167daf.html
The bug in jenny that @aelaraji found:

Jenny has to look for the metadata fields, it must find the # prev = ... line. To do so, I naively wrote something along these lines:


for line in content.splitlines():
    if line.startswith('# prev = '):
        ...


Problem is, we use \u2028 a lot in twtxt feeds and Python interprets those as line separators as well. That’s not what we want here. Jenny must only split at a \n.

Now @prologic had a quote/copy of some of his metadata fields in a twt. Like so:


# prev = foo bar


Perfectly legitimate, but now jenny found the # prev = *twice* (once in the actual header, once in a twt), didn’t know what to do, and thus did not fetch the archived feeds. 🤦

Should be fixed in this commit: https://www.uninformativ.de/git/jenny/commit/6e8ce5afdabd5eac22eae4275407b3bd2a167daf.html
The bug in jenny that @aelaraji found:

Jenny has to look for the metadata fields, it must find the # prev = ... line. To do so, I naively wrote something along these lines:


for line in content.splitlines():
    if line.startswith('# prev = '):
        ...


Problem is, we use \\u2028 a lot in twtxt feeds and Python interprets those as line separators as well. That’s not what we want here. Jenny must only split at a \\n.

Now @prologic had a quote/copy of some of his metadata fields in a twt. Like so:


# prev = foo bar


Perfectly legitimate, but now jenny found the # prev = *twice* (once in the actual header, once in a twt), didn’t know what to do, and thus did not fetch the archived feeds. 🤦

Should be fixed in this commit: https://www.uninformativ.de/git/jenny/commit/6e8ce5afdabd5eac22eae4275407b3bd2a167daf.html
@movq What's you definition of "complete thread"? ;-) There might be feeds participating in the conversation that you have no idea of.

But yes, this has a nice discoverability bonus. And even simpler than a hash, that's right.
@movq use @xuu pod as default instead, as he keeps the cache as long as I used to keep mine when I ran Yarn. @prologic's pod expires then way too soon.
@movq Yeah, I think so.
This is a bug in jenny. 🤦
This is a bug in jenny. 🤦
This is a bug in jenny. 🤦
This is a bug in jenny. 🤦
@prologic Oh so *that’s* how it works? The front page only shows the latest twt of each feed? 🤔*
@prologic Oh so *that’s* how it works? The front page only shows the latest twt of each feed? 🤔*
@prologic Oh so *that’s* how it works? The front page only shows the latest twt of each feed? 🤔*
@prologic Oh so *that’s* how it works? The front page only shows the latest twt of each feed? 🤔*
[47°09′41″S, 126°43′52″W] Transponder malfunction
No, something is fishy. It didn’t fetch @prologic’s archived feeds and now only 969 of his twts are in my maildir. 🤔
No, something is fishy. It didn’t fetch @prologic’s archived feeds and now only 969 of his twts are in my maildir. 🤔
No, something is fishy. It didn’t fetch @prologic’s archived feeds and now only 969 of his twts are in my maildir. 🤔
No, something is fishy. It didn’t fetch @prologic’s archived feeds and now only 969 of his twts are in my maildir. 🤔
@aelaraji Yep, I just tried. It’s not that easy to verify, though. 😅 It *looks* fine to me. The number of twts in the maildir has gone down from 61759 to 34787 – but that’s probably because I unfollowed lots of (presumably dead) feeds in the last few weeks. 🥴
@aelaraji Yep, I just tried. It’s not that easy to verify, though. 😅 It *looks* fine to me. The number of twts in the maildir has gone down from 61759 to 34787 – but that’s probably because I unfollowed lots of (presumably dead) feeds in the last few weeks. 🥴
@aelaraji Yep, I just tried. It’s not that easy to verify, though. 😅 It *looks* fine to me. The number of twts in the maildir has gone down from 61759 to 34787 – but that’s probably because I unfollowed lots of (presumably dead) feeds in the last few weeks. 🥴
@aelaraji Yep, I just tried. It’s not that easy to verify, though. 😅 It *looks* fine to me. The number of twts in the maildir has gone down from 61759 to 34787 – but that’s probably because I unfollowed lots of (presumably dead) feeds in the last few weeks. 🥴