# I am the Watcher. I am your guide through this vast new twtiverse.
#
# Usage:
# https://watcher.sour.is/api/plain/users View list of users and latest twt date.
# https://watcher.sour.is/api/plain/twt View all twts.
# https://watcher.sour.is/api/plain/mentions?uri=:uri View all mentions for uri.
# https://watcher.sour.is/api/plain/conv/:hash View all twts for a conversation subject.
#
# Options:
# uri Filter to show a specific users twts.
# offset Start index for quey.
# limit Count of items to return (going back in time).
#
# twt range = 1 60515
# self = https://watcher.sour.is?uri=https://twtxt.net/user/prologic/twtxt.txt&offset=56791
# next = https://watcher.sour.is?uri=https://twtxt.net/user/prologic/twtxt.txt&offset=56891
# prev = https://watcher.sour.is?uri=https://twtxt.net/user/prologic/twtxt.txt&offset=56691
@movq No that's okay. I happen to agree with you really, I just wanted to get a bit of a vibe on using cryptography in general and the idea of signing feeds. It's not particularly about a problem being solved, just gauging your opinions/thoughts on this ๐
@aelaraji We just have to write better clients ๐คฃ I have figured out how to detect edits FWIW, but haven't gone from R&D phase to an actual design and implementation. But it is possible to detect an edit as well as a similarity score and the matching Twts.
@aelaraji We just have to write better clients ๐คฃ I have figured out how to detect edits FWIW, but haven't gone from R&D phase to an actual design and implementation. But it is possible to detect an edit as well as a similarity score and the matching Twts.
@quark I think I may have lost my original feed because it was written with the old software before Yarn social became a thing.
@quark I think I may have lost my original feed because it was written with the old software before Yarn social became a thing.
@sorenpeter not really you're really forming a cryptographic chain of twts, that are cryptographically provable by anyone, at least in one direction ). It's called content addressing. Your propose scheme while simple doesn't do this.
@sorenpeter not really you're really forming a cryptographic chain of twts, that are cryptographically provable by anyone, at least in one direction ). It's called content addressing. Your propose scheme while simple doesn't do this.
@lyse and @movq and possibly @aelaraji and even @cuaxolotl -- I'm very curious to understand and hear thoughts, pros and cons or other feelings about introducing the notation of a feed's identify using cryptography? If we were to keep things simple, and use what's commonly available, for example SSH ED25519 keys? using the ssh-keygen -Y sign or ssh-keygen -Y verify tools already available? Maybe in combination with @xuu 's idea of generating a random unique ID for your feed, say # id = and signing it with your ED25519 key? ๐
@lyse and @movq and possibly @aelaraji and even @cuaxolotl -- I'm very curious to understand and hear thoughts, pros and cons or other feelings about introducing the notation of a feed's identify using cryptography? If we were to keep things simple, and use what's commonly available, for example SSH ED25519 keys? using the ssh-keygen -Y sign or ssh-keygen -Y verify tools already available? Maybe in combination with @xuu 's idea of generating a random unique ID for your feed, say # id = and signing it with your ED25519 key? ๐
@falsifian You mean the idea of being able to inline # url = changes in your feed?
> Whatever gets used, it would be nice to be able to rotate identities. I like @lyseโs idea for that.
@falsifian You mean the idea of being able to inline # url = changes in your feed?
> Whatever gets used, it would be nice to be able to rotate identities. I like @lyseโs idea for that.
This scheme also only support threading off a specific Twt of someone's feed. What if you're not replying to anyone in particular?
This scheme also only support threading off a specific Twt of someone's feed. What if you're not replying to anyone in particular?
Noting that this scheme cannot support disjoint threads that should be merged together once either party discovers each other ๐
Noting that this scheme cannot support disjoint threads that should be merged together once either party discovers each other ๐
@movq It does now as of several weeks ago or so ๐
@movq It does now as of several weeks ago or so ๐
@lyse Yea I think your idea of inclining url changes in the feed works perfectly as long as folks remember to do so I guess? ๐ค
@lyse Yea I think your idea of inclining url changes in the feed works perfectly as long as folks remember to do so I guess? ๐ค
@bender Seems to have used the hash correctly here ๐คฃ
@bender Seems to have used the hash correctly here ๐คฃ
Holy shot! what an old thread ๐คฃ
Holy shot! what an old thread ๐คฃ
@cuaxolotl Gitea is plenty fast for me ๐ Even with a SQLite database ๐คฃ
@cuaxolotl Gitea is plenty fast for me ๐ Even with a SQLite database ๐คฃ
I dunno whether any of this is actually true ๐คทโโ๏ธ The LLM ๐คcould have made (hallucinated) this shit up ๐คฃ
I dunno whether any of this is actually true ๐คทโโ๏ธ The LLM ๐คcould have made (hallucinated) this shit up ๐คฃ
WiscKey's approach to handling key-value pairs in SSDs offers several advantages:
1. It minimizes I/O amplification by separating keys and values, allowing for more efficient storage and retrieval.
2. The design leverages the SSD's performance characteristics, utilizing sequential writes and parallel random reads to enhance throughput and reduce latency.
3. WiscKey maintains excellent insert and lookup performance while improving SSD lifespan by reducing the number of erase cycles required.
WiscKey's approach to handling key-value pairs in SSDs offers several advantages:
1. It minimizes I/O amplification by separating keys and values, allowing for more efficient storage and retrieval.
2. The design leverages the SSD's performance characteristics, utilizing sequential writes and parallel random reads to enhance throughput and reduce latency.
3. WiscKey maintains excellent insert and lookup performance while improving SSD lifespan by reducing the number of erase cycles required.
WiscKey minimizes CPU usage compared to LevelDB by eliminating the log file, which reduces the CPU cost associated with encoding and writing key-value pairs. While LevelDB has higher CPU usage due to its single writer protocol and background compaction using one thread, WiscKey's architecture allows for garbage collection to be decoupled and performed later, minimizing its impact on foreground performance. Consequently, WiscKey generally exhibits lower CPU usage during workloads, except when utilizing multiple background threads for prefetching.
WiscKey minimizes CPU usage compared to LevelDB by eliminating the log file, which reduces the CPU cost associated with encoding and writing key-value pairs. While LevelDB has higher CPU usage due to its single writer protocol and background compaction using one thread, WiscKey's architecture allows for garbage collection to be decoupled and performed later, minimizing its impact on foreground performance. Consequently, WiscKey generally exhibits lower CPU usage during workloads, except when utilizing multiple background threads for prefetching.
The four critical ideas in the design of WiscKey are:
1. Separation of keys from values, with only keys stored in the LSM-tree and values in a separate log file.
2. Utilization of the parallel random-read characteristic of SSD devices to handle unsorted values during range queries.
3. Implementation of unique crash-consistency techniques to manage the value log efficiently.
4. Optimization of performance by removing the LSM-tree log without sacrificing consistency, reducing system-call overhead from small writes.
The four critical ideas in the design of WiscKey are:
1. Separation of keys from values, with only keys stored in the LSM-tree and values in a separate log file.
2. Utilization of the parallel random-read characteristic of SSD devices to handle unsorted values during range queries.
3. Implementation of unique crash-consistency techniques to manage the value log efficiently.
4. Optimization of performance by removing the LSM-tree log without sacrificing consistency, reducing system-call overhead from small writes.
# Summary of WiscKey: Separating Keys from Values in SSD-Conscious Storage
- Authors: Lanyue Lu, Thanumalayan Sankaranarayana Pillai, Andrea C. Arpaci-Dusseau, Remzi H. Arpaci-Dusseau
- Conference: 14th USENIX Conference on File and Storage Technologies (FAST '16)
- Key Concept: WiscKey is a key-value store that separates keys from values to minimize I/O amplification, optimizing performance for SSDs.
- Performance: WiscKey outperforms LevelDB and RocksDB in various workloads, achieving up to 111ร faster loading and improved random lookup speeds.
- Design Goals: Focus on low write/read amplification, SSD optimization, and support for modern features like range queries.
# Summary of WiscKey: Separating Keys from Values in SSD-Conscious Storage
- Authors: Lanyue Lu, Thanumalayan Sankaranarayana Pillai, Andrea C. Arpaci-Dusseau, Remzi H. Arpaci-Dusseau
- Conference: 14th USENIX Conference on File and Storage Technologies (FAST '16)
- Key Concept: WiscKey is a key-value store that separates keys from values to minimize I/O amplification, optimizing performance for SSDs.
- Performance: WiscKey outperforms LevelDB and RocksDB in various workloads, achieving up to 111ร faster loading and improved random lookup speeds.
- Design Goals: Focus on low write/read amplification, SSD optimization, and support for modern features like range queries.
@quark Meh I lost the plot ages ago ๐คฃ
@quark Meh I lost the plot ages ago ๐คฃ
@movq Bah you're right, that's a mistake and easily fixed ๐
@movq Bah you're right, that's a mistake and easily fixed ๐
@movq I tend to agree too, I think the focus should be on fixing and supporting Edits first ๐
@movq I tend to agree too, I think the focus should be on fixing and supporting Edits first ๐
@quark We will fix this soonโข ๐
@quark We will fix this soonโข ๐
@aelaraji So what is it about @sorenpeter's feed that's screwed with your client? (Jenny?) ๐ค Kind of curious now ๐คฃ
@aelaraji So what is it about @sorenpeter's feed that's screwed with your client? (Jenny?) ๐ค Kind of curious now ๐คฃ
@quark It _would_ also be possible to use the search engine here too I _think_ ๐ค i.e: https://search.twtxt.net
@quark It _would_ also be possible to use the search engine here too I _think_ ๐ค i.e: https://search.twtxt.net
@quark Looks like that would work according to the patch I just read ๐
@quark Looks like that would work according to the patch I just read ๐
These then become useful in filters like what you see here: 
These then become useful in filters like what you see here:
@aelaraji Ahh that's interesting! ๐ง One of my original goals when I started out building Yarn.social was to also be a source of news, blogs, and whatever else that could be roughly/easily translated into a Twtxt feed. I'm not sure if others do something similar, but that's why I built feeds.twtxt.net, which consumes RSS/Atom and produces Twtxt feeds.
My only desire one day is to build a "Feed Builder" of sorts that allows one to say, for example, construct a Slashdot feed but without AI hype, or as another example, a BBC/ABC feed that's a digest once or twice per day.
@aelaraji Ahh that's interesting! ๐ง One of my original goals when I started out building Yarn.social was to also be a source of news, blogs, and whatever else that could be roughly/easily translated into a Twtxt feed. I'm not sure if others do something similar, but that's why I built feeds.twtxt.net, which consumes RSS/Atom and produces Twtxt feeds.
My only desire one day is to build a "Feed Builder" of sorts that allows one to say, for example, construct a Slashdot feed but without AI hype, or as another example, a BBC/ABC feed that's a digest once or twice per day.
@aelaraji Good man ๐คฃ I keep getting this bloody AI hype from various news feeds I subscribe to via Twtxt like Slashdot cough ๐คฆโโ๏ธ
@aelaraji Good man ๐คฃ I keep getting this bloody AI hype from various news feeds I subscribe to via Twtxt like Slashdot cough ๐คฆโโ๏ธ
@bender Hahahahaha ๐คฃ Me neither ๐คฎ
@bender Hahahahaha ๐คฃ Me neither ๐คฎ
@bender you're probably right, but by that argument, cryptocurrency should be innate too right? as we no longer hear about that gold awful hype, right? ๐ค
@bender you're probably right, but by that argument, cryptocurrency should be innate too right? as we no longer hear about that gold awful hype, right? ๐ค
@movq Haha same and I'm sick of it! It's ruining innovation in anything else ๐คฆโโ๏ธ
@movq Haha same and I'm sick of it! It's ruining innovation in anything else ๐คฆโโ๏ธ
@aelaraji it's definitely your social bubble. ๐คฃ you need to use twtxt more as your daily driver ๐คฃ
@aelaraji it's definitely your social bubble. ๐คฃ you need to use twtxt more as your daily driver ๐คฃ
That's an interesting side effect to the new Discover feature that I added sometime ago that only displays one post per feed. That is when you're not logged in and viewing my pod's front page. You can pretty easily and roughly see what the monthly active view account is just by looking at the pager size. ๐ค
That's an interesting side effect to the new Discover feature that I added sometime ago that only displays one post per feed. That is when you're not logged in and viewing my pod's front page. You can pretty easily and roughly see what the monthly active view account is just by looking at the pager size. ๐ค
Amazingly though it seems to be slightly better to VPN in. ๐ค 
Amazingly though it seems to be slightly better to VPN in. ๐ค
But you know speedtest.net I believe is a bit of a liar and I'm quite sure they do something to make sure the speed test come up good even remote areas the real speed test my actual surfer infrastructure is quite piss poor ๐คฃ 
But you know speedtest.net I believe is a bit of a liar and I'm quite sure they do something to make sure the speed test come up good even remote areas the real speed test my actual surfer infrastructure is quite piss poor ๐คฃ
Even though we're quite a ways from any suburban areas, even with the Internet access via cell towers this poor, using my pod is still very snappy. ๐ 
Even though we're quite a ways from any suburban areas, even with the Internet access via cell towers this poor, using my pod is still very snappy. ๐
When will the AI hype die down?
When will the AI hype die down?
@falsifian One of the nice things I think is that you can almost assuredly trust that the hash is a correct representation of the thread because it was computed via our content, addressing in the first place, so all you need to do yes copy it ๐
@falsifian One of the nice things I think is that you can almost assuredly trust that the hash is a correct representation of the thread because it was computed via our content, addressing in the first place, so all you need to do yes copy it ๐
@falsifian I think I wrote a very similar program and go myself actually and you're right we do have to change the way we encode hashes.
@falsifian I think I wrote a very similar program and go myself actually and you're right we do have to change the way we encode hashes.
@falsifian All very good points ๐ by the way, how did you find two pieces of content that hash the same when taking the last N characters of the base32 and coded hash?
@falsifian All very good points ๐ by the way, how did you find two pieces of content that hash the same when taking the last N characters of the base32 and coded hash?