# I am the Watcher. I am your guide through this vast new twtiverse.
# 
# Usage:
#     https://watcher.sour.is/api/plain/users              View list of users and latest twt date.
#     https://watcher.sour.is/api/plain/twt                View all twts.
#     https://watcher.sour.is/api/plain/mentions?uri=:uri  View all mentions for uri.
#     https://watcher.sour.is/api/plain/conv/:hash         View all twts for a conversation subject.
# 
# Options:
#     uri     Filter to show a specific users twts.
#     offset  Start index for quey.
#     limit   Count of items to return (going back in time).
# 
# twt range = 1 54
# self = https://watcher.sour.is?uri=https://twtxt.alessandrocutolo.it/twtxt.txt&offset=54
@zvava @bender At first I added it without thinking when planning the possible fields based on other UIs I was researching.

I was about to discard it but after thinking about it a bit I noticed that the services allowing to have a separated nick and display_name could unlock some good uses.

For example some added context or at-a-glance information like pronouns or statuses (like Artist [Accepting commissions] or App Name (v2.5)) while other used a more readable version of the nick (blog.domain.com became Person Name's Blog).

Of course it is absolutely optional and it can be safely ignored, but with my vision of being able to build more that a pure twtxt clients, giving it a first-class support just like the other known fields felt right to me.
@lyse Great job!

I suggested it because I did it in the past, but never used it on bigger works.

In my case I did it exclusively on really small projects and used a thin rubber head hammer to prevent deforming the metal.
@movq I can confirm.

An intern practicing with turtle had an error when launching it the first time because it was missing tkinter which it use internally.
@lyse I think you should be able to find some even in general stores in the hardware section.
@movq So damn true.

I have a friend that might lock himself out of his home if there's a power outage while I keep removing apps and devices from my daily lives instead.

I recently switched from all the todo apps I used to sticky notes on my monitors and a pocket notebook for sketching and quick notes.
@thecanine So cool!

It reminds me of the monsters in Heart of Darkness on PSX (just replayed the other day).

https://www.youtube.com/shorts/ZnGOpT5NKxc
Hello again everyone! A little update on my twtxt client.

I think it's finally shaping a bit better now, but... ☝️

As I'm trying to put all the parts together, I decided to build multiple parallel UIs, to ensure I don't accidentally create a structure that is more rigid than planned.

I already decided on a UI that I would want to use for myself, it would be inspired by moshidon, misskey and some other "social feeds" mock-ups I found on dribbble.

I also plan on building a raw HTML version (for anyone wanting to do a full DIY client).

I would love to get any suggestions of what you would like to see (and possibly use) as a client, by sharing a link, app/website name or even a sketch made by you on paper.

I think I'll pick a third and maybe a fourth design to build together with the two already mentioned.

For reference, the screens I think of providing are (some might be optional or conditionally/manually hidable):

- Global / personal timeline screen
- Profile screen (with timeline)
- Thread screen
- Notifications screen or popup (both valid)
- DM list & chat screens (still planning, might come later)
- Settings screen (it'll probably be a hard coded form, but better mention it)
- Publish / edit post screen or popup (still analysing some use cases, as some "engines" might not have direct publishing support)

I also plan on adding two optional metadata fields:

- display_name: To show a human readable alternative for a nick, it fallback to nick if not defined
- banner: Using the same format as avatar but the image expected is wider, inspired by other socials around

I also plan on supporting any metadata provided, including a dynamically parsable regex rule format for those extra fields, this should allow anyone to build new clients that don't limit themselves to just the social aspect of twtxt, hoping to see unique ways of using twtxt! 🀞
@bender The first format use the subject extension while the other is a new format that is inspired by mentions format, the first one should be compatible but I'm not sure, if it's used verbatim by the client it would work, but if we consider the new proposal for it to have an optional part it wont work on clients without changes.
@movq While using the a frament is pretty nice, I think we can have a twtxt only format if the formatting seems to be a problem.
@lyse I think will be bad if handled incorrectly.

The client must reference both properly or it would miss posts, including both this way is a bit pointless if you can't use the hash or url separately.

Being a highly likely a breaking change anyway I think @zvava proposal looks much better.
@zvava That was my greatest concern with how it is currently handled, I'm afraid to break threads even by fixing a typo.

Handling it via the pod might work but I think it's not the best approach, external feeds and clients don't usually use a pod api but their own implementation, so any workaround won't work there.

That's why my proposals addressed those issues:

- the idea of using a "key" instead of the url (with the url as a fallback), the key could even be a public key so it can be used verifieable in crypto functions
- using the timestamp to prevent content changes to break threads (plus being simpler to implement)
- using an explicit thread reference with an alternative subject format (like [#THREAD_ID] Hello world and replies with (#REPLY_ID) Ahoy) so the content can change without affecting the thread reference, and anyone can use their own schemes freely
@zvava I just have plastic plants at home too.
@zvava Don't say that.

The eye candy is always good to have.
@zvava I'm not sure, I could just set up a cors-anywhere via docker in a minute and it would work the same.

Still, I could write one with just a dozen lines of Go or Node.js, I might consider writing one after the client is working decently.
Amazing game, Love the Peer-to-peer with other players and be able to communicate with them.
So smart, would be great to sneak in as an easter egg in a app.
I always wanted to build a terrarium, but after cleaning my keyboard today, I think I already have one...
@lyse I can suggest you a trick to do a "cold" welding.

Using a copper wire or a similarly malleable material, pass it through a drilled hole, hammer it on one end until flat, then do the same on the other side.

It does the same job of a rivet but it's flatter and look nicer on both sides, it's of course weaker but still strong enough for small objects.

It's sometimes used to reduce risk of deformities due to heat in hand-crafted jewelry and to reduce costs of small tools.
I love it!

Giving the user multiple choices to do the same things is what is great about protocols in general.
@zvava CORS is our worst enemy. πŸ₯·

I too had the same issue being a browser-based request, so the only solution is using a proxy.

For testing (and real personal use) I rely on this one https://corsproxy.io/.

In my client, I first check if the source allows me to fetch it without issues first and fallback to prefixing with a proxy if it gives an error.

For security reasons the client don't give you a readable error for CORS, so you must use a catch-all for that, if it fails again with the proxy you can deal with any other errors it throws as you normally would (preferably outside of the fetch function).

After the fetching responded, I store the response.url value to fetch it again for updates without having to do extra calls (you can store it verbatim or as a flag to be able to change the proxy later).

Here an extract of my code:

t
export async function fetchWithProxy(url, proxy=null) {
    return await fetch(url).catch(err => {
        if (!proxy) throw err;
        return fetch(`${proxy}${encodeURIComponent(url)}`);
    });
}

// Using it with
const res = await fetchWithProxy('https://twtxt.net/user/zvava/twtxt.txt', 'https://corsproxy.io/?');

// Get the working url (direct or through proxy)
const fetchingURL = res.url;

// Get the twtxt feed content (or handle errors)
const text = await res.text();


I also plan to allow the user to define a custom proxy field, I like the solution used by Delta.chat in their android app, where you can define the URL format with a variable https://my-proxy?$TWTXT_URL since it allows you to define with more freedom any proxy without a prefix format.

If the idea of using a third-party proxy is not to the user liking they can use a self-hosted solution like cors-anywhere or build their own (with twtxt it should just be a GET).
@bender Yes and no.

To build a compliant PWA you need to provide a webmanifest json and a service worker.

Those requirements are not directly part of this project.

You can build the client as a standalone PWA or even as a widget inside an existing page.

The general steps are closer to how you would include a third-party library in an existing project, by importing it as a dependency and using it in your website.

I'm pretty sure most users would expect a PWA (me included) so I plan to provide a ready-made template ready to be deployed as is.
Hi everyone, here's a little introduction of my twtxt client (still WIP).

The client I'm developing is a single tenant project that runs entirely in the browser (it might use an optional backend).

It's entirely based on native web-components and vanilla JS, it is designed to act closer to a toolkit than a full-fledged client, allowing users to "DIY" their own interface with pure html or plain javascript functions.

Users can also build their own *engines* by including a global javascript object that implement the defined internal API (TBD).

I'm planning to build a system that is easy enough to build and use with any skill level, using only pure html (with a homebrew minimal template engine) or via plain JS (I'll be also providing some pre-made templates too).

Everything can be self-hosted on any static hosting provider, this allows to spread twtxt within communities like Neocities and similarly hosted websites (basically any Indieweb/Smallweb/Digital garden website and any of the common GitHub/Lab/Berg/lify Pages).

It will be probably named something like TxtCraft or craf.txt but I'm not really sure yet... πŸ€” (Maybe some suggestions could help)

I'm still in the experimental phase, so there's no decent source-code to share yet, but it will soon enough!
@lyse Beautiful handwork, how did you seal the corners? I don't see and hole or anything.

BTW, That Sheet Metal Dude is something else himself, skilled enough to teach others, can work properly with self-imposed contraints, care about safety and is humble enough to be wiling to learn from others, a true craftman worthy of respect.
@lyse Thanks!

The way I have it setup right now would take me more work to integrate with your script though.

I'm keeping things dumb on purpose, I just have to finish my client to have all the cool buttons that do it for me.
@prologic I too, self-host various services on a VPS (and considering buying a mini PC to keep at home instead).

I use most of it as a hosting platform for personal use only and as a remote development environment (I do share a couple of tools with a friend though).

But given the costant risks of DDoS, hacking, bots, etc. I keep any of my public facing resources purely static and on separate hosting providers (without lock-ins of course).

Lately, I began using homebrew PWAs with CouchDB as a sync database, this way I get a fantastic local-first experience and also have total control of my data, that also sync in a locally hosted backup instance in real-time.

Also, I was already aware of Salty.im, but what I'm thinking is a more feature complete solution that even my family can use quickly, Delta.chat with the new chatmail provider (self-hostable) might be the solution for my needs.

But I'm still thinking if it's worth the trouble. I might just drop everything and only use safe channels to speak with them (free 24/7 family tech-support is easy to manage πŸ˜†).

Also, I'll be waiting for the day you'll share with us your story, I'm pretty curious about it!
@prologic I think nobody will stop you if you replace the current hashing with SHA-256 if you call it *improvementβ„’* πŸ˜‰*
That's what I'm using right now, while my own client is still in the making.

A simple bash script to write a post in a mktemp file then clean it with regex.
I don't even bother to hash the replies, I just open https://twtxt.net and copy the hash by hand since I'm checking the new posts from there anyway (temporarily, as I might end up DoS-ing everyone's feed in my client right now).
@prologic Don't worry about it!

I also getting angry thinking how this Chat Control crap will escalate to.

I'm already thinking of countermeasures and self-hosted alternatives, while searching lists of affected apps and services to replace/drop in the worst scenario (and probably devices).
@zvava Amazing! I would love to see all the specs described this way. 🀩
@prologic Well, personally I would, as I already do for user feeds in my client.

That's why part of my proposal was to allow custom strings and be free from a specific format that need periodical upgrades, but it's not much of a problem in the end.

I'll adapt to what we can get out of this.
@prologic I think a counter in the client is not a good choice given the decentralized nature of twtxt, especially if someone use multiple cients together.

After thinking about it for a while I got to two solutions:

Proposal 1: Thread syntax (using subject)

Each post have an implicit and an *optional* explicit root reference:

- Implicit (no action needed, all data required are already there)
- URL + timestamp

- Explicit (subject required)
- Identity (client generated)
- External reference
- Random value

We then add include a "root" subject in each post for generating explicit theads:
1. [ROOT_ID] (REPLY_ID): simpler with no need of prefixes
2. (root:ROOT_ID) (reply:REPLY_ID): more complex but could allow expansions
- (rt:ROOT_ID) (re:REPLY_ID): same but with a compact version
- ($ROOT_ID) (>REPLY_ID): same but with a single characters

Each post can have both references, like the current hash approach the reference can be treated as a simple string and don't have a real meaning.

Using a custom reference this way allows a client to decide how to generate them:
- Identity: can be a content hash or signature or anything else, without enforcing how it is generated we can upgrade the algorithm/length freely
- External references: can be provided from another system (Eg. 7e073bd345, *yarnsocial/yarn* latest commit)
- Random value: like a UUID (Eg. 9a0c34ed-d11e-447e-9257-0a0f57ef6e07)

Proposal 2: Threaded mentions (featuring zvava)

Inspired by @zvava's solution it could be simplified into: #<nick url#timestamp> or #<url#timestamp>

It can be shown like a mentions or hidden like a subject.

If we're using thinking of using a counter in the client, I think there's no point in avoiding the timestamp anymore.*
@prologic While it might work if you want to keep both, I think the point was to be able to use one or the other, if we still have to generate the hash anyway it might be pointless to use this format.
@prologic I admit that I was a bit confused about the meaning of the message, at least I understood it was a "yes" from the last sentence. πŸ˜…
@bender Well, you guessed correctly! 😁

Would be nice to have a fixed fee for that, a car is a car anywhere in the world...
@prologic That is really great to hear!

If there are opposing opinions we either build a bridge or provide a new parallel road.

Also, I wouldn't call my opinion a "stance", I just wish for a better twtxt thanks to everyone's effort.

The last thing we need to do is decide a proper format for the location-based version.

My proposal is to keep the "Subject extension" unchanged and include the reference to the mention like this:


// Current hash format: starts with a '#'
(#hash) here's text
(#hash) @<nick url> here's text

// New location format: valid URL-like + '#' + TIMESTAMP (verbatim format of feed source)
(url#timestamp) here's text
(url#timestamp) @<nick url> here's text


I think the timestamp should be referenced verbatim to prevent broken references with multiple variations (especially with the many timezones out there) which would also make it even easier to implement for everyone.

I'm sure we can get @zvava, @lyse and everyone else to help on this one.

I personally think we should also consider allowing a generic format to build on custom references, this would allow for creating threads using any custom source (manual, computed or external generated), maybe using a new "Topic extension", here's some examples.


// New format for custom references: starts with a '!' maybe?
(!custom) here's text
(!custom) @<nick url> here's text

// A possible "Topic" parse as a thread root:
[!custom] start here
[custom] simpler format


This one is just an idea of mine, but I feel it can unleash new ways of using twtxt.
@itsericwoodward I used the dates as is for indexing them as string, the ISO format allows for free auto sorting.
@bender What?! In my country you have to pay 100€ every 10 years of which about 75% are just taxes...
@lyse @prologic Can't we find a middle ground and support both?

The thread is defined by two parts:

1. The hash
2. The subject

The client/pod generate the *hash* and index it in it's database/cache, then it simply query the subject of other posts to find the related posts, right?

In my own client current implementation (using hashes), the only calculation is in the hash generation, the rest is a verbatim copy of the subject (minus the # character), if this is the common implemented approach then adding the location based one is somewhat simple.

s
function setPostIndex(post) {
    // Current hash approach
    const hash = createHash(post.url, post.timestamp, post.content);

    // New location approach
    const location = post.url + '#' + post.timestamp;

    // Unchanged (probably)
    const subject = post.subject;

    // Index them all
    addToIndex(hash, post);
    addToIndex(location, post);
    addToIndex(subject, post);
}

// Both should work if the index contains both versions
getThreadBySubject('#abcdef') => [post1, post2, post3]; // Hash
getThreadBySubject('https://example.com#2025-01-01T12:00:00') => [post1, post2, post3]; // Location


As I said before, the mention is already location based @<example https://example.com/twtxt.txt>, so I think we should keep that in consideration.

Of course this will lead to a bit of fragmentation (without merging the two) but I think this can make everyone happy.

Otherwise, the only other solution I can think of is a different approach where the value doesn't matter, allowing to use anything as a reference (hash, location, git commit) for greater flexibility and freedom of implementation (this probably need the use of a fixed "header" for each post, but it can be seen as a separate extension).
@movq Happy equinox!

It looks amazing from the map, you probably can't tell even by looking from space.
@kat Oh! A new place to spam dad jokes. πŸ₯³
@prologic I can see the issues mentioned, but I think some can be fixed.

1. The current hash relies on a url field too, by specification, it will use the first # url = <URL> in the feed's metadata if present, that too can be different from the fetching source, if that field changes it would break the existing hashes too, a better solution would be to use a non-URL key like # feed_id = <UNIQUE_RANDOM_STRING> with the url as fallback.

2. We can prevent duplications if the reference uses that same url field too or the client "collapse" any reference of all the urls defined in the metadata.

3. I agree that hashing based on content is good, but we still use the URL as part of the hashing, which is just a field in the feed, easily replicable by a bot, also noting that edits can also break the hash, for this issue an alternative solution (E.g. a private key not included in the feed) should be considered.

4. For offline reading the source would be downloaded already, the fetching of non followed feeds would fill the gap in the same way mentions does, maybe I'm missing some context on this one.

5. To prevent collisions there was a discussion on extending the hash (forgot if that was already fixed or not), but without a fallback that would break existing clients too, we should think of a parallel format that maintains current implementations unchanged, we are already backward compatible with the original that don't use threads at all, a mention style format for that could be even more user-friendly for those clients.

We should also keep in mind that the current mention format is already location based (@<example https://example.com/twtxt.txt>) so I'm not that worried about threads working the same way.

Hope to see some other thought about this matter. πŸ€“
@thecanine Thanks!

Looking forward to it!✌️
This looks so huggable, it feels like a plushie! πŸ₯°

After seeing so many resolutions, I think you're only missing a *matrioska* version of it. πŸͺ†
@lyse Yeah, the format is just an idea of how it could work.

The order of SOURCE > POST does make more sense indeed.
@prologic thanks, I already follow @important_dev_news too.

BTW, the feed on https://feeds.twtxt.net/ seem down? It says it's in maintenance.
@prologic Me neither, if there's any important news others usually tell me anyway. 😌
While working on the Discoverability for my twtxt client (it runs client-side) I found out that Chrome doesn't allow to set a custom user agent. πŸ™ƒ

I thought it was a general thing for browsers, but it that was actually allowed in a newer specification, yet it's still not implemented in Chrome, it does work in Firefox though.
@zvava I kinda fixed the issue by not stripping the timestamp at all.

Seems that more feeds work correctly this way. πŸ€”
@thecanine With a progressive web app (PWA) you can have a native like experience without having to trouble yourself with building a second project that act as a client.

You can even "wrap" it into a packaged installation and publish it on stores, theres even projects to streamline it https://www.pwabuilder.com/.
My proposal is taken from https://texudus.readthedocs.io/en/latest/ BTW.
@zvava @lyse I also think a location based reference might be better.

A thread is a single post of a single feed as a root, but the hash has the drawback of not referencing the source, in a distributed network like twtxt it might leave some people out of the whole conversation.

I suggest a simpler format, something like: (#<TIMESTAMP URL>)

This solves three issues:

- Easier referencing: no need to generate a hash, just copy the timestamp and url, it's also simpler to implement in a client without the rish of collisions when putting things together
- Fetchable source: you can find the source within the reference and construct the thread from there
- Allow editing: If a post is modified the hash becomes invalid since it depends on [ timestamp, url, content ]
@zvava @lyse @movq I also was wondering how to handle this.

Currently my regex is like this: /@<((?<nick>[^\s]+)\s)?(?<url>\w+:\/\/[^>]+)>/g

It takes everything until the space and the nick is optional.
Hello everyone! πŸ‘‹

After a long while away, I'm back on twtxt with this new feed.

Some of you might remember me as justamoment@twtxt.net, that was a test account I made for trying things out, but I ended up keeping it more than planned.

I also tried other social platforms in search of a place that felt right for me.

In the end twtxt was the one that ticked all of my boxes:

- Slow social: it act more like a feed reader and I really appreciate that there's no flood of content that I can't keep up with.
- No server needed: I absolutely love to have total control over my content, I tend to avoid having moving parts that might break, plus you can put your feed under version control and it's all backed up.
- Ownership: I can put my feed anywhere I want and nobody can decide if I can access it or not.
- For hackers: a single .txt file allows me to join a community, how cool is that!

This is why I decided to build my own twtxt client, one that allows you to decide how the feed is presented on your "instance".

It's still in the making but I'll try to share a bit of it once I defined how things should work.

Coincidentally, I discovered that @itsericwoodward and @zvava were also building a twtxt client, seems like twtxt is set to grow!
Hello world, everyone!