I've not checked actual file rotation. With
max_twts_per_rotation set to 100 and me posting ~ once a week first roatation will take place in two years ;-)
I've not checked actual file rotation. With
> I feel like
README will need a rework soon. There’s a lot of options now. Or maybe a manpage instead.For example that
local_twtxt_dir MUST end in path separator should be mentioned somewhere ;-)
I've a problem with
local_twtxt_file not beeing supported any more. Being forced to use twtxt.txt as file name breaks at least my URL.
> The user is supposed to run it manually or via a periodic system
> service. The recommended period is a month but could be less.
So me doing it weekly is a bit over cautious. It's often overseen by users that they are supposed to perform this task regularly.
* Advent of Code
* Carlsen-Nepomniachtchi
btrfs scrub e.g. once a week.I'm using
btrfs scrub -B /dev/xyz and mail the result to myself.
https://www.creative-webdesign.de/en/software/keepass-plugins/multi-cert-keyprovider
We leave master password empty. Each person needs an own certificate to access the database file.
Not using a master password makes it easy to add or remove people with access w/o changing (and sharing) a master password.
SRCFILES = $(wildcard *)
# remove existing *.gz (actually doubles entries)
CLEANSRC = $(SRCFILES:.gz=)
DSTFILES = $(addsuffix .gz, $(CLEANSRC))
%.gz: %
\tgzip -c $< > $<.gz
all: $(DSTFILES)
You must not have subdirectories in that folder, though.
SRCFILES = $(wildcard *)
# remove existing *.gz (actually doubles entries)
CLEANSRC = $(SRCFILES:.gz=)
DSTFILES = $(addsuffix .gz, $(CLEANSRC))
%.gz: %
gzip -c $< > $<.gz
all: $(DSTFILES)
You must not have subdirectories in that folder, though.
Actually in nano you would have to ctrl-k ctrl-k ctrl-x y
Writing a twt is more similiar to writing a commit message. Git does quite some checks to detect that nothing **new** was written and happily discards a commit if you just leave the editor. You don't need any special action, just quit your editor. Git will take care for the rest.
But it's OK as it is. I just didn't expect that I have to select and delete all to discard a twt. So it's C-x h C-w C-x C-c for me.
I only know this "feature" from my revision control software where I get "abort: empty Commit message" or "Aborting commit due to empty commit message" when I do not change whatever is already in there. Can be quite some text about which files changed and so on.
I hit "reply" hotkey and my editor comes up.
With or without writing something I close my editor **without saving the content**.
Of course I close it by C-x C-c, not by :q! ;-)
Jenny finds the temp file unchanged, e.g. it's content is the same as it was when my editor was started. I would like that jenny discards the reply then.
Autosaving is no problem either. Real editors do this to a temporary (kind of backup) file. Only in case of a crash that file is consulted and the user is asked if she would like to continue with that stored content.
But I can't reproduce any weird behaviour right now. I've tried to "blackhole" twt.nfld.uk temporarily. That does not have any effect.
I've also tried to force twt.nfld.uk to deliver an empty twtxt. That does not have any effect either.
So I guess everything is fine with jenny.
I have wrapped jenny into some shell script to versionize
~/.cache/jenney. This way I have better data if anything unexprected is showing again.
eleven and utf8test, https://search.twtxt.net is the only follower. Maybe you can stop it to follow those twtxts? They were meant for testing purposes only.
jenny -f. Have now jennys cache under revision control, automatically commiting changes after each fetch. Let's see if this helps finding a (possible) bug.
diff --git a/jenny b/jenny
index b47c78e..20cf659 100755
--- a/jenny
+++ b/jenny
@@ -278,7 +278,8 @@ def prefill_for(email, reply_to_this, self_mentions):
def process_feed(config, nick, url, content, lasttwt):
nick_address, nick_desc = decide_nick(content, nick)
url_for_hash = decide_url_for_hash(content, url)
- new_lasttwt = parse('1800-01-01T12:00:00+00:00').timestamp()
+ # new_lasttwt = parse('1800-01-01T12:00:00+00:00').timestamp()
+ new_lasttwt = None
for line in twt_lines_from_content(content):
res = twt_line_to_mail(
@@ -296,7 +297,7 @@ def process_feed(config, nick, url, content, lasttwt):
twt_stamp = twt_date.timestamp()
if lasttwt is not None and lasttwt >= twt_stamp:
continue
- if twt_stamp > new_lasttwt:
+ if not new_lasttwt or twt_stamp > new_lasttwt:
new_lasttwt = twt_stamp
mailname_new = join(config['maildir_target'], 'new', twt_hash)
lasttwts is reset to -5364619200.0 every time no new content wasfetched for example if if-modified-since did not produce new twts?
It looks like
lasttwts is reset to -5364619200.0 every time no new content wasfetched for example if if-modified-since did not produce new twts?
lastmods, lastseen and lasttwts it's easy to test.Works like a charm!
lastmods, lastseen and lasttwts it's easy to test.\n\nWorks like a charm!
if-modified-since ;-)I've almost only timestamps -5364619200.0...
Diff looks good to me!
if-modified-since ;-)\nI've almost only timestamps -5364619200.0...\nDiff looks good to me!
I'll test it tomorrow. Thank's for starting this feature!
glob () is an O(n) algorithm\nYes, I see. But don't underestimate OS caching for files and directories!\nIf you look up files in the same directory many times then OS may use cached results from earlier lookups.\nI'm not totally sure but I believe this is how things work for both, Windows and Linux at least.
> (#el7d3ja) I believe
glob () is an O(n) algorithmYes, I see. But don't underestimate OS caching for files and directories!
If you look up files in the same directory many times then OS may use cached results from earlier lookups.
I'm not totally sure but I believe this is how things work for both, Windows and Linux at least.
F in their name.\n\nI see that in mail_file_exists jenny does not consider flagged messages when testing if a message already exists.\n\nI understand that looking up only 12 combinations is faster than reading huge directories. I'm astonished that globbing would be slower. Learning something new every day...
When I look in my twtxt maildir for duplicated messages they all have
F in their name.I see that in
mail_file_exists jenny does not consider flagged messages when testing if a message already exists.I understand that looking up only 12 combinations is faster than reading huge directories. I'm astonished that globbing would be slower. Learning something new every day...
I just pulled it, works like a charm (as expected) ;-)
I'm not a Python programmer, so please bear with me.
The doc about encodings does also mention:
If you require a different encoding, you can manually set the Response.encoding property
Wouldn't that be a one liner like (Ruby example)?
'some text'.force_encoding('utf-8')
I understand that you do not want to interfere with
requests. On the other hand we know that received data must be utf-8 (by twtxt spec) and it does burden "publishers" to somehow add charset property to content-type header. But again I'm not sure what "the right thing to do" (TM) is.
requests. On the other hand we know that received data must be utf-8 (by twtxt spec) and it does burden "publishers" to somehow add charset property to content-type header. But again I'm not sure what "the right thing to do" (TM) is.
content-type: text/plain leaving out charset declaration).\n\nAfter following utf8test twtxt myself I now see that jenny does not handle it as UTF-8 when charset is missing from HTTP header, just like @quark has observed.\n\nSo should jenny treat twtxt files always as UTF-8 encoded? I'm not sure about this.
Exactly, you see correct UTF-8 encoded version (even with
content-type: text/plain leaving out charset declaration).After following utf8test twtxt myself I now see that
jenny does not handle it as UTF-8 when charset is missing from HTTP header, just like @quark has observed.So should
jenny treat twtxt files always as UTF-8 encoded? I'm not sure about this.
Sorry, I should have mentioned your twt #vjjdara where you already described the same idea.
Applause!
\o/
AddCharset utf-8 .txt to .htaccess. Especially I did not change encoding of stackeffect.txt.
> I believe Yarn assumes utf-8 anyway which is why we don’t see encoding issues
Are you sure? I think in #kj2c5oa @quark mentioned exactly that problem. My logs say "jenny/latest" was fetching my twtxt for quark.
All I did to fix this was to adding
AddCharset utf-8 .txt to .htaccess. Especially I did not change encoding of stackeffect.txt.
Don't miss step 0 (I should have made this a separate point): having a meta header promising appending twts with strictly monotonically increasing timestamps.
> (Also, I’d first like to see the pagination thingy implemented.)
In jenny I would like to see "don't process previously fetched twts" AKA "Allow the user to archive/delete old twts" feature implemented ;-)
charset: utf-8 by default.\n\nI had to set/modify .htaccess to correct that.\n\nIt would have been easier if there had been a charset header entry "overwriting" what http server is delivering.\n\nWhat do you think?
I myself stumbled upon .txt files not being delivered with
charset: utf-8 by default.I had to set/modify
.htaccess to correct that.It would have been easier if there had been a charset header entry "overwriting" what http server is delivering.
What do you think?
Additionally to pagination also range request should be used to reduce traffic.
I understand that there are corner cases making this a complicated matter.
I would like to see a meta header saying that the given twtxt is append only with increasing timestamps so that a simple strategy can detect valid content fetched per range request.
1. read meta part per range request
2. read last fetched twt at expected range (as known from last fetch)
3. if fetched content starts with expected twt then process rest of data
4. if fetched content doesn't start with expected twt discard all and fall back to fetching whole twtxt
Pagination (e.g. archiving old content in a different file) will lead to point 4.
Of course especially pods should support range requests, correct @prologic?
next and prev pointers seems a really amazing solution.\n\nAs in RFC5005 there should also be a meta header pointing to the **main** URL, e.g. current or baseurl or something like that. This way hashes can calculated correctly even for archived twts.
Following the discussion about pagination (paging) I think that's the right thing to do.
Fetching the same content again and again with only a marginal portion of actually new twts is unbearable and does not scale in any way. It's not only a waste of bandwidth but with increasing number of fetchers it will also become a problem for pods to serve all requests.
Because it's so easy to implement and simple to understand, splitting twtxt file in parts with
next and prev pointers seems a really amazing solution.As in RFC5005 there should also be a meta header pointing to the **main** URL, e.g.
current or baseurl or something like that. This way hashes can calculated correctly even for archived twts.
> I’m curious, what is your use case for deleting twts?
Not just deleting, also sorting into other folders is impossible.
It also doesn't scale in the long term. When I cannot delete twts then I have a full copy of every twtxt I follow - forever. That's a waste of bandwidth and disk space.
D~d>1m and then fetched by !jenny -f. This brings back all deleted twts. Isn't lastmods used to skip older twts?
No, it would be sufficient to skip avatar discovery when metadata does contain an avatar.
Thank you, that's the correct one.
Still I have this in my logs (first access of "eleven" by yarnd):
ip.ip.ip.ip - - [21/Oct/2021:20:05:36 +0000] "GET /eleven.txt HTTP/2.0" 200 344 "-" "yarnd/0.2.0@46bea3f (Pod: twtxt.net Support: https://twtxt.net/support)"
ip.ip.ip.ip - - [21/Oct/2021:20:05:36 +0000] "HEAD /avatar.png HTTP/2.0" 200 0 "-" "yarnd/0.2.0@46bea3f (Pod: twtxt.net Support: https://twtxt.net/support)"
And I guess without avatar.png sitting there I would have seen even more requests like /eleven.txt/avatar.png.
I've copied stackeffect.png to avatar.png to make yarnd happy when accessing stackeffect.txt.
So in this setup yarnd fetched eleven.txt along with avatar.png which belongs to another twtxt. This feels buggy.
date -Is and emacs. Off course all is UTF-8 encoded, but now Apache also announces content-type: text/plain; charset=utf-8
E.g. it could refetch only last twtext line of last request to make sure it starts off at correct position.
I guess there are twtxt files that only grow, then this will save a lot bandwidth over time.
For twtxt files that "forget" older content this situation would be detected and as a fallback the whole twtxt file could then be fetched.
;-)
Personally I started looking at some twtxt files with curl and saw information about avatar images.
I assumed that to be sort of standard and mentioned my avatar image in my stackeffect.txt. But it was not "avatar.png".
Later I saw in logfiles that the info was totally ignored and instead several "avatar.png" locations were tried by the pulling side.
When information in "header" of twtxt file were respected one could easily change avatar file to one with a new filename and there would be no caching problem.