Publ changes since 0.7.31:
Accept:, properly allowing multiple templates with the same name and providing reasonable fallback behaviorContent-Type handling in generalNote that in order to upgrade to 0.7.35 you’ll also need to restrict your Python environment to use a Python version < 3.13; more on that in a bit.
Pushl changes since v0.3.5:
Accept: headerSo, let’s talk about these projects and some other related stuff.
Publ has quite a lot of tech debt.
Most of the testing is done in an ad-hoc smoke-testing manner, and this has sometimes made it difficult for me to work on new features. (Of course I also am the only person working on these features, so I only prioritize things that I personally need.)
One particular issue is that it relies heavily on Pony. Pony is a great ORM for doing quick application prototyping stuff, but unfortunately it’s been a problem for quite some time:
To address many of these problems, I’ve been wanting to move to another database abstraction; I will most likely use SQLAlchemy. Whether I use its ORM or not is something I’m still trying to decide.
Similarly, full-text search uses whoosh, which has been more or less abandoned. I have updated to use whoosh-reloaded, a community fork that has fixed most of the more egregious issues, but there are still a lot of issues with it:
Unfortunately there aren’t as many existing full-text search implementations for Python. My expectation for now is that I’ll roll my own using the algorithms described in Bart de Goede’s article, although this doesn’t feel like a great way to do things either.
Finally, the Markdown engine currently used by Publ has been abandoned and is unlikely to continue to working in the long term. It’s always been a bit of an operational problem, as well, because it relies on Hoedown (which has been abandoned for ages and never even got updated with HTML 5 support) and requires being able to either retrieve architecture-specific binaries or being able to build them yourself. The build process has been mostly easy for most users, but it’s still not ideal.
Fortunately, I’ve been playing with other Markdown implementations on other projects, and I’m fairly certain that I’ll be happy with Mistune instead. Switching to that will require reimplementing a bunch of stuff in Publ, but it’s all stuff I’d been wanting to fix anyway.
In particular, I’ve been wanting to figure out a way to templatize footnotes (for example, letting people make use of Tufte sidenotes or putting a <details> reveal inline or after the current paragraph or the like), and there’s a lot that could be better about how image sets are currently handled, and ideally I’d be able to templatize those as well.
The main ways that I’ve handled comments on Publ-based sites is either:
Both of these are Fine™ for simple usage but they run into a bunch of issues on larger sites, and also the UX just isn’t really where I’d like it to be.
I wrote a much more detailed blog post on my main site, but the short version is that I’d like to make a comment system that works more closely with Publ (or any other publishing framework) that stores things locally and supports both local posts and webmention (both sending and receiving), and which would also accept user data from the publishing stack.
I’m thinking it would take the form of a Python library that you can embed into an app (with easy hooks for Flask), but would also offer its own Flask frontend for hosting it as an embeddable app instance that can be used from non-Python things.
Native ActivityPub support has been at the back of my mind for a while. Having its own built-in webmention endpoint would also make for a nice spot to start adding in ActivityPub, since ActivityPub verbs aren’t fundamentally different from Webmention verbs.
The main thing this would bring to the table is being able to set up various outboxes for different views (for example, @blog@example.com for just blog posts or @all@example.com for everything), and then also being able to make use of the user permissions to send private entries as DMs to the authorized subscribers, reducing the need for private entry stubs (which are bad UX all around and which I only adopted as a compromise because magic links have problems and feed readers still don’t support bearer tokens or Ticket Auth).
Also, having had to actually consult the docs while building a new Publ-based website, I’ve come to realize just how bad a disaster the manual currently is. It could really do with some reorganization at the very least.
I have a lot of stuff I want to work on and hopefully I get somewhere with some of these things this year. If you’d like to help out, you can make code contributions, or you can make financial contributions via Ko-Fi, Patreon, or GitHub Sponsors. But no pressure.
]]>Thanks to @seirdy for bringing this to my attention.
]]>Per the above, the main code branch1 is now called main, and you include and exclude link rel attributes with --rel-include and --rel-exclude, in keeping with some long-overdue naming convention updates.
We’re all in this together.
]]>h-feed/h-entry) feedsNow all you “RSS is dead” folks have no excuse not to use Pushl. 😘
Changes since the last version:
There are three ways in which an entry URL may change:
The entry item’s link in the feed changed, and the old link now redirects
Previously, Pushl would try retrieving the old URL, would get redirected to the new URL, and then would only send updates for any target URLs which changed, coming from the new URL. It would also see the new URL as a separate entry and send out pings from it. At best, this meant that there would be duplicated pings from the old and new URLs.
Now, it sees that the source URL has changed and re-sends all of the old pings from the old URL, which lets the endpoint know that the URL has changed and should be updated. (It also sends a fresh set of pings from the new URL, which will at least theoretically be ignored by the endpoint, if it’s implemented correctly.)
The entry item’s link in the feed changed, and the old link now generates an error
This path hasn’t changed at all; Pushl saw the old URL as a deletion, and the new one as an addition, and either way would send out pings for all targets from both URLs.
The feed linked to it via a redirection URL (such as a short link or a tracking link), and the redirection changed
Previously, Pushl would not see this as as change at all in the entry, and would not send out any updates at all for old targets. This causes all sorts of potential failures not worth enumerating.
Now, it sees that the destination changed, and re-sends all pings from the old target URL, telling the endpoint to update them.
However, this has now introduced a bug which I didn’t realize until writing up this explanation: it only works if the old redirected URL also redirects to the new entry URL, in which case it behaves like case 1 above. If the old URL now errors, this will appear as a deletion to the endpoint, and all of the mentions go away – and Pushl won’t ever try to re-send them.
So, I’ve already made a bugfix (which will go out in the next Pushl release, whenever that is) where the pings are always sent from the original URL in the feed. This way, it’s up to the endpoint to see that the redirection updated. This puts the onus on the endpoint to be able to track changes in redirections in the source URL, or always use the source URL verbatim even if it’s a redirection. Either way, Pushl is trying its hardest to do the right thing in a tricky situation.
Update: I rolled back that bugfix, because it had the much worse effect of breaking rel="canonical". A better solution would be to re-send the old pings from the old URL instead. Which is now implemented and tested working.
In an ideal world, case 3 simply wouldn’t happen, but unfortunately, a lot of feeds do this for various reasons. Feedburner is particularly egregious about it, but a lot of blogging platforms do it naturally; Tumblr, for example. And, heck, it’s easy to configure Publ to do that too (and it always does this for unauthorized posts, as part of the privacy model).
Anyway. Consider this fix my day 1 effort for the IndieWeb 2019 Challenge.
]]>rel="canonical" or rel="self" when determining which URL to send a WebSub ping for--self-pings parameterI also added the ability to ping the Internet Archive Wayback Machine for outgoing links if the target has changed (relative to the usual If-Modified-Since/If-None-Match tests).
Pushl will now also log warnings for two useful situations:
The way it handles canonical URLs is also now improved; if a page has <link rel="canonical"> it will use that, otherwise it will use the final URL that is the result of chasing redirects.
Right now it’s a little spammy (in that it’ll tell you about redirection mismatches for all links, not just ones with a Webmention or Pingback endpoint), but the next version will address that.
]]>So, I fixed this bug by making the context part of what dedupes the actions.
Every time I work on Pushl I feel like it could use a major rewrite, incidentally. This is one of those times.
]]>