Publ + Pushl releases and a bunch of plans
There’s been a few releases of both Publ (now on 0.7.35) and Pushl (now on 0.4.0). A pretty decent amount has changed!
There’s been a few releases of both Publ (now on 0.7.35) and Pushl (now on 0.4.0). A pretty decent amount has changed!
Publ changes since 0.7.31:
- Fix some error handling issues causing an ISE
- Add support for HTTP
Accept:
, properly allowing multiple templates with the same name and providing reasonable fallback behavior - Improve the
Content-Type
handling in general - Fix some markup-safe handling bugs
Note that in order to upgrade to 0.7.35 you’ll also need to restrict your Python environment to use a Python version < 3.13; more on that in a bit.
Pushl changes since v0.3.5:
- Tidy up some code rot
- Actually send an
Accept:
header - Removed lxml + Pingback support, which has never actually been useful
So, let’s talk about these projects and some other related stuff.
Tech debt
Publ has quite a lot of tech debt.
Most of the testing is done in an ad-hoc smoke-testing manner, and this has sometimes made it difficult for me to work on new features. (Of course I also am the only person working on these features, so I only prioritize things that I personally need.)
Pony
One particular issue is that it relies heavily on Pony. Pony is a great ORM for doing quick application prototyping stuff, but unfortunately it’s been a problem for quite some time:
- It makes proper unit testing significantly more difficult (it’s actually the main reason the testing is in such a sorry state right now)
- New Python versions usually end up breaking its clever code generator (which is why Publ 0.7.35 requires you to restrict your Python version)
- It never gained support for schema migrations (which, thankfully, Publ was designed to not need to begin with, but this hampers some of my other plans)
- General community support just never really materialized
To address many of these problems, I’ve been wanting to move to another database abstraction; I will most likely use SQLAlchemy. Whether I use its ORM or not is something I’m still trying to decide.
Whoosh
Similarly, full-text search uses whoosh, which has been more or less abandoned. I have updated to use whoosh-reloaded, a community fork that has fixed most of the more egregious issues, but there are still a lot of issues with it:
- Its locking behavior is difficult to work with (and can cause a lot of operational difficulty)
- Its ingest is slow and not easily threadable
- Updating the index can be difficult and fragile
- It stores the index on the local filesystem which can be a problem for many deployment scenarios
- It puts way too much structured into its structured queries, and is overkill for the kinds of query representation Publ needs
Unfortunately there aren’t as many existing full-text search implementations for Python. My expectation for now is that I’ll roll my own using the algorithms described in Bart de Goede’s article, although this doesn’t feel like a great way to do things either.
Misaka
Finally, the Markdown engine currently used by Publ has been abandoned and is unlikely to continue to working in the long term. It’s always been a bit of an operational problem, as well, because it relies on Hoedown (which has been abandoned for ages and never even got updated with HTML 5 support) and requires being able to either retrieve architecture-specific binaries or being able to build them yourself. The build process has been mostly easy for most users, but it’s still not ideal.
Fortunately, I’ve been playing with other Markdown implementations on other projects, and I’m fairly certain that I’ll be happy with Mistune instead. Switching to that will require reimplementing a bunch of stuff in Publ, but it’s all stuff I’d been wanting to fix anyway.
In particular, I’ve been wanting to figure out a way to templatize footnotes (for example, letting people make use of Tufte sidenotes or putting a <details>
reveal inline or after the current paragraph or the like), and there’s a lot that could be better about how image sets are currently handled, and ideally I’d be able to templatize those as well.
Comments
The main ways that I’ve handled comments on Publ-based sites is either:
Both of these are Fine™ for simple usage but they run into a bunch of issues on larger sites, and also the UX just isn’t really where I’d like it to be.
I wrote a much more detailed blog post on my main site, but the short version is that I’d like to make a comment system that works more closely with Publ (or any other publishing framework) that stores things locally and supports both local posts and webmention (both sending and receiving), and which would also accept user data from the publishing stack.
I’m thinking it would take the form of a Python library that you can embed into an app (with easy hooks for Flask), but would also offer its own Flask frontend for hosting it as an embeddable app instance that can be used from non-Python things.
ActivityPub
Native ActivityPub support has been at the back of my mind for a while. Having its own built-in webmention endpoint would also make for a nice spot to start adding in ActivityPub, since ActivityPub verbs aren’t fundamentally different from Webmention verbs.
The main thing this would bring to the table is being able to set up various outboxes for different views (for example, @blog@example.com
for just blog posts or @all@example.com
for everything), and then also being able to make use of the user permissions to send private entries as DMs to the authorized subscribers, reducing the need for private entry stubs (which are bad UX all around and which I only adopted as a compromise because magic links have problems and feed readers still don’t support bearer tokens or Ticket Auth).
Documentation
Also, having had to actually consult the docs while building a new Publ-based website, I’ve come to realize just how bad a disaster the manual currently is. It could really do with some reorganization at the very least.
In conclusion
I have a lot of stuff I want to work on and hopefully I get somewhere with some of these things this year. If you’d like to help out, you can make code contributions, or you can make financial contributions via Ko-Fi, Patreon, or GitHub Sponsors. But no pressure.