Publ: Development Blog

Entries tagged pushl, import or planning

Pushl v0.2.13

Posted Sunday, December 1 at 9:10 PM (2 months ago)

I’ve released a new version of Pushl.

Changes since the last version:

  • Added support for tracking entry URL changes
  • Finally got around to adding type annotations and static analysis

Pushl v0.2.12

Posted Thursday, October 3 at 1:49 AM (4 months ago)

I have now released Pushl v0.2.12. The following is new:

  • It now respects rel="canonical" or rel="self" when determining which URL to send a WebSub ping for
  • You can now send self-pings using the --self-pings parameter
  • Miscellaneous code cleanups

Pushl v0.2.11

Posted Monday, September 23 at 9:33 AM (5 months ago)

In trying to fix what looked like a bug in Pushl (which turned out to be a bug in one of the services I was pinging), I did a bunch of much-needed code cleanup and refactoring.

I also added the ability to ping the Internet Archive Wayback Machine for outgoing links if the target has changed (relative to the usual If-Modified-Since/If-None-Match tests).

Pushl will now also log warnings for two useful situations:

  • An outgoing link generates a 400-class error (403/404/410/etc.)
  • An outgoing webmention has a different canonical URL than what’s being pinged (improved since v0.2.8)

The way it handles canonical URLs is also now improved; if a page has <link rel="canonical"> it will use that, otherwise it will use the final URL that is the result of chasing redirects.

Pushl v0.2.8

Posted Wednesday, August 21 at 10:21 PM (6 months ago)

I’ve released v0.2.8 of Pushl, which fixes an issue with Webmention and Pingback where it was over-optimistically setting the link target. It will also warn you if the link target doesn’t match with the actual page, so you can update your links accordingly.

Right now it’s a little spammy (in that it’ll tell you about redirection mismatches for all links, not just ones with a Webmention or Pingback endpoint), but the next version will address that.

Pushl v0.2.7

Posted Monday, August 12 at 5:25 PM (6 months ago)

Around a month ago a bunch of my webmention stuff broke on my site, and I just figured out what was causing it. Pushl was getting confused by the fact that I had multiple feeds which provided the same content, and some of them were in a no-webmentions context. The no-webmentions ones were getting processed first, which was preventing the webmention-context versions from actually being processed.

So, I fixed this bug by making the context part of what dedupes the actions.

Every time I work on Pushl I feel like it could use a major rewrite, incidentally. This is one of those times.

Publ 0.4.6, Authl 0.1.5

Posted Friday, July 26 at 12:36 AM (7 months ago)

Updated some packages.

Main things with Publ since the last release:

  • Internal cleanups to how caching happens
  • Stop spuriously-caching a bunch of stuff; in particular login/logout endpoint URLs no longer get cached
  • Various cleanups
  • Improve the way that built-in templates are managed
  • Initial cruddy implementation of an admin authentication dashboard (although this isn’t quite ready for prime time)

The only Authl change is that email identities are now given as a full mailto: URL; going forward all identity strings will be full URLs. This simplifies the UX for admin dashboards, in particular, and removes some ambiguity.

v0.4.0 released! Let’s talk future!

Posted Monday, May 13 at 11:11 PM (9 months ago)

Oh wow, I finally closed out the Publ 0.4.0 milestone. So, wow, this is a pretty big deal for me.

Changes since 0.3.23:

So that closes out the last three issues before I was ready to consider this usable by a wider audience. Neat.

There’s a lot of stuff I want to do for 1.0 of course, but also more importantly I need to vastly improve the documentation on this site, as well as collect things to make it easier for people to get started with Publ. I would absolutely invite people to make suggestions; have you looked into Publ, and been confused about what exactly to do? Please let me know!

The best ways to do so are to either hop onto my Discord channel or to open issues against the documentation site.

Pushl v0.2.5 (not a joke!)

Posted Monday, April 1 at 11:29 AM (10 months ago)

I have released v0.2.5 of Pushl. The changes since v0.2.4:

  • Improved the logging levels a bunch, making them more useful
  • Added the ability to only send WebSub for feeds

The latter improvement makes it so that if your site is accessible from multiple URLs (e.g. http and https, or multiple domain names), it won’t send multiple Webmentions to everyone with each possible URL. This helps to cut down on spamminess to sites which don’t detect multi-origin pings (such as most IndieWeb blogs or fed.brid.gy).

I also finally added a tools page to this website, to collect useful things that make Publ work better with other things. I also intend to add various useful quality-of-life things like an image cropping tool, and whatever else might occur to me down the road.

Someday I need to get around to making a proper “Features” page for this site, too. Someday…

Pushl v0.2.4, now with a proper user-agent

Posted Friday, March 15 at 5:29 PM (11 months ago)

While trying to figure out some weird access patterns on the day-job site I had the realization Pushl wasn’t actually specifying a user-agent, so it was just coming through as the generic aiohttp one, which isn’t very friendly.

Now it sends a reasonable user-agent by default, and this can be overridden by the --user-agent flag if you want to for your own analytics or whatever.

Oh, and I had quietly released 0.2.3 a few days ago; there were just some minor internal changes to logging and also declaring Pushl as beta, rather than alpha, software.

Pushl 0.2.2

Posted Sunday, March 10 at 6:25 PM (11 months ago)

I’ve done a bunch more work on Pushl to try to get it more stable. In particular, I’ve made it so that it will only recurse into feeds that are on domains that were declared in the initial requests, and I seem to have cleared up some cases which were causing it to hang and also added a global timeout which will, hopefully, prevent it from hanging indefinitely.

I do wish I could figure out what is causing the hangs when they do happen though. Oh well. Some discussion of the issue below the cut.

Pushl v0.2.1 released

Posted Thursday, March 7 at 10:27 PM (a year ago)

I’ve been working on getting Pushl much more stable and reliable, particularly around a persistent “too many open files” error I was having, which turned out to be primarily due to a fd leak in the caching routines. Oops.

Anyway, there’s also seemingly a problem with how aiohttp manages its connection pool, at least on macOS, so I’ve disabled connection keep-alive by default. However, if you still want to use keep-alive, there’s now a --keepalive option to allow you to do that. I’m finding that it doesn’t really improve performance all that much anyway.

This is feeling beta-ready but I’ll give it a few days for other issues to shake out first.

Pushl v0.2.0 released

Posted Thursday, March 7 at 12:05 AM (a year ago)

So, I just released v0.2.0 of Pushl. It was a pretty big change, in that I pretty much rewrote all the networking stuff, and fixed some pretty ridiculous bugs with the caching implementation as well.

The main thing is now it’s using async I/O instead of thread-per-connection, so it’s way more efficient and also times out correctly.

And oh gosh, I had so many tiny but critical errors in the way caching was implemented – no wonder it kept on acting as if there was no cached state. Yeesh.

Anyway, I’ll let this run on my site for a few days and if I like what I see I’ll upgrade it to beta status on PyPI.

v0.3.19, now with extra tagging goodness!

Posted Monday, March 4 at 3:47 PM (a year ago)

I’ve released Publ v0.3.19, which now finally has a tagging system, which is only one of the oldest issues that was still open.

Here’s a list of what’s been added or changed since 0.3.18:

Credits

I want to thank Karina Antonio for implementing image cropping.

v0.3.18, now with better asset management!

Posted Wednesday, February 27 at 9:38 PM (a year ago)

I’ve just released v0.3.18, with the following changes:

  • Add date grouping properties to entry
  • Add a pages property to view
  • Provide the current category object to the error handler
  • Support linking to non-image/non-entry local files
  • Added, then removed, some performance micro-optimizations that only caused problems

More details about the major changes below!

Update: I released a hotfix as 0.3.18.1 because there was a last-minute bug that snuck in while I was trying to silence a new pylint error. Oops.

An early-alpha Movable Type importer

Posted Wednesday, February 20 at 3:42 PM (a year ago)

For those folks who want to import their content from Movable Type over to Publ, I’ve finally gotten around to writing an importer. Currently it only attempts to convert entry content and category metadata, and only using SQLite-formatted database dumps.

See its README.md for the (incredibly rough) usage instructions.

Eventually I want to try to automatically convert templates from MT’s scripting language to Jinja-Publ templates, although there’s a bunch of stuff that’s going to be difficult to port across and a lot of stuff is just plain not feasible to even try, so don’t expect that to become a major thing any time soon.

Pushl v0.1.7

Posted Monday, January 14 at 9:28 PM (a year ago)

I ended up doing some more work on Pushl and have now released v0.1.7. The major changes:

  • Did a bunch of refactoring to make the code a little cleaner and handle configuration more appropriately
  • Added a configurable timeout for connections (which now defaults to 15)
  • Added a --version option on the command line arguments

Also, some suggested usage ideas below the cut!

Pushl v0.1.6 released

Posted Sunday, January 13 at 8:48 PM (a year ago)

It’s been a while since I’ve updated Pushl but today I released v0.1.6. It includes the following fixes:

  • Now it supports Pingback as well as Webmention
  • Improved the threading defaults and connection pooling
  • Also checks entries for updates even if the feed didn’t change (in case something changed in the more text or page metadata or whatever)

Anyway, it should just be a pip install --upgrade pushl (or pipenv update) away.

Pushl v0.1.5

Posted Saturday, December 22 at 1:35 AM (a year ago)

While I’m fixing random stuff in Publ, I figured I’d finally fix some problems with Pushl too. Nothing major here, just:

  • Stability: Fixed a bug where feeds that don’t declare links caused the worker to die before entries got processed
  • Performance: Now we use a global connection pool (so connections can be reused)
  • Fixed a minor correctness issue with archive feeds (which actually doesn’t make any difference in the real world but whatever)

v0.3.11

Posted Saturday, December 15 at 1:08 AM (a year ago)

v0.3.11 is now released, with the following changes:

  • A more complete fix for how to handle image sets and inline images with respect to paragraphs
  • Better cleanup for spurious empty paragraphs
  • Improved internal entry link handling

Detailed descriptions of the changes are below.