Publ changes:

• Added support for IndieAuth Ticket Auth, allowing aware readers to automatically log in on your behalf and get private entries
• Added support for HTML 5 lazy image loading; all images are now lazy-loaded by default

Authl changes:

• Improve the meta robots rules on the login form
• Add IndieWeb endpoint discovery to the profile

Ticket Auth is a pretty cool protocol which I have a lot of optimism for. You know how one of the big problems with self-hosted blogging is that there’s no way to share private posts on your feed without revealing that you have private posts and requiring people to log in to actually read them? Ticket Auth is a huge solution for that.

There’s a few things that need to happen before it actually gets used, though. The main thing is that feed readers need to get support for Ticket Auth, and there also needs to be a way to associate your user profile with the Ticket Auth endpoint. In an IndieWeb context this is pretty straightforward; on your profile page you’d have something like:

<link rel="ticket_endpoint" href="https://feed-reader.example/tickets">

and when you sign in, Publ sees that you have that ticket endpoint and initiates the flow to it. Then the feed reader has a bearer token that it can use to fetch the secure feed on your behalf.

However, outside of IndieAuth it gets a little trickier. There’s technically no requirement that someone uses the IndieAuth login flow to get a ticket, but the protocol relies pretty heavily on there being a tight coupling between the identity provider and the feed reader. The most straightforward way to support this in non-IndieWeb feed readers is to simply have feed readers grow IndieAuth support, though, and then use that to sign in to the IndieWeb sites you’re following.

There’s probably some other mechanism that could be used to provide matching third-party identity between a feed reader and, say, a Twitter identity, but that’s getting so far off into the weeds.

My recommendation to anyone implementing a new feed reader in this day and age: integrate with existing IndieAuth login flows, and/or allow folks to identify themselves with a public IndieWeb profile when they log in with a username and password. Then you can provide them a public profile page that also has the Ticket Auth endpoint, and then we can finally break free of walled-garden social media.

Authl 0.3.1 fixes the problem with the IndieAuth login flow, and Publ 0.5.8 fixes the problem with the Bearer token flow.

If you don’t understand what any of that means, the short form is: please update your package versions. You might also want to change your secret key while you’re at it; even if you don’t have any private content yet, someone could possibly have used this hole to log in as you in case you ever do post private content.

It’s incredibly unlikely, of course! As far as I know I’m the only active user of Publ (aside from my old day job where they are definitely not using the authentication stuff at all). But I felt that full disclosure is a good idea anyway.

Also, if you have your own IndieAuth implementation that you want to check for proper identity sanitization, the Authl GitHub repository has an identity tester that should be easy to deploy. It’s worth testing against that to make sure that your identity verification is working correctly!

To use it, run it somewhere that’s visible to your IndieAuth login flow; for example, if you’re testing locally, you can do something like:

$ FLASK_APP=Authl/test/rogue-indieauth.py flask run -p 6789

and then you can use http://localhost:6789 as your identity; you can also add arbitrary path elements (e.g. http://localhost:6789/alice). Then when you try to log in as IndieAuth it’ll prompt you for what you want your canonical identity to look like (for example, http://example.com/ or http://localhost:6789/bob), and then see how your login flow deals with it.

At some point I’ll probably spin up a public instance of this, as well.

There’s also a deficiency in the IndieAuth spec regarding how to verify the path part of an identity URL; see this open issue if you want to see more and/or participate in the discussion.

Authl and Publ currently follow my proposal for the path validation, where for example http://example.com/alice can identify as http://example.com/alice/ or http://example.com/alice/blog/ but not as http://example.com/bob or http://example.com/alice_is_bob. This makes it technically stricter than the current public specification, but it’s also a lot safer especially for multi-user websites such as tilde.club or any random WordPress installation or whatever.

Anyway. This is just my long-winded way of saying, oops, I hecked up, but I fixed it, and maybe other people hecked up too and it’s worth testing.

EDIT: Oh I also forgot to mention that no, AutoAuth isn’t actually supported yet. But I’m working on it!

Out of the box, Publ authentication does support a shared cookie jar; if you can provide your cookies to your feed reader in some way, then things will Just Work. Unfortunately, I don’t know of any feed readers that actually support this, at least not easily. (Back when most browsers had a feed reader built-in this was a lot simpler. But time marches on.)

The two mechanisms which seemed most promising are AutoAuth and “magic links,” where users get signed URLs that come pre-authenticated and show the full authorized content for that user. AutoAuth is still in a draft phase that’s stuck in a chicken-and-egg situation (and also requires a lot of buy-in to IndieWeb protocols, which is still a pill too large to swallow for most of the folks who follow my blog), so magic feed links seemed like the best path forward.

I even got so far as to draft out an implementation, but there’s a few bad issues with it which just made me opt not to.

Feed discovery

Right now, when people want to subscribe to a feed, they usually point their feed reader at the URL for the website, and then let the reader software discover the feed. Usually there will be a <link> tag that provides the feed URL, like:

<link rel="alternate" type="application/atom+xml" title="Atom feed" href="feed" />

Sometimes there may be more than one of these <link> tags for different styles of feed; for example, it might have both RSS and Atom versions, or there might be a choice between full-content and summary, or a comments feed, and so on. Some feed readers will show a list and allow the user to select which feed to use, while others will simply use the first one.

In the case of a magic link, however, these links are only provided to the person who is logged in. An external feed reader won’t be logged in, and therefore won’t see the magic link.

So, an alternate discovery mechanism must be provided; usually this will take the form of a widget in the corner of the page where clicking on it will expand a text box you can select the (possibly really long) feed link from and then paste that into your feed reader. There is no standard approach to doing this, and is confusing and weird.

Item sharing

The bigger problem, however, and the reason I decided to abandon the project entirely, is that the way that Atom specifies item sharing makes this incredibly dangerous.

Atom provides a way of sharing items; Feed on Feeds implements this, for example. If you share an item, its Atom entry looks like this:

<entry>
<id>urn:uuid:dacd7607-380e-526d-b688-60d8d334bde7</id>
<link href="https://beesbuzz.biz/comics/journal/3675-ADDitive" rel="alternate" type="text/html"/>
<title type="html">Journal: ADDitive</title>
<summary type="html">
<!-- actual item content goes here -->
</summary>
<updated>2019-10-15T23:14:52Z</updated>
<source>
  <id>https://beesbuzz.biz/</id>
  <link href="https://beesbuzz.biz/" rel="alternate" type="text/html"/>
  <link href="https://beesbuzz.biz/feed" rel="self" type="application/atom+xml"/>
  <title>busybee</title>
</source>
</entry>

That <source> block is what to look at; namely, the rel="self". It provides a link back to the original feed. This means that if someone were to share an item from an authenticated feed – regardless of the privacy of the item itself – it would also share the authenticated feed URL. This can be very, very bad.

So what are the alternatives?

As stated in the preamble, the two major alternatives are shared cookies, and AutoAuth. Neither is a perfect solution.

Shared cookies are great if you can synchronize your session cookies between your browser and your feed reader. (This is especially feasible if your feed reader is hosted in your browser.) Most feed readers don’t work that way. So, you could export your cookies to be used by the feed reader (which hopefully uses the presence of a cookie to avoid deduping subscriptions! I’m pretty sure Feed On Feeds doesn’t!), but if cookies have an expiration date on them (which Publ cookies absolutely do) this means having to re-export periodically.

Some sort of feed metadata indicating that there is a login/auth mechanism available might be workable; something like <link rel="authenticate"> which prompts the browser to pop up some sort of proxy popup so that it can intercept the login cookie, for example. This might be a nice middle ground to AutoAuth without requiring every user to buy in fully to the IndieWeb experience.

(And, of course, supporting AutoAuth would be ideal for those who do.)

I think having some sort of “hey, please log in” metadata in the feed is also helpful if only because it gives a cue to a subscriber that there’s something to authenticate against in the first place. But this purpose is already served by having empty “private post” stub entries…

Other things to consider

While I’m ramble-thinking about this stuff, I’d also like to see a better mechanism for dealing with authentication around WebSub. As far as I’ve seen, there are three kinds of WebSub push:

1. Full-feed “fat ping” (i.e. the push notification contains the entire feed content)
2. Update-only “fat ping” (i.e. it only contains the new/changed items)
3. Notification-only “thin ping” (it only sends a notification of the update and then the recipient needs to do a pull of the content, once notified)

The WebSub model doesn’t really have any provisions for determining authentication/authorization status, as there’s no mechanism for associating authentication stuff with the subscription topic. In case 3 it doesn’t really matter – the client will just provide its normal content-pull credentials – but in cases 1 and 2 it matters quite a lot, as the content needs to be pre-filtered through the authentication layer.

For what it’s worth, on all of my sites I use SuperFeedr as my WebSub hub, which does case 2 (actually a particularly annoying version of it where it only pushes new items, rather than including changed items). It definitely doesn’t provide the extensibility required for authenticated WebSub, and I doubt that this is anything they ever would add even if a standard were to be adopted. So, I think for the non-thin push case, it would become necessary to have a different hub. Switchboard appears to do a full-content push (case 1 above) and doesn’t currently support AutoAuth; however, given the author, I would expect it to add that functionality when it becomes more commonplace.

In the meantime, I think I’ll continue on with my unauthorized stub entries; they’re annoying to unauthorized users and they leak the fact I’m posting private entries to the world, but at least they prompt people to sign in and notify folks that there might be something for them to read. And for better or worse it also works with my POSSE setup.

Conclusion

Software is hard.

Part of this update was to also refactor how OAuth is handled, so it’ll be a lot easier for me to add more OAuth-based providers in the future; hopefully I’ll have direct support for Twitter, GitHub, and maybe even Facebook in the near-ish future. But for now, between Mastodon, email, and IndieAuth, I think I have all of my own personal needs taken care of.

Feel free to make suggestions for other identity providers in the Authl issue tracker, though!

Well you’re in luck, I wrote a simple-ish script for that. (You’ll probably also want to see the accompanying stylesheet too.) And it doesn’t even require that you use Publ – it should work with any CMS, static or dynamic. The only requirement is that you use either webmention.io or something that has a similar enough retrieval API.

I wrote more about it on my blog, where you can also see it in use. For now, I’m just going to use the sample site repository to manage it (and issues against it).

It’s MIT-licensed, so feel free to use it wherever and however you want and to modify it for your needs. I might improve it down the road but for now it’s mostly just a quick itch-scratching hack that does things the way I want it to.

Originally I was hoping to have a major performance optimization, in the form of having rewritten Pushl from thread-per-connection to async operation, but unfortunately I ran into a bunch of problems with it. Mostly that I was running into a “too many open files” error and I couldn’t figure out what was causing a descriptor leak. I have the work-in-progress branch online if anyone wants to take a look at it.

Anyway, the reason I went down this route is because I added WebSub subscriber support to my fork of Feed-On-Feeds, which makes it so that WebSub-enabled RSS and Atom feeds will push their updates to your reader instead of having to wait for a polling interval.

You can read more about some of my other thoughts on a blog entry that quickly devolves into a rant, if you’re so inclined.

Also, recently someone pointed out to me fed.brid.gy which makes it easy to turn a static site into an ActivityPub source. At some point I’ll experiment with setting up Publ for this; it looks like it’s just a matter of adding a couple of additional route rules to Publ, so that will probably go into an advanced configuration guide if I ever get around to making such a thing. (Or it could actually be added to Publ directly but there isn’t much of a reason for that, IMO.)

As far as I can tell, it works great, but I’m also not going to actually merge this to master or push it to production. Read on to see why!

When I started Publ, it was with the very specific goal of providing the publishing mechanism for a content store. Given content files and templates, format them into reasonable web-friendly HTML and Atom and whatever else the templates do, in a basically-stateless manner, and do a great job of providing high-quality image renditions and generated CSS and so on, while also supporting things like legacy URL mapping and automatic redirection and so on.

Basically, be like a static publishing system, but dynamic.

I want Publ to be the publishing and formatting part of an open, dynamic, independent web. I do truly believe in the IndieWeb mission, and absolutely want to see it become successful! However, I feel like Publ’s role is best served as one of many interlocking parts to make this work.

Supporting Webmention correctly means having the following:

• Automatically send out Webmention pings based on content changing
• Automatically receive and handle Webmention pings as they come in
• Provide structured metadata about the page content so that pings get credibly formatted on the receipient’s end

It also has a few other fun implications, like:

• Have a canonical URL for the content, generated by the publishing system itself, regardless of how the incoming traffic is routed to it
• Send updates as often as things change, but no more often than that
• Keep track of updates being sent (rather than just spamming everyone else with spurious updates)
• Generally, be stateful

And being stateful is a thing that flies in the face of the overall Publ design.

(So is the canonical URL thing, for reasons not worth getting into here, but briefly, things like staging/local/debug instances vs. the final production endpoint with any number of domain name normalizations and selections of protocol and so on. It’s messy and not something that’s easy to configure, and is impossible to detect in a way that’s consistent with what the outside world sees.)

The way I see it, this stuff is better left to external tools. Publ is for managing and formatting content. There are plenty of other tools that can be integrated into Publ templates to provide IndieWeb functionality in the way that seems most fitting to the person running the site.

For receiving WebMentions you can use webmention.io or brid.gy or this Disqus-like webmention endpoint thingamajig that’s currently in early alpha. For sending WebMentions you can use Telegraph or webmention-tools or ronkyuu or any number of other tools for sending mentions out. All of the configuration for the inbound webmentions, and for your semantic markup on your templates and so on, belongs in your templates, for what you think makes the most sense for your setup. There is no reason for it to actually be integrated into Publ.

A similar story exists for WebSub. Publ doesn’t know – nor should it have to know – what templates correspond to RSS or Atom feeds. It doesn’t know (or want to know) which of those feeds are affected by which category posts. It shouldn’t have to be configured with any mappings for which categories trigger which templates' updates, or which hub(s) those updates should be published to.

These are all better served by external tools that you, the publisher, configures to work the way you want them to.

So my proposal for supporting this stuff:

• Have a cron job that checks your feeds to see if they’ve updated
• When they do, run the appropriate tools to do the job of sending out notifications

To this end, I might start a separate project that lets you set up these mappings yourself, to be run as a sort of controller thing. It wouldn’t be at all integrated into Publ; indeed, it would work with any CMS that exports a feed or whatever. It would be configured with the following:

• One or more feeds to check
• Various plugins/actions to perform on those feeds

For example, a site’s master feed would have a WebMention task which would go through every entry in the feed, and does the following:

1. Fetch every referenced item
2. If the item has changed, parse it for all <a href> elements and send WebMentions

And they could also have a WebSub task which would simply:

1. See if the feed has changed
2. If so, send a WebSub notification to the hub of your choice

These tools can be simple and elegant and work universally across all of your sites, not just Publ ones! Got a Wordpress site? This will work with it! Got a Tumblr site? This will work with it! Got a YouTube channel? This will work with it.

And in the Publ case this also settles the canonical URL issue because it’s based on what the external tool sees, which is what the rest of the Internet should see – not based on whatever whimsical, fragile, momentary state might be the case due to it running on localhost:5000 or your private firewalled staging/preflight instance or whatever.

And this also gets along perfectly with another notion I had going into this – having services to syndicate content into a Publ site by reformatting an external Atom feed into an entry, for example. Want to automatically post all your Flickr items to a gallery on your Publ site? Use their Atom API. And the very same kind of simple tool can, for example, re-post your itch.io devlogs or your Tumblr reblogs or your YouTube videos or whatever.

I am entirely a fan of small tools doing simple things, and composing tools together. The world of CMSes already has too many Swiss army knives. Let’s keep our tools simple, doing a few things well, and have them work together with other tools.

I will absolutely have Publ support Webmention and WebSub and so on, in the best way possible – by doing nothing, and not getting in the way of other tools that can do it better.

Publ is a puzzle piece, not a complete solution.