aiohttp one, which isn’t very friendly.Now it sends a reasonable user-agent by default, and this can be overridden by the --user-agent flag if you want to for your own analytics or whatever.
Oh, and I had quietly released 0.2.3 a few days ago; there were just some minor internal changes to logging and also declaring Pushl as beta, rather than alpha, software.
]]>I do wish I could figure out what is causing the hangs when they do happen though. Oh well. Some discussion of the issue below the cut.
So, there are two main tasks, process_feed and process_entry, which can both be spawned by the command line processor, and which can also spawn each other. (process_feed generally spawns process_entry as a matter of course, process_entry only spawns process_feed if -r is set.)
Both of these tasks will asynchronously fetch the data for the item itself, but then will gather a list of additional tasks to start in parallel, such as sending off WebSub/WebMention notifications or the aforementioned additional feed and entry processing tasks. And, because of the way asyncio works, the last thing each task does is wait for its pending tasks to complete.
The thing is, the only thing that ever hangs is that pending wait!
I’ve added a lot of logging to everything to see where every part of every process begins and ends, in a way that I can match things up in pairs, and every single individual task completes. But that await asyncio.wait(pending) will sometimes just wait forever. If I inspect the list of pending tasks when this does happen, every one is in the done state, so asyncio.wait should just be returning for them. But they aren’t.
It’s not even deterministic, which means that there’s probably something timing-related. Which would make me worry about there being a deadlock, but… there’s nowhere that a deadlock could sneak in, either. Any time a task is fired off it’s done as a new instance (except for the specific case of getting a webmention endpoint, which is cached using async_lru but doesn’t have any dependencies on anything that has a pending list, and isn’t a thing that’s hanging anyway), any duplicated work is discarded before any await statement (so there’s no way any cyclic dependencies are happening), all local file access is non-asynchronous, and like, when it does hang, the usual pattern is that there will be 2-3 process_feed tasks waiting on 6-7 process_entry tasks, which have all completed all of their async work but are waiting on their pending tasks.
I’m sure there’s just some dang typo somewhere that is causing something weird to happen, although pylint and flake8 haven’t found any of the usual telltale signs of that.
But of course, now that I’ve written a blog entry about trying to diagnose the problem, I can’t get the problem to recur, even on things that used to reproduce it 100%. WHATEVER.
]]>Anyway, there’s also seemingly a problem with how aiohttp manages its connection pool, at least on macOS, so I’ve disabled connection keep-alive by default. However, if you still want to use keep-alive, there’s now a --keepalive option to allow you to do that. I’m finding that it doesn’t really improve performance all that much anyway.
This is feeling beta-ready but I’ll give it a few days for other issues to shake out first.
]]>The main thing is now it’s using async I/O instead of thread-per-connection, so it’s way more efficient and also times out correctly.
And oh gosh, I had so many tiny but critical errors in the way caching was implemented – no wonder it kept on acting as if there was no cached state. Yeesh.
Anyway, I’ll let this run on my site for a few days and if I like what I see I’ll upgrade it to beta status on PyPI.
]]>See its README.md for the (incredibly rough) usage instructions.
Eventually I want to try to automatically convert templates from MT’s scripting language to Jinja-Publ templates, although there’s a bunch of stuff that’s going to be difficult to port across and a lot of stuff is just plain not feasible to even try, so don’t expect that to become a major thing any time soon.
]]>--version option on the command line argumentsAlso, some suggested usage ideas below the cut!
An installation guide is available in the project README, but the short version is to make sure you have Python 3 available and then run the following at a command prompt:
pip3 install pushl
which should do everything you need to install it. (On Linux or macOS may need to do sudo pip3 install pushl depending on how your system is set up.)
The main use for Pushl is to send Webmention and Pingbacks from any arbitrary blog to link targets, regardless of blogging platform (for example, using Jekyll, Movable Type, Pelican, or, of course, Publ). But it can be used for a lot more than that!
For example, the -e/--entry flag can be used to send webmentions from a specific page; for example:
pushl -e http://example.com/blog/page/12345
And if this page embeds feed discovery tags, you can combine that with -r to also recursively apply to its feeds; for example:
pushl -re http://forum.example.com/
This works especially well with forum software such as phpBB and XenForo, both of which support feed discovery. And this will help website publishers to know when their content is being discussed, with forum posts appearing as “pingbacks” on their site!
Of course, when using it with a forum or a sporadically-updating blog or whatever you’ll probably want it to be in a cron job. There’s more information about how to set that up in the project README.
]]>Anyway, it should just be a pip install --upgrade pushl (or pipenv update) away.
Well you’re in luck, I wrote a simple-ish script for that. (You’ll probably also want to see the accompanying stylesheet too.) And it doesn’t even require that you use Publ – it should work with any CMS, static or dynamic. The only requirement is that you use either webmention.io or something that has a similar enough retrieval API.
I wrote more about it on my blog, where you can also see it in use. For now, I’m just going to use the sample site repository to manage it (and issues against it).
It’s MIT-licensed, so feel free to use it wherever and however you want and to modify it for your needs. I might improve it down the road but for now it’s mostly just a quick itch-scratching hack that does things the way I want it to.
]]>