Here is an implementation of a basic tag browser that allows users to filter content based on multiple criteria.

Note that you’ll also want to implement the crawler mitigations.

First, you’ll need to modify your content view with tag_filter='ALL'; for example, at the top of your template, have a line like:

{%- set view = view(tag_filter='ALL') -%}

Next, this block will give you a tag filter that shows the top 10 tags present in the current view in addition to any tags that are currently selected, with a toggle to show all tags that have more than one entry:

{%- if not user.is_bot and category.tags(recurse=view.spec.recurse) -%}
<div id="tags"><h3>Topic Tags</h3>

    {% macro tag_link(view) -%}
    {{view.link(template=template,all_tags=1 if request.args.all_tags else None)}}
    {%- endmacro %}

    <ul>
        {%- if view.tags -%}
        <li class="clear"><a href="{{tag_link(view(tag=None))}}">Clear filter</a></li>
        {% endif %}

        {% set tags = category.tags(tag=view.spec.tag if view.spec.tag else None,
            tag_filter='ALL',recurse=view.spec.recurse)
            | sort(attribute='count',reverse=True) %}
        {% for name,count in (tags if request.args.all_tags else tags[:10+view.tags|length]) %}
            {%- if count > 1 and
                (request.args.all_tags or loop.index < 10 or name in view.tags) -%}
                <li class="{{name in view.tags and 'selected' or ''}}"><a rel="nofollow" href="{{tag_link(view.tag_toggle(name))}}">{{name}}</a> ({{count}})</li>
            {%- endif -%}
        {% endfor %}

    {% if request.args.all_tags %}
        <li class="top-only"><a rel="nofollow" href="{{view.link(template=template)}}">Top tags only</a></li>
    {% elif tags|length > 10 %}
        <li class="all"><a rel="nofollow" href="{{view.link(template=template,all_tags=1)}}">Show all tags</a></li>
    {% endif %}
    </ul>
</div>
{% endif %}

This stylesheet snippet will add useful styling to the tag list, as well:

#tags ul {
    list-style-type: none;
}
#tags li {
    text-indent: 1em hanging;
}
#tags li:before {
    content: '☐\00a0';
}

#tags li.clear:before {
    content: '☒\00a0';
    font-weight: bold;
}

#tags li.selected:before {
    content: '☑︎\00a0';
}

#tags li.all, #tags li.top-only {
    font-style: italic;
}
#tags li.all:before, #tags li.top-only:before {
    content: '';
}

Here is a template snippet for rendering out a calendar for a particular view.

{%- macro render_calendar(view, week_start=7) -%}

{%- if not view.is_current and view.newest -%}
{%- set month = view.newest.date.floor('month') -%}
{%- else -%}
{%- set month = arrow.now().floor('month') -%}
{%- endif -%}

{%- set content = view(date=month.format('YYYY-MM')) -%}
{%- set firstday,_ = month.span('week',week_start=week_start) -%}
{%- set _,lastday = month.ceil('month').span('week',week_start=week_start) -%}

<table class="calendar">
    <thead>
        <tr>{%- for day in arrow.Arrow.range('day',firstday,firstday.shift(days=6)) -%}
            <th>{{day.format('ddd')}}</th>
        {%- endfor -%}</tr>
    </thead>
    <tbody>

        {%- for week in arrow.Arrow.range('week',firstday,lastday) -%}
        <tr>
            {%- for day in arrow.Arrow.range('day',week,week.shift(days=6)) -%}
            {%- set items = content(date=day.format('YYYYMMDD')) -%}
            <td class="{{['entries' if items.entries else 'empty',
            'thismonth' if day.month==month.month else 'othermonth',
            'today' if day.date() == arrow.get().date() else 'future' if day.date() > arrow.get().date() else 'past']|join(' ')}}"><time datetime="{{day.format('YYYY-MM-DD')}}">{{day.format('D')}}</time>
                {%- if items.entries -%}
                <ul>
                    {%- for entry in items.entries -%}
                    <li class="{{entry.type}}"><a href="{{entry.link}}">{{entry.title}}</a></li>
                    {%- endfor -%}
                </ul>
                {%- endif -%}
            </td>
            {%- endfor -%}
        </tr>
        {%- endfor -%}
    </tbody>
</table>

{%- endmacro -%}

You can now use render_calendar to display a calendar that shows days with items on it, where the month will be based on the most recent visible item in the view.

The week_start parameter indicates which day of the week the week starts on (1 = Monday, 7 = Sunday).

Each cell in the table will have the following CSS classes:

• entries if there are items on that day
• empty for days with no items
• thismonth for days which are on the current month
• othermonth for days which come from neighboring months
• today for the current date
• future for days in the future
• past for days in the past

In this day and age it’s necessary to have mitigations in place to prevent badly-behaving web crawlers from taking down every single website. Both legitimate search bots and things like AI/LLM crawlers do a bunch of nasty tricks to try to extract as much detail as possible from a website, even when signals are present to indicate which pages are worth crawling.

This recipe is a starting point for implementing a simple “sentience check” into Publ websites, which has shown itself to be just as effective as more heavyweight options such as Anubis or Cloudflare’s “managed challenge” CAPTCHA. Setting it up is pretty simple:

1. Add the following functions to your app.py:

   import arrow
   import flask
   import werkzeug.exceptions
   
   def keymaster(sid):
       """ Generates a salted token for the browser """
       import hashlib
   
       parts = [
           str(sid),
           flask.request.remote_addr,
           flask.request.headers.get('User-Agent')
       ]
       token = hashlib.md5('|'.join(parts).encode('utf-8'))
       return token.digest()
   
   @app.before_request
   def antiscraper():
       """ Dissuade aggressive bots from pummeling the site """
   
       # Logged-in users have passed the test already
       if publ.user.get_active():
           return
   
       # Send possible crawlers to the login page
       score = len(list(flask.request.args.items(True)))
       if score > 1:
           # Check for an existing sentience token
           try:
               sid, token = flask.session['vinz']
               if (arrow.now().shift(hours=-1) < arrow.get(float(sid)) < arrow.now() and
                       keymaster(sid) == token):
                   return
           except (KeyError, ValueError, arrow.ParserError):
               pass
   
           raise werkzeug.exceptions.TooManyRequests("Sentience test")
   
       return
   
   @app.route('/_zuul', methods=['POST'])
   def gatekeeper():
       """ Check the test response and set the salted token upon passing """
       try:
           sid = float(flask.request.form['sid'])
           if arrow.get(sid) > arrow.now():
               # Someone's trying to set a token that'll last longer
               raise werkzeug.exceptions.BadRequest("Hello time traveler")
           if arrow.get(sid) < arrow.now().shift(minutes=-5):
               # Someone took a while to respond to the form
               raise werkzeug.exceptions.TooManyRequests("Try again")
       except ValueError:
           raise werkzeug.exceptions.BadRequest("Nice try")
   
       redir = flask.request.form['redir']
       flask.session['vinz'] = sid, keymaster(sid)
       return flask.redirect(f'{redir}', code=303)
   

2. Add the following template as templates/429.html:

   <!DOCTYPE html>
   <html><head><title>Sentience test</title>
   <script>
   window.addEventListener("load", () => {
       document.forms['proxy'].submit();
   });
   </script>
   </head>
   <body>
   <h1>Sentience check</h1>
   
   <form method="POST" id='proxy' action="{{url_for('gatekeeper')}}">
       <input type="hidden" name="redir" value="{{request.full_path}}">
       <input type="hidden" name="sid" value="{{arrow.now().format('X')}}">
       <input type="submit" value="I'm actually here">
   </form>
   </body>
   </html>
   

Out of the box, this will present a sentience check to anyone who is exhibiting basic bad-crawler behavior, which will be skipped for anything that has a cookie indicating that the test has previously been passed. For folks running browsers with JavaScript the test should automatically pass, as well.

The test is very simple; it just indicates that the form has been submitted within the past hour and that the agent submitting the form still has the same IP address and browser user agent, as those values will be stable during a particular browsing session and tend to be randomized by the AI crawlers. Keep in mind that there may be some situations in which the IP address for a legitimate user is randomized on a per-request basis, though (such as certain VPN or caching proxy configurations, or particularly dysfunctional CGNAT deployments).

Because of the rise in badly-behaved website scraping bots, it can be very helpful for site performance to have a fronting CDN, such as Cloudflare, in front of your origin server. But there are a few things you need to keep in mind.

Caching

When configuring caching, it is very important that you exclude the following paths from any aggressive caching behavior:

• /_cb/
• /_login and /_login/
• /_logout and /_logout/

If these paths get cached, it may lead to unpredictable behavior for user login and logout.

Image CDN

There should be no need for any additional fronting image CDN on Publ, although having it enabled shouldn’t hurt. However, because Publ manages its own cache lifetime and the CDN is not aware of this, some strange behaviors may occur in extreme edge cases.

Setting the CDN to aggressively prefetch images or to use status 103 “Early Hints,” if possible, is probably a good idea.

For smaller Publ sites, the suggested configuration is to use SQLite for the index database, as it is low-maintenance and performs very well due to being in-process and taking advantage of the operating system’s disk cache.

However, sometimes it’s helpful to use a larger-scale database system for the deployment, primarily to gain access to finer-grained locking. This is especially useful in situations where there are many thousands of entries and a desire to keep the site running at full capacity during a reindex.

Here is a configuration snippet that allows you to use MySQL/MariaDB on your Publ site:

# ...

if 'DATABASE_URL' in os.environ:
    import urllib.parse
    parsed = urllib.parse.urlparse(os.environ['DATABASE_URL'])
    user = re.match(r'(.*):(.*)@(.*)', parsed.netloc)
    db_config = {
            'provider': parsed.scheme,
            'user': user.group(1),
            'password': user.group(2),
            'host': user.group(3),
            'database': parsed.path[1:],
            # charset and collation must be specified, as the MySQL defaults do
            # not properly support emoji and other 4-byte characters
            'charset': 'utf8mb4',
            'collation': 'utf8mb4_bin',
        }
else:
    db_config = {'provider': 'sqlite', 'filename': os.path.join(APP_PATH, 'index.db')}

config = {
    'database_config': db_config,
    # ...

Then when running the site, set an environment variable such as:

DATABASE_URL='mysql://username:password@server/dbname'

If migrating from SQLite to MySQL, it is a good idea to create the index first before flipping the configuration:

DATABASE_URL='mysql://username:password@server/dbname' poetry run flask publ reindex

An alternate way to store the database configuration is to put it into a local db_config.py file, like so:

db_config = {
    'provider': 'mysql',
    'user': 'db username',
    'password': 'db password',
    'host': 'localhost',
    'database': 'my_site',
    'charset': 'utf8mb4',
    'collation': 'utf8mb4_bin'
}

# ...

try:
    from .db_config import db_config
except ImportError:
    db_config = {'provider': 'sqlite', 'filename': os.path.join(APP_PATH, 'index.db')}

config = {
    'databse_config': db_config,
    # ...
}

but as always it is important to ensure the security of this file; environment-based configuration is traditionally considered to be much easier to secure and prevent mishaps such as accidentally checking it into source control.

The administration dashboard is available to anyone who is in the administration group, as configured by the user configuration and application configuration. It is accessed by going to the /_admin path on the website; on this site you can log in as test:admin for a demonstration.

For this to be available, Publ needs to be configured with authentication, and there must be at least one user in the administrators group.

The dashboard provides information about the operation of the site; of particular interest, it shows the recent authentication activity, such as the recent users, and both authorized and unauthorized accesses to protected entries.

Any time a user identity is presented, you can copy the link itself to add to your user configuration, or you can visit the user’s profile link (designated with an ℹ︎ symbol) to view their user profile.

The user object provides the following:

• identity: The identity URL of the user

• humanize: A humanized version of the identity URL

• name: The display name of the user

• profile: The user profile; see the Authl documentation for the relevant keys.

  Note that this will not necessarily be available, depending on how and when the user logged in. For example, if the database has been fully reset (due to e.g. a Publ upgrade or a server migration) since they last logged in, or if the user logged in from a different instance on a load-balanced configuration using a per-instance database, the profile will likely not be present.

• groups: A list of the groups they belong to (not including the user’s identity group)

• auth_groups: The full list of matching authentication groups (including the user’s own identity group)

• is_admin: True if the user is a member of the administrative group

• auth_type: How the user’s authentication was obtained; possible values:

  • 'session': Normal login flow/session cookie
  • 'token': Bearer token

• scope: The user’s permission scopes, if applicable (typically if the login was via a bearer token)

• last_login: The last time the user logged in

  This may not be available, per the same rules as user.profile

• last_seen: The last time the user was active on the site

• is_bot: Whether the page is currently being viewed by a declared bot

• token(lifetime,scope=None): A function to generate an HTTP Bearer token for preauthorization.

  Arguments:

  • lifetime: How long the token is good for, in seconds
  • scope: The access scopes that should be associated with the token

  See using bearer tokens for more information.

For the most part you should only use user.name to address the user, and possibly use user.groups to check for particular group membership if that is something you want to show them. For example, if you use user groups as a means of managing memberships or rewards or the like, you could do something like:

Hello, {{user.name}}.
{% if 'current_members' in user.groups %}
    Your membership is current!
{% else %}
    You aren't currently a member. Please consider <a href="/subscribe">becoming one</a>.
{% endif %}

When extending Publ using additional Python functions, the current user can be retrieved with:

import publ.user
user = publ.user.get_active()

In all contexts you can check the truthiness of the user object to see if someone is logged in; bots do not count as logged-in users. For example, in a template:

{% if user %}
    <p>Hello user {{user.name}}!</p>
{% elif user.is_bot %}
    <p>Hello bot {{user.name}}!</p>
{% else %}
    <P>Howdy, stranger!</p>
{% endif %}

Publ extends the default flask command with some additional commands, which can be used for various useful things. In addition to this page, you can get detailed help from the command line with:

flask publ [command] --help

from within your Publ virtual environment (e.g. poetry run flask publ --help or env/bin/flask publ --help).

For usage of the flask command itself, see the Flask CLI documentation.

reindex

This command lets you force a database reindex, and will wait until the index finishes. This is useful for deployment scripts and the like.

It takes two arguments:

• --fresh/-f: Start with a fresh index (i.e. remove all the cached data); this is useful if something weird has happened and you want a fresh start (if something weird has happened, please open an issue!)
• --quiet/-q: Don’t show the progress indicator

token

This command can be used to generate an IndieAuth token for scripting purposes, such as for use with Pushl, or for testing authentication automatically.

For example:

TOKEN=$(flask publ token https://example.com)
curl -H "Authorization: Bearer $TOKEN" https://example.com/feed

will fetch the https://example.com/feed page with a token identifying the user as https://example.com (as generated by your Publ site).

Note that the session key will need to match between wherever you’re running this and the actual site. If they do not match for some reason (for example, because you’re running in a different configuration) this token will not be valid.

This takes the following arguments:

• --scope/-s: Generate the token with the specified scope (defaults to none)
• --lifetime/-l: How long the token should be valid for, in seconds (default: 3600)

Note that token scopes are not currently used by Publ itself; this is provided largely to make it easier to extend Publ via the Python API, such as implementing a mechanism to automatically post content from external sources.

normalize

This command allows you to bulk-rename entries within one or more categories with a templatized filename. By default it gives things a name of {date} {sid} {title}, so e.g. names like:

20190204 3 Welcome to my new blog.md
20200101 1942 Happy new year!.md
20200704 DELETED Ugh never mind.md

Example usage:

### Give 'blog' entries a name like '20210101 Hello everyone.md'
flask publ normalize -af "{date} {title}" blog

### Apply the default name to published entries in 'articles' and 'recipes'
flask publ normalize articles recipes

### Apply hyper-specific naming to every entry in the site
flask publ normalize -rf "{date}-{time}-{id}-{slug}"

It accepts the following arguments:

• -r/--recurse: Include subdirectories
• -a/--all: Apply to all entries, not just reachable ones (published, scheduled, hidden)
• -n/--dry-run: Show, but don’t apply, changes
• -f/--format TEXT: Filename format to use
• -v/--verbose: Show detailed actions

And the following format tokens can be used in the string provided to -f/--format:

• {date}: The entry’s publish date, in YYYYMMDD format
• {time}: The entry’s publish time, in HHMMSS format
• {id}: The entry’s ID
• {status}: The entry’s publish status
• {sid}: If the entry is reachable, the ID, otherwise the status
• {title}: The entry’s title, normalized to filename-safe characters
• {slug}: The entry’s slug text
• {type}: The entry’s type

Note that entries in DRAFT status always get an {id} of DRAFT, even if an ID has been assigned.

When templates start to get particularly complex, Publ can start to slow down; this is one of the tradeoffs when using a dynamic site generator, rather than a static one. Fortunately, it is fairly straightforward to configure Publ with caching; here are some pointers on how.

Basics

Entire textbooks can (and have been) written on caching and their optimum usage. This is not intended to be an entire textbook, but rather a set of brief-ish guidelines for how to configure Publ’s caching mechanism.

What is a cache?

At its most basic and general, a cache is a temporary storage area where intermediate results get stored for a while.

Fundamentally, there are two algorithms involved in a cache: key generation, and result generation.

The key is basically a filename that tells the cache where to look for a result, and this is computed based on the unique aspects that are involved in generating the result. In the case of Publ’s page cache, the key is based on what’s being looked at (the selected category, template, entry, and URL parameters), how it’s being looked at (the URL), and who’s looking at it (the current user, if any).

The result is the result of performing the operation that’s being cached. In this case, this is the rendered content of the template.

The goal in any cache is to correctly produce the right result while reusing those results as much as possible, while taking less time on average than what it would take to simply compute the result each time.

Performance tradeoffs

When setting up a cache, you need to consider what the right balance is between memory usage, performance, and immediacy. These three things are at odds with one another and there is fundamentally no perfect solution that maximizes all three.

Generally-speaking there are two statistics we are looking for in trying to optimize a cache:

• Hit rate: How often a page render can be reused; we want this to be as high as possible
• Staleness: How out-of-date a page render can be; we want this to be as low as tolerable

There is a third statistic which also matters, namely the active cache size (also called the “hot size,” among other things) – this is simply the total amount of storage used by page renders.

To achieve these goals, there are two “knobs” of particular interest: maximum cache size, and expiration time.

Increasing the expiration time will increase the hit rate, while also increasing the staleness. To understand why, if a page is only accessed once every 15 minutes, and the expiration time is only 5 minutes, then the cached result will be expired before the page is accessed again, making the cache useless. But, if the expiration time is one hour, then if a page changes at all, the update won’t appear until an hour after it was rendered.

The other knob, maximum cache size, is simply a measure of how much can be kept in the cache. As long as the expiration time is finite, the cache will never grow to be infinitely large, but you might still want to put a limit on how large it can grow. Raising the cache size won’t necessarily increase the hit rate, but decreasing the cache size to be less than the active cache size will decrease it.¹

A note on staleness

To mitigate the staleness problem, Publ uses aspects of when content was last edited as part of its caching mechanism, so if content changes that might affect a page, all previous versions of that page will be discarded.² So, while many caching systems recommend only setting the expiration time to the maximum amount of time you can tolerate a change not being available, Publ does not have this issue.

However, some things that Publ can do will still become stale. For example, things that are essentially non-deterministic (such as things based on random numbers) or based on the current time (such as entry.date.humanize()) will still be subject to becoming stale. So if you use any of these things in your templates, you’ll want to set your expiration time accordingly. Some guidelines:

• If you use entry.date.humanize(), an expiration time of around 1 hour is probably fine (although this will make a recent entry’s relative time appear to be wrong for its first hour of existence or so)
• If you use random number generation for whatever reason (such as displaying a random image on a particular page), the expiration time should be for however long you are willing to tolerate the number staying the same
• If you want to display the current time on the page for some reason³, the expiration time should be for the most “slow” you’re willing to have the clock be.

Also, if you plan on using any sort of monitoring service (munin, Pingdom, etc.) to measure page load time over time (to get an ongoing performance indicator), it’s a good idea to set your expiration time to a nearby prime number; for example, if you want your expiration time to be around an hour, try values of 3593 or 3659 or the like. This helps to prevent sampling bias and gives a much better overall average.

Configuration

Publ uses flask-caching for its caching layer. The short version of its documentation is it provides caching backends for a number of common caching servers (Redis and a few variants of MemcacheD), as well as file-based and in-process caching.

If you have Redis or MemcacheD available, definitely use those; they have the advantage of persisting beyond your application’s run time, and going into a common shared memory pool for all sites that use them, which can be way more memory-efficient than the in-process cache. They also both support distributed caches.

If you are self-hosting and trying to decide which cache server to go with, I suggest MemcacheD, as it’s by far the easiest to install and configure, and most distributions pre-configure it with sensible defaults.

If you don’t have any of those (for example, you’re on shared hosting that doesn’t allow you to run your own services), your best choices⁴ are FileSystemCache (file-based) and SimpleCache (in-process), which each have strengths and weaknesses. Both of them suffer from not being configurable based on actual storage size; they can only track the number of items in the hot cache. SimpleCache is faster, but also makes your process take more memory – a big concern if you’re on a shared system where memory usage is limited. FileSystemCache is generally friendlier and fast enough (and also will take advantage of the operating system’s file cache which makes it kinda-sorta similar to using a caching server), but its performance disadvantage can end up slowing the site down in many cases. But if you can look at the state of the filesystem, FileSystemCache gives you better visibility into what’s happening with the cache, which can give you better ideas for how to tune it.

In general, I’d recommend this priority order:

1. MemcacheD or Redis
2. FileSystemCache (if you have lots of storage available)
3. SimpleCache (if you have lots of RAM available)

Regardless of caching backend, in your application’s cache configuration you should configure the following:

• CACHE_KEY_PREFIX: Set this to some random but fixed string (this helps to prevent certain kinds of attacks that are possible if you share the cache with someone else)
• CACHE_DEFAULT_TIMEOUT: Set this to your expiration time, as discussed above

Some backend-specific notes are below.

MemcacheD

In order to use MemcacheD you’ll also need to add a MemcacheD client to your Python environment. I generally use python-memcached as it’s easier to install (especially in cross-platform scenarios), but libmc and pylibmc both have some advantages in performance-critical and large-scale scenarios, being slightly faster as well as supporting consistent hashing.

If you’re on Google AppEngine, a suitable client is already provided to you and there is no reason to add your own.

SimpleCache

Set the value of CACHE_THRESHOLD to limit the number of items that can be kept in the cache. As noted above, there is no way to limit the size directly, so if you have a specific allocation limit in mind, try to figure out what the average size of a page is (in bytes), and divide your allocation limit by that. That is the value you should use.

FileSystemCache

Similarly to SimpleCache, this uses CACHE_THRESHOLD to limit the number of items in the cache, rather than the total storage size. Since filesystem space is usually much more plentiful than RAM, there’s no reason to be particularly miserly with this. Unless you start to run out of storage space, anyway

You should also make sure that the CACHE_DIR is set to a directory that you have write access to, and you should set CACHE_OPTIONS to {'mode': 0o600} (the default).