Here is an implementation of a basic tag browser that allows users to filter content based on multiple criteria.

Note that you’ll also want to implement the crawler mitigations.

First, you’ll need to modify your content view with tag_filter='ALL'; for example, at the top of your template, have a line like:

{%- set view = view(tag_filter='ALL') -%}

Next, this block will give you a tag filter that shows the top 10 tags present in the current view in addition to any tags that are currently selected, with a toggle to show all tags that have more than one entry:

{%- if not user.is_bot and category.tags(recurse=view.spec.recurse) -%}
<div id="tags"><h3>Topic Tags</h3>

    {% macro tag_link(view) -%}
    {{view.link(template=template,all_tags=1 if request.args.all_tags else None)}}
    {%- endmacro %}

    <ul>
        {%- if view.tags -%}
        <li class="clear"><a href="{{tag_link(view(tag=None))}}">Clear filter</a></li>
        {% endif %}

        {% set tags = category.tags(tag=view.spec.tag if view.spec.tag else None,
            tag_filter='ALL',recurse=view.spec.recurse)
            | sort(attribute='count',reverse=True) %}
        {% for name,count in (tags if request.args.all_tags else tags[:10+view.tags|length]) %}
            {%- if count > 1 and
                (request.args.all_tags or loop.index < 10 or name in view.tags) -%}
                <li class="{{name in view.tags and 'selected' or ''}}"><a rel="nofollow" href="{{tag_link(view.tag_toggle(name))}}">{{name}}</a> ({{count}})</li>
            {%- endif -%}
        {% endfor %}

    {% if request.args.all_tags %}
        <li class="top-only"><a rel="nofollow" href="{{view.link(template=template)}}">Top tags only</a></li>
    {% elif tags|length > 10 %}
        <li class="all"><a rel="nofollow" href="{{view.link(template=template,all_tags=1)}}">Show all tags</a></li>
    {% endif %}
    </ul>
</div>
{% endif %}

This stylesheet snippet will add useful styling to the tag list, as well:

#tags ul {
    list-style-type: none;
}
#tags li {
    text-indent: 1em hanging;
}
#tags li:before {
    content: '☐\00a0';
}

#tags li.clear:before {
    content: '☒\00a0';
    font-weight: bold;
}

#tags li.selected:before {
    content: '☑︎\00a0';
}

#tags li.all, #tags li.top-only {
    font-style: italic;
}
#tags li.all:before, #tags li.top-only:before {
    content: '';
}

This functionality is particularly useful for implementing crawler mitigations, which have been quite effective with the current onslaught of badly-behaved crawlers.

I have also finally begun to collect some useful recipes.

Here is a template snippet for rendering out a calendar for a particular view.

{%- macro render_calendar(view, week_start=7) -%}

{%- if not view.is_current and view.newest -%}
{%- set month = view.newest.date.floor('month') -%}
{%- else -%}
{%- set month = arrow.now().floor('month') -%}
{%- endif -%}

{%- set content = view(date=month.format('YYYY-MM')) -%}
{%- set firstday,_ = month.span('week',week_start=week_start) -%}
{%- set _,lastday = month.ceil('month').span('week',week_start=week_start) -%}

<table class="calendar">
    <thead>
        <tr>{%- for day in arrow.Arrow.range('day',firstday,firstday.shift(days=6)) -%}
            <th>{{day.format('ddd')}}</th>
        {%- endfor -%}</tr>
    </thead>
    <tbody>

        {%- for week in arrow.Arrow.range('week',firstday,lastday) -%}
        <tr>
            {%- for day in arrow.Arrow.range('day',week,week.shift(days=6)) -%}
            {%- set items = content(date=day.format('YYYYMMDD')) -%}
            <td class="{{['entries' if items.entries else 'empty',
            'thismonth' if day.month==month.month else 'othermonth',
            'today' if day.date() == arrow.get().date() else 'future' if day.date() > arrow.get().date() else 'past']|join(' ')}}"><time datetime="{{day.format('YYYY-MM-DD')}}">{{day.format('D')}}</time>
                {%- if items.entries -%}
                <ul>
                    {%- for entry in items.entries -%}
                    <li class="{{entry.type}}"><a href="{{entry.link}}">{{entry.title}}</a></li>
                    {%- endfor -%}
                </ul>
                {%- endif -%}
            </td>
            {%- endfor -%}
        </tr>
        {%- endfor -%}
    </tbody>
</table>

{%- endmacro -%}

You can now use render_calendar to display a calendar that shows days with items on it, where the month will be based on the most recent visible item in the view.

The week_start parameter indicates which day of the week the week starts on (1 = Monday, 7 = Sunday).

Each cell in the table will have the following CSS classes:

• entries if there are items on that day
• empty for days with no items
• thismonth for days which are on the current month
• othermonth for days which come from neighboring months
• today for the current date
• future for days in the future
• past for days in the past

In this day and age it’s necessary to have mitigations in place to prevent badly-behaving web crawlers from taking down every single website. Both legitimate search bots and things like AI/LLM crawlers do a bunch of nasty tricks to try to extract as much detail as possible from a website, even when signals are present to indicate which pages are worth crawling.

This recipe is a starting point for implementing a simple “sentience check” into Publ websites, which has shown itself to be just as effective as more heavyweight options such as Anubis or Cloudflare’s “managed challenge” CAPTCHA. Setting it up is pretty simple:

1. Add the following functions to your app.py:

   @app.before_request
   def antiscraper():
       """ If something looks like a scraper, give it a sentience check """
       import arrow
       import flask
       import werkzeug.exceptions
   
       # Logged-in users have passed the test already
       if publ.user.get_active():
           return
   
       try:
           # Check to see if the session data matches the user's data
           if (flask.session['addr'] == flask.request.remote_addr and
               arrow.get(flask.session['sid']) > arrow.now().shift(days=-3) and
               flask.session['ua'] == flask.request.headers.get('User-Agent')):
               return
       except (KeyError, arrow.ParserError):
           pass
   
       # Send possible crawlers to the sentience test
       # Initial score: number of items in the GET arguments
       score = len(list(flask.request.args.items(True)))
   
       # add any other custom signals to the score
   
       if score > 2:
           raise werkzeug.exceptions.TooManyRequests("Sentience test")
   
       return
   
   @app.route('/_zuul', methods=['POST'])
   def gatekeeper():
       """ Sentience check callback """
       import arrow
       import flask
   
       redir = flask.request.form['redir']
       flask.session['sid'] = flask.request.form['sid']
       flask.session['addr'] = flask.request.remote_addr
       flask.session['ua'] = flask.request.headers.get('User-Agent')
       return flask.redirect(f'{redir}', code=303)
   

2. Add the following template as templates/429.html:

   <!DOCTYPE html>
   <html><head><title>Sentience test</title>
   <script>
   window.addEventListener("load", () => {
       document.forms['proxy'].submit();
   });
   </script>
   </head>
   <body>
   <h1>Sentience check</h1>
   
   <form method="POST" id='proxy' action="{{url_for('gatekeeper')}}">
       <input type="hidden" name="redir" value="{{request.full_path}}">
       <input type="hidden" name="sid" value="{{arrow.now()}}">
       <input type="submit" value="I'm actually here">
   </form>
   </body>
   </html>
   

Out of the box, this will present a sentience check to anyone who is exhibiting basic bad-crawler behavior, which will be skipped for anything that has a cookie indicating that the test has previously been passed. For folks running browsers with JavaScript the test should automatically pass, as well.

The test is very simple; it just indicates that the form has been submitted within the past three days and that the agent submitting the form still has the same IP address and browser user agent, as those values will be stable during a particular browsing session and tend to be randomized by the AI crawlers. Keep in mind that there may be some situations in which the IP address for a legitimate user is randomized on a per-request basis, though (such as certain VPN or caching proxy configurations, or particularly dysfunctional CGNAT deployments).

I’ve also finally documented how to provide your own token store for more exotic configurations, although that, much like Sam Reich, has been here the whole time.

In order to upgrade you’ll have to remove any duplicate UUIDs. To find entries with duplicate UUIDs you can run something like:

find . -name '*.md' -print0 | xargs -0 grep 'UUID' | cut -f2- -d: | sort | uniq -c | sort -n

which will show how many entries have each UUID. For every UUID that has a count greater than 1, you’ll have to do something like:

grep -rl bc3ced56-93c3-4252-aa85-48fb85b253de .

which will then list all of the entries with UUID bc3ced56-93c3-4252-aa85-48fb85b253de, and then you can decide which one(s) to remove the UUID from, and then reindex the site (with poetry flask publ reindex) to generate the new UUIDs.

This will help to clear up weird issues where items don’t appear as new in feeds, and also paves the way for some future functionality I’ve been wanting to implement which relies on the second U in UUID being true.

Note that since this change modifies the database, it will require a full site reindex. I recommend fixing any known UUID issues before upgrading Publ.

… And of course with a major rewrite that I decided to push out after midnight after a very long day of coding, of course it had some edge-case bugs, right on the edge cases I was trying to fix to begin with. Whoops. Now we’re on 0.8.3.

This is also the first release announcement since v0.7.40, so here’s what’s changed since then:

• Fixed a bug with render arguments not being passed in correctly for card images
• Improved the mechanism for adding fallback images to a card
• Support Unicode characters (and internal periods, as seen on this very entry!) in URLs and tags

Note that any affected URLs for existing entries may not change until the site has been fully reindexed. If this matters to you, run flask publ reindex -f.

Because of the rise in badly-behaved website scraping bots, it can be very helpful for site performance to have a fronting CDN, such as Cloudflare, in front of your origin server. But there are a few things you need to keep in mind.

Caching

When configuring caching, it is very important that you exclude the following paths from any aggressive caching behavior:

• /_cb/
• /_login and /_login/
• /_logout and /_logout/

If these paths get cached, it may lead to unpredictable behavior for user login and logout.

Image CDN

There should be no need for any additional fronting image CDN on Publ, although having it enabled shouldn’t hurt. However, because Publ manages its own cache lifetime and the CDN is not aware of this, some strange behaviors may occur in extreme edge cases.

Setting the CDN to aggressively prefetch images or to use status 103 “Early Hints,” if possible, is probably a good idea.