trashpanda.cc

Deploying an Eleventy site to render.com

2024-09-03T00:00:00Z

tl;dr If you try to deploy the 'hello, world!' Eleventy site to Render, it will fail with a heifsave: Unsupported compression (via Template render error) error. To fix this, you need to remove the avif image type from .eleventy.config.images.js

slightly longer; might read Handling AVIF image types requires HEIC libraries to be installed on the system that builds the eleventy site, and render.com doesn't have this. Hence the error, and hence the need to remove this as a valid image type in the file that controls image processing.

even longer; not strictly relevant While render.com is a nice brand name, it doesn't make for very good search engine optimisation because the results are inevitably polluted by references to 'render' as in the verb 'to convert something'. It doesn't make finding problem resolutions any easier...k

How to manage a statically-generated website from iOS devices

2024-08-21T00:00:00Z

tl;dr with the aid of two apps, you can create and manage content for websites with git-based static site publishing workflows like Hugo, Eleventy and others.

Why is this worth doing?

If you've outgrown the blog-as-a-service platforms like Wordpress or Typepad, running your own site often involves using tools like Hugo or Eleventy.

They're basically static site generators - create content in text files, use templates to define layouts, run the content through a processing engine, and then publish the resulting HTML and assets to a web server. They're fast, powerful, flexible and decouple the content management from the hosting.

Many of this type of tools can also take advantage of git-based deployment - you configure something like Github Actions to react to pushes to the site's repository by building the content and transferring it over to the hosting platform. Some hosting services like Render will do the same thing in the opposite direction - they watch the respository for changes, and fetch and build the updates when something pushed.

Regardless of what tool, source code respository and hosting platforms are involved, the content creation process is usually similar. You edit the source files in some text editor, then commit the changes using the command line or a Git client. That's straight-forward enough on a desktop system, but tends not to play nicely with mobile devices.

I spend much more time using my mobile devices than I do sat in front of a Macbook, so I wanted a way of being able to CRUD content on my sites with the tools that I've got in my hand. The good news is that it's a) possible and b) very straight-forward, albeit at the cost of a couple of (very reasonably priced imho) apps.

The process also doesn't need any changes to either the publishing workflows or the hosting platform.

How it works

For the purposes of this explanation, I've got the following moving parts:

Hugo as my static site generator
Github as the source code repository
Render as my hosting platform

The desktop workflow goes:

Create a new post in the local repo copy of the site's source code using a text editor
Commit the changes to Github
Render watches the repository
When it detects changes, it checks out the updated source, builds the static site content, and deploys to its content distribution network automatically.

This process is going to reproduce that workflow, but with mobile tools:

A local copy of the site's source code will live on an iPad (you can subsitute iPhone for iPad as needed, because the process is exactly the same)
A text editor app will be used to create and/or update the site source
A Git client app will handle pushing and pulling to the Github repo
All other components and workflows will remain exactly the same and unchanged.

For this you'll need an iOS Git client (I'm going to use Working Copy) and an iOS text editor that can open and save files in Markdown format (I'm going to use Textastic). Both of these are paid apps with free trials - the total outlay will be about 35€, but to me that's a reasonable price for the flexibility of being able to manage my site on the go.

Setup

Setup is split into two parts - the Git component, and the text editor

Setting up Working Copy as the Git client

Install Working Copy from the App Store
Clone the site's respository from Github
Enable Working Copy as a file source using these instructions

Open the local files in Textastic

Install Textastic from the App Store
Add the repo to the sidebar
1. In the Textastic sidebar, tap the Add external folder option
2. Tap on the Working Copy entry in the Locations section of the sidebar
3. Open the folder for the repo
4. Navigate down to the content files (e.g. Hugo's content files normally live in ./content/posts )
5. Tap Open to add the folder to the Textastic side bar

Operation

Create a new post file in Textastic (I've found the easiest way to do this is long-press on an existing post file, tap the Duplicate option to clone it, then edit the new file to overwrite the existing content)
Switch to Working Copy
Create a new commit
Push the commit to the remote repo
Wait for the changes to be built and published.

Conclusion

For a total outlay of around 40€, it's possible get the best of both worlds - the utility of a mobile device that you've always got to hand; and the power and flexibility of a modern static site generator. It feels like a step closer to a world where table and laptops are functionally-equivalent - we're not there quite yet, but being able to manage sites without being dependent on "monolithic" client apps is definitely progress.

Keeping email uncluttered with a "waiting for" folder

2024-08-21T00:00:00Z

About the single easiest thing you can do to keep on top of an email inbox is not to use it for long term storage.

The simplistic way of achieving this is to delete every message once it's been read. But the problem with this approach is that not every message can be deleted once it's been read. Suppose it's details of an order that you'll need to follow up on, or the contact details for a meeting that's going to take place next week? You'll need to keep these, at least for a while, but ideally you don't want them hanging around cluttering your inbox.

The solution is very straight-forward.

Create a new folder called "waiting-for", and pin it st the top of the folders list
When an email comes in, read it.
If it's not needed any more, delete it.
If it's relevant to something that you're waiting for, drag it into the "waiting for" folder
When the order arrives, or after the meeting, or whatever - delete the relevant message from the "waiting for" folder
Periodically scan through the "waiting for" folder for anything that's late, and delete anything that's aged out

My information management process

2024-08-19T12:00:00Z

Reading about other people's tools and systems is always interesting, so I figured I'd sit down and record what I've come up with.

It's a process that's evolved over the years to try to capture and manage information - calling it an "information management system" would be grandiose and pretentious, but this workflow is what (currently)works for me.

On a daily basis I read a lot of things that could be useful at some point in the future, but unless I explicity try and store them somehow, they get lost. Online search is becoming ever-less useful in the age of AI slop, and a lot of online content is only fleeting. And that's after I've allowed for my imperfect memory and recall.

The process has two basic components:

somewhere to store text-based information in a way that's as flexible as possible
tags and search to (hopefully) retrieve info when needed.

I've tried to end up with something as low-friction as possible, on the basis that if it's easy to use and as far as possible always within reach, actually using it will become an embedded habit. That helps when trying to capture "fleeting thoughts" - flashes of what passed for insight in the shower, that kind of thing.

Principles

From what I've evolved, I can reverse-engineer some "principles" which makes everything sound way more calculated and sophisticated than it actually is.

Don’t try to remember anything

This is something that I picked up decades ago from Dave Allen's "Getting Things Done" book. (My) human memory is imperfect, so don't rely on it. If there's a chance that a piece of information might be wanted again in the future, throw it into the system as soon as you find it, and rely on the system to surface it again. This approach works equally-well for both information and future tasks

Keep the tools at hand

If the tools aren't immediately to hand, you're back on relying on human memory in order to remember to get the info into the system. Therefore, put the tools where you can reach them - which in my case means a iPhone, an iPad and a Macbook in that order.

Keep the friction minimal

If the ingestion process is slow and painful, you'll end up not bothering. Ideally, a single click or tap should be enough. I'm not quite there with my processes, but I'm close enough for this to take just a few seconds in the moment - which is fast enough.

Good enough is better than perfect

There's no such thing as an ideal system, so don't bother trying to shoot for perfection - instead, aim for the sweet spot between "minimal friction" and "workable solution". That helps to resist the temptation to replace using the system with tweaking the system.

Allow it to evolve...

See above - the available tooling evolves over time, and so do the circumstances your're operating in. The occasional tweak is needed to keep the processes relevant, so give yourself permission to do that.

...but try to stick with some consistency

The flip side of evolution is constant tinkering, and ideally you want a system that is familiar enough to operate with muscle memory (see the point about keeping the friction minimal). Chopping and changing tooling every 10 minutes will get in the way of that, so resiste the new shiny for shiny's sake as much as possible.

Where the information comes from

With some principles as a background, where does the information actually come from? In my case, it's overwhelmingly digital - in rough descending order of frequency, the sources are:

web
email newsletters (which tend to resolve to the web)
RSS feeds (which are shortcuts to the web)
Posts on services like Mastodon and LinkedIn (which are often shortcuts as well)
Kindle books

The emphasis on web-based sources that means that something web-centric is going to cover at least 80% of my needs. That has two important benefits - the solution is roughly one-size-fits-all, and I've no need for more complex workflows like forwarding email and so on.

Tools

These are the building blocks:

Text notes

I create text notes using the Bear app. This checks off multiple requirements:

the content is text-based and stored in Markdown
Bear is relatively open in terms of format, so I can get the information in and out easily
it's cross-platform, in the sense that it works on phone, iPad and Mac
it's exceptionally well-designed, so actually using it is pleasurable
it plays nicely with the platform, with first-class support for share sheets and x-url linking

Handling web content

I use Omnivore for summarising long-form web content. This leverages four Omnivore features - the one-click "save for later", the format-stripping that converts the page into text-only form, highlighting of chunks within the text, and the ability to export those highlights back out to Bear.

I don't use Omnivore as a bookmarks manager, although it could do this. Once I've read a page, it gest archived. Bookmarks get stored as notes within Bear, and I've got an item on my to-do list to figure out what better/more flexible approach could be. For now, though, it's good enough.

Handling RSS feeds

I use RSS feeds purely as a way of scanning through a large number of sites very rapidly. NetNewsWire runs on all three devices, and I open the posts that look interesting in Safari before using Bear or Omnivore to deal with them.

Kindle

Ebooks are a mixed blessing - the idea of being able to carry around an effectively infinite number of books on a device weighing grams is the future already being here; but Kindles also facilitates building a backlog of hundreds of books because it's just too easy to click on another one.

I'm also not super-happy about being locked into a specific vendor ecosystem, and some parts of the experience are clunky. But a Kindle is good enough, even if deep-down I'd prefer the tactile experience of dragging a highlighter across a paper page.

Dealing with content

My basic process is:

skim the content in whatever inbox it's landed
open it in Safari if it looks interesting
using the Safari share sheet, add a note into Bear
stick hashtags in the note before saving

If the content is long-form and looks worth reading in more detail, I'll save it into Omnivore. Periodically I'll trawl through Omnivore to see what's there, and use the highlighting function to mark the intersesting parts. Then the "export highlights" function creates a blob of text that can be pushed into Bear via the share sheet, together with a link to the original and any hashtags that I add.

A combination of hashtags and search within Bear is how I find things again. I'm pretty liberal with tagging, on the basis that I'm trying to aid keyword searches rather than build an alternative to the Dewey Decimal system.

I do use a few specific tags, though - #ideas and #sparkfile for "interesting things that might turn into a future side project if only I had the time", and specific tags for things that I've got going on at the moment (#moving is an example for collecting everything relating to a pending house move).

If there's something specific that I want to deal with and it's important enough to become an item on a to-do list, then I'll use the share sheet within Bear to create a to-do item in the built-in iOS reminders app. That comes along with an x-url link to the specific note within Bear.

Some of the things I don’t do

There are a few things that I just don't bother with.

any kind of AI Summarisation is supposed to be one of AI's killer features, but that feels largely pointless. The process of scanning through an article and making highlights of the important bits is part of the comprehension process to me, so outsourcing that wouldn't win me back any time.
using dicatation or voice notes to create content I'd feel a bit self-conscious doing that in the firstplace, and I'd need to be some kind of voice-to-text service in the loop. Maybe it would save time, but it's not something I've bothered with (yet).
using an external bookmarking service I have done in the past, but it feels a bit like opening myself up to ransom if I end up relying on someone else's goodwill or product roadmap.
expect this process to last As the workflow stands it's evolved over years, so there's no reason to expect that I've got it right permanentily this time.

Tool improvements

There's definite room for improvement - on the to-do list is to have a go at building my own bookmarking persistence in a more database-y way, for example. The Bear share sheet functionality could benefit from some way of searching or navigating within notes. Reading content might be a bit easier on a larger iPhone, traded off against the convenience of something that just about fits in a pocket.

My dream is some way of seamlessly connecting digital workflows and content with an analogue paper notebook, but that feels a bit like nuclear fusion - always due at some intederminate point in the future.

Overall

Is this perfect? No. Would this work for anyone else? Probably not, everybody's requirements are slightly different and the point about a workflow that sticks is that it has to fit with the needs of the individual.

Half-Assing a Web Game with ChatGPT and Friends

2024-06-20T00:00:00Z

I’ve been playing around with AI tools to help create software for a while, but the upcoming UK election was the kick-in-the-behind that I needed to try something a bit more serious.

I had an idea for a silly election night distraction - a sort of bingo-style web game that you could use to track the unfolding catastrophe for the ruling Conservative Party whilst knocking back shots as prominent politicians were cast into the political oblivion.

My front end web development skills are not sufficiently up-to-date to build that in the limited time available, so I figured I’d try to achieve that by leaning on ChatGPT, Perplexity and Mistral to help me out.

It worked, as does the site (after a fashion). And the experience was a fascinating one.

There are definitely benefits in being able to get LLMs to help out:

I was way quicker than I would have been, had I only had documentation and tutorials to call on
I could get going with a surface understanding of what I was trying to achieve, once I’d broken the problem down into the main building blocks
In the end, I delivered a working solution before I lost patience (this isn’t specifically LLM-related, but there is definitely a mean-time-to-giving-up with side projects like this if you can’t get results within a certain timeframe.

But there are also downsides:

I ended up down several dead ends due to the LLM hallucinating - the sheer confidence with which an LLM delivers an answer makes it very easy to blindly cut-and-paste, and it’s not always obvious where the model lost the plot
My awareness of hallucination potential meant it wasn’t always obvious where the LLM was right and the problem was that I’d introduced the bug
Using a single LLM didn’t really work, because of the dead-end problem - what worked around this was having several LLMs on the go simultaneously. As they didn’t hallucinate consistently, asking the same question to another model was generally the route out of the dead-end.

In summary, then:

I got something built quicker
The quality is what you’d expect from an enthusiastic amateur - it’s a long way from robust, tested, well-architected production code

That last point is the main problem.

LLMs are the junk food of the software industry. We rely on them because they’re an initially satisfying quick fix, and just like junk food, you’ll suffer long-term consequences if you rely on them too much.

LLM-generated code is the result of feeding together different solutions to the same problem through a statistical mincer, and the results are not going to be consistent in terms of the small details that can make the difference between a robust code base and shoddy one.

Aspects like naming conventions, and error handling patterns, and a myriad of other seemingly-trivial factors: they might not make the different between code that works or not, but they WILL make the challenge of maintaining and extending the code much much harder.

The bigger problem, though, is with the human in the loop.

If the solution has been glued together with generated code, you’re probably won’t have the conceptual understanding of the structure that’s needed to reliably understand the solution as a whole.

Code is important to a product, but building it relies on developing and maintaining intrinsic mental models - you have to be able to conceive and visualise abstract strictures of data and flows in your minds eye.

Database records don’t exist in any meaningful physical sense, for example - to understand the structure of the data you’re working with, you have to be able to visualise abstract structures and “spin” them in different directions as you’re working with them. To a very large extent, that’s THE key skill for developers - technical knowledge will get you so far, but lacking that kind of mental ability is going to be a significant handicap.

That skill is as good a definition of general intelligence as I can think of, so you can see where I’m going here. If general intelligence is needed to build robust software, and LLMs aren’t generally intelligent by any practical measure, you can’t rely on them to write your software for you.

LLMs will speed you up, but in that sense they’re not so different to a better Google search, or improved abstractions from the underlying technology like libraries and frameworks.

I don’t expect software engineering is going to go the way of buggy whip manufacturing for a while, and hopefully not in the time I’ve got left in the industry. It will change, just like it has constantly for the last 60 or 70 years, but I wouldn’t bet on the future of an organisations who are currently rushing to replace their software developers with ChatGPT licenses.

Running Jest tests alongside an Express server

2024-05-29T00:00:00Z

Here's the scenario: you've got an Express project that you're testing with Jest. You've got the server running, fire off your test suite, and you hit this error:

listen EADDRINUSE: address already in use :::3000

The root cause is that your test suite is trying to start a server instance on the default port, and failing because there's already a server running on that port.

The workaround is to persuade Jest to start its server on a different port so that it can co-exist with the production server.

That needs a bit of adaption in the server, the test suite and package.json.

1. Updating the server code

Start by defining a const for the port number:

dotenv.config();
const port = process.env.PORT || 3000;

Now wrap the server in a startServer function:

export const app: Express = express();
export function startServer(port: any) {
return app.listen(port, () => {
console.log(Server is running on port ${port});
});
}

And start the server, but only if we're not in the test environment :

if (process.env.NODE_ENV !== 'test') {
startServer(port);
}

2. Updating the test suite

Add a beforeAll() step to start the server on a testing port:

let server: any;
const testPort = 4000;
beforeAll((done) => {
// Start server on non-standard port
server = startServer(testPort);
done();
});

and an afterAll() step to tear down the server after the tests have run:

afterAll((done) => {
// Clean up all nocks
server.close(done);
});

3. Updating `package.json`

Specify the Node environment in the test command:

"scripts": {
"server": "tsc && node ./dist/server.js",
"build": "tsc",
"dev": "NODE_ENV=dev node --env-file=.env --watch -r ts-node/register src/server.ts",
"test": "NODE_ENV=test jest --verbose --detectOpenHandles",
"test:watch": "NODE_ENV=test jest --watch --verbose",
"lint": "eslint 'src/**/*.{js,ts}'"
},

This sets the NODE_ENV var in the environment spawned by the test process, which in turn forces the server to start on a non-production port.

Summary

The complete files should look something like this:

`server.ts`

import dotenv from 'dotenv';
import express, { Express, Request, Response } from "express";
dotenv.config();
const port = process.env.PORT || 3000;
export const app: Express = express();
export function startServer(port: any) {
return app.listen(port, () => {
console.log(Server is running on port ${port});
});
}
if (process.env.NODE_ENV !== 'test') {
startServer(port);
}
// ROUTES GO HERE

`server.test.ts`

import { app, startServer } from '../src/server';
let server: any;
const testPort = 4000;
beforeAll((done) => {
// Start server on non-standard port
server = startServer(testPort);
done();
});
afterAll((done) => {
server.close(done);
});
describe('The tests...', () => {
// TESTS GO HERE
});

`package.json`

{
...
"scripts": {
"server": "tsc && node ./dist/server.js",
"build": "tsc",
"dev": "NODE_ENV=development node --env-file=.env --watch -r ts-node/register src/server.ts",
"test": "jest --verbose --detectOpenHandles",
"test:watch": "NODE_ENV=test jest --watch --verbose",
"lint": "eslint 'src/**/*.{js,ts}'"
},
...
}

Data Rention

2024-05-01T00:00:00Z

Designing and implementing data security gets a whole lot easier if you start the exercise with the question "Why the hell are we keeping this data in the first place?"

In Praise of My Toaster

2024-04-07T00:00:00Z

This is my toaster.

It’s the oldest gadget I own.

It's got a retro style all of it's own, so it's immune to the currents and trends of fashion.

Over the years it's developed a patina of scratches and dents that only add to the aesthetic. Every dent a story, like the ding in the top where I dropped my favourite mug and saw it explode into fragments.

And while it was several times the price of something made of plastic, it's been amortised over 20 years and has probably worked out cheaper in the long run.

Last week I had to replace the element, which was a half-hour job involving a screwdriver and a pair of pliers. Now it's good for another 20 years.

In the process of replacing the broken part, I realised that in many ways, this might be the perfect gadget.

As a toaster, it works exceptionally well.

It gives feedback: The timer is clockwork, and the operation is simple. Turn further for darker toast. Once you've decided on your perfect level of toasting, that's embedded in memory as an angle of the wrist. You don't even need to look at the numbers - optimal setting is completely haptic.

It doesn't need monitoring: The buzzing of the clockwork provides an audible signal that it's running. As the timer runs down towards the end, the sound subtly changes. The end of the cycle is announced with a solid clunk as a finale, if the olafactory feedback provided by the smell wasn't enough.

It's easy to intervene: The lever that flips the toast up out of the slots is independent of the timer, so you can check the degree of browning without interrupting operations. That's also a failsafe - if you've miscalculated the timings, you can just bash down on the knob and the slices will be ejected out before burning too badly. Bash enthusiastically enough and time it right, and you can catch them on a plate

As an example of a gadget, it's also exceptional.

It's simple: Aside from the element it’s completely mechanical. There's no electronics to fail silently, or software to be buggy.Electrical problems can be diagnosed with simple tools. It doesn't take anything more sophisticated than a screwdriver to disassemble. The way it comes apart and goes back together again is completely obvious.

It's fixable: It's non-obsolescent. Components like the element and the clockwork mechanism will eventually fail because they're under stress - but they are readily-replaceable because the manufacturer has committed to maintaining enough inventory. At a push, it would be possible to create ersatz parts myself - rewinding an element with nichrome wire or 3D-printing a plastic part.

There are some things it can’t do. It can’t send me push notifications when the toast is ready, for example. But in some ways this is actually a net positive. If this toaster was online, it would be a countdown to obsolescence. The functionality would rely on some kind of backend web service that would last as long as the provider stayed interested. It would be at constant risk of enshittification through upsells, or my details being leaked and payments hijacked.

Do I actually need a push notification from my toaster. Can’t I rely on the audible cues of the clockwork timer, and the olfactory feedback of the toast browning? If I really wanted the connectivity, I could plug it into a current-sensing socket, and fire a message when the element stopped drawing power.

That would have the advantage of decoupling the problems - the toaster continues to do what the toaster is meant to do, and nothing more. I could switch the digital parts at will to mitigate the obsolescence and enshittification, and introduce new functionality as it develops. Rather than throwing the toaster away because it's not metaverse-compatible, augment it with the components needed to bridge the gap between the physical and virtual worlds.

There's a temptation to take the analogue approach to extremes, and slip into a kind of neo-luddite mindset that rejects any kind of connectivity, or to assume enshittification is the ultimate fate of any service. But there are some situations where it's very hard to see why there could be digital benefits, and the creation of two slices of nicely-browned toast has to be one of them.

Does It Move A Scooter?

2024-04-05T00:00:00Z

The hardest problem in a startup is figuring out what not to do.

Unless you’re very lucky, you’re resource-constrained - there simply isn’t enough money, time or people to do everything that could be done. At the same time, every stakeholder has their own opinion about what’s most important.

There’s the ever-present risk of doing the wrong thing, or worse, getting stuck in an analysis trap and not doing anything.

What you need is a quick rule-of-thumb that you can use to gauge whether something’s worth doing or not.

I learned this the hard way working on the project from hell - building an e-scooter sharing service which kicked off in a February and absolutely had to be up and running by that summer.

Every aspect of the service was built from scratch, from the hardware platform, to the mobile app, to the payment system.

Unsurprisingly, it was chaos - that scope would have been a challenge for a team twice the size with a timeline twice as long.

We spent a lot of time trying to figure out what needed to be built and what could be left for later, and the process of deciding which was which took almost as long as the building did. It also caused massive amounts of friction and argument.

After a particularly fractious debate, probably about something disproportionately minor, we hit the point where everyone looked at each other and said the same thing - “there has to be a better way of deciding these things”

I can’t remember who it was who came up with the idea, but it was simple and to the point:

If there was disagreement about whether something should be done, we ask a question to short-cut the debate.

“Does it move a scooter?”

That was a simple phrase, but it turned out to be a small act of genius. Within a day or two, it was written in huge multicolor letters across a whiteboard, and it was our standard way of prioritizing.

The approach works on multiple levels:

It’s focused. The goal of the project was to get people riding scooters, and scooters that can’t move can’t be ridden.

It’s simple. It doesn’t need a lengthy analysis process; there’s no scoring matrices or cost-benefit analysis. Ultimately it’s a ‘yes’ or ‘no’. True, it does strip nuance from complex situations, but it also encourages getting to the point.

It’s quick. There’s a finite amount of time you can spend arguing as a team about a simple question. If the answer isn’t quick, it’s a hint that the issue isn’t clear enough or the understanding isn’t shared.

It’s communicable. We ended up with it as a kind of team slogan, and it was so simple that it could be explained to business stakeholders 🤪

It’s uniting. In adverse situations like crunch projects, you need things that the team can unite around - and small-scale rituals like resolving arguments by pointing to the whiteboard and chanting “does it move a scooter” in unison can have a disproportionately-large effect on maintaining morale.

That exact phrase is only going to work for you if you’re in the scooter business, but there will be an equivalent in your domain.

There’s a temptation to make strategy bigger and more complex than it needs to be, because strategy is supposed to be complicated. But early stage startups are all about survival, so making things simple speeds things up.

Interacting with a JMAP API - a rapid-start

2024-03-29T00:00:00Z

IMAP is one of the predominant email protocols out there, but it's a) very old and b) somewhat painful to work with. To address some of these problems, JMAP was designed as IMAP-for-the-21st-ceentury, particularly for client usage.

Although it's an official IETF standard, it doesn't seem to have got that much traction - but is being used and supported by Fastmail, who also happen to be my long-term (nearly 20 years) email provider of choice.

That means I needed to get to grips with it in order to hack together my overly-complicated blog-content-from-markdown hack.

The problem with not being widely-adopted is the dearth of actual practical examples for ~~copy-and-pasting~~ inspiration. What follows is my get-things-up-and-running process, hopefully as a springboard for more complex usecases.

Basics

You interact with the API by sending HTTP POST requests with an authorisation header and a JSON payload. The payload contains the command you want to run along with any necessary criteria such as search terms.

Fastmail's authorisation requires an Authorization header with a value in the format Bearer <API token>)
The API token needs to be created on the Fastmail dashboard
The method calls are sent as JSON text with an application/json type in the POST body

Command format

The POST body has two parts:

a using array, which doesn't change from method to method and defines the spec being used
a methodCalls array which contains the actual methods you want to call, along with their parameters

The method calls have a format in the shape of object/action - objects are Mailbox, Email, Thread and so on; actions are verbs like get, set, query etc.

The main commands we'll be using are Mailbox/get, Email/query and Email/get.

The overall shape of the body looks like this:

{
  "using": [
    "urn:ietf:params:jmap:core",
    "urn:ietf:params:jmap:mail"
  ],
  "methodCalls": [
    <calls go here>
  ]
}

Getting your account ID

The first step is to figure out your Fastmail account ID. This requires making an authenticated GET request to https://api.fastmail.com/.well-known/jmap, which returns a JSON response containing details of your account and what rights the API key allows.

Buried in middle somewhere is a primaryAccounts key, which should look something like this:

"primaryAccounts": {
    "urn:ietf:params:jmap:core": "u123456ab",
    "urn:ietf:params:jmap:mail": "u123456ab"
  },

Make a note of the value of the <xxx>:mail key, you'll need it for all further requests.

Request a list of folders

Most processes are going to involve getting a list of available messages in a specific folder, and the prerequisite for that is a folder ID.

The request

Let's start by getting a list of folders (aka mailboxes in JMAP-speak). Send this JSON body in a POST request:

{
  "using": [
    "urn:ietf:params:jmap:core",
    "urn:ietf:params:jmap:mail"
  ],
  "methodCalls": [
    [
      "Mailbox/get",
      {
        "accountId": "u123456ab",
        "ids": null
      },
      "abc"
    ]
  ]
}

Breaking down the methodCalls section, there's three parts:

the command we're sending, in this case Mailbox/get
the parameters for this command, in this case the accountId and the folders ids we want to match. By sending null, the server will respond with a list of all folders/mailboxes
the trailing abc is effectively a variable name - the results of this Mailbox/get command will be available to subsquent commands by referencing the abc variable, which allows you to create composite commands that are chained together

The response

The respose is a blob of JSON containing the methodResponse array. That in turn contains a list of folders and their various properties - the keys we're interested in look like this:

{
  "methodResponses": [
    [
      "Mailbox/get",
      {
        "list": [
          {
            ...
            "name": "Inbox",
            "id": "abcdef1234",
            ...
          },
        ...
      },
      "abc"
    ]
  ],
  "latestClientVersion": "",
  "sessionState": "cyrus-0;p-e21c23889a;s-6606885adc92256b"
}

The name key contains the human-readable folder name, and the id is the corresponding folder ID

Getting a list of emails

Retrieving a specific email is a two-stage process: finding its ID, then retrieving the email itself:

Retrieving a list of emails in a specific folder

This uses the Email/query command with a filter (this example also throws in a sort order for good measure). It's the same process - make a POST request with the auth headers, and send over the JSON payload in the body.

This is the full payload:

{
  "using": [
    "urn:ietf:params:jmap:core",
    "urn:ietf:params:jmap:mail"
  ],
  "methodCalls": [
    [
      "Email/query",
      {
        "accountId": "u123456ab",
        "filter": {
          "inMailbox": "abcdef1234"
        },
        "sort": [
          {
            "property": "receivedAt",
            "isAscending": false
          }
        ],
        "limit": 500
      },
      "xzy"
    ]
  ]
}

Breaking this down, we have:

a filter, which searches in a specific mailbox with ID abcdef1234
a sort order, which in this case is by property receivedAt and most-recent first
a limit, in case there's a lot of results. If that happens, you can start using paging, but that's beyond the scope of this simple example.

The result is a JSON response which looks like this:

{
  "methodResponses": [
    [
      "Email/query",
      {
        "ids": [
          "Mcae60028353b175d88882abe",
          "M999a6d73062302c1592daebe",
          "M38493717a69686cbb4fc4d6f",
        ],
        "sort": [
          {
            "isAscending": false,
            "property": "receivedAt"
          }
        ],
        "filter": {
          "inMailbox": "abcdef1234"
        },
        "position": 0,
        "collapseThreads": false,
        "queryState": "1871939:0",
        "canCalculateChanges": true,
        "accountId": "u123456ab",
        "total": 3
      },
      "xyz"
    ]
  ],
  "latestClientVersion": "",
  "sessionState": "cyrus-0;p-e21c23889a;s-6606885adc92256b"
}

The first and most useful part is the ids array, which contains a list of IDs for actual emails. You also get a confirmation of the sort order and filter that was used, and some metadata about the query itself.

position relates to the paging if that took place; collapseThreads relates to how message threads are handled, and the total value is the number of records returned which can be useful when processing the results.

Retrieving a specific email

Now that we've got message IDs, we can finish off by retrieving a specific one. This uses the Email/get method, to which you pass a specific message ID(s):

{
  "using": [
    "urn:ietf:params:jmap:core",
    "urn:ietf:params:jmap:mail"
  ],
  "methodCalls": [
    [
      "Email/get",
        {
          "accountId": "abcdef1234",
          "properties": null,
          "ids": [
            "Mcae60028353b175d88882abe"
          ]
        },
      "pqr"
    ]
  ]
}

The result (assuming the message exists) should be a lump of JSON that contains the message itself along with a pile of metadata:

{
"methodResponses": [
[
"Email/get",
{
"accountId": "abcdef1234",
"state": "1872326",
"notFound": [],
"list": [
{
"attachments": [],
"mailboxIds": {
"abcdef1234": true
},
"size": 91829,
"htmlBody": [
{
"language": null,
"location": null,
"blobId": "G67f2b026af5a35d77985d0dd57edb93364d2736",
"name": null,
"type": "text/html",
"cid": null,
"size": 61902,
"charset": "utf-8",
"partId": "2",
"disposition": null
}
],
"blobId": "G438f25a1713b0b3445b3ecd76a2d33d43337856",
"from": [
{
"email": "sz-magazin@newsletter.sueddeutsche.de",
"name": "SZ-Magazin Das Beste"
}
],
"textBody": [
{
"disposition": null,
"size": 13300,
"charset": "utf-8",
"partId": "1",
"cid": null,
"type": "text/plain",
"name": null,
"language": null,
"location": null,
"blobId": "G05569b6bd4104d589a7efdece3d2f5d7dcbfe1e"
}
],
"inReplyTo": null,
"id": "M438f25a1713b0b4451b3ecd",
"sentAt": "2024-03-30T07:59:53+01:00",
"to": [
{
"name": null,
"email": "tim@duckett.de"
}
],
"receivedAt": "2024-03-30T06:59:59Z",
"sender": null,
"hasAttachment": false,
"threadId": "Td27cfc407a520026",
"keywords": {
"$x-me-annot-2": true,
"$ismailinglist": true,
"$canunsubscribe": true
},
"preview": "Außerdem: Warum das Wort »später« so gefährlich ist Sollte der Newsletter nicht korrekt angezeigt werden, klicken Sie bitte hier Illustration: iStock / by Malte Mueller",
"subject": "Lust auf ein neues Leben?",
"cc": null,
"bcc": null,
"messageId": [
"0.0.1A.354.1DA826FDE08E6A0.0@uspmta120023.emarsys.net"
],
"replyTo": null,
"bodyValues": {},
"references": null
}
]
},
"a"
]
],
"latestClientVersion": "",
"sessionState": "cyrus-0;p-e21c23889a;s-6607b80d3ff0a0fe"
}

This is an example of an HTML-formatted message - there's a text preview which is contained in the preview key, but there's one more step required to get the content of the message itself. This is available in two formats:

HTML, which is referred to by the blobId contained in the htmlBody section
text, which is referred to by the blobId contained in the textBody section

Confusingly, there's a third blobId in the list section which refers to the message itself in case you want retrieve that as a block of text.

Retrieving the contents of a specific email

This final step is different to the others, because it's a straight-forward GET request:

GET https://www.fastmailusercontent.com/jmap/download/<accountId>/<blobId>/<filename>?type=<type>

The parameters here are:

accountId - the account ID as used before
blobId- the ID of the specific blob you want, either HTML or text
filename - this appears to have no effect as far as I can tell, but needs to be present; you can send any string
type - the MIME type in which you want the content encoded. This makes no practical difference in terms of the content returned, but does show up in the content-type response header in case your code needs this. In this example, it makes most sense to send application/html

Getting email attachments

Although that final GET request is the end point if what you want is the message, my usecase was slightly different and I wanted any attachments.

The good news here is that the process of getting attachments is the same as the process of getting email contents - you send a GET request with a blob ID.

Attachment blob IDs will show up in the attachments array at the top of the email payload:

{
"methodResponses": [
[
"Email/get",
{
"accountId": "abcdef1234",
"state": "1872326",
"notFound": [],
"list": [
{
"attachments": [
"G67f2b026af5a35d77985d0dd45edb93364d2736"
],
"mailboxIds": {
"abcdef1234": true
},
<SNIP>
}

Grab the blob ID(s) from the attachments array, feed these into the GET requests, and you can download and process attachments as needed.

Finishing up

This is only the minimal basics needed to get messages and attachments; there's a lot more that the protocol can do. The docs are here, and you can use that information to extend the examples above to build a completely-functional email client if that's what you need.

There's a Postman collection of the relevant API calls here.

LinkedIn, how I loathe thee

2024-03-19T00:00:00Z

LinkedIn sits firmly in the “it has to exist, but wouldn’t life be better if it didn’t have to?” quadrant of existence.

For the basics of a network of connections, and a way for others to find me, it works. But every time I close the tab, I can’t help thinking “this should work better than it does”.

The endless scrolling encourages mindless interaction. There’s no “end” - it’s never “done” and finished with, so when is “enough” is a decision I have to make
The random nature of the feed makes it virtually impossible not to lose things unless you you remember to bookmark something immediately
Once a post has gone, you’re stuck with trying to remember enough detail to search
Speaking of which, LinkedIn search sucks, and sucks badly
Feed filtering is more unreliable than not, so the default is drown in garbage
The feed doesn’t appear to prioritise your contacts (I assume to drive more connections)

All of this creates the feeling that the interaction patterns are optimised for some product manager’s perception of “engagement”, rather than my utility.

Fixing this shouldn’t be hard. There’s no reason why a useful UI can’t coexist with the river of broetry from the hustle pros and Gary Vee wannabes.

Provide a reverse-chronological feed of content from my contacts, and when it’s all read, it’s done - basically the pattern of Twitter before Elon Musk burned it to the ground.

Bearblogging

2024-03-19T00:00:00Z

It’s truth universally acknowledged that a person in possession of opposable thumbs an a modicum of programming skills must be in want of an over-elaborate web publishing workflow.

My basic blog setup is based on Hugo, which is a static site generator that renders Markdown files into web pages. I write the Markdown files locally, then push the changes to GitHub, after which a deployment job in Render is triggered. This checks out the latest revision, builds it, and published the resulting static HTML.

It all works very well, but it does depend on creating, committing and pushing Markdown files. That’s not easy to do on something that isn’t a laptop - not that Markdown is difficult, but there’s no good combination of both iOS GitHub client and an iOS Markdown editor.

What iOS and Mac do have is Bear, which is a beautifully-designed notes tool that I use extensively as an outboard brain. It’s ideal for banging out short and medium length files, but it doesn’t fit at all into a Hugo workflow.

Until now. The process I’ve come up with isn’t exactly elegant, but it is seamless.

I write the content in Bear, and email the resulting file to myself as a Markdown file
My mail server files the email into the right folder based on sender and subject
Periodically, a cron job wakes up a Node app running in a Docker container on Render by pinging a webhook URL
The Node app checks for waiting email; grabs the message; downloads the attachment, renders it into a Hugo-style Markdown file, saves it locally, then pushes the new file up to GitHub.
The my website’s Render project watches its repo, and whenever a change is detected it runs the deploy process I described above.
I can force the whole process by pinging the webhook URL to wake the Node app up manually.

No, this was not exactly trivial to setup.

Yes, there are quite a lot of moving parts.

Yes, I probably did take it all a bit far. It’s a bit over the top just for the sake of sticking with a specific text editor. On the other hand, it was a learning process, particularly the “devopsy” bits of building and deploying Docker containers; and wrangling the connections to IMAP servers.

Will I actually use this, to keep this site updated more often? To be determined…

Sleepwalking into a new dark age

2024-03-08T00:00:00Z

The report into the extensive attack which took out the British Library has just been published, and it doesn’t make for good reading. It’s another data point that that reinforces my theory that we’re sleepwalking into a new dark ages.

Briefly, the British Library - the ultimate custodian of every printed and published resource in Britain since the 18th century and before - was crippled by an attack that brought down systems and exfiltrated gigabytes of data. It’s still struggling to get all systems back online, and there’s no timeline for when - or even if - the damage will finally be repaired.

The report is partial, because much of the evidence of how the attack took place has been lost in the rubble. But there’s some depressing conclusions.

The attack succeeded in the first place because systems were complex, outdated and unmaintained. Cost pressures led to a tangle of contractors and suppliers, which in turn allowed otherwise-addressable vulnerabilities to creep in, and resulted in a lack of institutional knowledge about how systems operated.

The damage was compounded by some systems being unrecoverable simply because they’re now obsolete - either they’re no longer maintained, or they’re incompatible with available operating systems.

The fact that the report’s been published is positive - it’s hard to imagine a commercial organization washing its laundry in public like this. There’s a chance that lessons will be learned and the future will be different, although past experience suggests that’s a lot easier to promise than deliver.

What strikes me reading the report is that at the same time that the sheer volume of content being created is accelerating, maintaining it for the long-term is getting increasing difficult. A document from 500 years ago can still be read and understood; a file stored on a medium from the last decade is probably lost forever now.

Setting up a Typescript-based Node project: the basics

2024-02-25T00:00:00Z

tl;dr: This is the simplest way I know of setting up a Node projeect using Typescript from scratch.

My weapon of choice for most development these days is Node, and because my formative code years were spent fighting with type-safe languages, I prefer Typescript to vanilla Javascript.

I start a lot more projects that I finish, but despite the number of times I've set things up from scratch, I always end up forgetting some step some point along the way. The point of this post is a note-to-self with a process to follow; and maybe an aide-memoire for others.

There's a cloneable repo containing this setup on Github.

The process

There are three stages:

Basic directory and Git things
Node and Typescript
Wiring it all up

Strictly speaking, step 3 isn't essential, but it's nice to make sure things work.

Directory and Git things

Create a directory for the project:

mkdir template-node-project
cd template-node-project

Initialise Git

git init

Add a .gitignore for later:

touch .gitignore

Add a README for later:

touch README.md

Setup the Node project

Initialise the Node project:

npm init -y

This will create a package.json file that contains default settings for things like the project name, version, license and so on. You can edit these as needed later.

Set up Typescript

Install Typescript, the Node dependencies, and the Node types:

npm i -D typescript ts-node @types/node

Initialise Typescript:

npx tsc --init

Add an .env file:

touch .env

Add a placeholder environment variable in the .env file:

GREETING=world

Update the package.json file to add a couple of extra settings:

Add an engines section

{
  "engines": {
    "node": ">=20.6.0"
  },
  "name": "template-node-project",
  "version": "1.0.0",
  "description": "",
  "main": "index.js",
  "scripts": {
    "build": "tsc",
    "dev": "node --env-file=.env --watch -r ts-node/register src/index.ts",
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "keywords": [],
  "author": "",
  "license": "ISC"
}

this defines:
- uses tsc as the compiler
- loads the .env file to set environment variables
- runs the Node process in 'watch' mode, so code changes are automatically reloaded
- runs the index.ts file

Add in some boilerplace code

Create a src directory for the source files:

mkdir src
cd src

Create an index.ts file:

mkdir src
touch ./src/index.ts

Add a basic 'hello, world' function to index.ts:

function helloWorld(): void {
  console.log(`Hello, ${process.env.GREETING}!`);
}
helloWorld();

Run the Node process

The Node process can be run with npm run dev

This picks up the script that was setup in the package.json file earlier - if everything's configued correctly you'll see the process fire up and run the helloWorld function:

npm run dev > template-node-project@0.0.1 dev > node --env-file=.env --watch -r ts-node/register src/index.ts

(node:98297) ExperimentalWarning: Watch mode is an experimental feature and might change at any time (Use node --trace-warnings ... to show where the warning was created) Hello, world! Completed running 'src/index.ts'

Enjoy the ride

2024-02-10T00:00:00Z

In research that should surprise absolutely no-one who’s ever been involved in software development, there’s evidence that generative AI tools are strongly-correlated with more copy/paste, less code refactoring and reuse, and higher code churn.

In other words, faster output, high volatility and lower long-term quality.

If I was going to extrapolate this to the industry as a whole, I’d expect a drop in demand for experienced software developers over the next couple of years as companies go all-in on the apparent savings...

...followed by an explosion in demand for people who can sort out the mess that was caused…

...and an absolute shortage of those skills thanks to the hollowing-out of the workforce while AI is the apparent answer.

Enjoy the ride!