Notifications for New Eleventy Posts in GitLab - Part 2

April 29, 2024

One of the challenges with deploying static sites is that there's nothing tracking any sort of site state, including when new content is published. This post presents a technique to identify newly published content on an Eleventy site and sending various notifications with content-specific data. Part 1 covered identifying the new posts and collecting post-specific data. Part 2 covers posting a status to Mastodon, posting a status to Bluesky, and sending an IndexNow notification for the new page.

Sending notifications for new posts #

Once the new posts are identified, any kind of notification could be sent, but the three implemented here are:

Posting a status to Mastodon
Posting a status to Bluesky
Sending an IndexNow notification

Managing secrets in GitLab #

The following sections use secrets, specifically API keys/tokens, to access those services securely. These secrets should not be kept in the project's git repository. For these examples, these are all stored as variables in GitLab, which are encrypted for security. They also have the following settings:

protected: Ensures the variable is only exposed on protected branches or protected tags. By default, these require project maintainer or owner permissions to execute and is typically the setting on the default branch.
masked: Uses GitLab's bult-in logic to redact the variable in any GitLab CI job log. As the GitLab docs note, this is on a best-effort basis, but is generally reliable (maybe sometimes too reliable and there's unnecessary masking - better to be safe than sorry).

These variables are then exposed to jobs on GitLab CI pipelines run on protected branches or tags as environment variables. For this example that's the default branch, which is where the site is published.

See the documentation for additional details on GitLab variables security.

Iterate through new posts providing notifications #

In part 1, a function was introduced to iterates through new posts to send notifications via an async queue.

(async () => {
    const posts = getNewPosts();
    if (posts.length === 0) {
        console.log('No new posts to submit');
        return;
    }
    const taskQueue = [];
    for (const post of posts) {
        console.log(`Submitting updates for ${post.url}`);
        taskQueue.push(
            postMastodonStatus(post),
            postBlueskyStatus(post),
            postIndexNow(post)
        );
    }

    const results = await Promise.allSettled(taskQueue);
    for (const result of results) {
        if (result.status === 'rejected') {
            console.error(result.reason.message);
        }
    }
})();

This function has been updated to call functions for each of the three desired notifications passing the post data. The implementation for the postMastodonStatus, postBlueskyStatus, and postIndexNow functions is detailed in the following sections. These are the three examples implemented with this technique, but could be augmented or replaced with other notifications.

Post Mastodon status #

To start posting to your applicable Mastodon instance via the API, you must first create an Application, and with that obtain an access token. This can be done on your applicable Mastodon instance at Preferences > Development > New application (at least it is in my case, the official documentation on doing this via the UI is, unfortunately, limited).

You'll need to provide application name, website, and the applicable scopes (you do not need to modify the default Redirect URI value). The default scopes give the application a lot of capability, for the actions here only read:statuses and write:statuses are required. Then click SUBMIT and you'll be returned to your list of applications. From there you can see the details for the application that was just created, and "Your access token" is the token value to use in the example below.

The access token is stored as the protected and masked variable MASTODON_TOKEN within GitLab. The postMastodonStatus function uses that token and the post information to post a status to Mastodon.

const postMastodonStatus = async (post) => {
    const accessToken = process.env.MASTODON_TOKEN;
    // Update for the applicable Mastadon instance
    const instanceUrl = 'https://fosstodon.org';
    const data = {
        status: `${post.description}\n\n${post.url}\n\n${post.tags.join(' ')}`
    };

    const response = await fetch(`${instanceUrl}/api/v1/statuses`, {
        body: JSON.stringify(data),
        headers: {
            Authorization: `Bearer ${accessToken}`,
            'Content-Type': 'application/json'
        },
        method: 'POST'
    });

    const postStatus = await response.json();
    if (!response.ok) {
        throw new Error(`Error posting Mastadon status: ${postStatus.error}`);
    }
    console.log(`Mastodon status successfully posted (ID: ${postStatus.id})`);
};

The function uses fetch to post the status with the following data:

The previously defined MASTODON_TOKEN environment variable with the access token.
The base URL of the applicable Mastodon instance.
A status message, which in this case includes the post description, URL, and hashtags (the tags array was previously formatted as hashtags).

If successful, a JSON response is returned and a success message is logged with the returned status ID. If not, an error is thrown with the response error.

Additional details on using the Mastodon API can be found in the the documentation.

Post Bluesky status #

To access Bluesky's API an App password must be created (please don't use your actual account password). The official docs, don't cover this. An App password can be created at Settings > Advanced > App Passwords > Add App Password. You'll be prompted for a name for the App, and then returned the password. Note that this is only opportunity to view the password, so capture it somewhere secure. If it's lost, it can be deleted and a replacement created.

The App password is stored as the protected and masked variable BSKY_PASSWORD within GitLab. In addition the associated Bluesky ID is stored as the protected and masked variable BSKY_ID within GitLab.

The postBlueskyStatus function uses these credentials and the post information to submit a post to Bluesky.

const { BskyAgent, RichText } = require('@atproto/api');

const postBlueskyStatus = async (post) => {
    // Use Bluesky agent based on API complexity
    const agent = new BskyAgent({
        service: 'https://bsky.social'
    });

    try {
        await agent.login({
            identifier: process.env.BSKY_ID,
            password: process.env.BSKY_PASSWORD
        });

        const message = `${post.description}\n\n${post.url}`;
        // Rich formatting in posts, for example links and mentions, must be
        // specified by byte offsets, so use the RichText capabilities to
        // detect them.
        const rt = new RichText({ text: message });
        await rt.detectFacets(agent);

        const postRecord = {
            $type: 'app.bsky.feed.post',
            createdAt: new Date().toISOString(),
            // Cards are not automatically created from OG tags in the link,
            // they must be explicitly added. Note this does not include
            // images, which must be referenced, and separately posted to
            // the API.
            embed: {
                $type: 'app.bsky.embed.external',
                external: {
                    description: post.description,
                    title: post.title,
                    uri: post.url
                }
            },
            facets: rt.facets,
            text: rt.text
        };

        await agent.post(postRecord);
        console.log('Bluesky status successfully posted');
    } catch (error) {
        throw new Error(`Error posting Bluesky status: ${error.message}`);
    }
};

Bluesky's AT protocol has granular control, but with that comes additional complexity. So, the Bluesky agent from the official API client is used:

The BSKY_ID and BSKY_PASSWORD environment variable are used to login.
The message includes the post description and URL. Any links (or mentions) in the message are not automatically detected, and must be specified by the start/end byte offset, but the API client provides the RichText class that is used to detect and provide this data (to linkify the URL).
The postRecord is constructed with all of this data. In addition, cards representing the URL are not automatically added, so an embed is included with the card data (description, title, and URL). In this case an image is not provided, although if required it must be submitted via the API as a separate request (see the docs for full details).

If successful, a success message is logged. If not, an error is thrown with the response error.

Additional details on using the AT protocol for Posts can be found in the the Bluesky documentation, as well as the API client documentation.

Post URL to IndexNow #

IndexNow is a standard that allows submission of a URL to enabled search engines, which is then shared with all IndexNow-enabled engines (Bing, for example, is one of the IndexNow enabled search engines). This avoids an indexing delay waiting for the new page to be organically discovered.

Details on using the API and obtaining an API key is available in the documentation. The API key is stored as the protected and masked variable INDEXNOW_API_KEY within GitLab. To validate the API key, a file is required to be deployed to the site. This file is named with the key, and its content includes the key, so the previously defined pages CI job script is updated to create the file.

pages:
  ...
  script:
    # build runs `npx @11ty/eleventy`
    - npm run build
    # Create IndexNow key file in the site output folder
    - echo $INDEXNOW_API_KEY > ./public/${INDEXNOW_API_KEY}.txt
  artifacts:
    # Ensure artifacts with the key file are not publicly visible
    public: false
    ...

This implementation allows the API key to be securely stored in a GitLab variable and the file created only for deployment. With that, the job artifacts is also updated with public: false to ensure that job artifacts are not made available publicly (only to project members). The variable is also masked, but this provides an extra layer of protection against accidental exposure.

The postIndexNow function uses the API key and the post information to submit the URL to IndexNow.

const postIndexNow = async (post) => {
    const apiKey = process.env.INDEXNOW_API_KEY;
    const indexNowUrl = 'https://api.indexnow.org/IndexNow';
    const postUrl = new URL(post.url);
    const { host } = postUrl;
    const data = {
        host,
        key: apiKey,
        keyLocation: `https://${host}/${apiKey}.txt`,
        urlList: [postUrl.toString()]
    };

    const response = await fetch(indexNowUrl, {
        body: JSON.stringify(data),
        headers: { 'Content-Type': 'application/json' },
        method: 'POST'
    });

    if (!response.ok) {
        throw new Error(
            `Error submitting URL to IndexNow: ${response.statusText}`
        );
    }
    console.log(`URL ${post.url} successfully submitted to IndexNow`);
};

The function uses fetch to post the status with the following data:

The previously defined INDEXNOW_API_KEY environment variable with the API key.
The website host is taken from the new post URL.
The location of the API key file that was previously created.
The list of URLs (in this case only one URL is submitted with each request, but this API endpoint does accept a list of URLs).

In this cases the update is posted to api.indexnow.org, although it could be posted to another IndexNow server (for example, www.bing.com) per the documentation.

If successful, a success message is logged. If not, an error is thrown with the response status.

Posting updates #

On each commit to main, the update script ./scripts/new-posts.js is run, as detailed in part 1.

If no new posts are detected, the logs show:

$ node ./scripts/new-posts.js
No new posts to submit

If there is a new post, the logs show (the logs from part 1 of this post):

$ node ./scripts/new-posts.js
Submitting updates for https://[MASKED]/posts/notifications-for-new-eleventy-posts-in-gitlab-part-1/
Mastodon status successfully posted (ID: 123456789012345678)
Bluesky status successfully posted
URL https://[MASKED]/posts/notifications-for-new-eleventy-posts-in-gitlab-part-1/ successfully submitted to IndexNow

In this case the domain is [MASKED] because it happens to be my Bluesky ID (the value for BSKY_ID, a masked variable that probably doesn't need to be masked).

Summary #

This post has detailed how to identify new posts in a Eleventy build, with summary data for those posts, and use that to post updates to Mastodon, Bluesky, and IndexNow. For reference, the final .gitlab-ci.yml, .eleventy.js, and ./scripts/new-posts.js files (the pieces covered in this post) are included here.

.gitlab-ci.yml

npm_install:
  image: node:20-alpine
  needs: []
  script:
    - npm ci
  artifacts:
    paths:
      - node_modules/

# Current sitemap must be retrieved before deploy for comparison.
get_current_sitemap:
  image: alpine:latest
  # With no needs, the job will run at the start of the pipeline
  needs: []
  script:
    - wget -O sitemap.xml https://<site>/sitemap.xml
  rules:
    # Site only deploys on the default branch
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
  artifacts:
    paths:
      - sitemap.xml

pages:
  image: node:20-alpine
  needs:
    - npm_install
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
  script:
    # build runs `npx @11ty/eleventy`
    - npm run build
    # Create IndexNow key file in the site output folder
    - echo $INDEXNOW_API_KEY > ./public/${INDEXNOW_API_KEY}.txt
  artifacts:
    # Ensure artifacts with the key file are not publicly visible
    public: false
    paths:
      - public/
      - posts.json

new_post_notification:
  image: node:20-alpine
  # Needs specifies artifacts to download as well as prerequisite jobs
  needs:
    # Provides node_modules folder
    - npm_install
    # Provides previous sitemap.xml
    - get_current_sitemap
    # Provides built site and posts.json
    - pages
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
  script:
    - node ./scripts/new-posts.js

.eleventy.js (unchanged from part 1)

'use strict';

const fs = require('node:fs');
const path = require('node:path');

// Global paths
const inputPath = 'src';
const outputPath = 'public';

const sanitizeTag = (tag) =>
    tag.toLowerCase().replaceAll(/[#/]/g, '').replaceAll(' ', '-');

module.exports = function (eleventyConfig) {

    // other configuration

    eleventyConfig.addFilter('stringify', (value) => JSON.stringify(value));

    eleventyConfig.addFilter('stringifyTags', (tags) =>
        JSON.stringify(tags.map((tag) => `#${sanitizeTag(tag)}`))
    );

    // Create collection of posts data for use in external notifications
    eleventyConfig.addCollection('postsData', (collection) =>
        collection.getFilteredByTag('posts').map((item) => ({
            date: item.date,
            description: item.data.description,
            inputPath: item.inputPath,
            outputPath: item.outputPath,
            tags: item.data.tags.filter((tag) => tag !== 'posts'),
            title: item.data.title,
            url: item.url
        }))
    );

    // Move the posts.json file to the root folder since not deployed
    eleventyConfig.on('eleventy.after', () => {
        const postsDataFilename = 'posts.json';
        fs.renameSync(path.join(outputPath, postsDataFilename), postsDataFilename);
    });

    return {
        dir: {
            input: inputPath,
            output: outputPath
        },
        // other configuration
    };
};

./scripts/new-posts.js

'use strict';

const fs = require('node:fs');
const { BskyAgent, RichText } = require('@atproto/api');

const sitemapFilename = 'sitemap.xml';
const postsFilename = 'posts.json';
const postsThreshold = 3;

const getNewPosts = () => {
    const sitemap = fs.readFileSync(sitemapFilename, 'utf8');
    const urlRegex = /<loc>(?<url>.+\/posts\/.+?)<\/loc>/g;
    const sitemapUrls = [...sitemap.matchAll(urlRegex)].map((match) => match.groups.url);

    const posts = JSON.parse(fs.readFileSync(postsFilename, 'utf8'));
    if (
        sitemapUrls.length === 0 ||
        posts.length === 0 ||
        Math.abs(sitemapUrls.length - posts.length) > postsThreshold
    ) {
        throw new Error(
            'Error: sitemap and posts data are invalid or out of sync'
        );
    }
    return posts.filter((post) => !sitemapUrls.includes(post.url));
};

const postMastodonStatus = async (post) => {
    const accessToken = process.env.MASTODON_TOKEN;
    // Update for the applicable Mastadon instance
    const instanceUrl = 'https://fosstodon.org';
    const data = {
        status: `${post.description}\n\n${post.url}\n\n${post.tags.join(' ')}`
    };

    const response = await fetch(`${instanceUrl}/api/v1/statuses`, {
        body: JSON.stringify(data),
        headers: {
            Authorization: `Bearer ${accessToken}`,
            'Content-Type': 'application/json'
        },
        method: 'POST'
    });

    const postStatus = await response.json();
    if (!response.ok) {
        throw new Error(`Error posting Mastadon status: ${postStatus.error}`);
    }
    console.log(`Mastodon status successfully posted (ID: ${postStatus.id})`);
};

const postBlueskyStatus = async (post) => {
    // Use Bluesky agent based on API complexity
    const agent = new BskyAgent({
        service: 'https://bsky.social'
    });

    try {
        await agent.login({
            identifier: process.env.BSKY_ID,
            password: process.env.BSKY_PASSWORD
        });

        const message = `${post.description}\n\n${post.url}`;
        // Rich formatting in posts, for example links and mentions, must be
        // specified by byte offsets, so use the RichText capabilities to
        // detect them.
        const rt = new RichText({ text: message });
        await rt.detectFacets(agent);

        const postRecord = {
            $type: 'app.bsky.feed.post',
            createdAt: new Date().toISOString(),
            // Cards are not automatically created from OG tags in the link,
            // they must be explicitly added. Note this does not include
            // images, which must be referenced, and separately posted to
            // the API.
            embed: {
                $type: 'app.bsky.embed.external',
                external: {
                    description: post.description,
                    title: post.title,
                    uri: post.url
                }
            },
            facets: rt.facets,
            text: rt.text
        };

        await agent.post(postRecord);
        console.log('Bluesky status successfully posted');
    } catch (error) {
        throw new Error(`Error posting Bluesky status: ${error.message}`);
    }
};

const postIndexNow = async (post) => {
    const apiKey = process.env.INDEXNOW_API_KEY;
    const indexNowUrl = 'https://api.indexnow.org/IndexNow';
    const postUrl = new URL(post.url);
    const { host } = postUrl;
    const data = {
        host,
        key: apiKey,
        keyLocation: `https://${host}/${apiKey}.txt`,
        urlList: [postUrl.toString()]
    };

    const response = await fetch(indexNowUrl, {
        body: JSON.stringify(data),
        headers: { 'Content-Type': 'application/json' },
        method: 'POST'
    });

    if (!response.ok) {
        throw new Error(
            `Error submitting URL to IndexNow: ${response.statusText}`
        );
    }
    console.log(`URL ${post.url} successfully submitted to IndexNow`);
};

(async () => {
    const posts = getNewPosts();
    if (posts.length === 0) {
        console.log('No new posts to submit');
        return;
    }
    const taskQueue = [];
    for (const post of posts) {
        console.log(`Submitting updates for ${post.url}`);
        taskQueue.push(
            postMastodonStatus(post),
            postBlueskyStatus(post),
            postIndexNow(post)
        );
    }

    const results = await Promise.allSettled(taskQueue);
    for (const result of results) {
        if (result.status === 'rejected') {
            console.error(result.reason.message);
        }
    }
})();