Posts on The blog of Peter Evans

Synthetic Monitoring: Building Confidence in Distributed Systems

Mon, 02 Feb 2026 09:00:00 +0000

Tangential to my main area of responsibility at GitHub, I recently worked with another engineer on a side project to create a synthetic monitoring service. It launched this week and so I thought I would share a little about what it is, and the motivation behind it.

Let’s say you’ve just finished deploying a new service or spun up infrastructure in a new region. The dashboards are all green. CPU utilization looks normal. Memory is fine. No errors in the logs. Everything appears healthy.

But is it actually working? 🤔

When you have little to no real traffic flowing through a new deployment, traditional monitoring doesn’t give you the full picture. Metrics can only tell you what’s happening with the traffic you have. If you have no traffic, you have no signal.

Low Traffic Challenges

Over the last couple of years, we’ve built GitHub in multiple regions for GitHub Enterprise Cloud with data residency. Each new region involves deploying a full stack of all of GitHub’s services, databases, caches, and so on. This has brought challenges around confidence in new deployments because, initially, there is little to no user traffic in these new regions.

For many service teams, the answer to this problem has been manual verification. Someone would log in, click around, create some test data, and confirm that things seemed to work. This approach has obvious drawbacks: it’s tedious, it doesn’t scale, and it’s prone to human error.

Synthetic Monitoring

Synthetic monitoring (also called “active monitoring”) is a technique where you simulate real user actions against your service to verify it’s working correctly. Unlike passive monitoring that watches logs and metrics, synthetic monitoring actively exercises your system and verifies the results. Think of it as having a tireless robot that continuously performs the same actions a real user would, reporting back on whether things worked and how long they took.

This concept isn’t new. Companies have been using synthetic monitoring for years to verify that their public-facing websites load correctly, often running tests from multiple geographic locations to ensure global availability.

What We Built

Our synthetic monitoring service runs a suite of tests against multiple deployment environments on a regular schedule. Each test simulates a real user workflow: creating resources, making API calls, verifying the results, and cleaning up after itself.

The tests run continuously, every few minutes, across all environments we care about. When something fails, we know about it immediately. When latency degrades, we see it in the telemetry. Most importantly, we can compare behavior across environments to spot anomalies that might otherwise go unnoticed.

We instrument every operation automatically, so engineers contributing new tests don’t need to think about metrics or observability. They simply write code that exercises the functionality they care about, and the framework handles the rest. This low barrier to entry will be crucial for adoption across the organization.

The Two-Stage Approach

One interesting challenge we encountered was testing asynchronous behavior. Many systems don’t complete all their work synchronously. Background jobs run, events propagate, and downstream systems react. If you only test the synchronous response, you might miss failures that happen after the initial request returns successfully.

To address this, we structured our tests in two stages. The first stage performs the initial operations and records what it created. The second stage runs after a delay, checking that asynchronous processes completed successfully. This gives us confidence not just in the immediate response, but in the entire workflow end-to-end.

Why This Matters

There’s something deeply satisfying about transforming an anxiety-inducing question into an automated, continuous answer. “Is this environment healthy?” used to require deep investigation. Now it’s a dashboard we can glance at.

Beyond the peace of mind, synthetic monitoring has proven valuable in unexpected ways. Recently, our telemetry showed elevated latency in one particular region. Before anyone could start investigating, AI-powered SRE automation had already correlated the anomaly with a scheduled failover test being run by our performance engineering team. What could have been a 30-minute investigation was resolved in seconds with a clear explanation. The synthetic monitoring data provided the signal, and AI connected the dots. 💪

We hope the service will also help us catch regressions earlier. Issues that might go unnoticed for hours or days in low-traffic environments should now surface within minutes. That faster feedback loop will mean faster fixes and less impact on users.

Looking Ahead

The synthetic monitoring service is now running, continuously verifying that our systems are healthy across multiple environments. It’s not a replacement for other forms of testing or monitoring, but it fills an important gap in our observability story.

If you’re facing similar challenges, I’d encourage you to explore synthetic monitoring. There’s nothing quite like being able to answer “Is it working?” with data instead of hope!

Pagination: Supporting Both Cursor and Page-Based Strategies

Mon, 26 Jan 2026 15:40:35 +0000

Pagination is one of those “solved problems” that keeps coming back to haunt us. Maybe you chose cursor-based pagination because it’s more efficient for large datasets. Maybe your mobile team chose page-based pagination because users want to jump to “page 5.” Now you need to support both, without breaking anyone! 😓

This is pretty much the scenario I faced recently, and so I thought I’d share the approach I took to support multiple pagination strategies in a single API while maintaining backward compatibility.

Pagination Strategy Tradeoffs

Let’s start with why this problem exists. Cursor-based and page-based pagination solve different problems.

Cursor-based pagination (also called keyset pagination):

Uses an opaque token pointing to a specific record
Handles concurrent inserts/deletes gracefully
Efficient for infinite scroll UIs
O(1) database performance regardless of offset

Page-based pagination:

Uses page numbers (page 1, page 2, etc.)
Familiar UI pattern (“Showing page 3 of 10”)
Users can jump to arbitrary pages
Suffers from offset performance issues on large datasets

The general advice tends to be “just pick cursor-based, it’s better.” But these tradeoffs show why that’s not always straightforward. Your API might have been designed for infinite scroll on mobile (cursor-based), but then a dashboard team needs “page 3 of 10” for their UI. Or you inherited a page-based system and now need to scale it without breaking existing clients. Requirements evolve, and sometimes the right answer is to support both.

A Unified Cursor Format

The foundation of this design is that the cursor is just an opaque string to clients. They shouldn’t parse it—they just pass it back. This means we can encode any pagination state inside it.

// Cursor represents pagination state that supports both strategies
type Cursor struct {
    ID   string `json:"id,omitempty"`   // For cursor-based pagination
    Page int64  `json:"page,omitempty"` // For page-based pagination
}

Encoded as base64 JSON, a cursor-based cursor looks like:

eyJpZCI6IjEyMzQ1In0=  →  {"id":"12345"}

And a page-based cursor:

eyJwYWdlIjoyfQ==  →  {"page":2}

Clients see an opaque string, but the server knows exactly what to do.

Decoding with Type Detection

When a request comes in, we need to figure out what kind of pagination the client is using:

type PaginationType int

const (
    PaginationTypeCursor PaginationType = iota
    PaginationTypePage
)

type DecodedCursor struct {
    Type PaginationType
    ID   string // For cursor-based
    Page int64  // For page-based
}

func DecodeCursor(cursor string) DecodedCursor {
    if cursor == "" {
        // Empty cursor = first page, default to cursor-based
        return DecodedCursor{Type: PaginationTypeCursor}
    }

    dec, err := base64.StdEncoding.DecodeString(cursor)
    if err != nil {
        return DecodedCursor{Type: PaginationTypeCursor}
    }

    var cur Cursor
    if err := json.Unmarshal(dec, &cur); err != nil {
        return DecodedCursor{Type: PaginationTypeCursor}
    }

    // Determine type based on which field is populated
    if cur.Page > 0 {
        return DecodedCursor{
            Type: PaginationTypePage,
            Page: cur.Page,
        }
    }
    return DecodedCursor{
        Type: PaginationTypeCursor,
        ID:   cur.ID,
    }
}

If we can’t decode the cursor, we assume cursor-based pagination beginning at the start. It’s generally preferable to degrade gracefully in this way than to throw an error.

Querying the Database

Here’s a basic example of how we go about translating the decoded cursor into actual database queries.

func (s *Store) ListItems(ctx context.Context, cursor DecodedCursor, limit int) ([]Item, error) {
    switch cursor.Type {
    case PaginationTypeCursor:
        return s.listByCursor(ctx, cursor.ID, limit)
    case PaginationTypePage:
        return s.listByPage(ctx, cursor.Page, limit)
    default:
        return s.listByCursor(ctx, "", limit)
    }
}

func (s *Store) listByCursor(ctx context.Context, afterID string, limit int) ([]Item, error) {
    query := `
        SELECT id, name, created_at 
        FROM items 
        WHERE ($1 = '' OR id > $1)
        ORDER BY id ASC
        LIMIT $2
    `
    return s.query(ctx, query, afterID, limit)
}

func (s *Store) listByPage(ctx context.Context, page int64, limit int) ([]Item, error) {
    offset := (page - 1) * int64(limit)
    query := `
        SELECT id, name, created_at 
        FROM items 
        ORDER BY id ASC
        LIMIT $1 OFFSET $2
    `
    return s.query(ctx, query, limit, offset)
}

You’ll notice cursor-based uses a WHERE id > $1 condition (efficient), while page-based uses OFFSET (less efficient for large offsets, but sometimes necessary for the UX).

Preserving Pagination Type Through Response Cycles

Something to watch out for is that the next cursor should use the same pagination type as the current request. If a client starts with page-based pagination, they’ll expect to stay in page-based pagination!

func EncodeNextCursor(currentPage DecodedCursor, lastID string) string {
    switch currentPage.Type {
    case PaginationTypePage:
        // Client is using pages, give them the next page
        return EncodePageCursor(currentPage.Page + 1)
    default:
        // Client is using cursors, give them the next cursor
        return EncodeCursorID(lastID)
    }
}

func EncodePageCursor(page int64) string {
    b, _ := json.Marshal(Cursor{Page: page})
    return base64.StdEncoding.EncodeToString(b)
}

func EncodeCursorID(id string) string {
    b, _ := json.Marshal(Cursor{ID: id})
    return base64.StdEncoding.EncodeToString(b)
}

Performance Considerations

It’s worth being upfront with API consumers about the tradeoffs:

Aspect	Cursor-Based	Page-Based
First page	Fast	Fast
Page 1000	Fast	Slow (OFFSET scan)
Concurrent writes	Consistent	May skip/duplicate
Jump to arbitrary page	Not supported	Supported

If you’re supporting page-based for UX reasons but have large datasets, you may want to consider:

Capping the maximum page number
Suggesting cursor-based for programmatic access
Using estimated counts instead of exact counts for “page X of Y”

Final Thoughts

Supporting multiple pagination strategies doesn’t have to mean duplicating your API. With a type-aware cursor format, I was able to:

Encode pagination state in opaque cursors — clients don’t need to understand the format
Detect pagination type on decode — the cursor tells us how to query
Preserve type through the request cycle — next cursor matches current strategy
Degrade gracefully — unknown/malformed cursors reset to page 1, cursor-based

This allowed the API I was working on to serve different clients with different needs through a single endpoint, without breaking changes. 🎉

The mystery of the decryption errors: A two‑line bug fix

Tue, 16 Dec 2025 10:14:32 +0100

In distributed systems the most unsettling failures are the ones that almost don’t fail. Everything “works,” but a tiny percentage of requests behave strangely. The impact might be low, but the implications can be serious, especially when encryption is involved.

This is the story of a small, gradually increasing rate of decryption errors in a production system, what it took to track them down, and the deceptively simple fix.🕵️

Copilot and AI agents have become a big part of my development and investigation workflow, but due to the nature of this bug it wasn’t very helpful during investigation. However, Copilot did inadvertently lead me to the root cause in the end!

The symptom

The system in question is a fairly typical backend service written in Go, with data stored in MySQL. Some of the data stored is sensitive and so encrypted at rest. It’s not really important what this data is, but if the data can’t be decrypted correctly there’s a fallback path that allows the system to continue functioning.

That fallback mattered, because we saw a small but steady trickle of decryption errors over time. The rate was growing in proportion to the volume of data being decrypted, and at a rate of less than 1% of all requests.

During these events, the encryption library was returning a clear error message:

Ciphertext could not be decrypted.

Even at a very low rate, that kind of error is concerning! It points to either:

the secret key used for decryption is sometimes wrong, or
the stored encrypted bytes are sometimes corrupted (or are corrupted after retrieval).

The first clue

One of the earliest observations was that when the error occurred, within a single request multiple encrypted fields would always fail decryption together. That was important because it suggested the problem wasn’t transitive, in the sense that the decryption could be retried and succeed. When decryption failed for one field, it consistently failed for all fields within that request.

This strongly suggested the problem was either:

something about how values are retrieved for that request, or
memory reuse / mutation within the process.

Narrowing down the bug

The investigation went through the kinds of steps you’d expect when debugging something that could be data corruption.

1) Validate the shape of the retrieved ciphertext

By instrumenting the length of the encrypted bytes coming back from the database, I could check whether truncation or unexpected formatting was involved. The ciphertext lengths were consistently as expected.

So not obviously truncated.

2) Add targeted tests around “encryption enabled/disabled” behaviour

There were cases where encrypted fields might be absent, and the system should interpret that as “encryption disabled for this record.” Tests confirmed that behaviour was correct and stable.

So not a logic bug around “missing encrypted fields.”

3) Correlate with larger result sets

A pattern emerged where most failures happened when fetching multiple records in a single query, generally greater than five rows.

So only manifests when scanning multiple rows.

4) Verify stored data by batch processing

I created a batch job (internally we call them “transitions”) that iterated through all stored rows in the database and attempted decryption. Everything decrypted successfully.

Very reassuring! The encrypted data in storage seemed fine.

5) Confirm the decryption secret wasn’t changing

Instrumentation and small refactors verified that the decryption secret being used in-process was stable and correct.

So not a case of “wrong key.”

The breakthrough

The batch job I created to verify the encrypted data had been scanning the encrypted bytes directly into a []byte. I had used Copilot coding agent to help write the transition quickly, so at the time I didn’t think much of it, but in the main application code the encrypted bytes were being scanned into a different type. That type was a custom type called Bytes that implemented the database/sql scanner interface.

When the batch job was modified to use the same Bytes scanner type, I was suddenly able to reproduce the decryption errors!💡

The bug

Many SQL drivers return []byte for binary columns, and they may reuse the backing array for performance as they iterate through rows. If your Scan method simply assigns that slice then you might be keeping a reference to a buffer that the driver will overwrite when scanning the next row.

b.Bytes = bytes

This is what was happening:

The SQL driver provides a []byte slice when scanning
The Scan method directly assigns this slice to b.Bytes
The SQL driver may reuse the underlying memory for this slice across multiple rows
By the time we try to decrypt the data in one of the rows, the data has been overwritten with data from subsequent rows

The ciphertext length was correct, but the bytes were just wrong! That’s what was producing our intermittent, confusing decryption failures.

The fix

The fix was exactly what you’d expect once you know the behaviour—make a defensive copy in Scan.

func (b *Bytes) Scan(value interface{}) error {
    if value == nil {
        b.Bytes, b.Valid = nil, false
        return nil
    }
    bytes, ok := value.([]byte)
    if !ok {
        return errors.New("failed to assert column value as type []byte")
    }
    b.Valid = true

    // Defensive copy to avoid referencing driver-owned memory
    b.Bytes = make([]byte, len(bytes))
    copy(b.Bytes, bytes)

    return nil
}

So this is a cautionary tale, that when implementing Scan you need to be aware that the []byte you receive may be reused by the driver! ⚠️

Commit signing with GitHub's Git database API

Fri, 20 Sep 2024 16:23:32 +0100

Commit signature verification is a feature where GitHub will mark signed commits as “verified” to give confidence that changes are from a trusted source. Some organizations require commit signing, and enforce it with branch protection rules.

In a recent major version update of create-pull-request, I added the ability to have commits signed with bot-generated tokens. You can read about this feature and its token requirements here.

In this post I want to touch on some of the technical aspects of implementing this feature using GitHub’s REST API endpoints for Git database. There are plenty of articles that deal with the basics of using this API. So, instead, I want to highlight some edge cases and considerations for delivering a well-rounded implementation.

Payload limits

The API has a 40MiB payload limit, which I think applies to all the endpoints, although you are only likely to notice it when creating large git blobs or trees. As of writing, this limit is not officially documented.

For blobs, it means that the size of a single file cannot exceed the limit. If you need to support files larger than the limit, then unfortunately, the Git database API cannot be used.

If you hit the limit when creating a large tree, you can modify the implementation to create a tree in multiple parts. Chunk the tree objects and create a tree per chunk in a chain, each referencing the previous.

  // Chunk the tree objects
  const chunkSize = 100
  const chunkedTreeObjects: TreeObject[][] = Array.from(
    {length: Math.ceil(treeObjects.length / chunkSize)},
    (_, i) => treeObjects.slice(i * chunkSize, i * chunkSize + chunkSize)
  )

  // Create a tree per chunk where the base_tree references the previous chunk
  let treeSha = parentCommit.tree
  for (let i = 0; i < chunkedTreeObjects.length; i++) {
    const {data: tree} = await this.octokit.rest.git.createTree({
      owner,
      repo
      base_tree: treeSha,
      tree: chunkedTreeObjects[i]
    })
    treeSha = tree.sha
  }

Rate limits

For most implementations it will probably make sense to implement blob creation with concurrency. Trees with a large number of blobs could take a long time to process in series. The key to doing this well is to deliberately slow calls enough to not hit GitHub’s rate limits.

There are many ways this could be done, depending on your language and/or framework.

  import pLimit from 'p-limit'

  // Limit async task concurrency for blob creation to 10
  const blobCreationLimit = pLimit(10)
  const treeObjects = await Promise.all(
    commit.changes.map(async ({path, mode, content}) => {
        const {data: blob} = await blobCreationLimit(() =>
          this.octokit.rest.git.createBlob({
            owner,
            repo,
            content,
            encoding: 'base64'
          })
        )
        return <TreeObject>{
          path,
          mode,
          sha: blob.sha,
          type: 'blob'
        }
    })
  )

Empty commits

For some implementations it may be necessary to handle the creation of empty commits. This can be done by setting the tree to the parent commit’s tree.

  // In the case of an empty commit, the tree references the parent's tree
  let treeSha = parentCommit.tree

  const {data: commit} = await this.octokit.rest.git.createCommit({
    owner,
    repo,
    parents: [parentCommit.sha],
    tree: treeSha,
    message: commitMessage
  })

Submodules

To create commits containing changes to a submodules’ commit SHA, create a tree object with type commit.

  return <TreeObject>{
    path,
    mode: '160000',
    sha: submoduleCommitSha,
    type: 'commit'
  }

Symlinks

To handle changes to symlinks, create a blob and make sure to read the symlink’s link. An easy coding mistake would be for the file IO code to follow the symlink path and read the content of the target.

  const content = fs.readlinkSync(path, {encoding: 'buffer'}).toString('base64')

  const {data: blob} = this.octokit.rest.git.createBlob({
    owner,
    repo
    content,
    encoding: 'base64'
  })

Complete example

That wraps up some of the edge cases I experienced when working on my own implementation. Check out create-pull-request for the full source code of the snippets.

How to rewrite Git commit messages non-interactively

Wed, 08 Nov 2023 11:23:13 +0000

I had a situation recently where I needed to rewrite the first commit message of a pull request branch, but I couldn’t use interactive commands because it was going to be automated in a workflow.

To explain the use case more fully, I was writing a GitHub Actions workflow that automatically creates a Jira issue for a pull request. The Jira issue key is then added to the pull request title. In addition, I also wanted to rewrite the first commit message of the pull request branch to include the Jira issue key. This associates it correctly with the Jira issue and allows the commits to be linked to from Jira.

Rewriting Git commit messages

It turns out that there are a number of ways to rewrite Git commit messages, but all of them seem to require interactive commands like git rebase -i or git commit --amend. Eventually I found a promising solution in this Stackoverflow answer.🔍 It emulates the behaviour of the interactive command git commit --fixup=reword:<commit_hash> to make it non-interactive.

The solution they outlined uses a git alias to rewrite the commit message non-interactively. I picked it apart into separate commands that can be run in a script or GitHub Actions workflow.

    NEWLINE=$'\n'
    amend_message="amend! $commit_hash${NEWLINE}${NEWLINE}$new_commit_message"

    git config --global user.email "${{ github.actor_id }}+${{ github.actor }}@users.noreply.github.com"
    git config --global user.name "${{ github.actor }}"

    git commit --allow-empty --only -m "$amend_message"
    GIT_SEQUENCE_EDITOR=: git rebase -i --autosquash "$commit_hash^"
    git push --force-with-lease

Notes:

The snippet above assumes two variables are set. The commit_hash of the commit to rewrite, and new_commit_message containing the string to replace the commit message with.
Getting the newlines to correctly render in the amend! message is tricky. Those lines are very deliberate to make sure the amend commit message is formatted correctly. This includes the double quotes around $amend_message in the git commit command.
The lines setting the user with git config include GitHub Actions context variables. Replace these if you are running the script elsewhere.

Thoughts on "The Staff Engineer's Path"

Fri, 30 Jun 2023 13:35:12 +0900

I recently read “The Staff Engineer’s Path” by Tanya Reilly. This post shares some thoughts and highlights some points I found interesting about the book.

I really enjoyed the book and I highly recommend it to engineers of all levels. Even if you aren’t at staff+ level it will give you a really good idea of what is expected and a “north star” to aim for in terms of professional and personal development. For staff+ level engineers it contains an enormous amount of great practical advice about how to approach the role. The first edition was only published in 2022, but I wish I had the opportunity to read this earlier in my career! I’m definitely going to be dipping into this book occasionally for guidance in future.

The three pillars

There can be a lot of ambiguity about what is expected of staff+ engineers, and even when the expectation is articulated, it can be difficult to know how to achieve it practically. In particular, I think it can be difficult to understand how to scale your impact from being team-level to organization-level.

This book seeks to give engineers a framework and methodology to approach the role. It unpacks the staff+ engineer role into three pillars of, big-picture thinking, execution, and leveling up (others). The pillars require a solid foundation of technical knowledge and experience, and together they will lead to achieving greater impact.

The sections of the book are dedicated to these three pillars. Breaking down the role in this way made the book and its advice feel well organized and understandable.

Big-picture thinking

The first pillar focuses on building the skills to see the big picture. Organizations need engineers that can see the big picture and provide context for making good decisions. If all engineers in the organization only have context relevant to their team’s interests and responsibilities, then decision making will always arrive at the local maxima. This means that solutions will be chosen that may be the best for the immediate scope of the team, but perhaps not the best for the organization as a whole.

The first part of this section focused on defining your role in terms of scope, responsibility and influence. The key take-away for me was that “your job will be a weird shape sometimes, and that’s OK.” There is no one-size-fits-all, and I think the point is that you need to be flexible and adapt to fit what the organization needs for your particular area of responsibility.

“At staff+ levels, your manager should be bringing you information and sharing context, but you should be telling them what’s important just as much as the other way around.”

I thought this was a great point and a good sign that an engineer at this level is building the wider context necessary to be effective in their role.

The section continues with discussing how to “map the terrain” of a role to better understand the context of how your area of responsibility fits into the bigger picture. The key point for me here was to talk with peers outside your group to understand their goals and what success looks like for them.

The final part of this section was about how to document a technical vision and strategy. This is the output of big-picture thinking and will form the basis for decision making. A technical vision is a picture of what we envision our systems to look like in the future. It’s the state that our systems will be in after all the work is done. It describes the destination, but importantly, it doesn’t describe the road to get there!

“(a technical vision) doesn’t set out to make all of the decisions, but it should remove sources of conflict or ambiguity and empower everyone to choose their own path while being confident that they’ll end up at the right place.”

Providing a technical vision with “guardrails” is a much more efficient use of a staff+ engineer’s time than trying to micro-manage every detail. The former approach scales impact much more effectively.

A technical strategy is a plan for how to achieve the vision. The author makes the point that there should be no pressure to innovate here or come up with “insightful game-changing solutions.” Good technical strategy is often pretty boring.

Project execution

This section deals with the second pillar and covers topics like time management, social capital, leading big projects, and dealing with stalled or blocked projects. As much as you might want to, you can’t do everything and be in control of every detail of a large project. Choosing how to spend time to execute on a project can be challenging, and the book does a good job of describing how and what to prioritize.

The author touches on the concept of “social capital,” which is the idea that you bank capital through a mix of trust, credibility, friendship, and favors for your peers. In turn, you can use this capital at opportune moments to help you achieve your goals.

You will build credibility as a professional every time you take on a chaotic situation and make it easier for everyone else to understand.

Using this social capital well is another important way of maximizing your influence in the organization and leveraging it to get things done at scale. Don’t waste your social capital on fights that in the big picture are unimportant, or you don’t really care about.

The book’s advice for leading big projects was also great, pointing out the need for clear definitions of success, driving it rather than letting things happen, and effective communication.

“…, and–the number one tool for success–writing things down.”

When leading big projects there will be times when you are faced with difficulties and are feeling out of your depth. The author suggests that voicing this uncertainty to everyone is generally not a good idea, and that you should find one or two peers that you can be open and unsure with. You need peers that you can be open with about the difficulties and will let you bounce ideas off them for validation.

Leveling up (others)

The final section of the book, dealing with the third pillar, focuses on leveling up others around you and how to be a good influence at scale. I think this was the most helpful section of the book to me personally. I’ve often struggled to understand practically how to achieve that part of my role. This book made me realize that “mentorship” and leveling up others can take many forms, not just limited to activities with direct interaction.

The first chapter of this section is titled “You’re a Role Model Now (Sorry),” and is all about how staff+ engineers are role models for the rest of the organization whether they like it or not. Our words and actions influence the engineering culture of the organization, for good or bad. Reading this section made me realize that being a good role model counts as “mentorship” and can have a powerful ability to level up others around you, often to a greater extent than more deliberate approaches. So on the one hand it made me feel much more confident that I have gone some way to fulfilling the mentorship aspect of my role, but on the other hand it made me realize how much of a responsibility it is to be a good role model. The author made me see that it has greater impact than I previously thought.

The book shares a lot of good advice for being a good role model. The advice is laid out as a general set of principles and good behaviors grouped into these categories:

Be competent
Be responsible
Remember the goal
Look ahead

One point in the “look ahead” section that I particularly liked and rang true with me was to “optimize for maintenance, not creation.” This can often cause the creation of new systems to take longer, but in my experience it’s almost always worth it. Most of our time is spent maintaining systems and if we can reduce that burden we will be able to move faster as an organization.

Another section I thought was great and had good advice was about “being a catalyst” in order to scale your ability to level up others.

“Creating robots, policies, and processes that reinforce your message scales further than being a guardrail for individual colleagues.”

Much of the advice in this section also focused on making sure that others can work autonomously, and are given the opportunities to do so.

Opportunities can be much more valuable than advice. Think about who you’re sponsoring and delegating to. Share the spotlight in your team.

Final thoughts

This was a great book and I highly recommend giving it a read. It’s given me a lot more confidence, and the way the book broke the role down has made it easier to see what areas I would like to focus on personally.

I loved that the advice was clear and practical, and really demystifies the core aspect of the role–how to have impact and influence at scale.

Using Postgres multirange types with jOOQ

Sat, 19 Mar 2022 17:54:23 +0900

PostgreSQL 14 introduces built-in multirange types. The existing range types store the beginning and end value of a single range, but the new multirange types can store a list of non-contiguous ranges.

By default Postgres outputs the built-in range types in a canonical form, where the lower bound is inclusive ([) and the upper bound is exclusive ()).

SELECT id_ranges FROM example_table;
 id_ranges
----------------
 {[1,21),[25,41),[45,51),[55,81)}
(1 row)

When adding a range contiguous with an existing range, the ranges are automatically merged so that it always outputs non-contiguous ranges.

UPDATE example_table SET id_ranges = id_ranges + '{[21, 24]}' RETURNING *;
 id_ranges
----------------
 {[1,41),[45,51),[55,81)}
(1 row)

Using with jOOQ

As of writing, support for multirange types is not yet available in jOOQ. The good news is that jOOQ has support for defining custom bindings and converters to work with these types.

For my use case I needed a binding for the int8multirange type, but this example could easily be modified to apply to other range types.

The first step is to define a data class for the type. The secondary constructor is optional for convenience.

data class Int8MultiRange(
    val value: List<LongRange> = emptyList()
) {
    constructor(vararg ranges: LongRange) : this(ranges.toList())
}

Then create a Converter which handles conversion between the string form of the int8multirange type and our Int8MultiRange class.

class Int8MultiRangeConverter : Converter<String, Int8MultiRange> {
    private val regex = "\\[(.*?),(.*?)\\)".toRegex()

    override fun from(databaseObject: String?): Int8MultiRange? {
        return databaseObject?.let {
            val matches = regex.findAll(databaseObject.toString())
            Int8MultiRange(
                matches.map {
                    val (start, end) = it.destructured
                    LongRange(
                        start.toLong(),
                        end.toLong() - 1
                    )
                }.toList()
            )
        }
    }

    override fun to(userObject: Int8MultiRange?): String? {
        return userObject?.let { ranges ->
            ranges.value.joinToString(separator = ",", prefix = "{", postfix = "}") {
                "[${it.first},${it.last + 1})"
            }
        }
    }

    override fun fromType(): Class<String> {
        return String::class.java
    }

    override fun toType(): Class<Int8MultiRange> {
        return Int8MultiRange::class.java
    }
}

Next, create a jOOQ Binding to make use of our converter.

class Int8MultiRangeBinding : Binding<String, Int8MultiRange> {
    override fun converter(): Converter<String, Int8MultiRange> {
        return Int8MultiRangeConverter()
    }

    override fun sql(ctx: BindingSQLContext<Int8MultiRange>) {
        ctx.render()
            .visit(DSL.`val`(ctx.convert(converter()).value()))
            .sql("::int8multirange")
    }

    ...
}

Finally, we can create a jOOQ field, setting the DataType with our binding.

    private val rangesField = field(
        name("example_table", "id_ranges"),
        SQLDataType.VARCHAR.asConvertedDataType(Int8MultiRangeBinding())
    )

See repository jOOQ-pg-int8multirange for a complete example with tests.

Gaps and islands: Merging contiguous ranges

Wed, 02 Mar 2022 22:16:23 +0900

I recently needed a solution to merge rows of contiguous ranges in a PostgreSQL table. The approach I took was based on solutions to the gaps and islands problem.

Note that you can avoid needing a solution like this if you are able to upgrade to PostgreSQL 14 and take advantage of multirange types. If not, read on!

Requirements for my particular use case:

Find gaps and islands between rows containing a numerical range, expressed as two columns, from_id and to_id.
Merge the islands (rows of contiguous ranges) into a single row.
Perform the merge and update the table in a single SQL transaction to avoid race conditions with concurrent processes.

Solution

This is the table we’ll use for the following examples. set_id is a set of ranges, and the merge operation targets a specific set. The EXCLUDE constraint is added to prevent overlapping ranges from being inserted into the table.

CREATE TABLE ranges (
  set_id integer NOT NULL,
  from_id bigint NOT NULL,
  to_id bigint NOT NULL,
  EXCLUDE USING GIST (
    set_id WITH =,
    int8range(from_id, to_id, '[]') WITH &&
  )
);

Identify islands

Identifying islands is done in two steps. The first step adds the column island_start, marking the start of an island.

SELECT
  *,
  CASE from_id - LAG(ranges.to_id)
      OVER (ORDER BY ranges.from_id ASC)
    WHEN NULL THEN 1
    WHEN 1 THEN 0
    ELSE 1
  END AS island_start
FROM ranges
WHERE set_id = 1;

The query uses the LAG window function to evaluate the previous row, and determine if the current row is the start of an island or not. Since the first row has no previous row, we must check for NULL to handle that case.

Here is an example result, showing the start of four islands have been marked.

set_id	from_id	to_id	island_start
1	1	10	1
1	11	15	0
1	16	20	0
1	25	30	1
1	31	40	0
1	45	50	1
1	55	60	1
1	61	80	0

The next step is to give each island a unique ID, so that we can identify which island each row belongs to.

WITH range_islands AS (
  SELECT
    *,
    CASE from_id - LAG(ranges.to_id)
        OVER (ORDER BY ranges.from_id ASC)
      WHEN NULL THEN 1
      WHEN 1 THEN 0
      ELSE 1
    END AS island_start
  FROM ranges
  WHERE set_id = 1
)
SELECT
  *,
  SUM(range_islands.island_start)
    OVER (ORDER BY range_islands.from_id ASC) AS island_id
FROM range_islands;

The query uses SUM as a windowed function over the island_start column in the result of our previous query. This creates a rolling sum, where each island start row increases the sum by one, giving us a unique ID.

Here is an example result, showing four islands with their unique IDs.

set_id	from_id	to_id	island_start	island_id
1	1	10	1	1
1	11	15	0	1
1	16	20	0	1
1	25	30	1	2
1	31	40	0	2
1	45	50	1	3
1	55	60	1	4
1	61	80	0	4

Merge islands

Once each row has an ID, identifying what island it belongs to, the next step is straightforward. We group by island_id and find the MIN and MAX of the contiguous ranges.

WITH range_islands AS (
  SELECT
    *,
    CASE from_id - LAG(ranges.to_id)
        OVER (ORDER BY ranges.from_id ASC)
      WHEN NULL THEN 1
      WHEN 1 THEN 0
      ELSE 1
    END AS island_start
  FROM ranges
  WHERE set_id = 1
),
range_island_ids AS (
  SELECT
    *,
    SUM(range_islands.island_start)
      OVER (ORDER BY range_islands.from_id ASC) AS island_id
  FROM range_islands
)
SELECT
  set_id,
  MIN(from_id) AS from_id,
  MAX(to_id) AS to_id
FROM range_island_ids
GROUP BY set_id, island_id;

Here is the result, showing the four merged islands.

set_id	from_id	to_id
1	1	20
1	25	40
1	45	50
1	55	80

Update islands

Updating the table with the merged rows takes place in two steps. Firstly, any rows that were identified as not being the start of an island can be deleted.

DELETE FROM ranges
USING range_islands
WHERE
  ranges.set_id = range_islands.set_id AND
  ranges.from_id = range_islands.from_id AND
  range_islands.island_start = 0

Secondly, the remaining rows representing the islands are updated with the to_id of the merged ranges.

UPDATE ranges SET
  to_id = merged_ranges.to_id
FROM merged_ranges
WHERE
  ranges.set_id = merged_ranges.set_id AND
  ranges.from_id = merged_ranges.from_id

That completes all the steps necessary to execute a merge of contiguous ranges in a single PostgreSQL transaction. See gaps-and-islands for a complete example. You can also check out the example in dbfiddle.

kdef: Declarative resource management for Kafka

Wed, 26 Jan 2022 22:35:12 +0900

Introducing kdef, a tool for declarative management of Kafka resources.

kdef aims to provide an easy way to manage resources in a Kafka cluster by having them defined explicitly in a human-readable format. Changes to resource definitions can be reviewed like code and applied to a cluster.

kdef was designed to support being run in a CI-CD environment, allowing teams to manage Kafka resource definitions in source control with pull requests (GitOps).

Features

Definition support for:
- Topics
- ACLs
- Per-broker configs
- Cluster-wide broker configs
YAML and JSON definition formats
TLS and SASL mechanisms (PLAIN, SCRAM, AWS_MSK_IAM)
CLI scripting support (input via stdin, JSON output, etc.)

Highlights

kdef shows a diff of exactly what is changing. This allows a dry-run of the changes to be reviewed easily before being applied.

Example output:

topic definition "tutorial_topic1" diff (local -> remote):
 {
   "apiVersion": "v1",
   "kind": "topic",
   "metadata": {
     "name": "tutorial_topic1"
   },
   "spec": {
     "configs": {
-      "retention.ms": "86400000"
+      "retention.ms": "43200000"
     },
     "deleteUndefinedConfigs": false,
-    "partitions": 3,
+    "partitions": 6,
     "replicationFactor": 2,
     "managedAssignments": {
       "balance": "new",
       "selection": "topic-cluster-use"
     },
     "maintainLeaders": false
   },
   "state": {
     "assignments": [
       [
         1,
         2
       ],
       [
         2,
         1
       ],
       [
         1,
         2
+      ],
+      [
+        2,
+        1
+      ],
+      [
+        1,
+        2
+      ],
+      [
+        2,
+        1
       ]
     ]
   }
 }

kdef is scripting-friendly, supporting features such as input via stdin and JSON output.

Passing a topic definition to kdef via stdin:

cat <<EOF | kdef apply - --dry-run
apiVersion: v1
kind: topic
metadata:
  name: tutorial_topic2
spec:
  configs:
    retention.ms: "86400000"
  partitions: 3
  replicationFactor: 2
EOF

JSON output from kdef being piped to jq:

kdef apply "definitions/topic/*.yml" --dry-run --json-output | jq -r '.[] | "\(.local.metadata.name):\n\(.diff)"'

Documentation

See the manual for installation, configuration and usage instructions.

Automating Gradle dependency updates with GitHub Actions

Fri, 05 Jun 2020 17:34:21 +0900

Using Gradle’s dependency locking feature we can create an automated process to periodically create a pull request for dependency updates.

See an example pull request using the method outlined in this article.

Configuring dependency locking

Firstly, make sure the gradle wrapper is up to date. This is necessary in order to use the feature preview in the next step.
```
gradle wrapper --gradle-version 6.5
```
Enable the ONE_LOCKFILE_PER_PROJECT feature preview in settings.gradle.kts. You can read more about this feature here.
```
rootProject.name = "example-api"

enableFeaturePreview("ONE_LOCKFILE_PER_PROJECT")
```
Add the following section to build.gradle.kts to version lock all configurations. See the documentation here if you would like to customise this for specific configurations.
```
dependencyLocking {
    lockAllConfigurations()
}
```

Optionally, add the following if you would like to create a lockfile for the buildscript section. This can be used to version lock plugins.

buildscript {
    repositories {
        mavenCentral()
        jcenter()
    }
    dependencies {
        classpath("com.jfrog.bintray.gradle:gradle-bintray-plugin:1.8.+")
    }
    configurations.classpath {
        resolutionStrategy.activateDependencyLocking()
    }
}

apply(plugin = "com.jfrog.bintray")

Write a gradle.lockfile for your current dependencies. If you followed step 4, you will also have a buildscript-gradle.lockfile.
```
./gradlew dependencies --write-locks
```
Check the lockfiles into source control. The lockfiles will now make sure that ./gradlew build uses strict versions from the lockfile.
Specify version ranges for your dependencies. The range should include all versions that you are happy to accept version updates for. For example, 1.2.+ for just patch updates, 1.+ for minor updates, and + to include major version updates.

Automate dependency updates

Add the following GitHub Actions workflow to periodically create a pull request containing dependency updates. The following example uses the create-pull-request action and executes once a week.

Note that if you want pull requests created by this action to trigger checks then a repo scoped PAT should be used instead of the default GITHUB_TOKEN. It is highly recommended to make sure checks run and build the new pull request in CI. This will verify that the dependency versions in the new lockfile will build and pass tests.

name: Update Dependencies
on:
  schedule:
    - cron:  '0 1 * * 1'
jobs:
  update-dep:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - uses: actions/setup-java@v1
        with:
          java-version: 1.8
      - name: Grant execute permission for gradlew
        run: chmod +x gradlew
      - name: Perform dependency resolution and write new lockfiles
        run: ./gradlew dependencies --write-locks
      - name: Create Pull Request
        uses: peter-evans/create-pull-request@v5
        with:
            token: ${{ secrets.PAT }}
            commit-message: Update dependencies
            title: Update dependencies
            body: |
              - Dependency updates
  
              Auto-generated by [create-pull-request][1]
          <span class="pl-ent">[1]</span>: <span class="pl-s">https://github.com/peter-evans/create-pull-request</span>
        <span class="pl-ent">branch</span>: <span class="pl-s">update-dependencies</span></pre></div>

See the code in this repository for a complete example.

ChatOps for GitHub Actions

Sun, 05 Jan 2020 16:30:12 +0900

“ChatOps” is a term widely credited to GitHub, referring to the practice of performing operations by typing commands in chat messaging applications.

While GitHub Actions has many ways to trigger workflows based on events that occur in a repository, it doesn’t have a particularly straightforward way to manually trigger a workflow.

Manually triggering workflows

You can trigger workflows manually by configuring them to listen for the repository_dispatch event, and then sending a call to the GitHub API. I found myself using this method a lot to test actions I was developing. The main problem with this was the awkwardness of calling the API using curl.

Another way to trigger workflows is to configure a workflow on the issue_comment event and parse slash commands from the comments. However, in repositories with a lot of activity, the workflow queue gets backed up very quickly trying to handle new issue_comment events and process the commands themselves.

I wanted to develop something that was a combination of these two methods.

Slash Command Dispatch

Slash Command Dispatch is a GitHub action that facilitates “ChatOps” by creating repository dispatch events for slash commands.

The action runs in issue_comment event workflows and checks comments for slash commands. When a valid command is found it creates a repository dispatch event that includes a payload containing full details of the command and its context.

Dispatching commands to be processed elsewhere keeps the workflow queue moving quickly and essentially enables parallel processing of workflows.

Demos

See the demos section of the slash-command-dispatch README for a number of live demos.

Getting started

Follow this guide to get started with a working /example command.

Command processing setup

Create a new repository called, for example, slash-command-processor. This will be the repository that commands are dispatched to for processing.

In your new repository, create the following workflow at .github/workflows/example-command.yml.

name: example-command
 on:
   repository_dispatch:
     types: [example-command]
 jobs:
   example:
     runs-on: ubuntu-latest
     steps:
       - name: Add reaction
         uses: peter-evans/create-or-update-comment@v1
         with:
           token: ${{ secrets.REPO_ACCESS_TOKEN }}
           repository: ${{ github.event.client_payload.github.payload.repository.full_name }}
           comment-id: ${{ github.event.client_payload.github.payload.comment.id }}
           reaction-type: hooray

Create a repo scoped Personal Access Token (PAT) by following this guide.
Go to your repository Settings -> Secrets and Add a new secret.

Name: REPO_ACCESS_TOKEN

Value: (The PAT created in step 3)

Command processing setup is complete! Now we need to setup command dispatch for our /example command.

Command dispatch setup

Choose a repository or create a new repository to dispatch commands from. This will be the repository where issue and pull request comments will be monitored for slash commands.

In the repository, create the following workflow at .github/workflows/slash-command-dispatch.yml.

Note: Change your-github-username/slash-command-processor to reference your command processor repository created in the previous section.

name: Slash Command Dispatch
 on:
   issue_comment:
     types: [created]
 jobs:
   slashCommandDispatch:
     runs-on: ubuntu-latest
     steps:
       - name: Slash Command Dispatch
         uses: peter-evans/slash-command-dispatch@v2
         with:
           token: ${{ secrets.REPO_ACCESS_TOKEN }}
           commands: example
           repository: your-github-username/slash-command-processor

Create a new repo scoped PAT, OR, use the one created at step 3 of the previous section.
Go to your repository Settings -> Secrets and Add a new secret.

Name: REPO_ACCESS_TOKEN

Value: (The PAT created in step 2)

Command dispatch setup is complete! Now let’s test our /example command.

Testing the command

Create a new GitHub Issue in the repository you chose to dispatch commands from.
Add a new comment with the text /example.

Once the command completes you should see all three reactions on your comment.

Now you can start to tweak the command and make it do something useful!

GitHub Actions: How to Automate Code Formatting in Pull Requests

Thu, 17 Oct 2019 20:13:45 +0900

Update: While the approach outlined in this post does work, my current recommendation would be to use a slash command style “ChatOps” solution for operations on pull requests. See slash-command-dispatch for such a solution.

Many programming languages have auto-formatting tools. The most common way to use these is client-side, either using git-hooks to format on pre-commit, or text editor plugins that format on save. Since they run client-side they all rely on engineers setting these tools up correctly. Failing to format before raising pull requests often means that checks will fail and corrections will need to be made. Wouldn’t it be great to have automated code formatting of pull request branches.

Automated code formatting of pull request branches

With a lot of trial and error, I discovered this is possible using an on: pull_request workflow in GitHub Actions. The following example uses the autopep8 action to format Python code.

Important caveat 1: Due to token restrictions on public repository forks these workflows do not work for pull requests raised from forks. Private repositories can be configured to enable workflows from forks to run without restriction.

Important caveat 2: If you have other pull request checks besides the following workflow then you must use a Personal Access Token instead of the default GITHUB_TOKEN. This is due to a deliberate limitation imposed by GitHub Actions that events raised by a workflow (such as push) cannot trigger further workflow runs. This is to prevent accidental “infinite loop” situations, and as an anti-abuse measure. Using a repo scoped Personal Access Token is an approved workaround. See here for further detail.

How it works:

When a pull request is raised the workflow executes as a check.
The code formatter executes and modifies files if necessary.
The workflow checks to see if any tracked files by git have been modified.
If modified files exist they are committed and pushed to the remote.
When using a repo scoped Personal Access Token instead of GITHUB_TOKEN, the push triggers all pull request checks to run again.

name: auto-format
on: pull_request
jobs:
  format:
    # Check if the PR is not from a fork
    if: github.event.pull_request.head.repo.full_name == github.repository
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v1
        with:
          ref: ${{ github.head_ref }}
      - name: autopep8
        uses: peter-evans/autopep8@v1
        with:
          args: --exit-code --recursive --in-place --aggressive --aggressive .
      - name: Check for modified files
        id: git-check
        run: echo "modified=$(if git diff-index --quiet HEAD --; then echo "false"; else echo "true"; fi)" >> $GITHUB_OUTPUT
      - name: Push changes
        if: steps.git-check.outputs.modified == 'true'
        run: |
          git config --global user.name 'Peter Evans'
          git config --global user.email 'peter-evans@users.noreply.github.com'
          git remote set-url origin https://x-access-token:${{ secrets.GITHUB_TOKEN }}@github.com/${{ github.repository }}
          git commit -am "Automated changes"
          git push

Further examples

Automated Python code formatting with Black via Black-action.

name: auto-format
on: pull_request
jobs:
  format:
    # Check if the PR is not from a fork
    if: github.event.pull_request.head.repo.full_name == github.repository
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v1
        with:
          ref: ${{ github.head_ref }}
      - name: black
        uses: lgeiger/black-action@v1.0.1
        with:
          args: .
      - name: Check for modified files
        id: git-check
        run: echo "modified=$(if git diff-index --quiet HEAD --; then echo "false"; else echo "true"; fi)" >> $GITHUB_OUTPUT
      - name: Push changes
        if: steps.git-check.outputs.modified == 'true'
        run: |
          git config --global user.name 'Peter Evans'
          git config --global user.email 'peter-evans@users.noreply.github.com'
          git remote set-url origin https://x-access-token:${{ secrets.GITHUB_TOKEN }}@github.com/${{ github.repository }}
          git commit -am "Automated changes"
          git push

Automated Javascript code formatting with Prettier.

name: auto-format
on: pull_request
jobs:
  format:
    # Check if the PR is not from a fork
    if: github.event.pull_request.head.repo.full_name == github.repository
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v1
        with:
          ref: ${{ github.head_ref }}
      - name: prettier
        run: npx prettier --write src/**/*.js
      - name: Check for modified files
        id: git-check
        run: echo "modified=$(if git diff-index --quiet HEAD --; then echo "false"; else echo "true"; fi)" >> $GITHUB_OUTPUT
      - name: Push changes
        if: steps.git-check.outputs.modified == 'true'
        run: |
          git config --global user.name 'Peter Evans'
          git config --global user.email 'peter-evans@users.noreply.github.com'
          git remote set-url origin https://x-access-token:${{ secrets.GITHUB_TOKEN }}@github.com/${{ github.repository }}
          git commit -am "Automated changes"
          git push

Automated Go code formatting with gofmt.

name: auto-format
on: pull_request
jobs:
  format:
    # Check if the PR is not from a fork
    if: github.event.pull_request.head.repo.full_name == github.repository
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v1
        with:
          ref: ${{ github.head_ref }}
      - name: gofmt
        run: gofmt -s -w .
      - name: Check for modified files
        id: git-check
        run: echo "modified=$(if git diff-index --quiet HEAD --; then echo "false"; else echo "true"; fi)" >> $GITHUB_OUTPUT
      - name: Push changes
        if: steps.git-check.outputs.modified == 'true'
        run: |
          git config --global user.name 'Peter Evans'
          git config --global user.email 'peter-evans@users.noreply.github.com'
          git remote set-url origin https://x-access-token:${{ secrets.GITHUB_TOKEN }}@github.com/${{ github.repository }}
          git commit -am "Automated changes"
          git push

See the following repositories for further details and examples involving automation of pull request workflows.

GitHub Actions: How to Create Pull Requests Automatically

Fri, 11 Oct 2019 19:07:35 +0900

Introducing one of the first GitHub Actions I wrote and published to the GitHub Marketplace. A generic action to automatically create a pull request for changes to your repository in the Actions workspace.

create-pull-request

Changes to a repository in the Actions workspace persist between steps in a workflow. The create-pull-request action is designed to be used in conjunction with other steps that modify or add files to your repository. The local changes will be automatically committed to a new branch and a pull request created.

Create Pull Request action will:

Check for repository changes in the Actions workspace. This includes:
- untracked (new) files
- tracked (modified) files
- commits made during the workflow that have not been pushed
Commit all changes to a new branch, or update an existing pull request branch.
Create a pull request to merge the new branch into the base—the branch checked out in the workflow.

There are many interesting use cases for this type of action, such as…

Process management bots
Synchronization with external data sources
Updating contract definitions (API, Swagger docs, etc.)
Code linting/formatting
Static analysis/fixes
Scheduled process management

Basic example workflow

Here is simple example that adds a dated report to a repository and raises a pull request.

on:
  schedule:
    - cron: '0 10 * * *'
name: create-pull-request workflow
jobs:
  createPullRequest:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - name: Create report file
        run: date +%s > report.txt
      - name: Create Pull Request
        uses: peter-evans/create-pull-request@v5
        with:
          commit-message: Add report file
          title: '[Example] Add report file'
          body: >
            This PR is auto-generated by 
            [create-pull-request](https://github.com/peter-evans/create-pull-request).
          labels: report, automated pr

This is an example of what pull requests created with the action look like on GitHub.

Example workflow to automate periodic dependency updates

This pattern will work well for updating any kind of static content from an external source. The following example workflow executes once a week and will create a pull request for any new npm dependency updates. It works best in combination with a build workflow triggered on push and pull_request. A Personal Access Token (PAT) can be used in order for the creation of the pull request to trigger further workflows. See the documentation here for further details.

name: Update Dependencies
on:
  schedule:
    - cron:  '0 10 * * 1'
jobs:
  update-dep:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - uses: actions/setup-node@v3
        with:
          node-version: '16'
      - name: Update dependencies
        run: |
          npx -p npm-check-updates ncu -u
          npm install
      - name: Create Pull Request
        uses: peter-evans/create-pull-request@v5
        with:
            token: ${{ secrets.PAT }}
            commit-message: Update dependencies
            title: Update dependencies
            body: |
              - Dependency updates
  
              Auto-generated by [create-pull-request][1]
          <span class="pl-ent">[1]</span>: <span class="pl-s">https://github.com/peter-evans/create-pull-request</span>
        <span class="pl-ent">branch</span>: <span class="pl-s">update-dependencies</span></pre></div>

The above workflow works best in combination with a build workflow triggered on push and pull_request.
name: CI
on:
  push:
    branches: [master]
  pull_request:
    branches: [master]
jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - uses: actions/setup-node@v3
        with:
          node-version: 16
      - run: npm ci
      - run: npm run test
      - run: npm run build
Further examples and documentation
For further examples of how create-pull-request action can be used, please see the list of examples in the documentation.
See the create-pull-request action repository for the latest version and full usage details.

Containerising Kotlin with Jib

Sun, 16 Jun 2019 14:01:23 +0900

Writing a Dockerfile to containerise an application can often be a non-trivial task. Many times I’ve spent hours fiddling with different base images and configurations and never being quite satisfied with the result. Well I recently tried Jib, one of Google’s container tools, and I love it! It builds optimised Docker and Open Container Initiative (OCI) spec images for JVM applications. For containerising JVM apps I will definitely try and use Jib where possible in future.

Jib images are distroless!

I had been interested in trying Google’s “Distroless” Docker images for a while, and so it was great to read that Jib uses these as base images by default. “Distroless” images are stripped down to the bare essentials. They contain only the application and its runtime dependencies. There is no bloat from unnecessary programs, shells, package managers, etc.

Jib images are reproducible

The layers, metadata, files and directories are all added to the image by Jib in a consistent order. Additionally, timestamps of all files and directories are set to one second after the Epoch, and the image creation time is set exactly to the Epoch. This makes the image build process deterministic and able to rebuild layers so that they have the exact same digest each time.

Don’t be surprised that it reports the image was created 49+ years ago⁠—it’s for reproducibility!

REPOSITORY                 TAG          IMAGE ID          CREATED           SIZE
peterevans/webservice      0.1          493233af1b87      49 years ago      137MB
peterevans/webservice      0.1.0        493233af1b87      49 years ago      137MB
peterevans/webservice      latest       493233af1b87      49 years ago      137MB

The result is that Jib produces lean, efficient and reproducible images that have a number of benefits.

An accurate diff of changes and provenance between image releases becomes much less of a burden
Reduced attack surface
Improves the signal to noise ratio of vulnerability scanners

Using Jib with Gradle’s Kotlin DSL

Here is a quick introduction of how to use Jib with Gradle’s Kotlin DSL. A complete example project is contained in this repository.

First add the plugin to build.gradle.kts

plugins {
    id("com.google.cloud.tools.jib") version "1.3.0"
}

Add a configuration section for Jib to build.gradle.kts specifying the image name. Don’t forget to prefix with the registry host for pushing to registries other than io.dockerhub.

jib {
    to {
        image = "peterevans/webservice"
    }
}

The jib gradle task will build and push to the registry. You might also need to specify an authentication method. This is the preferred way to build Jib images as it is daemonless. There is no requirement for your CI environment to be running the Docker daemon.

gradle jib

The jibDockerBuild gradle task will build the image and load it into the Docker daemon. This can be useful if during your CI release process you want to smoke test the image after being built but before being pushed to the registry.

gradle jibDockerBuild

Image Configuration

Labels

Labels can be specified as a map under jib.container.labels as follows.

jib {
    container {
        labels = mapOf(
            "maintainer" to "Peter Evans <mail@peterevans.dev>",
            "org.opencontainers.image.title" to "webservice",
            "org.opencontainers.image.description" to "An example webservice",
            "org.opencontainers.image.version" to "$version",
            "org.opencontainers.image.authors" to "Peter Evans <mail@peterevans.dev>",
            "org.opencontainers.image.url" to "https://github.com/peter-evans/kotlin-jib",
            "org.opencontainers.image.vendor" to "https://peterevans.dev",
            "org.opencontainers.image.licenses" to "MIT"
        )
    }
}

Further Container Configuration

You can customise almost everything about the image including JVM flags, environment variables, volumes, ports, etc. Further configuration options can be found in the documentation here.

jib {
    container {
        jvmFlags = listOf(
            "-server",
            "-Djava.awt.headless=true",
            "-XX:InitialRAMFraction=2",
            "-XX:MinRAMFraction=2",
            "-XX:MaxRAMFraction=2",
            "-XX:+UseG1GC",
            "-XX:MaxGCPauseMillis=100",
            "-XX:+UseStringDeduplication"
        )
        environment = mapOf(
            "USERNAME" to "user1",
            "PASSWORD" to "1234")
        workingDirectory = "/webservice"
        volumes = listOf("/data")
        ports = listOf("8080")
        args = listOf("--help")
    }
}

See the code in this repository for a more complete example of using Jib to containerise Kotlin.

Smoke Testing Containers

Thu, 14 Mar 2019 21:18:44 +0900

Smoke testing is a class of testing designed to determine if something is ready for more rigorous testing. The terminology appears to have originated from plumbing where smoke is used to test for leaks in a closed system of pipes. It also seems to be widely used in electronics to refer to the practice of turning on a new piece of hardware for the first time and considering it a success if none of the components overheat and start to smoke.

In software engineering, smoke testing is used to reveal basic failures that can allow us to immediately reject a software build. It is also sometimes called “Build Verification Testing,” “Confidence Testing,” or “Sanity Testing.” The tests are usually a fast, simple check of major functionality to confirm that a build is ready to be subject to further testing. It can prevent flawed artifacts being released into QA environments, wasting both time and resources.

With smoke tests we can address basic questions, such as…

Does my component/service run?
Are interfaces accessible?
Do the main features appear to work?

But I already have unit tests!

Unit tests and other types of testing that run before the final artifact is produced will not catch problems with packaging. Smoke tests are executed against the final artifact that will be published and/or deployed to a QA environment. In some cases the artifact will just be a compiled binary and there might be very little that can go wrong. Containers, however, are a minefield of dependencies and security updates and it can be difficult to have a high degree of confidence about your final container image without testing it! Of course we can test it locally, but it should be automated in any good CI/CD pipeline. We should be able to update our Dockerfile and git push knowing that if we have broken something obvious it will be caught. This is where smoke tests can save us from causing a serious fire!

Smoke testing Docker containers with CircleCI

There are various ways to smoke test containers but I will outline a couple of techniques using CircleCI. CircleCI is probably the trickiest I’ve experienced due to the way they isolate the job container from the Docker build when using the docker executor type. There is an option to avoid this by using the machine executor type that runs jobs in a dedicated, ephemeral VM. However, they warn that this method may incur additional fees in a future pricing update. So the following techniques assume use of the docker executor type.

Networking containers

To run Docker commands you must specify the setup_remote_docker key to create a remote Docker environment. This is completely isolated from your job container so if, for example, you attempt to curl a running Docker container you will find it has no connectivity.

curl: (7) Failed to connect to localhost port 8080: Connection refused

To run our smoke tests, in whatever form they take, we need to package them in a container of their own and network them to the container we want to test. Here are two examples using publicly available Docker images to verify our service is running and returning a response from the /healthcheck endpoint.

An example using the Docker image appropriate/curl to execute a curl request.

- run:
    name: Start the service and perform healthcheck
    command: |
      docker run -d –name my-service my-service
      docker run –network container:my-service appropriate/curl –retry 10 –retry-connrefused http://localhost:8080/healthcheck

An similar example using the Docker image jwilder/dockerize to wait for a response from the healthcheck endpoint.

- run:
    name: Start the service and perform healthcheck
    command: |
      docker run -d –name my-service my-service
      docker run –network container:my-service jwilder/dockerize -wait http://localhost:8080/healthcheck -timeout 120s -wait-retry-interval 5s

Custom smoke tests

Using curl or dockerize to simply check for a 200 OK response will most likely not be enough to adequately smoke test our container. Those tools do, however, perform a necessary role of making sure that the service is up and running before we execute further tests.

Smoke tests can take whatever form you like provided they are executable via a container. The following example demonstrates executing a Postman collection of smoke tests with the postman/newman Docker image. Note that in order to make our test files (in this case a JSON Postman collection) available to the test executor we create a separate container to store them. This container defines a volume at the path we want our test files to be accessible in the executor container. That volume is then mounted in the executor container using the --volumes-from flag to make the test files available at runtime.

- run:
    name: Execute smoke tests
    command: |
      docker run -d --name my-service my-service
      # Create a container called "smoke-tests" to store our smoke test files
      docker create -v /etc/newman --name smoke-tests alpine:3.4 /bin/true
      # Copy test files from local directory 'smoke-tests' to the container
      docker cp smoke-tests/. smoke-tests:/etc/newman
      # Wait for service to be up and running
      docker run --network container:my-service jwilder/dockerize -wait http://localhost:8080/healthcheck -timeout 120s -wait-retry-interval 5s
      # Run smoke tests
      docker run --network container:my-service --volumes-from smoke-tests -t postman/newman:4.4.0-alpine run my-service.postman_collection.json

See the code in this repository for a complete example of these techniques.

Mutation Testing

Tue, 31 Jul 2018 15:48:26 +0900

Mutation testing is a type of testing designed to assess the quality of unit tests. This method is also sometimes described as “fault based testing” as it deliberately creates faults in software.

How mutation testing frameworks work

Small syntactic changes are made to the application code. Each change, or mutation, is applied to a separate copy of the code thus creating many versions. These versions of the application code are described as “mutants.”
The unit test suite is run against the original application code and all the mutant copies.
The unit test results are compared between the original application code and mutants. If the unit test results are different then that mutant is said to be “killed.” If the unit test results are the same, the mutant is said to have “survived.”
The tally of killed and survived mutants is displayed along with the code mutation for each surviving mutant.

Here is an example where an if statement is modified to always equate to false. Despite this change the unit test results did not change. This indicates the unit testing is poor and not covering all logical cases.

Mutator: IfStatement
-       if(result.length < arr[x].length) {
+       if(false) {

Ran all tests for this mutant.
Mutant survived!

Why use mutation testing?

For engineers, mutation testing is a great tool to support the development of unit test suites. In both Test Driven Development (TDD) and writing unit tests after development, mutation testing helps fill in gaps in test suites. Furthermore, difficult to kill mutations can often highlight problems with the code itself. This forces engineers to rethink their logic and refactor, ultimately leading to higher quality code.

Mutation testing can also be beneficial for managers or anybody involved in evaluating code quality and individual performance. In an ideal world, all code should be peer reviewed to a high standard, but it can be difficult to achieve and maintain within a team. Code quality is often crudely evaluated by the unit test suite coverage of a codebase. However, code coverage is a very poor indictor of the quality of the codebase and its test suite. Mutation testing can provide a metric, albeit not perfect, for the quality of unit testing that is verifiable without needing to look directly at the code itself.

Mutation Testing Frameworks

See here for a good list of frameworks for many different languages.

Mutation testing with Stryker

The following is an example using Stryker, a mutation testing framework for the JavaScript ecosystem. The System under test (SUT) is a Node.js module containing a simple function that returns the longest word in a sentence.

module.exports.longestWord = function(str) {
  if (typeof str !== "string") return "";

  var arr = str.match(/\w[a-z]{0,}/gi);
  if (arr === null || arr.length === 0) return "";

  var result = arr[0];
  for(var x = 1; x < arr.length; x++) {
    if(result.length < arr[x].length) {
      result = arr[x];
    } 
  }
  return result;
}

The following test alone provides close to 100% code coverage but doesn’t actually verify that the function does what it is supposed to do. This test with no assertions is an extreme example and would no doubt be caught by code review, but just shows how easy it is to create poor unit tests.

  it('should return the longest word', function(done) {
    var sentence = 'Ladies and gentlemen, the truth is that mutants are very real, and that they are among us.';
    stringFormatter.longestWord(sentence);
    done();
  });

Running the Stryker mutation test framework reveals that the unit testing is very poor. 20 mutants survived!

Ran 1.00 tests per mutant on average.
----------|---------|----------|-----------|------------|----------|---------|
File      | % score | # killed | # timeout | # survived | # no cov | # error |
----------|---------|----------|-----------|------------|----------|---------|
All files |    9.09 |        2 |         0 |         20 |        0 |       0 |
index.js  |    9.09 |        2 |         0 |         20 |        0 |       0 |
----------|---------|----------|-----------|------------|----------|---------|

Killing those 20 mutants required an additional four unit tests as shown below. Further unit tests could be written but for the purposes of this example only those required to kill the mutants are shown.

describe('longest word', function() {

  it('should return the longest word', function(done) {
    var sentence = 'Ladies and gentlemen, the truth is that mutants are very real, and that they are among us.';
    stringFormatter.longestWord(sentence);
    done();
  });

  it('should return the longest word', function(done) {
    var sentence = 'Ladies and gentlemen, the truth is that mutants are very real, and that they are among us.';
    var result = stringFormatter.longestWord(sentence);
    assert.equal(result, 'gentlemen');
    done();
  });

  it('should return the first longest word found', function(done) {
    var result = stringFormatter.longestWord('mutation testing gnitset noitatum');
    assert.equal(result, 'mutation');
    done();
  });

  it('should return an empty string when an empty string is provided', function(done) {
    var result = stringFormatter.longestWord('');
    assert.equal(result, '');
    done();
  });

  it('should return an empty string for non-string input', function(done) {
    var result = stringFormatter.longestWord(123);
    assert.equal(result, '');
    done();
  });

});

Mutation testing in production

If you have a large codebase you will soon find that testing against hundreds or thousands of mutants takes a long time! Rather than hooking it directly in to your CI pipeline you might consider running the process once per night.

Sample code for this example can be found at https://github.com/peter-evans/mutation-testing

Lightweight Architecture Decision Records

Thu, 24 May 2018 15:47:59 +0900

“Lightweight Architecture Decision Records is a technique for capturing important architectural decisions along with their context and consequences.”

— “Lightweight Architecture Decision Records”, ThoughtWorks Technology Radar

Making architecture and design decisions

In my experience architecture decisions are often made verbally while gathered around a whiteboard. This is a great way to hash out the design while you’re still exploring the architecture. However, if no record of that process is made six months later most people will be hard pressed to remember the full context and reason why decisions were made and important details will be forgotten.

When joining a team with established systems and software it’s often very easy to criticise the way it was designed. However, those decisions may well have been the best course of action given the context and constraints at the time. Documenting these architecture and design decisions and the context they were made in can be a valuable record for future team members and external oversight.

These days many teams favour highly readable code and tests rather than writing extensive documentation. Few engineers enjoy writing documentation so what is the minimum we need to document to maintain an effective record of architecture decisions?

Lightweight Architecture Decision Records

The idea of recording architecture decisions has been written about in various forms for a while, but a version was proposed in this excellent blog post by Michael Nygard.

“An architecture decision record is a short text file in a format similar to an Alexandrian pattern that describes a set of forces and a single decision in response to those forces.”

He proposes a short document with just five sections:

Title
Context
Decision
Status [Proposed, Accepted, Deprecated, Superseded]
Consequences

The beauty of this format is its simplicity. It requires very little training and even engineers skeptical of the value of recording architecture decisions will be able to accept the low effort required.

Here is an example template.

# 1. Title

## Status

[Proposed, Accepted, Deprecated, Superseded]

## Context

## Decision

## Consequences

Tips for using Lightweight Architecture Decision Records

Use markdown and store along with the component it relates to in source control
Number the files sequentially
Keep it brief and use plain, easy to understand language
Peer review as you would code
For cross cutting decisions that affect multiple components consider making a separate “architecture” repository
When you make a decision document it immediately!

Real-world example

This is a real-world example of using Lightweight Architecture Decision Records by GOV.UK for their migration to AWS. https://github.com/alphagov/govuk-aws/tree/master/docs/architecture/decisions

How to Host Swagger Documentation With Github Pages

Wed, 02 May 2018 15:23:13 +0900

This article describes how use the Swagger UI to dynamically generate beautiful documentation for your API and host it for free with GitHub Pages.

An example API specification can be seen hosted at https://peter-evans.github.io/swagger-github-pages.

Steps

Download the latest stable release of the Swagger UI here.
Extract the contents and copy the “dist” directory to the root of your repository.
Move the file “index.html” from the directory “dist” to the root of your repository.
```
mv dist/index.html .
```
Copy the YAML specification file for your API to the root of your repository.

Edit index.html and change the url property to reference your local YAML file.

    const ui = SwaggerUIBundle({
        url: "swagger.yaml",
    ...

Then fix any references to files in the “dist” directory.

...
<link rel="stylesheet" type="text/css" href="dist/swagger-ui.css" >
<link rel="icon" type="image/png" href="dist/favicon-32x32.png" sizes="32x32" />
<link rel="icon" type="image/png" href="dist/favicon-16x16.png" sizes="16x16" />    
...
<script src="dist/swagger-ui-bundle.js"> </script>
<script src="dist/swagger-ui-standalone-preset.js"> </script>    
...

Go to the settings for your repository at https://github.com/{github-username}/{repository-name}/settings and enable GitHub Pages.
Browse to the Swagger documentation at https://{github-username}.github.io/{repository-name}/.

The sample code and API specification can be found at https://github.com/peter-evans/swagger-github-pages.

Candidate Selection Using Iterative Soft-Thresholding

Wed, 21 Jun 2017 15:47:07 +0900

This article describes one way to use soft-thresholding to select the statistically best candidates from a sorted list. This algorithm was introduced to me as an alternative to setting a hard threshold, i.e. selecting a fixed number of the best candidates. Using an iterative soft-thresholding algorithm a variable number of candidates can be selected depending on the distribution of the values.

In the following example the best candidates are selected from a sorted list. Setting a hard threshold of three will of course always select the top three candidates. However, it is clear from looking at the distribution of the values that only the top two could be considered as candidates. This soft-thresholding algorithm allows us to select just those candidates.

How the algorithm works

In each iteration the algorithm compares the mean and the median of the values remaining in the list. Any values higher than the minimum of the mean and median are discarded. The process is repeated until exit conditions are satisfied or until there is only one value remaining.

Sample code

The following sample python code is a simple example to demonstrate how iterative soft-thresholding can be implemented. The sorted list values are randomly generated on each execution of the script. Executing a number of times shows how the number of selected candidates varies based on the distribution.

import numpy as np
import numpy.random as npr

# The maximum number of candidates to select
max_candidates = 3
# The sorted list
sorted_list = sorted(npr.random_integers(1, 100, size=25))
print("Sorted list of candidates:", sorted_list)

while len(sorted_list) > 1:
    # Output remaining candidates
    print("Remaining candidates:", len(sorted_list))

    # Calculate the mean, median and standard deviation of the list
    mean = np.mean(sorted_list)
    median = np.median(sorted_list)
    std = np.std(sorted_list)

    # If the standard deviation is zero the list values are identical
    if std == 0:
        sorted_list = sorted_list[:max_candidates]
        break

    # Stop iterating if the exit conditions for the distribution are met
    if (abs(mean - median) < 0.1 * max(mean, median)) & (std < 0.5 * mean):
        if len(sorted_list) <= max_candidates:
            break

    # Remove any values less or equal to the minimum of the mean and median
    sorted_list = [_ for _ in sorted_list if _ <= min(mean, median)]


# Results after soft threshold iterations
print("=" * 24)
print("Selected candidates:", sorted_list)

One candidate is selected:

~$ python soft_thresholding.py
Sorted list of candidates: [2, 10, 11, 20, 22, 23, 27, 29, 35, 39, 43, 44, 49, 57, 58, 61, 65, 66, 68, 83, 83, 91, 94, 94, 99]
Remaining candidates: 25
Remaining candidates: 13
Remaining candidates: 7
Remaining candidates: 3
========================
Selected candidates: [2]

Two candidates are selected:

~$ python soft_thresholding.py
Sorted list of candidates: [1, 2, 11, 12, 12, 27, 32, 34, 35, 37, 38, 44, 46, 48, 50, 59, 60, 60, 62, 71, 71, 75, 77, 80, 91]
Remaining candidates: 25
Remaining candidates: 12
Remaining candidates: 5
Remaining candidates: 2
========================
Selected candidates: [1, 2]

Three candidates are selected:

~$ python soft_thresholding.py
Sorted list of candidates: [2, 3, 4, 5, 5, 6, 12, 12, 16, 17, 20, 21, 26, 27, 32, 34, 41, 53, 55, 58, 59, 61, 72, 86, 96]
Remaining candidates: 25
Remaining candidates: 13
Remaining candidates: 6
Remaining candidates: 3
========================
Selected candidates: [2, 3, 4]

Fine tuning

The maximum number of candidates can be modified in the sample code. The output of the algorithm will be any number of candidates up to this value.

max_candidates = 3

The algorithm will continue to iterate until the exit conditions are satisfied. These can be fine tuned to be less or more sensitive. In general, if the candidates are very close in value then we want to stop iterating because all of them will be good potential candidates. If the distribution is sparse then we want to keep iterating.

These are the exit conditions for asymmetrical and symmetrical distributions in the sample code.

abs(mean - median) < 0.1 * max(mean, median)

std < 0.5 * mean

The fixed values of 0.1 and 0.5 allow the algorithm to be tuned. Decreasing these values will make the exit condition less sensitive and the algorithm will keep iterating. Increasing the value will cause the algorithm to exit sooner.

How to Wait for Container X Before Starting Y

Sun, 05 Mar 2017 15:08:24 +0900

The healthcheck property was originally introduced in the 2.1 Compose file format and is now part of the Compose Specification used by recent versions of Docker Compose. This allows a check to be configured in order to determine whether or not containers for a service are “healthy.”

How can I wait for container X before starting Y?

This is a common problem and in earlier versions of docker-compose requires the use of additional tools and scripts such as wait-for-it and dockerize. Using the healthcheck parameter the use of these additional tools and scripts is often no longer necessary.

Waiting for PostgreSQL to be “healthy”

A particularly common use case is a service that depends on a database, such as PostgreSQL. We can configure docker-compose to wait for the PostgreSQL container to startup and be ready to accept requests before continuing.

The following healthcheck has been configured to periodically check if PostgreSQL is ready using the pg_isready command. See the documentation for the pg_isready command here.

healthcheck:
  test: ["CMD-SHELL", "pg_isready -U postgres"]
  interval: 10s
  timeout: 5s
  retries: 5

If the check is successful the container will be marked as healthy. Until then it will remain in an unhealthy state. For more details about the healthcheck parameters interval, timeout and retries see the documentation here.

Services that depend on PostgreSQL can then be configured with the depends_on parameter as follows:

depends_on:
  postgres-database:
    condition: service_healthy

Waiting for PostgreSQL before starting Kong

In this complete example docker-compose waits for the PostgreSQL service to be “healthy” before starting Kong, an open-source API gateway. It also waits for an additional ephemeral container to complete Kong’s database migration process.

version: '3.9'  # optional since Compose v1.27.0

services:
  kong-database:
    image: postgres:9.5
    container_name: kong-postgres
    environment:
      - POSTGRES_USER=kong
      - POSTGRES_DB=kong
      - POSTGRES_HOST_AUTH_METHOD=trust
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U postgres"]
      interval: 10s
      timeout: 5s
      retries: 5

  kong-migration:
    image: kong
    container_name: kong-migration
    depends_on:
      kong-database:
        condition: service_healthy
    environment:
      - KONG_DATABASE=postgres
      - KONG_PG_HOST=kong-database
    command: kong migrations bootstrap

  kong:
    image: kong
    container_name: kong
    restart: always
    depends_on:
      kong-database:
        condition: service_healthy
      kong-migration:
        condition: service_started
    links:
      - kong-database:kong-database
    ports:
      - 8000:8000
      - 8443:8443
      - 8001:8001
      - 8444:8444
    environment:
      - KONG_DATABASE=postgres
      - KONG_PG_HOST=kong-database
      - KONG_PG_DATABASE=kong
      - KONG_ADMIN_LISTEN=0.0.0.0:8001

Test it out with:

docker-compose up -d

Wait until all services are running:

Test by querying Kong’s admin endpoint:

curl http://localhost:8001/

Sample code can be found at https://github.com/peter-evans/docker-compose-healthcheck