Kuranto itself runs in-process alongside the integration test suite, so that we do not need to build an image for it, and deploy it in the Kind cluster.

Once the Kind setup is done, the overall testing process is pretty simple, and runs as Rust integration tests:

Talk to the Forgejo API to create a repository
Create a workflow from a fixture file
(Optionally) Create files in the repository to support the workflow execution
Dispatch a workflow run
Check the result of the triggered run

Fixtures are located under fixtures/, tests are inside e2e.rs.

Running tests #

If you have mise installed, just run mise e2e.

Otherwise, run ./scripts/e2e.sh from the root of the repository. You'll need a few dependencies:

kind
kubectl
jq
curl
envsubst (either the one from gettext, or the Go version packaged in mise is fine)
OpenSSH for ssh and ssh-keygen
Either shasum or sha256sum

Either way, this will create a local Kind cluster, deploy Forgejo and the runner, then run the Rust tests. Once the initial setup is done, you can re-run the script again, which will reuse the existing cluster.

Writing tests and workflows #

There are a few rules to follow when writing workflows for tests:

The workflow needs to be manually runnable, meaning it needs a trigger on workflow_dispatch
Jobs need to select kubernetes as the label for runs-on

Also, a few macros are available to simplify the Rust writing process, we'll go through a few kinds of tests showing them.

Once you've written your workflow and its associated test, you need to add it to the test suite in workflow_suite(), similarly to existing tests.

Standalone workflows #

These are workflows that can run without any other supporting files in the repository. Here's an example of such a workflow:

name: Basic

on:
  workflow_dispatch:

jobs:
  build:
    runs-on: kubernetes
    steps:
      - name: It works
        run: echo "it works"

Given this workflow written in a basic.yml file in fixtures/, we can then create the corresponding test in e2e.rs:

async fn runs_basic_workflow() {
    run_standalone_workflow!("basic.yml", Status::Success);
}

That's it. The macro will read the fixture, create a repository, create the workflow, run it, and ensure it succeeds.

Workflows with supporting files #

These are, for example, workflows that build code, read files, etc. A simple example is one that read a file:

name: Checkout

on:
  workflow_dispatch:

jobs:
  build:
    runs-on: kubernetes
    steps:
      - uses: actions/checkout@v7
      - name: It works
        run: |
          cat test.txt | grep test

Here's how we can write its corresponding test:

async fn runs_workflow_with_checkout_action() {
    let (forgejo, repository_name) = setup_workflow!("checkout.yml");
    forgejo
        .create_file(&repository_name, "test.txt", "test")
        .await;
    ensure_workflow!(forgejo, repository_name, "checkout.yml", Status::Success);
}

Here, we split the process into three parts:

Create the repository, write the workflow file; done by setup_workflow!
Create the file we need for the workflow
Trigger a run and making sure it succeeds; done by ensure_workflow!

Debugging #

After the script exits, you can export a new kubeconfig for the Kind cluster and inspect the cluster:

kind get kubeconfig --name kuranto-e2e > /tmp/kuranto-e2e.kubeconfig
export KUBECONFIG=/tmp/kuranto-e2e.kubeconfig
kubectl -n forgejo-e2e get pods
kubectl -n forgejo-e2e logs deployment/forgejo-runner -c runner
kubectl -n forgejo-e2e logs deployment/forgejo-runner -c ssh-relay
kubectl -n forgejo-e2e describe deployment forgejo-runner
kubectl -n forgejo-e2e port-forward --address 127.0.0.1 service/forgejo 3000:80

You can access Forgejo at http://127.0.0.1:3000 to inspect repositories and workflow runs.

You can also delete the entire cluster if you need to: kind delete cluster --name kuranto-e2e.

Why Kind #

Testcontainers #

So, initially, this started as testcontainers. Each test would spawn a Forgejo testcontainer, as well as a K3s testcontainer, then deploy everything in it, then run the test itself.

This worked but it was by far a lot more complex than what we have now:

There were more than a thousand lines of code added to what we currently have, just to spawn the containers themselves
We needed to have a registry testcontainer to push our E2E image to
It was a little bit brittle and some hacks were needed to make everything communicate correctly (especially as I'm writing this from macOS and Docker runs in a VM...)
It was pretty slow
Debugging was a nightmare, everything was cleaned up on failure, you would need to sprinkle sleeps in your test, fetch the temporary kube config, etc.
- This one might have been a skill issue, I didn't have the courage to investigate this further

Minikube #

After testcontainers, we moved to Minikube, either running in a VM for macOS, or in Docker for Linux. It worked a lot better than testcontainers, but still had its issues.

It was still pretty slow, but you only paid most of its cost once, then you could run multiple batches of tests pretty quickly. The testcontainers setup would pay this cost every run (again it might have been possible to reuse containers, but I didn't feel like continuing with this setup at that point).

The downside to using Minikube is that we accumulate a little bit of cruft along the way, since we never clean up repositories, but that's a pretty small issue (especially since we can just recreate the containers without recreating minikube itself and we start from a blank state again) compared to all the benefits we got:

One-time setup
Easy debugging, just go onto the deployed Forgejo and see everything happening
Connect to Minikube easily to debug deployment issues if any

Now #

Now we're using Kind, which is a little bit faster to setup than Minikube is in my testing (not by much but still), and most importantly, we run Kuranto in-process in the integration test suite, and we don't build its Docker image for the tests.

This has made a "cold start" E2E test run much faster than it did before, going from 10-15 minutes to around 4 minutes (on my computer at least)