Reporting your CI results to Button

Run your tests on your own machines and send the results to your project on btn.dev, where everyone who can see the project can see which commits passed, on which machines and with which Pythons.

What you need

• A project on btn.dev, and to be an owner of its org. Owners make the tokens below.
• The client, client.py: one Python file that needs Python 3.11 or later and nothing else. It is in the Button repository at packages/button_ci/src/button_ci/client.py. Keep a copy outside your project, for example in ~/.local/share/button-ci/.

The commands on this page write button-ci for the client. Either make an alias:

alias button-ci='python3 ~/.local/share/button-ci/client.py'

or type python3 ~/.local/share/button-ci/client.py wherever this page says button-ci.

1. Link the repository

On your project's Settings page, under Repositories, give the repository's URL, for example https://github.com/you/your-project, and press Link repository. Its results are kept against that URL.

2. Make a seat token

A seat is a machine that runs the tests: your laptop, a desktop, a server. On the same Settings page, under CI seat tokens, give the seat a name (laptop) and press Make token.

The token starts btnci_ and is shown once only, so copy it now. Save it in a file that only you can read, outside every repository:

mkdir -p ~/.config/button-ci
pbpaste > ~/.config/button-ci/your-project.token   # or paste it in with an editor
chmod 600 ~/.config/button-ci/your-project.token

The page that shows the token also shows the settings for each of the project's repositories, ready to copy.

Anyone holding a token can post results for the project, so give one only to people you trust to run your tests, one token per seat. If a token leaks, Revoke it on the Settings page and make another; a revoked token stops working at once.

3. Set the environment

On the machine that runs the tests, in its shell profile or wherever you keep such settings:

export BUTTON_CI_REPORT=https://btn.dev/your-org/your-project/-/ci/github.com/you/your-project/-/results/
export BUTTON_CI_TOKEN_FILE=~/.config/button-ci/your-project.token

BUTTON_CI_REPORT is the address the token page showed you. You can give the token itself in BUTTON_CI_TOKEN instead of a file. If your checkout's origin is a path on disk rather than a URL, also set BUTTON_CI_REPOSITORY=https://github.com/you/your-project.

The client names the machine by its short host name. To choose another name, set BUTTON_CI_MACHINE.

4. Run your tests through the client

Put button-ci run and the name of the check in front of the command you already use, with -- between them:

button-ci run suite -- pytest
button-ci run lint -- ruff check .

The command runs as it always does and the client exits with its exit code, so scripts and hooks still see a failure. The client records one result for the commit you are on. It does not touch the network.

The result counts only if your checkout is clean: with changes to tracked files that are not committed, what was tested is not any commit, so the result is kept on your machine and never sent.

Counts of tests. If the command writes a JUnit XML report, the result carries how many tests passed, failed, errored and were skipped, and which failed. With pytest, add --junitxml:

button-ci run suite -- pytest --junitxml=/tmp/junit.xml

Without a report you still get passed or failed, from the exit code.

With tox. Each tox environment is a cell of the matrix. Either run each one through the client, naming the cell:

for cell in py312 py313 py314; do
    button-ci run suite --cell $cell -- tox -e $cell -- --junitxml=/tmp/$cell.xml
done

(the --junitxml after the second -- reaches pytest if your tox.ini passes {posargs} on), or change the test command in your tox.ini once:

[testenv]
pass_env = BUTTON_CI_*
commands = {env:BUTTON_CI_RUN:} pytest {posargs}

and run:

BUTTON_CI_RUN="python3 $HOME/.local/share/button-ci/client.py run suite --" tox -p

The client takes each cell's name from tox. With BUTTON_CI_RUN unset, tox runs pytest as before.

Declare your matrix. So that a cell nobody ran does not look like a pass, tell the client which cells the project has. With tox, keep the environments that are cells of the check:

export BUTTON_CI_MATRIX="$(tox -l | grep '^py')"

5. Send the results

button-ci submit

This sends every result not yet sent. If btn.dev does not answer, nothing is lost: the results stay on your machine and the next submit sends them. To see what your machine has recorded for the current commit, run button-ci status.

6. Read the results

• On your project's summary, the Repositories box shows each repository's most recent commit with results, and whether it passed.
• The repository's page lists its commits, newest first. Each check shows one result per cell, machine and kind of run, with its counts; click one for the details.
• A commit's page shows every result for that commit, and the names of the failing tests.

A commit is green when every result shown passed, and red when any failed. It is incomplete when nothing failed but a declared cell has no result; that cell shows dashed, as "declared, not run". If you run a check again on the same machine, the newer result is the one shown, and the older one is kept behind it.

You may also see these marks:

• results disagree: one machine passed a cell and another failed it.
• N declared cells not run: as above.
• runner found red and no seat result: about runs made by an automatic runner rather than on someone's own machine.

Who sees the results

The results are as visible as the project. On a public project anyone can read them, including each result's machine name, platform and failing test names. On a members-only project, only the people who can see the project can. Posting needs a seat token, whoever you are: being signed in is not enough.