Documentation

Overview

sphinx-ci generates an AI quiz from each Pull Request diff. The developer proves they understand their own code before the merge — or they go back and read it again. Comment @sphinx-ci on a PR, the Sphinx generates a quiz from the diff, the dev answers, and the merge is unlocked or blocked based on the score.

How it works

1. 1GitHub Action sends the PR diff to the hub when a comment matches the trigger keyword.
2. 2The hub generates N questions via Claude using your own Anthropic API key.
3. 3A status check is posted on the commit, and a comment with the quiz link is posted on the PR.
4. 4The developer answers in the browser. The score is calculated server-side.
5. 5The status check is updated to success or failure based on the score.

Quickstart

Five steps to get sphinx-ci running on a repo.

1. 1

   Sign in on sphinx-ci

   Go to sphinx-ci.dev and click "Get started with GitHub". The app requests the repo scope so it can post comments and status checks on your PRs.

2. 2

   Configure your repo

   In the dashboard, open the Repos tab, find the repo you want to protect, click Configure, set the quiz parameters (questions, passing score, attempts, language, trigger keyword), then click Generate API key. Copy the spx_… key.

3. 3

   Add secrets and variables to your GitHub repo

   In your GitHub repo settings, under Secrets and variables → Actions, add PR_QUIZ_API_KEY (the spx_… key) and ANTHROPIC_API_KEY (your sk-ant-… key) as secrets, and PR_QUIZ_HUB_URL (https://sphinx-ci.dev) as a variable.

4. 4

   Add the workflow file

   Copy .github/workflows/pr-quiz.yml from the sphinx-ci repo to your own repo's main branch.

5. 5

   (Optional) Enable branch protection

   In Settings → Branches, add a branch protection rule for main, check Require status checks to pass before merging, and add the pr-quiz status check. Without this, the quiz is informational but won't block the merge.

Workflow file

The workflow file lives at .github/workflows/pr-quiz.yml in your repo. It triggers on PR comments containing the trigger keyword, sends the diff to the hub, and reports back. The full file is in the sphinx-ci GitHub repo — here is the shape:

name: PR Quiz

on:
  issue_comment:
    types: [created]

permissions:
  contents: read
  pull-requests: write
  statuses: write

jobs:
  quiz:
    if: github.event.issue.pull_request && contains(github.event.comment.body, '@sphinx-ci')
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0
      - name: Generate quiz
        env:
          PR_QUIZ_API_KEY: ${{ secrets.PR_QUIZ_API_KEY }}
          PR_QUIZ_HUB_URL: ${{ vars.PR_QUIZ_HUB_URL }}
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
        run: |
          # See the full file in the repo for the complete script
          curl -X POST "$PR_QUIZ_HUB_URL/api/quiz" \
            -H "X-API-Key: $PR_QUIZ_API_KEY" \
            -d "{ ... }"

GITHUB_TOKEN is provided automatically by GitHub Actions. The Anthropic key stays in your GitHub repo secrets and is sent directly to the hub at request time.

Configuration

All quiz parameters are configurable per repo from the dashboard.

To change settings, go to Dashboard → Repos, click Edit on the repo, change the values, then Save. You can also Reset key to invalidate the current key, or Revoke to remove the configuration entirely.

Usage

1. 1A developer opens a PR.
2. 2Someone comments @sphinx-ci on the PR.
3. 3The Sphinx generates a quiz and posts a comment with the link.
4. 4The developer answers the quiz in their browser.
5. 5A result comment is posted on the PR. Passed → merge unlocked. Not yet → retry link if attempts remaining. Out of attempts → status check failure, merge blocked.

Teams & organizations

sphinx-ci works with GitHub organization repos. One admin signs in, configures the repo, adds the secrets and the workflow. From then on, all developers with access to the repo can trigger and take quizzes — they don't need a sphinx-ci account.

Granting access to organization repos

1. 1Go to github.com/settings/applications.
2. 2Find sphinx-ci in the list and click it.
3. 3Request access for your organization, then have an org admin approve.
4. 4Or: an org admin can pre-approve sphinx-ci from Organization settings → Third-party access.

Self-hosting

If you want to host your own sphinx-ci instance:

1. 1Create a Vercel account, a PostgreSQL database (Neon, Supabase, or Vercel Postgres), and a GitHub OAuth App pointing to your domain.
2. 2Configure environment variables on Vercel: DATABASE_URL, NEXT_PUBLIC_APP_URL, GITHUB_CLIENT_ID, GITHUB_CLIENT_SECRET, AUTH_SECRET.
3. 3Deploy by connecting the repo to Vercel — migrations run automatically during the build.

The full self-hosting guide is in the README on GitHub.

Troubleshooting