Using KwikQA

Running in CI

kwikqa ci runs in your CI job after your tests fail. It analyzes the run as analyze does, and writes the failure report, kwikqa-report.html, with each failure group's evidence. Then it publishes a summary:

on GitHub Actions, the job summary and one pull-request comment;
on Azure Pipelines, a build summary and one pull-request thread on Azure Repos;
on GitLab CI/CD, the summary in the job's log and one merge-request note;
on Jenkins, the summary in the console log and one comment on the pull request, on GitHub or GitLab;
on any CI system, an optional Slack or Microsoft Teams message, through your own webhooks.

It reads every report analyze reads, so a JUnit, TestNG, pytest, or Cucumber suite gets the same summary as a Playwright one: pass the report with --report, as in Other test frameworks. With a committed policy, kwikqa fix can also repair a failing test in the job and open the repair as a draft pull request, which a person reviews and merges: see Fix pull requests.

1. Create a CI token

In the dashboard, open Account, name a token after where it will live (for example "GitHub Actions"), and create it. Copy it right away: it is shown once. Store it as a secret named KWIKQA_TOKEN in your CI system.

Running KwikQA with the token approves its diagnosis requests, and each request's preview is printed in the job's log, so every run shows what was sent. Each run counts as one analysis. On a team plan, CI tokens created by anyone on the team draw from the team's shared pool.

2. GitHub Actions: the KwikQA Action

Add the KwikQA Action as a step after your tests:

# .github/workflows/e2e.yml (excerpt)
permissions:
  contents: read
  pull-requests: write # only for the comment

steps:
  - uses: actions/checkout@v7
  - run: npm ci && npx playwright install --with-deps
  - run: npx playwright test

  - if: failure()
    uses: kwikqa/action@v1
    with:
      kwikqa-token: ${{ secrets.KWIKQA_TOKEN }}

With Cypress or WebdriverIO, run your tests as usual (npx cypress run, npx wdio run wdio.conf.ts) with KwikQA's reporter configured, and the Action finds its reports in test-results/kwikqa/. With mochawesome, JUnit, or another framework's report, set the report input.

On a failed run you get:

The job summary, on the run's page. It shows the counts, then one row per failure group with its size, owner, and evidence grade. It also lists the flaky tests and links to the report.
One pull-request comment with the same summary. A re-run of the job updates the comment instead of adding another, and a later run with no failures updates it to say so. Each job of a workflow keeps its own comment.
The failure report, kept as the run's kwikqa-report artifact. It holds each group's tests, diagnosis, and the evidence the diagnosis cites.

Inputs

Input	Default	Effect
`kwikqa-token`	none	Your CI token, from a secret. Without it, the run is analyzed locally and isn't diagnosed.
`github-token`	`github.token`	Writes the pull-request comment. The job needs `pull-requests: write`.
`report`	found	The test report, a file or a directory, if it isn't in a default location.
`framework`	none	For a JUnit report, the framework that wrote it: `playwright`, `cypress`, or `webdriverio`.
`working-directory`	`.`	The package that holds your tests.
`output`	`kwikqa-report.html`	Where to write the failure report, relative to `working-directory`.
`local`	`false`	`true` analyzes without asking a model, even with a token.
`notify`	`on-failure`	When to post to Slack and Teams: `on-failure` or `always`.
`slack-webhook-url`, `teams-webhook-url`	none	Your chat webhooks, from secrets.
`upload-report`	`true`	Keep the failure report as an artifact of the run.
`artifact-name`	`kwikqa-report`	The report artifact's name. Give each job of a matrix its own.
`version`	pinned	The kwikqa version to run. Each release of the Action names the one it was tested with.

Pull requests from forks

GitHub gives a fork's workflows no secrets and a read-only token. On such a pull request, KwikQA analyzes the run locally, writes the report and the job summary, and skips the comment. Don't switch to pull_request_target to get the token: that event runs the fork's code with your repository's secrets.

Without the Action

The Action runs npx kwikqa ci and uploads the report. You can run those two steps yourself:

- if: failure()
  run: npx kwikqa ci
  env:
    KWIKQA_TOKEN: ${{ secrets.KWIKQA_TOKEN }}
    GITHUB_TOKEN: ${{ github.token }}

- if: failure()
  uses: actions/upload-artifact@v7
  with:
    name: kwikqa-report
    path: kwikqa-report.html

3. Azure Pipelines

Run kwikqa ci as a step that runs only when an earlier step failed, and publish the report:

# azure-pipelines.yml (excerpt)
steps:
  - script: npx playwright test
  - script: npx kwikqa ci
    condition: failed()
    env:
      KWIKQA_TOKEN: $(KWIKQA_TOKEN)
      SYSTEM_ACCESSTOKEN: $(System.AccessToken)
      KWIKQA_SLACK_WEBHOOK_URL: $(KWIKQA_SLACK_WEBHOOK_URL) # optional
  - publish: kwikqa-report.html
    artifact: kwikqa-report
    condition: failed()

Store KWIKQA_TOKEN as a secret variable of the pipeline. Azure Pipelines gives a step a secret, and System.AccessToken, only through the step's env. The test step's failure keeps the build failed.

Build summary. The summary is a "KwikQA" section on the run's Extensions tab.
Pull-request thread. On an Azure Repos pull request, the summary is also one thread. A re-run of the same pipeline job updates it. It's created closed, so it never holds up the pull request under a comment-resolution policy.
Permission. The thread needs "Contribute to pull requests" for the pipeline's build service. Set it in Project settings → Repositories → your repository → Security, for "project Build Service (organization)". Without it, the log names the identity that was refused.
Other repositories. A pipeline for a GitHub or Bitbucket repository gets the build summary only.
Forks. A fork's build gets no secrets unless the pipeline allows it. KwikQA analyzes it locally and skips the thread.

4. GitLab CI/CD

# .gitlab-ci.yml (excerpt)
e2e:
  image: mcr.microsoft.com/playwright:v1.64.1-noble
  script:
    - npm ci
    - npx playwright test || { npx kwikqa ci; exit 1; }
  artifacts:
    when: always
    expose_as: KwikQA report
    paths:
      - kwikqa-report.html

The exit 1 keeps the job failed: kwikqa ci exits with the analysis's code, which is usually 0.

Variables. Add KWIKQA_TOKEN and GITLAB_TOKEN under Settings → CI/CD → Variables. Mark them masked, and leave them unprotected, or a feature branch's pipeline won't get them.
The token. GitLab's own CI_JOB_TOKEN can't write notes, so GITLAB_TOKEN is a token of your own, with at least the Reporter role. The note is posted as that token's user.
- A fine-grained personal access token needs only your project, with Work Item read, create, and update. GitLab's merge-request notes come under Work Item. Make it a dedicated user's if you can.
- Otherwise, a token with the api scope: a legacy personal access token, or a project or group access token (Premium or Ultimate on GitLab.com, any tier when self-managed).
Merge-request note. The summary is printed in the job's log and posted as one note on the merge request. A re-run of the same job updates it. A merge request pipeline names its merge request. A branch pipeline gets a note when its branch has exactly one open merge request in the project.
The report. expose_as puts a "KwikQA report" link on the merge request.
Forks. A fork's merge request pipeline runs in the fork, without your project's variables. KwikQA analyzes it locally and skips the note.

5. Jenkins

In a Pipeline, run kwikqa ci when the tests fail, with its tokens bound only around it. The agent needs Node.js 20.11 or later.

// Jenkinsfile (excerpt)
stage('e2e') {
  steps {
    sh 'npm ci'
    sh 'npx playwright test'
  }
  post {
    failure {
      withCredentials([
        string(credentialsId: 'kwikqa-token', variable: 'KWIKQA_TOKEN'),
        string(credentialsId: 'github-token', variable: 'GITHUB_TOKEN'),
      ]) {
        sh 'npx kwikqa ci'
      }
      archiveArtifacts artifacts: 'kwikqa-report.html', allowEmptyArchive: true
    }
  }
}

Summary. It's printed in the build's console log, with a link to the build, where the archived report is.
Pull-request comment. In a multibranch project, a pull request's build knows its pull request from the branch source. Jenkins has no pull-request API of its own, so KwikQA comments through the pull request's host, and a rebuild updates the comment:
- GitHub, including GitHub Enterprise, with GITHUB_TOKEN: a fine-grained token with Pull requests read and write on the repository, or a classic token with the repo scope. A GitHub App credential works too, bound with usernamePassword(credentialsId: 'github-app', usernameVariable: 'GITHUB_APP', passwordVariable: 'GITHUB_TOKEN'). Its token lasts an hour, so bind it around kwikqa ci, as above.
- GitLab, with GITLAB_TOKEN, a token as for GitLab CI/CD.
- Bitbucket and other hosts get no comment yet.
Forks. Jenkins gives a fork's pull request build the credentials its Jenkinsfile binds, and that build runs the fork's code, its package.json included. If you build pull requests from forks you don't trust, don't bind tokens in their builds. The branch source's trust setting only decides whose Jenkinsfile runs.
The report. Jenkins serves an archived page with its script and styles blocked, and the report stays readable that way. For its styles and filters too, set a Resource Root URL on another host name, under Manage Jenkins → System → "Serve resource files from another domain". Don't loosen Jenkins' Content-Security-Policy instead.

6. Fix pull requests

In GitHub Actions, Azure Pipelines, and GitLab CI/CD, kwikqa fix can repair a failing test in the job and open the repair as a draft pull request, a merge request on GitLab. Nothing lands until a person reviews and merges it. Opt in by committing a policy in kwikqa.config.json:

"ci": { "fix": { "pullRequests": "draft", "retry": false } }

Then run npx kwikqa fix after kwikqa ci, in the same job, with the host's token. Fetch the full history, so the diagnosis can see the commit that changed your app.

GitHub Actions

permissions:
  contents: write # the fix branch
  pull-requests: write # the comment and the fix pull request

steps:
  - uses: actions/checkout@v7
    with:
      fetch-depth: 0
  - run: npm ci && npx playwright install --with-deps
  - run: npx playwright test

  - if: failure()
    uses: kwikqa/action@v1
    with:
      kwikqa-token: ${{ secrets.KWIKQA_TOKEN }}
  - if: failure()
    run: npx kwikqa fix
    env:
      KWIKQA_TOKEN: ${{ secrets.KWIKQA_TOKEN }}
      GITHUB_TOKEN: ${{ github.token }}

Also turn on "Allow GitHub Actions to create and approve pull requests" in the repository's or organization's Actions settings. GitHub doesn't start workflows for a pull request the job's GITHUB_TOKEN opens: close and reopen it, or push to it, to run them, or give kwikqa fix a token from your own GitHub App (actions/create-github-app-token) as GITHUB_TOKEN. GitHub Free has no draft pull requests in private repositories, and KwikQA opens only drafts.

Azure Pipelines

- checkout: self
  fetchDepth: 0
# … your tests and the kwikqa ci step …
- script: npx kwikqa fix
  condition: failed()
  env:
    KWIKQA_TOKEN: $(KWIKQA_TOKEN)
    SYSTEM_ACCESSTOKEN: $(System.AccessToken)

For an Azure Repos repository. The build service, "project Build Service (organization)", needs "Contribute", "Create branch", and "Contribute to pull requests" on the repository.

GitLab CI/CD

variables:
  GIT_DEPTH: "0"
e2e:
  script:
    - npm ci
    - npx playwright test || { npx kwikqa ci; npx kwikqa fix; exit 1; }

GITLAB_TOKEN needs the api scope and at least the Developer role, to push the fix branch. A project access token keeps your own name off the merge request. A fine-grained personal access token isn't verified for this yet.

What it does

Repairs and validates on the runner. It re-runs the failing test once to confirm the failure, asks for one repair, and applies it to the runner's own checkout. The repaired test must pass three times in a row, with retries off. A failure is rolled back, and retried once only with "retry": true. It reuses the diagnosis kwikqa ci made, and one CI fix counts as one fix.
Opens one draft. A new branch, kwikqa/fix-<id>, named after the failure group, with one commit built through your code host's API, so the runner's git writes nothing. For a pull request's run it goes into that pull request's branch; for any other branch's run, such as a nightly suite on main, into that branch. It's labeled kwikqa, and its description gives the diagnosis, the repair's intent, the validation, and which models were asked.
Exactly the validated change. If the target branch changed the repaired file since the run, nothing is committed. KwikQA never moves a branch it didn't create, and never merges, approves, or marks a pull request ready.
One per failure group. While a group's fix is open, a later run of the same failure asks for no repair and costs no fix; it updates one note on the draft instead. A kwikqa/fix- branch left after a closed pull request also holds the group back until you delete it.
Only under the committed policy. The policy counts only if kwikqa.config.json is exactly as committed, and only for the runner's checkout. Each request, re-run, and patch is printed in the log with what approved it.
When it can't. A fork's pull request, a run on a tag, and a job without the token are refused before any request, so they cost nothing. On Jenkins, fix still needs a terminal.

7. Slack and Microsoft Teams

KwikQA posts the run to incoming webhooks you store as CI secrets. It keeps no webhook anywhere, and never prints one.

Slack. Create an incoming webhook for the channel, and store its URL as KWIKQA_SLACK_WEBHOOK_URL.
Teams. In the channel, add the Workflows template "Post to a channel when a webhook request is received", and store its URL as KWIKQA_TEAMS_WEBHOOK_URL. Office 365 connector webhooks were retired in May 2026.

Pass them to the Action as slack-webhook-url and teams-webhook-url, or set them as environment variables for npx kwikqa ci. An empty variable counts as unset, so check that the secret has a value if nothing is posted.

A message names the repository and branch, links the pull request or commit and the run, and gives the counts. Then it lists each failure group by its category, where it fails, its size, and its owner, and gives the flaky count. It holds no error text, evidence, or source code: a channel is a wider audience than a build log, so the message says where to look, and the report shows what failed.

KwikQA posts only when tests failed, unless you pass --notify always (the Action's notify input). A failed post is retried once and reported in the log.

Other CI systems

On other CI systems, npx kwikqa ci writes the failure report and posts to Slack and Teams, but writes no summary or comment yet. Keep kwikqa-report.html as a build artifact.

npx playwright test || { npx kwikqa ci; exit 1; }

The exit 1 keeps the build failed: kwikqa ci exits with the analysis's code, which is usually 0.

It never fails your build

A summary, comment, or message that can't be published is reported in the log and skipped. Without a KwikQA token, with a revoked one, or past your plan's limit, the run is analyzed locally, and the summary says why it wasn't diagnosed. kwikqa ci exits with the analysis's own exit code.

Without an account

Without a token, kwikqa ci sends nothing to KwikQA. It groups the failures, writes the report and the summary, and says the run wasn't diagnosed. npx kwikqa analyze --local does the same analysis without publishing.

Tips

Use a separate token for each CI system so you can revoke one without affecting the others.
The jobs of one GitHub Actions matrix share a job name, so they share one comment. Analyze a sharded run once, after merging its reports, and give each job its own artifact-name.
A diagnosis made in CI isn't reused by a later fix on your machine, because the run and checkout differ. When a failure can be repaired, the log says so: let kwikqa fix repair it in the job under a committed policy, or download the report artifact and run analyze and fix locally.
For the fullest Cypress or WebdriverIO diagnosis, run npx kwikqa analyze --capture on your machine: it re-runs the failing specs and records each failure's page, which a CI report doesn't contain.