Skip to content
Autowright

Failure Cases

Failure Cases

A failure case is a structured representation of a test failure that Fixwright uses to analyze and fix issues. This page explains the format and how to generate failure cases.

Failure Case Structure

interface FailureCase {
  /** Unique identifier */
  id: string;


  /** Current processing status */
  status: 'pending' | 'processing' | 'fixed' | 'unfixable';


  /** When the failure occurred */
  timestamp: string;


  /** Script information */
  script: {
    name: string;
    path: string;
    repository?: {
      url: string;
      branch: string;
      commit?: string;
    };
  };


  /** Failure details */
  failure: {
    stepName: string;
    checkpointName?: string;


    error: {
      type: string;
      message: string;
      stack?: string;
    };


    sourceLocation: {
      file: string;
      startLine: number;
      endLine: number;
    };


    sourceCode: string;


    artifacts: {
      screenshot?: string;
      dom?: string;
      console?: string;
    };


    url?: string;
    title?: string;
  };
}

Generating Failure Cases

From Stepwright Results

Use generateFailureCase to create a failure case from a Stepwright result:

import { Stepwright, generateFailureCase } from '@korvol/stepwright';


const script = Stepwright.create('My Test')
  .config({ screenshotOnFailure: true, domOnFailure: true })
  .step('Click button', async (ctx) => {
    await ctx.page.click('#submit-btn');
  });


const result = await script.run();


if (!result.success && result.failedStep) {
  const failureCase = generateFailureCase(result, {
    scriptInfo: {
      name: 'My Test',
      path: 'scripts/my-test.ts',
    },
    repository: {
      url: 'https://github.com/org/repo',
      branch: 'main',
    },
  });


  // Save for Fixwright processing
  await fs.writeFile(
    `./failure-cases/${failureCase.id}.json`,
    JSON.stringify(failureCase, null, 2)
  );
}

Manual Creation

Create failure cases manually for custom integrations:

import { generateId } from '@korvol/fixwright';


const failureCase: FailureCase = {
  id: generateId(),
  status: 'pending',
  timestamp: new Date().toISOString(),


  script: {
    name: 'Login Flow',
    path: 'tests/login.spec.ts',
    repository: {
      url: 'https://github.com/org/repo',
      branch: 'main',
    },
  },


  failure: {
    stepName: 'Click login button',
    checkpointName: 'Authentication',


    error: {
      type: 'TimeoutError',
      message: 'Timeout 30000ms exceeded waiting for selector "#login-btn"',
      stack: '...stack trace...',
    },


    sourceLocation: {
      file: 'tests/login.spec.ts',
      startLine: 15,
      endLine: 17,
    },


    sourceCode: `await page.click('#login-btn');`,


    artifacts: {
      screenshot: './artifacts/login-failure.png',
      dom: './artifacts/login-failure.html',
      console: './artifacts/login-console.log',
    },


    url: 'https://app.example.com/login',
    title: 'Login - App',
  },
};

Failure Case Lifecycle

┌─────────┐    ┌────────────┐    ┌───────┐
│ PENDING │───▶│ PROCESSING │───▶│ FIXED │
└─────────┘    └────────────┘    └───────┘
                     │
                     ▼
              ┌───────────┐
              │ UNFIXABLE │
              └───────────┘

Status: pending

Failure case created and waiting to be processed:

{
  status: 'pending',
  // ... rest of failure case
}

Status: processing

Currently being analyzed by Fixwright:

fixwright.on('attempt:start', (attempt) => {
  console.log('Processing:', attempt.failureCase.id);
  // Status is now 'processing'
});

Status: fixed

Successfully fixed and PR created:

fixwright.on('failure:completed', (summary) => {
  if (summary.status === 'fixed') {
    console.log('Fixed!', summary.prUrl);
  }
});

Status: unfixable

Could not be fixed after all attempts:

fixwright.on('failure:completed', (summary) => {
  if (summary.status === 'unfixable') {
    console.log('Could not fix:', summary.reason);
  }
});

Artifacts

Screenshot

A PNG image of the page state at failure:

artifacts: {
  screenshot: './artifacts/step-3-failure.png',
}

DOM Snapshot

HTML content of the page at failure:

artifacts: {
  dom: './artifacts/step-3-failure.html',
}

Console Logs

Browser console messages:

artifacts: {
  console: './artifacts/step-3-console.log',
}

Failure Sources

File Source

Watch a directory for new failure cases:

fixwright.useFileSource('./failure-cases');

Directory structure:

failure-cases/
├── fc-2024-01-15-abc123.json
├── fc-2024-01-15-def456.json
└── fc-2024-01-16-ghi789.json

Database Source

Process failures from a database:

import { DatabaseFailureSource } from '@korvol/fixwright';


const source = new DatabaseFailureSource({
  connectionString: process.env.DATABASE_URL,
  tableName: 'failure_cases',
  pollInterval: 5000,
});


fixwright.useSource(source);

Custom Source

Implement a custom failure source:

import { FailureSource } from '@korvol/fixwright';


class MyCustomSource extends FailureSource {
  async start(): Promise<void> {
    // Start watching for failures
    this.pollQueue();
  }


  async stop(): Promise<void> {
    // Clean up
  }


  private async pollQueue() {
    // Fetch from queue
    const failure = await this.myQueue.pop();
    if (failure) {
      this.emit('failure', failure);
    }
  }
}


fixwright.useSource(new MyCustomSource());

Best Practices

Include All Artifacts

More context helps Claude fix correctly:

.config({
  screenshotOnFailure: true,
  domOnFailure: true,
  // Console logs are collected automatically
})

Use Absolute Paths

Ensure artifact paths are accessible:

artifacts: {
  screenshot: path.resolve('./artifacts/failure.png'),
  dom: path.resolve('./artifacts/failure.html'),
}

Include Repository Info

Required for PR creation:

script: {
  name: 'My Test',
  path: 'tests/my-test.ts',
  repository: {
    url: 'https://github.com/org/repo',
    branch: 'main',
    commit: 'abc123',  // Optional but helpful
  },
}

Store Error Stack Traces

Helps identify the exact failure location:

error: {
  type: error.name,
  message: error.message,
  stack: error.stack,  // Include full stack
}

Integration with CI/CD

Generate and store failure cases in CI:

# GitHub Actions example
- name: Run Tests
  run: npx tsx scripts/run-tests.ts
  continue-on-error: true


- name: Upload Failure Cases
  if: failure()
  uses: actions/upload-artifact@v4
  with:
    name: failure-cases
    path: ./failure-cases/

Then process failures separately:

- name: Download Failure Cases
  uses: actions/download-artifact@v4
  with:
    name: failure-cases


- name: Fix Failures
  run: npx tsx scripts/fix-failures.ts
  env:
    ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
    GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

Next Steps