E2E Testing

Overview

The E2E stage validates that your deployed application functions correctly. Party Bus does not prescribe specific test cases, but every project must implement meaningful E2E tests that satisfy CtF requirements.

Cross-IL Setups

E2E testing across ILs is not supported. The staging environment must be in the same IL as the source code. In this situation, E2E testing will need to be done locally, and results captured and passed along to Cyber.

How the E2E job works

The E2E job:

• Runs in a dedicated test container.
• Clones your repository.
• Does not use your application image.
• Executes npm run test:e2e-ci.

E2E execution flow

flowchart TD
    A[GitLab Pipeline]
    B[Dedicated Cypress Container]
    C[Clone Repository]
    D["Run `npm run test:e2e-ci`"]
    E["Staging Environment (`ZAP_URL`)"]

    A --> B
    B --> C
    C --> D
    D --> E

Requirements

Prerequisites

Before you begin:

• Implement an E2E test suite that validates core application functionality.
• Ensure your E2E tests satisfy any applicable CtF and SD Elements requirements not covered by earlier pipeline stages.
• Deploy your application to a staging environment. E2E tests run against staging.
• Configure your tests to use the Party Bus-provided ZAP_URL environment variable as the application's baseUrl.
• If your application uses P1 authentication headers or JWT-based authorization, create a test JWT for E2E testing.
• If using Cypress:
  • Include a package.json file in your repository.
  • Define a test:e2e-ci script in package.json.
  • Configure your Cypress pipeline settings (recommended: cypress/config/pipeline.config.js).
  • Store screenshots in cypress/screenshots and videos in cypress/videos.
• If using Pytest, set the E2E_PYTEST variable to your E2E test entry point.
• If using an E2E framework other than Cypress or Pytest, submit a Help Desk ticket before implementation.
• If your staging environment resides in a different IL than your source code, plan to execute E2E tests locally because cross-IL E2E testing is not supported.

Checklist

Before enabling the E2E stage, confirm that:

• [ ] The application is deployed to staging.
• [ ] E2E tests pass locally.
• [ ] test:e2e-ci executes successfully.
• [ ] The pipeline uploads screenshots and videos.
• [ ] The E2E stage passes in GitLab.

Required for CtF

1. Every pipeline must include an E2E testing stage.

Per Project Requirements

At a minimum, define one E2E test suite for the front-end and execute it in every pipeline. This approach verifies application functionality regardless of which pipeline is used.

2. E2E tests must:

• Validate core application functionality.
• Verify any SD Elements requirements not covered by earlier pipeline stages. If you are not familiar with SD Elements or have not gone through the CtF Process, please submit a CtF request to start the process.

3. All tests are performed on the front-end, but should encompass any back-end functionality as well.

• Treat the application as a single system during E2E testing. Tests should validate E2E functionality rather than individual services.

What must be tested

Your E2E tests should demonstrate that the deployed application functions as expected from an end-user perspective. At a minimum, your test suite should:

• Validate one or more primary user workflows.
• Verify core application functionality after deployment.
• Exercise any required back-end services or APIs used by the application.
• Verify SD Elements requirements that are not covered by earlier pipeline stages.
• Validate application-specific functionality required for CtF authorization.

There is no required minimum number of E2E tests or code coverage threshold. Product teams are responsible for determining the scope of testing needed to demonstrate application integrity. During a CtF review, the CAT may recommend or require additional tests to validate specific security controls.

TIP

Focus on validating user workflows rather than individual functions. Unit and integration tests should verify implementation details, while E2E tests should verify that the deployed application works as expected.

Supported frameworks

Party Bus supports the following E2E frameworks:

• Cypress (default)
• pytest

Other frameworks require a Help Desk request.

Configuration

ZAP_URL

Party Bus automatically provides the ZAP_URL environment variable for the E2E job. Configure your test framework to use this value as the application's baseUrl.

Do not hardcode application URLs in your tests.

Example:

module.exports = {
  e2e: {
    baseUrl: process.env.ZAP_URL
  }
}

JWT

If your application uses P1 authentication headers or JWT-based authorization, configure a test JWT for E2E testing. For more information, see the SSO, Authentication, and Authorization guide.

Gateway

E2E Gateway

The E2E gateway bypasses Keycloak during testing.

Your application should not receive Keycloak redirects or authentication errors. If it does, contact the Help Desk.

Artifacts

Party Bus automatically archives test artifacts from the following directories:

• cypress/screenshots
• cypress/videos

These artifacts can help diagnose failures that occur during pipeline execution. Review these artifacts in the GitLab job after the pipeline completes.

Cypress

During the E2E stage, GitLab clones your repository into a dedicated Cypress container and runs npm run test:e2e-ci. Your application image is not used to execute the tests.

As a result, you do not need to install Cypress or copy your E2E test files into your application image. Your package.json, Cypress configuration, and test files only need to exist in your Git repository.

Repository layout

The recommended repository structure is:

├── package.json
└── cypress/
    ├── config/
    │   └── pipeline.config.js
    ├── e2e/
    ├── screenshots/
    └── videos/

package.json

Your repository must include a package.json file with a test:e2e-ci script. During pipeline execution, Party Bus runs the following command from the root of your cloned repository: npm run test:e2e-ci.

Note for non-Node.js applications

Even if your application is not built with Node.js (for example, .NET, Java, Go, or Python), you still need a package.json file. Party Bus uses this file only to define and execute the test:e2e-ci script during the E2E job. It does not mean your application itself must use Node.js or npm.

The following is the minimum supported package.json:

{
  "name": "dotnet-world",
  "version": "1.0.0",
  "scripts": {
    "test:e2e-ci": "npx cypress run --browser chrome --config-file cypress/config/pipeline.config.js"
  }
}

pipeline.config.js

This command executes the test:e2e-ci script defined in your package.json. A typical configuration is:

{
  "scripts": {
    "test:e2e-ci": "npx cypress run --browser chrome --config-file cypress/config/pipeline.config.js"
  }
}

The recommended location for the pipeline configuration file is cypress/config/pipeline.config.js.

Although some existing projects may use different locations because of legacy configurations, new projects should follow this directory structure whenever possible.

Specify a Cypress version

Customers manage their own Cypress version. To use a specific version, create a GitLab CI/CD variable named CYPRESS_VERSION and set its value to the desired Iron Bank cypress-included image tag (for example, 14.3.0). Available image versions are published in Iron Bank through Registry1.

For more information about writing Cypress tests, refer to the official Cypress documentation.

Pytest

You will need to set E2E_PYTEST to your E2E entrypoint. By default, Party Bus looks for pytests/e2e_pytest.py.

Data isolation

E2E tests run against the staging environment. Because E2E and staging share the same cluster, test data is not isolated from other staging data.

WARNING

At this time, unsupported databases and their isolation are the responsibility of the product team.

FAQs

How does the JWT work for E2E tests?

Since auth-service and Keycloak are not present in the E2E gateway, requests will hit your app without a JWT. It is recommended that your tests add a mock JWT header.

View an example implementation in our SSO, Authentication, and Authorization User Guide.

What URL should my pipeline E2E tests use?

Configure your test framework to use the Party Bus-provided ZAP_URL environment variable as the application's baseUrl.

How do E2E tests communicate with my application?

Unlike most pipeline jobs, the E2E test job is permitted to access your deployed staging environment. Cypress runs tests against the configured application URL, typically specified by the baseUrl setting in the Cypress configuration.

Using the baseUrl configuration is recommended instead of hardcoding URLs in test files.

Is Cypress the only supported E2E testing framework?

No. Cypress is the default and most commonly used framework. Python-based E2E testing with pytest is also supported. Support for additional languages and frameworks may require additional Platform One pipeline support.

Are E2E tests required for ATO?

Yes. The E2E pipeline job must pass to satisfy ATO pipeline requirements.

There is no required minimum code coverage for E2E tests. Teams should implement tests that provide meaningful validation of application behavior. During an ATO review, the CAT may recommend or request additional E2E tests to demonstrate specific security controls.

Related content and references

Submit requests to the Help Desk

• Submit a CtF request

Review supporting documentation

• Review our Pipeline Basics article
• Review our SSO, Authentication, and Authorization guide