S3

Overview

Party Bus provides access to Amazon S3 for applications running on P1. Authentication is performed using IRSA, allowing workloads to securely access S3 using short-lived AWS credentials without managing long-lived AWS access keys.

By default, the use of AWS access keys and secret keys is not permitted and requires approval from the Cyber team.

Before your application can access Amazon S3, an S3 bucket must be provisioned for your project. If your application does not already have an S3 bucket, follow the bucket provisioning process described in the Initial bucket provisioning section.

This guide explains how to configure your application to access S3, the available endpoint options (including FIPS endpoints), and the process for requesting additional S3 buckets when needed.

Prerequisites

Before configuring your application to use Amazon S3 through Party Bus, ensure the following prerequisites are met:

• Your application is deployed on the Party Bus platform.
• An Amazon S3 bucket has been provisioned for your application.
• Your application has a Kubernetes service account configured for AWS authentication.
• If your use case requires AWS access keys and secret keys instead of IRSA, approval from the Cyber team is required before credentials can be issued.

Authentication

IRSA (Recommended)

Party Bus uses IRSA as the default and recommended authentication method for Amazon S3. IRSA enables workloads running on Kubernetes to securely authenticate to AWS by assuming an IAM role associated with a Kubernetes service account, using short-lived AWS credentials instead of long-lived access keys.

When an application is configured to use IRSA, supported AWS SDKs and the AWS CLI automatically obtain temporary AWS credentials for the workload using its Kubernetes service account identity. These credentials are automatically refreshed as needed, allowing applications to authenticate without managing or rotating AWS access keys.

By default, applications should not use AWS access keys and secret keys. Requests for static credentials require approval from the Cyber team and should only be used when IRSA is not a viable option.

If your application requires an exception to use static credentials, see Temporary S3 Credentials.

Temporary S3 credentials

When IRSA cannot be used, teams may request temporary S3 credentials for one-time data extraction, transfer, or loading. These are one-time, short-term, 12-hour credentials. Please coordinate with your BAM or TAM to request temporary S3 credentials.

INFO

S3 is not a traditional file system (block storage) but instead is object storage. This means that folder paths do not need to be created when generating files, since the first time a file is created, the path is created as part of the file name.

Bucket management

Initial bucket provisioning

Before an application can use Amazon S3, an S3 bucket must be provisioned.

To request an S3 bucket:

1. Work with your BAM or TAM to ensure the requested information is captured and any billing requirements are addressed.
2. Ensure the bucket request is documented on the appropriate COT ticket.
3. MDO will create the S3 bucket.

Once the bucket has been created, configure your application to access it using IRSA.

Requesting additional S3 buckets

Your team can have additional S3 buckets created.

1. Contact your BAM or TAM to review any billing implications and ensure the additional bucket(s) are documented on your COT.
2. Once the COT has been updated, submit an MDO ticket requesting the additional S3 bucket(s).
3. The MDO team will create the bucket(s).

WARNING

MDO cannot create additional buckets until the request is reflected on the associated COT.

Application configuration

Configure the service account (IRSA)

To allow your application to authenticate to S3 using IRSA, add the serviceAccountName role to your kustomization deployment.yaml manifest. Example:

serviceAccountName: s3-role

Configure the bucket name

To use the correct S3 bucket name, the S3_BUCKET_NAME variable prefix can be generated using the below:

envFrom:
  - configMapRef:
      name: cluster-prefix
env:
  - name: NAMESPACE
    valueFrom:
      fieldRef:
        fieldPath: metadata.namespace
  - name: S3_BUCKET_NAME
    value: pb-mdo-$(NAMESPACE)-$(CLUSTER_PREFIX)

Configure the endpoint

If your application requires an endpoint and region, you can specify it below:

deployment.yaml

env:
  - name: S3_ENDPOINT_URL
    value: https://s3.us-gov-west-1.amazonaws.com

INFO

Some SDKs expect explicit values for the AWS_REGION variable. If so, set the following in your deployment.yaml.

deployment.yaml

env:
  - name: AWS_REGION
    value: us-gov-west-1

SDK usage examples

INFO

The examples below use the AWS SDK default credential provider chain, which automatically detects IRSA credentials when configured.

Java

We recommend using AWS SDK for Java, as it supports IRSA credentials by default.

1. Maven dependencies (pom.xml):

<dependencies>
    <dependency>
        <groupId>com.amazonaws</groupId>
        <artifactId>aws-java-sdk-s3</artifactId>
        <version>1.12.542</version>
    </dependency>
    <dependency>
        <groupId>com.amazonaws</groupId>
        <artifactId>aws-java-sdk-sts</artifactId>
        <version>1.12.542</version>
    </dependency>
</dependencies>

2. Java example code to list S3 buckets using IRSA:

import com.amazonaws.auth.WebIdentityTokenCredentialsProvider;
import com.amazonaws.services.s3.AmazonS3;
import com.amazonaws.services.s3.AmazonS3ClientBuilder;
import com.amazonaws.services.s3.model.Bucket;

public class IRSAExample {
    public static void main(String[] args) {
        String region = System.getenv("AWS_REGION");
        if (region == null || region.isEmpty()) {
            System.err.println("AWS_REGION environment variable is not set.");
            return;
        }
        try {
            AmazonS3 s3Client = AmazonS3ClientBuilder.standard()
                .withCredentials(WebIdentityTokenCredentialsProvider.create())
                .withRegion(region)
                .build();
            System.out.println("S3 Buckets:");
            for (Bucket bucket : s3Client.listBuckets()) {
                System.out.println("- " + bucket.getName());
            }
        } catch (Exception e) {
            System.err.println("Error accessing AWS S3 using IRSA: " + e.getMessage());
            e.printStackTrace();
        }
    }
}

Spring boot

1. Add AWS dependencies in Gradle (build.gradle):

dependencies {
  implementation platform('io.awspring.cloud:spring-cloud-aws-dependencies:3.1.1')
  implementation 'io.awspring.cloud:spring-cloud-aws-starter-s3'
  implementation platform('software.amazon.awssdk:bom:2.26.12')
  implementation 'software.amazon.awssdk:s3'
  implementation 'software.amazon.awssdk:sts'
}

2. Configure AWS properties (application.yml):

spring:
  cloud:
    aws:
      s3:
        enabled: true
        endpoint: "${S3_ENDPOINT_URL}"
        path-style-access-enabled: true
      region:
        static: "${AWS_REGION}"
  <appName>:
    bucketName: "${S3_BUCKET_NAME}"

3. Access Bucket Name via ConfigurationProperties:

import org.springframework.boot.context.properties.ConfigurationProperties;

@ConfigurationProperties("spring.<appName>")
public record AppConfiguration(String bucketName) {}

4. IRSA Credentials Setup: Spring Boot will automatically detect and use IRSA credentials via StsWebIdentityTokenFileCredentialsProvider. No additional configuration is required.

Javascript

The AWS SDK for JS uses the AWS_WEB_IDENTITY_TOKEN_FILE and AWS_ROLE_ARN by default.

1. Install Dependencies:

npm install @aws-sdk/client-s3 dotenv

2. Main file (app.js):

const { S3Client, ListObjectsV2Command } = require("@aws-sdk/client-s3");

const bucketName = process.env.S3_BUCKET_NAME;
const awsRegion = process.env.AWS_REGION;

const s3Client = new S3Client({ region: awsRegion });

async function listBucketObjects() {
  try {
    const command = new ListObjectsV2Command({ Bucket: bucketName });
    const data = await s3Client.send(command);
    if (!data.Contents || data.Contents.length === 0) {
      console.log(\`Bucket "\${bucketName}" is empty.\`);
      return;
    }
    console.log(\`Objects in "\${bucketName}":\`);
    data.Contents.forEach((item) => {
      console.log(\`- \${item.Key}\`);
    });
  } catch (error) {
    console.error("Error listing bucket objects:", error);
  }
}

listBucketObjects();

Python

We recommend using boto3 for AWS integration in Python.

1. Install boto3:

pip install boto3

2. Python script:

import boto3
import os

def handle_s3():
    bucket_name = os.getenv("S3_BUCKET_NAME")
    if not bucket_name:
        return "S3_BUCKET_NAME environment variable is not configured."
    s3 = boto3.resource('s3')
    try:
        objects = list(s3.Bucket(bucket_name).objects.all())
        print(f"Objects in bucket '{bucket_name}':")
        for obj in objects:
            print(f"- {obj.key}")
        return f"{bucket_name} contains {len(objects)} objects."
    except Exception as e:
        return f"Error accessing bucket: {e}"

if __name__ == "__main__":
    print(handle_s3())

Go

We recommend using the official AWS SDK for Go (aws-sdk-go-v2).

1. Install AWS SDK for Go v2:

go get github.com/aws/aws-sdk-go-v2/aws
go get github.com/aws/aws-sdk-go-v2/config
go get github.com/aws/aws-sdk-go-v2/service/s3

2. Sample Go script:

package main

import (
    "context"
    "fmt"
    "log"
    "os"

    "github.com/aws/aws-sdk-go-v2/config"
    "github.com/aws/aws-sdk-go-v2/service/s3"
)

func main() {
    bucketName := os.Getenv("S3_BUCKET_NAME")
    if bucketName == "" {
        log.Fatal("Environment variable S3_BUCKET_NAME is not set.")
    }

    cfg, err := config.LoadDefaultConfig(context.TODO())
    if err != nil {
        log.Fatalf("Failed to load AWS configuration: %v", err)
    }

    s3Client := s3.NewFromConfig(cfg)

    resp, err := s3Client.ListObjectsV2(context.TODO(), &s3.ListObjectsV2Input{
        Bucket: &bucketName,
    })
    if err != nil {
        log.Fatalf("Failed to list objects: %v", err)
    }

    fmt.Printf("Objects in bucket '%s':
", bucketName)
    for _, item := range resp.Contents {
        fmt.Printf("- %s
", *item.Key)
    }
}

FIPS S3 endpoints

Applications requiring FIPS-compliant S3 communication should use the following endpoint:

deployment.yaml

env:
  - name: S3_ENDPOINT_URL
    value: https://s3-fips.us-gov-west-1.amazonaws.com

Troubleshooting

AccessDenied errors

Verify:

• The Kubernetes service account is configured correctly.
• The IAM role has permissions for the bucket.
• The application is using the correct bucket name.

Unable to locate credentials

Verify:

• IRSA is configured.
• The pod is using the correct service account.
• AWS SDK version supports web identity credentials.

Related content and references

Review supporting documentation

Review the official Amazon S3 resources

Submit requests to the Help Desk and get support

• Ask for help in the Party Bus Value Stream Support channel
• Submit a support request with the P1 Help Desk