Description

The AWSRoleBasedS3Receiver enables secure file uploads to an AWS S3 bucket. It authenticates using an AWS IAM Role, which defines the permissions and access policies required to interact with the S3 service.


Role-based S3 access in AWS works by allowing trusted identities (like users, applications, or services) to assume a role that has specific permissions to access Amazon S3. This method eliminates the need for permanent credentials, offering fine-grained, temporary, and auditable access to S3 buckets.


This Receiver supports secure, automated file transfers in role-controlled AWS environments, eliminating the use of individual credentials.


NOTE: To upload files to an S3 bucket using individual user credentials (Client ID and Secret Key) rather than an IAM Role, use the AmazonS3Receiver. 


In This Article


Key Features

  • Secure Access - Leverages IAM Roles instead of access keys for heightened security.
  • Temporary Credentials - Integrates with AWS STS to obtain short-lived credentials.
  • Permission Scope - Role defines what actions can be performed (e.g., s3:PutObject, s3:ListBucket) and on which resources (e.g., specific buckets or prefixes).


How Does it Work?

The Receiver uploads files from the defined payload folder to a specified S3 bucket or folder within that bucket. It can be used with another Receiver to upload generated test data files.

The following steps are required to upload generated files to a folder within an S3 bucket:

  1. Set up a Project to generate the file(s) in the desired output format (e.g., JSON, Delimited, XML). 
  2. Assign the AWSRoleBasedS3Receiver as a second receiver to the Domain(s). 
    • Flat Files - Assign directly to the Domain from which the files are being generated.
    • Nested Files - Assign to the Merge Domain so that the final merged files are sent to the S3 bucket.
  3. Configure the AWSRoleBasedS3Receiver parameters accordingly. 
    • (Optional) Specify the type of files that should be transferred from the Payload location (e.g., txt, xls).
    • (Optional) Specify how many requests (threads) can be sent at the same time. 
  4. Generate the files using either the command line or an alternative method such as a script.
  5. If everything is set up correctly, the generated file(s) will be successfully uploaded to the specified folder within the S3 bucket. 


Receiver Parameters

The following parameters can be defined for the AWSRoleBasedS3Receiver. Items with an asterisk (*) are required. 

  • roleARN* - Defines the Amazon Resource Name (ARN) of the IAM Role used to access AWS resources securely. 
  • sessionName* - Define the session identifier used in the STS authentication request. For example, you can use "genrocket" to indicate the request was initiated by GenRocket, making it easier to identify who is connecting. 
  • region* - Specifies the AWS geographical location (e.g., is-east-1, ap-south-1) where services and resources like S3 buckets are hosted. Choosing the correct region ensures lower latency, compliance with data residency requirements, and optimized performance. 
  • bucketName* - Defines the name of the bucket on the S3 server.
  • foldersWithinBucket - Defines the subfolders within the bucket (e.g., user/address. user/address/city).
  • payloadFilesPath* - Defines the base location of the files that will be sent. 
  • payloadFilesSubDir - Defines the naming convention for subdirectories within the payloadFilesPath (e.g., data1, data2, data3).
  • payloadFileDescriptor - Defines the descriptor name (file type) to be sent from the Payload SubDirectory. Examples:
    • txt - sends only files whose extension is txt
    • xls - sends only files whose extension is xls
  • logRequestCount* - Defines the number of requests that are executed before logging a message to the console. 
  • threadCount* - Defines the number of requests that may be sent simultaneously. 
  • reponseOutput* - Defines if an output response file should be produced, and if so, how it should be produced.
  • responseFilePath - Defines the location where the response file will be saved.
  • responseFileName - Defines the name of the response file. 


Receiver Attributes Property Keys

No property keys are necessary for this Receiver.


Use Case - Generate Delimited Files and Upload to a Folder in an S3 Bucket

A user wants to generate 10,000 customer records in a delimited file format, with each file containing 100 records, and have the following naming convention: Customer1.txt, Customer2.txt, Customer3.txt, and so on. Additionally, they want the generated files to be securely transferred to a designated folder in an S3 bucket.

Project Setup

To achieve this, the user must set up the Project Version to generate delimited files. Additionally, they must assign and configure the AWSRoleBasedS3Receiver. 
  • Domain - Customer
  • Domain loopCount - 10000
  • Attributes - firstName, lastName, ssn, phoneNumber, creditCardNumber
  • Generators - NameGen, NameGen, SSNGen, PhoneNumberGen, CreditCardGen
  • Receivers - DelimitedFileReceiver, AWSRoleBasedS3Receiver
  • Scenario - CustomerScenario



DelimitedFileReceiver Configuration

The default values are used for all parameters except the subDir parameter. Generated delimited files will be stored in a folder called "bankData".


The user has also configured the Dynamic File Configuration for the DelimitedFileReceiver. It is set to generate 100 records per delimited file. Each file will be named Customer#.txt (e.g., Customer1.txt, Customer2.txt, Customer3.txt).



AWSRoleBasedS3Receiver Configuration

The AWSRoleBasedS3Receiver connects via the IAM Role to transfer the files. The files stored in the 'bankData' folder will be transferred to the 'genrocket-data' S3 bucket in the 'bank' folder. Only files with a '.txt' extension will be transferred from the payload folder. Logs will be provided for every 10 requests.