Description

The DelimitedFileSegmentQueueMergeReceiver is identical to the DelimitedFileSegmentMergeReceiver in that it merges multiple segment data into a Delimited file formatted output. The two Receivers are different in the following way: 

  • DelimitedFileSegmentQueueMergeReceiver - Performs the merging process in real-time by monitoring segment queues while the segment files are being generated and saved to disk.

  • DelimitedFileSegmentMergeReceiver - Performs the merging process after all segments have been completed generating their respective file to disk. 


In This Article


When Should this Receiver be Used?

  • Any time a user wants to generate large volumes of nested delimited file output and speed up the test data generation process.


Is it Dependent on Other Receivers?

  • Yes, the SegmentDataQueueReceiver generates the Domain data segments to queue.

  • The DelimitedFileSegmentQueueMergeReceiver monitors the generated segment queues and performs the merging process in real time.


Receiver Parameters

The DelimitedFileSegmentQueueMergeReceiver requires that the following parameters are defined. Items with an asterisk* are required.

  • outputPath* - Defines the location to store the newly merged file. The default value is "#{resource.output.directory}". 
  • outputSubDirectory - Defines the prefix name of subdirectories that are auto-created under the outputPath and then appended with a number. The default value is "data". (Example: data1, data2, data2, ...)
  • configName* - Defines the name of the configuration file. The default value is "Config.xml".
  • configPath* - Defines the location where the configuration file and one or more template files are stored. The default value is "#{resource.output.directory}".
  • configSubDir - The subdirectory under the config path where configuration and template files are stored. The default value is "config".
  • segmentPath* -  Defines the path of the segment directory where all segment subdirectories can be found. The default value is "#{resource.output.directory}".
  • segmentSubDirectory - Defines the subdirectory under the segmentPath where segment files can be found. The default value is "segments".
  • filesPerOutputSubDir* - Defines the number of files to be generated per output subdirectory. The default value is "100".
  • columnDelimiter* - Defines the value that separates each column. The default value is a comma.
  • queuePeekRetryCount* - Defines the number of times the SegmentDirectoryManager will excuse a queue peekFirst() to pull a path and filename from the queue. Each attempt will pause for one second. If the queue is empty after the final attempt, a GenRocket exception will be thrown, and the process will be halted. The default value is "20".
  • lineDelimiter - Defines the value that separates each line. The default value is "/".
  • overrideFileName - This parameter allows you to override the output file name that is defined in the configuration file. Also, it gives the ability to modify the output file name with the help of the Engine API. 
  • deleteOutputSubDir - Defines whether to delete the outputSubDir or not. The default value is "true".
  • multipleFileOutput - Defines whether the nested records will go in 1 file or multiple files based on the loop count of the Root Domain. The value "false" means that only 1 file will be created for all the nested records. The value "true" means that 1 file will be created for each nested record, resulting in multiple records. The default value is "false".


Receiver Attribute Property Keys

There are no property keys necessary for this Receiver.


Configuration File

The DelimitedFileSegmentQueueMergeReceiver requires a configuration file to help facilitate the formatting of the data output. The example configuration file below defines the following:

  • fileNameSegments- The fileNameSegment tag defines the file naming convention for the Delimited format file that is being generated. For example:
    • Delimited-1.txt, Delimited-2.txt, and so on
  • segments - The segment files from the segment tag will be loaded and used to create the merged output.
  • segmentsHierarchy- Defines the hierarchical structure of the Domains. 


The Configuration File must be downloaded to the user's local machine and placed into the location specified by these two Receiver parameters in order to generate data: configPath and configSubDir (if defined).




How to Create the Configuration File 

  1. Select a Project and Project Version within the Project Dashboard.
  2. Select the Configuration Management Tab from the Management pane.
  3. Click on the New Configuration button.


  4. Select the Delimited Format radio button and click Select.


  5. Enter the required parameters to create a configuration file and then click Save.
    • Name: Defines the unique name for the Delimited Format configuration.
    • Config File Name: Defines the name of the configuration file that DelimitedFileSegmentQueueMergeReceiver will read to create a Delimited Format file.
    • Output File Name Format: Defines the format for the name of the output file (e.g., Output1.txt)
    • Segment Files: Defines the various segment files used to build nested Delimited Format.

       
  6. Arrange the Segments in a hierarchy by dragging/dropping the segments. Click on Done once finished.



  7. Click Preview to view the Configuration File in a Preview window. 



Sample Configuration File

<Config>
  <fileNameSegments>
    <fileNameSegment segmentType='Constant' value='Delimited' />
    <fileNameSegment segmentType='Constant' value='-' />
    <fileNameSegment segmentType='Counter' value='1' />
    <fileNameSegment segmentType='Constant' value='.txt' />
  </fileNameSegments>
  <segments>
    <segment domainName='Address' segmentFile='AddressSegment.xml' partitionSegmentSubDir='' />
    <segment domainName='Department' segmentFile='DepartmentSegment.xml' partitionSegmentSubDir='' />
    <segment domainName='User' segmentFile='UserSegment.xml' partitionSegmentSubDir='' />
    <segment domainName='Organization' segmentFile='OrganizationSegment.xml' partitionSegmentSubDir='' />
  </segments>
  <segmentHierarchy>
    <segment domainName='Organization'>
      <segment domainName='Department'>
        <segment domainName='User'>
          <segment domainName='Address' />
        </segment>
      </segment>
    </segment>
  </segmentHierarchy>
</Config>


Example User Story

A tester wants to generate nested Delimited Format data for four Domains. They want the segment merging process to happen in real time rather than after all the segments have been created and saved to disk. The data will be generated for four Domains: 

  • Organization Domain - Generate 3 Organizations (1 Organization per Delimited File)
  • Department Domain - Generate 3 Departments per Organization
  • User Domain - Generate 3 Users per Department
  • Address Domain - Generate 1 Address per User


Please note that this Receiver is designed to speed up the process for large amounts of data. A simple example is provided to quickly show how it works.


Setup Prerequisites

The tester has already set up the Project by completing the following setup tasks:

  1. Completed the GenRocket Runtime Installation
  2. Created a Project with the default Project Version
  3. Created the four Domains with Attributes
  4. Made the required Generator changes for Attributes
  5. Set loop counts for four Domains
    • Organization = 3
    • Department = 3
    • User = 3
    • Address = 1
  6. Created a Scenario for each of the four Domains
  7. Set Parent/Child/Sibling Relationships (Manually or through Domain Referencing Wizard)



Note: Not every one of the above steps is necessary when Domains are imported. The Domains in this example have been created manually within the Project. 


Step 1 - Add the SegmentDataQueueReceiver to each Domain

The tester adds the SegmentDataQueueReceiver to each of the four Domains for generating the data segments. Please look at SegmentDataQueueReceiver and How do I add a Receiver to a Domain? to learn more.


Organization Domain Example


Department Domain Example


User Domain Example


Address Domain


Step 2 - Create a Configuration File

The tester creates a Configuration File for the Delimited Format.




Step 3 - Create a Merge Domain and Merge Scenario

The tester then creates a Merge Domain and Scenario. The Merge Domain is used to merge the generated data segments. This Domain is created automatically for some Domain imports but can also be created manually when needed.


Step 4 - Add DelimitedFileSegmentQueueMergeReceiver to Merge Domain

Next, the tester adds the DelimitedFileSgementQueueMergeReceiver to the Merge Domain to merge the segments from the queue in real-time. 



Additionally, the tester configures the following Receiver Parameters: 

  • configName - The configuration file name created during Step 2 of this example user story (i.e., Config.xml).
  • multipleFileOutput - Set to "True" to generate 1 file for each Organization (Root Domain).



Step 5 - Create a Scenario Chain with all Scenarios

The tester creates a Scenario Chain containing all the Domain Scenarios (including the Merge Domain Scenario). 


Note: The Merge Scenario must be the last Scenario within the Scenario Chain to merge the Domain data segments. 



Step 6 - Download Files and Generate Test Data

The tester completes the following tasks to generate their data:

Step 6a - Download the Configuration File

  • Clicks on the Download (Cloud) icon to download the file to their local computer.


  • Places the file into a config subdirectory located in the path defined in the configPath Receiver Parameter.




Step 6b - Download the Scenario Chain.

  • Click on the Download (Cloud) icon to download the Scenario Chain.


  • Place the Scenario Chain file into the path defined within the resource.output.directory Organization Resource.



Step 6c - Run the Command at the Command Line

Opens a Command Prompt or Terminal window and sets the directory to the appropriate path:


Command

genrocket -r <ScenarioChainName.grs>


Command Example

genrocket -r AllScenarioChain.grs



Sample Output

For this example, three output files will be created in the delimited file format (one for each organization).


Each file contains three departments and three users per department. Each user has one address. 


Delimited-1.txt


Delimited-2.txt


Delimited-3.txt