config.md

Config file

main structure

Example

{
    "total_number_of_rows":100000,
    "number_of_rows_per_batch":10,
    "sinks":[],
    "lookup_fields":[],
    "fields":[]
}

Field	Type	Description	Possible values
total_number_of_rows	int	Number of rows to be generated
number_of_rows_per_batch	int	Number of rows to be generated per batch (parallely )
sinks	array	Where to output the generated data to
lookup_fields	array	Field values that are generated by taking as input another field
fields	array	Field to be generated

sinks

Sink documentation

lookup_fields

The values of the field is generated by reading a dictionary using as an input a field of the row.

Example

"lookup_fields":[
    {
        "lookup_name":"product_id_price",
        "source_field_name":"product_id",
        "mapping":{
            "product1":15,
            "product2":30,
            "product3":70,
            "product4":68,
            "product5":24,
            "product6":100
        }
    }
]

Field	Type	Description	Possible values
lookup_name	string	Name of the lookup
source_field_name	string	Name of the field that should be used for performing the lookup
mapping	dictionary	Dictionary that returns the lookup value for a given key(source_field_name)

fields

Fields documentation