JOLT Mapping

JOLT (JSON-to-JSON Object Language for Transform) is a powerful library used to transform JSON data. JOLT is integrated into the flows we provide and can be used in Mapping Steps to perform dynamic data transformations. This documentation provides an overview of key JOLT methods, along with examples and string operations that can be useful in customizing JSON data within your flows.

The methods and examples provided in this document do not cover all JOLT usage or methods. For more detailed usage examples and explanations, you can refer to the JOLT Demo website, which includes an in-built JOLT engine to test your own modifications, or consult the JOLT framework source code for more information on how JOLT operates and the available methods.

JOLT fails silently: if the JOLT specification syntax is correct and every keyword or method is used properly, JOLT will not generate an error. As a result, when running the code, if the JOLT engine cannot find an object key in the input or is unable to map a value to the designated output key, it will fail silently. The result will not have the expected outcome, and the missing key/values will not trigger any exception message.

What is JOLT?​

JOLT is a framework for transforming JSON documents into other JSON documents using declarative transformation specifications. It is highly flexible and allows mapping, modifying, and restructuring JSON data efficiently. JOLT is particularly suited for data normalization, enrichment, or reformatting tasks in flows.

JOLT Methods​

The basic structure of a JOLT specification is as follows:

[
    {
        "operation": "<method name>",
        "spec": {
            ... method argument specs ...
        }
    }
]

Each method is defined by using “operation” key, shift, remove etc. after which “spec” key is used for providing arguments of how input should be transformed by the method. Any number of “operations” can be supplied in the root array which is referred to as JOLT Chain.

JOLT Chains​

Integrator uses JOLT Chains in mapping steps. A JOLT chain specification is a sequence of transformations that are applied to JSON data in order. Each transformation in the chain builds on the output of the previous one. This allows for complex and multi-step JSON restructuring. Below is a simple example:

Input JSON:

{
  "product": {
    "id": "123",
    "name": "Shirt",
    "price": 1000,
    "discount": 200,
    "internalCode": "ABC-123"
  },
  "stock": {
    "location1": 10,
    "location2": 5
  }
}

JOLT Chain Specification:

[
  {
    "operation": "shift",
    "spec": {
    "product": {
        "id": "details.productId",
        "name": "details.productName",
        "price": "details.productPrice",
        "discount": "details.discountAmount",
        "#basic": "details.productType"
    },
    "stock": "stockDetails"
    }
  },
  {
    "operation": "modify-default-beta",
    "spec": {
    "details": {
        "finalPrice": "=intSubtract(@(1,productPrice),@(1,discountAmount))"
    }
    }
  },
  {
    "operation": "remove",
    "spec": {
    "details": {
        "discountAmount": ""
    }
    }
  }
]

Result of specification:

{
  "details" : {
    "productType" : "basic",
    "productId" : "123",
    "productName" : "Shirt",
    "productPrice" : 1000,
    "finalPrice" : 800
  },
  "stockDetails" : {
    "location1" : 10,
    "location2" : 5
  }
}

The specification is applied in order:

1. Shift: Restructure and rename fields.

2. Modify-default-beta: Add a calculated field (finalPrice).

3. Remove: Clean up by removing unnecessary fields (discountAmount).

Shift Method​

The shift operation is used to map and restructure data from the input JSON into the output JSON.

Example:

//Input:
{
  "product": {
    "id": "123",
    "name": "Shirt",
    "price": 1000,
    "discount": 200,
    "internalCode": "ABC-123",
    "unusedValue": "foobar" 
  },
  "stock": {
    "location1": 10,
    "location2": 5
  }
}

//Specification:
[
  {
    "operation": "shift",
    "spec": {
    "product": {
            "id": ["details.productId", "details.productSku"],
        "name": "details.productName",
        "price": "details.productPrice",
        "discount": "details.discountAmount",
        "#basic": "details.productType"
    },
    "stock": "stockDetails"
    }
  }
]

//Result:
{
  "details" : {
    "productType" : "basic",
    "productId" : "123",
    "productSku" : "123",
    "productName" : "Shirt",
    "productPrice" : 1000,
    "discountAmount" : 200
  },
  "stockDetails" : {
    "location1" : 10,
    "location2" : 5
  }
}

In this case:

• product.id is mapped to details.productId and details.productSku

• product.name is mapped to details.productName.

• product.price is mapped to details.productPrice.

• product.discount is mapped to details.discountAmount.

• stock is directly moved to stockDetails.

Please note that destination mapping values are given in an array to use 1 input key as multiple values.

Default Method​

The "default" operation checks if the specified field exists in the JSON. If the field is missing or null, it assigns the default value from the spec, if the field already exists with a value, it remains unchanged.

Example:

//Input:
{
  "product": {
    "id": "123",
    "name": "Shirt",
    "price": 1000,
    "discount": 200,
    "internalCode": "ABC-123",
    "unusedValue": "foobar"
  },
  "stock": {
    "location1": 10,
    "location2": 5
  }
}

//Specification:
[
  {
    "operation": "default",
    "spec": {
    "product": {
            "productType": "basic"
    },
    "stock": {
        "location1": 999
}
    }
  }
]

//Result:
{
  "product" : {
    "id" : "123",
    "name" : "Shirt",
    "price" : 1000,
    "discount" : 200,
    "internalCode" : "ABC-123",
    "unusedValue" : "foobar",
    "productType" : "basic"
  },
  "stock" : {
    "location1" : 10,
    "location2" : 5
  }
}

In this case:

• Since product.productType value does not exist, “basic” value is set by default

• Since the stock.location1 value already exists in the input data, the value in the output does not change.

Modify Default Method​

The modify-default-beta operation in JOLT is used to perform modifications to a JSON structure. It allows you to update existing fields or create new fields with new calculated or default values.

This operation can utilize functions and expressions to compute values dynamically based on the data in the JSON.

• Supports functions like =intSum, =intMultiply, and others for calculations.

• Allows referencing other fields in the JSON using special path syntax.

• You can assign fixed values like strings or numbers directly without any computation.

Path Syntax:

• @(1,<path>): Refers to the current JSON structure after the last operation and fetches a value from the JSON for computation.

• Example: @(1,details.productPrice) retrieves the value of details.productPrice from the current structure.

Example:

//Input:
{
  "product": {
    "id": "123",
    "name": "Shirt",
    "price": 1000,
    "discount": 200,
    "internalCode": "ABC-123",
    "unusedValue": "foobar"
  },
  "stock": {
    "location1": 10,
    "location2": 5
  }
}

//Specification:
[
  {
    "operation": "modify-default-beta",
    "spec": {
    "product": {
        "finalPrice": "=intSubtract(@(1,price),@(1,discount))",
        "fullName": "=concat(@(1,name),' ',@(1,internalCode))"
    }
    }
  }
]

//Result:
{
  "product" : {
    "id" : "123",
    "name" : "Shirt",
    "price" : 1000,
    "discount" : 200,
    "internalCode" : "ABC-123",
    "unusedValue" : "foobar",
    "finalPrice" : 800,
    "fullName" : "Shirt ABC-123"
  },
  "stock" : {
    "location1" : 10,
    "location2" : 5
  }
}

In this case:

• finalPrice is calculated using price and discount values, with intSubtract method.

• fullName is calculated using name and internalCode values, with concat method.

JOLT Supports a variety of Math, string, list methods. Some popular ones are, intSum, doubleSum, intSubtract, avg, divide, max, min for math functions and concat, toUpper, toLower for string methods, more methods and examples over them can be found at Jolt Demo website or Jolt framework source codes.

Remove Method​

The remove operation removes specific fields from the JSON.

Example:

//Input:
{
  "product": {
    "id": "123",
    "name": "Shirt",
    "price": 1000,
    "discount": 200,
    "internalCode": "ABC-123",
    "unusedValue": "foobar",
    "valueToRemove1": "1",
    "valueToRemove2": "2",
    "valueToRemove3": "3"
  },
  "stock": {
    "location1": 10,
    "location2": 5
  }
}

//Specification:
[
  {
    "operation": "remove",
    "spec": {
    "product": {
        "unusedValue": "",
        "valueToRemove*": ""
    }
    }
  }
]

//Result:
{
  "product" : {
    "id" : "123",
    "name" : "Shirt",
    "price" : 1000,
    "discount" : 200,
    "internalCode" : "ABC-123"
  },
  "stock" : {
    "location1" : 10,
    "location2" : 5
  }
}

In this case,

• “unusedValue” is removed from the input by specifying “unusedValue” in the specification.

• “valueToRemove1”, “valueToRemove2”, etc. are removed from the input by matching with “valueToRemove” key, “” character can be used to match with everything.

Working With Arrays or Objects in JOLT​

When working with arrays or dynamic keys in JSON data, JOLT provides special symbols like * and & to iterate over arrays and capture keys or values dynamically. Here’s a detailed explanation with examples:

Using * for Iterating Over Arrays​

The * symbol acts as a wildcard and is used to match every element in an array or object. It allows you to iterate through arrays and process each element individually.

Example:

//Input:
{
  "products": [
    { "id": "101", "name": "Laptop" },
    { "id": "102", "name": "Mouse" },
    { "id": "103", "name": "Keyboard" }
  ]
}

//Specification:
[
  {
    "operation": "shift",
    "spec": {
    "products": {
        "*": {
        "id": "productIds[]",
        "name": "productNames[]"
        }
    }
    }
  }
]

//Result:
{
  "productIds" : [ "101", "102", "103" ],
  "productNames" : [ "Laptop", "Mouse", "Keyboard" ]
}

• products.*: Matches every element in the products array.

• id and name: Extract the corresponding fields from each array element.

• productIds[] and productNames[]: Appends the values to an array in the output named productIds and productNames created in the root object.

Please note that, productIds and productNames are created in the result objects root not in their respective product objects. & value can be used for specifying which product the value belongs to.

Using & for Capturing Keys or Values Dynamically​

The & symbol is used to capture the matched key or index dynamically from the input JSON. It is especially useful for creating mappings where keys or indices are needed in the output.

Example:

//Input:
{
  "stock": {
    "location1": 10,
    "location2": 20,
    "location3": 30,
    "foobar": 5
  }
}

//Specification:
[
  {
    "operation": "shift",
    "spec": {
    "stock": {
        "location*": "stockDetails.&"
    }
    }
  }
]

//Result:
{
  "stockDetails" : {
    "location1" : 10,
    "location2" : 20,
    "location3" : 30
  }
}

• stock.location*: Matches every key-value pair in the stock object starting with location.

• &: Captures the key (e.g., location1) dynamically and uses it as a key in the output JSON.

Using #, $ and @ for Transposing Array and Objects​

You can combine *, & and @ to iterate over arrays or objects and dynamically map their keys or values using # or $.

• # is used for counting on the right hand side of specification. Since on an object & is used for accessing the key value instead of index, # lets you create an index based on how many keys have been matched so far.

• $ is used for using the key name as a value on the left hand side of specification.

• @ basically refers to a value, it can be used on both left hand side and right hand side of the specification, there are examples on right hand side usage on “Modify Default Method” examples.

Example:

//Input:
{
  "products": [
    {
    "id": "101",
    "name": "Laptop",
    "stock": {
        "warehouse1": 5,
        "warehouse2": 10
    }
    },
    {
    "id": "102",
    "name": "Mouse",
    "stock": {
        "warehouse1": 15,
        "warehouse2": 20
    }
    }
  ]
}

//Specification:
[
  {
    "operation": "shift",
    "spec": {
    "products": {
        "*": {
        "id": "products[&1].productId",
        "name": "products[&1].productName",
        "stock": {
            "*": {
            "$": "products[&3].stock[#2].location",
            "@": "products[&3].stock[#2].stock"
            }
        }
        }
    }
    }
  }
]

//Result:
{
  "products" : [ {
    "productId" : "101",
    "productName" : "Laptop",
    "stock" : [ {
    "location" : "warehouse1",
    "stock" : 5
    }, {
    "location" : "warehouse2",
    "stock" : 10
    } ]
  }, {
    "productId" : "102",
    "productName" : "Mouse",
    "stock" : [ {
    "location" : "warehouse1",
    "stock" : 15
    }, {
    "location" : "warehouse2",
    "stock" : 20
    } ]
  } ]
}

Explanation

• Products array is iterated using *, for each product, [&1] is used for referencing the index of each product making sure when the values inside are used they are copied to the correct product:

  • id value is copied as productId

  • name value is copied as productName

  • Each stock value is iterated over, in this context [&3] refers to the product's index while [&2] would refer to the key name (e.g. warehouse1). [#2] value instead refers to the count of matched keys (for warehouse1 it is 0, warehouse2 it is 1)

    • For each stock value, the key is passed as a value using $ and copied over to product [&3] and warehouse [#2] as “location”

    • For each stock value, the current value is received using @ and copied over to product [&3] and warehouse [#2] as “stock”