Quick wins for a faster PC:
Clear out junk files and repair common Windows errorsFree Scan →Scan for outdated or missing drivers - takes under a minuteDriver Scan →Duplicate detection is a common transformation requirement in Mule 4 integrations, especially when validating incoming payloads, cleaning data before sending it downstream, or identifying repeated records in API requests. With DataWeave 2.0, you can handle this cleanly using functions such as groupBy, filterObject, pluck, and sizeOf.
The basic idea is to group array values by their own value, inspect each group, and keep only the entries that appear more than once. From there, you can return just the duplicate values, include occurrence counts, or adapt the same pattern to arrays of objects by grouping on a specific field such as id, email, or name.
As an Amazon Associate I earn from qualifying purchases.
Understanding the Duplicate Detection Requirement
Before writing a DataWeave 2.0 transformation, it is useful to define what “duplicate” means for the array you are processing. In a simple array such as ["A", "B", "A", "C"], the value "A" is duplicated because it appears more than once. In Mule 4 integrations, this type of check is common when validating incoming payloads, detecting repeated identifiers, cleaning reference data, or preventing duplicate records from being sent to a downstream system.
Do these 3 things before closing this tab:
1Clear out junk files and repair common Windows errors2Scan for outdated or missing drivers - takes under a minute3Repair Windows errors before they cause bigger problemsThe required output also matters. Sometimes you only need to know whether duplicates exist. In other cases, you need a list of duplicated values, the original repeated items, or a count of how many times each value appears. These are different requirements and they lead to slightly different DataWeave expressions. For example, returning ["A"] is different from returning ["A", "A"], and both are different from returning an object such as { "A": 2 }.
#1 Best Overall
- Hardcover journal with 240 line-ruled pages (120 sheets)
- Built-in elastic closure and ribbon bookmark
- Includes an expandable inner storage pocket and a pen holder
For simple arrays, duplicate detection usually compares the array values directly. This works well for strings, numbers, booleans, and other scalar values. A practical approach in DataWeave is to group the values first, then filter the groups whose size is greater than one. The grouped structure makes it clear which values appear once and which values appear mulle times.
When the array contains objects, the comparison should usually be based on one or more fields rather than the whole object. For example, in an array of customer records, two records may be considered duplicates if they share the same customerId or email, even if other fields such as name or lastUpdated are different. In that case, the duplicate detection expression should group by the selected key field, not by the entire object.
Typical duplicate detection outputs
- Boolean result: returns whether any duplicate exists, such as
trueorfalse. - Unique duplicate values: returns each duplicated value once, such as
["A", "B"]. - Repeated items: returns the actual repeated entries from the original array.
- Duplicate counts: returns each duplicated value with its occurrence count, such as
{ "A": 3, "B": 2 }. - Object duplicates by field: returns records or keys that repeat based on fields such as
id,email, ororderNumber.
Consider the array ["apple", "banana", "apple", "orange", "banana", "apple"]. If the requirement is to return duplicate values only once, the expected result is ["apple", "banana"]. If the requirement is to count duplicates, the expected result is closer to { "apple": 3, "banana": 2 }. If the requirement is to keep all repeated occurrences, the output may include mulle instances of the same value. Defining this expected shape first keeps the DataWeave script simple and avoids confusion later in the flow.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
In Mule 4, duplicate detection is often placed in a Transform Message component before validation, routing, or persistence. The transformation can produce a clean structure that an error handler, choice router, or logging step can use. Once the requirement is clear, DataWeave functions such as groupBy, filterObject, pluck, filter, and sizeOf can be combined to return exactly the duplicate information needed.
Sample Input Array and Expected Output
Before writing the DataWeave expression, it helps to define a small input array and agree on what the duplicate result should look like. In Mule 4, this array might come from a JSON payload, a variable, a database result, or a field extracted from a larger object. For a simple case, assume the payload is an array of strings where some values appear more than once.
[
"apple",
"banana",
"orange",
"apple",
"grape",
"banana",
"apple"
]
There are several valid outputs depending on the integration requirement. If the goal is only to know which values are duplicated, the expected output should contain each duplicate value once. For the sample input above, both "apple" and "banana" are duplicates, while "orange" and "grape" appear only once.
[
"apple",
"banana"
]
In other scenarios, the consuming system may need the duplicate value along with the number of times it appears. This is useful for validation responses, audit logs, error messages, and reconciliation flows where the duplicate count matters. In that case, the expected output can be shaped as an array of objects.
The Tool Desk
Outbyte Driver Updater FREEFix the driver behind crashes, sound loss and screen glitchesFind Drivers →Outbyte PC Repair FREEClear out junk files and repair common Windows errorsFree Scan →[
{
"value": "apple",
"count": 3
},
{
"value": "banana",
"count": 2
}
]
If the requirement is to return every repeated occurrence rather than each duplicate value once, the output will be different. For example, after the first "apple" is seen, the next two "apple" entries are repeated occurrences. After the first "banana" is seen, the second "banana" is also a repeated occurrence. This format is less common for validation, but it can be useful when preserving the original sequence of duplicate entries.
[
"apple",
"banana",
"apple"
]
Example with numbers
The same approach works for numeric arrays. DataWeave can group, filter, and count numbers just like strings, as long as the comparison value is consistent. Given this input:
[
101,
102,
103,
101,
104,
102
]
A duplicate-only result would be:
[
101,
102
]
A count-based result would be:
[
{
"value": 101,
"count": 2
},
{
"value": 102,
"count": 2
}
]
Choosing the expected output shape
- Return unique duplicate values when the flow only needs to reject or flag repeated entries.
- Return values with counts when the response must explain how many times each value appears.
- Return repeated occurrences when order and repeated positions are relevant to downstream processing.
These sample inputs and outputs provide a clear target for the DataWeave transformations in the next sections. Once the desired shape is defined, functions such as groupBy, filterObject, pluck, sizeOf, and distinctBy can be combined to produce the exact duplicate format required by the Mule flow.
Recommended Free Tools
Finding Duplicate Values with groupBy
In Mule 4 DataWeave 2.0, groupBy is one of the most practical functions for detecting duplicate values because it reorganizes an array into buckets. Each bucket is created from a value in the array, and all matching entries are placed together. Once the values are grouped, any bucket with more than one item represents a duplicated value.
For a simple array such as [1, 2, 3, 2, 4, 1, 5], the groupBy function can group each number by its own value:
%dw 2.0
output application/json
var numbers = [1, 2, 3, 2, 4, 1, 5]
---
numbers groupBy ((item) -> item)
The result is an object where each property name is the grouped value and each property value is an array containing all matching items:
Rank #2
- Colorful statistics humor for data scientists, analysts, researchers, and evidence-driven thinkers who believe claims deserve numbers, charts, and reproducible results.
- A lively choice for labs, classrooms, analytics teams, coding sessions, and research events. Add this data-minded joke to your regular rotation.
- Hardcover journal with 240 line-ruled pages (120 sheets)
- Built-in elastic closure and ribbon bookmark
- Includes an expandable inner storage pocket and a pen holder
{
"1": [1, 1],
"2": [2, 2],
"3": [3],
"4": [4],
"5": [5]
}
This grouped structure makes duplicate detection straightforward. Values such as 1 and 2 have arrays with a size greater than 1, so they are duplicates. Values such as 3, 4, and 5 appear only once.
Free tools Windows power users keep installed
One-click scans. No signup required.
Filtering grouped values to find duplicates
After grouping, use filterObject to keep only the entries where the grouped array contains more than one item. This returns an object that contains only the duplicated values and their repeated occurrences:
%dw 2.0
output application/json
var numbers = [1, 2, 3, 2, 4, 1, 5]
var groupedNumbers = numbers groupBy ((item) -> item)
---
groupedNumbers filterObject ((items, key) -> sizeOf(items) > 1)
The output is:
{
"1": [1, 1],
"2": [2, 2]
}
This format is useful when the integration needs both the duplicate value and the matching items that caused the duplication. For example, it can be used before logging validation failures, rejecting records, or building an error response that shows which values were repeated in the payload.
Returning duplicate values as an array
If only the duplicate values are needed, apply keysOf after filtering the grouped object. Since object keys are strings in JSON output, numeric values grouped as keys are returned as strings unless they are explicitly converted back to numbers.
%dw 2.0
output application/json
var numbers = [1, 2, 3, 2, 4, 1, 5]
var groupedNumbers = numbers groupBy ((item) -> item)
---
keysOf(groupedNumbers filterObject ((items, key) -> sizeOf(items) > 1))
The output is:
[
"1",
"2"
]
To return the duplicate values as numbers, map each key with as Number:
%dw 2.0
output application/json
var numbers = [1, 2, 3, 2, 4, 1, 5]
var groupedNumbers = numbers groupBy ((item) -> item)
---
keysOf(groupedNumbers filterObject ((items, key) -> sizeOf(items) > 1))
map ((key) -> key as Number)
The output becomes:
[
1,
2
]
This same pattern can be adapted for strings, identifiers, email addresses, product codes, or other scalar values. For case-sensitive string comparisons, values such as "ABC" and "abc" are treated as different groups. If those should be considered the same, normalize the value inside groupBy, for example by using lower(item) when grouping string values.
Returning Only Duplicate Items
After using groupBy, the next common step is to return only the values that appear more than once. In DataWeave 2.0, this usually means grouping the array by value, selecting groups whose size is greater than one, and then converting those selected groups back into a simple array of duplicate values.
For a simple input array like this:
%dw 2.0
output application/json
var numbers = [1, 2, 3, 2, 4, 5, 3, 6]
---
numbers
You can return only the duplicated values by combining groupBy, filterObject, and keysOf:
%dw 2.0
output application/json
var numbers = [1, 2, 3, 2, 4, 5, 3, 6]
---
numbers
groupBy $
filterObject ((items, value) -> sizeOf(items) > 1)
keysOf
The result is:
[
"2",
"3"
]
One detail to be aware of is that keysOf returns object keys as strings. Since groupBy creates an object where each grouped value becomes a key, numeric values such as 2 and 3 are returned as "2" and "3". If the downstream system expects numbers instead of strings, you can cast the keys back to numbers:
%dw 2.0
output application/json
var numbers = [1, 2, 3, 2, 4, 5, 3, 6]
---
numbers
groupBy $
filterObject ((items, value) -> sizeOf(items) > 1)
keysOf
map ($ as Number)
This returns:
Rank #3
- Field Notes Notebooks Cover: A5 notebook cover Waterproof wear-resistant and, It Fits Most 6.7 X 9.8 Notebooks, Planners, Journals, And Sketchbooks. The card slots and pockets inside can organize your cards, tickets, cash, etc. The pen holder can keep the pen in place.
- Travelers Notebook Cover: This pocket notebook cover Two Pockets On Front Cover, Is Also Designed With Three Card Slots And Two Pockets. you Can Store Your Small Notebook In The Right Pocket And Cash In The Left Pocket It'S Also fits for Storing Your Credit Cards, Id Cards, Or Other Small Accessories, Like U-Disk, Sd Card, Charging Cable Etc.It'S A Versatile And Practical Accessory That'S fits for Everyday Use.
- Waterproof Canvas Pockets: Journal Notebook Cover Made With Classy Heavy Duty Waxed Canvas, Durable And Water-Resistant, To Properly Hold Your Field Notes, It'S fits for Outdoor Adventures Or Business Travel.
- Pocket Zipper Notebook Cover With Pen Holder: A Designed Pen Holder On Front Cover To Holder Multiple Pens Or Small Tools For Easy Access. It Also Has A Pen Loop Design In The Middle To Securely Hold Your Pen So That You Can Use It Anytime, This Field Notes Planner Is fits for Keeping All Your Writing Tools Organized And Easily Accessible.
- A5 Planner Cover Fits Most Occasion: This Journal Cover Makes A Thoughtful & Long Lasting Your Family And Friends As An Anniversary, Birthday Present, It'S fits for Anyone Who Values Organization And Practicality, Such As Writers, Business Professionals, Journalists, And Travelers.
[
2,
3
]
Returning duplicate occurrences instead of unique duplicate values
Sometimes you do not want only the unique duplicated values. You may want every item that belongs to a duplicated group. For example, from [1, 2, 3, 2, 4, 5, 3, 6], the expected result may be [2, 2, 3, 3] instead of [2, 3]. In that case, filter the grouped values and flatten the remaining arrays:
%dw 2.0
output application/json
var numbers = [1, 2, 3, 2, 4, 5, 3, 6]
---
numbers
groupBy $
filterObject ((items, value) -> sizeOf(items) > 1)
pluck ((items, value) -> items)
flatten
The result is:
[
2,
2,
3,
3
]
This version preserves the duplicated occurrences, which is useful when you need to reject, audit, or display all repeated entries rather than only listing which values were repeated.
Using filter and count-style comparison
For smaller arrays, another readable approach is to use filter and compare each item against the number of matching items in the same array:
%dw 2.0
output application/json
var numbers = [1, 2, 3, 2, 4, 5, 3, 6]
---
numbers filter ((item) ->
sizeOf(numbers filter ($ == item)) > 1
)
This returns all duplicate occurrences:
[
2,
3,
2,
3
]
If you want unique duplicate values from this result, apply distinctBy:
%dw 2.0
output application/json
var numbers = [1, 2, 3, 2, 4, 5, 3, 6]
---
(numbers filter ((item) ->
sizeOf(numbers filter ($ == item)) > 1
)) distinctBy $
For larger arrays, the groupBy approach is usually cleaner and avoids repeatedly scanning the same array for each item. The filter approach can still be useful in quick transformations or when the expression needs to stay close to the original array structure.
Counting Duplicate Occurrences
After identifying which values are duplicated, the next common requirement is to return how many times each duplicate appears. In Mule 4 DataWeave 2.0, this is usually done by combining groupBy, filterObject, and sizeOf. The idea is to group equal values together, keep only groups with more than one element, and then transform each group into a count.
For a simple array such as:
%dw 2.0
output application/json
var numbers = [10, 20, 10, 30, 20, 20, 40]
---
numbers
You can return each duplicate value with its occurrence count like this:
%dw 2.0
output application/json
var numbers = [10, 20, 10, 30, 20, 20, 40]
var groupedNumbers = numbers groupBy ((item) -> item)
---
groupedNumbers
filterObject ((items, key) -> sizeOf(items) > 1)
mapObject ((items, key) -> {
(key): sizeOf(items)
})
The output is:
{
"10": 2,
"20": 3
}
In this result, the object key represents the duplicated value, and the value represents the number of times it appeared in the original array. DataWeave object keys are strings, so numeric values such as 10 and 20 appear as "10" and "20" in the JSON output.
Crashes, No Sound, or Screen Glitches?
Random freezes, missing sound and display glitches usually trace back to one bad driver. Find and replace yours safely.Free scan · under a minutePC Slower Than It Used to Be?
A free scan shows the junk files, broken settings and background clutter dragging Windows down - then fixes them in one click.Free scan · Windows 10 & 11Returning counts as an array of objects
Sometimes an array is easier to consume than an object, especially when sending the result to an API, writing to a database, or displaying it in a table. You can convert the grouped result into a list of objects using pluck:
%dw 2.0
output application/json
var numbers = [10, 20, 10, 30, 20, 20, 40]
---
(numbers groupBy ((item) -> item))
filterObject ((items, key) -> sizeOf(items) > 1)
pluck ((items, key) -> {
value: key as Number,
count: sizeOf(items)
})
The output is:
[
{
"value": 10,
"count": 2
},
{
"value": 20,
"count": 3
}
]
This structure is often more practical because each duplicate is represented as a separate record. If the original array contains strings, remove the as Number conversion and return value: key instead.
Rank #4
- Full-Size Daily Folio A5 Cover: Features a structured writing board, organized storage, & room for up to 4 slim A5 notebooks or larger notes up to 6" × 10"—ideal for work bags, backpacks, & briefcases
- A5 Notebook Compatible & Refillable: Fits popular A5 notebooks including Moleskine, Leuchtturm, Field Notes A5, 5 × 8 padfolio, & other standard inserts. Includes a dotted refill 5.5" × 8.3" journal with 64 pages
- Multi-Function Writing Organizer: Four elastic bands hold notebooks, refills, or Traveler’s Notebook inserts, while an exterior multi-pen pocket, interior pen pocket & fountain pen loop organize writing tools
- Designed for Writers On the Move: Extra pockets keep cards, notes, USB drives & other small essentials handy. A reliable companion for journalers, professionals, travelers, & field writers who value organized carry, & lasting craftsmanship
- Waxed Canvas & Leather Craftsmanship: Built with rugged waxed canvas, genuine leather spine, reinforced stitching, & wraparound zipper closure for durable everyday carry and a secure, comfortable grip
Including all values with their counts
If you need counts for every value, including values that occur only once, simply remove the duplicate filter:
Do these 3 things before closing this tab:
1Clear out junk files and repair common Windows errors2Fix the driver behind crashes, sound loss and screen glitches3Repair Windows errors before they cause bigger problems%dw 2.0
output application/json
var numbers = [10, 20, 10, 30, 20, 20, 40]
---
(numbers groupBy ((item) -> item))
pluck ((items, key) -> {
value: key as Number,
count: sizeOf(items),
duplicated: sizeOf(items) > 1
})
This returns a full frequency list and adds a boolean field that clearly indicates whether the value is duplicated. That pattern works well when downstream systems need both the counts and the duplicate status without recalculating anything.
Adapting counts for arrays of objects
The same counting approach can be applied to object arrays by grouping on a specific field. For example, to count repeated email addresses:
%dw 2.0
output application/json
var users = [
{ id: 1, email: "[email protected]" },
{ id: 2, email: "[email protected]" },
{ id: 3, email: "[email protected]" }
]
---
(users groupBy ((user) -> user.email))
filterObject ((items, email) -> sizeOf(items) > 1)
pluck ((items, email) -> {
email: email,
count: sizeOf(items),
records: items
})
This returns the duplicated email, how many times it appears, and the matching records. Keeping the original records in the result is useful when you need to inspect conflicting rows or pass them into another validation step.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Handling Arrays of Objects in DataWeave
Duplicate detection often moves beyond simple arrays such as ["A", "B", "A"]. In Mule 4 integrations, the payload is commonly an array of objects, and the duplicate condition is usually based on one or more fields. For example, two customer records may be considered duplicates when they share the same email, while two order lines may be duplicates when they share the same sku. DataWeave 2.0 handles this cleanly by using groupBy with a field selector.
Grouping objects by a single field
Given an array of customer objects, the following script groups records by email, filters the groups that contain more than one record, and returns the duplicated email values:
%dw 2.0
output application/json
var customers = [
{ id: 1, name: "Ana", email: "[email protected]" },
{ id: 2, name: "Ben", email: "[email protected]" },
{ id: 3, name: "Ana M", email: "[email protected]" },
{ id: 4, name: "Cara", email: "[email protected]" }
]
---
customers
groupBy $.email
filterObject ((records, email) -> sizeOf(records) > 1)
pluck ((records, email) -> email)
The output is an array of duplicate field values:
[
"[email protected]"
]
Returning duplicate records instead of duplicate values
In many flows, the next step is not just to know which value is duplicated, but to inspect or route the full records that caused the duplication. After filtering the grouped object, use pluck to get the grouped arrays and flatten to return a single array of records:
Recommended Free Tools
%dw 2.0
output application/json
var customers = [
{ id: 1, name: "Ana", email: "[email protected]" },
{ id: 2, name: "Ben", email: "[email protected]" },
{ id: 3, name: "Ana M", email: "[email protected]" },
{ id: 4, name: "Cara", email: "[email protected]" }
]
---
customers
groupBy $.email
filterObject ((records, email) -> sizeOf(records) > 1)
pluck ((records, email) -> records)
flatten
This returns both objects that share the same email address:
[
{ "id": 1, "name": "Ana", "email": "[email protected]" },
{ "id": 3, "name": "Ana M", "email": "[email protected]" }
]
Including duplicate counts for object arrays
If the result needs to show each duplicated value and how many times it appears, return an object from pluck. This format is useful for validation responses, audit logs, or error payloads sent back to an API consumer:
%dw 2.0
output application/json
var customers = [
{ id: 1, name: "Ana", email: "[email protected]" },
{ id: 2, name: "Ben", email: "[email protected]" },
{ id: 3, name: "Ana M", email: "[email protected]" },
{ id: 4, name: "Cara", email: "[email protected]" }
]
The Tool Desk
Outbyte Driver Updater FREEFix the driver behind crashes, sound loss and screen glitchesFind Drivers →Outbyte PC Repair FREEClear out junk files and repair common Windows errorsFree Scan →---
customers
groupBy $.email
filterObject ((records, email) -> sizeOf(records) > 1)
pluck ((records, email) -> {
email: email,
count: sizeOf(records),
records: records
})
For composite duplicates, build a grouping value from mulle fields. For example, duplicate order lines may be identified by the combination of orderId and sku. A safe pattern is to concatenate fields with a delimiter that will not appear in the data, or to normalize the fields before grouping when case and spacing should not create separate groups:
Best Value
- PREMIUM QUALITY: Made with sturdy and padded lining materials, This Journal Supplies Storage Case offers long-lasting protection for your personal organizer against dust, scratches, and other damage. Smooth zippper closure to keep your items put in and out easily.
- LARGE CAPACITY: L8.5"*W3"*H12". Perfect for B5/A5 size notebook/planner/journal, or book up to 12" x 8" x 1.25". 2 front pockets, and multiple inner organizing pockets allow you to store everything you need in one bag, such as ipads, celephone, charger,stickers,passport, papers, handmade accessories, school/office stationeries or travel gear.
- PEN SLOTS DESIGN: This journal supplies case with 9 pen slots for your pens, pencil, highlighters. 2 cards slots for your differents cards.
- EASY TO CARRY: The sturdy handles and adjustable shoulder strap make you can carry it comfortably, whether you are traveling or studying in a coffee shop.
- MULTI-FUNCTIONAL: This journal supplies organizer can protect your journal, calendar, planner accessories or DIY projects. You can use it as a book cover/bible cover, pen case, stationery pencil bag, purse style handbag. It is a nice choice for your family and friends on Easter, Christmas and Halloween.
%dw 2.0
output application/json
var lines = [
{ orderId: "O-100", sku: "SKU-1", qty: 2 },
{ orderId: "O-100", sku: "SKU-2", qty: 1 },
{ orderId: "O-100", sku: "SKU-1", qty: 5 },
{ orderId: "O-200", sku: "SKU-1", qty: 3 }
]
---
lines
groupBy ((item) -> item.orderId ++ "|" ++ item.sku)
filterObject ((records, groupKey) -> sizeOf(records) > 1)
pluck ((records, groupKey) -> {
duplicateKey: groupKey,
count: sizeOf(records),
records: records
})
When adapting these examples, handle missing or null fields deliberately. If several objects have null for the grouped field, they will be grouped together and may appear as duplicates. If that is not desired, filter those records before grouping, such as filter $.email != null. For case-insensitive checks, group by a normalized expression like lower($.email), and for values entered by users, consider trimming spaces with trim($.email) before applying groupBy.
Common Pitfalls and Best Practices
Duplicate detection in Mule 4 DataWeave 2.0 is usually straightforward, but small implementation details can change the result significantly. The most common issue is comparing values that look similar but are not actually equal. For example, the string “101” and the number 101 are different values, and “ABC” is different from “abc”. Before using groupBy or filter, decide whether values should be normalized with functions such as lower, trim, or explicit type casting.
PC Slower Than It Used to Be?
A free scan shows the junk files, broken settings and background clutter dragging Windows down - then fixes them in one click.Free scan · Windows 10 & 11Outdated Drivers Are Slowing You Down
One free scan finds every outdated or missing driver and matches the right update for your exact hardware.Free scan · exact hardware matchFor simple arrays, keep the duplicate check based on a consistent key. If the array may contain mixed casing or extra spaces, normalize the value before grouping:
%dw 2.0
output application/json
var items = ["A001", "a001", " A002 ", "A003", "A002"]
---
items
groupBy ((item) -> lower(trim(item)))
filterObject ((value, key) -> sizeOf(value) > 1)
pluck ((value, key) -> key)
This returns the normalized duplicate keys rather than the original values. If the original values matter, return the grouped array instead. That lets you preserve exactly what came in while still detecting duplicates using a cleaned comparison value.
Another frequent pitfall is returning every repeated item when the requirement is to return each duplicate value only once. For example, filtering an array with countBy or repeated filter may produce ["A001", "A001", "A001"] when the desired output is only ["A001"]. Using groupBy, then filterObject, then pluck is usually clearer when the output should contain unique duplicate values:
%dw 2.0
output application/json
var items = ["A001", "A002", "A001", "A003", "A002"]
---
items
groupBy $
filterObject ((groupedItems, key) -> sizeOf(groupedItems) > 1)
pluck ((groupedItems, key) -> key)
For arrays of objects, avoid grouping by the whole object unless the entire object must match exactly. In most integration flows, duplicates are based on a business field such as id, email, orderNumber, or a composite key. Be explicit about that key, and handle missing or null values deliberately:
%dw 2.0
output application/json
var customers = [
{ id: "C001", email: "[email protected]" },
{ id: "C002", email: "[email protected]" },
{ id: "C001", email: "[email protected]" },
{ id: null, email: "[email protected]" }
]
---
customers
filter ((customer) -> customer.id != null)
groupBy ((customer) -> customer.id)
filterObject ((groupedCustomers, id) -> sizeOf(groupedCustomers) > 1)
When returning counts, choose an output structure that downstream systems can consume easily. An array of objects is often more API-friendly than a grouped object because it has stable field names:
%dw 2.0
output application/json
var items = ["A001", "A002", "A001", "A003", "A002", "A002"]
---
items
groupBy $
filterObject ((value, key) -> sizeOf(value) > 1)
pluck ((value, key) -> {
value: key,
count: sizeOf(value)
})
For best results, separate duplicate detection into small transformation steps: normalize the comparison value, group by the selected key, filter groups with more than one item, and shape the response. This keeps the DataWeave readable, makes testing easier, and reduces surprises when the same pattern is adapted from simple arrays to arrays of objects.
Frequently Asked Questions
How do I return only the values that appear more than once in a DataWeave array?
Use groupBy to group the array by value, then filter groups where the size is greater than 1. For example, with [1, 2, 2, 3, 4, 4], grouping by $ lets you identify 2 and 4 as duplicates. You can then return the grouped keys or flatten the duplicate groups depending on the output you need.
How can I get duplicate values along with their occurrence counts?
After grouping the array, map each group to an object that contains the value and its count. A common output format is [{ value: "A", count: 3 }, { value: "B", count: 2 }]. Filter the grouped result with sizeOf($) > 1 before mapping if you only want duplicate entries.
Quick wins for a faster PC:
Scan for outdated or missing drivers - takes under a minuteDriver Scan →Repair Windows errors before they cause bigger problemsFix Now →What is the difference between returning duplicate values and returning duplicate items?
Returning duplicate values usually means returning each duplicated value once, such as ["A", "B"]. Returning duplicate items means returning all repeated occurrences from the original array, such as ["A", "A", "B", "B", "B"]. In DataWeave, the first approach typically uses grouped keys, while the second approach uses the grouped arrays and flattens them.
How do I find duplicates in an array of objects using a specific field?
Use groupBy with the field that defines uniqueness, such as groupBy $.email or groupBy $.id. Then filter the grouped records where the group size is greater than 1. This lets you detect duplicate customers, users, orders, or products based on a business key rather than the entire object.
What should I watch out for when detecting duplicates in DataWeave?
Be consistent with data types, because 1 and "1" are not the same value. Normalize strings when needed by using functions such as lower or trim before grouping, especially for emails or names. Also consider how null or missing fields should be handled when grouping arrays of objects.
Bottom Line
Finding duplicates in Mule 4 DataWeave 2.0 is straightforward once you choose the output you need: use grouping when you want counts, filtering when you only need repeated values, and key-based grouping when working with arrays of objects. These patterns are easy to adapt for IDs, emails, product codes, or any field that must be checked for repetition.
As a next step, start with a simple groupBy-based solution, then shape the result to match your integration requirement—either a list of duplicate values, a count per value, or the original duplicate records for downstream validation and error handling.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
