Overview
The Duplicates node is a data preparation node in a Weave Workflow that drops duplicates from the upstream data. “Duplicate” is defined by a Unique Key — one or more columns whose values together identify a row’s identity. Two rows with matching Unique Key values are duplicates of each other; the node drops all but one.
Duplicates nodes are added from the Data Preparation category of the node panel alongside Cleansing, Imputation, Filter, OutlierTreatment, Binning, and Trimming.
Note: “Workflow” is the in-UI term for what some Weave documentation calls a pipeline. This reference uses “Workflow.”
When to use it
- Enforcing row uniqueness on chosen columns before downstream joins (joining against duplicate rows produces row multiplication).
- Reducing a transactional dataset to one row per entity — one record per customer, one record per product, one record per session.
- Cleaning data that arrived from sources where the same logical record was loaded multiple times.
Configuration
Unique Key
The Unique Key defines what makes two rows duplicates of each other. Required configuration.
| Select (default Select All) | Opens a column-selection popup. By default, every column is checked (Select All). Deselect columns to narrow the duplicate definition. |
| Unique Key composition | Single-column key: one row per distinct value of that column survives. Multi-column key: distinct combinations of the selected columns survive. |
| Column order significance | When Partition and Ranking is enabled, the first column in the Unique Key selection serves as the sort key. Reorder the selection if a different column should drive the sort. |
| Apply | Commits the column selection. |
Partition and Ranking (optional)
Partition and Ranking controls the sort (ascending or descending). When unchecked, the kept row is determined by upstream order; when checked, rows within each duplicate group are sorted by the first Unique Key column and the first row is kept.
| Partition and Ranking (checkbox) | Off: no explicit row-selection logic; first-encountered row per duplicate group is kept. On: enables the Sorting dropdown for explicit selection. |
| Sorting | Dropdown with Select (placeholder), Ascending, Descending. Ascending keeps the smallest / earliest value of the first Unique Key column per group; Descending keeps the largest / latest. |
| Sort key source | The first column in the Unique Key selection. For a single-column Unique Key, that column is the sort key. For a multi-column composite Unique Key, the first column in the selection is used. The remaining Unique Key columns participate in duplicate identification but not in sort ordering. |
Three-layer progressive configuration
Duplicates uses a progressive disclosure pattern — each layer is optional except the first, and enabling a layer reveals additional controls.
| Unique Key | Yes (always required) | Column-selection popup | Defines what “duplicate” means. |
| Partition and Ranking | Optional checkbox | Sorting dropdown | Controls which row in each duplicate group survives, sorted by the first Unique Key column. |
| Filter Table | Optional checkbox | Filter button → Filter Criteria popup | Filters the clean records. |
Key behaviors
Unique Key defines duplicate identity. Two rows are duplicates if and only if they have matching values in every column of the Unique Key. A wider Unique Key (more columns) makes duplicates harder to find — fewer drops. A narrower Unique Key (fewer columns) makes duplicates more common — more drops. The default Select All produces the most conservative behavior (only fully-identical rows are duplicates).
Three layers compose cleanly. Unique Key + Partition and Ranking + Filter Table can all be configured on the same node. The combination produces a focused deduplication.
Progressive disclosure mirrors common usage. The simplest case (drop all duplicates) needs only the Unique Key. Adding Partition and Ranking when row selection matters; adding Filter Table when scope restriction matters. The UI surfaces only what’s needed by hiding the optional layers behind their checkboxes.
Filter Criteria structure matches other Weave filter UIs. The Column / Filter Condition / Input / Action / Logic / Add Group / Delete Group structure is consistent with the canonical Filter Criteria UI used in the Filter node.