How to Deduplicate Leads in n8n Before They Reach Your CRM

Lead deduplication is not one filter. A reliable n8n workflow separates repeated submissions from repeated people, normalizes the values used for comparison, checks persistent records, and lets the CRM enforce uniqueness whenever the destination supports it.

First identify which duplicate you are stopping

The Facebook Lead Ads workflow uses a Meta lead ID to stop one submission from creating several sheet rows. This guide goes further by handling several sources and an existing CRM full of contacts.

Choose identity keys before building nodes

Decide field ownership too. A new form may fill a missing phone number, but it should not silently replace a verified CRM email, owner, lifecycle stage, or consent value. Deduplication decides which record to use. A separate update policy decides which fields may change.

A practical n8n workflow shape

Lead trigger
  → Edit Fields: normalize keys
  → Remove Duplicates: current batch or repeated event
  → Data Table or CRM: persistent lookup
  → Switch: new / exact match / conflict
  → CRM: create or update
  → Data Table: record source key and CRM ID
  → Alert only when review or failure is needed

Start with one lead source and one CRM. The same pattern can later sit behind Facebook forms, website forms, imports, chat, and email without giving each source a different duplicate policy.

Step 1: normalize the fields once

Add an Edit Fields node immediately after the trigger. Preserve the original values for audit and display, then create separate comparison fields:

source_key
email_original
email_key
phone_original
phone_key
full_name
source
received_at

For an email comparison key, an n8n expression can trim whitespace and use lowercase:

{{ ($json.email || '').trim().toLowerCase() }}

Build source_key from the provider and its stable ID, for example facebook:lead_8472 or website:submission_193. Do not concatenate name and timestamp and call that stable. For phone numbers, remove presentation punctuation and normalize with the correct country context. Simply stripping every non-digit character can make two different international numbers look deceptively similar.

Step 2: use Remove Duplicates for the narrow job it does well

For duplicate rows arriving together, configure Remove Duplicates → Remove Items Repeated Within Current Input, choose Selected Fields, and compare source_key. This is ideal for imports and batched records.

For the same source event arriving in later executions, the node also supports Remove Items Processed in Previous Executions. Use Value Is New and deduplicate on source_key. Scope can be the individual node or the workflow. Current n8n documentation says the default history size is 10,000 items.

Step 3: add a persistent lookup

Option A: n8n Data Table for a small lead ledger

A Data Table is a practical first version for a small business that does not yet have strong CRM lookup rules. Create a table such as lead_identity with these columns:

source_key
email_key
phone_key
crm_id
status
first_seen_at
last_seen_at

The current Data Table node can check whether a row exists, get rows, insert, update, and upsert using conditions. Search the strongest available key before the CRM write, then upsert the final crm_id only after the CRM succeeds.

n8n describes Data Tables as light to moderate storage. The default total limit across all tables in an instance is 50 MB, although self-hosted instances can change it. That makes a table useful as a compact identity ledger, not a replacement for a mature CRM or warehouse.

Option B: search the CRM directly

If the CRM already owns the customer record, query it first. The current n8n HubSpot node, for example, supports searching contacts and a Create/Update a contact operation. Other CRM nodes expose different operations, so check the destination's exact contract before copying a pattern.

Prefer a CRM record ID or custom unique source ID once you have one. HubSpot's current Contacts API can retrieve by email and batch upsert by email or a custom unique identifier. It also notes that partial upserts are not supported when email is the identifier, which is one reason not to send an incomplete payload blindly.

Step 4: route new, matched, and conflicting leads

Use a Switch node with three explicit outcomes:

1. No match: create the contact, then save the returned CRM ID with the source key.
2. One safe match: update only allowed fields and attach the new source or submission as an activity.
3. Conflict: do not create or merge automatically. Send the candidate record IDs and incoming values to an owner for review.

A conflict includes more than two search results. It also includes email matching contact A while phone matches contact B, or a source ID pointing to a record whose verified identity now disagrees with the payload. Preserve the lead and notify someone. Silent deletion is not deduplication.

Step 5: make the CRM write duplicate-safe

Use the destination's create/update or upsert operation when it can enforce a unique identifier. This is stronger than a separate search followed by create because two n8n executions can run at nearly the same time, both see no result, and both create a contact before either writes the ledger.

If the CRM cannot upsert atomically, use its unique-field constraint, serialize the critical section, or put the identity key in a database with a unique index. Catch the destination's duplicate response and retrieve the existing record instead of treating every conflict as an unrecoverable failure.

After a successful write, upsert the source key and returned CRM ID into the Data Table. Then send notifications or start follow-up. This ordering prevents the team from receiving a success alert for a contact that never reached the CRM.

Test with a small duplicate matrix

Common mistakes

• Comparing raw email or phone values without normalization.
• Using name as a unique key.
• Assuming a source submission ID identifies the same person across every channel.
• Dropping duplicates without recording the new campaign, form, message, or timestamp.
• Overwriting verified CRM fields with blank or lower-confidence form values.
• Running search and create concurrently without a destination uniqueness control.
• Using a temporary in-workflow list as the only long-term duplicate barrier.

Useful questions before publishing

Can Remove Duplicates replace a CRM lookup?

No. It is useful for repeated items and remembered values, but it does not understand an existing customer record, alternate email, shared phone, merge history, or field ownership.

Should email or phone be the main key?

Use the strongest verified identifier your process reliably collects. Email is often the simplest contact key. Phone can be useful after country-aware normalization. Keep a stable source ID for replay protection even when email is the CRM match key.

Should a duplicate lead be discarded?

Usually not. The person may have submitted a new service request or responded to a different campaign. Reuse the contact record, then preserve the new submission as an activity, note, deal, or source event.

Privacy and operating rules

• Store only the identity fields needed for matching and follow-up.
• Restrict Data Tables, execution data, and CRM credentials to the responsible project members.
• Set retention rules for rejected and ambiguous records.
• Do not place full lead payloads in chat alerts when a secure CRM link is enough.
• Document who resolves conflicts and how quickly the review queue should be checked.

For retries around state-changing API calls, use the separate duplicate-safe n8n API retry pattern. That guide protects one operation from being repeated. This guide protects CRM identity across leads and sources.

Sources checked

• n8n Remove Duplicates operations, scope, history, and default history size
• n8n Data Tables uses, access, and storage limitations
• n8n Data Table row checks and upsert operation
• n8n HubSpot contact search and create/update operations
• HubSpot contact identifiers and batch upsert behavior