Building vault-deduplicator
with Claude Code
From a one-liner question to a fully-automated AI dedup pipeline
user
~/Repos/parsidion
$
claude
Claude Code — parsidion
✓ Vault context loaded · 1,531 notes indexed
What we'll build
- Add --no-index flag to vault-merge for batch workflows
- Run vault-merge --scan to find 50 near-duplicate pairs
- Launch 10 parallel Haiku agents to evaluate & merge 46 pairs
- Create the vault-deduplicator agent codifying the workflow
- Sync all docs and commit
vault-merge
parallel agents
46 merges
haiku model
Does vault-merge re-index?
A simple question reveals a missing feature
user
→
does vault-merge trigger re-index after a merge?
claude
→
grep -r "update_index" skills/parsidion/scripts/vault_merge.py
No matches found
No — vault-merge does not trigger a re-index after merging.
You'd need to run it manually:
$ uv run --no-project ~/.claude/skills/parsidion/scripts/update_index.py
Want me to add an automatic re-index call at the end of a successful merge?
user
→
yes, but add a --no-index option to disable it
Gap identified: batch workflows need --no-index to skip per-merge rebuilds
and do one final rebuild at the end — much faster for 50+ merges.
Adding --no-index to vault-merge
Three targeted edits to vault_merge.py
1 — import subprocess
import argparse, re, shutil, struct, sqlite3
+import subprocess
import sys
from pathlib import Path
2 — add _rebuild_index() helper
def _rebuild_index() -> None:
index_script = Path(__file__).parent / "update_index.py"
if not index_script.exists():
index_script = Path.home() / ".claude/skills/parsidion/scripts/update_index.py"
subprocess.run(["uv", "run", str(index_script)], check=True, ...)
3 — add flag + call after git commit
parser.add_argument("--no-index", action="store_true",
help="Skip rebuilding the vault index after a successful merge.")
# after git_commit_vault():
if not args.no_index:
_rebuild_index()
✓ vault_merge.py updated · synced to ~/.claude/skills/parsidion/scripts/
vault-merge --scan
Embedding similarity reveals 50 near-duplicate pairs
user
$
vault-merge --scan
Found 50 near-duplicate pair(s) (threshold=0.92):
1. [0.9995] burning/ship-fractal-implementation-0929
burning/ship-fractal-implementation-0843
→ vault-merge ship-fractal-implementation-0929 ship-fractal-implementation-0843
2. [0.9980] burning/ship-fractal-implementation-0929
burning/ship-fractal-implementation
3. [0.9977] qdrant/capacity-planning-essentials-0928
qdrant/capacity-planning-essentials
...
50. [0.9833] Patterns/terminal-emulator-core-architecture-patterns
Patterns/terminal-emulator-core-architecture-patterns-0801
All pairs scored ≥ 0.9833 — well above the 0.92 threshold. Most are
timestamped variants (e.g. foo-0929 vs foo) created when the
summarizer wrote the same insight across multiple sessions.
Identifying Chain Dependencies
Some notes appear in multiple pairs — order matters
Before launching agents, I identified 6 chain groups where the same stem
appears as both NOTE_A in one pair and NOTE_B in another. These must be
processed sequentially within a single agent.
Chains (sequential)
ship-fractal: 3 variants
0929 + 0843 → then base + 0929
par-fractal: 3 variants
0804 + 0803 → then base + 0804
pydantic-fix: 3 variants
0758 + 0759 → then base + 0758
ai-service-client: 3 variants
0802 + base → then 0803 + base
tmux-click-jitter: 3 variants
1028 + base → then base + tmux/
parsidion-arch: 3 variants
0806 + base → then Patterns/ + Projects/
Strategy
◆ 10 parallel Haiku agents
◆ Chains stay within 1 agent
◆ Independent pairs batched 5/agent
◆ --no-index on every merge
◆ One index rebuild at the end
◆ Skip redundant pairs after chain
◆ Cross-folder pairs: careful eval
◆ Prefer base name as NOTE_A
6 chains
~36 independent pairs
4 pairs needing careful review
10 Parallel Haiku Agents
Each agent reads both notes, decides validity, executes merge
claude
→
Agent × 10 (model: haiku, parallel)
1
ship-fractal chain
⟳ running
2
qdrant/research batch
⟳ running
3
debugging batch
⟳ running
4
tmux + parsidion chains
⟳ running
5
patterns batch
⟳ running
6
pydantic + ai-service chains
⟳ running
7
par-fractal chain + colbert
⟳ running
8
debugging/patterns batch
⟳ running
9
cross-named + special
⟳ running
10
patterns batch 6
⟳ running
Each agent: (1) Read both notes (2) Evaluate same topic?
(3) If YES → vault-merge A B --no-index --execute
(4) Report decision + reason
Agents 1–4 Complete
Ship-fractal chain, qdrant batch, debugging batch, tmux+parsidion chains
🔗
Agent 1 — ship-fractal chain
2 merged · 0929+0843 then base+0929
✓
Agent 2 — qdrant/research (pairs 3,5,6,7,8)
5 merged · all identical content
✓
Agent 3 — debugging/qdrant (pairs 9,10,12,13,15)
5 merged · same session_id on each pair
🔗
Agent 4 — tmux chain + parsidion arch chain
3 merged · 1 skipped (tmux/ note missing)
Agent 4 detail:
Pair 14: tmux-click-jitter-1028 → tmux-click-jitter ✓ merged
Pair 11: tmux-click-jitter (Debugging/) ↔ click-jitter (tmux/) ⚠ note missing
Pair 16: parsidion-arch-0806 → parsidion-arch ✓ merged
Pair 32: parsidion-arch-gap (Patterns/) → parsidion-arch ✓ merged
15 merged so far
1 skipped
Agents 5–7 Complete
Patterns batch, pydantic+ai-service chains, par-fractal chain
✓
Agent 5 — patterns batch (pairs 17-21)
5 merged · synknot, mandelbrot, parsplat, frontmatter
🔗
Agent 6 — pydantic chain + ai-service chain
5 merged · 6 notes consolidated (chains)
🔗
Agent 7 — par-fractal chain + colbert/brush/yaml
6 merged · 0804+0803 then base+0804
Agent 6 chain detail:
pydantic-fix: 0758+0759 → base+0758 (3 notes → 1) ✓
pydantic-pitfalls: 0759 → base (2 notes → 1) ✓
ai-service: 0802+base → 0803+base (3 notes → 1) ✓
Agent 7 chain detail:
par-fractal: 0804+0803 → base+0804 (3 notes → 1) ✓
31 merged so far
chains preserved ordering
Agents 8–10 Complete
Debugging/patterns, cross-named specials, final patterns batch
✓
Agent 8 — agentchrome, arkit, par-term, codable, ai-writing
5 merged · identical session_id on all pairs
⚠
Agent 9 — cross-named pairs (careful eval)
5 merged · same session_id confirmed same content
✓
Agent 10 — secret-env, mainactor, fastapi, claudevault, terminal
5 merged · classic summarizer duplicates
Agent 9 cross-name decisions:
path-traversal-vault-boundary-fixes
↔ vault-path-traversal-boundary-fixes ✓ MERGED same session_id SEC-001/005
image-tagging-pipeline-architecture
↔ ai-image-tagging-pipeline-architecture ✓ MERGED same par_image_tag project
46 merged total
4 skipped
~170 wikilinks updated
All Agents Done — Rebuilding Index
Final tally and one rebuild to rule them all
claude
→
uv run --no-project ~/.claude/skills/parsidion/scripts/update_index.py
Updated CLAUDE.md: 1531 notes indexed, 490 tags; 8 MANIFEST.md file(s) generated
Embeddings: incremental rebuild launched in background
--no-index saved ~46 sequential index rebuilds. Instead: one parallel
batch of merges, one final rebuild. Significant time savings on large vaults.
1,531 notes
490 tags
embeddings rebuilding in background
Creating vault-deduplicator Agent
Codifying the workflow so any future session can run it
user
→
create a new agent for the dedupe process we just ran, following existing naming conventions
agents/vault-deduplicator.md
---
name: vault-deduplicator
description: >
Use when you need to find and merge
near-duplicate notes in ~/ClaudeVault/.
Scans, evaluates, merges, rebuilds.
Trigger on: "deduplicate the vault",
"find duplicate notes", "merge duplicate
vault notes", "vault has duplicates" ...
model: haiku
color: cyan
---
4-step procedure
- Step 1 — vault-merge --scan
- Step 2 — Identify chain dependencies
- Step 3 — Batch parallel Agent launches (Haiku)
- Step 4 — One final update_index.py
✓ Follows vault-* naming pattern
✓ Haiku model (utility task)
✓ Cyan color (distinct from others)
✓ Synced to ~/.claude/agents/
Syncing All Docs
7 parallel agents check each doc for implementation drift
user
→
update docs · also update skill to be aware of it
claude
→
Agent × 8 (7 docs + SKILL.md update)
ARCHITECTURE.md ✓ Added vault-deduplicator to ToC, Mermaid diagram, component
details, and both file layout trees
README.md ✓ No drift — already accurate
MCP.md ✓ Added --migrate-subfolders/--execute flags; removed brittle model size
MCPL.md ✓ Fixed --activate/--deactivate to accept index or path
EMBEDDINGS.md ✓ Documented auto-detection of dimension mismatch on incr. rebuild
EMBEDDINGS_EVAL.md ✓ Fixed "sortable by column" → "pre-sorted by MRR descending"
AGENTCHROME.md ✓ Updated fallback: curl OR Claude Code Web Fetch tool
SKILL.md: Added vault-deduplicator agent table + ## Vault Merge section
with --no-index batch-merge pattern · synced to ~/.claude/skills/
Commit & Ship
9 files, one atomic commit
user
→
commit all
On branch main · ahead of origin/main by 19 commits
Changes to commit:
M docs/AGENTCHROME.md
M docs/ARCHITECTURE.md
M docs/EMBEDDINGS.md
M docs/EMBEDDINGS_EVAL.md
M docs/MCP.md
M docs/MCPL.md
M skills/parsidion/SKILL.md
M skills/parsidion/scripts/vault_merge.py
A agents/vault-deduplicator.md
9 files changed, 232 insertions(+), 13 deletions(-)
[main a8a3c5d] feat(vault): add vault-deduplicator agent and vault-merge --no-index flag
Merged: vault-deduplicator agent · --no-index flag · 7 doc syncs · SKILL.md update
What We Built
A fully-automated vault deduplication pipeline
Features shipped
- vault-merge --no-index flag for batch workflows
- _rebuild_index() auto-called after merge
- vault-deduplicator agent (haiku, cyan)
- 10 parallel Haiku agents merged 46/50 pairs
- ~170 wikilinks updated across vault
- 1,531 notes re-indexed, 490 tags
- 7 docs synced, SKILL.md updated
Commands / workflow
vault-merge --scan
vault-merge A B --no-index --execute
vault-merge A B --execute (auto re-index)
vault-deduplicator agent trigger
update_index.py final rebuild
46 duplicate pairs merged
vault-deduplicator agent
--no-index batch pattern
commit a8a3c5d