TalviDocs

Clean duplicate and similar files

Compare partial matches, inspect changed pages, and review a deletion candidate.

What you will achieve

Periodically review your datasets for duplicate or similar files, compare the differences, and choose which version to keep. Run this review after substantial imports or as part of your regular content maintenance.

1. Open the suggested matches

Open Talvi Data Cleaner. The opening screen shows the available datasets, the last completed run, and the three stages of a cleaning session. Select Begin Talvi Data Cleaner if no session is active, then choose Duplicate Files.

Talvi Data Cleaner opening screen and Begin Talvi Data Cleaner button

The percentage helps you prioritise comparisons. It is a similarity signal, not a recommendation to delete a file. A high match may still contain an important update.

2. Select a partial match

Select a group showing a partial match, such as 87% in this example, then select a file under Candidates. The comparison places the Master alongside the Candidate. Use the match that you want to investigate in your own dataset; the workflow is the same for other scores.

Select a partial-match group in Duplicate Files; file and dataset names hidden

Select Full Screen to give both documents and their file seekers more space.

Full-screen comparison of a partial match, with Master and Candidate file seekers

3. Inspect the first red marker

Click the first red dot, at the left of the Master’s File seeker, to jump to the first indicated difference. Scroll within the page to read the marked edit, then compare the corresponding page in the Candidate. Use its page control or file seeker if you need to align the views.

Red markers point to differences to inspect; the page preview lets you read the actual change. Continue through the remaining markers before deciding whether the changes should be retained. In this example, the edited document and original share much of their content, but several pages differ.

First red marker on page 1, showing the edited section beside the original

4. Select the file to remove

Decide which version your agents should retain. If the edited file is the version to keep, select Mark for deletion for the version you no longer need in Candidates. The checked row shows it has been selected for the queue. The candidate’s Delete control also marks it; when marked, that control becomes Cancel, which undoes the selection.

Candidate selected for the deletion queue, with its similarity score displayed and filename hidden

Master and Candidate are comparison roles, not guarantees of which file is newest or correct. Use the swap control if you need to compare in the other direction. Keep both when their differences serve a useful purpose.

5. Review the deletion queue

Close the comparison and open Deletion Queue. Review all selections, including decisions already present in the session. Select Commit deletions to inspect the final list; the final Delete action is permanent. See the removal recipe for the final review and test option.

Result: redundant files are identified through a content comparison, with selected removals reviewed before they are applied.

See also: Talvi Data Cleaner · Update your AI agent’s data.

Talvi documentation · v1.0