Does this skill delete images permanently?

The skill provides an option to deduplicate, which removes samples from the FiftyOne dataset. It is highly recommended to review the flagged duplicates in the FiftyOne App UI before performing the final deletion.

What is the difference between an exact and a near-duplicate?

Exact duplicates are byte-for-byte identical files. Near-duplicates are visually similar images (e.g., different compression, slight crops, or lighting changes) identified through deep learning embeddings.

How do I adjust the sensitivity of the search?

You can adjust the 'threshold' parameter in the similarity operators; lower values (e.g., 0.1) find only near-identical images, while higher values (e.g., 0.5) will capture more loosely similar content.

Do I need a GPU to use this skill?

A GPU is not strictly required, as the skill can use CPU-optimized models like MobileNetV2, but a GPU will significantly speed up embedding computation for large datasets.

FiftyOne Image Deduplication

Name: FiftyOne Image Deduplication
Author: AdonaiVera

byAdonaiVera

0•

데이터 과학 및 ML

Identifies and removes duplicate or visually similar images in FiftyOne datasets using deep learning embeddings.

The FiftyOne Find Duplicates skill empowers computer vision engineers to clean and optimize their datasets by identifying redundant visual data. By leveraging FiftyOne's brain similarity operators and deep learning embeddings, this skill can detect both byte-for-byte exact duplicates and near-duplicate images that share visual characteristics but differ slightly in composition or quality. It streamlines the dataset curation process, helping users improve model performance by removing noise, reducing training time, and ensuring a diverse data distribution through automated deduplication and clustering workflows.

주요 기능

01Automated detection of exact and near-duplicate images

02Configurable similarity thresholds for granular deduplication control

030 GitHub stars

04Direct integration with FiftyOne brain operators for efficient dataset cleaning

05Deep learning-based similarity computation using MobileNetV2 and other models

06Interactive duplicate review within the FiftyOne App

사용 사례

01Identifying and removing redundant samples to reduce storage and labeling costs

02Clustering visually similar content to analyze dataset diversity

03Cleaning large computer vision datasets before model training

What are Skills?·How to Install

Install with 🐟 Skill.Fish

npx skillfish add adonaivera/fiftyone-skills fiftyone-find-duplicates

For use in Claude.ai and ChatGPT

Download Skill