Does this skill delete images permanently?

The skill provides an option to deduplicate, which removes samples from the FiftyOne dataset. It is highly recommended to review the flagged duplicates in the FiftyOne App UI before performing the final deletion.

What is the difference between an exact and a near-duplicate?

Exact duplicates are byte-for-byte identical files. Near-duplicates are visually similar images (e.g., different compression, slight crops, or lighting changes) identified through deep learning embeddings.

How do I adjust the sensitivity of the search?

You can adjust the 'threshold' parameter in the similarity operators; lower values (e.g., 0.1) find only near-identical images, while higher values (e.g., 0.5) will capture more loosely similar content.

Do I need a GPU to use this skill?

A GPU is not strictly required, as the skill can use CPU-optimized models like MobileNetV2, but a GPU will significantly speed up embedding computation for large datasets.

FiftyOne Image Deduplication

Name: FiftyOne Image Deduplication
Author: AdonaiVera

byAdonaiVera

Data Science & ML

Identifies and removes duplicate or visually similar images in FiftyOne datasets using deep learning embeddings.

About

The FiftyOne Find Duplicates skill empowers computer vision engineers to clean and optimize their datasets by identifying redundant visual data. By leveraging FiftyOne's brain similarity operators and deep learning embeddings, this skill can detect both byte-for-byte exact duplicates and near-duplicate images that share visual characteristics but differ slightly in composition or quality. It streamlines the dataset curation process, helping users improve model performance by removing noise, reducing training time, and ensuring a diverse data distribution through automated deduplication and clustering workflows.

Key Features

Automated detection of exact and near-duplicate images
Configurable similarity thresholds for granular deduplication control
0 GitHub stars
Direct integration with FiftyOne brain operators for efficient dataset cleaning
Deep learning-based similarity computation using MobileNetV2 and other models
Interactive duplicate review within the FiftyOne App

Use Cases

Identifying and removing redundant samples to reduce storage and labeling costs
Clustering visually similar content to analyze dataset diversity
Cleaning large computer vision datasets before model training

About

Key Features

Automated detection of exact and near-duplicate images
Configurable similarity thresholds for granular deduplication control
0 GitHub stars
Direct integration with FiftyOne brain operators for efficient dataset cleaning
Deep learning-based similarity computation using MobileNetV2 and other models
Interactive duplicate review within the FiftyOne App

Use Cases

Identifying and removing redundant samples to reduce storage and labeling costs
Clustering visually similar content to analyze dataset diversity
Cleaning large computer vision datasets before model training