Duplicate Image Detection

## Why it matters

A gallery might list the same photo under a CDN URL, a cache-busted URL with query strings, and a srcset variant. A logo appears in the header, the footer and the about page. When you bulk download, you end up with the same image three, five, or twenty times over.

Matching on weight plus dimensions catches these copies without opening every image file. If two images are exactly the same byte size and pixel dimensions, they're almost certainly identical. Downloading once saves bandwidth, drive space and time.

## How it works

Scalpel Images reads two signals for every image: its file size (from the Resource Timing API's encodedBodySize for cached images, or computed from the data URI payload for inlined images) and its pixel dimensions (naturalWidth and naturalHeight). Then it groups images that share both values.

This approach is cheap. You already read dimensions and size to power the filters, so duplicate detection costs almost nothing extra. It needs no pixel decoding and no perceptual hashing. The memory overhead is tiny: a map keyed by width-x-height-x-bytes.

The method catches most real duplicates. Two genuinely different images are unlikely to have the exact same file size and dimensions. The false negative rate is very low (you might miss an image that's been re-encoded at the same quality, but that's rare). The false positive rate is also low, with one exception.

## What does not matter

Exact URL matching is not needed. A duplicate might be served from a different domain (cdn1.example.com/img.jpg vs cdn2.example.com/img.jpg), or with cache-busting query strings (img.jpg?v=1 vs img.jpg?v=2). Size plus dimensions bypass all that noise.

Also, a tiny tracking pixel (1×1 PNG, a few hundred bytes) might happen to be the same size as another 1×1 image. This is the false positive problem: you could accidentally group unrelated tiny images. Scalpel Images filters this out by only grouping images above a minimum area (around 100 pixels). Below that threshold, the probability of accidental collision is high enough that the filtering gain outweighs the duplicates you'd catch.

Perceptual hashing is a different method altogether. It decodes every image, computes a fingerprint of its visual content, and matches hashes that are similar. This catches images that are rescaled, recompressed, or slightly modified. But it's expensive (you have to decode every image) and it produces false positives (similar-looking but different images). For deduplication at scale, byte size plus dimensions is the right trade-off.

## Code example

Here's how you'd detect duplicates in JavaScript:

const images = [
  { url: 'https://cdn.example.com/photo.jpg', width: 1200, height: 800, size: 85000 },
  { url: 'https://cdn.example.com/photo.jpg?v=1', width: 1200, height: 800, size: 85000 },
  { url: 'https://other-cdn.example.com/photo.jpg', width: 1200, height: 800, size: 85000 },
  { url: 'https://cdn.example.com/logo.png', width: 200, height: 200, size: 15000 },
];

const duplicateGroups = new Map();

images.forEach((img) => {
  // Group by width-height-size
  const key = `${img.width}x${img.height}x${img.size}`;
  if (!duplicateGroups.has(key)) {
    duplicateGroups.set(key, []);
  }
  duplicateGroups.get(key).push(img);
});

// Print groups with more than one image
duplicateGroups.forEach((group, key) => {
  if (group.length > 1) {
    console.log(`Duplicate group (${key}):`, group.map(img => img.url));
  }
});
Copy

## How Scalpel Images shows it

Every non-primary image in a duplicate group carries a DUP badge on its tile. You can see which images are duplicates at a glance. If you check the "Hide duplicates" option, the popup collapses every duplicate group to show you just one representative image from each set. This makes it trivial to spot exactly which files you actually need.