[FEATURE REQUEST] remove duplicates while keeping external library image

[traktuner]

Hello!
I currently have the following issue:
I imported many pictures into immich via cli some time ago which are present in photo album folders on my NAS drive.
Not long ago, the external library feature was implemented which I am using a lot.
So I have many duplicates now, because lots of albums I manually imported are now part of some external libraries.
It would be really cool if you could add a routine (or separate deduplicate function) to check if assets in the user library match a file exactly which is present in an external library, and if yes - trash the uploaded picture (and update the albums with the "new" file instead)
When I do that now, it would like to remove the one from the external library but of cours that is a read only volume.
Hope that makes sense!

Thank you very much for that useful project!

[simulot]

Yes it makes sense. Your external library is not writable through the immich server.

A possible way would be for immich-go to have direct access to your external library and let him remove the duplicate file, and not through the immich API.

Albums would be updated to points to the version of the image that would remain.

Once done, immich-go could launch the immich job to refresh the library and let it remove its references to that file.

Is it something that could fit?

[traktuner]

Hello!
For me, it should be the other way around. The external library ist the immutable library which should not be touched. The imported pictures in my own default library should be treated as duplicates (because I imported many albums while the external library feature was not implemented yet).
So there would be a scenario where immich-go detects a duplicate image, one in an external library, one in the default library, the one in the default library should get trashed and albums with that image should be updated with the one from the external library.
Also, a possible scenario could be that immich-go detects 3 duplicates for the same image, 2 in an external library, one in the default library - same as above - remove the pic in the default library and update albums with one from an external library.

I think this can be done through the immich api...

[simulot]

Thanks for the precision.
Yes this makes a lot more sense this way.

The deduplication should keep the external library version, and rebuild albums on this basis.

The initial functionality of deduplication was to remove lower quality images and keep to best one. What if the best photo resides in the immich internal ?

[johny-mnemonic]

Do you have any plans to add this functionality by any chance? 🙏

I just started with Immich and I am facing the same issue.
I first used your great tool to import my Google takeout to Immich and then connected my NAS as external library. Inevitably that ended with thousands of duplicates and I would like to keep in Immich only the images that are not already on my NAS.

Especially when the external library is in read-only mode, there is no point in dropping the duplicate from it. It will (hopefully) re-appear after next re-scan of the external library.

[simulot]

I'm busy refactoring the existing code.
I can't promise a quick implementation.

Stay tuned.

[johny-mnemonic]

OK, thanks for quick reply. Keep your pace.

Would you remember what is the decision process when it finds two identical photos? How it selects the one to drop? Maybe we (users) can come up with some temporary workaround how to skew it the way we want 😉

[MikeR984]

I'm thinking of doing a Photoshop batch action on all of the images in my external with the aim of increasing the file size by 0.1mb or so, meaning when I run the duplicate removal it should always pick the external file. Pretty dirty way of doing it but it might work?

[simulot]

totally dirty!
I'm still working (slowly) to the new version....

[traktuner]

Good point! I assume that won't happen very often, but if it does, maybe use the default immich-go behaviour and keep the one with the better quality, so in the end, nothing happens because the external libraries are read-only.
I guess it would be enough to do very good logging that there still are duplicates even with the "-yes" flag because nothing was done?

[MRobi1]

I would assume they would likely both be the same quality.