Velero backups failing due to blob missing from the repository

[dharanui]

error message: data path backup failed: Failed to run kopia backup: Failed to find previous kopia snapshot manifests for si default@default:snapshot-data-upload-download/kopia/glowroot/data-glowroot-cassandra-0: unable to find manifest entries: failed to get manifests with labels map[hostname:default path:snapshot-data-upload-download/kopia/glowroot/data-glowroot-cassandra-0 type:snapshot username:default]: error to find manifests: unable to load manifest contents: error loading manifest content: error getting cached content: failed to get blob with ID qf639e8abd6afa38f0b504f5e4643753e-sf3d16207b6fc75a5130: BLOB not found

Also tried to connect to kopia s3 repository and genuinely the blob is not available. How to recover the missing blob if the source volume is still available?

velero version: 1.12
velero plugin for aws: v1.7.0

[Lyndon-Li]

There is no way to recover a missing repo blob from the source volume, you can only run a new backup and create blobs for the new backup.

[dharanui]

@Lyndon-Li thanks for the confirmation!
Running new backup will search for existing blobs and fails if it doesnt find. Is there a way to instruct kopia through velero to run a fresh backup like kopia offers kopia snapshot create --force-hash=100 ?

And what might be possible reason for missing blob?

[kaovilai]

Running new backup will search for existing blobs and fails if it doesnt find.

What is the error when creating a new backup and failing?

[dharanui]

It gives the same error: error to find manifests: unable to load manifest contents: error loading manifest content: error getting cached content: failed to get blob with ID qf639e8abd6afa38f0b504f5e4643753e-sf3d16207b6fc75a5130: BLOB not found

[kaovilai]

If a new clean backup cannot be created, then I agree we need a mitigation to force kopia to create fresh backup/repo assuming there is nothing useful still usable being overwritten.

[dharanui]

Yes, kopia offers --force-hash=100 flag to create everything fresh, but i dont see a way to do that through the velero aws plugin.