cache gets verifies crc checksum on stored cache items

[assafvayner]

FIX XET-142.

Previously we removed the chunk cache validation on the cache items on the get path since it turned out to be where all the time was spent on cache operations. This PR reintroduces checksum validation, now that we are using checksums rather than more expensive hashes.

Notably the verification is still rather expensive since it requires reading the entire file, whereas a simple cache get may only need to seek to 1 section of the file.

Verification is done only once on a best effort basis, i.e. it is not strictly enforced that the verification work is done only once, if 2 threads attempt to verify concurrently, double work is done. See VerificationCell.

[seanses]

Why do we need to verify the cache content? Is there anything that may corrupt (or maliciously manipulate) the cache data on disk?

[assafvayner]

Why do we need to verify the cache content? Is there anything that may corrupt (or maliciously manipulate) the cache data on disk?

This is to mitigate the risk of user error more than anything, i.e. the user or some program modified the cache files. I am personally of the opinion that we could rip out this validation, and instead drop it or replace with just a magic word check.