Files restored using restore_keys dont trigger upload at the end

[ryangraham-ge]

We want to manage caching of the .eslintcache file in nodejs projects. Its different because its a file that isnt committed because it causes merge conflicts.

In the pre-hook, we'd like buildkite to load the most recent version. And on post-hook, we want to upload the new version (ideally only pushing if the file changed).

We've tried a few different setups, and this is the closest to what we want:

eslint-cache: &eslint-cache
  id: eslint
  key: "v1-cache-{{ id }}-$BUILDKITE_COMMIT"
  backend: s3
  restore-keys:
    - 'v1-cache-{{ id }}-'
  paths:
    - .eslintcache

Expected processing

1. Assumption is the current commit id is 222. A cache currently exists for commit id 111. The app workdir is /app.
2. Buildkite-cache plugin looks for a cache for the current commit id (222). There should be no match, so we use restore-keys to load a previous cache.
3. The code sorts the existing cache items by last modified and sends back the most recent that matches our restore-keys. In this case, we find the cache for commit id 111, v1-cache-eslint-111.tar available.
4. v1-cache-eslint-111.tar is loaded into the workdir to /app/v1-cache-eslint-111.tar and extracted.
5. Pre-hook complete. Now we can run yarn lint. This looks at the extracted .eslintcache file, uses it to speed up the lint command, and slightly updates the .eslintcache to match the current files in the app.
6. Post-hook begins. We look for an existing tar cache for the commit id 222, /app/v1-cache-eslint-222.tar. The file doesnt exist. So we compress the local .eslintcache into a tar with that name, and upload it to s3.

Actual processing

• On step 4, the v1-cache-eslint-111.tar file is pulled, but saved locally as /app/v1-cache-eslint-222.tar. This makes step 6 believe we dont have any updates, and therefore ignores the upload.

Workaround

We can force this to work as we'd like by changing the linting step from yarn lint to yarn lint && rm v1-cache-eslint-*. This will force step6 to not see any existing cache, and therefore upload the new version. But it feels a little hacky to us.

The situation seems to be caused by the TAR_FILE var being overwritten in https://github.com/gencer/cache-buildkite-plugin/blob/master/lib/backends/s3.bash#L89, and I dont think its intentional behavior. What are your thoughts?

[gencer]

Hey @ryangraham-ge!

When you say:

On step 4, the v1-cache-eslint-111.tar file is pulled, but saved locally as /app/v1-cache-eslint-222.tar. This makes step 6 believe we dont have any updates, and therefore ignores the upload.

I think we've a bug there. Will look ASAP and get back to you for this case.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.