You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This was initially reported by @pipex on the delta server repo but it should really be reported here. Felipe's complete description:
When downloading images with deltas, the target state endpoint provides a target image name as registry2.balena-cloud.com/v2/<imageName>@<imageDigest>.
The supervisor then requests a delta from the source image (on device) to the target image, and gets back a delta address as registry2.balena-cloud.com/v2/<imageName>:delta-<deltaId>.
That image gets pulled by the engine, and gets assigned another digest, which means that the image is no longer queryable by the original digest, making the supervisor image database the only way to maintain a relation between the original image and the image on disk.
See below for an example of feedback from the engine when pulling target images with and without deltas.
Downloading with deltas
Before state
root@cceab6e:~# balena images
REPOSITORY TAG IMAGE ID CREATED SIZE
livepush-supervisor-12.8.7 latest 582baa1eef67 10 minutes ago 527MB
registry2.balena-cloud.com/v2/2f787b4a0f149d9f05ebb9f202f4c7ff latest 29afe3a0cd02 2 weeks ago 62.9MB
balenalib/armv7hf-alpine-supervisor-base 3.11 a6b88078ad7a 14 months ago 12.1MB
balena-healthcheck-image latest 851163c78e4a 17 months ago 4.85kB
First release
Target state image: registry2.balena-cloud.com/v2/872a7de303e661af19fcf1ea09d12fc8@sha256:9ec0e488aaa12a82f10d8ac2e88a342d0f7527e8a32f76789f82c8820e985176
Supervisor logs
[info] Applying target state
[event] Event: Apply config change in progress {}
[event] Event: Apply config change success {}
[event] Event: Network creation {}
[event] Event: Docker image download {"image":{"name":"registry2.balena-cloud.com/v2/872a7de303e661af19fcf1ea09d12fc8@sha256:9ec0e488aaa12a82f10d8ac2e88a342d0f7527e8a32f76789f82c8820e985176","appId":1839100,"serviceId":1102814,"serviceName":"hello","imageId":3729845,"releaseId":1834408,"dependent":0,"dockerImageId":null}}
[info] Internet Connectivity: OK
[event] Event: Image downloaded {"image":{"name":"registry2.balena-cloud.com/v2/872a7de303e661af19fcf1ea09d12fc8@sha256:9ec0e488aaa12a82f10d8ac2e88a342d0f7527e8a32f76789f82c8820e985176","appId":1839100,"serviceId":1102814,"serviceName":"hello","imageId":3729845,"releaseId":1834408,"dependent":0,"dockerImageId":null}}
[event] Event: Service install {"service":{"appId":1839100,"serviceId":1102814,"serviceName":"hello","releaseId":1834408}}
[event] Event: Service installed {"service":{"appId":1839100,"serviceId":1102814,"serviceName":"hello","releaseId":1834408}}
[event] Event: Service start {"service":{"appId":1839100,"serviceId":1102814,"serviceName":"hello","releaseId":1834408}}
[event] Event: Service started {"service":{"appId":1839100,"serviceId":1102814,"serviceName":"hello","releaseId":1834408}}
[debug] Spawning journald with: chroot /mnt/root journalctl -a -S 2021-06-17 20:13:56 -o json CONTAINER_ID_FULL=b8034ae13de507703e66099fca735fc8d6060ce7d964d305cd5ff31cbd95c2cb
[debug] Finished applying target state
[success] Device state apply success
root@cceab6e:~# balena images --digests
REPOSITORY TAG DIGEST IMAGE ID CREATED SIZE
livepush-supervisor-12.8.7 latest <none> 582baa1eef67 16 minutes ago 527MB
registry2.balena-cloud.com/v2/872a7de303e661af19fcf1ea09d12fc8 <none> sha256:9ec0e488aaa12a82f10d8ac2e88a342d0f7527e8a32f76789f82c8820e985176 86fdf87bb519 2 hours ago 214MB
registry2.balena-cloud.com/v2/2f787b4a0f149d9f05ebb9f202f4c7ff latest sha256:ba9079a6d38c425d3b5ffad2dd7b5b0d39c25c978ded7727e3cc4e0af6e54b2b 29afe3a0cd02 2 weeks ago 62.9MB
balenalib/armv7hf-alpine-supervisor-base 3.11 sha256:80a6cf83f367aab2f0fe1f3c0dd5bc7da5e321d738d07c845a38dfff05a92c3b a6b88078ad7a 14 months ago 12.1MB
balena-healthcheck-image latest <none> 851163c78e4a 17 months ago 4.85kB
Second release (deltas)
Target state image: registry2.balena-cloud.com/v2/2236dbe3386015de90cef16f9d55c201@sha256:135044860e52ad8b6070b858f680faf16fc7ca7e6dd4a85a8cd64ee879a141ce
Supervisor logs:
[info] Applying target state
[debug] Replacing container for service hello because of config changes:
[debug] Non-array fields: {"added":{},"deleted":{"command":{},"entrypoint":{},"environment":{},"labels":{}},"updated":{"image":"registry2.balena-cloud.com/v2/2236dbe3386015de90cef16f9d55c201@sha256:135044860e52ad8b6070b858f680faf16fc7ca7e6dd4a85a8cd64ee879a141ce","workingDir":""}}
[debug] Replacing container for service hello because of config changes:
[debug] Non-array fields: {"added":{},"deleted":{"command":{},"entrypoint":{},"environment":{},"labels":{}},"updated":{"image":"registry2.balena-cloud.com/v2/2236dbe3386015de90cef16f9d55c201@sha256:135044860e52ad8b6070b858f680faf16fc7ca7e6dd4a85a8cd64ee879a141ce","workingDir":""}}
[event] Event: Service kill {"service":{"appId":1839100,"serviceId":1102814,"serviceName":"hello","releaseId":1834408}}
[debug] Replacing container for service hello because of config changes:
[debug] Non-array fields: {"added":{},"deleted":{"command":{},"entrypoint":{},"environment":{},"labels":{}},"updated":{"image":"registry2.balena-cloud.com/v2/2236dbe3386015de90cef16f9d55c201@sha256:135044860e52ad8b6070b858f680faf16fc7ca7e6dd4a85a8cd64ee879a141ce","workingDir":""}}
[event] Event: Service exit {"service":{"appId":1839100,"serviceId":1102814,"serviceName":"hello","releaseId":1834408}}
[event] Event: Service stop {"service":{"appId":1839100,"serviceId":1102814,"serviceName":"hello","releaseId":1834408}}
[event] Event: Delta image download {"image":{"name":"registry2.balena-cloud.com/v2/2236dbe3386015de90cef16f9d55c201@sha256:135044860e52ad8b6070b858f680faf16fc7ca7e6dd4a85a8cd64ee879a141ce","appId":1839100,"serviceId":1102814,"serviceName":"hello","imageId":3729854,"releaseId":1834414,"dependent":0,"dockerImageId":null}}
DELTA SOURCE registry2.balena-cloud.com/v2/872a7de303e661af19fcf1ea09d12fc8@sha256:9ec0e488aaa12a82f10d8ac2e88a342d0f7527e8a32f76789f82c8820e985176 ID sha256:86fdf87bb5193bb64535251c1f9e80e08dc7ca30d6f1c53d9b49d67eaf936e1f
DELTA TARGET registry2.balena-cloud.com/v2/2236dbe3386015de90cef16f9d55c201@sha256:135044860e52ad8b6070b858f680faf16fc7ca7e6dd4a85a8cd64ee879a141ce ID null
[debug] delta([hello] registry2.balena-cloud.com/v2/872a7de303e661af19fcf1ea09d12fc8@sha256:9ec0e488aaa12a82f10d8ac2e88a342d0f7527e8a32f76789f82c8820e985176): Starting delta to registry2.balena-cloud.com/v2/2236dbe3386015de90cef16f9d55c201@sha256:135044860e52ad8b6070b858f680faf16fc7ca7e6dd4a85a8cd64ee879a141ce
[debug] delta([hello] registry2.balena-cloud.com/v2/872a7de303e661af19fcf1ea09d12fc8@sha256:9ec0e488aaa12a82f10d8ac2e88a342d0f7527e8a32f76789f82c8820e985176): Applying balena delta...
[debug] delta([hello] registry2.balena-cloud.com/v2/872a7de303e661af19fcf1ea09d12fc8@sha256:9ec0e488aaa12a82f10d8ac2e88a342d0f7527e8a32f76789f82c8820e985176): Using registry auth token
DELTA IMAGE registry2.balena-cloud.com/v2/2236dbe3386015de90cef16f9d55c201:delta-b124973c5cc57951
[debug] delta([hello] registry2.balena-cloud.com/v2/872a7de303e661af19fcf1ea09d12fc8@sha256:9ec0e488aaa12a82f10d8ac2e88a342d0f7527e8a32f76789f82c8820e985176): Delta applied successfully
[event] Event: Image downloaded {"image":{"name":"registry2.balena-cloud.com/v2/2236dbe3386015de90cef16f9d55c201@sha256:135044860e52ad8b6070b858f680faf16fc7ca7e6dd4a85a8cd64ee879a141ce","appId":1839100,"serviceId":1102814,"serviceName":"hello","imageId":3729854,"releaseId":1834414,"dependent":0,"dockerImageId":null}}
[error] Error from the API: 503
[error] Non-200 response from the API! Status code: 503 - message: Error: "The API cannot handle your request right now"
[error] at sendReportPatch (/usr/src/app/src/device-state/current-state.ts:95:9)
[error] at runMicrotasks (<anonymous>)
[error] at processTicksAndRejections (internal/process/task_queues.js:97:5)
[error] at /usr/src/app/src/device-state/current-state.ts:154:3
[error] at /usr/src/app/src/device-state/current-state.ts:207:4
Removing sha256:86fdf87bb5193bb64535251c1f9e80e08dc7ca30d6f1c53d9b49d67eaf936e1f
[event] Event: Image removal {"image":{"id":1,"name":"registry2.balena-cloud.com/v2/872a7de303e661af19fcf1ea09d12fc8@sha256:9ec0e488aaa12a82f10d8ac2e88a342d0f7527e8a32f76789f82c8820e985176","appId":1839100,"serviceId":1102814,"serviceName":"hello","imageId":3729845,"releaseId":1834408,"dependent":0,"dockerImageId":"sha256:86fdf87bb5193bb64535251c1f9e80e08dc7ca30d6f1c53d9b49d67eaf936e1f"}}
[event] Event: Image removed {"image":{"id":1,"name":"registry2.balena-cloud.com/v2/872a7de303e661af19fcf1ea09d12fc8@sha256:9ec0e488aaa12a82f10d8ac2e88a342d0f7527e8a32f76789f82c8820e985176","appId":1839100,"serviceId":1102814,"serviceName":"hello","imageId":3729845,"releaseId":1834408,"dependent":0,"dockerImageId":"sha256:86fdf87bb5193bb64535251c1f9e80e08dc7ca30d6f1c53d9b49d67eaf936e1f"}}
[event] Event: Service install {"service":{"appId":1839100,"serviceId":1102814,"serviceName":"hello","releaseId":1834414}}
[event] Event: Service installed {"service":{"appId":1839100,"serviceId":1102814,"serviceName":"hello","releaseId":1834414}}
[event] Event: Service start {"service":{"appId":1839100,"serviceId":1102814,"serviceName":"hello","releaseId":1834414}}
[event] Event: Service started {"service":{"appId":1839100,"serviceId":1102814,"serviceName":"hello","releaseId":1834414}}
[debug] Spawning journald with: chroot /mnt/root journalctl -a -S 2021-06-17 20:17:06 -o json CONTAINER_ID_FULL=4172e01e658de47918321834ffef702bebae2cee860abefa7c77d096a015521f
[debug] Finished applying target state
[success] Device state apply success
Querying image:
root@cceab6e:~# balena inspect registry2.balena-cloud.com/v2/2236dbe3386015de90cef16f9d55c201@sha256:135044860e52ad8b6070b858f680faf16fc7ca7e6dd4a85a8cd64ee879a141ce | jq .[]
Error: No such object: registry2.balena-cloud.com/v2/2236dbe3386015de90cef16f9d55c201@sha256:135044860e52ad8b6070b858f680faf16fc7ca7e6dd4a85a8cd64ee879a141ce
Image state after
root@cceab6e:~# balena images --digests
REPOSITORY TAG DIGEST IMAGE ID CREATED SIZE
livepush-supervisor-12.8.7 latest <none> 582baa1eef67 20 minutes ago 527MB
registry2.balena-cloud.com/v2/2236dbe3386015de90cef16f9d55c201 delta-b124973c5cc57951 sha256:cd4f2a794ddc4df61b7325c79a3c400f3e57c4a0925de0ec3c550c99ef8cbd87 587671fc12ab 2 hours ago 214MB
registry2.balena-cloud.com/v2/2f787b4a0f149d9f05ebb9f202f4c7ff latest sha256:ba9079a6d38c425d3b5ffad2dd7b5b0d39c25c978ded7727e3cc4e0af6e54b2b 29afe3a0cd02 2 weeks ago 62.9MB
balenalib/armv7hf-alpine-supervisor-base 3.11 sha256:80a6cf83f367aab2f0fe1f3c0dd5bc7da5e321d738d07c845a38dfff05a92c3b a6b88078ad7a 14 months ago 12.1MB
balena-healthcheck-image latest <none> 851163c78e4a 17 months ago 4.85kB```
Removing the database (same target as previous case)
Target state image: registry2.balena-cloud.com/v2/2236dbe3386015de90cef16f9d55c201@sha256:135044860e52ad8b6070b858f680faf16fc7ca7e6dd4a85a8cd64ee879a141ce
Supervisor logs
[info] Applying target state
[debug] Ensuring device is provisioned
[info] Reporting initial configuration
[event] Event: Service kill {"service":{"appId":1839100,"serviceId":1102814,"serviceName":"hello","releaseId":1834414}}
[event] Event: Service kill {"service":{"appId":1839100,"serviceId":1102814,"serviceName":"hello","releaseId":1834414}}
[debug] Starting current state report
[debug] Starting target state poll
[debug] Spawning journald with: chroot /mnt/root journalctl -a --follow -o json _SYSTEMD_UNIT=balena.service
[event] Event: Service exit {"service":{"appId":1839100,"serviceId":1102814,"serviceName":"hello","releaseId":1834414}}
[event] Event: Service already stopped {"service":{"appId":1839100,"serviceId":1102814,"serviceName":"hello","releaseId":1834414}}
[event] Event: Service stop error {"error":{"message":"(HTTP code 409) unexpected - removal of container 4172e01e658de47918321834ffef702bebae2cee860abefa7c77d096a015521f is already in progress ","stack":"Error: (HTTP code 409) unexpected - removal of container 4172e01e658de47918321834ffef702bebae2cee860abefa7c77d096a015521f is already in progress \n at /usr/src/app/node_modules/docker-modem/lib/modem.js:257:17\n at getCause (/usr/src/app/node_modules/docker-modem/lib/modem.js:287:7)\n at Modem.buildPayload (/usr/src/app/node_modules/docker-modem/lib/modem.js:256:5)\n at IncomingMessage.<anonymous> (/usr/src/app/node_modules/docker-modem/lib/modem.js:232:14)\n at IncomingMessage.emit (events.js:322:22)\n at endReadableNT (_stream_readable.js:1187:12)\n at processTicksAndRejections (internal/process/task_queues.js:84:21)"},"service":{"appId":1839100,"serviceId":1102814,"serviceName":"hello","releaseId":1834414}}
[event] Event: Service stop {"service":{"appId":1839100,"serviceId":1102814,"serviceName":"hello","releaseId":1834414}}
[event] Event: Network removal {}
[event] Event: Apply config change in progress {}
[event] Event: Apply config change success {}
[event] Event: Network creation {}
[event] Event: Docker image download {"image":{"name":"registry2.balena-cloud.com/v2/2236dbe3386015de90cef16f9d55c201@sha256:135044860e52ad8b6070b858f680faf16fc7ca7e6dd4a85a8cd64ee879a141ce","appId":1839100,"serviceId":1102814,"serviceName":"hello","imageId":3729854,"releaseId":1834414,"dependent":0,"dockerImageId":null}}
[event] Event: Image downloaded {"image":{"name":"registry2.balena-cloud.com/v2/2236dbe3386015de90cef16f9d55c201@sha256:135044860e52ad8b6070b858f680faf16fc7ca7e6dd4a85a8cd64ee879a141ce","appId":1839100,"serviceId":1102814,"serviceName":"hello","imageId":3729854,"releaseId":1834414,"dependent":0,"dockerImageId":null}}
[event] Event: Service install {"service":{"appId":1839100,"serviceId":1102814,"serviceName":"hello","releaseId":1834414}}
[event] Event: Service installed {"service":{"appId":1839100,"serviceId":1102814,"serviceName":"hello","releaseId":1834414}}
[event] Event: Service start {"service":{"appId":1839100,"serviceId":1102814,"serviceName":"hello","releaseId":1834414}}
[info] Internet Connectivity: OK
[event] Event: Service started {"service":{"appId":1839100,"serviceId":1102814,"serviceName":"hello","releaseId":1834414}}
[debug] Spawning journald with: chroot /mnt/root journalctl -a -S 2021-06-17 20:24:03 -o json CONTAINER_ID_FULL=edfe8a5383d54976ec1a14103c91db9be3bcec885dd68d6bb38c93fe78f216e4
[debug] Finished applying target state
[success] Device state apply success
REPOSITORY TAG DIGEST IMAGE ID CREATED SIZE
livepush-supervisor-12.8.7 latest <none> 582baa1eef67 29 minutes ago 527MB
registry2.balena-cloud.com/v2/2236dbe3386015de90cef16f9d55c201 delta-b124973c5cc57951 sha256:135044860e52ad8b6070b858f680faf16fc7ca7e6dd4a85a8cd64ee879a141ce 587671fc12ab 3 hours ago 214MB
registry2.balena-cloud.com/v2/2236dbe3386015de90cef16f9d55c201 delta-b124973c5cc57951 sha256:cd4f2a794ddc4df61b7325c79a3c400f3e57c4a0925de0ec3c550c99ef8cbd87 587671fc12ab 3 hours ago 214MB
registry2.balena-cloud.com/v2/2f787b4a0f149d9f05ebb9f202f4c7ff latest sha256:ba9079a6d38c425d3b5ffad2dd7b5b0d39c25c978ded7727e3cc4e0af6e54b2b 29afe3a0cd02 2 weeks ago 62.9MB
balenalib/armv7hf-alpine-supervisor-base 3.11 sha256:80a6cf83f367aab2f0fe1f3c0dd5bc7da5e321d738d07c845a38dfff05a92c3b a6b88078ad7a 14 months ago 12.1MB
balena-healthcheck-image latest <none> 851163c78e4a 17 months ago 4.85kB
No deltas were used in the second case but the proper digest got tagged with the same delta tag. Since querying the supervisor for the image before the database removal does not return a proper value, the supervisor tries to delete the whole image. Since the engine sees the layers already there, it knows just to tag the image
The text was updated successfully, but these errors were encountered:
This was initially reported by @pipex on the delta server repo but it should really be reported here. Felipe's complete description:
When downloading images with deltas, the target state endpoint provides a target image name as
registry2.balena-cloud.com/v2/<imageName>@<imageDigest>.The supervisor then requests a delta from the source image (on device) to the target image, and gets back a delta address as
registry2.balena-cloud.com/v2/<imageName>:delta-<deltaId>.That image gets pulled by the engine, and gets assigned another digest, which means that the image is no longer queryable by the original digest, making the supervisor image database the only way to maintain a relation between the original image and the image on disk.
See below for an example of feedback from the engine when pulling target images with and without deltas.
Downloading with deltas
Before state
First release
Target state image: registry2.balena-cloud.com/v2/872a7de303e661af19fcf1ea09d12fc8@sha256:9ec0e488aaa12a82f10d8ac2e88a342d0f7527e8a32f76789f82c8820e985176
Supervisor logs
Second release (deltas)
Target state image: registry2.balena-cloud.com/v2/2236dbe3386015de90cef16f9d55c201@sha256:135044860e52ad8b6070b858f680faf16fc7ca7e6dd4a85a8cd64ee879a141ce
Supervisor logs:
Removing the database (same target as previous case)
Target state image: registry2.balena-cloud.com/v2/2236dbe3386015de90cef16f9d55c201@sha256:135044860e52ad8b6070b858f680faf16fc7ca7e6dd4a85a8cd64ee879a141ce
Supervisor logs
No deltas were used in the second case but the proper digest got tagged with the same delta tag. Since querying the supervisor for the image before the database removal does not return a proper value, the supervisor tries to delete the whole image. Since the engine sees the layers already there, it knows just to tag the image
The text was updated successfully, but these errors were encountered: