error pulling image configuration: download failed after attempts=1: unknown blob

[jmouel]

docker-registry-proxy is able to cache all of my images, with one exception: registry.k8s.io/kube-state-metrics/kube-state-metrics

• I've verified that the image can be pulled without docker-registry-proxy.
• I'm using rpardini/docker-registry-proxy:0.6.2-debug.

Client log:

> docker pull registry.k8s.io/kube-state-metrics/kube-state-metrics:v2.5.0

v2.5.0: Pulling from kube-state-metrics/kube-state-metrics
b0b160e41cf3: Pulling fs layer 
4a3980d81ef5: Pulling fs layer 
error pulling image configuration: download failed after attempts=1: unknown blob

Proxy log:

{"access_time":"01/Dec/2022:19:37:31 +0000","upstream_cache_status":"","method":"GET","uri":"/v2/","request_type":"unknown","status":"200","bytes_sent":"0","upstream_response_time":"0.104","host":"registry.k8s.io","proxy_host":"registry.k8s.io","upstream":"34.107.244.51:443"}
{"access_time":"01/Dec/2022:19:37:31 +0000","upstream_cache_status":"HIT","method":"HEAD","uri":"/v2/kube-state-metrics/kube-state-metrics/manifests/v2.5.0","request_type":"manifest-secondary","status":"200","bytes_sent":"0","upstream_response_time":"","host":"registry.k8s.io","proxy_host":"registry.k8s.io","upstream":""}
{"access_time":"01/Dec/2022:19:37:31 +0000","upstream_cache_status":"HIT","method":"GET","uri":"/v2/kube-state-metrics/kube-state-metrics/manifests/sha256:09a36e2be1dbda6009641235d90be8627c9616d42a0b887b770fc92c1753b74a","request_type":"manifest-by-digest","status":"200","bytes_sent":"1665","upstream_response_time":"","host":"registry.k8s.io","proxy_host":"registry.k8s.io","upstream":""}
{"access_time":"01/Dec/2022:19:37:32 +0000","upstream_cache_status":"HIT","method":"GET","uri":"/v2/kube-state-metrics/kube-state-metrics/manifests/sha256:502b5e5b3f394103badf0dc78b5e3a004b50c33062fa35c17ca094dc9b552c28","request_type":"manifest-by-digest","status":"200","bytes_sent":"739","upstream_response_time":"","host":"registry.k8s.io","proxy_host":"registry.k8s.io","upstream":""}
{"access_time":"01/Dec/2022:19:37:32 +0000","upstream_cache_status":"MISS","method":"GET","uri":"/v2/kube-state-metrics/kube-state-metrics/blobs/sha256:b0b160e41cf35d4a469a1209cfd553534b248053060de060cbbc4a5d912ad5f0","request_type":"blob-by-digest","status":"302","bytes_sent":"820","upstream_response_time":"0.102 : 0.320","host":"registry.k8s.io","proxy_host":"us-west2-docker.pkg.dev","upstream":"34.107.244.51:443 : 74.125.135.82:443"}
{"access_time":"01/Dec/2022:19:37:32 +0000","upstream_cache_status":"MISS","method":"GET","uri":"/v2/kube-state-metrics/kube-state-metrics/blobs/sha256:64c53325f62ed939131f8cad80c0e0f99016abdf012955386048b094e477ae50","request_type":"blob-by-digest","status":"302","bytes_sent":"820","upstream_response_time":"0.102 : 0.320","host":"registry.k8s.io","proxy_host":"us-west2-docker.pkg.dev","upstream":"34.107.244.51:443 : 74.125.135.82:443"}
{"access_time":"01/Dec/2022:19:37:32 +0000","upstream_cache_status":"MISS","method":"GET","uri":"/v2/kube-state-metrics/kube-state-metrics/blobs/sha256:4a3980d81ef5e2ae25e42212cc5501796f3d3596c88e92e739fe422d5bd6fce9","request_type":"blob-by-digest","status":"302","bytes_sent":"820","upstream_response_time":"0.102 : 0.321","host":"registry.k8s.io","proxy_host":"us-west2-docker.pkg.dev","upstream":"34.107.244.51:443 : 74.125.135.82:443"}
{"access_time":"01/Dec/2022:19:37:32 +0000","upstream_cache_status":"","method":"GET","uri":"/artifacts-downloads/namespaces/k8s-artifacts-prod/repositories/images/downloads/AHi26s-hfTjt1XsIrPM3uXRwGSJ_dKmyrkkBRNgON5xB-oxXmXNhN9uJidTe6TZ_N2UabCTVXBxPAHb9ub2dfChQTAFk6tkBfieXWPiS9vcegn3D44Iob7KPwQivCUei6eNrtYiSzWJnfuVwBI9EcGE761JFU-vDMA670cRS2QQPaK7JNXbQo6EL_4M2t9uljfZtHCZRB-qZBlgrtrFaw4QOMpmRKVVr-byK2VieIoafjeP_YzQwwitMEP5vnA0dpXhp4ckEYn1EYJ690KmnMcfewUZ6AgHGWvz8vfToOlMwOMbIsc1FWtAPNujwuO_Fx_tfBJeI43H02o98KOQbIXam2g_JOpJIttxSexdJlmG_VJbbFvvtnBuwXKwSeU3kqYDZBzB1zuA8VVpplZC4QN9ROMgbBW6ttsg5ldby7ondtOQWjsdo4VzsIRb_l8qbBUwLwTkR-fnuj23ukIcE4AVHgCC4L1fm6DZZAxWnra8mJ3cAYiraJqYe1s38LH34E_JUT3JH-3su5VTntZSgyqrFgXU_wneYrIK9NWHWyu8XYmYbIs1iDHyt6iedfeGYlZSCG4MuFNe-Igye9ZiEMfCLOI2aJg7cr3ifYgr0JlfatAZFFLcyyaX2NyHqfJZqr5OMSEwoBUPIoE6ykbYvM-Wq3nWgx5wWX_2a-FkVSP2_KL9SnqExFYlQkpZmSrKSgJ6_m7PSRruS","request_type":"unknown","status":"404","bytes_sent":"19","upstream_response_time":"0.099","host":"registry.k8s.io","proxy_host":"registry.k8s.io","upstream":"34.107.244.51:443"}
{"access_time":"01/Dec/2022:19:37:32 +0000","upstream_cache_status":"","method":"GET","uri":"/artifacts-downloads/namespaces/k8s-artifacts-prod/repositories/images/downloads/AHi26s_Sf3rHF44nEHy_RlJCsBekdofRidJoJdiiHK0l9pzwywAr3U_VLaZdMUxanNfSlyy9wyoXaaXTupqgp4gclljGMT0AMzrvtpKiiFrAorZuvNlVfl9Puqh2YsROei7aZTUhXJ9i1h6DGtCXPzGZjdxQ4iYPih_zpoqGJl5KonrNR58KI-1Ixb2pDFAC1Jw6r-vPeabRLymdwbjozYTG9UoyyiLM76K4tykU8L2oDX_PYoX3nqanC2LPcOj_4fJ2ITlU7q12sJO1d-QwcB7iDaS1nhl3Ckd56X-sXRH6x3GotvGYCVqeiRg5iunNSejA3QtMGSXz9erm853D3NZQJbCmU5xv66DULaE2QQ83pks1rUWCWoBjzQNtYRi9-HfunnxVckiCLU1KlSUMpq3a6qrPc9n6X1PeYvsTorvhjnl3BjquUm7ETufBHvyFmXC8oaXh5aJv9hs0C5mBX1eJ1hX96kzMVdw-QGpMoNJlvfCs1ViaBqHKEdWT1aF0R2VqlNMKixYu7NjBjoYN6Ilho1smZI1Q1WJjoEUjjkHSuFCN6sRWfKof4l7RI34dpKmaJ_8WDWQprb1dNPsjhgGq2dpw4wT8sau-1N6uyy_TtdHrXeDdN6STP1MjgJ6tYk_DszTDwz51jfLXc7F9_5disfxI4tl_LIHtYYHiLuSZcJsOwAupPD7zMwIue2DLvZxYKPn2rFl4","request_type":"unknown","status":"404","bytes_sent":"19","upstream_response_time":"0.100","host":"registry.k8s.io","proxy_host":"registry.k8s.io","upstream":"34.107.244.51:443"}
{"access_time":"01/Dec/2022:19:37:32 +0000","upstream_cache_status":"","method":"GET","uri":"/artifacts-downloads/namespaces/k8s-artifacts-prod/repositories/images/downloads/AHi26s-zyDoQPMeuqms-83rvCaGDub9EUJfzVgH3Uy30ybZ1PUbxuxUrXwp3Lv_BWAWs5zgjuurLHYPTRj9aYIibC5DPu7aKr8s74UUu0hN4e5Ibrl4srLkpO-uL3DEL8WMXxeWJ2WKqSUq6VVJgfZawv8ACieI6b7P1twyUHKlfDmEwtEnOUaQWG52LrwBOgEHiy8lPfUhnR0tYhmZ5EjZTCHJOHQQN4WhE4WHq5rFFTDXhsfJ-9QUc5PvoVW3qOF5H2a--BGFkZu1VYsVDXy6paYB8KgiXBB5iblUlW-VO6iRiRhWxV7LsCUOdr00AZLqIqjW8cLHrKixzFm19sJaaFMr6DEAw74rvJ5TBexyqnRN6aEWLszt4mbpdPAGPNTrPuT72pPo1glRO5R153y3nvK-uxEWX0HmD0PjLfmmYqIurQz4-8Aa3jolaslFqLcKY1Y0v2qS_HsYUetwV9eLBMHMg8nx1lxvMUfODm0XLBbBQwocg9sq-3MqvyJQZFi7iQJy67oR0q6jlnXaLjGkNe9GoTeHgwebt0FoI0hdfZTAe0zD9LoG0AQkWlDJdIIN4C9CgH5Fdaqqdi7OJ8XZitVj9Clc_cUc0U-C7t72zPcxTpkoih3nhf8fb8AuURc59AB_K_h7awrK3LNff5p6ZekoLJ47Dw6Dd1YT6lAcT9HLjJ3NHy_KXdyhlYhYgLeM7y34OQLc8","request_type":"unknown","status":"404","bytes_sent":"19","upstream_response_time":"0.100","host":"registry.k8s.io","proxy_host":"registry.k8s.io","upstream":"34.107.244.51:443"}
{"access_time":"01/Dec/2022:19:37:32 +0000","upstream_cache_status":"MISS","method":"GET","uri":"/v2/kube-state-metrics/kube-state-metrics/blobs/sha256:b0b160e41cf35d4a469a1209cfd553534b248053060de060cbbc4a5d912ad5f0","request_type":"blob-by-digest","status":"302","bytes_sent":"820","upstream_response_time":"0.104 : 0.295","host":"registry.k8s.io","proxy_host":"us-west2-docker.pkg.dev","upstream":"34.107.244.51:443 : 74.125.135.82:443"}
{"access_time":"01/Dec/2022:19:37:33 +0000","upstream_cache_status":"MISS","method":"GET","uri":"/v2/kube-state-metrics/kube-state-metrics/blobs/sha256:4a3980d81ef5e2ae25e42212cc5501796f3d3596c88e92e739fe422d5bd6fce9","request_type":"blob-by-digest","status":"302","bytes_sent":"820","upstream_response_time":"0.104 : 0.322","host":"registry.k8s.io","proxy_host":"us-west2-docker.pkg.dev","upstream":"34.107.244.51:443 : 74.125.135.82:443"}
{"access_time":"01/Dec/2022:19:37:33 +0000","upstream_cache_status":"","method":"GET","uri":"/artifacts-downloads/namespaces/k8s-artifacts-prod/repositories/images/downloads/AHi26s_1xtmsE04mXjTA4MrmwZP5Vs-9OxugR-tKxCYtI31VHt-N2DzAmUEKR9xy1qknsQTMpsVJFhrtfttaX3tCcoPwhNObpzYdcUxD6WFxgZIgfNydfwDP3E1PR7sPBsxbCfCjGierlEitsZeNYv6xsy8VP7n0lHfCWt8nE9AKqDaAvY4zdNFDvvuIsDWZ06hkJxq-BvB8yU63riicGL7HAKT-R-7HPKXLuotNYKcH3464llOoc-JSn1lRILSkT7t5uGljHZ01QohlStOlcKERqbSy1ugK-q8HHa3WCysPoBrIOv4MI50Go6fc9qMDDT6cuerZcQkc5_aAAelfuVZOH8OJVcKP2xlZ6ds18S1G4WAAcP_LzcP5Nr184KSotpLQ-aIn7EDxTf8YLbrCb7135GQjgY7aTqTQ5KK41FqJ353knuII_YpRd6JXOKszhNvK_rjvCjgkNNW5Dx_lx7WU_dP_A2N-A6cuBQQ7vjr6F4vvTEegx6IXXYYdOCelsOL9mt4V_48XvzEgfT1uGVcmHGH1AaiKFW7eq8KmNWzd-U-ll2zwSoGQ4hyEZw1TNxFzylpFrUdiIn9OTw_SsWeHjUCPdchmy3hcmMTUEd1KFQY_KgqjT2oiBMiDkHGoEOqhCENVbHXkaPjzziVRFpz1vVJd_w51P4GJ3WKmsb92y6rSpQhQJDw0HJLotIsWKQfJsaD9KxGc","request_type":"unknown","status":"404","bytes_sent":"19","upstream_response_time":"0.098","host":"registry.k8s.io","proxy_host":"registry.k8s.io","upstream":"34.107.244.51:443"}
{"access_time":"01/Dec/2022:19:37:33 +0000","upstream_cache_status":"","method":"GET","uri":"/artifacts-downloads/namespaces/k8s-artifacts-prod/repositories/images/downloads/AHi26s8ypzx983srXbTD3ny_am6I46tfv7rLB5LiPfDZWewNVbI76ZuDZdFCT8lfE8Rf9u7h73-MBbsqsyK4eeCHC3yQyIMhbKIVzjvowDG9hP6abf1KtK8kaXyO8xQ4Eor1NoNRN1diiGmd87FDh6IzCpmAPs1kb9G4TjmNF2-m5MTRkAEmCorbuIxJgupuPf3wvxwS4GNFzN7qV-42jMIrGtysp2yUk2ucY9-5gh01inQ81GuKVQ24Vtsb5KteiPXTGkWjAGs8j5A-EP2XtS3O1cciK3OwW_ZCmdJHOnwirMwy6pvz7xMjBMsBe-MaHjw4b9k4Z8ySzjpALCO-bdMGI5-gzTyVOiyBvO8uMzKD3Hp2pSyvqvlIpz9gcuPPJlfk7G4de7ILPa0podXzh-I3dEmlm2S5MEpD1o-HPBv71WRz7Dqae1-EbmfNDAgPlkPUexXu_0LmyHGcQMz8NWfNijQi1D_Z0GXvmKGRDXJemP-IEh2QxVaJ4CB9a1nTH3U20Qx1vxpSgCVWLYO9jqG0vTu2T2rG6lc6PRD1rUwj2Ldq1p0mtklVlaorrxgJy5EWakdUc-V_z03te5M2XlXlQQacQjp-OJ1_DkxGD0zyxRRB-oGA4zycjXKpEPZVuFspuEXz5txf6Mm_i_T2jjMWspn4EDZ7IjAmI4XaCsNxuJyFbx4_oaLAuuD2Ilo2Sa1SOE9m7wX1","request_type":"unknown","status":"404","bytes_sent":"19","upstream_response_time":"0.108","host":"registry.k8s.io","proxy_host":"registry.k8s.io","upstream":"34.107.244.51:443"}

[cheretbe]

Same here. Everything from registry.k8s.io fails to download

docker pull registry.k8s.io/kube-controller-manager:v1.25.3
v1.25.3: Pulling from kube-controller-manager
79e0d8860fad: Pulling fs layer 
78ef12c4eeb6: Pulling fs layer 
f04c5f7b8e15: Pulling fs layer 
error pulling image configuration: download failed after attempts=1: unknown blob

Without docker-registry-proxy images are pulled OK. docker-registry-proxy version is 0.6.4