Fails to open terminal to an application container with: OCI runtime exec failed

[thgreasi]

Description

When trying from the hostOS to connect to a container using balena exec -it <container> sh we get:

OCI runtime exec failed: exec failed: container_linux.go:349: starting container process caused "process_linux.go:93: starting setns process caused "fork/exec /proc/self/exe: no such file or directory"": unknown

This also happens from the webterminal as well.

Output of balena-engine version:

Client:
 Version:           19.03.18
 API version:       1.40
 Go version:        go1.12.17
 Git commit:        840aacc77b6c600b3b929fe9e4d9356a322b9e5b
 Built:             Tue Jun  8 15:38:05 2021
 OS/Arch:           linux/arm64
 Experimental:      false

Server:
 Engine:
  Version:          19.03.18
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.12.17
  Git commit:       840aacc77b6c600b3b929fe9e4d9356a322b9e5b
  Built:            Tue Jun  8 15:38:05 2021
  OS/Arch:          linux/arm64
  Experimental:     true
 containerd:
  Version:          1.2.0+unknown
  GitCommit:        
 runc:
  Version:          1.0.0-rc4+dev
  GitCommit:        dc9208a3303feef5b3839f4323d9beb36df0a9dd
 balena-engine-init:
  Version:          0.13.0
  GitCommit:        949e6fa-dirty

Output of balena-engine info:

Client:
 Debug Mode: false

Server:
 Containers: 6
  Running: 6
  Paused: 0
  Stopped: 0
 Images: 7
 Server Version: 19.03.18
 Storage Driver: overlay2
  Backing Filesystem: <unknown>
  Supports d_type: true
  Native Overlay Diff: true
 Logging Driver: journald
 Cgroup Driver: systemd
 Plugins:
  Volume: local
  Network: bridge host null
  Log: journald json-file local
 Swarm: 
  NodeID: 
  Is Manager: false
  Node Address: 
 Runtimes: bare runc
 Default Runtime: runc
 Init Binary: balena-engine-init
 containerd version: 
 runc version: dc9208a3303feef5b3839f4323d9beb36df0a9dd
 init version: 949e6fa-dirty (expected: fec3683b971d9)
 Kernel Version: 5.4.83-v8
 Operating System: balenaOS 2.80.3+rev1
 OSType: linux
 Architecture: aarch64
 CPUs: 4
 Total Memory: 1.856GiB
 Name: bcfb28e
 ID: KHSA:H3SN:PQWD:ARCX:C7D7:RP3C:6DNX:RK7G:2UIO:OLSH:ERU7:S4AX
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: true
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: No swap limit support

Additional environment details (device type, OS, etc.):
RPi4 balenaOS 2.80.3+rev1 supervisor 12.7.0

See: https://jel.ly.fish/b342978c-ab0c-404b-8bd3-b5f33c7864f9
See: https://jel.ly.fish/301adc3a-0b81-45f3-a4e0-e8a58bba628d

[klutchell]

After restarting the engine the issue was resolved.

This is key info from one of the support threads.

[klutchell]

Related: https://github.com/balena-io/balena-io/issues/2280

[jellyfish-bot]

[cywang117] This has attached https://jel.ly.fish/b81b0085-a3da-403a-b406-7877ddbdb82a

[cywang117]

The issue appears in the above support thread, with similar engine version and info. Restarting the engine with systemctl restart balena fixed the issue as mentioned above.

root@6d6f52f:~# balena-engine version
Client:
 Version:           19.03.13-dev
 API version:       1.40
 Go version:        go1.12.17
 Git commit:        074a481789174b4b6fd2d706086e8ffceb72e924
 Built:             Tue Jan 19 13:26:33 2021
 OS/Arch:           linux/arm64
 Experimental:      false

Server:
 Engine:
  Version:          19.03.13-dev
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.12.17
  Git commit:       074a481789174b4b6fd2d706086e8ffceb72e924
  Built:            Tue Jan 19 13:26:33 2021
  OS/Arch:          linux/arm64
  Experimental:     true
 containerd:
  Version:          1.2.0+unknown
  GitCommit:        
 runc:
  Version:          
  GitCommit:        dc9208a3303feef5b3839f4323d9beb36df0a9dd
 balena-engine-init:
  Version:          0.13.0
  GitCommit:        949e6fa-dirty

root@6d6f52f:~# balena-engine info
Client:
 Debug Mode: false

Server:
 Containers: 2
  Running: 2
  Paused: 0
  Stopped: 0
 Images: 2
 Server Version: 19.03.13-dev
 Storage Driver: overlay2
  Backing Filesystem: <unknown>
  Supports d_type: true
  Native Overlay Diff: true
 Logging Driver: journald
 Cgroup Driver: systemd
 Plugins:
  Volume: local
  Network: bridge host null
  Log: journald json-file local
 Swarm: 
  NodeID: 
  Is Manager: false
  Node Address: 
 Runtimes: bare runc
 Default Runtime: runc
 Init Binary: balena-engine-init
 containerd version: 
 runc version: dc9208a3303feef5b3839f4323d9beb36df0a9dd
 init version: 949e6fa-dirty (expected: fec3683b971d9)
 Kernel Version: 5.4.83-v8
 Operating System: balenaOS 2.67.3+rev4
 OSType: linux
 Architecture: aarch64
 CPUs: 4
 Total Memory: 3.756GiB
 Name: 6d6f52f
 ID: S7ZK:KJXJ:A2WL:I4BZ:NLIW:Q5Y6:ENL6:M4GO:RSNA:CRMP:AEUK:S6C5
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: true
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: API is accessible on http://127.0.0.1:2375 without encryption.
         Access to the remote API is equivalent to root access on the host. Refer
         to the 'Docker daemon attack surface' section in the documentation for
         more information: https://docs.docker.com/engine/security/security/#docker-daemon-attack-surface
WARNING: No swap limit support

[jellyfish-bot]

[cywang117] This has attached https://jel.ly.fish/6c42a7d7-ff57-417c-a0d5-e4de3d04dc76

[jellyfish-bot]

[majorz] This has attached https://jel.ly.fish/702cd95b-1a14-413f-a749-ee2bedbe2cf5