Investigate running x86 kubernetes on M1

[ilya-zuyev]

Docker Desktop for M1 now supports running x86 containers

lyaz@host --- /tmp » docker version
Client:
 Cloud integration: 1.0.14
 Version:           20.10.6
 API version:       1.41
 Go version:        go1.16.3
 Git commit:        370c289
 Built:             Fri Apr  9 22:46:57 2021
 OS/Arch:           darwin/arm64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.6
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.13.15
  Git commit:       8728dd2
  Built:            Fri Apr  9 22:44:13 2021
  OS/Arch:          linux/arm64
  Experimental:     true
 containerd:
  Version:          1.4.4
  GitCommit:        05f951a3781f4f2c1911b05e61c160e9c30eaa8e
 runc:
  Version:          1.0.0-rc93
  GitCommit:        12644e614e25b05da6fd08a38ffa0cfe1903fdec
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0
ilyaz@host --- /tmp » arch
arm64
ilyaz@host --- /tmp » docker run -it --platform linux/amd64 ubuntu
root@29557202deef:/# arch
x86_64
root@29557202deef:/# ps aux
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         1  1.6  0.1 222560 13832 pts/0    Ssl  04:35   0:00 /usr/bin/qemu-x86_64 /usr/bin/bash
root        18  0.0  0.1 223956  8916 ?        Rl+  Jun21   0:00 /usr/bin/ps aux
root@29557202deef:/#

We need to investigate if it's possible to run x86_64 kicbase there.
If yes, we could give a lot of flexibility to our users, including building mixed arm/x86 minikube clusters

It's qemu and probably is pretty slow, but worths a shot

[afbjorklund]

On a side note, we have been using this to run aarch64 processes on amd64 for a while now...

https://www.docker.com/blog/getting-started-with-docker-for-arm-on-linux/

It is also possible to run the whole OS (system), but then you lose the KVM/HVF acceleration.

https://qemu-project.gitlab.io/qemu/about/ (for "system" vs "user mode")

Sometimes it has to be done, like when it is not possible to cross-compile for instance.
The Go compiler, and some other software, still have issues when running under QEMU.

We would require multi-node for deploying VM workers of different architecture, though.
But it should be possible to join an emulated worker node to a native control plane node...

[MadhavJivrajani]

/remove-kind design
/kind feature
kind/design is migrated to kind/feature, see kubernetes/community#6144 for more details

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[MPLew-is]

Is there any particular reason this was closed? This would be a huge feature to have, to run x86 images on an M1 cluster.

[yuvipanda]

Would love to have this be opened again!

[MPLew-is]

For what it's worth for anyone else in the same position, I've migrated to colima for this purpose and have been quite pleased. Emulation works flawlessly, supports Docker contexts (rather than having to use minikube docker-env), and the filesystem mounting is much more stable than Minikube's.

[66Ton99]

It should be natively supported by vfkit driver