[Build Guide] build 3FS with docker #78
Comments
|
Thanks for your guide, I used the compiled program in the container as follows:
This error has occurred. Have you encountered it
|
|
根据帖子提示构建的Dockerfile，你们可以尝试构建： # 使用 Ubuntu 22.04 作为基础镜像
FROM ubuntu:22.04
# 设置非交互模式，避免 apt 交互提示
ENV DEBIAN_FRONTEND=noninteractive
# 更新并安装 3FS 构建所需的依赖
RUN apt update && apt install -y \
cmake \
libuv1-dev \
liblz4-dev \
liblzma-dev \
libdouble-conversion-dev \
libdwarf-dev \
libunwind-dev \
libaio-dev \
libgflags-dev \
libgoogle-glog-dev \
libgtest-dev \
libgmock-dev \
clang-format-14 \
clang-14 \
clang-tidy-14 \
lld-14 \
libgoogle-perftools-dev \
google-perftools \
libssl-dev \
gcc-12 \
g++-12 \
libboost-all-dev \
cargo \
git \
wget \
meson \
ninja-build \
dpkg \
&& rm -rf /var/lib/apt/lists/*
# 安装 FoundationDB
WORKDIR /tmp
RUN wget https://github.com/apple/foundationdb/releases/download/7.1.67/foundationdb-server_7.1.67-1_amd64.deb \
https://github.com/apple/foundationdb/releases/download/7.1.67/foundationdb-clients_7.1.67-1_amd64.deb && \
dpkg -i foundationdb-server_7.1.67-1_amd64.deb foundationdb-clients_7.1.67-1_amd64.deb && \
rm foundationdb-server_7.1.67-1_amd64.deb foundationdb-clients_7.1.67-1_amd64.deb
# 构建并安装 fuse（忽略 ninja install 错误）
WORKDIR /tmp
RUN wget https://github.com/libfuse/libfuse/releases/download/fuse-3.16.2/fuse-3.16.2.tar.gz && \
tar -zxvf fuse-3.16.2.tar.gz && \
mkdir -p fuse-3.16.2/build && \
cd fuse-3.16.2/build && \
meson setup .. && \
ninja && \
ninja install || true
# 设置构建目录环境变量
ENV BUILD_DIR=/build
# 创建工作目录，并克隆 3FS 源码进行构建
WORKDIR ${BUILD_DIR}
RUN git clone https://github.com/deepseek-ai/3fs.git && \
cd 3fs && \
git submodule update --init --recursive && \
./patches/apply.sh && \
cmake -S . -B build \
-DCMAKE_CXX_COMPILER=clang++-14 \
-DCMAKE_C_COMPILER=clang-14 \
-DCMAKE_BUILD_TYPE=RelWithDebInfo \
-DCMAKE_EXPORT_COMPILE_COMMANDS=ON && \
cmake --build build -j $(nproc)
# 将 3FS 编译生成的 bin 目录加入 PATH
ENV PATH="${BUILD_DIR}/3fs/build/bin:${PATH}"
CMD ["/bin/bash"]但是，我的思路是确保其能够类似 Ceph 进行多节点集群化部署，并直接管理本地磁盘，如果能借鉴 Ceph 的容器化架构，将 3FS 的各组件解耦成独立容器，但是采用同一镜像，不同的启动命令参数以启动不同角色容器，支持多节点集群部署，并利用 systemd 管理容器的生命周期，这样方便迁移和组件改造升级。 3FS（Fire-Flyer File System）由 集群管理（Cluster Manager） 、 元数据服务（Metadata Service） 、 存储服务（Storage Service） 和 客户端（Client） 四部分组成 (3FS/docs/design_notes.md at main · deepseek-ai/3FS · GitHub)。 那么比如：为存储服务编写/etc/systemd/system/3fs-storage@.service模板， [Unit]
Description=3FS Cluster Manager (mgmtd) Container
After=docker.service
Requires=docker.service
[Service]
Restart=always
ExecStartPre=-/usr/bin/docker rm -f 3fs-mgmtd
ExecStart=/usr/bin/docker run --name 3fs-mgmtd --net=host --privileged \
-v /etc/3fs:/opt/3fs/etc:ro \
deepseek/3fs:latest mgmtd_main
ExecStop=/usr/bin/docker stop 3fs-mgmtd
[Install]
WantedBy=multi-user.target使每块磁盘对应一个实例名启动（实例名可采用磁盘标识或节点ID） docker run --name 3fs-storage-%i \
--hostname %H \
--net=host --ipc=host --ulimit nofile=1048576:1048576 \
--privileged \
-v /dev:/dev \
-v /etc/3fs:/opt/3fs/etc:ro \
-e NODE_ID=%i -e CLUSTER_ID=stage \
deepseek/3fs:latest storage_main然后mgmtd 组建容器类似： docker run --name 3fs-mgmtd --net=host --privileged \
-v /etc/3fs:/opt/3fs/etc:ro \
deepseek/3fs:latest mgmtd_main我感觉完全docker化目前还尚早，等市场反应，稍后应该会有各家集成。 |
|
另外，追加一个精简镜像构建： #####################################
# Stage 1：构建阶段 – 编译 3FS 二进制
#####################################
FROM ubuntu:22.04 AS builder
# 设置非交互模式
ENV DEBIAN_FRONTEND=noninteractive
# 安装编译所需依赖
RUN apt update && apt install -y \
cmake \
libuv1-dev \
liblz4-dev \
liblzma-dev \
libdouble-conversion-dev \
libdwarf-dev \
libunwind-dev \
libaio-dev \
libgflags-dev \
libgoogle-glog-dev \
libgtest-dev \
libgmock-dev \
clang-format-14 \
clang-14 \
clang-tidy-14 \
lld-14 \
libgoogle-perftools-dev \
google-perftools \
libssl-dev \
gcc-12 \
g++-12 \
libboost-all-dev \
cargo \
git \
wget \
meson \
ninja-build \
dpkg \
&& rm -rf /var/lib/apt/lists/*
# 安装 FoundationDB（用于元数据存储）
WORKDIR /tmp
RUN wget https://github.com/apple/foundationdb/releases/download/7.1.67/foundationdb-server_7.1.67-1_amd64.deb \
https://github.com/apple/foundationdb/releases/download/7.1.67/foundationdb-clients_7.1.67-1_amd64.deb && \
dpkg -i foundationdb-server_7.1.67-1_amd64.deb foundationdb-clients_7.1.67-1_amd64.deb && \
rm foundationdb-server_7.1.67-1_amd64.deb foundationdb-clients_7.1.67-1_amd64.deb
# 构建 fuse（可选，如果你需要在客户端容器中使用 FUSE）
WORKDIR /tmp
RUN wget https://github.com/libfuse/libfuse/releases/download/fuse-3.16.2/fuse-3.16.2.tar.gz && \
tar -zxvf fuse-3.16.2.tar.gz && \
mkdir -p fuse-3.16.2/build && \
cd fuse-3.16.2/build && \
meson setup .. && \
ninja && \
ninja install || true
# 克隆并构建 3FS
ENV BUILD_DIR=/build
WORKDIR ${BUILD_DIR}
RUN git clone --depth 1 --recurse-submodules https://github.com/deepseek-ai/3fs.git && \
cd 3fs && \
./patches/apply.sh && \
cmake -S . -B build \
-DCMAKE_CXX_COMPILER=clang++-14 \
-DCMAKE_C_COMPILER=clang-14 \
-DCMAKE_BUILD_TYPE=RelWithDebInfo \
-DCMAKE_EXPORT_COMPILE_COMMANDS=ON && \
cmake --build build -j $(nproc) && \
# 剥除二进制调试信息以减小体积
strip build/bin/*
#####################################
# Stage 2：运行时阶段 – 精简镜像
#####################################
FROM ubuntu:22.04
ENV DEBIAN_FRONTEND=noninteractive
# 安装运行时所需的最低依赖（仅安装必须的运行库）
RUN apt update && apt install -y \
libfuse3-3 \
libssl-dev \
libuv1 \
liblz4-1 \
liblzma5 \
libdouble-conversion3 \
libdwarf1 \
libunwind8 \
libaio1 \
libgflags2.2 \
libgoogle-glog0v5 \
&& rm -rf /var/lib/apt/lists/*
WORKDIR /opt/3fs
# 将构建阶段生成的 3FS 二进制复制到运行镜像中
COPY --from=builder /build/3fs/build/bin/ ./bin/
# 同时复制一份默认配置和文档（根据项目实际情况调整）
COPY --from=builder /build/3fs/configs/ ./etc/
COPY --from=builder /build/3fs/deploy/README.md ./README.md
# 设置 PATH，保证容器启动时可以直接调用 3FS 各服务二进制
ENV PATH="/opt/3fs/bin:${PATH}"
# 默认进入 bash，可在启动时通过指定命令来选择具体服务（如 mgmtd_main、meta_main、storage_main 等）
CMD ["/bin/bash"]示例: docker run --name 3fs-mgmtd --net=host --privileged 3fs-build-lite mgmtd_main以及磁盘： docker run --name 3fs-storage-10001 --net=host --privileged -v /dev:/dev 3fs-build-lite storage_main |
你网络硬件不支持，没有IB网卡吧， docker run --privileged --device=/dev/infiniband:/dev/infiniband -it 3fs-build注意设备 |
这个问题刚刚解决了，是我的clickhouse没有配置正确。 |
Thanks to deepseek, we have access to the outstanding filesystem 3FS.
If you want to compile or use 3FS, here is a guide to compiling with docker for you.
Step 1: create container
Step2: install dependencies
2.1 install dependencies
$ apt update $ apt install -y cmake libuv1-dev liblz4-dev liblzma-dev libdouble-conversion-dev libdwarf-dev libunwind-dev \ libaio-dev libgflags-dev libgoogle-glog-dev libgtest-dev libgmock-dev clang-format-14 clang-14 clang-tidy-14 lld-14 \ libgoogle-perftools-dev google-perftools libssl-dev gcc-12 g++-12 libboost-all-dev cargo git g++ wget meson2.2 install foundationdb
2.3 build and install fuse
Step 3: build 3FS
It will generates binrary shown as below once build success:
The text was updated successfully, but these errors were encountered: