Archive size limited to 4GB when downloading selected photos from the web app

[voxxys]

The bug

When I select many photos in the web application (so that cumulative size of selected photos is larger than 4 GB) and download this selection, the archive of 4 GB size is prepared and downloaded. The archive can be unpacked and contains only part of selected photos, i.e. the first N oldest photos that could fit into these 4 GB.

The OS that Immich Server is running on

raspbian 10 buster

Version of Immich Server

v1.100.0

Version of Immich Mobile App

Platform with the issue

• Server
• Web
• Mobile

Your docker-compose.yml content

name: immich

services:
  immich-server:
    container_name: immich_server
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    command: ['start.sh', 'immich']
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      - /etc/localtime:/etc/localtime:ro
    env_file:
      - .env
    ports:
      - 2283:3001
    depends_on:
      - redis
      - database
    restart: always

  immich-microservices:
    container_name: immich_microservices
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    # extends: # uncomment this section for hardware acceleration - see https://immich.app/docs/features/hardware-transcoding
    #   file: hwaccel.transcoding.yml
    #   service: cpu # set to one of [nvenc, quicksync, rkmpp, vaapi, vaapi-wsl] for accelerated transcoding
    command: ['start.sh', 'microservices']
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      - /etc/localtime:/etc/localtime:ro
    env_file:
      - .env
    depends_on:
      - redis
      - database
    restart: always

  immich-machine-learning:
    container_name: immich_machine_learning
    # For hardware acceleration, add one of -[armnn, cuda, openvino] to the image tag.
    # Example tag: ${IMMICH_VERSION:-release}-cuda
    image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release}
    # extends: # uncomment this section for hardware acceleration - see https://immich.app/docs/features/ml-hardware-acceleration
    #   file: hwaccel.ml.yml
    #   service: cpu # set to one of [armnn, cuda, openvino, openvino-wsl] for accelerated inference - use the `-wsl` version for WSL2 where applicable
    volumes:
      - model-cache:/cache
    env_file:
      - .env
    restart: always

  redis:
    container_name: immich_redis
    image: registry.hub.docker.com/library/redis:6.2-alpine@sha256:51d6c56749a4243096327e3fb964a48ed92254357108449cb6e23999c37773c5
    restart: always

  database:
    container_name: immich_postgres
    image: registry.hub.docker.com/tensorchord/pgvecto-rs:pg14-v0.2.0@sha256:90724186f0a3517cf6914295b5ab410db9ce23190a2d9d0b9dd6463e3fa298f0
    environment:
      POSTGRES_PASSWORD: ${DB_PASSWORD}
      POSTGRES_USER: ${DB_USERNAME}
      POSTGRES_DB: ${DB_DATABASE_NAME}
    volumes:
      - pgdata:/var/lib/postgresql/data
    restart: always

volumes:
  pgdata:
  model-cache:

Your .env content

# You can find documentation for all the supported env variables at https://immich.app/docs/install/environment-variables

# The location where your uploaded files are stored
UPLOAD_LOCATION=./app/library

# The Immich version to use. You can pin this to a specific version like "v1.71.0"
IMMICH_VERSION=release

# Connection secrets for postgres and typesense. You should change these to random passwords
DB_PASSWORD=*****

# The values below this line do not need to be changed
###################################################################################
DB_HOSTNAME=immich_postgres
DB_USERNAME=postgres
DB_DATABASE_NAME=immich

REDIS_HOSTNAME=immich_redis

LOG_LEVEL=log

Reproduction steps

1. In the web app, select many photos with cumulative size of more than 4 GB and click download option in the menu. Alternatively, open an album with the size of more than 4 GB and click the download icon.
2. Observe the archive of exactly 4 GB size being prepared and downloaded.
3. Unpack the archive and check the files downloaded. The downloaded files comprise only a part of the original selection that fit 4 GB.

Additional information

I've checked that this bug happens:

• on two different accounts;
• on two different PCs;
• on Chrome and Firefox.

The file system of the disk where the archive is downloaded to is NTFS.

[jrasm91]

I'm pretty sure I have downloaded large archives. Are you perhaps using a reverse proxy? Does the download work without it?

[voxxys]

I just tried to download without using reverse proxy and it behaves the same, still limited to 4 GB

[voxxys]

I explored a bit more and figured out that all selected photos are eventually downloaded, it just happens in 4 GB parts (first the [ALBUM_NAME]+1-[DATETIME] is prepared and saved, then the next part etc.

It wat not obvious to me because:

• the overlays showing the progress bar for these parts (in the lower left corner) don't appear all at once but are added successively, i.e. the progress bar for the second part only appears after the first part is completed and saved to the disk;
• when the progress bar of the second part reached completion, I got a prompt from the browser requesting me to allow multiple files to be saved. When I did, it did save the second part archive. Initially, I didn't closely observe the files being downloaded for the whole time, probably missed this prompt and ended up with only the first part being saved.

So there is no issue with file download, as long as you wait for the second archive part to be completed and then allow downloading multiple files when prompted. Also, this probably only happens the first time you try to download a multipart archive, unless you reset permissions for that page.

There's no bug but I guess some things could help prevent this confusion:

• Maybe the fact that multiple parts will be successively prepared and saved could be made more obvious in the UI, e.g. creating all the progress bars for archive parts at the same time (so that a user can see how many there are) and then progressing them one at a time?
• The permission to download multiple files could be obtained at once, when user triggers a multipart download, rather than at the completion of the second part. A download of this size takes a lot of time and the user will not necessarily still be looking at the same tab when the second part is ready and the prompt appears.
• The fact that large downloads are broken up in 4 GB pieces could be reflected in documentation (I searched and did not find any notion of this fact anywhere, which made me think that the first download being 4 GB was already a bug, not that there was an issue with the further parts).

[jrasm91]

Ah, I think you are right. I was going to make it configurable at one point but never got to it. I forgot it broke up the download into archives.

[lbckmnn]

@jrasm91 can this be changed somehow?
My album is a little bit over 4GB.
I had some troubles with the individual zip files (archive tool reports the first one as corrupt).

[jrasm91]

Yes this should be an easy one. I'll try to add an option to update it.

[bubundas17]

This option is needed.
I have shared a album over 100GB. And they can't download them at once.