Smart Search doesn't work. Unknown error: Error: Machine learning request '{"clip":{"textual":{"modelName":"ViT-B-32__openai"}}}' failed for all URLs

[wlodzimierz-dzis]

The bug

Hello team,

After updating to latest verstion v1.125.7 smart serach stopped working. I'm using Immich server on Synology NAS as a host and remote ML with microservices on PC for remote jobs. Both remote and host have the same versions.

Failed to search smart (Immich Server Error)

Server version

Immich server logs

[Nest] 32 - 02/01/2025, 6:09:58 PM WARN [Api:MachineLearningRepositoryxx39mh3a] Machine learning request to "http://192.168.50.217:3003" failed: fetch failed
[Nest] 32 - 02/01/2025, 6:10:09 PM WARN [Api:MachineLearningRepositoryxx39mh3a] Machine learning request to "http://immich-machine-learning:3003" failed: fetch failed
[Nest] 32 - 02/01/2025, 6:10:09 PM ERROR [Api:ErrorInterceptor~xx39mh3a] Unknown error: Error: Machine learning request '{"clip":{"textual":{"modelName":"ViT-B-32__openai"}}}' failed for all URLs
Error: Machine learning request '{"clip":{"textual":{"modelName":"ViT-B-32__openai"}}}' failed for all URLs
at MachineLearningRepository.predict (/usr/src/app/dist/repositories/machine-learning.repository.js:38:15)
at process.processTicksAndRejections (node:internal/process/task_queues:105:5)
at async MachineLearningRepository.encodeText (/usr/src/app/dist/repositories/machine-learning.repository.js:61:26)
at async SearchService.searchSmart (/usr/src/app/dist/services/search.service.js:60:27)

Immich remote ML logs

[02/01/25 15:19:38] INFO Starting gunicorn 23.0.0
[02/01/25 15:19:38] INFO Listening at: http://[::]:3003 (9)
[02/01/25 15:19:38] INFO Using worker: app.config.CustomUvicornWorker
[02/01/25 15:19:38] INFO Booting worker with pid: 10
[02/01/25 15:19:43] INFO Started server process [10]
[02/01/25 15:19:43] INFO Waiting for application startup.
[02/01/25 15:19:43] INFO Created in-memory cache with unloading after 300s
of inactivity.
[02/01/25 15:19:43] INFO Initialized request thread pool with 12 threads.
[02/01/25 15:19:43] INFO Application startup complete.
[02/01/25 15:46:37] INFO Starting gunicorn 23.0.0
[02/01/25 15:46:37] INFO Listening at: http://[::]:3003 (9)
[02/01/25 15:46:37] INFO Using worker: app.config.CustomUvicornWorker
[02/01/25 15:46:37] INFO Booting worker with pid: 10
[02/01/25 15:46:43] INFO Started server process [10]
[02/01/25 15:46:43] INFO Waiting for application startup.
[02/01/25 15:46:43] INFO Created in-memory cache with unloading after 300s
of inactivity.
[02/01/25 15:46:43] INFO Initialized request thread pool with 12 threads.
[02/01/25 15:46:43] INFO Application startup complete.
[02/01/25 15:50:16] INFO Loading visual model 'ViT-B-32__openai' to memory
[02/01/25 15:50:16] INFO Setting execution providers to
['CUDAExecutionProvider', 'CPUExecutionProvider'],
in descending order of preference
[02/01/25 15:50:19] INFO Loading detection model 'buffalo_l' to memory
[02/01/25 15:50:19] INFO Setting execution providers to
['CUDAExecutionProvider', 'CPUExecutionProvider'],
in descending order of preference
[02/01/25 15:50:22] INFO Loading recognition model 'buffalo_l' to memory
[02/01/25 15:50:22] INFO Setting execution providers to
['CUDAExecutionProvider', 'CPUExecutionProvider'],

The OS that Immich Server is running on

Synology Nas as a host, Windows 11 with Nvidia GPU as a remote ML

Version of Immich Server

v.1.125.7

Version of Immich Mobile App

v.1.125.7

Platform with the issue

• Server
• Web
• Mobile

Your docker-compose.yml content

=================================================HOST================================

services:
  immich-redis:
    image: redis
    container_name: Immich-REDIS
    hostname: immich-redis
    security_opt:
      - no-new-privileges:true
    healthcheck:
      test: ["CMD-SHELL", "redis-cli ping || exit 1"]
    user: 1026:100
    environment:
      - TZ=Europe/Warsaw
    ports:
      - 6379:6379
    volumes:
      - /volume1/docker/immich/redis:/data:rw
    restart: on-failure:5

  immich-db:
    image: tensorchord/pgvecto-rs:pg16-v0.2.0
    container_name: Immich-DB
    hostname: immich-db
    security_opt:
      - no-new-privileges:true
    healthcheck:
      test: ["CMD", "pg_isready", "-q", "-d", "immich", "-U", "immichuser"]
      interval: 10s
      timeout: 5s
      retries: 5
    volumes:
      - /volume1/docker/immich/db:/var/lib/postgresql/data:rw
    ports:
      - 5433:5432
    environment:
      - TZ=Europe/Warsaw
      - POSTGRES_DB=immich
      - POSTGRES_USER=immichuser
      - POSTGRES_PASSWORD=immichpw
    restart: on-failure:5

  immich-server:
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    container_name: Immich-SERVER
    hostname: immich-server
    user: 1026:100
    security_opt:
      - no-new-privileges:true
    env_file:
      - stack.env
    ports:
      - 2283:2283
    volumes:
      - /volume1/docker/immich/upload:/usr/src/app/upload:rw
    restart: on-failure:5
    depends_on:
      immich-redis:
        condition: service_healthy
      immich-db:
        condition: service_started

  immich-machine-learning:
    image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release}
    container_name: Immich-LEARNING
    hostname: immich-machine-learning
    user: 1026:100
    security_opt:
      - no-new-privileges:true
    env_file:
      - stack.env
    volumes:
      - /volume1/docker/immich/upload:/usr/src/app/upload:rw
      - /volume1/docker/immich/cache:/cache:rw
      - /volume1/docker/immich/matplotlib:/matplotlib:rw
    environment:
      - MPLCONFIGDIR=/matplotlib
    restart: on-failure:5
    depends_on:
      immich-db:
        condition: service_started
        

=================================================REMOTE==========================================


services:
  immich-microservices:
    container_name: immich_microservices
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities:
                - gpu
                - compute
                - video
    volumes:
      - immich-uploads:/usr/src/app/upload:rw
      - immich-external-library:/usr/src/app/external/me:rw
      - /etc/localtime:/etc/localtime:ro
    env_file:
      - stack.env
    environment:
      - PUID=1026
      - PGID=100
    restart: always
  
  immich-machine-learning:
    container_name: immich_machine_learning
    image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release}-cuda
    ports:
      - 3003:3003
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities:
                - gpu
    env_file:
      - stack.env
    environment:
      - PUID=1026
      - PGID=100
    volumes:
      - 'C:\\Users\\pc\\model-cache:/cache:rw'
    restart: always

volumes:
  immich-uploads:
    driver_opts:
        type: "nfs"
        o: "addr=IP,nfsvers=3"
        device: ":volume1/docker/immich/upload"
  immich-external-library:
    driver_opts:
        type: "nfs"
        o: "addr=IP,nfsvers=3"
        device: ":volume1/data/..."

Your .env content

NODE_ENV=production
TZ=Europe/Warsaw
DB_HOSTNAME=immich-db
DB_USERNAME=immichuser
DB_DATABASE_NAME=immich
REDIS_HOSTNAME=immich-redis
LOG_LEVEL=log
DISABLE_REVERSE-GEOCODING=false
REVERSE_GEOCODING_PRECISION=3
PUBLIC_LOGIN_PAGE_MESSAGE=-
IMMICH_WEB_URL=http://immich-web:3000
IMMICH_SERVER_URL=http://immich-server:3001
IMMICH_MACHINE_LEARNING_URL=http://immich-machine-learning:3003
IMMICH_WORKERS_INCLUDE=api




============================================HOST=====================================================

DB_HOSTNAME=IP
REDIS_HOSTNAME=IP
DB_USERNAME=immichuser
DB_DATABASE_NAME=immich
DB_PORT=5433
TZ=Europe/Warsaw
IMMICH_VERSION=release
REDIS_PORT=6379
IMMICH_WORKERS_EXCLUDE=api

Reproduction steps

1.Search in the smart search
2.You will get an error
3.
...

Relevant log output

[Nest] 32  - 02/01/2025, 5:10:17 PM    WARN [Api:MachineLearningRepository~bhi5w349] Machine learning request to "http://192.168.50.217:3003" failed: fetch failed

[Nest] 32  - 02/01/2025, 5:10:28 PM    WARN [Api:MachineLearningRepository~bhi5w349] Machine learning request to "http://immich-machine-learning:3003" failed: fetch failed

[Nest] 32  - 02/01/2025, 5:10:28 PM   ERROR [Api:ErrorInterceptor~bhi5w349] Unknown error: Error: Machine learning request '{"clip":{"textual":{"modelName":"ViT-B-32__openai"}}}' failed for all URLs

Error: Machine learning request '{"clip":{"textual":{"modelName":"ViT-B-32__openai"}}}' failed for all URLs

    at MachineLearningRepository.predict (/usr/src/app/dist/repositories/machine-learning.repository.js:38:15)

    at process.processTicksAndRejections (node:internal/process/task_queues:105:5)

    at async MachineLearningRepository.encodeText (/usr/src/app/dist/repositories/machine-learning.repository.js:61:26)

    at async SearchService.searchSmart (/usr/src/app/dist/services/search.service.js:60:27)

Additional information

One important thing to mention, after rerunning Smart Search and Face Detection jobs I can see remote GPU in use in task manager and PC is reachable from NAS.