docker-worker-1 | core.plugin.manager.exc.PluginInvokeError: PluginInvokeError: {"args":{},"error_type":"RuntimeError","message":"can't start new thread"}

[yuyeshuhuai]

The error always comes out after user uses Dify for a while.
Sometimes it happens in the phase of knowledge base indexing( based on LLM for embedding), sometimes in the phase of user communicating with LLM. I am sure that is all relevant to calling LLM.

Once this comes out, I will restart docker containers until it shows up next time.
Did you have this kind of problem? Any suggestion will be appreciated.
Logs in detail as follows:

docker-worker-1 | 2025-03-11 09:56:27.368 ERROR [Dummy-23] [indexing_runner.py:96] - consume document failed
docker-worker-1 | Traceback (most recent call last):
docker-worker-1 | File "/app/api/core/indexing_runner.py", line 73, in run
docker-worker-1 | documents = self._transform(
docker-worker-1 | ^^^^^^^^^^^^^^^^
docker-worker-1 | File "/app/api/core/indexing_runner.py", line 706, in _transform
docker-worker-1 | documents = index_processor.transform(
docker-worker-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^
docker-worker-1 | File "/app/api/core/rag/index_processor/processor/parent_child_index_processor.py", line 83, in transform
docker-worker-1 | child_nodes = self._split_child_nodes(
docker-worker-1 | ^^^^^^^^^^^^^^^^^^^^^^^^
docker-worker-1 | File "/app/api/core/rag/index_processor/processor/parent_child_index_processor.py", line 186, in _split_child_nodes
docker-worker-1 | child_documents = child_splitter.split_documents([document_node])
docker-worker-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
docker-worker-1 | File "/app/api/core/rag/splitter/text_splitter.py", line 96, in split_documents
docker-worker-1 | return self.create_documents(texts, metadatas=metadatas)
docker-worker-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
docker-worker-1 | File "/app/api/core/rag/splitter/text_splitter.py", line 81, in create_documents
docker-worker-1 | for chunk in self.split_text(text):
docker-worker-1 | ^^^^^^^^^^^^^^^^^^^^^
docker-worker-1 | File "/app/api/core/rag/splitter/fixed_text_splitter.py", line 68, in split_text
docker-worker-1 | chunks_lengths = self._length_function(chunks)
docker-worker-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
docker-worker-1 | File "/app/api/core/rag/splitter/fixed_text_splitter.py", line 38, in _token_encoder
docker-worker-1 | return embedding_model_instance.get_text_embedding_num_tokens(texts=texts)
docker-worker-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
docker-worker-1 | File "/app/api/core/model_manager.py", line 244, in get_text_embedding_num_tokens
docker-worker-1 | self._round_robin_invoke(
docker-worker-1 | File "/app/api/core/model_manager.py", line 370, in _round_robin_invoke
docker-worker-1 | return function(*args, **kwargs)
docker-worker-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^
docker-worker-1 | File "/app/api/core/model_runtime/model_providers/__base/text_embedding_model.py", line 65, in get_num_tokens
docker-worker-1 | return plugin_model_manager.get_text_embedding_num_tokens(
docker-worker-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
docker-worker-1 | File "/app/api/core/plugin/manager/model.py", line 313, in get_text_embedding_num_tokens
docker-worker-1 | for resp in response:
docker-worker-1 | ^^^^^^^^
docker-worker-1 | File "/app/api/core/plugin/manager/base.py", line 189, in _request_with_plugin_daemon_response_stream
docker-worker-1 | self._handle_plugin_daemon_error(error.error_type, error.message)
docker-worker-1 | File "/app/api/core/plugin/manager/base.py", line 221, in _handle_plugin_daemon_error
docker-worker-1 | raise PluginInvokeError(description=message)
docker-worker-1 | core.plugin.manager.exc.PluginInvokeError: PluginInvokeError: {"args":{},"error_type":"RuntimeError","message":"can't start new thread"}
docker-worker-1 | 2025-03-11 09:56:27.381 INFO [Dummy-23] [document_indexing_task.py:76] - Processed dataset: fd49a7a6-0946-4512-acbb-9f297a1538ed latency: 0.0918743850197643
docker-plugin_daemon-1 | [GIN] 2025/03/11 - 09:56:27 | 200 | 18.583202ms | 172.19.0.9 | GET "/plugin/2d64604d-dae0-4e51-9cdb-0ebebf24b427/management/models?page=1&page_size=256"