Memory leak

[filippocarone]

hawthorne_lettera_scarlatta.epub.gz

When processing this file using the docker container, memory utilization grows continuously. Command line:

/ebook2audiobook.sh --hea
dless --ebook ../hawthorne_lettera_scarlatta.epub -
-language ita

[DrewThomasson]

If your willing I'd also love to see if this is a issue purely in the docker

Or also in the code in general

Try running the

ebook2audiobook.sh

And see if this issue also happens through that

[DrewThomasson]

Include any logs and details such also

No amount of details is too much for us 👍

[DrewThomasson]

say....

Are you running into issues with the docker image after running multiple books through it?

And its just having this increase in ram usage after each book is processed in order without wiping and restarting the docker image from the base image scratch? 🤔

[DrewThomasson]

Testing rn with this

alice_in_wonderland.txt

In the gui in stnadard settings

docker was launched with a upper cpu memory limit of 4gb with this command for my test

docker run -it -p 7860:7860 --platform=linux/amd64 --memory=4g athomasson2/ebook2audiobook:latest python app.py

edit: I am testing it with this book because there are errors with your.epub file as its seen as a dir for some reason

[DrewThomasson]

seems to be going.... well XD

it's BARELY hanging on rn but still progressing

log_so_far.txt

Update: still going strong

log_so_far2.txt

Update still going strong

log_so_far3.txt

[DevonGrandahl]

This has been happening with every novel-length book I try.

• Docker on Windows 10
• Mostly using a ~2k sentence book
• Mostly using Bryan Cranston's voice
• Mid/low-end mini PC w/ 8gb of available RAM.

It fails consistently after a few hours (getting about 4-6% of the way through) once it hits the memory threshold. Unrestricted or set to 4gb, doesn't seem to matter. Logs don't show anything interesting, just dies mid-sentence.

I've reverted to using the piper-TTS image for now, which is still awesome. I'll comment if I figure out anything useful.

[DrewThomasson]

Interesting

Yeah that's weird then

[DrewThomasson]

Even if you turn off sentence splitting?

[DrewThomasson]

@DevonGrandahl

Any chance you could show us the book you're using?

[DevonGrandahl]

Even if you turn off sentence splitting?

Trying this now, I previously misread as turn on sentence splitting.

The book is Penpal by Dathan Auerbach. I own a copy. What's the best way for me to get it to you?

[DrewThomasson]

@DevonGrandahl

Discord

Don't want anyone thinking we're distributing books to the public illegally

[DevonGrandahl]

No luck with text splitting off and a 4gb memory limit. It crashed this time at 3.8%.

[DrewThomasson]

Anyone who has issues with this keep posting ✨🫶🏻

this is just as a helpful list of legacy things that might work for you in the meantime

Other LEGACY versions of ebook2audiobook that might not have this issue (I'm not updating them tho as these will be integrated eventually)

Legacy Ebook2Audiobook v1.0

Legacy Ebook2AudiobookpiperTTS

Legacy Ebook2AudiobookStyleTTS

Legacy Ebook2AudiobookEspeak

[ROBERT-MCDOWELL]

@DevonGrandahl
Could you provide:

• the next text after the last sentence converted (see the terminal when you run eb2ab)
• the CPU and GPU of your PC

[DevonGrandahl]

Sure! It doesn't fail at the same spot every time.

The last lines in the logs:

2025-01-02 21:20:49 94/2396 Sentence: Small towns lack many of the luxuries of larger towns or cities; what few stores there are close down early, 
2025-01-02 21:20:49 
2025-01-02 21:21:47 
Processing 3.88%: : 94/2396

The next line is:

traveling events don’t stop there because they probably missed your small dot on the map, and there aren’t many police or hospitals at your disposal.

The PC is a ProDesk mini PC I use as a server.
Processor: Intel(R) Core(TM) i7-6700T CPU @ 2.80GHz, 2808 Mhz, 4 Core(s), 8 Logical Processor(s)
RAM: 16gb (8gb available to Docker)
GPU: Intel HD Graphics 530 (Integrated)

[ROBERT-MCDOWELL]

ha you said it doesn't fail at the same spot right? everytime it's random?

[DevonGrandahl]

Yep, seems to be random. It's never made it past ~8%.

Trying the legacy v1.0 image now.

Update: the v1.0 image just passed this sentence in maybe 1/3rd the time. Using Attenborough and no memory limit.

Update: Aaaaand it crashed. Went much faster, but still crashed when memory topped out.

[ROBERT-MCDOWELL]

so I don't think it's related to ebook2audiobook, but more how your OS is managing the docker.
it can be also a RAM failure...

[DevonGrandahl]

RAM failure would be a strange thing to happen to multiple people at the same time trying to run the app. Plenty of apps run inside docker with no issue, so I'm also not sure about the Windows/Docker management issue. Could be, though!

I tried mounting Docker volumes for the tmp & audiobook directories with no luck, but I have a hunch I did that wrong. Will try again, since that could help with RAM usage.

[ROBERT-MCDOWELL]

failure is maybe not the righ word, but more how the OS is managing the docker RAM....
maybe there is also something else out of Docker causing trouble to your docker system.
try to reboot with the minimum services using RAM and virtual RAM and try again to see if it's better.

[filippocarone]

The memory leak happens also when running in native mode. I'll try next with text splitting off.

[ROBERT-MCDOWELL]

I don't think it's related to enable_text_splitting but more a coqui-tts issue... if you say that around 8% it fails, so there is something somewhere where the memory is not freed....

[filippocarone]

Attaching logs and a couple of screenshots of memory utilization (see the clock at the top left of the screen).

log.txt

[ROBERT-MCDOWELL]

ok there is already something wrong with the processes running. In any case it should be 3 or 4 processes, unless you are several user on it so it will explain your memory "leak" which is not a leak but just a need of more RAM since more users...

[filippocarone]

Those different PIDs are actually threads of the same process. Only one instance is running, by just one user - this is a desktop computer. What I wanted to show with the 2 screenshots is that in just 7 minutes RAM usage has increased by ca. 2.6 GB (from 5.3 to 7.9).

[ROBERT-MCDOWELL]

it should'n be threads since ebook2audiobook is running in multiprocessing and it's NOT allowing threading. So a race occurs, which can explain the increase of your RAM. now it needs to undertsand why threads are running....
are you on windows 11?

[filippocarone]

I'm running on Linux, Ubuntu 24.10.

[ROBERT-MCDOWELL]

ok try to run a conversion, then check the PID of each thread, and try to kill all but one with kill -SIGTERM , then provide the log

[ROBERT-MCDOWELL]

@DevonGrandahl Keep in mind that a very few users have this problem. so if it was a major issue the entire project won't be useable. the issue is elsewhere for sure. are you saying it's the same in native mode? btw I just saw you are on windows 10, not 11 right?

[DevonGrandahl]

Yep, I'm seeing the same issue in native mode. The Google Collab is also in native mode, I think.

Correct, Windows 10.

[ROBERT-MCDOWELL]

windows 10 could be the issue.... I also saw on coqui-tts forums users having the same issue on windows 10 even with 16GB RAM.
but the original coqui-tts repo closed it as "won't fix'.. however some found a way to stabilize the memory by cutting sentence to max 100 chars and a new tts instance on each (not sure about this one). maybe with gc.collect() it can avoid to create a new tts instance, however I can code a new condition to split more the sentences in case of cpu mode. The thing I still don't catch is my laptop test is 17 years old core 2 duo with 4GB RAM, and some tests I did were like around 30,000 sentences on CPU, and after 3 days the RAM was the same, at the max + virtual memory but no crash, in native mode. windows 11

[filippocarone]

I tried to create a new TTS object at every sentence and then execute a gc.collect(), but memory grew even faster. I also tried to set to None the params and session objects to see if there was something holding references to other objects, but also that did not reduce memory utilization.

[ROBERT-MCDOWELL]

read carefully my comment above please

[filippocarone]

On the debian 12.8 VM in Virtualbox memory is not growing (it grows by approx 1MB per sentence instead of 10s or 100s of MB per sentence like in Ubuntu 24.10). There could be something related to the kernel/OS which influences how memory is managed resulting in a leak. This is weird.
Anyway, I'll try a full run and see how it behaves until the end.

[ROBERT-MCDOWELL]

any VM is still dependent on the OS host so windows10. then docker is doing the mapping and for sure the map behavior differs from each VM to other

[DevonGrandahl]

Just tried this on a Windows 11 gaming machine and the memory climb seems much more reasonable. Can't leave this running to see if it ever crashes, but it seems like it would probably work.

[filippocarone]

A virtualbox VM is different from a docker container in many respects and also from a memory management standpoint. My host is not windows10, but ubuntu 24.10. Anyway it reached 8GB of RAM after 417 sentences, and the process was killed by the OS as the VM has 9 GB of RAM allocated. So it grew, but slower than on ubuntu 24.10.

[ROBERT-MCDOWELL]

run a TTS A.I. on a VM is not reasonable btw.... it's ok for test, but production not.

[filippocarone]

This whole issue may be related to the comments made here: https://stackoverflow.com/questions/15455048/releasing-memory-in-python
It would be interesting to know if there are Linux systems where the memory over allocation doesn't happen and understand what influences that behavior.

[ROBERT-MCDOWELL]

Linux has many way to share and manage the RAM. depend also how the application (torch in our case) is programmed.

[ROBERT-MCDOWELL]

Did you try in headless mode to see if it's the same?

[filippocarone]

GUI and CLI behave the same wrt RAM usage.

[DevonGrandahl]

Confirmed Win11 does not have this issue. Used Native (non-docker) via GUI, GPU mode, Penpal epub, and Bryan Cranston's voice. Thanks for the help!

Very strange that Win10 and some linux distros are struggling.

[ROBERT-MCDOWELL]

@DevonGrandahl it's rare when I'm wrong (just kidding) :)
anyhow, my intuition is you can maybe trick how the virtual memory and such related to it manage the shared stuff and so on.... on reddit you will maybe have an answer.
About linux, the kernel has a lot of various RAM and shared memory options that maybe one option can break torch.... maybe some deep research on the net will give an answer too.

[maxkuminov]

I am experiencing this same issue with the docker container. I have tried the image from DockerHub and building one from source with the same result. If I give the container 4GB of memory it fails after a while. Without the limit it will use all of the system's RAM.

I think the important piece of information is that I am using the CPU mode since I am running this on an older PC. This issue might be specific to CPU, and not GPU modes.

[filippocarone]

Are you running docker on Linux or Windows? And which exact version?

[maxkuminov]

Are you running docker on Linux or Windows? And which exact version?
I am running Linux Mint 21.3 on an older system with no GPU

docker version
Client:
Version: 24.0.7
API version: 1.43
Go version: go1.21.1
Git commit: 24.0.7-0ubuntu2~22.04.1
Built: Wed Mar 13 20:23:54 2024
OS/Arch: linux/amd64
Context: default

Server:
Engine:
Version: 24.0.7
API version: 1.43 (minimum version 1.12)
Go version: go1.21.1
Git commit: 24.0.7-0ubuntu222.04.1
Built: Wed Mar 13 20:23:54 2024
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.7.12
GitCommit:
runc:
Version: 1.1.12-0ubuntu222.04.1
GitCommit:
docker-init:
Version: 0.19.0
GitCommit:

System:
Kernel: 5.15.0-130-generic x86_64 bits: 64 compiler: gcc v: 11.4.0 Desktop: Cinnamon 6.0.4
tk: GTK 3.24.33 wm: muffin vt: 7 dm: LightDM 1.30.0 Distro: Linux Mint 21.3 Virginia
base: Ubuntu 22.04 jammy
Machine:
Type: Desktop Mobo: MSI model: B85-G41 PC Mate(MS-7850) v: 1.0 serial:
UEFI: American Megatrends v: 2.3 date: 09/18/2013
CPU:
Info: quad core model: Intel Core i5-4570 bits: 64 type: MCP smt: arch: Haswell
rev: 3 cache: L1: 256 KiB L2: 1024 KiB L3: 6 MiB
Speed (MHz): avg: 3565 high: 3596 min/max: 800/3600 cores: 1: 3535 2: 3558 3: 3596 4: 3571
bogomips: 25602
Flags: avx avx2 ht lm nx pae sse sse2 sse3 sse4_1 sse4_2 ssse3 vmx
Graphics:
Device-1: Intel Xeon E3-1200 v3/4th Gen Core Processor Integrated Graphics
vendor: Micro-Star MSI driver: i915 v: kernel ports: active: HDMI-A-2 empty: HDMI-A-1,VGA-1
bus-ID: 00:02.0 chip-ID: 8086:0412 class-ID: 0300
Display: x11 server: X.Org v: 1.21.1.4 driver: X: loaded: modesetting unloaded: fbdev,vesa
gpu: i915 display-ID: :0 screens: 1
Screen-1: 0 s-res: 1920x1200 s-dpi: 96 s-size: 508x317mm (20.0x12.5") s-diag: 599mm (23.6")
Monitor-1: HDMI-2 mapped: HDMI-A-2 model: Dell U2415 serial: res: 1920x1200 hz: 60
dpi: 94 size: 518x324mm (20.4x12.8") diag: 611mm (24.1") modes: max: 1920x1200 min: 720x400
OpenGL: renderer: Mesa Intel HD Graphics 4600 (HSW GT2) v: 4.6 Mesa 23.2.1-1ubuntu3.1~22.04.3
direct render: Yes
Audio:
Device-1: Intel Xeon E3-1200 v3/4th Gen Core Processor HD Audio vendor: Micro-Star MSI
driver: snd_hda_intel v: kernel bus-ID: 00:03.0 chip-ID: 8086:0c0c class-ID: 0403
Device-2: Intel 8 Series/C220 Series High Definition Audio vendor: Micro-Star MSI
driver: snd_hda_intel v: kernel bus-ID: 00:1b.0 chip-ID: 8086:8c20 class-ID: 0403
Device-3: VIA VT1720/24 [Envy24PT/HT] PCI Multi-Channel Audio driver: snd_ice1724 v: kernel
bus-ID: 04:01.0 chip-ID: 1412:1724 class-ID: 0401
Sound Server-1: ALSA v: k5.15.0-130-generic running: yes
Sound Server-2: PulseAudio v: 15.99.1 running: yes
Sound Server-3: PipeWire v: 0.3.48 running: yes
Network:
Device-1: Realtek RTL8111/8168/8411 PCI Express Gigabit Ethernet vendor: Micro-Star MSI
driver: r8169 v: kernel pcie: speed: 2.5 GT/s lanes: 1 port: e000 bus-ID: 02:00.0
chip-ID: 10ec:8168 class-ID: 0200
IF: enp2s0 state: up speed: 1000 Mbps duplex: full mac:
IF-ID-1: br-0f60959ee27c state: down mac:
IF-ID-2: br-4da904ac038f state: down mac:
IF-ID-3: br-86141811424d state: down mac:
IF-ID-4: br-f381557bcdde state: down mac:
IF-ID-5: docker0 state: down mac:
Drives:
Local Storage: total: 4.55 TiB used: 1.36 TiB (29.8%)
ID-1: /dev/sda vendor: Seagate model: ST4000DM004-2CV104 size: 3.64 TiB speed: 6.0 Gb/s
type: HDD rpm: 5425 serial: rev: 0001 scheme: GPT
ID-2: /dev/sdb vendor: Samsung model: SSD 860 EVO 1TB size: 931.51 GiB speed: 6.0 Gb/s
type: SSD serial: rev: 4B6Q scheme: GPT
Partition:
ID-1: / size: 915.32 GiB used: 838.53 GiB (91.6%) fs: ext4 dev: /dev/sdb2
ID-2: /boot/efi size: 511 MiB used: 6.1 MiB (1.2%) fs: vfat dev: /dev/sdb1
Swap:
ID-1: swap-1 type: file size: 2 GiB used: 0 KiB (0.0%) priority: -2 file: /swapfile
USB:
Hub-1: 1-0:1 info: Full speed or root hub ports: 2 rev: 2.0 speed: 480 Mb/s chip-ID: 1d6b:0002
class-ID: 0900
Hub-2: 1-1:2 info: Intel Integrated Rate Matching Hub ports: 6 rev: 2.0 speed: 480 Mb/s
chip-ID: 8087:8008 class-ID: 0900
Hub-3: 2-0:1 info: Full speed or root hub ports: 2 rev: 2.0 speed: 480 Mb/s chip-ID: 1d6b:0002
class-ID: 0900
Hub-4: 2-1:2 info: Intel Integrated Rate Matching Hub ports: 6 rev: 2.0 speed: 480 Mb/s
chip-ID: 8087:8000 class-ID: 0900
Hub-5: 3-0:1 info: Hi-speed hub with single TT ports: 12 rev: 2.0 speed: 480 Mb/s
chip-ID: 1d6b:0002 class-ID: 0900
Device-1: 3-3:2 info: SCM Microsystems SCR331-LC1 / SCR3310 SmartCard Reader type: Smart Card
driver: N/A interfaces: 1 rev: 2.0 speed: 12 Mb/s power: 76mA chip-ID: 04e6:5116 class-ID: 0b00
serial:
Hub-6: 3-4:3 info: ASIX AX68002 KVM Switch SoC ports: 7 rev: 1.0 speed: 12 Mb/s power: 100mA
chip-ID: 0b95:6802 class-ID: 0900
Device-1: 3-4.1:5 info: Holtek Keyboard type: Keyboard,Mouse driver: hid-generic,usbhid
interfaces: 2 rev: 1.1 speed: 12 Mb/s power: 100mA chip-ID: 04d9:2323 class-ID: 0301
Device-2: 3-4.2:6 info: Logitech Marble Mouse (4-button) type: Mouse
driver: hid-generic,usbhid interfaces: 1 rev: 1.1 speed: 12 Mb/s power: 50mA chip-ID: 046d:c408
class-ID: 0301
Device-3: 3-10:4 info: Brother Industries DCP-L2540DW type: Printer driver: usblp
interfaces: 2 rev: 2.0 speed: 480 Mb/s power: 2mA chip-ID: 04f9:0328 class-ID: 0701
serial:
Hub-7: 4-0:1 info: Super-speed hub ports: 6 rev: 3.0 speed: 5 Gb/s chip-ID: 1d6b:0003
class-ID: 0900
Sensors:
System Temperatures: cpu: 29.8 C mobo: 27.8 C
Fan Speeds (RPM): N/A

[ROBERT-MCDOWELL]

I am experiencing this same issue with the docker container. I have tried the image from DockerHub and building one from source with the same result. If I give the container 4GB of memory it fails after a while. Without the limit it will use all of the system's RAM.

I think the important piece of information is that I am using the CPU mode since I am running this on an older PC. This issue might be specific to CPU, and not GPU modes.

no it has nothing to do with CPU

[DevonGrandahl]

After trying some longer books (4k sentences) this is happening on my Win11 PC too. It gets ~1.5k sentences in before RAM is full and it crashes.
Book length: 4k sentences (Horror Movie by Paul Tremblay, IIRC)
Voice: Morgan Freeman
PC: Windows 11 Laptop (LG Gram)
Mode: GPU via Gradio UI, Native (not Docker)
GPU: NVIDIA RTX3050 4GB
RAM: 16GB

Logs are gone, but I do plan to try again with sentence splitting off, as I saw that suggestion earlier. I'll try to capture logs on that run.

[ROBERT-MCDOWELL]

text splitting is fixed for all languages on the next update with a completely rebased function.
the option text_splitting fo xtts seems not really efficient btw. on or off, no significants differences.
but, are you using GPU or CPU?
logs are gone? how if you have the terminal open so just copy and a paste or take a screenshot with your smartphone if it's fully blocked

[LexiconCode]

A wise move here is to build some enhanced logging around this particular issue. In addition, allowing a user to submit logs with ease via the interface. Some thought would have to be given where it's submitted too and user privacy.

A simple method is simply to zip logs and then place them on the user's desktop where they can drag and drop to an issue.

[ROBERT-MCDOWELL]

@LexiconCode I'm working on it. parallely maybe an automatic report via url -> email.
a console terminal in gradio is on rails too, not sure it will be ready for the next version though.

[Mr-Jiggledaddy]

Just had this happen to me as well, on windows 10 running native (not docker).

Actually wasn't even processing a book, just had the program open idling waiting to start the next book, and it randomly started filling my ram (quite quickly, filled 32gb of ram in <10minutes).

Closed my browser first and it cleared the excess memory usage, even with the cmd window still running. Still stable after another hour idling with no browser open.

Not sure if there are any logs I could pull for that, but if there is let me know and I'd be happy to upload them.

[ROBERT-MCDOWELL]

@Mr-Jiggledaddy weird since some users reported here the same issue in headless mode....