usb_host (imx8mp_iot_gate): hang on USB-client restart #5500
Labels
Comments
trimpim
added a commit
to trimpim/genode
that referenced
this issue
Apr 1, 2025
This test restarts the usb_hid client in a loop. Issue genodelabs#5500
trimpim
added a commit
to trimpim/genode-imx
that referenced
this issue
Apr 1, 2025
|
Update: The last few log lines are: After a disconnect / connect cycle of the keyboard the run script continued again. |
|
Please correct me if I'm wrong but you wrote: nothing further happens, usb_hid will not be stopped / started again resp. usb_hid also wasn't stopped / started any more which reads for me that the restart mechanism stopped working. |
|
@chelmuth yes, it looks like, the mechanism of starting / stopping |
|
Update: on the UP-Xtreme there is also a resource request (after about 10 hours): |
trimpim
added a commit
to trimpim/genode
that referenced
this issue
Apr 14, 2025
This test restarts the usb_hid client in a loop. Issue genodelabs#5500
trimpim
added a commit
to trimpim/genode-imx
that referenced
this issue
Apr 14, 2025
alex-ab
pushed a commit
to alex-ab/genode
that referenced
this issue
Apr 22, 2025
This test restarts the usb_hid client in a loop. Issue genodelabs#5500
alex-ab
added a commit
to alex-ab/genode
that referenced
this issue
Apr 22, 2025
alex-ab
added a commit
to alex-ab/genode
that referenced
this issue
Apr 22, 2025
The "[init -> usb]" component is actually an init instance in your usb test run script, which should not go out of caps in this scenario, as far as I can say. I added two commits on my issue_5500 branch, which reports the usage of caps/ram of "[init -> usb]" and its children. @trimpim: maybe you can add and see whether the caps of the init instance decreases over time. I will let it run here also, by now the numbers don't change (but it did not run 10 hours by now), output is alike: |
|
Sorry, the child "usb" of the "init -> usb" requests the resources, so well, looking now at the resource usage of the usb host driver in the init state report. |
alex-ab
pushed a commit
to alex-ab/genode-imx
that referenced
this issue
Apr 23, 2025
alex-ab
added a commit
to alex-ab/genode-imx
that referenced
this issue
Apr 23, 2025
alex-ab
added a commit
to alex-ab/genode-imx
that referenced
this issue
Apr 23, 2025
alex-ab
added a commit
to alex-ab/genode-imx
that referenced
this issue
Apr 23, 2025
alex-ab
pushed a commit
to alex-ab/genode
that referenced
this issue
Apr 23, 2025
alex-ab
added a commit
to alex-ab/genode
that referenced
this issue
Apr 23, 2025
|
I can reproduce the resource request. It turns out that memory is leaked in the usb_host driver, which is related to the task_struct not freed up. Since ever more memory is allocated, also caps are used (for the dataspace) and so the usb_host driver runs out of resources after some while. In issue #4575 there was already a commit to handle the free up, so I revived it (thx @cnuke for the hint). I added two commits to my issue_5500 and imx/issue_5500 branch. With those commits the resource usage of the usb_host driver seems to be stable. |
|
@trimpim: regarding the hang, with the resource fixes, I can't reproduce it by now on the imx8 iot gateway. May you please re-try ? If it persists, the potential difference are of course your usb hid device and my here, which are not identical. Can you please try to use, just for testing, another usb input device attached, to see whether things change ? |
|
@alex-ab I will test your changes and come back with the results. |
|
@alex-ab I have now tested with two different keyboards I do have here. With one I still see the problem, with the other I do not observe it. keyboard that ends up "hanging": Keyboard that keeps working: Still the system recovers when I disconnect the keyboard. |
alex-ab
added a commit
to alex-ab/genode
that referenced
this issue
Apr 23, 2025
alex-ab
added a commit
to alex-ab/genode
that referenced
this issue
Apr 23, 2025
alex-ab
pushed a commit
to alex-ab/genode
that referenced
this issue
Apr 23, 2025
alex-ab
added a commit
to alex-ab/genode
that referenced
this issue
Apr 23, 2025
alex-ab
added a commit
to alex-ab/genode
that referenced
this issue
Apr 23, 2025
alex-ab
added a commit
to alex-ab/genode
that referenced
this issue
Apr 23, 2025
|
@chelmuth that is a good idea. I created the small test program (see attached files). When I run this program I can use the keyboard as normal in a text editor. When the keyboard is attached, the key presses appear in the editor, when the keyboard is detached not. what the program prints about the device: The output from #include <iostream>
#include <numbers>
#include <ostream>
#include <unistd.h>
#include <./libusb-1.0/libusb.h>
void print_info(libusb_device *dev);
int main()
{
// init libusb context
libusb_context *context = NULL;
libusb_init(&context);
libusb_set_option(context, LIBUSB_OPTION_LOG_LEVEL, LIBUSB_LOG_LEVEL_WARNING);
// get list of device
int cnt = 0;
libusb_device **dev_list = NULL;
cnt = libusb_get_device_list(context, &dev_list);
std::cout << "Number of devices : " << cnt << std::endl;
uint16_t const vendor { 0x1c11 };
uint16_t const product { 0xb04d };
libusb_device *keyboard { nullptr };
for (int i = 0; i < cnt; ++i) {
libusb_device *dev { dev_list[i] };
libusb_device_descriptor dev_desc;
libusb_get_device_descriptor(dev, &dev_desc);
if ((dev_desc.idVendor == vendor) &&
(dev_desc.idProduct == product)) {
keyboard = dev_list[i];
}
}
print_info(keyboard);
libusb_device_handle *keyboard_handle;
if (libusb_open(keyboard, &keyboard_handle) == 1) {
std::cerr << "error claiming device";
exit(1);
}
while (true) {
std::cout << "detach kernel driver" << std::endl;
libusb_detach_kernel_driver(keyboard_handle, 0);
libusb_detach_kernel_driver(keyboard_handle, 1);
libusb_detach_kernel_driver(keyboard_handle, 2);
libusb_detach_kernel_driver(keyboard_handle, 3);
libusb_detach_kernel_driver(keyboard_handle, 4);
libusb_detach_kernel_driver(keyboard_handle, 5);
libusb_detach_kernel_driver(keyboard_handle, 6);
sleep(10);
libusb_attach_kernel_driver(keyboard_handle, 0);
libusb_attach_kernel_driver(keyboard_handle, 1);
libusb_attach_kernel_driver(keyboard_handle, 2);
libusb_attach_kernel_driver(keyboard_handle, 3);
libusb_attach_kernel_driver(keyboard_handle, 4);
libusb_attach_kernel_driver(keyboard_handle, 5);
libusb_attach_kernel_driver(keyboard_handle, 6);
std::cout << "attach kernel driver" << std::endl;
sleep(10);
}
libusb_close(keyboard_handle);
libusb_free_device_list(dev_list, 1);
libusb_exit(context);
return 0;
}
void print_info(libusb_device *dev)
{
libusb_device_descriptor dev_desc;
libusb_get_device_descriptor(dev, &dev_desc);
std::cout << std::endl
<< std::endl;
std::cout << "Device Class : " << (int)dev_desc.bDeviceClass << std::endl;
std::cout << "ProductID : " << std::hex << dev_desc.idProduct << std::endl;
std::cout << "VendorID : " << dev_desc.idVendor << std::endl;
std::cout << "-------Configs-------" << std::endl;
libusb_config_descriptor *config = NULL;
libusb_interface inter;
libusb_interface_descriptor inter_desc;
libusb_endpoint_descriptor ep_desc;
libusb_get_active_config_descriptor(dev, &config);
if (config != NULL)
{
for (uint8_t i = 0; i < config->bNumInterfaces; ++i)
{
std::cout << " Config" << (int)i << std::endl;
inter = config->interface[i];
for (int j = 0; j < inter.num_altsetting; ++j)
{
inter_desc = inter.altsetting[j];
std::cout << " Interface Num : "
<< (int)inter_desc.bInterfaceNumber << std::endl;
std::cout << " Class : "
<< std::hex << (int)inter_desc.bInterfaceClass << std::endl;
for (int k = 0; k < inter_desc.bNumEndpoints; ++k)
{
ep_desc = inter_desc.endpoint[k];
std::cout << " Endpoint address : "
<< std::dec << (int)ep_desc.bEndpointAddress << std::endl;
std::cout << " Max Packet size : "
<< (int)ep_desc.wMaxPacketSize << std::endl;
std::cout << " Interval for polling : "
<< (int)ep_desc.bInterval << std::endl;
}
}
}
libusb_free_config_descriptor(config);
}
}cmake_minimum_required(VERSION 3.10)
project(card_server VERSION 0.1)
set(CMAKE_CXX_STANDARD 23)
set(CMAKE_CXX_STANDARD_REQUIRED True)
add_compile_options(-Wall -Wextra -Wpedantic -Werror)
include_directories(../include)
add_executable(
libusb_test
main.cc
)
find_library(libudev /usr/lib/x86_64-linux-gnu/)
target_link_libraries(
libusb_test PRIVATE
/usr/lib/x86_64-linux-gnu/libusb-1.0.so
) |
|
It is interesting, that this keyboard provides a |
alex-ab
added a commit
to alex-ab/genode
that referenced
this issue
Apr 25, 2025
|
@trimpim: please re-run part V |
|
@alex-ab part V results |
|
@trimpim: the upload of the log file seems to be failed |
|
@alex-ab I will place the logs to https://webmail2.trimpim.ch/gapfruit/trimpim/ from now on. |
|
@trimpim: thanks! According to the current debugging state, the poll_device task never leaves, in the bad case, the repos/dde_linux/src/lib/lx_emul/usb.c |
alex-ab
added a commit
to alex-ab/genode
that referenced
this issue
Apr 25, 2025
|
@trimpim: please re-run with part VI |
|
@alex-ab uploaded the file. |
alex-ab
added a commit
to alex-ab/genode
that referenced
this issue
Apr 25, 2025
|
@trimpim: thanks, please re-run part VII - seems handle_control_request() does not return |
|
@alex-ab uploaded the file. |
alex-ab
added a commit
to alex-ab/genode
that referenced
this issue
Apr 25, 2025
|
@trimpim: please re-run part VIII, which just shows the timeout value which is hopefully not 0 (wait forever) The code doesn't get back from usb_control_msg in repos/dde_linux/src/lib/lx_emul/usb.c: |
|
@alex-ab uploaded part VIII result. As a side info, I'll be off after lunch. |
|
@trimpim: thanks, well, the timeout value is 0 (wait forever). From my side I will stop debugging, I think we have enough information to discuss/think about it next week. |
|
@alex-ab thanks for you efforts, a good weekend to you all. |
alex-ab
added a commit
to alex-ab/genode
that referenced
this issue
Apr 28, 2025
|
@trimpim: may you please try the new commit, which restricts the client provided timeouts to some range (arbitrarily chosen by now). How does this change the behaviour in your case ? You may also play with removing the _warn_once_timeout check to see whether other timeout values unequal zero are seen. |
|
@alex-ab I uploaded At around 89 Seconds |
|
@trimpim: thanks, does the test loop continue afterwards ? |
|
@alex-ab yes, once |
alex-ab
pushed a commit
to alex-ab/genode
that referenced
this issue
Apr 29, 2025
This test restarts the usb_hid client in a loop. Issue genodelabs#5500
alex-ab
pushed a commit
to alex-ab/genode
that referenced
this issue
Apr 29, 2025
alex-ab
added a commit
to alex-ab/genode
that referenced
this issue
Apr 29, 2025
alex-ab
added a commit
to alex-ab/genode
that referenced
this issue
Apr 29, 2025
|
I pushed a clean issue_5500 branch with the commits for usage and/or testing on staging after the Sculpt 25.04 release. I moved the old debug branch to issue_5500_debug. |
|
@alex-ab thanks, I will integrate these soon. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
When restarting a misbehaving libc application and the USB device driver it uses, there was
no communication anymore until the device got disconnected and connected again.
So I created the test run script
usb_host_client_restart.run, whereusb_hidis started / stopped every 8 seconds. Using this,I made the following observations:
usb_hidis started and stopped as expectedusb_hidis running,event_dumplogs two events per key-press,and
usb_hidis started and stopped as expectedusb_hidis stopped, everything works asexpected after
usb_hidis restarted.usb_hidis stopped, the following happens:usb_hidstarts again.event_dumpprints the first of the two eventsusb_hidwill not be stopped / started againusb_hostissues:If I execute the test on a PC target (UP-Xtreme) or
quemu-x86_64the events always are reported as expected. The long term run on this hardware is still running.The text was updated successfully, but these errors were encountered: