Add the ability to retry when looking for hot added device nodes

[katiewasnothere]

There are two main issues to solve when hot adding devices in the UVM:

• We have no generic way to know when a device is actually ready to use and the relevant /dev nodes have been created.
• We have no way to know how many /dev nodes to look for unless we had device and vendor specific handling in our code.

Given that our existing support for assigned devices attempts to handle devices in a very device- and vendor- generic way, there is a race condition between when we hot add devices to the UVM for the first time during container creation and when the /dev nodes have been created. We attempt to minimize this by adding retry logic for when we try to find related /dev nodes.

This does not entirely solve our problems, since there could be a case where we find only a subset of the desired /dev nodes and return. However, for our current testing and use cases 500ms seems to be enough to capture all /dev nodes. We should investigate alternatives to this approach.

[helsaawy]

nits aside, does the context used for devicePathsFromPCIPath have a timeout to prevent infinite looping?
also, rather than a long retry time, maybe we add an initial delay (~250ms or something) to allow the devices to show up?

[katiewasnothere]

nits aside, does the context used for devicePathsFromPCIPath have a timeout to prevent infinite looping? also, rather than a long retry time, maybe we add an initial delay (~250ms or something) to allow the devices to show up?

@helsaawy I'll add in an explicit deadline before we call devicePathsFromPCIPath. For the initial delay, I'm hesitant to do that just because if the dev nodes have all shown up by the time we try to find them, which does sometimes happen, we're adding an unnecessary delay, which I'm concerned the perf team would be unhappy with.

[anmaxvl]

LGTM, given other feedback is addressed.