You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The tt-mlir runtime now leaves every tensor on device and can accept inputs already on device. Use that to avoid unnecessary movements of tensors to and from the device.
For example, when running inference of a model in a loop, we are currently sending the weights/constant tensors on each iteration. This is obviosly not ideal.
Another example would be in training scenario. We can now reuse outputs of, let's say, forward() program on device as inputs to the backward() program. Without needing to move outputs of forward() to the host and then running the backward() program which will move them back to the device.
The text was updated successfully, but these errors were encountered:
The
tt-mlir
runtime now leaves every tensor on device and can accept inputs already on device. Use that to avoid unnecessary movements of tensors to and from the device.For example, when running inference of a model in a loop, we are currently sending the weights/constant tensors on each iteration. This is obviosly not ideal.
Another example would be in training scenario. We can now reuse outputs of, let's say,
forward()
program on device as inputs to thebackward()
program. Without needing to move outputs offorward()
to the host and then running thebackward()
program which will move them back to the device.The text was updated successfully, but these errors were encountered: