Perception Autonomy Cleanup #450
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
TomaszTB
commented
Nov 7, 2024
| * @param newPixelFormat the PixelFormat of the new image. | ||
| * @return A new {@link RawImage} with the same intrinsics & metadata, but with a different image. | ||
| */ | ||
| public RawImage replaceImage(Mat newCpuImageMat, PixelFormat newPixelFormat) |
There was a problem hiding this comment.
Realized this was missing
| @@ -222,22 +246,12 @@ public Instant getAcquisitionTime() | |||
|
|
|||
| public int getWidth() | |||
| { | |||
| if (hasCpuImage()) | |||
There was a problem hiding this comment.
These checks became unnecessary when CameraIntrinsics were added
| * @param depthImage 16UC1 depth image, used to get points of detected objects | ||
| */ | ||
| public void runYOLODetection(YOLOv8ObjectDetector yoloDetector, RawImage colorImage, RawImage depthImage) | ||
| { | ||
| if (yoloDetector.isReady() && !yoloExecutorService.isShutdown()) | ||
| { | ||
| // Acquire the images | ||
| if (colorImage.get() == null || depthImage.get() == null) |
There was a problem hiding this comment.
Moved calls to get here so it guarantees you "get" the images before they're deallocated.
| if (yoloDetectionResults.containsKey(lastRunDetectorIndex)) | ||
| yoloDetectionResults.remove(lastRunDetectorIndex).destroy(); | ||
| yoloDetectionResults.put(lastRunDetectorIndex, yoloResults); | ||
| synchronized (yoloDetectionResults) |
There was a problem hiding this comment.
I think the detection results may have been getting deallocated while the annotated image thread was using them, so I put think synchronized block for now.
Really, this class needs an overhaul.
| @@ -23,7 +23,7 @@ public class YOLOv8DetectionResults | |||
| private final MatVector outputBlobs; | |||
| private final RawImage detectionImage; | |||
| private final FloatIndexer outputMasksIndexer; | |||
| private final Map<YOLOv8DetectionOutput, Mat> objectMasks = new HashMap<>(); | |||
| private final Map<YOLOv8DetectionOutput, RawImage> objectMasks = new HashMap<>(); | |||
There was a problem hiding this comment.
Using RawImage instead of Mat just to be extra safe about memory allocation/deallocation here
| @@ -127,19 +127,22 @@ public boolean encodeNextFrame(Consumer<AVPacket> packetConsumer) | |||
| return error != AVERROR_EOF(); | |||
| } | |||
|
|
|||
| public boolean flush(Consumer<AVPacket> packetConsumer) | |||
There was a problem hiding this comment.
Turns out we should be flushing out the encoders once all the frames are sent (a few packets might've stayed in the encoder, waiting to be grouped together)
| @@ -139,6 +136,10 @@ private String getDeviceInfo(rs2_device rs2Device, int deviceInfoEnum) | |||
|
|
|||
| private int updateDeviceCount() | |||
| { | |||
| realsense2.rs2_delete_device_list(devices); | |||
| devices = realsense2.rs2_query_devices(context, error); | |||
There was a problem hiding this comment.
Feels like this is correct (performing another query when we want to update the device count). Seems to have solved an issue I was having
| import java.time.Instant; | ||
| import java.util.function.Supplier; | ||
|
|
||
| public class RealsenseImageSensor extends ImageSensor |
There was a problem hiding this comment.
Tempted to name it RealSenseImageSensor to be consistent with the brand name
TomaszTB
force-pushed
the
feature/perception-autonomy-cleanup-2
branch
2 times, most recently
from
79a1355 to
188efc7
Compare
TomaszTB
force-pushed
the
feature/perception-autonomy-cleanup-2
branch
from
188efc7 to
0f4c80a
Compare
TomaszTB
force-pushed
the
feature/perception-autonomy-cleanup-2
branch
3 times, most recently
from
4eb90b2 to
061aaba
Compare
TomaszTB
force-pushed
the
feature/perception-autonomy-cleanup-2
branch
from
061aaba to
3d898b3
Compare
calvertdw
approved these changes
Dec 16, 2024
ds58
approved these changes
Dec 16, 2024
TomaszTB
force-pushed
the
feature/perception-autonomy-cleanup-2
branch
from
3d898b3 to
3bdc53b
Compare
rjgriffin42
pushed a commit
that referenced
this pull request
Jan 8, 2025
* Added basic Perception Autonomy Process with ZED * WIP - added ImageSensor + YOLO * Added RealsenseImageSensor * Reset ROS2DemandGraphNode to develop * Created ImageSensorPublishThread * Minor ImageSensor cleanup * Use PausableLoopingThread, add YOLO * Cleanup * Correct YOLO input to BGR * Add scene graph update thread, fix YOLO null pointer issue * WIP - Adding planar regions * WIP - adding ZEDSVOPlayback * Fixes * Improve planar regions extraction * Working planar regions * Added ArUco detection thread * Added ArUco to SceneGraph update * Added Height Map * Added ICP * Added Behavior Tree Update Thread * Cleanup * Remove old perception autonomy process * Allow specifying ZED input type * Polishing
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Added abstract ImageSensor + new NadiaPerceptionAutonomyProcess which can be used with real or simulated sensors.