Rendervulkan.cpp: Use less conservative (memory and execution) barriers

[sharkautarch]

This PR tries to optimize barrier usage w/ shaders by getting ahead-of-time information of the shader usage pattern in vulkan_composite()

I've tested this PR for syncronization hazards w/ the Vulkan Validation Layer's sync + queueSubmit() validation.
The only sync hazard it showed was a Present-After-Write around vkQueuePresentKHR that is also detected w/ ValveSoftware/gamescope/master (not sure if it is a false positive or is a real issue...).

I will note that I had to force .oldLayout for shaders to be VK_IMAGE_LAYOUT_GENERAL instead of VK_IMAGE_LAYOUT_UNDEFINED, because the validation layers would report sync hazards from what it detects is a layout transition from undefined -> general...
EDIT: fixed the issue mentioned in the crossed out part

[misyltoad]

I really dislike the idea of encoding certain task types. We should probably make this more akin to a proper render graph.

[sharkautarch]

I really dislike the idea of encoding certain task types. We should probably make this more akin to a proper render graph.

uhh yeah I guess...
I mean gamescope's compute shaders mostly just execute one after the other, so the most use out of this PR is to remove unnecessary parts of memory barrier.
Tho I guess that a rendergraph would be more useful w/ the barriers around reshade...
( and also now that I think about it, with the syncronization2 extension, technically it should be possible to allow part of a reshade shader and the part of gamescope's compute shader that just loads the shader's samplers to overlap in execution... but that's definitely getting into super complicated territory)
I'm sure someone will figure that part out at some point

[sharkautarch]

I really dislike the idea of encoding certain task types. We should probably make this more akin to a proper render graph.

@Joshua-Ashton
Now that I think about it, maybe a framegraph would be better for gamescope than a rendergraph
Unfortunately, the only open source implementation of a framegraph that I could find is this archived one: https://github.com/azhirnov/FrameGraph

NEW UPDATE:
It seems like Vulkan Scene Graph supports rendergraph-like synchronization w/ compute
Tho I'm not sure if it fully supports compute-only workloads, and I'm not sure how well it can be integrated w/ gamescope...
Here's a promising example I found demonstrating its support for compute work:
https://github.com/vsg-dev/vsgExamples/blob/b77dcd91fd4901ff04d6c3bcca5176ffec0be0ff/examples/state/vsgcomputevertex/vsgcomputevertex.cpp

EDIT
Now that I've looked more at the Vulkan Scene Graph thing, I'm not confident that it's well-suited for compute work.
However, I've found a newer Rendergraph framework that explicitly supports synchronization w/ compute work (wherein it supports attaching compute pipelines and seems to be able to automatically handle the syncronization for it, which would happen outside of any renderpass): Vuk
Example given w/ compute: https://github.com/martty/vuk/blob/master/examples/08_pipelined_compute.cpp
Also, it seems like Vuk doesn't require you to associate the rendergraph with a window, which Vulkan Scene Graph seems to require
Tho it does seem to have quite a bunch of dependencies...

[sharkautarch]

Going to close this, since it is clear that a rendergraph-based syncronization approach is what's desired.
I might look into hooking up Vuk rendergraph into gamescope in the future