Avoid host allocations for by-offload arguments to GPU kernels

[e-kayrakli]

This PR addresses a performance issue first noted by @stonea in the context of Coral. Today, I had a chat with @Guillaume-Helbecque who was also suffering partly from it.

This issue arises when:

1. multiple GPUs in a node are used, and
2. LICM is not able to peel off the array metadata before we generate the GPU kernel

When (2), we pass arrays by offload -- we copy the record onto GPU memory and pass that record as an argument to the kernel. This involves allocating memory for the record on device-accessible memory. Now, most likely because oversight (on my part), that allocation is done on page-locked host memory instead. (Probably because I was thinking that object instances should be host-accessible so that they can be initialized, but we're talking about an already initialized record that we'll bit-copy anyways).

Page-locked allocations are globally synchronous. In other words, they will block all other GPUs, or be blocked until all other GPUs are free. That is why this is especially problematic with (1).

This PR adjusts the memory allocation to be on device memory proper.

Resolves #24936

Test:

• nvidia
• amd
• some EX testing as I noted it to be problematic in the past

[stonea]

Assuming tests pass and work on EX then it LGTM.

So chpl_gpu_mem_alloc was allocating things on the host (as page locked memory)? I suppose the idea is the is the gpu (runtime) module's version of 'mem_alloc', which may or may not actually be on the gpu given context? Anyway, not something needed for this PR but I wonder if there's some better name we can come up with for this function to avoid the confusion going forward.

[e-kayrakli]

Created https://github.com/Cray/chapel-private/issues/6662

[e-kayrakli]

Some example directories, coral and jacobi is clear on AMD gpus on an EX. I am merging this.