Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use container-alpha1v2 endpoint #62

Merged
merged 23 commits into from
Apr 11, 2024
Merged

Use container-alpha1v2 endpoint #62

merged 23 commits into from
Apr 11, 2024

Conversation

pditommaso
Copy link
Contributor

This PR uses the container-alpha1v2 endpoint. See seqeralabs/wave#435

Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>
Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>
Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>
@munishchouhan
Copy link
Member

munishchouhan commented Mar 28, 2024

  1. Test with conda packages:
munish.chouhan@Munishs-MacBook-Pro wave-cli % ./app/build/native/nativeCompile/wave --wave-endpoint https://00ffbc112673.ngrok.app --conda-package salmon --conda-package bwa
00ffbc112673.ngrok.app/wt/1842754d0bd4/hrma017/dev:salmon_bwa--fe16f788a519f251
munish.chouhan@Munishs-MacBook-Pro wave-cli % docker pull 00ffbc112673.ngrok.app/wt/1842754d0bd4/hrma017/dev:salmon_bwa--fe16f788a519f251
salmon_bwa--fe16f788a519f251: Pulling from wt/1842754d0bd4/hrma017/dev
af107e978371: Pull complete 
d92c8c3cf3aa: Pull complete 
fbe23ba65ac8: Pull complete 
4f4fb700ef54: Pull complete 
7312325a77fb: Pull complete 
426740d77448: Pull complete 
fcbff2cf876a: Pull complete 
d05634e10fa6: Pull complete 
2c874f7590ab: Pull complete 
e4ceed76320e: Pull complete 
8714c42f20bc: Pull complete 
c2d577554e64: Pull complete 
e837b8180e8a: Pull complete 
Digest: sha256:452c94f22e229f2d0ec6c3a153efae4eb32e4b8856a1204b498fe9455b0d4897
Status: Downloaded newer image for 00ffbc112673.ngrok.app/wt/1842754d0bd4/hrma017/dev:salmon_bwa--fe16f788a519f251
00ffbc112673.ngrok.app/wt/1842754d0bd4/hrma017/dev:salmon_bwa--fe16f788a519f251

  1. Test singularity build with conda packages : failed
munish.chouhan@Munishs-MacBook-Pro wave-cli % ./app/build/native/nativeCompile/wave --wave-endpoint https://00ffbc112673.ngrok.app --conda-package salmon --conda-package bwa  -s --platform arm64 --freeze --tower-token "eyJ0aWQiOiAyfS5jNDk0MjhlNjAwNWViMzI2OGRmZGM0Mjg5MzlmYzI5M2ZjNTFmZDQz" --tower-endpoint http://localhost:8008/api --build-repo "docker.io/hrma017/dev"
null

@munishchouhan
Copy link
Member

if the add --await to the failed test, I get this error:

wave-cli % ./app/build/native/nativeCompile/wave --wave-endpoint https://00ffbc112673.ngrok.app --conda-package salmon -s --platform arm64 --freeze --tower-token "eyJ0aWQiOiAyfS5jNDk0MjhlNjAwNWViMzI2OGRmZGM0Mjg5MzlmYzI5M2ZjNTFmZDQz" --tower-endpoint http://localhost:8008/api --build-repo "docker.io/hrma017/dev" --await 
dev.failsafe.FailsafeException: java.net.ConnectException
        at dev.failsafe.FailsafeExecutor.call(FailsafeExecutor.java:444)
        at dev.failsafe.FailsafeExecutor.get(FailsafeExecutor.java:115)
        at io.seqera.wave.cli.Client.safeApply(Client.java:166)
        at io.seqera.wave.cli.Client.httpSend(Client.java:170)
        at io.seqera.wave.cli.Client.awaitImage(Client.java:206)
        at io.seqera.wave.cli.App.run(App.java:431)
        at io.seqera.wave.cli.App.main(App.java:223)
        at java.base@21.0.2/java.lang.invoke.LambdaForm$DMH/sa346b79c.invokeStaticInit(LambdaForm$DMH)
Caused by: java.net.ConnectException
        at java.net.http@21.0.2/jdk.internal.net.http.HttpClientImpl.send(HttpClientImpl.java:951)
        at java.net.http@21.0.2/jdk.internal.net.http.HttpClientFacade.send(HttpClientFacade.java:133)
        at io.seqera.wave.cli.Client.lambda$httpSend$1(Client.java:171)
        at dev.failsafe.Functions.lambda$toCtxSupplier$11(Functions.java:236)
        at dev.failsafe.Functions.lambda$get$0(Functions.java:46)
        at dev.failsafe.internal.RetryPolicyExecutor.lambda$apply$0(RetryPolicyExecutor.java:75)
        at dev.failsafe.SyncExecutionImpl.executeSync(SyncExecutionImpl.java:176)
        at dev.failsafe.FailsafeExecutor.call(FailsafeExecutor.java:437)
        ... 7 more
Caused by: java.net.ConnectException
        at java.net.http@21.0.2/jdk.internal.net.http.common.Utils.toConnectException(Utils.java:1028)
        at java.net.http@21.0.2/jdk.internal.net.http.PlainHttpConnection.connectAsync(PlainHttpConnection.java:227)
        at java.net.http@21.0.2/jdk.internal.net.http.AsyncSSLConnection.connectAsync(AsyncSSLConnection.java:56)
        at java.net.http@21.0.2/jdk.internal.net.http.Http1Exchange.sendHeadersAsync(Http1Exchange.java:312)
        at java.net.http@21.0.2/jdk.internal.net.http.Exchange.lambda$responseAsyncImpl0$8(Exchange.java:567)
        at java.net.http@21.0.2/jdk.internal.net.http.Exchange.checkFor407(Exchange.java:447)
        at java.net.http@21.0.2/jdk.internal.net.http.Exchange.lambda$responseAsyncImpl0$9(Exchange.java:571)
        at java.base@21.0.2/java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:934)
        at java.base@21.0.2/java.util.concurrent.CompletableFuture.uniHandleStage(CompletableFuture.java:950)
        at java.base@21.0.2/java.util.concurrent.CompletableFuture.handle(CompletableFuture.java:2372)
        at java.net.http@21.0.2/jdk.internal.net.http.Exchange.responseAsyncImpl0(Exchange.java:571)
        at java.net.http@21.0.2/jdk.internal.net.http.Exchange.responseAsyncImpl(Exchange.java:423)
        at java.net.http@21.0.2/jdk.internal.net.http.Exchange.responseAsync(Exchange.java:415)
        at java.net.http@21.0.2/jdk.internal.net.http.MultiExchange.responseAsyncImpl(MultiExchange.java:413)
        at java.net.http@21.0.2/jdk.internal.net.http.MultiExchange.lambda$responseAsyncImpl$7(MultiExchange.java:454)
        at java.base@21.0.2/java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:934)
        at java.base@21.0.2/java.util.concurrent.CompletableFuture.uniHandleStage(CompletableFuture.java:950)
        at java.base@21.0.2/java.util.concurrent.CompletableFuture.handle(CompletableFuture.java:2372)
        at java.net.http@21.0.2/jdk.internal.net.http.MultiExchange.responseAsyncImpl(MultiExchange.java:444)
        at java.net.http@21.0.2/jdk.internal.net.http.MultiExchange.lambda$responseAsync0$2(MultiExchange.java:346)
        at java.base@21.0.2/java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:1150)
        at java.base@21.0.2/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
        at java.base@21.0.2/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1773)
        at java.net.http@21.0.2/jdk.internal.net.http.HttpClientImpl$DelegatingExecutor.execute(HttpClientImpl.java:177)
        at java.base@21.0.2/java.util.concurrent.CompletableFuture.completeAsync(CompletableFuture.java:2719)
        at java.net.http@21.0.2/jdk.internal.net.http.MultiExchange.responseAsync(MultiExchange.java:299)
        at java.net.http@21.0.2/jdk.internal.net.http.HttpClientImpl.sendAsync(HttpClientImpl.java:1047)
        at java.net.http@21.0.2/jdk.internal.net.http.HttpClientImpl.send(HttpClientImpl.java:930)
        ... 14 more
Caused by: java.nio.channels.UnresolvedAddressException
        at java.base@21.0.2/sun.nio.ch.Net.checkAddress(Net.java:137)
        at java.base@21.0.2/sun.nio.ch.Net.checkAddress(Net.java:145)
        at java.base@21.0.2/sun.nio.ch.SocketChannelImpl.checkRemote(SocketChannelImpl.java:842)
        at java.base@21.0.2/sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:865)
        at java.net.http@21.0.2/jdk.internal.net.http.PlainHttpConnection.lambda$connectAsync$1(PlainHttpConnection.java:210)
        at java.base@21.0.2/java.security.AccessController.executePrivileged(AccessController.java:114)
        at java.base@21.0.2/java.security.AccessController.doPrivileged(AccessController.java:571)
        at java.net.http@21.0.2/jdk.internal.net.http.PlainHttpConnection.connectAsync(PlainHttpConnection.java:212)
        ... 40 more

@munishchouhan
Copy link
Member

It seems that the freeze functionality of alpha1v2/container endpoint is not correctly working

@munishchouhan
Copy link
Member

test is failing beacuse now freeze is also returning the image name in targetImage and not in containerImage

return freeze
? resp.containerImage
: resp.targetImage;
}

@munishchouhan
Copy link
Member

munishchouhan commented Apr 1, 2024

As per the wave change in the below mentioned link, we are null to containerImage incase of V2, so I will make the change in wave-cli to only output targetImage
https://github.com/seqeralabs/wave/blob/d805e0c05f03bf622bc38f02a6696773d3ea7eb3/src/main/groovy/io/seqera/wave/controller/ContainerTokenController.groovy#L213-L215

Signed-off-by: munishchouhan <hrma017@gmail.com>
@munishchouhan
Copy link
Member

  1. Test singularity build with conda packages : success
./app/build/native/nativeCompile/wave --wave-endpoint https://00ffbc112673.ngrok.app --conda-package salmon --conda-package bwa  -s --platform arm64 --freeze --tower-token "eyJ0aWQiOiAyfS5jNDk0MjhlNjAwNWViMzI2OGRmZGM0Mjg5MzlmYzI5M2ZjNTFmZDQz" --tower-endpoint http://localhost:8008/api --build-repo "docker.io/hrma017/dev"
oras://docker.io/hrma017/dev:salmon_bwa--9c3881bd9217f26e

@munishchouhan
Copy link
Member

  1. Test singularity and freeze with spack
./app/build/native/nativeCompile/wave --wave-endpoint https://00ffbc112673.ngrok.app --spack-package salmon  -s --platform arm64 --freeze --tower-token "eyJ0aWQiOiAyfS5jNDk0MjhlNjAwNWViMzI2OGRmZGM0Mjg5MzlmYzI5M2ZjNTFmZDQz" --tower-endpoint http://localhost:8008/api --build-repo "hrma017/dev"
oras://hrma017/dev:salmon--0988bf00c31a019f
  1. Test using docker and spack packages
./app/build/native/nativeCompile/wave --wave-endpoint https://00ffbc112673.ngrok.app --spack-package salmon   --tower-token "eyJ0aWQiOiAyfS5jNDk0MjhlNjAwNWViMzI2OGRmZGM0Mjg5MzlmYzI5M2ZjNTFmZDQz" --tower-endpoint http://localhost:8008/api
00ffbc112673.ngrok.app/wt/59463b9c4ef9/hrma017/dev:salmon--e04bf60e6cea0058
  1. Test using docker, spack and arm64
munish.chouhan@Munishs-MacBook-Pro wave-cli % ./app/build/native/nativeCompile/wave --wave-endpoint https://00ffbc112673.ngrok.app --conda-package salmon  --platform arm64
00ffbc112673.ngrok.app/wt/015dba406cf3/hrma017/dev:salmon--906c138318fdd09d
Screenshot 2024-04-01 at 14 04 14

@pditommaso
Copy link
Contributor Author

Using the new endpoint the await needs to be re-implemented so that when the buildId is returned in the response the cached flag is false, the status endpoint should be check until it returns COMPLETED

@munishchouhan
Copy link
Member

Using the new endpoint the await needs to be re-implemented so that when the buildId is returned in the response the cached flag is false, the status endpoint should be check until it returns COMPLETED

This change needs BuildStatusResponse in wave-cli also, to access the status, I will create a PR in libseqera to add the same there

Signed-off-by: munishchouhan <hrma017@gmail.com>
@munishchouhan
Copy link
Member

depends upon seqeralabs/libseqera#20

@munishchouhan
Copy link
Member

@pditommaso as now we are using status endpoint, should we check if the build succeeded or not and if only succeeded then only output the image name while using await or some error message?

@pditommaso
Copy link
Contributor Author

Let's just check the for complete status for now. It could be added the error check later

Signed-off-by: munishchouhan <hrma017@gmail.com>
Signed-off-by: munishchouhan <hrma017@gmail.com>
Signed-off-by: munishchouhan <hrma017@gmail.com>
@munishchouhan munishchouhan marked this pull request as draft April 3, 2024 10:23
@munishchouhan
Copy link
Member

converting it to draft, getting error while using await for cached image, will fix and make it ready for review
error:

12:19:59.291 [main] DEBUG io.seqera.wave.cli.Client - Wave connection failure - attempt: 1
java.net.ConnectException: null
        at java.net.http@21.0.2/jdk.internal.net.http.HttpClientImpl.send(HttpClientImpl.java:951)
        at java.net.http@21.0.2/jdk.internal.net.http.HttpClientFacade.send(HttpClientFacade.java:133)
        at io.seqera.wave.cli.Client.lambda$httpSend$1(Client.java:166)
        at dev.failsafe.Functions.lambda$toCtxSupplier$11(Functions.java:236)
        at dev.failsafe.Functions.lambda$get$0(Functions.java:46)
        at dev.failsafe.internal.RetryPolicyExecutor.lambda$apply$0(RetryPolicyExecutor.java:75)
        at dev.failsafe.SyncExecutionImpl.executeSync(SyncExecutionImpl.java:176)
        at dev.failsafe.FailsafeExecutor.call(FailsafeExecutor.java:437)
        at dev.failsafe.FailsafeExecutor.get(FailsafeExecutor.java:115)
        at io.seqera.wave.cli.Client.safeApply(Client.java:161)
        at io.seqera.wave.cli.Client.httpSend(Client.java:165)
        at io.seqera.wave.cli.Client.awaitImage0(Client.java:209)
        at io.seqera.wave.cli.Client.awaitImage(Client.java:196)
        at io.seqera.wave.cli.App.run(App.java:432)
        at io.seqera.wave.cli.App.main(App.java:223)

Signed-off-by: munishchouhan <hrma017@gmail.com>
@munishchouhan
Copy link
Member

munishchouhan commented Apr 3, 2024

This error is coming because now v2 endpoint is sending the containerImage name in targetImage

https://github.com/seqeralabs/wave/blob/d805e0c05f03bf622bc38f02a6696773d3ea7eb3/src/main/groovy/io/seqera/wave/controller/ContainerTokenController.groovy#L204-L207

and that's making imageToManifestUri method to create wrong url

final URI manifest = imageToManifestUri(image);

@munishchouhan
Copy link
Member

I think when its freeze, we should only check the build endpoint in await state
I will make the changes accordingly

Signed-off-by: munishchouhan <hrma017@gmail.com>
Signed-off-by: munishchouhan <hrma017@gmail.com>
@munishchouhan
Copy link
Member

@pditommaso can you please check the await functionality change and see if this is the intended functionality?

https://github.com/seqeralabs/wave-cli/blob/aea2ff3b51fcd5b91d46f4ef3ed2c4b90e6aed70/app/src/main/java/io/seqera/wave/cli/Client.java#L194-261

Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>
Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>
@pditommaso
Copy link
Contributor Author

I gave it another try to simplify it a bit

Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>
Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>
@munishchouhan
Copy link
Member

munishchouhan commented Apr 4, 2024

  1. Test with conda packages with await:
    New client:
% ./app/build/native/nativeCompile/wave --wave-endpoint https://00ffbc112673.ngrok.app --conda-package salmon --conda-package bwa --await

00ffbc112673.ngrok.app/wt/a1c9ae30c6b3/hrma017/dev:salmon_bwa--fe16f788a519f251

Old client:

% wave --wave-endpoint https://00ffbc112673.ngrok.app --conda-package salmon  --await

00ffbc112673.ngrok.app/wt/dc50ed2b3b09/hrma017/dev:salmon--88bbb93b1e6558b4
  1. Test singularity build with conda packages with await:
    New client:
% ./app/build/native/nativeCompile/wave -s --platform arm64 --freeze --wave-endpoint https://00ffbc112673.ngrok.app --conda-package salmon    --tower-token "eyJ0aWQiOiAyfS5jNDk0MjhlNjAwNWViMzI2OGRmZGM0Mjg5MzlmYzI5M2ZjNTFmZDQz" --tower-endpoint http://localhost:8008/api --build-repo "docker.io/hrma017/dev" --await

oras://docker.io/hrma017/dev:salmon--c1f2ecd9bcb7e5db

Old client:

% wave -s --platform arm64 --freeze --wave-endpoint https://00ffbc112673.ngrok.app --conda-package Jedi    --tower-token "eyJ0aWQiOiAyfS5jNDk0MjhlNjAwNWViMzI2OGRmZGM0Mjg5MzlmYzI5M2ZjNTFmZDQz" --tower-endpoint http://localhost:8008/api --build-repo "docker.io/hrma017/dev" --await

oras://docker.io/hrma017/dev:Jedi--e2a91ac9d44064b4

@munishchouhan
Copy link
Member

Test sending the same requests again requests: success no wait time

munish.chouhan@Munishs-MacBook-Pro wave-cli % ./app/build/native/nativeCompile/wave --wave-endpoint https://00ffbc112673.ngrok.app --conda-package salmon --conda-package bwa --await
00ffbc112673.ngrok.app/wt/6dc014f73ec3/hrma017/dev:salmon_bwa--fe16f788a519f251
munish.chouhan@Munishs-MacBook-Pro wave-cli % wave --wave-endpoint https://00ffbc112673.ngrok.app --conda-package salmon  --await
00ffbc112673.ngrok.app/wt/f78d9a321cde/hrma017/dev:salmon--88bbb93b1e6558b4
munish.chouhan@Munishs-MacBook-Pro wave-cli % ./app/build/native/nativeCompile/wave -s --platform arm64 --freeze --wave-endpoint https://00ffbc112673.ngrok.app --conda-package salmon    --tower-token "eyJ0aWQiOiAyfS5jNDk0MjhlNjAwNWViMzI2OGRmZGM0Mjg5MzlmYzI5M2ZjNTFmZDQz" --tower-endpoint http://localhost:8008/api --build-repo "docker.io/hrma017/dev" --await
oras://docker.io/hrma017/dev:salmon--c1f2ecd9bcb7e5db
munish.chouhan@Munishs-MacBook-Pro wave-cli % wave -s --platform arm64 --freeze --wave-endpoint https://00ffbc112673.ngrok.app --conda-package Jedi    --tower-token "eyJ0aWQiOiAyfS5jNDk0MjhlNjAwNWViMzI2OGRmZGM0Mjg5MzlmYzI5M2ZjNTFmZDQz" --tower-endpoint http://localhost:8008/api --build-repo "docker.io/hrma017/dev" --await
oras://docker.io/hrma017/dev:Jedi--e2a91ac9d44064b4

@munishchouhan
Copy link
Member

Testing after deleting images from the repo and invalidating the cache, and repeating the commands is also successful

@munishchouhan munishchouhan marked this pull request as ready for review April 4, 2024 13:41
@marcodelapierre
Copy link
Contributor

@munishchouhan Is this still relevant ? seqeralabs/wave#423 (comment)

Thank you

@munishchouhan
Copy link
Member

@munishchouhan Is this still relevant ? seqeralabs/wave#423 (comment)

Thank you

I think this is a rare scenario, so we can ignore this in this effort.

Now I only one item still open about the delay in between in status API calls

Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>
Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>
@pditommaso
Copy link
Contributor Author

Let's finish this fixing the wait loop and the await timeout

@munishchouhan
Copy link
Member

Let's finish this fixing the wait loop and the await timeout

sure, working on it

munishchouhan and others added 6 commits April 10, 2024 15:37
Signed-off-by: munishchouhan <hrma017@gmail.com>
Signed-off-by: munishchouhan <hrma017@gmail.com>
Signed-off-by: munishchouhan <hrma017@gmail.com>
Signed-off-by: munishchouhan <hrma017@gmail.com>
Signed-off-by: munishchouhan <hrma017@gmail.com>
Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>
@pditommaso pditommaso merged commit a923aac into master Apr 11, 2024
11 checks passed
@pditommaso pditommaso deleted the container-alpha1v2 branch April 11, 2024 09:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants