Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Errors when deploying newest version of Knative CRDs #5759

Closed
gunpuz opened this issue Nov 14, 2023 · 15 comments · Fixed by #5769
Closed

Errors when deploying newest version of Knative CRDs #5759

gunpuz opened this issue Nov 14, 2023 · 15 comments · Fixed by #5769
Assignees

Comments

@gunpuz
Copy link

gunpuz commented Nov 14, 2023

Expected Behavior

Knative CRDS would deploy without STDERRs

Actual Behavior

Standart error is reported about incorrect CRD:

> kubectl apply -f https://github.com/knative/serving/releases/download/knative-v1.12.0/serving-crds.yaml
customresourcedefinition.apiextensions.k8s.io/certificates.networking.internal.knative.dev unchanged
customresourcedefinition.apiextensions.k8s.io/configurations.serving.knative.dev unchanged
customresourcedefinition.apiextensions.k8s.io/clusterdomainclaims.networking.internal.knative.dev unchanged
customresourcedefinition.apiextensions.k8s.io/ingresses.networking.internal.knative.dev unchanged
customresourcedefinition.apiextensions.k8s.io/metrics.autoscaling.internal.knative.dev unchanged
customresourcedefinition.apiextensions.k8s.io/podautoscalers.autoscaling.internal.knative.dev unchanged
customresourcedefinition.apiextensions.k8s.io/revisions.serving.knative.dev unchanged
customresourcedefinition.apiextensions.k8s.io/routes.serving.knative.dev unchanged
customresourcedefinition.apiextensions.k8s.io/serverlessservices.networking.internal.knative.dev unchanged
customresourcedefinition.apiextensions.k8s.io/services.serving.knative.dev unchanged
customresourcedefinition.apiextensions.k8s.io/images.caching.internal.knative.dev unchanged
The CustomResourceDefinition "domainmappings.serving.knative.dev" is invalid: status.storedVersions[0]: Invalid value: "v1alpha1": must appear in spec.versions

Steps to Reproduce the Problem

  1. kubectl apply -f https://github.com/knative/serving/releases/download/knative-v1.12.0/serving-crds.yaml

Additional Info

Copy link

knative-prow bot commented Nov 14, 2023

@gunpuz: GitHub didn't allow me to assign the following users: user.

Note that only knative members with read permissions, repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time.
For more information please see the contributor guide

In response to this:

Expected Behavior

Knative CRDS would deploy without STDERRs

Actual Behavior

Standart error is reported about incorrect CRD:

> kubectl apply -f https://github.com/knative/serving/releases/download/knative-v1.12.0/serving-crds.yaml
customresourcedefinition.apiextensions.k8s.io/certificates.networking.internal.knative.dev unchanged
customresourcedefinition.apiextensions.k8s.io/configurations.serving.knative.dev unchanged
customresourcedefinition.apiextensions.k8s.io/clusterdomainclaims.networking.internal.knative.dev unchanged
customresourcedefinition.apiextensions.k8s.io/ingresses.networking.internal.knative.dev unchanged
customresourcedefinition.apiextensions.k8s.io/metrics.autoscaling.internal.knative.dev unchanged
customresourcedefinition.apiextensions.k8s.io/podautoscalers.autoscaling.internal.knative.dev unchanged
customresourcedefinition.apiextensions.k8s.io/revisions.serving.knative.dev unchanged
customresourcedefinition.apiextensions.k8s.io/routes.serving.knative.dev unchanged
customresourcedefinition.apiextensions.k8s.io/serverlessservices.networking.internal.knative.dev unchanged
customresourcedefinition.apiextensions.k8s.io/services.serving.knative.dev unchanged
customresourcedefinition.apiextensions.k8s.io/images.caching.internal.knative.dev unchanged
The CustomResourceDefinition "domainmappings.serving.knative.dev" is invalid: status.storedVersions[0]: Invalid value: "v1alpha1": must appear in spec.versions

Steps to Reproduce the Problem

  1. kubectl apply -f https://github.com/knative/serving/releases/download/knative-v1.12.0/serving-crds.yaml

Additional Info

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@gunpuz gunpuz changed the title STDERR when deploying newest version of Knative CRDs Errors when deploying newest version of Knative CRDs Nov 15, 2023
@jasonsmithio
Copy link
Contributor

I just deployed on a GKE cluster without issue. Tell me more about your setup

@gunpuz
Copy link
Author

gunpuz commented Nov 18, 2023

What info would be usefull?

@jasonsmithio
Copy link
Contributor

What version of kubernetes? Where is it hosted (MiniKube, GKE, EKS, etc.)? What else is installed on the cluster (specifically at the time of deploying knative)

@gunpuz
Copy link
Author

gunpuz commented Nov 23, 2023

Its hosted on Ubuntu: microk8s
Kubernetes version: 1.28

There are already some custom services, helm charts... , including Knative Serving, Knative Eventing

@ReToCode
Copy link
Member

v1alpha1 version of DomainMapping was deprecated in 1.11, see https://github.com/knative/serving/releases/tag/knative-v1.11.0. It looks like you were still using them when migrating to version 1.12 which will cause the error. Make sure you update your DomainMapping objects to the new version before installing 1.12.

@gunpuz
Copy link
Author

gunpuz commented Nov 23, 2023

What is the recommended upgrade procedure? Is there any docs for this? You could give some hint, please :)

@jasonsmithio
Copy link
Contributor

I would assume just updating the domainmapping yamls to reflect the new API

https://knative.dev/docs/serving/services/custom-domains/#procedure

@dprotaso
Copy link
Member

So serving has a post install job after upgrading that needs to run

eg. https://github.com/knative/serving/releases/download/knative-v1.12.1/serving-post-install-jobs.yaml

This typically will set things up for the next release. If you run that job now prior to upgrading you should be fine.

We sorta mention it in the link below but I think we could do a better job documenting it

https://knative.dev/docs/install/yaml-install/serving/serving-installation-files/

@ReToCode
Copy link
Member

/assign

I'll add a bit more to the update docs.

@gunpuz this job will migrate the existing objects on the cluster, but you still need to update the version in your own instances of DomainMapping if you store them in git or similar (like GitOps, ArgoCD).

@gunpuz
Copy link
Author

gunpuz commented Dec 4, 2023

Unfortunately the post-install script did not help.

Uninstalling just Knative Serving and installing it again solved this issue.
Not a production scenario, but it worked well for development.
So this issue seems to be related only to Knative itself as everything else was left as it is.

P.S. @ReToCode
In docs, should there be?:

kubectl create -f https://github.com/knative/serving/releases/download/knative-v1.12.2/serving-post-install-jobs.yaml

instead of

kubectl apply -f https://github.com/knative/serving/releases/download/knative-v1.12.2/serving-post-install-jobs.yaml

ref kubernetes/kubernetes#44501

@ReToCode
Copy link
Member

ReToCode commented Dec 4, 2023

Uninstalling just Knative Serving and installing it again solved this issue.

Hm hard to say what went wrong there. Are you able to reproduce the issue so we can try to get to the same state?

P.S. @ReToCode In docs, should there be

Right, #5785

@gunpuz
Copy link
Author

gunpuz commented Dec 5, 2023

@ReToCode so should we reopen this issue then?

For example:

Step 1: Add older Knative

kubectl apply -f https://github.com/knative/serving/releases/download/knative-v1.10.2/serving-crds.yaml
kubectl wait --for=condition=Established --all crd
kubectl apply -f https://github.com/knative/serving/releases/download/knative-v1.10.2/serving-core.yaml
kubectl apply -f https://github.com/knative/net-kourier/releases/download/knative-v1.10.0/kourier.yaml

#kubectl patch configmap/config-network --namespace knative-serving --type merge --patch '{"data":{"ingress-class":"kourier.ingress.networking.knative.dev"}}'
# Adjustment for PowerShell
kubectl patch configmap/config-network --namespace knative-serving --type merge --patch '{\"data\":{\"ingress-class\":\"kourier.ingress.networking.knative.dev\"}}'

# Wait for Knative to become ready before starting knative services
kubectl wait --for=condition=ready --timeout=60s -n knative-serving --all pods
kubectl wait --for=condition=ready --timeout=60s -n kourier-system --all pods

Step 2: Update to newer Knative

kubectl apply -f https://github.com/knative/serving/releases/download/knative-v1.12.2/serving-crds.yaml
kubectl wait --for=condition=Established --all crd
kubectl apply -f https://github.com/knative/serving/releases/download/knative-v1.12.2/serving-core.yaml
kubectl apply -f https://github.com/knative/net-kourier/releases/download/knative-v1.12.1/kourier.yaml

Check if error is reported:

The CustomResourceDefinition "domainmappings.serving.knative.dev" is invalid: status.storedVersions[0]: Invalid value: "v1alpha1": must appear in spec.versions

Step 3: Run post install job

kubectl create -f https://github.com/knative/serving/releases/download/knative-v1.12.2/serving-post-install-jobs.yaml

job.batch/storage-version-migration-serving-f87s5 created

Step 4: Check this migration job logs to see anything interesting

{"severity":"INFO","timestamp":"2023-12-05T09:09:30.6460017Z","caller":"logging/config.go:80","message":"Unable to read vcs.revision from binary"}
{"severity":"INFO","timestamp":"2023-12-05T09:09:30.6467751Z","logger":"storage-migrator","caller":"migrate/main.go:69","message":"Migrating 5 group resources"}
{"severity":"INFO","timestamp":"2023-12-05T09:09:30.6468262Z","logger":"storage-migrator","caller":"migrate/main.go:72","message":"Migrating group resource services.serving.knative.dev"}
{"severity":"INFO","timestamp":"2023-12-05T09:09:30.6593467Z","logger":"storage-migrator","caller":"migrate/main.go:72","message":"Migrating group resource configurations.serving.knative.dev"}
{"severity":"INFO","timestamp":"2023-12-05T09:09:30.666545Z","logger":"storage-migrator","caller":"migrate/main.go:72","message":"Migrating group resource revisions.serving.knative.dev"}
{"severity":"INFO","timestamp":"2023-12-05T09:09:30.6756363Z","logger":"storage-migrator","caller":"migrate/main.go:72","message":"Migrating group resource routes.serving.knative.dev"}
{"severity":"INFO","timestamp":"2023-12-05T09:09:30.6784452Z","logger":"storage-migrator","caller":"migrate/main.go:72","message":"Migrating group resource domainmappings.serving.knative.dev"}
{"severity":"INFO","timestamp":"2023-12-05T09:09:30.6817267Z","logger":"storage-migrator","caller":"migrate/main.go:82","message":"Migration complete"}

Only thing that is "unusual" is - Unable to read vcs.revision from binary

Step 5: Run step 2 to check if error got away

@ReToCode
Copy link
Member

ReToCode commented Dec 5, 2023

Hm you can not update from 1.10 to 1.12 (see https://knative.dev/docs/install/upgrade/). Does the same thing occur when you do it 1.10 -> 1.11 -> 1.12?

@gunpuz
Copy link
Author

gunpuz commented Dec 7, 2023

This should could be the cause then if minor versions cannot be skipped.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants