fixed review comments

datahub-project · Mar 4, 2025 · 5765025 · 5765025
1 parent 4edd575
commit 5765025
Show file tree

Hide file tree

Showing 10 changed files with 540 additions and 217 deletions.
diff --git a/metadata-ingestion/docs/sources/vertexai/README.md b/metadata-ingestion/docs/sources/vertexai/README.md
diff --git a/metadata-ingestion/docs/sources/vertexai/vertexai_pre.md b/metadata-ingestion/docs/sources/vertexai/vertexai_pre.md
@@ -1,14 +1,16 @@
+Ingesting metadata from VertexAI requires using the **Vertex AI** module.
 
+#### Prerequisites
+Please refer to the [Vertex AI documentation](https://cloud.google.com/vertex-ai/docs) for basic information on Vertex AI.
 
-#### Credential to access to GCP
-1. Follow the section on credentials to access Vertex AI  [GCP docs](https://cloud.google.com/docs/authentication/provide-credentials-adc#how-to).
+#### Credentials to access to GCP
+Please read the section to understand how to set up application default Credentials to GCP [GCP docs](https://cloud.google.com/docs/authentication/provide-credentials-adc#how-to).
 
 #### Create a service account and assign roles
 
-1. Setup a ServiceAccount as per [GCP docs](https://cloud.google.com/iam/docs/creating-managing-service-accounts#iam-service-accounts-create-console)
-   and assign the previously created role to this service account.
-2. Download a service account JSON keyfile.
-   Example credential file:
+1. Setup a ServiceAccount as per [GCP docs](https://cloud.google.com/iam/docs/creating-managing-service-accounts#iam-service-accounts-create-console) and assign the previously created role to this service account.
+2. Download a service account JSON keyfile. 
+- Example credential file:
 
 ```json
 {
@@ -27,15 +29,15 @@
 
 3. To provide credentials to the source, you can either:
 
-   Set an environment variable:
+- Set an environment variable:
 
    ```sh
    $ export GOOGLE_APPLICATION_CREDENTIALS="/path/to/keyfile.json"
    ```
 
    _or_
 
-   Set credential config in your source based on the credential json file. For example:
+- Set credential config in your source based on the credential json file. For example:
 
    ```yml
    credential:

diff --git a/metadata-ingestion/docs/sources/vertexai/vertexai_recipe.yml b/metadata-ingestion/docs/sources/vertexai/vertexai_recipe.yml
@@ -3,7 +3,8 @@ source:
   config:
     project_id: "acryl-poc"
     region:  "us-west2"
-#    credential:
+# Note that GOOGLE_APPLICATION_CREDENTIALS or credential section below is required for authentication.
+#   credential:
 #      private_key: '-----BEGIN PRIVATE KEY-----\\nprivate-key\\n-----END PRIVATE KEY-----\\n'
 #      private_key_id: "project_key_id"
 #      client_email: "client_email"

diff --git a/metadata-ingestion/src/datahub/ingestion/source/bigquery_v2/bigquery_config.py b/metadata-ingestion/src/datahub/ingestion/source/bigquery_v2/bigquery_config.py
@@ -18,7 +18,7 @@
 from datahub.ingestion.glossary.classification_mixin import (
     ClassificationSourceConfigMixin,
 )
-from datahub.ingestion.source.common.credentials import GCPCredential
+from datahub.ingestion.source.common.gcp_credentials_config import GCPCredential
 from datahub.ingestion.source.data_lake_common.path_spec import PathSpec
 from datahub.ingestion.source.sql.sql_config import SQLCommonConfig, SQLFilterConfig
 from datahub.ingestion.source.state.stateful_ingestion_base import (

diff --git a/...ub/ingestion/source/common/credentials.py → ...n/source/common/gcp_credentials_config.py b/...ub/ingestion/source/common/credentials.py → ...n/source/common/gcp_credentials_config.py