Use PyPI distribution for MeltanoLabs tap-postgres #1951
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
edgarrmondragon
requested review from
tayloramurphy and
pnadolny13
as code owners
edgarrmondragon
temporarily deployed
to
preview
GitHub Actions
Inactive
✅ Deploy Preview for meltano-hub ready!
To edit notification comments on pull requests, go to your Netlify site configuration. |
|
Testing plugin
Auto-generated README.md
|
| Setting | Required | Default | Description |
|---|---|---|---|
| replication_slot_name | False | tappostgres | Name of the replication slot to use for logical replication. Must be unique for parallel extractions. Only applicable when replication_method is LOG_BASED.- Contain only letters, numbers, and underscores. - Be less than or equal to 63 characters. - Not start with 'pg_'. |
| host | False | None | Hostname for postgres instance. Note if sqlalchemy_url is set this will be ignored. |
| port | False | 5432 | The port on which postgres is awaiting connection. Note if sqlalchemy_url is set this will be ignored. |
| user | False | None | User name used to authenticate. Note if sqlalchemy_url is set this will be ignored. |
| password | False | None | Password used to authenticate. Note if sqlalchemy_url is set this will be ignored. |
| database | False | None | Database name. Note if sqlalchemy_url is set this will be ignored. |
| max_record_count | False | None | Optional. The maximum number of records to return in a single stream. |
| sqlalchemy_url | False | None | Example postgresql://[username]:[password]@localhost:5432/[db_name] |
| filter_schemas | False | None | If an array of schema names is provided, the tap will only process the specified Postgres schemas and ignore others. If left blank, the tap automatically determines ALL available Postgres schemas. |
| dates_as_string | False | 0 | Defaults to false, if true, date, and timestamp fields will be Strings. If you see ValueError: Year is out of range, try setting this to True. |
| json_as_object | False | 0 | Defaults to false, if true, json and jsonb fields will be Objects. |
| ssh_tunnel | False | None | SSH Tunnel Configuration, this is a json object |
| ssh_tunnel.enable | False | 0 | Enable an ssh tunnel (also known as bastion server), see the other ssh_tunnel.* properties for more details |
| ssh_tunnel.host | False | None | Host of the bastion server, this is the host we'll connect to via ssh |
| ssh_tunnel.username | False | None | Username to connect to bastion server |
| ssh_tunnel.port | False | 22 | Port to connect to bastion server |
| ssh_tunnel.private_key | False | None | Private Key for authentication to the bastion server |
| ssh_tunnel.private_key_password | False | None | Private Key Password, leave None if no password is set |
| ssl_enable | False | 0 | Whether or not to use ssl to verify the server's identity. Use ssl_certificate_authority and ssl_mode for further customization. To use a client certificate to authenticate yourself to the server, use ssl_client_certificate_enable instead. Note if sqlalchemy_url is set this will be ignored. |
| ssl_client_certificate_enable | False | 0 | Whether or not to provide client-side certificates as a method of authentication to the server. Use ssl_client_certificate and ssl_client_private_key for further customization. To use SSL to verify the server's identity, use ssl_enable instead. Note if sqlalchemy_url is set this will be ignored. |
| ssl_mode | False | verify-full | SSL Protection method, see postgres documentation for more information. Must be one of disable, allow, prefer, require, verify-ca, or verify-full. Note if sqlalchemy_url is set this will be ignored. |
| ssl_certificate_authority | False | ~/.postgresql/root.crl | The certificate authority that should be used to verify the server's identity. Can be provided either as the certificate itself (in .env) or as a filepath to the certificate. Note if sqlalchemy_url is set this will be ignored. |
| ssl_client_certificate | False | ~/.postgresql/postgresql.crt | The certificate that should be used to verify your identity to the server. Can be provided either as the certificate itself (in .env) or as a filepath to the certificate. Note if sqlalchemy_url is set this will be ignored. |
| ssl_client_private_key | False | ~/.postgresql/postgresql.key | The private key for the certificate you provided. Can be provided either as the certificate itself (in .env) or as a filepath to the certificate. Note if sqlalchemy_url is set this will be ignored. |
| ssl_storage_directory | False | .secrets | The folder in which to store SSL certificates provided as raw values. When a certificate/key is provided as a raw value instead of as a filepath, it must be written to a file before it can be used. This configuration option determines where that file is created. |
| default_replication_method | False | FULL_TABLE | Replication method to use if there is not a catalog entry to override this choice. One of FULL_TABLE, INCREMENTAL, or LOG_BASED. |
| stream_maps | False | None | Config object for stream maps capability. For more information check out Stream Maps. |
| stream_map_config | False | None | User-defined config values to be used within map expressions. |
| faker_config | False | None | Config for the Faker instance variable fake used within map expressions. Only applicable if the plugin specifies faker as an additional dependency (through the singer-sdk faker extra or directly). |
| faker_config.seed | False | None | Value to seed the Faker generator for deterministic output: https://faker.readthedocs.io/en/master/#seeding-the-generator |
| faker_config.locale | False | None | One or more LCID locale strings to produce localized output for: https://faker.readthedocs.io/en/master/#localization |
| flattening_enabled | False | None | 'True' to enable schema flattening and automatically expand nested properties. |
| flattening_max_depth | False | None | The max depth to flatten schemas. |
| batch_config | False | None | Configuration for BATCH message capabilities. |
| batch_config.encoding | False | None | Specifies the format and compression of the batch files. |
| batch_config.encoding.format | False | None | Format to use for batch files. |
| batch_config.encoding.compression | False | None | Compression format to use for batch files. |
| batch_config.storage | False | None | Defines the storage layer to use when writing batch files |
| batch_config.storage.root | False | None | Root path to use when writing batch files. |
| batch_config.storage.prefix | False | None | Prefix to use when writing batch files. |
A full list of supported settings and capabilities is available by running: tap-postgres --about
Version info
tap-postgres v[could not be detected], Meltano SDK v0.44.3
Usage info
melty-bot % tap-postgres --help
Usage: tap-postgres [OPTIONS]
Execute the Singer tap.
Options:
--version Display the package version.
--about Display package metadata and settings.
--format [json|markdown] Specify output style for --about
--config TEXT Configuration file location or 'ENV' to use
environment variables.
--discover Run the tap in discovery mode.
--test TEXT Use --test to sync a single record for each
stream. Use --test=schema to test schema output
without syncing records.
--catalog FILE Use a Singer catalog file with the tap.
--state FILE Use a bookmarks file for incremental replication.
--help Show this message and exit.
Detected capabilities
- ✅ 'discover'
- ✅ 'catalog'
- ✅ 'state'
- ✅ 'about'
JSON Metadata
{
"name": "tap-postgres",
"description": "Singer tap for Postgres.",
"version": "[could not be detected]",
"sdk_version": "0.44.3",
"supported_python_versions": null,
"capabilities": [
"catalog",
"state",
"discover",
"about",
"stream-maps",
"schema-flattening",
"batch"
],
"settings": {
"type": "object",
"properties": {
"replication_slot_name": {
"type": [
"string",
"null"
],
"pattern": "^(?!pg_)[A-Za-z0-9_]{1,63}$",
"default": "tappostgres",
"description": "Name of the replication slot to use for logical replication. Must be unique for parallel extractions. Only applicable when replication_method is LOG_BASED.- Contain only letters, numbers, and underscores. - Be less than or equal to 63 characters. - Not start with 'pg_'."
},
"host": {
"type": [
"string",
"null"
],
"description": "Hostname for postgres instance. Note if sqlalchemy_url is set this will be ignored."
},
"port": {
"type": [
"integer",
"null"
],
"default": 5432,
"description": "The port on which postgres is awaiting connection. Note if sqlalchemy_url is set this will be ignored."
},
"user": {
"type": [
"string",
"null"
],
"description": "User name used to authenticate. Note if sqlalchemy_url is set this will be ignored."
},
"password": {
"type": [
"string",
"null"
],
"description": "Password used to authenticate. Note if sqlalchemy_url is set this will be ignored.",
"secret": true,
"writeOnly": true
},
"database": {
"type": [
"string",
"null"
],
"description": "Database name. Note if sqlalchemy_url is set this will be ignored."
},
"max_record_count": {
"type": [
"integer",
"null"
],
"description": "Optional. The maximum number of records to return in a single stream."
},
"sqlalchemy_url": {
"type": [
"string",
"null"
],
"description": "Example postgresql://[username]:[password]@localhost:5432/[db_name]",
"secret": true,
"writeOnly": true
},
"filter_schemas": {
"type": [
"array",
"null"
],
"items": {
"type": [
"string"
]
},
"description": "If an array of schema names is provided, the tap will only process the specified Postgres schemas and ignore others. If left blank, the tap automatically determines ALL available Postgres schemas."
},
"dates_as_string": {
"type": [
"boolean",
"null"
],
"default": false,
"description": "Defaults to false, if true, date, and timestamp fields will be Strings. If you see ValueError: Year is out of range, try setting this to True."
},
"json_as_object": {
"type": [
"boolean",
"null"
],
"default": false,
"description": "Defaults to false, if true, json and jsonb fields will be Objects."
},
"ssh_tunnel": {
"type": [
"object",
"null"
],
"properties": {
"enable": {
"type": [
"boolean",
"null"
],
"default": false,
"description": "Enable an ssh tunnel (also known as bastion server), see the other ssh_tunnel.* properties for more details"
},
"host": {
"type": [
"string",
"null"
],
"description": "Host of the bastion server, this is the host we'll connect to via ssh"
},
"username": {
"type": [
"string",
"null"
],
"description": "Username to connect to bastion server"
},
"port": {
"type": [
"integer",
"null"
],
"default": 22,
"description": "Port to connect to bastion server"
},
"private_key": {
"type": [
"string",
"null"
],
"description": "Private Key for authentication to the bastion server",
"secret": true,
"writeOnly": true
},
"private_key_password": {
"type": [
"string",
"null"
],
"description": "Private Key Password, leave None if no password is set",
"secret": true,
"writeOnly": true
}
},
"description": "SSH Tunnel Configuration, this is a json object"
},
"ssl_enable": {
"type": [
"boolean",
"null"
],
"default": false,
"description": "Whether or not to use ssl to verify the server's identity. Use ssl_certificate_authority and ssl_mode for further customization. To use a client certificate to authenticate yourself to the server, use ssl_client_certificate_enable instead. Note if sqlalchemy_url is set this will be ignored."
},
"ssl_client_certificate_enable": {
"type": [
"boolean",
"null"
],
"default": false,
"description": "Whether or not to provide client-side certificates as a method of authentication to the server. Use ssl_client_certificate and ssl_client_private_key for further customization. To use SSL to verify the server's identity, use ssl_enable instead. Note if sqlalchemy_url is set this will be ignored."
},
"ssl_mode": {
"type": [
"string",
"null"
],
"default": "verify-full",
"description": "SSL Protection method, see [postgres documentation](https://www.postgresql.org/docs/current/libpq-ssl.html#LIBPQ-SSL-PROTECTION) for more information. Must be one of disable, allow, prefer, require, verify-ca, or verify-full. Note if sqlalchemy_url is set this will be ignored."
},
"ssl_certificate_authority": {
"type": [
"string",
"null"
],
"default": "~/.postgresql/root.crl",
"description": "The certificate authority that should be used to verify the server's identity. Can be provided either as the certificate itself (in .env) or as a filepath to the certificate. Note if sqlalchemy_url is set this will be ignored."
},
"ssl_client_certificate": {
"type": [
"string",
"null"
],
"default": "~/.postgresql/postgresql.crt",
"description": "The certificate that should be used to verify your identity to the server. Can be provided either as the certificate itself (in .env) or as a filepath to the certificate. Note if sqlalchemy_url is set this will be ignored."
},
"ssl_client_private_key": {
"type": [
"string",
"null"
],
"default": "~/.postgresql/postgresql.key",
"description": "The private key for the certificate you provided. Can be provided either as the certificate itself (in .env) or as a filepath to the certificate. Note if sqlalchemy_url is set this will be ignored."
},
"ssl_storage_directory": {
"type": [
"string",
"null"
],
"default": ".secrets",
"description": "The folder in which to store SSL certificates provided as raw values. When a certificate/key is provided as a raw value instead of as a filepath, it must be written to a file before it can be used. This configuration option determines where that file is created."
},
"default_replication_method": {
"type": [
"string",
"null"
],
"default": "FULL_TABLE",
"description": "Replication method to use if there is not a catalog entry to override this choice. One of `FULL_TABLE`, `INCREMENTAL`, or `LOG_BASED`.",
"enum": [
"FULL_TABLE",
"INCREMENTAL",
"LOG_BASED"
]
},
"stream_maps": {
"type": [
"object",
"null"
],
"properties": {},
"title": "Stream Maps",
"description": "Config object for stream maps capability. For more information check out [Stream Maps](https://sdk.meltano.com/en/latest/stream_maps.html)."
},
"stream_map_config": {
"type": [
"object",
"null"
],
"properties": {},
"title": "User Stream Map Configuration",
"description": "User-defined config values to be used within map expressions."
},
"faker_config": {
"type": [
"object",
"null"
],
"properties": {
"seed": {
"oneOf": [
{
"type": [
"number"
]
},
{
"type": [
"string"
]
},
{
"type": [
"boolean"
]
},
{
"type": "null"
}
],
"title": "Faker Seed",
"description": "Value to seed the Faker generator for deterministic output: https://faker.readthedocs.io/en/master/#seeding-the-generator"
},
"locale": {
"oneOf": [
{
"type": [
"string"
]
},
{
"type": "array",
"items": {
"type": [
"string"
]
}
},
{
"type": "null"
}
],
"title": "Faker Locale",
"description": "One or more LCID locale strings to produce localized output for: https://faker.readthedocs.io/en/master/#localization"
}
},
"title": "Faker Configuration",
"description": "Config for the [`Faker`](https://faker.readthedocs.io/en/master/) instance variable `fake` used within map expressions. Only applicable if the plugin specifies `faker` as an additional dependency (through the `singer-sdk` `faker` extra or directly)."
},
"flattening_enabled": {
"type": [
"boolean",
"null"
],
"title": "Enable Schema Flattening",
"description": "'True' to enable schema flattening and automatically expand nested properties."
},
"flattening_max_depth": {
"type": [
"integer",
"null"
],
"title": "Max Flattening Depth",
"description": "The max depth to flatten schemas."
},
"batch_config": {
"type": [
"object",
"null"
],
"properties": {
"encoding": {
"type": [
"object",
"null"
],
"properties": {
"format": {
"type": [
"string",
"null"
],
"title": "Batch Encoding Format",
"description": "Format to use for batch files.",
"enum": [
"jsonl",
"parquet"
]
},
"compression": {
"type": [
"string",
"null"
],
"title": "Batch Compression Format",
"description": "Compression format to use for batch files.",
"enum": [
"gzip",
"none"
]
}
},
"title": "Batch Encoding Configuration",
"description": "Specifies the format and compression of the batch files."
},
"storage": {
"type": [
"object",
"null"
],
"properties": {
"root": {
"type": [
"string",
"null"
],
"title": "Batch Storage Root",
"description": "Root path to use when writing batch files."
},
"prefix": {
"type": [
"string",
"null"
],
"title": "Batch Storage Prefix",
"description": "Prefix to use when writing batch files."
}
},
"title": "Batch Storage Configuration",
"description": "Defines the storage layer to use when writing batch files"
}
},
"title": "Batch Configuration",
"description": "Configuration for BATCH message capabilities."
}
},
"$schema": "https://json-schema.org/draft/2020-12/schema"
}
}
edgarrmondragon
temporarily deployed
to
preview
GitHub Actions
Inactive
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.