-
Notifications
You must be signed in to change notification settings - Fork 63
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
docs: add Oracle Batch Query connector docs
Inspired by postgres docs
- Loading branch information
Showing
2 changed files
with
29 additions
and
1 deletion.
There are no files selected for viewing
1 change: 0 additions & 1 deletion
1
site/docs/reference/Connectors/capture-connectors/OracleDB/flashback.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
29 changes: 29 additions & 0 deletions
29
site/docs/reference/Connectors/capture-connectors/OracleDB/oracle-batch.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,29 @@ | ||
# Oracle Batch Query Connector | ||
|
||
This connector captures data from Oracle into Flow collections by periodically | ||
executing queries and translating the results into JSON documents. | ||
|
||
For local development or open-source workflows, [`ghcr.io/estuary/source-oracle-batch:dev`](https://ghcr.io/estuary/source-oracle-batch:dev) provides the latest version of the connector as a Docker image. You can also follow the link in your browser to see past image versions. | ||
|
||
We recommend using our [Oracle CDC Connector](http://go.estuary.dev/source-oracle) instead | ||
if possible. Using CDC provides lower latency data capture, delete and update events, and usually | ||
has a smaller impact on the source database. | ||
|
||
However there are some circumstances where this might not be feasible. Perhaps you need | ||
to capture from a managed Oracle instance which doesn't support logminer, has resource constraints | ||
which make logminer unviable or perhaps you need to capture the contents of a view or the result of an ad-hoc query. | ||
That's the sort of situation this connector is intended for. | ||
|
||
The number one caveat you need to be aware of when using this connector is that **it will | ||
periodically execute its update query over and over**. At the default polling interval of | ||
5 minutes, a naive `SELECT * FROM foo` query against a 100 MiB view will produce 30 GiB/day | ||
of ingested data, most of it duplicated. | ||
|
||
This is why the connector's autodiscovery logic only returns ordinary tables of data, because | ||
in that particular case we can use the `ORA_ROWSCN` system column as a cursor and ask the database | ||
to `SELECT ORA_ROWSCN, foo.* FROM foo WHERE ORA_ROWSCN > $1;`. | ||
|
||
If you start editing these queries or manually adding capture bindings for views or to run | ||
ad-hoc queries, you need to either have some way of restricting the query to "just the new | ||
rows since last time" or else have your polling interval set high enough that the data rate | ||
`<DatasetSize> / <PollingInterval>` is an amount of data you're willing to deal with. |