Enhance Google Drive Connector for Multi-Folder, Multi-Account, and Concurrent Processing

[Sakiruto]

The current Google Drive Airbyte Connector is limited to syncing a single folder per connection and supports only one Google Drive account. This makes it inefficient for users needing to sync multiple folders, specific files, and multiple Google Drive accounts. Additionally, the lack of built-in concurrency results in slow data extraction.

This proposal aims to enhance the Google Drive Connector to:

• Support multiple folders and files per connection.
• Enable syncing data from multiple Google Drive accounts.
• Introduce configurable concurrency settings for parallel processing.

Current Limitations

1. Single Folder Restriction

• Each Airbyte connection only supports one folder_id.
• Users requiring data from multiple folders must run the code dynamically multiple times (which is not optimal)

2. Single Google Drive Account Limitation

• Each connection is tied to a single Google Drive account via an OAuth token.
• Users needing to sync data from multiple Google Drives must manually create multiple connections.

3. No Support for Multi-File Syncing

• The connector only supports fetching entire folders.
• There is no built-in mechanism to specify and sync specific files dynamically from within the Airbyte UI.

4. No Built-in Concurrency or Parallel Processing

• Data extraction happens sequentially, making the sync process slow.
• Users lack control over resource allocation for parallel sync operations.

Proposed Enhancements

1. Multi-Folder & Multi-File Syncing

• Modify the connector to allow users to specify multiple folders and individual files per connection.
• Introduce configuration options for selecting specific files along with folders.

2. Multi-Account Support

• Enable the use of multiple OAuth tokens in a single connection
• Provide a mechanism to switch between multiple Google Drive accounts dynamically.

3. Configurable Concurrency

• Introduce an option in the Airbyte UI to specify the number of parallel threads for data extraction.
• Optimize API requests to reduce latency and improve sync speeds.

Why This Matters

This enhancement is a major requirement for our startup, as we deal with multiple data sources across various Google Drive accounts. Contributing these improvements to Airbyte will not only help our use case but also benefit the broader Airbyte community.

I would love to collaborate with the maintainers to contribute these features and make the Google Drive connector more powerful and efficient. Looking forward to discussing the feasibility and roadmap for these enhancements.

Would love to hear your thoughts on how best to move forward!
Note: I have used a bit of AI assistance to structure the issue content for clarity and completeness.