Destination BigQuery + Snowflake : Disable type and dedupe step controlled by flag in Connector spec #31686

gisripa · 2023-10-20T23:01:18Z

What

Adds ability to disable generating the final Typed+Deduped tables through a flag in ConnectorSpecification.

Other minor fixes

closes #11122 - Copied contents from https://github.com/airbytehq/airbyte-platform/blob/v0.43.2/airbyte-webapp/docs/HowTo-ConnectionSpecification.md
closes #31720 - Pending GSM credentials updates. local tests pass with minor fixes

BigQuery UI

Snowflake UI

vercel · 2023-10-20T23:01:24Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
airbyte-docs	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Oct 26, 2023 8:33pm

github-actions · 2023-10-20T23:01:41Z

Before Merging a Connector Pull Request

Wow! What a great pull request you have here! 🎉

To merge this PR, ensure the following has been done/considered for each connector added or updated:

PR name follows PR naming conventions
Breaking changes are considered. If a Breaking Change is being introduced, ensure an Airbyte engineer has created a Breaking Change Plan.
Connector version has been incremented in the Dockerfile and metadata.yaml according to our Semantic Versioning for Connectors guidelines
You've updated the connector's metadata.yaml file any other relevant changes, including a breakingChanges entry for major version bumps. See metadata.yaml docs
Secrets in the connector's spec are annotated with airbyte_secret
All documentation files are up to date. (README.md, bootstrap.md, docs.md, etc...)
Changelog updated in docs/integrations/<source or destination>/<name>.md with an entry for the new version. See changelog example
Migration guide updated in docs/integrations/<source or destination>/<name>-migrations.md with an entry for the new version, if the version is a breaking change. See migration guide example
If set, you've ensured the icon is present in the platform-internal repo. (Docs)

If the checklist is complete, but the CI check is failing,

Check for hidden checklists in your PR description
Toggle the github label checklist-action-run on/off to re-run the checklist CI.

evantahler · 2023-10-20T23:45:36Z

airbyte-integrations/connectors/destination-bigquery/src/main/resources/spec.json

+        "type": "boolean",
+        "default": false,
+        "description": "Disable Writing Final Tables. Warning! The data format in _airbyte_data is likely stable but there are no guarantees that other metadata columns will remain the same in future versions",
+        "title": "Disable Final Tables. (Warning! Unstable option; Columns in raw table schema might change between versions)",


@Hesperide - is this message a good enough deterrent?

evantahler · 2023-10-23T15:02:01Z

docs/connector-development/connector-specification-reference.md

+While iterating on your specification, you can preview what it will look like in the UI in realtime by following the instructions below.
+1. Open the `ConnectorForm` preview component in our deployed Storybook at: https://components.airbyte.dev/?path=/story/connector-connectorform--preview
+2. Press `raw` on the `connectionSpecification` property, so you will be able to paste a JSON structured string
+3. Set the string you want to preview the UI for
+4. When submitting the form you can see a preview of the values in the "Actions" tab


Nice addition! Be sure to share this at our next team demos, or make a loom showing this off.

zero credit to me, just resurrected contents from a deleted file and inlined them :D

airbyte-oss-build-runner · 2023-10-24T03:44:51Z

destination-bigquery test report (commit `e005987ae1`) - ❌

⏲️ Total pipeline duration: 11mn29s

Step	Result
Build connector tar	✅
Java Connector Unit Tests	✅
Build destination-bigquery docker image for platform(s) linux/amd64	✅
Java Connector Integration Tests	❌
Validate metadata for destination-bigquery	✅
Connector version semver check	✅
Connector version increment check	❌
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-bigquery test

airbyte-oss-build-runner · 2023-10-24T03:52:47Z

destination-snowflake test report (commit `e005987ae1`) - ❌

⏲️ Total pipeline duration: 07mn45s

Step	Result
Build connector tar	✅
Java Connector Unit Tests	✅
Build destination-snowflake docker image for platform(s) linux/amd64	✅
Java Connector Integration Tests	✅
Validate metadata for destination-snowflake	✅
Connector version semver check	✅
Connector version increment check	❌
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-snowflake test

airbyte-oss-build-runner · 2023-10-24T16:35:06Z

destination-snowflake test report (commit `34e72b97c9`) - ❌

⏲️ Total pipeline duration: 10mn25s

Step	Result
Build connector tar	✅
Java Connector Unit Tests	✅
Build destination-snowflake docker image for platform(s) linux/amd64	✅
Java Connector Integration Tests	✅
Validate metadata for destination-snowflake	✅
Connector version semver check	✅
Connector version increment check	❌
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-snowflake test

airbyte-oss-build-runner · 2023-10-24T16:42:36Z

destination-bigquery test report (commit `34e72b97c9`) - ❌

⏲️ Total pipeline duration: 07mn18s

Step	Result
Build connector tar	✅
Java Connector Unit Tests	✅
Build destination-bigquery docker image for platform(s) linux/amd64	✅
Java Connector Integration Tests	✅
Validate metadata for destination-bigquery	✅
Connector version semver check	✅
Connector version increment check	❌
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-bigquery test

edgao

had some minor comments but generally looks good! very annoyed by how github has decided to show this diff but not sure how to fix it without doing actual git stuff

...in/java/io/airbyte/integrations/base/destination/typing_deduping/BaseTypingDedupingTest.java

...ns/destination/bigquery/typing_deduping/BigQueryGcsRawOverrideDisableTypingDedupingTest.java

...irbyte/integrations/base/destination/typing_deduping/NoOpTyperDeduperWithV1V2Migrations.java

airbyte-oss-build-runner · 2023-10-25T02:21:39Z

destination-bigquery test report (commit `db3d1169db`) - ❌

⏲️ Total pipeline duration: 09mn39s

Step	Result
Build connector tar	✅
Java Connector Unit Tests	✅
Build destination-bigquery docker image for platform(s) linux/amd64	✅
Java Connector Integration Tests	✅
Validate metadata for destination-bigquery	✅
Connector version semver check	✅
Connector version increment check	❌
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-bigquery test

airbyte-oss-build-runner · 2023-10-25T02:29:20Z

destination-snowflake test report (commit `db3d1169db`) - ❌

⏲️ Total pipeline duration: 07mn30s

Step	Result
Build connector tar	✅
Java Connector Unit Tests	✅
Build destination-snowflake docker image for platform(s) linux/amd64	✅
Java Connector Integration Tests	✅
Validate metadata for destination-snowflake	✅
Connector version semver check	✅
Connector version increment check	❌
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-snowflake test

airbyte-oss-build-runner · 2023-10-25T17:23:43Z

destination-snowflake test report (commit `af9623b8c4`) - ❌

⏲️ Total pipeline duration: 08mn42s

Step	Result
Build connector tar	✅
Java Connector Unit Tests	✅
Build destination-snowflake docker image for platform(s) linux/amd64	✅
Java Connector Integration Tests	✅
Validate metadata for destination-snowflake	✅
Connector version semver check	✅
Connector version increment check	❌
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-snowflake test

airbyte-oss-build-runner · 2023-10-25T17:32:21Z

destination-bigquery test report (commit `af9623b8c4`) - ❌

⏲️ Total pipeline duration: 08mn27s

Step	Result
Build connector tar	✅
Java Connector Unit Tests	✅
Build destination-bigquery docker image for platform(s) linux/amd64	✅
Java Connector Integration Tests	✅
Validate metadata for destination-bigquery	✅
Connector version semver check	✅
Connector version increment check	❌
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-bigquery test

Signed-off-by: Gireesh Sreepathi <[email protected]>

airbyte-oss-build-runner · 2023-10-26T19:47:15Z

destination-bigquery test report (commit `6698ae592a`) - ✅

⏲️ Total pipeline duration: 10mn50s

Step	Result
Build connector tar	✅
Java Connector Unit Tests	✅
Build destination-bigquery docker image for platform(s) linux/amd64	✅
Java Connector Integration Tests	✅
Validate metadata for destination-bigquery	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-bigquery test

airbyte-oss-build-runner · 2023-10-26T19:56:26Z

destination-snowflake test report (commit `6698ae592a`) - ✅

⏲️ Total pipeline duration: 08mn59s

Step	Result
Build connector tar	✅
Java Connector Unit Tests	✅
Build destination-snowflake docker image for platform(s) linux/amd64	✅
Java Connector Integration Tests	✅
Validate metadata for destination-snowflake	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-snowflake test

docs/release_notes/upgrading_to_destinations_v2.md

Co-authored-by: Evan Tahler <[email protected]>

gisripa requested a review from a team as a code owner October 20, 2023 23:01

gisripa marked this pull request as draft October 20, 2023 23:01

octavia-squidington-iii added the area/connectors Connector related issues label Oct 20, 2023

octavia-squidington-iii added connectors/destination/bigquery connectors/destination/snowflake labels Oct 20, 2023

evantahler reviewed Oct 20, 2023

View reviewed changes

octavia-squidington-iii added the area/documentation Improvements or additions to documentation label Oct 21, 2023

vercel bot deployed to Preview October 21, 2023 00:45 View deployment

evantahler reviewed Oct 23, 2023

View reviewed changes

gisripa force-pushed the gireesh/30454 branch from 83c2127 to 3c0a72d Compare October 23, 2023 23:40

gisripa changed the title ~~[WIP] Destination BigQuery + Snowflake : Disable type and dedupe step controlled by flag in Connector spec~~ Destination BigQuery + Snowflake : Disable type and dedupe step controlled by flag in Connector spec Oct 24, 2023

gisripa marked this pull request as ready for review October 24, 2023 03:31

gisripa requested a review from a team October 24, 2023 03:34

gisripa self-assigned this Oct 24, 2023

gisripa changed the base branch from master to edgao/skip_raw_dedup October 24, 2023 18:58

edgao reviewed Oct 24, 2023

View reviewed changes

octavia-squidington-iv requested a review from a team October 24, 2023 21:35

gisripa changed the base branch from edgao/skip_raw_dedup to master October 25, 2023 02:31

gisripa force-pushed the gireesh/30454 branch from db3d116 to af9623b Compare October 25, 2023 17:13

gisripa removed the request for review from a team October 25, 2023 22:36

gisripa and others added 21 commits October 26, 2023 12:33

unused imports and unthrown exceptions

751fe17

Signed-off-by: Gireesh Sreepathi <[email protected]>

wire the NoOpTypeDedupe conditionally

f9239be

Signed-off-by: Gireesh Sreepathi <[email protected]>

Automated Commit - Formatting Changes

abf286e

Fixes #11122

7a9c002

Signed-off-by: Gireesh Sreepathi <[email protected]>

Extend tests to verify raw table only inserts

a1b4262

Signed-off-by: Gireesh Sreepathi <[email protected]>

add log stmts for standard consumer

ca640d5

Signed-off-by: Gireesh Sreepathi <[email protected]>

stuff

d8fc7d0

test cleanup

c917c54

test cleanup

0eb8d41

Automated Commit - Formatting Changes

0685074

Working integration tests with skip_raw_dedupe merge

aee046e

Signed-off-by: Gireesh Sreepathi <[email protected]>

Automated Commit - Formatting Changes

34997cc

override tests

c9538d9

Signed-off-by: Gireesh Sreepathi <[email protected]>

Add skipTD impl for snowflake

0851874

Signed-off-by: Gireesh Sreepathi <[email protected]>

Automated Commit - Formatting Changes

c1f7a68

Fixes 1st_1t removal failed tests

e24ee1b

Signed-off-by: Gireesh Sreepathi <[email protected]>

Automated Commit - Formatting Changes

38cc3b4

pr comments

37189e8

Signed-off-by: Gireesh Sreepathi <[email protected]>

bump connector versions

958d30c

Signed-off-by: Gireesh Sreepathi <[email protected]>

remove redundant assertion

299ad75

Signed-off-by: Gireesh Sreepathi <[email protected]>

docs update

6698ae5

Signed-off-by: Gireesh Sreepathi <[email protected]>

gisripa force-pushed the gireesh/30454 branch from f07d590 to 6698ae5 Compare October 26, 2023 19:34

vercel bot deployed to Preview October 26, 2023 19:35 View deployment

evantahler reviewed Oct 26, 2023

View reviewed changes

docs/release_notes/upgrading_to_destinations_v2.md Outdated Show resolved Hide resolved

Update docs/release_notes/upgrading_to_destinations_v2.md

c5dba08

Co-authored-by: Evan Tahler <[email protected]>

vercel bot deployed to Preview October 26, 2023 20:33 View deployment

gisripa merged commit 3e76471 into master Oct 26, 2023
15 checks passed

gisripa deleted the gireesh/30454 branch October 26, 2023 20:46

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Destination BigQuery + Snowflake : Disable type and dedupe step controlled by flag in Connector spec #31686

Destination BigQuery + Snowflake : Disable type and dedupe step controlled by flag in Connector spec #31686

gisripa commented Oct 20, 2023 •

edited

Loading

vercel bot commented Oct 20, 2023 •

edited

Loading

github-actions bot commented Oct 20, 2023 •

edited by gisripa

Loading

evantahler Oct 20, 2023

evantahler Oct 23, 2023

gisripa Oct 23, 2023 •

edited

Loading

airbyte-oss-build-runner commented Oct 24, 2023

airbyte-oss-build-runner commented Oct 24, 2023

airbyte-oss-build-runner commented Oct 24, 2023

airbyte-oss-build-runner commented Oct 24, 2023

edgao left a comment

airbyte-oss-build-runner commented Oct 25, 2023

airbyte-oss-build-runner commented Oct 25, 2023

airbyte-oss-build-runner commented Oct 25, 2023

airbyte-oss-build-runner commented Oct 25, 2023

airbyte-oss-build-runner commented Oct 26, 2023

airbyte-oss-build-runner commented Oct 26, 2023

Destination BigQuery + Snowflake : Disable type and dedupe step controlled by flag in Connector spec #31686

Destination BigQuery + Snowflake : Disable type and dedupe step controlled by flag in Connector spec #31686

Conversation

gisripa commented Oct 20, 2023 • edited Loading

What

Other minor fixes

BigQuery UI

Snowflake UI

vercel bot commented Oct 20, 2023 • edited Loading

github-actions bot commented Oct 20, 2023 • edited by gisripa Loading

Before Merging a Connector Pull Request

evantahler Oct 20, 2023

Choose a reason for hiding this comment

evantahler Oct 23, 2023

Choose a reason for hiding this comment

gisripa Oct 23, 2023 • edited Loading

Choose a reason for hiding this comment

airbyte-oss-build-runner commented Oct 24, 2023

destination-bigquery test report (commit e005987ae1) - ❌

airbyte-oss-build-runner commented Oct 24, 2023

destination-snowflake test report (commit e005987ae1) - ❌

airbyte-oss-build-runner commented Oct 24, 2023

destination-snowflake test report (commit 34e72b97c9) - ❌

airbyte-oss-build-runner commented Oct 24, 2023

destination-bigquery test report (commit 34e72b97c9) - ❌

edgao left a comment

Choose a reason for hiding this comment

airbyte-oss-build-runner commented Oct 25, 2023

destination-bigquery test report (commit db3d1169db) - ❌

airbyte-oss-build-runner commented Oct 25, 2023

destination-snowflake test report (commit db3d1169db) - ❌

airbyte-oss-build-runner commented Oct 25, 2023

destination-snowflake test report (commit af9623b8c4) - ❌

airbyte-oss-build-runner commented Oct 25, 2023

destination-bigquery test report (commit af9623b8c4) - ❌

airbyte-oss-build-runner commented Oct 26, 2023

destination-bigquery test report (commit 6698ae592a) - ✅

airbyte-oss-build-runner commented Oct 26, 2023

destination-snowflake test report (commit 6698ae592a) - ✅

gisripa commented Oct 20, 2023 •

edited

Loading

vercel bot commented Oct 20, 2023 •

edited

Loading

github-actions bot commented Oct 20, 2023 •

edited by gisripa

Loading

gisripa Oct 23, 2023 •

edited

Loading

destination-bigquery test report (commit `e005987ae1`) - ❌

destination-snowflake test report (commit `e005987ae1`) - ❌

destination-snowflake test report (commit `34e72b97c9`) - ❌

destination-bigquery test report (commit `34e72b97c9`) - ❌

destination-bigquery test report (commit `db3d1169db`) - ❌

destination-snowflake test report (commit `db3d1169db`) - ❌

destination-snowflake test report (commit `af9623b8c4`) - ❌

destination-bigquery test report (commit `af9623b8c4`) - ❌

destination-bigquery test report (commit `6698ae592a`) - ✅

destination-snowflake test report (commit `6698ae592a`) - ✅