Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

java CDK: improve blobstore module structure #35285

Merged
merged 8 commits into from
Feb 15, 2024
Merged

java CDK: improve blobstore module structure #35285

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
a173b0b
java CDK: rename azure-blob-storage-destinations to azure-destinations
postamar Feb 14, 2024
b0189d3
gradle: remove :airbyte-integrations:bases from gradle
postamar Feb 14, 2024
6e6aa8a
flesh out gcs-destinations
postamar Feb 15, 2024
575848c
bump cdk version and update changelog
postamar Feb 15, 2024
732f3a8
destination bigquery and gcs use cdk 0.20.8
postamar Feb 15, 2024
ff96ef0
bump connector versions and changelogs
postamar Feb 15, 2024
1f10333
useLocalCdk = false
postamar Feb 15, 2024
e86ce13
fix destination-gcs
postamar Feb 15, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions airbyte-cdk/java/airbyte-cdk/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -166,6 +166,7 @@ MavenLocal debugging steps:

| Version | Date | Pull Request | Subject |
|:--------|:-----------|:-----------------------------------------------------------|:---------------------------------------------------------------------------------------------------------------------------------------------------------------|
| 0.20.8 | 2024-02-15 | [\#35285](https://github.com/airbytehq/airbyte/pull/35285) | Improve blobstore module structure. |
| 0.20.7 | 2024-02-13 | [\#35236](https://github.com/airbytehq/airbyte/pull/35236) | output logs to files in addition to stdout when running tests |
| 0.20.6 | 2024-02-12 | [\#35036](https://github.com/airbytehq/airbyte/pull/35036) | Add trace utility to emit analytics messages. |
| 0.20.5 | 2024-02-13 | [\#34869](https://github.com/airbytehq/airbyte/pull/34869) | Don't emit final state in SourceStateIterator there is an underlying stream failure. |
Expand Down
0 ...re-blob-storage-destinations/build.gradle → ...rbyte-cdk/azure-destinations/build.gradle
View file
Open in desktop
File renamed without changes.
0 ...bc/copy/azure/AzureBlobStorageConfig.java → ...bc/copy/azure/AzureBlobStorageConfig.java
View file
Open in desktop
File renamed without changes.
0 ...y/azure/AzureBlobStorageStreamCopier.java → ...y/azure/AzureBlobStorageStreamCopier.java
View file
Open in desktop
File renamed without changes.
0 .../AzureBlobStorageStreamCopierFactory.java → .../AzureBlobStorageStreamCopierFactory.java
View file
Open in desktop
File renamed without changes.
2 changes: 1 addition & 1 deletion airbyte-cdk/java/airbyte-cdk/core/src/main/resources/version.properties
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1 +1 @@
version=0.20.7
version=0.20.8
14 changes: 14 additions & 0 deletions airbyte-cdk/java/airbyte-cdk/gcs-destinations/build.gradle
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,20 @@
java {
// TODO: rewrite code to avoid javac wornings in the first place
compileJava {
options.compilerArgs += "-Xlint:-deprecation"
}
compileTestFixturesJava {
options.compilerArgs += "-Xlint:-deprecation"
}
}

dependencies {
implementation project(':airbyte-cdk:java:airbyte-cdk:dependencies')
implementation project(':airbyte-cdk:java:airbyte-cdk:core')
api project(':airbyte-cdk:java:airbyte-cdk:s3-destinations')

api 'com.google.cloud:google-cloud-storage:2.32.1'

testFixturesApi project(':airbyte-cdk:java:airbyte-cdk:s3-destinations')
testFixturesApi testFixtures(project(':airbyte-cdk:java:airbyte-cdk:s3-destinations'))
}
0 ...s/destination/gcs/BaseGcsDestination.java → ...s/destination/gcs/BaseGcsDestination.java
View file
Open in desktop
File renamed without changes.
0 ...destination/gcs/GcsDestinationConfig.java → ...destination/gcs/GcsDestinationConfig.java
View file
Open in desktop
File renamed without changes.
0 ...s/destination/gcs/GcsNameTransformer.java → ...s/destination/gcs/GcsNameTransformer.java
View file
Open in desktop
File renamed without changes.
0 ...destination/gcs/GcsStorageOperations.java → ...destination/gcs/GcsStorageOperations.java
View file
Open in desktop
File renamed without changes.
0 ...s/destination/gcs/avro/GcsAvroWriter.java → ...s/destination/gcs/avro/GcsAvroWriter.java
View file
Open in desktop
File renamed without changes.
0 ...n/gcs/credential/GcsCredentialConfig.java → ...n/gcs/credential/GcsCredentialConfig.java
View file
Open in desktop
File renamed without changes.
0 .../gcs/credential/GcsCredentialConfigs.java → .../gcs/credential/GcsCredentialConfigs.java
View file
Open in desktop
File renamed without changes.
0 ...ion/gcs/credential/GcsCredentialType.java → ...ion/gcs/credential/GcsCredentialType.java
View file
Open in desktop
File renamed without changes.
0 ...redential/GcsHmacKeyCredentialConfig.java → ...redential/GcsHmacKeyCredentialConfig.java
View file
Open in desktop
File renamed without changes.
0 ...ons/destination/gcs/csv/GcsCsvWriter.java → ...ons/destination/gcs/csv/GcsCsvWriter.java
View file
Open in desktop
File renamed without changes.
0 ...destination/gcs/jsonl/GcsJsonlWriter.java → ...destination/gcs/jsonl/GcsJsonlWriter.java
View file
Open in desktop
File renamed without changes.
0 ...ination/gcs/parquet/GcsParquetWriter.java → ...ination/gcs/parquet/GcsParquetWriter.java
View file
Open in desktop
File renamed without changes.
0 ...destination/gcs/util/GcsS3FileSystem.java → ...destination/gcs/util/GcsS3FileSystem.java
View file
Open in desktop
File renamed without changes.
0 ...ations/destination/gcs/util/GcsUtils.java → ...ations/destination/gcs/util/GcsUtils.java
View file
Open in desktop
File renamed without changes.
0 ...destination/gcs/writer/BaseGcsWriter.java → ...destination/gcs/writer/BaseGcsWriter.java
View file
Open in desktop
File renamed without changes.
0 ...ination/gcs/GcsDestinationConfigTest.java → ...ination/gcs/GcsDestinationConfigTest.java
View file
Open in desktop
File renamed without changes.
0 ...ion/gcs/avro/GcsAvroFormatConfigTest.java → ...ion/gcs/avro/GcsAvroFormatConfigTest.java
View file
Open in desktop
File renamed without changes.
0 ...stination/gcs/avro/GcsAvroWriterTest.java → ...stination/gcs/avro/GcsAvroWriterTest.java
View file
Open in desktop
File renamed without changes.
0 ...ation/gcs/csv/GcsCsvFormatConfigTest.java → ...ation/gcs/csv/GcsCsvFormatConfigTest.java
View file
Open in desktop
File renamed without changes.
0 ...n/gcs/jsonl/GcsJsonlFormatConfigTest.java → ...n/gcs/jsonl/GcsJsonlFormatConfigTest.java
View file
Open in desktop
File renamed without changes.
0 ...destination/gcs/util/ConfigTestUtils.java → ...destination/gcs/util/ConfigTestUtils.java
View file
Open in desktop
File renamed without changes.
0 ...tions/src/test/resources/test_config.json → ...tions/src/test/resources/test_config.json
View file
Open in desktop
File renamed without changes.
0 ...AvroParquetDestinationAcceptanceTest.java → ...AvroParquetDestinationAcceptanceTest.java
View file
Open in desktop
File renamed without changes.
0 ...nation/gcs/GcsAvroTestDataComparator.java → ...nation/gcs/GcsAvroTestDataComparator.java
View file
Open in desktop
File renamed without changes.
0 ...GcsBaseAvroDestinationAcceptanceTest.java → ...GcsBaseAvroDestinationAcceptanceTest.java
View file
Open in desktop
File renamed without changes.
0 .../GcsBaseCsvDestinationAcceptanceTest.java → .../GcsBaseCsvDestinationAcceptanceTest.java
View file
Open in desktop
File renamed without changes.
0 ...BaseCsvGzipDestinationAcceptanceTest.java → ...BaseCsvGzipDestinationAcceptanceTest.java
View file
Open in desktop
File renamed without changes.
0 ...csBaseJsonlDestinationAcceptanceTest.java → ...csBaseJsonlDestinationAcceptanceTest.java
View file
Open in desktop
File renamed without changes.
0 ...seJsonlGzipDestinationAcceptanceTest.java → ...seJsonlGzipDestinationAcceptanceTest.java
View file
Open in desktop
File renamed without changes.
0 ...BaseParquetDestinationAcceptanceTest.java → ...BaseParquetDestinationAcceptanceTest.java
View file
Open in desktop
File renamed without changes.
0 ...ion/gcs/GcsDestinationAcceptanceTest.java → ...ion/gcs/GcsDestinationAcceptanceTest.java
View file
Open in desktop
File renamed without changes.
38 changes: 19 additions & 19 deletions airbyte-cdk/java/airbyte-cdk/s3-destinations/build.gradle
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,32 +12,32 @@ java {
}

dependencies {
api 'com.amazonaws:aws-java-sdk-s3:1.12.647'

implementation project(':airbyte-cdk:java:airbyte-cdk:dependencies')
implementation project(':airbyte-cdk:java:airbyte-cdk:core')
implementation project(':airbyte-cdk:java:airbyte-cdk:typing-deduping')
implementation project(':airbyte-cdk:java:airbyte-cdk:db-destinations')

implementation ('com.github.airbytehq:json-avro-converter:1.1.0') { exclude group: 'ch.qos.logback', module: 'logback-classic'}
implementation 'com.github.alexmojaki:s3-stream-upload:2.2.4'
implementation 'org.apache.avro:avro:1.11.3'
implementation 'org.apache.commons:commons-csv:1.10.0'
implementation 'org.apache.commons:commons-text:1.11.0'
implementation ('org.apache.hadoop:hadoop-aws:3.3.6') { exclude group: 'com.amazonaws', module: 'aws-java-sdk-bundle' }
implementation 'org.apache.hadoop:hadoop-common:3.3.6'
implementation 'org.apache.hadoop:hadoop-mapreduce-client-core:3.3.6'
implementation 'org.apache.parquet:parquet-avro:1.13.1'
// Re-export dependencies for gcs-destinations.
api 'com.amazonaws:aws-java-sdk-s3:1.12.647'
api ('com.github.airbytehq:json-avro-converter:1.1.0') { exclude group: 'ch.qos.logback', module: 'logback-classic'}
api 'com.github.alexmojaki:s3-stream-upload:2.2.4'
api 'org.apache.avro:avro:1.11.3'
api 'org.apache.commons:commons-csv:1.10.0'
api 'org.apache.commons:commons-text:1.11.0'
api ('org.apache.hadoop:hadoop-aws:3.3.6') { exclude group: 'com.amazonaws', module: 'aws-java-sdk-bundle' }
api 'org.apache.hadoop:hadoop-common:3.3.6'
api 'org.apache.hadoop:hadoop-mapreduce-client-core:3.3.6'
api 'org.apache.parquet:parquet-avro:1.13.1'
runtimeOnly 'com.hadoop.gplcompression:hadoop-lzo:0.4.20'

testImplementation 'org.mockito:mockito-inline:5.2.0'

testFixturesImplementation project(':airbyte-cdk:java:airbyte-cdk:dependencies')
testFixturesImplementation testFixtures(project(':airbyte-cdk:java:airbyte-cdk:dependencies'))
testFixturesImplementation project(':airbyte-cdk:java:airbyte-cdk:core')
testFixturesImplementation testFixtures(project(':airbyte-cdk:java:airbyte-cdk:core'))
testFixturesImplementation project(':airbyte-cdk:java:airbyte-cdk:typing-deduping')
testFixturesImplementation testFixtures(project(':airbyte-cdk:java:airbyte-cdk:typing-deduping'))
testFixturesImplementation project(':airbyte-cdk:java:airbyte-cdk:db-destinations')
testFixturesImplementation testFixtures(project(':airbyte-cdk:java:airbyte-cdk:db-destinations'))
testFixturesApi project(':airbyte-cdk:java:airbyte-cdk:dependencies')
testFixturesApi testFixtures(project(':airbyte-cdk:java:airbyte-cdk:dependencies'))
testFixturesApi project(':airbyte-cdk:java:airbyte-cdk:core')
testFixturesApi testFixtures(project(':airbyte-cdk:java:airbyte-cdk:core'))
testFixturesApi project(':airbyte-cdk:java:airbyte-cdk:typing-deduping')
testFixturesApi testFixtures(project(':airbyte-cdk:java:airbyte-cdk:typing-deduping'))
testFixturesApi project(':airbyte-cdk:java:airbyte-cdk:db-destinations')
testFixturesApi testFixtures(project(':airbyte-cdk:java:airbyte-cdk:db-destinations'))
}
2 changes: 1 addition & 1 deletion airbyte-cdk/java/airbyte-cdk/settings.gradle
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ include ':airbyte-cdk:java:airbyte-cdk:db-sources'
include ':airbyte-cdk:java:airbyte-cdk:db-destinations'
include ':airbyte-cdk:java:airbyte-cdk:s3-destinations'
include ':airbyte-cdk:java:airbyte-cdk:typing-deduping'
include ':airbyte-cdk:java:airbyte-cdk:azure-blob-storage-destinations'
include ':airbyte-cdk:java:airbyte-cdk:azure-destinations'
include ':airbyte-cdk:java:airbyte-cdk:gcs-destinations'
include ':airbyte-cdk:java:airbyte-cdk:datastore-bigquery'
include ':airbyte-cdk:java:airbyte-cdk:datastore-mongo'
Expand Down
7 changes: 3 additions & 4 deletions airbyte-integrations/connectors/destination-bigquery/build.gradle
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,13 +3,12 @@ plugins {
}

airbyteJavaConnector {
cdkVersionRequired = '0.20.2'
cdkVersionRequired = '0.20.8'
features = [
'datastore-bigquery',
'db-destinations',
'gcs-destinations',
's3-destinations',
'datastore-bigquery',
'typing-deduping',
'gcs-destinations',
]
useLocalCdk = false
}
Expand Down
2 changes: 1 addition & 1 deletion airbyte-integrations/connectors/destination-bigquery/metadata.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ data:
connectorSubtype: database
connectorType: destination
definitionId: 22f6c74f-5699-40ff-833c-4a879ea40133
dockerImageTag: 2.4.8
dockerImageTag: 2.4.9
dockerRepository: airbyte/destination-bigquery
documentationUrl: https://docs.airbyte.com/integrations/destinations/bigquery
githubIssueLabel: destination-bigquery
Expand Down
4 changes: 2 additions & 2 deletions airbyte-integrations/connectors/destination-gcs/build.gradle
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,8 +3,8 @@ plugins {
}

airbyteJavaConnector {
cdkVersionRequired = '0.19.0'
features = ['db-destinations', 's3-destinations', 'gcs-destinations']
cdkVersionRequired = '0.20.8'
features = ['db-destinations', 'gcs-destinations']
useLocalCdk = false
}

Expand Down
2 changes: 1 addition & 1 deletion airbyte-integrations/connectors/destination-gcs/metadata.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ data:
connectorSubtype: file
connectorType: destination
definitionId: ca8f6566-e555-4b40-943a-545bf123117a
dockerImageTag: 0.4.5
dockerImageTag: 0.4.6
dockerRepository: airbyte/destination-gcs
githubIssueLabel: destination-gcs
icon: googlecloudstorage.svg
Expand Down
1 change: 1 addition & 0 deletions docs/integrations/destinations/bigquery.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -210,6 +210,7 @@ tutorials:

| Version | Date | Pull Request | Subject |
|:--------|:-----------|:-----------------------------------------------------------|:----------------------------------------------------------------------------------------------------------------------------------------------------------------|
| 2.4.9 | 2024-02-15 | [35285](https://github.com/airbytehq/airbyte/pull/35285) | Adopt CDK 0.20.8 |
| 2.4.8 | 2024-02-12 | [35144](https://github.com/airbytehq/airbyte/pull/35144) | Adopt CDK 0.20.2 |
| 2.4.7 | 2024-02-12 | [35111](https://github.com/airbytehq/airbyte/pull/35111) | Adopt CDK 0.20.1 |
| 2.4.6 | 2024-02-09 | [34575](https://github.com/airbytehq/airbyte/pull/34575) | Adopt CDK 0.20.0 |
Expand Down
1 change: 1 addition & 0 deletions docs/integrations/destinations/gcs.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -237,6 +237,7 @@ Under the hood, an Airbyte data stream in Json schema is first converted to an A
| Version | Date | Pull Request | Subject |
| :------ | :--------- | :--------------------------------------------------------- | :------------------------------------------------------------------------------------------------------------------------- |
| 0.4.6 | 2024-02-15 | [35285](https://github.com/airbytehq/airbyte/pull/35285) | Adopt CDK 0.20.8 |
| 0.4.5 | 2024-02-08 | [34745](https://github.com/airbytehq/airbyte/pull/34745) | Adopt CDK 0.19.0 |
| 0.4.4 | 2023-07-14 | [#28345](https://github.com/airbytehq/airbyte/pull/28345) | Increment patch to trigger a rebuild |
| 0.4.3 | 2023-07-05 | [#27936](https://github.com/airbytehq/airbyte/pull/27936) | Internal code update |
Expand Down
5 changes: 1 addition & 4 deletions settings.gradle
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -169,15 +169,12 @@ include ':airbyte-cdk:java:airbyte-cdk:db-destinations'
include ':airbyte-cdk:java:airbyte-cdk:db-sources'
include ':airbyte-cdk:java:airbyte-cdk:s3-destinations'
include ':airbyte-cdk:java:airbyte-cdk:typing-deduping'
include ':airbyte-cdk:java:airbyte-cdk:azure-blob-storage-destinations'
include ':airbyte-cdk:java:airbyte-cdk:azure-destinations'
include ':airbyte-cdk:java:airbyte-cdk:gcs-destinations'
include ':airbyte-cdk:java:airbyte-cdk:datastore-bigquery'
include ':airbyte-cdk:java:airbyte-cdk:datastore-mongo'
include ':airbyte-cdk:java:airbyte-cdk:datastore-postgres'

include ':airbyte-integrations:bases:base'
include ':airbyte-integrations:bases:base-java'
include ':airbyte-integrations:bases:base-normalization'
include ':airbyte-integrations:connector-templates:generator'
include ':airbyte-integrations:connectors-performance:source-harness'
include ':airbyte-integrations:connectors-performance:destination-harness'
Expand Down
Loading