diff --git a/connectors/aws/aws-s3/LICENSE.txt b/connectors/aws/aws-s3/LICENSE.txt new file mode 100644 index 0000000000..85fdd16e79 --- /dev/null +++ b/connectors/aws/aws-s3/LICENSE.txt @@ -0,0 +1,5 @@ +Copyright Camunda Services GmbH and/or licensed to Camunda Services GmbH under one or more contributor license agreements and licensed to you under a proprietary license. +You may not use this file except in compliance with the proprietary license. +The proprietary license can be either the Camunda Self-Managed Free Edition license (available on Camunda’s website) or the Camunda Self-Managed Enterprise Edition license (a copy you obtain when you contact Camunda). +The Camunda Self-Managed Free Edition comes for free but only allows for usage of the software (file) in non-production environments. +If you want to use the software (file) in production, you need to purchase the Camunda Self-Managed Enterprise Edition. \ No newline at end of file diff --git a/connectors/aws/aws-s3/README.md b/connectors/aws/aws-s3/README.md new file mode 100644 index 0000000000..1811d34698 --- /dev/null +++ b/connectors/aws/aws-s3/README.md @@ -0,0 +1,152 @@ +# AWS S3 Connector + +Camunda Outbound Connector to interact with the content of an S3 bucket + +DISCLAIMER: You are responsible for your AWS configuration in your environment, keep in mind that you have to make +sure that you keep credentials in a safe place and only give access to specific resources, and be as restrictive as +possible. This is not a security tutorial for AWS. You should definitively know what you are doing! + +## Compatibility + +- JDK 21+ +- Camunda Platform v8.7.x +- Connector SDK v8.7.x +- AWS SDK v2.x + +## Features + +- Upload a generated file to an AWS S3 bucket +- Delete a file from an AWS S3 bucket +- Download a file from an AWS S3 bucket +- Files are saved in the local filesystem to allow interaction between activities + + +## Setup + +### Connector configuration + +| name | description | example | +|-----------------|--------------------------------------------|-----------------------------| +| `accessKey` | the AWS access key for the authorized user | `secrets.AWS_ACCESS_KEY` | +| `secretKey` | the AWS secret key for the authorized user | `secrets.AWS_SECRET_KEY` | +| `region` | the AWS region of your S3 bucket | eu-central-1 | +| `bucketName` | the name of the S3 bucket | camunda-s3-connector-bucket | +| `objectKey` | path + file name in the s3 bucket | `="invoice/"+fileName` | +| `operationType` | what to do on s3 | `PUT_OBJECT` | +| `filePath` | absolute path to the file to upload | `=filePath` | +| `contentType` | the content type of the content | `=contentType` | + +NOTE: please do not put secrets directly into your configuration. See the secrets section for more details. + +#### How it looks in the modeler +how it looks like in the modeler + +### required AWS Resources +- S3 bucket (non-public) with server-side encryption and versioning enabled +- IAM User with putObject, deleteObject and getObject rights to the bucket +- Access key and Secret key for the IAM user + +### create AWS Resources if not yet present +#### 1. create AWS S3 Bucket: + First, you need to log into the AWS management console and navigate to S3. + There you can create a new bucket by setting your AWS region and a unique name. + + +Moreover, making sure that the “Block all public access” option is enabled to keep the bucket private. + + +#### 2. create IAM User and Access/Security Key +Once the bucket has been set up, you must create an IAM (Identity and Access Management) user in the IAM area. +To do this, navigate to IAM > Users and add a new user. + +##### Get credentials from AWS + +In order to access AWS from your connector you need the above mentioned user with a IAM policy for S3. Additionally +you need to generated credentials for the user. You can do this over the AWS console in IAM: + +- Log into your AWS console +- Go to IAM > Users +- Select your created user, e.g. `camunda-s3-connector-user` +- Go to Security credentials > Access keys +- Click `Create access key` +- Select `Third-party service` +- Check that you understand the risk of having a permanent access key +- add a tag if wished +- Click on create and save your credentials in a save place + +ATTENTION: There are many ways to achieve this in IAM, I only describe the easiest, +but possibly not the safest way to do it + + +#### 3. IAM policy + +Next, you create an IAM policy that grants the user authorization to upload, delete and retrieve objects in the bucket. +To do this, you must create an IAM policy in the IAM area and create a JSON policy with the corresponding authorizations. + +```json +{ + "Version": "2012-10-17", + "Statement": [ + { + "Sid": "AllowPutAndDeleteInS3", + "Effect": "Allow", + "Action": [ + "s3:PutObject", + "s3:DeleteObject", + "s3:GetObject" + ], + "Resource": "arn:aws:s3:::/*" + } + ] +} +``` +Afterwards, you need to add the created policy to your previously created IAM user. + +## Runtime + +### Running the connector + +- You can run it as a standalone connector described here: [run a standalone connector](https://github.com/NovatecConsulting/camunda-aws-s3-connector/tree/main/connector-aws-s3-standalone) +- Or you can run it together with some JobWorkers described here: [run a connector](https://github.com/NovatecConsulting/camunda-aws-s3-connector/tree/main/connector-aws-s3-example) + +### Handling secrets +Since your connector needs to run in a custom connector runtime, you cannot just add secrets in the cloud console since +they will not be auto-magically transported into your connector runtime. You can provide them by: + +- adding them as environment variables (e.g. when you use the SpringBoot connector runtime) +- adding them as an env file to your docker container + +NOTE: This behaviour will most likely be improved in the future + +## File Handling + +The connector has two file adapters: + +- *cloud file adapter* to S3 +- *local file adapter* to the local file system + +They are implementations of the [Connector File API](./fileapi) + +### Why is this necessary? +If you are handling a lot of files (maybe even big files) it is a best practice approach to NOT store your file or the content +in a process variable for several reasons: + +- Zeebe currently only support variables < 2MB +- The files become part of the process state +- You have no way to clean it up (yet) + +With the local file adapter the file can be written to a temp directory and the file path can be handed to other activities. +If you want to move files to another process instance, just upload it back to S3 and start another process with the +input variables needed to download the file again from S3. + +### Current Restrictions + +It is currently not possible to receive process instance specific variables like the process instance key in the connector +or with a feel expression. Both improvements exist as a feature request. The only way to get an instance key is by calling +the value from an activated job in a JobWorker and setting it as a process variable for others to pick up. + +NOTE: the process instance key is used for generating file paths so local and remote files are not overwritten by other +instances + +### This connector was initially developed by the BPM team at [Novatec Consulting GmbH](https://www.novatec-gmbh.de) + \ No newline at end of file diff --git a/connectors/aws/aws-s3/assets/awsScreenshots/screenshot_bucket_creation.png b/connectors/aws/aws-s3/assets/awsScreenshots/screenshot_bucket_creation.png new file mode 100644 index 0000000000..222fba80c7 Binary files /dev/null and b/connectors/aws/aws-s3/assets/awsScreenshots/screenshot_bucket_creation.png differ diff --git a/connectors/aws/aws-s3/assets/awsScreenshots/screenshot_public_access.png b/connectors/aws/aws-s3/assets/awsScreenshots/screenshot_public_access.png new file mode 100644 index 0000000000..ffe21c5eec Binary files /dev/null and b/connectors/aws/aws-s3/assets/awsScreenshots/screenshot_public_access.png differ diff --git a/connectors/aws/aws-s3/assets/camundaScreenshots/screenshot_client_properties.png b/connectors/aws/aws-s3/assets/camundaScreenshots/screenshot_client_properties.png new file mode 100644 index 0000000000..9cf814ac8d Binary files /dev/null and b/connectors/aws/aws-s3/assets/camundaScreenshots/screenshot_client_properties.png differ diff --git a/connectors/aws/aws-s3/assets/camundaScreenshots/screenshot_clustertab.png b/connectors/aws/aws-s3/assets/camundaScreenshots/screenshot_clustertab.png new file mode 100644 index 0000000000..bad67c218b Binary files /dev/null and b/connectors/aws/aws-s3/assets/camundaScreenshots/screenshot_clustertab.png differ diff --git a/connectors/aws/aws-s3/assets/connector-config-example.png b/connectors/aws/aws-s3/assets/connector-config-example.png new file mode 100644 index 0000000000..fea9ea23bf Binary files /dev/null and b/connectors/aws/aws-s3/assets/connector-config-example.png differ diff --git a/connectors/aws/aws-s3/assets/connector-file-api.png b/connectors/aws/aws-s3/assets/connector-file-api.png new file mode 100644 index 0000000000..4675fabb28 Binary files /dev/null and b/connectors/aws/aws-s3/assets/connector-file-api.png differ diff --git a/connectors/aws/aws-s3/assets/example-process.png b/connectors/aws/aws-s3/assets/example-process.png new file mode 100644 index 0000000000..8c10f1b5ee Binary files /dev/null and b/connectors/aws/aws-s3/assets/example-process.png differ diff --git a/connectors/aws/aws-s3/assets/novatec.png b/connectors/aws/aws-s3/assets/novatec.png new file mode 100644 index 0000000000..f3a60cf9ee Binary files /dev/null and b/connectors/aws/aws-s3/assets/novatec.png differ diff --git a/connectors/aws/aws-s3/connector-aws-s3/element-templates/connector-aws-s3.json b/connectors/aws/aws-s3/connector-aws-s3/element-templates/connector-aws-s3.json new file mode 100644 index 0000000000..40a99d99cb --- /dev/null +++ b/connectors/aws/aws-s3/connector-aws-s3/element-templates/connector-aws-s3.json @@ -0,0 +1,204 @@ +{ + "$schema": "https://unpkg.com/@camunda/zeebe-element-templates-json-schema/resources/schema.json", + "name": "Connector AWS S3", + "id": "6f60159e-f5f5-49d0-805b-9320aab39ee5", + "description": "Manage your files on AWS S3", + "version": 2, + "icon": { + "contents": "data:image/svg+xml;base64,PD94bWwgdmVyc2lvbj0iMS4wIiBlbmNvZGluZz0iVVRGLTgiPz4KPHN2ZyB3aWR0aD0iODBweCIgaGVpZ2h0PSI4MHB4IiB2aWV3Qm94PSIwIDAgNjAgNjAiIHZlcnNpb249IjEuMSIgeG1sbnM9Imh0dHA6Ly93d3cudzMub3JnLzIwMDAvc3ZnIiB4bWxuczp4bGluaz0iaHR0cDovL3d3dy53My5vcmcvMTk5OS94bGluayI+CiAgICA8dGl0bGU+Q29ubmVjdG9yL0xvZ28vUzM8L3RpdGxlPgogICAgPGcgaWQ9IkNvbm5lY3Rvci9Mb2dvL1MzIiBzdHJva2U9Im5vbmUiIHN0cm9rZS13aWR0aD0iMSIgZmlsbD0ibm9uZSIgZmlsbC1ydWxlPSJldmVub2RkIj4KICAgICAgICA8ZyBpZD0iQ29ubmVjdG9yL0xvZ28vUzMvU3F1YXJlIiBmaWxsPSIjRkZGRkZGIj4KICAgICAgICAgICAgPHJlY3QgaWQ9IlJlY3RhbmdsZSIgeD0iMCIgeT0iMCIgd2lkdGg9IjYwIiBoZWlnaHQ9IjYwIj48L3JlY3Q+CiAgICAgICAgPC9nPgogICAgICAgIDxnIGlkPSJDb25uZWN0b3IvTG9nby9TMy9CdWNrZXQiIHRyYW5zZm9ybT0idHJhbnNsYXRlKDIuOTk5OTAwLCAxLjk5OTYwMCkiIGZpbGw9IiMwMDAwMDAiPgogICAgICAgICAgICA8cGF0aCBzdHJva2U9ImJsYWNrIiBzdHJva2Utd2lkdGg9IjEuNSIgZD0iTTQ3LjgzNiwzMC44OTMgTDQ4LjIyLDI4LjE4OSBDNTEuNzYxLDMwLjMxIDUxLjgwNywzMS4xODYgNTEuODA2MDEzMiwzMS4yMSBDNTEuOCwzMS4yMTUgNTEuMTk2LDMxLjcxOSA0Ny44MzYsMzAuODkzIEw0Ny44MzYsMzAuODkzIFogTTQ1Ljg5MywzMC4zNTMgQzM5Ljc3MywyOC41MDEgMzEuMjUsMjQuNTkxIDI3LjgwMSwyMi45NjEgQzI3LjgwMSwyMi45NDcgMjcuODA1LDIyLjkzNCAyNy44MDUsMjIuOTIgQzI3LjgwNSwyMS41OTUgMjYuNzI3LDIwLjUxNyAyNS40MDEsMjAuNTE3IEMyNC4wNzcsMjAuNTE3IDIyLjk5OSwyMS41OTUgMjIuOTk5LDIyLjkyIEMyMi45OTksMjQuMjQ1IDI0LjA3NywyNS4zMjMgMjUuNDAxLDI1LjMyMyBDMjUuOTgzLDI1LjMyMyAyNi41MTEsMjUuMTA2IDI2LjkyOCwyNC43NjEgQzMwLjk4NiwyNi42ODIgMzkuNDQzLDMwLjUzNSA0NS42MDgsMzIuMzU1IEw0My4xNyw0OS41NjEgQzQzLjE2Myw0OS42MDggNDMuMTYsNDkuNjU1IDQzLjE2LDQ5LjcwMiBDNDMuMTYsNTEuMjE3IDM2LjQ1Myw1NCAyNS40OTQsNTQgQzE0LjQxOSw1NCA3LjY0MSw1MS4yMTcgNy42NDEsNDkuNzAyIEM3LjY0MSw0OS42NTYgNy42MzgsNDkuNjExIDcuNjMyLDQ5LjU2NiBMMi41MzgsMTIuMzU5IEM2Ljk0NywxNS4zOTQgMTYuNDMsMTcgMjUuNSwxNyBDMzQuNTU2LDE3IDQ0LjAyMywxNS40IDQ4LjQ0MSwxMi4zNzQgTDQ1Ljg5MywzMC4zNTMgWiBNMiw4LjQ3OCBDMi4wNzIsNy4xNjIgOS42MzQsMiAyNS41LDIgQzQxLjM2NCwyIDQ4LjkyNyw3LjE2MSA0OSw4LjQ3OCBMNDksOC45MjcgQzQ4LjEzLDExLjg3OCAzOC4zMywxNSAyNS41LDE1IEMxMi42NDgsMTUgMi44NDMsMTEuODY4IDIsOC45MTMgTDIsOC40NzggWiBNNTEsOC41IEM1MSw1LjAzNSA0MS4wNjYsMCAyNS41LDAgQzkuOTM0LDAgMCw1LjAzNSAwLDguNSBMMC4wOTQsOS4yNTQgTDUuNjQyLDQ5Ljc3OCBDNS43NzUsNTQuMzEgMTcuODYxLDU2IDI1LjQ5NCw1NiBDMzQuOTY2LDU2IDQ1LjAyOSw1My44MjIgNDUuMTU5LDQ5Ljc4MSBMNDcuNTU1LDMyLjg4NCBDNDguODg4LDMzLjIwMyA0OS45ODUsMzMuMzY2IDUwLjg2NiwzMy4zNjYgQzUyLjA0OSwzMy4zNjYgNTIuODQ5LDMzLjA3NyA1My4zMzQsMzIuNDk5IEM1My43MzIsMzIuMDI1IDUzLjg4NCwzMS40NTEgNTMuNzcsMzAuODQgQzUzLjUxMSwyOS40NTYgNTEuODY4LDI3Ljk2NCA0OC41MjIsMjYuMDU1IEw1MC44OTgsOS4yOTMgTDUxLDguNSBaIiBpZD0iUzNfTG9nbyI+PC9wYXRoPgogICAgICAgIDwvZz4KICAgIDwvZz4KPC9zdmc+" + }, + "category": { + "id": "connectors", + "name": "Connectors" + }, + "appliesTo": [ + "bpmn:Task" + ], + "elementType": { + "value": "bpmn:ServiceTask" + }, + "groups": [ + { + "id": "authentication", + "label": "Authentication" + }, + { + "id": "operation", + "label": "Select Operation" + }, + { + "id": "operationDetails", + "label": "Operation Details" + }, + { + "id": "output", + "label": "Output Mapping" + }, + { + "id": "errors", + "label": "Error Handling" + } + ], + "properties": [ + { + "type": "Hidden", + "value": "io.camunda.connector.awss3:aws-s3:1", + "binding": { + "type": "zeebe:taskDefinition:type" + } + }, + { + "label": "Access Key", + "description": "Provide an access key of a user with permissions to access the specified AWS S3 bucket", + "group": "authentication", + "type": "String", + "binding": { + "type": "zeebe:input", + "name": "authentication.accessKey" + }, + "constraints": { + "notEmpty": true + } + }, + { + "label": "Secret Key", + "description": "Provide a secret key of a user with permissions to access the specified AWS S3 bucket", + "group": "authentication", + "type": "String", + "binding": { + "type": "zeebe:input", + "name": "authentication.secretKey" + }, + "constraints": { + "notEmpty": true + } + }, + { + "id": "operationType", + "group": "operation", + "type": "Dropdown", + "value": "PUT_OBJECT", + "choices": [ + { + "name": "Upload file", + "value": "PUT_OBJECT" + }, + { + "name": "Delete file", + "value": "DELETE_OBJECT" + }, + { + "name": "Download file", + "value": "GET_OBJECT" + } + ], + "binding": { + "type": "zeebe:input", + "name": "requestDetails.operationType", + "key": "requestDetails.operationType" + } + }, + { + "label": "AWS Region", + "description": "Specify an AWS region", + "group": "operationDetails", + "type": "String", + "feel": "optional", + "binding": { + "type": "zeebe:input", + "name": "requestDetails.region" + }, + "constraints": { + "notEmpty": true, + "maxLength": 255 + } + }, + { + "label": "Bucket name", + "description": "Enter the name of your s3 bucket", + "group": "operationDetails", + "type": "String", + "feel": "optional", + "binding": { + "type": "zeebe:input", + "name": "requestDetails.bucketName" + }, + "constraints": { + "notEmpty": true, + "maxLength": 255 + } + }, + { + "label": "Object key", + "description": "Provide a key for your upload, relative from the bucket (e.g. my/files/message.xml)", + "group": "operationDetails", + "type": "String", + "feel": "optional", + "binding": { + "type": "zeebe:input", + "name": "requestDetails.objectKey" + }, + "constraints": { + "notEmpty": true, + "maxLength": 255 + } + }, + { + "label": "File Path", + "description": "Provide the path to a local file, default is the same as objectKey", + "group": "operationDetails", + "type": "String", + "feel": "optional", + "binding": { + "type": "zeebe:input", + "name": "requestDetails.filePath" + } + }, + { + "label": "Content Type", + "description": "Provide a content type (e.g. application/xml)", + "group": "operationDetails", + "type": "String", + "feel": "optional", + "binding": { + "type": "zeebe:input", + "name": "requestDetails.contentType" + }, + "constraints": { + "notEmpty": false + } + }, + { + "label": "Result Variable", + "description": "Name of variable to store the response in", + "group": "output", + "type": "String", + "binding": { + "type": "zeebe:taskHeader", + "key": "resultVariable" + } + }, + { + "label": "Result Expression", + "description": "Expression to map the response into process variables", + "group": "output", + "type": "Text", + "feel": "required", + "binding": { + "type": "zeebe:taskHeader", + "key": "resultExpression" + } + }, + { + "label": "Error Expression", + "description": "Expression to handle errors. Details in the documentation.", + "group": "errors", + "type": "Text", + "feel": "required", + "binding": { + "type": "zeebe:taskHeader", + "key": "errorExpression" + } + } + ] +} diff --git a/connectors/aws/aws-s3/connector-aws-s3/pom.xml b/connectors/aws/aws-s3/connector-aws-s3/pom.xml new file mode 100644 index 0000000000..06bf5aabd7 --- /dev/null +++ b/connectors/aws/aws-s3/connector-aws-s3/pom.xml @@ -0,0 +1,31 @@ + + + 4.0.0 + + io.camunda.connector + connector-aws-s3-parent + 8.7.0-SNAPSHOT + ../pom.xml + + + connector-aws-s3 + io.camunda.connector.awss3 + connector-aws-s3 + + + 21 + 21 + UTF-8 + + + + + io.camunda.connector.fileapi + connector-aws-fileapi + 8.7.0-SNAPSHOT + + + + \ No newline at end of file diff --git a/connectors/aws/aws-s3/connector-aws-s3/src/main/java/io/camunda/connector/awss3/in/ConnectorAdapter.java b/connectors/aws/aws-s3/connector-aws-s3/src/main/java/io/camunda/connector/awss3/in/ConnectorAdapter.java new file mode 100644 index 0000000000..1965a362b1 --- /dev/null +++ b/connectors/aws/aws-s3/connector-aws-s3/src/main/java/io/camunda/connector/awss3/in/ConnectorAdapter.java @@ -0,0 +1,48 @@ +package io.camunda.connector.awss3.in; + +import io.camunda.connector.api.annotation.OutboundConnector; +import io.camunda.connector.api.outbound.OutboundConnectorContext; +import io.camunda.connector.api.outbound.OutboundConnectorFunction; +import io.camunda.connector.fileapi.ProcessFileCommand; +import io.camunda.connector.fileapi.model.RequestData; +import io.camunda.connector.awss3.in.model.ConnectorRequest; +import io.camunda.connector.awss3.in.model.ConnectorResponse; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.io.IOException; + +@OutboundConnector( + name = "AWSS3", + inputVariables = {"authentication", "requestDetails"}, + type = "info.novatec.bpm:aws-s3:1") +public class ConnectorAdapter implements OutboundConnectorFunction { + + private static final Logger logger = LoggerFactory.getLogger(ConnectorAdapter.class); + + private ProcessFileCommand processFileCommand; + + public ConnectorAdapter() {} + + public ConnectorAdapter(ProcessFileCommand processFileCommand) { + this.processFileCommand = processFileCommand; + } + + @Override + public Object execute(OutboundConnectorContext context) throws IOException { + ConnectorRequest request = context.bindVariables(ConnectorRequest.class); + logger.info("Executing connector with request {}", request); + return execute(request); + } + + private ConnectorResponse execute(ConnectorRequest request) throws IOException { + RequestData requestData = RequestMapper.mapRequest(request); + RequestData result = switch (request.getRequestDetails().getOperationType()) { + case DELETE_OBJECT -> processFileCommand.deleteFile(requestData); + case PUT_OBJECT -> processFileCommand.uploadFile(requestData); + case GET_OBJECT -> processFileCommand.downloadFile(requestData); + }; + return new ConnectorResponse(result); + } + +} diff --git a/connectors/aws/aws-s3/connector-aws-s3/src/main/java/io/camunda/connector/awss3/in/RequestMapper.java b/connectors/aws/aws-s3/connector-aws-s3/src/main/java/io/camunda/connector/awss3/in/RequestMapper.java new file mode 100644 index 0000000000..6ae3763dfa --- /dev/null +++ b/connectors/aws/aws-s3/connector-aws-s3/src/main/java/io/camunda/connector/awss3/in/RequestMapper.java @@ -0,0 +1,28 @@ +package io.camunda.connector.awss3.in; + +import io.camunda.connector.fileapi.model.RequestData; +import io.camunda.connector.awss3.in.model.ConnectorRequest; + +import java.util.Objects; + +public class RequestMapper { + + public static RequestData mapRequest(ConnectorRequest request) { + return RequestData.builder() + .authenticationKey(request.getAuthentication().getAccessKey()) + .authenticationSecret(request.getAuthentication().getSecretKey()) + .region(request.getRequestDetails().getRegion()) + .bucket(request.getRequestDetails().getBucketName()) + .key(request.getRequestDetails().getObjectKey()) + .filePath( + // fallback to objectKey + Objects.requireNonNullElse( + request.getRequestDetails().getFilePath(), + request.getRequestDetails().getObjectKey() + ) + ) + .contentType(request.getRequestDetails().getContentType()) + .build(); + } + +} diff --git a/connectors/aws/aws-s3/connector-aws-s3/src/main/java/io/camunda/connector/awss3/in/model/AuthenticationRequestData.java b/connectors/aws/aws-s3/connector-aws-s3/src/main/java/io/camunda/connector/awss3/in/model/AuthenticationRequestData.java new file mode 100644 index 0000000000..fc5a87c80e --- /dev/null +++ b/connectors/aws/aws-s3/connector-aws-s3/src/main/java/io/camunda/connector/awss3/in/model/AuthenticationRequestData.java @@ -0,0 +1,18 @@ +package io.camunda.connector.awss3.in.model; + +import jakarta.validation.constraints.NotEmpty; +import lombok.Data; +import lombok.ToString; + +@Data +public class AuthenticationRequestData { + + @NotEmpty + @ToString.Exclude + private String accessKey; + + @NotEmpty + @ToString.Exclude + private String secretKey; + +} diff --git a/connectors/aws/aws-s3/connector-aws-s3/src/main/java/io/camunda/connector/awss3/in/model/ConnectorRequest.java b/connectors/aws/aws-s3/connector-aws-s3/src/main/java/io/camunda/connector/awss3/in/model/ConnectorRequest.java new file mode 100644 index 0000000000..46c4dc6647 --- /dev/null +++ b/connectors/aws/aws-s3/connector-aws-s3/src/main/java/io/camunda/connector/awss3/in/model/ConnectorRequest.java @@ -0,0 +1,17 @@ +package io.camunda.connector.awss3.in.model; + +import jakarta.validation.Valid; +import jakarta.validation.constraints.NotNull; +import lombok.Data; + +@Data +public class ConnectorRequest { + + @Valid + @NotNull + private AuthenticationRequestData authentication; + + @Valid + @NotNull + private RequestDetails requestDetails; +} diff --git a/connectors/aws/aws-s3/connector-aws-s3/src/main/java/io/camunda/connector/awss3/in/model/ConnectorResponse.java b/connectors/aws/aws-s3/connector-aws-s3/src/main/java/io/camunda/connector/awss3/in/model/ConnectorResponse.java new file mode 100644 index 0000000000..3b86a70eb6 --- /dev/null +++ b/connectors/aws/aws-s3/connector-aws-s3/src/main/java/io/camunda/connector/awss3/in/model/ConnectorResponse.java @@ -0,0 +1,23 @@ +package io.camunda.connector.awss3.in.model; + +import io.camunda.connector.fileapi.model.RequestData; +import lombok.Data; +import lombok.NoArgsConstructor; + +@Data +@NoArgsConstructor +public class ConnectorResponse { + + private String bucketName; + private String objectKey; + private String filePath; + private String contentType; + + public ConnectorResponse(RequestData request) { + this.bucketName = request.getBucket(); + this.objectKey = request.getKey(); + this.filePath = request.getFilePath(); + this.contentType = request.getContentType(); + } + +} diff --git a/connectors/aws/aws-s3/connector-aws-s3/src/main/java/io/camunda/connector/awss3/in/model/OperationType.java b/connectors/aws/aws-s3/connector-aws-s3/src/main/java/io/camunda/connector/awss3/in/model/OperationType.java new file mode 100644 index 0000000000..360ca72edc --- /dev/null +++ b/connectors/aws/aws-s3/connector-aws-s3/src/main/java/io/camunda/connector/awss3/in/model/OperationType.java @@ -0,0 +1,5 @@ +package io.camunda.connector.awss3.in.model; + +public enum OperationType { + PUT_OBJECT, DELETE_OBJECT, GET_OBJECT +} diff --git a/connectors/aws/aws-s3/connector-aws-s3/src/main/java/io/camunda/connector/awss3/in/model/RequestDetails.java b/connectors/aws/aws-s3/connector-aws-s3/src/main/java/io/camunda/connector/awss3/in/model/RequestDetails.java new file mode 100644 index 0000000000..bebc8cfb68 --- /dev/null +++ b/connectors/aws/aws-s3/connector-aws-s3/src/main/java/io/camunda/connector/awss3/in/model/RequestDetails.java @@ -0,0 +1,26 @@ +package io.camunda.connector.awss3.in.model; + +import jakarta.validation.constraints.NotEmpty; +import jakarta.validation.constraints.NotNull; +import lombok.Data; + +@Data +public class RequestDetails { + + @NotEmpty + private String bucketName; + + @NotEmpty + private String objectKey; + + private String filePath; + + @NotEmpty + private String region; + + @NotNull + private OperationType operationType; + + private String contentType; + +} diff --git a/connectors/aws/aws-s3/connector-aws-s3/src/main/java/io/camunda/connector/awss3/out/cloud/CloudClientFactory.java b/connectors/aws/aws-s3/connector-aws-s3/src/main/java/io/camunda/connector/awss3/out/cloud/CloudClientFactory.java new file mode 100644 index 0000000000..2a2b7ad7cc --- /dev/null +++ b/connectors/aws/aws-s3/connector-aws-s3/src/main/java/io/camunda/connector/awss3/out/cloud/CloudClientFactory.java @@ -0,0 +1,41 @@ +package io.camunda.connector.awss3.out.cloud; + +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; +import software.amazon.awssdk.auth.credentials.AwsBasicCredentials; +import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider; +import software.amazon.awssdk.regions.Region; +import software.amazon.awssdk.services.s3.S3Client; +import software.amazon.awssdk.services.s3.S3ClientBuilder; + +import java.net.URI; + +public class CloudClientFactory { + + private String endpointOverride; + + public CloudClientFactory() { + } + + public CloudClientFactory(String endpointOverride) { + this.endpointOverride = endpointOverride; + } + + private static final Logger logger = LoggerFactory.getLogger(CloudClientFactory.class); + + public S3Client createClient(String accessKey, String secretKey, String region) { + AwsBasicCredentials credentials = AwsBasicCredentials.create(accessKey, secretKey); + logger.info("Initialized AWS client for region: {}", region); + + S3ClientBuilder builder = S3Client.builder() + .credentialsProvider(() -> StaticCredentialsProvider.create(credentials).resolveCredentials()) + .region(Region.of(region)); + + if (endpointOverride != null && !endpointOverride.isBlank()) { + logger.info("AWS endpoint override: {}", endpointOverride); + builder.endpointOverride(URI.create(endpointOverride)); + } + return builder.build(); + } + +} diff --git a/connectors/aws/aws-s3/connector-aws-s3/src/main/java/io/camunda/connector/awss3/out/cloud/CloudFileAdapter.java b/connectors/aws/aws-s3/connector-aws-s3/src/main/java/io/camunda/connector/awss3/out/cloud/CloudFileAdapter.java new file mode 100644 index 0000000000..5fca92f2c0 --- /dev/null +++ b/connectors/aws/aws-s3/connector-aws-s3/src/main/java/io/camunda/connector/awss3/out/cloud/CloudFileAdapter.java @@ -0,0 +1,76 @@ +package io.camunda.connector.awss3.out.cloud; + +import io.camunda.connector.fileapi.RemoteFileCommand; +import io.camunda.connector.fileapi.model.FileContent; +import io.camunda.connector.fileapi.model.RequestData; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; +import software.amazon.awssdk.core.ResponseInputStream; +import software.amazon.awssdk.core.sync.RequestBody; +import software.amazon.awssdk.services.s3.S3Client; +import software.amazon.awssdk.services.s3.model.*; + +import java.io.IOException; + +public class CloudFileAdapter implements RemoteFileCommand { + + private static final Logger logger = LoggerFactory.getLogger(CloudFileAdapter.class); + + private final CloudClientFactory clientFactory; + + public CloudFileAdapter(CloudClientFactory clientFactory) { + this.clientFactory = clientFactory; + } + + public void deleteFile(RequestData requestData) { + try (S3Client s3Client = clientFactory.createClient(requestData.getAuthenticationKey(), requestData.getAuthenticationSecret(), requestData.getRegion())) { + DeleteObjectRequest awsRequest = DeleteObjectRequest.builder() + .bucket(requestData.getBucket()) + .key(requestData.getKey()) + .build(); + logger.info("Delete object: {}", requestData.getBucket() + "/" + requestData.getKey()); + logger.debug("request {}", awsRequest); + DeleteObjectResponse response = s3Client.deleteObject(awsRequest); + logger.info("Object deleted: {}", requestData.getBucket() + "/" + requestData.getKey()); + logger.debug("response {}", response); + } + } + + public void putFile(RequestData requestData, FileContent fileContent) { + try (S3Client s3Client = clientFactory.createClient(requestData.getAuthenticationKey(), requestData.getAuthenticationSecret(), requestData.getRegion())) { + PutObjectRequest awsRequest = PutObjectRequest.builder() + .bucket(requestData.getBucket()) + .key(requestData.getKey()) + .contentType(fileContent.getContentType()) + .contentLength(fileContent.getContentLength()) + .build(); + logger.info("Put object: {}", requestData.getBucket() + "/" + requestData.getKey()); + logger.debug("request {}", awsRequest); + PutObjectResponse awsResponse = s3Client.putObject(awsRequest, RequestBody.fromBytes(fileContent.getContent())); + logger.info("Object put: {}", requestData.getBucket() + "/" + requestData.getKey()); + logger.debug("response {}", awsResponse); + } + } + + public FileContent getFile(RequestData requestData) throws IOException { + try (S3Client s3Client = clientFactory.createClient(requestData.getAuthenticationKey(), requestData.getAuthenticationSecret(), requestData.getRegion())) { + GetObjectRequest awsRequest = GetObjectRequest.builder() + .bucket(requestData.getBucket()) + .key(requestData.getKey()) + .build(); + logger.info("Get object: {}", requestData.getBucket() + "/" + requestData.getKey()); + logger.debug("request {}", awsRequest); + try (ResponseInputStream object = s3Client.getObject(awsRequest)) { + FileContent result = FileContent.builder() + .content(object.readAllBytes()) + .contentType(object.response().contentType()) + .contentLength(object.response().contentLength()) + .build(); + logger.info("Object received: {}", requestData.getBucket() + "/" + requestData.getKey()); + logger.debug("response {}", object.response()); + return result; + } + } + } + +} diff --git a/connectors/aws/aws-s3/connector-aws-s3/src/main/java/io/camunda/connector/awss3/out/local/LocalFileAdapter.java b/connectors/aws/aws-s3/connector-aws-s3/src/main/java/io/camunda/connector/awss3/out/local/LocalFileAdapter.java new file mode 100644 index 0000000000..2bf02d59bf --- /dev/null +++ b/connectors/aws/aws-s3/connector-aws-s3/src/main/java/io/camunda/connector/awss3/out/local/LocalFileAdapter.java @@ -0,0 +1,64 @@ +package io.camunda.connector.awss3.out.local; + +import io.camunda.connector.fileapi.LocalFileCommand; +import io.camunda.connector.fileapi.exceptions.LocalFileException; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.io.IOException; +import java.io.InputStream; +import java.io.OutputStream; +import java.nio.file.Files; +import java.nio.file.Path; + +public class LocalFileAdapter implements LocalFileCommand { + + private final Path baseDir; + Logger logger = LoggerFactory.getLogger(LocalFileAdapter.class); + + public LocalFileAdapter(Path baseDir) { + this.baseDir = baseDir; + logger.info("Initialized local file adapter base path: {}", baseDir); + } + + public Path saveFile(byte[] content, String filePath) throws IOException { + Path file = baseDir.resolve(filePath); + if (Files.exists(file)) { + throw new LocalFileException(String.format("The file already exists: %s", filePath)); + } + Path directories = Files.createDirectories(file.getParent()); + logger.info("Created directories {}", directories); + logger.info("Writing file to {}", filePath); + try (OutputStream stream = Files.newOutputStream(file)) { + stream.write(content); + logger.debug("{} bytes written to disk", content.length); + return file; + } + } + + public byte[] loadFile(String filePath) throws IOException { + Path file = baseDir.resolve(filePath); + if (!Files.exists(file)) { + throw new LocalFileException(String.format("The file doesn't exist: %s", filePath)); + } + logger.info("Reading file from {}", filePath); + try (InputStream stream = Files.newInputStream(file)) { + byte[] bytes = stream.readAllBytes(); + logger.debug("{} bytes read from disk", bytes.length); + return bytes; + } + } + + @Override + public void deleteFile(String filePath) throws IOException { + Path file = baseDir.resolve(filePath); + logger.info("Deleting file {}", filePath); + boolean deleted = Files.deleteIfExists(file); + if (deleted) { + logger.debug("File deleted from disk: {}", filePath); + } else { + logger.debug("File didn't exist: {}", filePath); + } + } + +} diff --git a/connectors/aws/aws-s3/connector-aws-s3/src/main/resources/META-INF/services/io.camunda.connector.api.outbound.OutboundConnectorFunction b/connectors/aws/aws-s3/connector-aws-s3/src/main/resources/META-INF/services/io.camunda.connector.api.outbound.OutboundConnectorFunction new file mode 100644 index 0000000000..93ee4148fd --- /dev/null +++ b/connectors/aws/aws-s3/connector-aws-s3/src/main/resources/META-INF/services/io.camunda.connector.api.outbound.OutboundConnectorFunction @@ -0,0 +1 @@ +io.camunda.connector.awss3.in.ConnectorAdapter \ No newline at end of file diff --git a/connectors/aws/aws-s3/connector-aws-s3/src/main/resources/logback.xml b/connectors/aws/aws-s3/connector-aws-s3/src/main/resources/logback.xml new file mode 100644 index 0000000000..5cf4dcffa8 --- /dev/null +++ b/connectors/aws/aws-s3/connector-aws-s3/src/main/resources/logback.xml @@ -0,0 +1,11 @@ + + + + %d{ISO8601} [%thread] %-5level %logger{36} - %msg%n + + + + + + + \ No newline at end of file diff --git a/connectors/aws/aws-s3/connector-aws-s3/src/test/java/io/camunda/connector/awss3/adapter/ConnectorAdapterContextTest.java b/connectors/aws/aws-s3/connector-aws-s3/src/test/java/io/camunda/connector/awss3/adapter/ConnectorAdapterContextTest.java new file mode 100644 index 0000000000..3d56aa4266 --- /dev/null +++ b/connectors/aws/aws-s3/connector-aws-s3/src/test/java/io/camunda/connector/awss3/adapter/ConnectorAdapterContextTest.java @@ -0,0 +1,147 @@ +package io.camunda.connector.awss3.adapter; + +import com.fasterxml.jackson.core.JsonProcessingException; +import com.fasterxml.jackson.databind.ObjectMapper; +import io.camunda.connector.api.error.ConnectorInputException; +import io.camunda.connector.awss3.in.model.AuthenticationRequestData; +import io.camunda.connector.awss3.in.model.ConnectorRequest; +import io.camunda.connector.awss3.in.model.OperationType; +import io.camunda.connector.awss3.in.model.RequestDetails; +import io.camunda.connector.test.outbound.OutboundConnectorContextBuilder; +import io.camunda.connector.test.outbound.OutboundConnectorContextBuilder.TestConnectorContext; +import org.junit.jupiter.api.Test; + +import static org.assertj.core.api.Assertions.assertThat; +import static org.assertj.core.api.Assertions.assertThatThrownBy; + +class ConnectorAdapterContextTest { + + @Test + void should_replace_secrets() throws JsonProcessingException { + // given + ConnectorRequest request = new ConnectorRequest(); + request.setAuthentication(getAuthentication()); + request.setRequestDetails(getDetails()); + + TestConnectorContext context = OutboundConnectorContextBuilder.create() + .secret("AWS_ACCESS_KEY", "abc") + .secret("AWS_SECRET_KEY", "123") + .variables(new ObjectMapper().writeValueAsString(request)) + .build(); + + // when + ConnectorRequest details = context.bindVariables(ConnectorRequest.class); + + // then + assertThat(details).extracting("authentication") + .extracting("accessKey") + .as("access key") + .isEqualTo("abc"); + + assertThat(details).extracting("authentication") + .extracting("secretKey") + .as("secret key") + .isEqualTo("123"); + } + + @Test + void should_fail_if_authentication_is_missing() throws JsonProcessingException { + // setup + ConnectorRequest request = new ConnectorRequest(); + request.setRequestDetails(getDetails()); + TestConnectorContext context = OutboundConnectorContextBuilder.create() + .secret("AWS_ACCESS_KEY", "abc") + .secret("AWS_SECRET_KEY", "123") + .variables(new ObjectMapper().writeValueAsString(request)) + .build(); + + // expect + assertThatThrownBy(() -> context.validate(request)) + .isInstanceOf(ConnectorInputException.class) + .hasMessage("jakarta.validation.ValidationException: Found constraints violated while validating input: \n - Property: authentication: Validation failed."); + } + + @Test + void should_fail_if_details_are_missing() throws JsonProcessingException { + // setup + ConnectorRequest request = new ConnectorRequest(); + AuthenticationRequestData authentication = getAuthentication(); + request.setAuthentication(authentication); + TestConnectorContext context = OutboundConnectorContextBuilder.create() + .secret("AWS_ACCESS_KEY", "abc") + .secret("AWS_SECRET_KEY", "123") + .variables(new ObjectMapper().writeValueAsString(request)) + .build(); + + // expect + assertThatThrownBy(() -> context.validate(request)) + .isInstanceOf(ConnectorInputException.class) + .hasMessage("jakarta.validation.ValidationException: Found constraints violated while validating input: \n - Property: requestDetails: Validation failed."); + } + + @Test + void should_fail_if_required_details_values_are_missing() throws JsonProcessingException { + // setup + ConnectorRequest request = new ConnectorRequest(); + request.setAuthentication(getAuthentication()); + request.setRequestDetails(new RequestDetails()); + var context = OutboundConnectorContextBuilder.create() + .secret("AWS_ACCESS_KEY", "abc") + .secret("AWS_SECRET_KEY", "123") + .variables(new ObjectMapper().writeValueAsString(request)) + .build(); + + // expect + assertThatThrownBy(() -> context.validate(request)) + .isInstanceOf(ConnectorInputException.class) + .hasMessageContainingAll( + "requestDetails.bucketName: Validation failed", + "requestDetails.region: Validation failed", + "requestDetails.objectKey: Validation failed", + "requestDetails.operationType: Validation failed" + ) + .hasMessageNotContainingAny( + "requestDetails.contentType: Validation failed", + "requestDetails.filePath: Validation failed" + ); + + } + + @Test + void should_fail_if_required_authentication_value_are_missing() throws JsonProcessingException { + // setup + ConnectorRequest request = new ConnectorRequest(); + request.setAuthentication(new AuthenticationRequestData()); + request.setRequestDetails(getDetails()); + var context = OutboundConnectorContextBuilder.create() + .secret("AWS_ACCESS_KEY", "abc") + .secret("AWS_SECRET_KEY", "123") + .variables(new ObjectMapper().writeValueAsString(request)) + .build(); + + // expect + assertThatThrownBy(() -> context.validate(request)) + .isInstanceOf(ConnectorInputException.class) + .hasMessageContainingAll("authentication.secretKey", "authentication.accessKey"); + + } + + private RequestDetails getDetails() { + RequestDetails details = new RequestDetails(); + details.setBucketName("bucket"); + details.setContentType("application/text"); + details.setFilePath("/tmp/invoice.txt"); + details.setObjectKey("/invoice/invoice-123.txt"); + details.setOperationType(OperationType.PUT_OBJECT); + details.setRegion("eu-central-1"); + return details; + } + + private AuthenticationRequestData getAuthentication() { + AuthenticationRequestData authentication = new AuthenticationRequestData(); + authentication.setAccessKey("secrets.AWS_ACCESS_KEY"); + authentication.setSecretKey("secrets.AWS_SECRET_KEY"); + return authentication; + } + +} \ No newline at end of file diff --git a/connectors/aws/aws-s3/connector-aws-s3/src/test/java/io/camunda/connector/awss3/adapter/ConnectorAdapterTest.java b/connectors/aws/aws-s3/connector-aws-s3/src/test/java/io/camunda/connector/awss3/adapter/ConnectorAdapterTest.java new file mode 100644 index 0000000000..53f2c2a3e9 --- /dev/null +++ b/connectors/aws/aws-s3/connector-aws-s3/src/test/java/io/camunda/connector/awss3/adapter/ConnectorAdapterTest.java @@ -0,0 +1,236 @@ +package io.camunda.connector.awss3.adapter; + +import com.fasterxml.jackson.databind.ObjectMapper; +import io.camunda.connector.awss3.in.ConnectorAdapter; +import io.camunda.connector.awss3.in.model.*; +import io.camunda.connector.awss3.out.cloud.CloudClientFactory; +import io.camunda.connector.awss3.out.cloud.CloudFileAdapter; +import io.camunda.connector.awss3.out.local.LocalFileAdapter; +import io.camunda.connector.fileapi.ProcessFileCommand; +import io.camunda.connector.fileapi.ProcessFileService; +import io.camunda.connector.test.outbound.OutboundConnectorContextBuilder; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.extension.ExtendWith; +import org.mockito.ArgumentCaptor; +import org.mockito.Mock; +import org.mockito.Mockito; +import org.mockito.junit.jupiter.MockitoExtension; +import software.amazon.awssdk.core.ResponseInputStream; +import software.amazon.awssdk.core.sync.RequestBody; +import software.amazon.awssdk.services.s3.S3Client; +import software.amazon.awssdk.services.s3.model.*; + +import java.io.ByteArrayInputStream; +import java.io.IOException; +import java.io.InputStream; +import java.nio.charset.StandardCharsets; +import java.nio.file.Path; + +import static org.assertj.core.api.Assertions.assertThat; +import static org.mockito.ArgumentMatchers.any; +import static org.mockito.Mockito.*; + +@ExtendWith(MockitoExtension.class) +class ConnectorAdapterTest { + + @Mock + CloudClientFactory factory; // mock component to S3 + + @Mock + LocalFileAdapter localFileAdapter; // mock component to local file system + + private ConnectorAdapter connector; + + @BeforeEach + public void setup() { + ProcessFileCommand processFileCommand = new ProcessFileService(new CloudFileAdapter(factory), localFileAdapter); + connector = new ConnectorAdapter(processFileCommand); + } + + @Test + void happy_path_aws_put_is_called_as_expected() throws IOException { + // given + S3Client client = Mockito.mock(S3Client.class); + when(factory.createClient(any(), any(), any())).thenReturn(client); + PutObjectResponse awsResult = createPutResponse(); + when(client.putObject(any(PutObjectRequest.class), any(RequestBody.class))).thenReturn(awsResult); + + byte[] fileBytes = "hello, world!".getBytes(StandardCharsets.UTF_8); + when(localFileAdapter.loadFile(anyString())).thenReturn(fileBytes); + + ConnectorRequest request = new ConnectorRequest(); + request.setAuthentication(getAuthentication()); + String filePath = "my/path/to/file.txt"; + request.setRequestDetails(getPutDetails("bucket", "path/file.txt", filePath)); + + ConnectorResponse expectedResponse = new ConnectorResponse(); + expectedResponse.setFilePath(filePath); + expectedResponse.setObjectKey("path/file.txt"); + expectedResponse.setBucketName("bucket"); + expectedResponse.setContentType("application/text"); + + OutboundConnectorContextBuilder.TestConnectorContext context = OutboundConnectorContextBuilder.create() + .secret("AWS_ACCESS_KEY", "abc") + .secret("AWS_SECRET_KEY", "123") + .variables(new ObjectMapper().writeValueAsString(request)) + .build(); + + // when + ConnectorResponse actualResult = (ConnectorResponse) connector.execute(context); + + // then + assertThat(actualResult).isEqualTo(expectedResponse); + + ArgumentCaptor requestCaptor = ArgumentCaptor.forClass(PutObjectRequest.class); + ArgumentCaptor bodyCaptor = ArgumentCaptor.forClass(RequestBody.class); + verify(client, times(1)).putObject(requestCaptor.capture(), bodyCaptor.capture()); + PutObjectRequest putRequest = requestCaptor.getValue(); + assertThat(putRequest.bucket()).isEqualTo("bucket"); + assertThat(putRequest.key()).isEqualTo("path/file.txt"); + assertThat(putRequest.contentLength()).isEqualTo(fileBytes.length); + assertThat(putRequest.contentType()).isEqualTo("application/text"); + + RequestBody requestBody = bodyCaptor.getValue(); + try (InputStream is = requestBody.contentStreamProvider().newStream()) { + assertThat(is.readAllBytes()).isEqualTo(fileBytes); + } + + verify(localFileAdapter, times(1)).loadFile(eq(filePath)); + verify(client, times(1)).putObject(any(PutObjectRequest.class), any(RequestBody.class)); + } + + @Test + void happy_path_aws_delete_is_called_as_expected() throws IOException { + // given + S3Client client = Mockito.mock(S3Client.class); + when(factory.createClient(any(), any(), any())).thenReturn(client); + + ConnectorRequest request = new ConnectorRequest(); + request.setAuthentication(getAuthentication()); + request.setRequestDetails(getDeleteDetails("bucket", "path/file.txt", "my/path/file.txt")); + + OutboundConnectorContextBuilder.TestConnectorContext context = OutboundConnectorContextBuilder.create() + .secret("AWS_ACCESS_KEY", "abc") + .secret("AWS_SECRET_KEY", "123") + .variables(new ObjectMapper().writeValueAsString(request)) + .build(); + + // when + ConnectorResponse actualResult = (ConnectorResponse) connector.execute(context); + + // then + ConnectorResponse response = new ConnectorResponse(); + response.setObjectKey("path/file.txt"); + response.setBucketName("bucket"); + response.setFilePath("my/path/file.txt"); + assertThat(actualResult).isEqualTo(response); + + ArgumentCaptor argumentCaptor = ArgumentCaptor.forClass(DeleteObjectRequest.class); + verify(client, times(1)).deleteObject(argumentCaptor.capture()); + DeleteObjectRequest deleteRequest = argumentCaptor.getValue(); + assertThat(deleteRequest.bucket()).isEqualTo("bucket"); + assertThat(deleteRequest.key()).isEqualTo("path/file.txt"); + + verify(localFileAdapter, times(1)).deleteFile(eq("my/path/file.txt")); + verify(client, times(1)).deleteObject(any(DeleteObjectRequest.class)); + + } + + @Test + void happy_path_aws_get_is_called_as_expected() throws IOException { + // given + S3Client client = Mockito.mock(S3Client.class); + when(factory.createClient(any(), any(), any())).thenReturn(client); + + String filePath = "my/path/to/file.txt"; + byte[] fileBytes = "hello, world!".getBytes(StandardCharsets.UTF_8); + when(localFileAdapter.saveFile(eq(fileBytes), eq(filePath))).thenReturn(Path.of(filePath)); + + GetObjectResponse response = GetObjectResponse.builder() + .contentLength(1L) + .contentType("application/text") + .build(); + ResponseInputStream result = new ResponseInputStream<>(response, new ByteArrayInputStream(fileBytes)); + when(client.getObject(any(GetObjectRequest.class))).thenReturn(result); + + ConnectorRequest request = new ConnectorRequest(); + request.setAuthentication(getAuthentication()); + request.setRequestDetails(getGetDetails("bucket", "path/file.txt", filePath)); + + ConnectorResponse expectedResponse = new ConnectorResponse(); + expectedResponse.setFilePath(filePath); + expectedResponse.setObjectKey("path/file.txt"); + expectedResponse.setBucketName("bucket"); + expectedResponse.setContentType("application/text"); + + OutboundConnectorContextBuilder.TestConnectorContext context = OutboundConnectorContextBuilder.create() + .secret("AWS_ACCESS_KEY", "abc") + .secret("AWS_SECRET_KEY", "123") + .variables(new ObjectMapper().writeValueAsString(request)) + .build(); + + // when + ConnectorResponse actualResult = (ConnectorResponse) connector.execute(context); + + // then + assertThat(actualResult).isEqualTo(expectedResponse); + + ArgumentCaptor requestCaptor = ArgumentCaptor.forClass(GetObjectRequest.class); + verify(client, times(1)).getObject(requestCaptor.capture()); + GetObjectRequest getRequest = requestCaptor.getValue(); + assertThat(getRequest.bucket()).isEqualTo("bucket"); + assertThat(getRequest.key()).isEqualTo("path/file.txt"); + + verify(localFileAdapter, times(1)).saveFile(eq(fileBytes), eq(filePath)); + verify(client, times(1)).getObject(any(GetObjectRequest.class)); + } + + + private static PutObjectResponse createPutResponse() { + return PutObjectResponse.builder() + .versionId("1234567") + .serverSideEncryption(ServerSideEncryption.AES256) + .checksumSHA256("foo") + .build(); + } + + private RequestDetails getPutDetails(String bucket, String key, String path) { + RequestDetails details = new RequestDetails(); + details.setBucketName(bucket); + details.setObjectKey(key); + details.setContentType("application/text"); + details.setFilePath(path); + details.setOperationType(OperationType.PUT_OBJECT); + details.setRegion("eu-central-1"); + return details; + } + + private RequestDetails getGetDetails(String bucket, String key, String path) { + RequestDetails details = new RequestDetails(); + details.setBucketName(bucket); + details.setObjectKey(key); + details.setContentType("application/text"); + details.setFilePath(path); + details.setOperationType(OperationType.GET_OBJECT); + details.setRegion("eu-central-1"); + return details; + } + + private RequestDetails getDeleteDetails(String bucket, String key, String path) { + RequestDetails details = new RequestDetails(); + details.setBucketName(bucket); + details.setObjectKey(key); + details.setFilePath(path); + details.setOperationType(OperationType.DELETE_OBJECT); + details.setRegion("eu-central-1"); + return details; + } + + private AuthenticationRequestData getAuthentication() { + AuthenticationRequestData authentication = new AuthenticationRequestData(); + authentication.setAccessKey("secrets.AWS_ACCESS_KEY"); + authentication.setSecretKey("secrets.AWS_SECRET_KEY"); + return authentication; + } +} \ No newline at end of file diff --git a/connectors/aws/aws-s3/connector-aws-s3/src/test/java/io/camunda/connector/awss3/adapter/in/process/RequestMapperTest.java b/connectors/aws/aws-s3/connector-aws-s3/src/test/java/io/camunda/connector/awss3/adapter/in/process/RequestMapperTest.java new file mode 100644 index 0000000000..bcb6765b3a --- /dev/null +++ b/connectors/aws/aws-s3/connector-aws-s3/src/test/java/io/camunda/connector/awss3/adapter/in/process/RequestMapperTest.java @@ -0,0 +1,84 @@ +package io.camunda.connector.awss3.adapter.in.process; + +import io.camunda.connector.fileapi.model.RequestData; +import io.camunda.connector.awss3.in.RequestMapper; +import io.camunda.connector.awss3.in.model.AuthenticationRequestData; +import io.camunda.connector.awss3.in.model.OperationType; +import io.camunda.connector.awss3.in.model.RequestDetails; +import io.camunda.connector.awss3.in.model.ConnectorRequest; +import org.junit.jupiter.api.Test; + +import static org.assertj.core.api.Assertions.assertThat; + +class RequestMapperTest { + + @Test + void mapping_is_as_expected() { + // given + ConnectorRequest request = new ConnectorRequest(); + request.setRequestDetails(getDetails()); + request.setAuthentication(getAuthentication()); + + // when + RequestData requestData = RequestMapper.mapRequest(request); + + // then + assertThat(requestData.getRegion()) + .as("region") + .isEqualTo(request.getRequestDetails().getRegion()); + assertThat(requestData.getBucket()) + .as("bucket") + .isEqualTo(request.getRequestDetails().getBucketName()); + assertThat(requestData.getKey()) + .as("key") + .isEqualTo(request.getRequestDetails().getObjectKey()); + assertThat(requestData.getFilePath()) + .as("file path") + .isEqualTo(request.getRequestDetails().getFilePath()); + assertThat(requestData.getContentType()) + .as("content type") + .isEqualTo(request.getRequestDetails().getContentType()); + assertThat(requestData.getAuthenticationKey()) + .as("access key") + .isEqualTo(request.getAuthentication().getAccessKey()); + assertThat(requestData.getAuthenticationSecret()) + .as("secret key") + .isEqualTo(request.getAuthentication().getSecretKey()); + } + + @Test + void mapping_is_as_expected_with_key_as_fallback_for_file_path() { + // given + ConnectorRequest request = new ConnectorRequest(); + request.setRequestDetails(getDetails()); + request.getRequestDetails().setFilePath(null); + request.setAuthentication(getAuthentication()); + + // when + RequestData requestData = RequestMapper.mapRequest(request); + + // then + assertThat(requestData.getFilePath()) + .as("file path") + .isEqualTo(request.getRequestDetails().getObjectKey()); + } + + private RequestDetails getDetails() { + RequestDetails details = new RequestDetails(); + details.setBucketName("bucket"); + details.setContentType("application/text"); + details.setFilePath("/tmp/invoice.txt"); + details.setObjectKey("/invoice/invoice-123.txt"); + details.setOperationType(OperationType.PUT_OBJECT); + details.setRegion("eu-central-1"); + return details; + } + + private AuthenticationRequestData getAuthentication() { + AuthenticationRequestData authentication = new AuthenticationRequestData(); + authentication.setAccessKey("secrets.AWS_ACCESS_KEY"); + authentication.setSecretKey("secrets.AWS_SECRET_KEY"); + return authentication; + } + +} \ No newline at end of file diff --git a/connectors/aws/aws-s3/connector-aws-s3/src/test/java/io/camunda/connector/awss3/adapter/out/local/LocalFileAdapterTest.java b/connectors/aws/aws-s3/connector-aws-s3/src/test/java/io/camunda/connector/awss3/adapter/out/local/LocalFileAdapterTest.java new file mode 100644 index 0000000000..40b430ddfc --- /dev/null +++ b/connectors/aws/aws-s3/connector-aws-s3/src/test/java/io/camunda/connector/awss3/adapter/out/local/LocalFileAdapterTest.java @@ -0,0 +1,76 @@ +package io.camunda.connector.awss3.adapter.out.local; + +import io.camunda.connector.fileapi.exceptions.LocalFileException; +import io.camunda.connector.awss3.out.local.LocalFileAdapter; +import lombok.SneakyThrows; +import org.junit.jupiter.api.AfterAll; +import org.junit.jupiter.api.BeforeAll; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Test; + +import java.io.IOException; +import java.nio.charset.StandardCharsets; +import java.nio.file.Files; +import java.nio.file.Path; + +import static org.assertj.core.api.Assertions.*; +class LocalFileAdapterTest { + + private static Path baseDir; + private LocalFileAdapter adapter; + + @BeforeAll + static void generateTempDir() throws IOException { + baseDir = Files.createTempDirectory("unit-test-"); + } + + @AfterAll + static void deleteTempDir() throws IOException { + for (Path path : baseDir) { + Files.deleteIfExists(path); + } + } + + @BeforeEach + void setUp() { + adapter = new LocalFileAdapter(baseDir); + } + + @Test + @SneakyThrows(LocalFileException.class) + void file_is_written_and_loaded() throws IOException { + adapter.saveFile("foo".getBytes(StandardCharsets.UTF_8), "foo.txt"); + byte[] bytes = adapter.loadFile("foo.txt"); + assertThat(new String(bytes)).isEqualTo("foo"); + adapter.deleteFile("foo.txt"); + } + + @Test + void parent_dirs_are_created() throws IOException { + Path resolve = baseDir.resolve(Path.of("1", "2", "3", "4")); + assertThat(Files.exists(resolve)).isFalse(); + adapter.saveFile("foo".getBytes(StandardCharsets.UTF_8), "1/2/3/4/foo.txt"); + assertThat(Files.exists(resolve)).isTrue(); + } + + @Test + void loading_missing_file_throws_exception() { + assertThatThrownBy(() -> adapter.loadFile("unknown.txt")) + .isExactlyInstanceOf(LocalFileException.class) + .hasMessageContaining("unknown.txt"); + } + + @Test + void saving_existing_file_throws_exception() throws IOException { + adapter.saveFile("text".getBytes(StandardCharsets.UTF_8), "known.txt"); + assertThatThrownBy(() -> adapter.saveFile("new text".getBytes(StandardCharsets.UTF_8), "known.txt")) + .isExactlyInstanceOf(LocalFileException.class) + .hasMessageContaining("known.txt"); + } + + @Test + void deleting_missing_file_throws_no_exception() { + assertThatNoException() + .isThrownBy(() -> adapter.deleteFile("anyfile.txt")); + } +} \ No newline at end of file diff --git a/connectors/aws/aws-s3/connector-aws-s3/src/test/resources/invoices/invoice.txt b/connectors/aws/aws-s3/connector-aws-s3/src/test/resources/invoices/invoice.txt new file mode 100644 index 0000000000..e2e107ac61 --- /dev/null +++ b/connectors/aws/aws-s3/connector-aws-s3/src/test/resources/invoices/invoice.txt @@ -0,0 +1 @@ +123456789 \ No newline at end of file diff --git a/connectors/aws/aws-s3/fileapi/pom.xml b/connectors/aws/aws-s3/fileapi/pom.xml new file mode 100644 index 0000000000..2f89144b21 --- /dev/null +++ b/connectors/aws/aws-s3/fileapi/pom.xml @@ -0,0 +1,23 @@ + + + 4.0.0 + + io.camunda.connector + connector-aws-s3-parent + 8.7.0-SNAPSHOT + ../pom.xml + + + connector-aws-fileapi + io.camunda.connector.fileapi + connector-aws-fileapi + + + 21 + 21 + UTF-8 + + + \ No newline at end of file diff --git a/connectors/aws/aws-s3/fileapi/src/main/java/io/camunda/connector/fileapi/LocalFileCommand.java b/connectors/aws/aws-s3/fileapi/src/main/java/io/camunda/connector/fileapi/LocalFileCommand.java new file mode 100644 index 0000000000..4708051b01 --- /dev/null +++ b/connectors/aws/aws-s3/fileapi/src/main/java/io/camunda/connector/fileapi/LocalFileCommand.java @@ -0,0 +1,14 @@ +package io.camunda.connector.fileapi; + +import java.io.IOException; +import java.nio.file.Path; + +public interface LocalFileCommand { + + Path saveFile(byte[] content, String filePath) throws IOException; + + byte[] loadFile(String filePath) throws IOException; + + void deleteFile(String filePath) throws IOException; + +} diff --git a/connectors/aws/aws-s3/fileapi/src/main/java/io/camunda/connector/fileapi/ProcessFileCommand.java b/connectors/aws/aws-s3/fileapi/src/main/java/io/camunda/connector/fileapi/ProcessFileCommand.java new file mode 100644 index 0000000000..07729f02c0 --- /dev/null +++ b/connectors/aws/aws-s3/fileapi/src/main/java/io/camunda/connector/fileapi/ProcessFileCommand.java @@ -0,0 +1,15 @@ +package io.camunda.connector.fileapi; + +import io.camunda.connector.fileapi.model.RequestData; + +import java.io.IOException; + +public interface ProcessFileCommand { + + RequestData uploadFile(RequestData request) throws IOException; + + RequestData deleteFile(RequestData request) throws IOException; + + RequestData downloadFile(RequestData request) throws IOException; + +} diff --git a/connectors/aws/aws-s3/fileapi/src/main/java/io/camunda/connector/fileapi/ProcessFileService.java b/connectors/aws/aws-s3/fileapi/src/main/java/io/camunda/connector/fileapi/ProcessFileService.java new file mode 100644 index 0000000000..15a7d17fe4 --- /dev/null +++ b/connectors/aws/aws-s3/fileapi/src/main/java/io/camunda/connector/fileapi/ProcessFileService.java @@ -0,0 +1,43 @@ +package io.camunda.connector.fileapi; +import io.camunda.connector.fileapi.model.FileContent; +import io.camunda.connector.fileapi.model.RequestData; + +import java.io.IOException; +import java.util.Objects; + +public class ProcessFileService implements ProcessFileCommand { + + private final RemoteFileCommand remoteFileCommand; + private final LocalFileCommand localFileCommand; + + public ProcessFileService(RemoteFileCommand remoteFileCommand, LocalFileCommand localFileCommand) { + this.remoteFileCommand = remoteFileCommand; + this.localFileCommand = localFileCommand; + } + + public RequestData uploadFile(RequestData request) throws IOException { + String contentType = Objects.requireNonNull(request.getContentType(), "Content type must be set"); + byte[] content = localFileCommand.loadFile(request.getFilePath()); + remoteFileCommand.putFile(request, FileContent.builder() + .content(content) + .contentLength((long) content.length) + .contentType(contentType) + .build() + ); + return request; + } + + public RequestData deleteFile(RequestData request) throws IOException { + localFileCommand.deleteFile(request.getFilePath()); + remoteFileCommand.deleteFile(request); + return request; + } + + public RequestData downloadFile(RequestData request) throws IOException { + FileContent response = remoteFileCommand.getFile(request); + localFileCommand.saveFile(response.getContent(), request.getFilePath()); + // overwrite with actual content type + request.setContentType(request.getContentType()); + return request; + } +} diff --git a/connectors/aws/aws-s3/fileapi/src/main/java/io/camunda/connector/fileapi/RemoteFileCommand.java b/connectors/aws/aws-s3/fileapi/src/main/java/io/camunda/connector/fileapi/RemoteFileCommand.java new file mode 100644 index 0000000000..872d2abaea --- /dev/null +++ b/connectors/aws/aws-s3/fileapi/src/main/java/io/camunda/connector/fileapi/RemoteFileCommand.java @@ -0,0 +1,12 @@ +package io.camunda.connector.fileapi; + +import java.io.IOException; + +import io.camunda.connector.fileapi.model.FileContent; +import io.camunda.connector.fileapi.model.RequestData; + +public interface RemoteFileCommand { + void deleteFile(RequestData requestData); + void putFile(RequestData requestData, FileContent fileContent) throws IOException; + FileContent getFile(RequestData requestData) throws IOException; +} diff --git a/connectors/aws/aws-s3/fileapi/src/main/java/io/camunda/connector/fileapi/exceptions/LocalFileException.java b/connectors/aws/aws-s3/fileapi/src/main/java/io/camunda/connector/fileapi/exceptions/LocalFileException.java new file mode 100644 index 0000000000..43bb64812a --- /dev/null +++ b/connectors/aws/aws-s3/fileapi/src/main/java/io/camunda/connector/fileapi/exceptions/LocalFileException.java @@ -0,0 +1,9 @@ +package io.camunda.connector.fileapi.exceptions; + +public class LocalFileException extends RuntimeException { + + public LocalFileException(String message) { + super(message); + } + +} diff --git a/connectors/aws/aws-s3/fileapi/src/main/java/io/camunda/connector/fileapi/exceptions/RemoteFileException.java b/connectors/aws/aws-s3/fileapi/src/main/java/io/camunda/connector/fileapi/exceptions/RemoteFileException.java new file mode 100644 index 0000000000..c757e718e2 --- /dev/null +++ b/connectors/aws/aws-s3/fileapi/src/main/java/io/camunda/connector/fileapi/exceptions/RemoteFileException.java @@ -0,0 +1,9 @@ +package io.camunda.connector.fileapi.exceptions; + +public class RemoteFileException extends RuntimeException { + + public RemoteFileException(String message) { + super(message); + } + +} diff --git a/connectors/aws/aws-s3/fileapi/src/main/java/io/camunda/connector/fileapi/model/FileContent.java b/connectors/aws/aws-s3/fileapi/src/main/java/io/camunda/connector/fileapi/model/FileContent.java new file mode 100644 index 0000000000..b3a0b59894 --- /dev/null +++ b/connectors/aws/aws-s3/fileapi/src/main/java/io/camunda/connector/fileapi/model/FileContent.java @@ -0,0 +1,14 @@ +package io.camunda.connector.fileapi.model; + +import lombok.Builder; +import lombok.Data; + +@Data +@Builder +public class FileContent { + + private byte[] content; + private String contentType; + private Long contentLength; + +} diff --git a/connectors/aws/aws-s3/fileapi/src/main/java/io/camunda/connector/fileapi/model/RequestData.java b/connectors/aws/aws-s3/fileapi/src/main/java/io/camunda/connector/fileapi/model/RequestData.java new file mode 100644 index 0000000000..92e04f6e2f --- /dev/null +++ b/connectors/aws/aws-s3/fileapi/src/main/java/io/camunda/connector/fileapi/model/RequestData.java @@ -0,0 +1,18 @@ +package io.camunda.connector.fileapi.model; + +import lombok.Builder; +import lombok.Data; + +@Data +@Builder +public class RequestData { + + private String authenticationKey; + private String authenticationSecret; + private String bucket; + private String key; + private String region; + private String filePath; + private String contentType; + +} diff --git a/connectors/aws/aws-s3/fileapi/src/test/java/ProcessFileServiceTest.java b/connectors/aws/aws-s3/fileapi/src/test/java/ProcessFileServiceTest.java new file mode 100644 index 0000000000..7ba6573731 --- /dev/null +++ b/connectors/aws/aws-s3/fileapi/src/test/java/ProcessFileServiceTest.java @@ -0,0 +1,97 @@ +import io.camunda.connector.fileapi.LocalFileCommand; +import io.camunda.connector.fileapi.ProcessFileService; +import io.camunda.connector.fileapi.RemoteFileCommand; +import io.camunda.connector.fileapi.model.FileContent; +import io.camunda.connector.fileapi.model.RequestData; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.extension.ExtendWith; +import org.mockito.ArgumentCaptor; +import org.mockito.Mock; +import org.mockito.junit.jupiter.MockitoExtension; + +import java.io.IOException; +import java.nio.charset.StandardCharsets; +import java.nio.file.Path; + +import static org.mockito.ArgumentMatchers.anyString; +import static org.mockito.ArgumentMatchers.eq; +import static org.mockito.Mockito.*; +import static org.assertj.core.api.Assertions.assertThat; + +@ExtendWith(MockitoExtension.class) +class ProcessFileServiceTest { + + @Mock + RemoteFileCommand remoteFileCommand; + + @Mock + LocalFileCommand localFileCommand; + + private ProcessFileService service; + + @BeforeEach + void setUp() { + service = new ProcessFileService(remoteFileCommand, localFileCommand); + } + + @Test + void file_service_for_upload_called_as_expected() throws IOException { + // given + RequestData requestData = RequestData.builder() + .filePath("myfile.txt") + .contentType("contentType") + .build(); + byte[] expectedContent = "foo".getBytes(StandardCharsets.UTF_8); + when(localFileCommand.loadFile(anyString())).thenReturn(expectedContent); + + // when + RequestData response = service.uploadFile(requestData); + + // then + verify(localFileCommand, times(1)).loadFile(eq("myfile.txt")); + ArgumentCaptor argumentCaptor = ArgumentCaptor.forClass(FileContent.class); + verify(remoteFileCommand, times(1)).putFile(eq(requestData), argumentCaptor.capture()); + FileContent value = argumentCaptor.getValue(); + assertThat(value.getContent()).as("content").isEqualTo(expectedContent); + assertThat(value.getContentLength()).as("content length").isEqualTo(expectedContent.length); + assertThat(value.getContentType()).as("content type").isEqualTo("contentType"); + assertThat(response).isEqualTo(requestData); + } + + @Test + void file_service_for_delete_called_as_expected() throws IOException { + // given + RequestData requestData = RequestData.builder() + .filePath("myfile.txt") + .build(); + + // when + RequestData response = service.deleteFile(requestData); + + // then + verify(localFileCommand, times(1)).deleteFile(eq("myfile.txt")); + verify(remoteFileCommand, times(1)).deleteFile(eq(requestData)); + assertThat(response).isEqualTo(requestData); + } + + @Test + void file_service_for_download_called_as_expected() throws IOException { + // given + RequestData requestData = RequestData.builder() + .filePath("myfile.txt") + .contentType("contentType") + .build(); + byte[] expectedContent = "foo".getBytes(StandardCharsets.UTF_8); + when(remoteFileCommand.getFile(eq(requestData))).thenReturn(FileContent.builder().content(expectedContent).build()); + when(localFileCommand.saveFile(eq(expectedContent), eq("myfile.txt"))).thenReturn(Path.of("myfile.txt")); + + // when + RequestData response = service.downloadFile(requestData); + + // then + verify(remoteFileCommand, times(1)).getFile(eq(requestData)); + verify(localFileCommand, times(1)).saveFile(eq(expectedContent), eq("myfile.txt")); + assertThat(response).isEqualTo(requestData); + } +} \ No newline at end of file diff --git a/connectors/aws/aws-s3/pom.xml b/connectors/aws/aws-s3/pom.xml new file mode 100644 index 0000000000..c669b7c079 --- /dev/null +++ b/connectors/aws/aws-s3/pom.xml @@ -0,0 +1,59 @@ + + + 4.0.0 + + io.camunda.connector + connector-aws-parent + 8.7.0-SNAPSHOT + ../pom.xml + + + aws-connector-s3-parent + connector-aws-s3-parent + Parent POM for AWS S3 + pom + 2024 + + + connector-aws-s3 + fileapi + + + + 21 + 21 + UTF-8 + + + + + Camunda Self-Managed Free Edition license + + https://camunda.com/legal/terms/cloud-terms-and-conditions/camunda-cloud-self-managed-free-edition-terms/ + + + + Camunda Self-Managed Enterprise Edition license + + + + + + io.camunda.connector + connector-aws-base + ${project.version} + + + software.amazon.awssdk + s3 + 2.29.9 + + + org.projectlombok + lombok + + + + \ No newline at end of file diff --git a/connectors/aws/pom.xml b/connectors/aws/pom.xml index 81eb5e5fda..79d13f0caf 100644 --- a/connectors/aws/pom.xml +++ b/connectors/aws/pom.xml @@ -26,6 +26,9 @@ aws-sagemaker aws-bedrock aws-textract + aws-s3 + aws-s3/fileapi + aws-s3/connector-aws-s3