Go Dcp Couchbase

Go Dcp Couchbase streams documents from Couchbase Database Change Protocol (DCP) and writes to Couchbase bucket in near real-time.

Features

• Less resource usage and higher throughput.
• Update multiple documents for a DCP event(see Example).
• Handling different DCP events such as expiration, deletion and mutation(see Example).
• **Managing inflight request size to Couchbase.
• Scale up and down by custom membership algorithms(Couchbase, KubernetesHa, Kubernetes StatefulSet or Static, see examples).
• Easily manageable configurations.

Concepts

General Concept

Merge at target bucket

Example

Struct Config

package main

import (
  "github.com/Trendyol/go-dcp-couchbase"
  "time"

  "github.com/Trendyol/go-dcp-couchbase/config"
  dcpConfig "github.com/Trendyol/go-dcp/config"
)

func main() {
  c, err := dcpcouchbase.NewConnectorBuilder(&config.Config{
    Dcp: dcpConfig.Dcp{
      Hosts:      []string{"localhost:8091"},
      Username:   "user",
      Password:   "password",
      BucketName: "dcp-test",
      Dcp: dcpConfig.ExternalDcp{
        Group: dcpConfig.DCPGroup{
          Name: "groupName",
          Membership: dcpConfig.DCPGroupMembership{
            RebalanceDelay: 3 * time.Second,
          },
        },
      },
      Metadata: dcpConfig.Metadata{
        Config: map[string]string{
          "bucket":     "dcp-test-meta",
          "scope":      "_default",
          "collection": "_default",
        },
        Type: "couchbase",
      },
      Debug: true,
    },
    Couchbase: config.Couchbase{
      Hosts:          []string{"localhost:8091"},
      Username:       "user",
      Password:       "password",
      BucketName:     "dcp-test-backup",
      RequestTimeout: 10 * time.Second,
    },
  }).SetMapper(dcpcouchbase.DefaultMapper).Build()
  if err != nil {
    panic(err)
  }

  defer c.Close()
  c.Start()
}

Configuration

Dcp Configuration

Check out on go-dcp

Couchbase Specific Configuration

Variable	Type	Required	Default	Description
couchbase.hosts	[]string	yes		Couchbase connection urls
couchbase.username	string	yes		Defines Couchbase username
couchbase.password	string	yes		Defines Couchbase password
couchbase.bucketName	string	yes		Defines Couchbase bucket name
couchbase.scopeName	string	no	_default	Defines Couchbase scope name
couchbase.collectionName	string	no	_default	Defines Couchbase collection name
couchbase.maxInflightRequests	int	no	2048	Maximum message count for Couchbase
couchbase.writePoolSizePerNode	int	no	1	Write connection pool size per node
couchbase.requestTimeout	time.Duration	no	3s	Maximum request waiting time
couchbase.secureConnection	bool	no	false	Enables secure connection.
couchbase.rootCAPath	string	no	false	Defines root CA path.
couchbase.maxQueueSize	int	no	2048	The maximum number of requests that can be queued waiting to be sent to a node. 2048 is default. Check this if you get queue overflowed or queue full.
couchbase.connectionBufferSize	uint	no	20971520	Defines connectionBufferSize.
couchbase.connectionTimeout	time.Duration	no	5s	Defines connectionTimeout.

Exposed metrics

For DCP related metrics see also.

Breaking Changes

Date taking effect	Date announced	Change	How to check
December 29, 2023	December 29, 2023	Mapper first arg changed to couchbase.EventContext	Compile project
November 14, 2023	November 14, 2023	Creating connector via builder	Compile project

Contributing

Go Dcp Couchbase is always open for direct contributions. For more information please check our Contribution Guideline document.

License

Released under the MIT License.