CASSGO-43: externally-defined type registration #1855
+2,701
−1,217
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Here's an example of a JSONB type: package mypackage
import "github.com/gocql/gocql"
var typeJSONB gocql.Type = 0x0080
func init() {
gocql.RegisterType(typeJSONB, "jsonb", jsonbCQLType{})
}
// jsonbCQLType implements the gocql.CQLType interface
type jsonbCQLType struct {
}
// Params implements the gocql.CQLType interface
func (j jsonbCQLType) Params(proto int) []reflect.Type {
return nil
}
// TypeInfoFromParams implements the gocql.CQLType interface
func (j jsonbCQLType) TypeInfoFromParams(proto int, params []interface{}) TypeInfo {
if len(params) != 0 {
panic(fmt.Errorf("expected 0 param for jsonb type, got %d", len(params)))
}
return typeJSONB
}
// TypeInfoFromString implements the gocql.CQLType interface
func (j jsonbCQLType) TypeInfoFromString(proto int, name string) TypeInfo {
if name != "" {
panic(fmt.Errorf("expected empty name for jsonb type, got %s", name))
}
return typeJSONB
}
// Marshal implements the gocql.CQLType interface
func (j jsonbCQLType) Marshal(info gocql.TypeInfo, value interface{}) ([]byte, error) {
return gocql.Marshal(gocql.TypeBlob, value)
}
// Unmarshal implements the gocql.CQLType interface
func (j jsonbCQLType) Unmarshal(info gocql.TypeInfo, value []byte, dest interface{}) error {
return gocql.Unmarshal(gocql.TypeBlob, value, dest)
}Or they can write their own marshal/unmarshal methods: func marshalJSONB(info gocql.TypeInfo, value interface{}) ([]byte, error) {
switch v := value.(type) {
case gocql.Marshaler:
return v.MarshalCQL(info)
case json.RawMessage:
return v, nil
case string:
return []byte(v), nil
case []byte:
return v, nil
}
if value == nil {
return nil, nil
}
rv := reflect.ValueOf(value)
t := rv.Type()
k := t.Kind()
switch {
case k == reflect.String:
return []byte(rv.String()), nil
case k == reflect.Slice && t.Elem().Kind() == reflect.Uint8:
return rv.Bytes(), nil
}
return nil, gocql.MarshalError(fmt.Sprintf("can not marshal %T into jsonb", value))
}
var typeSliceOfBytes = reflect.TypeOf([]byte(nil))
func unmarshalJSONB(info gocql.TypeInfo, data []byte, value interface{}) error {
switch v := value.(type) {
case gocql.Unmarshaler:
return v.UnmarshalCQL(info, data)
case *json.RawMessage:
if data != nil {
*v = append((*v)[:0], data...)
} else {
*v = nil
}
return nil
case *string:
*v = string(data)
return nil
case *[]byte:
if data != nil {
*v = append((*v)[:0], data...)
} else {
*v = nil
}
return nil
case *interface{}:
if data != nil {
*v = append(json.RawMessage{}, data...)
} else {
*v = nil
}
return nil
}
rv := reflect.ValueOf(value)
if rv.Kind() != reflect.Ptr {
return gocql.UnmarshalError(fmt.Sprintf("can not unmarshal jsonb into non-pointer %T", value))
}
rv = rv.Elem()
t := rv.Type()
k := t.Kind()
switch {
case k == reflect.String:
rv.SetString(string(data))
return nil
case k == reflect.Slice && t.Elem().Kind() == reflect.Uint8:
if data != nil {
if rv.Type() != typeSliceOfBytes {
rv.Set(reflect.AppendSlice(rv.Slice(0, 0), reflect.ValueOf(data).Convert(t)))
} else {
rv.Set(reflect.AppendSlice(rv.Slice(0, 0), reflect.ValueOf(data)))
}
} else {
rv.Set(reflect.Zero(t))
}
return nil
}
return gocql.UnmarshalError(fmt.Sprintf("can not unmarshal jsonb into %T", value))
} |
jameshartig
force-pushed
the
external-types
branch
from
6b5029b to
71ccb42
Compare
jameshartig
force-pushed
the
external-types
branch
3 times, most recently
from
61387c3 to
0abeec5
Compare
jameshartig
force-pushed
the
external-types
branch
from
0abeec5 to
963a6f6
Compare
|
I could use some tests of custom types but given that all of the native types are using most of the same code, I feel confident in the guts of the code. |
jameshartig
force-pushed
the
external-types
branch
2 times, most recently
from
1c53a5b to
0203176
Compare
That makes sense. Going to switch this back to draft so I can resolve those conflicts and take a stab at cleaning up the UDT case we talked about in Slack. |
|
I simplified the |
jameshartig
force-pushed
the
external-types
branch
from
0203176 to
6b5029b
Compare
|
Initially, some of the unmarshal benchmarks were worse so I inlined some of the type switching which is a bit gross but massively improved the benchmarks. |
jameshartig
force-pushed
the
external-types
branch
3 times, most recently
from
4ce416c to
88ed1af
Compare
The new RegisterType function can be used to register externally-defined
types. You'll need to define your own marshalling and unmarshalling code
as well as a TypeInfo implementation. The name and id MUST not collide
with existing and future native CQL types.
A lot of the type handling was refactored to use the new format for
native types.
Pointers to empty interfaces are accepted by Scan and are used to build
the maps in MapScan and SliceMap.
inet columns are now unmarshaled as net.IP which is a breaking change.
goos: linux
goarch: amd64
pkg: github.com/gocql/gocql
cpu: AMD EPYC 7B13
│ old.txt │ new.txt │
│ sec/op │ sec/op vs base │
SingleConn-16 25.92µ ± 1% 25.74µ ± 1% ~ (p=0.481 n=10)
ParseRowsFrame-16 1053.5n ± 2% 737.1n ± 1% -30.03% (p=0.000 n=10)
Unmarshal_BigInt-16 18.91n ± 0% 21.67n ± 0% +14.60% (p=0.000 n=10)
Unmarshal_Blob-16 19.38n ± 0% 21.62n ± 1% +11.56% (p=0.000 n=10)
Unmarshal_Boolean-16 16.57n ± 0% 19.20n ± 1% +15.84% (p=0.000 n=10)
Unmarshal_Date-16 18.48n ± 0% 25.71n ± 1% +39.15% (p=0.000 n=10)
Unmarshal_Decimal-16 196.5n ± 2% 206.9n ± 1% +5.32% (p=0.000 n=10)
Unmarshal_Double-16 16.27n ± 1% 18.57n ± 1% +14.14% (p=0.000 n=10)
Unmarshal_Duration-16 28.55n ± 0% 29.23n ± 0% +2.40% (p=0.000 n=10)
Unmarshal_Float-16 16.65n ± 1% 18.83n ± 0% +13.09% (p=0.000 n=10)
Unmarshal_Int-16 19.47n ± 1% 22.92n ± 1% +17.72% (p=0.000 n=10)
Unmarshal_Inet-16 30.28n ± 1% 41.27n ± 0% +36.27% (p=0.000 n=10)
Unmarshal_SmallInt-16 19.09n ± 0% 21.93n ± 1% +14.91% (p=0.000 n=10)
Unmarshal_Time-16 16.57n ± 0% 18.80n ± 0% +13.39% (p=0.000 n=10)
Unmarshal_Timestamp-16 16.60n ± 0% 18.80n ± 1% +13.22% (p=0.000 n=10)
Unmarshal_TinyInt-16 16.66n ± 0% 22.61n ± 1% +35.73% (p=0.000 n=10)
Unmarshal_UUID-16 17.87n ± 0% 19.12n ± 1% +6.97% (p=0.000 n=10)
Unmarshal_Varchar-16 18.77n ± 1% 21.62n ± 1% +15.24% (p=0.000 n=10)
Unmarshal_List-16 217.1n ± 0% 222.0n ± 1% +2.28% (p=0.000 n=10)
Unmarshal_Set-16 219.5n ± 1% 221.3n ± 1% +0.82% (p=0.001 n=10)
Unmarshal_Map-16 392.7n ± 1% 403.0n ± 1% +2.62% (p=0.000 n=10)
FramerReadTypeInfo-16 224.7n ± 4% 228.5n ± 1% ~ (p=0.138 n=10)
ConnStress-16 9.255µ ± 15% 10.312µ ± 10% +11.42% (p=0.006 n=10)
ConnRoutingKey-16 178.4n ± 3% 211.7n ± 1% +18.60% (p=0.000 n=10)
WikiCreateSchema-16 520.9m ± 3% 555.3m ± 3% +6.62% (p=0.000 n=10)
WikiCreatePages-16 1.492m ± 1% 1.504m ± 2% ~ (p=0.165 n=10)
WikiSelectAllPages-16 1.959m ± 1% 1.998m ± 3% ~ (p=0.075 n=10)
WikiSelectSinglePage-16 1.496m ± 1% 1.510m ± 1% ~ (p=0.052 n=10)
WikiSelectPageCount-16 1.658m ± 2% 1.674m ± 1% ~ (p=0.075 n=10)
Unmarshal_TupleStrings-16 401.6n ± 1%
Unmarshal_TupleInterfaces-16 424.2n ± 1%
geomean 502.2n 538.0n +9.11%
│ old.txt │ new.txt │
│ B/op │ B/op vs base │
SingleConn-16 3.110Ki ± 0% 3.110Ki ± 0% ~ (p=0.224 n=10)
ParseRowsFrame-16 1112.0 ± 0% 856.0 ± 0% -23.02% (p=0.000 n=10)
Unmarshal_BigInt-16 0.000 ± 0% 0.000 ± 0% ~ (p=1.000 n=10) ¹
Unmarshal_Blob-16 0.000 ± 0% 0.000 ± 0% ~ (p=1.000 n=10) ¹
Unmarshal_Boolean-16 0.000 ± 0% 0.000 ± 0% ~ (p=1.000 n=10) ¹
Unmarshal_Date-16 0.000 ± 0% 0.000 ± 0% ~ (p=1.000 n=10) ¹
Unmarshal_Decimal-16 96.00 ± 0% 96.00 ± 0% ~ (p=1.000 n=10) ¹
Unmarshal_Double-16 0.000 ± 0% 0.000 ± 0% ~ (p=1.000 n=10) ¹
Unmarshal_Duration-16 0.000 ± 0% 0.000 ± 0% ~ (p=1.000 n=10) ¹
Unmarshal_Float-16 0.000 ± 0% 0.000 ± 0% ~ (p=1.000 n=10) ¹
Unmarshal_Int-16 0.000 ± 0% 0.000 ± 0% ~ (p=1.000 n=10) ¹
Unmarshal_Inet-16 4.000 ± 0% 4.000 ± 0% ~ (p=1.000 n=10) ¹
Unmarshal_SmallInt-16 0.000 ± 0% 0.000 ± 0% ~ (p=1.000 n=10) ¹
Unmarshal_Time-16 0.000 ± 0% 0.000 ± 0% ~ (p=1.000 n=10) ¹
Unmarshal_Timestamp-16 0.000 ± 0% 0.000 ± 0% ~ (p=1.000 n=10) ¹
Unmarshal_TinyInt-16 0.000 ± 0% 0.000 ± 0% ~ (p=1.000 n=10) ¹
Unmarshal_UUID-16 0.000 ± 0% 0.000 ± 0% ~ (p=1.000 n=10) ¹
Unmarshal_Varchar-16 0.000 ± 0% 0.000 ± 0% ~ (p=1.000 n=10) ¹
Unmarshal_List-16 32.00 ± 0% 32.00 ± 0% ~ (p=1.000 n=10) ¹
Unmarshal_Set-16 32.00 ± 0% 32.00 ± 0% ~ (p=1.000 n=10) ¹
Unmarshal_Map-16 248.0 ± 0% 248.0 ± 0% ~ (p=1.000 n=10) ¹
FramerReadTypeInfo-16 160.00 ± 0% 96.00 ± 0% -40.00% (p=0.000 n=10)
ConnStress-16 2.670Ki ± 0% 2.667Ki ± 0% ~ (p=0.068 n=10)
ConnRoutingKey-16 32.00 ± 0% 32.00 ± 0% ~ (p=1.000 n=10) ¹
WikiCreateSchema-16 53.38Ki ± 1% 53.03Ki ± 1% -0.65% (p=0.045 n=10)
WikiCreatePages-16 3.866Ki ± 0% 3.864Ki ± 0% ~ (p=0.563 n=10)
WikiSelectAllPages-16 28.74Ki ± 0% 28.74Ki ± 0% ~ (p=1.000 n=10)
WikiSelectSinglePage-16 3.305Ki ± 0% 3.303Ki ± 0% ~ (p=0.426 n=10)
WikiSelectPageCount-16 2.816Ki ± 0% 2.815Ki ± 0% ~ (p=0.319 n=10)
Unmarshal_TupleStrings-16 126.0 ± 0%
Unmarshal_TupleInterfaces-16 126.0 ± 0%
geomean ² -2.66% ²
¹ all samples are equal
² summaries must be >0 to compute geomean
│ old.txt │ new.txt │
│ allocs/op │ allocs/op vs base │
SingleConn-16 37.00 ± 0% 37.00 ± 0% ~ (p=1.000 n=10) ¹
ParseRowsFrame-16 21.00 ± 0% 13.00 ± 0% -38.10% (p=0.000 n=10)
Unmarshal_BigInt-16 0.000 ± 0% 0.000 ± 0% ~ (p=1.000 n=10) ¹
Unmarshal_Blob-16 0.000 ± 0% 0.000 ± 0% ~ (p=1.000 n=10) ¹
Unmarshal_Boolean-16 0.000 ± 0% 0.000 ± 0% ~ (p=1.000 n=10) ¹
Unmarshal_Date-16 0.000 ± 0% 0.000 ± 0% ~ (p=1.000 n=10) ¹
Unmarshal_Decimal-16 4.000 ± 0% 4.000 ± 0% ~ (p=1.000 n=10) ¹
Unmarshal_Double-16 0.000 ± 0% 0.000 ± 0% ~ (p=1.000 n=10) ¹
Unmarshal_Duration-16 0.000 ± 0% 0.000 ± 0% ~ (p=1.000 n=10) ¹
Unmarshal_Float-16 0.000 ± 0% 0.000 ± 0% ~ (p=1.000 n=10) ¹
Unmarshal_Int-16 0.000 ± 0% 0.000 ± 0% ~ (p=1.000 n=10) ¹
Unmarshal_Inet-16 1.000 ± 0% 1.000 ± 0% ~ (p=1.000 n=10) ¹
Unmarshal_SmallInt-16 0.000 ± 0% 0.000 ± 0% ~ (p=1.000 n=10) ¹
Unmarshal_Time-16 0.000 ± 0% 0.000 ± 0% ~ (p=1.000 n=10) ¹
Unmarshal_Timestamp-16 0.000 ± 0% 0.000 ± 0% ~ (p=1.000 n=10) ¹
Unmarshal_TinyInt-16 0.000 ± 0% 0.000 ± 0% ~ (p=1.000 n=10) ¹
Unmarshal_UUID-16 0.000 ± 0% 0.000 ± 0% ~ (p=1.000 n=10) ¹
Unmarshal_Varchar-16 0.000 ± 0% 0.000 ± 0% ~ (p=1.000 n=10) ¹
Unmarshal_List-16 2.000 ± 0% 2.000 ± 0% ~ (p=1.000 n=10) ¹
Unmarshal_Set-16 2.000 ± 0% 2.000 ± 0% ~ (p=1.000 n=10) ¹
Unmarshal_Map-16 5.000 ± 0% 5.000 ± 0% ~ (p=1.000 n=10) ¹
FramerReadTypeInfo-16 4.000 ± 0% 4.000 ± 0% ~ (p=1.000 n=10) ¹
ConnStress-16 33.00 ± 0% 33.00 ± 0% ~ (p=1.000 n=10)
ConnRoutingKey-16 3.000 ± 0% 3.000 ± 0% ~ (p=1.000 n=10) ¹
WikiCreateSchema-16 773.0 ± 1% 663.5 ± 1% -14.17% (p=0.000 n=10)
WikiCreatePages-16 51.00 ± 2% 51.00 ± 2% ~ (p=0.628 n=10)
WikiSelectAllPages-16 338.0 ± 0% 338.0 ± 0% ~ (p=1.000 n=10) ¹
WikiSelectSinglePage-16 45.00 ± 0% 45.00 ± 0% ~ (p=1.000 n=10) ¹
WikiSelectPageCount-16 40.00 ± 0% 40.00 ± 0% ~ (p=1.000 n=10) ¹
Unmarshal_TupleStrings-16 8.000 ± 0%
Unmarshal_TupleInterfaces-16 8.000 ± 0%
geomean ² -2.16% ²
¹ all samples are equal
² summaries must be >0 to compute geomean
Patch by James Hartig for CASSGO-43
jameshartig
force-pushed
the
external-types
branch
from
88ed1af to
33fab21
Compare
I do think we can roll some of those back when Go 1.24 comes out and there's improved Swiss table maps. I'll need to rerun the benchmarks but it would be nice to get rid of all of that boilerplate. |
We probably want to be careful with requiring such a recent go version, if it's not crucial then we should delay it as much as possible |
|
Is this ready for review btw? |
Yes, but I would like a chance to write some more tests. I think it's in a good place though for review besides the tests.
Yeah, my comment was meant for the distant future. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
CASSGO-43: externally-defined type registration
The new RegisterType function can be used to register externally-defined
types. You'll need to define your own marshalling and unmarshalling code
as well as a TypeInfo implementation. The name and id MUST not collide
with existing and future native CQL types.
A lot of the type handling was refactored to use the new format for
native types.
Pointers to empty interfaces are accepted by Scan and are used to build
the maps in MapScan and SliceMap.
inet columns are now unmarshaled as net.IP which is a breaking change.