Signature::Coercible with user defined implicit casting #14440

jayzhan211 · 2025-02-03T13:37:38Z

Which issue does this PR close?

Part of Epic: Simplify functions signature with LogicalType #13301.
It would be helpful for Fix Type Coercion for UDF Arguments #14268

Rationale for this change

Inspired from Fix Type Coercion for UDF Arguments #14268 from @shehabgamin we used to define some implicit coercion inside datafusion which turns out it is not flexible and doesn't work for all the use case. We turn it explicitly and customizable by default.

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

jayzhan211 · 2025-02-03T13:40:04Z

datafusion/expr-common/src/signature.rs

+
+#[derive(Debug, Clone, Eq, PartialOrd, Hash)]
+pub struct ParameterType {
+    pub param_type: LogicalTypeRef,


param_type: target type of function signature
allowed_casts: implicit coercion allowed to cast to target type.

For example,
param_type: string
allowed_casts: binary, int

Valid: All are casted to string func(string) func(binary) func(int) Invalid: func(float or other types)

I am worried that Vec<Vec<ParameterType>> might become challenging to reason about. It would also be confusing about when one use this new signature rather than Signature::Coercable 🤔

Given this seems very similar to Signature::Coercable, and Signature::Coercable mirrors what we want pretty well, could add some new information there on the allowed coercions rather than an entire new type signature. Something like extending Coercable with rules could be used to coerce to the target type:

pub enum TypeSignature { ... Coercible(Vec<Coercion>), ... }

Where Coercion looks like

struct Coercion { desired_type: TypeSignatureClass, allowed_casts: ... // also includes an option for User Defined? }

🤔

I would like to have a breaking change to Signature::Coercible too, the only concern is whether this cause regression or large impact to downstream

@shehabgamin If we replace CoercibleV2 with Signature::Coercible would it be a large change we should be concerned to?

If not, I'll replace Signature::Coercible with Signature::CoercibleV2.

Signed-off-by: Jay Zhan <[email protected]>

jayzhan211 · 2025-02-07T01:55:22Z

datafusion/functions/src/string/repeat.rs

                vec![
-                    TypeSignatureClass::Native(logical_string()),
-                    TypeSignatureClass::Native(logical_int64()),


In v1 version, any integer is casted to the defined NativeType's default casted type which is i64 in this case.

jayzhan211 · 2025-02-07T01:55:39Z

datafusion/functions/src/string/repeat.rs

+                    // Accept all integer types but cast them to i64
+                    Coercion {
+                        desired_type: TypeSignatureClass::Native(logical_int64()),
+                        allowed_casts: vec![TypeSignatureClass::Integer],


Without this, i32 is rejected.

jayzhan211 · 2025-02-07T02:00:08Z

datafusion/functions/src/string/ascii.rs

+            signature: Signature::coercible_v2(
+                vec![Coercion {
+                    desired_type: TypeSignatureClass::Native(logical_string()),
+                    allowed_casts: vec![TypeSignatureClass::Native(logical_binary())],


Coercing binary to string is now easily customizable

Signed-off-by: Jay Zhan <[email protected]>

alamb · 2025-02-07T19:17:17Z

I will try and review this carefully over the weekend

Maybe @shehabgamin has some time to take a look too

shehabgamin · 2025-02-08T08:25:54Z

I will try and review this carefully over the weekend

Maybe @shehabgamin has some time to take a look too

I will review over the weekend as well!

alamb

Thank you @jayzhan211 -- I really like where this is heading. The Coercion idea I think is the missing concept that DataFusion has been missing

It seems like there are three kinds of Coercion I know about:

Coercing to a (single) specific data type
Coercing to one of a specific logical type (eg. one of the String types)
Coercing two types to a "comparable" / compatible one (where comparison is differently defined)

It seems in all cases, the operation that is desired is:

Given an existing Arrow::DataType and a coercion (that describes a desired type) what, if any cast, can be applied so the coersion is satisfied

If we could encapsulate this into Coercion somehow and update all our operations in terms of Coercion I think we would be able to stop churning this code.

Also once it is in Coercion then there is a natural place for other systems to provide their own coercion rules

Questions:

Can we make Signature::Coercable general enough to represent the same thing as Signature::Numeric and Signature::String ? (if so perhaps we can deprecate them as a follow on PR)

alamb · 2025-02-08T14:38:36Z

datafusion/expr-common/src/signature.rs

-    /// For example, `Coercible(vec![logical_float64()])` accepts
-    /// arguments like `vec![Int32]` or `vec![Float32]`
-    /// since i32 and f32 can be cast to f64
+    /// [`Coercion`] contains not only the desired type but also the allowed casts.


this API makes a lot of sense to me-- in fact I think it is pretty close to being able to express most other signatures.

alamb · 2025-02-08T14:48:48Z

datafusion/expr-common/src/signature.rs

@@ -431,6 +463,35 @@ impl TypeSignature {
    }
 }

+fn get_possible_types_from_signature_classes(


this would make more sense to me as a method on TypeSignatureClass

This function is used in information.schema to list all possible signature combination, but it is too granular than necessary, sometimes we can use NativeType or a new struct that represent a set of DataType and it should be enough.

@goldmedal
If there is function requires Integer, we don't need to list all possible i8, i16, i32 or i64 but integer instead. I think we need a output other than Vec<DataType> but something that could combines both DataType or a set of type similar to NativeType. Otherwise we will generate tons of DataType combination for Coercible signature and I guess it is not readable for information.schema

I think it is "simplified" because of some reasons, if we add all possible types, the combination will be huge. This function is used in information.schema that shows the possible signatures a function has. Is listing all possible DataType combination helpful?

@jayzhan211 I'm moving the conversation to this thread to consolidate.

I totally understand your concern, here are my thoughts:

I think we should rename this function and add a comment for clarity.

information.schema is not reporting accurate information if we only list a subset of possible types. I like your idea of using other output that isn't Vec<DataType>.

alamb · 2025-02-08T15:15:13Z

datafusion/expr-common/src/signature.rs

@@ -225,6 +229,45 @@ impl Display for TypeSignatureClass {
    }
 }

+impl TypeSignatureClass {
+    /// Returns the default cast type for the given `TypeSignatureClass`.  


I am not familiar what a "default cast type" is

I looked at the definition of LogicalType:defaut_cast_for and that didn't help me either (no comments)
https://github.com/apache/datafusion/blob/08d3b656d761a8e0d9e7f7c35c7b76cfec00e095/datafusion/common/src/types/native.rs#L201-L200

🤔

alamb · 2025-02-08T15:35:30Z

datafusion/expr-common/src/signature.rs

+#[derive(Debug, Clone, Eq, PartialOrd)]
+pub struct Coercion {
+    pub desired_type: TypeSignatureClass,
+    pub allowed_casts: Vec<TypeSignatureClass>,


What is allowed_castss? Does that represent the source types that this coercion applies to?

If so, perhaps we could name this filed allowed_source_types

+1 for allowed_source_types

alamb · 2025-02-08T15:37:56Z

datafusion/expr-common/src/signature.rs

@@ -460,6 +521,44 @@ fn get_data_types(native_type: &NativeType) -> Vec<DataType> {
    }
 }

+#[derive(Debug, Clone, Eq, PartialOrd)]
+pub struct Coercion {


I really like the idea of Coercion.

Can this idea be used for user defined coercions or coercions to specific (Arrow) types?

Also, is there any way to allow users to provide their own coercion rules? For example, if Sail / @shehabgamin wants to support converting numeric values to strings automatically, would he be express that?

+1 this would be really great to have!

Similar to ascii that binary to string is supported, you add numeric types in allowed_source_types

@jayzhan211 , I believe @alamb 's question (please correct me if I'm wrong) is about creating functionality for a downstream user to override the default signature of a UDF in order to provide their own coercion rules.

For example, something like this:

let scalar_expr = ScalarExprBuilder::new(AsciiFunc::new(), args) .with_signature(Signature::any(1, Volatility::Immutable)) .build() .map(Arc::new)?;

This is the conversation we were having here as well:
#14296 (comment)
#14296 (comment)

shehabgamin

@jayzhan211 Thanks for taking on this initiative, I like the direction of where this is going!

My biggest take away is that is_matched_type + default_casted_type together seem too ambiguous/restrictive as is. Together they add assumptions tightly coupled to the implementation which makes it hard for users to tailor logic according to their needs. I also worry that there isn't clarity into what's going on unless you carefully dig into the code. IMO the logic should be very simple and as follows:

If source_type is desired_type OR source_type is one of allowed_casts, then if needed: desired_type.default_cast_for(source_type).

It would also be really great if we could find an elegant way to handle DataType::Timestamp with a wildcard for both TimeUnit and TZ (where the TZ wildcard also matches None). Some effort has already been made regarding the signature (which might be sufficient for now), but it would also be useful to be able to use the wildcard in the return_type function as well.

shehabgamin · 2025-02-09T04:55:54Z

datafusion/expr-common/src/signature.rs

+            }
+            // This is an existing use case for casting string to timestamp, since we don't have specific unit and timezone from string,
+            // so we use the default timestamp type with nanosecond precision and no timezone
+            // TODO: Consider allowing the user to specify the default timestamp type instead of having it predefined in DataFusion when we have more use cases


The user should be allowed to specify the default timestamp type.

shehabgamin · 2025-02-09T05:05:21Z

datafusion/expr-common/src/signature.rs

+    match signature_classes {
+        TypeSignatureClass::Native(l) => get_data_types(l.native()),
+        TypeSignatureClass::Timestamp => {
+            vec![


The rest of the TimeUnits are missing. It would be great if TIMEZONE_WILDCARD encompassed None and it would also be great if there was a wildcard for TimeUnit.

I think it is "simplified" because of some reasons, if we add all possible types, the combination will be huge. This function is used in information.schema that shows the possible signatures a function has. Is listing all possible DataType combination helpful?

shehabgamin · 2025-02-09T05:06:02Z

datafusion/expr-common/src/signature.rs

+            vec![DataType::Date64]
+        }
+        TypeSignatureClass::Time => {
+            vec![DataType::Time64(TimeUnit::Nanosecond)]


This should encompass all possible DataType::Time32 and DataType::Time64. Also same comment as above regarding TimeUnit wildcard.

shehabgamin · 2025-02-09T05:08:43Z

datafusion/expr-common/src/signature.rs

+            ]
+        }
+        TypeSignatureClass::Date => {
+            vec![DataType::Date64]


This should be DataType::Date32 and DataType::Date64

shehabgamin · 2025-02-09T05:10:04Z

datafusion/expr-common/src/signature.rs

+            vec![DataType::Time64(TimeUnit::Nanosecond)]
+        }
+        TypeSignatureClass::Interval => {
+            vec![DataType::Interval(IntervalUnit::DayTime)]


This should encompass all possible DataType::Interval

shehabgamin · 2025-02-09T05:12:07Z

datafusion/expr-common/src/signature.rs

+            vec![DataType::Interval(IntervalUnit::DayTime)]
+        }
+        TypeSignatureClass::Duration => {
+            vec![DataType::Duration(TimeUnit::Nanosecond)]


This should encompass all possible DataType::Duration. Also same comment as above regarding TimeUnit wildcard.

shehabgamin · 2025-02-09T05:12:23Z

datafusion/expr-common/src/signature.rs

+            vec![DataType::Duration(TimeUnit::Nanosecond)]
+        }
+        TypeSignatureClass::Integer => {
+            vec![DataType::Int64]


This should encompass all possible ints.

shehabgamin · 2025-02-09T05:14:07Z

datafusion/expr-common/src/signature.rs

+#[derive(Debug, Clone, Eq, PartialOrd)]
+pub struct Coercion {
+    pub desired_type: TypeSignatureClass,
+    pub allowed_casts: Vec<TypeSignatureClass>,


+1 for allowed_source_types

shehabgamin · 2025-02-09T05:16:46Z

datafusion/expr-common/src/signature.rs

@@ -460,6 +521,44 @@ fn get_data_types(native_type: &NativeType) -> Vec<DataType> {
    }
 }

+#[derive(Debug, Clone, Eq, PartialOrd)]
+pub struct Coercion {


+1 this would be really great to have!

shehabgamin · 2025-02-09T05:27:34Z

datafusion/common/src/types/builtin.rs

@@ -47,3 +49,11 @@ singleton!(LOGICAL_FLOAT64, logical_float64, Float64);
 singleton!(LOGICAL_DATE, logical_date, Date);
 singleton!(LOGICAL_BINARY, logical_binary, Binary);
 singleton!(LOGICAL_STRING, logical_string, String);
+
+// TODO: Extend macro
+// TODO: Should we use LOGICAL_TIMESTAMP_NANO to distinguish unit and timzeone?


It would be great if there was a wildcard to match all the units and timezones.

TypeSignatureClass::Timestamp acts as a wildcard, covering all possible units and timezones. If you're considering introducing a similar concept in NativeType, we should also discuss whether we need to include Numeric, Integer, or Float as part of it.

jayzhan211 · 2025-02-09T10:53:51Z

datafusion/expr-common/src/signature.rs

@@ -209,14 +210,13 @@ impl TypeSignature {
 #[derive(Debug, Clone, Eq, PartialEq, PartialOrd, Hash)]
 pub enum TypeSignatureClass {
    Timestamp,
-    Date,


We can use NativeType::Date instead, not need TypeSignatureClass::Date

jayzhan211 · 2025-02-09T11:53:57Z

but it would also be useful to be able to use the wildcard in the return_type function as well.

Using TypeSignatureClass:Timestamp can represent wildcard of timestamp. Define wildcard of timestamp for the return type might not be possible since we return arrow type and it doesn't have wildcard type

jayzhan211 · 2025-02-11T00:53:09Z

datafusion/expr/src/type_coercion/functions.rs

+                .any(|t| is_matched_type(t, &current_logical_type)) {
+                    // If the condition is met which means `implicit coercion`` is provided so we can safely unwrap
+                    let default_casted_type = param.default_casted_type().unwrap();
+                    let casted_type = default_casted_type.default_cast_for(current_type)?;


Didn't find more simplified logic

shehabgamin · 2025-02-11T01:56:25Z

@jayzhan211 I will re-review by tomorrow EOD!

jayzhan211 · 2025-02-11T10:33:22Z

I revisit the binary to string for ascii but it seems either Postgres and Duckdb correctly support it.

postgres=# select ascii('0xa');
 ascii 
-------
    48
(1 row)

postgres=# select ascii(X'a');
ERROR:  function ascii(bit) does not exist
LINE 1: select ascii(X'a');
               ^
HINT:  No function matches the given name and argument types. You might need to add explicit type casts.
postgres=# select ascii(x'a');
ERROR:  function ascii(bit) does not exist
LINE 1: select ascii(x'a');
               ^
HINT:  No function matches the given name and argument types. You might need to add explicit type casts.
postgres=# select ascii('xa');
 ascii 
-------
   120
(1 row)

D select ascii('0xa');
┌──────────────┐
│ ascii('0xa') │
│    int32     │
├──────────────┤
│           48 │
└──────────────┘
D select ascii(x'a');
┌─────────────┐
│ ascii('xa') │
│    int32    │
├─────────────┤
│         120 │
└─────────────┘
D select ascii('xa');
┌─────────────┐
│ ascii('xa') │
│    int32    │
├─────────────┤
│         120 │
└─────────────┘

Note that DuckDB process ascii(x'a') as ascii('xa') which is not binary

shehabgamin · 2025-02-12T05:12:27Z

datafusion/expr-common/src/signature.rs

+    ///
+    /// Get all possible types for `information_schema` from the given `TypeSignature`
+    //
+    // TODO: Make this function private
    pub fn get_possible_types(&self) -> Vec<Vec<DataType>> {


Nice comment, can we rename the function too?

shehabgamin · 2025-02-12T05:25:31Z

@jayzhan211 Should we port any relevant tests from the old PR?
#14268

jayzhan211 · 2025-02-12T05:27:48Z

@jayzhan211 Should we port any relevant tests from the old PR? #14268

or close this first then revisit #14268

BTW, most of the binary-to-string conversions mentioned in #14268 might not be ideal for DataFusion. We should reconsider them.

github-actions bot added logical-expr Logical plan and expressions functions labels Feb 3, 2025

jayzhan211 commented Feb 3, 2025

View reviewed changes

jayzhan211 mentioned this pull request Feb 3, 2025

Fix Type Coercion for UDF Arguments #14268

Open

jayzhan211 force-pushed the signature-v2 branch from 23953dd to 806c6a6 Compare February 6, 2025 13:46

github-actions bot added the sqllogictest SQL Logic Tests (.slt) label Feb 6, 2025

jayzhan211 marked this pull request as draft February 7, 2025 00:38

jayzhan211 added 7 commits February 7, 2025 08:38

coerciblev2

386c7ed

Signed-off-by: Jay Zhan <[email protected]>

repeat

579ffef

Signed-off-by: Jay Zhan <[email protected]>

fix possible types

104da43

replace all coerciblev1

aae48ff

cleanup

f585136

remove specialize logic

bad7348

comment

c99e986

jayzhan211 force-pushed the signature-v2 branch from 806c6a6 to c99e986 Compare February 7, 2025 01:49

github-actions bot added the common Related to common crate label Feb 7, 2025

jayzhan211 changed the title ~~Draft: coercible signature~~ Signature::Coercible with user defined implicit casting Feb 7, 2025

jayzhan211 commented Feb 7, 2025

View reviewed changes

jayzhan211 added 2 commits February 7, 2025 10:15

err msg

07f97d0

ci escape

da84394

jayzhan211 marked this pull request as ready for review February 7, 2025 03:57

jayzhan211 added 3 commits February 7, 2025 20:23

rm coerciblev1

e0a889e

Signed-off-by: Jay Zhan <[email protected]>

fmt

7a78a6d

rename

2f8c2ad

jayzhan211 added the api change Changes the API exposed to users of the crate label Feb 7, 2025

jayzhan211 requested a review from alamb February 7, 2025 13:04

alamb reviewed Feb 8, 2025

View reviewed changes

shehabgamin reviewed Feb 9, 2025

View reviewed changes

jayzhan211 added 4 commits February 9, 2025 16:13

Merge branch 'main' of github.com:apache/datafusion into signature-v2

f02bf6e

rename

8054915

refactor

62da381

make default_casted_type private

231d75b

jayzhan211 commented Feb 9, 2025

View reviewed changes

jayzhan211 added 2 commits February 9, 2025 18:54

cleanup

44250fc

fmt

2f6b5d6

integer

5387e5e

This was referenced Feb 10, 2025

[DISCUSSION] 2025 Q1-Q2 Roadmap #14580

Open

coercion of input types in coalesce leads to type unsupported arrow cast #14581

Open

jayzhan211 commented Feb 11, 2025

View reviewed changes

jayzhan211 requested review from alamb and shehabgamin February 11, 2025 00:54

rm binary for ascii

181f2ab

rm unused

d98e4b6

shehabgamin reviewed Feb 12, 2025

View reviewed changes

jayzhan211 added 3 commits February 12, 2025 13:29

Merge branch 'main' of github.com:apache/datafusion into signature-v2

cb30d91

conflit

2cd8f5f

fmt

6ce159f

Signature::Coercible with user defined implicit casting #14440

Are you sure you want to change the base?

Signature::Coercible with user defined implicit casting #14440

Conversation

jayzhan211 commented Feb 3, 2025 • edited Loading

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alamb commented Feb 7, 2025

shehabgamin commented Feb 8, 2025

alamb left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jayzhan211 Feb 9, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shehabgamin left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jayzhan211 Feb 9, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jayzhan211 commented Feb 9, 2025

Choose a reason for hiding this comment

shehabgamin commented Feb 11, 2025

jayzhan211 commented Feb 11, 2025 • edited Loading

Choose a reason for hiding this comment

shehabgamin commented Feb 12, 2025

jayzhan211 commented Feb 12, 2025 • edited Loading

jayzhan211 commented Feb 3, 2025 •

edited

Loading

jayzhan211 Feb 9, 2025 •

edited

Loading

jayzhan211 Feb 9, 2025 •

edited

Loading

jayzhan211 commented Feb 11, 2025 •

edited

Loading

jayzhan211 commented Feb 12, 2025 •

edited

Loading