Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[docs](sql-statement) Update create sql block rule #2032

Merged
merged 44 commits into from
Feb 13, 2025
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
44 commits
Select commit Hold shift + click to select a range
2571869
fix-deadlink-1220
KassieZ Dec 20, 2024
232d764
Merge branch 'apache:master' into master
KassieZ Dec 20, 2024
83837f7
fix-deadlink-1220
KassieZ Dec 20, 2024
f11fe14
Merge branch 'apache:master' into master
KassieZ Dec 20, 2024
9100e6e
Merge branch 'master' of https://github.com/KassieZ/doris-website
KassieZ Dec 20, 2024
e922768
Merge branch 'apache:master' into master
KassieZ Dec 24, 2024
68ab9d4
Merge branch 'apache:master' into master
KassieZ Dec 25, 2024
7e9d4ed
Merge branch 'apache:master' into master
KassieZ Dec 25, 2024
f43dbc4
Merge branch 'apache:master' into master
KassieZ Dec 30, 2024
d33973f
Merge branch 'apache:master' into master
KassieZ Jan 2, 2025
0d76b02
Merge branch 'apache:master' into master
KassieZ Jan 2, 2025
98f62ad
Merge branch 'apache:master' into master
KassieZ Jan 2, 2025
7e0cc95
Merge branch 'apache:master' into master
KassieZ Jan 3, 2025
3c00633
Merge branch 'apache:master' into master
KassieZ Jan 3, 2025
b205323
Merge branch 'apache:master' into master
KassieZ Jan 3, 2025
0e40530
Merge branch 'apache:master' into master
KassieZ Jan 6, 2025
1b0f17a
Merge branch 'apache:master' into master
KassieZ Jan 6, 2025
4f01201
Merge branch 'apache:master' into master
KassieZ Jan 6, 2025
b39fab9
Merge branch 'apache:master' into master
KassieZ Jan 9, 2025
18072b4
Merge branch 'apache:master' into master
KassieZ Jan 14, 2025
16f056b
Merge branch 'apache:master' into master
KassieZ Jan 15, 2025
c9c3fbf
Merge branch 'apache:master' into master
KassieZ Jan 16, 2025
9da32b2
Merge branch 'apache:master' into master
KassieZ Jan 17, 2025
001e451
delete table value function invalid docs
KassieZ Jan 17, 2025
130e917
Merge branch 'apache:master' into master
KassieZ Jan 17, 2025
bddd144
Merge branch 'apache:master' into master
KassieZ Jan 17, 2025
db04cd3
Merge branch 'apache:master' into master
KassieZ Jan 17, 2025
adc02f5
Merge branch 'apache:master' into master
KassieZ Jan 19, 2025
22194db
Merge branch 'apache:master' into master
KassieZ Jan 20, 2025
401d0f7
Merge branch 'apache:master' into master
KassieZ Jan 21, 2025
52fa2c0
Merge branch 'apache:master' into master
KassieZ Jan 21, 2025
9435b49
Merge branch 'apache:master' into master
KassieZ Jan 24, 2025
57e3d5b
Merge branch 'apache:master' into master
KassieZ Jan 24, 2025
1958186
Merge branch 'apache:master' into master
KassieZ Feb 6, 2025
2e15158
Merge branch 'apache:master' into master
KassieZ Feb 7, 2025
775b02a
Merge branch 'apache:master' into master
KassieZ Feb 8, 2025
f4a9f1d
Merge branch 'apache:master' into master
KassieZ Feb 10, 2025
19cf140
Merge branch 'apache:master' into master
KassieZ Feb 11, 2025
f088062
Merge branch 'apache:master' into master
KassieZ Feb 11, 2025
3e720e8
Merge branch 'apache:master' into master
KassieZ Feb 12, 2025
82e175c
1
KassieZ Feb 12, 2025
ef1853d
update
KassieZ Feb 12, 2025
f3849c3
fix
KassieZ Feb 12, 2025
d02f52f
Merge branch 'master' into update-create-sql_block_rule
KassieZ Feb 12, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
230 changes: 131 additions & 99 deletions docs/sql-manual/sql-statements/data-governance/CREATE-SQL_BLOCK_RULE.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,127 +27,159 @@ under the License.

## Description

This statement creates a SQL blocking rule. it can restrict any kind of sql statements(no matter DDL or DML statement).
This statement is used to create an SQL block rule.

Supports configuring SQL blacklists by user:
SQL_BLOCK_RULE can be used to restrict users from performing certain operations, such as avoiding scanning large amounts of data.

- Refuse to specify SQL by regular matching
- Check if a sql reaches one of these limits by setting partition_num, tablet_num, cardinality
- partition_num, tablet_num, cardinality can be set together, once a query reaches one of these limits, the query will be intercepted

grammar:
## Syntax

```sql
CREATE SQL_BLOCK_RULE rule_name
[PROPERTIES ("key"="value", ...)];
CREATE SQL_BLOCK_RULE <rule_name>
PROPERTIES (
-- property
<property>
-- Additional properties
[ , ... ]
)
```

Parameter Description:
## Required Parameters

**1. `<rule_name>`**

> The name of the rule.

**2. `<property>`**

> The properties of the rule can be divided into three categories: SQL execution, scan limitation, and switch.

> The SQL execution and scan limitation categories are mutually exclusive, meaning that an SQL block rule can only restrict one of them.

> **SQL Execution Category**

> There are two types, representing regular expression matching and exact matching, respectively. Only one of them can be chosen.
>
> - sql: The matching rule (based on regular expression matching, special characters need to be escaped, for example, `select *` should be written as `select \\*`). When a user executes an SQL statement, the system will use the SQL set here as a regular expression to match the SQL submitted by the user. If it matches, the execution of the SQL will be blocked.
> - sqlHash: The MD5 hash value of the SQL. This is mainly used in conjunction with slow logs. Users do not need to calculate the hash value themselves. For example, if a slow log shows that a particular SQL is running slowly, you can copy the `SqlHash` from `fe.audit.log` and create an SQL_BLOCK_RULE to restrict the execution of this SQL.
>
> **Scan Limitation Category**
> When a user initiates a query, the query optimizer will calculate the number of partitions, tablets, and rows of data that need to be scanned for each table. The following properties can be used to limit these three numbers, either all at once or just some of them.
> - partition_num: The maximum number of partitions that a table will scan.
> - tablet_num: The maximum number of tablets that a table will scan.
> - cardinality: The number of rows of data that a table will scan.
>
> **Switch Category**
>
> - global: Whether the rule is effective for all users. The default is false. If it is not set to true, the rule needs to be applied to a specific user through the `set property` command.
> - enable: Whether the blocking rule is enabled. The default is true.

- sql: matching rule (based on regular matching, special characters need to be translated,for example`select *`use`select \\*`), optional, the default value is "NULL"
- sqlHash: sql hash value, used for exact matching, we will print this value in `fe.audit.log`, optional, this parameter and sql can only be selected one, the default value is "NULL"
- partition_num: the maximum number of partitions a scan node will scan, the default value is 0L
- tablet_num: The maximum number of tablets that a scanning node will scan, the default value is 0L
- cardinality: the rough scan line number of a scan node, the default value is 0L
- global: Whether to take effect globally (all users), the default is false
- enable: whether to enable blocking rules, the default is true

## Access Control Requirements

The user executing this SQL command must have at least the following permissions:

| Privilege | Object | Notes |
| ------------ | ------ | ----- |
| ADMIN_PRIV | Global | |

## Example

1. Create a block rule named test_rule
1. Create a rule that prevents all users from executing `select * from order_analysis`

```sql
CREATE SQL_BLOCK_RULE test_rule
PROPERTIES(
"sql"="select \\* from order_analysis",
"global"="false",
"enable"="true"
);
```
```sql
CREATE SQL_BLOCK_RULE test_rule
PROPERTIES(
"sql"="select \\* from order_analysis",
"global"="true",
"enable"="true"
);
```

When we execute the SQL defined in the rule, it will return an exception error, as shown below:

>Notes:
>
>That the sql statement here does not end with a semicolon

When we execute the sql we just defined in the rule, an exception error will be returned. The example is as follows:

```sql
select * from order_analysis;
ERROR 1064 (HY000): errCode = 2, detailMessage = sql match regex sql block rule: order_analysis_rule
```

```sql
mysql> select * from order_analysis;
ERROR 1064 (HY000): errCode = 2, detailMessage = sql match regex sql block rule: order_analysis_rule
```

2. Create a rule that prevents scanning more than 30 partitions of the same table and restricts the query data volume to no more than 10 billion rows

2. Create test_rule2, limit the maximum number of scanned partitions to 30, and limit the maximum scan base to 10 billion rows. The example is as follows:

```sql
CREATE SQL_BLOCK_RULE test_rule2
PROPERTIES (
"partition_num" = "30",
"cardinality" = "10000000000",
"global" = "false",
"enable" = "true"
);
CREATE SQL_BLOCK_RULE test_rule2
PROPERTIES
(
"partition_num" = "30",
"cardinality" = "10000000000",
"global" = "true",
"enable" = "true"
);
```
3. Create SQL BLOCK RULE with special chars

```sql
CREATE SQL_BLOCK_RULE test_rule3
PROPERTIES
(
"sql" = "select count\\(1\\) from db1.tbl1"
);
CREATE SQL_BLOCK_RULE test_rule4
PROPERTIES
(
"sql" = "select \\* from db1.tbl1"
);
```
4. In SQL_BLOCK_RULE, SQL matching is based on regular expressions. If want to match more patterns of SQL, need to write the corresponding regex. For example, to ignore spaces in SQL and not query tables that start with 'order_', as shown below:

```sql
CREATE SQL_BLOCK_RULE test_rule4
PROPERTIES(
"sql"="\\s*select\\s*\\*\\s*from order_\\w*\\s*",

3. SQL matching is based on regular expressions. If you want to match more patterns of SQL, you need to write the corresponding regular expressions. For example, ignoring spaces in SQL and preventing queries on tables starting with "order", as shown below:


```sql
CREATE SQL_BLOCK_RULE test_rule3
PROPERTIES(
"sql"="\\s*select\\s*\\*\\s*from order_\\w*\\s*",
"global"="true",
"enable"="true"
);
```

4. Create a rule that is only applicable to specific users


```sql
CREATE SQL_BLOCK_RULE test_rule4
PROPERTIES(
"sql"="select \\* from order_analysis",
"global"="false",
"enable"="true"
);
```
);
```

Apply the rule to user jack

### APPENDIX
Here are some commonly used regular expressions:
> . :Matches any single character except for a newline character \n.
>
> * :Matches the preceding element zero or more times. For example, a matches zero or more 'a'.
>
> + :Matches the preceding element one or more times. For example, a+ matches one or more 'a'.
>
> ? :Matches the preceding element zero or one time. For example, a? matches zero or one 'a'.
>
> [] :Used to define a character set. For example, [aeiou] matches any one vowel letter.
>
> [^] :In a character set, use ^ to indicate negation, matching characters that are not in the set. For example, [^0-9] matches any non-digit character.
>
> () :Used for grouping expressions and applying quantifiers. For example, (ab)+ matches consecutive 'ab'.
>
> | :Represents logical OR. For example, a|b matches 'a' or 'b'.
>
> ^ :Matches the beginning of a string. For example, ^abc matches a string that starts with 'abc'.
>
> $ :Matches the end of a string. For example, xyz$ matches a string that ends with 'xyz'.
>
> \ :Used to escape special characters to treat them as ordinary characters. For example, \\. matches the period character '.'.
>
> \s :Matches any whitespace character, including spaces, tabs, newline characters, etc.
>
> \d :Matches any digit character, equivalent to [0-9].
>
> \w :Matches any word character, including letters, digits, and underscores, equivalent to [a-zA-Z0-9_].

## Keywords
```sql
SET PROPERTY FOR 'jack' 'sql_block_rules' = 'test_rule4';
```

## Others

Common regular expressions are as follows:

TextCopy

```text
CREATE, SQL_BLCOK_RULE
```
. : Matches any single character except the newline character `\n`.

* : Matches the preceding element zero or more times. For example, a* matches zero or more 'a's.

+ : Matches the preceding element one or more times. For example, a+ matches one or more 'a's.

? : Matches the preceding element zero or one time. For example, a? matches zero or one 'a'.

[] : Defines a character set. For example, [aeiou] matches any vowel.

[^] : When used in a character set, ^ indicates negation, matching characters not in the set. For example, [^0-9] matches any non-digit character.

() : Groups expressions, allowing quantifiers to be applied to them. For example, (ab)+ matches consecutive 'ab's.

| : Indicates logical OR. For example, a|b matches 'a' or 'b'.

^ : Matches the beginning of a string. For example, ^abc matches strings starting with 'abc'.

$ : Matches the end of a string. For example, xyz$ matches strings ending with 'xyz'.

\ : Escapes special characters, making them ordinary characters. For example, \\. matches the period character '.'.

\s : Matches any whitespace character, including spaces, tabs, newlines, etc.

## Best Practice
\d : Matches any digit character, equivalent to [0-9].

\w : Matches any word character, including letters, digits, and underscores, equivalent to [a-zA-Z0-9_].
```
Loading
Loading