Skip to content

Commit c72ca72

Browse files
norton120Fokko
andauthored
Documented row_filter expressions (#1862)
# Rationale for this change scan's `row_filter` param is not super intuitive. I got tired of reading over the expression and parser code as I'm trying to build out statements, so I had some docs made up. # Are these changes tested? They are docs only, so not really? # Are there any user-facing changes? Yes there are docs for the expression and string syntaxes of `row_filter` now. <!-- In the case of user-facing changes, please add the changelog label. --> --------- Co-authored-by: Fokko Driesprong <[email protected]>
1 parent ae11ba4 commit c72ca72

File tree

3 files changed

+433
-0
lines changed

3 files changed

+433
-0
lines changed

mkdocs/docs/SUMMARY.md

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -24,6 +24,8 @@
2424
- [Configuration](configuration.md)
2525
- [CLI](cli.md)
2626
- [API](api.md)
27+
- [Row Filter Syntax](row-filter-syntax.md)
28+
- [Expression DSL](expression-dsl.md)
2729
- [Contributing](contributing.md)
2830
- [Community](community.md)
2931
- Releases

mkdocs/docs/expression-dsl.md

Lines changed: 259 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,259 @@
1+
<!--
2+
- Licensed to the Apache Software Foundation (ASF) under one or more
3+
- contributor license agreements. See the NOTICE file distributed with
4+
- this work for additional information regarding copyright ownership.
5+
- The ASF licenses this file to You under the Apache License, Version 2.0
6+
- (the "License"); you may not use this file except in compliance with
7+
- the License. You may obtain a copy of the License at
8+
-
9+
- http://www.apache.org/licenses/LICENSE-2.0
10+
-
11+
- Unless required by applicable law or agreed to in writing, software
12+
- distributed under the License is distributed on an "AS IS" BASIS,
13+
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14+
- See the License for the specific language governing permissions and
15+
- limitations under the License.
16+
-->
17+
18+
# Expression DSL
19+
20+
The PyIceberg library provides a powerful expression DSL (Domain Specific Language) for building complex row filter expressions. This guide will help you understand how to use the expression DSL effectively. This DSL allows you to build type-safe expressions for use in the `row_filter` scan argument.
21+
22+
They are composed of terms, predicates, and logical operators.
23+
24+
## Basic Concepts
25+
26+
### Terms
27+
28+
Terms are the basic building blocks of expressions. They represent references to fields in your data:
29+
30+
```python
31+
from pyiceberg.expressions import Reference
32+
33+
# Create a reference to a field named "age"
34+
age_field = Reference("age")
35+
```
36+
37+
### Predicates
38+
39+
Predicates are expressions that evaluate to a boolean value. They can be combined using logical operators.
40+
41+
#### Literal Predicates
42+
43+
```python
44+
from pyiceberg.expressions import EqualTo, NotEqualTo, LessThan, LessThanOrEqual, GreaterThan, GreaterThanOrEqual
45+
46+
# age equals 18
47+
age_equals_18 = EqualTo("age", 18)
48+
49+
# age is not equal to 18
50+
age_not_equals_18 = NotEqualTo("age", 18)
51+
52+
# age is less than 18
53+
age_less_than_18 = LessThan("age", 18)
54+
55+
# Less than or equal to
56+
age_less_than_or_equal_18 = LessThanOrEqual("age", 18)
57+
58+
# Greater than
59+
age_greater_than_18 = GreaterThan("age", 18)
60+
61+
# Greater than or equal to
62+
age_greater_than_or_equal_18 = GreaterThanOrEqual("age", 18)
63+
```
64+
65+
#### Set Predicates
66+
67+
```python
68+
from pyiceberg.expressions import In, NotIn
69+
70+
# age is one of 18, 19, 20
71+
age_in_set = In("age", [18, 19, 20])
72+
73+
# age is not 18, 19, oer 20
74+
age_not_in_set = NotIn("age", [18, 19, 20])
75+
```
76+
77+
#### Unary Predicates
78+
79+
```python
80+
from pyiceberg.expressions import IsNull, NotNull
81+
82+
# Is null
83+
name_is_null = IsNull("name")
84+
85+
# Is not null
86+
name_is_not_null = NotNull("name")
87+
```
88+
89+
#### String Predicates
90+
91+
```python
92+
from pyiceberg.expressions import StartsWith, NotStartsWith
93+
94+
# TRUE for 'Johnathan', FALSE for 'Johan'
95+
name_starts_with = StartsWith("name", "John")
96+
97+
# FALSE for 'Johnathan', TRUE for 'Johan'
98+
name_not_starts_with = NotStartsWith("name", "John")
99+
```
100+
101+
### Logical Operators
102+
103+
You can combine predicates using logical operators:
104+
105+
```python
106+
from pyiceberg.expressions import And, Or, Not
107+
108+
# TRUE for 25, FALSE for 67 and 15
109+
age_between = And(
110+
GreaterThanOrEqual("age", 18),
111+
LessThanOrEqual("age", 65)
112+
)
113+
114+
# FALSE for 25, TRUE for 67 and 15
115+
age_outside = Or(
116+
LessThan("age", 18),
117+
GreaterThan("age", 65)
118+
)
119+
120+
# NOT operator
121+
not_adult = Not(GreaterThanOrEqual("age", 18))
122+
```
123+
124+
## Advanced Usage
125+
126+
### Complex Expressions
127+
128+
You can build complex expressions by combining multiple predicates and operators:
129+
130+
```python
131+
from pyiceberg.expressions import And, Or, Not, EqualTo, GreaterThan, LessThan, In
132+
133+
# (age >= 18 AND age <= 65) AND (status = 'active' OR status = 'pending')
134+
complex_filter = And(
135+
And(
136+
GreaterThanOrEqual("age", 18),
137+
LessThanOrEqual("age", 65)
138+
),
139+
Or(
140+
EqualTo("status", "active"),
141+
EqualTo("status", "pending")
142+
)
143+
)
144+
145+
# NOT (age < 18 OR age > 65)
146+
age_in_range = Not(
147+
Or(
148+
LessThan("age", 18),
149+
GreaterThan("age", 65)
150+
)
151+
)
152+
```
153+
154+
### Type Safety
155+
156+
The expression DSL provides type safety through Python's type system. When you create expressions, the types are checked at runtime:
157+
158+
```python
159+
from pyiceberg.expressions import EqualTo
160+
161+
# This will work
162+
age_equals_18 = EqualTo("age", 18)
163+
164+
# This will raise a TypeError if the field type doesn't match
165+
age_equals_18 = EqualTo("age", "18") # Will fail if age is an integer field
166+
```
167+
168+
## Best Practices
169+
170+
1. **Use Type Hints**: Always use type hints when working with expressions to catch type-related errors early.
171+
172+
2. **Break Down Complex Expressions**: For complex expressions, break them down into smaller, more manageable parts:
173+
174+
```python
175+
# Instead of this:
176+
complex_filter = And(
177+
And(
178+
GreaterThanOrEqual("age", 18),
179+
LessThanOrEqual("age", 65)
180+
),
181+
Or(
182+
EqualTo("status", "active"),
183+
EqualTo("status", "pending")
184+
)
185+
)
186+
187+
# Do this:
188+
age_range = And(
189+
GreaterThanOrEqual("age", 18),
190+
LessThanOrEqual("age", 65)
191+
)
192+
193+
status_filter = Or(
194+
EqualTo("status", "active"),
195+
EqualTo("status", "pending")
196+
)
197+
198+
complex_filter = And(age_range, status_filter)
199+
```
200+
201+
## Common Pitfalls
202+
203+
1. **Null Handling**: Be careful when using `IsNull` and `NotNull` predicates with required fields. The expression DSL will automatically optimize these cases:
204+
- `IsNull` (and `IsNaN` for doubles/floats) on a required field will always return `False`
205+
- `NotNull` (and `NotNaN` for doubles/floats) on a required field will always return `True`
206+
207+
2. **String Comparisons**: When using string predicates like `StartsWith`, ensure that the field type is actually a string type.
208+
209+
## Examples
210+
211+
Here are some practical examples of using the expression DSL:
212+
213+
### Basic Filtering
214+
215+
```python
216+
217+
from datetime import datetime
218+
from pyiceberg.expressions import (
219+
And,
220+
EqualTo,
221+
GreaterThanOrEqual,
222+
LessThanOrEqual,
223+
GreaterThan,
224+
In
225+
)
226+
227+
active_adult_users_filter = And(
228+
EqualTo("status", "active"),
229+
GreaterThanOrEqual("age", 18)
230+
)
231+
232+
233+
high_value_customers = And(
234+
GreaterThan("total_spent", 1000),
235+
In("membership_level", ["gold", "platinum"])
236+
)
237+
238+
date_range_filter = And(
239+
GreaterThanOrEqual("created_at", datetime(2024, 1, 1)),
240+
LessThanOrEqual("created_at", datetime(2024, 12, 31))
241+
)
242+
```
243+
244+
### Multi-Condition Filter
245+
246+
```python
247+
from pyiceberg.expressions import And, Or, Not, EqualTo, GreaterThan
248+
249+
complex_filter = And(
250+
Not(EqualTo("status", "deleted")),
251+
Or(
252+
And(
253+
EqualTo("type", "premium"),
254+
GreaterThan("subscription_months", 12)
255+
),
256+
EqualTo("type", "enterprise")
257+
)
258+
)
259+
```

0 commit comments

Comments
 (0)