Skip to content

Commit aa9b8fa

Browse files
pyramationpyramation
committed
docs: add PostgreSQL 18 AST upgrade analysis
Comprehensive analysis of AST changes between PG17 and PG18 based on libpg_query 17-latest to 18-latest-dev comparison. Key findings: - 5 new node types (ReturningExpr, ReturningOption, ReturningClause, ATAlterConstraint) - 1 removed node (SinglePartitionSpec) - Major breaking changes: RETURNING clause restructuring, RowCompareType -> CompareType - New features: temporal constraints, virtual generated columns, constraint enforceability - Migration effort: Medium-High (less than PG13->PG17, more than PG16->PG17)
1 parent 132971c commit aa9b8fa

File tree

1 file changed

+311
-0
lines changed

1 file changed

+311
-0
lines changed

‎upgrade-18.md‎

Lines changed: 311 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,311 @@
1+
# PostgreSQL 18 AST Upgrade Analysis
2+
3+
This document analyzes the AST changes between PostgreSQL 17 and PostgreSQL 18 based on the [libpg_query comparison](https://github.com/pganalyze/libpg_query/compare/17-latest...18-latest-dev).
4+
5+
## Summary
6+
7+
The PG17 to PG18 upgrade represents a **moderate breaking change**. While not as severe as the PG13 to PG17 migration, there are significant structural changes that will require updates to the parser, deparser, and transformer packages.
8+
9+
The primary changes center around:
10+
1. Enhanced RETURNING clause support (OLD/NEW table references per SQL:2011)
11+
2. Temporal/period constraint support (WITHOUT OVERLAPS)
12+
3. Constraint enforceability (ENFORCED/NOT ENFORCED)
13+
4. Virtual generated columns
14+
5. Comparison type generalization
15+
16+
## New Node Types (5 added)
17+
18+
### ReturningExpr
19+
New expression node for handling RETURNING clause expressions with OLD/NEW context.
20+
21+
```protobuf
22+
message ReturningExpr {
23+
Node xpr = 1;
24+
int32 retlevelsup = 2;
25+
bool retold = 3;
26+
Node retexpr = 4;
27+
}
28+
```
29+
30+
### ReturningOption
31+
Options for RETURNING clause specifying OLD or NEW table reference.
32+
33+
```protobuf
34+
message ReturningOption {
35+
ReturningOptionKind option = 1;
36+
string value = 2;
37+
int32 location = 3;
38+
}
39+
```
40+
41+
### ReturningClause
42+
New structured RETURNING clause replacing the simple `returning_list`.
43+
44+
```protobuf
45+
message ReturningClause {
46+
repeated Node options = 1;
47+
repeated Node exprs = 2;
48+
}
49+
```
50+
51+
### ATAlterConstraint
52+
New node for ALTER CONSTRAINT operations with enhanced options.
53+
54+
```protobuf
55+
message ATAlterConstraint {
56+
string conname = 1;
57+
bool alter_enforceability = 2;
58+
bool is_enforced = 3;
59+
bool alter_deferrability = 4;
60+
bool deferrable = 5;
61+
bool initdeferred = 6;
62+
bool alter_inheritability = 7;
63+
bool noinherit = 8;
64+
}
65+
```
66+
67+
## Removed Nodes (1 removed)
68+
69+
### SinglePartitionSpec
70+
Removed (was an empty message in PG17 anyway).
71+
72+
## Major Breaking Changes
73+
74+
### 1. RETURNING Clause Restructuring
75+
76+
**Impact: HIGH** - Affects all DML statements
77+
78+
The `returning_list` field (repeated Node) has been replaced with `returning_clause` (ReturningClause) in:
79+
- `InsertStmt`
80+
- `DeleteStmt`
81+
- `UpdateStmt`
82+
- `MergeStmt`
83+
84+
**Before (PG17):**
85+
```protobuf
86+
message InsertStmt {
87+
// ...
88+
repeated Node returning_list = 5;
89+
// ...
90+
}
91+
```
92+
93+
**After (PG18):**
94+
```protobuf
95+
message InsertStmt {
96+
// ...
97+
ReturningClause returning_clause = 5;
98+
// ...
99+
}
100+
```
101+
102+
This enables SQL:2011 syntax like:
103+
```sql
104+
INSERT INTO t VALUES (1) RETURNING OLD AS o, NEW AS n, *;
105+
DELETE FROM t RETURNING OLD.*, NEW.*;
106+
```
107+
108+
### 2. RowCompareType Replaced by CompareType
109+
110+
**Impact: HIGH** - Breaking enum change
111+
112+
The `RowCompareType` enum has been removed and replaced with a more general `CompareType` enum.
113+
114+
**Removed:**
115+
```protobuf
116+
enum RowCompareType {
117+
ROWCOMPARE_LT = 1;
118+
ROWCOMPARE_LE = 2;
119+
ROWCOMPARE_EQ = 3;
120+
ROWCOMPARE_GE = 4;
121+
ROWCOMPARE_GT = 5;
122+
ROWCOMPARE_NE = 6;
123+
}
124+
```
125+
126+
**Added:**
127+
```protobuf
128+
enum CompareType {
129+
COMPARE_INVALID = 1;
130+
COMPARE_LT = 2;
131+
COMPARE_LE = 3;
132+
COMPARE_EQ = 4;
133+
COMPARE_GE = 5;
134+
COMPARE_GT = 6;
135+
COMPARE_NE = 7;
136+
COMPARE_OVERLAP = 8; // New for temporal
137+
COMPARE_CONTAINED_BY = 9; // New for temporal
138+
}
139+
```
140+
141+
**RowCompareExpr** field changed:
142+
- `rctype` (RowCompareType) -> `cmptype` (CompareType)
143+
144+
### 3. Query Node Field Additions and Renumbering
145+
146+
**Impact: MEDIUM** - Field position shifts
147+
148+
New fields added to `Query`:
149+
- `has_group_rte` (field 15) - New boolean for GROUP RTE tracking
150+
- `returning_old_alias` (field 27) - Alias for OLD table in RETURNING
151+
- `returning_new_alias` (field 28) - Alias for NEW table in RETURNING
152+
153+
All subsequent fields are renumbered (16-45 instead of 15-42).
154+
155+
### 4. Constraint Node Enhancements
156+
157+
**Impact: MEDIUM** - New fields for temporal and enforceability
158+
159+
New fields added to `Constraint`:
160+
- `is_enforced` (field 5) - For NOT ENFORCED constraints
161+
- `generated_kind` (field 12) - For virtual generated columns
162+
- `without_overlaps` (field 15) - For temporal PRIMARY KEY/UNIQUE
163+
- `fk_with_period` (field 27) - For temporal foreign keys
164+
- `pk_with_period` (field 28) - For temporal foreign keys
165+
166+
**Removed:** `inhcount` field
167+
168+
## Moderate Changes
169+
170+
### Var Node
171+
Added `varreturningtype` (VarReturningType enum) at field 9 for RETURNING OLD/NEW context.
172+
173+
### CreateStmt
174+
Added `nnconstraints` (field 8) for separate NOT NULL constraint handling.
175+
176+
### IndexStmt
177+
Added `iswithoutoverlaps` (field 19) for temporal index support.
178+
179+
### SortGroupClause
180+
Added `reverse_sort` (field 4) for explicit sort direction tracking.
181+
182+
### RangeTblEntry
183+
Added `groupexprs` (field 30) for GROUP RTE support.
184+
185+
### VariableSetStmt
186+
Added `jumble_args` (field 4) and `location` (field 6).
187+
188+
### FunctionParameter
189+
Added `location` (field 5) for better error reporting.
190+
191+
### ArrayExpr and A_ArrayExpr
192+
Added `list_start` and `list_end` fields for precise source location tracking.
193+
194+
### A_Expr
195+
Added `rexpr_list_start` and `rexpr_list_end` fields for IN-list location tracking.
196+
197+
### IntoClause
198+
`view_query` changed from generic `Node` to specific `Query` type.
199+
200+
## New Enums
201+
202+
### ReturningOptionKind
203+
```protobuf
204+
enum ReturningOptionKind {
205+
RETURNING_OPTION_OLD = 1;
206+
RETURNING_OPTION_NEW = 2;
207+
}
208+
```
209+
210+
### VarReturningType
211+
```protobuf
212+
enum VarReturningType {
213+
VAR_RETURNING_DEFAULT = 1;
214+
VAR_RETURNING_OLD = 2;
215+
VAR_RETURNING_NEW = 3;
216+
}
217+
```
218+
219+
### CompareType
220+
See above - replaces RowCompareType with additional temporal comparison operators.
221+
222+
## Modified Enums
223+
224+
### AlterTableType
225+
- **Removed:** `AT_CheckNotNull`
226+
- All subsequent values renumbered (shifted down by 1)
227+
228+
### ConstrType
229+
- **Added:** `CONSTR_ATTR_ENFORCED` (15), `CONSTR_ATTR_NOT_ENFORCED` (16)
230+
231+
### RTEKind
232+
- **Added:** `RTE_GROUP` (10) for GROUP BY optimization
233+
234+
### JoinType
235+
- **Added:** `JOIN_RIGHT_SEMI` (7)
236+
- Subsequent values renumbered
237+
238+
## New SQL Keywords/Tokens
239+
240+
- `ENFORCED` - For constraint enforceability
241+
- `OBJECTS_P` - New keyword
242+
- `PERIOD` - For temporal constraints
243+
- `VIRTUAL` - For virtual generated columns
244+
245+
**Removed:** `RECHECK`
246+
247+
## Migration Effort Estimate
248+
249+
### Deparser Updates Required
250+
251+
1. **ReturningClause handling** - Must deparse new structure instead of simple list
252+
2. **CompareType enum** - Update RowCompareExpr deparsing
253+
3. **New constraint syntax** - WITHOUT OVERLAPS, ENFORCED/NOT ENFORCED
254+
4. **Virtual generated columns** - New GENERATED ALWAYS AS ... VIRTUAL syntax
255+
256+
### Transformer Updates Required
257+
258+
1. **V17ToV18Transformer** - New transformer needed
259+
2. **RETURNING clause transformation** - Convert between list and clause structures
260+
3. **CompareType mapping** - Map old RowCompareType values to new CompareType
261+
4. **Constraint field mapping** - Handle new fields with defaults
262+
263+
### Types Package Updates
264+
265+
1. Regenerate from new protobuf definitions
266+
2. Update all affected interfaces
267+
3. Add new node types and enums
268+
269+
## Comparison with Previous Upgrades
270+
271+
| Upgrade | New Nodes | Removed Nodes | Breaking Changes | Effort |
272+
|---------|-----------|---------------|------------------|--------|
273+
| PG13 -> PG14 | ~5 | 0 | funcformat, A_Const | High |
274+
| PG14 -> PG15 | ~3 | 0 | Boolean primitive | Medium |
275+
| PG15 -> PG16 | ~2 | 0 | JSON functions | Low |
276+
| PG16 -> PG17 | ~3 | 0 | JSON types | Low |
277+
| **PG17 -> PG18** | **5** | **1** | **RETURNING, CompareType** | **Medium-High** |
278+
279+
## Recommended Approach
280+
281+
1. **Phase 1: Types Generation**
282+
- Update proto file to 18-latest
283+
- Regenerate @pgsql/types, @pgsql/enums, @pgsql/utils
284+
- Update runtime schema
285+
286+
2. **Phase 2: Deparser Updates**
287+
- Add ReturningClause visitor
288+
- Update RowCompareExpr to use CompareType
289+
- Add new constraint syntax support
290+
- Add temporal syntax support
291+
292+
3. **Phase 3: Transformer**
293+
- Create V17ToV18Transformer
294+
- Handle returning_list -> returning_clause conversion
295+
- Handle RowCompareType -> CompareType mapping
296+
- Add default values for new fields
297+
298+
4. **Phase 4: Testing**
299+
- Add fixtures for new PG18 syntax
300+
- Verify round-trip parsing/deparsing
301+
- Test transformation from PG17 ASTs
302+
303+
## Conclusion
304+
305+
The PG17 to PG18 upgrade is more significant than the PG15-16 or PG16-17 upgrades but less disruptive than PG13-14. The main challenges are:
306+
307+
1. The RETURNING clause restructuring affects all DML statements
308+
2. The RowCompareType -> CompareType change requires careful enum mapping
309+
3. New temporal/period features add complexity to constraint handling
310+
311+
However, most existing SQL will continue to work without changes. The new features (temporal constraints, enhanced RETURNING, virtual columns) are additive and won't break existing AST structures for queries that don't use them.

0 commit comments

Comments
 (0)