This PR adds basic support for lambda functions to the parser. They are not supported anywhere else yet and not bound yet, but the plan is to use them later on in functions that can apply to lists.

Lambda expressions look like this:

class LambdaExpression : public ParsedExpression {
public:
	string capture_name;
	unique_ptr<ParsedExpression> expression;
};

Example syntax:

SELECT map(i, x -> x + 1) FROM (VALUES (list_value(1, 2, 3))) tbl(i);

Ha I think I may have been responsible for those in the first place. All that hard work 😭. Great this is cleaned up!

Previously a CASE statement with multiple WHEN ... THEN ... nodes would be transformed into a chain of case statements in the transformer. In this PR we modify the case so that this step is only performed during binding. The case statement now looks like this after the transformer phase:

struct CaseCheck {
	unique_ptr<ParsedExpression> when_expr;
	unique_ptr<ParsedExpression> then_expr;
};

//! The CaseExpression represents a CASE expression in the query
class CaseExpression : public ParsedExpression {
	vector<CaseCheck> case_checks;
	unique_ptr<ParsedExpression> else_expr;
};

NULLIF(a, b) used to be transformed into CASE WHEN a=b THEN NULL ELSE a in the transformer phase. In this PR I have changed this to instead be transformed to a regular function call nullif(a, b) and have created a macro that does this transformation during binding instead.

Nice. Thank you, Mark.

All the changes are implemented now, lambda functions now look like this:

class LambdaExpression : public ParsedExpression {
	vector<string> parameters;
	unique_ptr<ParsedExpression> expression;
};

I also fixed several operator precedence rules so that lambda arrows take priority over other operators, which causes e.g. x -> x + 1 AND y + 1 to be correctly parsed as x + 1 AND y + 1 without requiring brackets.

select map(i, (x, y) -> x + y) from tbl;
-- lambda: parameters { x, y }, function: x + y
select map(i, x -> x + 1) from (values (list_value(1, 2, 3))) tbl(i);
-- lambda: parameters { x }, function: x + 1
select map(i, x -> x + 1 AND y + 1) from (values (list_value(1, 2, 3))) tbl(i);
-- lambda: parameters { x }, function: x -> x + 1 AND y + 1

Excellent. We missed that it is possible with COPY but have found that all the info we needed is here. We can probably close this issue.

Performance went down ~10% so need to investigate what's going on there.

This PR refactors and extends the Parquet reader. A major feature addition is the support for nested types in Parquet files, which are mapped to DuckDB's STRUCT and LIST types. Under the hood the Parquet reader now does zero-copy of strings, which should increase performance.

That makes a lot of sense; will do. Thanks for the feedback!

Mark, thanks for explaining. To clarify the terminology, lambda in filter(array_column, x -> x > int_column) has a single capture, e.g. int_column, and a single argument, e.g. x. In map_filter(m, k > 10 and v < 0) we have a lambda with no captures and 2 arguments: k and v. Hence, I think we should rename capture_name above into something like argument_names and make it a vector, not a single value.

+optional type specification for lambda

As for captures, the parser will not do anything besides transforming the expression. It is up to the binder to actually resolve columns. i.e. filter(array_column, x -> x > int_column) will pass the parser just fine and generate a lambda expression containing the following:

capture_name: x
expression: `Comparison(Column(x), Column(int_column), GREATER_THAN)`

The binder is then in charge of resolving "x" back to the lambda, and "int_column" to another data source (e.g. a table).

It supports only single argument captures right now, e.g. filter(array_column, x -> x > column) works, but map_filter(m, (k, v) -> k > 10 and v < 0) does not. I can have a look at extending the lambdas to support multiple captures.

Mark, this is great. To confirm, does this PR include support for multiple arguments for lambda, e.g. map_filter(m, (k, v) -> k > 10 AND v < 0) and does it include support for captures, e.g. filter(array_column, x -> x > int_column)?

CC @mbasmanova

This PR adds basic support for lambda functions to the parser. They are not supported anywhere else yet and not bound yet, but the plan is to use them later on in functions that can apply to lists.

Lambda expressions look like this:

class LambdaExpression : public ParsedExpression {
public:
	string capture_name;
	unique_ptr<ParsedExpression> expression;
};

Example syntax:

SELECT map(i, x -> x + 1) FROM (VALUES (list_value(1, 2, 3))) tbl(i);

Thank you, Mark.

Is this necessary for correctness purposes; or just an optimization? Not that I disagree with adding this, just for clarification. I would like to move it to a function, use an expression_map_t instead of a vector, and also use it for the bound_aggr.filter.

I would like some more test cases:

• Query with many different filter clauses (e.g. 5 aggregates, 5 different filters)
• Filter with some more complex aggregates: COVAR_POP (multiple input columns), STRING_AGG (strings) and ARRAY_AGG (lists)
• DISTINCT aggregates

Also; looking at these tests I would not be surprised if all of them use the perfect hash aggregate. You can force the regular hash aggregate to be used by using very spaced out groups (e.g. [0, 10000000, 20000000, ....]).

The change here to an empty if followed by an else doesn't make much sense. Perhaps a left-over from earlier code?

This seems like the exact same code as the regular AggregateHashtable. I would unify it with that code by using a static function (AggregateHashtable::UpdateAggregate(...)).

This seems overly complicated; why not just add the filter one after the regular payload of an aggregate?

"FILTER " + ...

Same here; can't this just refer to the filter as computed in the projection above it?

"FILTER " + ...

Maybe add "FILTER " before this to the output, to make it clear that this is a filter op (similar to how it appears in a SQL statement).

This seems very complicated; can't this just refer to the filter as computed in the projection above it?