Ha I think I may have been responsible for those in the first place. All that hard work 😭. Great this is cleaned up!

Nice. Thank you, Mark.

Excellent. We missed that it is possible with COPY but have found that all the info we needed is here. We can probably close this issue.

Performance went down ~10% so need to investigate what's going on there.

This PR refactors and extends the Parquet reader. A major feature addition is the support for nested types in Parquet files, which are mapped to DuckDB's STRUCT and LIST types. Under the hood the Parquet reader now does zero-copy of strings, which should increase performance.

Mark, thanks for explaining. To clarify the terminology, lambda in filter(array_column, x -> x > int_column) has a single capture, e.g. int_column, and a single argument, e.g. x. In map_filter(m, k > 10 and v < 0) we have a lambda with no captures and 2 arguments: k and v. Hence, I think we should rename capture_name above into something like argument_names and make it a vector, not a single value.

+optional type specification for lambda

Mark, this is great. To confirm, does this PR include support for multiple arguments for lambda, e.g. map_filter(m, (k, v) -> k > 10 AND v < 0) and does it include support for captures, e.g. filter(array_column, x -> x > int_column)?

CC @mbasmanova

Thank you, Mark.

That is extremely interesting, but leads to some questions.

1. I am assuming the above would be single-threaded, is that correct? Are there portions of the workload run in parallel?
2. Could the "SELECT * FROM read_csv_auto('test.csv'" include extended sql syntax (join, filter, aggregate, expressions, order)?

DuckDB is powerful as it can also write Parquet as well as read it among other formats. This naturally brings up a question that can DuckDB be used to stream data from source to destination while doing "local" transformations that do not require full table presence (like perhaps sorting without an index).

In our tests we have noticed (@jupiter) that DuckDB uses quite a lot memory (compared to e.g. nodejs streaming solution) when reading e.g. from CSV file (read_csv_auto) and exporting it to Parquet without any transformations or sorting. Does this mean that source data size contributes/scales with DuckDB memory consumption? What if the CSV file would be "infinite", never ending file?

Yes, I think RE2 is POSIX and differs from PCRE in this manner. Some negative lookahead regexps can be converted to posivite and adding NOT on SQL level, so this helps us. But I think it would nicer to have PCRE supported.

@lnkuiper if you look at this, consider cleaning up https://github.com/cwida/duckdb/tree/master/test/ldbc which has an old version of the LDBC queries.

Hi, not sure if this is still relevant. I managed to properly shutdown a duckdb on Windows, so that I did not get error when reopening it (as in issue #323):

` library(duckdb)

FAILS

con <- dbConnect(duckdb(), dbdir="test.duckdb") dbWriteTable(con, "iris", iris, overwrite = TRUE) dbDisconnect(con) con <- dbConnect(duckdb(), dbdir="test.duckdb")

Fehler in initialize(value, ...) :

duckdb_startup_R: Failed to open database

REMEDIES FAILURE, THEN SUCCEDS

dbDisconnect(con, shutdown=TRUE)

Warnmeldung:

Connection already closed.

con <- dbConnect(duckdb(), dbdir="test.duckdb") dbWriteTable(con, "iris", iris, overwrite = TRUE) dbDisconnect(con, shutdown=TRUE)

SUCCEDS :)

con <- dbConnect(duckdb(), dbdir="test.duckdb") dbWriteTable(con, "iris", iris, overwrite = TRUE) dbDisconnect(con, shutdown=TRUE)

con <- dbConnect(duckdb(), dbdir="test.duckdb") ## no error dbWriteTable(con, "iris", iris, overwrite = TRUE) dbDisconnect(con, shutdown=TRUE) `

We have this kind of regular expression, which works e.g. with NodeJS string.match(), but not with DuckDB.

regexp_matches(header_content_type, ' *(?![tT][eE][xX][tT]/[Hh][Tt][Mm][Ll]).*') 

invalid perl operator: (?!

Is this a bug or a feature? :)

While Auto-Incrementing ideas are more useful, common, and idiomatic is an OLTP store, they can be very useful for tracking changesets (especially for caching) in OLAP analytical tasks. Towards that end, it would be great to have the ability to specify an AUTO INCREMENT policy on a column (or something more advanced like PostgreSQLs Serial flag). While It's easy enough to do this manually with a prior COUNT(*) query, a write-lock, and bulk insert statements, the only way to add such a column when using a scanner/reader like read_csv is to add a new column and manually UPDATE into that column (thereby ~defeating the purpose of those fast import mechanisms). Thoughts?

In general, we should provide Android packages. It appears like different SDKs should be used to create those builds, and then there is some CPU-differences in resulting builds. If possible, integrate those binaries into the normal JDBC driver, but I have my doubts.

I' m a C++ beginner. I try to run C++ example in CLion and debug it (Win10 64).

I get error like "-lduckdb failed" so I download the duckdb library from websites and put it in my MinGW directory.

Now CLion can find the library but not recognized.

"D:\JetBrains\CLion 2020.3.1\bin\cmake\win\bin\cmake.exe" --build D:\code\duckdb\examples\embedded-c++\cmake-build-debug-mingw --target all -- -j 6
[ 50%] Linking CXX executable example.exe
D:/mingw/mingw32/bin/../lib/gcc/i686-w64-mingw32/8.1.0/../../../../lib/duckdb.dll: file not recognized: File format not recognized
collect2.exe: error: ld returned 1 exit status
mingw32-make.exe[2]: *** [CMakeFiles\example.dir\build.make:106: example.exe] Error 1
mingw32-make.exe[1]: *** [CMakeFiles\Makefile2:95: CMakeFiles/example.dir/all] Error 2
mingw32-make.exe: *** [Makefile:103: all] Error 2

Are there any suggestions to solve this problem?

Does DuckDB provide regression analysis functions?

This PR gives a (potentially over-complicated binding process, please check that) implementation of the filter clause for aggregates #896

good catch

Certainly:

echo -e '42\n43\n44' > /tmp/dummy

COPY a(b) FROM '/tmp/dummy';
SELECT * FROM a;

┌───┬────┐
│ i │ b  │
├───┼────┤
│ 1 │ 42 │
│ 2 │ 43 │
│ 3 │ 44 │
└───┴────┘

oh neat! is there any way to use this alongside read_csv/COPY?

This is again the SQLite shell that does this, not DuckDB

Two options, 1) pull those columns into R, and run lm there. 2) Implement a recursive CTE that computes the fit.

How about using a sequence? For example

CREATE SEQUENCE seq;
CREATE TABLE a (i INTEGER DEFAULT NEXTVAL('seq'), b INTEGER);
INSERT INTO a (b) VALUES (42), (43);
SELECT * FROM a;

Result:

┌───┬────┐
│ i │ b  │
├───┼────┤
│ 1 │ 42 │
│ 2 │ 43 │
└───┴────┘

While Auto-Incrementing ideas are more useful, common, and idiomatic is an OLTP store, they can be very useful for tracking changesets (especially for caching) in OLAP analytical tasks. Towards that end, it would be great to have the ability to specify an AUTO INCREMENT policy on a column (or something more advanced like PostgreSQLs Serial flag). While It's easy enough to do this manually with a prior COUNT(*) query, a write-lock, and bulk insert statements, the only way to add such a column when using a scanner/reader like read_csv is to add a new column and manually UPDATE into that column (thereby ~defeating the purpose of those fast import mechanisms). Thoughts?