All of the row result methods here: https://duckdb.org/docs/api/python

• fetchdf
• fetchall
• fetchnumpy will put store the query results into memory. If the DuckDB database is sufficiently large, we will run out of memory.

Can you provide an iterator-based method that won't create a list like fetchall? I would recommend just having fetchall return an iterator. That way, you can still call list(fetchall) if you want a list of results.

I would recommend two different APIs for appending.

1. Via dataframes: Append a dataframe to an existing table. This is already possible via the APIs you built, but it could. be slightly simpler for users. Eg:

table.append_df(df)

That would basically create a virtual reference to the dataframe, insert the values in, and then delete the virtual reference.

2. Via the appender API you recommended

While you are correct that this will add latency b/c of Python object allocation overhead, this might still pale in comparison to the overhead of writing to disk?

This will be especially useful when writing entries to the DuckDB that are too big to fit into memory. This appender API would let us basically add things in batch (into memory) and then flush it to disk.

It seems that there is a bug: after all data has been Fetched, the parquet reader returns nullptr instead of an empty DataChunk

Could you please post a full example of what goes wrong in a new issue? Not sure I understand.

Ah, actually, disregard that, it seems that the behaviour of duckdb has changed in query-result: Used to be:

	//! Fetches a DataChunk from the query result. Returns an empty chunk if the result is empty, or nullptr on failure.
	virtual unique_ptr<DataChunk> Fetch() = 0;

Now:

	//! Fetches a DataChunk of normalized (flat) vectors from the query result.
	//! Returns nullptr if there are no more results to fetch.
	DUCKDB_API virtual unique_ptr<DataChunk> Fetch();

So, it used to return an empty chunk, now it returns a nullptr.

https://github.com/cwida/duckdb/blob/436f6455f6e48b571bf5ba0812332f08d0bd65f4/CMakeLists.txt#L138

➜  ./build/release/duckdb -version
0.2.4-dev0 436f6455f
➜  duckdb git:(436f6455f) git describe --tags --abbrev=0git describe --tags --abbrev=0
error: option `abbrev' expects a numerical value
➜  duckdb git:(436f6455f) git describe --tags --abbrev=0
v0.2.3
➜  duckdb git:(436f6455f) git describe --tags --long
v0.2.3-0-g436f6455f

It seems that there is a bug: after all data has been Fetched, the parquet reader returns nullptr instead of an empty DataChunk

Oh I mean something not so specific, just some basic tips as Appender or transactions but for the read side.. ..but since you insist.. :) .. I wrote a simple read test using the old SQL queries. The improvement compared to SQLite is about 20X. Very impressive. The heavy part of my DB is a sort of timed series (key, blob, timestamp). I got millions of these tuple in a recording session. When replaying the data, I need to retrieve the last blob for a given (key, time) or retrieve all tuples in a given time range. The thing is not exactly this way but is just to explain a little more.

So, back to my test, I executed a read banchmark:

• a loop calling Execute -> about 21sec
• a loop calling Query -> about 6sec
• SQLite -> 120sec

The query for the Prepare is "SELECT type, value, MAX(time) FROM attribute WHERE object_id = $1 AND time <= $2 GROUP BY type, value;".

The Query call instead is that auto result = con->Query("SELECT type, value, MAX(time) FROM attribute WHERE object_id = " + std::to_string(object_id) + " AND time <= " + std::to_string(time) + " GROUP BY type, value;");

I don't understand why the Prepare + Execute is slower than the simple Query call.

As discovered in PR #1075, the following query runs into an infinite loop on the LDBC SNB SF0.1 data set.

WITH RECURSIVE search_graph(link, depth) AS (
		SELECT 17592186044856, 0 -- the big number is the start person Id
		UNION ALL
		(SELECT distinct k_person2id, x.depth+1
		FROM knows, search_graph x
		WHERE x.link = k_person1id)
)
select * from search_graph

The same code terminates under Postgres (v12.4):

      link      | depth 
----------------+-------
 17592186044856 |     0
(1 row)

PR #1239 fixed this, the query in https://github.com/cwida/duckdb/issues/1088#issuecomment-723650858 now works:

┌────────────┐
│ max(depth)  │
├────────────┤
│ 2           │
└────────────┘

Thanks @lnkuiper!

That is hard to answer without more details about what you are doing/trying to do/what API you are using. Perhaps you could expand a bit on your problem, the speed you are encountering and the speed you expect?

The "R package Windows" workflow consistently fails, also on the main branch. I'll take a look.

Thank you Mark! Yeah I know transactions will remove some overhead (same for SQLite, huge speed improvement). Have you some tips for me to speed up the reading? Transaction is a nop in this case? Thanks

This is fully expected; the appender is optimized for exactly this use case. Prepare avoids e.g. re-running the optimization and planning phase but still needs to do more than just the Appender. For example, it sets up the query context and creates a result object for every query that is run. There is likely room for improving that somewhat, but using the appender is preferable since the code there is much simpler and faster.

Another thing you should do is wrap multiple statements in a BEGIN TRANSACTION and COMMIT block. This also applies to the appender. If you do not do this but run in auto commit mode, there will be the additional overhead of starting and committing transactions, which includes syncing data to disk if you are not running in in-memory mode. This is extremely slow. i.e. your code should look like this:

con.BeginTransaction();
for(...) {
   con.Execute(1, 2, 3, 4);
}
con.Commit();

Or like this with the appender:

con.BeginTransaction();
for(...) {
	appender.AppendRow(1, 2, 3, 4);
}
con.Commit();

Sorry, I wrote connection.Query but I was talking about the Appender.

Hi, I'm trying to speed up a data recorder/replayer based on SQLite.

I coded an application to transfer data from old DB to DuckDB. As first attempt I used the connection.Prepared + Execute methods to insert data. The tranfert speed wasn't that high.. Also using the C-api the transfert seems really slow. Then I re-wrote the thing using connection.Query obtaining much better performances.

There is something I completely missed or this beaviour is correct? Thanks

This speeds up build time for the debug build from > 7 to just shy of 2 minutes.

Build with cold cache: https://github.com/krlmlr/duckdb/actions/runs/514705674

Build with warm cache: https://github.com/krlmlr/duckdb/actions/runs/514741712

We should consider one of the following options, to avoid corruption of the GitHub action used here:

• Fork https://github.com/hendrikmuhs/ccache-action in the cwida organization and use our forked action
• Embed the action in this repository, use with the syntax ./.github/actions/ccache-action
• Use a full SHA1 to refer to the current state of the action

We also need to enable ccache for the other builds. The size of the cache for the debug build is ~200 MB, this means there's enough space to store caches for other build variants. (The limit is 5 GB per repository, after that caches are evicted.)

You can also insert into a table from any query, including from a pandas dataframe. For example:

INSERT INTO test_df_table SELECT * FROM test_df_view

Does that resolve the issue, or would you propose a different method of appending?

The problem with porting the Appender as-is is that constructing Python objects is really expensive. Perhaps it would help if you could tell us what you would envision this API to look like. A straightforward wrapper like e.g.:

appender = con.append("table");
appender.append(1, 2);
appender.append(1, 2);

Would still be very slow because of all the Python object allocation and de-allocation overhead.

Likely caused by my IN optimization. @hannesmuehleisen please assign me.

It would be really helpful to be able to use the Appender API: https://duckdb.org/docs/data/appender

in Python. Unfortunately, the best way to do this in Python right now is the Pandas dataframe option: https://duckdb.org/docs/api/python This isn't a great option when you want to bulk append to an existing table.

Followup of #1006.

This commit adds the Postgres-compatible regexp_split_to_array alias to string_split_regex.

cc @lnkuiper

never mind, this worked well:

ALTER TABLE gh ADD COLUMN i BIGINT DEFAULT NEXTVAL('seq');

Thank you very much

CREATE SEQUENCE seq;
CREATE TABLE gh (i BIGINT DEFAULT NEXTVAL('seq'), time TIME, count INTEGER);
INSERT INTO gh SELECT * FROM read_csv_auto('github.csv');

Error: Binder Error: table gh has 3 columns but 2 values were supplied

-- Is there a way to handle this in cases where we don't know the column names for the given csv?

INSERT INTO gh(time,count) SELECT * FROM read_csv_auto('github.csv');
SELECT * FROM gh LIMIT 5;
┌───┬──────────┬───────┐
│ i │   time   │ count │
├───┼──────────┼───────┤
│ 1 │ 01:00:00 │ 2     │
│ 2 │ 04:00:00 │ 3     │
│ 3 │ 05:00:00 │ 1     │
│ 4 │ 08:00:00 │ 1     │
│ 5 │ 09:00:00 │ 3     │
└───┴──────────┴───────┘

Somewhat of a follow-up to this question: https://github.com/cwida/duckdb/issues/1330

if you have one process continually writing to a DB, can you copy that file and be sure that it won’t be in a corrupt or somewhat indeterminate state?

(obviously, it would be hard to predict which writes made it in or not. I’m copying the dB file while it’s being written might lead it to be in a corrupt state. Basically not sure if writes are atomic, etc)

I have a use-case where I have one process continually writing to a DB and multiple readers who are reading from the DB.

Does DuckDB support for this use-case? Docs say multireader is ok, but not sure if that also allows for one writer.

This PR moves the SelectStatement::cte_map to QueryNode::cte_map, so that queries like the following work:

SELECT 1 UNION ALL (WITH cte AS (SELECT 42) SELECT * FROM cte);

Before, an error would be thrown because the binder was not able to find cte.

This was found due to #1088 , which should now also be fixed.

Happy to receive any feedback.

Task #1023

Since the unsigned types are built-in I think the most relevant changes are in the cast operators. Let me know if I forgot any interface or if I should add any other relevant tests.