I am really excited for the changes w.r.t. performance in the merged PRs!

This will allow us to further improve some kernels :+1:

No worries @jorgecarleitao -- I think this was likely to happen given the backup of PRs waiting to go on to master :)

Thanks a lot, @Dandandan and @alamb , and sorry for the mess 😞

@maxburke I predict sometime in the next 24 hours. We are working through the Rust PR backlog (though it is fairly large). I also believe this one is important as tokio flows upwards through the Rust ecosystem

The fix is merged, so hopefully master is back 🟢 ✅

🚀

FYI @jorgecarleitao I think some change on master and maybe some parquet related changes caused a compilation error on master. This fixes the compilation error.

For anyone else following along, the error this PR fixes looks like the following (from https://github.com/apache/arrow/runs/1729540155):

  Compiling arrow-flight v3.0.0-SNAPSHOT (/__w/arrow/arrow/rust/arrow-flight)
   Compiling tonic v0.3.1
   Compiling datafusion v3.0.0-SNAPSHOT (/__w/arrow/arrow/rust/datafusion)
error[E0061]: this function takes 2 arguments but 1 argument was supplied
   --> datafusion/src/physical_plan/parquet.rs:712:25
    |
712 |             data_buffer.resize(data_buffer.len() + data_size);
    |                         ^^^^^^ ----------------------------- supplied 1 argument
    |                         |
    |                         expected 2 arguments

error: aborting due to previous error

That seems like a reasonable request. Could you please report this feature request on Arrow's JIRA?.

I ran this locally and it fixes the build for me. I plan to merge this in prior to CI finishing to get master back to green

cc @alamb

https://issues.apache.org/jira/browse/ARROW-11321

This one is next in line for merging @jorgecarleitao and I have our eyes on it... Once a few more tests have completed on https://github.com/apache/arrow/commits/master we'll get it in

Yeah those submodules seems to cause some issues lately :/. Should be fixed now @jorgecarleitao

Well, it broke master... 🤣 @Dandandan already has a fix https://github.com/apache/arrow/pull/9269 💯

🎉

This has an unrelated change on a submodule?

likely. It was kind of expected, as it did some backward incompatible changes. I was trying to merge it first to avoid breaking, but I guess I was not fast for the speed on which PRs are merged into master after the green light on the mailing list :P

@jorgecarleitao I think this PR broke master:

   --> datafusion/src/physical_plan/parquet.rs:712:25
    |
712 |             data_buffer.resize(data_buffer.len() + data_size);
    |                         ^^^^^^ ----------------------------- supplied 1 argument
    |                         |
    |                         expected 2 arguments
    |
note: associated function defined here
   --> rust/arrow/src/buffer.rs:833:12
    |
833 |     pub fn resize(&mut self, new_len: usize, value: u8) {

The way it is at the moment you will try the next directory if you get a permissions error or non-existing directory but you won't try the next directory if you tried three times and failed. I think this is probably ok, just making sure this is the behavior you want.

Why << 17? Won't this leave the last 17 bits as 0? It appears PID is at least 32 bits.

Why not simply use a static/anon-namespace functions for TryCreatingDirectory and MakeBaseName? MakeRandomName is already one.

Why are you applying std::move to a return value?

I think the feature to be able to repartition an in memory table is useful, as the repartitioning only needs to be applied once, and repartition itself is cheap (at the same node). Doing this when loading data is very useful for in-memory analytics as we can benefit from mutliple cores after loading the data.

The speed up from repartitioning is very big (mainly on aggregates), on my (8-core machine): ~5-7x on query 1 and 12 versus a single partition, and a smaller (~30%) difference for query 5 when using 16 partition. q1/q12 also have very high cpu utilization.

@jorgecarleitao maybe this is of interest to you, as you mentioned you are looking into multi-threading. I think this would be a "high level" way to get more parallelism, also in the logical plan. I think in some optimizer rules and/or dynamically we can do repartitions, similar to what's described here https://issues.apache.org/jira/browse/ARROW-9464

Benchmarks after repartitioning (16 partitions):

PR (16 partitions)

Query 12 iteration 0 took 33.9 ms
Query 12 iteration 1 took 34.3 ms
Query 12 iteration 2 took 36.9 ms
Query 12 iteration 3 took 33.6 ms
Query 12 iteration 4 took 35.1 ms
Query 12 iteration 5 took 38.8 ms
Query 12 iteration 6 took 35.8 ms
Query 12 iteration 7 took 34.4 ms
Query 12 iteration 8 took 34.2 ms
Query 12 iteration 9 took 35.3 ms
Query 12 avg time: 35.24 ms

Master (1 partition):

Query 12 iteration 0 took 245.6 ms
Query 12 iteration 1 took 246.4 ms
Query 12 iteration 2 took 246.1 ms
Query 12 iteration 3 took 247.9 ms
Query 12 iteration 4 took 246.5 ms
Query 12 iteration 5 took 248.2 ms
Query 12 iteration 6 took 247.8 ms
Query 12 iteration 7 took 246.4 ms
Query 12 iteration 8 took 246.6 ms
Query 12 iteration 9 took 246.5 ms
Query 12 avg time: 246.79 ms

PR (16 partitions):

Query 1 iteration 0 took 138.6 ms
Query 1 iteration 1 took 142.2 ms
Query 1 iteration 2 took 125.8 ms
Query 1 iteration 3 took 102.4 ms
Query 1 iteration 4 took 105.9 ms
Query 1 iteration 5 took 107.0 ms
Query 1 iteration 6 took 109.3 ms
Query 1 iteration 7 took 109.9 ms
Query 1 iteration 8 took 108.8 ms
Query 1 iteration 9 took 112.0 ms
Query 1 avg time: 116.19 ms

Master (1 partition):

Query 1 iteration 0 took 640.6 ms
Query 1 iteration 1 took 640.0 ms
Query 1 iteration 2 took 632.9 ms
Query 1 iteration 3 took 634.6 ms
Query 1 iteration 4 took 630.7 ms
Query 1 iteration 5 took 630.7 ms
Query 1 iteration 6 took 631.9 ms
Query 1 iteration 7 took 635.5 ms
Query 1 iteration 8 took 639.0 ms
Query 1 iteration 9 took 638.3 ms
Query 1 avg time: 635.43 ms

PR (16 partitions)

Query 5 iteration 0 took 465.8 ms
Query 5 iteration 1 took 428.0 ms
Query 5 iteration 2 took 435.0 ms
Query 5 iteration 3 took 407.3 ms
Query 5 iteration 4 took 435.7 ms
Query 5 iteration 5 took 437.4 ms
Query 5 iteration 6 took 411.2 ms
Query 5 iteration 7 took 432.0 ms
Query 5 iteration 8 took 436.8 ms
Query 5 iteration 9 took 435.6 ms
Query 5 avg time: 432.47 ms

Master (1 partition)

Query 5 iteration 0 took 660.6 ms
Query 5 iteration 1 took 634.4 ms
Query 5 iteration 2 took 626.4 ms
Query 5 iteration 3 took 628.0 ms
Query 5 iteration 4 took 635.3 ms
Query 5 iteration 5 took 631.1 ms
Query 5 iteration 6 took 631.3 ms
Query 5 iteration 7 took 639.4 ms
Query 5 iteration 8 took 634.3 ms
Query 5 iteration 9 took 639.0 ms
Query 5 avg time: 635.97 ms

I find myself trying to remember the exact incantation to create a StringDictionaryBuilder so I figured I would add it as a doc example

Also if I want to integrate only plasma storage into my project, what's the suggested way to do so? c_glib or other way?