profile
viewpoint
If you are wondering where the data of this site comes from, please visit https://api.github.com/users/jakirkham/events. GitMemory does not store any data, but only uses NGINX to cache data for a period of time. The idea behind GitMemory is simply to give users a better reading experience.

dask/dask-jobqueue 158

Deploy Dask on job schedulers like PBS, SLURM, and SGE

dask/dask-image 134

Distributed image processing

gpuopenanalytics/pynvml 73

Provide Python access to the NVML library for GPU diagnostics

dask/dask-drmaa 41

Deploy Dask on DRMAA clusters

conda-forge/docker-images 28

Repository to host the Docker images files used in conda-forge

dask-image/dask-ndfilters 5

A library for using N-D filters with Dask Arrays

aspp-apac/2019-parallel-python 1

Parallel python class material for the 2019 summer school in Canberra.

conda-forge/conda-forge-anvil 1

conda-forge conda-constructor

dask-image/dask-ndmorph 1

A library of N-D morphological operations for Dask Arrays

jakirkham/ac2018-dl-gpu 1

Notebooks from AnacondaCON 2018 Deep Learning with GPUs tutorial

pull request commentconda-forge/dask-feedstock

Release 2021.7.2

Thanks James! 😀

jrbourbeau

comment created time in 6 days

pull request commentconda-forge/distributed-feedstock

Release 2021.7.2

Oops sorry didn't see this James 😬

jrbourbeau

comment created time in 6 days

pull request commentdask/distributed

Deserialization: zero-copy merge subframes when possible

UCX uses a mix of NumPy arrays and RMM DeviceBuffers depending on if it is in CPU or GPU memory. That said, wouldn't worry about that. We can probably refresh that layer to benefit from some of the same things we are doing for TCP.

gjoseph92

comment created time in 6 days

pull request commentdask/dask

Require NumPy 1.18+ & Pandas 1.0+

Thanks all! 😄 Hoping this simplifies maintenance going forward 🙂

jakirkham

comment created time in 6 days

delete branch jakirkham/dask

delete branch : req_np17_pd1

delete time in 6 days

issue commentdask/dask

`pyarrow=5.0.0` compatiblity

Looks like PR ( https://github.com/dask/dask/pull/7967 ) has a proposed Dask fix

jrbourbeau

comment created time in 6 days

push eventjakirkham/distributed

Matthew Rocklin

commit sha 3f1b25097feb683c228e63b7165ac1a568e01330

Include maximum shard size in second `to_frames` method (#5145) Websockets are weird in that they have two different Comm objects. Previously we added the maximum shard size setting to one, but not the other

view details

Florian Jetter

commit sha 1999c158521db8a1ec356f4389c8e9c1666b6a94

Fix a deadlock connected to task stealing and task deserialization (#5128) * Fix a deadlock connected to task stealing and deserialization If a task is stolen while the task runspec is being deserialized this allows for an edge case where the executing_count is never decreased again such that the ready queue is never worked off * Simplify exception handling for Worker.execute * remove test about unintended behaviour * function naming fix

view details

James Bourbeau

commit sha 7392de636c3dc6e52f08d9d3fbd171ff3177b4d1

bump version to 2021.07.2

view details

push time in 6 days

push eventjakirkham/dask

Julia Signell

commit sha 03747f2df21e0e1947e2758604eb5de4ace731e3

Fix read-csv name - when path is different, use different name for task (#7942)

view details

Freyam Mehta

commit sha 5fbbf90a59c19c197875345fb215d536fc451b8d

Change graphviz font family to sans (#7931) * changed font to roboto and cleaned code * reverted square shape for all nodes in low level graph * retrigger CI * Changed font to Helvetica

view details

Peter Andreas Entschev

commit sha c8718d5714601234f2d40b502a8b83a4ef2e3df4

Adds missing prepend/append functionality to da.diff (#7946) * Add prepend kwarg to da.diff * Add da.diff prepend tests * Add append kwarg to da.diff * Add da.diff append tests * Raise on negative da.diff order * Fix da.diff append raise test when n==0 * Use meta_from_array in da.diff prepend/append * Add CuPy tests for da.diff

view details

Zhengnan Zhao

commit sha 724907f20d01783aa0a3629982d716672a595d1a

Remove skips from doctests (5 of 6) (#7864)

view details

Zhengnan Zhao

commit sha 69502ffecb2fba61db7ad3e38a9833b1d841c69a

Remove skips from doctests (4 of 6) (#7865)

view details

James Bourbeau

commit sha 5eac452d4d85f24814d64a3eaebf2bf531c253e2

Add deprecation warning for top-level `ucx` and `rmm` config values (#7956)

view details

James Bourbeau

commit sha 39b383aa09e41f25921213a0ed929c872ba9a4ae

Temporarily pin `pyarrow<5` in CI (#7960)

view details

Matthew Rocklin

commit sha ae51ab3c1dfd495c5a3e9f7dd12fb7541998ab04

Avoid use of `Delayed` in `to_parquet` (#7958) This removes our ability to perform graph optimization. Instead we use a dask.dataframe.Scalar class here. We should probably do this elsewhere as well.

view details

Freyam Mehta

commit sha 3d65627f163819a932fdbc8bf77f241a2f3b5fb4

Add `dask.array` SVG to the HTML Repr (#7886) New Features - Added chunks to the collection_annotations - Added collection_annotations to MaterializedLayer class - Added the Dask Array SVG of the chunks to the HTML Repr

view details

James Bourbeau

commit sha ad6f837ea11f03a32c7bb51d6e33516d93e85148

bump version to 2021.07.2

view details

push time in 6 days

issue commentconda-forge/ctng-compilers-feedstock

System headers not picked up by `gcc`

Thanks Isuru :)

For example...

$ gcc -xc -E -v --sysroot=/ - 
Reading specs from /datasets/jkirkham/miniconda/envs/rapids2108dev/bin/../lib/gcc/x86_64-conda-linux-gnu/9.3.0/specs
COLLECT_GCC=gcc
Target: x86_64-conda-linux-gnu
Configured with: /home/conda/feedstock_root/build_artifacts/ctng-compilers_1618239181388/work/.build/x86_64-conda-linux-gnu/src/gcc/configure --build=x86_64-build_pc-linux-gnu --host=x86_64-build_pc-linux-gnu --target=x86_64-conda-linux-gnu --prefix=/home/conda/feedstock_root/build_artifacts/ctng-compilers_1618239181388/work/gcc_built --with-sysroot=/home/conda/feedstock_root/build_artifacts/ctng-compilers_1618239181388/work/gcc_built/x86_64-conda-linux-gnu/sysroot --enable-languages=c,c++,fortran,objc,obj-c++ --with-pkgversion='crosstool-NG 1.24.0.133_b0863d8_dirty' --enable-__cxa_atexit --disable-libmudflap --enable-libgomp --disable-libssp --enable-libquadmath --enable-libquadmath-support --enable-libsanitizer --enable-libmpx --with-gmp=/home/conda/feedstock_root/build_artifacts/ctng-compilers_1618239181388/work/.build/x86_64-conda-linux-gnu/buildtools --with-mpfr=/home/conda/feedstock_root/build_artifacts/ctng-compilers_1618239181388/work/.build/x86_64-conda-linux-gnu/buildtools --with-mpc=/home/conda/feedstock_root/build_artifacts/ctng-compilers_1618239181388/work/.build/x86_64-conda-linux-gnu/buildtools --with-isl=/home/conda/feedstock_root/build_artifacts/ctng-compilers_1618239181388/work/.build/x86_64-conda-linux-gnu/buildtools --enable-lto --enable-threads=posix --enable-target-optspace --enable-plugin --enable-gold --disable-nls --disable-multilib --with-local-prefix=/home/conda/feedstock_root/build_artifacts/ctng-compilers_1618239181388/work/gcc_built/x86_64-conda-linux-gnu/sysroot --enable-long-long --enable-default-pie
Thread model: posix
gcc version 9.3.0 (crosstool-NG 1.24.0.133_b0863d8_dirty) 
COLLECT_GCC_OPTIONS='-E' '-v' '-mtune=generic' '-march=x86-64'
 /datasets/jkirkham/miniconda/envs/rapids2108dev/bin/../libexec/gcc/x86_64-conda-linux-gnu/9.3.0/cc1 -E -quiet -v -iprefix /datasets/jkirkham/miniconda/envs/rapids2108dev/bin/../lib/gcc/x86_64-conda-linux-gnu/9.3.0/ -isysroot / - -mtune=generic -march=x86-64
ignoring duplicate directory "/datasets/jkirkham/miniconda/envs/rapids2108dev/bin/../lib/gcc/../../lib/gcc/x86_64-conda-linux-gnu/9.3.0/include"
ignoring nonexistent directory "/home/conda/feedstock_root/build_artifacts/ctng-compilers_1618239181388/work/gcc_built/x86_64-conda-linux-gnu/sysroot/include"
ignoring duplicate directory "/datasets/jkirkham/miniconda/envs/rapids2108dev/bin/../lib/gcc/../../lib/gcc/x86_64-conda-linux-gnu/9.3.0/include-fixed"
ignoring duplicate directory "/datasets/jkirkham/miniconda/envs/rapids2108dev/bin/../lib/gcc/../../lib/gcc/x86_64-conda-linux-gnu/9.3.0/../../../../x86_64-conda-linux-gnu/include"
#include "..." search starts here:
#include <...> search starts here:
 /datasets/jkirkham/miniconda/envs/rapids2108dev/bin/../lib/gcc/x86_64-conda-linux-gnu/9.3.0/include
 /datasets/jkirkham/miniconda/envs/rapids2108dev/bin/../lib/gcc/x86_64-conda-linux-gnu/9.3.0/include-fixed
 /datasets/jkirkham/miniconda/envs/rapids2108dev/bin/../lib/gcc/x86_64-conda-linux-gnu/9.3.0/../../../../x86_64-conda-linux-gnu/include
 /usr/include
End of search list.

Note: /usr/include shows up at the end

charlesbluca

comment created time in 6 days

MemberEvent

Pull request review commentconda-forge/staged-recipes

[WIP] Add `arm-variant` package

+arm_variant_type:  # [aarch64]+  - sbsa           # [aarch64]

We could use features. It's kind of a blunt tool. So it would only prioritize one and there wouldn't be an order with any others

Agree this could make sense, but left it out initially. If we agree generally it should be added, we can do this

jakirkham

comment created time in 6 days

PullRequestReviewEvent

Pull request review commentconda-forge/staged-recipes

[WIP] Add `arm-variant` package

+package:+  name: arm-variant+  version: 1.0.0++build:+  number: 0+  string: "{{ arm_variant_type }}"+  skip: true  # [not aarch64]++test:+  commands:+    - exit 0

To keep the linter happy :)

jakirkham

comment created time in 6 days

PullRequestReviewEvent

issue commentconda-forge/cudatoolkit-feedstock

Initial CUDA 11.3 Conda Packages

cc @seibert

jakirkham

comment created time in 6 days

issue commentconda-forge/cudatoolkit-feedstock

Initial CUDA 11.3 Conda Packages

It might make sense to do away with building against every minor version at the same time we move to these packages and use CUDA Enhanced Compatibility instead. This will cutdown the amount of rebuilding we need to do while still supporting CUDA 11.0+

jakirkham

comment created time in 6 days

issue commentconda-forge/cudatoolkit-feedstock

Shim packages

Should add that for lightening deployments we can look at using nvprune to target things more carefully and cutdown on size. This is a little tricky to do in the general case (like how we build in conda-forge), but maybe we can envision final build steps that do this kind of splitting either at the end of a CI job or on a user's machine. There are pros & cons to this though that we would want to evaluate carefully

jaimergp

comment created time in 6 days

issue commentconda-forge/cudatoolkit-feedstock

Shim packages

More generally am interested in getting the new packages squared away ( https://github.com/conda-forge/cudatoolkit-feedstock/issues/62 ). This will help us solve a lot of things including adding new CUDA versions, better splitting, providing access to build tooling, etc. We can also take that opportunity to leverage CUDA enhanced compatibility and lighten our build matrices.

jaimergp

comment created time in 6 days

issue commentconda-forge/cudatoolkit-feedstock

Shim packages

With CUDA 11.3+ it will be less of a problem I guess, but still some libraries are well into the hundreds anyway. We can add this only for <=11.2?

The move to 11.3+ already seems pretty fraught. Would rather not add one more thing to the list

jaimergp

comment created time in 6 days

MemberEvent

pull request commentconda-forge/cudatoolkit-feedstock

Add SBSA aarch64 CTK pkgs for CUDA 11.X

Drafted a metapackage and submitted as PR ( https://github.com/conda-forge/staged-recipes/pull/15806 ). PTAL :)

Ethyling

comment created time in 6 days

push eventjakirkham/staged-recipes

John Kirkham

commit sha 19b7524ce721d5a0904d7ad230d81dd7f997c03c

Add `arm-variant` package

view details

push time in 6 days

PR opened conda-forge/staged-recipes

[WIP] Add `arm-variant` package

Adds a metapackage to allow selection between different ARM variants. Currently SBSA and Tegra

cc @Ethyling @raydouglass @dicta @kkraus14 @jaimergp @leofang @quasiben

Checklist

  • [ ] Title of this PR is meaningful: e.g. "Adding my_nifty_package", not "updated meta.yaml".
  • [ ] License file is packaged (see here for an example).
  • [ ] Source is from official source.
  • [ ] Package does not vendor other packages. (If a package uses the source of another package, they should be separate packages or the licenses of all packages need to be packaged).
  • [ ] If static libraries are linked in, the license of the static library is packaged.
  • [ ] Build number is 0.
  • [ ] A tarball (url) rather than a repo (e.g. git_url) is used in your recipe (see here for more details).
  • [ ] GitHub users listed in the maintainer section have posted a comment confirming they are willing to be listed there.
  • [ ] When in trouble, please check our knowledge base documentation before pinging a team.
+53 -0

0 comment

3 changed files

pr created time in 6 days

push eventjakirkham/staged-recipes

John Kirkham

commit sha 7feacc4ab0f2cd7431b668737e267f4aeb05eb41

Add `arm-variant` package

view details

push time in 6 days

push eventjakirkham/staged-recipes

John Kirkham

commit sha 0300458e74dc0b3f216ad63f60240b13ddcc35f8

Add `arm-variant` package

view details

push time in 6 days

create barnchjakirkham/staged-recipes

branch : add_arm-variant

created branch time in 6 days

push eventjakirkham/staged-recipes

Alex Kaszynski

commit sha ec3ea46b4dfe0133f9d821be95365031e4d54600

add yum req and attempt cython fix

view details

Zebedee Nicholls

commit sha 01bcb2a409aaf72602b53642a88d0f9e655c602a

Add mesmer recipe

view details

Zebedee Nicholls

commit sha 09a5e7f5b925298e2431994b91a3dcdfcdfb24db

Fix up dependencies

view details

Zebedee Nicholls

commit sha 5454c48520f34aaf271349fadb73c7b69e7b8f1b

Fix license identifier

view details

Alex Kaszynski

commit sha a64bfad2042b99082fd96023602d30a7c7d4330a

bump version

view details

Zebedee Nicholls

commit sha e277f4400b4bf1a282d232ad983c9bdb2d3181b1

Use version with packaged license file

view details

Zebedee Nicholls

commit sha cf203eb344d159bf4c86249db2bf76bc5015eb7a

Make error more helpful

view details

sarthakpati

commit sha 1cd723c5b441376bf3aedf7c8e0451a169825c4d

generic name

view details

Wolf Vollprecht

commit sha b3ac6e75132851742b9460eeed4de705aa0b1b4c

add vcpkg-tool

view details

Daniel Bast

commit sha e656905d9945894ae81102eda97deba084a39a56

Merge pull request #15564 from izahn/r-wto add r-wto

view details

GitHub Actions on github.com/conda-forge/staged-recipes

commit sha 09f5b643f28092d54892f46d329d743e575d2447

Removed recipe (r-wto) after converting into feedstock. [ci skip]

view details

Wolf Vollprecht

commit sha e7cc18c97777eb3879888a57c61aa73442e1e36d

add run constrained for current vcpkg package

view details

Jenna Lipscomb

commit sha b993e64180315d9fb473d1d8c28a609cfbf915e4

Update download url

view details

Jenna Lipscomb

commit sha aff365fbf168ebf0cbfb9d83c1e59af42a70f66f

Correct sha256

view details

Jenna Lipscomb

commit sha 961dae4939be6f28a24984338a7a990ec5d14881

Add license

view details

Jenna Lipscomb

commit sha 1ff8217566f3f6b2cb540d1233db546e1a51de12

remove license file extension

view details

Alex Kaszynski

commit sha 6cb6a57fd56a61a4c8bb19129e47af23a6d80387

update checksum

view details

Alex Kaszynski

commit sha 4939da2f18f27fe535debab5b7044508b450d4eb

add cython

view details

Wolf Vollprecht

commit sha 72cbb4b79dee7df310c65dc1ee5fed6e1893ca2e

Merge pull request #15418 from tlambert-forks/napari-bioformats rename napari-pims-bioformats to napari-bioformats

view details

GitHub Actions on github.com/conda-forge/staged-recipes

commit sha c7e4efd18f4e44d892e8cc7955cde66cfc5cbe19

Removed recipe (napari-bioformats) after converting into feedstock. [ci skip]

view details

push time in 6 days

issue commentdask/distributed

Are reference cycles a performance problem?

It's possible to tell Cython not to GC certain Cython extension objects. Am less clear on if that will work in our use case or whether it will be a good idea. Should add Cython extension types already disable features like the trashcan by default.

A separate, but also interesting idea in Cython (that I believe I've raised before) is to use Cython freelist for commonly used extension types. This cuts down on the allocation/deallocation cost of a particular type (especially if it is created/deleted frequently). That said, I don't think we've narrowed down what objects this would be useful for or how many objects we tend to use. Both would really be needed to make this useful.

More generally while we have seen some pains around GC. I don't think we have narrowed it down to particular objects (though please correct me if things have changed here I may be a bit out-of-date). Think this is very important to determine where effort is best spent. For example we might think Task* and *State objects and spend a bunch of time improving allocation/deallocation costs around them only to learn it doesn't help.

To put a finer point on this, we have seen that turning off GC seems to be help during reads and MsgPack unpackb specifically ( https://github.com/dask/distributed/issues/4881 ). More recently we have identified large memory usage during serialization causing some pain ( https://github.com/dask/distributed/issues/5107 ). Large memory usage while creating a lot of small temporary objects during serialization may be exactly the sort of thing that could trigger heavy GC usage (and some of these other problems discussed). If this is true, optimizing Task* and *State objects wouldn't help much. Instead being more careful to use memoryviews ( https://github.com/dask/distributed/pull/5112 ), investigating other serialization libraries like msgspec, and possibly an overhaul of serialization altogether ( https://github.com/dask/distributed/pull/4923 ) may all be needed to address this kind of issue.

mrocklin

comment created time in 6 days

pull request commentconda-forge/cudatoolkit-feedstock

Add SBSA aarch64 CTK pkgs for CUDA 11.X

SGTM. I'll go ahead and start a PR to staged-recipes using that name and we can go from there 🙂

Ethyling

comment created time in 6 days

issue commentdask/distributed

Are reference cycles a performance problem?

In general this is the kind of structure that GC is designed to handle. It's hard to know when a large self-referential object can be cleaned up. So this is what GC checks for. Maybe this is already obvious, but thought it deserved stating to set expectations.

mrocklin

comment created time in 6 days