David Duvenaud duvenaud University of Toronto

Democratizing Deep-Learning for Drug Discovery, Quantum Chemistry, Materials Science and Biology

Official Code for Invertible Residual Networks

Code for the paper "Learning Differential Equations that are Easy to Solve"

Kernel structure discovery research code - likely to be unstable

I am in [research] stepped in so far that, should I wade no more, Returning were as tedious as go o'er. -MacBeth

Optimizing control variates for black-box gradient estimation

Graduate topics course on learning discrete latent structure.

Repo for a paper about constructing priors on very deep models.

Code for density estimation with nonparametric cluster shapes.

There's still lots to flesh out, but the basic index set for sets is working.

The next step will be to try using it in some more pandas-like examples.

There was a strange issue with the test for sort failing on my machine (but apparently not when run as part of CI?), which I also fixed.

+204 -13

0 comment

2 changed files

pr created time in 18 hours

create barnchduvenaud/dex-lang

created branch time in 18 hours

On safe-names-dev,

instance {i j} [Ix i, Ix j] Ix (i=>j)
getSize = \(). intpow (size j) (size i)
ordinal = \k. 0                          -- placeholder logic
unsafeFromOrdinal = \k. for i':i. (0@j)  -- placeholder logic

:p for i:((Fin 2)=>(Fin 3)).
i

gives

Compiler bug!

Not implemented: (%runState
1
(\h':Type ref:(Ref h' Int32).
v6:(Unit & Int32) =
%runState
v5
(\h'1:Type ref1:(Ref h'1 Int32).
v6:(Unit & Int32) =
%runState
v2
(\h'2:Type ref2:(Ref h'2 Int32).
v6:Unit =
%while
(\_ign:Unit.
v6:Int32 = %get ref1
v7:Word8 = %igt v6 0
v8:Bool = %toEnum Bool v7
v9:Bool =
case v8 of
-> False
->
v9:Int32 = %get ref1
v10:Int32 = %irem v9 2
v11:Word8 = %ieq v10 1
v12:Bool = %toEnum Bool v11
v13:Unit =
case v12 of
-> ()
->
v13:Int32 = %get ref
v14:Int32 = %get ref2
v15:Int32 = %imul v13 v14
ref := v15

v14:Int32 = %get ref2
v15:Int32 = %get ref2
v16:Int32 = %imul v14 v15
v17:Unit = ref2 := v16
v18:Int32 = %get ref1
v19:Int32 = %shr v18 1
v20:Unit = ref1 := v19
True

v10:Word8 = %dataConTag v9
v10)
v6)
ProjectElt  v6)
ProjectElt  v6))
CallStack (from HasCallStack):
error, called at src/lib/Interpreter.hs:151:10 in dex-0.1.0.0-DR9NUIpDDJVCDBC6gQeZlk:Interpreter

This is not a blocking issue for anything important.

created time in 18 hours

Here's a MWE:

data Set a [Ord a] =
AsSet n:Int elements:(Fin n => a)

data SetIx a l:(Set a) [Ord a] =
MkSetIx Int

gives

Type error:Couldn't synthesize a class dictionary for: (Ord a)

data SetIx a l:(Set a) [Ord a] =
^^^^^

This isn't a blocking issue.

created time in 19 hours

In trying to build custom index sets, I'm hitting limitations in being able to store information at the type level. Specifically, I think the limiting factor is that datatypes can only take types as arguments in their definitions.

What I'd like to be able to write is:

data TableInType n a table:n=>a [Ix n] =
MkTableInType              -- Doesn't store any data except in the type!

But the syntax is not supported.

If that worked, I think I could write the rest of an Ix instance for this custom type:

def unsafeFindInTable {n a} [Eq a] (xs:n=>a) (x:a) : n =
matches = for i:n. xs.i == x
(AsList _ resultTable) = argFilter id matches
resultTable.(0@_)

instance {n:Type} {a:Type} {table:n=>a} [Ix n] Ix (TableInType n a table)
size = size n
ordinal = unsafeFindInTable table
unsafeFromOrdinal = \i. table.(unsafeFromOrdinal i)

def tableToTableInType {n a} (table:(n=>a)) : Type =
MkTableInType n a table

names = ["Alice", "Bob", "Charlie"]
enumerateNames : (tableToTableInType names)=>Int =
for i. ordinal i

:p enumerateNames

closed time in 4 days

duvenaud

Never mind, I realized that if I put parentheses around the table type, as in data TableInType n a table:(n=>a) [Ix n] = then it does parse.

duvenaud

comment created time in 4 days

In trying to build custom index sets, I'm hitting limitations in being able to store information at the type level. Specifically, I think the limiting factor is that datatypes can only take types as arguments in their definitions.

What I'd like to be able to write is:

data TableInType n a table:n=>a [Ix n] =
MkTableInType              -- Doesn't store any data except in the type!

But the syntax is not supported.

If that worked, I think I could write the rest of an Ix instance for this custom type:

def unsafeFindInTable {n a} [Eq a] (xs:n=>a) (x:a) : n =
matches = for i:n. xs.i == x
(AsList _ resultTable) = argFilter id matches
resultTable.(0@_)

instance {n:Type} {a:Type} {table:n=>a} [Ix n] Ix (TableInType n a table)
size = size n
ordinal = unsafeFindInTable table
unsafeFromOrdinal = \i. table.(unsafeFromOrdinal i)

def tableToTableInType {n a} (table:(n=>a)) : Type =
MkTableInType n a table

names = ["Alice", "Bob", "Charlie"]
enumerateNames : (tableToTableInType names)=>Int =
for i. ordinal i

:p enumerateNames

created time in 4 days

This is great! For starters, I'm going to try to add an Ix instance for tables, and then see if I can make the FFT demo enforce power-of-2-sized inputs in the types.

apaszke

comment created time in 5 days

PullRequestReviewEvent

it's doing bubble / insertion sort with quadratic time cost. However, if breaks the list in half recursively, it'll be doing parallel mergesort. Currently it'll do the quadratic time version. --- TODO(Ix)-instance {a b} [Ix a, Ix b] Ix {left:a | right:b}-  dummy_ix__ = 0--def concatTable {a b v} (leftin: a=>v) (rightin: b=>v) : ({left:a | right:b }=>v) =+def concatTable {a b v} (leftin: a=>v) (rightin: b=>v) : ((a|b)=>v) =   -- Note: It'd be nicer to use (a | b)=>v but it's not currently supported.

This comment is now obsolete!

apaszke

comment created time in 5 days

PullRequestReviewEvent

instance Fractional Float64 instance Fractional Float32   divide = \x y. %fdiv x y +'## Index set interface and instances++interface Ix n+  size n : Int+  ordinal : n -> Int+  unsafeFromOrdinal n : Int -> n++def Range (low:Int) (high:Int) : Type = %IntRange low high+def Fin (n:Int) : Type = Range 0 n++-- A max(a, 0) that is friendly to our algebraic simplifier+def clampNonnegPrim (a: Int) : Int =+  z : Int = 0+  isNeg = %ilt a z+  %select isNeg z a++-- TODO: Define Range in surface syntax+instance {l h} Ix (Range l h)+  size = clampNonnegPrim \$ h - l+  ordinal = \i. %cast Int i+  unsafeFromOrdinal = \i. %cast (Range l h) i++instance {q:Type} {i:q} [Ix q] Ix (i..)+  size = size q - ordinal i+  ordinal = \j. %cast Int j+  unsafeFromOrdinal = \j. %cast (i..) j++instance {q:Type} {i:q} [Ix q] Ix (i<..)+  size = size q - ordinal i - 1+  ordinal = \j. %cast Int j+  unsafeFromOrdinal = \j. %cast (i<..) j++instance {q:Type} {i:q} [Ix q] Ix (..i)+  size = ordinal i + 1+  ordinal = \j. %cast Int j+  unsafeFromOrdinal = \j. %cast (..i) j++instance {q:Type} {i:q} [Ix q] Ix (..<i)+  size = ordinal i+  ordinal = \j. %cast Int j+  unsafeFromOrdinal = \j. %cast (..<i) j++instance {q:Type} {i:q} {j:q} [Ix q] Ix (j..i)+  size = clampNonnegPrim \$ ordinal i - ordinal j + 1+  ordinal = \k. %cast Int k+  unsafeFromOrdinal = \k. %cast (j..i) k++instance {q:Type} {i:q} {j:q} [Ix q] Ix (j..<i)+  size = clampNonnegPrim \$ ordinal i - ordinal j+  ordinal = \k. %cast Int k+  unsafeFromOrdinal = \k. %cast (j..<i) k++instance {q:Type} {i:q} {j:q} [Ix q] Ix (j<..i)+  size = clampNonnegPrim \$ ordinal i - ordinal j+  ordinal = \k. %cast Int k+  unsafeFromOrdinal = \k. %cast (j<..i) k++instance {q:Type} {i:q} {j:q} [Ix q] Ix (j<..<i)+  size = clampNonnegPrim \$ ordinal i - ordinal j - 1+  ordinal = \k. %cast Int k+  unsafeFromOrdinal = \k. %cast (j<..<i) k++instance Ix Unit+  size = 1+  ordinal = \_. 0+  unsafeFromOrdinal = \_. ()++def iota (n:Type) [Ix n] : n=>Int = view i. ordinal i++'## Arithmetic instances for table types++instance {a n} [Add a] Add (n=>a)+  add = \xs ys. for i. xs.i + ys.i+  sub = \xs ys. for i. xs.i - ys.i+  zero = for _. zero++instance {a n} [Add a] Add (i:n => (i..) => a)  -- Upper triangular tables+  add = \xs ys. for i. xs.i + ys.i+  sub = \xs ys. for i. xs.i - ys.i+  zero = for _. zero++instance {a n} [Add a] Add (i:n => (..i) => a)  -- Lower triangular tables+  add = \xs ys. for i. xs.i + ys.i+  sub = \xs ys. for i. xs.i - ys.i+  zero = for _. zero

Are these instances for triangular tables even necessary anymore?

apaszke

comment created time in 5 days

Yeah, I'm waiting to rebase a few PRs like this until the other tests pass on the dev branch.

duvenaud

comment created time in 7 days

startedgregversteeg/esh_dynamics

started time in 14 days

create barnchduvenaud/dex-lang

created branch time in a month

startedprinceton-vl/lietorch

started time in a month

push eventduvenaud/dex-lang

commit sha 097965d5094924d349fe904398e0cd729aacbe8d

Shortened helper functions in simplex demo.

push time in a month

push eventduvenaud/dex-lang

commit sha cac2d908368584a0dacc57d48fd6dcc5395dba81

Tidied up simplex example.

push time in a month

push eventduvenaud/dex-lang

commit sha c36776cfb9c0afc6a41f008ded4ec85c79367785

Fixed a bug in simplex example, and parallelized another loop.

push time in a month

push eventduvenaud/dex-lang

commit sha bffddd9652e0c9d77e46b383a0971892552446a1

Turn some stateful loops in simplex example into parallel loops.

push time in a month

push eventduvenaud/dex-lang

commit sha f6e271b5938e03090b55e64dcb9f99412686b79a

Added simplex example to makefile tests.

push time in 2 months

push eventduvenaud/dex-lang

commit sha 1f3b663e217db17b9bcbce6f4972ef4b9fe2fc00

Updated comments and syntax of simplex example.

push time in 2 months

push eventduvenaud/dex-lang

commit sha 906f1fce5038225ae1d78f9175f71bcff97764e8

Query for GPU architecture before CUDA compilation (#515)

commit sha c8a5296714012e8899225ea945a41ea39ab065ab

fix navbar not showing up in static HTML docs

commit sha 68e07022921d898a3962337ae3154b2c101f4b48

commit sha 73782ee97454f63879fd7581a4cb25bcc61186b6

Kernel regression example (#541)

commit sha 5d9597ca54c1992231ec69a67ec9746281f6c234

Center main output section in dex-web

commit sha c852ccccb6c6b4fb35aea2c7a41108f880151d8b

Example : Quaternions literate code (#544)

commit sha f40ca177c474bb232028505f4501ddfdd98e0587

Add a Storable instance for Float32 Co-authored-by: Zoe Chen <zoe.chenzizhao@gmail.com> Co-authored-by: Adam Paszke <apaszke@google.com> Co-authored-by: David Duvenaud <duvenaud@gmail.com>

commit sha bdb157bc63a11b58c551b2478d3e250ce4c4da2c

Add castPtr() function to prelude Co-authored-by: Zoe Chen <zoe.chenzizhao@gmail.com> Co-authored-by: Adam Paszke <apaszke@google.com> Co-authored-by: David Duvenaud <duvenaud@gmail.com>

commit sha bf0bb29a6826935fee93f2db6c0f196baabdf8b2

Move `ErrType` and related utils into its own module.

Put `LabeledItems` and friends into a separate module.

commit sha 84cd39058904fca916c0b38b31811beee71bdfb8

Start adding core data types and classes for our safer naming system.

commit sha 5294555de02998dd16141c40f2115d97864c9209

Compile dexrt with png support

commit sha 89864f609cd465b90bc05b6acb12961ffcb5cab0

Add `HasSafeVersion{E,B}` classes for the safe/unsafe conversion bridge.

commit sha b9e3fb644755c50e96ae2b46be1d343d609aff10

Minor fix for documentation in tutorial.

commit sha 33a385aec59f4a3b98a2f66997c9531c0387a05e

Transition Python tests to use (the more standard) unittest

commit sha b1208d34a93bd7821d660a7a84c4bbd7cfd18e80

Restore support for builds with profiling

commit sha 3273fc3e76c9a7eebaec27a8c5b338c8d48e256d

push time in 2 months

I hope this isn't too esoteric. I'm trying to build a type that carries a list inside of it. The use case is typing partial differential equations, where it's useful to keep track of a list of which partials a particular PDE depends on.

Here's a MWE:

def A (AsList n t : List Int) : Type = Fin n

def constFunc (a : A list) : Int = 0

myList = AsList 1 
myA : A myList = 0@_

:p constFunc myA
Type error:
Expected: (Fin ((\((AsList n _)). n) a))
Actual: (Fin ((\((AsList n _)). n) b))
(Solving for: [a:(List Int32), b:(List Int32)])

:p constFunc myA
^^^

I initially tried to put tables in the types instead of lists, but encountered (seemingly erroneous) "Ambiguous type" errors when typechecking. At least this approach typechecks when defining the functions.

created time in 2 months

startedrtqichen/residual-flows

started time in 2 months

started time in 2 months