profile
viewpoint

levelgraph/levelgraph 1200

Graph database JS style for Node.js and the Browser. Built upon LevelUp and LevelDB.

Raynos/graphics 40

Efficient data structures that represent renderable scenes

danmason/crux 0

Open Time Store

refset/0a-.github.io 0

Compiled by Jekyll: https://github.com/0a-/0a-jekyll

refset/3dtv 0

THREE.js Samples for side-by-side 3d tvs

refset/abci-host 0

Clojure host/server for Tendermint's ABCI protocol.

refset/about 0

Sourcegraph blog, feature announcements, and website (about.sourcegraph.com)

startedlambdaisland/edn-lines

started time in a day

startedmthom/scryer-prolog

started time in 2 days

startedlogseq/logseq

started time in 6 days

startedluciodale/navi

started time in 7 days

fork refset/gungnir

A high level, data driven database library for Clojure data mapping.

https://kwrooijen.github.io/gungnir/

fork in 8 days

startedkwrooijen/gungnir

started time in 8 days

push eventjuxt/crux

Jeremy Taylor

commit sha e322a00a1c1c8fb81b9928ee20f6201505605b92

Docs - fix and clarify standalone configurations

view details

push time in 9 days

create barnchrefset/crux

branch : txfn-ctx-is-inconsistent

created branch time in 9 days

push eventrefset/crux

push time in 9 days

push eventrefset/crux

Daniel Mason

commit sha 1ef158b351e540a7a894f158986207dbbe0a97ae

Add container definitions to start zookeeper and kafka in bench Add kafka backed node to bench Add specific CPU and Memory values for each container in bench task Add 'KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR' to broker container Necessary when running a single-node cluster. Lower memory usage of benchmark tasks

view details

Daniel Mason

commit sha ce764c6e0608437736d5bb2448887987a53a6294

Make the bench on 'sorted-maps' only run on one node type

view details

James Henderson

commit sha 6efddde2625bb4eca2ea2064d8dea3b026e7b878

Merge branch 'bench-start-kafka'

view details

James Henderson

commit sha 45154006d288a08cca89a96916cfa9f6c5c5040c

don't need spec to conform a tx-event, it's just destructuring, and spec's costly

view details

James Henderson

commit sha ca0b540782754ee8df7e50db10fd086616a82843

downgrading put pre-commit check to an assert now that we're proactively indexing docs in the consumer

view details

James Henderson

commit sha 3a8bd10cc5350a165aaf021d454cfc33373c1521

not sleeping if we've consumed transactions, let's get straight on with it

view details

James Henderson

commit sha b84a6697aa076e22c1f7dc009fb8e381af3d6235

shortcutting creating a tx-log if we know there's no more transactions

view details

James Henderson

commit sha ab10e335f7eb2f2f97be0faef6ea8bcd20c1414f

Merge branch 'tx-consumer-performance'

view details

James Henderson

commit sha e0b19195751891e957077548d4e167e77b5d0899

sorted maps microbench example code

view details

James Henderson

commit sha 95c5508c7ff4c300ee0941853b5462af2efbf461

not creating extra sorted maps when hashing, we just serialise them in order

view details

James Henderson

commit sha 5a5b8a5ee9d8b599decb740cb2a212d245a3d037

Merge branch 'sorted-maps-685'

view details

James Henderson

commit sha e456aa10423dece665e1a02ae0aaf7c5a94416ad

index version 6 now that we're hashing sets differently

view details

James Henderson

commit sha f0d8f6f4c056dd1e3de2ac1b3845907ab004f5cf

prepare 1.8.0

view details

Daniel Mason

commit sha cbfb774941ee75cff5c3cc0eb8b54bc133c38a1e

Fix docker release flow on circleci

view details

Daniel Mason

commit sha 8cb82d434b60cbe4bc21658bd4ba5486fb7c98ab

Add HTML formatting for the nightly benchmark email.

view details

Jon Pither

commit sha c1330124774f9e66b2a32888a4b465ab054adefd

Trimming blank lines

view details

Jon Pither

commit sha 2efc0f124be488908422e5ead4a5bd0460e66718

Fixtures wrong way around to exercise kafka using a centralised doc-store

view details

Daniel Mason

commit sha 953c3d911bd816f3ca2dd2c76cdc94e33b0f89da

Increase timeout on tx-log poll to that of the doc topic poll

view details

Daniel Mason

commit sha 50a0c60eec86ee526e6f9c5ad1a8cea73bc428d0

Fix minor issues within watdiv_datomic

view details

Daniel Mason

commit sha 392d641a0d49d9fe89364978ef9d32a73b6f53b0

Force watdiv bench to exit after running Currently, seemingly due to a bug, the queries do not free up their resources properly and the program will run indefinitely after completing. For the moment, using a direct system call to exit the program after the bench has completed allows this to be ran.

view details

push time in 9 days

create barnchrefset/crux

branch : txfn-return-ops-forked-ctx-failure

created branch time in 9 days

push eventrefset/crux

msladecek

commit sha a3355081d41c775240bfdd3c14c16113001a3507

docs: remove references to broken links sources of breakages: - branch name changed from `master` to `develop` in seancorfield/next-jdbc - example `standalone_webservice` was removed in 89671f1adcf0ec3ec01dae16a37729f72c865951

view details

James Henderson

commit sha 1742e1e80b70dde123ac098ad3a7712ec5380638

nREPL only passes a Writer here, not a PrintWriter

view details

Daniel Mason

commit sha 567ce44c0de7b6f537a48ed73500bbc17c8c0e99

Add dropwizard `status-reporter` - augments status with gauges.

view details

Daniel Mason

commit sha 245245a924bcff6de48ad48092240c96767bce14

Add meters/timers into returned crux.metrics map.

view details

James Henderson

commit sha ee64f264044685db9d16449a007f762ce7f5600f

giving the original node access to the full status so that it can be returned from the HTTP server

view details

Daniel Mason

commit sha 80e8bd1a7065417834e4b933341dfcfbd6e4a670

Output statuses as description lists.

view details

Daniel Mason

commit sha 84978229563521960631730682aea7fc862c3b3b

Ensure can still read topology options if using file/objects

view details

Daniel Mason

commit sha 8d67af08aeab1423c635e15a6a4b52e377d650ce

Render timers/meters as nested maps.

view details

Daniel Mason

commit sha 6fd1ace97a211c0cba8f514ec26a0f2fa8788343

Add further information from timer snapshot to metrics status

view details

Daniel Mason

commit sha 99318ce74b77c981a3b75b9c5ea77bdee8b7080d

Add `sort-maps` to sort status maps. This could potentially be used elsewhere in rendering the map.

view details

Daniel Mason

commit sha 436a47aa873c11d5be81e97d5101ae34c33632b7

Rename meter/timer keys.

view details

Daniel Mason

commit sha 4a2375d623cd6ff98c73af464fec8244ada4b6ae

Unnecessary mapcats->map

view details

Daniel Mason

commit sha 76e4bf6966a83572b2dcd5dab2e9faf5798efe6c

Convert nanos->ms, add units.

view details

James Henderson

commit sha e3703ede36e552648787990806903867672c442e

Merge branch 'add-metrics-to-status'

view details

Daniel Mason

commit sha 7fd1cb580c57d05a496cf5dec56068d71cfc7a2f

Various resolution of dependencies

view details

Daniel Mason

commit sha 67ba7c7b2b13aa8e23d9f846c0fefc4e262eccf6

Test level dependency resolution.

view details

James Henderson

commit sha ab8af0810e845777a68f6bda2a6cbd1390c0454e

moving crux-test deps to test profile to get rid of a whole load of dependency conflict resolution

view details

James Henderson

commit sha b6235c4d35f6f23da9cba62b0b87206b7f788914

Merge branch 'dependency-warning-bash'

view details

James Henderson

commit sha 6bcde346afcc9f4d71d021846a1dd10d953aa223

missed a few dep conflicts at the top level of the project

view details

James Henderson

commit sha 3b05dea08d61c6115fa2729d022dc9d0d7c74b31

add project to crux.query * using the document store to return the correct cardinalities of pull values * add '*' syntax

view details

push time in 9 days

startedthegeez/databrowser

started time in 13 days

push eventdanmason/crux

Jeremy Taylor

commit sha a5079e91042575cdbe1d0dccfff0c512f2f7ef77

WIP styling and bug fix

view details

Jeremy Taylor

commit sha 504bdc08b044125e87581f20075b9f1e4087cb8d

WIP rebase

view details

Jeremy Taylor

commit sha 9272d1b541aa9e41b75d85e3f1f7eb2094249984

WIP pre-rebase 2

view details

push time in 13 days

created tagdanmason/crux

tag20.06-1.9.0

Open Time Store

created time in 13 days

create barnchdanmason/crux

branch : dan/data-browser-edits

created branch time in 13 days

startedCirru/bisection-key

started time in 14 days

issue commentjuxt/crux

Crux can restrict access to documents based on a user's permissions (ABAC)

@malcolmsparks and I are experimenting with a set of authorisation decorator capabilities covering:

  1. RDF-like property and class inheritance using Datalog rules
  2. Using those same rules for implementing primitive authz
  3. XACML-inspired ABAC modelling
  4. High-level EQL with embedded Crux Datalog resolvers (i.e. unrelated to (eql/project ?e ...))
  5. Authz-aware EQL resolvers

WIP (across various branches): https://github.com/crux-labs/crux-auth

jarohen

comment created time in 15 days

fork refset/transistor

https://crates.io/crates/transistor

fork in 15 days

startednaomijub/transistor

started time in 15 days

startedp-kolacz/org-linkz

started time in 20 days

issue commentjuxt/crux

Standalone event-log-kv-store configuration is confusing

Regardless of how the mechanism works, I see two options:

  1. Prevent use of kv persistence without a persisted event-log OR
  2. fix: 2.a. event-log-dir not configured causes open-tx-log to throw as no protocol on ingest client AND 2.b. entity API not working on node restart when event-log-dir not configured
jarohen

comment created time in 22 days

starteddbs-leipzig/gradoop

started time in 23 days

push eventrefset/crux

Daniel Mason

commit sha a121bdf06b367f5a95f8aff58de3a81021e6a6d9

Ensure we use batched ids. #902

view details

Daniel Mason

commit sha f7150590c4e1b3177bf3057a7265c0e87eaf56ae

Add :crux.config/file-path type.

view details

Daniel Mason

commit sha 8b88172dcf174868a618db5140639b5f9380e392

Replace `crux.config/string` usages with `crux.config/file-path`

view details

Daniel Mason

commit sha 09f60a0454775dd89946f26621ef68dd7919814b

Replace unnecessary conversions to string.

view details

Daniel Mason

commit sha fdbff026b745d96ef25e474be3128e131f0a3944

Use Path instead of `string`.

view details

James Henderson

commit sha 1dde3bce30a27dafac1ad4dd03becb2943957cf4

Merge branch 'accept-file-for-filepath-985'

view details

James Henderson

commit sha 0704e0a151b8ed8961ea08e47e0cdbaf40e6a964

reordering query namespace, move functions closer to usage

view details

James Henderson

commit sha 842575fc3c06280cd5f86240933d2f9dda4a7926

combining eager/lazy query implementations, running queries on caller thread, no more query thread pool

view details

James Henderson

commit sha 2bc672c5188e579f11be56cb299f22795e8b09d7

only applying timeouts to eager queries

view details

James Henderson

commit sha 1923fb564fcb2306c4fe5fc01fc84cbe80e1e0ea

Merge branch 'combining-query-implementations-846'

view details

Daniel Mason

commit sha a1a125fc29cd7b8ff92d1c022ab25fecac2f2b14

Use arrow icon for 'console' dropdown.

view details

Daniel Mason

commit sha fa59b68b2c44a4b392a765b4f561001a83029ab6

Dropdown arrows for validtime/transaction time.

view details

Daniel Mason

commit sha d84eee76ced9b91a8d0474c8c79fac92e92f019b

Make 'query history' be within own tab.

view details

Daniel Mason

commit sha 141e7b86b411d40259ddbe3314b940737691080b

Editing tab link style

view details

Daniel Mason

commit sha 5038bce93e31e245863da5eaedb25bb21b0ea7c1

Changes to valid time/transaction time labels.

view details

Daniel Mason

commit sha a21df1cc0ad67c54710e2bbe262221c7346218b6

WIP - Add tutorials to query/entity editors + improved validation. Need to make sure the contents from entity root get properly replaced

view details

Daniel Mason

commit sha d862f8478f82f860b67e8b09c7f0f88337fc519d

Remove footer - rework header contents.

view details

Daniel Mason

commit sha 3a6ba5443b2f5a61ccbe19f74dd15ad8ba5e1760

Show form-pane by default, if not already set.

view details

Daniel Mason

commit sha e1e47a80fbfaf873c239a65368d3585c5a2fa490

Changes to status on http-server, and status route to SPA.

view details

Daniel Mason

commit sha ec7a4ef80d41a171dffce24e123b9bad0daa007c

WIP - Work on homepage contents, styling.

view details

push time in 23 days

starteddjjolicoeur/datamaps

started time in a month

startedjjttjj/wm

started time in a month

PR closed juxt/crux

Docs changes

WIP - to resolve #747 and more.

+47 -47

1 comment

3 changed files

refset

pr closed time in a month

pull request commentjuxt/crux

Docs changes

Closing this PR following https://github.com/juxt/crux/commit/8f152c61155f152e2f98062295b6d6b287d0210e

refset

comment created time in a month

push eventjuxt/crux

Jeremy Taylor

commit sha 8f152c61155f152e2f98062295b6d6b287d0210e

Docs updates to satisfy #747 part-1

view details

push time in a month

push eventrefset/crux

Håkan Råberg

commit sha d41fc2bf439c4215924a812de8c41518b842ffa3

Fixing two issues with buffer usage that could lead to bugs.

view details

James Henderson

commit sha 047a928e147eb24bfe48f5e14cd5d2091b539ff6

add repro for #443

view details

Håkan Råberg

commit sha a6d8e4c2bf04e10fe039018b413e9efff72fc16c

Allowing users to explicitly set which vars to bind before running rules/or-joins (#980, fixes #946) * Initial steps to fix up rules a bit. * Should not expand into or-joins with literals in the head. * Default to old behaviour when there are no bound vars.

view details

Håkan Råberg

commit sha 13214ef4dc01c8db7b872b93b2e8b0b30980844b

Rules bound vars, take 2 (#981) * Remove large commented out test that was really more a spike for leaf-vars. * Removing old explicit transitive dependency thing, can't recall why it's needed, tests pass. Test showing fix for #946. * Whitespace.

view details

James Henderson

commit sha d6d9ab2bafbb3e29a79a7742ce03edc356c760fb

long out-of-date todo

view details

James Henderson

commit sha fe1258123da651c01f85df119858edaaba6375d8

guard against nil `found-k` in `advance-iterator-to-hash-cache-value`

view details

James Henderson

commit sha 15ee2a0c170d7f15ce525eb5d60f6f6f173dbb19

test basic query against speculative db too

view details

James Henderson

commit sha 5ca5cd7b74504c163b122b7d5e7d059d29bec4b4

Separate out tx-ingester so that we can implement multiple tx-consumers

view details

James Henderson

commit sha 0aa1bfb01f2107cf2bc4d0ae618e1beef3c6ab20

implement tx-consumer independently in each implementation, so that it doesn't have to poll

view details

James Henderson

commit sha 111bd419bd9031648c307387a0f4b2207cb68c22

Merge branch 'tx-ingester'

view details

James Henderson

commit sha 96446946f70b6b4c65223bfa5a0e803158f5f82b

bus/listen accepts multiple event-types

view details

James Henderson

commit sha d7f4dd5a8d1833c5c45415079aa52b183cfe92d7

bus has an 'await' fn that can be used to block until we receive a certain event type

view details

James Henderson

commit sha 8d4a792ee03f475b000ff23d5c7a727d866dd411

await-tx/await-tx-time use bus's await

view details

James Henderson

commit sha 78a6a4e3d9f126208c3e7a4f17b3382931e77bd3

Merge branch 'no-polling-await-tx'

view details

James Henderson

commit sha d943708aa81b94342b220364cc07ffe0031f362e

previously await-tx introduced enough delay, now it doesn't. success, I think?! :)

view details

James Henderson

commit sha 18b1215b529afece93c0b4c2d1fa72a44295a6d4

allow overriding derived project version with env vars

view details

James Henderson

commit sha 8270fd1a1931e77962cebb65c8c4ee1f1faf7d89

not returning a db value if the speculative tx doesn't commit.

view details

James Henderson

commit sha 749a5b5a6ec839fd643712df6d979bbdcb318f59

javadoc/docstring for with-tx

view details

James Henderson

commit sha 6e667e7f30b17480d020c34efffc32d8f87f0c6c

remove unnecessary doto

view details

James Henderson

commit sha 1b2a05fde2b701a408b8ae5f84bdb5a9270aab0f

lost the commit where we default committed to true

view details

push time in a month

push eventrefset/crux

Jeremy Taylor

commit sha ff3e16ece0e77cd45d2841886642e5af19202f85

Typos

view details

push time in a month

push eventrefset/crux

James Henderson

commit sha c9004a9d1479a798e020bdb0a9a8daf8a72bb342

add easy `*readings-limit*` dynamic var to ts-devices for quick n easy micro-benchmarking

view details

Daniel Mason

commit sha 2d43ec0c90456aa529f6d41725027812a3f81e1d

Add link to S3 uploads of Javadocs

view details

James Henderson

commit sha 4494a6b6237a54a055d0f486eb262113fef1d000

correct env var in bench readme

view details

James Henderson

commit sha 57ea94be234cf4f12b566f4718a1da7d7921557e

first pass at CONTRIBUTING.adoc

view details

James Henderson

commit sha eb1078bb278a486df9867ecd549d0d5b96e1a4b3

without a newline between header and paragraph, this doesn't render

view details

Håkan Råberg

commit sha e779e85b54a6b0ecb9897e37024673496589a4c7

Index store (#815) Introducing the IndexStore protocol. Changing boundary of Indexer so the TxConsumer now contains all logic. Currently all logic is still delegating to the default KvIndexer/KvIndexStore.

view details

James Henderson

commit sha 54e635a54a6cdc13c8c64689e79b39a56b4aa39e

S3 doc-store happy path

view details

James Henderson

commit sha 0d869ebfbcc808440c8c2e72a0a5e60f170c2f11

add S3Configurator to configure S3 requests

view details

James Henderson

commit sha d3973255a875f0a1027cd5d8e6d103994e9b138c

add ability to choose S3 serde

view details

James Henderson

commit sha 8fbb909295a7b9150c5eff4e32bdf14973db90de

S3 tests running locally

view details

James Henderson

commit sha 941ca4b3dd4fb31412806703e386958cb236e642

Add CF template for the test S3 bucket

view details

James Henderson

commit sha 8fce7d0118944dea426d03b2bffbb6018edc5016

not throwing a massive log message if it's just a 404

view details

James Henderson

commit sha d58ee523c08099cfe4f037f39cfa16c4e145a884

crux-s3 README

view details

James Henderson

commit sha 258b080998f933c4dff12d0449dca34bb5783b40

Merge branch 's3-doc-store-810'

view details

James Henderson

commit sha d702d5b923f0a3493b9ffb9c0df6c45dc5ba32b8

Using `:crux/` prefix for events rather than an internal Crux ns

view details

James Henderson

commit sha 107e7f39f80983f3ec6ccaab2897edf3731372be

bus listeners can be detached too

view details

James Henderson

commit sha 2ac562598f9b5c7344a497d69b47274d8e71a6cc

exposing a public bus

view details

James Henderson

commit sha a239ad558a41afd5f27a3119a7b7aa55245d96e4

add `with-tx-ops?` option to `:crux/indexed-tx` event

view details

James Henderson

commit sha 3388ad8094f5813bfd09ae3c8254f83dee60c4db

add docstrings/javadoc for the public bus

view details

James Henderson

commit sha a3a36fd9bb63c102c1d437b01f9fa95e9f876d31

add public bus documentation to 'transactions' docs

view details

push time in a month

startedMaggieAppleton/digital-gardeners

started time in a month

startedwilkerlucio/spec-coerce

started time in a month

startedAshKyd/ui95

started time in a month

startedvvvvalvalval/scope-capture

started time in a month

startedcsm/crux-aws

started time in a month

startedParkerICI/datalog-json-parser

started time in a month

issue commenttonsky/datascript

VAET index

Yeah, it's certainly a tradeoff. I suspect that without VAET the deletion time cost might be too unpredictable at a large-enough scale to maintain consistent throughput, whereas the space (+ time) costs of inserting into VAET are amortised which makes it a much safer default for a highly-distributed production system like Datomic.

I think the non-deletion use-cases can be described as "exploratory" queries, where your data set is large (many attributes) and not well understood (can't predict which attributes might be relevant). I imagine Roam's knowledge graph "backlinks" feature might be a good use-case for benefiting from a native VAET (backlink!) index, if such a graph ever grew large enough (i.e. multi-user).

Incidentally, I just came across these discussions that mention Datomic's ability to do wildcard reverse lookups with the Entity API via (keys (.cache (.touch ent))):

  • https://stackoverflow.com/questions/15629085/in-datomic-how-is-it-possible-to-find-out-what-keys-are-available-for-reverse-l
  • https://stackoverflow.com/questions/14189647/get-all-fields-from-a-datomic-entity
  • (admittedly I can't see this behaviour officially documented anywhere, and I've not confirmed that it works with my own eyes)

Providing a similar index to VAET is something that the Crux team has been contemplating recently, motivated by our work on a generic entity navigation UI. Crux avoids the deletion issue by choosing not to enforce referential integrity, i.e. there are no explicit reference-type attributes, only values and ID that might happen to correspond.

I'll leave it at that for my contributions to this thread but hopefully it's useful context for some. I remember thinking about this very question several years ago, so it's good to have finally written it down :)

darkleaf

comment created time in a month

issue openedjuxt/crux

Users can configure an allow-list of predicate functions available during queries

Users should be able to control which core/built-in predicates and "custom" predicates (reachable via fully qualified names) are accessible during queries. This will help to secure the execution of queries that are constructed by untrusted clients. This could be achieved via a new topology option with a set of names for the allowed functions.

This is an intermediate towards more comprehensively resolving https://github.com/juxt/crux/issues/166

created time in a month

issue commenttonsky/datascript

VAET index

I'm saying that if you have many different :db.type/ref attributes then you can't query [?e _ :some-id] without naively scanning through all combinations in AVET.

I don't know exactly how VAET would be wired up internally, but I would expect to see something else more advanced than the EAVT filter here: https://github.com/tonsky/datascript/blob/6e80073355ef35bf0b0d94afd3dd5d0f97ded2c1/src/datascript/db.cljc#L504

Again, I doubt most users would be storing enough attributes or overall data in DataScript to warrant changing anything. Most of the time people know exactly which attributes they need in their queries.

For the curious, this is where near enough the same kind of scanning occurs for deleting entities: https://github.com/tonsky/datascript/blob/6e80073355ef35bf0b0d94afd3dd5d0f97ded2c1/src/datascript/db.cljc#L1278

darkleaf

comment created time in a month

startedfacebookincubator/LogDevice

started time in a month

issue commenttonsky/datascript

VAET index

VAET is only necessary when you don't know which reverse reference-type attributes might be applicable for a given entity. For a small DataScript-sized system this is probably never an issue. At a larger scale, with potentially millions of attributes, you need VAET in order to do explorative navigation of the graph efficiently (when you can't be sure which reverse attributes might be relevant beforehand). Without VAET you have to scan through all the possible combinations in AVET.

darkleaf

comment created time in a month

fork refset/gist

gist upper enterprise ontology

fork in a month

startedsemanticarts/gist

started time in a month

push eventrefset/crux

James Henderson

commit sha c9004a9d1479a798e020bdb0a9a8daf8a72bb342

add easy `*readings-limit*` dynamic var to ts-devices for quick n easy micro-benchmarking

view details

Daniel Mason

commit sha 2d43ec0c90456aa529f6d41725027812a3f81e1d

Add link to S3 uploads of Javadocs

view details

James Henderson

commit sha 4494a6b6237a54a055d0f486eb262113fef1d000

correct env var in bench readme

view details

James Henderson

commit sha 57ea94be234cf4f12b566f4718a1da7d7921557e

first pass at CONTRIBUTING.adoc

view details

James Henderson

commit sha eb1078bb278a486df9867ecd549d0d5b96e1a4b3

without a newline between header and paragraph, this doesn't render

view details

Håkan Råberg

commit sha e779e85b54a6b0ecb9897e37024673496589a4c7

Index store (#815) Introducing the IndexStore protocol. Changing boundary of Indexer so the TxConsumer now contains all logic. Currently all logic is still delegating to the default KvIndexer/KvIndexStore.

view details

James Henderson

commit sha 54e635a54a6cdc13c8c64689e79b39a56b4aa39e

S3 doc-store happy path

view details

James Henderson

commit sha 0d869ebfbcc808440c8c2e72a0a5e60f170c2f11

add S3Configurator to configure S3 requests

view details

James Henderson

commit sha d3973255a875f0a1027cd5d8e6d103994e9b138c

add ability to choose S3 serde

view details

James Henderson

commit sha 8fbb909295a7b9150c5eff4e32bdf14973db90de

S3 tests running locally

view details

James Henderson

commit sha 941ca4b3dd4fb31412806703e386958cb236e642

Add CF template for the test S3 bucket

view details

James Henderson

commit sha 8fce7d0118944dea426d03b2bffbb6018edc5016

not throwing a massive log message if it's just a 404

view details

James Henderson

commit sha d58ee523c08099cfe4f037f39cfa16c4e145a884

crux-s3 README

view details

James Henderson

commit sha 258b080998f933c4dff12d0449dca34bb5783b40

Merge branch 's3-doc-store-810'

view details

James Henderson

commit sha d702d5b923f0a3493b9ffb9c0df6c45dc5ba32b8

Using `:crux/` prefix for events rather than an internal Crux ns

view details

James Henderson

commit sha 107e7f39f80983f3ec6ccaab2897edf3731372be

bus listeners can be detached too

view details

James Henderson

commit sha 2ac562598f9b5c7344a497d69b47274d8e71a6cc

exposing a public bus

view details

James Henderson

commit sha a239ad558a41afd5f27a3119a7b7aa55245d96e4

add `with-tx-ops?` option to `:crux/indexed-tx` event

view details

James Henderson

commit sha 3388ad8094f5813bfd09ae3c8254f83dee60c4db

add docstrings/javadoc for the public bus

view details

James Henderson

commit sha a3a36fd9bb63c102c1d437b01f9fa95e9f876d31

add public bus documentation to 'transactions' docs

view details

push time in a month

created tagrefset/crux

tag20.06-1.9.0

Open Time Store

created time in a month

fork refset/crux-typescript-healthcare-demo

Demo Crux Usage from TypeScript in the Healthcare Context

fork in a month

fork refset/abci-host

Clojure host/server for Tendermint's ABCI protocol.

fork in a month

issue commentjuxt/crux

tx-fn breaks indexing

Hi @nivekuil thanks for identifying and reporting the issue so thoroughly.

I believe the problem here is happening during the reading of the function, rather than the execution, and I can't see that our existing tests have accounted for this level of error. The underlying issue is that the e logical variable needs quoting.

This version with the quoted e doesn't break the indexer:

(def tx
  (crux/submit-tx node
                  [[:crux.tx/put
                    {:crux.db/id :delete-where-from
                     :crux.db/fn
                     '(fn [ctx from-id]
                        (let [edges (crux.api/q
                                     (crux.api/db ctx)
                                     {:find ['e] :where [['e :from from-id]]})]
                          (mapv (fn [edge-id] [:crux.tx/delete (first edge-id)]) edges)))}]]))

We will hopefully be able turn around another release with a fix for this very soon. Sorry for the rough edges...your patience in the meantime is very appreciated :)

nivekuil

comment created time in 2 months

startedtonsky/grumpy

started time in 2 months

issue commentjuxt/crux

Approximate dates for Beta? Evaluating

Hi @jeveloper our beta release plans quietly morphed into today's latest 1.9 release. You can consider this version a stable foundation for all manner of evaluations and long-term support (production users should always get in touch with us directly though!).

Feel free to reach out directly for any further questions: jdt@juxt.pro

jeveloper

comment created time in 2 months

issue closedjuxt/crux

Approximate dates for Beta? Evaluating

Hello there team!

Thank you for amazing work.

I am evaluating this product for a potential use for a project. How likely will you be able to reach Beta by November/December or end of year? 2019

Thank you kindly and feel free to reach directly if this is sensitive.

closed time in 2 months

jeveloper

fork refset/shan-shui-inf

Procedurally generated Chinese landscape painting.

fork in 2 months

startedLingDong-/shan-shui-inf

started time in 2 months

startedirshadshalu/music-grid

started time in 2 months

starteddaanzu/kaldi-active-grammar

started time in 2 months

issue closedjuxt/crux

Performance disparity between raw HTTP REST queries and built-in HTTP client

When querying an empty Crux node spun up on a DigitalOcean box from the crux-standalone Docker image (in-memory KV store, no txes yet), the empty result set comes back ~2x as fast via the HTTP REST API when compared to the built-in HTTP client API for the same query.

Compare running this query (which should return the empty set as a result):

(ns crux-test
  (:require [dotenv :refer (env app-env)]
            [taoensso.timbre :as log]
            [org.httpkit.client :as http]))

(def crux-node-url (env "CRUX_NODE_URL"))

(defn q [query]
  (http/post
    (str crux-node-url "/query")
    {:headers {"Content-Type" "application/edn"}
     :body    (pr-str {:query query})}))

(time (read-string (slurp (:body @(q '{:find  [?last-name]
                                         :where [[?person :first-name "John"]
                                                 [?person :last-name ?last-name]]})))))
"Elapsed time: 353.379041 msecs"
=> #{}

If I run it several times, the elapsed time converges to ~170ms.

Compared to the built-in HTTP client:

(def crux-client (crux/new-api-client crux-node-url)

(let [db (crux/db crux-client)] ;; not profiling call to crux/db
    (time (crux/q db '{:find  [?last-name]
                       :where [[?person :first-name "John"]
                               [?person :last-name ?last-name]]})))
"Elapsed time: 329.54181 msecs"
=> #{}

This query time stays at around ~330ms no matter how many times I repeat it, which still feels suuuper slow to me for a query against an empty DB.

Could this be HTTP pipelining that's resulting in the time difference?

When I run the query directly on the DigitalOcean box, the query returns in about 18ms.

closed time in 2 months

theronic

issue commentjuxt/crux

Performance disparity between raw HTTP REST queries and built-in HTTP client

I will close this for the time being but let's reopen it again as soon as anyone manages to consistently reproduce the issue. Please feel free to continue the conversation on Zulip/Slack for any further help with debugging your specific setup, or alternatively, respond here if you discover any new clues about what's been happening :)

I hope it's not too big a blocker for your development work. I can certainly see that it would be useful to have a polished "Get Crux running on Digital Ocean" tutorial to save you from this kind of headache in future!

theronic

comment created time in 2 months

startedbob-cd/bob

started time in 2 months

startedumlaeute/v4l2loopback

started time in 2 months

startedreplikativ/datahike-frontend

started time in 2 months

fork refset/crux-dynamodb

Crux TX-Log on dynamodb.

fork in 2 months

startedordnungswidrig/crux-dynamodb

started time in 2 months

issue commentjuxt/crux

Microsecond resolution for valid-time?

Hi @stevana we don't natively support higher-resolution timestamps just yet, although we have been thinking about making this configurable eventually, to open up support for nanoTime() nanoseconds. Whilst there is no immediate roadmap agenda for this work we would be keen to hear more about your use-case/requirements - we could potentially accelerate a decision and implementation.

Right now we rely on 64-bit Java Dates that represent valid-time and transaction-time as milliseconds since 1970. As it happens, I have recently played around with manipulating these 64 bit value purely in userspace to encode entirely different time ranges, including vector clock information, so I know it's already possible to support microseconds if you are willing to use a few "translation" functions on the way in & out of the Crux tx operations and db+history APIs. The downside of encoding microseconds like this is that you will need to constrain the range of expressible timestamps at the other end of the scale, although I doubt your use-case will make use of those bits anyway :)

This encoding/decoding experiment I wrote for a Hybrid Logical Clock (as in https://jaredforsyth.com/posts/hybrid-logical-clocks/) sort of illustrates what I mean:

(def ^:const total-bsize 64)

(def ^:const ms-bsize 42) ;; (= (Date. max-ms) #inst "2109-05-15T07:35:11.103-00:00")
;; min time supported 1970-01-01
;; max time supported #inst "2109-05-15T07:35:11.103-00:00"
(def ^:const count+node-bsize (- total-bsize ms-bsize)) ;; 22b

(def ^:const count-bsize 10) ;; max count 1024 ;; max nodes 4096
(def ^:const node-bsize (- count+node-bsize count-bsize))

(def ^:const not-count-bsize (- total-bsize count-bsize))
(def ^:const not-node-bsize (- total-bsize node-bsize))

(def ^:const max-ms (dec (bit-set 0 ms-bsize)))
(def ^:const max-count (dec (bit-set 0 count-bsize)))
(def ^:const max-node (dec (bit-set 0 node-bsize)))

(defn- hlc->long [{:keys [ms count node]}]
  {:pre [(<= 0 ms)
         (<= ms max-ms)
         (<= count max-count)
         (<= node max-node)]}
  (bit-or node
          (bit-shift-left count node-bsize)
          (bit-shift-left ms count+node-bsize)))

(defn- long->hlc [l]
  {:ms (unsigned-bit-shift-right l count+node-bsize)
   :count (-> (bit-shift-left l ms-bsize)
                (unsigned-bit-shift-right not-count-bsize))
   :node (-> (bit-shift-left l not-node-bsize)
             (unsigned-bit-shift-right not-node-bsize))})

(defn hlc-long->date [l]
  (Date. (:ms (long->hlc l))))

;; (long->hlc (hlc->long {:ms (inst-ms (Date.)) :count 13 :node 1})) ;; test
stevana

comment created time in 2 months

issue commentjuxt/crux

Performance disparity between raw HTTP REST queries and built-in HTTP client

Thanks for ruling out the binding config as a possible reason.

I've done some sanity checking to confirm everything is happening as we would expect when running various combinations of nodes & clients in a local repl and also when using the container version locally (juxt/crux-standalone:20.05-1.8.4-alpha). Nothing jumps out that suggests any reasons why you might be seeing different HTTP behaviours depending on how you generate the HTTP requests (remote client vs. normal Clojure http call).

Are you able to reproduce this problem with a local container setup?

Can you describe more about how Digital Ocean is adding SSL? Is there a load balancer?

Another option is to inspect your network traffic for clues using a proxy, perhaps something like this: https://www.charlesproxy.com/ (I have no idea if this is a good tool or not, but it looks like it could be helpful for you)

theronic

comment created time in 2 months

issue commentjuxt/crux

Performance disparity between raw HTTP REST queries and built-in HTTP client

Hi @theronic it looks to me like there must be an extra http roundtrip happening. Before we dig in further though, can you try forcing the client to definitely use httpkit (as opposed to clj-http) and see what happens?

See the comments: https://github.com/juxt/crux/blob/master/crux-http-client/src/crux/remote_api_client.clj#L29

theronic

comment created time in 2 months

startedthheller/wasm-pack-cljs

started time in 2 months

issue commentjuxt/crux

Unused arg bound to false causes query to not return any results

@Vaelatern hey, you can clone the repo and do a lein sub install in the project root to create a local snapshot version that you can depend on temporarily from your other project.

jarohen

comment created time in 2 months

startedvlaaad/reveal

started time in 3 months

startedoracle/graal

started time in 3 months

fork refset/dataworks

Hot-swap API endpoints and stream processors using Clojure and Dataworks

fork in 3 months

startedacgollapalli/dataworks

started time in 3 months

IssuesEvent

issue commentcrux-labs/crux-decorators

Problem with aggregation, sums not correct

Reopening until there's an official resolution, but thanks again!

tsaixingwei

comment created time in 3 months

startedclojure/clojure

started time in 3 months

issue commentjuxt/crux

Transaction function impls are passed a context

  • Ideally, in addition to the context of the start of transaction, also a "current" view of the db/context at the point of the operation in processing of transaction vector
jarohen

comment created time in 3 months

startedjpmonettas/clograms

started time in 3 months

issue commentjuxt/crux

Standalone event-log-kv-store configuration is confusing

There has been additional confusion resulting from the likelihood of accidentally typing :crux.standalone/event-log-kv-store instead of :crux.standalone/event-log-kv - perhaps this can somehow be solved at the same time.

jarohen

comment created time in 3 months

push eventjuxt/crux

Jeremy Taylor

commit sha 7686dd011d37e7bb0426cb9a4a508fd26b8bd032

Update crux-node-1 architecture for document store

view details

push time in 3 months

issue commentcrux-labs/crux-decorators

Problem with aggregation, sums not correct

(As mentioned on the Zulip) This is a consequence of a change we made a few months ago, which is to deduplicate the lazy results, but the aggregation decorator was relying on the results not being deduplicated. Sorry for the confusion. See https://github.com/juxt/crux/issues/631 for context.

We are unlikely to restore the old behaviour soon, if at all, but a workaround is to add an extra noop select:

'{:aggr {:partition-by [?gender]
   :select {?money-sum [0 (+ acc ?money)]
            ?e-noop [0 ?e]}}
   :where [[?e :gender ?gender]
           [?e :money ?money]]})

This gives me:

[{:e-noop :ashley, :money-sum 3700, :gender :female}
 {:e-noop :mark, :money-sum 700, :gender :male}]
tsaixingwei

comment created time in 3 months

startedcldwalker/table

started time in 3 months

issue closedjuxt/crux

Extend `q` to also select meta DB attributes

https://clojurians.slack.com/archives/CG3AM2F7V/p1561214889099800

I found that oftentimes I want to know when a entity has been transacted, for example when I want to sort results by recency. Currently I'm adding a :entity/tx-time field manually. It would be great if I could simply query for tx-time, even valid-time when provided.

I also noticed that history returns a content-hash. For entity, this would be valuable to be able to tell whether and entity changed before transacting it.

I think besides :crux.tx/tx-id all other keys could come in handy and I wouldn't know what the harm would be in providing tx-id as well.

:crux.tx/tx-time
:crux.db/valid-time
:crux.db/content-hash
:crux.tx/tx-id

closed time in 3 months

den1k

issue commentjuxt/crux

Extend `q` to also select meta DB attributes

After much reflection I think it is essential that q must only be aware of its context as a single point-in-time, provided by the db coordinates. Giving an engine designed for point-in-time queries access to any underlying history (vt or tt) would undermine the design goal of providing consistent queries against valid time.

For example, a query run at a specific vt (and current tt) really needs to return the same results when run at a later tt, assuming that the set of valid documents is identical for both points in time - however, if your query is given information about the "start" valid time of a given document, then you will break the desired consistency if you retroactively put a new version of that document later than that start time but before the vt of the query.

Reifying the valid-time information inside the document is the right way to go, should you need it, and this approach also enables support for efficient range searches when querying (as per other domain-level #inst attributes). It may feel somewhat like duplication or a waste of storage, but in the scheme of things, for most use-cases, it's pretty minimal.

When we officially release transaction functions (resolving https://github.com/juxt/crux/issues/121) we'll add some example functions to make reifying tx-time more feasible too (vt is easy enough to do in user space already).

den1k

comment created time in 3 months

issue closedjuxt/crux

Add CORS functionality to edge as an independent library

Not simply extending https://github.com/juxt/edge/tree/master/lib/crux.ig but creating e.g. crux.ig.cors

closed time in 3 months

refset

issue commentjuxt/crux

Add CORS functionality to edge as an independent library

No longer relevant.

refset

comment created time in 3 months

issue commentjuxt/crux

HTTP/2 support for HTTP API

Thanks for opening this! I agree HTTP/2 is an important feature, and it is particularly relevant at the moment whilst we're working on various HTTP-related things.

It's probably worth noting that Vert.x is a strong contender.

We will update this with a more complete analysis & response when we find some time over the next couple of weeks :)

jorinvo

comment created time in 3 months

startedAntipurity/on-ai-article

started time in 3 months

startedAntipurity/on-ai-article

started time in 3 months

startedAntipurity/conceptual

started time in 3 months

fork refset/BootstrapLab

Visually bootstrap a self-sustaining system, and take it from there.

fork in 3 months

startedjdjakub/BootstrapLab

started time in 3 months

fork refset/fdat.cljc

Function serialization between Clojure processes and dialects

fork in 3 months

starteddvlopt/fdat.cljc

started time in 3 months

startedjuxt/tick

started time in 3 months

more