profile
viewpoint
If you are wondering where the data of this site comes from, please visit https://api.github.com/users/ept/events. GitMemory does not store any data, but only uses NGINX to cache data for a period of time. The idea behind GitMemory is simply to give users a better reading experience.
Martin Kleppmann ept University of Cambridge Cambridge, UK http://martin.kleppmann.com Distributed systems researcher at University of Cambridge; author of Designing Data-Intensive Applications; formerly Rapportive/LinkedIn

automerge/automerge 10329

A JSON-like data structure (a CRDT) that can be modified concurrently by different users, and merged again automatically.

ept/ddia-references 2782

Literature references for “Designing Data-Intensive Applications”

automerge/mpl 243

a p2p document synchronization system for automerge

ept/crdt-website 154

Source of the crdt.tech website

ept/avrodoc 137

Documentation tool for Avro schemas

ept/dist-sys 34

Distributed systems lecture notes

ept/cap-critique 16

Source of paper “A critique of the CAP theorem”

ept/blog 13

Source of my personal blog, using Markdown, Jekyll and Heroku

ept/byzantine-eventual 10

Byzantine Eventual Consistency

ept/debs-keynote 8

Keynote at the 15th ACM International Conference on Distributed and Event-Based Systems (DEBS)

push eventautomerge/automerge-rs

Andrew Jeffery

commit sha 9d6733716ce572849c570cf566a1df72b8f5b332

Change ActorId to be exact size array

view details

push time in an hour

PR opened automerge/automerge-rs

Reviewers
Change ActorId to be exact size array enhancement
+173 -172

0 comment

15 changed files

pr created time in an hour

create barnchautomerge/automerge-rs

branch : actorid-array

created branch time in an hour

push eventautomerge/automerge-rs

Andrew Jeffery

commit sha 7a282c8a668828f03db08775c960f636428ac13a

More preallocations

view details

push time in 2 hours

push eventautomerge/automerge-rs

Andrew Jeffery

commit sha ca5ebf29fb1cfa0be28bb539d8d64bf730c16a50

Preallocate ops in new_map_or_table

view details

push time in 2 hours

push eventautomerge/automerge-rs

Andrew Jeffery

commit sha 14269e11170ee727080baa8f983893d740768fbe

Remove result for incorporate_new_op

view details

push time in 2 hours

push eventautomerge/automerge-rs

Andrew Jeffery

commit sha fe83d938725d23a119bb0b314a47dded7b572809

Use swap remove instead of remove to avoid O(n)

view details

Andrew Jeffery

commit sha 80ee73a5b29485a8793fcddc773c5b65f9f25b92

Use default compression level for better balance of performance

view details

push time in 3 hours

Pull request review commentautomerge/automerge-rs

Flatten objtype

 pub enum Diff { pub struct MapDiff {     pub object_id: ObjectId,     #[serde(rename = "type")]-    pub obj_type: MapType,+    pub map_type: MapType,

Pushed

jeffa5

comment created time in 4 hours

push eventautomerge/automerge-rs

Andrew Jeffery

commit sha 43d32dc33de1cdfd9e47f7252716e06328becfcd

Remove maptype and seqtype from Diffs

view details

push time in 4 hours

Pull request review commentautomerge/automerge-rs

Flatten objtype

 pub enum Diff { pub struct MapDiff {     pub object_id: ObjectId,     #[serde(rename = "type")]-    pub obj_type: MapType,+    pub map_type: MapType,

Ah, I was thinking about doing this but stopped short. I'll follow through with this now.

jeffa5

comment created time in 4 hours

PR opened automerge/automerge-rs

Backend rearrangements

Using more preallocation where we know the length of the collection. Also using Cow for operation_key to avoid always cloning it.

+60 -39

0 comment

5 changed files

pr created time in 5 hours

create barnchautomerge/automerge-rs

branch : backend-perf

created branch time in 5 hours

fork tejashtarun14/ddia-references

Literature references for “Designing Data-Intensive Applications”

fork in 5 hours

Pull request review commentautomerge/automerge-rs

Flatten objtype

 pub enum Diff { pub struct MapDiff {     pub object_id: ObjectId,     #[serde(rename = "type")]-    pub obj_type: MapType,+    pub map_type: MapType,

Rather than having these map_type and seq_type tags, would it be worth also splitting the diffs up. E.g MapDiff, TableDiff, ListDiff, TextDiff?

jeffa5

comment created time in 6 hours

PR opened automerge/automerge-rs

Reviewers
Add SortedVec struct to ensure preds are sorted enhancement
+461 -411

0 comment

20 changed files

pr created time in 6 hours

create barnchautomerge/automerge-rs

branch : typed-sortedvec

created branch time in 6 hours

issue openedautomerge/automerge

Sequence replace: Automerge vs. other DDSes

Hello. I am testing some of the corner case resolutions of various DDSes and Automerge seems to behave unexpectedly in this specific case:

it('Insert at index vs replace [index]', () => { // <= This test passes
	s1 = Automerge.change(Automerge.init(), doc => doc.a = ['a'])
	s2 = Automerge.merge(Automerge.init(), s1)
	s1 = Automerge.change(s1, doc => {
			doc.a.deleteAt(0, 1)
			doc.a.insertAt(0, '1')
		}
	)
	s2 = Automerge.change(s2, doc => doc.a.insertAt(0, '2'))
	s1 = Automerge.merge(s1, s2)
	assert.deepStrictEqual(s1.a.join(''), '12') // Would have expected either '21' or a non-deterministic output.
})

Other DDSes I have tried this on either deterministically produce the merge state ['2', '1'] (that's the case for Fluid sequence and text-unicode (for both "left" and "right" tie-braking)) or mark the edits as conflicted (that's the case for OT-JSON1).

Is the outcome in Automerge in fact non-deterministic and I just haven't managed to trigger the other outcome (is there are reliable way to do that?)?

Am I wrong in expecting the outcome to be ['2', '1'] for Automerge?

Thank you!

created time in a day

startedept/ddia-references

started time in a day

delete branch automerge/automerge-rs

delete branch : mutation-tracker-rollback

delete time in a day

push eventautomerge/automerge-rs

Andrew Jeffery

commit sha 98dbd6150e05966a32c47c8a7c9d539be724ec54

Mutation tracker rollback (#165) * Make set and delete operations return the old value * Add rollback to MutableDocument * Use rollback in optimistically_apply_change

view details

push time in a day

PR merged automerge/automerge-rs

Mutation tracker rollback enhancement performance

This implements rollback on a MutableDocument. This is useful to handle when the user applies a change to the document that causes an error (e.g. missing index in insert). This mainly avoids us having to clone the current state each time and lets us make the hot path quicker and cheaper.

Thankfully when we overwrite a value in the document (map or seq), including deletes, we get the old value back so we can get away with not having to clone things and this ends up just delaying the drop call until we have finished with all of our changes.

tldr: cloning a large document for each change is expensive, tracking the individual changes for rollback is much cheaper

+583 -142

1 comment

6 changed files

jeffa5

pr closed time in a day

Pull request review commentautomerge/automerge-rs

Mutation tracker rollback

 impl LocalChange {     } } +// A container for either a multivalue or a multigrapheme+enum MultiThing {+    Value(MultiValue),+    Grapheme(MultiGrapheme),+}++impl MultiThing {+    fn into_value(self) -> MultiValue {+        match self {+            Self::Value(value) => value,+            Self::Grapheme(_) => unreachable!(),+        }+    }++    fn into_grapheme(self) -> MultiGrapheme {+        match self {+            Self::Value(_) => unreachable!(),+            Self::Grapheme(grapheme) => grapheme,+        }+    }+}++enum LocalOperationForRollback {

Ah yes, that's a much better idea! (I've also added a SetList as it expects the old value to always exist unlike in a map so avoids an unwrap).

jeffa5

comment created time in a day

Pull request review commentautomerge/automerge-rs

Mutation tracker rollback

 impl LocalChange {     } } +// A container for either a multivalue or a multigrapheme+enum MultiThing {+    Value(MultiValue),+    Grapheme(MultiGrapheme),+}++impl MultiThing {+    fn into_value(self) -> MultiValue {+        match self {+            Self::Value(value) => value,+            Self::Grapheme(_) => unreachable!(),

I've now removed MultiThing

jeffa5

comment created time in a day

push eventautomerge/automerge-rs

Andrew Jeffery

commit sha 75d5633c3aacf281e3ee4380b59eb100ed813c66

Remove MultiThing

view details

push time in a day

startedept/ddia-references

started time in a day

Pull request review commentautomerge/automerge-rs

Mutation tracker rollback

 impl LocalChange {     } } +// A container for either a multivalue or a multigrapheme+enum MultiThing {+    Value(MultiValue),+    Grapheme(MultiGrapheme),+}++impl MultiThing {+    fn into_value(self) -> MultiValue {+        match self {+            Self::Value(value) => value,+            Self::Grapheme(_) => unreachable!(),+        }+    }++    fn into_grapheme(self) -> MultiGrapheme {+        match self {+            Self::Value(_) => unreachable!(),+            Self::Grapheme(grapheme) => grapheme,+        }+    }+}++enum LocalOperationForRollback {

We could avoid the MultiThing wrapper by making the operations specific to the data type right? i.e Set for everything which needs a MultiValue and SetText for anything which needs a MultiGrapheme?

jeffa5

comment created time in a day

Pull request review commentautomerge/automerge-rs

Mutation tracker rollback

 impl LocalChange {     } } +// A container for either a multivalue or a multigrapheme+enum MultiThing {+    Value(MultiValue),+    Grapheme(MultiGrapheme),+}++impl MultiThing {+    fn into_value(self) -> MultiValue {+        match self {+            Self::Value(value) => value,+            Self::Grapheme(_) => unreachable!(),

Rather than using unreachable it might make more sense to return Option<MultiValue> and then expect at the use site, that way it's a bit more obvious to the type system that something could panic and we get slightly more informative panic messages.

jeffa5

comment created time in a day

push eventautomerge/automerge-rs

Andrew Jeffery

commit sha f224940fdcde5c3cc2379ffb0ccfa97c5e0c2e43

Fix Frontend debug struct name (#172)

view details

dependabot[bot]

commit sha 599c619a4deec4a6be5d220feb7650e2995758b7

Bump glob-parent from 5.1.1 to 5.1.2 in /automerge-backend-wasm (#166) Bumps [glob-parent](https://github.com/gulpjs/glob-parent) from 5.1.1 to 5.1.2. - [Release notes](https://github.com/gulpjs/glob-parent/releases) - [Changelog](https://github.com/gulpjs/glob-parent/blob/main/CHANGELOG.md) - [Commits](https://github.com/gulpjs/glob-parent/compare/v5.1.1...v5.1.2) --- updated-dependencies: - dependency-name: glob-parent dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

view details

Andrew Jeffery

commit sha 4821cdd766f6250cce86fc1c38886851d61aaf91

Calculate pred for deletion from text before deleting (#174)

view details

Andrew Jeffery

commit sha 8b3938c2e77f911a7328eb78b349c76c19b7b478

Rename UncompressedChange to Change (#173) It wasn't really uncompressed as we have compressed and uncompressed changes in the backend. It is just not encoded into the binary format. The module separation (protocol vs backend) should help with the distinction.

view details

Andrew Jeffery

commit sha 79239cc6a493b4a95cf0b042e8f5b764b3711fdb

Improve load performance when no cursors are present (#171) * Add save and load to benchmarks * Improve load performance when no cursors are found

view details

Andrew Jeffery

commit sha 0f0e9e827ad704a276143555a36be2a59c54ec66

New value zero copy multivalue (#170) * Remove copying when making a multivalue from a newvalue * Optimise cursors in new value * Make key be a value not reference * Rename union to extend

view details

Andrew Jeffery

commit sha c3cf09578289b5dd53a29f930c493a54519b353c

Make set and delete operations return the old value

view details

Andrew Jeffery

commit sha e3f575a22c776ed52cf6dffebfbc2b0740f8e8e5

Add rollback to MutableDocument

view details

Andrew Jeffery

commit sha 477d50e38ee587739b3d733284feaddda0e2cbf4

Use rollback in optimistically_apply_change

view details

Andrew Jeffery

commit sha 8be1148cccd6d693dd38029bc54fe9c61df20426

Update expects

view details

push time in a day

delete branch automerge/automerge-rs

delete branch : new-value-zero-copy-multivalue

delete time in a day