profile
viewpoint
Jane Lusby yaahc The Zcash Foundation San Francisco https://yaah.dev Rust, Linux, distributed systems, storage, networking, error handling

rust-lang/rust-clippy 5049

A bunch of lints to catch common mistakes and improve your Rust code

tokio-rs/tracing 979

Application level tracing for Rust.

RustBeginners/awesome-rust-mentors 345

A list of rust lang mentors tagged by topics of interest

hawkw/mycelium 16

🍄

yaahc/adhocerr 4

Crate for the quick creation of anonymous error types from strings

iykekings/deno_doctest 3

Proposal on how deno doctest will work, inspired by rustdoc --test

yaahc/btparse 3

A minimal deserialization library for std::backtrace::Backtrace's debug format

hdevalence/tcn-backend-simulator 1

experimental in-memory TCN protocol backend and simulator

issue openedyaahc/eyre

Make EyreHandlers composible

Right now eyre's design essentially requires that there's a single "Handler" type that is globally used in the application. Every error Report that is created will use the globally installed constructor for this handler type to create a new handler to go with the report to store additional context and print the error report + context. This is then meant to be paired with handler().downcast::<MyExpectedHandlerType>() when attempting to insert additional context into a Report as it propagates up the stack.

The problem is, if you are a library author who wishes to leverage eyre to hook additional functionality into error reporting you end up mandating that your users use your expected error report handler type, and if they ever install a different one all your downcasts will fail.

For a while my main plan for solving this problem has been to leverage https://github.com/mystor/object-provider to allow data to flow both ways across the dyn trait boundary without needing to downcast. This at least lets additional context be stored, so long as your users install a handler that will carry and provide your context the way you want them too, but this seems onerous and easy to forget to do.

I was looking at https://github.com/kellpossible/sentry-eyre today and trying to see what problems they were trying to solve and might run into and it dawned on me that what they might want is the ability to capture and send error report to sentry whenever they're printed, regardless of where the printing happens. The act of reporting it to a log or to a user would notify the back end.

That plus another PR that was opened today, https://github.com/yaahc/color-eyre/pull/67, which enables composition of panic handlers, made me realize that this same approach might be very useful for EyreHandler as well. If it was possible to take the Box<dyn EyreHandler> out of a report and replace it with another one we could effectively allow libraries to tap into the error handler behind a report to add additional context and functionality to reporters.

Problems

So now I need to figure out how or if this can be done. The first issue that comes to mind is losing the ability to downcast back to the original type when you wrap it, thus breaking existing integrations such as .suggestion as provided by color-eyre. It might be possible to work around this by doing some shenanigans in the downcast impls in eyre, similar to how it already supports downcasting to two alternate types for errors constructed with wrap_err.

Another alternative is to build the composition into the trait itself by adding an inner() method or something, essentially an equivalent to source() on the Error trait. Though I think I'd rather leverage the object-provider crate for this, which lets us solve more than one problem at a time.

Yet another alternative is to allow the Report type to store multiple handlers, which are called sequentially during reporting. This would then probably require some changes to the external interface of eyre so that it can downcast to the correct types but it also seems like it is probably the most intuitive approach to use. I'd likely opt to deprecate the existing handler interface which I doubt anyone uses, rather than making a breaking change, since I already know from experience that that causes immediate issues to be filed in this repo, lol. Report does not match Report is not a fun error.

created time in a day

pull request commentrust-lang/rust-clippy

lints: clarify rc_buffer and add caveats

looks great,

@bors r+

Fishrock123

comment created time in a day

PullRequestReviewEvent

startedkellpossible/sentry-eyre

started time in a day

CommitCommentEvent

push eventyaahc/rust

Jane Lusby

commit sha b3eda0701462b8334147c47f500c8ab01b2e5e46

cyclesss

view details

push time in a day

create barnchyaahc/rust

branch : typeof-errors

created branch time in a day

pull request commentrust-lang/rust

try enabling typeof for fun error messages

r? @eddyb

yaahc

comment created time in a day

PR opened rust-lang/rust

try enabling typeof for fun error messages
+7 -3

0 comment

2 changed files

pr created time in a day

create barnchtokio-rs/tracing

branch : instrument

created branch time in a day

PR opened tokio-rs/tracing

Reviewers
make fields in instrument opt in

<!-- Thank you for your Pull Request. Please provide a description above and review the requirements below.

Bug fixes and new features should include tests.

Contributors guide: https://github.com/tokio-rs/tracing/blob/master/CONTRIBUTING.md -->

Motivation

<!-- Explain the context and why you're making that change. What is the problem you're trying to solve? If a new feature is being added, describe the intended use case that feature fulfills. -->

Solution

<!-- Summarize the solution and provide any necessary context needed to understand the code change. -->

+4 -64

0 comment

2 changed files

pr created time in a day

issue commentyaahc/eyre

eyre Result is not cmp

we might be able to expose some sort of ptr equality? I don't know. Seems fraught.

gilescope

comment created time in a day

issue commentyaahc/color-eyre

Direct access to panic handler (to combine it with other panic handlers)

Sounds like a great idea :D

Some Suggestions:

  • Instead of exposing a method to convert into the Box<dyn Fn...> implement the Into trait
  • remove the install_panic_hook fn and add an install fn to both EyreHook and PanicHook that calls the relevant `set_hook(self.into());
  • remove installed_printer() entirely and copy the appropriate filters needed by Eyre when printing backtraces into the eyre hook
kellpossible

comment created time in a day

issue commentrust-lang/project-error-handling

Prototype `core::error::Error` to prove that stabilizing `fn backtrace` will not prevent a later move of the error trait to core

I'm still unconvinced that hooks are the best solution, here is my analysis of the trade offs.

  • Both traits and hooks are essentially ways to define a vtable
  • Traits carry the vtable as part of the object via a wide pointer
  • hooks store the vtable in a single static location
  • With traits you can have many implementations of the vtable in the same binary
  • with hooks you can must have exactly one implementation, unless you never attempt to call the hooks

To my pattern matching brain traits feel like a universally quantified form of dynamic dispatch whereas these ad-hoc hooks feel like an existentially quantified form of dynamic dispatch.

The question is, which is more appropriate?

Previously @withoutboats raised concerns about stability when using traits. I don't think this is a fundamental issue though. We could trivially wrap the trait in an opaque type and make the trait itself unstable, which is essentially what we're doing with the hooks. Either way if the user is ever allowed to provide the vtable, whether via traits or hooks, adding new items to that vtable will require handling the default case.

yaahc

comment created time in a day

issue commentrust-lang/project-error-handling

Move `std::io::Error` to core.

The type io::Error contains a Box inside of it, regardless of what traits it implements: https://github.com/rust-lang/rust/blob/master/library/std/src/io/error.rs#L58-L73

Wondering if this could maybee be worked around the way panic does with BoxMeUp... Seems sketch tho. I don't think I want to focus on this short term over https://github.com/rust-lang/project-error-handling/issues/3 which completely blocks this issue already.

indolering

comment created time in 2 days

issue commentrust-lang/project-error-handling

Prototype `core::error::Error` to prove that stabilizing `fn backtrace` will not prevent a later move of the error trait to core

I'm now trying to prototype the actual backtrace hook in core implementation to prove that all works and is something we could merge. I'm not really sure what I'm doing so I'm going to write out what I understand as the necessary steps so that more people more experienced with working on the language can give feedback and guidance:

  • [ ] Create a Backtrace type in core with the same interface as the current Backtrace type but without the implementation.
    • [ ] a function to conditionally construct a backtrace
    • [ ] a function to force construction of a backtrace
    • [ ] a function to access a backtrace status
  • [ ] Add hooks that
yaahc

comment created time in 2 days

issue commentrust-lang/project-error-handling

Move `std::io::Error` to core.

@indolering are you looking to also avoid having it bring in a dependency on alloc? std::io::Error has a lot of API dependencies on Box.

indolering

comment created time in 2 days

push eventZcashFoundation/zebra

Jane Lusby

commit sha 85fe56b55a432adfe11cd8592b3d285267703358

fix crashes at least

view details

push time in 2 days

push eventZcashFoundation/zebra

Jane Lusby

commit sha 4d3351f6a7fe5019ee8a96c812af066c5c0d8d5b

fix backwards asserts

view details

push time in 2 days

push eventZcashFoundation/zebra

Jane Lusby

commit sha 694e46e2f0ec2a6d49df1a7639aa70bfe3f16e99

clean a little more

view details

push time in 2 days

push eventZcashFoundation/zebra

Jane Lusby

commit sha 1d40aeedd1ace41b7f8bfc3e4f1bf59909b6ea45

cleanup prune impl

view details

push time in 2 days

push eventZcashFoundation/zebra

Jane Lusby

commit sha 4bcd6d87ab4dbae12d2df0240245e8105c079235

implement queued block pruning

view details

push time in 2 days

push eventZcashFoundation/zebra

Jane Lusby

commit sha cb1c85334998808837ecec0f4052afdcddf02916

for the diff

view details

push time in 2 days

push eventZcashFoundation/zebra

Jane Lusby

commit sha 854ec360495483faaec0dd6b0fbcfe0ba7389caf

Start updating RFC to match changes

view details

push time in 2 days

push eventZcashFoundation/zebra

Jane Lusby

commit sha 888c7806f534f5fae475f980c49eecf8102c3e15

propagate back results over channel

view details

push time in 2 days

create barnchZcashFoundation/zebra

branch : chainset-impl

created branch time in 2 days

PR opened ZcashFoundation/zebra

Implement the remaining portion of RFC0005: State Updates

<!-- Thank you for your Pull Request. Please provide a description above and review the requirements below.

Bug fixes and new features should include tests.

Contributors guide: https://zebra.zfnd.org/CONTRIBUTING.html -->

Motivation

<!-- Explain the context and why you're making that change. What is the problem you're trying to solve? In some cases there is not a problem and this can be thought of as being the motivation for your change. -->

Solution

<!-- Summarize the solution and provide any necessary context needed to understand the code change. -->

Related Issues

<!-- Please link to any existing GitHub issues pertaining to this PR. -->

+693 -182

0 comment

20 changed files

pr created time in 2 days

Pull request review commentZcashFoundation/zebra

Sapling note commitment tree

 impl fmt::Debug for Root {     } } -impl From<SaplingNoteCommitmentTree> for Root {-    fn from(_tree: SaplingNoteCommitmentTree) -> Self {-        // TODO: The Sapling note commitment tree requires a Pedersen-        // hash function, not SHA256.--        // let mut hash_writer = Sha256dWriter::default();-        // sapling_note_commitment_tree-        //     .zcash_serialize(&mut hash_writer)-        //     .expect("A Sapling note commitment tree must serialize.");-        // Self(hash_writer.finish())+/// Sapling Note Commitment Tree+#[derive(Clone, Debug, Default, Eq, PartialEq)]+#[cfg_attr(any(test, feature = "proptest-impl"), derive(Arbitrary))]+struct NoteCommitmentTree {+    /// The root node of the tree (often used as an anchor).+    root: Root,+    /// The height of the tree (maximum height for Sapling is 32).+    height: u8,+    /// The number of leaves (note commitments) in this tree.+    count: u32,+} +impl From<Vec<NoteCommitment>> for NoteCommitmentTree {+    fn from(_values: Vec<NoteCommitment>) -> Self {         unimplemented!();     } } -impl SaplingNoteCommitmentTree {-    /// Get the Jubjub-based Pedersen hash of root node of this merkle-    /// tree of commitment notes.-    pub fn hash(&self) -> [u8; 32] {-        unimplemented!();+impl From<Vec<jubjub::Fq>> for NoteCommitmentTree {+    fn from(values: Vec<jubjub::Fq>) -> Self {+        if values.is_empty() {+            return NoteCommitmentTree {+                root: Root::default(),+                height: 0,+                count: 0,+            };+        }++        let count = values.len() as u32;+        let mut height = 0u8;+        let mut current_layer: Vec<[u8; 32]> =+            values.into_iter().map(|cm_u| cm_u.to_bytes()).collect();++        while usize::from(height) < MERKLE_DEPTH {+            let mut next_layer_up = vec![];++            while !current_layer.is_empty() {+                let left = current_layer.remove(0);+                let right;+                if current_layer.is_empty() {+                    right = EMPTY_ROOTS[height as usize];+                } else {+                    right = current_layer.remove(0);+                }

TIOLI: since arrays are copy if their elements are you should be able to write it like this without any performance costs:

                let right = current_layer
                    .get(0)
                    .cloned()
                    .unwrap_or_else(|| EMPTY_ROOTS[height as usize]);
dconnolly

comment created time in 2 days

Pull request review commentZcashFoundation/zebra

Sapling note commitment tree

 impl fmt::Debug for Root {     } } -impl From<SaplingNoteCommitmentTree> for Root {-    fn from(_tree: SaplingNoteCommitmentTree) -> Self {-        // TODO: The Sapling note commitment tree requires a Pedersen-        // hash function, not SHA256.--        // let mut hash_writer = Sha256dWriter::default();-        // sapling_note_commitment_tree-        //     .zcash_serialize(&mut hash_writer)-        //     .expect("A Sapling note commitment tree must serialize.");-        // Self(hash_writer.finish())+/// Sapling Note Commitment Tree+#[derive(Clone, Debug, Default, Eq, PartialEq)]+#[cfg_attr(any(test, feature = "proptest-impl"), derive(Arbitrary))]+struct NoteCommitmentTree {+    /// The root node of the tree (often used as an anchor).+    root: Root,+    /// The height of the tree (maximum height for Sapling is 32).+    height: u8,

We probably want to use a custom strategy here so we don't generate invalid heights

https://altsysrq.github.io/proptest-book/proptest-derive/modifiers.html#strategy

dconnolly

comment created time in 2 days

Pull request review commentZcashFoundation/zebra

Sapling note commitment tree

 impl fmt::Debug for Root {     } } -impl From<SaplingNoteCommitmentTree> for Root {-    fn from(_tree: SaplingNoteCommitmentTree) -> Self {-        // TODO: The Sapling note commitment tree requires a Pedersen-        // hash function, not SHA256.--        // let mut hash_writer = Sha256dWriter::default();-        // sapling_note_commitment_tree-        //     .zcash_serialize(&mut hash_writer)-        //     .expect("A Sapling note commitment tree must serialize.");-        // Self(hash_writer.finish())+/// Sapling Note Commitment Tree+#[derive(Clone, Debug, Default, Eq, PartialEq)]+#[cfg_attr(any(test, feature = "proptest-impl"), derive(Arbitrary))]+struct NoteCommitmentTree {+    /// The root node of the tree (often used as an anchor).+    root: Root,+    /// The height of the tree (maximum height for Sapling is 32).+    height: u8,+    /// The number of leaves (note commitments) in this tree.+    count: u32,+} +impl From<Vec<NoteCommitment>> for NoteCommitmentTree {+    fn from(_values: Vec<NoteCommitment>) -> Self {         unimplemented!();     } } -impl SaplingNoteCommitmentTree {-    /// Get the Jubjub-based Pedersen hash of root node of this merkle-    /// tree of commitment notes.-    pub fn hash(&self) -> [u8; 32] {-        unimplemented!();+impl From<Vec<jubjub::Fq>> for NoteCommitmentTree {+    fn from(values: Vec<jubjub::Fq>) -> Self {+        if values.is_empty() {+            return NoteCommitmentTree {+                root: Root::default(),+                height: 0,+                count: 0,+            };+        }++        let count = values.len() as u32;+        let mut height = 0u8;+        let mut current_layer: Vec<[u8; 32]> =+            values.into_iter().map(|cm_u| cm_u.to_bytes()).collect();++        while usize::from(height) < MERKLE_DEPTH {

it looks like height us used as a usize in a few places but as a u8 in only one. Is it worth changing this to usize by default?

dconnolly

comment created time in 2 days

PullRequestReviewEvent

Pull request review commentZcashFoundation/zebra

Sapling note commitment tree

 pub fn pedersen_hash_to_point(domain: [u8; 8], M: &BitVec<Lsb0, u8>) -> jubjub::  /// Pedersen Hash Function ///+/// This is technically returning 255 (l_MerkleSapling) bits, not 256.+///

should we new-type this to ensure we don't mix them up in the future?

dconnolly

comment created time in 2 days

PullRequestReviewEvent

issue commentrust-lang/project-error-handling

Prototype `core::error::Error` to prove that stabilizing `fn backtrace` will not prevent a later move of the error trait to core

Here's a link to a related zulip discussion thread: https://rust-lang.zulipchat.com/#narrow/stream/257204-project-error-handling/topic/Stabilizing.20Backtrace.20.2F.20Core.20Error.20Proof.20of.20Concept/near/211291775

Once we've talked through potential solutions to the orphan rule issue more I'll come back and summarize the potential steps forward in the top level issue.

yaahc

comment created time in 2 days

pull request commenttokio-rs/tracing

Implementing first draft of an env builder api

I'm tagging @davidbarsky and @yaahc as I imagine y'all might be interested in this. :)

I do love API design :D

b::level(Level::TRACE).child(
        "my_crate",
        b::level(Filter::Inherit)
            .child("foo", b::level(Level::WARN))
            .child("bar", b::level(Level::ERROR)),
    );```

the children being associated with the level is a little strange looking at first but good sounding the more I think about it.

the `b::level` seems a bit redundant, would it be possible to write the API like this instead?

```rust
Level::TRACE.child(
    "my_crate",
    Filter::Inherit
        .child("foo", Level::WARN)
        .child("bar", Level::ERROR)
);
spacekookie

comment created time in 3 days

PR opened rust-lang/highfive

remove yaahc from clippy review rotation

going to focus on the error handling project group for the time being

+2 -2

0 comment

1 changed file

pr created time in 3 days

push eventyaahc/highfive

Jane Lusby

commit sha 3399d926d94c2b7a692b30e6494317fc084f4d7a

remove yaahc from clippy review rotation

view details

push time in 3 days

fork yaahc/highfive

Github hooks to provide an encouraging atmosphere for new contributors

fork in 3 days

Pull request review commentZcashFoundation/zebra

Add tests for `Chain` implementation

 impl Arbitrary for Header {                     merkle_root: merkle_root_hash,                     root_bytes,                     time: Utc.timestamp(timestamp, 0),-                    difficulty_threshold,+                    difficulty_threshold: CompactDifficulty(545259519),

it's this but evaluated so it doesn't have to export the constants, will be replaced once we can construct a CompactDifficulty from an Expanded difficulty

https://github.com/ZcashFoundation/zebra/blob/main/zebra-chain/src/work/difficulty/tests.rs#L148

yaahc

comment created time in 3 days

PullRequestReviewEvent

Pull request review commentZcashFoundation/zebra

Add tests for `Chain` implementation

 impl Arbitrary for Header {                     merkle_root: merkle_root_hash,                     root_bytes,                     time: Utc.timestamp(timestamp, 0),-                    difficulty_threshold,+                    difficulty_threshold: CompactDifficulty(545259519),

ahahaha, one sec let me find where I calculated this

yaahc

comment created time in 3 days

PullRequestReviewEvent

issue commenttokio-rs/tracing

instrument macro: omit `self` by default

Assigning this to myself and starting to work on this now. Based on discussions with @hawkw I think we're going to try just skipping all fields by default and make logging of function parameters entirely opt in as of tracing 0.2.

carllerche

comment created time in 3 days

Pull request review commentZcashFoundation/zebra

Add tests for `Chain` implementation

 impl Ord for Chain {         }     } }++#[cfg(test)]+mod tests {+    use transaction::Transaction;++    use std::mem;++    use zebra_chain::serialization::ZcashDeserializeInto;+    use zebra_chain::{+        parameters::{Network, NetworkUpgrade},+        LedgerState,+    };+    use zebra_test::prelude::*;++    use self::assert_eq;+    use super::*;++    /// Helper trait for constructing "valid" looking chains of blocks+    trait FakeChainHelper {+        fn make_fake_child(&self) -> Arc<Block>;+    }++    impl FakeChainHelper for Block {+        fn make_fake_child(&self) -> Arc<Block> {+            let parent_hash = self.hash();+            let mut child = Block::clone(self);+            let mut transactions = mem::take(&mut child.transactions);+            let mut tx = transactions.remove(0);++            let input = match Arc::make_mut(&mut tx) {+                Transaction::V1 { inputs, .. } => &mut inputs[0],+                Transaction::V2 { inputs, .. } => &mut inputs[0],+                Transaction::V3 { inputs, .. } => &mut inputs[0],+                Transaction::V4 { inputs, .. } => &mut inputs[0],+            };++            match input {+                transparent::Input::Coinbase { height, .. } => height.0 += 1,+                _ => panic!("block must have a coinbase height to create a child"),+            }++            child.transactions.push(tx);+            child.header.previous_block_hash = parent_hash;++            Arc::new(child)+        }+    }++    #[test]+    fn construct_empty() {+        zebra_test::init();+        let _chain = Chain::default();+    }++    #[test]+    fn construct_single() -> Result<()> {+        zebra_test::init();+        let block = zebra_test::vectors::BLOCK_MAINNET_434873_BYTES.zcash_deserialize_into()?;++        let mut chain = Chain::default();+        chain.push(block);++        assert_eq!(1, chain.blocks.len());++        Ok(())+    }++    #[test]+    fn construct_many() -> Result<()> {+        zebra_test::init();++        let mut block: Arc<Block> =+            zebra_test::vectors::BLOCK_MAINNET_434873_BYTES.zcash_deserialize_into()?;+        let mut blocks = vec![];++        while blocks.len() < 100 {+            let next_block = block.make_fake_child();+            blocks.push(block);+            block = next_block;+        }++        let mut chain = Chain::default();++        for block in blocks {+            chain.push(block);+        }++        assert_eq!(100, chain.blocks.len());++        Ok(())+    }++    fn arbitrary_chain(height: block::Height) -> BoxedStrategy<Vec<Arc<Block>>> {+        Block::partial_chain_strategy(+            LedgerState {+                tip_height: height,+                is_coinbase: true,+                network: Network::Mainnet,+            },+            100,+        )+    }++    prop_compose! {+        fn arbitrary_chain_and_count()+            (chain in arbitrary_chain(NetworkUpgrade::Blossom.activation_height(Network::Mainnet).unwrap()))+            (count in 1..chain.len(), chain in Just(chain)) -> (NoDebug<Vec<Arc<Block>>>, usize)+        {+            (NoDebug(chain), count)+        }+    }++    #[test]+    fn forked_equals_pushed() -> Result<()> {+        zebra_test::init();++        proptest!(|((chain, count) in arbitrary_chain_and_count())| {

as these are currently designed we will almost certainly want to disable these tests by default and add another CI step that builds the application in --release to run these tests

yaahc

comment created time in 3 days

PullRequestReviewEvent

push eventZcashFoundation/zebra

dependabot[bot]

commit sha d5ce5eeee2a83b87268e8d99e4128004bfb5c155

build(deps): bump structopt from 0.3.17 to 0.3.18 Bumps [structopt](https://github.com/TeXitoi/structopt) from 0.3.17 to 0.3.18. - [Release notes](https://github.com/TeXitoi/structopt/releases) - [Changelog](https://github.com/TeXitoi/structopt/blob/master/CHANGELOG.md) - [Commits](https://github.com/TeXitoi/structopt/compare/v0.3.17...v0.3.18) Signed-off-by: dependabot[bot] <support@github.com>

view details

Jane Lusby

commit sha 352721bd887d30e845b093ee1d913ea862dc87e6

Implement RFC5: State updates `Chain` type (#1069) * Begin work on RFC5 implementation * I think this is necessary * holy shit supertrait implemented via subtrait * implement most of the chain functions * change to slightly better name * implement fork * fix outpoint handling in Chain struct * update expect for work * resolve review comment * split utxo into two sets * update the Chain definition * just a little more * update comment * Apply suggestions from code review Co-authored-by: teor <teor@riseup.net> * apply changes from code review * remove allow attribute in zebra-state/lib.rs * Update zebra-state/src/memory_state.rs Co-authored-by: teor <teor@riseup.net> * merge ChainSet type into MemoryState * rename state impl types * Add error messages to asserts * add module doc comment * update RFC for utxos * add missing header Co-authored-by: teor <teor@riseup.net>

view details

Jane Lusby

commit sha 48bc6138cc4964273b7735f00126fa17d6f8d799

Merge branch 'main' into state-rfc-impl-tests

view details

push time in 3 days

push eventZcashFoundation/zebra

Jane Lusby

commit sha 352721bd887d30e845b093ee1d913ea862dc87e6

Implement RFC5: State updates `Chain` type (#1069) * Begin work on RFC5 implementation * I think this is necessary * holy shit supertrait implemented via subtrait * implement most of the chain functions * change to slightly better name * implement fork * fix outpoint handling in Chain struct * update expect for work * resolve review comment * split utxo into two sets * update the Chain definition * just a little more * update comment * Apply suggestions from code review Co-authored-by: teor <teor@riseup.net> * apply changes from code review * remove allow attribute in zebra-state/lib.rs * Update zebra-state/src/memory_state.rs Co-authored-by: teor <teor@riseup.net> * merge ChainSet type into MemoryState * rename state impl types * Add error messages to asserts * add module doc comment * update RFC for utxos * add missing header Co-authored-by: teor <teor@riseup.net>

view details

push time in 3 days

Pull request review commentZcashFoundation/zebra

Implement RFC5: State updates `Chain` type

 if the block is not in any non-finalized chain:     the `block_by_height` tree (to get the block data).  -### TODO document request for utxo is available+### `Request::AwaitUtxo(OutPoint)` -utxo_is_available:-* is it in spent? Return false-* is it in created? Return true-* is it in the permanent state? Return true-* otherwise, Return false+Returns++- `Response::Utxo(transparent::Output)`++- (non-finalized) if any `Chains` contain an `OutPoint` in their `created_utxos` and not their `spent_utxo` get the `transparent::Output` from `OutPoint`'s transaction+- (finalized) else if `OutPoint` is in `utxos_by_outpoint` return the associated `transparent::Output`.

applied this in the test PR to avoid needing a new test and approval

yaahc

comment created time in 3 days

PullRequestReviewEvent

push eventZcashFoundation/zebra

Jane Lusby

commit sha 84de5d7da4c07e85edb9f77fe318aa70d41b6a06

add module doc comment

view details

Jane Lusby

commit sha 40e6035e8e99aee2913edcc3a6a096d404b9c4ba

update RFC for utxos

view details

Jane Lusby

commit sha c6002a6a0accdb148241832b55b25965dfbc2831

add missing header

view details

Jane Lusby

commit sha 14f48a32371cf0bced5681e56394fb2ecf2ea600

Merge branch 'state-rfc-impl' into state-rfc-impl-tests

view details

Jane Lusby

commit sha a1c19f41c148e3d46129e210a217e8d77bc4eb3b

apply change from chain impl PR

view details

push time in 3 days

push eventZcashFoundation/zebra

Jane Lusby

commit sha 57be0d3fde960a48f3de9a997f41f9e4d8081298

working proptest for Chain

view details

push time in 3 days

Pull request review commentZcashFoundation/zebra

State RFC Tests Only - Used for Review

 impl Ord for Chain {         }     } }++#[cfg(test)]+mod tests {+    use proptest::collection::vec;+    use transaction::Transaction;++    use std::mem;++    use zebra_chain::parameters::{Network, NetworkUpgrade};+    use zebra_chain::serialization::ZcashDeserializeInto;+    use zebra_test::prelude::*;++    use self::assert_eq;+    use super::*;++    /// Helper trait for constructing "valid" looking chains of blocks+    trait FakeChainHelper {+        fn make_fake_child(&self) -> Arc<Block>;+    }++    impl FakeChainHelper for Block {+        fn make_fake_child(&self) -> Arc<Block> {+            let parent_hash = self.hash();+            let mut child = Block::clone(self);+            let mut transactions = mem::take(&mut child.transactions);+            let mut tx = transactions.remove(0);++            let input = match Arc::make_mut(&mut tx) {+                Transaction::V1 { inputs, .. } => &mut inputs[0],+                Transaction::V2 { inputs, .. } => &mut inputs[0],+                Transaction::V3 { inputs, .. } => &mut inputs[0],+                Transaction::V4 { inputs, .. } => &mut inputs[0],+            };++            match input {+                transparent::Input::Coinbase { height, .. } => height.0 += 1,+                _ => panic!("block must have a coinbase height to create a child"),+            }++            child.transactions.push(tx);+            child.header.previous_block_hash = parent_hash;++            Arc::new(child)+        }+    }++    #[test]+    fn construct_empty() {+        zebra_test::init();+        let _chain = Chain::default();+    }++    #[test]+    fn construct_single() -> Result<()> {+        zebra_test::init();+        let block = zebra_test::vectors::BLOCK_MAINNET_434873_BYTES.zcash_deserialize_into()?;++        let mut chain = Chain::default();+        chain.push(block);++        assert_eq!(1, chain.blocks.len());++        Ok(())+    }++    #[test]+    fn construct_many() -> Result<()> {+        zebra_test::init();++        let mut block: Arc<Block> =+            zebra_test::vectors::BLOCK_MAINNET_434873_BYTES.zcash_deserialize_into()?;

I don't think we should worry about the genesis block, this is only supposed to ever handle non-finalized data, so the genesis block should never pass through this data structure, or even any pre sapling blocks.

teor2345

comment created time in 4 days

PullRequestReviewEvent

push eventZcashFoundation/zebra

Jane Lusby

commit sha c6002a6a0accdb148241832b55b25965dfbc2831

add missing header

view details

push time in 4 days

push eventZcashFoundation/zebra

Jane Lusby

commit sha 40e6035e8e99aee2913edcc3a6a096d404b9c4ba

update RFC for utxos

view details

push time in 4 days

push eventZcashFoundation/zebra

Jane Lusby

commit sha 84de5d7da4c07e85edb9f77fe318aa70d41b6a06

add module doc comment

view details

push time in 4 days

push eventZcashFoundation/zebra

Jane Lusby

commit sha 7227a4a7dfa8bd679220a268def5b2371ba7dd9e

Merge branch 'main' into state-rfc-impl

view details

Jane Lusby

commit sha 6af09c760a756a2b401da81a2a9a51ecc739f68d

Merge branch 'state-rfc-impl' into state-rfc-impl-tests

view details

push time in 4 days

push eventZcashFoundation/zebra

Jane Lusby

commit sha 0b4e974c9e7b24591f97e04fc937e96862e144c6

export proptest impls for use in downstream crates (#1092) * export proptest impls for use in downstream crates * add testjob for disabled feature in zebra-chain * run rustfmt * try to fix github actions syntax * differentiate name * prove that github action tests zebra-chain build without features * revert change from last commit now that test is running * remove accidentally introduced newline * Update .github/workflows/ci.yml Co-authored-by: Deirdre Connolly <deirdre@zfnd.org> Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>

view details

Jane Lusby

commit sha 7227a4a7dfa8bd679220a268def5b2371ba7dd9e

Merge branch 'main' into state-rfc-impl

view details

push time in 4 days

push eventZcashFoundation/zebra

Jane Lusby

commit sha 0b4e974c9e7b24591f97e04fc937e96862e144c6

export proptest impls for use in downstream crates (#1092) * export proptest impls for use in downstream crates * add testjob for disabled feature in zebra-chain * run rustfmt * try to fix github actions syntax * differentiate name * prove that github action tests zebra-chain build without features * revert change from last commit now that test is running * remove accidentally introduced newline * Update .github/workflows/ci.yml Co-authored-by: Deirdre Connolly <deirdre@zfnd.org> Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>

view details

Jane Lusby

commit sha cd9b10d1c40f9ea70add43513ed4666762b475b3

Merge branch 'main' into state-rfc-impl-tests

view details

push time in 4 days

push eventZcashFoundation/zebra

Jane Lusby

commit sha 0b4e974c9e7b24591f97e04fc937e96862e144c6

export proptest impls for use in downstream crates (#1092) * export proptest impls for use in downstream crates * add testjob for disabled feature in zebra-chain * run rustfmt * try to fix github actions syntax * differentiate name * prove that github action tests zebra-chain build without features * revert change from last commit now that test is running * remove accidentally introduced newline * Update .github/workflows/ci.yml Co-authored-by: Deirdre Connolly <deirdre@zfnd.org> Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>

view details

push time in 4 days

delete branch ZcashFoundation/zebra

delete branch : proptest-impl

delete time in 4 days

PR opened ZcashFoundation/zebra

Add tests for `Chain` implementation

<!-- Thank you for your Pull Request. Please provide a description above and review the requirements below.

Bug fixes and new features should include tests.

Contributors guide: https://zebra.zfnd.org/CONTRIBUTING.html -->

Motivation

<!-- Explain the context and why you're making that change. What is the problem you're trying to solve? In some cases there is not a problem and this can be thought of as being the motivation for your change. -->

Solution

<!-- Summarize the solution and provide any necessary context needed to understand the code change. -->

Related Issues

<!-- Please link to any existing GitHub issues pertaining to this PR. -->

+998 -207

0 comment

53 changed files

pr created time in 4 days

create barnchZcashFoundation/zebra

branch : state-rfc-impl-tests

created branch time in 4 days

push eventZcashFoundation/zebra

push time in 4 days

pull request commentZcashFoundation/zebra

Implement RFC5: State updates

@teor2345 https://discordapp.com/channels/676527656170160146/716086297210650634/758474653207101480

yaahc

comment created time in 4 days

push eventZcashFoundation/zebra

Jane Lusby

commit sha f345570f77230233efef047ea987d2e5c6f2e637

Update .github/workflows/ci.yml Co-authored-by: Deirdre Connolly <deirdre@zfnd.org>

view details

push time in 4 days

push eventZcashFoundation/zebra

Jane Lusby

commit sha 2708f070aeef53362b93f5aec4e96bcae613bfde

checkpoint so I can split off arbitrary changes into a PR

view details

Jane Lusby

commit sha c021e80feef263d7869563347c181f742f50b12e

export proptest impls for use in downstream crates

view details

Jane Lusby

commit sha ba1549a72094a6320a341edaca31de0f0c524798

add testjob for disabled feature in zebra-chain

view details

Jane Lusby

commit sha 18105f93158df3095df7ed733ae93b0eecadb8be

run rustfmt

view details

Jane Lusby

commit sha 1b887973acefae50b8edfed6a8270f40e0de5137

try to fix github actions syntax

view details

Jane Lusby

commit sha 1ac35b6ab26839f2d0980dadac244a2473590b94

differentiate name

view details

Jane Lusby

commit sha 5a4db20dae73d8ab4896152dd0558b8dc5ecd1b7

prove that github action tests zebra-chain build without features

view details

Jane Lusby

commit sha 4a7da3abd1c7b4bca903ebc275737aebf3d24e2b

revert change from last commit now that test is running

view details

Jane Lusby

commit sha 4fbc7a51ecf83ad7fc1c11f28266b1b41e373754

remove accidentally introduced newline

view details

Jane Lusby

commit sha 452a241787496eb2d1282bd1040df4211dddda99

Merge branch 'proptest-impl' into state-rfc-impl

view details

push time in 4 days

push eventZcashFoundation/zebra

Jane Lusby

commit sha 4fbc7a51ecf83ad7fc1c11f28266b1b41e373754

remove accidentally introduced newline

view details

push time in 4 days

push eventZcashFoundation/zebra

Jane Lusby

commit sha 4a7da3abd1c7b4bca903ebc275737aebf3d24e2b

revert change from last commit now that test is running

view details

push time in 4 days

push eventZcashFoundation/zebra

Jane Lusby

commit sha 5a4db20dae73d8ab4896152dd0558b8dc5ecd1b7

prove that github action tests zebra-chain build without features

view details

push time in 4 days

push eventZcashFoundation/zebra

Jane Lusby

commit sha 1ac35b6ab26839f2d0980dadac244a2473590b94

differentiate name

view details

push time in 4 days

push eventZcashFoundation/zebra

Jane Lusby

commit sha 1b887973acefae50b8edfed6a8270f40e0de5137

try to fix github actions syntax

view details

push time in 4 days

push eventZcashFoundation/zebra

Jane Lusby

commit sha 18105f93158df3095df7ed733ae93b0eecadb8be

run rustfmt

view details

push time in 4 days

push eventZcashFoundation/zebra

Jane Lusby

commit sha ba1549a72094a6320a341edaca31de0f0c524798

add testjob for disabled feature in zebra-chain

view details

push time in 4 days

create barnchZcashFoundation/zebra

branch : proptest-impl

created branch time in 4 days

pull request commentZcashFoundation/zebra

Implement RFC5: State updates

I want to get the two proptests that henry mentioned implemented before merging, just having a bit of trouble figuring out how to make arbitrary fake chains :sweat_smile:

yaahc

comment created time in 4 days

issue openedrust-lang/project-error-handling

Research: Globally consistent error reporting formats

Problem

Right now there's no easy way to ensure that all errors are printed with a consistent error report format such as via eyre::Report + color-eyre. Errors that have not been converted to an error reporting type prior to panicking in functions like unwrap cannot then be correctly reported via the panic handler because the error is converted to a string and discarded. Even if it was not such as via panic!(error) directly you can only downcast to the exact type, so your error reporting hook would need an exhaustive list of all error types that it should try downcasting to, which is infeasible.

Possible Solutions

Carry original type in payloads for panics and add support for casting from &dyn Any to &dyn Error

One possible approach to this would be to adjust panics to include the data they were created with in the payload along with the formatted message and then add support for somehow extracting error trait objects back out of those &dyn Any playloads. Ideally the solution for extracting trait objects of other kinds from &dyn Any would not be specific to &dyn Error. If we could create a language / library level feature for casting between arbitrary trait objects or even only from &dyn Any to arbitrary trait objects that the underlying type also implements that would be ideal.

@oli-obk has suggested the possibility of using linker shenanigans to accomplish this:

linker tricks could make that actually feasible to implement like each type would allocate a named value and each impl block would append to that list not sure how we'd get the order right

@DianaNites has pointed out that the following libraries may be useful as sources of prior art

  • https://crates.io/crates/linkme
  • https://crates.io/crates/intertrait

created time in 4 days

push eventZcashFoundation/zebra

Jane Lusby

commit sha c4b8a4d7f310374f9e4a00b2621253fdbe8c3460

Add error messages to asserts

view details

push time in 4 days

issue openedrust-lang/project-error-handling

Research: Error Return Traces

I'd like to pursue the possibility of supporting error return traces in rust, based loosely on Ziglang's feature by the same name. I've written up some notes on this in the past in this Pre-RFC fact finding document and in This partial proof of concept written for this RFC.

created time in 4 days

pull request commentrust-lang/rust-clippy

Add `rc_buffer` lint for checking Rc<String> and friends

@bors r+

rschoon

comment created time in 4 days

Pull request review commentZcashFoundation/zebra

Implement RFC5: State updates

-pub struct MemoryState {-    // TODO+#![allow(dead_code)]+use std::{+    cmp::Ordering,+    collections::BTreeSet,+    collections::{BTreeMap, HashMap, HashSet},+    ops::Deref,+    sync::Arc,+};++use zebra_chain::{+    block::{self, Block},+    primitives::Groth16Proof,+    sapling, sprout, transaction, transparent,+    work::difficulty::PartialCumulativeWork,+};++use crate::service::QueuedBlock;++/// The state of the chains in memory, incuding queued blocks.+#[derive(Debug, Default)]+pub struct NonFinalizedState {

@teor2345 I applied your change to split up the queued blocks here but I went ahead and switch it and the SledState to different names. lmk what you think

yaahc

comment created time in 4 days

PullRequestReviewEvent

push eventZcashFoundation/zebra

Jane Lusby

commit sha 8a9b3194f23e3b7d0b7b3eb02f049e6f2397e299

rename state impl types

view details

push time in 4 days

Pull request review commentZcashFoundation/zebra

Implement RFC5: State updates

 handled by `#[derive(Default)]`.     all its blocks have been `pop`ped  -### `ChainSet` Type-[chainset-type]: #chainset-type+### `MemoryState` Type+[memorystate-type]: #memorystate-type -The `ChainSet` type represents the set of all non-finalized state. It+The `MemoryState` type represents the set of all non-finalized state. It consists of a set of non-finalized but verified chains and a set of unverified blocks which are waiting for the full context needed to verify them to become available. -`ChainSet` is defined by the following structure and API:+`MemoryState` is defined by the following structure and API:  ```rust-struct ChainSet {-    chains: BTreeSet<Chain>,+/// The state of the chains in memory, incuding queued blocks.+#[derive(Debug, Default)]+pub struct MemoryState {

@teor2345 I applied your change to split up the queued blocks here but I went ahead and switch it and the SledState to different names. lmk what you think

yaahc

comment created time in 4 days

PullRequestReviewEvent

push eventZcashFoundation/zebra

Jane Lusby

commit sha dec13465cb76243ff116b01b2307e3b881ae20b5

merge ChainSet type into MemoryState

view details

push time in 4 days

push eventZcashFoundation/zebra

Jane Lusby

commit sha b00ea96918dceb80e1326f2957f43175a84a479b

Update zebra-state/src/memory_state.rs Co-authored-by: teor <teor@riseup.net>

view details

push time in 4 days

push eventZcashFoundation/zebra

Jane Lusby

commit sha 563bc555cf2fc7977cd74b88d0c7551c4259a86c

remove allow attribute in zebra-state/lib.rs

view details

push time in 4 days

Pull request review commentZcashFoundation/zebra

Implement RFC5: State updates

+use std::{+    cmp::Ordering,+    collections::BTreeSet,+    collections::{BTreeMap, HashMap, HashSet},+    ops::Deref,+    sync::Arc,+};++use zebra_chain::{+    block::{self, Block},+    primitives::Groth16Proof,+    sapling, sprout, transaction, transparent,+    work::difficulty::PartialCumulativeWork,+    work::difficulty::Work,+};++use crate::{service::QueuedBlock, BoxError};+ pub struct MemoryState {     // TODO }++struct ChainSet {+    chains: BTreeSet<Chain>,++    queued_blocks: BTreeMap<block::Hash, QueuedBlock>,+    queued_by_parent: BTreeMap<block::Hash, Vec<block::Hash>>,+    queued_by_height: BTreeMap<block::Height, Vec<block::Hash>>,+}++impl ChainSet {+    pub fn finalize(&mut self) -> Arc<Block> {+        todo!()+    }++    pub fn queue(&mut self, block: QueuedBlock) {+        todo!()+    }++    fn process_queued(&mut self, new_parent: block::Hash) {+        todo!()+    }++    fn commit_block(&mut self, block: QueuedBlock) -> Option<block::Hash> {+        todo!()+    }+}++#[derive(Debug, Default, Clone)]+struct Chain {+    blocks: BTreeMap<block::Height, Arc<Block>>,+    height_by_hash: HashMap<block::Hash, block::Height>,+    tx_by_hash: HashMap<transaction::Hash, (block::Height, usize)>,++    created_utxos: HashSet<transparent::OutPoint>,+    spent_utxos: HashSet<transparent::OutPoint>,+    sprout_anchors: HashSet<sprout::tree::Root>,+    sapling_anchors: HashSet<sapling::tree::Root>,+    sprout_nullifiers: HashSet<sprout::Nullifier>,+    sapling_nullifiers: HashSet<sapling::Nullifier>,+    partial_cumulative_work: PartialCumulativeWork,+}++impl Chain {

Sorry, I know I'm the worst about this, ty for the patient reminders :sweat_smile:

yaahc

comment created time in 4 days

Pull request review commentZcashFoundation/zebra

Implement RFC5: State updates

+use std::{

I'll draft this after I've written tests

yaahc

comment created time in 5 days

Pull request review commentZcashFoundation/zebra

Implement RFC5: State updates

 #![doc(html_root_url = "https://doc.zebra.zfnd.org/zebra_state")] #![warn(missing_docs)] #![allow(clippy::try_err)]+#![allow(unused_imports, unused_variables, dead_code)]

I'll add a checkbox to the original comment on this PR to remind me to remove these before we merge this.

yaahc

comment created time in 5 days

Pull request review commentZcashFoundation/zebra

Implement RFC5: State updates

 if the block is not in any non-finalized chain:     the `block_by_height` tree (to get the block data).  +### TODO document request for utxo is available++utxo_is_available:+* is it in spent? Return false+* is it in created? Return true+* is it in the permanent state? Return true+* otherwise, Return false

this is going to be an external API so we won't be using Chain as an input, probably block::Hash or the Arc<Block>? I'll check the async script RFC to make sure these are consistent.

yaahc

comment created time in 5 days

Pull request review commentZcashFoundation/zebra

Implement RFC5: State updates

 struct Chain { Push a block into a chain as the new tip  1. Update cumulative data members-    - Add block to end of `self.blocks`     - Add hash to `height_by_hash`-    - for each `transaction` in `block`-      - add key: `transaction.hash` and value: `(height, tx_index)` to `tx_by_hash`-    - Add new utxos and remove consumed utxos from `self.utxos`-    - Add anchors to the appropriate `self.<version>_anchors`-    - Add nullifiers to the appropriate `self.<version>_nullifiers`+    - For each `transaction` in `block`+      - Add key: `transaction.hash` and value: `(height, tx_index)` to `tx_by_hash`+      - Add created utxos to `self.created_utxos`+      - Add spent utxos to `self.spent_utxos`+      - Add anchors to the appropriate `self.<version>_anchors`+      - Add nullifiers to the appropriate `self.<version>_nullifiers`     - Add work to `self.partial_cumulative_work`+    - Add block to end of `self.blocks`

I think moving the work to the top is a good idea but I'd prefer the block to be at the bottom because of ownership issues, if we insert it into the list immediately we have to make a clone which isn't a big deal but it seemed easy to avoid. The way I structured the implementation removes the manipulation of the block from the manipulation of the rest of the cummulative data members, so maybe it would be good to have both the "update cummulative work" and "for each transaction" bits indented and within an "update cummulative members" section. I'll reorganize things based on this and you let me know what you think.

yaahc

comment created time in 5 days

Pull request review commentZcashFoundation/zebra

Implement RFC5: State updates

 pub enum Transaction {         expiry_height: block::Height,         /// The net value of Sapling spend transfers minus output transfers.         value_balance: Amount,-        /// The shielded data for this transaction, if any.-        shielded_data: Option<ShieldedData>,         /// The JoinSplit data for this transaction, if any.         joinsplit_data: Option<JoinSplitData<Groth16Proof>>,+        /// The shielded data for this transaction, if any.+        shielded_data: Option<ShieldedData>,

I talked to henry about doing this a while ago and just remembered to do it. The fact that shielded_data was above joinsplit_data despite being the newer member was confusing for me, I reordered this to reflect when they were added to the protocol.

yaahc

comment created time in 5 days

PullRequestReviewEvent
PullRequestReviewEvent

push eventZcashFoundation/zebra

Jane Lusby

commit sha fd9eca1e091aadc866c33b3d20ed29a73c9fe2cb

apply changes from code review

view details

push time in 4 days

more