profile
viewpoint
Max Inden mxinden https://max-inden.de Rust, Peer-to-Peer, Prometheus

mxinden/elimination-backoff-stack 1

Lock-free elimination back-off stack

mxinden/1k-validators-be 0

Thousand Validator Program backend services.

mxinden/a-chat 0

A simple chat implemented on top of async-std

mxinden/alertmanager 0

Prometheus Alertmanager

mxinden/am-ui 0

nothing to see here

PR opened libp2p/rust-libp2p

Prepare release of `libp2p-v0.31` and all unreleased sub-crates.

Prepare release of libp2p-v0.31 and all unreleased sub-crates.

+22 -22

0 comment

22 changed files

pr created time in an hour

push eventlibp2p/rust-libp2p

Roman Borschel

commit sha 65c4bf88dca8df114bf92fe069060a646bc2246b

[multistream] Make the lazy variant more interoperable. (#1855) * Make the lazy variant interoperable. The remaining optimisation for `V1Lazy` for a listener in the negotiation, whereby the listener delays flushing of the multistream version header, is hereby removed. The remaining effect of `V1Lazy` is only on the side of the dialer, which delays flushing of its singular protocol proposal in order to send it together with the first application data (or an attempt is made to read from the negotiated stream, which similarly triggers a flush of the protocol proposal). This permits `V1Lazy` dialers to be interoperable with `V1` listeners. The remaining theoretical pitfall whereby application data gets misinterpreted as another protocol proposal by a listener remains, however unlikely. `V1` remains the default, but we may eventually risk just making this lazy dialer flush a part of the default `V1` implementation, removing the dedicated `V1Lazy` version identifier. * Update CHANGELOG * Separate versions from mere header lines. Every multistream-select version maps to a specific header line, but there may be different variants of the same multistream-select version using the same header line, i.e. the same wire protocol. * Cleanup * Update misc/multistream-select/CHANGELOG.md

view details

push time in 2 hours

PR merged libp2p/rust-libp2p

[multistream] Make the lazy variant more interoperable.

Closes https://github.com/libp2p/rust-libp2p/issues/1853. The remaining optimisation for V1Lazy for a listener in the negotiation, whereby the listener delays flushing of the multistream version header, is hereby removed. The only remaining effect of V1Lazy as compared to V1 is on the side of the dialer, which delays flushing of its singular protocol proposal (i.e. if it has no alternatives) in order to send it together with the first application data, e.g. a request, or an attempt is made to read from the negotiated stream, which similarly triggers a flush of the pending protocol proposal, as before. This permits V1Lazy dialers to be interoperable with V1 listeners. The remaining theoretical pitfall whereby application data gets misinterpreted as another protocol proposal by a listener remains, however unlikely.

V1 remains the default, but after sufficient experience we may eventually be bold enough to make this lazy dialer flush a part of the default V1 implementation, removing the dedicated V1Lazy version identifier. As an experiment I temporarily changed the default upgrade strategy to V1Lazy for all substreams and at least all the tests and the ipfs-kad example worked fine.

+141 -126

0 comment

7 changed files

romanb

pr closed time in 2 hours

issue closedlibp2p/rust-libp2p

Change wire protocol name of the "multistream-lazy" variant

The rust-libp2p implementation of multistream-select, in addition to the standard V1 protocol version, also supports a non-standard V1Lazy variant. Link to the documentation.

While this variant removes a round-trip-time in many circumstances, its drawback is that it isn't compatible with the non-Rust versions of libp2p. For that reason, nobody uses it.

I suggest here that we:

  • Change the wire name of V1Lazy to be the same as V1, therefore making indistinguishable from the standard one.
  • Remove the fact that the listener delays the response, as it is, as far as I understand, the reason why this protocol isn't the same as the standard one. The listener should immediately send back the response to the protocol request, even though it might result in two packets.

closed time in 2 hours

tomaka

PR closed libp2p/rust-libp2p

Send only validated messages
+11783 -1281

0 comment

25 changed files

blacktemplar

pr closed time in 2 hours

PR opened libp2p/rust-libp2p

Send only validated messages
+11783 -1281

0 comment

25 changed files

pr created time in 2 hours

issue openedspacejam/sled

API for checkpoint / snapshot / backup system

Use Case:

Rocks equivalent:

  • https://github.com/facebook/rocksdb/wiki/Checkpoints
  • https://docs.rs/rocksdb/0.15.0/rocksdb/checkpoint/struct.Checkpoint.html

The primary use case here is exactly as described in the Rocks use case. Full & incremental backups. In my specific use case, I am building a distributed data storage system which uses async-raft for consensus, which includes a protocol where snapshots are sent to new nodes to bring them up-to-speed. Good stuff.

Proposed Change:

The API I am envisioning is as follows.

  • sled::DB gets a new method checkpoint(path: impl AsRef<Path>), which does the work of generating a new DB checkpoint which will be written to the given path.
  • Given the nature of checkpoints, it seems logical that this method only be exposed on the DB type and not on Trees (maintainers would definitely know better on this one).
  • Some discussion is merited around incremental checkpoints / backups and what patterns would be best for this.

Who Benefits From The Change(s)?

Anyone and everyone looking for full / incremental backups, and the various use cases which emerge from that capability.

Alternative Approaches

Instead of calling this new API method checkpoint, we could call it backup.

created time in 6 hours

PublicEvent

created repositorymjgallag/runhub-demo

created time in 13 hours

Pull request review commentlibp2p/rust-libp2p

[multistream] Make the lazy variant more interoperable.

 # 0.9.0 [unreleased] +- Make the `V1Lazy` upgrade strategy more interoperable with `V1`.

How about this:

- Make the `V1Lazy` upgrade strategy more interoperable with `V1`. Specifically,
  the listener now behaves identically with `V1` and `V1Lazy`. Furthermore, the
  multistream-select protocol header is now also identical, making `V1` and `V1Lazy`
  indistinguishable on the wire. The remaining central effect of `V1Lazy` is that the dialer,
  if it only supports a single protocol in a negotiation, optimistically settles on that
  protocol without immediately flushing the negotiation data (i.e. protocol proposal)
  and without waiting for the corresponding confirmation before it is able to start
  sending application data, expecting for the used protocol to be confirmed with
  the response.
romanb

comment created time in 14 hours

push eventlibp2p/rust-libp2p

Roman S. Borschel

commit sha 86d228122284cd678dc431361ae544e7bdba1750

Tweak docs of StreamMuxer::poll_event

view details

dependabot[bot]

commit sha c01ac52745bbd0360076099a048183d93c123610

Update webpki-roots requirement from 0.20 to 0.21 (#1851) * Update webpki-roots requirement from 0.20 to 0.21 Updates the requirements on [webpki-roots](https://github.com/ctz/webpki-roots) to permit the latest version. - [Release notes](https://github.com/ctz/webpki-roots/releases) - [Commits](https://github.com/ctz/webpki-roots/compare/v/0.20.0...v/0.21.0) Signed-off-by: dependabot[bot] <support@github.com> * transports/websocket: Update changelog Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Max Inden <mail@max-inden.de>

view details

dependabot[bot]

commit sha 6f6c22a8faec2979a4e6f4623d2c8133048da14f

Update rustls requirement from 0.18.0 to 0.19.0 Updates the requirements on [rustls](https://github.com/ctz/rustls) to permit the latest version. - [Release notes](https://github.com/ctz/rustls/releases) - [Changelog](https://github.com/ctz/rustls/blob/main/OLDCHANGES.md) - [Commits](https://github.com/ctz/rustls/compare/v/0.18.0...v/0.19.0) Signed-off-by: dependabot[bot] <support@github.com>

view details

push time in 17 hours

push eventlibp2p/rust-libp2p

Roman S. Borschel

commit sha 86d228122284cd678dc431361ae544e7bdba1750

Tweak docs of StreamMuxer::poll_event

view details

dependabot[bot]

commit sha c01ac52745bbd0360076099a048183d93c123610

Update webpki-roots requirement from 0.20 to 0.21 (#1851) * Update webpki-roots requirement from 0.20 to 0.21 Updates the requirements on [webpki-roots](https://github.com/ctz/webpki-roots) to permit the latest version. - [Release notes](https://github.com/ctz/webpki-roots/releases) - [Commits](https://github.com/ctz/webpki-roots/compare/v/0.20.0...v/0.21.0) Signed-off-by: dependabot[bot] <support@github.com> * transports/websocket: Update changelog Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Max Inden <mail@max-inden.de>

view details

dependabot[bot]

commit sha 9d3b9bec71bf55575d9c5705bc0060600b2c4ac6

Update bytes requirement from 0.5 to 0.6 Updates the requirements on [bytes](https://github.com/tokio-rs/bytes) to permit the latest version. - [Release notes](https://github.com/tokio-rs/bytes/releases) - [Changelog](https://github.com/tokio-rs/bytes/blob/master/CHANGELOG.md) - [Commits](https://github.com/tokio-rs/bytes/compare/v0.5.6...v0.6.0) Signed-off-by: dependabot[bot] <support@github.com>

view details

push time in 17 hours

delete branch libp2p/rust-libp2p

delete branch : dependabot/cargo/webpki-roots-0.21

delete time in 17 hours

startedmjgallag/runhub

started time in 18 hours

created repositorymjgallag/runhub

created time in 18 hours

issue commentspacejam/sled

ALICE-like write permuter

Hmm, perhaps we should try fsyncing the directories and configuration file when creating a new database. I'd recommend turning on debugging symbols in your release profile, so that the static vulnerability reports are more readable.

spacejam

comment created time in 18 hours

PR closed libp2p/rust-libp2p

Reviewers
Improve muxer docs
+4 -0

1 comment

1 changed file

Demi-Marie

pr closed time in 18 hours

pull request commentlibp2p/rust-libp2p

Improve muxer docs

Superseded by https://github.com/libp2p/rust-libp2p/commit/86d228122284cd678dc431361ae544e7bdba1750.

Demi-Marie

comment created time in 18 hours

push eventlibp2p/rust-libp2p

Roman S. Borschel

commit sha 86d228122284cd678dc431361ae544e7bdba1750

Tweak docs of StreamMuxer::poll_event

view details

push time in 18 hours

issue commentspacejam/sled

ALICE-like write permuter

actyx is considering using sled. I was tasked with evaluating it. In particular the properties we care about are that the data is durable after a flush completed successfully. For this purpose I wrote a simple alice workload and checker https://github.com/dvc94ch/alice-sled and obtained the following results:

-------------------------------------------------------------------------------
ALICE tool version 0.0.1. Please go through the documentation, particularly the
listed caveats and limitations, before deriving any inferences from this tool.
-------------------------------------------------------------------------------
Parsing traces to determine logical operations ...
Logical operations:
0	mkdir("db", parent=510920, mode='0777', inode=510921)
1	mkdir("db/blobs", parent=510921, mode='0777', inode=510922)
2	creat("db/conf", parent=510921, mode='0666', inode=510923)
3	append("db/conf", offset=0, count=58, inode=510923)
4	append("db/conf", offset=58, count=4, inode=510923)
5	creat("db/db", parent=510921, mode='0666', inode=510924)
6	creat("db/blobs/295161", parent=510922, mode='0666', inode=510925)
7	append("db/blobs/295161", offset=0, count=4, inode=510925)
8	append("db/blobs/295161", offset=4, count=1, inode=510925)
9	append("db/blobs/295161", offset=5, count=327748, inode=510925)
10	fsync("db/db", size=0, inode=510924)
11	append("db/db", offset=0, count=295177, inode=510924)
12	fsync("db/db", size=295177, inode=510924)
-------------------------------------
Finding vulnerabilities...
(Dynamic vulnerability) Across-syscall atomicity, sometimes concerning durability: Operations 0 until 12 need to be atomically persisted
(Static vulnerability) Across-syscall atomicity: Operation B-/home/dvc/ipld/alice-sled/target/release/sled-workload:0x56044bbde468[None] until B-/usr/lib/libpthread-2.32.so:0x7f451db6757b[fsync]
Done finding vulnerabilities.

Not sure yet how to interpret them, but I assume they are false positives. Next is dumping our existing sqlite data into sled and seeing if it blows up.

spacejam

comment created time in 18 hours

issue commentlibp2p/rust-libp2p

Limit the number of outgoing connections per second

Is the time component, i.e. per second, really important? We already have a means to limit the number of concurrent pending as well as established connections, separately for incoming/outgoing with https://github.com/libp2p/rust-libp2p/pull/1848.

tomaka

comment created time in 19 hours

issue closedlibp2p/rust-libp2p

Add a way to limit the number of active connections

We currently have no way to limit the number of active connections. In the case of TCP/IP, this can be bad, as the available resources are limited.

I think the system should be in the Swarm, as the main objective of a limit here is to prevent DoS attacks.

closed time in 19 hours

tomaka

issue commentlibp2p/rust-libp2p

Add a way to limit the number of active connections

Should be resolved as of https://github.com/libp2p/rust-libp2p/pull/1848.

tomaka

comment created time in 19 hours

Pull request review commentlibp2p/rust-libp2p

[multistream] Make the lazy variant more interoperable.

 impl<TInner> Negotiated<TInner> {                         }                     }; -                    if let Message::Header(v) = &msg {-                        if *v == version {-                            *this.state = State::Expecting { io, protocol, version };+                    match (&msg, version) {+                        (Message::Header(Version::V1), Some(Version::V1)) => {

My proposal is https://github.com/libp2p/rust-libp2p/pull/1855/commits/70f041b8a79fb0f6e21048577e71188345dc2c25.

romanb

comment created time in 20 hours

push eventlibp2p/rust-libp2p

Roman Borschel

commit sha 1bd013c8432b88690b845594cf656a8e6756a28a

[swarm] Configurable and "infinite" scores for external addresses. (#1842) * Add "infinite" scores for external addresses. Extend address scores with an infinite cardinal, permitting addresses to be retained "forever" or until explicitly removed. Expose (external) address scores on the API. * Update swarm/src/registry.rs Co-authored-by: Pierre Krieger <pierre.krieger1708@gmail.com> * Fix compilation. * Update CHANGELOG Co-authored-by: Pierre Krieger <pierre.krieger1708@gmail.com>

view details

Roman Borschel

commit sha cef75ab7e11fa37650c106d89b7ef5409d442b00

[core/swarm] Refactor and extend configurable connection limits. (#1848) * Refactor and extend configurable connection limits. To better track different connection counts, permit configurable limits for these counts and make these available for inspection efficiently, introduce dedicated connection counters via a `ConnectionCounters` structure that is exposed on the API via the `NetworkInfo`. All connection or connection states that are counted in this way can also have effective configurable limits. * Cleanup * Add missing file. * Refine naming and config API. * Update core/CHANGELOG.md Co-authored-by: Max Inden <mail@max-inden.de> * Update core/CHANGELOG.md Co-authored-by: Max Inden <mail@max-inden.de> Co-authored-by: Max Inden <mail@max-inden.de>

view details

Pierre Krieger

commit sha 5e21806a3842c68bc33fd31b23c7ddc849fc006a

Some petty changes head of QUIC (#1854)

view details

Max Inden

commit sha e3af8b7d03bdf7e4b156481102a6a961c23a8946

*: Apply clippy suggestions and enable clippy on CI (#1850) * *: Apply clippy suggestions * .github: Add clippy check * protocols/kad/record: Implement custom PartialEq for ProviderRecord

view details

dependabot[bot]

commit sha faa8850d0294ed051f9d2b3d66ac6df92d6b6bb1

Update bytes requirement from 0.5 to 0.6 Updates the requirements on [bytes](https://github.com/tokio-rs/bytes) to permit the latest version. - [Release notes](https://github.com/tokio-rs/bytes/releases) - [Changelog](https://github.com/tokio-rs/bytes/blob/master/CHANGELOG.md) - [Commits](https://github.com/tokio-rs/bytes/compare/v0.5.6...v0.6.0) Signed-off-by: dependabot[bot] <support@github.com>

view details

push time in 20 hours

push eventlibp2p/rust-libp2p

Roman Borschel

commit sha cef75ab7e11fa37650c106d89b7ef5409d442b00

[core/swarm] Refactor and extend configurable connection limits. (#1848) * Refactor and extend configurable connection limits. To better track different connection counts, permit configurable limits for these counts and make these available for inspection efficiently, introduce dedicated connection counters via a `ConnectionCounters` structure that is exposed on the API via the `NetworkInfo`. All connection or connection states that are counted in this way can also have effective configurable limits. * Cleanup * Add missing file. * Refine naming and config API. * Update core/CHANGELOG.md Co-authored-by: Max Inden <mail@max-inden.de> * Update core/CHANGELOG.md Co-authored-by: Max Inden <mail@max-inden.de> Co-authored-by: Max Inden <mail@max-inden.de>

view details

Pierre Krieger

commit sha 5e21806a3842c68bc33fd31b23c7ddc849fc006a

Some petty changes head of QUIC (#1854)

view details

Max Inden

commit sha e3af8b7d03bdf7e4b156481102a6a961c23a8946

*: Apply clippy suggestions and enable clippy on CI (#1850) * *: Apply clippy suggestions * .github: Add clippy check * protocols/kad/record: Implement custom PartialEq for ProviderRecord

view details

dependabot[bot]

commit sha 773898d26f47c9fbb2ef520c097f6aa113605918

Update rustls requirement from 0.18.0 to 0.19.0 Updates the requirements on [rustls](https://github.com/ctz/rustls) to permit the latest version. - [Release notes](https://github.com/ctz/rustls/releases) - [Changelog](https://github.com/ctz/rustls/blob/main/OLDCHANGES.md) - [Commits](https://github.com/ctz/rustls/compare/v/0.18.0...v/0.19.0) Signed-off-by: dependabot[bot] <support@github.com>

view details

push time in 20 hours

Pull request review commentlibp2p/rust-libp2p

[multistream] Make the lazy variant more interoperable.

 impl<TInner> Negotiated<TInner> {                         }                     }; -                    if let Message::Header(v) = &msg {-                        if *v == version {-                            *this.state = State::Expecting { io, protocol, version };+                    match (&msg, version) {+                        (Message::Header(Version::V1), Some(Version::V1)) => {

What I had in mind is that you wouldn't decode a Version, but a new enum called for example VersionHandshake that would only contain V1 without any parameter.

I see, separating these two kinds of versions. I will give that a try.

romanb

comment created time in 21 hours

Pull request review commentlibp2p/rust-libp2p

[multistream] Make the lazy variant more interoperable.

 impl<TInner> Negotiated<TInner> {                         }                     }; -                    if let Message::Header(v) = &msg {-                        if *v == version {-                            *this.state = State::Expecting { io, protocol, version };+                    match (&msg, version) {+                        (Message::Header(Version::V1), Some(Version::V1)) => {

It would always be decoded from the wire with dialer_lazy_flush: false since that option is local.

What I had in mind is that you wouldn't decode a Version, but a new enum called for example VersionHandshake that would only contain V1 without any parameter.

I don't know if that makes sense, and I'm also not strongly opinionated. I'm fine with these changes.

romanb

comment created time in 21 hours

Pull request review commentlibp2p/rust-libp2p

[multistream] Make the lazy variant more interoperable.

 impl<TInner> Negotiated<TInner> {                         }                     }; -                    if let Message::Header(v) = &msg {-                        if *v == version {-                            *this.state = State::Expecting { io, protocol, version };+                    match (&msg, version) {+                        (Message::Header(Version::V1), Some(Version::V1)) => {

The fact that we assume that version can never be V1Lazy looks a bit like a hack to me.

By definition of these versions, they're now indistinguishable on the wire and it is always expected to receive V1. So receiving V1Lazy is considered an error (in the decoder really).

In an ideal world, we would differentiate the Version enum (containing V1 and V1Lazy) from the VersionHandshake (containing only V1), no?

Since V1Lazy is now really the same negotiation protocol as V1, only differing subtly in the behaviour of the dialer, it may be cleaner to separate this "lazy flush yes/no" setting from the Version, making it a separate configuration option for the dialer only, since V1Lazy used for a listener now has no effect at all. However, for transport and substream upgrades we currently only have this choice of Version without a concept of separate parameters for the roles of dialer and/or listener. Maybe Version::V1 { dialer_lazy_flush: bool } would already be an improvement, though be another API change. It would always be decoded from the wire with dialer_lazy_flush: false since that option is local. Should I give that a try? So due to the fact that this would require larger API changes, together with my hope that V1Lazy will eventually disappear altogether, merging its behaviour into V1, led me to the current approach, but I'm open to alternatives.

romanb

comment created time in 21 hours

Pull request review commentlibp2p/rust-libp2p

[multistream] Make the lazy variant more interoperable.

 pub enum Version {     /// [1]: https://github.com/libp2p/specs/blob/master/connections/README.md#protocol-negotiation     /// [2]: https://github.com/multiformats/multistream-select     V1,-    /// A lazy variant of version 1 that is identical on the wire but delays-    /// sending of protocol negotiation data as much as possible.-    ///-    /// Delaying the sending of protocol negotiation data can result in-    /// significantly fewer network roundtrips used for the negotiation,-    /// up to 0-RTT negotiation.+    /// A "lazy" variant of version 1 that is identical on the wire but whereby+    /// the dialer delays flushing protocol negotiation data in order to combine+    /// it with initial application data, thus performing 0-RTT negotiation.     ///-    /// 0-RTT negotiation is achieved if the dialer supports only a single-    /// application protocol. In that case the dialer immedidately settles+    /// This strategy is only applicable for the node with the role of "dialer"+    /// in the negotiation and only if the dialer supports just a single+    /// application protocol. In that case the dialer immedidately "settles"     /// on that protocol, buffering the negotiation messages to be sent     /// with the first round of application protocol data (or an attempt     /// is made to read from the `Negotiated` I/O stream).     ///-    /// A listener receiving a `V1Lazy` header will similarly delay sending-    /// of the protocol confirmation.  Though typically the listener will need-    /// to read the request data before sending its response, thus triggering-    /// sending of the protocol confirmation, which, in absence of additional-    /// buffering on lower layers will result in at least two response frames-    /// to be sent.-    ///-    /// `V1Lazy` is specific to `rust-libp2p`: While the wire protocol-    /// is identical to `V1`, delayed sending of protocol negotiation frames-    /// is only safe under the following assumptions:-    ///-    ///   1. The dialer is assumed to always send the first multistream-select-    ///      protocol message immediately after the multistream header, without-    ///      first waiting for confirmation of that header. Since the listener-    ///      delays sending the protocol confirmation, a deadlock situation may-    ///      otherwise occurs that is only resolved by a timeout. This assumption-    ///      is trivially satisfied if both peers support and use `V1Lazy`.+    /// A listener will behave identically to `V1`. This ensures interoperability with `V1`.+    /// Notably, it will immediately send the multistream header as well as the protocol+    /// confirmation, resulting in multiple frames being sent on the underlying transport.+    /// Nevertheless, if the listener supports the protocol that the dialer optimistically+    /// settled on, it can be a 0-RTT negotiation.     ///-    ///   2. When nesting multiple protocol negotiations, the listener is either-    ///      known to support all of the dialer's optimistically chosen protocols-    ///      or there is no intermediate protocol without a payload and none of-    ///      the protocol payloads has the potential for being mistaken for a-    ///      multistream-select protocol message. This avoids rare edge-cases whereby-    ///      the listener may not recognize upgrade boundaries and erroneously-    ///      process a request despite not supporting one of the intermediate-    ///      protocols that the dialer committed to. See [1] and [2].+    /// > **Note**: `V1Lazy` is specific to `rust-libp2p`. The wire protocol is identical to `V1`+    /// > and generally interoperable with peers only supporting `V1`. Nevertheless, there is a+    /// > pitfall that is rarely encountered: When nesting multiple protocol negotiations, the+    /// > listener should either be known to support all of the dialer's optimistically chosen+    /// > protocols or there is must be no intermediate protocol without a payload and none of+    /// > the protocol payloads must have the potential for being mistaken for a multistream-select+    /// > protocol message. This avoids rare edge-cases whereby the listener may not recognize

Yes, see https://github.com/multiformats/go-multistream/issues/20 for more details (that link is also in the code).

romanb

comment created time in a day

more