profile
viewpoint
Jacob Finkelman Eh2406 @SEMCOG Detroit

holmgr/cargo-sweep 237

A cargo subcommand for cleaning up unused build files generated by Cargo

Eh2406/quadrature 9

Fast Numerical Integration

pubgrub-rs/pubgrub 7

PubGrub version solving algorithm implemented in Rust

Eh2406/ThinkBayes-rs 6

Unofficial rewrite of ThinkBayes in Rust

Eh2406/rust-reader 3

light weight Windows screen reader

Eh2406/numpy-rust 2

This is a test of using numpy with rust-cpython.

UDST/developer 1

Redesigned UrbanSim developer/pro forma models

Eh2406/cargo 0

The Rust package manager

PR opened pubgrub-rs/pubgrub

fix: don't panic on Range::none() dependencies

This fixes #36. By filtering out the meaningless Term Negative(Range::none) in the specific case when it is generated from a dependency on Range::none().

+30 -18

0 comment

5 changed files

pr created time in an hour

create barnchpubgrub-rs/pubgrub

branch : bad_dependencies

created branch time in an hour

issue commentpubgrub-rs/pubgrub

Is the bump method on the Version trait actually necessary?

Is it worth thinking even more meta, can we take a Range as a user provided type? What functionality do we need from a Range?

mpizenberg

comment created time in 4 hours

Pull request review commentpubgrub-rs/pubgrub

let dependency provider priorities packages

 pub fn resolve<P: Package, V: Version>(             .should_cancel()             .map_err(|err| PubGrubError::ErrorShouldCancel(err))?; -        state.unit_propagation(next)?;+        state.unit_propagation(next.clone())?; -        // Pick the next package.-        let (p, term) = match state.partial_solution.pick_package(dependency_provider)? {-            None => {-                return state.partial_solution.extract_solution().ok_or_else(|| {-                    PubGrubError::Failure(-                        "How did we end up with no package to choose but no solution?".into(),-                    )-                })-            }-            Some(x) => x,-        };-        next = p.clone();-        let available_versions =-            dependency_provider-                .list_available_versions(&p)-                .map_err(|err| PubGrubError::ErrorRetrievingVersions {-                    package: p.clone(),-                    source: err,-                })?;+        let iter = state.partial_solution.potential_packages();+        if iter.is_none() {+            drop(iter);

The borrow checker did not like using a match on iter. This if ... is_none ... drop is a workaround. I believe this is a case where Polonius could help, when and if it lands in rustc.

Eh2406

comment created time in 4 hours

PullRequestReviewEvent

pull request commentrust-lang/cargo

List available packages if providing `--package` with an empty value

This looks good to me, @bors r? @ehuss for a second opinion.

weihanglo

comment created time in 5 hours

pull request commentpubgrub-rs/pubgrub

refactor!: define Dependencies type

This is a type alias for now, but I'd like to see if we could make it a struct and hide currently exposed Map alias in a separate PR.

If someday we want to add any other form of output, like statistics or as a dependency graph, we could add them as methods to this new type. So that seems like a good way to future proof things.

aleksator

comment created time in 20 hours

push eventpubgrub-rs/pubgrub

Eh2406

commit sha 1513bff56707400bea31a3d520f3bc150e20299e

feat: remove Hash req on DependencyProvider

view details

push time in 2 days

pull request commentpubgrub-rs/pubgrub

let dependency provider priorities packages

switched to Range. I (at this time of night) could not get the lifetimes to work for &Range so used Borrow<Range<V>>

Eh2406

comment created time in 2 days

push eventpubgrub-rs/pubgrub

Alex Tokarev

commit sha cbc6d4d4d1ecf8f3200569282b6f4c81135a4db2

refactor: apply Clippy suggestions (#52) * refactor: apply trivial Clippy suggestions * refactor: cargo clippy --fix * refactor: rewrite if chain using cmp() and match * feat: derive Default for OfflineDependencyProvider

view details

Eh2406

commit sha 39243a7ccf1fc3b0dd05d9e3dca1b6631836e734

feat: add a pick_package

view details

Eh2406

commit sha 88df6de7d939c4b04618927bf4e1fd0c00acf9b4

feat!: replace list_available_versions with make_decision

view details

Eh2406

commit sha a5fbf06bb54c9a4ff2ab87adbb36fdc8c34a5b6a

feat: remove Hash req on DependencyProvider

view details

push time in 2 days

issue commentpubgrub-rs/pubgrub

Move incompatibility identifiers outside of the incompatibility struct?

We can definitely leave it for later! It was just another option to keep in mind. (Especially in related conversations around the use of pub for ids.)

I don't think it's safe to assume that ids from addresses will remain the same

I think the Incompatibility can't move while we have a reference to it. At any time we can dereference any old copy of the reference we have. If it has been moved behind our back, then :boom: Undefined behavior in safe code. Whether I got the incantation correctly to get the location of the Incompatibility and not some other location... is a place to be careful.

mpizenberg

comment created time in 2 days

push eventpubgrub-rs/pubgrub

Alex Tokarev

commit sha 25b865187ecc71e8089b3f3f280caf897deb602e

ci: allow ! in commits to indicate breaking changes (#51) This change works around a bug in `commitlint`: https://github.com/conventional-changelog/commitlint/issues/2226

view details

Eh2406

commit sha 82dad8780af36e0afa2bd68c987c71f034e283f6

feat: add a pick_package

view details

Eh2406

commit sha daca2b65645e60f0b1fca3858425596850ad53c1

feat!: replace list_available_versions with make_decision

view details

Eh2406

commit sha 8ae3d6668b3acb05565d9da9bd07cf19e138781f

feat: remove Hash req on DependencyProvider

view details

push time in 2 days

push eventpubgrub-rs/pubgrub

Alex Tokarev

commit sha 25b865187ecc71e8089b3f3f280caf897deb602e

ci: allow ! in commits to indicate breaking changes (#51) This change works around a bug in `commitlint`: https://github.com/conventional-changelog/commitlint/issues/2226

view details

Eh2406

commit sha ddeacca018a1a89a018f09f4b7b8b3c763069a6a

refactor: use typed-arena to remove ids

view details

push time in 2 days

push eventpubgrub-rs/pubgrub

Alex Tokarev

commit sha 25b865187ecc71e8089b3f3f280caf897deb602e

ci: allow ! in commits to indicate breaking changes (#51) This change works around a bug in `commitlint`: https://github.com/conventional-changelog/commitlint/issues/2226

view details

Eh2406

commit sha b8e22d25a50e3dbfb2ebe5269d7bc9627403daa7

test: add two new benchmarks

view details

Eh2406

commit sha 98dcb35a8f183365147d09fcedb6528d31949aa5

test: shrink the benchmarks

view details

push time in 2 days

PullRequestReviewEvent
PullRequestReviewEvent

push eventpubgrub-rs/pubgrub

Jacob Finkelman

commit sha 25a1c8d63196d6ebbbf7cd0cf4a1eb291f5a94b1

test: switch to criterion (#47)

view details

Eh2406

commit sha 0e7b36892a69b63aea712f65269278bc2ae2c0d7

test: add two new benchmarks

view details

Eh2406

commit sha 5530de283c2d75a24c7fe8a212417756add0a61d

test: shrink the benchmarks

view details

push time in 2 days

push eventpubgrub-rs/pubgrub

Jacob Finkelman

commit sha 25a1c8d63196d6ebbbf7cd0cf4a1eb291f5a94b1

test: switch to criterion (#47)

view details

Eh2406

commit sha 0ed03b44186115134b32f69bc8d34aca535536b8

feat: add a pick_package

view details

Eh2406

commit sha ab09a3d7fc899177a363e7c31cc3bd595d5e2c60

feat!: replace list_available_versions with make_decision

view details

Eh2406

commit sha 8357220caa683c5e7faaeabbb5577ce920a7206a

feat: remove Hash req on DependencyProvider

view details

push time in 2 days

push eventpubgrub-rs/pubgrub

Jacob Finkelman

commit sha 25a1c8d63196d6ebbbf7cd0cf4a1eb291f5a94b1

test: switch to criterion (#47)

view details

Eh2406

commit sha 6093e4cbb692c2375bbabf112712b558071ace53

refactor: use typed-arena to remove ids

view details

push time in 2 days

Pull request review commentpubgrub-rs/pubgrub

test: switch to criterion

 // SPDX-License-Identifier: MPL-2.0+use std::time::Duration; -#![feature(test)]-extern crate test;-use test::Bencher;+extern crate criterion;+use self::criterion::*; -use pubgrub::solver::{resolve, OfflineDependencyProvider};+use pubgrub::solver::{resolve, DependencyProvider, OfflineDependencyProvider}; use pubgrub::version::NumberVersion; -#[cfg(feature = "serde")]-#[bench]-/// This is an entirely synthetic benchmark. It may not be realistic.-fn large_case(b: &mut Bencher) {-    let s = std::fs::read_to_string("test-examples/large_case_u16_NumberVersion.ron").unwrap();-    let dependency_provider: OfflineDependencyProvider<u16, NumberVersion> =-        ron::de::from_str(&s).unwrap();--    // bench-    b.iter(|| {-        let _ = resolve(&dependency_provider, 0, 0);-    });+fn bench_nested(c: &mut Criterion) {+    let mut group = c.benchmark_group("large_cases");+    group.measurement_time(Duration::from_secs(10));

It is also very dependent on optimizations, as the code gets faster the time can go down. I picked 10 working on top of a bunch of wip perf commits. I will change it to 20 and we can adjust as needed.

Unfortunately, confidence_interval does not affect the measurement time only the amount of difference that is declared regressed.

Eh2406

comment created time in 2 days

PullRequestReviewEvent

push eventpubgrub-rs/pubgrub

Eh2406

commit sha 49651d8564385745d3dfa6a509b80705a0855bbc

test: switch to criterion

view details

push time in 2 days

issue commentpubgrub-rs/pubgrub

Move incompatibility identifiers outside of the incompatibility struct?

Is it using their memory addresses as identifiers?

Yes. It is tricky to find a good alternative. We can do a linear scan of the Arena and use the index. But we would have to do it every time because Incompatibility can not impl Hash nor Ord, so ___Map<&'arena Incompatibility, Id> is out. Whatever we chews, it needs to become a method with good documentation of what it is doing.

if the 'arena lifetime doesn't creep up to the exposed API that could be acceptable even though quite annoying

Yes, it should not be in the public interface. If for no other reason then that means that the Arena with all the temporary Incompatibilitys will not yet have been dropped.

maybe there is a safe arena version somewhere?

The fundamental method provided by typed-arena is alloc(&self, value: T) -> &mut T. Somehow any arena with that method needs garantly that T will not be moved while self exists. I think that is fundamentally going to require unsafe somewhere.

There are lots of other arena crates with different apis. For example there is id-arena with no unsafe, but it's alloc does not return a reference it returns an opaque ID. We'd need to pass around a reference to the Arena for any place that wants to look at the contents of the Incompatibility. Fundamentally this is a variation on approach (2), and definitely worth considering if we decide on approach (2).

mpizenberg

comment created time in 2 days

push eventpubgrub-rs/pubgrub

Eh2406

commit sha 0715ab0b09e63583cce93ae236565a00710c71d5

refactor: use typed-arena to remove ids

view details

push time in 3 days

push eventpubgrub-rs/pubgrub

Eh2406

commit sha 9fb1ecd8a290c145d193f37e5f4dcfc4e7fddc74

refactor: use typed-arena to remove ids

view details

push time in 3 days

push eventpubgrub-rs/pubgrub

Eh2406

commit sha d5b157e28a5155cd77a7cb3956622bd1b0a93737

wip: remove ids

view details

push time in 3 days

issue commentpubgrub-rs/pubgrub

Move incompatibility identifiers outside of the incompatibility struct?

If I understand correctly all incompatibilities should be a copy of an item in incompatibility_store. That pattern is called an arena. That suggests an approach (4) use a crate like typed_arena so that we normally work with a &'arena Incompatibility, specifically so DerivedFrom can store &'arena Incompatibility<'arena, P, V> and we get rid of ids entirely.

Advantages:

  • no more ids
  • less memory usage (as we don't have more than one copy of an Incompatibility)
  • less clones as &_ impls copy

Disadvantages:

  • another dependency, one with unsafe code
  • an 'arena lifetime on almost everything
  • gnarly (for rust) error messages when other lifetimes get (incorrectly) entangled

And yes I did make sure this worked before suggesting it. Progress at no-ids branch.

mpizenberg

comment created time in 3 days

create barnchpubgrub-rs/pubgrub

branch : no-ids

created branch time in 3 days

pull request commentpubgrub-rs/pubgrub

PubGrub guide

I know this is wip but I just read it over and it looks grand so far! Thank you!

mpizenberg

comment created time in 3 days

pull request commentpubgrub-rs/pubgrub

let dependency provider priorities packages

I force pushed to fix the commit names. I also addressed @mpizenberg's comments. I have 2 remaining concerns:

  1. The module documentation still refers to list_available_versions. I could use some help describing the new method.
  2. I added a new trait Constraints to hide Term is this a good idea? Alternatively we can make Term public with only the one method available, or as we are filtering to positive terms we can convert to a Range which is already public.
Eh2406

comment created time in 3 days

push eventpubgrub-rs/pubgrub

Eh2406

commit sha 3f05880dca04d95b106d5ae37986526af6ac1c52

feat: add a pick_package

view details

Eh2406

commit sha 69a3061564deff5ed9671488222f880880878e0f

feat!: replace list_available_versions with make_decision

view details

Eh2406

commit sha 5e0d57d113607ae622fcac81765977859a1c7b4d

feat: remove Hash req on DependencyProvider

view details

push time in 3 days

pull request commentpubgrub-rs/pubgrub

refactor: apply Clippy suggestions

If there are a few lintes that are holding this PR back, then we should split them out and merge the clear cut parts.

aleksator

comment created time in 3 days

PR opened pubgrub-rs/pubgrub

let dependency provider priorities packages

This follows up on #40.

The first commit explores pick_package approach as described in https://github.com/pubgrub-rs/pubgrub/issues/40#issuecomment-714865685. The second commit explores the make_decision approach. The third commit uses a BTreeMap to remove Hash req on OfflineDependencyProvider

Beadback is definitely appreciated.

+178 -146

0 comment

7 changed files

pr created time in 3 days

create barnchpubgrub-rs/pubgrub

branch : priorities

created branch time in 3 days

push eventholmgr/cargo-sweep

Jonas Platte

commit sha 037155c153c50028571156c690a6d8da0830c147

Fix typo in main.rs

view details

push time in 3 days

PR merged holmgr/cargo-sweep

Fix typo in main.rs
+1 -1

0 comment

1 changed file

jplatte

pr closed time in 3 days

issue commentpubgrub-rs/pubgrub

feat: let the dependency provider choose a package

So I experimented with adding a pick_package that takes a impl Iterator<Item = (P, Term<V>)> and returns a Result<P,. There is a provided implementation that does fewest versions using list_available_versions, but it can be overwritten. For example OfflineDependencyProvider can use the same heuristic with many fewer allocations in its impl. This was going great! So time to clean the api, Can we prevent a bad implementation from returning a P that is not in the Iterator? (Yes) Can we limit how much of Terms api we make public? (Yes)

Then I noticed that the only thing we do with the resulting P was pass it to list_available_versions and the only thing we do with that was take the first in the list. So do we want to make a more radical change to the api by doing away with list_available_versions and having pick_package return a Result<(P, V),?

mpizenberg

comment created time in 4 days

pull request commentpubgrub-rs/pubgrub

refactor: apply Clippy suggestions

The commits are atomic so I'd be ok with rebase, but in general @mpizenberg prefers squash. So I'll leave it up to him.

aleksator

comment created time in 5 days

Pull request review commentpubgrub-rs/pubgrub

refactor: apply Clippy suggestions

 pub struct PartialSolution<P: Package, V: Version> {     memory: Memory<P, V>, } +pub type PackageTerm<P, V> = (P, Term<V>);

I have no methods to add to it so type alias works.

aleksator

comment created time in 5 days

PullRequestReviewEvent
PullRequestReviewEvent

Pull request review commentpubgrub-rs/pubgrub

refactor: apply Clippy suggestions

 pub struct PartialSolution<P: Package, V: Version> {     memory: Memory<P, V>, } +pub type PackageTerm<P, V> = (P, Term<V>);

Why is this here instead of in incompatibility.rs?

aleksator

comment created time in 5 days

pull request commentpubgrub-rs/pubgrub

refactor: apply Clippy suggestions

Lots of little things that each look good. This overall looks very good. I will try to revue one commit at time to ensure I like each thing.

aleksator

comment created time in 5 days

push eventpubgrub-rs/pubgrub

Eh2406

commit sha 2ce0d4423322fbf73fa5453344c9cf8bf6d23f8d

test: switch to criterion

view details

push time in 5 days

pull request commentpubgrub-rs/pubgrub

test: switch to criterion

Benchmarks are very sensitive to other things going on at the same time. Criterion uses statistics to accurately measure the smallest differences. I can get a reliable performance difference by changing the number of tabs open in a minimized chrome.

I think criterion_main builds a harness, but if so maybe we need a PR to there docs to use consistent terminology.

Eh2406

comment created time in 5 days

PR opened pubgrub-rs/pubgrub

test: switch to criterion

This switches our benchmark(s) to use criterion. Newer versions of criterion works better with slower tests, so it works out pretty well.

This automatically adds a benchmark for each file in test-examples/, removing the duplicated code pointed out in https://github.com/pubgrub-rs/pubgrub/pull/34#issuecomment-708426561. This also allows decoupling between the code and the ron files. For example if in #43 we decide not to store all the ron files in tree, this will run the benchmarks on only what is in the file system.

+44 -15

0 comment

2 changed files

pr created time in 5 days

create barnchpubgrub-rs/pubgrub

branch : criterion

created branch time in 5 days

issue commentrust-lang/cargo

Cargo ingores branch in .gitmodules when fetching submodules of dependencies

does this happen with https://doc.rust-lang.org/cargo/reference/config.html#netgit-fetch-with-cli set? If so it is a bug in libgit2.

turbocool3r

comment created time in 5 days

pull request commentrust-lang/cargo

New namespaced features implementation.

This was clearly a lot of work. Thank you!

ehuss

comment created time in 6 days

pull request commentpubgrub-rs/pubgrub

perf: precompute intersections (2x speedup)

By the way, I am a native speaker, but my spelling and grammar are atrocious. So I deeply appreciate your correction of any mistakes you find.

Eh2406

comment created time in 6 days

push eventpubgrub-rs/pubgrub

Jacob Finkelman

commit sha 0f5c913e1590a9cc48a622e98baf46a8087d8280

perf: precompute intersections (2x speedup) (#37) * perf: precompute intersections (2x speedup) * style: fix grammar in function names Co-authored-by: Alex Tokarev <aleksator@gmail.com>

view details

Eh2406

commit sha 280851ff71bd34c9e2cd6b49abc08efb306867ee

test: add two new benchmarks

view details

Alex Tokarev

commit sha 84ae067c1effbed4f75318c95118daba763c0cd0

docs: correct benchmark descriptions

view details

Eh2406

commit sha e8e31213cd40d25fb40072e28ef84bdcf7d25996

test: shrink the benchmarks

view details

push time in 6 days

Pull request review commentpubgrub-rs/pubgrub

perf: precompute intersections (2x speedup)

 impl<P: Package, V: Version> PartialSolution<P, V> {             .rposition(|(l, _)| *l == decision_level)             .unwrap_or(self.history.len() - 1);         *self = Self::from_assignments(-            self.history-                .to_owned()+            std::mem::take(&mut self.history)

Because we are building a new self and its history and assigning it to *self.

old code:

let copy = self.history.to_owned();
let new_self = Self::from_assignments(copy);
*self = new_self;

Note how no one uses the self.history after the copy. It is just replaced as part of the last line.

Eh2406

comment created time in 6 days

PullRequestReviewEvent

push eventpubgrub-rs/pubgrub

Eh2406

commit sha ec93c71ac290b977d82bd723990690b2b0ac094f

test: compare with a sat solver (#38)

view details

Matthieu Pizenberg

commit sha bf9d2872ec212e68bdec7bf6b2ba9b683b90f836

docs(readme): remove link to API docs of the dev branch (#42)

view details

Alex Tokarev

commit sha ad4f48e52af40e9d695b43907eede564465f4409

ci: check that commit messages follow project guidelines (#41)

view details

Eh2406

commit sha 8957829972b32f5da2b5f8d39e4a9be7d5278971

test: add two new benchmarks

view details

Alex Tokarev

commit sha 8f0e3290625131a9953ff5c4a01a7ca39504ca00

docs: correct benchmark descriptions

view details

Eh2406

commit sha 4e40774181e0c25438ade6aac98a6f463202f8c0

test: shrink the benchmarks

view details

push time in 8 days

push eventpubgrub-rs/pubgrub

Eh2406

commit sha ec93c71ac290b977d82bd723990690b2b0ac094f

test: compare with a sat solver (#38)

view details

Matthieu Pizenberg

commit sha bf9d2872ec212e68bdec7bf6b2ba9b683b90f836

docs(readme): remove link to API docs of the dev branch (#42)

view details

Alex Tokarev

commit sha ad4f48e52af40e9d695b43907eede564465f4409

ci: check that commit messages follow project guidelines (#41)

view details

Matthieu Pizenberg

commit sha 37dc24946db997b4f53b51ac4552a08f82ae549c

perf: precompute intersections (2x speedup)

view details

push time in 8 days

issue commentrust-lang/cargo

Please support cargo update --offline

What command are you running and what error messages are you getting?

Silentdoer

comment created time in 8 days

pull request commentpubgrub-rs/pubgrub

test: compare with a sat solver

This test does have a great shape, in that it takes any set of dependencies and can return if there a correct answer. As such I would like to set up tests that runs this against the ron files to ensure that the benchmarks are not going fast by getting the wrong answer. CC #43.

Eh2406

comment created time in 8 days

issue commentpubgrub-rs/pubgrub

Large files in the repository

I especially love the point about whether any new benchmarks are substantially different than the existing one. I'd much prefer to be optimizing something that runs in 0.03 sec then a comparable program that runs in 0.3 sec. Any analyses or scripts to as assess that would be great!

mpizenberg

comment created time in 8 days

issue commentpubgrub-rs/pubgrub

merge incompatibilities smartly

This is functionality we do not yet have. Can we keep it open?

Eh2406

comment created time in 9 days

issue commentmpizenberg/pubgrub-rs

Other functionality from the Dart implementation

So having seen the discussion so far I agree that when we are in an org it is worth making a separate repo to experiment with adding this functionality without changes to pubgrub-rs. I still think that Natalie decided to work these things into the the Dart implementation and so we should check if she had a good reason. It may be that for separation of concerns reasons we don't want to copy her decision, but I would like to understand it.

Instead of responding to the conversation on each feature, I will wait until we have that new repo and make a separate issue for each. If that is ok with you?

Eh2406

comment created time in 9 days

push eventmpizenberg/pubgrub-rs

Eh2406

commit sha c7fbd331c004a0cfd2eaf1bd956a6d8b7fab0bad

tests: compare with a sat solver

view details

push time in 9 days

pull request commentmpizenberg/pubgrub-rs

tests: compare with a sat solver

What is the advantage of the binary encoding logic which is hard to think about over just a double loop?

I don't know if it makes a practical difference for our use case. The double loop adds n*(n-1)/2 clauses. The binary encoding adds n*log_bits(n) clauses. The linked paper found the binary encoding to be much faster in there use cases.

Calling assume replaces the current set of assumed literals. So I don't think we need to clone.

As such the difference between self.solver.solve() and self.solver.model().is_some(), is effectively nothing. Although the former is newer, faster, and clearer so let me fix that.

Eh2406

comment created time in 9 days

pull request commentmpizenberg/pubgrub-rs

tests: compare with a sat solver

Yes the code that was ported is code I originally wrote. So we don't have to deal with that question this time. But thank you for keeping an eye on it!

Eh2406

comment created time in 9 days

issue commentrust-lang/cargo

resolver error messages as good as PubGrub

By the way progress is being made on a Rust rewrite! Discussion at:

  • https://rust-lang.zulipchat.com/#narrow/stream/260232-t-cargo.2FPubGrub and
  • https://rust-lang.zulipchat.com/#narrow/stream/246057-t-cargo/topic/PubGrub.20algorithm.20discussion

and for now the code is at:

  • https://github.com/mpizenberg/pubgrub-rs
Eh2406

comment created time in 10 days

issue openedmpizenberg/pubgrub-rs

Other functionality from the Dart implementation

The Dart implementation has some functionality that is not in the blog post. I do not know the the Dart implementation thoroughly, so if I missed something feel free to add to these notes.

  • "features" (in cargo) are a form of build flag that can enable optional dependencies. They were added to the dart implementation but where never exposed in the user interface.
  • "replacements" (in cargo the patch section and the replace) are a way to substitute the code for one package with some other code. In dart it is called dependency-overrides.
  • "pre-release" (in cargo) is part of the semver spec that allows versions like 1.0.0-alpha. I don't know how dart supports this but I believe it does.
  • "lockfiles" is a way to encode and reuse the output from a previous resolution, and even if the dependencies have changed keep things as similar as possible. In Cargo this is done by a set of heuristics that are part of the dependency provider, and it is a mess. In dart this is somehow worked into the resolver.

This is all functionality that Cargo and Dart have where the next steps for exploring is "look at how Dart did it". Even if Darts way of doing things does not work for Cargo it is probably worth making available in the library for other use cases. I will be writing up a separate issue for functionality that Cargo has that Dart does not.

created time in 10 days

push eventmpizenberg/pubgrub-rs

Eh2406

commit sha df919ae1f6f406cd1b642c95876ac1190d20c6b2

tests: compare with a sat solver

view details

push time in 10 days

PR opened mpizenberg/pubgrub-rs

tests: compare with a sat solver

This is a follow up https://github.com/mpizenberg/pubgrub-rs/pull/28, it uses the varisat library to make sure that we only reten Err when there is no solution. Ported from Cargo.

+143 -0

0 comment

3 changed files

pr created time in 10 days

create barnchmpizenberg/pubgrub-rs

branch : prop-sat

created branch time in 10 days

PR opened mpizenberg/pubgrub-rs

perf: precompute intersections (2x speedup)

This is a cleaned up version of @mpizenberg's work in https://github.com/mpizenberg/pubgrub-rs/issues/33#issuecomment-707165555

+84 -70

0 comment

4 changed files

pr created time in 10 days

push eventmpizenberg/pubgrub-rs

Matthieu Pizenberg

commit sha 3ee2333c82d6537c83ead15f8d4a3e3ea95a0116

perf: precompute intersections (2x speedup)

view details

push time in 10 days

push eventmpizenberg/pubgrub-rs

Matthieu Pizenberg

commit sha f1c336475f8b4df1c9f9b7249c95fac44eb642e2

perf: precompute intersections (2x speedup)

view details

push time in 10 days

push eventmpizenberg/pubgrub-rs

Alex Tokarev

commit sha 6fc11e6a925437b88befea5f3d99b988dc84f9ce

feat: use SPDX licence identifier (#25)

view details

Jacob Finkelman

commit sha 007f34aa5452a938ab7b7732fcc9155ae483aa7a

perf: choose package with fewest versions matching (30x speedup) (#32) * perf: choose package with fewest matching versions (30x speedup) * docs: change heuristic description for picking packages Co-authored-by: Alex Tokarev <aleksator@gmail.com>

view details

Jacob Finkelman

commit sha f297599c4f2356d2d940c67e0b019a409b536137

perf: don't redundantly add incompatibilities (1.5x speedup) (#29)

view details

Alex Tokarev

commit sha f8ad9385089b0419133672598261c602f14012bc

ci: check formatting and documentation (#31) * style: reformat CI files * ci: deduplicate CI workflows * style: rename CI workflows * ci: cache explicit cargo directories https://doc.rust-lang.org/cargo/guide/cargo-home.html recommends to cache only these directories from ~/.cargo: bin/ registry/index/ registry/cache/ git/db/ * ci: add Cargo version to the cache key * ci: use the latest stable Rust for testing * ci: check formatting * ci: check documentation builds without warnings * ci: use nightly compiler for generating docs * ci: cache documentation files * ci: run CI on master push * ci: run CI on schedule once a week * ci: remove ~/.cargo/bin from cache Since we don't use `cargo install` in our CI pipeline, this directory is going to be empty, so we might as well remove it from the cache.

view details

Jacob Finkelman

commit sha 055fbecd250a6f1b10a0d95f8c1a4efed342545c

test: port property tests from Cargo (#28) * test: add a mostly direct proptest port from Cargo * style: clean up * test: return OfflineDependencyProvider directly * test: add a first prop test * test: use ints instead of Strings to avoid expensive clones * test: check that algorithm is deterministic * test: port over prop_minimum_version_errors_the_same * test: port prop_removing_a_dep_cant_break * test: port prop_limited_independence_of_irrelevant_alternatives * test: port meta_test_deep_trees_from_strategy * ci: don't shrink on ci * feat: add a timeout * style: apply review suggestions * style: change license header to SPDX * docs: link to DependencyProvider.callback * style: proofread naming and comments * style: rename ReversedDependencyProvider * rename `callback` to `should_cancel` Co-authored-by: Alex Tokarev <aleksator@gmail.com>

view details

Matthieu Pizenberg

commit sha 48f09aaa35c217ef13f480a500ccb32d5dee3fef

perf: precompute intersections (2x speedup)

view details

push time in 10 days

delete branch mpizenberg/pubgrub-rs

delete branch : proptest-index

delete time in 10 days

push eventmpizenberg/pubgrub-rs

Jacob Finkelman

commit sha 055fbecd250a6f1b10a0d95f8c1a4efed342545c

test: port property tests from Cargo (#28) * test: add a mostly direct proptest port from Cargo * style: clean up * test: return OfflineDependencyProvider directly * test: add a first prop test * test: use ints instead of Strings to avoid expensive clones * test: check that algorithm is deterministic * test: port over prop_minimum_version_errors_the_same * test: port prop_removing_a_dep_cant_break * test: port prop_limited_independence_of_irrelevant_alternatives * test: port meta_test_deep_trees_from_strategy * ci: don't shrink on ci * feat: add a timeout * style: apply review suggestions * style: change license header to SPDX * docs: link to DependencyProvider.callback * style: proofread naming and comments * style: rename ReversedDependencyProvider * rename `callback` to `should_cancel` Co-authored-by: Alex Tokarev <aleksator@gmail.com>

view details

Eh2406

commit sha b7498a1edd34ad5da92b31b9f19e6bb27848d0ba

test: add two new benchmarks

view details

Alex Tokarev

commit sha 2e90679cdac2612f46e3e394e49290e3795f0837

docs: correct benchmark descriptions

view details

Eh2406

commit sha f6fdd1c61a3a83838f9a36462a1f5a7259ddfa2b

tests: shrink the benchmarks

view details

push time in 10 days

issue commentrust-lang/cargo

Please support cargo update --offline

I find this surprising. In what way does --offline not work with update? Can you show the error you are getting?

Silentdoer

comment created time in 10 days

issue commentrust-lang/cargo

when feature is not enable, should not generate to Cargo.lock and not compile

Can you make an example of the behavior you are seeing, and describe what you want it to do?

Silentdoer

comment created time in 10 days

push eventmpizenberg/pubgrub-rs

Alex Tokarev

commit sha f8ad9385089b0419133672598261c602f14012bc

ci: check formatting and documentation (#31) * style: reformat CI files * ci: deduplicate CI workflows * style: rename CI workflows * ci: cache explicit cargo directories https://doc.rust-lang.org/cargo/guide/cargo-home.html recommends to cache only these directories from ~/.cargo: bin/ registry/index/ registry/cache/ git/db/ * ci: add Cargo version to the cache key * ci: use the latest stable Rust for testing * ci: check formatting * ci: check documentation builds without warnings * ci: use nightly compiler for generating docs * ci: cache documentation files * ci: run CI on master push * ci: run CI on schedule once a week * ci: remove ~/.cargo/bin from cache Since we don't use `cargo install` in our CI pipeline, this directory is going to be empty, so we might as well remove it from the cache.

view details

Eh2406

commit sha 299c932ec56cba20748f3e7af203f861c157704e

test: add two new benchmarks

view details

Alex Tokarev

commit sha ad309bf2e4a60da049f2a151994644f694c536b5

docs: correct benchmark descriptions

view details

Eh2406

commit sha d2d78e52b9db55c1d308a46c1982afe01aa76e2d

tests: shrink the benchmarks

view details

push time in 11 days

pull request commentmpizenberg/pubgrub-rs

Proptest index

rebased and updated the name to should_cancel.

Eh2406

comment created time in 11 days

push eventmpizenberg/pubgrub-rs

Alex Tokarev

commit sha f8ad9385089b0419133672598261c602f14012bc

ci: check formatting and documentation (#31) * style: reformat CI files * ci: deduplicate CI workflows * style: rename CI workflows * ci: cache explicit cargo directories https://doc.rust-lang.org/cargo/guide/cargo-home.html recommends to cache only these directories from ~/.cargo: bin/ registry/index/ registry/cache/ git/db/ * ci: add Cargo version to the cache key * ci: use the latest stable Rust for testing * ci: check formatting * ci: check documentation builds without warnings * ci: use nightly compiler for generating docs * ci: cache documentation files * ci: run CI on master push * ci: run CI on schedule once a week * ci: remove ~/.cargo/bin from cache Since we don't use `cargo install` in our CI pipeline, this directory is going to be empty, so we might as well remove it from the cache.

view details

Eh2406

commit sha 46e901da4b655170452514574574dff8160e4380

a mostly direct port from Cargo

view details

Eh2406

commit sha 40b5cc1b665165c945d4242732e786fe5fb397e2

clean it up some

view details

Eh2406

commit sha ec95516eac11c542d5707bb331ffa5b4705267e0

return OfflineDependencyProvider directly

view details

Eh2406

commit sha 0174b7664b1eb73e65dd50328f24a09a97d16c9f

a first prop test

view details

Eh2406

commit sha a483769b1b9a8638c033fb4fd588372f5bcd3358

if cloning Strings are so expensive try using ints

view details

Eh2406

commit sha 38b64188a94f4eaabcc689a8fc52102531aaeb39

test allgorithm is deterministic

view details

Eh2406

commit sha cbe77a4ad43a2c298c541335d3924d4da31d9fae

poet over prop_minimum_version_errors_the_same

view details

Eh2406

commit sha ab01e88f6f8a5595b2e8f04ca225630fa2f95fbf

port prop_removing_a_dep_cant_break

view details

Eh2406

commit sha 65b52d06694149945f0b5cc88d745c0b8a295744

port prop_limited_independence_of_irrelevant_alternatives

view details

Eh2406

commit sha 7a2550d2b77f0c8a44e9cbdfa42824dd59dda03a

port meta_test_deep_trees_from_strategy

view details

Eh2406

commit sha c7ca561f83329e4bf5af13346ce43d13b4c46394

ci: dont shrink on ci

view details

Eh2406

commit sha bf378633f1a3c611da474d119ae0568caa5fd763

tests: add a timeout

view details

Eh2406

commit sha bc6d8e99b4f613adb0bae38e258f8314fe286a18

suggestions

view details

Alex Tokarev

commit sha d9b3f5642c7016c4a4f146e8b05b909a18fb1d8e

style: change license header to SPDX

view details

Alex Tokarev

commit sha 49e6e73a27287e5b7263a54e72d7844d83651cf1

docs: link to DependencyProvider.callback

view details

Alex Tokarev

commit sha 4cc553112b19210d53d910f299f9cdfc0d8b17c3

style: proofread naming and comments

view details

Alex Tokarev

commit sha 64bbaa55c0a54846dd6c6aa95d4fd092517c7153

style: rename ReversedDependencyProvider

view details

Eh2406

commit sha bff9acd2af957bde348784d453785dbd723acd66

rename to `should_cancel`

view details

push time in 11 days

issue commentmpizenberg/pubgrub-rs

don't panic: on bad dependencies

This is a hack but is the resolve when making dependencies I added:

            Some(x) => {
                if x.values().any(|r| r == &Range::none()) {
                    // todo: what is the correct Incompatibility?
                    state.add_incompatibility(|id| {
                        Incompatibility::unavailable_dependencies(id, p.clone(), v.clone())
                    });
                    continue;
                }
                x
            },

Witch gets this test to pass. What would be a more elegant solution?

Eh2406

comment created time in 11 days

issue openedmpizenberg/pubgrub-rs

don't panic: on bad dependencies

Minimizing from the panic found in https://github.com/mpizenberg/pubgrub-rs/pull/34#issuecomment-708764474. I get the small test case of:

#[test]
fn should_always_find_a_satisfier() {
    let mut dependency_provider = OfflineDependencyProvider::<_, NumberVersion>::new();
    dependency_provider.add_dependencies("a", 0, vec![("b", Range::none())]);

    // Run the algorithm.
    let _ = resolve(&dependency_provider, "a", 0);
}

While this is a silly input, we should not panic on it.

created time in 11 days

pull request commentmpizenberg/pubgrub-rs

Benchmark v2

Garbage in, garbage out. I am deleting blindly, I would not be surprised if some of the inputs are nonsensical.

Also, based on using the timeout functionality from #28 I now know that the >1min/iter is not calling callback. So even more interesting.

I will endeavor to find a minimal example leading to that panic.

Eh2406

comment created time in 11 days

push eventmpizenberg/pubgrub-rs

Eh2406

commit sha cfcff1d8e7b7e15d3d4b3c2cba2f45a467f3a890

tests: shrink the benchmarks

view details

push time in 12 days

pull request commentmpizenberg/pubgrub-rs

Benchmark v2

I have it set up to run in a loop working on case_2. Progress has slowed. However It is often finding that by removing one line it can make it take longer then 1 min per / iter. So that is definitely interesting. It all so often finds input that give 'We should always find a satisfier if called in the right context.'. I don't know is that an interesting case worth making a minimal example of or is that a case of GIGO. What do you think?

Eh2406

comment created time in 12 days

pull request commentmpizenberg/pubgrub-rs

Benchmark v2

I hacked something together, and have had some success making the files smaller with disproportionately smaller runtime changes.

114,568 -> 79,381 large_case_1_u16_NumberVersion.ron 125,430 -> 99,792 large_case_2_u16_NumberVersion.ron 111,299 -> 21,404 large_case_u16_NumberVersion.ron

Eh2406

comment created time in 12 days

push eventmpizenberg/pubgrub-rs

Eh2406

commit sha 9c19255e3c2b91cb78d736556b955ee5da92ee82

tests: shrink the benchmarks

view details

push time in 12 days

pull request commentmpizenberg/pubgrub-rs

Proptest index

https://rust-lang.zulipchat.com/#narrow/stream/260232-t-cargo.2FPubGrub/topic/How.20to.20add.20a.20timeout.3F

Eh2406

comment created time in 12 days

Pull request review commentmpizenberg/pubgrub-rs

Check formatting and documentation on CI

+name: CI+on:+  pull_request:+  push:+    branches: [ master, dev ]+  schedule: [ cron: "0 6 * * 4" ]++env:+  CARGO_TERM_COLOR: always++jobs:+  test:+    name: Build and test+    runs-on: ubuntu-latest+    steps:+      - uses: actions/checkout@v2+      - name: Install stable Rust+        uses: actions-rs/toolchain@v1+        with:+          toolchain: stable+          profile: minimal++      - name: Get Cargo version+        id: cargo_version+        run: echo "::set-output name=version::$(cargo -V | tr -d ' ')"+        shell: bash++      - name: Download cache+        uses: actions/cache@v2+        with:+          path: |+            ~/.cargo/bin/

.cargo/bin/ I will asked on https://rust-lang.zulipchat.com/#narrow/stream/246057-t-cargo/topic/why.20cache.20.2Ecargo.2Fbin.2F.20.3F

It it is recommended by the book then lets do it. If the book is wrong for our needs we can do a follow up PR.

aleksator

comment created time in 12 days

PullRequestReviewEvent

pull request commentmpizenberg/pubgrub-rs

Benchmark v2

Also is there a file size small enough that would be ok?

I have been thinking about how to minimize them further. My thought is right a script that:

  1. Calculates a (runtime)/(file size)
  2. Deletes some random lines in the input
  3. Get a new (runtime)/(file size) if it is lower than reset the file
  4. go to 2
Eh2406

comment created time in 12 days

pull request commentmpizenberg/pubgrub-rs

Benchmark v2

That was defintly on my todo list. I have done it in the past with a homegrown macro.

BTW: @mpizenberg sounded like he is not ready for merge https://github.com/mpizenberg/pubgrub-rs/issues/33#issuecomment-707204692

Eh2406

comment created time in 12 days

pull request commentmpizenberg/pubgrub-rs

Proptest index

That works!

Eh2406

comment created time in 12 days

Pull request review commentmpizenberg/pubgrub-rs

Check formatting and documentation on CI

+name: CI+on:+  pull_request:+  push:+    branches: [ master, dev ]+  schedule: [ cron: "0 6 * * 4" ]++env:+  CARGO_TERM_COLOR: always++jobs:+  test:+    name: Build and test+    runs-on: ubuntu-latest+    steps:+      - uses: actions/checkout@v2+      - name: Install stable Rust+        uses: actions-rs/toolchain@v1+        with:+          toolchain: stable+          profile: minimal++      - name: Get Cargo version+        id: cargo_version+        run: echo "::set-output name=version::$(cargo -V | tr -d ' ')"+        shell: bash++      - name: Download cache+        uses: actions/cache@v2+        with:+          path: |+            ~/.cargo/bin/

Won't the rust install have set this up? Why cache as well?

aleksator

comment created time in 12 days

PullRequestReviewEvent

pull request commentmpizenberg/pubgrub-rs

Proptest index

Are we leaving those for later?

I interpret "Good to merge in my opinion, just waiting for @aleksator feedback" as a yes.

I like @aleksator's renames. I see the point about the name callback, can we have something more descriptive? Like callback and hook give no indication of why or when there called. Maybe check_up, check_in, heartbeat?

Also do we want that callback to take an argument in case we have diagnostics we can share about the progress?

Eh2406

comment created time in 12 days

pull request commentmpizenberg/pubgrub-rs

Benchmark v2

They are both smaller then large_case_u16_NumberVersion but take longer to run. If they turn out to have related performance profiles, then it may makes sense to only keep one.

I am open to more realistic, or size efficient benchmarks. I do think the benchmarks should not depend on live data, (for example crates.io) nor on complicated generation code. As I want to be sure that the benchmarks are comparable over time.

Eh2406

comment created time in 13 days

Pull request review commentmpizenberg/pubgrub-rs

Proptest index

+// This Source Code Form is subject to the terms of the Mozilla Public+// License, v. 2.0. If a copy of the MPL was not distributed with this+// file, You can obtain one at http://mozilla.org/MPL/2.0/.++use std::{collections::BTreeSet as Set, error::Error};++use pubgrub::range::Range;+use pubgrub::solver::{resolve, DependencyProvider, OfflineDependencyProvider};+use pubgrub::type_aliases::Map;+use pubgrub::{+    error::PubGrubError, package::Package, report::DefaultStringReporter, report::Reporter,+    version::NumberVersion, version::Version,+};++use proptest::collection::{btree_map, vec};+use proptest::prelude::*;+use proptest::sample::Index;+use proptest::string::string_regex;++/// The same as DP but it prefers the opposite versions.+/// If DP returns versions from newest to oldest, this returns them from oldest to newest.+#[derive(Clone)]+struct ReversedDependencyProvider<DP>(DP);++impl<P: Package, V: Version, DP: DependencyProvider<P, V>> DependencyProvider<P, V>+    for ReversedDependencyProvider<DP>+{+    // Lists only from remote for simplicity+    fn list_available_versions(&self, p: &P) -> Result<Vec<V>, Box<dyn Error>> {+        self.0.list_available_versions(p).map(|mut v| {+            v.reverse();+            v+        })+    }++    // Caches dependencies if they were already queried+    fn get_dependencies(&self, p: &P, v: &V) -> Result<Option<Map<P, Range<V>>>, Box<dyn Error>> {+        self.0.get_dependencies(p, v)+    }+}++/// The same as DP but it has a time out.+#[derive(Clone)]+struct TimeoutDependencyProvider<DP> {+    dp: DP,+    start_time: std::time::Instant,+    call_count: std::cell::Cell<u64>,+    max_calls: u64,+}++impl<DP> TimeoutDependencyProvider<DP> {+    fn new(dp: DP, max_calls: u64) -> Self {+        Self {+            dp,+            start_time: std::time::Instant::now(),+            call_count: std::cell::Cell::new(0),+            max_calls,+        }+    }+}++impl<P: Package, V: Version, DP: DependencyProvider<P, V>> DependencyProvider<P, V>+    for TimeoutDependencyProvider<DP>+{+    // Lists only from remote for simplicity+    fn list_available_versions(&self, p: &P) -> Result<Vec<V>, Box<dyn Error>> {+        self.dp.list_available_versions(p)+    }++    // Caches dependencies if they were already queried+    fn get_dependencies(&self, p: &P, v: &V) -> Result<Option<Map<P, Range<V>>>, Box<dyn Error>> {+        self.dp.get_dependencies(p, v)+    }++    fn callback(&self) -> Result<(), Box<dyn Error>> {+        assert!(self.start_time.elapsed().as_secs() < 60);+        let calls = self.call_count.get();+        assert!(calls < self.max_calls);+        self.call_count.set(calls + 1);+        Ok(())+    }+}++#[test]+#[should_panic]+fn callback_can_panic() {+    let mut dependency_provider = OfflineDependencyProvider::<_, NumberVersion>::new();+    dependency_provider.add_dependencies(0, 0, vec![(666, Range::any())]);++    // Run the algorithm.+    let _ = resolve(+        &TimeoutDependencyProvider::new(dependency_provider, 1),+        0,+        0,+    );+}++fn string_names() -> impl Strategy<Value = String> {+    string_regex("[A-Za-z][A-Za-z0-9_-]{0,5}")+        .unwrap()+        .prop_filter("reserved names", |n| {+            // root is the name of the thing being compiled+            // so it would be confusing to have it in the index+            // bad is a name reserved for a dep that won't work+            n != "root" && n != "bad"+        })+}++/// This generates a random registry index.+/// Unlike vec((Name, Ver, vec((Name, VerRq), ..), ..)+/// This strategy has a high probability of having valid dependencies+pub fn registry_strategy<N: Package + Ord>(+    name: impl Strategy<Value = N>,+    bad_name: N,+) -> impl Strategy<+    Value = (+        OfflineDependencyProvider<N, NumberVersion>,+        Vec<(N, NumberVersion)>,+    ),+> {+    let max_crates = 40;+    let max_versions = 15;+    let shrinkage = 40;+    let complicated_len = 10usize;++    // If this is false than the crate will depend on the nonexistent "bad"+    // instead of the complex set we generated for it.+    let allow_deps = prop::bool::weighted(0.99);++    let a_version = ..(max_versions as u32);++    let list_of_versions = btree_map(a_version, allow_deps, 1..=max_versions)+        .prop_map(move |ver| ver.into_iter().collect::<Vec<_>>());++    let list_of_crates_with_versions = btree_map(name, list_of_versions, 1..=max_crates);++    // each version of each crate can depend on each crate smaller then it.+    // In theory shrinkage should be 2, but in practice we get better trees with a larger value.+    let max_deps = max_versions * (max_crates * (max_crates - 1)) / shrinkage;++    let raw_version_range = (any::<Index>(), any::<Index>());+    let raw_dependency = (any::<Index>(), any::<Index>(), raw_version_range);++    fn order_index(a: Index, b: Index, size: usize) -> (usize, usize) {+        use std::cmp::{max, min};+        let (a, b) = (a.index(size), b.index(size));+        (min(a, b), max(a, b))+    }++    let list_of_raw_dependency = vec(raw_dependency, ..=max_deps);++    // By default a package depends only on other packages that have a smaller name,+    // this helps make sure that all things in the resulting index are DAGs.+    // If this is true then the DAG is maintained with grater instead.+    let reverse_alphabetical = any::<bool>().no_shrink();++    (+        list_of_crates_with_versions,+        list_of_raw_dependency,+        reverse_alphabetical,+        1..(complicated_len + 1),+    )+        .prop_map(+            move |(crate_vers_by_name, raw_dependencies, reverse_alphabetical, complicated_len)| {+                let mut list_of_pkgid: Vec<(+                    (N, NumberVersion),+                    Option<Vec<(N, Range<NumberVersion>)>>,+                )> = crate_vers_by_name+                    .iter()+                    .flat_map(|(name, vers)| {+                        vers.iter().map(move |x| {+                            (+                                (name.clone(), NumberVersion::from(x.0)),+                                if x.1 { Some(vec![]) } else { None },+                            )+                        })+                    })+                    .collect();+                let len_all_pkgid = list_of_pkgid.len();+                for (a, b, (c, d)) in raw_dependencies {+                    let (a, b) = order_index(a, b, len_all_pkgid);+                    let (a, b) = if reverse_alphabetical { (b, a) } else { (a, b) };+                    let ((dep_name, _), _) = list_of_pkgid[a].to_owned();+                    if &(list_of_pkgid[b].0).0 == &dep_name {+                        continue;+                    }+                    let s = &crate_vers_by_name[&dep_name];+                    let s_last_index = s.len() - 1;+                    let (c, d) = order_index(c, d, s.len());++                    if let (_, Some(deps)) = &mut list_of_pkgid[b] {+                        deps.push((+                            dep_name,+                            if c == 0 && d == s_last_index {+                                Range::any()+                            } else if c == 0 {+                                Range::strictly_lower_than(s[d].0 + 1)+                            } else if d == s_last_index {+                                Range::higher_than(s[c].0)+                            } else if c == d {+                                Range::exact(s[c].0)+                            } else {+                                Range::between(s[c].0, s[d].0 + 1)+                            },+                        ))+                    }+                }++                let mut dependency_provider = OfflineDependencyProvider::<N, NumberVersion>::new();++                let complicated_len = std::cmp::min(complicated_len, list_of_pkgid.len());+                let complicated: Vec<_> = if reverse_alphabetical {+                    &list_of_pkgid[..complicated_len]+                } else {+                    &list_of_pkgid[(list_of_pkgid.len() - complicated_len)..]+                }+                .iter()+                .map(|(x, _)| (x.0.clone(), x.1))+                .collect();++                for ((name, ver), deps) in list_of_pkgid {+                    dependency_provider.add_dependencies(+                        name,+                        ver,+                        deps.unwrap_or_else(|| vec![(bad_name.clone(), Range::any())]),+                    );+                }++                (dependency_provider, complicated)+            },+        )+}++/// This test is to test the generator to ensure+/// that it makes registries with large dependency trees+#[test]+fn meta_test_deep_trees_from_strategy() {+    use proptest::strategy::ValueTree;+    use proptest::test_runner::TestRunner;++    let mut dis = [0; 21];++    let strategy = registry_strategy(0u16..665, 666);+    let mut test_runner = TestRunner::deterministic();+    for _ in 0..128 {+        let (dependency_provider, cases) = strategy+            .new_tree(&mut TestRunner::new_with_rng(+                Default::default(),+                test_runner.new_rng(),+            ))+            .unwrap()+            .current();++        for (name, ver) in cases {+            let res = resolve(&dependency_provider, name, ver);+            dis[res+                .as_ref()+                .map(|x| std::cmp::min(x.len(), dis.len()) - 1)+                .unwrap_or(0)] += 1;+            if dis.iter().all(|&x| x > 0) {+                return;+            }+        }+    }++    panic!(+        "In {} tries we did not see a wide enough distribution of dependency trees! dis: {:?}",+        dis.iter().sum::<i32>(),+        dis+    );+}++proptest! {+    #![proptest_config(ProptestConfig {+    max_shrink_iters:+        if std::env::var("CI").is_ok() {+            // This attempts to make sure that CI will fail fast,+            0+        } else {+            // but that local builds will give a small clear test case.+            2048 // u32::MAX+        },+        result_cache: prop::test_runner::basic_result_cache,+        .. ProptestConfig::default()+    })]++    #[test]+    /// This test is mostly for profiling+    fn prop_passes_string(+        (dependency_provider, cases) in registry_strategy(string_names(), "bad".to_owned())+    )  {+        for (name, ver) in cases {+            let _ = resolve(&TimeoutDependencyProvider::new(dependency_provider.clone(), 50_000), name, ver);+        }+    }++    #[test]+    /// This test is mostly for profiling+    fn prop_passes_int(+        (dependency_provider, cases) in registry_strategy(0u16..665, 666)+    )  {+        for (name, ver) in cases {+            let _ = resolve(&TimeoutDependencyProvider::new(dependency_provider.clone(), 50_000), name, ver);+        }+    }++    #[test]+    /// This tests whether the algorithm is still deterministic+    fn prop_same_on_repeated_runs(+        (dependency_provider, cases) in registry_strategy(0u16..665, 666)+    )  {+        for (name, ver) in cases {+            let one = resolve(&TimeoutDependencyProvider::new(dependency_provider.clone(), 50_000), name, ver);+            for _ in 0..3 {+                match (&one, &resolve(&TimeoutDependencyProvider::new(dependency_provider.clone(), 50_000), name, ver)) {+                    (Ok(l), Ok(r)) => assert_eq!(l, r),+                    (Err(PubGrubError::NoSolution(derivation_l)), Err(PubGrubError::NoSolution(derivation_r))) => {+                        prop_assert_eq!(+                            format!("{}", DefaultStringReporter::report(&derivation_l)),+                            format!("{}", DefaultStringReporter::report(&derivation_r))+                        )},+                    _ => panic!("not the same result")+                }+            }+        }+    }++    #[test]+    /// [ReversedDependencyProvider] change what order the candidates+    /// are tried but not the existence of a solution+    fn prop_reversed_version_errors_the_same(+        (dependency_provider, cases) in registry_strategy(0u16..665, 666)+    )  {+        let minimal_provider = ReversedDependencyProvider(dependency_provider.clone());+        for (name, ver) in cases {+            let l = resolve(&TimeoutDependencyProvider::new(dependency_provider.clone(), 50_000), name, ver);+            let r = resolve(&TimeoutDependencyProvider::new(minimal_provider.clone(), 50_000), name, ver);+            match (&l, &r) {+                (Ok(_), Ok(_)) => (),+                (Err(_), Err(_)) => (),+                _ => panic!("not the same result")+            }+        }+    }++    #[test]+    fn prop_removing_a_dep_cant_break(+        (dependency_provider, cases) in registry_strategy(0u16..665, 666),+        indexes_to_remove in prop::collection::vec((any::<prop::sample::Index>(), any::<prop::sample::Index>(), any::<prop::sample::Index>()), 1..10)+    )  {+        let packages: Vec<_> = dependency_provider.packages().collect();+        let mut removed_provider = dependency_provider.clone();+        for (package_idx, version_idx, dep_idx) in indexes_to_remove {+            let package = package_idx.get(&packages);+            let versions = dependency_provider.list_available_versions(package).unwrap();+            let version = version_idx.get(&versions);+            let dependencys: Vec<_> = dependency_provider.get_dependencies(package, version).unwrap().unwrap().into_iter().collect();+            if !dependencys.is_empty() {+                let dependency = dep_idx.get(&dependencys).0.clone();+                removed_provider.add_dependencies(+                    **package,+                    *version,+                    dependencys.into_iter().filter(|x| x.0 != dependency)+                )+            }+        }+        for (name, ver) in cases {+            if resolve(&TimeoutDependencyProvider::new(dependency_provider.clone(), 50_000), name, ver).is_ok() {+                prop_assert!(+                    resolve(&TimeoutDependencyProvider::new(removed_provider.clone(), 50_000), name, ver).is_ok(),

We will definitely have to work on all of these tests, to ensure they maintain a good signal to noise ratio. If they turn out not to pull their weight, we can alter or remove! Just to clarify are you ok with merging as is?

Eh2406

comment created time in 13 days

PullRequestReviewEvent

issue commentmpizenberg/pubgrub-rs

Don't require a clone in Solver's methods

We can change resolve to only use iterators. So I started with returning Iterator<Item = V>, but Vec is not an iterator it is an IntoIterator, so lets try:

pub trait DependencyProvider<P: Package, V: Version> {
    type IV: IntoIterator<Item = V>;
...
    fn list_available_versions(&self, package: &P) -> Result<Self::IV, Box<dyn Error>>;

    type ID: IntoIterator<Item = (P, Range<V>)>;
...
    fn get_dependencies(
        &self,
        package: &P,
        version: &V,
    ) -> Result<Option<Self::ID>, Box<dyn Error>>;
...
}

This works for Vec<V>, BTreeSet<V>, and im::Vector<V>, but impl IntoIterarater is not supported (yet?) for associated types. So anything more complicated than a plane owned type is really painful or does not work. So no BTreeMap.keys().cloned().rev(), not even Box<Vec<V>>.

Looking back, I may just have rediscovered @mpizenberg's point:

But I think I tried it and the compiler was not happy with my usage of impl.

So if impl does not work lets try using dyn, but dyn can't live on its own so it is really Box<dyn Iterator<Item = V> + '_>.

pub trait DependencyProvider<P: Package, V: Version> {
...
    fn list_available_versions<'s>(
        &'s self,
        package: &P,
    ) -> Result<Box<dyn Iterator<Item = V> + 's>, Box<dyn Error>>;
...
    fn get_dependencies<'s>(
        &'s self,
        package: &P,
        version: &V,
    ) -> Result<Option<Box<dyn Iterator<Item = (P, Range<V>)> + 's>>, Box<dyn Error>>;
...
}

This lets us do all kinds of fancy things, but we have an allocation anyway. Is it better to have a Box<_> vs a Vec?

Looking back, I may just have rediscovered @mpizenberg's point:

Maybe a dyn was needed and I didn't know if it was worth it.

So not having found a good api, I am leaning toward leaving things alone.

Eh2406

comment created time in 13 days

pull request commentmpizenberg/pubgrub-rs

WIP testing CI changes

I think it would be reasonable to ask in https://rust-lang.zulipchat.com/#narrow/stream/242791-t-infra. And the rest of the conversation sounds eminently reasonable. I am lucky to be working with such thoughtful people!

aleksator

comment created time in 13 days

more