profile
viewpoint
Daniel Dunbar ddunbar @apple San Francisco, CA http://minormatter.com/ddunbar I work on build systems and the Swift Package Manager at Apple. I love testing infrastructure. Previously of Clang, @KLEE, and Blender3D.

apple/swift-llbuild2 202

A fresh take on a low-level build system API.

ddunbar/zcov 45

Code coverage reporting tool

ddunbar/PDSample 15

Poisson-disk sampling code as described in "A Spatial Data Structure for Fast Poisson-Disk Sample Generation".

apple/swift-tools-support-async 9

Common infrastructural helpers on top of NIO for llbuild2 and swiftpm-on-llbuild2 projects. This is NOT a general purpose package and is unlikely to ever become stable.

ddunbar/CCGSubSurf 9

Catmull Clark Gridding Subdivision Surface Library

FranzBusch/swift-evolution 8

This maintains proposals for changes and user-visible enhancements to the Swift Programming Language.

ddunbar/swift-evolution 3

This maintains proposals for changes and user-visible enhancements to the Swift Programming Language.

ddunbar/robusta 2

Taste Testing Web App

ddunbar/swift-package-manager 2

The Package Manager for the Swift Programming Language

pull request commentapple/swift-package-manager

[Collections] CLI 'remove' results in "database is locked" error

@swift-ci please smoke test

yim-lee

comment created time in a minute

PR opened apple/swift-package-manager

[Collections] CLI 'remove' results in "database is locked" error

Motivation: Using the package-collection CLI, the remove command results in "database is locked" error even though the collection is removed successfully. This is because a SQLitePackageCollectionsStorage is created at the beginning of each command run, and in the initializer the time-consuming populateTargetTrie is called. When the command finishes running, it triggers SQLitePackageCollectionsStorage to be closed, which in turn closes the SQLite connection, but this is done while populateTargetTrie is still running and thus causes the "database is locked" error.

Modifications:

  • Initializing PackageCollections can be expensive and should only be done once in a command
  • Add and set isShuttingDown flag when SQLitePackageCollectionsStorage.close is called so populateTargetTrie knows that it should stop.
  • Try db.close in a second attempt after allowing time for database operations to react to isShuttingDown flag
  • populateTargetTrie should memoize its result

Result: No "database is locked" error.

+127 -101

0 comment

2 changed files

pr created time in a minute

pull request commentapple/swift-package-manager

[PackageEditor] Mechanical editing functionality for package manifests

Hi @abertelrud,

I think it would be pretty easy to expose this functionality to library clients by adding a new library product for the PackageSyntax module. Including PackageSyntax in libSwiftPM could work too, but would cause issues if somebody tried to switch between a version compiled with/without syntax support. My understanding is that Xcode currently always uses the version of libSwiftPM in the default toolchain, so maybe that isn't a problem.

If we decide not to use SwiftSyntax for source locations in error messages, splitting this code out into its own package is also an option that would simplify the build since there'd be no bootstrapping involved, but it would require some more infrastructure setup upfront.

I think there would probably also need to be some control over which product of a multi-product package gets linked (in addPackageDependency() but that's a detail.

Yeah, right now the add-dependency command doesn't add any of the dependency's products to targets. The PackageSyntax APIs support it, but I couldn't come up with a good CLI interface to specify the dependency product and target it should be added to as a pair.

owenv

comment created time in 2 hours

pull request commentapple/swift-package-manager

[PackageEditor] Mechanical editing functionality for package manifests

Hi @owenv, I wanted to pick back up the conversation about this. Firstly, this is awesome, and will be a great improvement! There are some technical complexities around the dependency on SwiftSyntax, but how hard do you think it would be to allow this to be used from clients of libSwiftPM? I haven't yet read all the diffs closely but it seems that today it would only be useable from the command line. Having it accessible through libSwiftPM (or a parallel library) would really help in order to access this functionality from IDEs, etc.

I think there would probably also need to be some control over which product of a multi-product package gets linked (in addPackageDependency() but that's a detail.

owenv

comment created time in 6 hours

Pull request review commentapple/swift-package-manager

[PackageEditor] Mechanical editing functionality for package manifests

 let package = Package(         .target(             /** High-level commands */             name: "Commands",-            dependencies: ["SwiftToolsSupport-auto", "Basics", "Build", "PackageGraph", "SourceControl", "Xcodeproj", "Workspace", "XCBuildSupport", "ArgumentParser"]),+            dependencies: ["SwiftToolsSupport-auto", "Basics", "Build", "PackageGraph", "SourceControl", "Xcodeproj", "Workspace", "XCBuildSupport", "ArgumentParser"] + (buildPackageSyntax ? ["PackageSyntax"] : []),+            swiftSettings: buildPackageSyntax ?+                [.define("BUILD_PACKAGE_SYNTAX")] :+                [.define("DONT_BUILD_PACKAGE_SYNTAX")]), // Unnecessary, but SPM errors if swiftSettings is empty.

Thanks. Looks like this will be in 5.4, which SwiftPM doesn't require yet, but perhaps could require once 5.4 is released.

owenv

comment created time in 6 hours

pull request commentapple/swift-package-manager

[Draft] Update model to support version-specific manifests

seems reasonable to me

yim-lee

comment created time in 6 hours

push eventapple/swift-package-manager

Boris Bügling

commit sha d9fefd794dc6ccc7f96e6a183a6e2755e162290c

Fix potential crash in repository cache fallback (#3221) It is possible that copying from the cache fails mid-way which can lead to SwiftPM hitting a precondition in the fallback. We should clear any potential leftover data before doing the fallback. rdar://73505115

view details

push time in 7 hours

PR merged apple/swift-package-manager

Reviewers
Fix potential crash in repository cache fallback

Motivation:

It is possible that copying from the cache fails mid-way which can lead to SwiftPM hitting a precondition in the fallback.

Modifications:

We should clear any potential leftover data before doing the fallback.

Result:

Hitting the precondition can be avoided.

rdar://73505115

+2 -0

2 comments

1 changed file

neonichu

pr closed time in 7 hours

push eventapple/swift-package-manager

Saleem Abdulrasool

commit sha f390de5e9ff4f479c079ea4f1dfbbe9150bcf054

PackageCollectionsModel: correct dependency (NFC) (#3224) PackageCollectionsModel has grown a dependency on Foundation, explicitly list it in the build.

view details

push time in 8 hours

PR merged apple/swift-package-manager

Reviewers
PackageCollectionsModel: correct dependency (NFC)

PackageCollectionsModel has grown a dependency on Foundation, explicitly list it in the build.

[One line description of your change]

Motivation:

[Explain here the context, and why you're making that change. What is the problem you're trying to solve.]

Modifications:

[Describe the modifications you've done.]

Result:

[After your change, what will change.]

+2 -0

1 comment

1 changed file

compnerd

pr closed time in 8 hours

pull request commentapple/swift-package-manager

PackageCollectionsModel: correct dependency (NFC)

@swift-ci please smoke test

compnerd

comment created time in 10 hours

PR opened apple/swift-package-manager

PackageCollectionsModel: correct dependency (NFC)

PackageCollectionsModel has grown a dependency on Foundation, explicitly list it in the build.

[One line description of your change]

Motivation:

[Explain here the context, and why you're making that change. What is the problem you're trying to solve.]

Modifications:

[Describe the modifications you've done.]

Result:

[After your change, what will change.]

+2 -0

0 comment

1 changed file

pr created time in 10 hours

Pull request review commentapple/swift-package-manager

Fix potential crash in repository cache fallback

 public class RepositoryManager {             } catch {                 // Fetch without populating the cache in the case of an error.                 print("Skipping cache due to an error: \(error)")+                // It is possible that we already created the directory before failing, so clear leftover data if present.+                try fileSystem.removeFileTree(repositoryPath)

I think what would make the most sense is to leave this as-is but change removeFileTree in TSC to not throw on ENOENT as suggested, but for the 5.4 cherry-pick we inline the ENOENT check to reduce the surface area of the change.

neonichu

comment created time in 10 hours

pull request commentapple/swift-package-manager

Fix potential crash in repository cache fallback

Starting smoke tests. This would be good to get into 5.4 is possible; it definitely seems safe enough.

neonichu

comment created time in 10 hours

pull request commentapple/swift-package-manager

Fix potential crash in repository cache fallback

@swift-ci please smoke test

neonichu

comment created time in 10 hours

Pull request review commentapple/swift-package-manager

Fix potential crash in repository cache fallback

 public class RepositoryManager {             } catch {                 // Fetch without populating the cache in the case of an error.                 print("Skipping cache due to an error: \(error)")+                // It is possible that we already created the directory before failing, so clear leftover data if present.+                try fileSystem.removeFileTree(repositoryPath)

That's possibly a follow-on fix down in TSC, but it looks as if the code in this particular diff is correct. I think it's right that try fileSystem.removeFileTree(repositoryPath) doesn't throw an error if the file isn't already there (much like mkdir doesn't complain if the directory is there). So the code in this PR is right but TSC could use a fix, it looks like. So I have no concerns about this PR. Thanks!

neonichu

comment created time in 10 hours

Pull request review commentapple/swift-package-manager

Fix potential crash in repository cache fallback

 public class RepositoryManager {             } catch {                 // Fetch without populating the cache in the case of an error.                 print("Skipping cache due to an error: \(error)")+                // It is possible that we already created the directory before failing, so clear leftover data if present.+                try fileSystem.removeFileTree(repositoryPath)

Today it does this:

    try FileManager.default.removeItem(atPath: path.pathString)
 }

So the race potential is there.

neonichu

comment created time in 10 hours

created tagapple/swift-integration-tests

tagswift-5.4-DEVELOPMENT-SNAPSHOT-2021-01-23-a

Automated tests for validating the generated Swift snapshots behave correctly

created time in 10 hours

created tagapple/swift-corelibs-libdispatch

tagswift-5.4-DEVELOPMENT-SNAPSHOT-2021-01-23-a

The libdispatch Project, (a.k.a. Grand Central Dispatch), for concurrency on multicore hardware

created time in 10 hours

created tagapple/swift-corelibs-xctest

tagswift-5.4-DEVELOPMENT-SNAPSHOT-2021-01-23-a

The XCTest Project, A Swift core library for providing unit test support

created time in 10 hours

created tagapple/swift-package-manager

tagswift-5.4-DEVELOPMENT-SNAPSHOT-2021-01-23-a

The Package Manager for the Swift Programming Language

created time in 10 hours

created tagapple/swift-llbuild

tagswift-5.4-DEVELOPMENT-SNAPSHOT-2021-01-23-a

A low-level build system, used by Xcode and the Swift Package Manager

created time in 10 hours

pull request commentapple/swift-nio

B2MD: Don't try to reclaim if continuing to parse

<h2>performance report</h2> <p>build id: 64</p> <p>timestamp: Mon Jan 25 15:58:09 UTC 2021</p> <h4>results</h4> <table border="1"> <tr><td>name</td><td>min</td><td>max</td><td>mean</td><td>std</td></tr> <tr> <td>write_http_headers</td> <td>0.004110444</td> <td>0.004279757</td> <td>0.0041426486</td> <td>5.1594001716823265e-05</td> </tr> <tr> <td>bytebuffer_write_12MB_short_string_literals</td> <td>0.483203897</td> <td>0.48940961</td> <td>0.48422683759999996</td> <td>0.0018462399421643216</td> </tr> <tr> <td>bytebuffer_write_12MB_short_calculated_strings</td> <td>0.484577209</td> <td>0.486211294</td> <td>0.48527404949999997</td> <td>0.000601038834323867</td> </tr> <tr> <td>bytebuffer_write_12MB_medium_string_literals</td> <td>0.168960909</td> <td>0.169761712</td> <td>0.16937743860000004</td> <td>0.00022167722977899914</td> </tr> <tr> <td>bytebuffer_write_12MB_medium_calculated_strings</td> <td>0.208064708</td> <td>0.208741362</td> <td>0.208333265</td> <td>0.0002243837979905615</td> </tr> <tr> <td>bytebuffer_write_12MB_large_calculated_strings</td> <td>0.19903573</td> <td>0.199709908</td> <td>0.19931986230000004</td> <td>0.0002053597538197298</td> </tr> <tr> <td>bytebuffer_lots_of_rw</td> <td>0.559071733</td> <td>0.561710337</td> <td>0.5599074786000001</td> <td>0.0007276857946937855</td> </tr> <tr> <td>bytebuffer_write_http_response_ascii_only_as_string</td> <td>0.038047042</td> <td>0.038675837</td> <td>0.03824477159999999</td> <td>0.00020779032732765198</td> </tr> <tr> <td>bytebuffer_write_http_response_ascii_only_as_staticstring</td> <td>0.030059245</td> <td>0.030645248</td> <td>0.030227688800000003</td> <td>0.00016322216770456725</td> </tr> <tr> <td>bytebuffer_write_http_response_some_nonascii_as_string</td> <td>0.038105832</td> <td>0.038983218</td> <td>0.038416592799999996</td> <td>0.0003183787596383341</td> </tr> <tr> <td>bytebuffer_write_http_response_some_nonascii_as_staticstring</td> <td>0.030029501</td> <td>0.03065042</td> <td>0.0302424615</td> <td>0.00017687913723045187</td> </tr> <tr> <td>no-net_http1_10k_reqs_1_conn</td> <td>0.132083112</td> <td>0.137595793</td> <td>0.1334378631</td> <td>0.0015594942860249017</td> </tr> <tr> <td>http1_10k_reqs_1_conn</td> <td>0.606579032</td> <td>0.613005897</td> <td>0.6100503603</td> <td>0.0019594381145977098</td> </tr> <tr> <td>http1_10k_reqs_100_conns</td> <td>0.610457883</td> <td>0.613089193</td> <td>0.611808005</td> <td>0.0009123549885941532</td> </tr> <tr> <td>future_whenallsucceed_100k_immediately_succeeded_off_loop</td> <td>0.092394239</td> <td>0.093070857</td> <td>0.09275236660000001</td> <td>0.0002604555969715789</td> </tr> <tr> <td>future_whenallsucceed_100k_immediately_succeeded_on_loop</td> <td>0.092472594</td> <td>0.100382957</td> <td>0.0937576841</td> <td>0.0023535751681096387</td> </tr> <tr> <td>future_whenallsucceed_100k_deferred_off_loop</td> <td>0.407019185</td> <td>0.409470371</td> <td>0.4082125245</td> <td>0.0008622535394841526</td> </tr> <tr> <td>future_whenallsucceed_100k_deferred_on_loop</td> <td>0.141268801</td> <td>0.143437442</td> <td>0.14206414620000002</td> <td>0.0006939853200084905</td> </tr> <tr> <td>future_whenallcomplete_100k_immediately_succeeded_off_loop</td> <td>0.034778065</td> <td>0.035359316</td> <td>0.0349476349</td> <td>0.0002168941566389012</td> </tr> <tr> <td>future_whenallcomplete_100k_immediately_succeeded_on_loop</td> <td>0.036017551</td> <td>0.036573046</td> <td>0.0362307511</td> <td>0.00018373022640240104</td> </tr> <tr> <td>future_whenallcomplete_100k_deferred_off_loop</td> <td>0.285274367</td> <td>0.294804012</td> <td>0.28981128070000006</td> <td>0.0031980644266867693</td> </tr> <tr> <td>future_whenallcomplete_100k_deferred_on_loop</td> <td>0.072961455</td> <td>0.078486487</td> <td>0.0738115126</td> <td>0.0016608493761774879</td> </tr> <tr> <td>future_reduce_10k_futures</td> <td>0.036888807</td> <td>0.037487007</td> <td>0.0372236473</td> <td>0.0002174108183555068</td> </tr> <tr> <td>future_reduce_into_10k_futures</td> <td>0.036307818</td> <td>0.03694978</td> <td>0.03651703359999999</td> <td>0.00020311200743497877</td> </tr> <tr> <td>channel_pipeline_1m_events</td> <td>0.1699677</td> <td>0.170294511</td> <td>0.17019143009999999</td> <td>9.340931979727299e-05</td> </tr> <tr> <td>websocket_encode_50b_space_at_front_1m_frames_cow</td> <td>0.81658132</td> <td>0.819069825</td> <td>0.8178039197999999</td> <td>0.000952986187808131</td> </tr> <tr> <td>websocket_encode_50b_space_at_front_1m_frames_cow_masking</td> <td>0.087669419</td> <td>0.08812115</td> <td>0.0878824089</td> <td>0.00020097865847809755</td> </tr> <tr> <td>websocket_encode_1kb_space_at_front_100k_frames_cow</td> <td>0.083358341</td> <td>0.084251957</td> <td>0.08361545499999999</td> <td>0.0002899888384560891</td> </tr> <tr> <td>websocket_encode_50b_no_space_at_front_1m_frames_cow</td> <td>0.810503493</td> <td>0.812418093</td> <td>0.8114393487000001</td> <td>0.0006654230995785179</td> </tr> <tr> <td>websocket_encode_1kb_no_space_at_front_100k_frames_cow</td> <td>0.083293199</td> <td>0.083724613</td> <td>0.08351567000000001</td> <td>0.00017753878526426651</td> </tr> <tr> <td>websocket_encode_50b_space_at_front_10k_frames</td> <td>0.010868453</td> <td>0.011251661</td> <td>0.010913308600000001</td> <td>0.00011901555575563299</td> </tr> <tr> <td>websocket_encode_50b_space_at_front_10k_frames_masking</td> <td>0.113256554</td> <td>0.11369838</td> <td>0.1135240974</td> <td>0.00020244812487439</td> </tr> <tr> <td>websocket_encode_1kb_space_at_front_1k_frames</td> <td>0.002041642</td> <td>0.0020556</td> <td>0.0020462544999999997</td> <td>4.192604474282945e-06</td> </tr> <tr> <td>websocket_encode_50b_no_space_at_front_10k_frames</td> <td>0.010645289</td> <td>0.01103475</td> <td>0.010693101899999998</td> <td>0.00012049732844382884</td> </tr> <tr> <td>websocket_encode_1kb_no_space_at_front_1k_frames</td> <td>0.002026564</td> <td>0.002037729</td> <td>0.0020305363000000004</td> <td>3.969302654959248e-06</td> </tr> <tr> <td>websocket_decode_125b_100k_frames</td> <td>0.139727401</td> <td>0.140273986</td> <td>0.14006792569999998</td> <td>0.0002031274469424156</td> </tr> <tr> <td>websocket_decode_125b_with_a_masking_key_100k_frames</td> <td>0.145263884</td> <td>0.145680594</td> <td>0.1455334739</td> <td>0.00017984257240180875</td> </tr> <tr> <td>websocket_decode_64kb_100k_frames</td> <td>0.145383133</td> <td>0.14595106</td> <td>0.14570450769999999</td> <td>0.000195397113183526</td> </tr> <tr> <td>websocket_decode_64kb_with_a_masking_key_100k_frames</td> <td>0.151427443</td> <td>0.152005661</td> <td>0.1517838567</td> <td>0.00018917559549738204</td> </tr> <tr> <td>websocket_decode_64kb_+1_100k_frames</td> <td>0.145209619</td> <td>0.145620094</td> <td>0.14545621700000003</td> <td>0.00016429064365933743</td> </tr> <tr> <td>websocket_decode_64kb_+1_with_a_masking_key_100k_frames</td> <td>0.150802445</td> <td>0.151429169</td> <td>0.15115108260000001</td> <td>0.00020435095025318043</td> </tr> <tr> <td>circular_buffer_into_byte_buffer_1kb</td> <td>0.05089386</td> <td>0.052086627</td> <td>0.0516768102</td> <td>0.00037591330344222265</td> </tr> <tr> <td>circular_buffer_into_byte_buffer_1mb</td> <td>0.094321349</td> <td>0.095236023</td> <td>0.0947759993</td> <td>0.00029347909084939824</td> </tr> <tr> <td>byte_buffer_view_iterator_1mb</td> <td>0.123135838</td> <td>0.123684405</td> <td>0.1234610924</td> <td>0.00022181639385271466</td> </tr> <tr> <td>byte_to_message_decoder_decode_many_small</td> <td>3.0e-08</td> <td>9.2e-08</td> <td>4.3e-08</td> <td>1.7575235101952087e-08</td> </tr> </table> <h4>comparison</h4> <table border="1"> <tr> <td>name</td> <td>current</td> <td>previous</td> <td>winner</td> <td>diff</td> </tr> <tr> <td>write_http_headers</td> <td>0.004110444</td> <td>0.004216906</td> <td>current</td> <td>-2%</td> </tr> <tr> <td>bytebuffer_write_12MB_short_string_literals</td> <td>0.483203897</td> <td>0.510516953</td> <td>current</td> <td>-5%</td> </tr> <tr> <td>bytebuffer_write_12MB_short_calculated_strings</td> <td>0.484577209</td> <td>0.512796961</td> <td>current</td> <td>-5%</td> </tr> <tr> <td>bytebuffer_write_12MB_medium_string_literals</td> <td>0.168960909</td> <td>0.169474784</td> <td>current</td> <td>0%</td> </tr> <tr> <td>bytebuffer_write_12MB_medium_calculated_strings</td> <td>0.208064708</td> <td>0.202418156</td> <td>previous</td> <td>2%</td> </tr> <tr> <td>bytebuffer_write_12MB_large_calculated_strings</td> <td>0.19903573</td> <td>0.199939794</td> <td>current</td> <td>0%</td> </tr> <tr> <td>bytebuffer_lots_of_rw</td> <td>0.559071733</td> <td>0.540619787</td> <td>previous</td> <td>3%</td> </tr> <tr> <td>bytebuffer_write_http_response_ascii_only_as_string</td> <td>0.038047042</td> <td>0.039544275</td> <td>current</td> <td>-3%</td> </tr> <tr> <td>bytebuffer_write_http_response_ascii_only_as_staticstring</td> <td>0.030059245</td> <td>0.029449223</td> <td>previous</td> <td>2%</td> </tr> <tr> <td>bytebuffer_write_http_response_some_nonascii_as_string</td> <td>0.038105832</td> <td>0.039502951</td> <td>current</td> <td>-3%</td> </tr> <tr> <td>bytebuffer_write_http_response_some_nonascii_as_staticstring</td> <td>0.030029501</td> <td>0.029662217</td> <td>previous</td> <td>1%</td> </tr> <tr> <td>no-net_http1_10k_reqs_1_conn</td> <td>0.132083112</td> <td>0.135701725</td> <td>current</td> <td>-2%</td> </tr> <tr> <td>http1_10k_reqs_1_conn</td> <td>0.606579032</td> <td>0.603989638</td> <td>previous</td> <td>0%</td> </tr> <tr> <td>http1_10k_reqs_100_conns</td> <td>0.610457883</td> <td>0.60396369</td> <td>previous</td> <td>1%</td> </tr> <tr> <td>future_whenallsucceed_100k_immediately_succeeded_off_loop</td> <td>0.092394239</td> <td>0.091241933</td> <td>previous</td> <td>1%</td> </tr> <tr> <td>future_whenallsucceed_100k_immediately_succeeded_on_loop</td> <td>0.092472594</td> <td>0.091219595</td> <td>previous</td> <td>1%</td> </tr> <tr> <td>future_whenallsucceed_100k_deferred_off_loop</td> <td>0.407019185</td> <td>0.380725357</td> <td>previous</td> <td>6%</td> </tr> <tr> <td>future_whenallsucceed_100k_deferred_on_loop</td> <td>0.141268801</td> <td>0.138594689</td> <td>previous</td> <td>1%</td> </tr> <tr> <td>future_whenallcomplete_100k_immediately_succeeded_off_loop</td> <td>0.034778065</td> <td>0.034553178</td> <td>previous</td> <td>0%</td> </tr> <tr> <td>future_whenallcomplete_100k_immediately_succeeded_on_loop</td> <td>0.036017551</td> <td>0.035686982</td> <td>previous</td> <td>0%</td> </tr> <tr> <td>future_whenallcomplete_100k_deferred_off_loop</td> <td>0.285274367</td> <td>0.281329017</td> <td>previous</td> <td>1%</td> </tr> <tr> <td>future_whenallcomplete_100k_deferred_on_loop</td> <td>0.072961455</td> <td>0.070851842</td> <td>previous</td> <td>2%</td> </tr> <tr> <td>future_reduce_10k_futures</td> <td>0.036888807</td> <td>0.037094506</td> <td>current</td> <td>0%</td> </tr> <tr> <td>future_reduce_into_10k_futures</td> <td>0.036307818</td> <td>0.036872139</td> <td>current</td> <td>-1%</td> </tr> <tr> <td>channel_pipeline_1m_events</td> <td>0.1699677</td> <td>0.170630752</td> <td>current</td> <td>0%</td> </tr> <tr> <td>websocket_encode_50b_space_at_front_1m_frames_cow</td> <td>0.81658132</td> <td>0.824742835</td> <td>current</td> <td>0%</td> </tr> <tr> <td>websocket_encode_50b_space_at_front_1m_frames_cow_masking</td> <td>0.087669419</td> <td>0.089263756</td> <td>current</td> <td>-1%</td> </tr> <tr> <td>websocket_encode_1kb_space_at_front_100k_frames_cow</td> <td>0.083358341</td> <td>0.085143509</td> <td>current</td> <td>-2%</td> </tr> <tr> <td>websocket_encode_50b_no_space_at_front_1m_frames_cow</td> <td>0.810503493</td> <td>0.824699586</td> <td>current</td> <td>-1%</td> </tr> <tr> <td>websocket_encode_1kb_no_space_at_front_100k_frames_cow</td> <td>0.083293199</td> <td>0.085723877</td> <td>current</td> <td>-2%</td> </tr> <tr> <td>websocket_encode_50b_space_at_front_10k_frames</td> <td>0.010868453</td> <td>0.011008313</td> <td>current</td> <td>-1%</td> </tr> <tr> <td>websocket_encode_50b_space_at_front_10k_frames_masking</td> <td>0.113256554</td> <td>0.114378624</td> <td>current</td> <td>0%</td> </tr> <tr> <td>websocket_encode_1kb_space_at_front_1k_frames</td> <td>0.002041642</td> <td>0.001774451</td> <td>previous</td> <td>15%</td> </tr> <tr> <td>websocket_encode_50b_no_space_at_front_10k_frames</td> <td>0.010645289</td> <td>0.010714274</td> <td>current</td> <td>0%</td> </tr> <tr> <td>websocket_encode_1kb_no_space_at_front_1k_frames</td> <td>0.002026564</td> <td>0.001763565</td> <td>previous</td> <td>14%</td> </tr> <tr> <td>websocket_decode_125b_100k_frames</td> <td>0.139727401</td> <td>0.141211362</td> <td>current</td> <td>-1%</td> </tr> <tr> <td>websocket_decode_125b_with_a_masking_key_100k_frames</td> <td>0.145263884</td> <td>0.146669302</td> <td>current</td> <td>0%</td> </tr> <tr> <td>websocket_decode_64kb_100k_frames</td> <td>0.145383133</td> <td>0.146671928</td> <td>current</td> <td>0%</td> </tr> <tr> <td>websocket_decode_64kb_with_a_masking_key_100k_frames</td> <td>0.151427443</td> <td>0.152412686</td> <td>current</td> <td>0%</td> </tr> <tr> <td>websocket_decode_64kb_+1_100k_frames</td> <td>0.145209619</td> <td>0.145377864</td> <td>current</td> <td>0%</td> </tr> <tr> <td>websocket_decode_64kb_+1_with_a_masking_key_100k_frames</td> <td>0.150802445</td> <td>0.151360845</td> <td>current</td> <td>0%</td> </tr> <tr> <td>circular_buffer_into_byte_buffer_1kb</td> <td>0.05089386</td> <td>0.041163118</td> <td>previous</td> <td>23%</td> </tr> <tr> <td>circular_buffer_into_byte_buffer_1mb</td> <td>0.094321349</td> <td>0.082286637</td> <td>previous</td> <td>14%</td> </tr> <tr> <td>byte_buffer_view_iterator_1mb</td> <td>0.123135838</td> <td>0.127521003</td> <td>current</td> <td>-3%</td> </tr> <tr> <td>byte_to_message_decoder_decode_many_small</td> <td>3.0e-08</td> <td>n/a</td> <td>n/a</td> <td>n/a%</td> </tr> </table> <p style="color: red">significant differences found</p>

weissi

comment created time in 13 hours

pull request commentapple/swift-nio

B2MD: Don't try to reclaim if continuing to parse

@swift-nio-bot test perf please

weissi

comment created time in 13 hours

PR closed apple/swift-nio

Reviewers
Add internal perf framework

Add a lightweight performance framework that can be used to run benchmarks against NIO family projects.

Motivation:

It seems extremely counter-intuitive to have to copy our performance testing code between repositories. This makes life much easier when setting up new projects, and has the added benefit of us being able to modify the framework in one place.

Modifications:

Define a Benchmark protocol inside _NIOPerformanceFramework that can be used in NIO family projects.

Result:

import _NIOPerformanceFramework

let benchmarks = <Make some benchmarks here>
try _PerformanceTester().runBenchmarks(benchmarks)
+106 -0

3 comments

2 changed files

Davidde94

pr closed time in 14 hours

pull request commentapple/swift-nio

B2MD: Don't try to reclaim if continuing to parse

<h2>performance report</h2> <p>build id: 63</p> <p>timestamp: Mon Jan 25 13:59:55 UTC 2021</p> <h4>results</h4> <table border="1"> <tr><td>name</td><td>min</td><td>max</td><td>mean</td><td>std</td></tr> <tr> <td>write_http_headers</td> <td>0.004188882</td> <td>0.004204817</td> <td>0.0041947150999999995</td> <td>5.5367931442507495e-06</td> </tr> <tr> <td>bytebuffer_write_12MB_short_string_literals</td> <td>0.514953796</td> <td>0.522127451</td> <td>0.5167123051000001</td> <td>0.002236250785873321</td> </tr> <tr> <td>bytebuffer_write_12MB_short_calculated_strings</td> <td>0.520044721</td> <td>0.52339962</td> <td>0.5211055441</td> <td>0.0010014937445814941</td> </tr> <tr> <td>bytebuffer_write_12MB_medium_string_literals</td> <td>0.175962492</td> <td>0.177628288</td> <td>0.1767210498</td> <td>0.0005766996865981445</td> </tr> <tr> <td>bytebuffer_write_12MB_medium_calculated_strings</td> <td>0.202629534</td> <td>0.205469466</td> <td>0.20359394809999998</td> <td>0.0010106406203804527</td> </tr> <tr> <td>bytebuffer_write_12MB_large_calculated_strings</td> <td>0.19415415</td> <td>0.194420487</td> <td>0.1943156198</td> <td>8.135899976414604e-05</td> </tr> <tr> <td>bytebuffer_lots_of_rw</td> <td>0.542248367</td> <td>0.544564254</td> <td>0.5430489058999999</td> <td>0.0007397981932335536</td> </tr> <tr> <td>bytebuffer_write_http_response_ascii_only_as_string</td> <td>0.039889707</td> <td>0.041174618</td> <td>0.040291333500000005</td> <td>0.000454849195329923</td> </tr> <tr> <td>bytebuffer_write_http_response_ascii_only_as_staticstring</td> <td>0.031262502</td> <td>0.03194131</td> <td>0.031503238700000005</td> <td>0.0002442903168504646</td> </tr> <tr> <td>bytebuffer_write_http_response_some_nonascii_as_string</td> <td>0.039941293</td> <td>0.040495158</td> <td>0.040127939200000004</td> <td>0.0002033644379623284</td> </tr> <tr> <td>bytebuffer_write_http_response_some_nonascii_as_staticstring</td> <td>0.031217415</td> <td>0.031968574</td> <td>0.0314836934</td> <td>0.00020515826417519148</td> </tr> <tr> <td>no-net_http1_10k_reqs_1_conn</td> <td>0.133780124</td> <td>0.136084141</td> <td>0.13464438019999997</td> <td>0.0007356379342750054</td> </tr> <tr> <td>http1_10k_reqs_1_conn</td> <td>0.603829121</td> <td>0.61215271</td> <td>0.6083909085999999</td> <td>0.003231923366961443</td> </tr> <tr> <td>http1_10k_reqs_100_conns</td> <td>0.604012774</td> <td>0.623513104</td> <td>0.6080245476</td> <td>0.005813451652932747</td> </tr> <tr> <td>future_whenallsucceed_100k_immediately_succeeded_off_loop</td> <td>0.090877671</td> <td>0.09191158</td> <td>0.0912127417</td> <td>0.00030775966562781703</td> </tr> <tr> <td>future_whenallsucceed_100k_immediately_succeeded_on_loop</td> <td>0.091241687</td> <td>0.098690124</td> <td>0.0926539214</td> <td>0.0022518882549987652</td> </tr> <tr> <td>future_whenallsucceed_100k_deferred_off_loop</td> <td>0.386995852</td> <td>0.398895128</td> <td>0.3925746771</td> <td>0.0036058623118188356</td> </tr> <tr> <td>future_whenallsucceed_100k_deferred_on_loop</td> <td>0.141541165</td> <td>0.143600191</td> <td>0.1424996025</td> <td>0.0006042067567035944</td> </tr> <tr> <td>future_whenallcomplete_100k_immediately_succeeded_off_loop</td> <td>0.03437789</td> <td>0.034974447</td> <td>0.0345534998</td> <td>0.0002091861248265649</td> </tr> <tr> <td>future_whenallcomplete_100k_immediately_succeeded_on_loop</td> <td>0.035606377</td> <td>0.036221939</td> <td>0.0358163928</td> <td>0.0001928028137379968</td> </tr> <tr> <td>future_whenallcomplete_100k_deferred_off_loop</td> <td>0.281882343</td> <td>0.297498934</td> <td>0.28721158570000005</td> <td>0.004706797788378821</td> </tr> <tr> <td>future_whenallcomplete_100k_deferred_on_loop</td> <td>0.073219557</td> <td>0.078865588</td> <td>0.074230352</td> <td>0.0016598807447323837</td> </tr> <tr> <td>future_reduce_10k_futures</td> <td>0.038797791</td> <td>0.039668638</td> <td>0.0390382565</td> <td>0.0002754011126695225</td> </tr> <tr> <td>future_reduce_into_10k_futures</td> <td>0.037078079</td> <td>0.037779274</td> <td>0.0373632085</td> <td>0.0002564569574759745</td> </tr> <tr> <td>channel_pipeline_1m_events</td> <td>0.171485898</td> <td>0.171701556</td> <td>0.1716224813</td> <td>5.8584043872505916e-05</td> </tr> <tr> <td>websocket_encode_50b_space_at_front_1m_frames_cow</td> <td>0.816186381</td> <td>0.81688932</td> <td>0.8164317970000001</td> <td>0.00019819959314344205</td> </tr> <tr> <td>websocket_encode_50b_space_at_front_1m_frames_cow_masking</td> <td>0.087609802</td> <td>0.088469753</td> <td>0.08794193430000001</td> <td>0.00029970310319642576</td> </tr> <tr> <td>websocket_encode_1kb_space_at_front_100k_frames_cow</td> <td>0.083690157</td> <td>0.084263189</td> <td>0.0838836645</td> <td>0.00021491278029793495</td> </tr> <tr> <td>websocket_encode_50b_no_space_at_front_1m_frames_cow</td> <td>0.815508139</td> <td>0.815914839</td> <td>0.8156034735000001</td> <td>0.00012021606614041492</td> </tr> <tr> <td>websocket_encode_1kb_no_space_at_front_100k_frames_cow</td> <td>0.083889126</td> <td>0.084432193</td> <td>0.084088338</td> <td>0.00021315585698993144</td> </tr> <tr> <td>websocket_encode_50b_space_at_front_10k_frames</td> <td>0.011019267</td> <td>0.011047573</td> <td>0.0110321167</td> <td>7.808792025659408e-06</td> </tr> <tr> <td>websocket_encode_50b_space_at_front_10k_frames_masking</td> <td>0.114329684</td> <td>0.114771226</td> <td>0.11458314950000001</td> <td>0.0001981640006488741</td> </tr> <tr> <td>websocket_encode_1kb_space_at_front_1k_frames</td> <td>0.002031037</td> <td>0.002053089</td> <td>0.0020371045</td> <td>6.795410448726965e-06</td> </tr> <tr> <td>websocket_encode_50b_no_space_at_front_10k_frames</td> <td>0.010686968</td> <td>0.01074255</td> <td>0.0107030367</td> <td>1.6154452534139943e-05</td> </tr> <tr> <td>websocket_encode_1kb_no_space_at_front_1k_frames</td> <td>0.002033118</td> <td>0.002376736</td> <td>0.0020742831</td> <td>0.00010699955162154237</td> </tr> <tr> <td>websocket_decode_125b_100k_frames</td> <td>0.140457306</td> <td>0.140974864</td> <td>0.14076146909999998</td> <td>0.000195719689875</td> </tr> <tr> <td>websocket_decode_125b_with_a_masking_key_100k_frames</td> <td>0.146419105</td> <td>0.147093115</td> <td>0.14683004430000002</td> <td>0.00021352672161845007</td> </tr> <tr> <td>websocket_decode_64kb_100k_frames</td> <td>0.146359375</td> <td>0.147207732</td> <td>0.1468502024</td> <td>0.0002468203525761656</td> </tr> <tr> <td>websocket_decode_64kb_with_a_masking_key_100k_frames</td> <td>0.152647879</td> <td>0.153413824</td> <td>0.1531535586</td> <td>0.0002667565875746893</td> </tr> <tr> <td>websocket_decode_64kb_+1_100k_frames</td> <td>0.146506825</td> <td>0.147223625</td> <td>0.1469510169</td> <td>0.00022623519824949145</td> </tr> <tr> <td>websocket_decode_64kb_+1_with_a_masking_key_100k_frames</td> <td>0.152236464</td> <td>0.153231858</td> <td>0.1526707551</td> <td>0.00030087874722716824</td> </tr> <tr> <td>circular_buffer_into_byte_buffer_1kb</td> <td>0.041187458</td> <td>0.041749491</td> <td>0.0413160897</td> <td>0.00021978118095407193</td> </tr> <tr> <td>circular_buffer_into_byte_buffer_1mb</td> <td>0.082323931</td> <td>0.082931614</td> <td>0.0825839546</td> <td>0.0002451232957881487</td> </tr> <tr> <td>byte_buffer_view_iterator_1mb</td> <td>0.125886379</td> <td>0.126452869</td> <td>0.1262114806</td> <td>0.0002386997032555081</td> </tr> <tr> <td>byte_to_message_decoder_decode_many_small</td> <td>2.9e-08</td> <td>5.3e-08</td> <td>3.619999999999999e-08</td> <td>7.885288693369302e-09</td> </tr> </table> <h4>comparison</h4> <table border="1"> <tr> <td>name</td> <td>current</td> <td>previous</td> <td>winner</td> <td>diff</td> </tr> <tr> <td>write_http_headers</td> <td>0.004188882</td> <td>0.004216906</td> <td>current</td> <td>0%</td> </tr> <tr> <td>bytebuffer_write_12MB_short_string_literals</td> <td>0.514953796</td> <td>0.510516953</td> <td>previous</td> <td>0%</td> </tr> <tr> <td>bytebuffer_write_12MB_short_calculated_strings</td> <td>0.520044721</td> <td>0.512796961</td> <td>previous</td> <td>1%</td> </tr> <tr> <td>bytebuffer_write_12MB_medium_string_literals</td> <td>0.175962492</td> <td>0.169474784</td> <td>previous</td> <td>3%</td> </tr> <tr> <td>bytebuffer_write_12MB_medium_calculated_strings</td> <td>0.202629534</td> <td>0.202418156</td> <td>previous</td> <td>0%</td> </tr> <tr> <td>bytebuffer_write_12MB_large_calculated_strings</td> <td>0.19415415</td> <td>0.199939794</td> <td>current</td> <td>-2%</td> </tr> <tr> <td>bytebuffer_lots_of_rw</td> <td>0.542248367</td> <td>0.540619787</td> <td>previous</td> <td>0%</td> </tr> <tr> <td>bytebuffer_write_http_response_ascii_only_as_string</td> <td>0.039889707</td> <td>0.039544275</td> <td>previous</td> <td>0%</td> </tr> <tr> <td>bytebuffer_write_http_response_ascii_only_as_staticstring</td> <td>0.031262502</td> <td>0.029449223</td> <td>previous</td> <td>6%</td> </tr> <tr> <td>bytebuffer_write_http_response_some_nonascii_as_string</td> <td>0.039941293</td> <td>0.039502951</td> <td>previous</td> <td>1%</td> </tr> <tr> <td>bytebuffer_write_http_response_some_nonascii_as_staticstring</td> <td>0.031217415</td> <td>0.029662217</td> <td>previous</td> <td>5%</td> </tr> <tr> <td>no-net_http1_10k_reqs_1_conn</td> <td>0.133780124</td> <td>0.135701725</td> <td>current</td> <td>-1%</td> </tr> <tr> <td>http1_10k_reqs_1_conn</td> <td>0.603829121</td> <td>0.603989638</td> <td>current</td> <td>0%</td> </tr> <tr> <td>http1_10k_reqs_100_conns</td> <td>0.604012774</td> <td>0.60396369</td> <td>previous</td> <td>0%</td> </tr> <tr> <td>future_whenallsucceed_100k_immediately_succeeded_off_loop</td> <td>0.090877671</td> <td>0.091241933</td> <td>current</td> <td>0%</td> </tr> <tr> <td>future_whenallsucceed_100k_immediately_succeeded_on_loop</td> <td>0.091241687</td> <td>0.091219595</td> <td>previous</td> <td>0%</td> </tr> <tr> <td>future_whenallsucceed_100k_deferred_off_loop</td> <td>0.386995852</td> <td>0.380725357</td> <td>previous</td> <td>1%</td> </tr> <tr> <td>future_whenallsucceed_100k_deferred_on_loop</td> <td>0.141541165</td> <td>0.138594689</td> <td>previous</td> <td>2%</td> </tr> <tr> <td>future_whenallcomplete_100k_immediately_succeeded_off_loop</td> <td>0.03437789</td> <td>0.034553178</td> <td>current</td> <td>0%</td> </tr> <tr> <td>future_whenallcomplete_100k_immediately_succeeded_on_loop</td> <td>0.035606377</td> <td>0.035686982</td> <td>current</td> <td>0%</td> </tr> <tr> <td>future_whenallcomplete_100k_deferred_off_loop</td> <td>0.281882343</td> <td>0.281329017</td> <td>previous</td> <td>0%</td> </tr> <tr> <td>future_whenallcomplete_100k_deferred_on_loop</td> <td>0.073219557</td> <td>0.070851842</td> <td>previous</td> <td>3%</td> </tr> <tr> <td>future_reduce_10k_futures</td> <td>0.038797791</td> <td>0.037094506</td> <td>previous</td> <td>4%</td> </tr> <tr> <td>future_reduce_into_10k_futures</td> <td>0.037078079</td> <td>0.036872139</td> <td>previous</td> <td>0%</td> </tr> <tr> <td>channel_pipeline_1m_events</td> <td>0.171485898</td> <td>0.170630752</td> <td>previous</td> <td>0%</td> </tr> <tr> <td>websocket_encode_50b_space_at_front_1m_frames_cow</td> <td>0.816186381</td> <td>0.824742835</td> <td>current</td> <td>-1%</td> </tr> <tr> <td>websocket_encode_50b_space_at_front_1m_frames_cow_masking</td> <td>0.087609802</td> <td>0.089263756</td> <td>current</td> <td>-1%</td> </tr> <tr> <td>websocket_encode_1kb_space_at_front_100k_frames_cow</td> <td>0.083690157</td> <td>0.085143509</td> <td>current</td> <td>-1%</td> </tr> <tr> <td>websocket_encode_50b_no_space_at_front_1m_frames_cow</td> <td>0.815508139</td> <td>0.824699586</td> <td>current</td> <td>-1%</td> </tr> <tr> <td>websocket_encode_1kb_no_space_at_front_100k_frames_cow</td> <td>0.083889126</td> <td>0.085723877</td> <td>current</td> <td>-2%</td> </tr> <tr> <td>websocket_encode_50b_space_at_front_10k_frames</td> <td>0.011019267</td> <td>0.011008313</td> <td>previous</td> <td>0%</td> </tr> <tr> <td>websocket_encode_50b_space_at_front_10k_frames_masking</td> <td>0.114329684</td> <td>0.114378624</td> <td>current</td> <td>0%</td> </tr> <tr> <td>websocket_encode_1kb_space_at_front_1k_frames</td> <td>0.002031037</td> <td>0.001774451</td> <td>previous</td> <td>14%</td> </tr> <tr> <td>websocket_encode_50b_no_space_at_front_10k_frames</td> <td>0.010686968</td> <td>0.010714274</td> <td>current</td> <td>0%</td> </tr> <tr> <td>websocket_encode_1kb_no_space_at_front_1k_frames</td> <td>0.002033118</td> <td>0.001763565</td> <td>previous</td> <td>15%</td> </tr> <tr> <td>websocket_decode_125b_100k_frames</td> <td>0.140457306</td> <td>0.141211362</td> <td>current</td> <td>0%</td> </tr> <tr> <td>websocket_decode_125b_with_a_masking_key_100k_frames</td> <td>0.146419105</td> <td>0.146669302</td> <td>current</td> <td>0%</td> </tr> <tr> <td>websocket_decode_64kb_100k_frames</td> <td>0.146359375</td> <td>0.146671928</td> <td>current</td> <td>0%</td> </tr> <tr> <td>websocket_decode_64kb_with_a_masking_key_100k_frames</td> <td>0.152647879</td> <td>0.152412686</td> <td>previous</td> <td>0%</td> </tr> <tr> <td>websocket_decode_64kb_+1_100k_frames</td> <td>0.146506825</td> <td>0.145377864</td> <td>previous</td> <td>0%</td> </tr> <tr> <td>websocket_decode_64kb_+1_with_a_masking_key_100k_frames</td> <td>0.152236464</td> <td>0.151360845</td> <td>previous</td> <td>0%</td> </tr> <tr> <td>circular_buffer_into_byte_buffer_1kb</td> <td>0.041187458</td> <td>0.041163118</td> <td>previous</td> <td>0%</td> </tr> <tr> <td>circular_buffer_into_byte_buffer_1mb</td> <td>0.082323931</td> <td>0.082286637</td> <td>previous</td> <td>0%</td> </tr> <tr> <td>byte_buffer_view_iterator_1mb</td> <td>0.125886379</td> <td>0.127521003</td> <td>current</td> <td>-1%</td> </tr> <tr> <td>byte_to_message_decoder_decode_many_small</td> <td>2.9e-08</td> <td>n/a</td> <td>n/a</td> <td>n/a%</td> </tr> </table> <p style="color: red">significant differences found</p>

weissi

comment created time in 15 hours

pull request commentapple/swift-nio

Add internal perf framework

@Lukasa making it a separate repository isn't something I'm against, but copying around code is - it's a fairly common concept that files shouldn't just be copied around like this. I'm not saying it's a major issue, but it's one that's easily solvable with minimal hassle.

Davidde94

comment created time in 15 hours

pull request commentapple/swift-nio

B2MD: Don't try to reclaim if continuing to parse

@swift-nio-bot test perf please

weissi

comment created time in 15 hours

pull request commentapple/swift-nio

Add internal perf framework

I think we don't necessarily agree on how bad the "copy around code" situation is.

The strong advantage of copying around the code is that there's no versioning problem. There is no risk of breaking API and no concern about behaviours changing, because each repository has a snapshot of the benchmark tool that will work for it. Given that the benchmark code is extremely small and doesn't change much, it doesn't present much of a burden.

So I think we need to make an affirmative statement of what problem, exactly, depending on this will solve.

Davidde94

comment created time in 15 hours

PR closed apple/swift-nio

Reviewers
[WIP] Add stream support to ChannelHandlerContext

Resolves #14

Motivation:

The API is easier to use when outbound writing a variety of data types.

Modifications:

Add a new Stream class, and a new method writeOutboundWithStream to ChannelHandlerContext. You provide a closure as the last argument to this method, which takes a Stream as it's only argument. You then write to the stream using the new <<< operator.

Multiple types are supported including:

  • String
  • StaticString
  • Substring
  • [UInt8]
  • FixedWidthInteger

Result:

var buffer = ByteBuffer()
buffer.writeString("Hello")
buffer.writeString("world")
buffer.writeInteger(1234)
context.writeAndFlush(NIOAny(buffer), promise: promise)

turns into

context.writeOutboundWithStream(promise: promise) { stream in
    stream <<< "hello" <<< "world" <<< 1234
}
+171 -0

1 comment

3 changed files

Davidde94

pr closed time in 15 hours

more