profile
viewpoint
David Ungar davidungar I love to program.

russellallen/self 519

Making the world safe for objects

smarr/RoarVM 163

RoarVM is a manycore Smalltalk Virtual Machine

krono/self 16

Making the world safe for objects

davidungar/miniPromiseKit 10

implementing a simplified, subset of PromiseKit for an upcoming book

davidungar/RoarVM 2

RoarVM is a manycore Smalltalk Virtual Machine

davidungar/Swift-dynamic-atomicity 1

Work in progress on optimizing Swift reference counting by using dynamic thread-escape tracking. Exploratory code, not production-quality.

davidungar/swift 0

The Swift Programming Language

davidungar/swift-corelibs-foundation 0

The Foundation Project, providing core utilities, internationalization, and OS independence

push eventdavidungar/swift-driver

David Ungar

commit sha fa144ec3f11fbc34f8f73dee1fce894540154e6d

WIP

view details

David Ungar

commit sha 4e208dff6c2872985eb02c70c268a695c48e1fee

test ignores fatal errors, still has extra junk in brent's check

view details

David Ungar

commit sha 1955f1d1a585008809d6971ee0ffa79de5269181

brute force

view details

David Ungar

commit sha f573cf9f3105173cbb8a1636a0431d03d566a33b

WIP

view details

push time in 12 hours

push eventdavidungar/swift-driver

David Ungar

commit sha 15882bfb51f51740bbad6590cd0097fa2daf0099

test passes, but driver fails

view details

push time in a day

push eventdavidungar/swift-driver

David Ungar

commit sha 0ffbe00b7bfbea6dc5bbe6bdbb6125a6fedd32e2

making linker inputs, but what of skipped jobs?

view details

David Ungar

commit sha 6c2179976cb3b7347f2fb0910c1d55523a2b943e

smuggles skipped jobs out and exits too soon; also wrong arch?

view details

push time in a day

push eventdavidungar/swift-driver

David Ungar

commit sha c97fdcd06f68afbe8ee7a05e45a989d6af9f636b

fixed 1

view details

David Ungar

commit sha 5d52f311b847f0267e0510e8c763014ba5bb498d

first wave scheduling OK

view details

push time in 2 days

push eventdavidungar/swift-driver

David Ungar

commit sha a4258df356865ce1de17e855b8cf57fd99ff4f65

fix bug

view details

push time in 2 days

push eventdavidungar/swift-driver

David Ungar

commit sha 9df301cf00cede0a1e9ee5b48fc9a6625df4ac11

Nondet

view details

push time in 2 days

push eventdavidungar/swift-driver

David Ungar

commit sha df5fc6c2003496a4f3cd800a1f7a08971082b112

2

view details

push time in 2 days

push eventdavidungar/swift-driver

David Ungar

commit sha 1041e1b591b4f7387687a9dba0bb32a9f02d8450

sort inputs

view details

David Ungar

commit sha 61fad08fa761047432a7cfdcfb08c266ef7ce21d

Woiks

view details

David Ungar

commit sha 3e480ad85d1a519f824a9ba71cf67743b01a0490

2

view details

push time in 2 days

push eventdavidungar/swift-driver

David Ungar

commit sha 23b3dce6265f090c9aaec9af4670a86f84d11e32

WIP

view details

push time in 2 days

push eventdavidungar/swift-driver

David Ungar

commit sha 95bb28be36c13f45272390f2b1af6a8774b6bce1

WIP

view details

push time in 2 days

push eventdavidungar/swift-driver

David Ungar

commit sha 5990a2dd1e72ad51b16a425fa8fd92a6f7740f48

wip test

view details

push time in 2 days

create barnchdavidungar/swift-driver

branch : incremental-10-18

created branch time in 2 days

push eventapple/swift-driver

David Ungar

commit sha 25e4b14587a01bc9f84510e178d7896b91aea8e8

show job lifecycle etc

view details

David Ungar

commit sha aab6ab011824f2fbbce52ed31cc7fd6b36079ba5

added simple test

view details

David Ungar

commit sha da72836c59a26d6e924bec2fccd3689984ca70f2

refine tests

view details

David Ungar

commit sha fbd3e4f629718b80b7bfa95640a6a06f0c810c60

Handle autolink-extracting platforms

view details

David Ungar

commit sha 750869db76f1d1603788e238be97d9c3086008b6

Merge pull request #311 from davidungar/incremental-10-16-show [Incremental] Testable debugging outputs & tests

view details

push time in 2 days

PR merged apple/swift-driver

Reviewers
[Incremental] Testable debugging outputs & tests
  1. Centralize the parsedOptions interrogation for -driver-show-job-lifecycle
  2. Use diagnostic remarks for -driver-show-job-lifecycle and -driver-show-incremental Why? Much easier to test, may not mess up Xcode's reading of the compiler's output. Why not? We'll have to fix some integration tests, as it differs from legacy front end.
  3. Add tests for these outputs
  4. Call back into BuildRecordInfo and IncrementalCompilationState when jobs finish.

Robert floated, and I like the idea of mimicking the legacy driver's messages. However, since we don't have the same sort of task queue (at the present), I'm not sure it's feasible.

+154 -26

5 comments

8 changed files

davidungar

pr closed time in 2 days

pull request commentapple/swift-driver

[Incremental] Testable debugging outputs & tests

Thanks, @owenv !

davidungar

comment created time in 2 days

pull request commentapple/swift-driver

[Incremental] Testable debugging outputs & tests

@swift-ci please test

davidungar

comment created time in 2 days

push eventdavidungar/swift-driver

David Ungar

commit sha fbd3e4f629718b80b7bfa95640a6a06f0c810c60

Handle autolink-extracting platforms

view details

push time in 2 days

pull request commentapple/swift-driver

[Incremental] Testable debugging outputs & tests

This looks good to me once the Linux tests are fixed - It looks like the it's failing because Linux adds an autolink-extract job to one of the test cases.

I did a quick search and found 15 integration tests that could depend on the old-style debugging output (because they use -driver-show-job-lifecycle or -driver-show-incremental). Updating those at some point seems manageable but might be a little more work than trying to accurately replicate the C++ output

Thanks, Owen, for looking at this. I'll take a whack at the Linux tests. Regarding the debugging output, I'm not sure whether we want a separate task queue, like the one in the legacy driver. Seems like it might be redundant with using llbuild. If we don't, then it might be undesirable to pretend there is one in the debugging output. I'm still wrestling with exactly how to get llbuild to work for us. The old driver batches up jobs on-demand as cores free up, depending on which jobs in the 2nd wave have already been scheduled.

davidungar

comment created time in 2 days

pull request commentapple/swift-driver

[Incremental] Testable debugging outputs & tests

@swift-ci please test

davidungar

comment created time in 3 days

push eventdavidungar/swift-driver

David Ungar

commit sha da72836c59a26d6e924bec2fccd3689984ca70f2

refine tests

view details

push time in 3 days

push eventdavidungar/swift-driver

David Ungar

commit sha aab6ab011824f2fbbce52ed31cc7fd6b36079ba5

added simple test

view details

push time in 4 days

pull request commentapple/swift-driver

[Incremental, WIP, DNM] Testable debugging outputs

@swift-ci please test

davidungar

comment created time in 4 days

PR opened apple/swift-driver

[Incremental, WIP, DNM] Testable debugging outputs
  1. Centralize the parsedOptions interrogation for -driver-show-job-lifecycle
  2. Use diagnostic remarks for -driver-show-job-lifecycle and -driver-show-incremental Why? Much easier to test, may not mess up Xcode's reading of the compiler's output. Why not? We'll have to fix some integration tests, as it differs from legacy front end.
  3. Add tests for these outputs (not done yet)
  4. Call back into BuildRecordInfo and IncrementalCompilationState when jobs finish.
+73 -21

0 comment

6 changed files

pr created time in 4 days

create barnchdavidungar/swift-driver

branch : incremental-10-16-show

created branch time in 4 days

push eventdavidungar/swift-driver

David Ungar

commit sha 5acd61fab5a5fee82f5473fad2b49c6ad4552c95

Remarks for show job lifecycle

view details

David Ungar

commit sha 6d09f2ef9cff0a1e72c6921eb522d443703cd17a

Tracing via diags

view details

push time in 4 days

PullRequestReviewEvent

push eventdavidungar/swift-driver

David Ungar

commit sha 5190803f06d4655faaaf0fe55c0e29da801da222

Scheduling 2nd wave with stub

view details

David Ungar

commit sha db97e53e94c5fec93649c22829020d6e98d9c9cc

Only plan for first wave

view details

push time in 6 days

create barnchdavidungar/swift-driver

branch : incremental-10-14

created branch time in 6 days

push eventdavidungar/swift-driver

David Ungar

commit sha 4783b01420f41d4398bd3e53e0aa192b47bb8d0b

1st wave, 2nd WIP

view details

push time in 6 days

create barnchdavidungar/swift-driver

branch : incremental-10-13

created branch time in 6 days

delete branch davidungar/swift-driver

delete branch : incremental-10-11b

delete time in 7 days

push eventapple/swift-driver

David Ungar

commit sha 55ed2e7712bdbe99c8630e40a66d970fb0350cd2

Move JobTracker.swift to FileTracker.swift

view details

David Ungar

commit sha aa4f9a838cc7bf9dc0836fc3244d707415c25305

Rename jobTracker to fileTracker, redo ModuleDependencyGraph

view details

David Ungar

commit sha 6b338155e602ccc8d110b7b23157297f5cd71d0c

Fix two blemishes

view details

David Ungar

commit sha 85e9701f092b83bc657827357f42e64f81f9aea0

Replayce FileTracker.swift with BidirectionalMap.swift

view details

David Ungar

commit sha 16829d1d399b429508dce09c7e8890448dee600c

Merge pull request #304 from davidungar/incremental-10-11b Prepare for scheduling first wave.

view details

push time in 7 days

PR merged apple/swift-driver

Reviewers
Prepare for scheduling first wave.

Evolve IncrementalCompilationState and ModuleDependencyGraph in preparation for scheduling the first wave. A future PR might move the sourcefile/swiftDeps boundary line a bit. At present it's mostly inside the ModuleDependencyGraph.

+1072 -816

4 comments

11 changed files

davidungar

pr closed time in 7 days

pull request commentapple/swift-driver

Prepare for scheduling first wave.

@swift-ci please test

davidungar

comment created time in 7 days

push eventdavidungar/swift-driver

David Ungar

commit sha 85e9701f092b83bc657827357f42e64f81f9aea0

Replayce FileTracker.swift with BidirectionalMap.swift

view details

push time in 7 days

pull request commentapple/swift-driver

Prepare for scheduling first wave.

@swift-ci please test

davidungar

comment created time in 7 days

push eventdavidungar/swift-driver

David Ungar

commit sha 6b338155e602ccc8d110b7b23157297f5cd71d0c

Fix two blemishes

view details

push time in 7 days

pull request commentapple/swift-driver

Prepare for scheduling first wave.

This looks good to me! But I'll let an expert (@CodaFi) weigh in for approval.

Appreciate you taking a look! Thank you.

davidungar

comment created time in 7 days

Pull request review commentapple/swift-driver

Prepare for scheduling first wave.

+//===------------------ FileTracker.swift ---------------------------------===//+//+// This source file is part of the Swift.org open source project+//+// Copyright (c) 2020 Apple Inc. and the Swift project authors+// Licensed under Apache License v2.0 with Runtime Library Exception+//+// See https://swift.org/LICENSE.txt for license information+// See https://swift.org/CONTRIBUTORS.txt for the list of Swift project authors+//+//===----------------------------------------------------------------------===//++extension ModuleDependencyGraph {+  // TODO: Incremental, check the relationship between batching and incremental compilation+  // Must delay batching until *after* incremental decisions++  ///Since the rest of the driver trucks in jobs, the correspondence must be tracked+  @_spi(Testing) public struct FileTracker {++    /// Keyed by swiftdeps filename, so we can get back to source files.+    private(set) var sourceFilesBySwiftDeps: [SwiftDeps: TypedVirtualPath] = [:]+    private(set) var swiftDepsBySourceFile: [TypedVirtualPath: SwiftDeps] = [:]+++    @_spi(Testing) public mutating func register(source: TypedVirtualPath,+                                                 swiftDeps: SwiftDeps) {+      sourceFilesBySwiftDeps[swiftDeps] = source+      swiftDepsBySourceFile[source] = swiftDeps+    }++    @_spi(Testing) public func swiftDeps(for sourceFile: TypedVirtualPath+    ) -> SwiftDeps {+      guard let swiftDeps = swiftDepsBySourceFile[sourceFile]+      else {+      fatalError("\(sourceFile) was not registered")

Again, thanks!

davidungar

comment created time in 7 days

PullRequestReviewEvent

Pull request review commentapple/swift-driver

Prepare for scheduling first wave.

 extension Diagnostic.Message {     )   }   static func remark_incremental_compilation_disabled(because why: String) -> Diagnostic.Message {-    .remark("incremental compilation has been disabled, because \(why).\n")+    .remark("Incremental compilation has been disabled, because \(why)")+  }+  static func remark_incremental_decision(because why: String) -> Diagnostic.Message {+    .remark("Incremental compilation decision: \(why)")+  }+}+++// MARK: - Scheduling the first wave++/// It is convenient to record the inputs to be compiled immediately by returning the inputs to be skipped until+/// further notice.+extension IncrementalCompilationState {+  private static func calculateScheduledAndSkippedInputs(+    buildRecordInfo: BuildRecordInfo,+    moduleDependencyGraph: ModuleDependencyGraph,+    outOfDateBuildRecord: BuildRecord,+    reportIncrementalDecision: (String) -> Void,+    reportIncrementalQueuing: (String, VirtualPath) -> Void+  ) -> (scheduled: Set<TypedVirtualPath>, skipped: Set<TypedVirtualPath>) {++    let changedInputs: [(TypedVirtualPath, InputInfo.Status)] = computeChangedInputs(+      buildRecordInfo: buildRecordInfo,+      moduleDependencyGraph: moduleDependencyGraph,+      outOfDateBuildRecord: outOfDateBuildRecord,+      reportIncrementalDecision: reportIncrementalDecision+    )+    let externalDependents = computeExternallyDependentInputs(+      buildTime: outOfDateBuildRecord.buildTime,+      fileSystem: buildRecordInfo.fileSystem,+      moduleDependencyGraph: moduleDependencyGraph,+      reportIncrementalDecision: reportIncrementalDecision)+    // Combine the first to obtain the inputs that definitely must be recompiled.+    let inputsRequiringCompilation = Set(changedInputs.map {$0.0} + externalDependents)+    for scheduledInput in inputsRequiringCompilation.sorted(by: {$0.file.name < $1.file.name}) {+      reportIncrementalQueuing("Queuing (initial):", scheduledInput.file)+    }++    // Sometimes, schedule inputs that depend on the changed inputs for the+    // first wave, even though they may not require compilation.+    // Any such inputs missed, will be found by the rereading of swiftDeps+    // as each first wave job finished.+    let speculativelyScheduledInputs = computeDependentsToSpeculativelySchedule(+      changedInputs: changedInputs,+      moduleDependencyGraph: moduleDependencyGraph,+      reportIncrementalDecision: reportIncrementalDecision)+    let additions = inputsRequiringCompilation.subtracting(speculativelyScheduledInputs)+    for addition in additions.sorted(by: {$0.file.name < $1.file.name}) {+      reportIncrementalQueuing("Queueing (dependent):", addition.file)+    }+    let scheduledInputs = inputsRequiringCompilation.union(additions)++    let skippedInputs = Set(buildRecordInfo.compilationInputModificationDates.keys)+      .subtracting(scheduledInputs)+    for skippedInput in skippedInputs.sorted(by: {$0.file.name < $1.file.name})  {+      reportIncrementalQueuing("Skipping:", skippedInput.file)+    }+    return (scheduled: scheduledInputs, skipped: skippedInputs)+  }++  /// Find the inputs that have changed since last compilation, or were marked as needed a builid

Thanks!

davidungar

comment created time in 7 days

PullRequestReviewEvent

pull request commentapple/swift-driver

Prepare for scheduling first wave.

@swift-ci Please test

davidungar

comment created time in 7 days

PR opened apple/swift-driver

Prepare for scheduling first wave.

Evolve IncrementalCompilationState and ModuleDependencyGraph in preparation for scheduling the first wave.

+1080 -816

0 comment

11 changed files

pr created time in 7 days

push eventdavidungar/swift-driver

David Ungar

commit sha aa4f9a838cc7bf9dc0836fc3244d707415c25305

Rename jobTracker to fileTracker, redo ModuleDependencyGraph

view details

push time in 7 days

push eventdavidungar/swift-driver

David Ungar

commit sha 7558d9954ec99985fbe23afbc93e58aba7e1bb41

Evolve module dependency graph

view details

push time in 7 days

push eventdavidungar/swift-driver

David Ungar

commit sha 4ea99f2827b4b405a7dd6cb7d442d9f7aa47c7ab

WIP module dep graph, its tests compile errors

view details

push time in 7 days

push eventdavidungar/swift-driver

David Ungar

commit sha e93ad3ea35253d2b7a4327c9843b0420eacb012f

WIP

view details

push time in 8 days

create barnchdavidungar/swift-driver

branch : incremental-10-11b

created branch time in 8 days

Pull request review commentapple/swift

[CodeCompletion] Remove parser hacks regarding type body fingerprints

 static bool areAnyDependentFilesInvalidated(       }); } +/// Get interface hash of \p SF including the type members in the file.+///+/// This is needed because SouceFile::getIntefaceHash() doesn't digest type body+/// fingerprints in the file.+void getInterfaceHashIncludingTypeMembers(SourceFile *SF,+                                          llvm::SmallString<32> &str) {+  /// FIXME: Gross. Hashing multiple "hash" values.+  llvm::MD5 hash;+  SF->getInterfaceHash(str);+  hash.update(str);++  std::function<void(IterableDeclContext *)> hashTypeBodyFingerprints =+      [&](IterableDeclContext *IDC) {+        if (auto fp = IDC->getBodyFingerprint())+          hash.update(*fp);+        for (auto *member : IDC->getParsedMembers())+          if (auto *childIDC = dyn_cast<IterableDeclContext>(member))+            hashTypeBodyFingerprints(childIDC);+      };++  for (auto *D : SF->getTopLevelDecls()) {+    if (auto IDC = dyn_cast<IterableDeclContext>(D))+      hashTypeBodyFingerprints(IDC);+  }++  llvm::MD5::MD5Result result;+  hash.final(result);+  str = result.digest();+}+

Thanks for explaining. That would make a great comment in some future PR, if it's not there already.

rintaro

comment created time in 8 days

PullRequestReviewEvent

push eventapple/swift-driver

David Ungar

commit sha 9bc4730afb768b58e4e45630bbcdf6b78c581b32

write build record

view details

David Ungar

commit sha 8ac661a1b886599775416f47becd028d0718bab3

First cut at a test

view details

David Ungar

commit sha 07b2b171e9def074507a640ce3da466d3be84574

Fixed up test

view details

David Ungar

commit sha 5e10d0d11d108f468f7abef6f27d7709fa25105e

Replace \c with backticks

view details

David Ungar

commit sha 5a1c9c1a9919e5ad51c5cea68eaa23e464cf16df

Remove support for "moduleonly" build record.

view details

David Ungar

commit sha 73cafdecfa7abb9fd895c09b9e50e6e6185fe917

Start diagnostics in lower case

view details

David Ungar

commit sha 9bd04adb459d085e7d6d53543775bc548dc57197

Move InputInfoMap.swift to BuildRecord.swift

view details

David Ungar

commit sha ba20519e05ef1f47c636314a5c45fea72b07d92c

Rename InputInfoMap to BuildRecord

view details

David Ungar

commit sha 4800715195874848cf6a86e1ffe31387d79829a2

Rename outOfDateMap to outOfDateBuildRecord

view details

David Ungar

commit sha e32fd167639c73ac7f85e544742489a2e063a48f

Centralize section name strings

view details

David Ungar

commit sha 18ba5b874e62ff05d46f3023a62cabf043e38103

Fix CMakeLists.txt

view details

David Ungar

commit sha ac4af7a1b25a970cc6d38f8b98b10d05b8447faa

Merge pull request #298 from davidungar/incremental-10-9c-write-build-rec [Incremental] Write build record & test

view details

push time in 9 days

PR merged apple/swift-driver

Reviewers
[Incremental] Write build record & test

Adds BuildRecordInfo to hold information needed to read and write build records. Writes build records whether incremental or not. Reads when incremental. Tests reading and writing.

+604 -338

7 comments

17 changed files

davidungar

pr closed time in 9 days

pull request commentapple/swift-driver

[Incremental] Write build record & test

@swift-ci please test

davidungar

comment created time in 9 days

push eventdavidungar/swift-driver

David Ungar

commit sha 18ba5b874e62ff05d46f3023a62cabf043e38103

Fix CMakeLists.txt

view details

push time in 9 days

Pull request review commentapple/swift-driver

[Incremental] Write build record & test

+//===--------------- BuildRecordInfo.swift --------------------------------===//

Thank you--yes, I remembered that too late, and I really appreciate the reminder. I will fix those.

davidungar

comment created time in 9 days

PullRequestReviewEvent

pull request commentapple/swift-driver

[Incremental] Write build record & test

@swift-ci please test

davidungar

comment created time in 9 days

push eventdavidungar/swift-driver

David Ungar

commit sha e32fd167639c73ac7f85e544742489a2e063a48f

Centralize section name strings

view details

push time in 9 days

pull request commentapple/swift-driver

[Incremental] Write build record & test

@swift-ci Thanks, @owenv , for your help. I think I've fixed the problems. Would you like to take another look?

davidungar

comment created time in 9 days

pull request commentapple/swift-driver

[Incremental] Write build record & test

@swift-ci please test

davidungar

comment created time in 9 days

Pull request review commentapple/swift-driver

[Incremental] Write build record & test

+//===--------------- BuildRecordInfo.swift --------------------------------===//+//+// This source file is part of the Swift.org open source project+//+// Copyright (c) 2014 - 2019 Apple Inc. and the Swift project authors+// Licensed under Apache License v2.0 with Runtime Library Exception+//+// See https://swift.org/LICENSE.txt for license information+// See https://swift.org/CONTRIBUTORS.txt for the list of Swift project authors+//+//===----------------------------------------------------------------------===//++import Foundation+import TSCBasic+import SwiftOptions++/// Holds information required to read and write the build record (aka compilation record)+/// This info is always written, but only read for incremental compilation.+@_spi(Testing) public class BuildRecordInfo {+  let buildRecordPath: VirtualPath+  let fileSystem: FileSystem+  let argsHash: String+  let actualSwiftVersion: String+  let timeBeforeFirstJob: Date+  let diagnosticEngine: DiagnosticsEngine+  let outputBuildRecordForModuleOnlyBuild: Bool+  let compilationInputModificationDates: [TypedVirtualPath: Date]++  var finishedJobResults  = [Job: ProcessResult]()++  init?(+    actualSwiftVersion: String,+    compilerOutputType: FileType?,+    diagnosticEngine: DiagnosticsEngine,+    fileSystem: FileSystem,+    moduleOutputInfo: ModuleOutputInfo,+    outputFileMap: OutputFileMap?,+    parsedOptions: ParsedOptions,+    recordedInputModificationDates: [TypedVirtualPath: Date]+  ) {+    // Cannot write a buildRecord without a path.+    guard let buildRecordPath = Self.computeBuildRecordPath(+            outputFileMap: outputFileMap,+            compilerOutputType: compilerOutputType,+            diagnosticEngine: diagnosticEngine)+    else {+      return nil+    }+    self.actualSwiftVersion = actualSwiftVersion+    self.argsHash = Self.computeArgsHash(parsedOptions)+    self.buildRecordPath = buildRecordPath+    self.compilationInputModificationDates =+      recordedInputModificationDates.filter { input, _ in+        input.type.isPartOfSwiftCompilation+      }+    self.diagnosticEngine = diagnosticEngine+    self.fileSystem = fileSystem+    // If we emit module along with full compilation, emit build record+    // file for '-emit-module' only mode as well.+    self.outputBuildRecordForModuleOnlyBuild = moduleOutputInfo.output?.isTopLevel ?? false+    self.timeBeforeFirstJob = Date()+   }++  private static func computeArgsHash(_ parsedOptionsArg: ParsedOptions+  ) -> String {+    var parsedOptions = parsedOptionsArg+    let hashInput = parsedOptions+      .filter { $0.option.affectsIncrementalBuild && $0.option.kind != .input}+      .map { $0.option.spelling }+      .sorted()+      .joined()+    return SHA256().hash(hashInput).hexadecimalRepresentation+  }++  /// Determine the input and output path for the build record+  private static func computeBuildRecordPath(+    outputFileMap: OutputFileMap?,+    compilerOutputType: FileType?,+    diagnosticEngine: DiagnosticsEngine+  ) -> VirtualPath? {+    // FIXME: This should work without an output file map. We should have+    // another way to specify a build record and where to put intermediates.+    guard let ofm = outputFileMap else {+      return nil+    }+    guard let partialBuildRecordPath =+            ofm.existingOutputForSingleInput(outputType: .swiftDeps)+    else {+      diagnosticEngine.emit(.warning_incremental_requires_build_record_entry)+      return nil+    }+    // In 'emit-module' only mode, use build-record filename suffixed with+    // '~moduleonly', so that module-only mode doesn't mess up build-record+    // file for full compilation.+    return compilerOutputType == .swiftModule+      ? partialBuildRecordPath.appendingToBaseName("~moduleonly")+      : partialBuildRecordPath+  }++  /// Write out the build record.+  /// \c Jobs must include all of the compilation jobs.+  /// \c Inputs will hold all the primary inputs that were not compiled because of incremental compilation+  func writeBuildRecord(_ jobs: [Job], _ skippedInputs: Set<TypedVirtualPath>? ) {+    let buildRecord = InputInfoMap(+      jobs: jobs,+      finishedJobResults: finishedJobResults,+      skippedInputs: skippedInputs,+      compilationInputModificationDates: compilationInputModificationDates,+      actualSwiftVersion: actualSwiftVersion,+      argsHash: argsHash,+      timeBeforeFirstJob: timeBeforeFirstJob)++    let contents: String+    do {  contents = try buildRecord.encode() }+    catch let InputInfoMap.Errors.notAbsolutePath(p) {+      diagnosticEngine.emit(+        .warning_could_not_write_build_record_not_absolutePath(p))+      return+    }

If I read the following code correctly (in Driver.swift):

 if let workingDirectory = self.workingDirectory {
      self.outputFileMap = outputFileMap?.resolveRelativePaths(relativeTo: workingDirectory)
    } else {
      self.outputFileMap = outputFileMap
    }

the warning will only happen if there is no working directory and the map contains a relative path.

davidungar

comment created time in 9 days

PullRequestReviewEvent

Pull request review commentapple/swift-driver

[Incremental] Write build record & test

 import Foundation @_implementationOnly import Yams  /// Holds the info about inputs needed to plan incremenal compilation+/// A.k.a. BuildRecord, but InputInfoMap is the legacy name public struct InputInfoMap {

I had thought of that, and you just pushed me over the top! Done!! (Thanks.)

davidungar

comment created time in 9 days

PullRequestReviewEvent

push eventdavidungar/swift-driver

David Ungar

commit sha 9bd04adb459d085e7d6d53543775bc548dc57197

Move InputInfoMap.swift to BuildRecord.swift

view details

David Ungar

commit sha ba20519e05ef1f47c636314a5c45fea72b07d92c

Rename InputInfoMap to BuildRecord

view details

David Ungar

commit sha 4800715195874848cf6a86e1ffe31387d79829a2

Rename outOfDateMap to outOfDateBuildRecord

view details

push time in 9 days

Pull request review commentapple/swift-driver

[Incremental] Write build record & test

 public struct InputInfo: Equatable { }  public extension InputInfo {-  enum Status {-    case upToDate, needsCascadingBuild, needsNonCascadingBuild, newlyAdded+  enum Status: Equatable {+    case upToDate,+         needsCascadingBuild,+         needsNonCascadingBuild,+         newlyAdded+  }+}++// MARK: - reading+extension InputInfo.Status {+  public init?(identifier: String) {+    switch identifier {+    case "": self = .upToDate+    case "!dirty": self = .needsCascadingBuild+    case "!private": self = .needsNonCascadingBuild+    default: return nil+    }+    assert(self.identifier == identifier, "Ensure reversibility")+  }

I'm not sure how to do that, given that there are two statuses with the same string:

 case .needsCascadingBuild, .newlyAdded:
      return "!dirty"
davidungar

comment created time in 9 days

PullRequestReviewEvent

Pull request review commentapple/swift-driver

[Incremental] Write build record & test

 public extension InputInfoMap {   } } +// MARK: - Creating and writing a new map+extension InputInfoMap {+  /// Create a new buildRecord for writing+  init(jobs: [Job],+       finishedJobResults: [Job: ProcessResult],+       skippedInputs: Set<TypedVirtualPath>?,+       compilationInputModificationDates: [TypedVirtualPath: Date],+       actualSwiftVersion: String,+       argsHash: String,+       timeBeforeFirstJob: Date+  ) {+    let jobResultsByInput = Dictionary(+      uniqueKeysWithValues:+        finishedJobResults.flatMap { job, result in+          job.primaryInputs.map { ($0, result)  }+        }+    )+    let inputInfosArray = compilationInputModificationDates+      .map { input, modDate -> (VirtualPath, InputInfo) in+        let status = InputInfo.Status(  wasSkipped: skippedInputs?.contains(input),+                                        jobResult: jobResultsByInput[input])+        return (input.file, InputInfo(status: status, previousModTime: modDate))+      }++    self.init(+      argsHash: argsHash,+      swiftVersion: actualSwiftVersion,+      buildTime: timeBeforeFirstJob,+      inputInfos: Dictionary(uniqueKeysWithValues: inputInfosArray)+    )+  }++  @_spi(Testing) public func encode() throws -> String {+      let pathsAndInfos = try inputInfos.map {+        input, inputInfo -> (String, InputInfo) in+        guard let path = input.absolutePath else {+          throw Errors.notAbsolutePath(input)+        }+        return (path.pathString, inputInfo)+      }+      let inputInfosNode = Yams.Node(+        pathsAndInfos+          .sorted {$0.0 < $1.0}+          .map {(Yams.Node($0.0, .implicit, .doubleQuoted), Self.encode($0.1))}+      )+      let fieldNodes = [+        ("version", Yams.Node(swiftVersion, .implicit, .doubleQuoted)),+        ("options", Yams.Node(argsHash, .implicit, .doubleQuoted)),+        ("build_time", Self.encode(buildTime)),+        ("inputs", inputInfosNode )+      ] .map { (Yams.Node($0.0), $0.1) }++    let inputInfoMapNode = Yams.Node(fieldNodes, .implicit, .block)+   // let options = Yams.Emitter.Options(canonical: true)+    return try Yams.serialize(node: inputInfoMapNode,+                              width: -1,+                              sortKeys: false)+  }++  private static func encode(_ date: Date, tag tagString: String? = nil) -> Yams.Node {+    let secsAndNanos = date.legacyDriverSecsAndNanos+    return Yams.Node(+      secsAndNanos.map {Yams.Node(String($0))},+      tagString.map {Yams.Tag(Yams.Tag.Name(rawValue: $0))} ?? .implicit,+      .flow)+  }++  private static func encode(_ inputInfo: InputInfo) -> Yams.Node {+    encode(inputInfo.previousModTime, tag: inputInfo.tag)+  }++}++extension Diagnostic.Message {+  static func warning_could_not_serialize_build_record(_ err: Error+  ) -> Diagnostic.Message {+    .warning("Next compile won't be incremental; Could not serialize build record: \(err.localizedDescription)")

Thanks. I have fixed mine. (There are others lurking.)

davidungar

comment created time in 9 days

PullRequestReviewEvent

push eventdavidungar/swift-driver

David Ungar

commit sha 73cafdecfa7abb9fd895c09b9e50e6e6185fe917

Start diagnostics in lower case

view details

push time in 9 days

Pull request review commentapple/swift-driver

[Incremental] Write build record & test

+//===--------------- BuildRecordInfo.swift --------------------------------===//+//+// This source file is part of the Swift.org open source project+//+// Copyright (c) 2014 - 2019 Apple Inc. and the Swift project authors+// Licensed under Apache License v2.0 with Runtime Library Exception+//+// See https://swift.org/LICENSE.txt for license information+// See https://swift.org/CONTRIBUTORS.txt for the list of Swift project authors+//+//===----------------------------------------------------------------------===//++import Foundation+import TSCBasic+import SwiftOptions++/// Holds information required to read and write the build record (aka compilation record)+/// This info is always written, but only read for incremental compilation.+@_spi(Testing) public class BuildRecordInfo {+  let buildRecordPath: VirtualPath+  let fileSystem: FileSystem+  let argsHash: String+  let actualSwiftVersion: String+  let timeBeforeFirstJob: Date+  let diagnosticEngine: DiagnosticsEngine+  let outputBuildRecordForModuleOnlyBuild: Bool+  let compilationInputModificationDates: [TypedVirtualPath: Date]++  var finishedJobResults  = [Job: ProcessResult]()++  init?(+    actualSwiftVersion: String,+    compilerOutputType: FileType?,+    diagnosticEngine: DiagnosticsEngine,+    fileSystem: FileSystem,+    moduleOutputInfo: ModuleOutputInfo,+    outputFileMap: OutputFileMap?,+    parsedOptions: ParsedOptions,+    recordedInputModificationDates: [TypedVirtualPath: Date]+  ) {+    // Cannot write a buildRecord without a path.+    guard let buildRecordPath = Self.computeBuildRecordPath(+            outputFileMap: outputFileMap,+            compilerOutputType: compilerOutputType,+            diagnosticEngine: diagnosticEngine)+    else {+      return nil+    }+    self.actualSwiftVersion = actualSwiftVersion+    self.argsHash = Self.computeArgsHash(parsedOptions)+    self.buildRecordPath = buildRecordPath+    self.compilationInputModificationDates =+      recordedInputModificationDates.filter { input, _ in+        input.type.isPartOfSwiftCompilation+      }+    self.diagnosticEngine = diagnosticEngine+    self.fileSystem = fileSystem+    // If we emit module along with full compilation, emit build record+    // file for '-emit-module' only mode as well.+    self.outputBuildRecordForModuleOnlyBuild = moduleOutputInfo.output?.isTopLevel ?? false+    self.timeBeforeFirstJob = Date()+   }++  private static func computeArgsHash(_ parsedOptionsArg: ParsedOptions+  ) -> String {+    var parsedOptions = parsedOptionsArg+    let hashInput = parsedOptions+      .filter { $0.option.affectsIncrementalBuild && $0.option.kind != .input}+      .map { $0.option.spelling }+      .sorted()+      .joined()+    return SHA256().hash(hashInput).hexadecimalRepresentation+  }++  /// Determine the input and output path for the build record+  private static func computeBuildRecordPath(+    outputFileMap: OutputFileMap?,+    compilerOutputType: FileType?,+    diagnosticEngine: DiagnosticsEngine+  ) -> VirtualPath? {+    // FIXME: This should work without an output file map. We should have+    // another way to specify a build record and where to put intermediates.+    guard let ofm = outputFileMap else {+      return nil+    }+    guard let partialBuildRecordPath =+            ofm.existingOutputForSingleInput(outputType: .swiftDeps)+    else {+      diagnosticEngine.emit(.warning_incremental_requires_build_record_entry)+      return nil+    }+    // In 'emit-module' only mode, use build-record filename suffixed with+    // '~moduleonly', so that module-only mode doesn't mess up build-record+    // file for full compilation.+    return compilerOutputType == .swiftModule+      ? partialBuildRecordPath.appendingToBaseName("~moduleonly")+      : partialBuildRecordPath

Thanks again, nice catch!

davidungar

comment created time in 9 days

PullRequestReviewEvent

push eventdavidungar/swift-driver

David Ungar

commit sha 5a1c9c1a9919e5ad51c5cea68eaa23e464cf16df

Remove support for "moduleonly" build record.

view details

push time in 9 days

Pull request review commentapple/swift-driver

[Incremental] Write build record & test

+//===--------------- BuildRecordInfo.swift --------------------------------===//+//+// This source file is part of the Swift.org open source project+//+// Copyright (c) 2014 - 2019 Apple Inc. and the Swift project authors+// Licensed under Apache License v2.0 with Runtime Library Exception+//+// See https://swift.org/LICENSE.txt for license information+// See https://swift.org/CONTRIBUTORS.txt for the list of Swift project authors+//+//===----------------------------------------------------------------------===//++import Foundation+import TSCBasic+import SwiftOptions++/// Holds information required to read and write the build record (aka compilation record)+/// This info is always written, but only read for incremental compilation.+@_spi(Testing) public class BuildRecordInfo {+  let buildRecordPath: VirtualPath+  let fileSystem: FileSystem+  let argsHash: String+  let actualSwiftVersion: String+  let timeBeforeFirstJob: Date+  let diagnosticEngine: DiagnosticsEngine+  let outputBuildRecordForModuleOnlyBuild: Bool+  let compilationInputModificationDates: [TypedVirtualPath: Date]++  var finishedJobResults  = [Job: ProcessResult]()++  init?(+    actualSwiftVersion: String,+    compilerOutputType: FileType?,+    diagnosticEngine: DiagnosticsEngine,+    fileSystem: FileSystem,+    moduleOutputInfo: ModuleOutputInfo,+    outputFileMap: OutputFileMap?,+    parsedOptions: ParsedOptions,+    recordedInputModificationDates: [TypedVirtualPath: Date]+  ) {+    // Cannot write a buildRecord without a path.+    guard let buildRecordPath = Self.computeBuildRecordPath(+            outputFileMap: outputFileMap,+            compilerOutputType: compilerOutputType,+            diagnosticEngine: diagnosticEngine)+    else {+      return nil+    }+    self.actualSwiftVersion = actualSwiftVersion+    self.argsHash = Self.computeArgsHash(parsedOptions)+    self.buildRecordPath = buildRecordPath+    self.compilationInputModificationDates =+      recordedInputModificationDates.filter { input, _ in+        input.type.isPartOfSwiftCompilation+      }+    self.diagnosticEngine = diagnosticEngine+    self.fileSystem = fileSystem+    // If we emit module along with full compilation, emit build record+    // file for '-emit-module' only mode as well.+    self.outputBuildRecordForModuleOnlyBuild = moduleOutputInfo.output?.isTopLevel ?? false+    self.timeBeforeFirstJob = Date()+   }++  private static func computeArgsHash(_ parsedOptionsArg: ParsedOptions+  ) -> String {+    var parsedOptions = parsedOptionsArg+    let hashInput = parsedOptions+      .filter { $0.option.affectsIncrementalBuild && $0.option.kind != .input}+      .map { $0.option.spelling }+      .sorted()+      .joined()+    return SHA256().hash(hashInput).hexadecimalRepresentation+  }++  /// Determine the input and output path for the build record+  private static func computeBuildRecordPath(+    outputFileMap: OutputFileMap?,+    compilerOutputType: FileType?,+    diagnosticEngine: DiagnosticsEngine+  ) -> VirtualPath? {+    // FIXME: This should work without an output file map. We should have+    // another way to specify a build record and where to put intermediates.+    guard let ofm = outputFileMap else {+      return nil+    }+    guard let partialBuildRecordPath =+            ofm.existingOutputForSingleInput(outputType: .swiftDeps)+    else {+      diagnosticEngine.emit(.warning_incremental_requires_build_record_entry)+      return nil+    }+    // In 'emit-module' only mode, use build-record filename suffixed with+    // '~moduleonly', so that module-only mode doesn't mess up build-record+    // file for full compilation.+    return compilerOutputType == .swiftModule+      ? partialBuildRecordPath.appendingToBaseName("~moduleonly")+      : partialBuildRecordPath+  }++  /// Write out the build record.+  /// \c Jobs must include all of the compilation jobs.+  /// \c Inputs will hold all the primary inputs that were not compiled because of incremental compilation

Thanks! Fixed.

davidungar

comment created time in 9 days

PullRequestReviewEvent

push eventdavidungar/swift-driver

David Ungar

commit sha 5e10d0d11d108f468f7abef6f27d7709fa25105e

Replace \c with backticks

view details

push time in 9 days

push eventapple/swift-driver

David Ungar

commit sha dbc33fce3d5c018e3ca3a055a1c6a2f8d3c333a8

Correctly create swiftdeps output

view details

David Ungar

commit sha fb0dbee3db4191e91846983be79c9937f7cbe11a

Merge pull request #302 from davidungar/incremental-10-10-fix-swiftdeps Correctly create swiftdeps output in a compile job.

view details

push time in 10 days

pull request commentapple/swift-driver

Correctly create swiftdeps output in a compile job.

@owenv Thanks!

davidungar

comment created time in 10 days

PullRequestReviewEvent
PullRequestReviewEvent

Pull request review commentapple/swift

[CodeCompletion] Remove parser hacks regarding type body fingerprints

 static bool areAnyDependentFilesInvalidated(       }); } +/// Get interface hash of \p SF including the type members in the file.+///+/// This is needed because SouceFile::getIntefaceHash() doesn't digest type body+/// fingerprints in the file.+void getInterfaceHashIncludingTypeMembers(SourceFile *SF,+                                          llvm::SmallString<32> &str) {+  /// FIXME: Gross. Hashing multiple "hash" values.+  llvm::MD5 hash;+  SF->getInterfaceHash(str);+  hash.update(str);++  std::function<void(IterableDeclContext *)> hashTypeBodyFingerprints =+      [&](IterableDeclContext *IDC) {+        if (auto fp = IDC->getBodyFingerprint())+          hash.update(*fp);+        for (auto *member : IDC->getParsedMembers())+          if (auto *childIDC = dyn_cast<IterableDeclContext>(member))+            hashTypeBodyFingerprints(childIDC);+      };++  for (auto *D : SF->getTopLevelDecls()) {+    if (auto IDC = dyn_cast<IterableDeclContext>(D))+      hashTypeBodyFingerprints(IDC);+  }++  llvm::MD5::MD5Result result;+  hash.final(result);+  str = result.digest();+}+

What if I’m doing a completion in a function in a struct in a closure? Is it still ok to miss a change in that struct? I’ll change my review to not block you so you can merge if this case won’t be a problem.

rintaro

comment created time in 10 days

PullRequestReviewEvent

pull request commentapple/swift-driver

Correctly create swiftdeps output in a compile job.

@swift-ci please test

davidungar

comment created time in 10 days

create barnchdavidungar/swift-driver

branch : incremental-10-10-fix-swiftdeps

created branch time in 10 days

pull request commentapple/swift-driver

[Incremental] Write build record & test

@swift-ci please test

davidungar

comment created time in 10 days

push eventdavidungar/swift-driver

David Ungar

commit sha 07b2b171e9def074507a640ce3da466d3be84574

Fixed up test

view details

push time in 10 days

pull request commentapple/swift-driver

[Incremental] Write build record & test

@swift-ci please test

davidungar

comment created time in 10 days

push eventdavidungar/swift-driver

David Ungar

commit sha 87fe8ca676d57c01ada6aed0dc3711217b3e6b7a

Fixed up test

view details

push time in 10 days

push eventdavidungar/swift-driver

David Ungar

commit sha 8ac661a1b886599775416f47becd028d0718bab3

First cut at a test

view details

push time in 10 days

PullRequestReviewEvent

push eventdavidungar/swift-driver

David Ungar

commit sha 9bc4730afb768b58e4e45630bbcdf6b78c581b32

write build record

view details

push time in 10 days

push eventdavidungar/swift-driver

David Ungar

commit sha a9c7850a3d0ef20f783fc63bc622cc60049ff6be

Clean up writing build record

view details

push time in 10 days

push eventdavidungar/swift-driver

Xi Ge

commit sha 337ddf4b42f2f32b30c312b9b10c173e47f9acc1

Don't complain about '-sanitize=' unsupported if not given at all for the freebsd target

view details

Xi Ge

commit sha 095937cb90b728ab122b8bac236f6d740327c701

Merge pull request #296 from nkcsgexi/sanitizer-flag-parse Don't complain about '-sanitize=' unsupported if not given at all for the freebsd target

view details

David Ungar

commit sha 5ae9bd879afd319d8714f1a7d0908e2d06af9bf8

Merge pull request #297 from davidungar/incremental-10-9b Add a `primaryInputs` field to `Job` which will be needed for incremental compilation.

view details

David Ungar

commit sha 01001adfe59eb062ef2c2d06df611c6c82857da7

write build record

view details

David Ungar

commit sha 6381e750014abe5c243b412ac74cdd2c2bf92c09

reorder init args

view details

David Ungar

commit sha 81d55fa5099faae775cd406b244905935af69076

Small cleanups

view details

push time in 11 days

push eventdavidungar/swift-driver

David Ungar

commit sha c9a781794df1c902c96f44d77f3e92b63011df16

reorder init args

view details

David Ungar

commit sha 9adcb5d7ec61e63190ff04bad91f9507cd2ce7fc

Small cleanups

view details

push time in 11 days

push eventapple/swift-driver

David Ungar

commit sha 9c66cc8e0f76bf757374e5b46cb2cb9bf9883830

add primaryInputs to jobs

view details

David Ungar

commit sha 5ae9bd879afd319d8714f1a7d0908e2d06af9bf8

Merge pull request #297 from davidungar/incremental-10-9b Add a `primaryInputs` field to `Job` which will be needed for incremental compilation.

view details

push time in 11 days

pull request commentapple/swift

[CodeCompletion] Remove parser hacks regarding type body fingerprints

PS: I think you need to walk the ASTScope tree instead of the IDC tree.

rintaro

comment created time in 11 days

more