profile
viewpoint

milseman/yaml-light 4

A light-weight Haskell wrapper with utility functions around HsSyck

milseman/swift 2

The Swift Programming Language

milseman/swift-evolution 2

This maintains proposals for changes and user-visible enhancements to the Swift Programming Language.

milseman/Haskell-Dnd-Utilities 1

Haskell libraries and utilities for playing/DMing Dnd sessions

milseman/MikesDots 0

My configuration files, maude, and misc setups

milseman/swift-corelibs-foundation 0

The Foundation Project, providing core utilities, internationalization, and OS independence

milseman/swift-corelibs-xctest 0

The XCTest Project, A Swift core library for providing unit test support

milseman/swift-integration-tests 0

Automated tests for validating the generated Swift snapshots behave correctly

milseman/swift-lldb 0

This is the version of LLDB that supports the Swift programming language & REPL.

pull request commentapple/swift-system

Syscall mock testing

@lorentey I've only enabled mocking in debug builds (best I can do to approximate testing), and I've conditionally compiled out all of the types and most of the infrastructure and use of the infrastructure. The mock_foo internal methods are still "present", but they're only called after a constant-foldable false branch and should be eliminated at compilation time.

milseman

comment created time in 4 hours

Pull request review commentapple/swift-system

Syscall mock testing

+/*+ This source file is part of the Swift System open source project++ Copyright (c) 2020 Apple Inc. and the Swift System project authors+ Licensed under Apache License v2.0 with Runtime Library Exception++ See https://swift.org/LICENSE.txt for license information+*/++// Syscall mocking support.+//+// NOTE: This is currently the bare minimum needed for System's testing purposes, though we do+// eventually want to expose some solution to users.+//+// Mocking is contextual, accessible through MockingDriver.withMockingEnabled. Mocking+// state, including whether it is enabled, is stored in thread-local storage. Mocking is only+// enabled in testing builds of System currently, to minimize runtime overhead of release builds.+//++public struct Trace {

Internal code that is never called can be dropped by the compiler (and even the linker), unless there is something preventing the optimization. I'll do a bit of both.

milseman

comment created time in 4 hours

PullRequestReviewEvent

push eventmilseman/swift-system

Michael Ilseman

commit sha 98a1531ea099b06796a3f56cbeaab8e7016211bb

Remove mocking infrastructure from non-mocking builds

view details

push time in 4 hours

Pull request review commentapple/swift-system

Syscall mock testing

 let targets: [PackageDescription.Target] = [     path: "Sources/System"),   .target(     name: "SystemInternals",-    dependencies: ["CSystem"]),+    dependencies: ["CSystem"],+    swiftSettings: [+      .define("ENABLE_MOCKING")

The best I can do, IIUC, is: .define("ENABLE_MOCKING", .when(configuration: .debug))

We also need executables as tests to build in debug and release at some point.

milseman

comment created time in 6 hours

PullRequestReviewEvent

Pull request review commentapple/swift-system

Syscall mock testing

+/*+ This source file is part of the Swift System open source project++ Copyright (c) 2020 Apple Inc. and the Swift System project authors+ Licensed under Apache License v2.0 with Runtime Library Exception++ See https://swift.org/LICENSE.txt for license information+*/++// Syscall mocking support.+//+// NOTE: This is currently the bare minimum needed for System's testing purposes, though we do+// eventually want to expose some solution to users.+//+// Mocking is contextual, accessible through MockingDriver.withMockingEnabled. Mocking+// state, including whether it is enabled, is stored in thread-local storage. Mocking is only+// enabled in testing builds of System currently, to minimize runtime overhead of release builds.+//++public struct Trace {

Another approach might be to just make all of this stuff internal, since the unit tests can always do a @testable import. Too bad we don't have a real notion of "public-for-testing" visibility level in Swift, as this is very much designed to be an API for testing different from other internal entities.

milseman

comment created time in 6 hours

Pull request review commentapple/swift-system

Syscall mock testing

+/*+ This source file is part of the Swift System open source project++ Copyright (c) 2020 Apple Inc. and the Swift System project authors+ Licensed under Apache License v2.0 with Runtime Library Exception++ See https://swift.org/LICENSE.txt for license information+*/++// Syscall mocking support.+//+// NOTE: This is currently the bare minimum needed for System's testing purposes, though we do+// eventually want to expose some solution to users.+//+// Mocking is contextual, accessible through MockingDriver.withMockingEnabled. Mocking+// state, including whether it is enabled, is stored in thread-local storage. Mocking is only+// enabled in testing builds of System currently, to minimize runtime overhead of release builds.+//++public struct Trace {+  public struct Entry: Hashable {+    var name: String+    var arguments: [AnyHashable]++    public init(name: String, _ arguments: [AnyHashable]) {+      self.name = name+      self.arguments = arguments+    }+  }++  private var entries: [Entry] = []+  private var firstEntry: Int = 0++  public var isEmpty: Bool { firstEntry >= entries.count }++  public mutating func dequeue() -> Entry? {+    guard !self.isEmpty else { return nil }+    defer { firstEntry += 1 }+    return entries[firstEntry]+  }++  internal mutating func add(_ e: Entry) {+    entries.append(e)+  }++  public mutating func clear() { entries.removeAll() }+}++// TODO: Track+public struct WriteBuffer {+  public var enabled: Bool = false++  private var buffer: [UInt8] = []+  private var chunkSize: Int? = nil++  internal mutating func write(_ buf: UnsafeRawBufferPointer) -> Int {+    guard enabled else { return 0 }+    let chunk = chunkSize ?? buf.count+    buffer.append(contentsOf: buf.prefix(chunk))+    return chunk+  }++  public var contents: [UInt8] { buffer }+}++public enum ForceErrno: Equatable {+  case none+  case always(errno: CInt)++  case counted(errno: CInt, count: Int)+}++// Provide access to the driver, context, and trace stack of mocking+public class MockingDriver {+  // Record syscalls and their arguments+  public var trace = Trace()++  // Mock errors inside syscalls+  public var forceErrno = ForceErrno.none++  // A buffer to put `write` bytes into+  public var writeBuffer = WriteBuffer()+}++#if os(macOS) || os(iOS) || os(watchOS) || os(tvOS)+import Darwin+#elseif os(Linux) || os(FreeBSD) || os(Android)+import Glibc+#else+#error("Unsupported Platform")+#endif++#if os(macOS) || os(iOS) || os(watchOS) || os(tvOS)+private func releaseObject(_ raw: UnsafeMutableRawPointer) -> () {+  Unmanaged<MockingDriver>.fromOpaque(raw).release()+}+#elseif os(Linux) || os(FreeBSD) || os(Android)+private func releaseObject(_ raw: UnsafeMutableRawPointer?) -> () {+  guard let object = raw else { return }+  Unmanaged<MockingDriver>.fromOpaque(object).release()+}+#else+#error("Unsupported Platform")+#endif+++internal let key: pthread_key_t = {+  var raw = pthread_key_t()+  guard 0 == pthread_key_create(&raw, releaseObject) else {+    fatalError("Unable to create key")+  }+  return raw+}()++internal var currentMockingDriver: MockingDriver? {+  #if !ENABLE_MOCKING+    fatalError("Contextual mocking in non-mocking build")+  #endif++  guard let rawPtr = pthread_getspecific(key) else { return nil }++  return Unmanaged<MockingDriver>.fromOpaque(rawPtr).takeUnretainedValue()+}++extension MockingDriver {+  /// Enables mocking for the duration of `f` with a clean trace queue+  /// Restores prior mocking status and trace queue after execution+  public static func withMockingEnabled(+    _ f: (MockingDriver) throws -> ()+  ) rethrows {+    let priorMocking = currentMockingDriver+    defer {+      if let object = priorMocking {+        pthread_setspecific(key, Unmanaged.passUnretained(object).toOpaque())+      } else {+        pthread_setspecific(key, nil)+      }+    }++    let driver = MockingDriver()+    guard 0 == pthread_setspecific(key, Unmanaged.passRetained(driver).toOpaque()) else {

Hmm, I think we should pass it unretained, fix the lifetime, and remove the releaseObject destructor above.

Currently, if the thread is killed and the TLS entry is non-NULL, releaseObject is called. If this is the top-most withMockingEnabled, then the entry will be set to NULL and there will be a leak. If this is a nested call, then the parent driver will be restored (AFAICT this is balanced), and this will leak. If the thread is killed during this call, then the "passRetained" will be paired with the the release in releaseObject, IIUC leakiness depends on whether there are other references to the driver alive.

But, if we pass it unretained and fix the lifetime, it will be balanced for the duration of the function and the defer above will remove the TLS entry anyways. If we don't have a releaseObject registered, then our leaking story if the thread is killed during this function is just the normal one for Swift.

milseman

comment created time in 6 hours

Pull request review commentapple/swift-system

Syscall mock testing

+/*+ This source file is part of the Swift System open source project++ Copyright (c) 2020 Apple Inc. and the Swift System project authors+ Licensed under Apache License v2.0 with Runtime Library Exception++ See https://swift.org/LICENSE.txt for license information+*/++import XCTest+import SystemInternals+import SystemPackage++internal protocol TestCase {+  // TODO: want a source location stack, more fidelity, kinds of stack entries, etc+  var file: StaticString { get }+  var line: UInt { get }++  // TODO: Instead have an attribute to register a test in a allTests var, similar to the argument parser.+  func runAllTests()+}+extension TestCase {+  func expectEqualSequence<S1: Sequence, S2: Sequence>(+    _ actual: S1, _ expected: S2,+    _ message: String? = nil+  ) where S1.Element: Equatable, S1.Element == S2.Element {+    if !actual.elementsEqual(expected) {+      fail(message)+    }+  }+  func expectEqual<E: Equatable>(+    _ actual: E, _ expected: E,+    _ message: String? = nil+  ) {+    if actual != expected {+      fail(message)+    }+  }+  func expectTrue(+    _ actual: Bool,+    _ message: String? = nil+  ) {+    expectEqual(true, actual, message)+  }+  func expectFalse(+    _ actual: Bool,+    _ message: String? = nil+  ) {+    expectEqual(false, actual, message)+  }++  func fail(_ reason: String? = nil) {+    XCTAssert(false, reason ?? "", file: file, line: line)+  }+}++internal struct MockTestCase: TestCase {+  var file: StaticString+  var line: UInt++  var expected: Trace.Entry+  var interruptable: Bool++  var body: (_ retryOnInterrupt: Bool) throws -> ()++  init(+    _ file: StaticString = #file,+    _ line: UInt = #line,+    name: String,+    _ args: AnyHashable...,+    interruptable: Bool,+    _ body: @escaping (_ retryOnInterrupt: Bool) throws -> ()+  ) {+    self.file = file+    self.line = line+    self.expected = Trace.Entry(name: name, args)+    self.interruptable = interruptable+    self.body = body+  }++  func runAllTests() {+    MockingDriver.withMockingEnabled { mocking in+      // Make sure we completely match the trace queue+      self.expectTrue(mocking.trace.isEmpty)+      defer { self.expectTrue(mocking.trace.isEmpty) }++      // Test our API mappings to the lower-level syscall invocation+      do {+        try body(true)+        self.expectEqual(mocking.trace.dequeue(), self.expected)+      } catch {+        self.fail()+      }++      // Test interupt behavior. Interruptable calls will be told not to+      // retry to catch the EINTR. Non-interruptable calls will be told to+      // retry, to make sure they don't spin (e.g. if API changes to include+      // interruptable)

My understanding is that is much more fuzzy. The man pages show one set of errnos, but in practice that is already out of date and can change version-to-version of the OS, as well as unanticipated corner cases involving things like network file systems.

milseman

comment created time in 6 hours

Pull request review commentapple/swift-system

Syscall mock testing

+/*+ This source file is part of the Swift System open source project++ Copyright (c) 2020 Apple Inc. and the Swift System project authors+ Licensed under Apache License v2.0 with Runtime Library Exception++ See https://swift.org/LICENSE.txt for license information+*/++import XCTest+import SystemInternals+import SystemPackage++internal protocol TestCase {+  // TODO: want a source location stack, more fidelity, kinds of stack entries, etc+  var file: StaticString { get }+  var line: UInt { get }++  // TODO: Instead have an attribute to register a test in a allTests var, similar to the argument parser.+  func runAllTests()+}+extension TestCase {+  func expectEqualSequence<S1: Sequence, S2: Sequence>(+    _ actual: S1, _ expected: S2,+    _ message: String? = nil+  ) where S1.Element: Equatable, S1.Element == S2.Element {+    if !actual.elementsEqual(expected) {+      fail(message)+    }+  }+  func expectEqual<E: Equatable>(+    _ actual: E, _ expected: E,+    _ message: String? = nil+  ) {+    if actual != expected {+      fail(message)+    }+  }+  func expectTrue(+    _ actual: Bool,+    _ message: String? = nil+  ) {+    expectEqual(true, actual, message)+  }+  func expectFalse(+    _ actual: Bool,+    _ message: String? = nil+  ) {+    expectEqual(false, actual, message)+  }++  func fail(_ reason: String? = nil) {+    XCTAssert(false, reason ?? "", file: file, line: line)+  }+}++internal struct MockTestCase: TestCase {+  var file: StaticString+  var line: UInt++  var expected: Trace.Entry+  var interruptable: Bool++  var body: (_ retryOnInterrupt: Bool) throws -> ()++  init(+    _ file: StaticString = #file,+    _ line: UInt = #line,+    name: String,+    _ args: AnyHashable...,+    interruptable: Bool,+    _ body: @escaping (_ retryOnInterrupt: Bool) throws -> ()+  ) {+    self.file = file+    self.line = line+    self.expected = Trace.Entry(name: name, args)+    self.interruptable = interruptable+    self.body = body+  }++  func runAllTests() {+    MockingDriver.withMockingEnabled { mocking in+      // Make sure we completely match the trace queue+      self.expectTrue(mocking.trace.isEmpty)+      defer { self.expectTrue(mocking.trace.isEmpty) }++      // Test our API mappings to the lower-level syscall invocation+      do {+        try body(true)+        self.expectEqual(mocking.trace.dequeue(), self.expected)+      } catch {+        self.fail()+      }++      // Test interupt behavior. Interruptable calls will be told not to+      // retry to catch the EINTR. Non-interruptable calls will be told to+      // retry, to make sure they don't spin (e.g. if API changes to include+      // interruptable)+      do {+        let oldErrno = mocking.forceErrno+        mocking.forceErrno = .always(errno: EINTR)+        defer { mocking.forceErrno = oldErrno }

Since we changed it make a new driver after withMockingEnabled, we don't have to restore this anymore. We can just assert that it is .none.

milseman

comment created time in 6 hours

PullRequestReviewEvent
PullRequestReviewEvent

pull request commentapple/swift-system

Initial port to Windows

This LGTM after I merge mocking at https://github.com/apple/swift-system/pull/8. I apologize in advance that it will cause you some merge conflicts around the system_* functions. Hopefully we can get most of the #if/#else out of there.

compnerd

comment created time in 10 hours

Pull request review commentapple/swift-system

Initial port to Windows

 public var system_errno: CInt { #endif  public func system_open(_ path: UnsafePointer<CChar>, _ oflag: Int32) -> CInt {+#if os(Windows)+  var fh: CInt = -1+  _ = _sopen_s(&fh, path, oflag, _SH_DENYNO, _S_IREAD | _S_IWRITE)+  return fh+#else   open(path, oflag)+#endif } +#if os(Windows)+public func system_open(+  _ path: UnsafePointer<CChar>, _ oflag: Int32, _ mode: CInt+) -> CInt {+  var fh: CInt = -1+  _ = _sopen_s(&fh, path, oflag, _SH_DENYNO, mode)+  return fh+}+#else public func system_open(   _ path: UnsafePointer<CChar>, _ oflag: Int32, _ mode: mode_t ) -> CInt {   open(path, oflag, mode) }+#endif  public func system_close(_ fd: Int32) -> Int32 {-  close(fd)+#if os(Windows)+  return _close(fd)+#else+  return close(fd)+#endif } +#if os(Windows)+public func system_read(+  _ fd: Int32, _ buf: UnsafeMutableRawPointer!, _ nbytes: CUnsignedInt+) -> CInt {+  _read(fd, buf, nbytes)+}+#else public func system_read(   _ fd: Int32, _ buf: UnsafeMutableRawPointer!, _ nbyte: Int ) -> Int {   read(fd, buf, nbyte) }+#endif  public func system_pread(   _ fd: Int32, _ buf: UnsafeMutableRawPointer!, _ nbyte: Int, _ offset: off_t ) -> Int {+#if os(Windows)+  let handle: intptr_t = _get_osfhandle(fd)+  if handle == /* INVALID_HANDLE_VALUE */ -1 { return Int(EBADF) }++  // NOTE: this is a non-owning handle, do *not* call CloseHandle on it+  let hFile: HANDLE = HANDLE(bitPattern: handle)!++  var ovlOverlapped: OVERLAPPED = OVERLAPPED()+  ovlOverlapped.OffsetHigh = DWORD(UInt32(offset >> 32) & 0xffffffff)+  ovlOverlapped.Offset = DWORD(UInt32(offset >> 0) & 0xffffffff)++  var nNumberOfBytesRead: DWORD = 0+  if !ReadFile(hFile, buf, DWORD(nbyte), &nNumberOfBytesRead, &ovlOverlapped) {+    let _ = GetLastError()+    // TODO(compnerd) map windows error to errno+    return Int(-1)+  }+  return Int(nNumberOfBytesRead)+#else   pread(fd, buf, nbyte, offset)+#endif }  public func system_lseek(   _ fd: Int32, _ off: off_t, _ whence: Int32 ) -> off_t {-  lseek(fd, off, whence)+#if os(Windows)+  // TODO(compnerd) should this use _lseeki64 to get larget file offsets?+  return _lseek(fd, off, whence)+#else+  return lseek(fd, off, whence)+#endif

To reduce the number of #if/#else inside of these functions, should Windows have a internal let lseek = _lseek, etc?

compnerd

comment created time in 10 hours

PullRequestReviewEvent

pull request commentapple/swift-system

Syscall mock testing

@lorentey I think this is ready to merge after your review. Could you make sure I'm doing sane things with Unmanaged?

milseman

comment created time in 10 hours

PR opened apple/swift-system

Syscall mock testing

Syscall mock testing

Adds mocking and tracing infrastructure, test infrastructure, and trace-based tests.

+640 -56

0 comment

10 changed files

pr created time in 10 hours

create barnchmilseman/swift-system

branch : mimus_polyglottos

created branch time in 10 hours

fork milseman/swift-system

Swift System provides idiomatic interfaces to system calls and low-level currency types.

fork in 10 hours

PR closed apple/swift-system

Reviewers
Syscall mock testing

Adds mocking and tracing infrastructure, test infrastructure, and trace-based tests.

TODO: Linux testing.

+573 -47

1 comment

7 changed files

milseman

pr closed time in 11 hours

pull request commentapple/swift-system

Syscall mock testing

For some reason Github isn't recognizing my latest pushes. Closing and re-opening

milseman

comment created time in 11 hours

issue commentapple/swift-system

Foundation additions

@lorentey do you know what the story is with cross-import overlays, especially in a package build environment?

benrimmington

comment created time in 11 hours

PullRequestReviewEvent

pull request commentapple/swift-system

Initial port to Windows

Right, so on nix you'd have an initializer taking a collection of bytes in some platform encoding (e.g. UTF-8) and on Windows you'd have an init taking (presumably) a collection of UInt16s. Both platforms can form paths from strings and both platforms can (with potential encoding error correction) go to a string. The storage representation of FilePath is hidden and opaque (even in the ABI).

compnerd

comment created time in 14 days

Pull request review commentapple/swift-system

Initial port to Windows

 public var system_errno: CInt { #endif  public func system_open(_ path: UnsafePointer<CChar>, _ oflag: Int32) -> CInt {+#if os(Windows)+  var fh: CInt = -1+  _ = _sopen_s(&fh, path, oflag, _SH_DENYNO, _S_IREAD | _S_IWRITE)+  return fh+#else   open(path, oflag)+#endif } +#if os(Windows)+public func system_open(+  _ path: UnsafePointer<CChar>, _ oflag: Int32, _ mode: CInt+) -> CInt {

I wonder if we should try to unify the interface around trivial type casts

compnerd

comment created time in 14 days

Pull request review commentapple/swift-system

Initial port to Windows

 public var system_errno: CInt {   get { Darwin.errno }   set { Darwin.errno = newValue } }+#elseif os(Windows)+public var system_errno: CInt {+  get {+    var value: CInt = 0+    // TODO(compnerd) handle the error?

What is there to handle?

compnerd

comment created time in 14 days

Pull request review commentapple/swift-system

Initial port to Windows

 public var system_errno: CInt { #endif  public func system_open(_ path: UnsafePointer<CChar>, _ oflag: Int32) -> CInt {+#if os(Windows)+  var fh: CInt = -1+  _ = _sopen_s(&fh, path, oflag, _SH_DENYNO, _S_IREAD | _S_IWRITE)

Is this the rough equivalent of POSIX open? E.g., what is the difference between a Windows file handle and a POSIX file descriptor?

compnerd

comment created time in 14 days

Pull request review commentapple/swift-system

Initial port to Windows

 public var system_errno: CInt { #endif  public func system_open(_ path: UnsafePointer<CChar>, _ oflag: Int32) -> CInt {+#if os(Windows)+  var fh: CInt = -1+  _ = _sopen_s(&fh, path, oflag, _SH_DENYNO, _S_IREAD | _S_IWRITE)+  return fh+#else   open(path, oflag)+#endif } +#if os(Windows)+public func system_open(+  _ path: UnsafePointer<CChar>, _ oflag: Int32, _ mode: CInt+) -> CInt {+  var fh: CInt = -1+  _ = _sopen_s(&fh, path, oflag, _SH_DENYNO, mode)+  return fh+}+#else public func system_open(   _ path: UnsafePointer<CChar>, _ oflag: Int32, _ mode: mode_t ) -> CInt {   open(path, oflag, mode) }+#endif  public func system_close(_ fd: Int32) -> Int32 {-  close(fd)+#if os(Windows)+  return _close(fd)+#else+  return close(fd)+#endif } +#if os(Windows)+public func system_read(+  _ fd: Int32, _ buf: UnsafeMutableRawPointer!, _ nbytes: CUnsignedInt+) -> CInt {+  _read(fd, buf, nbytes)+}+#else public func system_read(   _ fd: Int32, _ buf: UnsafeMutableRawPointer!, _ nbyte: Int ) -> Int {   read(fd, buf, nbyte) }+#endif  public func system_pread(   _ fd: Int32, _ buf: UnsafeMutableRawPointer!, _ nbyte: Int, _ offset: off_t ) -> Int {+#if os(Windows)+  let handle: intptr_t = _get_osfhandle(fd)+  if handle == /* INVALID_HANDLE_VALUE */ -1 { return Int(EBADF) }++  // NOTE: this is a non-owning handle, do *not* call CloseHandle on it+  let hFile: HANDLE = HANDLE(bitPattern: handle)!++  var ovlOverlapped: OVERLAPPED = OVERLAPPED()+  ovlOverlapped.OffsetHigh = DWORD(UInt32(offset >> 32) & 0xffffffff)+  ovlOverlapped.Offset = DWORD(UInt32(offset >> 0) & 0xffffffff)++  var nNumberOfBytesRead: DWORD = 0+  if !ReadFile(hFile, buf, DWORD(nbyte), &nNumberOfBytesRead, &ovlOverlapped) {+    let _ = GetLastError()+    // TODO(compnerd) map windows error to errno+    return Int(-1)+  }+  return Int(nNumberOfBytesRead)

Could you explain more what is going on here? Does Windows not support pread/pwrite? We will have to decide on a case-by-case basis what should be handled by availability and what should be handled by implementing the functionality ourselves (but this doesn't look too bad). This is definitely worthy of being extracted into a helper function and given a good name (and comments!) since it's non-obvious to a non-Windows contributor what's going on.

compnerd

comment created time in 14 days

Pull request review commentapple/swift-system

Initial port to Windows

+/*+ This source file is part of the Swift System open source project++ Copyright (c) 2020 Apple Inc. and the Swift System project authors+ Licensed under Apache License v2.0 with Runtime Library Exception++ See https://swift.org/LICENSE.txt for license information+*/++#if os(Windows)++import ucrt

What is this import used for in this file? We also want to make sure that System doesn't pull in any other modules, so make sure all imports from the System module are @_implementationOnly

compnerd

comment created time in 14 days

PullRequestReviewEvent
PullRequestReviewEvent

issue commentapple/swift-system

Better Way Of Defining Linux System Constants

Thanks for posting this. We're definitely in a tough due to language/compiler limitations around emitting these constants, hence why we have the hard-coded values. Ideally, we'd get language support to just forward the constants from the headers, but in the meantime we may have to resort to code-gen for this.

ldo

comment created time in 15 days

issue commentapple/swift-system

Question: why are @available directives commented out?

@BasThomas definitely. I'm trying to put together some more extensive documentation and this should be part of that.

BasThomas

comment created time in 15 days

PullRequestReviewEvent

Pull request review commentapple/swift-system

Draft: FilePath.ComponentView

+/*+ This source file is part of the Swift System open source project++ Copyright (c) 2020 Apple Inc. and the Swift System project authors+ Licensed under Apache License v2.0 with Runtime Library Exception++ See https://swift.org/LICENSE.txt for license information+*/++// @available(macOS 9999, iOS 9999, watchOS 9999, tvOS 9999, *)+extension FilePath {+  public struct Component: Hashable {+    // NOTE: For now, we store a slice of FilePath's storage representation. We'd like to+    // have a small-slice representation in the future since the majority of path+    // components would easily fit in the 3 words of storage.+    //+    internal var slice: FilePath.Storage.SubSequence++    // TODO: It would be nice to have a ComponentKind. Prefix (Windows only)+    // is an important piece of information that has to be parsed from the+    // front of the path.++    internal init(_ slice: FilePath.Storage.SubSequence) {+      self.slice = slice+      self.invariantCheck()+    }+  }+  public struct ComponentView {+    internal var path: FilePath+  }++  public var components: ComponentView {+    get { ComponentView(path: self) }+    set { self = newValue.path }+  }+}++// @available(macOS 9999, iOS 9999, watchOS 9999, tvOS 9999, *)+extension FilePath.Component {+  // WARNING: Return value is dependent on self+  fileprivate var unsafeCChars: UnsafeBufferPointer<CChar> {+    // Array must implement wCSIA with stable address...+    // TODO: A stable address byte buffer (and slice) would work better here...+    slice.withContiguousStorageIfAvailable { $0 }!+  }++  // WARNING: Return value is dependent on self+  fileprivate var unsafeUInt8s: UnsafeBufferPointer<UInt8> {+    unsafeCChars._asUInt8+  }++  fileprivate var count: Int { slice.count }++  public var isRoot: Bool {+    if isSeparator(slice.first!) {+      assert(count == 1)+      return true+    }+    return false+  }++  // TODO: ensure this all gets easily optimized away in release...+  fileprivate func invariantCheck() {+    defer { _fixLifetime(self) }++    // TODO: should this be a debugPrecondition? One can make a component+    // explicitly from a string, or maybe it should be a hard precondition+    // inside the EBSL init and a assert/debug one here...+    assert(isRoot || unsafeCChars.allSatisfy { !isSeparator($0) } )++    // TODO: Are we forbidding interior null?+    assert(unsafeUInt8s.isEmpty || unsafeUInt8s.last != 0)+  }+}++// @available(macOS 9999, iOS 9999, watchOS 9999, tvOS 9999, *)+extension String {+  /// Creates a string by interpreting the path component's content as UTF-8.+  ///+  /// - Parameter component: The path component to be interpreted as UTF-8.+  ///+  /// If the content of the path component+  /// isn't a well-formed UTF-8 string,+  /// this initializer removes invalid bytes or replaces them with U+FFFD.+  /// This means that, depending on the semantics of the specific file system,+  /// conversion to a string and back to a path+  /// might result in a value that's different from the original path.+  public init(decoding component: FilePath.Component) {+    defer { _fixLifetime(component) }+    self.init(decoding: component.unsafeUInt8s, as: UTF8.self)+  }++  /// Creates a string from a path component, validating its UTF-8 contents.+  ///+  /// - Parameter component: The path component to be interpreted as UTF-8.+  ///+  /// If the contents of the path component+  /// isn't a well-formed UTF-8 string,+  /// this initializer returns `nil`.+  public init?(validatingUTF8 component: FilePath.Component) {+    // TODO: use a failing initializer for String when one is added...+    defer { _fixLifetime(component) }+    let str = String(decoding: component)+    guard str.utf8.elementsEqual(component.unsafeUInt8s) else { return nil }+    self = str+  }++  // TODO: Consider a init?(validating:), keeping the encoding agnostic in API and+  // dependent on file system.+}+++// @available(macOS 9999, iOS 9999, watchOS 9999, tvOS 9999, *)+extension FilePath.Component: CustomStringConvertible, CustomDebugStringConvertible {++  /// A textual representation of the path component.+  @inline(never)+  public var description: String { String(decoding: self) }++  /// A textual representation of the path component, suitable for debugging.+  public var debugDescription: String { self.description.debugDescription }+}++// @available(macOS 9999, iOS 9999, watchOS 9999, tvOS 9999, *)+extension FilePath.Component: ExpressibleByStringLiteral {+  // TODO: Invariant that there's only one component...+  // Should we even do this, or rely on FilePath from a literal and overloads?+  //+  public init(stringLiteral: String) {+    self.init(stringLiteral)+  }++  // TODO: Invariant that there's only one component...+  // Should we even do this, or rely on FilePath from a literal and overloads?+  //+  public init(_ string: String) {+    let path = FilePath(string)+    precondition(path.components.count == 1)+    self = path.components.first!+    self.invariantCheck()+  }+}+++private var canonicalSeparator: CChar { Int8(bitPattern: UInt8(ascii: "/")) }++// TODO: For Windows, this becomes a little more complicated...+private func isSeparator(_ c: CChar) -> Bool { c == canonicalSeparator }+++// @available(macOS 9999, iOS 9999, watchOS 9999, tvOS 9999, *)+private func separatedComponentBytes<C: Collection>(+  _ components: C, addLeadingSeparator: Bool = false, addTrailingSeparator: Bool = false+) -> Array<CChar> where C.Element == FilePath.Component {+  var result = addLeadingSeparator ? [canonicalSeparator] : []+  defer { _fixLifetime(components) }+  let normalized = components.lazy.filter { !$0.isRoot }.map { $0.unsafeCChars }.joined(separator: [canonicalSeparator])+  result.append(contentsOf: normalized)++  if addTrailingSeparator && (result.isEmpty || !isSeparator(result.last!)) {+    result.append(canonicalSeparator)+  }+  return result+}++// @available(macOS 9999, iOS 9999, watchOS 9999, tvOS 9999, *)+extension FilePath.ComponentView: BidirectionalCollection {+  public typealias Element = FilePath.Component+  public struct Index: Comparable, Hashable {+    internal typealias Storage = FilePath.Storage.Index++    internal var _storage: Storage++    public static func < (lhs: Self, rhs: Self) -> Bool { lhs._storage < rhs._storage }++    fileprivate init(_ idx: Storage) {+      self._storage = idx+    }+  }++  public var startIndex: Index { Index(path.bytes.startIndex) }++  // Use the index of the guaranteed null terminator+  public var endIndex: Index { Index(path.bytes.indices.last!) }++  // Find the end of the component starting at `i`.+  private func parseComponentEnd(startingAt i: Index.Storage) -> Index.Storage {+    if isSeparator(path.bytes[i]) {+      // Special case: leading separator signifies root+      assert(i == path.bytes.startIndex)+      return path.bytes.index(after: i)+    }++    return path.bytes[i...].firstIndex(where: { isSeparator($0) }) ?? endIndex._storage+  }++  // Find the start of the component after the end of the prior at `i`+  private func parseNextComponentStart(+    afterComponentEnd i: Index.Storage+  ) -> Index.Storage {+    assert(i != endIndex._storage)+    if !isSeparator(path.bytes[i]) {+      assert(i == path.bytes.index(after: path.bytes.startIndex))+      // TODO: what about when we're done parsing and we have null terminator?+    }+    return path.bytes[i...].firstIndex(where: { !isSeparator($0) }) ?? endIndex._storage+  }++  public func index(after i: Index) -> Index {+    let end = parseComponentEnd(startingAt: i._storage)+    if Index(end) == endIndex {+      return endIndex+    }+    return Index(parseNextComponentStart(afterComponentEnd: end))+  }++  // Find the start of the component prior to the  after the end of the prior at `i`+  private func parseComponentStart(+    endingAt i: Index.Storage+  ) -> Index.Storage {+    assert(i != startIndex._storage)++    return path.bytes[i...].firstIndex(where: { !isSeparator($0) }) ?? startIndex._storage+  }++  // Chew through separators until we get to a component end+  private func parseComponentEnd(fromStart i: Index) -> Index.Storage {+    let slice = path.bytes[..<i._storage]+    return slice.lastIndex(where: { isSeparator($0) }) ?? startIndex._storage+  }++  public func index(before i: Index) -> Index {+    var slice = path.bytes[..<i._storage]+    while let c = slice.last, isSeparator(c) {+      slice.removeLast()+    }+    while let c = slice.last, !isSeparator(c) {+      slice.removeLast()+    }++    return Index(slice.endIndex)+  }++  public subscript(position: Index) -> FilePath.Component {+    let i = position+    let end = parseComponentEnd(startingAt: i._storage)+    return FilePath.Component(path.bytes[i._storage ..< end])+  }+}++// @available(macOS 9999, iOS 9999, watchOS 9999, tvOS 9999, *)+extension FilePath.ComponentView: RangeReplaceableCollection {+  public init() {+    self.init(path: FilePath())+  }++  public mutating func replaceSubrange<C: Collection>(+    _ subrange: Range<Index>, with newElements: C+  ) where Element == C.Element {+    let (lowerBound, upperBound) = (subrange.lowerBound, subrange.upperBound)++    let pathRange = lowerBound._storage ..< upperBound._storage+    guard !newElements.isEmpty else {+      path.bytes.removeSubrange(pathRange)+      return+    }++    // Insertion skips roots+    let hasNewComponents = !newElements.lazy.filter { !$0.isRoot }.isEmpty++    // Explicitly add a trailing separator if+    //   not at end and next character is not a separator+    let atEnd = upperBound == endIndex+    let trailingSeparator = !atEnd && !isSeparator(path.bytes[upperBound._storage])++    // Explicitly add a preceding separator if+    //   replacing front with absolute components (unless redundant by trailing separator),+    //   preceding character is not a separator (which implies at the end)+    let atStart = lowerBound == startIndex

For POSIX, it seems to denote implementation defined path resolution:

A pathname consisting of a single <slash> shall resolve to the root directory of the process. A null pathname shall not be successfully resolved. If a pathname begins with two successive <slash> characters, the first component following the leading <slash> characters may be interpreted in an implementation-defined manner, although more than two leading <slash> characters shall be treated as a single <slash> character.

https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap04.html#tag_04_13

milseman

comment created time in 22 days

Pull request review commentapple/swift-system

Draft: FilePath.ComponentView

+/*+ This source file is part of the Swift System open source project++ Copyright (c) 2020 Apple Inc. and the Swift System project authors+ Licensed under Apache License v2.0 with Runtime Library Exception++ See https://swift.org/LICENSE.txt for license information+*/++// @available(macOS 9999, iOS 9999, watchOS 9999, tvOS 9999, *)+extension FilePath {+  public struct Component: Hashable {+    // NOTE: For now, we store a slice of FilePath's storage representation. We'd like to+    // have a small-slice representation in the future since the majority of path+    // components would easily fit in the 3 words of storage.+    //+    internal var slice: FilePath.Storage.SubSequence++    // TODO: It would be nice to have a ComponentKind. Prefix (Windows only)+    // is an important piece of information that has to be parsed from the+    // front of the path.++    internal init(_ slice: FilePath.Storage.SubSequence) {+      self.slice = slice+      self.invariantCheck()+    }+  }+  public struct ComponentView {+    internal var path: FilePath+  }++  public var components: ComponentView {+    get { ComponentView(path: self) }+    set { self = newValue.path }+  }+}++// @available(macOS 9999, iOS 9999, watchOS 9999, tvOS 9999, *)+extension FilePath.Component {+  // WARNING: Return value is dependent on self+  fileprivate var unsafeCChars: UnsafeBufferPointer<CChar> {+    // Array must implement wCSIA with stable address...+    // TODO: A stable address byte buffer (and slice) would work better here...+    slice.withContiguousStorageIfAvailable { $0 }!+  }++  // WARNING: Return value is dependent on self+  fileprivate var unsafeUInt8s: UnsafeBufferPointer<UInt8> {+    unsafeCChars._asUInt8+  }++  fileprivate var count: Int { slice.count }++  public var isRoot: Bool {+    if isSeparator(slice.first!) {+      assert(count == 1)+      return true+    }+    return false+  }++  // TODO: ensure this all gets easily optimized away in release...+  fileprivate func invariantCheck() {+    defer { _fixLifetime(self) }++    // TODO: should this be a debugPrecondition? One can make a component+    // explicitly from a string, or maybe it should be a hard precondition+    // inside the EBSL init and a assert/debug one here...+    assert(isRoot || unsafeCChars.allSatisfy { !isSeparator($0) } )++    // TODO: Are we forbidding interior null?+    assert(unsafeUInt8s.isEmpty || unsafeUInt8s.last != 0)+  }+}++// @available(macOS 9999, iOS 9999, watchOS 9999, tvOS 9999, *)+extension String {+  /// Creates a string by interpreting the path component's content as UTF-8.+  ///+  /// - Parameter component: The path component to be interpreted as UTF-8.+  ///+  /// If the content of the path component+  /// isn't a well-formed UTF-8 string,+  /// this initializer removes invalid bytes or replaces them with U+FFFD.+  /// This means that, depending on the semantics of the specific file system,+  /// conversion to a string and back to a path+  /// might result in a value that's different from the original path.+  public init(decoding component: FilePath.Component) {+    defer { _fixLifetime(component) }+    self.init(decoding: component.unsafeUInt8s, as: UTF8.self)+  }++  /// Creates a string from a path component, validating its UTF-8 contents.+  ///+  /// - Parameter component: The path component to be interpreted as UTF-8.+  ///+  /// If the contents of the path component+  /// isn't a well-formed UTF-8 string,+  /// this initializer returns `nil`.+  public init?(validatingUTF8 component: FilePath.Component) {+    // TODO: use a failing initializer for String when one is added...+    defer { _fixLifetime(component) }+    let str = String(decoding: component)+    guard str.utf8.elementsEqual(component.unsafeUInt8s) else { return nil }+    self = str+  }++  // TODO: Consider a init?(validating:), keeping the encoding agnostic in API and+  // dependent on file system.+}+++// @available(macOS 9999, iOS 9999, watchOS 9999, tvOS 9999, *)+extension FilePath.Component: CustomStringConvertible, CustomDebugStringConvertible {++  /// A textual representation of the path component.+  @inline(never)+  public var description: String { String(decoding: self) }++  /// A textual representation of the path component, suitable for debugging.+  public var debugDescription: String { self.description.debugDescription }+}++// @available(macOS 9999, iOS 9999, watchOS 9999, tvOS 9999, *)+extension FilePath.Component: ExpressibleByStringLiteral {+  // TODO: Invariant that there's only one component...+  // Should we even do this, or rely on FilePath from a literal and overloads?+  //+  public init(stringLiteral: String) {+    self.init(stringLiteral)+  }++  // TODO: Invariant that there's only one component...+  // Should we even do this, or rely on FilePath from a literal and overloads?+  //+  public init(_ string: String) {+    let path = FilePath(string)+    precondition(path.components.count == 1)+    self = path.components.first!+    self.invariantCheck()+  }+}+++private var canonicalSeparator: CChar { Int8(bitPattern: UInt8(ascii: "/")) }++// TODO: For Windows, this becomes a little more complicated...+private func isSeparator(_ c: CChar) -> Bool { c == canonicalSeparator }+++// @available(macOS 9999, iOS 9999, watchOS 9999, tvOS 9999, *)+private func separatedComponentBytes<C: Collection>(+  _ components: C, addLeadingSeparator: Bool = false, addTrailingSeparator: Bool = false+) -> Array<CChar> where C.Element == FilePath.Component {+  var result = addLeadingSeparator ? [canonicalSeparator] : []+  defer { _fixLifetime(components) }+  let normalized = components.lazy.filter { !$0.isRoot }.map { $0.unsafeCChars }.joined(separator: [canonicalSeparator])+  result.append(contentsOf: normalized)++  if addTrailingSeparator && (result.isEmpty || !isSeparator(result.last!)) {+    result.append(canonicalSeparator)+  }+  return result+}++// @available(macOS 9999, iOS 9999, watchOS 9999, tvOS 9999, *)+extension FilePath.ComponentView: BidirectionalCollection {+  public typealias Element = FilePath.Component+  public struct Index: Comparable, Hashable {+    internal typealias Storage = FilePath.Storage.Index++    internal var _storage: Storage++    public static func < (lhs: Self, rhs: Self) -> Bool { lhs._storage < rhs._storage }++    fileprivate init(_ idx: Storage) {+      self._storage = idx+    }+  }++  public var startIndex: Index { Index(path.bytes.startIndex) }++  // Use the index of the guaranteed null terminator+  public var endIndex: Index { Index(path.bytes.indices.last!) }++  // Find the end of the component starting at `i`.+  private func parseComponentEnd(startingAt i: Index.Storage) -> Index.Storage {+    if isSeparator(path.bytes[i]) {+      // Special case: leading separator signifies root+      assert(i == path.bytes.startIndex)+      return path.bytes.index(after: i)+    }++    return path.bytes[i...].firstIndex(where: { isSeparator($0) }) ?? endIndex._storage+  }++  // Find the start of the component after the end of the prior at `i`+  private func parseNextComponentStart(+    afterComponentEnd i: Index.Storage+  ) -> Index.Storage {+    assert(i != endIndex._storage)+    if !isSeparator(path.bytes[i]) {+      assert(i == path.bytes.index(after: path.bytes.startIndex))+      // TODO: what about when we're done parsing and we have null terminator?+    }+    return path.bytes[i...].firstIndex(where: { !isSeparator($0) }) ?? endIndex._storage+  }++  public func index(after i: Index) -> Index {+    let end = parseComponentEnd(startingAt: i._storage)+    if Index(end) == endIndex {+      return endIndex+    }+    return Index(parseNextComponentStart(afterComponentEnd: end))+  }++  // Find the start of the component prior to the  after the end of the prior at `i`+  private func parseComponentStart(+    endingAt i: Index.Storage+  ) -> Index.Storage {+    assert(i != startIndex._storage)++    return path.bytes[i...].firstIndex(where: { !isSeparator($0) }) ?? startIndex._storage+  }++  // Chew through separators until we get to a component end+  private func parseComponentEnd(fromStart i: Index) -> Index.Storage {+    let slice = path.bytes[..<i._storage]+    return slice.lastIndex(where: { isSeparator($0) }) ?? startIndex._storage+  }++  public func index(before i: Index) -> Index {+    var slice = path.bytes[..<i._storage]+    while let c = slice.last, isSeparator(c) {+      slice.removeLast()+    }+    while let c = slice.last, !isSeparator(c) {+      slice.removeLast()+    }++    return Index(slice.endIndex)+  }++  public subscript(position: Index) -> FilePath.Component {+    let i = position+    let end = parseComponentEnd(startingAt: i._storage)+    return FilePath.Component(path.bytes[i._storage ..< end])+  }+}++// @available(macOS 9999, iOS 9999, watchOS 9999, tvOS 9999, *)+extension FilePath.ComponentView: RangeReplaceableCollection {+  public init() {+    self.init(path: FilePath())+  }++  public mutating func replaceSubrange<C: Collection>(+    _ subrange: Range<Index>, with newElements: C+  ) where Element == C.Element {+    let (lowerBound, upperBound) = (subrange.lowerBound, subrange.upperBound)++    let pathRange = lowerBound._storage ..< upperBound._storage+    guard !newElements.isEmpty else {+      path.bytes.removeSubrange(pathRange)+      return+    }++    // Insertion skips roots+    let hasNewComponents = !newElements.lazy.filter { !$0.isRoot }.isEmpty++    // Explicitly add a trailing separator if+    //   not at end and next character is not a separator+    let atEnd = upperBound == endIndex+    let trailingSeparator = !atEnd && !isSeparator(path.bytes[upperBound._storage])++    // Explicitly add a preceding separator if+    //   replacing front with absolute components (unless redundant by trailing separator),+    //   preceding character is not a separator (which implies at the end)+    let atStart = lowerBound == startIndex

TODO: Account for // special case...

Multiple successive <slash> characters are considered to be the same as one <slash>, except for the case of exactly two leading <slash> characters.

https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap03.html#tag_03_271

milseman

comment created time in 22 days

PullRequestReviewEvent
PullRequestReviewEvent

issue closedapple/swift-system

Divine Intervention

closed time in 23 days

jefferycline1

issue commentapple/swift-system

Divine Intervention

I didn't catch if this issue had any content originally, but now it is blank. Closing.

jefferycline1

comment created time in 23 days

issue commentapple/swift-system

Question: why are @available directives commented out?

This is due to language/tooling issues. We'd like the ability to add availability based on conditional compilation for ABI-stable builds of Swift System while not adding availability for package builds (i.e. a source dependency). For now, we include, but comment out, the availability declarations matching what has already been shipped in binary releases.

BasThomas

comment created time in 23 days

Pull request review commentapple/swift-system

Syscall mock testing

+/*+ This source file is part of the Swift System open source project++ Copyright (c) 2020 Apple Inc. and the Swift System project authors+ Licensed under Apache License v2.0 with Runtime Library Exception++ See https://swift.org/LICENSE.txt for license information+*/++// Syscall mocking support.+//+// NOTE: This is currently the bare minimum needed for System's testing purposes, though we do+// eventually want to expose some solution to users.+//+// Mocking is contextual, accessible through MockingDriver.withMockingEnabled. Mocking+// state, including whether it is enabled, is stored in thread-local storage. Mocking is only+// enabled in testing builds of System currently, to minimize runtime overhead of release builds.+//++public struct Trace {+  public struct Entry: Hashable {+    var name: String+    var arguments: [AnyHashable]++    public init(name: String, _ arguments: [AnyHashable]) {+      self.name = name+      self.arguments = arguments+    }+  }++  private var entries: [Entry] = []+  private var firstEntry: Int = 0++  public var isEmpty: Bool { firstEntry >= entries.count }++  public mutating func dequeue() -> Entry? {+    guard !self.isEmpty else { return nil }+    defer { firstEntry += 1 }+    return entries[firstEntry]+  }++  internal mutating func add(_ e: Entry) {+    entries.append(e)+  }++  public mutating func clear() { entries.removeAll() }+}++// TODO: Track+public struct WriteBuffer {+  public var enabled: Bool = false++  private var buffer: [UInt8] = []+  private var chunkSize: Int? = nil++  internal mutating func write(_ buf: UnsafeRawBufferPointer) -> Int {+    guard enabled else { return 0 }+    let chunk = chunkSize ?? buf.count+    buffer.append(contentsOf: buf.prefix(chunk))+    return chunk+  }++  public var contents: [UInt8] { buffer }+}++public enum ForceErrno {+  case none+  case always(errno: CInt)++  case counted(errno: CInt, count: Int)+}++// Provide access to the driver, context, and trace stack of mocking+public class MockingDriver {+  // Whether to bypass this shim and go straight to the syscall+  public var enableMocking = false++  // Record syscalls and their arguments+  public var trace = Trace()++  // Mock errors inside syscalls+  public var forceErrno = ForceErrno.none++  // A buffer to put `write` bytes into+  public var writeBuffer = WriteBuffer()+}++#if os(macOS) || os(iOS) || os(watchOS) || os(tvOS)+import Darwin+#elseif os(Linux) || os(FreeBSD) || os(Android)+import Glibc+#else+#error("Unsupported Platform")+#endif++internal let key: pthread_key_t = {+  var raw = pthread_key_t()+  func releaseObject(_ raw: UnsafeMutableRawPointer) -> () {+    Unmanaged<MockingDriver>.fromOpaque(raw).release()+  }+  guard 0 == pthread_key_create(&raw, releaseObject) else {+    fatalError("Unable to create key")+  }+  // TODO: All threads are sharing the same object; this is wrong

withMockingEnabled does swap out the state, but I think we can simplify it by having it swap out a reference. Good idea, and we should also add a test for that.

milseman

comment created time in a month

PullRequestReviewEvent
PullRequestReviewEvent

Pull request review commentapple/swift

[stdlib] Add `_StringGuts.init(immortal:isASCII:)`

 extension _StringGuts {     self.init(_StringObject(smol))   } +  @_alwaysEmitIntoClient   @inlinable @inline(__always)-  internal init(_ bufPtr: UnsafeBufferPointer<UInt8>, isASCII: Bool) {+  internal init(immortal bufPtr: UnsafeBufferPointer<UInt8>, isASCII: Bool) {     self.init(_StringObject(immortal: bufPtr, isASCII: isASCII))   }

For back-deployment, yes.

benrimmington

comment created time in 3 months

Pull request review commentapple/swift

[stdlib] Add `_StringGuts.init(immortal:isASCII:)`

 extension String.UTF16View {   @inlinable @inline(__always)   internal var _shortHeuristic: Int { return 32 } }++extension _StringGuts {+  @available(*, unavailable, renamed: "_StringGuts.init(immortal:isASCII:)")+  @inlinable @inline(__always)+  internal init(_ bufPtr: UnsafeBufferPointer<UInt8>, isASCII: Bool) {+    self.init(_StringObject(immortal: bufPtr, isASCII: isASCII))+  }+}

CC @lorentey , does unavailable impact the ABI and compatibility story?

benrimmington

comment created time in 3 months

Pull request review commentapple/swift

[stdlib] Add `_StringGuts.init(immortal:isASCII:)`

 extension String: TextOutputStream {   }    public mutating func _writeASCII(_ buffer: UnsafeBufferPointer<UInt8>) {-    self._guts.append(_StringGuts(buffer, isASCII: true))+    if let smol = _SmallString(buffer) {+      _guts.append(_StringGuts(smol))+    } else {+      _guts.reserveCapacity(_guts.utf8Count + buffer.count)+      _guts.appendInPlace(buffer, isASCII: true)+    }

Out of curiosity, why did you choose reserveCapacity rather than prepareForAppendInPlace? reserveCapacity is not meant to handle append-like growth and though it might happen to put the string into a state for appendInPlace that's not part of its contract. All of this is handled by the existing append taking a StringGuts, hence why we were just calling it directly.

Since the contents of the append are non-escaping, should we just form an immortal guts?

benrimmington

comment created time in 3 months

pull request commentapple/swift

[SE-0163] Migrate from deprecated Unicode APIs (part 3)

LGTM

benrimmington

comment created time in 3 months

Pull request review commentapple/swift

[SE-0163] Migrate from deprecated Unicode APIs

 extension String { }  extension String.UnicodeScalarView: _CustomPlaygroundQuickLookable {-  @available(swift, deprecated: 4.2/*, obsoleted: 5.0*/, message: "UnicodeScalarView.customPlaygroundQuickLook will be removed in Swift 5.0")+  @available(swift, deprecated: 4.2, /*obsoleted: TODO,*/ message: "UnicodeScalarView.customPlaygroundQuickLook will be removed in a future Swift version")   public var customPlaygroundQuickLook: _PlaygroundQuickLook {     return .text(description)   } } -//===--- Slicing Support --------------------------------------------------===//--// @available(swift,deprecated: 5.0, renamed: "Unicode.UTF8")+@available(swift, deprecated: 100000, /*obsoleted: TODO,*/ renamed: "Unicode.UTF8")

Also, that batch of SEs were part of a larger attempt to reformulate how strings were formulated that didn't pan out in practice. They have several portions that weren't fully thought through, much less implemented, so I don't think we want to blindly apply them without some reevaluation.

benrimmington

comment created time in 3 months

Pull request review commentapple/swift

[stdlib] Add `_StringGuts.init(immortal:isASCII:)`

 extension String: TextOutputStream {   }    public mutating func _writeASCII(_ buffer: UnsafeBufferPointer<UInt8>) {-    self._guts.append(_StringGuts(buffer, isASCII: true))+    if let smol = _SmallString(buffer) {+      _guts.append(_StringGuts(smol))+    } else {+      _guts.reserveCapacity(_guts.utf8Count + buffer.count)+      _guts.appendInPlace(buffer, isASCII: true)+    }

Were you finding this to produce better performance than what was there before?

benrimmington

comment created time in 3 months

Pull request review commentapple/swift

[stdlib] Add `_StringGuts.init(immortal:isASCII:)`

 extension _StringGuts {   }    @inlinable @inline(__always)-  internal init(_ bufPtr: UnsafeBufferPointer<UInt8>, isASCII: Bool) {+  internal init(immortal bufPtr: UnsafeBufferPointer<UInt8>, isASCII: Bool) {     self.init(_StringObject(immortal: bufPtr, isASCII: isASCII))   } +  @usableFromInline+  internal init(_ bufPtr: UnsafeBufferPointer<UInt8>, isASCII: Bool) {

Do you know where this is being called from? I want to make sure this change doesn't accidentally affect callers we're not sure of. You could consider adding a label copying or some such to make it clearer what will happen.

benrimmington

comment created time in 3 months

pull request commentapple/swift

[SE-0163] Migrate from deprecated Unicode APIs

I realize this went through SE quite a while ago, but I think it's a bad idea to formally remove the Unicode aliases, especially for UTF8/UTF16.

benrimmington

comment created time in 3 months

Pull request review commentapple/swift

[stdlib][string] Refactors String(repeating:count:)

 extension String {       self = count == 0 ? "" : repeatedValue       return     }+    guard !repeatedValue.isEmpty else {+      self = ""+      return+    } -    // TODO(String performance): We can directly call appendInPlace-    var result = String()-    result.reserveCapacity(repeatedValue._guts.count &* count)-    for _ in 0..<count {-      result += repeatedValue+    var repeatedValue = repeatedValue+    self = repeatedValue.withUTF8 { repeatedUTF8 in+      String(_uninitializedCapacity: repeatedUTF8.count * count) { buffer in

Yes, it has to be the same. withUTF8 doesn't include the null terminator (nor does the UTF8View of course).

valeriyvan

comment created time in 3 months

Pull request review commentapple/swift

[stdlib][string] Refactors String(repeating:count:)

 extension String {       self = count == 0 ? "" : repeatedValue       return     }+    guard !repeatedValue.isEmpty else {+      self = ""+      return+    } -    // TODO(String performance): We can directly call appendInPlace-    var result = String()-    result.reserveCapacity(repeatedValue._guts.count &* count)-    for _ in 0..<count {-      result += repeatedValue+    var repeatedValue = repeatedValue+    self = repeatedValue.withUTF8 { repeatedUTF8 in+      String(_uninitializedCapacity: repeatedUTF8.count * count) { buffer in+        var total = 0+        for i in 0..<count {+          let offset = i &* repeatedUTF8.count+          let range = offset ..< offset + repeatedUTF8.count+          _ = UnsafeMutableBufferPointer(rebasing: buffer[range])+            .initialize(from: repeatedUTF8)+          total += range.count+        }+        return total

If you're going to track total separately from the uninitialized capacity, could you add an assert that they're the same?

valeriyvan

comment created time in 3 months

Pull request review commentapple/swift

[stdlib][string] Refactors String(repeating:count:)

 StringTests.test("NormalizationCheck/Opaque") #endif } +StringTests.test("StringRepeating/SingleAsciiCharacterCount10") {+    expectEqual("xxxxxxxxxx", String(repeating: "x", count: 10))+}++StringTests.test("StringRepeating/SingleAsciiCharacterCount1") {+    expectEqual("x", String(repeating: "x", count: 1))+}++StringTests.test("StringRepeating/EmptyStringCount10") {+    expectEqual("", String(repeating: "", count: 10))+}++StringTests.test("StringRepeating/SingleAsciiCharacterCount0") {+    expectEqual("", String(repeating: "x", count: 0))+}++StringTests.test("StringRepeating/MultipleAsciiCharactersCount2") {+    expectEqual("abcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyz",+                String(repeating: "abcdefghijklmnopqrstuvwxyz", count: 2))+}++StringTests.test("StringRepeating/SingleCyrilicCharacterCount5") {+    expectEqual("яяяяя", String(repeating: "я", count: 5))+}++StringTests.test("StringRepeating/MultipleCyrilicCharactersCount2") {+    expectEqual("абвгґдеєжзиіїйклмнопрстуфхцчшщьюяабвгґдеєжзиіїйклмнопрстуфхцчшщьюя",+                String(repeating: "абвгґдеєжзиіїйклмнопрстуфхцчшщьюя", count: 2))+}++StringTests.test("StringRepeating/\\u{1F1F8}\\u{1F1FA}Count2") {+    expectEqual("\u{1F1F8}\u{1F1FA}\u{1F1F8}\u{1F1FA}" /* 🇸🇺🇸🇺 */,+                String(repeating: "\u{1F1F8}\u{1F1FA}" /* 🇸🇺 */, count: 2))+}++StringTests.test("StringRepeating/\\u{301}cafeCount5") {+    expectEqual("\u{301}cafécafécafécafécafe",+                String(repeating: "\u{301}cafe", count: 5))+}+

Thanks for the increased testing!

valeriyvan

comment created time in 3 months

Pull request review commentapple/swift

[stdlib][string] Adds benchmark for String(repeating:count:)

+//===--- StringRepeating.swift -------------------------------------------===//+//+// This source file is part of the Swift.org open source project+//+// Copyright (c) 2014 - 2019 Apple Inc. and the Swift project authors+// Licensed under Apache License v2.0 with Runtime Library Exception+//+// See https://swift.org/LICENSE.txt for license information+// See https://swift.org/CONTRIBUTORS.txt for the list of Swift project authors+//+//===----------------------------------------------------------------------===//++import TestsUtils++public let StringRepeating = [+  BenchmarkInfo(name: "StringRepeating/SingleAsciiCharacterCount10",+                runFunction: run_singleAsciiCharacterCount10,+                tags: [.validation, .api, .String]),+  BenchmarkInfo(name: "StringRepeating/SingleAsciiCharacterCount1",+                runFunction: run_singleAsciiCharacterCount1,+                tags: [.validation, .api, .String]),+  BenchmarkInfo(name: "StringRepeating/EmptyStringCount10",+                runFunction: run_emptyStringCount10,+                tags: [.validation, .api, .String]),+  BenchmarkInfo(name: "StringRepeating/SingleAsciiCharacterCount0",+                runFunction: run_singleAsciiCharacterCount0,+                tags: [.validation, .api, .String]),+  BenchmarkInfo(name: "StringRepeating/26AsciiCharactersCount2",+                runFunction: run_26AsciiCharactersCount2,+                tags: [.validation, .api, .String]),+  BenchmarkInfo(name: "StringRepeating/SingleCyrilicCharacterCount5",+                runFunction: run_singleCyrilicCharacterCount5,+                tags: [.validation, .api, .String]),+  BenchmarkInfo(name: "StringRepeating/33CyrilicCharactersCount2",+                runFunction: run_33CyrilicCharactersCount2,+                tags: [.validation, .api, .String]),+  BenchmarkInfo(name: "StringRepeating/U1F1F8U1F1FACount2",+                runFunction: run_U1F1F8U1F1FACount2,+                tags: [.validation, .api, .String]),+  BenchmarkInfo(name: "StringRepeating/U301cafeCount5",+                runFunction: run_U301cafeCount5,+                tags: [.validation, .api, .String]),+  BenchmarkInfo(name: "StringRepeating/LongMixedStringCount100",+                runFunction: run_longMixedStringCount100,+                tags: [.validation, .api, .String])+]++@inline(never)+func repeating(_ i: String, count: Int) -> String {+  let s = String(repeating: getString(i), count: count)+  return s+}++@inline(never)+public func run_singleAsciiCharacterCount10(N: Int) {+    for _ in 1...5000*N {+        blackHole(repeating("x", count: 10))+    }+}++@inline(never)+public func run_singleAsciiCharacterCount1(N: Int) {+    for _ in 1...5000*N {+        blackHole(repeating("x", count: 1))+    }+}

Are you seeing any meaningful difference between this and the 10 versions above? That would be surprising.

valeriyvan

comment created time in 3 months

Pull request review commentapple/swift

[stdlib][string] Adds benchmark for String(repeating:count:)

+//===--- StringRepeating.swift -------------------------------------------===//+//+// This source file is part of the Swift.org open source project+//+// Copyright (c) 2014 - 2019 Apple Inc. and the Swift project authors+// Licensed under Apache License v2.0 with Runtime Library Exception+//+// See https://swift.org/LICENSE.txt for license information+// See https://swift.org/CONTRIBUTORS.txt for the list of Swift project authors+//+//===----------------------------------------------------------------------===//++import TestsUtils++public let StringRepeating = [+  BenchmarkInfo(name: "StringRepeating/SingleAsciiCharacterCount10",+                runFunction: run_singleAsciiCharacterCount10,+                tags: [.validation, .api, .String]),+  BenchmarkInfo(name: "StringRepeating/SingleAsciiCharacterCount1",+                runFunction: run_singleAsciiCharacterCount1,+                tags: [.validation, .api, .String]),+  BenchmarkInfo(name: "StringRepeating/EmptyStringCount10",+                runFunction: run_emptyStringCount10,+                tags: [.validation, .api, .String]),+  BenchmarkInfo(name: "StringRepeating/SingleAsciiCharacterCount0",+                runFunction: run_singleAsciiCharacterCount0,+                tags: [.validation, .api, .String]),+  BenchmarkInfo(name: "StringRepeating/26AsciiCharactersCount2",+                runFunction: run_26AsciiCharactersCount2,+                tags: [.validation, .api, .String]),+  BenchmarkInfo(name: "StringRepeating/SingleCyrilicCharacterCount5",+                runFunction: run_singleCyrilicCharacterCount5,+                tags: [.validation, .api, .String]),+  BenchmarkInfo(name: "StringRepeating/33CyrilicCharactersCount2",+                runFunction: run_33CyrilicCharactersCount2,+                tags: [.validation, .api, .String]),+  BenchmarkInfo(name: "StringRepeating/U1F1F8U1F1FACount2",+                runFunction: run_U1F1F8U1F1FACount2,+                tags: [.validation, .api, .String]),+  BenchmarkInfo(name: "StringRepeating/U301cafeCount5",+                runFunction: run_U301cafeCount5,+                tags: [.validation, .api, .String]),+  BenchmarkInfo(name: "StringRepeating/LongMixedStringCount100",+                runFunction: run_longMixedStringCount100,+                tags: [.validation, .api, .String])+]

What is the rationale for this set of cases? Some seem redundant, and I would think we'd be more interested in small vs non-small forms

valeriyvan

comment created time in 3 months

Pull request review commentapple/swift

[stdlib] adds fast path to internal func foreignErrorCorrectedGrapheme

 extension _StringGuts {         startingAt: String.Index(_encodedOffset: start)       ).0))     }+    let range = start..<end+    let from = self._object.cocoaObject++    let capacity = 16 * 4

What is the relationship between this capacity and the actual fixed array size? Why this capacity? Seems like 16 code units would be more than enough for anything non-pathological. What use cases were you seeing?

valeriyvan

comment created time in 3 months

more