profile
viewpoint
If you are wondering where the data of this site comes from, please visit https://api.github.com/users/jrudolph/events. GitMemory does not store any data, but only uses NGINX to cache data for a period of time. The idea behind GitMemory is simply to give users a better reading experience.
Johannes Rudolph jrudolph Lightbend Freiburg, Germany https://virtual-void.net

jrudolph/akka-http-scala-js-websocket-chat 327

An example app that integrates akka-http and scala-js to implement a websocket chat

jrudolph/bootpgm 15

Demonstration of a Windows Boot Program using Window's Native API

jrudolph/better-future-exceptions 12

A POC for better exception reporting for futures

jrudolph/akka-graal 8

Testing Graal VM AOT compilation for Akka

jrudolph/akka-http-workshop 4

Source code and instructions for the akka-http tutorial at bobkonf 2016 in Berlin

jrudolph/android-plugin 2

An sbt plugin for Android development in Scala

jrudolph/bit-extractors 2

Scala extractors for pattern matching values from bits

jrudolph/akka 1

Akka Project

danielwegener/spray 0

A suite of scala libraries for building and consuming RESTful web services on top of Akka: lightweight, asynchronous, non-blocking, actor-based, testable

jrudolph/akka-grpc 0

Akka gRPC

pull request commentspray/spray-json

WIP quick experiment building with scala 3

ProductFormats don't work any more, probably because Scala compilation changed the way fields are encoded

Ultimately, we would want to use derivation in Scala 3, but I guess it would be nice to support it in a simpler way until then?

raboof

comment created time in 3 days

pull request commentspray/spray-json

WIP quick experiment building with scala 3

Cool, I fixed all the compilation issues but now it seems that

  • some printing facility changed, now escaping unicode characters
  • ProductFormats don't work any more, probably because Scala compilation changed the way fields are encoded
raboof

comment created time in 3 days

push eventraboof/spray-json

Johannes Rudolph

commit sha 9ade68e31bddb67d6a0a32d88d10521ee928fb9e

more type annotations

view details

Johannes Rudolph

commit sha 97c7c7dee5d31ef43bdc1f208753813c1d8d1b78

Scala 3 compatible by-name param usage

view details

push time in 3 days

created tagakka/akka-http

tagv10.2.5-M2

The Streaming-first HTTP server/module of Akka

created time in 3 days

release akka/akka-http

v10.2.5-M2

released time in 3 days

pull request commentspray/spray-json

WIP quick experiment building with scala 3

I added a fix for the self-type issue but the tests don't compile cleanly any more, mostly because the new version of specs is not syntax compatible with the old.

raboof

comment created time in 3 days

push eventraboof/spray-json

Johannes Rudolph

commit sha ff53f9d244d8afac7f560e41b89736cf474087e4

some Scala 3 fixes

view details

push time in 3 days

push eventraboof/spray-json

Johannes Rudolph

commit sha 6ab4676bed3a87a0c9b25002c48bd26c63fb3788

fix self type for Scala 3

view details

push time in 3 days

delete branch jrudolph/akka-http

delete branch : optimize-multiplexer

delete time in 3 days

push eventakka/akka-http

Johannes Rudolph

commit sha 867a8a5ec35fb20f1bcc8ce00a63e04f8db66586

http2: use mutable.Queue for state in multiplexer (#3873) Co-authored-by: Arnout Engelen <arnout@engelen.eu>

view details

push time in 3 days

PR merged akka/akka-http

http2: use mutable.Queue for state in multiplexer tested

To avoid expensive frequent updates to persistent collections like Set or Vector. Beneficial side-effect: states can all be constant values.

+103 -91

6 comments

4 changed files

jrudolph

pr closed time in 3 days

Pull request review commentakka/akka-http

http2: use mutable.Queue for state in multiplexer

 private[http2] trait Http2MultiplexerSupport { logic: GraphStageLogic with Stage          if (isDebugEnabled && newState.name != oldState.name) recordStateChange(oldState.name, newState.name)         if (allDataFlushed(newState)) onAllDataFlushed()+        allowReadingIncomingFrames(controlFrameBuffer.size < settings.outgoingControlFrameBufferSize)       } -      private[http2] sealed trait MultiplexerState extends Product {+      sealed trait MultiplexerState extends Product {         def name: String = productPrefix          def onPull(): MultiplexerState+        @silent("references private")         def pushControlFrame(frame: FrameEvent): MultiplexerState         def connectionWindowAvailable(): MultiplexerState         def enqueueOutStream(streamId: Int): MultiplexerState         def closeStream(streamId: Int): MultiplexerState -        protected def sendDataFrame(streamId: Int, sendableOutstreams: immutable.Set[Int]): MultiplexerState = {+        protected def sendDataFrame(streamId: Int): MultiplexerState = {           val maxBytesToSend = currentMaxFrameSize min connectionWindowLeft           val result = pullNextFrame(streamId, maxBytesToSend)-          val frame = result.frame-          pushFrameOut(frame)-          connectionWindowLeft -= frame.payload.length+          def send(frame: DataFrame): Unit = {+            pushFrameOut(frame)+            connectionWindowLeft -= frame.payload.length+          }            result match {-            case PullFrameResult.SendFrame(_, hasMore) =>-              if (hasMore) WaitingForNetworkToSendData(sendableOutstreams + streamId)-              else {-                val remainingStreams = sendableOutstreams - streamId-                if (remainingStreams.isEmpty) Idle-                else WaitingForNetworkToSendData(remainingStreams)+            case PullFrameResult.SendFrame(frame, hasMore) =>+              send(frame)+              if (hasMore) {+                sendableOutstreams += streamId+                WaitingForNetworkToSendData+              } else {+                if (sendableOutstreams.isEmpty) Idle+                else WaitingForNetworkToSendData               }-            case PullFrameResult.SendFrameAndTrailer(_, trailer) =>-              WaitingForNetworkToSendControlFrames(Vector(trailer), sendableOutstreams - streamId)+            case PullFrameResult.SendFrameAndTrailer(frame, trailer) =>+              send(frame)+              controlFrameBuffer += trailer+              WaitingForNetworkToSendControlFrames+            case PullFrameResult.NothingToSend =>

I added #3882 to make an attempt to revisit this complexity.

jrudolph

comment created time in 3 days

PullRequestReviewEvent

issue openedakka/akka-http

Cleanup Http2StreamHandling vs. Http2Multiplexer

There have been some attempts to clean up the interface between stream handling and multiplexer but some issues still remain:

  • The call path of Sending.pullNextFrame -> OutStream.nextFrame / endStreamIfPossible is more complex than necessary.
  • A stream might enqueue itself multiple times because the above call path makes it harder to keep track in OutStream whether a stream is currently enqueued or not.
  • We needed to introduce NothingToSend to deal with duplicate entries in the sendableOutStream queue in #3873. This can be removed once we can ensure that no streams are enqueued multiple times.

created time in 3 days

push eventjrudolph/akka-http

Johannes Rudolph

commit sha 4c614ef826114a735f14f70746ecccedf1263e1c

add another comment about things work

view details

push time in 3 days

Pull request review commentakka/akka-http

http2: use mutable.Queue for state in multiplexer

 private[http2] trait Http2MultiplexerSupport { logic: GraphStageLogic with Stage          if (isDebugEnabled && newState.name != oldState.name) recordStateChange(oldState.name, newState.name)         if (allDataFlushed(newState)) onAllDataFlushed()+        allowReadingIncomingFrames(controlFrameBuffer.size < settings.outgoingControlFrameBufferSize)       } -      private[http2] sealed trait MultiplexerState extends Product {+      sealed trait MultiplexerState extends Product {         def name: String = productPrefix          def onPull(): MultiplexerState+        @silent("references private")         def pushControlFrame(frame: FrameEvent): MultiplexerState         def connectionWindowAvailable(): MultiplexerState         def enqueueOutStream(streamId: Int): MultiplexerState         def closeStream(streamId: Int): MultiplexerState -        protected def sendDataFrame(streamId: Int, sendableOutstreams: immutable.Set[Int]): MultiplexerState = {+        protected def sendDataFrame(streamId: Int): MultiplexerState = {           val maxBytesToSend = currentMaxFrameSize min connectionWindowLeft           val result = pullNextFrame(streamId, maxBytesToSend)-          val frame = result.frame-          pushFrameOut(frame)-          connectionWindowLeft -= frame.payload.length+          def send(frame: DataFrame): Unit = {+            pushFrameOut(frame)+            connectionWindowLeft -= frame.payload.length+          }            result match {-            case PullFrameResult.SendFrame(_, hasMore) =>-              if (hasMore) WaitingForNetworkToSendData(sendableOutstreams + streamId)-              else {-                val remainingStreams = sendableOutstreams - streamId-                if (remainingStreams.isEmpty) Idle-                else WaitingForNetworkToSendData(remainingStreams)+            case PullFrameResult.SendFrame(frame, hasMore) =>+              send(frame)+              if (hasMore) {+                sendableOutstreams += streamId+                WaitingForNetworkToSendData+              } else {+                if (sendableOutstreams.isEmpty) Idle+                else WaitingForNetworkToSendData               }-            case PullFrameResult.SendFrameAndTrailer(_, trailer) =>-              WaitingForNetworkToSendControlFrames(Vector(trailer), sendableOutstreams - streamId)+            case PullFrameResult.SendFrameAndTrailer(frame, trailer) =>+              send(frame)+              controlFrameBuffer += trailer+              WaitingForNetworkToSendControlFrames+            case PullFrameResult.NothingToSend =>

I found another reason for the current logic:

With the queues we allow duplicates in the queue to be able to keep enqueueOutStream idempotent (without having to scan the queue for existing elements). We currently might add duplicates to the queue e.g. in OutStream.increaseWindow.

To get rid of NothingToSend, we would have to ensure that a stream is never put into the queue twice by keeping a flag on the OutStream side whether we are currently enqueued or not. Not sure if that's worth it right now.

jrudolph

comment created time in 3 days

PullRequestReviewEvent

Pull request review commentakka/akka-http

http2: use mutable.Queue for state in multiplexer

 private[http2] trait Http2MultiplexerSupport { logic: GraphStageLogic with Stage          if (isDebugEnabled && newState.name != oldState.name) recordStateChange(oldState.name, newState.name)         if (allDataFlushed(newState)) onAllDataFlushed()+        allowReadingIncomingFrames(controlFrameBuffer.size < settings.outgoingControlFrameBufferSize)       } -      private[http2] sealed trait MultiplexerState extends Product {+      sealed trait MultiplexerState extends Product {         def name: String = productPrefix          def onPull(): MultiplexerState+        @silent("references private")         def pushControlFrame(frame: FrameEvent): MultiplexerState         def connectionWindowAvailable(): MultiplexerState         def enqueueOutStream(streamId: Int): MultiplexerState         def closeStream(streamId: Int): MultiplexerState -        protected def sendDataFrame(streamId: Int, sendableOutstreams: immutable.Set[Int]): MultiplexerState = {+        protected def sendDataFrame(streamId: Int): MultiplexerState = {           val maxBytesToSend = currentMaxFrameSize min connectionWindowLeft           val result = pullNextFrame(streamId, maxBytesToSend)-          val frame = result.frame-          pushFrameOut(frame)-          connectionWindowLeft -= frame.payload.length+          def send(frame: DataFrame): Unit = {+            pushFrameOut(frame)+            connectionWindowLeft -= frame.payload.length+          }            result match {-            case PullFrameResult.SendFrame(_, hasMore) =>-              if (hasMore) WaitingForNetworkToSendData(sendableOutstreams + streamId)-              else {-                val remainingStreams = sendableOutstreams - streamId-                if (remainingStreams.isEmpty) Idle-                else WaitingForNetworkToSendData(remainingStreams)+            case PullFrameResult.SendFrame(frame, hasMore) =>+              send(frame)+              if (hasMore) {+                sendableOutstreams += streamId+                WaitingForNetworkToSendData+              } else {+                if (sendableOutstreams.isEmpty) Idle+                else WaitingForNetworkToSendData               }-            case PullFrameResult.SendFrameAndTrailer(_, trailer) =>-              WaitingForNetworkToSendControlFrames(Vector(trailer), sendableOutstreams - streamId)+            case PullFrameResult.SendFrameAndTrailer(frame, trailer) =>+              send(frame)+              controlFrameBuffer += trailer+              WaitingForNetworkToSendControlFrames+            case PullFrameResult.NothingToSend =>

So the motivation is that looking for the stream in closeStream is more expensive than just keeping it in there until we encounter it here and go to WaitingForData / WaitingForNetworkToSendData.onPull()

Yes.

I guess it depends on how often closeStream is called. The intuition would say, probably not so often, as it's only needed if a stream is cancelled and that should be exceptional. So, removing single elements in O(n) would be fine as long as not a whole bunch of streams out of many is removed and we run into a O(n^2) situation.

Maybe I'll run another benchmark and see if that would happen in normal situations.

jrudolph

comment created time in 3 days

PullRequestReviewEvent

push eventjrudolph/akka-http

Johannes Rudolph

commit sha 8bb88293f0b71ebd54f5baaf63d0731708a4eb44

Apply suggestions from code review Co-authored-by: Arnout Engelen <arnout@engelen.eu>

view details

push time in 3 days

Pull request review commentakka/akka-http

http2: use mutable.Queue for state in multiplexer

 private[http2] trait Http2MultiplexerSupport { logic: GraphStageLogic with Stage         }         def connectionWindowAvailable(): MultiplexerState = this // nothing to do, as there is no data to send         def enqueueOutStream(streamId: Int): MultiplexerState =-          if (connectionWindowLeft == 0) WaitingForConnectionWindow(immutable.TreeSet(streamId))-          else sendDataFrame(streamId, Set.empty)+          if (connectionWindowLeft == 0) {+            sendableOutstreams += streamId+            WaitingForConnectionWindow+          } else sendDataFrame(streamId)         def closeStream(streamId: Int): MultiplexerState = this       }        /** Not yet pulled but data waiting to be sent */-      private[http2] case class WaitingForNetworkToSendControlFrames(controlFrameBuffer: immutable.Vector[FrameEvent], sendableOutstreams: immutable.Set[Int]) extends MultiplexerState {-        require(controlFrameBuffer.nonEmpty)-        allowReadingIncomingFrames(controlFrameBuffer.size < settings.outgoingControlFrameBufferSize)-        def onPull(): MultiplexerState = controlFrameBuffer match {-          case first +: remaining =>-            pushFrameOut(first)-            allowReadingIncomingFrames(remaining.length < settings.outgoingControlFrameBufferSize)-            if (remaining.isEmpty && sendableOutstreams.isEmpty) Idle-            else if (remaining.isEmpty) WaitingForNetworkToSendData(sendableOutstreams)-            else copy(remaining, sendableOutstreams)+      case object WaitingForNetworkToSendControlFrames extends MultiplexerState {+        def onPull(): MultiplexerState = {+          val first = controlFrameBuffer.dequeue()+          pushFrameOut(first)+          if (controlFrameBuffer.isEmpty && sendableOutstreams.isEmpty) Idle+          else if (controlFrameBuffer.isEmpty) WaitingForNetworkToSendData+          else this+        }+        def pushControlFrame(frame: FrameEvent): MultiplexerState = {+          controlFrameBuffer += frame+          this         }-        def pushControlFrame(frame: FrameEvent): MultiplexerState = copy(controlFrameBuffer = controlFrameBuffer :+ frame)         def connectionWindowAvailable(): MultiplexerState = this-        def enqueueOutStream(streamId: Int): MultiplexerState =-          if (!sendableOutstreams.contains(streamId))-            copy(sendableOutstreams = sendableOutstreams + streamId)-          else-            this+        def enqueueOutStream(streamId: Int): MultiplexerState = {+          sendableOutstreams += streamId+          this+        } -        def closeStream(streamId: Int): MultiplexerState =-          if (sendableOutstreams.contains(streamId)) {-            val sendableExceptClosed = sendableOutstreams - streamId-            copy(sendableOutstreams = sendableExceptClosed)-          } else-            this+        def closeStream(streamId: Int): MultiplexerState = {+          // leave stream in sendableOutstreams, to be skipped in sendNextFrame

Indeed, thanks.

jrudolph

comment created time in 3 days

PullRequestReviewEvent

push eventakka/akka

Johannes Rudolph

commit sha 82aa15e6ab7b93cc874067d10e87caef8417fb7c

io: probe TCP socket when reading before registering interest (#30354) Just asking once for more data is cheaper than instantly updating epoll.

view details

push time in 3 days

delete branch jrudolph/akka

delete branch : tcp-avoid-read-interest-if-not-necessary

delete time in 3 days

PR merged akka/akka

io: probe TCP socket when reading before registering interest tested

Just asking once for more data is cheaper than instantly updating epoll.

+18 -19

3 comments

3 changed files

jrudolph

pr closed time in 3 days

issue commentjrudolph/json-lenses

Jsonpath: support && and ||

You would need to introduce new classes

case class And(p1: Predicate, p2: Predicate) extends Predicate
// same for or

in JsonPath.scala, then add a parser for that in JsonPathParser.scala, and an implementation/conversion in JsonPathIntegration.scala.

Amerousful

comment created time in 4 days

issue commentjrudolph/json-lenses

Jsonpath: support && and ||

I'm not planning any new development myself right now but I'm fine with releasing new versions if there are contributions to be released.

This feature should be relatively easy to add, so I'd be happy to look at a PR and release a new version when needed.

Amerousful

comment created time in 4 days

issue commentStarefossen/docker-github-pages

Could not find i18n-0.8.6 in any of the sources

Removing Gemfile.lock helped for me

wcoder

comment created time in 5 days

PullRequestReviewEvent

Pull request review commentakka/akka-http

core: simplify HeaderCompression

 private[http2] object HeaderCompression extends GraphStage[FlowShape[FrameEvent,    val shape = FlowShape(eventsIn, eventsOut) -  def createLogic(inheritedAttributes: Attributes): GraphStageLogic = new HandleOrPassOnStage[FrameEvent, FrameEvent](shape) with StageLogging {-    val currentMaxFrameSize = Http2Protocol.InitialMaxFrameSize+  def createLogic(inheritedAttributes: Attributes): GraphStageLogic = new GraphStageLogic(shape) with StageLogging with InHandler with OutHandler { logic =>+    setHandlers(eventsIn, eventsOut, this+    )+    private val currentMaxFrameSize = Http2Protocol.InitialMaxFrameSize      val encoder = new akka.http.shaded.com.twitter.hpack.Encoder(Http2Protocol.InitialMaxHeaderTableSize)     val os = new ByteArrayOutputStream(128) -    become(Idle)--    object Idle extends State {-      val handleEvent: PartialFunction[FrameEvent, Unit] = {-        case ack @ SettingsAckFrame(s) =>-          applySettings(s)-          push(eventsOut, ack)--        case ParsedHeadersFrame(streamId, endStream, kvs, prioInfo) =>-          kvs.foreach {-            case (key, value: String) =>-              encoder.encodeHeader(os, key, value, false)-            case (key, value) =>-              throw new IllegalStateException(s"Didn't expect key-value-pair [$key] -> [$value](${value.getClass}) here.")-          }-          val result = ByteString.fromArrayUnsafe(os.toByteArray) // BAOS.toByteArray always creates a copy-          os.reset()-          if (result.size <= currentMaxFrameSize) push(eventsOut, HeadersFrame(streamId, endStream, endHeaders = true, result, prioInfo))-          else {-            val first = HeadersFrame(streamId, endStream, endHeaders = false, result.take(currentMaxFrameSize), prioInfo)+    def onPull(): Unit = pull(eventsIn)+    def onPush(): Unit = grab(eventsIn) match {+      case ack @ SettingsAckFrame(s) =>+        applySettings(s)+        push(eventsOut, ack)+      case ParsedHeadersFrame(streamId, endStream, kvs, prioInfo) =>+        kvs.foreach {+          case (key, value: String) =>+            encoder.encodeHeader(os, key, value, false)+          case (key, value) =>+            throw new IllegalStateException(s"Didn't expect key-value-pair [$key] -> [$value](${value.getClass}) here.")+        }+        val result = ByteString.fromArrayUnsafe(os.toByteArray) // BAOS.toByteArray always creates a copy+        os.reset()+        if (result.size <= currentMaxFrameSize) push(eventsOut, HeadersFrame(streamId, endStream, endHeaders = true, result, prioInfo))+        else {+          val first = HeadersFrame(streamId, endStream, endHeaders = false, result.take(currentMaxFrameSize), prioInfo) -            emit(eventsOut, first)-            setHandler(eventsOut, new OutHandler {-              var remainingData = result.drop(currentMaxFrameSize)+          push(eventsOut, first)+          setHandler(eventsOut, new OutHandler {+            private var remainingData = result.drop(currentMaxFrameSize) -              def onPull(): Unit = {-                val thisFragment = remainingData.take(currentMaxFrameSize)-                val rest = remainingData.drop(currentMaxFrameSize)-                val last = rest.isEmpty+            def onPull(): Unit = {+              val thisFragment = remainingData.take(currentMaxFrameSize)+              val rest = remainingData.drop(currentMaxFrameSize)+              val last = rest.isEmpty -                push(eventsOut, ContinuationFrame(streamId, endHeaders = last, thisFragment))-                if (last) become(Idle)-                else remainingData = rest-              }-            })-          }-      }--      def applySettings(s: immutable.Seq[Setting]): Unit =-        s foreach {-          case Setting(SettingIdentifier.SETTINGS_HEADER_TABLE_SIZE, size) =>-            log.debug("Applied SETTINGS_HEADER_TABLE_SIZE({}) in header compression", size)-            // 'size' is strictly spoken unsigned, but the encoder is allowed to-            // pick any size equal to or less than this value (6.5.2)-            if (size >= 0) encoder.setMaxHeaderTableSize(os, size)-            else encoder.setMaxHeaderTableSize(os, Int.MaxValue)-          case _ => // ignore, not applicable to this stage+              push(eventsOut, ContinuationFrame(streamId, endHeaders = last, thisFragment))+              if (last) setHandler(eventsOut, logic)

We sent something in the line above, so there should be pull.

jrudolph

comment created time in 5 days