profile
viewpoint
If you are wondering where the data of this site comes from, please visit https://api.github.com/users/creachadair/events. GitMemory does not store any data, but only uses NGINX to cache data for a period of time. The idea behind GitMemory is simply to give users a better reading experience.

creachadair/cityhash 2

A straightforward transliteration of the CityHash non-cryptographic hash algorithm from C++ into Go. Based on https://github.com/google/cityhash

creachadair/bitstream 1

A Go package to support reading and writing of variable-width binary values to or from a stream of bytes.

creachadair/cache 1

Packages that implement string-keyed caching of byte buffers.

creachadair/ansi 0

A library for manipulating ANSI escape codes in Go.

creachadair/atomicfile 0

All-or-nothing file replacement using atomic renames.

creachadair/badgerstore 0

An implementation of the blob.Store interface over BadgerDB.

creachadair/bitcaskstore 0

An implementation of blob.Store on bitcask

creachadair/blitzmail 0

A client implementation of the BlitzMail protocol.

creachadair/boltstore 0

An implementation of the blob.Store interface over BoltDB.

push eventtendermint/tendermint

M. J. Fromberger

commit sha 60cba7402d959472dcbf0f0c1620917cd7de05be

Report errors from a new subscription.

view details

push time in 9 minutes

push eventtendermint/tendermint

M. J. Fromberger

commit sha 277548579d7360629527a270d527d47b9750e0af

Signal where signaling is OK

view details

M. J. Fromberger

commit sha b11e6edfb1e84f9ac492abfa0c0c026733b3fc45

Separate fields.

view details

M. J. Fromberger

commit sha 0908ff7ebecb087441943abe034641d796893464

Un-pointer the mutex field.

view details

push time in 31 minutes

Pull request review commenttendermint/tendermint

pubsub: Use a dynamic queue for buffered subscriptions

 var ( // 2) channel which is closed if a client is too slow or choose to unsubscribe // 3) err indicating the reason for (2) type Subscription struct {-	id  string-	out chan Message+	id    string+	out   chan Message+	queue *queue.Queue  	canceled chan struct{}+	stop     func() 	mtx      tmsync.RWMutex 	err      error }  // NewSubscription returns a new subscription with the given outCapacity. func NewSubscription(outCapacity int) *Subscription {-	return &Subscription{+	sub := &Subscription{ 		id:       uuid.NewString(),-		out:      make(chan Message, outCapacity),+		out:      make(chan Message), 		canceled: make(chan struct{}),++		// N.B. The output channel is always unbuffered. For an unbuffered+		// subscription that was already the case, and for a buffered one the+		// queue now serves as the buffer.+	}++	if outCapacity == 0 {+		sub.stop = func() { close(sub.canceled) }+	} else {+		q, err := queue.New(queue.Options{+			SoftQuota: outCapacity,+			HardLimit: outCapacity,+		})+		if err != nil {+			panic("queue creation failed: " + err.Error())+		}+		sub.queue = q+		sub.stop = func() { q.Close(); close(sub.canceled) }++		// Start a goroutine to bridge messages from the queue to the channel.+		// TODO(creachadair): This is a temporary hack until we can change the+		// interface not to expose the channel directly.+		go func() {+			for {+				next, err := q.Wait(context.Background())+				if err != nil {

It is not necessary to nil-check next, the only way it can be nil is if the caller put a nil value into the queue (which is valid, but we don't do that here).

Yes, the background context is intentional—in this case we don't actually want to time out the wait. But it is also, as you say, temporary: Once we no longer need to bridge values into the output channel, a real context will get plugged in on the client side.

Re: cleanup. There isn't anything to clean up. The output channel is never closed (by design), and we only return from the service routine if the queue is closed, which happens when the subscription is shut down. So I think we are safe. Plumbing in a logger is possible, but I think not worth the effort since I am planning to make bigger changes to the API soon.

creachadair

comment created time in 31 minutes

PullRequestReviewEvent

Pull request review commenttendermint/tendermint

pubsub: Use a dynamic queue for buffered subscriptions

 var ( // 2) channel which is closed if a client is too slow or choose to unsubscribe // 3) err indicating the reason for (2) type Subscription struct {-	id  string-	out chan Message+	id    string+	out   chan Message+	queue *queue.Queue  	canceled chan struct{}+	stop     func() 	mtx      tmsync.RWMutex 	err      error }  // NewSubscription returns a new subscription with the given outCapacity. func NewSubscription(outCapacity int) *Subscription {-	return &Subscription{+	sub := &Subscription{ 		id:       uuid.NewString(),-		out:      make(chan Message, outCapacity),+		out:      make(chan Message), 		canceled: make(chan struct{}),++		// N.B. The output channel is always unbuffered. For an unbuffered+		// subscription that was already the case, and for a buffered one the+		// queue now serves as the buffer.+	}++	if outCapacity == 0 {+		sub.stop = func() { close(sub.canceled) }+	} else {+		q, err := queue.New(queue.Options{+			SoftQuota: outCapacity,+			HardLimit: outCapacity,+		})+		if err != nil {+			panic("queue creation failed: " + err.Error())

Yes, that was the reason. But it turns out nothing else (at least inside Tendermint) is using this function, so I unexported it and added an error to the signature.

creachadair

comment created time in 31 minutes

PullRequestReviewEvent

Pull request review commenttendermint/tendermint

pubsub: Use a dynamic queue for buffered subscriptions

+// Package queue implements a dynamic FIFO queue with a fixed upper bound+// and a flexible quota mechanism to handle bursty load.+package queue++import (+	"context"+	"errors"+	"sync"+)++var (+	// ErrQueueFull is returned by the Add method of a queue when the queue has+	// reached its hard capacity limit.+	ErrQueueFull = errors.New("queue is full")++	// ErrNoCredit is returned by the Add method of a queue when the queue has+	// exceeded its soft quota and there is insufficient burst credit.+	ErrNoCredit = errors.New("insufficient burst credit")++	// ErrQueueClosed is returned by the Add method of a closed queue, and by+	// the Wait method of a closed empty queue.+	ErrQueueClosed = errors.New("queue is closed")++	// Sentinel errors reported by the New constructor.+	errHardLimit   = errors.New("hard limit must be > 0 and ≥ soft quota")+	errBurstCredit = errors.New("burst credit must be non-negative")+)++// A Queue is a limited-capacity FIFO queue of arbitrary data items.+//+// A queue has a soft quota and a hard limit on the number of items that may be+// contained in the queue. Adding items in excess of the hard limit will fail+// unconditionally.+//+// For items in excess of the soft quota, a credit system applies: Each queue+// maintains a burst credit score. Adding an item in excess of the soft quota+// costs 1 unit of burst credit. If there is not enough burst credit, the add+// will fail.+//+// The initial burst credit is assigned when the queue is constructed. Removing+// items from the queue adds additional credit if the resulting queue length is+// less than the current soft quota. Burst credit is capped by the hard limit.+//+// A Queue is safe for concurrent use by multiple goroutines.+type Queue struct {+	mu *sync.Mutex // protects the fields below++	softQuota int     // adjusted dynamically (see Add, Remove)+	hardLimit int     // fixed for the lifespan of the queue+	queueLen  int     // number of entries in the queue list+	credit    float64 // current burst credit++	closed      bool+	nempty      *sync.Cond+	back, front *entry++	// The queue is singly-linked. Front points to the sentinel and back points+	// to the newest entry. The oldest entry is front.link if it exists.+}++// New constructs a new empty queue with the specified options.  It reports an+// error if any of the option values are invalid.+func New(opts Options) (*Queue, error) {+	if opts.HardLimit <= 0 || opts.HardLimit < opts.SoftQuota {+		return nil, errHardLimit+	}+	if opts.BurstCredit < 0 {+		return nil, errBurstCredit+	}+	if opts.SoftQuota <= 0 {+		opts.SoftQuota = opts.HardLimit+	}+	if opts.BurstCredit == 0 {+		opts.BurstCredit = float64(opts.SoftQuota)+	}+	sentinel := new(entry)+	mu := new(sync.Mutex)+	return &Queue{+		mu:        mu,+		softQuota: opts.SoftQuota,+		hardLimit: opts.HardLimit,+		credit:    opts.BurstCredit,+		nempty:    sync.NewCond(mu),+		back:      sentinel,+		front:     sentinel,+	}, nil+}++// Add adds item to the back of the queue. It reports an error and does not+// enqueue the item if the queue is full or closed, or if it exceeds its soft+// quota and there is not enough burst credit.+func (q *Queue) Add(item interface{}) error {+	q.mu.Lock()+	defer q.mu.Unlock()++	if q.closed {+		return ErrQueueClosed+	}++	if q.queueLen >= q.softQuota {+		if q.queueLen == q.hardLimit {+			return ErrQueueFull+		} else if q.credit < 1 {+			return ErrNoCredit+		}++		// Successfully exceeding the soft quota deducts burst credit and raises+		// the soft quota. This has the effect of reducing the credit cap and the+		// amount of credit given for removing items to better approximate the+		// rate at which the consumer is servicing the queue.+		q.credit--+		q.softQuota = q.queueLen + 1+	}+	e := &entry{item: item}+	q.back.link = e+	q.back = e+	q.queueLen+++	if q.queueLen == 1 { // was empty+		q.nempty.Broadcast()+	}+	return nil+}++// Remove removes and returns the frontmost (oldest) item in the queue and+// reports whether an item was available.  If the queue is empty, Remove+// returns nil, false.+func (q *Queue) Remove() (interface{}, bool) {+	q.mu.Lock()+	defer q.mu.Unlock()++	if q.queueLen == 0 {+		return nil, false+	}+	return q.popFront(), true+}++// Wait blocks until q is non-empty, and then returns the frontmost (oldest)+// item from the queue. If ctx ends before an item is available, Wait returns a+// nil value and a context error. If the queue is closed while it is still+// empty, Wait returns nil, ErrQueueClosed.+func (q *Queue) Wait(ctx context.Context) (interface{}, error) {+	// If the context terminates, wake the waiter.+	ctx, cancel := context.WithCancel(ctx)+	defer cancel()+	go func() { <-ctx.Done(); q.nempty.Broadcast() }()

We're safe—when the condition wakes, we will always check again whether the queue is empty. If it is, we'll see the context is done and report an error.

creachadair

comment created time in 31 minutes

PullRequestReviewEvent

Pull request review commenttendermint/tendermint

pubsub: Use a dynamic queue for buffered subscriptions

+// Package queue implements a dynamic FIFO queue with a fixed upper bound+// and a flexible quota mechanism to handle bursty load.+package queue++import (+	"context"+	"errors"+	"sync"+)++var (+	// ErrQueueFull is returned by the Add method of a queue when the queue has+	// reached its hard capacity limit.+	ErrQueueFull = errors.New("queue is full")++	// ErrNoCredit is returned by the Add method of a queue when the queue has+	// exceeded its soft quota and there is insufficient burst credit.+	ErrNoCredit = errors.New("insufficient burst credit")++	// ErrQueueClosed is returned by the Add method of a closed queue, and by+	// the Wait method of a closed empty queue.+	ErrQueueClosed = errors.New("queue is closed")++	// Sentinel errors reported by the New constructor.+	errHardLimit   = errors.New("hard limit must be > 0 and ≥ soft quota")+	errBurstCredit = errors.New("burst credit must be non-negative")+)++// A Queue is a limited-capacity FIFO queue of arbitrary data items.+//+// A queue has a soft quota and a hard limit on the number of items that may be+// contained in the queue. Adding items in excess of the hard limit will fail+// unconditionally.+//+// For items in excess of the soft quota, a credit system applies: Each queue+// maintains a burst credit score. Adding an item in excess of the soft quota+// costs 1 unit of burst credit. If there is not enough burst credit, the add+// will fail.+//+// The initial burst credit is assigned when the queue is constructed. Removing+// items from the queue adds additional credit if the resulting queue length is+// less than the current soft quota. Burst credit is capped by the hard limit.+//+// A Queue is safe for concurrent use by multiple goroutines.+type Queue struct {+	mu *sync.Mutex // protects the fields below

I re-worked the code to populate the condition variable separately, and removed the pointer from the field. I'm still happy to add a comment also, if you still think it's necessary.

creachadair

comment created time in 31 minutes

PullRequestReviewEvent

Pull request review commenttendermint/tendermint

pubsub: Use a dynamic queue for buffered subscriptions

+// Package queue implements a dynamic FIFO queue with a fixed upper bound+// and a flexible quota mechanism to handle bursty load.+package queue++import (+	"context"+	"errors"+	"sync"+)++var (+	// ErrQueueFull is returned by the Add method of a queue when the queue has+	// reached its hard capacity limit.+	ErrQueueFull = errors.New("queue is full")++	// ErrNoCredit is returned by the Add method of a queue when the queue has+	// exceeded its soft quota and there is insufficient burst credit.+	ErrNoCredit = errors.New("insufficient burst credit")++	// ErrQueueClosed is returned by the Add method of a closed queue, and by+	// the Wait method of a closed empty queue.+	ErrQueueClosed = errors.New("queue is closed")++	// Sentinel errors reported by the New constructor.+	errHardLimit   = errors.New("hard limit must be > 0 and ≥ soft quota")+	errBurstCredit = errors.New("burst credit must be non-negative")+)++// A Queue is a limited-capacity FIFO queue of arbitrary data items.+//+// A queue has a soft quota and a hard limit on the number of items that may be+// contained in the queue. Adding items in excess of the hard limit will fail+// unconditionally.+//+// For items in excess of the soft quota, a credit system applies: Each queue+// maintains a burst credit score. Adding an item in excess of the soft quota+// costs 1 unit of burst credit. If there is not enough burst credit, the add+// will fail.+//+// The initial burst credit is assigned when the queue is constructed. Removing+// items from the queue adds additional credit if the resulting queue length is+// less than the current soft quota. Burst credit is capped by the hard limit.+//+// A Queue is safe for concurrent use by multiple goroutines.+type Queue struct {+	mu *sync.Mutex // protects the fields below++	softQuota int     // adjusted dynamically (see Add, Remove)+	hardLimit int     // fixed for the lifespan of the queue+	queueLen  int     // number of entries in the queue list+	credit    float64 // current burst credit++	closed      bool+	nempty      *sync.Cond+	back, front *entry++	// The queue is singly-linked. Front points to the sentinel and back points+	// to the newest entry. The oldest entry is front.link if it exists.+}++// New constructs a new empty queue with the specified options.  It reports an+// error if any of the option values are invalid.+func New(opts Options) (*Queue, error) {+	if opts.HardLimit <= 0 || opts.HardLimit < opts.SoftQuota {+		return nil, errHardLimit+	}+	if opts.BurstCredit < 0 {+		return nil, errBurstCredit+	}+	if opts.SoftQuota <= 0 {+		opts.SoftQuota = opts.HardLimit+	}+	if opts.BurstCredit == 0 {+		opts.BurstCredit = float64(opts.SoftQuota)+	}+	sentinel := new(entry)+	mu := new(sync.Mutex)+	return &Queue{+		mu:        mu,+		softQuota: opts.SoftQuota,+		hardLimit: opts.HardLimit,+		credit:    opts.BurstCredit,+		nempty:    sync.NewCond(mu),+		back:      sentinel,+		front:     sentinel,+	}, nil+}++// Add adds item to the back of the queue. It reports an error and does not+// enqueue the item if the queue is full or closed, or if it exceeds its soft+// quota and there is not enough burst credit.+func (q *Queue) Add(item interface{}) error {+	q.mu.Lock()+	defer q.mu.Unlock()++	if q.closed {+		return ErrQueueClosed+	}++	if q.queueLen >= q.softQuota {+		if q.queueLen == q.hardLimit {+			return ErrQueueFull+		} else if q.credit < 1 {+			return ErrNoCredit+		}++		// Successfully exceeding the soft quota deducts burst credit and raises+		// the soft quota. This has the effect of reducing the credit cap and the+		// amount of credit given for removing items to better approximate the+		// rate at which the consumer is servicing the queue.+		q.credit--+		q.softQuota = q.queueLen + 1+	}+	e := &entry{item: item}+	q.back.link = e+	q.back = e+	q.queueLen+++	if q.queueLen == 1 { // was empty+		q.nempty.Broadcast()

This one can be Signal (and now is). The others need to be broadcast, because in those cases we really do want all threads waiting on this condition to wake up and-recheck.

creachadair

comment created time in 31 minutes

PullRequestReviewEvent

push eventtendermint/tendermint

M. J. Fromberger

commit sha 1fd706054227950190710e9b38f6f978a7c42274

pubsub: Use distinct client IDs for test subscriptions. (#7178) Fixes #7176. Some of the benchmarks create a bunch of different subscriptions all sharing the same query. These were all using the same client ID, which violates one of the subscriber rules. Ensure each subscriber gets a unique ID. This has been broken as long as this library has been in the repo—I tracked it back to bb9aa85d and it was already failing there, so I think this never really worked. I'm not sure these test anything useful, but at least now they run.

view details

Sam Kleinman

commit sha 6108d0a9b53888f179760262def787267cf7e524

config: expose ability to write config to arbitrary paths (#7174)

view details

M. J. Fromberger

commit sha d981f7de2dad9ea21b4c930587e9ba26b41b8070

Add v0.35 to the configs for building the docs website.

view details

push time in an hour

push eventtendermint/tendermint

M. J. Fromberger

commit sha 1fd706054227950190710e9b38f6f978a7c42274

pubsub: Use distinct client IDs for test subscriptions. (#7178) Fixes #7176. Some of the benchmarks create a bunch of different subscriptions all sharing the same query. These were all using the same client ID, which violates one of the subscriber rules. Ensure each subscriber gets a unique ID. This has been broken as long as this library has been in the repo—I tracked it back to bb9aa85d and it was already failing there, so I think this never really worked. I'm not sure these test anything useful, but at least now they run.

view details

Sam Kleinman

commit sha 6108d0a9b53888f179760262def787267cf7e524

config: expose ability to write config to arbitrary paths (#7174)

view details

M. J. Fromberger

commit sha e78ce2c480f994ae735170653778fe3cfe19b9c8

Simulate queueing parameters with Markov conditionals.

view details

M. J. Fromberger

commit sha 22470008ec36fec7659d4ed0cc6d4cf725c3819f

Implement a dynamic queue. A bounded-length explicit queue with a soft quota and a credit system for burst capacity. The queue is safe for concurrent use and supports writer shutdown.

view details

M. J. Fromberger

commit sha 8a2d456b3fb1fe8fed982579d2a547c2ed4a09e2

Add unit tests for the queue package.

view details

M. J. Fromberger

commit sha 5558505a709d6d6c4040c0d397ca5fa9b4fe6aa5

Plug in the queue for buffered subscriptions. Unbuffered subscriptions still use the channel, and the channel is still part of the interface to avoid breaking existing use.

view details

M. J. Fromberger

commit sha 446792081afc4e4bd485db74eb921dca7d932ef7

Drop the simulator code.

view details

M. J. Fromberger

commit sha 350fc655784184c7b1be1c9d9ea3dabf027311de

Satiate the linter.

view details

push time in an hour

push eventtendermint/tendermint

M. J. Fromberger

commit sha 1fd706054227950190710e9b38f6f978a7c42274

pubsub: Use distinct client IDs for test subscriptions. (#7178) Fixes #7176. Some of the benchmarks create a bunch of different subscriptions all sharing the same query. These were all using the same client ID, which violates one of the subscriber rules. Ensure each subscriber gets a unique ID. This has been broken as long as this library has been in the repo—I tracked it back to bb9aa85d and it was already failing there, so I think this never really worked. I'm not sure these test anything useful, but at least now they run.

view details

Sam Kleinman

commit sha 6108d0a9b53888f179760262def787267cf7e524

config: expose ability to write config to arbitrary paths (#7174)

view details

M. J. Fromberger

commit sha 5b80c210cc3112afc96d7d113fd9792efaae1b3b

WIP: add callgraph package

view details

M. J. Fromberger

commit sha ad37e98d07604e841296ccdb3084df0894c7ceea

Add package bicall and basic tests. Also: Update go-cmp.

view details

M. J. Fromberger

commit sha 7ec2551db7be1e92d2b025d7c34138e79253034e

Add findbuiltin tool

view details

M. J. Fromberger

commit sha 5849e95866a54f68e8b55ec4b65cbab3afc110f0

go mod tidy

view details

push time in an hour

push eventtendermint/tendermint

M. J. Fromberger

commit sha 1fd706054227950190710e9b38f6f978a7c42274

pubsub: Use distinct client IDs for test subscriptions. (#7178) Fixes #7176. Some of the benchmarks create a bunch of different subscriptions all sharing the same query. These were all using the same client ID, which violates one of the subscriber rules. Ensure each subscriber gets a unique ID. This has been broken as long as this library has been in the repo—I tracked it back to bb9aa85d and it was already failing there, so I think this never really worked. I'm not sure these test anything useful, but at least now they run.

view details

Sam Kleinman

commit sha 6108d0a9b53888f179760262def787267cf7e524

config: expose ability to write config to arbitrary paths (#7174)

view details

M. J. Fromberger

commit sha ca0bf9a53de347b5cc99934d82831d871764bb53

Add x/architecture.md. Combine and expand some of my previous notes, and add a block diagram.

view details

M. J. Fromberger

commit sha f0bef53a58581f300f91078709f08c0a48d23167

Add x/match package. Helper functions for matching strings on patterns. Useful for constructing label filters.

view details

M. J. Fromberger

commit sha d4648cb55d7ccbc757830b8171faa39a3968a84c

Add package x/feed. This package provides a simple fixed-capacity buffered item feed. Writes to the feed are non-blocking up to the capacity. This is an essay toward a reworking of the Tendermint event system into two layers: This package is meant to handle the "inner" layer, which gathers and sequences the raw event records from various parts of the system. The consumer of the raw event stream will index, filter, buffer, and dispatch events to other consumers, e.g., RPC clients and pubsub subscribers.

view details

M. J. Fromberger

commit sha 0f79e08ee3679f6f0604e0fbb45c01fc73184746

Add package x/hub. This package defines a dispatcher that consumes items from a feed, optionally indexes them, and dispatches them to dynamically registered listeners. Each listener maintains a fixed-length queue of undelievered items, beyond which new items are discarded. The key types defined here are: Hub: streams items from a feed to an indexer and listeners. Listener: a buffered receiver of indexed items. Listeners can be registered and unregistered with a hub at any time. Closing a listener detaches it from the hub, but retains any queued items. WHen the hub exits, it closes all its registered listeners.

view details

M. J. Fromberger

commit sha ad6954dbd7c9a9ba37e24300cbe4f7de8e1b9e73

x/hub: Remove configurable eviction policy. This feature was too complicated for its value. Even for the common case of evicting the oldest, it copies the whole queue for each eviction, which is common case for a slow reader, exactly the case we're trying to reduce the cost of. Making a special case of that would make it even more complicated. Also, calling back into user code allows it to block the hub. We could just document that isn't allowed, but Hyrum is right over there. I wouldn't drop this feature just for that, but this was another nail.

view details

M. J. Fromberger

commit sha 88682be69c1a7d44f1527cb78b2d974d5072cb58

Add package x/index. This package defines a basic interface for reading and writing item data into a persistent index. The Store interface defines the capabilities needed from an underlying storage engine, which should be implementable without difficulty on top of a key/value store or a tabular DBMS.

view details

M. J. Fromberger

commit sha b9fceffc5c602a6069fdd482107d84d1a0c186a5

Add package x/index/memstore. This is a trivial in-memory implementation of the index.Store interface, sufficient to use for testing.

view details

M. J. Fromberger

commit sha 40f57e5f6a2d9674822bc97fae3604be4b3557d3

Add x/demo package. This is a working demonstration harness for the feed/hub prototype. Running the program starts a JSON-RPC service (over HTTP) that allows the caller to post events, query the index, and subscribe and consume items from the hub via listeners. Usage: demo -listen [host]:port The service endpoint is http://[host]:port/rpc. Content-Type: application/json Content: JSON-RPC requests or batches. To simulate an active node, the server runs a "heartbeat" that periodically injects new items into the feed.

view details

push time in an hour

push eventtendermint/tendermint

M. J. Fromberger

commit sha 1fd706054227950190710e9b38f6f978a7c42274

pubsub: Use distinct client IDs for test subscriptions. (#7178) Fixes #7176. Some of the benchmarks create a bunch of different subscriptions all sharing the same query. These were all using the same client ID, which violates one of the subscriber rules. Ensure each subscriber gets a unique ID. This has been broken as long as this library has been in the repo—I tracked it back to bb9aa85d and it was already failing there, so I think this never really worked. I'm not sure these test anything useful, but at least now they run.

view details

Sam Kleinman

commit sha 6108d0a9b53888f179760262def787267cf7e524

config: expose ability to write config to arbitrary paths (#7174)

view details

M. J. Fromberger

commit sha 9f2cb62f5105d27753296440df216babbe923102

Make ABCI semantic versions match the TM version. Since ABCI and Tendermint are built from the same repository, it is no longer necessary to version them independently. Move the ABCI version constants to be variables and copy them from the linker-assigned TM version. Note that the package initialization rules ensure these assignments are evaluated coherently, https://golang.org/ref/spec#Package_initialization.

view details

M. J. Fromberger

commit sha ed4ca7cd56b322793b3e7fe0e80c90c1d34df37f

Update usage.

view details

M. J. Fromberger

commit sha 2a6a45ee0a108549fa888fba7bae91e89980a003

Update default version.

view details

M. J. Fromberger

commit sha df91536c517297e78af802115dd56e11b16792b4

Add a changelog entry.

view details

M. J. Fromberger

commit sha bfa598fb5c883ca54182a1f4008d3ef48abaee1d

Revised proposal: Match ABCI to Tendermint on major/minor.

view details

M. J. Fromberger

commit sha 52cb733d3f50b0f744d88596fe5c0c8da8f10efa

Drop the panic. Some of our test infrastructure apparently does some weird build shenanigans that are probably not worth debugging.

view details

push time in an hour

pull request commenttendermint/tendermint

pubsub: Use distinct client IDs for test subscriptions.

Good catch. Just so I understand this correctly, if Subscribe is called with the same clientID it just returns the same existing subscription and thus to actually make multiple different subscriptions the clientID needs to be unique. Were there any other tests that weren't configured correctly?

I believe the existing tests catch this case correctly already. These benchmarks weren't being run by default in CI, and as far as I can tell they would have failed at basically any time in the history of this code.

creachadair

comment created time in an hour

PR opened tendermint/tendermint

pubsub: Use distinct client IDs for test subscriptions. S:automerge

Fixes #7176. Some of the benchmarks create a bunch of different subscriptions all sharing the same query. These were all using the same client ID, which violates one of the subscriber rules. Ensure each subscriber gets a unique ID.

This has been broken as long as this library has been in the repo—I tracked it back to bb9aa85d and it was already failing there, so I think this never really worked. I'm not sure these test anything useful, but at least now they run.

+2 -1

0 comment

1 changed file

pr created time in 9 hours

create barnchtendermint/tendermint

branch : mjf/bad-benchmarks

created branch time in 9 hours

issue commenttendermint/tendermint

libs/pubsub: Benchmark*ClientsOneQuery fails with "already subscribed"

Thanks—I noticed those benchmarks have been broken for a while now, at least as far back as v0.33. I meant to go back and try to bisect for a working version, but it would probably make more sense to just delete the bad benchmarks.

odeke-em

comment created time in 9 hours

push eventtendermint/tendermint

M. J. Fromberger

commit sha 6e30da30e2cd45e83cd6b5e2078c829e0c67d4e7

Satiate the linter.

view details

push time in 9 hours

push eventtendermint/tendermint

M. J. Fromberger

commit sha 7e2fd6f39adfa1144bd3a7aed396629eb7d4e4dd

Drop the simulator code.

view details

push time in 9 hours

PR opened tendermint/tendermint

pubsub: Use a dynamic queue for buffered subscriptions

This change is another increment of the work described in #7156, and a follow-up to #7070. As before, this change preserves the existing API of the pubsub package.

Note to reviewers: This change does not apply to the "privileged" unbuffered subscriber. I plan to address that in a subsequent change.

Synopsis

Replace the buffered channel used to deliver events to buffered subscribers with an explicit queue. The queue provides a soft quota and burst credit mechanism: Clients that usually keep up can survive occasional bursts, without allowing truly slow clients to hog resources indefinitely.

Discussion

Event subscriptions in Tendermint currently use a fixed-length Go channel as a queue. When the channel fills up, the publisher immediately terminates the subscription. This prevents slow subscribers from creating memory pressure on the node by not servicing their queue fast enough.

The problem is that event publishing is bursty, and a large collection of events occasionally arrives all at once (e.g., when a block is committed). Enqueuing on the Go channel is in-process, and fast relative to a websocket subscriber: Even a client actively servicing its queue may not be able to keep up and avoid disconnection. This causes UX issues (see, for example, #6729).

The explicit queue (internal/libs/queue) also includes a hard cap on the queue size, but the soft quota mechanism means the hard cap can safely be set much higher. The queue uses a "credit" scheme to model the consumer. Consumers that do not keep up with the steady state will use up their credit and be disconnected, while consumers that "keep up" are given more headroom.

+560 -14

0 comment

6 changed files

pr created time in 10 hours

push eventtendermint/tendermint

M. J. Fromberger

commit sha 324371d8519b330b754730d7b328704386c48269

Add unit tests for the queue package.

view details

M. J. Fromberger

commit sha 125adc3f2c084bbe836e3964ffa38577a4c20d01

Plug in the queue for buffered subscriptions. Unbuffered subscriptions still use the channel, and the channel is still part of the interface to avoid breaking existing use.

view details

push time in 10 hours

push eventtendermint/tendermint

William Banfield

commit sha 4d09a6c25d9aea3a0209c648467493ba5b4ac63d

abci: fix readme link (#7173)

view details

M. J. Fromberger

commit sha adc21468bcf001e9b3f8e084c485a77ce15e6dcc

Add v0.35 to the configs for building the docs website.

view details

push time in 10 hours

push eventtendermint/tendermint

William Banfield

commit sha 4d09a6c25d9aea3a0209c648467493ba5b4ac63d

abci: fix readme link (#7173)

view details

M. J. Fromberger

commit sha 40c46e94ddfa5a55231a25f5b2280d56b3b37343

Simulate queueing parameters with Markov conditionals.

view details

M. J. Fromberger

commit sha e6af6c3e540db1b37dc656ecab8e1477b61e3c57

Implement a dynamic queue. A bounded-length explicit queue with a soft quota and a credit system for burst capacity. The queue is safe for concurrent use and supports writer shutdown.

view details

M. J. Fromberger

commit sha d3ece6cf4293286b34606fe7e31867031c5effbf

Plug in the queue for buffered subscriptions. Unbuffered subscriptions still use the channel, and the channel is still part of the interface to avoid breaking existing use.

view details

M. J. Fromberger

commit sha 1f11877ee4594a0d13aefdb10cdd45e815c76f83

Add unit tests for the queue package.

view details

push time in 10 hours

push eventtendermint/tendermint

William Banfield

commit sha 4d09a6c25d9aea3a0209c648467493ba5b4ac63d

abci: fix readme link (#7173)

view details

M. J. Fromberger

commit sha 491306bba2e875b7be0ce1b46413f356eddcd967

WIP: add callgraph package

view details

M. J. Fromberger

commit sha efbebafad91ac43a441a09d7bd5f4924329a562c

Add package bicall and basic tests. Also: Update go-cmp.

view details

M. J. Fromberger

commit sha 6ae7bac1592300e09b6f5dce4aae3b75e8a0f41a

Add findbuiltin tool

view details

M. J. Fromberger

commit sha 11309ce5c6604ba9b40d1e3f53c43abdbae82db3

go mod tidy

view details

push time in 10 hours

push eventtendermint/tendermint

William Banfield

commit sha 4d09a6c25d9aea3a0209c648467493ba5b4ac63d

abci: fix readme link (#7173)

view details

M. J. Fromberger

commit sha 1980f703b7db433ecd8d8e212b18eae7cdb99036

Add x/architecture.md. Combine and expand some of my previous notes, and add a block diagram.

view details

M. J. Fromberger

commit sha 92816ce4487ef22ea5e0035ba00e4c32a85a6415

Add x/match package. Helper functions for matching strings on patterns. Useful for constructing label filters.

view details

M. J. Fromberger

commit sha e454af22a31353e269cd4334f8feefe419a348cb

Add package x/feed. This package provides a simple fixed-capacity buffered item feed. Writes to the feed are non-blocking up to the capacity. This is an essay toward a reworking of the Tendermint event system into two layers: This package is meant to handle the "inner" layer, which gathers and sequences the raw event records from various parts of the system. The consumer of the raw event stream will index, filter, buffer, and dispatch events to other consumers, e.g., RPC clients and pubsub subscribers.

view details

M. J. Fromberger

commit sha 697918d72d295a2e96aaa0f5c52dbae6c1106c3e

Add package x/hub. This package defines a dispatcher that consumes items from a feed, optionally indexes them, and dispatches them to dynamically registered listeners. Each listener maintains a fixed-length queue of undelievered items, beyond which new items are discarded. The key types defined here are: Hub: streams items from a feed to an indexer and listeners. Listener: a buffered receiver of indexed items. Listeners can be registered and unregistered with a hub at any time. Closing a listener detaches it from the hub, but retains any queued items. WHen the hub exits, it closes all its registered listeners.

view details

M. J. Fromberger

commit sha 3314b5e07feb8bb2e70e321ea37e923dfadc7de1

x/hub: Remove configurable eviction policy. This feature was too complicated for its value. Even for the common case of evicting the oldest, it copies the whole queue for each eviction, which is common case for a slow reader, exactly the case we're trying to reduce the cost of. Making a special case of that would make it even more complicated. Also, calling back into user code allows it to block the hub. We could just document that isn't allowed, but Hyrum is right over there. I wouldn't drop this feature just for that, but this was another nail.

view details

M. J. Fromberger

commit sha 07d885b15112a5736e257fc5215dd001ac8a415f

Add package x/index. This package defines a basic interface for reading and writing item data into a persistent index. The Store interface defines the capabilities needed from an underlying storage engine, which should be implementable without difficulty on top of a key/value store or a tabular DBMS.

view details

M. J. Fromberger

commit sha 51586d03ff7a022219cf6c1b03393f2265d93de9

Add package x/index/memstore. This is a trivial in-memory implementation of the index.Store interface, sufficient to use for testing.

view details

M. J. Fromberger

commit sha 35e1bbf8b858577be3f7cbe52c46f1557090c7ba

Add x/demo package. This is a working demonstration harness for the feed/hub prototype. Running the program starts a JSON-RPC service (over HTTP) that allows the caller to post events, query the index, and subscribe and consume items from the hub via listeners. Usage: demo -listen [host]:port The service endpoint is http://[host]:port/rpc. Content-Type: application/json Content: JSON-RPC requests or batches. To simulate an active node, the server runs a "heartbeat" that periodically injects new items into the feed.

view details

push time in 10 hours

push eventtendermint/tendermint

William Banfield

commit sha 4d09a6c25d9aea3a0209c648467493ba5b4ac63d

abci: fix readme link (#7173)

view details

M. J. Fromberger

commit sha 41bc6647e7109b6652a667816819d19cf70e2a49

Make ABCI semantic versions match the TM version. Since ABCI and Tendermint are built from the same repository, it is no longer necessary to version them independently. Move the ABCI version constants to be variables and copy them from the linker-assigned TM version. Note that the package initialization rules ensure these assignments are evaluated coherently, https://golang.org/ref/spec#Package_initialization.

view details

M. J. Fromberger

commit sha 3f6d7733b368e56e719560ea0402e206aa952ff1

Update usage.

view details

M. J. Fromberger

commit sha 8cbbd56d531ff42968d359a2afdbd930e51e54f3

Update default version.

view details

M. J. Fromberger

commit sha 2bf45d79da72c7a198de68092021c036f54b3f3a

Add a changelog entry.

view details

M. J. Fromberger

commit sha 5b772e77fd455b37bb8e2caaffef314f0d5bad96

Revised proposal: Match ABCI to Tendermint on major/minor.

view details

M. J. Fromberger

commit sha 197793e13fe98ccbc59b0c8736ac530865ccc676

Drop the panic. Some of our test infrastructure apparently does some weird build shenanigans that are probably not worth debugging.

view details

push time in 10 hours