profile
viewpoint

graydon/awesome-c 2

A curated list of awesome C frameworks, libraries and software.

graydon/awesome-scalability 1

The Patterns of Scalable, Reliable, and Performant Large-Scale Systems

davidungar/swift 0

The Swift Programming Language

graydon/1ml 0

1ML prototype interpreter

graydon/8086tiny 0

Official repository for 8086tiny: a tiny PC emulator/virtual machine

graydon/adapton.rust 0

Nominal Adapton in Rust

graydon/aliceml 0

A functional programming language based on Standard ML, extended with support for concurrent, distributed, and constraint programming

graydon/alternative-internet 0

A collection of interesting new networks and tech aiming at decentralisation (in some form).

graydon/antidote 0

SyncFree Reference Platform

graydon/ApprovalTests.cpp 0

Native ApprovalTests for C++ on Linux, Mac and Windows

pull request commentstellar/stellar-core

replace all remaining LRUcache by RandomEvictionCache

r+ 84435aa

MonsieurNicolas

comment created time in 4 hours

fork graydon/modules-papers

A collection of papers on modules.

fork in 13 hours

startedak3n/modules-papers

started time in 13 hours

pull request commentstellar/stellar-core

fix crash on shutdown (as to get the proper error message)

r+ 7c74acf

MonsieurNicolas

comment created time in 13 hours

startedmitsuba-renderer/enoki

started time in 4 days

fork graydon/parallel-hashmap

A family of header-only, very fast and memory-friendly hashmap and btree containers.

https://greg7mdp.github.io/parallel-hashmap/

fork in 4 days

startedgreg7mdp/parallel-hashmap

started time in 4 days

fork graydon/ssabook

Mirror of InriaForge SSABook repository: https://gforge.inria.fr/projects/ssabook/ (scheduled for retirement at the end of 2020)

https://pfalcon.github.io/ssabook/latest/

fork in 5 days

startedpfalcon/ssabook

started time in 5 days

push eventgraydon/edelcrantz

Graydon Hoare

commit sha 6579df068e80c0ead43f7684617cadbadd0c0bd1

Mutex-guard the response FuturesUnordered so futures will be Send.

view details

push time in 5 days

startedjfaure/Nimzo

started time in 5 days

startedgamozolabs/fzero_fuzzer

started time in 5 days

startednvaccess/nvda

started time in 6 days

fork graydon/TablaM

A general purpose relational language

fork in 7 days

startedTablam/TablaM

started time in 7 days

fork graydon/awesome-compilers

:sunglasses: Curated list of awesome resources on Compilers, Interpreters and Runtimes

http://aalhour.com/awesome-compilers

fork in 7 days

startedaalhour/awesome-compilers

started time in 7 days

startedtimescale/timescaledb

started time in 7 days

pull request commentstellar/stellar-core

Manual close end point now allows to close manually once

Fine questions! I await @MonsieurNicolas input as I do not have a clear sense of the answers myself, clearly :)

marta-lokhova

comment created time in 7 days

startedFastFilter/fastfilter_cpp

started time in 7 days

fork graydon/fastfilter_cpp

Fast Approximate Membership Filters (C++)

fork in 7 days

pull request commentstellar/stellar-core

Manual close end point now allows to close manually once

Looks fairly good but I am a little concerned with a few points:

  • The names "once" and "all" are a little uninformative / non-obvious to users, and passing a boolean flag named once that we check the negation of seems a little hard to follow; especially when combined with the fact that "all" needs the flag to be set, and once needs FORCE_SCP to not be set. Overall it just feels confusing. This is subjective and you're welcome to tell me "this is the best that can be done!" but it's sorta a gut sense I get reading it, that I'll have a hard time editing the code in the future / telling what it's supposed to do in which combinations.
  • More worryingly: "once" can be called more than once, and I'm not sure it's actually safe to call Herder::bootstrap more than once. Maybe? Might also cause havoc.
  • I'm not 100% clear on why you need to restoreScpState in the non-FORCE_SCP path of bootstrap. What's different about it?
  • Overall I suspect the close-once flag might serve a subtly-different purpose than the MANUAL_CLOSE flag. If I understand correctly, MANUAL_CLOSE was added as a kind of escape hatch to let running networks that have "lost leadership" get forced back into operation by an operator. This is, I believe, why it is also not hidden behind BUILD_TESTS: so that production nodes can be nudged back into SCP when they have given up on it or are all waiting for one another. This might no longer be necessary at all (given the changes you landed in https://github.com/stellar/stellar-core/pull/2612) but if it is true (or if we want to keep it around for purposes of backward compatibility with scripts, or mere paranoia) then I think that's different than what you're adding here. I think what you're adding here is a one-time, idempotent trigger that negates --wait-for-consensus mode and says ok, stop waiting now. If that's true I might suggest:
    • Move --wait-for-consensus behind BUILD_TESTS if we think nobody needs to do this in production (I think .. not?)
    • Rename Config.FORCE_SCP to Config.WAIT_FOR_CONSENSUS to reduce the number of double negatives. Limit the residue of the FORCE_SCP name to the deprecation warnings and back-compat config option parser.
    • Add a new HTTP method called, say, bootstrap that is behind BUILD_TESTS that calls bootstrap iff WAIT_FOR_CONSENSUS is true and bootstrap hasn't happened yet. If bootstrap has happened, return an error indicating that.

How does that sound to you?

marta-lokhova

comment created time in 8 days

fork graydon/terminusdb-store

a tokio-enabled data store for triple data

fork in 8 days

startedterminusdb/terminusdb-store

started time in 8 days

fork graydon/terminusdb-server

open source model driven graph database for knowledge graph representation.

fork in 8 days

startedterminusdb/terminusdb-server

started time in 8 days

startedcomunica/comunica

started time in 8 days

pull request commentstellar/stellar-core

change local history location in configs

r+ 47841c5

sui77

comment created time in 11 days

pull request commentstellar/stellar-core

change local history location in configs

Looks good to me, thanks!

sui77

comment created time in 11 days

pull request commentstellar/stellar-core

Do not wait for in-progress merges on shutdown

r+ 7148ea6

marta-lokhova

comment created time in 11 days

Pull request review commentstellar/stellar-core

Do not wait for in-progress merges on shutdown

 Bucket::merge(BucketManager& bucketManager, uint32_t maxProtocolVersion,                              mc, ctx, doFsync);      BucketEntryIdCmp cmp;+    size_t iter = 0;+     while (oi || ni)     {+        // Check if the merge should be stopped every few entries+        if (++iter >= 1000)+        {+            iter = 0;+            if (bucketManager.isShutdown())+            {+                // Stop merging, as BucketManager is now shutdown+                // This is safe as temp file has not been adopted yet,+                // so it will be removed with the tmp dir+                return nullptr;

Ah, I somehow missed that checkState checks validity. Well and good!

marta-lokhova

comment created time in 11 days

push eventgraydon/stellar-core

Graydon Hoare

commit sha 6dab5dbad1bb62b532400656a0cb38b2dcf8aea8

Set fail and/or badbit exception masks on fstreams.

view details

Latobarita

commit sha 558dfa83cc29a923880d5077186f655e9959a6fb

Merge pull request #2637 from graydon/set-fstream-exception-masks Set fail and/or badbit exception masks on fstreams. Reviewed-by: MonsieurNicolas

view details

Graydon Hoare

commit sha a16dd6c373b94fa87c239bc40a402b292a418f86

git-ignore .idea (CLion) and .clangd

view details

Graydon Hoare

commit sha 52c08ce60979ed8d3935a076da2de18580f327b9

Add a new MODE_DOES_CATCHUP flag to Config (non-user-configurable)

view details

Graydon Hoare

commit sha 233b0203b2d9ba73f450006c701a557d283bf68f

Cancel timers in HerderImpl::shutdown to avoid spurious desyncs.

view details

Graydon Hoare

commit sha 9036b409646e05bd2c10b64a3712e2b474d12781

Rename getBufferedLedger to getFirstBufferedLedger, add getLastBufferedLedger.

view details

Graydon Hoare

commit sha 93bcd9b6d0ef24fd169dbf48df1857980e7aa7ef

Adjust hash-link propagation code in verifyHistoryOfSingleCheckpoint. - Rename mTrustedEndLedger to mTrustedMaxLedger, and make parameter name match. Generally try to use min/max terminology rather than begin/start/end/last since the order of replay is quite mixed: min-to-max within each checkpoint, max-to-min between checkpoints. - Rename mVerifiedLedgerRangeStart to mMaxVerifiedLedgerOfMinCheckpoint. - Rename misleading 'nextCheckpointFirstLedger' local to 'incoming'. It's not the next checkpoint's first ledger, it's the hash-link _from_ the next checkpoint's first (min) ledger, denoting _this_ checkpoint's last (max) ledger. - Move load-incoming / save-outgoing steps of loop to be adjacent. - Switch mTrustedMaxLedger to a std::future and add outgoing promise & future mVerifiedMinLedgerPrev to propagate trust from one VerifyLedgerChainWork to the next in a BatchWork. - Add optional output stream to write checkpoint hashes while propagating.

view details

Graydon Hoare

commit sha ebeb2f42ad0873f9d5e63faadc471922fa45d9a3

Add historywork/WriteVerifiedCheckpointHashesWork.cpp

view details

Graydon Hoare

commit sha 29fd9c96ce6d222913fdedf92922d974cea2238f

Add verify-checkpoints subcommand.

view details

Graydon Hoare

commit sha 2f79970d1e78d404a8603331e0b9674021bc4eab

Support loading trusted checkpoint hashes from file in "catchup" command.

view details

Graydon Hoare

commit sha 1581822c51d01297fb8dbf5605f5459f05c7bd8e

Document "verify-checkpoints" command and "--trusted-checkpoint-hashes" option to "catchup"

view details

Graydon Hoare

commit sha 46ba216835670fde59a58919609ea47fe3df0e1e

Add end-to-end test of checkpoint hash verification.

view details

push time in 11 days

PR opened stellar/stellar-core

Set fail and/or badbit exception masks on fstreams.

This just sets badbit on ifstream exception masks, and failbit|badbit on ofstreams.

Failbit is not set on ifstreams because read operations typically signal completion by reading at-the-end and failing, setting both failbit and eof. Yet more magic and beauty of the iostreams library.

+16 -1

0 comment

14 changed files

pr created time in 12 days

push eventgraydon/stellar-core

Graydon Hoare

commit sha 6dab5dbad1bb62b532400656a0cb38b2dcf8aea8

Set fail and/or badbit exception masks on fstreams.

view details

push time in 12 days

create barnchgraydon/stellar-core

branch : set-fstream-exception-masks

created branch time in 12 days

push eventgraydon/stellar-core

Graydon Hoare

commit sha 952c17ed4b37cb84c87c77ed6e234dbde992076c

Set output file exception mask to failbit | badbit.

view details

push time in 12 days

Pull request review commentstellar/stellar-core

Verify ledger chain command

+// Copyright 2020 Stellar Development Foundation and contributors. Licensed+// under the Apache License, Version 2.0. See the COPYING file at the root+// of this distribution or at http://www.apache.org/licenses/LICENSE-2.0++#include "WriteVerifiedCheckpointHashesWork.h"+#include "catchup/CatchupConfiguration.h"+#include "catchup/CatchupRange.h"+#include "catchup/DownloadApplyTxsWork.h"+#include "catchup/VerifyLedgerChainWork.h"+#include "history/FileTransferInfo.h"+#include "history/HistoryManager.h"+#include "historywork/BatchDownloadWork.h"+#include "ledger/LedgerManager.h"+#include "ledger/LedgerRange.h"+#include "main/Application.h"+#include "util/Logging.h"+#include "work/ConditionalWork.h"+#include <Tracy.hpp>+#include <fmt/format.h>++namespace stellar+{++class TmpDirOwnerWork : public Work+{+    TmpDir const mTmpDir;++  public:+    TmpDirOwnerWork(Application& app, std::string const& name)+        : Work(app, name, RETRY_NEVER)+        , mTmpDir(mApp.getTmpDirManager().tmpDir(getName()))+    {+    }+    BasicWork::State+    doWork() override+    {+        return State::WORK_SUCCESS;+    }+    TmpDir const&+    getTmpDir()+    {+        return mTmpDir;+    }+};++WriteVerifiedCheckpointHashesWork::WriteVerifiedCheckpointHashesWork(+    Application& app, LedgerNumHashPair rangeEnd, std::string const& outputFile,+    std::shared_ptr<HistoryArchive> archive)+    : BatchWork(app, "write-verified-checkpoint-hashes")+    , mRangeEnd(rangeEnd)+    , mCurrCheckpoint(rangeEnd.first)+    , mArchive(archive)+    , mOutputFile(std::make_shared<std::ofstream>(outputFile))+{+    if (mArchive)+    {+        CLOG(INFO, "History")+            << "WriteVerifiedCheckpointHashesWork: selected archive "+            << mArchive->getName();+    }+    if (!*mOutputFile)

Ah! You're talking about setting the stream's exception mask, sorry, I misread. For sure, makes sense (I had forgotten this facility even existed :P)

It's a good point about not setting this everywhere we should. I'll open another PR shortly.

graydon

comment created time in 13 days

pull request commentstellar/stellar-core

Verify ledger chain command

Fixed the error (was holding a &-ref to invalid memory!) in e618550b57f878af3d3eaef482f4002743c8db48, should be good to squash and land now.

graydon

comment created time in 13 days

push eventgraydon/stellar-core

Jon Jove

commit sha cb9008f53d4c165ab28a5d3e06ae237f95a3d5e6

Fix ManageOfferOpFrameBase::insertLedgerKeysToPrefetch again

view details

MonsieurNicolas

commit sha 7b581185c41d4a49d6c4009d95c0d825e7bf528b

warning police

view details

MonsieurNicolas

commit sha f41504d33e383abba316e33090e98bf773431ac6

fix compile error - conform to C++ 14 struct initialization is a C99 extension https://en.cppreference.com/w/c/language/struct_initialization that was only added in C++ 20 as designator initializers https://en.cppreference.com/w/cpp/language/aggregate_initialization

view details

Latobarita

commit sha 01ab2af163011c194ded22a32543a139be831a2a

Merge pull request #2611 from MonsieurNicolas/windowsCompileFixes20200707 Fix compile error (non C++ 14 compliant code) Reviewed-by: MonsieurNicolas

view details

Graydon Hoare

commit sha bc22f0b576ee2e8ff675ca3a4b49638a90cfe2c6

log getpeername errors more explicitly

view details

Latobarita

commit sha d2d0a2cbdc65bb53f776758890f3dc676ea67b1f

Merge pull request #2614 from graydon/getpeername-error-logging log getpeername errors explicitly Reviewed-by: MonsieurNicolas

view details

Terence Rokop

commit sha 84c94dcc8488372131cc27e323fbff287350d17c

Issue #2615: Add missing string to log message I managed to drop the intended column string in a database schema upgrade log message in PR 2593 for issue 2570.

view details

Latobarita

commit sha 13de97fe7707e9c4dc58ee5b60c0112f1908d40b

Merge pull request #2616 from rokopt/issue-2615-empty-column-strings Issue #2615: Add missing string to log message Reviewed-by: jonjove

view details

Latobarita

commit sha cb05a741b4464eff7e67d3fbd3ae021074d6fb3c

Merge pull request #2600 from jonjove/fix-offer-prefetching Fix ManageOfferOpFrameBase::insertLedgerKeysToPrefetch again Reviewed-by: MonsieurNicolas

view details

Siddharth Suresh

commit sha bb33f5327de53b64b097f9f57e1e5c13c2dd64b7

increase catchup timeout in testcases for address sanitizer

view details

Siddharth Suresh

commit sha 7867dfcca27b83c1324fef1947d08aab542290f9

add SECTION so overlapping protocol versions don't conflict and cause Catch to skip test

view details

Siddharth Suresh

commit sha 770cd8308b91390830e440bbdac5f47faec0301c

bump protocol version

view details

Jon Jove

commit sha bba5707ab0646c342d46ce048d2fbedc1aca6332

Add new LedgerEntry and LedgerKey types for ClaimableBalance

view details

Jon Jove

commit sha 288ceb43f4ef28ee120ceaba36c5afc0e2794723

Refactor LedgerCmp and expand it for new LedgerEntryTypes

view details

Jon Jove

commit sha a4a1ea086e3e51ce6da388e05efb28d79a942786

Update LedgerEntryKey for ClaimableBalance

view details

Jon Jove

commit sha 74b7f964d6cb7a0f14d270b41b540faa7bb40d9e

Update LedgerTestUtils for ClaimableBalance

view details

Jon Jove

commit sha 901e6bbc2d23dc9826d92d30305b1bf941b47e1c

Update LedgerHashUtils for ClaimableBalance

view details

Jon Jove

commit sha ea6830315ac15041b68845e45e100cf86c5a03a1

Update LedgerTxnTests for ClaimableBalance

view details

Jon Jove

commit sha 6c71d5606178c7d4f2b4cd7b3765b4057f35c736

Implement SQL for ClaimableBalance

view details

Siddharth Suresh

commit sha ba6155091bcc41aa986bb0613f2c66405b7160a6

Use claimable balance SQL in LedgerTxn

view details

push time in 13 days

issue openedservo/pathfinder

glyph placeholders / non-rendering text in demos

Ubuntu 18.04 LTS, fairly stock environment, I checked out pathfinder and did cd examples/canvas_nanovg/ then cargo run --release and I got the following display with missing glyph placeholders:

image

Since I have no idea what the cause is -- looks like the fonts are in the tree even! -- I'll just list everything else about the software & hardware context:

Intel(R) Xeon(R) CPU E5-2680 v2 @ 2.80GHz

4.15.0-111-generic #112-Ubuntu SMP Thu Jul 9 20:32:34 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

NVIDIA Corporation GP107 [GeForce GTX 1050 Ti]

X.Org version: 1.19.6


$ rustup show
Default host: x86_64-unknown-linux-gnu
rustup home:  /home/graydon/.rustup

installed toolchains
--------------------

stable-x86_64-unknown-linux-gnu
nightly-2019-07-18-x86_64-unknown-linux-gnu
nightly-x86_64-unknown-linux-gnu (default)

active toolchain
----------------

nightly-x86_64-unknown-linux-gnu (default)
rustc 1.44.0-nightly (14061868b 2020-04-10)

$ nvidia-smi 
Wed Jul 22 12:39:19 2020       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 440.100      Driver Version: 440.100      CUDA Version: 10.2     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GTX 105...  Off  | 00000000:05:00.0  On |                  N/A |
| 31%   38C    P0    N/A /  75W |   2168MiB /  4038MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+

created time in 14 days

fork graydon/simde

Implementations of SIMD instruction sets for systems which don't natively support them.

https://simd-everywhere.github.io/blog/

fork in 14 days

startedsimd-everywhere/simde

started time in 14 days

pull request commentstellar/stellar-core

Verify ledger chain command

I somehow managed to break the test in this in the meantime I guess while addressing Marta's comments. Will need to go figure out how. Out of time for tonight though.

graydon

comment created time in 14 days

Pull request review commentstellar/stellar-core

Verify ledger chain command

+// Copyright 2020 Stellar Development Foundation and contributors. Licensed+// under the Apache License, Version 2.0. See the COPYING file at the root+// of this distribution or at http://www.apache.org/licenses/LICENSE-2.0++#include "WriteVerifiedCheckpointHashesWork.h"+#include "catchup/CatchupConfiguration.h"+#include "catchup/CatchupRange.h"+#include "catchup/DownloadApplyTxsWork.h"+#include "catchup/VerifyLedgerChainWork.h"+#include "history/FileTransferInfo.h"+#include "history/HistoryManager.h"+#include "historywork/BatchDownloadWork.h"+#include "ledger/LedgerManager.h"+#include "ledger/LedgerRange.h"+#include "main/Application.h"+#include "util/Logging.h"+#include "work/ConditionalWork.h"+#include <Tracy.hpp>+#include <fmt/format.h>++namespace stellar+{++class TmpDirOwnerWork : public Work+{+    TmpDir const mTmpDir;++  public:+    TmpDirOwnerWork(Application& app, std::string const& name)+        : Work(app, name, RETRY_NEVER)+        , mTmpDir(mApp.getTmpDirManager().tmpDir(getName()))+    {+    }+    BasicWork::State+    doWork() override+    {+        return State::WORK_SUCCESS;+    }+    TmpDir const&+    getTmpDir()+    {+        return mTmpDir;+    }+};++WriteVerifiedCheckpointHashesWork::WriteVerifiedCheckpointHashesWork(+    Application& app, LedgerNumHashPair rangeEnd, std::string const& outputFile,+    std::shared_ptr<HistoryArchive> archive)+    : BatchWork(app, "write-verified-checkpoint-hashes")+    , mRangeEnd(rangeEnd)+    , mCurrCheckpoint(rangeEnd.first)+    , mArchive(archive)+    , mOutputFile(std::make_shared<std::ofstream>(outputFile))+{+    if (mArchive)+    {+        CLOG(INFO, "History")+            << "WriteVerifiedCheckpointHashesWork: selected archive "+            << mArchive->getName();+    }+    if (!*mOutputFile)

I'm not sure I understand -- we are detecting badbit and throwing on error here. Setting badbit again doesn't make sense to me.

graydon

comment created time in 14 days

Pull request review commentstellar/stellar-core

Verify ledger chain command

+#pragma once++#include "util/NonCopyable.h"+#include <exception>+#include <memory>++// Copyright 2020 Stellar Development Foundation and contributors. Licensed+// under the Apache License, Version 2.0. See the COPYING file at the root+// of this distribution or at http://www.apache.org/licenses/LICENSE-2.0++// This is a helper class somewhat like a std::shared_future but with no+// integration with async / threading / blocking. It's just a sharable,+// write-once variable you can poll for readiness and store either a value or an+// exception in. Supports "dataflow"-style programming.++namespace stellar+{+template <typename T> class WriteOnceSharedVar+{+    struct Impl : public NonMovableOrCopyable+    {+        std::unique_ptr<T> mValue;+        std::exception_ptr mException;+    };++    std::shared_ptr<Impl> mImpl;++  public:+    WriteOnceSharedVar() : mImpl(std::make_shared<Impl>())+    {+    }++    // Helper to construct in written state.+    WriteOnceSharedVar(T const& v) : mImpl(std::make_shared<Impl>())+    {+        setValue(v);+    }++    bool+    hasValue() const+    {+        assert(mImpl);+        return static_cast<bool>(mImpl->mValue);+    }++    bool+    hasException() const+    {+        assert(mImpl);+        return static_cast<bool>(mImpl->mException);+    }++    void+    setValue(T const& v)+    {+        if (hasValue())+        {+            throw std::runtime_error("write-once variable value already set");+        }+        if (hasException())+        {+            throw std::runtime_error(+                "write-once variable exception already set");+        }+        mImpl->mValue = std::make_unique<T>(v);+    }++    // An "idempotent" set that either sets the value if it's not set,+    // or asserts that the parameter is equal to the already-set value.+    //+    // If an exception is already set, it is left in place and neither

N/A at this point, dropping this class entirely.

graydon

comment created time in 14 days

Pull request review commentstellar/stellar-core

Verify ledger chain command

+#pragma once++#include "util/NonCopyable.h"+#include <exception>+#include <memory>++// Copyright 2020 Stellar Development Foundation and contributors. Licensed+// under the Apache License, Version 2.0. See the COPYING file at the root+// of this distribution or at http://www.apache.org/licenses/LICENSE-2.0++// This is a helper class somewhat like a std::shared_future but with no+// integration with async / threading / blocking. It's just a sharable,+// write-once variable you can poll for readiness and store either a value or an+// exception in. Supports "dataflow"-style programming.++namespace stellar+{+template <typename T> class WriteOnceSharedVar+{+    struct Impl : public NonMovableOrCopyable+    {+        std::unique_ptr<T> mValue;+        std::exception_ptr mException;+    };++    std::shared_ptr<Impl> mImpl;++  public:+    WriteOnceSharedVar() : mImpl(std::make_shared<Impl>())+    {+    }++    // Helper to construct in written state.+    WriteOnceSharedVar(T const& v) : mImpl(std::make_shared<Impl>())+    {+        setValue(v);+    }++    bool+    hasValue() const+    {+        assert(mImpl);+        return static_cast<bool>(mImpl->mValue);+    }++    bool+    hasException() const+    {+        assert(mImpl);+        return static_cast<bool>(mImpl->mException);+    }++    void+    setValue(T const& v)+    {+        if (hasValue())+        {+            throw std::runtime_error("write-once variable value already set");+        }+        if (hasException())+        {+            throw std::runtime_error(+                "write-once variable exception already set");+        }+        mImpl->mValue = std::make_unique<T>(v);+    }++    // An "idempotent" set that either sets the value if it's not set,+    // or asserts that the parameter is equal to the already-set value.+    //+    // If an exception is already set, it is left in place and neither+    // of the above actions occurs.+    void+    setValueIdem(T const& v)+    {+        if (!hasException())+        {+            if (hasValue())+            {+                assert(getValue() == v);

N/A at this point, dropping this class entirely.

graydon

comment created time in 14 days

Pull request review commentstellar/stellar-core

Verify ledger chain command

+#pragma once++#include "util/NonCopyable.h"+#include <exception>+#include <memory>++// Copyright 2020 Stellar Development Foundation and contributors. Licensed+// under the Apache License, Version 2.0. See the COPYING file at the root+// of this distribution or at http://www.apache.org/licenses/LICENSE-2.0++// This is a helper class somewhat like a std::shared_future but with no+// integration with async / threading / blocking. It's just a sharable,+// write-once variable you can poll for readiness and store either a value or an+// exception in. Supports "dataflow"-style programming.++namespace stellar+{+template <typename T> class WriteOnceSharedVar+{+    struct Impl : public NonMovableOrCopyable+    {+        std::unique_ptr<T> mValue;+        std::exception_ptr mException;+    };++    std::shared_ptr<Impl> mImpl;++  public:+    WriteOnceSharedVar() : mImpl(std::make_shared<Impl>())+    {+    }++    // Helper to construct in written state.+    WriteOnceSharedVar(T const& v) : mImpl(std::make_shared<Impl>())+    {+        setValue(v);+    }++    bool+    hasValue() const+    {+        assert(mImpl);+        return static_cast<bool>(mImpl->mValue);+    }++    bool+    hasException() const+    {+        assert(mImpl);+        return static_cast<bool>(mImpl->mException);+    }++    void+    setValue(T const& v)+    {+        if (hasValue())+        {+            throw std::runtime_error("write-once variable value already set");+        }+        if (hasException())+        {+            throw std::runtime_error(+                "write-once variable exception already set");+        }+        mImpl->mValue = std::make_unique<T>(v);+    }++    // An "idempotent" set that either sets the value if it's not set,+    // or asserts that the parameter is equal to the already-set value.+    //+    // If an exception is already set, it is left in place and neither+    // of the above actions occurs.+    void+    setValueIdem(T const& v)+    {+        if (!hasException())+        {+            if (hasValue())+            {+                assert(getValue() == v);+            }+            else+            {+                setValue(v);+            }+        }+    }++    void+    setRuntimeError(std::string const& msg)+    {+        setException(std::runtime_error(msg));+    }++    // Sets an exception "idempotently", leaving an existing

N/A at this point, dropping this class entirely.

graydon

comment created time in 14 days

Pull request review commentstellar/stellar-core

Verify ledger chain command

+#pragma once++#include "util/NonCopyable.h"+#include <exception>+#include <memory>++// Copyright 2020 Stellar Development Foundation and contributors. Licensed+// under the Apache License, Version 2.0. See the COPYING file at the root+// of this distribution or at http://www.apache.org/licenses/LICENSE-2.0++// This is a helper class somewhat like a std::shared_future but with no+// integration with async / threading / blocking. It's just a sharable,+// write-once variable you can poll for readiness and store either a value or an+// exception in. Supports "dataflow"-style programming.++namespace stellar+{+template <typename T> class WriteOnceSharedVar+{+    struct Impl : public NonMovableOrCopyable+    {+        std::unique_ptr<T> mValue;+        std::exception_ptr mException;+    };++    std::shared_ptr<Impl> mImpl;++  public:+    WriteOnceSharedVar() : mImpl(std::make_shared<Impl>())+    {+    }++    // Helper to construct in written state.+    WriteOnceSharedVar(T const& v) : mImpl(std::make_shared<Impl>())+    {+        setValue(v);+    }++    bool+    hasValue() const+    {+        assert(mImpl);+        return static_cast<bool>(mImpl->mValue);+    }++    bool+    hasException() const+    {+        assert(mImpl);+        return static_cast<bool>(mImpl->mException);+    }++    void+    setValue(T const& v)+    {+        if (hasValue())+        {+            throw std::runtime_error("write-once variable value already set");+        }+        if (hasException())+        {+            throw std::runtime_error(+                "write-once variable exception already set");+        }+        mImpl->mValue = std::make_unique<T>(v);+    }++    // An "idempotent" set that either sets the value if it's not set,+    // or asserts that the parameter is equal to the already-set value.+    //+    // If an exception is already set, it is left in place and neither+    // of the above actions occurs.+    void+    setValueIdem(T const& v)+    {+        if (!hasException())+        {+            if (hasValue())+            {+                assert(getValue() == v);+            }+            else+            {+                setValue(v);+            }+        }+    }++    void+    setRuntimeError(std::string const& msg)+    {+        setException(std::runtime_error(msg));+    }++    // Sets an exception "idempotently", leaving an existing+    // exception in place if it already exits. Does not compare+    // exceptions (they are not generally comparable anyways).+    void+    setRuntimeErrorIdem(std::string const& msg)+    {+        if (!hasException())+        {+            setRuntimeError(msg);+        }+    }++    template <typename E>+    void+    setException(E exn) const+    {+        setException(std::make_exception_ptr(exn));+    }++    void+    setException(std::exception_ptr ptr) const+    {+        if (hasValue())+        {+            throw std::runtime_error("write-once variable value already set");+        }+        if (hasException())+        {+            throw std::runtime_error(+                "write-once variable exception already set");+        }+        mImpl->mException = ptr;+    }++    T const&+    getValue() const+    {+        if (!hasValue())+        {+            throw std::runtime_error("write-once variable has no value");+        }+        if (hasException())+        {+            throw std::runtime_error("write-once variable has exception");

Yeah for some reason I convinced myself that std::future was only really related to async execution and that it'd be overkill. I just redid this on the basis of std::promise and while it's a little clunky (you have to arrange for someone to keep the promise alive, you can't convert directly from values to read/write pairs and the future-end isn't enough to keep the shared state alive, and you also separately have to arrange for a transfer of the future out of it into a shared_future in order to facilitate multiple access) it's .. probably for the best to go this way. Agreed.

graydon

comment created time in 14 days

push eventgraydon/stellar-core

Graydon Hoare

commit sha 124ca80097a178b84af3ddc1cdcc640b2037de71

Use std::{promise,future} not WriteOnceSharedVar

view details

push time in 14 days

Pull request review commentstellar/stellar-core

Do not wait for in-progress merges on shutdown

 Bucket::merge(BucketManager& bucketManager, uint32_t maxProtocolVersion,                              mc, ctx, doFsync);      BucketEntryIdCmp cmp;+    size_t iter = 0;+     while (oi || ni)     {+        // Check if the merge should be stopped every few entries+        if (++iter >= 1000)+        {+            iter = 0;+            if (bucketManager.isShutdown())+            {+                // Stop merging, as BucketManager is now shutdown+                // This is safe as temp file has not been adopted yet,+                // so it will be removed with the tmp dir+                return nullptr;

I don't think this is generally a safe thing to do -- FutureBucket's contract is that its mOutputBucketFuture returns a non-null shared pointer. See for example FutureBucket.cpp:260, in the function FutureBucket::resolve() where we call getHash() on the returned value from mOutputBucketFuture.get().

And more generally FutureBucket::resolve() is supposed to also return a non-null shared_ptr. The whole lifecycle of FutureBucket assumes that it runs to completion, not that it can be interrupted and left in an invalid state, eg. BucketLevel::commit() calls setCurr(mNextCurr.resolve()) and so on.

Granted none of these things are likely to happen if we're in the process of shutting down, but changing the contract this way feels .. kinda worrying to me.

Maybe we could approach it by throwing an exception in the merge function, in this shutdown condition? I believe this will be caught by the std::packaged_task doing the merge and will re-propagate on access to its caller (eg. if anyone calls get() it'll throw). It might also be worth adding a release assert that the future is valid() in any access paths; it looks like we're not doing that currently and we should probably fail if it's ever not.

marta-lokhova

comment created time in 14 days

startedmicrosoft/ProcMon-for-Linux

started time in 15 days

startedgraphsketcher/GraphSketcher

started time in 15 days

startedballista-compute/ballista

started time in 15 days

push eventgraydon/edelcrantz

Graydon Hoare

commit sha 1de67d243638168a6ed2485644a5854134c8dd0c

description

view details

push time in 18 days

push eventgraydon/edelcrantz

Graydon Hoare

commit sha da25d04f551dddc925cd58e72c5ad7ffa9da4df0

gitignore

view details

push time in 18 days

create barnchgraydon/edelcrantz

branch : master

created branch time in 18 days

created repositorygraydon/edelcrantz

created time in 18 days

starteddeislabs/krustlet

started time in 23 days

startedByron/git-oxide

started time in 23 days

startedmicrosoft/rust_win_etw

started time in 23 days

startedyeslogic/fathom

started time in 23 days

issue commentstellar/stellar-core

Figure out how to instrument performance critical areas

@jonjove Absolutely. There are maybe 4 things to note here:

  1. If you're trying to differentiate zones in the trace that have a relatively small (dozens-to-hundreds) set of possible values -- i.e. values that you're interested in grouping by for aggregate-analysis purposes -- then you should probably just attach the values to the zones in question using ZoneValue(int64) or ZoneText(char*) annotations. This will show up as annotations and the zone inspection dialogs will allow grouping and subsetting based on them. There are examples of this already in stellar-core's code.
  2. If you're trying to record "one time" event-detail messages that you just want to inspect in the timeline as they pass, but they're not going to be meaningfully grouped or anything, you should probably emit a TracyMessage(char*) which will just show up as a point-in-time message in the timeline and the messages log. There are not currently any uses of this in stellar-core but it's very straightforward.
  3. If you want to watch some numeric value that varies at a relatively high frequency and across many values (so is less appropriate for ZoneValue) you can plot it like a metric underneath the timeline using TracyPlot(char*, int64). There are instances of this in stellar-core presently.
  4. We do not do this presently, but we can if it matters: tracy supports a way for the client (stellar-core) to present a bunch of "parameters" to the user operating the server (GUI), using TracyParameterRegister and TracyParameterSetup. These appear in the form of named numbers or booleans in the GUI that the user can adjust to change how the collection is proceeding, interactively; they are communicated back to a client-provided callback that we can do whatever we want with in stellar-core. They are there to allow the analyst to dynamically turn on or off subsets of instrumentation, if there are some that are particularly egregious in the volume of data they produce or such.
jonjove

comment created time in 25 days

startedfrankmcsherry/columnation

started time in a month

issue commentstellar/stellar-core

Discussion - historical data tables future in core

I'd be ok keeping a circular log in memory that we dump on error, of even repeatedly writing (asynchronously) to disk in the buckets dir if it were helpful. it'd add a fair bit of write overhead but they'd just be sequential and bufferable, and I don't think we're usually keeping the disk saturated in the idle periods between closes.

MonsieurNicolas

comment created time in a month

startedjanet-lang/janet

started time in a month

PR opened stellar/stellar-core

log getpeername errors explicitly

This just improves reporting-precision of errors from getpeername / remote peer resolution, to help diagnose rare network conditions.

+13 -1

0 comment

2 changed files

pr created time in a month

create barnchgraydon/stellar-core

branch : getpeername-error-logging

created branch time in a month

push eventgraydon/stellar-core

Graydon Hoare

commit sha c0f9056aeca8e6227e1dd5288859c13af8ebc47e

Document "verify-checkpoints" command and "--trusted-checkpoint-hashes" option to "catchup"

view details

push time in a month

push eventgraydon/stellar-core

Terence Rokop

commit sha 0eb4454ea85e9badebac4a872cac4754f315b29e

Add a small test-code-injection interface Add a mechanism, using the Factory pattern, for production code to create injection points that can call back into test-defined code (or perhaps other applications in the future; parameters used by the injected code can vary with each test or other use, including, in effect, the method called, by the use of lambdas). A test client can create one derived class of Database for each injection point or combination of injection points (into methods of class Database -- other classes could also use this mechanism for injecting code into their methods) to override with methods that do something. The derived class's constructor may take any list of parameter types; each call to createTestApplication() (or Application::create()) passes its own list of parameters of those types. The derived class may store the parameters received by its constructor, so that it is effectively called with parameters even though the production code calls a method with no parameters. This means in particular that TestApplication itself does not need to acquire new members to allow tests to store parameters to injection point methods. A single derived class of Database may be used by multiple test cases even to call, in effect, different injected methods, because the parameters passed by createTestApplication() to the constructor of the derived class of TestApplication may include lambdas. This checkin also adds one use of the new mechanism, actBeforeDBSchemaUpgrade(), by production code to create an injection point.

view details

Terence Rokop

commit sha 0af8465f95ff67668b6885225dc1987c97a2f88d

Utilities for decoding base-64-encoded opaque XDR

view details

Terence Rokop

commit sha 244cd1840964f7f9f90c93c14513069e08801306

Maps between vectors and SQL records Introduce three generic functions for operating on database records in bulk: - selectMap() takes a function which maps a SQL record to an element of a client-defined (templated) type, and produces a function which takes a SQL SELECT statement and produces a vector of the client-defined type comprising the list of selected records mapped through the client-defined function. - updateMap() takes a function which maps an element of a client-defined (templated) type to a SQL UPDATE statement, and produces a function which generates a series of SQL UPDATE statements from a vector of the client-defined type. - selectUpdateMap() produces the composition of updateMap() following selectMap(), namely, a function which takes a SQL SELECT statement and produces for each selected record an UPDATE statement derived from a transformation applied to the record via client-defined functions which interpret the record as an element of a client-defined type and then use that interpretation to choose some update to perform on it.

view details

Terence Rokop

commit sha 1c4510b8c0acc11df0a13fdffb7374ddfa4af100

DB schema upgrade to make extensions opaque Upgrade the DB schema from 12 to 13. The new schema makes all of the extensions in all ledger entry types opaque to the database code itself. Those extensions are the extension to LedgerEntry itself, which is stored in the table for each LedgerEntry type (account, trustline, data, offer), and the extensions which are specific to each type. The idea of making the extensions opaque is that they can be extended in the future without further changes to either the DB schema or the DB code, as the database itself will simply continue to treat the extensions as variable-length, opaque XDR. The cost is that we can't query on components of the extensions (which we don't do currently, and won't do with any of the extensions that we currently have planned). The general LedgerEntry extension, as well as the extensions to DataEntry and OfferEntry, are currently unused -- they are unions with the only version being 0 and no content besides that fixed version number, and they are not stored in any database columns at all. The upgrade code therefore simply creates columns for them and populates all of the cells of the new columns with base-64 encodings of opaque XDR representing that trivial tagged union. The two extensions that are already used are those specific to AccountEntry and TrustLineEntry, both of which contain the same thing, namely a Liabilities structure (which contains two fields, one for buying and one for selling). Those extensions are tagged unions with a genuine version option -- version 0 for entries that have never yet had liabilities, and then version 1 when liabilities are first created for the entry. Existing databases have NULL in the "buyingliabilities" and "sellingliabilities" columns for the version-0 entries, and non-NULL values, namely base-64 encodings of the XDR for the Liabilities structure, for the version-1 entries. Therefore, the upgrade code for AccountEntry and TrustLineEntry performs a conversion: for each record, it reads the "buyingliabilities" and "sellingliabilities" columns, which are 64-bit integers transparent to the database, and generates encoded XDR, opaque to the database, representing an extension field which, in those cases, has a Liabilities field, and has the values read from the old columns filled in. (If the old liabilities fields for a record are NULL, then the new "extension" column for that record is also NULL -- the new live database code will recognize that NULL values of those two extensions are possible.) Once all of the records have had their new "extension" columns written, the upgrade code deletes the old "buyingliabilities" and "sellingliabilities" columns.

view details

Terence Rokop

commit sha 2957ab216f64c115b69b4c7fb8578477f57d594d

Live DB code for schema 13 with opaque extensions Modify the live database code for all of the LedgerEntry types -- AccountEntry, TrustLineEntry, DataEntry, and OfferEntry -- to use database schema 13, in which the extensions for each of those individual entry types, as well as the extension for LedgerEntry itself which is also stored in the table for whatever specific type a given LedgerEntry has, are just base-64-encoded opaque XDR as far as the database is concerned, regardless of how the ledger-transaction code inteprets those extensions, now or in the future.

view details

Terence Rokop

commit sha 5801f31f822d74c9cd2c0af623e3ed52c85d5cea

Test upgrades to database schema 13 Test upgrades to database schema 13, in which all extensions in all ledger entry types become opaque XDR as far as the database is concerned. The tests use the test-code-injection mechanism which allow them to run arbitrary code between the creation of a database (which is done in an old schema version, MIN_SCHEMA_VERSION) and the upgrading of that database to the current schema version (SCHEMA_VERSION). They create ledger entries of each of the four types, using raw SQL so that they can create them in the old format which the live production code no longer uses. After the database upgrade completes (which it has done by the time that createTestApplication() returns), the tests validate that the contents of the ledger entries are intact and logically unchanged (although their representations in the database have changed, with some columns having been created and some having been deleted, and some contents being encoded differently).

view details

Siddharth Suresh

commit sha f891619cb1302a4dd00e41e08685c8a24b600a9d

run LedgerTxnTests with postgres

view details

Latobarita

commit sha ed3212fdc27e153ed34e6749d6bb95875913702c

Merge pull request #2603 from sisuresh/postgres_for_ledger_txn_tests Run LedgerTxnTests with Postgres Reviewed-by: jonjove

view details

Hidenori

commit sha d9303f8a2b2d1777f1b1602a37839a60fd145668

Introduce a constant for the maximum nesting level for quorums

view details

Hidenori

commit sha b23ca5eea3e6747d08bf63526317ebf949686493

Add tests to ensure that the constant is used in both places and the meaning of the level is consistent across the codebase.

view details

Latobarita

commit sha 8dc8718db0b2fbffe24321f1b193a0301a9a79fe

Merge pull request #2602 from hidenori-shinohara/quourm-level Introduce a constant for the maximum nesting level for quorums Reviewed-by: graydon

view details

Latobarita

commit sha 7ab017b888b9f138070307bb1785eb433ddcd5d1

Merge pull request #2593 from rokopt/issue-2570-pr-4 Issue #2570: DB schema change to make extensions opaque Reviewed-by: MonsieurNicolas

view details

Siddharth Suresh

commit sha 080a2454475796f2bb45e176ae6d1bcd8daeb008

Add option to use a close time offset for max time bound validity check

view details

Siddharth Suresh

commit sha e18ca355fd2242139fc4c623913edb42f9b609dd

Use upper bound estimate for tx validity pre-consensus

view details

Latobarita

commit sha bf8486bd99bc8d19e569b765d3244e523d5c3e18

Merge pull request #2608 from sisuresh/tx_flooding Improve precision of transaction expiration while flooding Reviewed-by: MonsieurNicolas

view details

Graydon Hoare

commit sha 96be911f9e328f42d69dc0e78d89ce0b9488ffe0

Add "verify-hit benchmarking" test to CryptoTests.cpp

view details

Graydon Hoare

commit sha 20b490c300f08a0dff448bd0d8d0e176054fa115

Make SHA256 class stack-allocated, remove shared global gHasher.

view details

Latobarita

commit sha 6a70b25ed98d789484adb2f78154a01758ef0580

Merge pull request #2605 from graydon/bug-2604-mutex-guard-ghasher Remove gHasher and use stack-local incremental SHA256 class. Reviewed-by: MonsieurNicolas

view details

Jon Jove

commit sha d8f2731627b080019c277f261370a3e076278676

Remove version-checks for LedgerTxn after protocol 13

view details

Latobarita

commit sha 84ab72e5a1aac7ff925059d7e26b9cd12366d5a8

Merge pull request #2599 from jonjove/remove-ltx-version-checks Remove version-checks for LedgerTxn after protocol 13 Reviewed-by: MonsieurNicolas

view details

push time in a month

Pull request review commentstellar/stellar-core

Verify ledger chain command

  #pragma once -#include "catchup/CatchupConfiguration.h"-#include "catchup/VerifyLedgerChainWork.h"-#include "history/HistoryArchive.h"-#include "historywork/BatchDownloadWork.h"-#include "historywork/GetHistoryArchiveStateWork.h"+#include "ledger/LedgerRange.h" #include "util/WriteOnceSharedVar.h"+#include "work/BatchWork.h" #include "work/Work.h"

Yep, fixed.

graydon

comment created time in a month

Pull request review commentstellar/stellar-core

Verify ledger chain command

+// Copyright 2020 Stellar Development Foundation and contributors. Licensed+// under the Apache License, Version 2.0. See the COPYING file at the root+// of this distribution or at http://www.apache.org/licenses/LICENSE-2.0++#include "historywork/WriteVerifiedCheckpointHashesWork.h"+#include "catchup/VerifyLedgerChainWork.h"+#include "history/HistoryManager.h"+#include "historywork/BatchDownloadWork.h"+#include "ledger/LedgerManager.h"+#include "ledger/LedgerRange.h"+#include "main/Application.h"+#include "util/Logging.h"+#include "work/ConditionalWork.h"+#include <Tracy.hpp>+#include <algorithm>+#include <fmt/format.h>++namespace stellar+{+;++WriteVerifiedCheckpointHashesWork::WriteVerifiedCheckpointHashesWork(+    Application& app, LedgerNumHashPair rangeEnd, std::string const& outputFile,+    std::shared_ptr<HistoryArchive> archive)+    : BatchWork(app, "write-verified-checkpoint-hashes")+    , mRangeEnd(rangeEnd)+    , mCurrCheckpoint(rangeEnd.first)+    , mArchive(archive)+    , mOutputFile(std::make_shared<std::ofstream>(outputFile))+{+    if (mArchive)+    {+        CLOG(INFO, "History") << "selected archive " << mArchive->getName();+    }+    if (!*mOutputFile)+    {+        throw std::runtime_error("error opening output file " + outputFile);+    }+    (*mOutputFile) << "[";+}++WriteVerifiedCheckpointHashesWork::~WriteVerifiedCheckpointHashesWork()+{+    // Each line of output made by a VerifyLedgerChainWork has a trailing comma,+    // and trailing commas are not a valid end of a JSON array; so we terminate+    // the array here with an entry that does _not_ have a trailing comma (and+    // identifies an invalid ledger number anyways).+    (*mOutputFile) << "\n[0, \"\"]\n]\n";

I want to support partial results so I've done what you suggest in the second part here, implemented a clear open/close pair called at reset as well as construct/destruct, and with an explicit truncate flag.

graydon

comment created time in a month

Pull request review commentstellar/stellar-core

Verify ledger chain command

 VerifyLedgerChainWork::onRun()         {             CLOG(INFO, "History") << "History chain [" << mRange.mFirst << ","                                   << mRange.last() << "] verified";+            if (mOutputStream)

Sure, might as well move it now. Fixed.

graydon

comment created time in a month

Pull request review commentstellar/stellar-core

Verify ledger chain command

 VerifyLedgerChainWork::verifyHistoryOfSingleCheckpoint()         return HistoryManager::VERIFY_STATUS_ERR_MISSING_ENTRIES;     } +    // We just finished scanning a checkpoint. We first grab the _incoming_+    // hash-link our caller (or previous call to this method) saved for us.+    auto incoming = mVerifiedAhead;++    {+        // While scanning the checkpoint, we saved (in `first`) the first+        // (lowest-numbered) ledger in this checkpoint while we were scanning+        // past it.+        //+        // We now write back (to `mVerifiedAhead`) the _outgoing_ hash-link and+        // expected sequence number of the next-lowest ledger number before this+        // checkpoint, which we'll encounter on the next call to this method+        // (processing the numerically-lower checkpoint) in the code below that+        // checks that the last ledger in a checkpoint agrees with the expected+        // _incoming_ hash-link.+        //+        // Note `mVerifiedAhead` is written here after being read moments+        // before. We're currently writing the value to be used in the _next_+        // call to this method.+        auto hash = make_optional<Hash>(first.header.previousLedgerHash);+        mVerifiedAhead = LedgerNumHashPair(first.header.ledgerSeq - 1, hash);+    }++    // We check to see if we just finished the first call to this method in this+    // object's work (true when this checkpoint-end landed on the+    // highest-numbered ledger in the range)     if (curr.header.ledgerSeq == mRange.last())     {-        assert(nextCheckpointFirstLedger.first == 0);-        nextCheckpointFirstLedger = mTrustedEndLedger;+        // If so, there should be no "saved" incoming hash-link value from+        // a previous iteration.+        assert(incoming.first == 0);++        // Instead, there _should_ be a value in the shared write-once+        // variable this work object reads its initial trust from.+        assert(mTrustedMaxLedger.hasValueOrException());++        if (mTrustedMaxLedger.hasException())

Fair. Fixed.

graydon

comment created time in a month

Pull request review commentstellar/stellar-core

Verify ledger chain command

 VerifyLedgerChainWork::verifyHistoryOfSingleCheckpoint()         return HistoryManager::VERIFY_STATUS_ERR_MISSING_ENTRIES;     } +    // We just finished scanning a checkpoint. We first grab the _incoming_+    // hash-link our caller (or previous call to this method) saved for us.+    auto incoming = mVerifiedAhead;++    {+        // While scanning the checkpoint, we saved (in `first`) the first+        // (lowest-numbered) ledger in this checkpoint while we were scanning+        // past it.+        //+        // We now write back (to `mVerifiedAhead`) the _outgoing_ hash-link and+        // expected sequence number of the next-lowest ledger number before this+        // checkpoint, which we'll encounter on the next call to this method+        // (processing the numerically-lower checkpoint) in the code below that+        // checks that the last ledger in a checkpoint agrees with the expected+        // _incoming_ hash-link.+        //+        // Note `mVerifiedAhead` is written here after being read moments+        // before. We're currently writing the value to be used in the _next_+        // call to this method.+        auto hash = make_optional<Hash>(first.header.previousLedgerHash);+        mVerifiedAhead = LedgerNumHashPair(first.header.ledgerSeq - 1, hash);+    }++    // We check to see if we just finished the first call to this method in this+    // object's work (true when this checkpoint-end landed on the+    // highest-numbered ledger in the range)     if (curr.header.ledgerSeq == mRange.last())     {-        assert(nextCheckpointFirstLedger.first == 0);-        nextCheckpointFirstLedger = mTrustedEndLedger;+        // If so, there should be no "saved" incoming hash-link value from+        // a previous iteration.+        assert(incoming.first == 0);

Sure, fixed.

graydon

comment created time in a month

fork graydon/project-m36

Project: M36 Relational Algebra Engine

fork in a month

startedagentm/project-m36

started time in a month

push eventgraydon/stellar-core

Terence Rokop

commit sha 0eb4454ea85e9badebac4a872cac4754f315b29e

Add a small test-code-injection interface Add a mechanism, using the Factory pattern, for production code to create injection points that can call back into test-defined code (or perhaps other applications in the future; parameters used by the injected code can vary with each test or other use, including, in effect, the method called, by the use of lambdas). A test client can create one derived class of Database for each injection point or combination of injection points (into methods of class Database -- other classes could also use this mechanism for injecting code into their methods) to override with methods that do something. The derived class's constructor may take any list of parameter types; each call to createTestApplication() (or Application::create()) passes its own list of parameters of those types. The derived class may store the parameters received by its constructor, so that it is effectively called with parameters even though the production code calls a method with no parameters. This means in particular that TestApplication itself does not need to acquire new members to allow tests to store parameters to injection point methods. A single derived class of Database may be used by multiple test cases even to call, in effect, different injected methods, because the parameters passed by createTestApplication() to the constructor of the derived class of TestApplication may include lambdas. This checkin also adds one use of the new mechanism, actBeforeDBSchemaUpgrade(), by production code to create an injection point.

view details

Terence Rokop

commit sha 0af8465f95ff67668b6885225dc1987c97a2f88d

Utilities for decoding base-64-encoded opaque XDR

view details

Terence Rokop

commit sha 244cd1840964f7f9f90c93c14513069e08801306

Maps between vectors and SQL records Introduce three generic functions for operating on database records in bulk: - selectMap() takes a function which maps a SQL record to an element of a client-defined (templated) type, and produces a function which takes a SQL SELECT statement and produces a vector of the client-defined type comprising the list of selected records mapped through the client-defined function. - updateMap() takes a function which maps an element of a client-defined (templated) type to a SQL UPDATE statement, and produces a function which generates a series of SQL UPDATE statements from a vector of the client-defined type. - selectUpdateMap() produces the composition of updateMap() following selectMap(), namely, a function which takes a SQL SELECT statement and produces for each selected record an UPDATE statement derived from a transformation applied to the record via client-defined functions which interpret the record as an element of a client-defined type and then use that interpretation to choose some update to perform on it.

view details

Terence Rokop

commit sha 1c4510b8c0acc11df0a13fdffb7374ddfa4af100

DB schema upgrade to make extensions opaque Upgrade the DB schema from 12 to 13. The new schema makes all of the extensions in all ledger entry types opaque to the database code itself. Those extensions are the extension to LedgerEntry itself, which is stored in the table for each LedgerEntry type (account, trustline, data, offer), and the extensions which are specific to each type. The idea of making the extensions opaque is that they can be extended in the future without further changes to either the DB schema or the DB code, as the database itself will simply continue to treat the extensions as variable-length, opaque XDR. The cost is that we can't query on components of the extensions (which we don't do currently, and won't do with any of the extensions that we currently have planned). The general LedgerEntry extension, as well as the extensions to DataEntry and OfferEntry, are currently unused -- they are unions with the only version being 0 and no content besides that fixed version number, and they are not stored in any database columns at all. The upgrade code therefore simply creates columns for them and populates all of the cells of the new columns with base-64 encodings of opaque XDR representing that trivial tagged union. The two extensions that are already used are those specific to AccountEntry and TrustLineEntry, both of which contain the same thing, namely a Liabilities structure (which contains two fields, one for buying and one for selling). Those extensions are tagged unions with a genuine version option -- version 0 for entries that have never yet had liabilities, and then version 1 when liabilities are first created for the entry. Existing databases have NULL in the "buyingliabilities" and "sellingliabilities" columns for the version-0 entries, and non-NULL values, namely base-64 encodings of the XDR for the Liabilities structure, for the version-1 entries. Therefore, the upgrade code for AccountEntry and TrustLineEntry performs a conversion: for each record, it reads the "buyingliabilities" and "sellingliabilities" columns, which are 64-bit integers transparent to the database, and generates encoded XDR, opaque to the database, representing an extension field which, in those cases, has a Liabilities field, and has the values read from the old columns filled in. (If the old liabilities fields for a record are NULL, then the new "extension" column for that record is also NULL -- the new live database code will recognize that NULL values of those two extensions are possible.) Once all of the records have had their new "extension" columns written, the upgrade code deletes the old "buyingliabilities" and "sellingliabilities" columns.

view details

Terence Rokop

commit sha 2957ab216f64c115b69b4c7fb8578477f57d594d

Live DB code for schema 13 with opaque extensions Modify the live database code for all of the LedgerEntry types -- AccountEntry, TrustLineEntry, DataEntry, and OfferEntry -- to use database schema 13, in which the extensions for each of those individual entry types, as well as the extension for LedgerEntry itself which is also stored in the table for whatever specific type a given LedgerEntry has, are just base-64-encoded opaque XDR as far as the database is concerned, regardless of how the ledger-transaction code inteprets those extensions, now or in the future.

view details

Terence Rokop

commit sha 5801f31f822d74c9cd2c0af623e3ed52c85d5cea

Test upgrades to database schema 13 Test upgrades to database schema 13, in which all extensions in all ledger entry types become opaque XDR as far as the database is concerned. The tests use the test-code-injection mechanism which allow them to run arbitrary code between the creation of a database (which is done in an old schema version, MIN_SCHEMA_VERSION) and the upgrading of that database to the current schema version (SCHEMA_VERSION). They create ledger entries of each of the four types, using raw SQL so that they can create them in the old format which the live production code no longer uses. After the database upgrade completes (which it has done by the time that createTestApplication() returns), the tests validate that the contents of the ledger entries are intact and logically unchanged (although their representations in the database have changed, with some columns having been created and some having been deleted, and some contents being encoded differently).

view details

Siddharth Suresh

commit sha f891619cb1302a4dd00e41e08685c8a24b600a9d

run LedgerTxnTests with postgres

view details

Latobarita

commit sha ed3212fdc27e153ed34e6749d6bb95875913702c

Merge pull request #2603 from sisuresh/postgres_for_ledger_txn_tests Run LedgerTxnTests with Postgres Reviewed-by: jonjove

view details

Hidenori

commit sha d9303f8a2b2d1777f1b1602a37839a60fd145668

Introduce a constant for the maximum nesting level for quorums

view details

Hidenori

commit sha b23ca5eea3e6747d08bf63526317ebf949686493

Add tests to ensure that the constant is used in both places and the meaning of the level is consistent across the codebase.

view details

Latobarita

commit sha 8dc8718db0b2fbffe24321f1b193a0301a9a79fe

Merge pull request #2602 from hidenori-shinohara/quourm-level Introduce a constant for the maximum nesting level for quorums Reviewed-by: graydon

view details

Latobarita

commit sha 7ab017b888b9f138070307bb1785eb433ddcd5d1

Merge pull request #2593 from rokopt/issue-2570-pr-4 Issue #2570: DB schema change to make extensions opaque Reviewed-by: MonsieurNicolas

view details

Siddharth Suresh

commit sha 080a2454475796f2bb45e176ae6d1bcd8daeb008

Add option to use a close time offset for max time bound validity check

view details

Siddharth Suresh

commit sha e18ca355fd2242139fc4c623913edb42f9b609dd

Use upper bound estimate for tx validity pre-consensus

view details

Latobarita

commit sha bf8486bd99bc8d19e569b765d3244e523d5c3e18

Merge pull request #2608 from sisuresh/tx_flooding Improve precision of transaction expiration while flooding Reviewed-by: MonsieurNicolas

view details

Graydon Hoare

commit sha 96be911f9e328f42d69dc0e78d89ce0b9488ffe0

Add "verify-hit benchmarking" test to CryptoTests.cpp

view details

Graydon Hoare

commit sha 20b490c300f08a0dff448bd0d8d0e176054fa115

Make SHA256 class stack-allocated, remove shared global gHasher.

view details

push time in a month

push eventgraydon/stellar-core

Graydon Hoare

commit sha ee4b90bc6962a95cac759ad3bbb00bbf1576b332

Add "verify-hit benchmarking" test to CryptoTests.cpp

view details

Graydon Hoare

commit sha 021177c52961d9265817b2822e16241353272ef3

Make SHA256 class stack-allocated, remove shared global gHasher.

view details

push time in a month

pull request commentstellar/stellar-core

Guard access to gHasher with a mutex.

Ok here's a better benchmark that's composed entirely of verify-hits with a before/after comparison with "remove the entire shared global and use a stack local", removing the indirection to a private class and not worrying about a little bleed-through of a small set of definitions from sodium headers (I checked and it's quite small):

shared global:

 Performance counter stats for './src/stellar-core test verify-hit benchmarking' (30 runs):

       2700.300198      task-clock (msec)         #    0.923 CPUs utilized            ( +-  1.21% )
               467      context-switches          #    0.173 K/sec                    ( +-  3.02% )
                 2      cpu-migrations            #    0.001 K/sec                    ( +- 12.80% )
             2,183      page-faults               #    0.808 K/sec                    ( +-  0.02% )
     7,844,273,314      cycles                    #    2.905 GHz                      ( +-  0.06% )
       567,236,379      stalled-cycles-frontend   #    7.23% frontend cycles idle     ( +-  0.96% )
    23,785,821,606      instructions              #    3.03  insn per cycle         
                                                  #    0.02  stalled cycles per insn  ( +-  0.00% )
       418,395,890      branches                  #  154.944 M/sec                    ( +-  0.02% )
           698,821      branch-misses             #    0.17% of all branches          ( +- 14.44% )

       2.925243781 seconds time elapsed                                          ( +-  1.13% )

stack local:

 Performance counter stats for './src/stellar-core test verify-hit benchmarking' (30 runs):

       2283.762593      task-clock (msec)         #    0.909 CPUs utilized            ( +-  0.22% )
               409      context-switches          #    0.179 K/sec                    ( +-  3.97% )
                 2      cpu-migrations            #    0.001 K/sec                    ( +- 12.97% )
             2,182      page-faults               #    0.956 K/sec                    ( +-  0.02% )
     7,788,646,217      cycles                    #    3.410 GHz                      ( +-  0.08% )
       556,192,231      stalled-cycles-frontend   #    7.14% frontend cycles idle     ( +-  1.20% )
    23,704,973,281      instructions              #    3.04  insn per cycle         
                                                  #    0.02  stalled cycles per insn  ( +-  0.01% )
       400,531,120      branches                  #  175.382 M/sec                    ( +-  0.05% )
           636,040      branch-misses             #    0.16% of all branches          ( +- 13.01% )

       2.511696622 seconds time elapsed                                          ( +-  0.23% )

In other words it's a perf win to do so, and it's simpler. Everyone should be happy!

graydon

comment created time in a month

startedspacejam/paxos

started time in a month

starteddistil/diffus

started time in a month

startedrossberg/hamlet

started time in a month

startedStanfordLegion/legion

started time in a month

issue openedgraydon/sixbit

Add crate feature to switch latin treatments

The choice of latin ranges has a not-necessarily-desirable design preference wired into it: you can't do mixed-case latin strings, so that you can do at least one sort of alphanumeric-with-symbols latin strings. This might not be the right choice.

In many contexts (eg. programming language tokenizers) the identifier range is [a-zA-Z0-9_] which happens to fit precisely in 63 codes, plus the 0 code that's reserved: this would make a decent alternative choice for the latin range.

(Apparently crate features are only supposed to be additive, so this has to be done by adding a separate type and making all the code polymorphic over the two types. Whatever.)

created time in a month

fork graydon/tarpc

An RPC framework for Rust with a focus on ease of use.

fork in a month

startedgoogle/tarpc

started time in a month

startedcucapra/turnt

started time in a month

startedapprovals/ApprovalTests.Net

started time in a month

startedapprovals/ApprovalTests.cpp

started time in a month

pull request commentstellar/stellar-core

Guard access to gHasher with a mutex.

Well, here's perf stat -r 3 stellar-core test 'sign and verify benchmarking':

with SHA1 cache:                                                                                                                                                                
                                                                                                                                                                                
Performance counter stats for './src/stellar-core test sign and verify benchmarking' (3 runs):                                                                                  
                                                                                                                                                                                
      14099.986837      task-clock (msec)         #    0.987 CPUs utilized            ( +-  2.52% )                                                                             
             2,145      context-switches          #    0.152 K/sec                    ( +-  2.39% )                                                                             
                 9      cpu-migrations            #    0.001 K/sec                    ( +-  6.42% )                                                                             
            63,424      page-faults               #    0.004 M/sec                    ( +-  0.22% )                                                                             
    47,165,524,808      cycles                    #    3.345 GHz                      ( +-  0.11% )                                                                             
     8,123,588,237      stalled-cycles-frontend   #   17.22% frontend cycles idle     ( +-  0.98% )                                                                             
   115,641,971,821      instructions              #    2.45  insn per cycle                                                                                                     
                                                  #    0.07  stalled cycles per insn  ( +-  0.00% )                                                                             
     1,458,902,624      branches                  #  103.468 M/sec                    ( +-  0.00% )                                                                             
        39,142,008      branch-misses             #    2.68% of all branches          ( +-  0.12% )                                                                             
                                                                                                                                                                                
      14.287589810 seconds time elapsed                                          ( +-  2.49% )                                                                                  
                                                                                                                                                                                
without SHA1 cache:                                                                                                                                                             
                                                                                                                                                                                
 Performance counter stats for './src/stellar-core test sign and verify benchmarking' (3 runs):                                                                                 
                                                                                                                                                                                
      14525.457126      task-clock (msec)         #    0.987 CPUs utilized            ( +-  3.05% )                                                                             
             2,225      context-switches          #    0.153 K/sec                    ( +-  3.13% )                                                                             
                12      cpu-migrations            #    0.001 K/sec                    ( +- 20.00% )                                                                             
            60,371      page-faults               #    0.004 M/sec                    ( +-  2.83% )                                                                             
    47,114,685,352      cycles                    #    3.244 GHz                      ( +-  0.20% )                                                                             
     8,028,694,902      stalled-cycles-frontend   #   17.04% frontend cycles idle     ( +-  1.01% )                                                                             
   115,651,189,437      instructions              #    2.45  insn per cycle                                                                                                     
                                                  #    0.07  stalled cycles per insn  ( +-  0.00% )                                                                             
     1,461,180,220      branches                  #  100.594 M/sec                    ( +-  0.02% )                                                                             
        39,244,894      branch-misses             #    2.69% of all branches          ( +-  0.12% )                                                                             
                                                                                                                                                                                
      14.720780018 seconds time elapsed                                          ( +-  3.00% )                                                                                  

Unfortunately that's a benchmark that's 100% misses so it's going to bury the difference of the cache in the cost of the verifies. I will work up a better test.

graydon

comment created time in a month

push eventgraydon/concorde

Graydon Hoare

commit sha f14e1b558373749302d9e15705197af33a4ab7d6

Update to new stateright and pergola, bump to 0.5.0

view details

push time in a month

push eventgraydon/pergola

Graydon Hoare

commit sha 73fb1aacd515744e8eada89654c4d4067b49436e

bump to 0.8 and fix tuple composition

view details

Graydon Hoare

commit sha c2c10904e3b63eb17feaa261ba19b23dbcca06f8

update README.md

view details

push time in a month

fork graydon/tinyset

Compact sets in rust

fork in a month

starteddroundy/tinyset

started time in a month

pull request commentstellar/stellar-core

Verify ledger chain command

Pushed some fixes, tests, and integration on the catchup side. Feature is roughly complete now.

graydon

comment created time in a month

push eventgraydon/stellar-core

Graydon Hoare

commit sha 8e069336093e47da5ef3195a0175fb237e945b12

Support loading trusted checkpoint hashes from file in "catchup" command.

view details

push time in a month

PR opened stellar/stellar-core

Guard access to gHasher with a mutex.

Resolves #2604 -- adds a mutex to protect access to gHasher. This wasn't accessed by multiple threads as far as I can tell already but of course it should be protected the same as the other static globals in this file.

+2 -0

0 comment

1 changed file

pr created time in a month

more