ACK fa5f46600fb98f1b35346bedc1a66c9019d01114: patch looks correct

Thanks for fixing flaky tests :)

ACK 1a520fc8dbe32b9f9fd4b38c4b70660a64e26e05: why not :)

ACK 1d5ac6062603e8ed494c8edacbdd314377e83627

Concept ACK

Thanks for improving testing!

Concept ACK: -Wswitch is good, and termination via assert(false); is better than UB :)

From the standard: "Flowing off the end of a function is equivalent to a return with no value; this results in undefined behavior in a value-returning function." (C++11, [stmt.return])

I think adding asserts should be making it clearer for people to see why the code is safe as well as automated tools.

I agree about that statement :)

In addition to that use I also think assertions can be useful to get deterministic termination in case where we otherwise would get a null pointer dereference (if our assumption of non-null turns out to be false).

Why is that desired?

One might think that a null pointer dereference is guaranteed to terminate execution. Unfortunately that is not the case: deterministic termination is the best outcome. Another possible outcome is what in the literature is called "optimization-unsafety" which can be problematic from a security perspective.

There is a neat paper covering optimization-safe code called "Towards Optimization-Safe Systems: Analyzing the Impact of Undefined Behavior" which I recommend for context.

Personally I think adding assertions can be a cheap way to get avoid some instances of potentially "optimization-unsafe code". That is my personal opinion: I know others might not agree and that is fine of course :)

(If you are getting this from an automated tool, are there reports from it around? I remember there were previously somewhere)

IIRC Coverity and Clang's Cross Translation Unit (CTU) static analysis flag these cases. I think most static analysers doing pointer analysis would flags some of these since it is non-trivall for them to infer non-nullptr in these cases AFAICT. I wouldn't be surprised if PVS Studio and cppcheck flags these too as an example.

I don't have any reports readily available, but feel free to check the instructions in #18920 for instructions on how to get Bitcoin Core reports from Clang Static Analysis, clang-tidy and cppcheck. It only takes a couple of minutes of setup :)

Hope that helps! :)

ACK a4b1588c6555b164fdd5e41164bf9358a5194660: patch looks correct :)

Concept ACK: nice cleanup!

Concept ACK

Concept ACK

I could refactor ATEC but that doesn't mean people will be eager to review it, …

FWIW I would be very eager to review a change which made AttemptToEvictConnection unit testable. As demonstrated it can be made unit testable with only few lines of code.

I've always been somewhat amazed about how little testing AttemptToEvictConnection has considering the importance of that function for the network. See #16660 (August 2019) and #19966 (September 2020) for context.

@jonatack

unlikely to be merged (unless someone of v high stature drives it)

I find it very concerning that you're worried that you're not "high stature" enough to suggest such a change.

Either it is in the best interest of our users to have the code merged, or it is not.

Who wrote the code simply doesn't enter that equation.

Concept ACK, but please add a AttemptToEvictConnection unit test for this scenario.

See #19966 on ideas on how to make AttemptToEvictConnection testable, or feel free to cherry-pick in 818dd2a04da1ae7329c58a79a4fade5ec62c549d and df1328fcecc3f359574fa53979402894b4a73a12 from #19972 ("tests/net: Add fuzzing harness for node eviction logic. Make node eviction logic testable.") .

Concept_ACK

Concept ACK: deduplication is good

Heh, thanks! Literally on the same line :) Sorry.

@ajtowns I'm leaving this one as up for grabs now that the first commit is included in #20138 :)

• Make Assert(…) (util/check.h) usable in all contexts.
• Use Assert(…) to make implicit assumptions explicit.

Before the first commit:

./chain.h:404:23: error: C++11 only allows consecutive left square brackets when introducing an attribute
        return (*this)[Assert(pindex)->nHeight] == pindex;
                      ^

Could someone restart Cirrus please? :)

Add fuzzing harness for CConnman.

See doc/fuzzing.md for information on how to fuzz Bitcoin Core. Don't forget to contribute any coverage increasing inputs you find to the Bitcoin Core fuzzing corpus repo.

Happy fuzzing :)

I see -Werror being removed here, but where is it re-added? :)

@naumenkogs

Oh, to be clear: I meant "peer triggerable" as the union of a.) "peer triggerable in low volume" (spammy but generally not a problem from a security perspective), b.) "peer triggerable in high volume" (may be used for disk fill attacks).

Note that the first case "peer triggerable in low volume" includes logging of events that is the result of "remote peer (mis)behaviour".

Generally I subscribe to this logging philosophy described by MarcoFalke in #19995: "I'd prefer if unconditional logs were only used for the init/shutdown sequence and local system errors, such as data corruption. Anything else the average user probably doesn't care about, and if they did, they could enable the corresponding debug category and provide enough disk space for the debug log file.", and ideally in combination with some kind of general mitigation like the one I suggested in #19995 ("Mitigate disk filling attacks by temporarily rate limiting LogPrintf(…)").

Which are (a)? Most seem (b) to me, and I'm not even sure they're that "uninteresting" (although the content could be improved to be user-friendly).

I think these are spammy/uninteresting:

src/flatfile.cpp:   LogPrintf("Pre-allocating up to position 0x%x in %s%05u.dat\n", new_size, m_prefix, pos.nFile);
src/validation.cpp: LogPrintf("Leaving block file %i: %s\n", nLastBlockFile, vinfoBlockFile[nLastBlockFile].ToString());

I think these are peer triggerable in the sense that they are the result of "remote peer (mis)behaviour" (and hopefully only in low volume):

src/net.cpp:        LogPrintf("socket send error %s\n", NetworkErrorString(nErr));
src/net.cpp:        LogPrintf("socket sending timeout: %is\n", nTime - pnode->nLastSend);
src/net.cpp:        LogPrintf("socket receive timeout: %is\n", nTime - pnode->nLastRecv);
src/net.cpp:        LogPrintf("ping timeout: %fs\n", 0.000001 * count_microseconds(GetTime<std::chrono::microseconds>() - pnode->m_ping_start.load()));

If someone comes up with a way in which these could be made peer triggerable in high volume then that should be discussed with security@ of course :)

(And if we had a mitigation in place we wouldn't have to worry much at all :))

ACK 76bbcc414f3001b16ac0101f738ed0b3e6f1a372: diff looks correct

@Crypt-iQ Sorry, I don't have any Mac to test on and I haven't seen this issue under Linux. I'm afraid I'll have to let someone else tackle this issue :)

As MarcoFalke noted in #19995:

I'd prefer if unconditional logs were only used for the init/shutdown sequence and local system errors, such as data corruption. Anything else the average user probably doesn't care about, and if they did, they could enable the corresponding debug category and provide enough disk space for the debug log file.

I agree that we use too much unconditional logging.

It is bad for two reasons:

• From a security perspective unnecessary unconditional logging risks open up for disk fill attacks.
• From a usability perspective unnecessary unconditional logging risks burying important log messages (anomalies) in logs about mundane expected stuff (non-anomalies).

I suggest gradually making non-critical unconditional logging conditional to reduce risk of disk fill attacks and to improve user experience.

More concretely that would mean changing from LogPrintf("…\n"); to LogPrint(DEBUG_CATEGORY, "…\n");.

Some candidates for changing from unconditional to conditional debug logging:

src/flatfile.cpp:   LogPrintf("Pre-allocating up to position 0x%x in %s%05u.dat\n", new_size, m_prefix, pos.nFile);
src/net.cpp:        LogPrintf("socket send error %s\n", NetworkErrorString(nErr));
src/net.cpp:        LogPrintf("socket sending timeout: %is\n", nTime - pnode->nLastSend);
src/net.cpp:        LogPrintf("socket receive timeout: %is\n", nTime - pnode->nLastRecv);
src/net.cpp:        LogPrintf("ping timeout: %fs\n", 0.000001 * count_microseconds(GetTime<std::chrono::microseconds>() - pnode->m_ping_start.load()));
src/validation.cpp: LogPrintf("Leaving block file %i: %s\n", nLastBlockFile, vinfoBlockFile[nLastBlockFile].ToString());

These are (AFAICT) either a.) peer triggerable or b.) uninteresting from a user perspective :)

Getting rid of the cases above as a start would significantly reduce the amount of log spam when in steady state (non-IBD).

ACK modulo CI failure :)

ACK 2418ec658ccd2e8e033bced0f5b7c183946940ac

If measurements show that this ever becomes a problem in practice it can be tackled in a follow-up.

TBH I'm much more worried about the possible file descriptor DoS vector that we would risk open up if the file descriptor were left open open.

Security first! :)

Make ComputeEntrySchnorr and ComputeEntryECDSA const to clarify contract.

re-ACK fa723e3d43e63e8424d97d21d8f2cc8136aba206: patch still looks correct

Add hyperlinks.

No text added :)

After some squeezing my description is now trimmed to less than two tweets of size, and roughly 2 % smaller than Chris' entry. Acceptable? :)

@dennisreimann I think Chris' entry is fine (567 bytes): it is roughly the length of two tweets (560 bytes). As a donor I appreciate two tweets of context to be able to curate a short list of donatees without having to click them all :)

I'll trim down my entry so that it significantly smaller than Chris' entry and we should hopefully be ready to go. Will push soonish! :)

re-ACK fa60f0dc039d2fa5c6d379f20dce964e43c52372: patch still looks correct

@dennisreimann Thanks a lot for your quick review! I've now tried hard to shorten it while still giving readers the relevant information. I managed to shorten it by removing two sentences. Is it acceptable now? :)

@MarcoFalke Thanks for your Concept ACK. Would you mind reviewing the code too? :)

Friendly review beg :)

Concept NACK:s are totally fine too: that would allow me to move forward with other fuzzing stuff without having to keep this PR updated :)

Although given it's commented out, it's just the difference of applying your own patches vs un-commenting the lines pre-build?

The reason it is commented out is that older versions of UBSan errors on suppressions it cannot parse. We'll need this in master uncommented eventually (when upgrading to Clang 12).

For me personally I already have the patch in my repo, but I'm hopefully not the only one building Bitcoin Core with the latest version of Clang (to get early notification of compilation issues/compiler bugs, to get the best libFuzzer and diagnostics goodies, etc.) we support so I thought it would be helpful to upstream it commented out for now :)

Removed my ACK (temporarily) in light of @MarcoFalke's comment about the unrelated change in src/net.cpp.

Hopefully that will be resolved soon and I'd be glad to re-review :)

Add practicalswift :)

re-ACK 9a9314aa454c88a577260406fed3cb391dc4e4a7: diff still looks correct

Looking forward to having this neat feature in master!

ACK ba8950ee0134a7958e3e9b041cd54d222feb09a1: diff looks correct!

Really looking forward to having this in master :)

Oh, thanks! Fix pushed.

Makes sense. I'll implement.

That's a good idea. I'll implement!

@MarcoFalke

Conceptually, I am not sure if it is good to fight the symptoms.

I fail to see how introducing a mitigation would be to "fight the symptoms".

The reason we use ASLR for example isn't that we've given up on the ambition to write vulnerability free code :)

The reason we're using mitigations is to make exploitation harder and hopefully turn a vulnerability into a normal bug.

Have you looked at the implementation in this PR? :)

The mitigation logic would only kick in in under extreme circumstances meaning that the only observable difference from a user perspective is that his/her node would be exploitable to disk fill attacks without the mitigation in place, whereas he/she would be safe with it.

Note that this is not a theoretical issue: we've had exploitable disk filling vulnerabilities in the past which this mitigation would have successfully turned into just another non-exploitable non-security bug.

I'd prefer if unconditional logs were only used for the init/shutdown sequence and local system errors, such as data corruption. Anything else the average user probably doesn't care about, and if they did, they could enable the corresponding debug category and provide enough disk space for the debug log file.

I agree that we should log more conservatively and ideally leave no room for disk filling attacks, but that doesn't preclude also protecting our users in case we fail to do so :)

Thanks for the ping @fanquake!

When doing large scale fuzzing with the goal of reaching as good coverage as possible I think we definitely need support for one-binary-per-target.

Don't take my word for it though: @kcc of C++ sanity fame who introduced the sanitizers, libFuzzer, OSS-Fuzz, CFI, etc. wrote this in https://github.com/bitcoin/bitcoin/issues/11045#issuecomment-334516409: "I always advocate for one-binary-per-target because it makes fuzzing more efficient." :)

With that said I think it could make sense to add a configure option which would build the fuzzing harnesses as one binary (with individual targets selected as a runtime argument) to speed up normal day-to-day compilation. That could hopefully be enabled by default to make sure the fuzzing harnesses still compiles are non-fuzzing code changes (currently that is often noticed only at CI stage).

Would that make sense?

ACK 3562c15be3dbf725197d719d58c8d78e2f2c6779: correct is better than incorrect :)

ACK fab93637867217266a810794d179487bb1eb8c69: patch looks correct

Concept ACK

ACK d1292f25f272401da0c58580521c74b1fa03a9ad

Concept ACK

While touching src/util/check.h, would you mind cherry-picking in 91bdd439987fb3f24f9b841fe88b3cf416b38f48 from #20122 to make Assert(…) usable in all contexts?

You're absolutely right: the conditional is redundant. Thanks for noticing! Redundancy removed :)

Please re-review! :)

Update UBSan suppressions file with suppressions needed for clang 12 (current trunk).

When extending the test/fuzz/integer fuzzer I noticed the following UBSan warning when fuzzing abs64(...):

runtime error: negation of -9223372036854775808 cannot be represented in type 'int64_t' (aka 'long'); cast to an unsigned type to negate this value to itself

Fuzzing harness:

diff --git a/src/test/fuzz/integer.cpp b/src/test/fuzz/integer.cpp
index 35d6804d4..bc158e5a2 100644
--- a/src/test/fuzz/integer.cpp
+++ b/src/test/fuzz/integer.cpp
@@ -40,6 +40,8 @@
 #include <set>
 #include <vector>

 void initialize()
 {
     SelectParams(CBaseChainParams::REGTEST);
@@ -82,6 +84,7 @@ void test_one_input(const std::vector<uint8_t>& buffer)
     (void)ComputeMerkleRoot(v256);
     (void)CountBits(u64);
     (void)DecompressAmount(u64);
+    (void)abs64(i64);
     (void)FormatISO8601Date(i64);
     (void)FormatISO8601DateTime(i64);
     // FormatMoney(i) not defined when i == std::numeric_limits<int64_t>::min()

Typically abs(I n) type functions are not defined when n == std::numeric_limits<I>::min() so it could be argued that this is expected, but perhaps the function could be rewritten in a way which guarantees that it gives the same behaviour across systems (instead of UB).

@theStack Now printing a log message. Please re-review :)

An example from mainnet where Bitcoin Core tries to connect to a fec0::/10 address learned from a mainnet peer.

The connections is blocked by Tor:

Oct 10 12:34:56.000 [warn] Rejecting SOCKS request for anonymous connection to private address fec0::5b4f:fa94:a34b:1731.