The reverse submarine swap - using an untrusted server to exchange lightning btc for chain btc - allows new nodes to bootstrap themselves into being useful for routing and useful for receiving. The mechanism of the swap is simple, a on chain buyer makes a request then an on chain address is funded with a script like

HASH160 <hash of preimage> EQUAL
IF <chain buyer pubkey>
ELSE <timeout> CHECKLOCKTIMEVERIFY
DROP <chain seller pubkey> ENDIF
CHECKSIG 

If the timeout has not been reached then the only way the chain coins can be claimed is using the preimage. The preimage is the secret lock of a lightning invoice and must be revealed in order to receive the lightning payment, thus an exchange can happen with no trust between the participants.

The problem with this is the CHECKLOCKTIMEVERIFY creates a confirmation urgency. In boltz-backend implementation even in a low fee moment the lockup fee rate was 95sat/byte+. Paying this level of fees to balance digs a very large hole that routing fees will take years to make up. Seems like lightning loop uses the same transaction dynamic, subject to the same confirmation urgency.

Naively, a script like

HASH160 <hash of preimage> EQUAL

would allow a server to put outputs up for sale indefinitely? But the server has the preimage as well so it could play dirty games like self sending the output with a large fee once the invoice is paid. Think it is important to be able to do reverse submarine swaps with low fees. Wondering if an asynchronous, counterparty flexible version of reverse sub-swaps is possible? The fees paid by boltz backend are too damn high!

Hey this is cool and btw great work on budgeting the balancing. Pulled and started running this branch to see if I can get a swap, but it started to swap with the original boltz server, so I made a small modification to make my boltz server the only choice. Left it run overnight, did not made contact.

Weird mypy error from Travis. Trivial rebase...

Yep, seems mypy==0.800 broke some things, I pinned it to 0.790 in another cleanup

Ooh:

2021-01-17T17:12:07.684Z DEBUG   024b9a1fa8e006f1e3937f65f66c408e6da8e1ca728ea43222a7381df1cc449605-connectd: peer_out WIRE_INIT

--- snipped ---

2021-01-17T17:12:09.129Z DEBUG   024b9a1fa8e006f1e3937f65f66c408e6da8e1ca728ea43222a7381df1cc449605-connectd: peer_in WIRE_INIT

So i receive your INIT 1 second and a half after sending mine to you. It must have been on different interfaces, and lightningd doesn't know what to do.

Here are the logs for your id previously to the crash:

# grep -B10000 '2021-01-17T17:12:09.139Z' ~lightningd/.lightning/debug.log |grep 024b9a1fa8e006f1e3937f65f66c408e6da8e1ca728ea43222a7381df1cc449605

--- REDACTED: normal channel operations, HTLCs addition, commitment revocation, all good --

2021-01-17T15:47:52.196Z DEBUG   024b9a1fa8e006f1e3937f65f66c408e6da8e1ca728ea43222a7381df1cc449605-channeld-chan#5497: peer_out WIRE_REVOKE_AND_ACK
2021-01-17T15:52:02.566Z DEBUG   024b9a1fa8e006f1e3937f65f66c408e6da8e1ca728ea43222a7381df1cc449605-channeld-chan#5497: Trying commit
2021-01-17T15:52:02.566Z DEBUG   024b9a1fa8e006f1e3937f65f66c408e6da8e1ca728ea43222a7381df1cc449605-channeld-chan#5497: Can't send commit: nothing to send
2021-01-17T16:01:33.244Z DEBUG   024b9a1fa8e006f1e3937f65f66c408e6da8e1ca728ea43222a7381df1cc449605-channeld-chan#5497: Trying commit
2021-01-17T16:01:33.244Z DEBUG   024b9a1fa8e006f1e3937f65f66c408e6da8e1ca728ea43222a7381df1cc449605-channeld-chan#5497: Can't send commit: nothing to send
2021-01-17T16:43:56.829Z DEBUG   024b9a1fa8e006f1e3937f65f66c408e6da8e1ca728ea43222a7381df1cc449605-channeld-chan#5497: Failed reading header: No route to host
2021-01-17T16:43:56.830Z INFO    024b9a1fa8e006f1e3937f65f66c408e6da8e1ca728ea43222a7381df1cc449605-channeld-chan#5497: Peer connection lost
2021-01-17T16:43:56.830Z DEBUG   024b9a1fa8e006f1e3937f65f66c408e6da8e1ca728ea43222a7381df1cc449605-channeld-chan#5497: Status closed, but not exited. Killing
2021-01-17T16:43:56.831Z INFO    024b9a1fa8e006f1e3937f65f66c408e6da8e1ca728ea43222a7381df1cc449605-chan#5497: Peer transient failure in CHANNELD_NORMAL: channeld: Owning subdaemon channeld died (9)
2021-01-17T16:43:56.831Z DEBUG   024b9a1fa8e006f1e3937f65f66c408e6da8e1ca728ea43222a7381df1cc449605-chan#5497: Will try reconnect in 1 seconds
2021-01-17T16:48:06.664Z DEBUG   024b9a1fa8e006f1e3937f65f66c408e6da8e1ca728ea43222a7381df1cc449605-chan#5497: Will try reconnect in 2 seconds
2021-01-17T16:48:06.664Z DEBUG   024b9a1fa8e006f1e3937f65f66c408e6da8e1ca728ea43222a7381df1cc449605-connectd: Failed connected out: [2400:6180:0:d0::5cd2:a001]:9735: Connection establishment: No route to host. 128.199.202.168:9735: Connection establishment: No route to host. [2400:6180:0:d0::5cd2:a001]:9735: Connection establishment: No route to host. Error connecting to ki5uack5xb4gicfnpfavzdibojfu3qvwtlg3ca7eyjxik7tgwfxummqd.onion: Tor server reply: TTL expired. ki5uack5xb4gicfnpfavzdibojfu3qvwtlg3ca7eyjxik7tgwfxummqd.onion:9735: Connection establishment: Connection refused. Error connecting to ytcdmglaxryimsz3z53fggzzqujblmu4ye4luz56ax4kb6dalbss5fqd.onion: Tor server reply: TTL expired. ytcdmglaxryimsz3z53fggzzqujblmu4ye4luz56ax4kb6dalbss5fqd.onion:9735: Connection establishment: Connection refused. 
2021-01-17T16:52:18.177Z DEBUG   024b9a1fa8e006f1e3937f65f66c408e6da8e1ca728ea43222a7381df1cc449605-chan#5497: Will try reconnect in 4 seconds
2021-01-17T16:52:18.177Z DEBUG   024b9a1fa8e006f1e3937f65f66c408e6da8e1ca728ea43222a7381df1cc449605-connectd: Failed connected out: [2400:6180:0:d0::5cd2:a001]:9735: Connection establishment: No route to host. 128.199.202.168:9735: Connection establishment: No route to host. [2400:6180:0:d0::5cd2:a001]:9735: Connection establishment: No route to host. Error connecting to ki5uack5xb4gicfnpfavzdibojfu3qvwtlg3ca7eyjxik7tgwfxummqd.onion: Tor server reply: TTL expired. ki5uack5xb4gicfnpfavzdibojfu3qvwtlg3ca7eyjxik7tgwfxummqd.onion:9735: Connection establishment: Connection refused. Error connecting to ytcdmglaxryimsz3z53fggzzqujblmu4ye4luz56ax4kb6dalbss5fqd.onion: Tor server reply: TTL expired. ytcdmglaxryimsz3z53fggzzqujblmu4ye4luz56ax4kb6dalbss5fqd.onion:9735: Connection establishment: Connection refused. 
2021-01-17T16:56:30.083Z DEBUG   024b9a1fa8e006f1e3937f65f66c408e6da8e1ca728ea43222a7381df1cc449605-chan#5497: Will try reconnect in 8 seconds
2021-01-17T16:56:30.084Z DEBUG   024b9a1fa8e006f1e3937f65f66c408e6da8e1ca728ea43222a7381df1cc449605-connectd: Failed connected out: [2400:6180:0:d0::5cd2:a001]:9735: Connection establishment: No route to host. 128.199.202.168:9735: Connection establishment: No route to host. [2400:6180:0:d0::5cd2:a001]:9735: Connection establishment: No route to host. Error connecting to ki5uack5xb4gicfnpfavzdibojfu3qvwtlg3ca7eyjxik7tgwfxummqd.onion: Tor server reply: TTL expired. ki5uack5xb4gicfnpfavzdibojfu3qvwtlg3ca7eyjxik7tgwfxummqd.onion:9735: Connection establishment: Connection refused. Error connecting to ytcdmglaxryimsz3z53fggzzqujblmu4ye4luz56ax4kb6dalbss5fqd.onion: Tor server reply: TTL expired. ytcdmglaxryimsz3z53fggzzqujblmu4ye4luz56ax4kb6dalbss5fqd.onion:9735: Connection establishment: Connection refused. 
2021-01-17T17:00:39.066Z DEBUG   024b9a1fa8e006f1e3937f65f66c408e6da8e1ca728ea43222a7381df1cc449605-chan#5497: Will try reconnect in 16 seconds
2021-01-17T17:00:39.067Z DEBUG   024b9a1fa8e006f1e3937f65f66c408e6da8e1ca728ea43222a7381df1cc449605-connectd: Failed connected out: [2400:6180:0:d0::5cd2:a001]:9735: Connection establishment: Network is unreachable. 128.199.202.168:9735: Connection establishment: No route to host. [2400:6180:0:d0::5cd2:a001]:9735: Connection establishment: Network is unreachable. Error connecting to ki5uack5xb4gicfnpfavzdibojfu3qvwtlg3ca7eyjxik7tgwfxummqd.onion: Tor server reply: TTL expired. ki5uack5xb4gicfnpfavzdibojfu3qvwtlg3ca7eyjxik7tgwfxummqd.onion:9735: Connection establishment: Connection refused. Error connecting to ytcdmglaxryimsz3z53fggzzqujblmu4ye4luz56ax4kb6dalbss5fqd.onion: Tor server reply: TTL expired. ytcdmglaxryimsz3z53fggzzqujblmu4ye4luz56ax4kb6dalbss5fqd.onion:9735: Connection establishment: Connection refused. 
2021-01-17T17:04:57.419Z DEBUG   024b9a1fa8e006f1e3937f65f66c408e6da8e1ca728ea43222a7381df1cc449605-chan#5497: Will try reconnect in 32 seconds
2021-01-17T17:04:57.419Z DEBUG   024b9a1fa8e006f1e3937f65f66c408e6da8e1ca728ea43222a7381df1cc449605-connectd: Failed connected out: [2400:6180:0:d0::5cd2:a001]:9735: Connection establishment: Network is unreachable. 128.199.202.168:9735: Connection establishment: No route to host. [2400:6180:0:d0::5cd2:a001]:9735: Connection establishment: Network is unreachable. Error connecting to ki5uack5xb4gicfnpfavzdibojfu3qvwtlg3ca7eyjxik7tgwfxummqd.onion: Tor server reply: TTL expired. ki5uack5xb4gicfnpfavzdibojfu3qvwtlg3ca7eyjxik7tgwfxummqd.onion:9735: Connection establishment: Connection refused. Error connecting to ytcdmglaxryimsz3z53fggzzqujblmu4ye4luz56ax4kb6dalbss5fqd.onion: Tor server reply: TTL expired. ytcdmglaxryimsz3z53fggzzqujblmu4ye4luz56ax4kb6dalbss5fqd.onion:9735: Connection establishment: Connection refused. 
2021-01-17T17:09:30.748Z DEBUG   024b9a1fa8e006f1e3937f65f66c408e6da8e1ca728ea43222a7381df1cc449605-chan#5497: Will try reconnect in 64 seconds
2021-01-17T17:09:30.749Z DEBUG   024b9a1fa8e006f1e3937f65f66c408e6da8e1ca728ea43222a7381df1cc449605-connectd: Failed connected out: [2400:6180:0:d0::5cd2:a001]:9735: Connection establishment: Network is unreachable. 128.199.202.168:9735: Connection establishment: No route to host. [2400:6180:0:d0::5cd2:a001]:9735: Connection establishment: Network is unreachable. Error connecting to ki5uack5xb4gicfnpfavzdibojfu3qvwtlg3ca7eyjxik7tgwfxummqd.onion: Tor server reply: TTL expired. ki5uack5xb4gicfnpfavzdibojfu3qvwtlg3ca7eyjxik7tgwfxummqd.onion:9735: Connection establishment: Connection refused. Error connecting to ytcdmglaxryimsz3z53fggzzqujblmu4ye4luz56ax4kb6dalbss5fqd.onion: Tor server reply: TTL expired. ytcdmglaxryimsz3z53fggzzqujblmu4ye4luz56ax4kb6dalbss5fqd.onion:9735: Connection establishment: Connection refused. 
2021-01-17T17:12:06.733Z DEBUG   024b9a1fa8e006f1e3937f65f66c408e6da8e1ca728ea43222a7381df1cc449605-connectd: Connected out, starting crypto
2021-01-17T17:12:07.684Z DEBUG   024b9a1fa8e006f1e3937f65f66c408e6da8e1ca728ea43222a7381df1cc449605-connectd: Connect OUT
2021-01-17T17:12:07.684Z DEBUG   024b9a1fa8e006f1e3937f65f66c408e6da8e1ca728ea43222a7381df1cc449605-connectd: peer_out WIRE_INIT
2021-01-17T17:12:09.094Z DEBUG   024b9a1fa8e006f1e3937f65f66c408e6da8e1ca728ea43222a7381df1cc449605-gossipd: seeker: disabling gossip
2021-01-17T17:12:09.129Z DEBUG   024b9a1fa8e006f1e3937f65f66c408e6da8e1ca728ea43222a7381df1cc449605-connectd: peer_in WIRE_INIT
2021-01-17T17:12:09.130Z DEBUG   024b9a1fa8e006f1e3937f65f66c408e6da8e1ca728ea43222a7381df1cc449605-chan#5497: Peer has reconnected, state CHANNELD_NORMAL
2021-01-17T17:12:09.135Z DEBUG   024b9a1fa8e006f1e3937f65f66c408e6da8e1ca728ea43222a7381df1cc449605-channeld-chan#5497: pid 278110, msgfd 67
2021-01-17T17:12:09.135Z DEBUG   024b9a1fa8e006f1e3937f65f66c408e6da8e1ca728ea43222a7381df1cc449605-chan#5497: Already have funding locked in (and ready to announce)

Interestingly you seem to disconnect for a small-ish time, reconnect, and then we crash. Could it have been the very hour at which you upgraded your node to 0.9.3rc1 (time in the logs is EST) ?

I see no clue in the logs that would indicate my node tried to reach you on both clearnet and Tor. Could it be a race condition that you reconnected at the very same time my timer tried a connection to you ?

Do you have logs for the previous minute? This log is very strange:

2021-01-17T17:12:09.129Z DEBUG   024b9a1fa8e006f1e3937f65f66c408e6da8e1ca728ea43222a7381df1cc449605-connectd: peer_in WIRE_INIT
2021-01-17T17:12:09.129Z **BROKEN** connectd: FATAL SIGNAL 11 (version v0.9.2)
[...]
2021-01-17T17:12:09.137Z **BROKEN** connectd: backtrace: connectd/connectd.c:511 (peer_connected) 0x55a146d50f75

OK, so we crash at the bottom of peer_connected. Except, we see this, too:

2021-01-17T17:12:09.130Z DEBUG   024b9a1fa8e006f1e3937f65f66c408e6da8e1ca728ea43222a7381df1cc449605-chan#5497: Peer has reconnected, state CHANNELD_NORMAL
2021-01-17T17:12:09.135Z DEBUG   024b9a1fa8e006f1e3937f65f66c408e6da8e1ca728ea43222a7381df1cc449605-channeld-chan#5497: pid 278110, msgfd 67
2021-01-17T17:12:09.135Z DEBUG   024b9a1fa8e006f1e3937f65f66c408e6da8e1ca728ea43222a7381df1cc449605-chan#5497: Already have funding locked in (and ready to announce)

Which cannot happen until connectd tells lightningd about the new connection, except connectd died before re-entering the io loop!

Do we somehow have two connections at once? We're not supposed to, and the logic there seems sound. This is a Tor connection, so obviously we're using the proxy, which is a less tested path...

From #clightning:

 (11:24:10) andytoshi: hiya, i'm trying to set up a new lightningd node, and when i run lightningd --network=bitcoin --log-level=debug --daemon
 (11:24:17) andytoshi: i get errors of the form fetchinvoice: Malformed JSON reply '2021-01-25T00:51:16.655Z DEBUG   plugin-offers: disabled itself at init: offers not enabled in config
 (11:24:43) andytoshi: there are a couple variants of this, but always some form of "something: failed to parse <a log line> as json"

Indeed, we close stdout, and it ends up being reused for some plugin. But the real problem is that we log to stdout by default, which doesn't make sense. If they really want to discard logs, they can use --log-file=/dev/null.

Self-restarting is an interesting thought. One nice aspect is that the new process would retain the same PID as the old process, so system service managers (like OpenRC) wouldn't notice anything awry at any instant.

Yeah, I learned this the hard way, and I now shutdown before make install.

We could have subdaemons send their version string on startup, then restart lightningd (after a few seconds' sleep) if the version is wrong. If we restart and our own version hasn't changed, then simply fail. I think upgrading running instances is a better choice than trying to keep them going?

Weird mypy error from Travis. Trivial rebase...

Yep, this is great! We generally dislike underscores in JSON parameters, but the JSON already outputs those names so consistency wins.

Tests would ofc be nice, so would docs ;)

Few small fixes I found while re-doing the dual-funding code for RBF

Changelog-None