profile
viewpoint
Michael Schmoock m-schmoock software archaeologist

ElementsProject/lightning 1776

c-lightning — a Lightning Network implementation in C

m-schmoock/lcpp 121

A Lua C PreProcessor

lightningd/plugins 107

Community curated plugins for c-lightning

m-schmoock/ludis86 16

udis86 Lua bindings

m-schmoock/bitcoin 0

Bitcoin Core integration/staging tree

m-schmoock/bitcoin-wallet 0

Bitcoin Wallet app for your Android device. Standalone Bitcoin node, no centralized backend required.

m-schmoock/bitcoinfaith 0

bitcoin-faith source code

m-schmoock/bitcoinj 0

A library for working with Bitcoin

m-schmoock/bitcointax 0

Calculate taxes based on exchange CSVs and also pubkey data

m-schmoock/bitcoin_fork_claimer 0

Script for transferring/claiming your coins on various Bitcoin forks

Pull request review commentlightningd/plugins

backup: Add the ability to compact the backups

 def read_metadata(self):     def add_change(self, entry: Change) -> bool:         typ = b'\x01' if entry.snapshot is None else b'\x02'         if typ == b'\x01':-            payload = b'\x00'.join([t.encode('UTF-8') for t in entry.transaction])+            stmts = [t.encode('UTF-8') if isinstance(t, str) else t for t in entry.transaction]+            payload = b'\x00'.join(stmts)

Thanks—yeah no problem, these changes are very contained and easy to port over.

cdecker

comment created time in an hour

pull request commentlightningd/plugins

chore: renames RPC feeadjusttoggle to feeadjust-toggle

Also: do we want feeadjuster-toggle or feeadjust-toggle ? I tend to the first. Second one is only better if we rename the whole plugin, which I dont like.

I think the current feeadjuster-toggle is good in the code. Maybe the commit message could be more precise, but does not matter.

m-schmoock

comment created time in 6 hours

pull request commentlightningd/plugins

chore: renames RPC feeadjusttoggle to feeadjust-toggle

tACK fb05a96 Now I can call feeadjuster-toggle (with dash) from command line and feeadjuster_toggle (with underscore) from python RPC.

m-schmoock

comment created time in 6 hours

issue commentElementsProject/lightning

Multiple channels not supported

Thanks to @niftynei's efforts we have an experimental implementation of dual-funding (compile with EXPERIMENTAL_FEATURES=1). Once the specification is finalized we expect other implementations to implement the proposal as well.

peername

comment created time in 7 hours

PR opened ElementsProject/lightning

hsmtool: fix a segfault on `dumponchaindescriptors` default network parameter

fixes https://github.com/ElementsProject/lightning/issues/4339

+2 -2

0 comment

1 changed file

pr created time in 7 hours

pull request commentlightningd/plugins

rebalance: improve the feeadjust availability check

What happens if I call this?

print(callable(getattr(plugin.rpc, "this_function_does_not_exist")))
m-schmoock

comment created time in 7 hours

issue commentElementsProject/lightning

Change sqlite page_size to 4096 - massive performance improvement

Suggest separate thread, maybe link some of the relevant text in this thread. Some of the other SCAN TABLEs in the query plan might be worth pursuing but as per my analysis the expected speed increase would be significantly lower.

I could create a short document for tweaking parameters of your SQLITE based on the underlying filesystem characteristics.

mb300sd

comment created time in 7 hours

pull request commentlightningd/plugins

rebalance: improve the feeadjust availability check

Looks good to me, thanks! :)

m-schmoock

comment created time in 7 hours

Pull request review commentlightningd/plugins

backup: Add the ability to compact the backups

 def read_metadata(self):     def add_change(self, entry: Change) -> bool:         typ = b'\x01' if entry.snapshot is None else b'\x02'         if typ == b'\x01':-            payload = b'\x00'.join([t.encode('UTF-8') for t in entry.transaction])+            stmts = [t.encode('UTF-8') if isinstance(t, str) else t for t in entry.transaction]+            payload = b'\x00'.join(stmts)

Good point, I'm fine with either, since this is at runtime only and doesn't change the backup files.

Feel free to switch it around if that makes it easier to rebase.

Sorry for breaking your PR by merging this. I'll review #189 asap.

cdecker

comment created time in 8 hours

pull request commentlightningd/plugins

feeadjuster: no underscore in method name

@m-schmoock Ok, understood. I will submit a PR later today.

gallizoltan

comment created time in 8 hours

pull request commentlightningd/plugins

rebalance: better feeadjuster integration

Just looks a bit clunky ;)

* b80c558 (origin/master, origin/HEAD, master) rebalance: better feeadjuster integration - turns off feeadjuster's automatic fee updates while rebalance running - runs feeadjust only once, at the end - measures time for rebalance
* 7c88402 feeadjuster: no underscore in method name

Ok, I think I just hit one Enter in the commit message instead of two. Will do better next time.

gallizoltan

comment created time in 8 hours

issue commentElementsProject/lightning

Change sqlite page_size to 4096 - massive performance improvement

@mb300sd, @ZmnSCPxj merging #4337 auto-closed this issue. Would you like me to reopen, or shall we create a separate thread for further sqlite3 optimizations?

mb300sd

comment created time in 8 hours

push eventElementsProject/lightning

ZmnSCPxj jxPCSnmZ

commit sha 4dfbee47f7c9bdf3ef1aa5fe84255514e924af2b

wallet/db.c: Speed up deletion of single peers. ChangeLog-Fixed: Database: Speed up deletion of peer especially when there is a long history with that peer.

view details

push time in 9 hours

PR merged ElementsProject/lightning

wallet/db.c: Speed up deletion of single peers.

Fixes #4325

I petition adding this to the 0.9.3 release candidate, as, it gives a massive speedup on peer deletion, does not change semantics of the database, just pure speedup, and the change is very small.

+42 -21

13 comments

4 changed files

ZmnSCPxj

pr closed time in 9 hours

issue closedElementsProject/lightning

Change sqlite page_size to 4096 - massive performance improvement

Disregard, see next reply

Issue and Steps to Reproduce

I'm having a strange issue with lightningd becoming unresponsive and taking up 100% of a cpu core. There's no output in the logs, and it's not responding to rpc commands, but I do notice that lightningd.sqlite3-journal is continuously growing.

It cleared up after 12 hours or so the first time I let it run, and I assumed it was some kind of database migration or cleanup after upgrading, but it's happening again without me doing anything.

getinfo output

It won't respond to getinfo, but I upgraded to the latest master as of this post.

closed time in 9 hours

mb300sd

pull request commentElementsProject/lightning

wallet/db.c: Speed up deletion of single peers.

ACK https://github.com/ElementsProject/lightning/pull/4337/commits/3ecca0c42252d191856937d290967bcd405cb57f

ZmnSCPxj

comment created time in 9 hours

pull request commentElementsProject/lightning

wallet/db.c: Speed up deletion of single peers.

My bad, must've failed on my side. Sorry for the lecture :-)

ZmnSCPxj

comment created time in 9 hours

issue commentElementsProject/lightning

Change sqlite page_size to 4096 - massive performance improvement

Thanks, that looks like a very big decrease in time to search. Now all we have left is SQLITE3 deletion LOL (which was not actually done in the tests here due to the FOREIGN KEY constraint failing).

Going over the query plan for deleting a peer, it looks like the largest bonuses are from indexing forwarded_payments.

SQLITE3 considers foreign key support to be optional --- it is the reason why we have to give PRAGMA foreign_keys=on; each and every time we open a db with it. Thus, we can expect that SQLITE3 does not really optimize well for this support.

From what I gather elsewhere, if foreign key constraints are involved, SQLITE will delete one row at a time, then check each dependent table if that row was involved in some ON DELETE dependency. In the case of our db schema: deleting a row in peers cascades deletes in channels which cascades into channel_htlcs which cascades into forwarded_payments.

We have a low number of channels rows per peers row (usually just one, though if we close and then open to the same peer in rapid succession I expect there might be two), so SCAN TABLE there will not be costly (though as the number of peers/channels rows grows that might become uncomfortable in the future). There are also only a small number of channel_state_changes per channels row as well, so SCAN TABLE there will not be costly. transaction_annotations seems to be intended to be JOINed with channeltxs. Now, channeltxs contains primarily the closing transactions of channels, and if the channel gets closed while it has a few hundred HTLCs in-flight, that would have a few hundred rows per channels row as well, so it is something we should consider in the near future to optimize as well.

But the real cake-taker is forwarded_payments. This is because we have to store every historical HTLC --- a theft attempt could take any state in our history, meaning all those ancient HTLCs remain potential theft vectors (we could have removed this need by rotating how transactions are laid out --- put the ability to revoke first before emitting the state, which I think was first presented in the Decker-Russell-Osuntokun paper, and which will be adopted by the alternate Poon-Dryja-style that Fournier is working on). So we have to remember each and every HTLC so we can correct them.

There could be several thousand rows of channel_htlcs per channels row, and because the forwarded_payments has a dependent key, SQLITE3 will delete one at a time, then scan the forwarded_payments table for the deleted row in channel_htlcs. And since there is no index on forwarded_payments(out_htlc_id) each deleted channel_htlcs row will trigger a scan of all rows of forwarded_payments, greatly multiplying the time taken.

Now, if you are primarily operating as a forwarding node, and not doing a lot of rebalances or sends or receives of money, then for every two rows in channel_htlcs you are likely to have a row in forwarded_payments. So the one-delete-at-a-time of channel_htlcs compounded with the linear SCAN TABLE of forwarded_payments to satisfy the foreign key constraint leads to awful, awful O( n^2 ) time consumption! Creating an index on forwarded_payments(out_htlc_id) should lead to O( n log n ) time instead.

Probably the Postgres advantage is that it might auto-create an index on columns with a foreign key constraint, and possibly batching multiple deletions on the parent table so it can make fewer range queries on the index of the child table, simply because Postgres foreign key support is not an afterthought like in SQLITE3.

Thanks, looks like the root fix is simply to add the index, which is benign on Postgres and will be a massive performance boost on SQLITE3. SQLITE3 deletions will probably still be slow, but shrug that is a lot harder to fix.

mb300sd

comment created time in 13 hours

issue commentElementsProject/lightning

Change sqlite page_size to 4096 - massive performance improvement

Restore went pretty quick for my RAID being degraded/rebuilding right now...

sqlite3 test-index.sqlite3
SQLite version 3.27.2 2019-02-25 16:06:06
Enter ".help" for usage hints.
sqlite> CREATE INDEX forwarded_payments_out_htlc_id ON forwarded_payments (out_htlc_id);
sqlite> PRAGMA foreign_keys=on;
sqlite> .timer on
sqlite> DELETE FROM peers ORDER BY id LIMIT 1;
Run Time: real 1.875 user 0.078349 sys 0.318645
Error: FOREIGN KEY constraint failed
sqlite> .quit

sqlite3 test-noindex.sqlite3
SQLite version 3.27.2 2019-02-25 16:06:06
Enter ".help" for usage hints.
sqlite> PRAGMA foreign_keys=on;
sqlite> .timer on
sqlite> DELETE FROM peers ORDER BY id LIMIT 1;
Run Time: real 469.100 user 260.638115 sys 208.137506
Error: FOREIGN KEY constraint failed
sqlite> .quit
mb300sd

comment created time in 14 hours

issue commentElementsProject/lightning

Change sqlite page_size to 4096 - massive performance improvement

I actually deleted it from the live vm, but I have daily incremental backups. Give me a few hours to run a restore into a temp vm and I'll give it a shot.

mb300sd

comment created time in 15 hours

issue commentElementsProject/lightning

Change sqlite page_size to 4096 - massive performance improvement

@mb300sd well, if you have a copy of your SQLITE3 database file still, could you try this and report the time spent?

  • Create two copies, one named with-index.sqlite3 the other named without-index.sqlite3.
  • For with-index:
    • sqlite3 with-index.sqlite3
    • CREATE INDEX forwarded_payments_out_htlc_id ON forwarded_payments (out_htlc_id);
    • PRAGMA foreign_keys=on; <- notice the plural, if you misspell foreign_keys it will not error so be careful.
    • .timer on
    • DELETE FROM peers ORDER BY id LIMIT 1; <- this should eventually fail with "foreign key error" but the timing should be accurate. If it complains about ORDER BY or LIMIT it means the sqlite3 is not compiled for this syntax though (it is optional) and that is a bit more complex to figure out.
    • .quit
  • For without-index.sqlite3
    • sqlite3 without-index.sqlite3
    • PRAGMA foreign_keys=on;
    • .timer on
    • DELETE FROM peers ORDER BY id LIMIT 1;
  • Delete the with-index.sqlite3 and without-index.sqlite3 to recover the space.

Totally optional though. I just need confirmation whether the proposed fix is large enough to have solved your issue; it is certainly large enough to be much more performant on the database I have, but I am just removing almost a second from a second of delete time.

mb300sd

comment created time in 17 hours

pull request commentElementsProject/lightning

wallet/db.c: Speed up deletion of single peers.

It is set, do not know why it is not working: Screenshot from 2021-01-19 08-38-06

ZmnSCPxj

comment created time in 17 hours

issue commentlightningd/plugins

feeadjuster: assert can trigger

I'm not sure what would be the best solution. Maybe we should write a @plugin.subscribe("invoice_payment") and a @plugin.subscribe("sendpay_success").

gallizoltan

comment created time in 21 hours

issue commentElementsProject/lightning

Change sqlite page_size to 4096 - massive performance improvement

@hosiawak @fiatjaf is correct. I used https://github.com/fiatjaf/mcldsp to migrate it. Was pretty smooth once we worked out some bugs. If your database is very old like mine was, there may be some leftover records causing foreign key violations, those are safe to manually delete from SQLite before migrating.

@ZmnSCPxj I don't think I'd feel comfortable giving out the database. It's quite a large node with a large amount of funds.

mb300sd

comment created time in a day

issue openedElementsProject/lightning

Implement new semantics for reply_channel_range

The proposal 826 is almost ready to be merged, but requires c-lightning to update as well.

created time in a day

issue openedElementsProject/lightning

hsmtool segfault

The dumponchaindescriptors command segfaults on mainnet if we don't provide a network parameter (bitcoin is the default).

Self-assignment..

created time in a day

issue commentElementsProject/lightning

Merge the per-peer daemons into a single peerd

Passing FDs around are certainly a pain point when considering Windows, and in that regard anything that removes FD-passing completely would be a plus. On the other hand we could just use sockets and special Windows-specific APIs for passing HANDLEs around (I would like to specifically deny that I know anything about the Windows API, and no, DWORD does not mean 32-bit unsigned integer).

Indeed there seem to be posix equivalents in Windows, but we have no experts at hand that could write the shims necessary, so removing the requirement seems to be the easiest way to get Windows support.

  • My proposal would retain the existing channeld/openingd/closingd daemons; the peerd would simply multiplex messges, between gossipd, and specific channel-level daemon.

Notice that my primary goal isn't to eliminate FD-passing, but rather remove the duplicated message handling code in each daemon (where each daemon is implemented just slightly differently), and reflect the reduction in specialization (due to dual-funding, splicing, etc), because the phases of a channel are not as clearly separated as we thought initially.

This proposal surfaced again due to the gargantuan #4293 which moved existing functionality between two daemons since the cutoff between the phases shifted slightly, and resulted in a 91 commit PR!

cdecker

comment created time in a day

issue openedElementsProject/lightning

connect ENOENT /home/ubuntu/testnet/ln/testnet/lightning-rpc

HI.

I migrated from regtest to tesnet and when I try to execute a lightning node rpc call I get:

connect ENOENT /home/ubuntu/testnet/ln/testnet/lightning-rpc

lightning-rpc when created is owned by root. In regtest in order to make it work I had to change ownership to the linux user I use. In testnet I can't make it work.

this is my docker-compose:

testnetln: image: elementsproject/lightningd container_name: testnetln command: - --network=testnet - --bitcoin-rpcconnect=testnetnode - --bitcoin-rpcport=18332 - --bitcoin-rpcuser=xxxxx - --bitcoin-rpcpassword=xxxxx - --bind-addr=0.0.0.0:9901 - --alias=bappTestnet - --log-level=debug - --plugin=/root/.lightning/ln_testnet.py environment: EXPOSE_TCP: "true" expose: - "9901" ports: - "9901:9901" volumes: - "/home/ubuntu/testnet/bitcoin:/etc/bitcoin" - "/home/ubuntu/testnet/ln:/root/.lightning" depends_on: - testnetnode networks: - testnet restart: unless-stopped

Thanks.

created time in a day

pull request commentElementsProject/lightning

wallet/db.c: Speed up deletion of single peers.

By the way, if you allow maintainers to push I can fix these tiny things up myself, rather than having to take a roundtrip to you :wink:

ZmnSCPxj

comment created time in a day

Pull request review commentElementsProject/lightning

wallet/db.c: Speed up deletion of single peers.

 struct db_query db_sqlite3_queries[] = {     }, }; -#define DB_SQLITE3_QUERY_COUNT 294+#define DB_SQLITE4_QUERY_COUNT 294

Typo.

#define DB_SQLITE3_QUERY_COUNT 294
ZmnSCPxj

comment created time in a day

more