Skip to content

Async BumpTransactionEventHandler #3752

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Jun 12, 2025

Conversation

joostjager
Copy link
Contributor

@joostjager joostjager commented Apr 28, 2025

Converts BumpTransactionEventHandler to async.

Changes the CoinSelectionSource and WalletSource traits to be async and provides WalletSourceSyncWrapper as a helper for users that want to implement a sync wallet source.

TestWalletSource is kept sync, to prevent a cascade of async conversions in tests.

Fixes #3540

@ldk-reviews-bot
Copy link

ldk-reviews-bot commented Apr 28, 2025

👋 I see @tnull was un-assigned.
If you'd like another reviewer assignemnt, please click here.

@joostjager joostjager force-pushed the async-wallet-bump branch 3 times, most recently from d9b1c75 to 873516b Compare May 1, 2025 11:08
Copy link

codecov bot commented May 1, 2025

Codecov Report

Attention: Patch coverage is 90.08621% with 23 lines in your changes missing coverage. Please review.

Project coverage is 90.78%. Comparing base (35748f6) to head (e7d43db).
Report is 16 commits behind head on main.

Files with missing lines Patch % Lines
lightning/src/events/bump_transaction.rs 89.92% 5 Missing and 9 partials ⚠️
lightning/src/events/bump_transaction_sync.rs 90.00% 8 Missing ⚠️
lightning/src/util/test_utils.rs 85.71% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #3752      +/-   ##
==========================================
+ Coverage   89.92%   90.78%   +0.86%     
==========================================
  Files         160      161       +1     
  Lines      129322   137021    +7699     
  Branches   129322   137021    +7699     
==========================================
+ Hits       116290   124393    +8103     
+ Misses      10345     9956     -389     
+ Partials     2687     2672      -15     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@joostjager joostjager force-pushed the async-wallet-bump branch 6 times, most recently from 4b4b898 to 3816c66 Compare May 5, 2025 11:37
@joostjager joostjager force-pushed the async-wallet-bump branch 13 times, most recently from 1a86521 to e1509f1 Compare May 14, 2025 09:55
@joostjager joostjager changed the title Async wallet bump Async BumpTransactionEventHandler May 14, 2025
@joostjager joostjager marked this pull request as ready for review May 14, 2025 11:06
@joostjager joostjager requested a review from tnull May 14, 2025 11:06
@joostjager joostjager added the weekly goal Someone wants to land this this week label May 14, 2025
Copy link
Contributor

@tnull tnull left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did a first / quickish pass. Generally looks good so far.

@@ -48,12 +48,14 @@ backtrace = { version = "0.3", optional = true }

libm = { version = "0.2", default-features = false }
inventory = { version = "0.3", optional = true }
tokio = { version = "1.35", features = [ "macros", "rt" ], optional = true }
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please set default-features = false here and below.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added. Why is this? I looked at the tokio crate and it seems there are no default features?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's 'best practice' and will also make sure we'd not add arbitrary dependencies if tokio decided to add default features in some release.

*commitment_tx_fee_satoshis,
anchor_descriptor,
) {
if let Err(_) = self
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IMO these would be a bit more readable if we did the the whole call including .await on a separate line and then simply did if res.is_err() { .. }

Copy link
Contributor Author

@joostjager joostjager May 15, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tried it, but I don't think it is an improvement with that temp var.

let res = self
	.handle_channel_close(
		*claim_id,
		*package_target_feerate_sat_per_1000_weight,
		commitment_tx,
		*commitment_tx_fee_satoshis,
		anchor_descriptor,
	)
	.await;
if res.is_err() {
	log_error!(
		self.logger,
		"Failed bumping commitment transaction fee for {}",
		commitment_tx.compute_txid()
	);
}

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also tried with map_err, but that doesn't work all that well with the parent fn without a return result.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also tried with map_err, but that doesn't work all that well with the parent fn without a return result.

How about:

				self.handle_channel_close(
					*claim_id,
					*package_target_feerate_sat_per_1000_weight,
					commitment_tx,
					*commitment_tx_fee_satoshis,
					anchor_descriptor,
				)
				.await
				.unwrap_or_else(|_| {
					log_error!(
						self.logger,
						"Failed bumping commitment transaction fee for {}",
						commitment_tx.compute_txid()
					);
				});

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah this is nicer. Wasn't aware of the unwrap_or_else on result. Updated.

#[allow(clippy::expect_used, clippy::diverging_sub_expression)]
{
return tokio::runtime::Builder::new_current_thread()
.enable_all()
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FWIW, I don't think we need to enable_all here, could just actually enable features we need. Same goes for all the tests that use [tokio::test], although not sure if it makes a big difference in practice.

Copy link
Contributor Author

@joostjager joostjager May 15, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I just expanded the macro. Can make customizations, but maybe it is better to keep to the original expansion? Also if it doesn't matter in practice...

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, no strong opinion. If we increase the number of tests using tokio, we might want to some benchmarks though, as the runtime of cargo test accumulates over time, so every second we can squeeze out of it would be appreciated.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed then. If we get more of these xtests, we might need to duplicate the macro or something like that.

let configs = [(false, false), (false, true), (true, false), (true, true)];
let mut last_err = None;
for (force_conflicting_utxo_spend, tolerate_high_network_feerates) in configs {
log_debug!(
self.logger,
"Attempting coin selection targeting {} sat/kW (force_conflicting_utxo_spend = {}, tolerate_high_network_feerates = {})",
target_feerate_sat_per_1000_weight,
force_conflicting_utxo_spend,
tolerate_high_network_feerates
);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: Indent off here.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't see the indent off. The closing parenthesis?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually the parenthesis and all arguments of log_debug too (they need to be indented by one tab).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't see the problem anymore in the diff now, but github still shows it on the old code fragment? In my IDE it all looks good with tabs, so I think it is solved.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nope, still there on HEAD:

Screenshot From 2025-05-16 10-47-39

Copy link
Contributor Author

@joostjager joostjager May 16, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, happened in the next commit. Fixed now.

@ldk-reviews-bot
Copy link

👋 The first review has been submitted!

Do you think this PR is ready for a second reviewer? If so, click here to assign a second reviewer.

@joostjager joostjager requested a review from TheBlueMatt June 9, 2025 14:17
@joostjager
Copy link
Contributor Author

Reviewers, didn't do fix up commits as the commit structure changed substantially. Thanks for your patience with this PR. Should have realized that the sync wrappers could avoid all the test changes that have now been reverted.

Copy link
Contributor

@tnull tnull left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Basically LGTM, some nits/comments.

Also double-checked this still works with LDK Node.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure having this module is parallel is the way to go. If you really prefer to move the wrappers to their own file, should they be in events/bump_transaction/blocking.rs (or sync.rs, but the latter is usually associated with synchronization primitives in Rust), for example?

Also, this module is missing a licensing header.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I put them in a separate file to avoid large files. Quite a few devs indicated that they did not like large files.

What is wrong with bump_transaction_sync.rs? It aligns with the types inside. Something like blocking.rs might grow big again.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is wrong with bump_transaction_sync.rs?

In terms of crate organization, it's just a file in-parallel even though it hold utilities for the bump_transaction logic. So would just make sense to have it part of the same sub-module rather an in-parallel module, IMO, but no blocker.

Something like blocking.rs might grow big again.

Not sure I'm following how the name relates to what/how much we put in a file?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, I missed that you suggested a bump_tx subdir. Made the change in a fix up commit. Generated docs look better now, with sync residing below bump_transaction. Let me know if this is what you had in mind.

This preparatory commit converts do_coin_selection from a closure into
a standard function.
@joostjager joostjager force-pushed the async-wallet-bump branch 2 times, most recently from 93e5d7e to 2fdcf05 Compare June 10, 2025 12:54
Copy link
Contributor

@tnull tnull left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Feel free to squash fixups from my side.

@joostjager
Copy link
Contributor Author

Squashed

@joostjager joostjager requested a review from tnull June 10, 2025 13:37
tnull
tnull previously approved these changes Jun 10, 2025
Copy link
Collaborator

@TheBlueMatt TheBlueMatt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please write a commit message for the last commit. Its +393 -105 and very far from trivial. It needs a textual explanation of the approach, why we're doing wrappers, etc.

@@ -11,6 +11,8 @@
//!
//! [`Event`]: crate::events::Event

pub mod sync;
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we're moving all the wrappers into a separate module let's definitely link to the wrapper types in the documentation for the traits/structs. When all the types are the in the same module things are pretty visible in docs, when they're not the sync wrappers become invisible and people will get confused and think they have to use the async version.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Links added

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am getting to see more of how rustdoc works with this PR. It's quite nice.

where
T::Target: WalletSourceSync,
{
/// Returns all UTXOs, with at least 1 confirmation each, that are available to spend. Wraps
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Usually we don't bother to write documentation for impls of traits as rustdoc doesn't do a good job of surfacing it, but also in this case its just repeating the trait docs?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed

}
}

/// A synchronous helper trait that can be used to implement [`CoinSelectionSource`] in a synchronous
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These docs seem to imply to me that I'm not supposed to use it directly (its a "helper trait") but that's not accurate.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated

Comment on lines 44 to 42
/// This wrapper isn't intended to be used directly, because that would risk blocking an async context. Instead, it is
/// built for you in [`WalletSync::new`].
#[doc(hidden)]
pub(crate) struct WalletSourceSyncWrapper<T: Deref>(T)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, I didn't realize we could make this non-pub. We don't really have to worry about verbose docs in that case, and probably shouldn't #[doc(hidden)] since it should show up in crate-internal documentation (cargo doc --document-private-items)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Indeed, no longer necessary. Removed.

where
T::Target: CoinSelectionSourceSync,
{
#[allow(dead_code)]
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, seems this came back after it had already been dropped? #3752 (comment)

Copy link
Contributor Author

@joostjager joostjager Jun 11, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Didn't remove all of it apparently. And also found a new for WalletSourceSyncWrapper that doesn't add much. Removed as well.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I mostly meant the dead_code cause its used now, but removing the new method is also nice.

@@ -0,0 +1,241 @@
// This file is Copyright its original authors, visible in version control
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: It seems more sensible to move bump_transaction.rs to bump_transaction/mod.rs to keep it and sync.rs together.

Copy link
Contributor Author

@joostjager joostjager Jun 11, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done. Although so many mod.rs files with actual code make it a bit harder to navigate. Not sure if rust has a way of keeping files in the same sub dir and still have a descriptive file name, without making it a sub module. And which is also in line with project conventions.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The only way to do that would be to have a separate module and re-export from the top-level one. I'm generally not a fan of that unless we have a good reason (eg the module is way too large), which this doesn't feel like.

@joostjager
Copy link
Contributor Author

Comments addressed, commit message expanded. Ready for another look.

Copy link
Contributor

@tnull tnull left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixups look good to me. Could be squashed from my side, once Matt saw them, too.

@joostjager joostjager requested review from tnull and TheBlueMatt and removed request for tnull June 11, 2025 12:28
@TheBlueMatt
Copy link
Collaborator

Feel free to squash

Copy link
Collaborator

@TheBlueMatt TheBlueMatt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Needs squash

This commit converts the `CoinSelectionSource` and `WalletSource`
traits to `async` and bubbles up the `async` keyword where
needed.

To preserve usability in a synchronous context, sync versions of
these traits are still provided and can be used via the
`WalletSync` and `BumpTransactionEventHandlerSync` wrappers.
Existing synchronous tests remain unchanged and also use
these wrappers.

Additionally, the `MaybeSync` and `MaybeSend` marker traits are
introduced to support `no_std` compilation. Due to limitations in
Rust 1.63's type inference, these marker traits are used more
frequently than might be necessary in later versions.
@joostjager
Copy link
Contributor Author

Squashed

@joostjager joostjager requested a review from TheBlueMatt June 11, 2025 18:29
Copy link
Collaborator

@TheBlueMatt TheBlueMatt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Gonna go ahead and land this since @tnull ACK'd it previously and LGTM'd the fixups. Will probably open a followup to remove a few allocations but no need to delay this.

@@ -1874,36 +1873,36 @@ impl Drop for TestScorer {

pub struct TestWalletSource {
secret_key: SecretKey,
utxos: RefCell<Vec<Utxo>>,
utxos: Mutex<Vec<Utxo>>,
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think its okay. Its certainly not ideal but it only kicks on with std, and the alternative is macro-ing the whole code and duplicating it all, which I'm not really sure is a better answer. Of course once Rust has proper impl Trait returns from trait methods we can drop this stuff, but it sounds like thats still a bit off :/

wallet_source: Arc::clone(&wallet_source),
bump_tx_handler: BumpTransactionEventHandler::new(
cfgs[i].tx_broadcaster, Arc::new(Wallet::new(Arc::clone(&wallet_source), cfgs[i].logger)),
wallet_source: wallet_source.clone(),
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Definitely prefer to leave Arc::clones over .clone()s. They're way more readable :)

Copy link
Contributor Author

@joostjager joostjager Jun 12, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't feel it. Maybe that feeling needs time to develop 😅

Copy link
Contributor

@tnull tnull Jun 12, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't feel it. Maybe that feeling needs time to develop 😅

Maybe. FWIW, I share the preference for Arc::clone, especially as it highlights that it's not a deep (i.e. costly) clone, but just cloning a reference..

@joostjager joostjager merged commit 7e6ca67 into lightningdevkit:main Jun 12, 2025
26 of 27 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
weekly goal Someone wants to land this this week
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Make BumpTransactionEventHandler async-optional
4 participants