Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ban solvers based on the settlements success rate #3263

Merged
merged 120 commits into from
Feb 26, 2025
Merged
Show file tree
Hide file tree
Changes from 103 commits
Commits
Show all changes
120 commits
Select commit Hold shift + click to select a range
5fe0dd6
Solver participation validator
squadgazzz Jan 29, 2025
5319945
Test
squadgazzz Jan 29, 2025
e65c328
Avoid rpc calls every time
squadgazzz Jan 29, 2025
fc3321b
Typo
squadgazzz Jan 29, 2025
0fbd61c
Docs
squadgazzz Jan 29, 2025
b1abfa0
Metrics
squadgazzz Jan 29, 2025
292dcff
Configurable validators
squadgazzz Jan 29, 2025
fe9ef5b
Fixed clap config
squadgazzz Jan 29, 2025
c5e3502
Refactoring
squadgazzz Jan 30, 2025
a9e6a3f
Config per solver
squadgazzz Jan 30, 2025
9a55fe2
Start using the new config
squadgazzz Jan 30, 2025
f9bdafd
Simplify to hashset
squadgazzz Jan 30, 2025
5fc831e
Nit
squadgazzz Jan 30, 2025
321f9bc
Use driver directly
squadgazzz Jan 30, 2025
f69e174
Notify external solvers
squadgazzz Jan 30, 2025
1504c48
/notify driver endpoint
squadgazzz Jan 30, 2025
9f0cbc8
OpenApi
squadgazzz Jan 30, 2025
7f86000
Nit
squadgazzz Jan 30, 2025
d233094
Solver's OpenAPI
squadgazzz Jan 30, 2025
105b9a7
Store drivers accepted for the feature
squadgazzz Jan 31, 2025
08924da
Comment
squadgazzz Jan 31, 2025
3154cd0
Use driver's name in metrics
squadgazzz Jan 31, 2025
47007c1
Nit
squadgazzz Jan 31, 2025
dfec727
Merge branch 'blacklist-failing-solvers' into notify-banned-solvers
squadgazzz Jan 31, 2025
e370871
Fix after merge
squadgazzz Jan 31, 2025
bb9059e
Send metrics about each found solver
squadgazzz Jan 31, 2025
3c591e4
Merge branch 'blacklist-failing-solvers' into notify-banned-solvers
squadgazzz Jan 31, 2025
6787d34
Cache only accepted solvers
squadgazzz Jan 31, 2025
85a3d86
Merge branch 'blacklist-failing-solvers' into notify-banned-solvers
squadgazzz Jan 31, 2025
921692f
Notify only accepted solvers
squadgazzz Jan 31, 2025
f3b6415
Minor refactoring
squadgazzz Jan 31, 2025
805946c
DB statistics guard
squadgazzz Jan 31, 2025
a2710c6
Refactoring
squadgazzz Jan 31, 2025
098113a
Merge branch 'blacklist-failing-solvers' into notify-banned-solvers
squadgazzz Jan 31, 2025
7ab2a38
Naming
squadgazzz Jan 31, 2025
cca115a
Merge branch 'notify-banned-solvers' into db-statistics-based-guard
squadgazzz Jan 31, 2025
43e650f
Banned notification
squadgazzz Jan 31, 2025
983dc41
Merge branch 'notify-banned-solvers' into db-statistics-based-guard
squadgazzz Jan 31, 2025
160e2d9
New ban reason notification
squadgazzz Jan 31, 2025
597d268
min_settlement_success_rate
squadgazzz Jan 31, 2025
7c87087
Separate configs
squadgazzz Jan 31, 2025
1f43009
Fix the tests
squadgazzz Feb 3, 2025
0d50991
Merge branch 'main' into blacklist-failing-solvers
squadgazzz Feb 3, 2025
634aec4
Merge branch 'blacklist-failing-solvers' into notify-banned-solvers
squadgazzz Feb 3, 2025
df4b77f
Merge branch 'notify-banned-solvers' into db-statistics-based-guard
squadgazzz Feb 3, 2025
366611d
Nits
squadgazzz Feb 7, 2025
e9a70f5
Trigger updates on the proposed_solution table insert
squadgazzz Feb 11, 2025
e220eaf
Nit
squadgazzz Feb 11, 2025
8745e6b
Merge branch 'blacklist-failing-solvers' into notify-banned-solvers
squadgazzz Feb 11, 2025
4bee676
Move notification to the infra mod
squadgazzz Feb 11, 2025
17f3f3a
Ban until timestamp notification
squadgazzz Feb 11, 2025
9ee6af7
Merge remote-tracking branch 'origin/notify-banned-solvers' into noti…
squadgazzz Feb 11, 2025
51832d4
Formatting
squadgazzz Feb 11, 2025
70d8907
Merge branch 'blacklist-failing-solvers' into notify-banned-solvers
squadgazzz Feb 11, 2025
b2b4c91
Doc
squadgazzz Feb 12, 2025
7ef7db2
Switch to max failure rate
squadgazzz Feb 12, 2025
57cb190
Naming
squadgazzz Feb 12, 2025
1b7829f
Merge branch 'notify-banned-solvers' into db-statistics-based-guard
squadgazzz Feb 12, 2025
8af1e4b
Fix after merge
squadgazzz Feb 12, 2025
e7ad14f
Notifications refactoring
squadgazzz Feb 12, 2025
cca70c8
Refactoring
squadgazzz Feb 12, 2025
7de4dc1
OpenAPI
squadgazzz Feb 12, 2025
9f4b5ad
Merge branch 'notify-banned-solvers' into db-statistics-based-guard
squadgazzz Feb 12, 2025
ee7f492
Naming
squadgazzz Feb 12, 2025
17ee52c
infra::Persistence
squadgazzz Feb 12, 2025
cba693a
Naming
squadgazzz Feb 12, 2025
5c266d8
Merge branch 'blacklist-failing-solvers' into notify-banned-solvers
squadgazzz Feb 12, 2025
4523b52
Formatting
squadgazzz Feb 12, 2025
f26d676
Merge branch 'notify-banned-solvers' into db-statistics-based-guard
squadgazzz Feb 12, 2025
0d57971
Fix after merge
squadgazzz Feb 12, 2025
c3c9433
Comment
squadgazzz Feb 14, 2025
58d7de1
Merge branch 'main' into blacklist-failing-solvers
squadgazzz Feb 14, 2025
fdc3afe
Merge branch 'main' into blacklist-failing-solvers
squadgazzz Feb 14, 2025
6bb7ee2
Merge branch 'blacklist-failing-solvers' into notify-banned-solvers
squadgazzz Feb 14, 2025
839488f
Merge branch 'notify-banned-solvers' into db-statistics-based-guard
squadgazzz Feb 14, 2025
b773f66
Api description
squadgazzz Feb 14, 2025
ab5c616
Drop error response
squadgazzz Feb 14, 2025
680e806
Naming
squadgazzz Feb 14, 2025
4f6cd1d
Comments
squadgazzz Feb 17, 2025
bdd33d0
Simplify the code
squadgazzz Feb 17, 2025
fd0fc27
Nits
squadgazzz Feb 17, 2025
38ad536
ISO timestamp
squadgazzz Feb 17, 2025
c76761a
As str
squadgazzz Feb 17, 2025
1a35e9d
Nit
squadgazzz Feb 17, 2025
171de85
Merge branch 'blacklist-failing-solvers' into notify-banned-solvers
squadgazzz Feb 17, 2025
d4b87bd
Fix after merge
squadgazzz Feb 17, 2025
e5250a5
Solver names in the log
squadgazzz Feb 17, 2025
2e3f4ab
Merge branch 'blacklist-failing-solvers' into notify-banned-solvers
squadgazzz Feb 17, 2025
173acea
Merge branch 'notify-banned-solvers' into db-statistics-based-guard
squadgazzz Feb 17, 2025
204984f
Fix after merge
squadgazzz Feb 17, 2025
051bd50
Naming
squadgazzz Feb 17, 2025
de8cfa2
Merge branch 'blacklist-failing-solvers' into notify-banned-solvers
squadgazzz Feb 17, 2025
e301f64
Merge branch 'notify-banned-solvers' into db-statistics-based-guard
squadgazzz Feb 17, 2025
4bb8640
Fixes unit tests
squadgazzz Feb 17, 2025
f5385fb
Merge branch 'blacklist-failing-solvers' into notify-banned-solvers
squadgazzz Feb 17, 2025
2256282
Merge branch 'notify-banned-solvers' into db-statistics-based-guard
squadgazzz Feb 17, 2025
de31f1e
Nit
squadgazzz Feb 18, 2025
127e2f5
Merge branch 'blacklist-failing-solvers' into notify-banned-solvers
squadgazzz Feb 18, 2025
4bc5805
Merge branch 'notify-banned-solvers' into db-statistics-based-guard
squadgazzz Feb 18, 2025
2985000
Fix after merge
squadgazzz Feb 18, 2025
a4a43e5
Fire notify and forget
squadgazzz Feb 18, 2025
4a81454
Copy enum
squadgazzz Feb 18, 2025
6f315bb
Merge branch 'notify-banned-solvers' into db-statistics-based-guard
squadgazzz Feb 18, 2025
b4c31f3
Redundant serde_as
squadgazzz Feb 19, 2025
ab98890
Log the found block
squadgazzz Feb 18, 2025
d362f9b
Redundant ordering
squadgazzz Feb 24, 2025
9b9ff24
Merge branch 'main' into blacklist-failing-solvers
squadgazzz Feb 26, 2025
e568f3b
Merge branch 'blacklist-failing-solvers' into notify-banned-solvers
squadgazzz Feb 26, 2025
f66d257
Minor
squadgazzz Feb 26, 2025
eba672c
Minor
squadgazzz Feb 26, 2025
ecd6e6a
Merge branch 'blacklist-failing-solvers' into notify-banned-solvers
squadgazzz Feb 26, 2025
1cc17ba
Merge branch 'notify-banned-solvers' into db-statistics-based-guard
squadgazzz Feb 26, 2025
2677d26
Formatting
squadgazzz Feb 26, 2025
3bae51f
Merge branch 'main' into notify-banned-solvers
squadgazzz Feb 26, 2025
6ea7aac
Merge branch 'notify-banned-solvers' into db-statistics-based-guard
squadgazzz Feb 26, 2025
88d9c71
Solvers-dto
squadgazzz Feb 26, 2025
301f87e
Merge branch 'notify-banned-solvers' into db-statistics-based-guard
squadgazzz Feb 26, 2025
06ee6b0
Solvers-dto
squadgazzz Feb 26, 2025
4971ff1
Merge branch 'main' into db-statistics-based-guard
squadgazzz Feb 26, 2025
92047e7
Fix after merge
squadgazzz Feb 26, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions crates/autopilot/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,7 @@ chrono = { workspace = true }
clap = { workspace = true }
contracts = { path = "../contracts" }
cow-amm = { path = "../cow-amm" }
dashmap = { workspace = true }
database = { path = "../database" }
derive_more = { workspace = true }
ethcontract = { workspace = true }
Expand Down
140 changes: 136 additions & 4 deletions crates/autopilot/src/arguments.rs
Original file line number Diff line number Diff line change
Expand Up @@ -245,6 +245,82 @@ pub struct Arguments {
/// Archive node URL used to index CoW AMM
#[clap(long, env)]
pub archive_node_url: Option<Url>,

/// Configuration for the solver participation guard.
#[clap(flatten)]
pub db_based_solver_participation_guard: DbBasedSolverParticipationGuardConfig,
}

#[derive(Debug, clap::Parser)]
pub struct DbBasedSolverParticipationGuardConfig {
/// Enables or disables the solver participation guard
#[clap(
id = "db_enabled",
long = "db-based-solver-participation-guard-enabled",
env = "DB_BASED_SOLVER_PARTICIPATION_GUARD_ENABLED",
default_value = "true"
)]
pub enabled: bool,

/// Sets the duration for which the solver remains blacklisted.
/// Technically, the time-to-live for the solver participation blacklist
/// cache.
#[clap(long, env, default_value = "5m", value_parser = humantime::parse_duration)]
pub solver_blacklist_cache_ttl: Duration,

#[clap(flatten)]
pub non_settling_solvers_finder_config: NonSettlingSolversFinderConfig,

#[clap(flatten)]
pub low_settling_solvers_finder_config: LowSettlingSolversFinderConfig,
}

#[derive(Debug, clap::Parser)]
pub struct NonSettlingSolversFinderConfig {
/// Enables search of non-settling solvers.
Comment on lines +279 to +280
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we still need the non-settling version if the low-settling version could be configured to cover that as well? (e.g. ban if 0% of all solutions went through)

Copy link
Contributor Author

@squadgazzz squadgazzz Feb 24, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The low-settling version cannot quickly ban the solver if it has 3 consecutive failed settlements since the logic is focused more on a zoomed-out picture. For example, if a solver had 90% of successful settlements and suddenly shut down, the low-settling metric would require much more time to ban this solver, whereas the non-settling guard would do that right after the 3 consecutive failures.

#[clap(
id = "non_settling_solvers_blacklisting_enabled",
long = "non-settling-solvers-blacklisting-enabled",
env = "NON_SETTLING_SOLVERS_BLACKLISTING_ENABLED",
default_value = "true"
)]
pub enabled: bool,

/// The number of last auctions to check solver participation eligibility.
#[clap(
id = "non_settling_last_auctions_participation_count",
long = "non-settling-last-auctions-participation-count",
env = "NON_SETTLING_LAST_AUCTIONS_PARTICIPATION_COUNT",
default_value = "3"
)]
pub last_auctions_participation_count: u32,
}

#[derive(Debug, clap::Parser)]
pub struct LowSettlingSolversFinderConfig {
/// Enables search of non-settling solvers.
#[clap(
id = "low_settling_solvers_blacklisting_enabled",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what is id used for?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

By default, clap uses the variable name as an ID. For this particular case, since all the params are flattened, there will be 2 identical IDs for the enabled fields, so clap fails to parse the args without this annotation.

long = "low-settling-solvers-blacklisting-enabled",
env = "LOW_SETTLING_SOLVERS_BLACKLISTING_ENABLED",
default_value = "true"
)]
pub enabled: bool,

/// The number of last auctions to check solver participation eligibility.
#[clap(
id = "low_settling_last_auctions_participation_count",
long = "low-settling-last-auctions-participation-count",
env = "LOW_SETTLING_LAST_AUCTIONS_PARTICIPATION_COUNT",
default_value = "100"
)]
pub last_auctions_participation_count: u32,

/// A max failure rate for a solver to remain eligible for
/// participation in the competition. Otherwise, the solver will be
/// banned.
#[clap(long, env, default_value = "0.9")]
pub solver_max_settlement_failure_rate: f64,
}

impl std::fmt::Display for Arguments {
Expand Down Expand Up @@ -290,6 +366,7 @@ impl std::fmt::Display for Arguments {
max_winners_per_auction,
archive_node_url,
max_solutions_per_solver,
db_based_solver_participation_guard,
} = self;

write!(f, "{}", shared)?;
Expand Down Expand Up @@ -373,6 +450,11 @@ impl std::fmt::Display for Arguments {
"max_solutions_per_solver: {:?}",
max_solutions_per_solver
)?;
writeln!(
f,
"db_based_solver_participation_guard: {:?}",
db_based_solver_participation_guard
)?;
Ok(())
}
}
Expand All @@ -384,6 +466,7 @@ pub struct Solver {
pub url: Url,
pub submission_account: Account,
pub fairness_threshold: Option<U256>,
pub requested_timeout_on_problems: bool,
}

#[derive(Debug, Clone, PartialEq, Eq, Hash)]
Expand Down Expand Up @@ -432,18 +515,31 @@ impl FromStr for Solver {
Account::Address(H160::from_str(parts[2]).context("failed to parse submission")?)
};

let fairness_threshold = match parts.get(3) {
Some(value) => {
Some(U256::from_dec_str(value).context("failed to parse fairness threshold")?)
let mut fairness_threshold: Option<U256> = Default::default();
let mut requested_timeout_on_problems = false;

if let Some(value) = parts.get(3) {
match U256::from_dec_str(value) {
Ok(parsed_fairness_threshold) => {
fairness_threshold = Some(parsed_fairness_threshold);
}
Err(_) => {
requested_timeout_on_problems =
value.to_lowercase() == "requested_timeout_on_problems";
}
}
None => None,
};

if let Some(value) = parts.get(4) {
requested_timeout_on_problems = value.to_lowercase() == "requested_timeout_on_problems";
}

Ok(Self {
name: name.to_owned(),
url,
fairness_threshold,
submission_account,
requested_timeout_on_problems,
})
}
}
Expand Down Expand Up @@ -640,6 +736,7 @@ mod test {
name: "name1".into(),
url: Url::parse("http://localhost:8080").unwrap(),
fairness_threshold: None,
requested_timeout_on_problems: false,
submission_account: Account::Address(H160::from_slice(&hex!(
"C02aaA39b223FE8D0A0e5C4F27eAD9083C756Cc2"
))),
Expand All @@ -655,6 +752,7 @@ mod test {
name: "name1".into(),
url: Url::parse("http://localhost:8080").unwrap(),
fairness_threshold: None,
requested_timeout_on_problems: false,
submission_account: Account::Kms(
Arn::from_str("arn:aws:kms:supersecretstuff").unwrap(),
),
Expand All @@ -673,6 +771,40 @@ mod test {
"C02aaA39b223FE8D0A0e5C4F27eAD9083C756Cc2"
))),
fairness_threshold: Some(U256::exp10(18)),
requested_timeout_on_problems: false,
};
assert_eq!(driver, expected);
}

#[test]
fn parse_driver_with_accepts_unsettled_blocking_flag() {
let argument =
"name1|http://localhost:8080|0xC02aaA39b223FE8D0A0e5C4F27eAD9083C756Cc2|requested_timeout_on_problems";
let driver = Solver::from_str(argument).unwrap();
let expected = Solver {
name: "name1".into(),
url: Url::parse("http://localhost:8080").unwrap(),
submission_account: Account::Address(H160::from_slice(&hex!(
"C02aaA39b223FE8D0A0e5C4F27eAD9083C756Cc2"
))),
fairness_threshold: None,
requested_timeout_on_problems: true,
};
assert_eq!(driver, expected);
}

#[test]
fn parse_driver_with_threshold_and_accepts_unsettled_blocking_flag() {
let argument = "name1|http://localhost:8080|0xC02aaA39b223FE8D0A0e5C4F27eAD9083C756Cc2|1000000000000000000|requested_timeout_on_problems";
let driver = Solver::from_str(argument).unwrap();
let expected = Solver {
name: "name1".into(),
url: Url::parse("http://localhost:8080").unwrap(),
submission_account: Account::Address(H160::from_slice(&hex!(
"C02aaA39b223FE8D0A0e5C4F27eAD9083C756Cc2"
))),
fairness_threshold: Some(U256::exp10(18)),
requested_timeout_on_problems: true,
};
assert_eq!(driver, expected);
}
Expand Down
6 changes: 5 additions & 1 deletion crates/autopilot/src/domain/competition/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -6,8 +6,12 @@ use {
};

mod participant;
mod participation_guard;

pub use participant::{Participant, Ranked, Unranked};
pub use {
participant::{Participant, Ranked, Unranked},
participation_guard::SolverParticipationGuard,
};

type SolutionId = u64;

Expand Down
Loading
Loading