Skip to content

Commit

Permalink
Handle rollback during postgres peer election
Browse files Browse the repository at this point in the history
Infrequently, the postgres peer election transaction returns `{:error,
:rollback}`. Now that return value is handled to prevent a match error.

The peer maintains its current `leader?` status on rollback—this may
cause inconsistency if the leader encounters an error and multiple
rollbacks happen in sequence. That tradeoff is acceptable because the
situation is unlikely and less of an issue than crashing the peer.

Closes #1007
  • Loading branch information
sorentwo committed Jan 4, 2024
1 parent 3ca84b9 commit 61e30f7
Showing 1 changed file with 17 additions and 8 deletions.
25 changes: 17 additions & 8 deletions lib/oban/peers/postgres.ex
Original file line number Diff line number Diff line change
Expand Up @@ -90,14 +90,23 @@ defmodule Oban.Peers.Postgres do

state =
:telemetry.span([:oban, :peer, :election], meta, fn ->
{:ok, state} =
Repo.transaction(state.conf, fn ->
state
|> delete_expired_peers()
|> upsert_peer()
end)

{state, %{meta | leader: state.leader?}}
fun = fn ->
state
|> delete_expired_peers()
|> upsert_peer()
end

case Repo.transaction(state.conf, fun) do
{:ok, state} ->
{state, %{meta | leader: state.leader?}}

{:error, :rollback} ->
# The peer maintains its current `leader?` status on rollback—this may cause
# inconsistency if the leader encounters an error and multiple rollbacks happen in
# sequence. That tradeoff is acceptable because the situation is unlikely and less of
# an issue than crashing the peer.
{state, meta}
end
end)

{:noreply, schedule_election(state)}
Expand Down

0 comments on commit 61e30f7

Please sign in to comment.