Get and store balances, some alternatives

[juanpabloaj]

I was thinking of some alternatives to get and store the balances locally (maybe in the data warehouse).

I have split it into how to get and when to get.

1. How get the balances?

The first part is how to get the balances, until now the simplest way that I found was

{:ok, %Binance.Account{balances: balances}} = Binance.get_account()
balances |> Enum.filter(fn(x) -> Decimal.gt?(x["free"], 0) end)

Maybe this endpoint is expensive. At least Binance's documentation gives a weight of 10 to this endpoint.

https://github.com/binance/binance-spot-api-docs/blob/master/rest-api.md#account-information-user_data

Get current account information.
...
Weight: 10

2. When get the balances?

Until now, I have thought about these alternatives

a. After receiving every Order event

# apps/data_warehouse/lib/data_warehouse/subscriber/worker.ex
  def handle_info(%Binance.Order{} = order, state) do
    ...

    get_balances_and_insert_in_repo()

    {:noreply, state}
  end

You're getting real-time updates from the balances but If get_account is an expensive endpoint, maybe this is not a good approach.

b. Every some period of time with a GenServer andsend_after

Similar to this

use GenServer
...
@ten_seconds 10000

def init(opts) do
  Process.send_after(self(), :tick, @ten_seconds)
  {:ok, opts}
end

def handle_info(:tick, state) do
  balances = get_balances()

  struct!(DataWarehouse.Schema.Balances, balances)
  |> DataWarehouse.Repo.insert()

  {:noreply, state}
end

https://stackoverflow.com/a/32097971/348081

c. Every some period of time with quantum-core

Similar to the previous option but using some dependencies similar to cron like quantum-core

https://github.com/quantum-elixir/quantum-core

This option could give you more control, more precision over the instant of the time to do the task: every hour at the minute 0, or every day at midnight, etc. But you need to add a new dependency.

What do you think?
Do you have another idea or suggestions?

Thanks in advance for your commentaries.

References

• https://elixirschool.com/blog/til-send-after/
• https://stackoverflow.com/questions/32085258/how-can-i-schedule-code-to-run-every-few-hours-in-elixir-or-phoenix-framework

[Cinderella-Man]

Hi @juanpabloaj

Is there any reason why you would need to query balances once again after the initial setup? I think you could get away with estimating the new balances and updating the DB accordingly after every trade?

I know that this isn't a perfect solution as in case of "dropping" some trades because of, i.e. full server failure, we wouldn't be up to date anymore. Although this could be potentially fixed by simply querying Binance at the start of the server?
(there's whole another topic about picking up partially made trades etc., but that's a discussion for another topic 😉 )

There's an additional advantage of calculating the new balances on our own - they can grow/shrink automatically as traders trade, as you would update the state in both the leader and the database "in real-time".

Let me know what do you think about this idea?

[thisiscetin]

Hi @juanpabloaj ,

I think if you are in control of your interactions with Binance, you should be able to estimate when the balance in your system stops matching the remote(Binance) the majority of the time. What I mean by this is, the time you execute an order, or an order is filled, etc. the probability of the balance being dirty and not matching in your side is getting higher and higher.

So, instead of fetching with an interval, I would think about a mechanism to mark the balance field as dirty after potentially balance-changing interactions. When you need it and you see it's dirty, you can make a balance check upfront. At the same time, you can complement that with a scheduled check, if dirty -> update balance -> mark dirty = false. For eventual consistency, you can go one step further and introduce an expiry time like 3/6/x hours. When the balance field expired, you can refresh the balance regardless of its being dirty or not.

But those complex solutions most of the time comes with a cost, I would try to go with the simplest approach, like refresh after every trade, until underlying technical implementation / API limits... do not allow me to do so.