Streaming / Checkpointing while using daft

[ghost]

To use daft for processing in streaming ( micro batch or similar ) , it should be possible to use the time travel feature of iceberg/delta to do checkpointing. Is there any plans for daft to do it or are there third party libraries that can help with this.

to summarize ... something like ..

spark.readStream.table(delta_table)
...
...
.checkpointLocation("xxxx").

[jaychia]

Hey @vibh3s! Could you elaborate on your use-case here? Streaming means quite different things depending on who you ask 😛

Daft today is mostly a batch engine, but micro-batching is definitely on the roadmap. Interested to learn more about how you're thinking about this problem though.

[ghost]

If you think of a delta or iceberg table. I am thinking of processing it with daft for the data that has been processed "since the last run"... Today I can do that with spark using checkpointing.

As a example , I would like to use the api
daft.read_deltalake(..) ...

But preferably would like to maintain a state so that the daft.read_deltalake would read from the last record from the previous daft.read_deltalake ....

I would see this as a utility on top of the api.

read_iceberg has the snapshot_id and I am assuming that I could use that .

[samster25]

Hi @vibh3s, We're currently building out checkpoint functionality in newest execution engine. We should be able to snapshot state of our execution engine and therefore recover or resume from prior checkpoints.