I thought this would be a short post, but loads happened this week! This is the second installment of fdri weeknotes, an effort to start sharing what we are working on more widely. Please feel free to reach out to domgin@ceh.ac.uk (also on teams) or anybody else in the FDRI project if anything here sounds interesting!
More duckdb bugs, issues coming from using duckdb in an api which doesn't seem to be a standard use case. Seems fixed for now, but the workarounds are adding up...
- Lots of ongoing Metadata discovery, this week was more on integrating configs into the time series pipeline
- Epimorphics discovery on api development proposal . Sensor things api spec discovery
We've started benchmarking the timeseries model aggregation speeds these tests are now baked into our CI, to help us keep track of when we make it faster/slower.
Started work on speeding up the dri-data-api . We have been seeing speeds a bit slower than our parquet performance testing. Again this could be a symptom of our tech choice of using duckb in an api, so far we have managed a 2x speedup of the api.
✨ UI
https://dri-ui.staging.eds.ceh.ac.uk/ (requires VPN to access)
Removed nextjs from the ui. We found the developer experience of using nextjs very poor, as noted in this open issue . We are also not using any of the features of nextjs beyond the router, so despite nextjs being the recommended way to start a new react project have replaced it with vite and react-router
New developer experience is instant compile times 🚀
| Title | Size |
|---|---|
| Nextjs Bundle Size | 7.4MB |
| New Bundle Size | 1.7MB |
That's a 77% decrease!
💸 removed secrets manager, paying for it and not using the features, switched to parameter store 🧊 Scaled up the maximum underlying nodes in our eks cluster to support more ongoing development
🧑🤝🧑 Relationship manager in timeseries model, so that additional context or metadata for the data can be tracked alongside the column.
Workstream has kicked off, JIRA Waiting on IT to push the phenocam images to a server where we can write code
Diagram of our plan up til the end of march, we are planning to start streaming the images into an AWS S3 bucket and updating the api and ui, so they can be presented in a gallery view.
- Metadata service and catalogue boundary discussions have started. Discovery ongoing, diagrams to come soon.