We've had reduced capacity this week with 3 people on leave, but have still seen some great progress!
WP1 have some devices and have started playing with them, we've been issuing certificates and trying to offer some form of self service to them. FDRI data is expected to start flowing in soon.
Last week we deleted nextjs and secrets manager, this week we were able to remove metabase and pnpm from our stack. It's great to consolidate and reduce what we managing but it might be tricky to keep up this pace of deletion..
As always our ui is here https://dri-ui.staging.eds.ceh.ac.uk/ (requires VPN)
We reorganised the code base ready for other projects to start collaborating in the same repo. This is the first time we are experimenting with a mono-repo approach and are excited for the NC-UK dev team to start contributing to the same repo so we can share knowledge more.
We also did some minor ui tweaks to make the cosmos site page charts a bit sleeker.
Last week we mentioned our api wasn't quite as performant as expected from our parquet/duckdb speed tests. Addressing this we were able to speed up the api performance by ~60% Without implementing caching!
There's also been some discovery on what a standards based API could look like. We attempted a bunch of different speed up ideas, and found increasing the number of duckdb threads manually to 64 and horizontally scaling the api to give the best result. Full results are on confluence
| Branch | run1 | run2 | run3 | run4 | run5 | mean | % change |
|---|---|---|---|---|---|---|---|
| main | 2.743 | 2.757 | 2.908 | 2.848 | 3.59 | 2.9692 | 0 |
| 64 duckdb threads | 1.664 | 1.82 | 1.61 | 1.641 | 1.752 | 1.6974 | 42.83308635 |
| 64 duckdb threads + 4 api deployments | 1.212 | 1.039 | 1.036 | 1.141 | 1.387 | 1.163 | 60.83120032 |
Brilliant progress one week in from the phenocam team, we have an s3 bucket setup, access to the images and almost have live phemocam images streaming into the s3 bucket. Still blocked waiting on IT to enable the red arrow in the diagram, but they have been working on this and expect it to be working very soon.
We are kicking off a book club on the dev team! We are still deciding on the book, currently designing data intensive applications is looking likely. Please reach out if you would like to be involved. The idea is we meet every 2 weeks to discuss a chapter of a book with no requirement to have read the chapter.
Wanted to give a shoutout to other weeknotes in the org. The rse team have some very nice open communication over on their github repo and are also doing weeknotes