Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

draft netcdf2zarr tool #64

Open
wants to merge 6 commits into
base: master
Choose a base branch
from
Open

draft netcdf2zarr tool #64

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
33c2de6
draft netcdf2zarr tool
annefou Nov 2, 2023
4f7581d
try to add test
annefou Nov 2, 2023
de484c2
Update xarray_netcdf2zarr.xml
bgruening Nov 2, 2023
b41b5f0
test against EU which has the new datatype
bgruening Nov 2, 2023
01612a7
try to comply with planemo lint
annefou Nov 2, 2023
fae7866
update tests for zarr
annefou Nov 2, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .github/workflows/pr.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,8 +16,8 @@ on:
- 'docs/**'
- '*'
env:
GALAXY_FORK: galaxyproject
GALAXY_BRANCH: release_23.1
GALAXY_FORK: usegalaxy-eu
GALAXY_BRANCH: release_23.1_europe
GALAXY_COPERNICUS_CDSAPIRC_KEY: ${{ secrets.GALAXY_COPERNICUS_CDSAPIRC_KEY }}
COPERNICUS_CDSAPIRC_KEY_FILE: ${{ secrets.COPERNICUS_CDSAPIRC_KEY_FILE }}
MAX_CHUNKS: 4
Expand Down
10 changes: 10 additions & 0 deletions tools/netcdf2zarr/.shed.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
categories:
- Climate Analysis
description: netCDF to Zarr conversion
homepage_url: https://docs.xarray.dev/en/latest/generated/xarray.Dataset.to_zarr.html
long_description: |
This tool is a wrapper for converting netCDF data into Zarr.
name: netcdf2zarr
owner: climate
remote_repository_url: https://github.com/NordicESMhub/galaxy-tools/tree/master/tools/netcdf2zarr
type: unrestricted
5 changes: 5 additions & 0 deletions tools/netcdf2zarr/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@

# netCDF to Zarr conversion

This tool allows you to generate NC Zarr from a netCDF file.

Empty file added tools/netcdf2zarr/test-data/output.dat
View file
Open in desktop
Empty file.
Binary file added tools/netcdf2zarr/test-data/output.nc
View file
Open in desktop
Binary file not shown.
9 changes: 9 additions & 0 deletions tools/netcdf2zarr/test-data/output.zarr/.zattrs
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
{
"ANALYSIS": "Europe, 20210616+[0H_0H]",
"history": "Model ENSEMBLE ANALYSIS",
"institution": "Data produced by Meteo France",
"project": "MACC-RAQ (http://macc-raq.gmes-atmosphere.eu)",
"source": "Data from ENSEMBLE model",
"summary": "ENSEMBLE model hourly ANALYSIS of NO2 concentration at the Surface from 20210616+[0H_0H] on Europe",
"title": "NO2 Air Pollutant ANALYSIS at the Surface"
}
3 changes: 3 additions & 0 deletions tools/netcdf2zarr/test-data/output.zarr/.zgroup
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
{
"zarr_format": 2
}
163 changes: 163 additions & 0 deletions tools/netcdf2zarr/test-data/output.zarr/.zmetadata
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,163 @@
{
"metadata": {
".zattrs": {
"ANALYSIS": "Europe, 20210616+[0H_0H]",
"history": "Model ENSEMBLE ANALYSIS",
"institution": "Data produced by Meteo France",
"project": "MACC-RAQ (http://macc-raq.gmes-atmosphere.eu)",
"source": "Data from ENSEMBLE model",
"summary": "ENSEMBLE model hourly ANALYSIS of NO2 concentration at the Surface from 20210616+[0H_0H] on Europe",
"title": "NO2 Air Pollutant ANALYSIS at the Surface"
},
".zgroup": {
"zarr_format": 2
},
"latitude/.zarray": {
"chunks": [
1
],
"compressor": {
"blocksize": 0,
"clevel": 5,
"cname": "lz4",
"id": "blosc",
"shuffle": 1
},
"dtype": "<f4",
"fill_value": "NaN",
"filters": null,
"order": "C",
"shape": [
1
],
"zarr_format": 2
},
"latitude/.zattrs": {
"_ARRAY_DIMENSIONS": [
"latitude"
],
"long_name": "latitude",
"units": "degrees_north"
},
"level/.zarray": {
"chunks": [
1
],
"compressor": {
"blocksize": 0,
"clevel": 5,
"cname": "lz4",
"id": "blosc",
"shuffle": 1
},
"dtype": "<f4",
"fill_value": "NaN",
"filters": null,
"order": "C",
"shape": [
1
],
"zarr_format": 2
},
"level/.zattrs": {
"_ARRAY_DIMENSIONS": [
"level"
],
"long_name": "level",
"units": "m"
},
"longitude/.zarray": {
"chunks": [
2
],
"compressor": {
"blocksize": 0,
"clevel": 5,
"cname": "lz4",
"id": "blosc",
"shuffle": 1
},
"dtype": "<f4",
"fill_value": "NaN",
"filters": null,
"order": "C",
"shape": [
2
],
"zarr_format": 2
},
"longitude/.zattrs": {
"_ARRAY_DIMENSIONS": [
"longitude"
],
"long_name": "longitude",
"units": "degrees_east"
},
"no2_conc/.zarray": {
"chunks": [
1,
1,
1,
2
],
"compressor": {
"blocksize": 0,
"clevel": 5,
"cname": "lz4",
"id": "blosc",
"shuffle": 1
},
"dtype": "<f4",
"fill_value": -999.0,
"filters": null,
"order": "C",
"shape": [
1,
1,
1,
2
],
"zarr_format": 2
},
"no2_conc/.zattrs": {
"_ARRAY_DIMENSIONS": [
"time",
"level",
"latitude",
"longitude"
],
"species": "Nitrogen Dioxide",
"standard_name": "mass_concentration_of_nitrogen_dioxide_in_air",
"units": "\u00b5g/m3",
"value": "hourly values"
},
"time/.zarray": {
"chunks": [
1
],
"compressor": {
"blocksize": 0,
"clevel": 5,
"cname": "lz4",
"id": "blosc",
"shuffle": 1
},
"dtype": "<f4",
"fill_value": "NaN",
"filters": null,
"order": "C",
"shape": [
1
],
"zarr_format": 2
},
"time/.zattrs": {
"_ARRAY_DIMENSIONS": [
"time"
],
"long_name": "ANALYSIS time from 20210616",
"units": "hours"
}
},
"zarr_consolidated_format": 1
}
20 changes: 20 additions & 0 deletions tools/netcdf2zarr/test-data/output.zarr/latitude/.zarray
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
{
"chunks": [
1
],
"compressor": {
"blocksize": 0,
"clevel": 5,
"cname": "lz4",
"id": "blosc",
"shuffle": 1
},
"dtype": "<f4",
"fill_value": "NaN",
"filters": null,
"order": "C",
"shape": [
1
],
"zarr_format": 2
}
7 changes: 7 additions & 0 deletions tools/netcdf2zarr/test-data/output.zarr/latitude/.zattrs
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
{
"_ARRAY_DIMENSIONS": [
"latitude"
],
"long_name": "latitude",
"units": "degrees_north"
}
Binary file added tools/netcdf2zarr/test-data/output.zarr/latitude/0
View file
Open in desktop
Binary file not shown.
20 changes: 20 additions & 0 deletions tools/netcdf2zarr/test-data/output.zarr/level/.zarray
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
{
"chunks": [
1
],
"compressor": {
"blocksize": 0,
"clevel": 5,
"cname": "lz4",
"id": "blosc",
"shuffle": 1
},
"dtype": "<f4",
"fill_value": "NaN",
"filters": null,
"order": "C",
"shape": [
1
],
"zarr_format": 2
}
7 changes: 7 additions & 0 deletions tools/netcdf2zarr/test-data/output.zarr/level/.zattrs
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
{
"_ARRAY_DIMENSIONS": [
"level"
],
"long_name": "level",
"units": "m"
}
Binary file added tools/netcdf2zarr/test-data/output.zarr/level/0
View file
Open in desktop
Binary file not shown.
20 changes: 20 additions & 0 deletions tools/netcdf2zarr/test-data/output.zarr/longitude/.zarray
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
{
"chunks": [
2
],
"compressor": {
"blocksize": 0,
"clevel": 5,
"cname": "lz4",
"id": "blosc",
"shuffle": 1
},
"dtype": "<f4",
"fill_value": "NaN",
"filters": null,
"order": "C",
"shape": [
2
],
"zarr_format": 2
}
7 changes: 7 additions & 0 deletions tools/netcdf2zarr/test-data/output.zarr/longitude/.zattrs
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
{
"_ARRAY_DIMENSIONS": [
"longitude"
],
"long_name": "longitude",
"units": "degrees_east"
}
Binary file added tools/netcdf2zarr/test-data/output.zarr/longitude/0
View file
Open in desktop
Binary file not shown.
26 changes: 26 additions & 0 deletions tools/netcdf2zarr/test-data/output.zarr/no2_conc/.zarray
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
{
"chunks": [
1,
1,
1,
2
],
"compressor": {
"blocksize": 0,
"clevel": 5,
"cname": "lz4",
"id": "blosc",
"shuffle": 1
},
"dtype": "<f4",
"fill_value": -999.0,
"filters": null,
"order": "C",
"shape": [
1,
1,
1,
2
],
"zarr_format": 2
}
12 changes: 12 additions & 0 deletions tools/netcdf2zarr/test-data/output.zarr/no2_conc/.zattrs
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
{
"_ARRAY_DIMENSIONS": [
"time",
"level",
"latitude",
"longitude"
],
"species": "Nitrogen Dioxide",
"standard_name": "mass_concentration_of_nitrogen_dioxide_in_air",
"units": "\u00b5g/m3",
"value": "hourly values"
}
Binary file added tools/netcdf2zarr/test-data/output.zarr/no2_conc/0.0.0.0
View file
Open in desktop
Binary file not shown.
20 changes: 20 additions & 0 deletions tools/netcdf2zarr/test-data/output.zarr/time/.zarray
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
{
"chunks": [
1
],
"compressor": {
"blocksize": 0,
"clevel": 5,
"cname": "lz4",
"id": "blosc",
"shuffle": 1
},
"dtype": "<f4",
"fill_value": "NaN",
"filters": null,
"order": "C",
"shape": [
1
],
"zarr_format": 2
}
7 changes: 7 additions & 0 deletions tools/netcdf2zarr/test-data/output.zarr/time/.zattrs
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
{
"_ARRAY_DIMENSIONS": [
"time"
],
"long_name": "ANALYSIS time from 20210616",
"units": "hours"
}
Binary file added tools/netcdf2zarr/test-data/output.zarr/time/0
View file
Open in desktop
Binary file not shown.
Empty file added tools/netcdf2zarr/test-data/timeseries.dat
View file
Open in desktop
Empty file.
Binary file added tools/netcdf2zarr/test-data/timeseries.nc
View file
Open in desktop
Binary file not shown.
1 change: 1 addition & 0 deletions tools/netcdf2zarr/test-data/timeseries.zarr/.zattrs
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{}
3 changes: 3 additions & 0 deletions tools/netcdf2zarr/test-data/timeseries.zarr/.zgroup
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
{
"zarr_format": 2
}
Loading