diff --git a/_quarto.yml b/_quarto.yml index 8100c33..0e0d564 100644 --- a/_quarto.yml +++ b/_quarto.yml @@ -87,6 +87,10 @@ website: contents: - text: ArcGIS in Python href: topics-2024/2024-04-05-ArcGIS/index.qmd + - text: Explore CMIP6 Data + href: topics-2024/2024-04-12-cmip6/index.qmd + - text: Parallel Computing + href: topics-2024/2024-04-19-dask/index.qmd format: html: theme: diff --git a/docs/index.html b/docs/index.html index dc429f9..730ebc0 100644 --- a/docs/index.html +++ b/docs/index.html @@ -251,7 +251,7 @@

echopype. Wu-Jung Lee from UW/APL (Python) Set-up
  • April 12th. Exploring CMIP6 data with pangeo Python tools. Eli Holmes. Pythia CMIP6 Cookbook
    • -
  • April 5th. Using ArcGIS via the arcgis Python API. Tim Haverland from OST/DisMap.
    • +
  • April 5th. Using ArcGIS via the arcgis Python API. Tim Haverland from OST/DisMap.
  • March 29th. Accessing acoustics data from AWS Open Data (echopype examples). Eli Holmes (Python)
  • March 22nd. 1pm PT CoastWatch tutorials in R. Sunny Hospital and Daisy Shi from CoastWatch (R)
  • March 15th. CoastWatch tutorials. Sunny Hospital and Dale Robinson from CoastWatch (Python)
    • diff --git a/docs/search.json b/docs/search.json index dedde82..f905398 100644 --- a/docs/search.json +++ b/docs/search.json @@ -6,6 +6,27 @@ "section": "Select an event to the left", "text": "Select an event to the left\n\nApril 19th. More acoustics data with echopype. Wu-Jung Lee (Python) Set-up\nApril 12th. Exploring CMIP6 data with pangeo Python tools. Eli Holmes. Pythia CMIP6 Cookbook\nApril 5th. Using ArcGIS via the arcgis Python API. Tim Haverland from OST/DisMap.\nMarch 29th. Accessing acoustics data from AWS Open Data (echopype examples). Eli Holmes (Python)\nMarch 22nd. 1pm PT CoastWatch tutorials in R. Sunny Hospital and Daisy Shi from CoastWatch (R)\nMarch 15th. CoastWatch tutorials. Sunny Hospital and Dale Robinson from CoastWatch (Python)\nMarch 8th. Using precipitation estimates from IMERG tutorial Eli Holmes (Python)" }, + { + "objectID": "topics-2024/2024-05-03-ocean-color/Read_plot_L2_chlorophyll_map_with_xarray.html", + "href": "topics-2024/2024-05-03-ocean-color/Read_plot_L2_chlorophyll_map_with_xarray.html", + "title": "NOAA Fisheries HackHours", + "section": "", + "text": "Author: Guoqing Wang (guoqing.wang@nasa.gov), OB.DAAC Scientist, NASA GSFC; SSAI\nRead Chlorophyll data from standard L2 file into xarray, and plot it into a map Xarray is a powerful Python library designed for working with labeled multi-dimensional arrays. It is particularly useful for handling NetCDF files. Matplotlib is a popular Python library for creating static, animated, and interactive visualizations in a variety of formats. It is widely used for data visualization and plotting. Cartopy is a Python library designed for geospatial data visualization on maps. It is built on top of Matplotlib and provides tools for creating maps, handling projections, and adding geographical features. This turtorial shows an example of using Xarray to read ocean color level 2 file and plot the chlorophyll map with matplotlib and cartopy.\n1. Import packages \n\nimport warnings\nwarnings.filterwarnings(\"ignore\")\n\n\n# Imports\nimport netCDF4 as nc # pip install netCDF4\nimport xarray as xr # pip install xarray\nimport os\nimport numpy as np\nimport matplotlib.pyplot as plt\n\nimport cartopy\nfrom cartopy.mpl.ticker import LongitudeFormatter, LatitudeFormatter\n\n2. To read file with different groups, need to find out what are the groups and what variables are inside each group.\n\n# Open the datasets with xarray\nfilename = 'AQUA_MODIS.20221007T175001.L2.OC.nc'\nds = xr.open_dataset(filename)\n# # Open with netcdf4 to check the groups and variables\n# f = nc.Dataset(filename, 'r')\n# print(f)\n# # print grouped variables\n# print(f.groups.keys())\n# print(f.groups['geophysical_data'].variables)\n# # if you just want to see the variable names, use the following code\n# print(f.groups['geophysical_data'].variables.keys())\n\n# Load attributes\nds_geo = xr.open_dataset(filename, group='geophysical_data') \nds_nav = xr.open_dataset(filename, group='navigation_data')\n\n\n# Use .data_vars to find variable names\n# ds['navigation_data'].data_vars\n# ds['geophysical_data'].data_vars\n\n# Read chlor_a, latitude, and longitude data from their respective groups\nlat = ds_nav.latitude\nlon = ds_nav.longitude\nchlor_a = ds_geo.chlor_a\nchlor = np.log10(chlor_a)\n\n3. Plot the chlorophyll-a data\n\nUse matplotlib\n\n\nplt.pcolormesh(lon,lat,np.squeeze(chlor),cmap = 'jet', vmin=-2, vmax=1.3)\nplt.title(ds.time_coverage_start)\nplt.xlabel('Longitude')\nplt.ylabel('Latitude')\nplt.colorbar(label='log10(Chlorophyll-a)',)\n\nplt.show()\n\n\n\n\n\nUse matplotlib and cartopy to plot map with basemap\n\n\n# Plot Chlor_a data\nfig = plt.figure(figsize=(10, 7))\nmap_projection = cartopy.crs.PlateCarree()\nax = plt.axes(projection=map_projection)\n\nim = ax.pcolormesh(lon, lat,np.squeeze(chlor),cmap = 'jet', vmin=-2, vmax=1.3)\nax.coastlines()\nax.add_feature(cartopy.feature.STATES, linewidth=0.5)\n\nax.set_xticks(np.linspace(-85, -55, 5), crs=map_projection)\nax.set_yticks(np.linspace(26, 48, 5), crs=map_projection)\nlon_formatter = LongitudeFormatter(zero_direction_label=True)\nlat_formatter = LatitudeFormatter()\nax.xaxis.set_major_formatter(lon_formatter)\nax.yaxis.set_major_formatter(lat_formatter)\n\nplt.colorbar(im, label='log10(Chlorophyll-a)')\nplt.title(ds.time_coverage_start)\n\nText(0.5, 1.0, '2022-10-07T17:50:01.477Z')" + }, + { + "objectID": "topics-2024/2024-05-03-ocean-color/Download_read_visualize_L2_products.html", + "href": "topics-2024/2024-05-03-ocean-color/Download_read_visualize_L2_products.html", + "title": "NOAA Fisheries HackHours", + "section": "", + "text": "Author: Guoqing Wang (guoqing.wang@nasa.gov), OB.DAAC Scientist, NASA GSFC; SSAI\nDownload, read, and visualize level-2 ocean color data\nThis tutorial shows an example of downloading, reading, and plot OB.DAAC standard level 2 ocean color file.\n\n# !pip install cartopy\n# !pip install netcdf4\n\n\nimport warnings\nwarnings.filterwarnings(\"ignore\")\n\n\nfrom matplotlib import pyplot as plt\nimport pandas as pd\n\nimport numpy as np\nimport urllib.request\nimport netCDF4 as nc\nimport cartopy #!pip install cartopy\nfrom cartopy.mpl.ticker import LongitudeFormatter, LatitudeFormatter\n\n1. Download file In this example, I will use the MODIS_AQUA L2 OC file on April 30, 2023 (AQUA_MODIS.20221007T175001.L2.OC.nc).\n\nSpecify your appkey Generate appkey for your Earthdata login credentials and update the variable “key” correspondingly.\n\n\nkey = 'abcd1234' # paste your appkey here.\nfilename = 'AQUA_MODIS.20221007T175001.L2.OC.nc' # put here the file name to download\nurls = r'https://oceandata.sci.gsfc.nasa.gov/ob/getfile/%s?appkey=%s'%(filename, key)\n\nurllib.request.urlretrieve(urls, filename) # download the file to the current path\n\n('AQUA_MODIS.20221007T175001.L2.OC.nc',\n <http.client.HTTPMessage at 0x7f3293265180>)\n\n\n2. Loading the Level-2 Dataset NASA ocean color data are stored in NetCDF files. They can be read with a few different Python modules. The most popular are netCDF4 and gdal. For this script we’ll focus on netCDF4. Loading a dataset is simple, just pass a NetCDF file path to netCDF4.Dataset().\n\n# open file and print the metadata\nf = nc.Dataset(filename, 'r')\n\nA NetCDF file consists of three fundamental components: metadata, dimensions, and variables. Variables encompass both metadata and data. The netCDF4 library enables us to retrieve the metadata and data linked to a NetCDF file.\n3. Retrieve Metadata When we print the dataset f, it provides details about the variables present in the file as well as the groups of variables.\n\nprint (f)\n\n<class 'netCDF4._netCDF4.Dataset'>\nroot group (NETCDF4 data model, file format HDF5):\n title: MODISA Level-2 Data\n product_name: AQUA_MODIS.20221007T175001.L2.OC.nc\n processing_version: R2022.0\n equatorCrossingLongitude: 107.35993\n orbit_number: 108659\n history: l2gen par=/data19/sdpsoper/vdc/vpu38/workbuf/A2022280175000.L1B_LAC.param metafile=AQUA_MODIS.20221007T175001.L2.OC.nc.meta\n instrument: MODIS\n platform: Aqua\n Conventions: CF-1.6 ACDD-1.3\n license: https://science.nasa.gov/earth-science/earth-science-data/data-information-policy/\n naming_authority: gov.nasa.gsfc.sci.oceandata\n id: R2022.0/L2/AQUA_MODIS.20221007T175001.L2.OC.nc\n date_created: 2022-11-29T16:08:55.000Z\n keywords_vocabulary: NASA Global Change Master Directory (GCMD) Science Keywords\n keywords: Earth Science > Oceans > Ocean Optics > Ocean Color\n standard_name_vocabulary: CF Standard Name Table v36\n institution: NASA Goddard Space Flight Center, Ocean Ecology Laboratory, Ocean Biology Processing Group\n creator_name: NASA/GSFC/OBPG\n creator_email: data@oceancolor.gsfc.nasa.gov\n creator_url: https://oceandata.sci.gsfc.nasa.gov\n project: Ocean Biology Processing Group (NASA/GSFC/OBPG)\n publisher_name: NASA/GSFC/OBPG\n publisher_url: https://oceandata.sci.gsfc.nasa.gov\n publisher_email: data@oceancolor.gsfc.nasa.gov\n identifier_product_doi_authority: http://dx.doi.org\n identifier_product_doi: 10.5067/AQUA/MODIS/L2/OC/2022\n processing_level: L2\n cdm_data_type: swath\n equatorCrossingDateTime: 2022-10-07T18:31:34.643Z\n spatialResolution: 1 km\n time_coverage_start: 2022-10-07T17:50:01.477Z\n time_coverage_end: 2022-10-07T17:54:59.854Z\n start_center_longitude: -66.70454\n start_center_latitude: 28.26412\n end_center_longitude: -72.15332\n end_center_latitude: 46.206642\n northernmost_latitude: 47.412445\n southernmost_latitude: 26.05069\n easternmost_longitude: -54.84424\n westernmost_longitude: -86.26568\n geospatial_lat_units: degrees_north\n geospatial_lon_units: degrees_east\n geospatial_lat_max: 47.412445\n geospatial_lat_min: 26.05069\n geospatial_lon_max: -54.84424\n geospatial_lon_min: -86.26568\n startDirection: Ascending\n endDirection: Ascending\n day_night_flag: Day\n earth_sun_distance_correction: 1.001212239265442\n dimensions(sizes): number_of_lines(2030), pixels_per_line(1354), bands_per_pixel(16), number_of_reflectance_location_values(10), pixel_control_points(1354), number_of_bands(24), number_of_reflective_bands(16)\n variables(dimensions): \n groups: sensor_band_parameters, scan_line_attributes, geophysical_data, navigation_data, processing_control\n\n\nAbove you can see information for the file format, data source, data version, citation, dimensions, and variables. In L2 ocean color data, the variables are put in different groups, the ones that we are interested in, such as Rrs, Chla, etc. are in “geophysical_data”, and lat, lon are in “navigation_data”\n\n# print grouped variables\nprint(f.groups.keys())\n\ndict_keys(['sensor_band_parameters', 'scan_line_attributes', 'geophysical_data', 'navigation_data', 'processing_control'])\n\n\n4. Dimensions Accessing dimensions is akin to accessing file metadata. Each dimension is stored as a dimension class that holds relevant information. To retrieve metadata for all dimensions, one can loop through all the available dimensions, as demonstrated below.\n\nfor dim in f.dimensions.values():\n print(dim)\n \n# Individual dimensions are accessed like so: f.dimensions['x'].\n\n<class 'netCDF4._netCDF4.Dimension'>: name = 'number_of_lines', size = 2030\n<class 'netCDF4._netCDF4.Dimension'>: name = 'pixels_per_line', size = 1354\n<class 'netCDF4._netCDF4.Dimension'>: name = 'bands_per_pixel', size = 16\n<class 'netCDF4._netCDF4.Dimension'>: name = 'number_of_reflectance_location_values', size = 10\n<class 'netCDF4._netCDF4.Dimension'>: name = 'pixel_control_points', size = 1354\n<class 'netCDF4._netCDF4.Dimension'>: name = 'number_of_bands', size = 24\n<class 'netCDF4._netCDF4.Dimension'>: name = 'number_of_reflective_bands', size = 16\n\n\n5. Variable Metadata Access variable metadata in the groups of “geophysical_data” and “navigation_data”.\n\n#print(f.groups['geophysical_data'].variables)\n# if you just want to see the variable names, use the following code\nprint(f.groups['geophysical_data'].variables.keys())\n\ndict_keys(['aot_869', 'angstrom', 'Rrs_412', 'Rrs_443', 'Rrs_469', 'Rrs_488', 'Rrs_531', 'Rrs_547', 'Rrs_555', 'Rrs_645', 'Rrs_667', 'Rrs_678', 'chlor_a', 'Kd_490', 'pic', 'poc', 'ipar', 'nflh', 'par', 'l2_flags'])\n\n\n6. Access Data Values The actual precipitation data values are accessed by array indexing, and a numpy array is returned. All variable data is returned as follows:\n\n# metadata of variable: chlor_a\nprint(f.groups['geophysical_data'].variables['chlor_a'])\n\n<class 'netCDF4._netCDF4.Variable'>\nfloat32 chlor_a(number_of_lines, pixels_per_line)\n long_name: Chlorophyll Concentration, OCI Algorithm\n units: mg m^-3\n standard_name: mass_concentration_of_chlorophyll_in_sea_water\n _FillValue: -32767.0\n valid_min: 0.001\n valid_max: 100.0\n reference: Hu, C., Lee Z., and Franz, B.A. (2012). Chlorophyll-a algorithms for oligotrophic oceans: A novel approach based on three-band reflectance difference, J. Geophys. Res., 117, C01011, doi:10.1029/2011JC007395.\npath = /geophysical_data\nunlimited dimensions: \ncurrent shape = (2030, 1354)\nfilling on\n\n\n\n# value of chlor_a\nchlor_a = f.groups['geophysical_data'].variables['chlor_a'][:]\nprint(chlor_a)\n\n[[0.01556490734219551 0.01509102713316679 0.015974054113030434 ... --\n 0.20154109597206116 0.18869897723197937]\n [0.01652761548757553 0.015768131241202354 0.015580163337290287 ... -- --\n --]\n [0.015834644436836243 0.01573110558092594 0.01611599512398243 ... -- --\n --]\n ...\n [-- 2.553077220916748 1.7888909578323364 ... -- -- --]\n [-- 4.523899078369141 3.566077470779419 ... -- -- --]\n [-- 2.478635549545288 2.4827561378479004 ... -- -- --]]\n\n\n\n# read information of lat, lon\nprint(f.groups['navigation_data'].variables.keys())\nlat = f.groups['navigation_data'].variables['latitude'][:]\nlon = f.groups['navigation_data'].variables['longitude'][:]\nprint(lat.shape)\nprint(lon.shape)\nprint(chlor_a.shape)\nchlor = np.log10(chlor_a)\n\ndict_keys(['longitude', 'latitude', 'cntl_pt_cols', 'cntl_pt_rows', 'tilt'])\n(2030, 1354)\n(2030, 1354)\n(2030, 1354)\n\n\n7. Visualization and mapping Here, we use cartopy for basemap creation and matplotlib to plot data onto the created map. The combination of Matplotlib and Cartopy provides a powerful toolkit for creating geospatial visualizations.\n\n# Plot Chlor_a data\nfig = plt.figure(figsize=(10, 7))\nmap_projection = cartopy.crs.PlateCarree()\nax = plt.axes(projection=map_projection)\n\nim = ax.pcolormesh(lon, lat,np.squeeze(chlor),cmap = 'jet', vmin=-2, vmax=1.3)\nax.coastlines()\nax.add_feature(cartopy.feature.STATES, linewidth=0.5)\n\nax.set_xticks(np.linspace(-90, -52.5, 5), crs=map_projection)\nax.set_yticks(np.linspace(20, 50, 5), crs=map_projection)\nlon_formatter = LongitudeFormatter(zero_direction_label=True)\nlat_formatter = LatitudeFormatter()\nax.xaxis.set_major_formatter(lon_formatter)\nax.yaxis.set_major_formatter(lat_formatter)\n\nplt.colorbar(im, label='log10(Chlorophyll-a)',)" + }, + { + "objectID": "topics-2024/2024-04-19-dask/Parallel_compute_with_dask.html", + "href": "topics-2024/2024-04-19-dask/Parallel_compute_with_dask.html", + "title": "NOAA Fisheries HackHours", + "section": "", + "text": "# import needed modules\nimport time, random\n\n# define our functions\ndef inc(x):\n time.sleep(random.random())\n return x + 1\n\ndef dec(x):\n time.sleep(random.random())\n return x - 1\n\ndef add(x, y):\n time.sleep(random.random())\n return x + y\n\n\n%%time\n\n# a sequential example with no parallelization\nresults = []\nfor x in range(20):\n result = inc(x)\n result = dec(result)\n results.append(result)\n\nprint(results)\n\n[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19]\nCPU times: user 0 ns, sys: 3.51 ms, total: 3.51 ms\nWall time: 22.5 s\n\n\n\n# import dask for parallel work\nfrom dask.distributed import Client, LocalCluster\n\n\n# Set up our cluster with default workers and threads\ncluster = LocalCluster(processes=False)\ncluster\n\n\n\n\n\n%%time\n# Set up a client for work\nclient = cluster.get_client()\n\nresults = []\nfor x in range(20):\n result = client.submit(inc, x)\n result = client.submit(dec, result)\n results.append(result)\n\nresults = client.gather(results)\nprint(results)\nclient.close()\n\n[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19]\nCPU times: user 332 ms, sys: 12.6 ms, total: 344 ms\nWall time: 3.66 s\n\n\n\n# When we are done we can close our dask cluster\ncluster.close()\n\n\n# Set up a new cluster with default 8 workers and 1 thread per worker\ncluster = LocalCluster(n_workers=8, processes=False, threads_per_worker=1)\ncluster\n\n\n\n\n\n%%time\n# Set up a client for work\nclient = cluster.get_client()\n\nresults = []\nfor x in range(20):\n result = client.submit(inc, x)\n result = client.submit(dec, result)\n results.append(result)\n\nresults = client.gather(results)\nprint(results)\nclient.close()\n\n[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19]\nCPU times: user 549 ms, sys: 77.6 ms, total: 627 ms\nWall time: 3.11 s\n\n\n\n# When we are done we can close our dask cluster\ncluster.close()\n\n\nimport coiled # use a Coiled cluster\n\n\n# Set up cluster in the cloud where we can grab more workers\ncluster = coiled.Cluster(n_workers=20) # run on a cluster in the cloud\nclient = cluster.get_client()\n\n\n\n\nmyst-parser 0.18.1 has requirement mdit-py-plugins~=0.3.1, but you have mdit-py-plugins 0.4.0.\n\n\n\nPackage - myst-parser, Pip check had the following issues that need resolving: \nmyst-parser 0.18.1 has requirement mdit-py-plugins~=0.3.1, but you have mdit-py-plugins 0.4.0.\n\n\n\n\n\n\n╭──────────────────────────────── Package Info ────────────────────────────────╮\n│ ╷ │\n│ Package │ Note │\n│ ╶──────────────────────────┼───────────────────────────────────────────────╴ │\n│ coiled_local_jovyan │ Source wheel built from /home/jovyan │\n│ ╵ │\n╰──────────────────────────────────────────────────────────────────────────────╯\n\n\n\n╭────────────────────────── Not Synced with Cluster ───────────────────────────╮\n│ ╷ ╷ │\n│ Package │ Error │ Risk │\n│ ╶─────────────┼──────────────────────────────────────────────────┼─────────╴ │\n│ myst-parser │ Pip check had the following issues that need │ Warning │\n│ │ resolving: │ │\n│ │ myst-parser 0.18.1 has requirement │ │\n│ │ mdit-py-plugins~=0.3.1, but you have │ │\n│ │ mdit-py-plugins 0.4.0. │ │\n│ ╵ ╵ │\n╰──────────────────────────────────────────────────────────────────────────────╯\n\n\n\n\n\n\n\n\n\n\n%%time\nresults = []\nfor x in range(20): # scale 100x\n result = client.submit(inc, x)\n result = client.submit(dec, result)\n results.append(result)\n\nresults = client.gather(results)\nprint(results)\n\n[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19]\nCPU times: user 136 ms, sys: 11.4 ms, total: 148 ms\nWall time: 2.33 s\n\n\nSet up on coiled with free GCP (or AWS or Azure) * go to coiled.io * sign-in * Open a terminal and run\nconda deactivate\nconda create -n coiled -c conda-forge coiled google-cloud-sdk\nconda install -c conda-forge google-cloud-sdk -n notebook\nconda activate coiled\ncoiled setup gcp\n\n```\n(coiled) jovyan@jupyter-eeholmes:~$ coiled setup aws\n╭────────────────────────────────────────────────────────────────────────────────────────╮\n│ Introduction │\n│ │\n│ This uses your AWS credentials to set up Coiled. │\n│ │\n│ This will do the following ... │\n│ 1. Create limited IAM roles and grant them to Coiled │\n│ 2. Check and expand your AWS quota if needed │\n│ 3. Create initial resources to deploy clusters │\n│ │\n│ This will not ... │\n│ 1. Create resources that cost money │\n│ 2. Grant Coiled access to your data │\n╰────────────────────────────────────────────────────────────────────────────────────────╯\nMissing: You don't have local AWS credentials.\nThat's ok, you can run setup from AWS CloudShell.\n\nRun setup from AWS CloudShell with the following steps:\n\n1. Go to https://console.aws.amazon.com/cloudshell\n2. Sign in to your AWS account\n (if you usually switch role or profile, you should do this)\n3. Run the following command in CloudShell:\n\n pip3 install coiled && \\ \n coiled login \\ \n --token b2bfb562be4a457da2a74efa \\\n --token 42713203-d012864e09b94db \\\n --token b82f66c7c273c6913213a9fa9 && \\\n coiled setup aws --region us-east-1\n```\n\n\ncoiled setup aws --quotas\n\nSyntaxError: invalid syntax (3831637974.py, line 1)\n\n\n\n627/148\n\n4.236486486486487\n\n\n\n20/8\n\n2.5\n\n\n\ncluster\n\n\n\n\n╭─────────────────────────────── Coiled Cluster ───────────────────────────────╮\n│ https://cloud.coiled.io/clusters/445452?account=eeholmes │\n╰──────────────────────────────────────────────────────────────────────────────╯\n╭────────────── Overview ──────────────╮╭─────────── Configuration ────────────╮\n│ ││ │\n│ Name: eeholmes-ef8daa0e ││ Region: us-east-1 │\n│ ││ │\n│ Scheduler Status: started ││ Scheduler: m6i.xlarge │\n│ ││ │\n│ Dashboard: ││ Workers: m6i.xlarge (15), Unknown │\n│ https://cluster-qajgq.dask.host?toke ││ (5) │\n│ n=rKD-dj80QhjCSklb ││ │\n│ ││ Workers Requested: 20 │\n╰──────────────────────────────────────╯╰──────────────────────────────────────╯\n╭──────────────── Dask Worker States (2024/04/16 19:09:24 UTC) ────────────────╮\n│ Provisioning ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0 │\n│ Booting Instance ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0 │\n│ Downloading environment ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0 │\n│ Ready ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━ 15 │\n│ Stopping ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0 │\n│ Stopped ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0 │\n│ Error ━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 5 │\n╰──────────────────────────────────────────────────────────────────────────────╯" + }, { "objectID": "topics-2023/index.html#select-an-event-to-the-left", "href": "topics-2023/index.html#select-an-event-to-the-left", @@ -233,5 +254,53 @@ "crumbs": [ "Earth Data Access in Python" ] + }, + { + "objectID": "topics-2024/2024-04-19-dask/Parallel_compute_coiled.html", + "href": "topics-2024/2024-04-19-dask/Parallel_compute_coiled.html", + "title": "Run simple example with coiled", + "section": "", + "text": "This will spin up virtual machines on our behalf.\nSet-up\n\nGo to coiled.io and set-up an account\n\nYou need to associate it with a cloud account on AWS, Azure or GCP (Google). You can get a free 12 month trial on these.\nI find the Google Cloud dashboard more intuitive to use. But the set up for coiled was difficult. I had to do it from my computer using Python and then it asked for me to install Google Cloud sdk CLI. I stopped at that point.\nAWS was easier. I already had a AWS account. I clicked buttons on coiled.io, logged into AWS as instructed, and got it linked.\n\nNow back to the Jupyter notebook.\n\nGo to a terminal and make sure you are in the same conda environment that you will run your notebook in\nRun coiled setup aws. I assume you have coiled module installed. If not do pip install coiled.\nIt is going to send you to the AWS Cloud Shell and will give you a link for that.\nIt will give you some code to run in the shell. Note I had to clean up the code to remove some extra spaces. I couldn’t just paste and run.\nWhen it is good, it will say it is authenticated.\n\nNow run `coiled\n\nExample of the output\n(coiled) jovyan@jupyter-eeholmes:~$ coiled setup aws\n╭────────────────────────────────────────────────────────────────────────────────────────╮\n│ Introduction │\n│ │\n│ This uses your AWS credentials to set up Coiled. │\n│ │\n│ This will do the following ... │\n│ 1. Create limited IAM roles and grant them to Coiled │\n│ 2. Check and expand your AWS quota if needed │\n│ 3. Create initial resources to deploy clusters │\n│ │\n│ This will not ... │\n│ 1. Create resources that cost money │\n│ 2. Grant Coiled access to your data │\n╰────────────────────────────────────────────────────────────────────────────────────────╯\nMissing: You don't have local AWS credentials.\nThat's ok, you can run setup from AWS CloudShell.\n\nRun setup from AWS CloudShell with the following steps:\n\n1. Go to https://console.aws.amazon.com/cloudshell\n2. Sign in to your AWS account\n (if you usually switch role or profile, you should do this)\n3. Run the following command in CloudShell:\n\n pip3 install coiled && \\\n coiled login \\\n --token b2bfb56 \\\n --token 42713203-d \\\n --token b82f66c && \\\n coiled setup aws --region us-east-1\n\n# import needed modules\nimport time, random\n\n# define our functions\ndef inc(x):\n time.sleep(random.random())\n return x + 1\n\ndef dec(x):\n time.sleep(random.random())\n return x - 1\n\ndef add(x, y):\n time.sleep(random.random())\n return x + y\n\n\n%%time\n\n# a sequential example with no parallelization\nresults = []\nfor x in range(20):\n result = inc(x)\n result = dec(result)\n results.append(result)\n\nprint(results)\n\n[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19]\nCPU times: user 0 ns, sys: 3.51 ms, total: 3.51 ms\nWall time: 22.5 s\n\n\n\nimport coiled # use a Coiled cluster\n\nSet up cluster in the cloud where we can grab more workers. These 15 workers will cost about $0.05 a minute and this job from start (set-up) to finish is like 5 minutes so $0.25. Almost all the time is the set-up of the workers. You can go to your dashboard on coiled to see how much compute time you used up.\n\ncluster = coiled.Cluster(n_workers=15) # run on a cluster in the cloud\nclient = cluster.get_client()\n\n\n\n\nmyst-parser 0.18.1 has requirement mdit-py-plugins~=0.3.1, but you have mdit-py-plugins 0.4.0.\n\n\n\nPackage - myst-parser, Pip check had the following issues that need resolving: \nmyst-parser 0.18.1 has requirement mdit-py-plugins~=0.3.1, but you have mdit-py-plugins 0.4.0.\n\n\n\n\n\n\n╭──────────────────────────────── Package Info ────────────────────────────────╮\n│ ╷ │\n│ Package │ Note │\n│ ╶──────────────────────────┼───────────────────────────────────────────────╴ │\n│ coiled_local_jovyan │ Source wheel built from /home/jovyan │\n│ ╵ │\n╰──────────────────────────────────────────────────────────────────────────────╯\n\n\n\n╭────────────────────────── Not Synced with Cluster ───────────────────────────╮\n│ ╷ ╷ │\n│ Package │ Error │ Risk │\n│ ╶─────────────┼──────────────────────────────────────────────────┼─────────╴ │\n│ myst-parser │ Pip check had the following issues that need │ Warning │\n│ │ resolving: │ │\n│ │ myst-parser 0.18.1 has requirement │ │\n│ │ mdit-py-plugins~=0.3.1, but you have │ │\n│ │ mdit-py-plugins 0.4.0. │ │\n│ ╵ ╵ │\n╰──────────────────────────────────────────────────────────────────────────────╯\n\n\n\n\n\n\n\n\n\n\n%%time\nresults = []\nfor x in range(20): # scale 100x\n result = client.submit(inc, x)\n result = client.submit(dec, result)\n results.append(result)\n\nresults = client.gather(results)\nprint(results)\n\n[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19]\nCPU times: user 140 ms, sys: 1.54 ms, total: 141 ms\nWall time: 2.18 s\n\n\n\n# Close our cluster as soon as we are done because running it uses $\nclient.close()\ncluster.close()" + }, + { + "objectID": "topics-2024/2024-04-19-dask/index.html", + "href": "topics-2024/2024-04-19-dask/index.html", + "title": "Parallel Computing in Python", + "section": "", + "text": "Image: Py - Openscapes", + "crumbs": [ + "Parallel Computing" + ] + }, + { + "objectID": "topics-2024/2024-04-19-dask/index.html#get-the-tutorials", + "href": "topics-2024/2024-04-19-dask/index.html#get-the-tutorials", + "title": "Parallel Computing in Python", + "section": "Get the tutorials", + "text": "Get the tutorials\nNever cloned the NOAAHackDays repo?\ncd ~\ngit clone https://github.com/nmfs-opensci/NOAAHackDays\nHave cloned it but need to update? This is going to destroy any changes that you made to the repo to make it match the current state of the repo on GitHub.\ngit reset --hard origin/master", + "crumbs": [ + "Parallel Computing" + ] + }, + { + "objectID": "topics-2024/2024-05-03-ocean-color/L3_files_read_plot.html", + "href": "topics-2024/2024-05-03-ocean-color/L3_files_read_plot.html", + "title": "NOAA Fisheries HackHours", + "section": "", + "text": "Author: Guoqing Wang (guoqing.wang@nasa.gov), OB.DAAC Scientist, NASA GSFC; SSAI\nRead and plot NASA ocean color level-3 data - 8-day chlorophyll a, monthly Rrs(412)\nThis tutorial shows an example of reading and plotting a level-3 dataset.\n\n# !pip install netCDF4\n# !pip install cmocean\n\n\nimport warnings\nwarnings.filterwarnings(\"ignore\")\n\n\nfrom matplotlib import pyplot as plt\n# from mpl_toolkits.basemap import Basemap\nimport pandas as pd\nimport netCDF4 as nc\nimport numpy as np\nimport xarray as xr # pip install xarray\nimport urllib.request\n\nimport cartopy\nfrom cartopy.mpl.ticker import LongitudeFormatter, LatitudeFormatter\n\n1. Download file In this example, the image to be used is MODIS AQUA L3 8day averaged 4km chlorophyll image for Sep 13-20, 2016 - Specify your appkey Generate an appkey for your Earthdata login credentials and update the following variable “key” correspondingly.\n\nkey = 'abcd1234' # paste your appkey here\nfilename = ['AQUA_MODIS.20160913_20160920.L3m.8D.CHL.chlor_a.4km.nc',\n 'AQUA_MODIS.20201001_20201031.L3m.MO.RRS.Rrs_412.4km.nc'] \n# L3 chlorophyll 8-day 4km file\n# L3 Rrs(412) monthly mean 4km file\n\nfor file in filename:\n\n urls = r'https://oceandata.sci.gsfc.nasa.gov/ob/getfile/%s?appkey=%s'%(file, key)\n\n urllib.request.urlretrieve(urls, file) # download the file to the current path\n\n2. Read and plot Level-3 8-day chlorophyll a\n\n# Open the datasets with xarray\nfilename = 'AQUA_MODIS.20160913_20160920.L3m.8D.CHL.chlor_a.4km.nc'\nf = nc.Dataset(filename, 'r')\n# Load attributes\nds = xr.open_dataset(filename)\n\n\n# Use .data_vars to find variable names\n# ds.data_vars\n\n# read chlor_a, latitude, and longitude data \nlat = ds.chlor_a.lat\nlon = ds.chlor_a.lon\nchlor_a = ds.chlor_a.values\nchlor = np.log10(chlor_a)\n\n\n# Plot the chlorophyll-a data\n\nplt.pcolormesh(lon,lat,np.squeeze(chlor),cmap = 'jet', vmin=-2, vmax=1.3)\nplt.title(ds.time_coverage_start)\nplt.xlabel('Longitude')\nplt.ylabel('Latitude')\nplt.colorbar(label='Chlorophyll-a',)\n\nplt.show()\n\n\n\n\n3. Read and plot monthly Rrs(412)\n\n# Open the datasets with xarray\nfilename = 'AQUA_MODIS.20201001_20201031.L3m.MO.RRS.Rrs_412.4km.nc'\nf = nc.Dataset(filename, 'r')\n# Load attributes\nds = xr.open_dataset(filename)\n\n\n# # Use .data_vars to find variable names\n# ds.data_vars\n\n# read Rrs, latitude, and longitude data \nlat = ds.Rrs_412.lat\nlon = ds.Rrs_412.lon\nRrs_412 = ds.Rrs_412.values\nlogRrs = np.log10(Rrs_412)\n\nPlot Rrs(412) with matplotlib and cartopy\n\n# Plot Rrs(412) data\nfig = plt.figure(figsize=(10, 7))\nmap_projection = cartopy.crs.PlateCarree()\nax = plt.axes(projection=map_projection)\n\nim = ax.pcolormesh(lon, lat,np.squeeze(logRrs),cmap = 'jet', vmin=-3, vmax=-1)\nax.coastlines()\nax.add_feature(cartopy.feature.BORDERS, linewidth=0.5)\n\nax.set_xticks(np.linspace(-180, 180, 5), crs=map_projection)\nax.set_yticks(np.linspace(-90, 90, 5), crs=map_projection)\nlon_formatter = LongitudeFormatter(zero_direction_label=True)\nlat_formatter = LatitudeFormatter()\nax.xaxis.set_major_formatter(lon_formatter)\nax.yaxis.set_major_formatter(lat_formatter)\n\nplt.colorbar(im, label='log10(Rrs(412))')\nplt.title(ds.time_coverage_start)\n\nplt.show()" + }, + { + "objectID": "topics-2024/2024-05-03-ocean-color/index.html", + "href": "topics-2024/2024-05-03-ocean-color/index.html", + "title": "Ocean Color Mapping", + "section": "", + "text": "Image:" + }, + { + "objectID": "topics-2024/2024-05-03-ocean-color/index.html#get-the-tutorials", + "href": "topics-2024/2024-05-03-ocean-color/index.html#get-the-tutorials", + "title": "Ocean Color Mapping", + "section": "Get the tutorials", + "text": "Get the tutorials\nNever cloned the NOAAHackDays repo?\ncd ~\ngit clone https://github.com/nmfs-opensci/NOAAHackDays\nHave cloned it but need to update? This is going to destroy any changes that you made to the repo to make it match the current state of the repo on GitHub.\ngit reset --hard origin/master" } ] \ No newline at end of file diff --git a/docs/sitemap.xml b/docs/sitemap.xml index 720d6ad..698c24c 100644 --- a/docs/sitemap.xml +++ b/docs/sitemap.xml @@ -4,6 +4,22 @@ https://nmfs-opensci.github.io/NOAAHackDays/topics-2024/index.html 2024-04-15T03:11:07.762Z + + https://nmfs-opensci.github.io/NOAAHackDays/topics-2024/2024-05-03-ocean-color/Read_plot_L2_chlorophyll_map_with_xarray.html + 2024-04-16T22:19:51.914Z + + + https://nmfs-opensci.github.io/NOAAHackDays/topics-2024/2024-05-03-ocean-color/Download_read_visualize_L2_products.html + 2024-04-16T22:20:25.811Z + + + https://nmfs-opensci.github.io/NOAAHackDays/topics-2024/2024-04-19-dask/Parallel_compute_with_dask.html + 2024-04-23T01:19:00.083Z + + + https://nmfs-opensci.github.io/NOAAHackDays/topics-2024/2024-04-12-cmip6/index.html + 2024-04-23T01:25:04.385Z + https://nmfs-opensci.github.io/NOAAHackDays/topics-2024/2024-04-05-ArcGIS/index.html 2024-04-15T03:29:01.710Z @@ -42,7 +58,7 @@ https://nmfs-opensci.github.io/NOAAHackDays/index.html - 2024-04-16T22:08:21.108Z + 2024-04-16T22:10:23.762Z https://nmfs-opensci.github.io/NOAAHackDays/reuse.html @@ -76,4 +92,20 @@ https://nmfs-opensci.github.io/NOAAHackDays/topics-2024/2024-04-05-ArcGIS/netcdffromthredds03.html 2024-04-05T20:40:29.161Z + + https://nmfs-opensci.github.io/NOAAHackDays/topics-2024/2024-04-19-dask/Parallel_compute_coiled.html + 2024-04-19T19:08:07.219Z + + + https://nmfs-opensci.github.io/NOAAHackDays/topics-2024/2024-04-19-dask/index.html + 2024-04-23T01:30:53.745Z + + + https://nmfs-opensci.github.io/NOAAHackDays/topics-2024/2024-05-03-ocean-color/L3_files_read_plot.html + 2024-04-16T22:20:31.851Z + + + https://nmfs-opensci.github.io/NOAAHackDays/topics-2024/2024-05-03-ocean-color/index.html + 2024-04-23T01:18:23.986Z + diff --git a/docs/topics-2024/2024-04-05-ArcGIS/index.html b/docs/topics-2024/2024-04-05-ArcGIS/index.html index 242531d..a7045b2 100644 --- a/docs/topics-2024/2024-04-05-ArcGIS/index.html +++ b/docs/topics-2024/2024-04-05-ArcGIS/index.html @@ -30,6 +30,7 @@ + @@ -165,6 +166,18 @@ ArcGIS in Python + +
  • +
    + + Explore CMIP6 Data +
    +
    • +
  • +
    + + Parallel Computing +
    • @@ -782,6 +795,15 @@

    Get the tutorials

    } }); +