Getting started¶

The xdatasets library enables users to effortlessly access a vast collection of earth observation datasets that are compatible with xarray formats.

The library adopts an opinionated approach to data querying and caters to the specific needs of certain user groups, such as hydrologists, climate scientists, and engineers. One of the functionalities of xdatasets is the ability to extract data at a specific location or within a designated region, such as a watershed or municipality, while also enabling spatial and temporal operations.

To use xdatasets, users must employ a query. For instance, a straightforward query to extract the variables t2m (2m temperature) and tp (Total precipitation) from the era5_reanalysis_single_levels dataset at two geographical positions (Montreal and Toronto) could be as follows:

query = {
    "datasets": {"era5_reanalysis_single_levels": {'variables': ["t2m", "tp"]}},
    "space": {
        "clip": "point", # bbox, point or polygon
        "geometry": {'Montreal' : (45.508888, -73.561668),
                     'Toronto' : (43.651070, -79.347015)
                    }
    }
}

An example of a more complex query would look like the one below.

Note Don’t worry! Below, you’ll find additional examples that will assist in understanding each parameter in the query, as well as the possible combinations.

This query calls the same variables as above. However, instead of specifying geographical positions, a GeoPandas.DataFrame is used to provide features (such as shapefiles or geojson) for extracting data within each of them. Each polygon is identified using the unique identifier Station, and a spatial average is computed within each one (aggregation: True). The dataset, initially at an hourly time step, is converted into a daily time step while applying one or more temporal aggregations for each variable as prescribed in the query. xdatasets ultimately returns the dataset for the specified date range and time zone.

query = {
    "datasets": {"era5_reanalysis_single_levels": {'variables': ["t2m", "tp"]}},
    "space": {
        "clip": "polygon", # bbox, point or polygon
        "averaging": True, # spatial average of the variables within each polygon
        "geometry": gdf,
        "unique_id": "Station" # unique column name in geodataframe
    },
    "time": {
        "timestep": "D",
        "aggregation": {"tp": np.nansum,
                        "t2m": [np.nanmax, np.nanmin]},

        "start": '2000-01-01',
        "end": '2020-05-31',
        "timezone": 'America/Montreal',
    },
}

Query climate datasets¶

In order to use xdatasets, you must import at least xdatasets, pandas, geopandas, and numpy. Additionally, we import pathlib to interact with files.

[1]:

import os
import warnings
from pathlib import Path

warnings.simplefilter("ignore")

os.environ["USE_PYGEOS"] = "0"
import geopandas as gpd

# Visualization
import hvplot.pandas  # noqa
import hvplot.xarray  # noqa-
import numpy as np
import pandas as pd
import panel as pn  # noqa

import xdatasets as xd

Clip on polygons with no averaging in space¶

First, let’s explore specific polygon features. With xdatasets, you can access geographical datasets, such as watershed boundaries linked to streamflow stations. These datasets follow a nomenclature where they are named after the hydrological dataset, with "_polygons" appended. For example, if the hydrological dataset is named deh, its corresponding watershed boundaries dataset will be labeled deh_polygons. The query below retrieves all polygons for the deh_polygons dataset.

gdf = xd.Query(
    **{
        "datasets": "deh_polygons"
}).data

gdf

As the data is loaded into memory, the process of loading all polygons may take some time. To expedite this, we recommend employing filters, as illustrated below. It’s important to note that the filters are consistent for both hydrological and corresponding geographical datasets. Consequently, only watershed boundaries associated with existing hydrological data will be returned.

[7]:

import xdatasets as xd

gdf = xd.Query(
    **{
        "datasets": {
            "deh_polygons": {
                "id": ["0421*"],
            }
        }
    }
).data.reset_index()

gdf

[7]:

	Station	Superficie	geometry
0	042102	623.479187	POLYGON ((-78.5712 46.70742, -78.57112 46.7073...
1	042103	579.479614	POLYGON ((-78.49014 46.64514, -78.4901 46.6450...

Let’s examine the geographic locations of the polygon features.

[8]:

gdf.hvplot(
    geo=True,
    tiles="ESRI",
    color="Station",
    alpha=0.8,
    width=750,
    height=450,
    legend="top",
    hover_cols=["Station", "Superficie"],
)

[8]:

Getting started¶

Query climate datasets¶

Clip by points (sites)¶

Clip on polygons with no averaging in space¶