Overview

This notebook includes guidelines for the Agriculture Adaptation Atlas’ collection of Data Exploration Notebooks. It suggests best practices for consistent styling, narrative structure, and example cells that can be used as the Alliance Bioversity team and the community creates their own notebooks.

Please note: this should be considered a living document that will be added to and adjusted as needed.

Creating a Narrative

To create a narrative for your Observable Notebook, we suggest following a general story arc and structure:

  1. Hero Image (with title). A succinct title capturing the premise of the Notebook. See below for additional information on creating a hero image with help from the Alliance Bioversity team.
  2. Introductory Paragraph. Introductory framing text for the Notebook.
  3. Narrative Sections. 3-4 sections that convey a narrative. They include headers, descriptive text, data presentations, and dynamic insights (if applicable).
  • 1-2 data presentations/visualizations per each section. As applicable, controls can be global and be synced across all sections
  • Brief descriptive text, explaining the section takeaway and any context about the chart.
  • If applicable, a set of dynamic Insights that update based on user selection of various controls of the charts, located below the chart
  1. Summary Paragraph. Summary text for the Notebook. This section may include
  • Higher-level dynamic insights that summarize the key takeaways of the Notebook
  • Next steps or actions users can take such as hyperlinks to additional resources.
  1. Methods & Sources. Methods & sources for the Notebook.
  2. Appendix. This large section includes any cells that power the Notebook. This should exist at the bottom of the Notebook to mitigate any disruption to the narrative.

Please reach out to us if you need help crafting your narrative, and we’d be happy to help.

Notebook Elements

Headers

We recommend using title case for headers. Main section headings should be a question which is answered by the narrative and figures in the following section. For example, instead of “Identifying Risk Hotspots”, consider using “Where Is Most At Risk?”

To best format the sections within the narrative, please use header tags:

  • Adding “#” before text creates an H1 header. Use it for headings of main sections.
  • Adding “##” before text creates an H2 header. Use it for chart titles.
  • Adding “###” before text creates an H3 header. Use it for subtitles within charts and for titles for data insights related to charts.

Because our content requires dynamic headers for translation, the standard Quarto header syntax needs a slight adjustment. There are several ways to create dynamic headers, but the method described here is the most straightforward and closely aligned with standard Quarto formatting:

```{ojs}
heading1 = _lang({en: "Heading 1", fr: "Titre 1"}) // define ojs var for heading
```

# ``{ojs} heading1` {#example-heading2} <!-- Use the ojs var as a heading & add an id-->

Note that the id for the heading is assigned. This is important for the TOC generation and is required.

Table of Contents

We have put together a table of contents function that should be used across notebooks. This can be imported from the ‘helpers’ directory. Below is an example of its use:

Show the code
import { atlasTOC } from "/helpers/uiComponents.ojs"

toc_bottom = atlasTOC({
  skip: ['notebook-title', 'appendix', 'source-code'], // heading ids to skip
  heading: `<b>In This Notebook</b>`,
  selector: "h1", // Selectors to include (e.g., "h1,h2,h3")
})

htl.html`
      <div class='floating-toc'>
      ${toc_bottom}
      </div>
      `

Notice that by defining the id’s for each header, we can specify which ones to ignore in the TOC. The function defaults to only show h1 headers but this can be adjusted by changing the selector variable, but in general it should stay as h1. You can copy the code above to use in notebooks and it should work with no modifications.

Dynamic Insights

As can be seen in the “What’s at Risk” notebook, we have developed dynamic insights within the text that changes based on the selection of various controls throughout the tool. To emphasize that this text is dynamic, we bold the dynamic portion of the sentence.

Dynamic Insights are to be placed in a subsection with an H3 header ““###”” below each visualization at the end of each main section.

These dynamic insights can be in partial or full sentence form, and in either an bulleted list or not (could vary depending on the specific insight).

Color Scales

Below are some example linear color scales you can use, all defined by endpoint color values.

Please note that color scales may need to be designed on a chart-by-chart basis. The color scales listed below are a few developed for various charts in the “What’s at Risk?” and “On-farm Solutions for Today” notebooks.

Regarding the direction to apply the scales, we have been mapping green (from the yellowGreen scale) to normatively “better” values and red (from the orangeRed scale) to normatively “worse” values.

Visualizations

To ensure a good user experience, all plots, maps, and figures within the notebooks should have a consistent fixed height to prevent page shifts during interactive changes. A maximum height of 800px is recommended, although it can be adjusted based on specific design requirements. The height parameter should be set, especially for maps, where different admins will have different sizes, resulting in large content shifts if not set. A maximum width of 1000px is suggested. Text should be large enough to be read with ease.

Most previous Data Exploration Notebooks have used the Observable Plot library for creating visualizations. Examples using this library can be found here or within the Atlas here. Some also use d3 to create more complex or interactive visualizations. Any JS plotting library should work fine so long as they allow:

  • Dynamic Plots: Plots should adapt dynamically based on user inputs such as region or crop selection, provided no data is unavailable.
  • Tooltips for Additional Information: Tooltips can enhance understanding but should not overwhelm the main plot. Plots should generally convey information succinctly and effectively.

Data Download buttons

Every figure should include a data download button that can be used to download the data used in the figure. For consistency and ease, there is a function in the /helpers folder that can be used to create the download button.

```{ojs}
import { downloadButton } from "/helpers/uiComponents.ojs"
data = [{a: 1, b: 2}, {a: 3, b: 4}, {a: 5, b: 6}]
downloadButton(data, filename, label = "Download")

```

Selectors

The Observable standard library comes with many inputs/selectors pre-included. These should be used when possible, rather than building selectors from scratch or importing them from elsewhere. Documentation on these selectors can be found here with many examples on ObservableHQ. We use a custom CSS to modify the styling of these inputs, these should be automatically applied on quarto render and preview.

When multiple selectors are used together - such as to select multiple admin levels - they should be grouped together in a single horizontal block using an Inputs.form, or other methods where necessary. When multiple selectors are used across a notebook to select the same variable in different places, there should be one master selector that is updated when other selectors are used (using Inputs.bind()). The below example shows how to do this, along with how to use the Inputs.form selector to group selectors together:

Images

Each notebook should include a single hero image in .webp format, optimized for minimal file size without noticeable quality loss. Save the image in the /images directory using the notebook’s name as the filename. Included in the /helpers folder is a function to crop an image to the required size of 800x175 pixels and convert it to webP, if not already in this format. It is not required to use this function, but images do need to be in webP and cropped to this size for best performance. If a specific part of the image should be used, it should be manually cropped, rather than using the function.

The below example shows how to use this function:

quarto run helpers/cropToWebP.ts 'default_im.webp' 'default_crop.webp'

Data

Storing Data

A lot of the processing of data is/should be done before hand due to the limitations of client-side processing and page performance. This pre-processed data should be stored in the Adaptation Atlas S3 bucket in a cloud-optimized format, such as parquet for tabular data. For development this data could be sotored elsewhere, but should be migrated to the Atlas S3 bucket for production. Get in touch with us to complete this. Files that may need to be updated should be stored in the S3 bucket to allow easier updating and version control.

To easily manage data and allow users to access the raw datasets being queried and pulled from the S3, data paths, alongside some metadata, should be stored in a JSON file in the notebook sub-directory in the /data folder. At minimum this should include a dataset name, source, and the sections the data is used. See the example below:

{
    "HeatData" : {
      "source": "https://digital-atlas.s3.us-east-1.amazonaws.com/heat.parquet",
      "name": "Historical and Future heat Data",
      "sections": [
        "Where is the heat?",
        "How hot will it be in the Future?"
      ]
    },
    "OtherData": {
      "source": "https://digital-atlas.s3.us-east-1.amazonaws.com/example.parquet",
      "name": "Some More Data",
      "sections": [
        "Section 1"
      ]
    }
}

Data can also be stored within the quarto project and loaded into quarto notebooks using File Attachments. This should be reserved for the translations and small files with a specific use - such as datasets needed for a single plot or files requiring a specific format. Any data stored within the project should be within a sub-directory in the /data folder with the notebook name.

Importing data

In general, DuckDB-wasm should be used for data import, querying, and some data processing. DuckDB can be used to directly query parquet data from the S3 bucket in an incredibly efficient manner. Some of the datasets used may be extremly large and contain may rows and columns - importing all of them at once would be terrible for page performance. We strongly suggest dynamically querying data by admin region and variable of interest to limit the about of data imported at a time. For large files, we suggest reading them as views, rather than tables, this can be seen in the below example. Some basic functions and data aggregation methods can also be performed using DuckDB, but JS methods can also be used for this, and are sometimes more efficient. Regarding data manipulation, it is recommended to consolidate data wrangling into a few OJS cells rather than splitting each processing step into seperate cells. For example, you can use a block within a js cell to define multiple lines (see Cells from Observable Documentation for examples).

To use DuckDB to import subsets data into the documents from the S3:

Show the code
viewof time = Inputs.radio(["historic", "2021_2040", "2041_2060"], {label: "Timeframe", value: "historic"})

db = {
  let db = await DuckDBClient.of()
  await db.query(`
    CREATE VIEW Hazard AS
    SELECT admin0_name, admin1_name, admin2_name, scenario, timeframe, hazard, value
    FROM read_parquet("s3://digital-atlas/hazards/hazard_timeseries_mean/annual/haz_means_adm.parquet")
  `);
  return await db;
}

data = {
 // The query below can be made dynamic using template literals
 // and it will automatically re-run when the interpolated values change.
  let query = `
    SELECT *
    FROM Hazard
    WHERE admin1_name IS NULL
    AND "hazard" IN ('HSH_max', 'HSH_mean', 'NDWL0', 'NDWS', 'NTx35', 'NTx40', 'TAI', 'THI_max', 'THI_mean', 'TAVG', 'PTOT')
    AND "timeframe" = '${time}'
    LIMIT 3`
  return db.query(query)
}

data

To load File Attachements follow this pattern:

```{ojs}  
data = await FileAttachment("/data/shared/atlas-region_admin0_simplified.json").json()
```

Important Note: Windows devices using certain web browsers have a error which results in cached parquet data becoming invalid when using DuckDB. This results in errors across any figure/section using this data when the page is reloaded or refreshed. The GitHub issue can be followed here. The solution to this is to prevent the use of cached parquet data on windows devices. There is an included function in the /helpers folder which itentifies if the device is a windows device and prevents the use of cached data though “cache busting”. It is required to use this function when loading parquet data into a notebook. Its use can be seen in the above example using DuckDB.

Country Boundaries and Geospatial data

For all spatial data within the Adaptation Atlas data explorations, use the following guidelines to maintain consistency and efficiency:

  1. Ensure that all maps, geographic selectors, and other geospatial elements utilize the Adaptation Atlas boundary dataset. These boundaries are based on the GAUL 2024, but include modifications so that country boundaries appear as each country represents themselves, including disputed areas.
  2. In general the Data Explorations should provide data to admin level 2 where possible, although data should be limited to the highest resolution data avaliable.
  3. Analysis and any raster extraction should use the highest resoltuon boundaries from the Atlas boundary dataset.
  4. Any visualizations within the data explorations should use the low or very low resolution boundaries, unless there is a valid reason for using higher resolution boundaries. This is to insure fast page load speeds, as there is a signifigant difference in file sizes between dataset resolution.
  5. Only one set of admin boundaries should be used in each data exploration - all datasets with a spatial “aspect” should be stored in raw tabular format and merged to the admin boundries within each notebook using the matching ‘admin0_name’, ‘admin1_name’, and ‘admin2_name’ columns. This will prevent multiple large boundary datasets from being loaded in each notebook.

Notebook Code

OJS in Quarto

OJS is slighty different than vanilla JavaScript and cells will automatically re-run when a variable they are dependent on is changed. The main differences can be outlined in the documentation. Quarto includes code blocks and OJS itself is based on cells, so it is important to note some differences. It is possible to have multiple OJS Cells within a quarto code block, and OJS cells can include blocks of JavaScript code. This can be seen in the below example, which return the same resutl using 3 different ways of structuring the code:

Example 1:

```{ojs}
x = 10

y = x + 5

result = y * 10
```

Example 2:

```{ojs}
x = 10
```

```{ojs}
y = x + 5
```

```{ojs}
result = y * 10
```

Example 3:

```{ojs}
result = {
  let x = 10
  let y = x + 5
  return y * 10
}
```

Example 1 and 2 are exactly the same, just split up across quarto code blocks. Both result in globals of x and y and result. Example 3 uses 1 quarto code block and 1 OJS cell with a block to return just result rather than the intermediate variables. In general, using a block to avoid assigning globals is preferred. Multiple OJS cells can be used in one Quarto code blocks, but aim for consistent and logical methods of organizing code.

Inline code blocks are useful for simple and short code such as selectors and text chunks. NOTE: These should only have 1 backtick at the start. The below example uses a extra backtick at the start as a hack to get the code to show correctly and prevent it from executing. For more about inline code, see the Quarto docs

This is some text in my document ``{ojs} "example".toUpperCase()`


### ``{ojs} "example_h3".toUpperCase()`

Modular code

Quarto Modules

Elements of a notebook can be split into multiple different .qmd documents. This can be useful in keeping code cleaner and more organized. This can be done using the {{< include _myFile.qmd >}} shortcode

A few important considerations when doing this: - Variables, Functions, and JS modules are shared between the imports because the include/merge is done on build/render. - This means having x defined in both or importing the same module in both will result in an error. - There are no performance implications of doing this, since the files are merged on build. - Any additional qmd module files used for these imports should be in the same directory as the main notebook, and imported modules should begin with an underscore so that quarto does not try to render them as standalone

Imports in OJS

OJS cells also have the ability to call in imports. Unlike the above quarto imports, this is done post-render on the client.This can include importing other OJS cells, using the .ojs extension, importing local JS code and modules, and importing modules from NPM. Although possible, this method should not be used to import large sections of a notebook - the above Quarto imports are better suited for this. However, this is useful for sharing functions and importing external libraries.

To import a local JS module or code in an OJS file (documented here):

```{ojs}
import { lang as Lang } from "/helpers/lang.js"

import { atlasTOC, atlasHero  } from "/helpers/uiComponents.ojs"
```

To import an NPM module or something from an observableHQ notebook:

```{ojs}
topojson = require("topojson") // Regular import
d3 = require("d3@7") // With a version
import {resolveWindowsCacheIssue} from "cbd32f92c155bf5b" // From an observableHQ notebook
initgeosJs = (await import("https://cdn.skypack.dev/geos-wasm")).default; // Directly from a CDN
duckdb = import("https://cdn.jsdelivr.net/npm/@duckdb/duckdb-wasm@1.28.1-dev215.0/+esm")

```

Folder and File structure

The main folder structure for the project is as follows:

.
├── index.qmd                   # Main index file required by Quarto
├── data
│   ├── notebook_name           # Named folder containing individual notebook data
│   │   ├── dataset1.json       # Notebook specific data not stored in S3
│   │   └── S3_data.json        # Paths to data stored externally in S3
│   │   └── translations.json   # Notebook specific translations
│   └── shared                  # Data shared across notebooks
├── helpers                     # Code and modules shared between notebooks
├── images
│   └── notebook_name.webp      # A single hero image for each notebook
├── notebooks                   # Code for all notebooks will end here
│   └── notebook_name           # Named folder containing individual notebook code
│       ├── notebook_name.qmd   # Main notebook file
│       ├── _module1.qmd        # Module imported into main notebook
│       ├── _module2.qmd
│       └── notebook_stye.css   # Notebook specific styling
├── _quarto.yml                 # Main project configuration
├── README.md                   # Main project README
├── _site                       # Output directory build by Quarto on render
└── styles.css                  # Main project styling

A concise and descriptive folder name should be used for each notebook, and this name should be used consistently across the main project folders (i.e. as the sub-directory names in the /notebooks, and /data folders and file names in the /images folder).

Translations

A small json of general translations can be found in the /data/shared directory. These may be useful to add in some basic translations, but each notebook will need a stand-alone translation document. This file is a good example of how the translations are handled within each notebook. Each notebook requires English and French at a minimum. More info on how translations are managed in the Atlas notebooks can be found here.

Shared Project Resources

Shared Data

At the moment the only data shared across all notebooks is the admin0 boundaries and some general translations. The admin1 and admin2 boundaries are stored and queried from the S3 bucket, but will be included in this folder in the future following a data update.

Helpers and functions

The /helpers folder contains code and modules that are used across the notebooks. This includes the lang module for the translations, some style builders to build elements on the page such as the TOC and hero image, and functions to handle other tasks. This folder also includes a post-render script which is automatically run to minify the CSS and JS files.

Styling

The main styling for the project can be found in the root directory under styles.css. Any styling that is notebook specific should be stored within the `/notebooks/notebook_name/” directory and applied using the yaml header at the top of the document/imported with OJS, or it should be including within the notebook code itself as a style block.

Other Documentation

Gotchas and Tips

  • Regularly delete the _site and .quarto files. These are auto generated by Quarto on build and can occasionally result in unexpected behavior and old versions of the notebooks to render.
  • There is a .gitignore in the project root you can use as a template to avoid committing the build files and other unwanted files.