Jupyter notebooks are interactive computing environments where prose and code can be combined. In the OGS project notebooks can be used to define complex benchmark workflows and its results can be converted to be shown on the OGS web page (see an example here). Notebooks can be used in two ways:
Consider direct conversion for the following use cases:
Tests/Data/requirements*.txt
for available packages).Create a new notebook file in web/contents/docs/[some-section]/my-page/my-page.ipynb
. It is important that the notebook filename is the same as the containing folder name!
If you use additional images put them into the my-page
-folder.
If the notebook result should appear as a page on the web documentation a frontmatter with some meta information (similar to regular web pages) is required as the first cell in the notebook:
Add a new cell and move it to the first position in the notebook
Cell type needs to be markdown
or raw
Add meta information e.g.:
+++
title = "BHE Meshing"
date = "2023-08-18"
author = "Joy Brato Shil, Haibing Shao"
+++
Make sure that you execute the cells in the notebook and save the notebook (with generated outputs).
To get a preview of the web page run the convert_notebooks
-script:
# You need the converter-tool nbconvert installed. Recommended way is to
# create and activate a virtual environment and install it there:
python -m venv .venv # or `python3 -m ...` on some systems
source .venv/bin/activate # .\.venv\Scripts\Activate.ps1 on Windows
pip install nbconvert
python web/scripts/convert_notebooks.py
cd web
hugo server
# open http://localhost:1313
The notebook needs some meta information (only title
, date
and author
is required) as outlined below.
Also make sure that you also provide necessary data files and please don’t use machine specific paths (e.g. assume that ogs
and other tools are in the PATH
).
These notebooks are part of the regular CI testing. Please try to keep the notebook execution time low.
Create a new notebook file in Tests/Data
(if it should appear in the benchmark gallery) or in web/content/docs
(e.g. for tutorials). Create it as a regular Markdown-file with Python code blocks. The notebook execution and conversion is done via Jupytext. See examples:
.ipynb
-notebook are also possible but Markdown-based notebooks are preferred)If the notebook result should appear as a page on the web documentation a frontmatter with some meta information (similar to regular web pages) is required as the first cell in the notebook:
+++
title = "SimplePETSc"
date = "2021-11-09"
author = "Lars Bilke"
image = "optional_gallery_image.png"
web_subsection = "small-deformations" # required for notebooks in Tests/Data only
+++
<-- Add Two newlines here to separate -->
<-- the frontmatter as its own cell -->
web_subsection
needs to be set to a sub-folder in web/content/docs/benchmarks (if not set the notebook page will not be linked from navigation bar / benchmark gallery on the web page)..ipynb
-based notebooks the frontmatter needs to be given in the first cell. See existing notebooks (e.g. SimpleMechanics.ipynb) for reference.The first cell after the frontmatter needs to be a markdown
-cell!
Tests/Data
only: Static images e.g. for the gallery image or to be used in Markdown cells have to be located in either images
- or figures
-subdirectories beneath the Notebook file! Otherwise they will not show up on the web site.
For image captions add a title in quotation marks after the image path, e.g.
![Alt text](figures/my_image.png "This will be the image caption.")
Please note that in merge request web previews static images are not shown at all.
Do not use machine-specific or absolute paths! See the following example to set up notebook output paths:
import os
from pathlib import Path
# On CI out_dir is set to the notebooks directory inside the build directory
# similar to regular benchmark tests. On local testing it will output to the
# notebooks source directory under a _out-subdirectory.
out_dir = Path(os.environ.get("OGS_TESTRUNNER_OUT_DIR", "_out"))
if not out_dir.exists():
out_dir.mkdir(parents=True)
# ...
# Run ogs; get input data from current directory; write to `out_dir`
! ogs my_project.prj -o {out_dir} > {out_dir}/log.txt
# OR with ogs6py:
# ... setup model ...
model.run_model(logfile=os.path.join(out_dir, "log.txt"), args=f"-o {out_dir}")
# Verify results; on failure assert with:
assert False
# or
raise SystemExit()
Do not write anything into the source directories. Use an out_dir
as above.
Assume that ogs
and other tools are in the PATH
.
In CI the notebooks are executed with all dependencies installed into a virtual environment in the build directory. The installed packages are defined in Test/Data/requirements.txt
. The same setup can be enabled locally by setting the CMake option OGS_USE_PIP=ON
. E.g.
cmake --preset release -DOGS_USE_PIP=ON # Creates the virtual environment
source ../build/release/.venv/bin/activate # Activates the virtual environment
jupyter lab # Starts Jupyter Lab
Add the notebook to CTest (example) with e.g.:
if(NOT OGS_USE_PETSC)
NotebookTest(NOTEBOOKFILE Mechanics/Linear/SimpleMechanics.ipynb RUNTIME 10)
# Notebooks in web/content need to be prefixed with 'notebook-'!
NotebookTest(NOTEBOOKFILE ../../web/content/docs/tutorials/bhe_meshing/notebook-bhe_meshing.md
PYTHON_PACKAGES openpyxl
RUNTIME 10)
endif()
NOTEBOOKFILE
is relative to Tests/Data
.PYTHON_PACKAGES
.web/content
it is important to prefix the notebook file name with notebook-
! The prefix is required to indicate Hugo that this is a notebook and not a regular markdown page.If your notebook should not appear on the website add the SKIP_WEB
-option to NotebookTest()
. This may be useful if the notebook serves as CI test only, e.g. comparing multiple simulation runs or doing performance measurements. But please also note that there will be no artifact produced (except for notebook errors which get reported as usual).
Then e.g. run all notebook test (-R nb
) in parallel (-j 4
) with:
# cd into build directory
source .venv/bin/activate # Is created with OGS_USE_PIP=ON, see above note on environment.
ctest -R nb -j 4 --output-on-failure
If you use the execution environment Jupytext is already configured and its usage is transparent:
.ipynb
-file is created in the background which stores outputsOn the web site or MR web previews on pages generated by a notebook there is a new banner:
bilke/binder-ogs-requirements
at GitHubPyVista (or VTK) requires a windowing environment for rendering. You can provide a virtual window with xvfb-run
:
sudo apt install libgl1-mesa-glx xvbf # install xvfb
xvfb-run -a ctest [...] # provide a virtual window to the ctest-run
This article was written by Lars Bilke. If you are missing something or you find an error please let us know.
Generated with Hugo 0.122.0
in CI job 473229
|
Last revision: July 26, 2024
Commit: [py] Some pytest fixes for PETSc config. f827096
| Edit this page on