Debugging the code

Sometimes, the code does not give you the result that you are expecting. This can be due to a variety of reasons, from misunderstandings or changes, system specific quirks, or bugs. You might also want to debug your code that uses pyAMReX or pyAMReX itself, as you implement new features during development.

This section gives a step-by-step guidance on how to systematically check what might be going wrong.

Debugging Workflow

Debugging Python processes works the same way as debugging regular applications. For investigating bugs on the Python extension side (e.g., in AMReX/pyAMReX/your application), the Python intepreter itself does not need to be compiled in debug mode.

Try the following steps to debug a simulation:

  1. Check the output text file, often called output.txt: are there warnings or errors present?

  2. On an HPC system, look for the job output and error files, usually called <name>.e... and <name>.o.... Read long messages from the top and follow potential guidance.

  3. If your simulation already created output data files: Check if they look reasonable before the problem occurred; are the initial conditions of the simulation as you expected? Do you spot numerical artifacts or instabilities that could point to missing resolution or unexpected/incompatible numerical parameters?

  4. Did the job output files indicate a crash? Check the Backtrace.<mpirank> files for the location of the code that triggered the crash. Backtraces are read from bottom (high-level) to top (most specific line that crashed).

  5. In case of a crash, Backtraces can be more detailed if you re-compile with debug flags: for example, try compiling with -DCMAKE_BUILD_TYPE=RelWithDebInfo (some slowdown) or even -DCMAKE_BUILD_TYPE=Debug (this will make the simulation way slower) and rerun.

  6. If debug builds are too costly, try instead compiling with -DAMReX_ASSERTIONS=ON to activate more checks and rerun.

  7. If the problem looks like a memory violation, this could be from an invalid field or particle index access. Try compiling with -DAMReX_BOUND_CHECK=ON (this will make the simulation very slow), and rerun.

  8. If the problem looks like a random memory might be used, try initializing memory with signaling Not-a-Number (NaN) values through the runtime option fab.init_snan = 1. Further useful runtime options are amrex.fpe_trap_invalid, amrex.fpe_trap_zero and amrex.fpe_trap_overflow (see details in the AMReX link below).

  9. On Nvidia GPUs, if you suspect the problem might be a race condition due to a missing host / device synchronization, set the environment variable export CUDA_LAUNCH_BLOCKING=1 and rerun.

  10. Consider simplifying your input options and re-adding more options after having found a working baseline.

Fore more information, see also the AMReX Debugging Manual.

Last but not least: the community of AMReX developers and users can help if you get stuck. Collect your above findings, describe where and what you are running and how you installed the code, describe the issue you are seeing with details and input files used and what you already tried. Can you reproduce the problem with a smaller setup (less parallelism and/or less resolution)? Report these details in a pyAMReX or AMReX GitHub issue.


See the AMReX debugger section on additional runtime parameters to

  • disable backtraces

  • rethrow exceptions

  • avoid AMReX-level signal handling

You will need to set those runtime options to work directly with debuggers.