Step 7 — Run the Solver on IBM Polus

By this point, the solver has several execution modes:

sequential
OpenMP
MPI
MPI + OpenMP
MPI + CUDA

Running them on a workstation is useful for development, but the coursework also asks us to execute and evaluate the implementations on IBM Polus.

Polus uses IBM Spectrum LSF as its batch scheduler. The practical workflow is therefore different from a local shell session:

connect to Polus
      ↓
prepare the software environment
      ↓
compile the executable
      ↓
describe the requested resources
      ↓
submit the job to LSF
      ↓
wait in the queue
      ↓
run on an allocated compute node
      ↓
collect logs and solution files

The SM25 repository captures this workflow in run_polus.sh and a collection of .lsf job files under config/.

This step explains how those pieces fit together.

1. Understand the target system

The official Polus documentation describes a five-node IBM POWER8 system, with one node also serving frontend functions.

The compute-node hardware includes

2 × 10-core IBM POWER8 CPUs
8 hardware threads per CPU core
2 × NVIDIA Tesla P100 GPUs
16 GB memory per GPU
NVLink

The documented software environment includes GNU and IBM compilers, OpenMP, IBM Spectrum MPI, Open MPI, IBM Spectrum LSF, and CUDA.

For our solver, this hardware naturally maps to the execution models developed in the previous steps:

POWER8 cores and hardware threads
        ↓
Sequential / OpenMP

multiple processes and nodes
        ↓
MPI / MPI+OpenMP

Tesla P100 GPUs
        ↓
MPI+CUDA

The cluster therefore gives us one environment in which all implementations can be tested under the same scheduler and filesystem.

2. Connect to Polus and work from the frontend

The Polus documentation provides SSH access through

polus.hpc.cs.msu.ru

or

polus.cs.msu.ru

and access is based on SSH keys.

A typical login command is

ssh <username>@polus.hpc.cs.msu.ru

The frontend is the place to prepare source code, inspect files, compile programs, and submit jobs.

The actual numerical experiments should be sent to the scheduler so that LSF can allocate CPU cores, GPUs, wall time, and compute nodes.

For the SM25 repository, the working directory should contain at least

Makefile
run_polus.sh
task.cpp
task_mpi.cpp
task_mpi_cuda.cpp

include/
src/
config/
solution/
log/

The run script creates solution/ and log/ when needed, so the main requirement is that the source tree and configuration files remain together.

3. Load the MPI environment used by the repository

The official Polus compilation guide documents both Spectrum MPI and Open MPI. It recommends Spectrum MPI for general use, while also providing Open MPI as a supported option.

The SM25 Polus script makes an explicit reproducibility choice and uses Open MPI:

module purge
module load OpenMPI/4.0.0
ulimit -s 10240

module purge clears previously loaded modules so that an older compiler or MPI environment does not silently remain active.

The repository then loads the exact Open MPI module expected by its Polus workflow.

This matters because the MPI compiler wrappers and runtime launcher should come from the same MPI environment:

mpicxx
mpiexec

Using one MPI implementation for compilation and another for execution can produce avoidable runtime problems.

For this project, the simplest rule is to reproduce the environment encoded in run_polus.sh when reproducing the repository experiments.

4. Compile on Polus, execute through LSF

Compilation happens before the job is submitted.

For example, the MPI implementation is built with

mpicxx -O3 -o task_mpi \
    task_mpi.cpp \
    src/conjugate_gradient.cpp \
    src/mpi_conjugate_gradient.cpp \
    -Iinclude \
    -lm \
    -std=c++11

The hybrid version adds OpenMP support:

mpicxx -O3 -o task_mpi_omp \
    task_mpi.cpp \
    src/conjugate_gradient.cpp \
    src/mpi_conjugate_gradient.cpp \
    -Iinclude \
    -lm \
    -std=c++11 \
    -fopenmp

The Polus GPU build uses

make ARCH=sm_60

which produces

task_mpi_cuda

through the repository Makefile.

After compilation, the executable is launched through LSF rather than being used as a long-running frontend process.

The central scheduler command is

bsub

with the general form

bsub [options] command [arguments]

5. Describe resources with bsub

The Polus LSF documentation defines several options that appear throughout the repository:

For example, the sequential jobs in run_polus.sh are submitted in the form

bsub \
    -J "seq_40x40" \
    -n 1 \
    -W 1 \
    -oo "log/stage_1/seq_40x40_cout.log" \
    -eo "log/stage_1/seq_40x40_cerr.log" \
    ./task_seq 40 40

The solver therefore runs only after LSF has allocated the requested resource.

The Polus documentation describes three queues:

short
normal
fullcluster

short is the default queue for short runs with a documented limit of 30 minutes, while normal is intended for jobs of up to three hours. fullcluster is used for full-cluster work and requires prior access.

For our experiments, wall-time requests should reflect the expected scale of the run instead of reserving unnecessarily large windows.

6. Use .lsf files for structured jobs

A long bsub command quickly becomes difficult to maintain, especially for hybrid and GPU runs.

LSF supports command files whose scheduler directives begin with

#BSUB

For example, a minimal MPI+OpenMP job can look like

#BSUB -n 2
#BSUB -W 1
#BSUB -J "mpi_omp_40_40_2_4"
#BSUB -o "log/stage_7/mpi_omp_40_40_2_4_cout.log"
#BSUB -e "log/stage_7/mpi_omp_40_40_2_4_cerr.log"

export OMP_NUM_THREADS=4
export OMP_PLACES=cores
export OMP_PROC_BIND=close

mpiexec ./task_mpi_omp 40 40

The repository contains exactly this configuration for the two-process, four-thread correctness run.

Submit a command file with

bsub < config/stage_7/mpi_omp_40_40_2_4.lsf

This separates two concerns cleanly:

solver executable
        ↓
numerical computation

.lsf file
        ↓
cluster resource configuration

The same binary can therefore be reused with several resource layouts.

7. Submit the OpenMP runs with explicit placement

OpenMP jobs need both a thread count and a processor-placement strategy.

The Polus documentation supports affinity requests such as

-R "affinity[core(N)]"

when OpenMP threads should be associated with CPU cores.

The repository uses this form for its smaller OpenMP tests:

bsub \
    -J "omp_40_40_4" \
    -W 1 \
    -R "affinity[core(4)]" \
    OMP_NUM_THREADS=4 \
    ./task_omp 40 40

For a larger single-node OpenMP experiment, the Polus documentation also provides the helper

/polusfs/lsf/openmp/launchOpenMP.py

for affinity-aware launching.

The repository uses it for the 32-thread \(800\times1200\) case:

bsub \
    -R "affinity[core(16)]" \
    OMP_NUM_THREADS=32 \
    /polusfs/lsf/openmp/launchOpenMP.py \
    ./task_omp 800 1200

This is a useful reminder that OMP_NUM_THREADS describes the OpenMP team, while LSF resource requests describe where that team is allowed to run.

8. Submit MPI jobs through the scheduler

The MPI executable is launched inside the LSF allocation with

mpiexec

For the \(40\times40\) correctness cases, run_polus.sh submits

1 MPI process
2 MPI processes
4 MPI processes

using jobs equivalent to

bsub -n 1 -W 1 mpiexec ./task_mpi 40 40
bsub -n 2 -W 1 mpiexec ./task_mpi 40 40
bsub -n 4 -W 1 mpiexec ./task_mpi 40 40

The larger MPI experiments extend the same pattern to the \(400\times600\) and \(800\times1200\) grids.

For example, the repository contains a Polus configuration for a 20-process \(800\times1200\) run:

#BSUB -n 20
#BSUB -W 30
#BSUB -J mpi_800_1200_20
#BSUB -oo "log/stage_9/mpi_800_1200_20_cout.log"
#BSUB -eo "log/stage_9/mpi_800_1200_20_cerr.log"
#BSUB -R "span[hosts=1]"

mpiexec ./task_mpi 800 1200

The resource expression

span[hosts=1]

asks LSF to place all requested CPU slots on one compute node.

Other LSF placement expressions, such as

span[ptile=N]

can limit the number of allocated slots placed on each host.

9. Submit the hybrid MPI+OpenMP runs

The Polus documentation treats hybrid execution as two parameters:

At the shell level, the pattern is

OMP_NUM_THREADS=T mpiexec ./task_mpi_omp ...

with LSF allocating the resources for the MPI processes.

The SM25 correctness configurations use four OpenMP threads per rank:

1 MPI process × 4 OpenMP threads
2 MPI processes × 4 OpenMP threads

The first of these is exactly the mpi_omp_40_40_2_4.lsf file shown in Section 6; the one-process configuration is the same file with -n 1.

The repository also contains a larger Polus configuration that deliberately uses the POWER8 hardware-thread structure:

#BSUB -n 20
#BSUB -W 30
#BSUB -R "span[hosts=1]"

export OMP_NUM_THREADS=8
export OMP_PROC_BIND=true
export OMP_PLACES=threads

mpiexec ./task_mpi_omp 800 1200

Here the job requests 20 MPI slots on one host and creates eight OpenMP threads inside each process.

This configuration is useful for exploring the POWER8 simultaneous multithreading capacity described by the Polus documentation.

The important point for the experiment report is to record both dimensions:

MPI process count
OpenMP threads per process

A label such as

20 MPI × 8 OpenMP

is much more informative than recording only a total thread count.

10. Request GPUs explicitly for MPI+CUDA

GPU jobs add the LSF

-gpu

resource request.

The Polus documentation describes the general form as

-gpu "num=...:mode=...:..."

and notes that each Polus node contains two GPUs.

The repository requests exclusive GPU access for the CUDA measurements.

A one-GPU job is configured as

#BSUB -n 1
#BSUB -W 30
#BSUB -J "mpi_cuda_800_1200_g1"
#BSUB -oo "log/stage_9/mpi_cuda_800_1200_g1_cout.log"
#BSUB -eo "log/stage_9/mpi_cuda_800_1200_g1_cerr.log"
#BSUB -gpu "num=1:mode=exclusive_process"

mpiexec ./task_mpi_cuda 800 1200

The two-GPU version requests two CPU slots and two exclusive GPUs:

#BSUB -n 2
#BSUB -W 30
#BSUB -J "mpi_cuda_800_1200_g2"
#BSUB -oo "log/stage_9/mpi_cuda_800_1200_g2_cout.log"
#BSUB -eo "log/stage_9/mpi_cuda_800_1200_g2_cerr.log"
#BSUB -R "span[ptile=2]"
#BSUB -gpu "num=2:mode=exclusive_process"

mpiexec ./task_mpi_cuda 800 1200

Inside the executable, the rank-to-device logic from Step 6 maps each node-local MPI rank to a visible CUDA device.

The allocation and execution layers therefore work together:

LSF
    reserves the GPUs
        ↓
MPI
    creates the processes
        ↓
local MPI rank
    selects a CUDA device
        ↓
CUDA kernels
    execute the local solver

Keeping GPU allocation in the .lsf file and GPU selection in the executable makes the resource request explicit and reproducible.

11. Monitor jobs and keep stdout and stderr

After submission, LSF assigns a numeric job identifier.

The standard command for inspecting the queue is

bjobs

and detailed information for one job can be requested with

bjobs -l <JOBID>

The Polus documentation lists common states including

PEND    waiting for resources
RUN     executing
DONE    completed successfully
EXIT    completed with a nonzero status

run_polus.sh captures the ID returned by bsub. For jobs submitted from a .lsf file, the form is

JOB_ID=$(bsub < "${LSF_FILE}" | awk '{print $2}' | tr -d '<>')

where ${LSF_FILE} is the job script path; the inline form used for the earlier stages (bsub ... ./task_seq 40 40) is parsed the same way.

It then waits for that specific job:

bwait -w "ended(${JOB_ID})"

This makes the experiment driver sequential at the job level:

submit one configuration
        ↓
wait for completion
        ↓
submit the next configuration

Each experiment writes separate stdout and stderr logs under directories such as

log/stage_1/
log/stage_2/
log/stage_5/
log/stage_7/
log/stage_9/

Keeping the error stream is important even when the program appears to have finished, because MPI, CUDA, or scheduler errors may be reported there.

12. Understand run_polus.sh as the experiment driver

The repository’s run_polus.sh combines environment setup, compilation, submission, waiting, and log organization.

It accepts

--stage
--stop_stage

so a subset of the experiment pipeline can be executed.

For example:

bash run_polus.sh --stage 5 --stop_stage 7

runs the internal script stages from 5 through 7.

These internal stage numbers belong to the automation script and are separate from the Step 1–8 structure of this walkthrough.

The script currently covers:

internal stage 1
    sequential correctness grids

internal stage 2
    OpenMP 1 / 4 / 16 thread correctness runs

internal stage 3
    larger OpenMP measurements

internal stage 5
    MPI 1 / 2 / 4 process correctness runs

internal stage 6
    larger MPI measurements

internal stage 7
    MPI+OpenMP correctness runs

internal stage 8
    larger MPI+OpenMP measurements

internal stage 9
    matched MPI, MPI+OpenMP, and MPI+CUDA batch runs

The last stage is especially useful for final comparisons because it runs pre-written .lsf configurations over several grid sizes.

The current size list is

800 × 1200
1600 × 2400
3200 × 4800

and for each size the script submits configurations for

MPI
MPI + OpenMP
MPI + CUDA with one GPU
MPI + CUDA with two GPUs

Before those GPU jobs, it builds the CUDA target with

make ARCH=sm_60

which is the architecture setting used by the repository for Polus.

13. Direct LSF submission and mpisubmit.pl

The official Polus documentation provides a convenience utility named

mpisubmit.pl

for MPI, OpenMP, MPI+OpenMP, and MPI+CUDA jobs.

Its interface exposes parameters such as

-p    MPI processes
-t    threads per process
-w    wall time
-g    GPU execution

For example, the documented style is conceptually

mpisubmit.pl -p 10 -t 8 -w 00:12 executable

The documentation recommends this helper for common parallel jobs and direct LSF usage for more specialized resource requests.

SM25 uses direct bsub commands and explicit .lsf files.

That choice makes several experiment details visible in version control:

job name
wall time
CPU-slot request
host placement
OpenMP placement variables
GPU count and sharing mode
stdout path
stderr path
executable and grid size

For a reproducibility-oriented project, these job files become part of the experiment configuration rather than temporary shell commands.

14. Preserve the execution configuration with the results

When a cluster run finishes, the numerical output alone is not enough to reconstruct the experiment.

For every timing result, keep the associated configuration:

grid size
executable
MPI process count
OpenMP thread count
GPU count
LSF placement request
wall-time request
compiler/build configuration
stdout log
stderr log

The SM25 repository already separates configuration and output into

config/
log/
solution/

which gives us a useful structure for this record.

At this stage, the purpose is to execute every implementation under controlled and documented resource settings.

The next step uses those outputs to answer the final questions of the coursework: whether the implementations agree numerically, how runtime changes with parallelism, where communication and data-transfer costs appear, and how to present speedup and efficiency in the final report.

Sources

The Polus-specific details in this step follow the official MSU HPC documentation (checked August 2026):

• IBM Polus hardware description
• Compilation guide
• IBM Spectrum LSF scheduler
• mpisubmit.pl script