Algorithmic Differentiation with dco/c++

The course consists of

lecture-style presentations
live coding of examples
hands-on group programming sessions

covering

first-order tangents and adjoints
beyond black-box adjoints: early intervention
beyond black-box adjoints: late intervention
second- and higher-order tangents and adjoints
extended set of elementals / implicit functions

Preparation

All participants should have access to the latest version of dco/c++.

All participants should be able to

built the sample codes in their favorite environment
run the sample codes (including simple dco/c++ examples)
reproduce the results given in the README.md files in the respective subdirectories
edit the source files for hands-on work

Download and Unpack the Sample Code

A zip archive of the sample code can be downloaded from here.

Build the Sample Code

Linux Makefiles are provided as a reference for the build process.

adapt them to your environment
"make" builds all
"make clean" cleans up

Integration into alternative build environments (e.g. Windows or MacOS) should be performed accordingly.

Run the Sample Code

Reference output is provided in the README.md files located in the

Heston
SDE
Toy

subdirectories.

Agenda

Introduction
- The chain rule of differentiation is associative. So what?
- Computational cost of gradients
- Back-propagation in machine learning
- Playground (generic stochastic differential equation): SDE/
- Hands-On: Check local installation
- Typical work flow
Algorithmic Differentiation (AD) of Differentiable Programms
- Single assignment code and directed acyclic graph (tape)
- Chain rule on tape
- Tangent AD
- Tangent AD by hand on toy code
  - Toy/hand/tangent.cpp
- Finite Differences (FD)
  - SDE/fd/gradient/
- Scalar Tangent AD with dco/c++ (dco::gt1s<T>::type)
  - SDE/gt1s/
- Vector Tangent AD with dco/c++ (dco::gt1v<T,S>::type)
  - SDE/gt1v/
- Hands-On:
  - Heston/primal/
  - Heston/gt1s/
  - Heston/gt1v/
- Adjoint AD
- Adjoint AD by hand on toy code
  - Toy/hand/adjoint.cpp
- Scalar Adjoint AD with dco/c++ (dco::ga1s<T>::type)
  - SDE/ga1s/
- Variants in SDE/ga1s/variants/
  - Mutable independent variables
  - Tape types and custom tape sizes
  - Mixed tape types
  - File tape
- Hands-On:
  - Heston/ga1s/
- Toward AD mission planning
- Differential invariant
Beyond Black-Box AD: Early Intervention
- Motivation
- Generic scenario: Toy/
- Early Recording and Late Back-Propagation (default)
  - Performance analysis
  - Implementation
    - Toy/dco/adjoint.cpp
  - Discussion (Behind the scenes)
- Early Back-Propagation
  - Performance analysis
  - Implementation
    - Toy/dco/early_backpropagation.cpp
  - Discussion (Behind the scenes)
- Early Preaccumulation;
  - Performance analysis
  - Implementation; Preaccumulation in
    - in adjoint mode
      - Toy/dco/early_preaccumulation_adjoint.cpp
    - in tangent mode
      - Toy/dco/early_preaccumulation_tangent.cpp
    - with jacobian_preaccumulator_t
      - Toy/dco/early_preaccumulation_jp.cpp
  - Discussion (Behind the scenes)
  - Variants
- Early Preaccumulation and Back-Propagation
  - Performance analysis
  - Implementation
    - Toy/dco/early_preaccumulation_and_backpropagation.cpp
  - Discussion (Behind the scenes)
- Hands-On:
  - SDE/early/back-propagation/ga1s/
  - SDE/early/preaccumulation/ga1s/
  - SDE/early/preaccumulation/gt1v/
  - SDE/early/preaccumulation/jacobian_preaccumulator_t/
  - SDE/early/preaccumulation_and_back-propagation/gt1v/
  - Heston/early/ (jacobian_preaccumulator_t)
Beyond Black-Box AD: Late Intervention
- Late Back-Propagation (default)
  - Toy/dco/gradient/adjoint.cpp
- Late Recording
  - Performance analysis
  - Implementation
    - Toy/dco/gradient/late_recording.cpp
  - Discussion (Behind the scenes)
- Late Preaccumulation
  - Performance analysis
  - Implementation; Preaccumulation in
    - adjoint mode
      - Toy/dco/gradient/late_preaccumulation_adjoint.cpp
    - tangent mode
      - Toy/dco/gradient/late_preaccumulation_tangent.cpp
  - Discussion (Behind the scenes)
  - Variants
External Adjoints
- External adjoints interface of dco/c++
  - Toy/dco/gradient/external_adjoint.cpp
- Hands-On:
  - SDE/external/adjoint_buckets/
  - SDE/external/adjoint_chunks/
  - SDE/external/adjoint_buckets_chunks/
Higher-Order AD
- Hessian by FD; SDE/fd/Hessian/
- Second-order tangent AD
- Second-order tangent AD with dco/c++
  - SDE/gt1s_gt1s/
- Second-order adjoint AD
- Second-order adjoint AD with dco/c++
  - SDE/gt1s_ga1s/
- Hands-On:
  - Heston/gt1s_gt1s/
  - Heston/gt1v_ga1s/
- Beyond black-box second-order adjoint AD with dco/c++
  - SDE/early/gt2s_*/
  - SDE/external/*/gt2s_*
- Further second-order AD
- Higher-order AD
- Higher-order differential invariants
Enhanced Elemental Functions
- BLAS
- Implicit functions
  - Embedded nonlinear equation
    - hand coding: ImplicitFunction/hand
    - external adjoint with dco/c++: ImplicitFunction/dco
  - Systems of linear equations
  - First-order optimality condition
  - Case study: Calibration of SDE;
  - Error analysis
- NAG AD Library

Material

slides
code