Seminar: Algorithmic Differentiation

Sample topics:

  • Algorithmic Differentiation (AD) for numerical methods [1,www]
  • first derivative models [1,2]
    • implementation by manual source transformation 
    • implementation by overloading
    • implementation by automatic source transformation
  • back-propagation in neural networks [9]
  • AD by overloading with dco/c++ [8,www]
  • AD by source transformation with TAPENADE [www]
  • second and higher derivative models [1,2]
  • exploitation of Jacobian sparsity [1,2,3]
  • data-flow reversal in first-order adjoints [1,10]
  • AD-specific data-flow analysis [5]
  • elimination techniques on computational graphs [2,4]
  • AD of numerical methods [6,7]
  • Expression templates for AD [11]

Literature:

  1. Naumann: The Art of Differentiating Computer Programs, SIAM, 2012.
  2. Griewank, Walther: Evaluating Derivatives, SIAM, 2008.
  3. Gebremedhin, Manne, Pothen: What Color Is Your Jacobian? Graph Coloring for Computing Derivatives, SIAM, 2005.
  4. Naumann: Optimal Accumulation of Jacobian Matrices by Elimination Methods on the Dual Computational Graph,
    Mathematical Programming, 2004.
  5. Hascoet, Naumann, Pascual: `To be recorded″ analysis in reverse-mode automatic differentiation, Future Generation Computer Systems, 2005.
  6. Naumann, Lotz: Algorithmic Differentiation of Numerical Methods: Tangent-Linear and Adjoint Direct Solvers for Systems of Linear Equations, AIB 2012-10.
  7. Naumann, Lotz, Leppkes, Towara: Algorithmic differentiation of numerical methods: Tangent and adjoint solvers for parameterized systems of nonlinear equations, ACM Transactions on Mathematical Software, 2015. (also: AIB 2014-07).
  8. Naumann, Leppkes, Lotz: dco/c++: Derivative Code by Overloading in C++: Introduction and Summary of Features, AIB 2016-08.
  9. Goodfellow, Bengio, Courville: The Deep Learning Book, MIT Press. In preparation. [www]
  10. Naumann: DAG Reversal is NP Complete, Journal of discrete algorithms, 2009.
  11. Phipps, Pawlowski: Efficient Expression Templates for Operator Overloading-based Automatic Differentiation. arXiv:1205.3506, 2012. [www]

Web links: