Nonlinear Optimization

written by: Matthew Goh, Andrew King, and Edwin Yik =Introduction= Various conditions and situations are not adequately described using linear systems. In this case, nonlinear optimization may be applied. Unlike linear optimization, the optimal operating condition does not exist at the boundaries.

=Quadratic Optimization= $$\quad f(x)= c-x^Tb+\frac{1}{2}x^TAx$$

To optimize, it is necessary to find when the gradient of f is equal to zero.

$$\quad \nabla f(x)= 0$$

$$\quad \nabla f(x)=b-Ax $$

$$\quad x_*=A^{-1} b$$

It may be possible to solve the optimal $$\quad x_*$$ by a linear equation, approximated by a Taylor series [1].

$$\quad f(x_*) =~ f(x)+(x_*-x)^T\nabla f(x) + \frac{1}{2}(x_*-x)^T \nabla \nabla\ f(x)(x_*-x)+...$$

=Iterative Methods= When direct methods cannot solve the equation (i.e. A is not symmetric positive definite), iterative methods are possible [1].

By starting with an initial guess of $$\quad x_i$$, an algorithm may lead to a $$\quad x_{i+1}$$ that better satisfies the equation. Through iteration, theoretically, $$\quad x_\infty=x$$.

=Applications= - Finance: Portfolio optimization

- Businesses: Optimize inventory

- Engineering: Rigid body dynamics

- Biochemistry: Kinetic modeling [2]

=Example: Typical Nonlinear 3d Curves=



(Image from [1])

As observed, the optimal condition does not necessarily exist at the boundary of the curve.

=Example: Quadratic Optimization=

$$\quad f(x)= \overrightarrow{c}^T\overrightarrow{x} + \frac {1}{2}\overrightarrow{x}^TQ\overrightarrow{x}$$

where

$$\quad \overrightarrow{c}^T=(c_1, c_2, ..., c_n)$$ $$\quad \overrightarrow{x}^T=(x_1, x_2, ..., x_n)$$

For a quadratic system, n=2, thus, Q (the quadratic term constant) is defined as a symmetric matrix as follows.

$$\quad Q=\begin{bmatrix} Q_1 & Q_3 \\ Q_3 & Q_2 \end{bmatrix}$$

Thus, multiplying out the $$\quad f $$,

$$\quad f(x)= (c_1x_1 + c_2x_2) + \frac{1}{2}(Q_1x_1^2 + 2 Q_3x_1x_2+Q_2x_2^2) $$

[Note -- can use a specific example -- R.Z.]

=References=
 * 1) Lippert, Ross A. "Introduction to non-linear optimization."  D.E. Shaw Research, February 25, 2008. http://www.mit.edu/~9.520/spring08/Classes/optlecture.pdf
 * 2) Mendes, Pedro and Kell, Douglas B. "Non-linear optimization of biochemical pathways: application to metabolic engineering and parameter estimation."  Journal of Bioinformatics, Volume 14, 869-883. 1998.
 * 3) "Introduction to Non-linear optimization."  Georgia Institute of Technology Systems Realization Laboratory.  http://www.srl.gatech.edu/education/ME6103/NLP-intro.ppt