- Convex Optimization Tutorial
- Home
- Introduction
- Linear Programming
- Norm
- Inner Product
- Minima and Maxima
- Convex Set
- Affine Set
- Convex Hull
- Caratheodory Theorem
- Weierstrass Theorem
- Closest Point Theorem
- Fundamental Separation Theorem
- Convex Cones
- Polar Cone
- Conic Combination
- Polyhedral Set
- Extreme point of a convex set
- Direction
- Convex & Concave Function
- Jensen's Inequality
- Differentiable Convex Function
- Sufficient & Necessary Conditions for Global Optima
- Quasiconvex & Quasiconcave functions
- Differentiable Quasiconvex Function
- Strictly Quasiconvex Function
- Strongly Quasiconvex Function
- Pseudoconvex Function
- Convex Programming Problem
- Fritz-John Conditions
- Karush-Kuhn-Tucker Optimality Necessary Conditions
- Algorithms for Convex Problems
- Convex Optimization Resources
- Convex Optimization - Quick Guide
- Convex Optimization - Resources
- Convex Optimization - Discussion
Sufficient & Necessary Conditions for Global Optima
Theorem
Let f be twice differentiable function. If $\bar{x}$ is a local minima, then $\bigtriangledown f\left ( \bar{x} \right )=0$ and the Hessian matrix $H\left ( \bar{x} \right )$ is a positive semidefinite.
Proof
Let $d \in \mathbb{R}^n$. Since f is twice differentiable at $\bar{x}$.
Therefore,
$f\left ( \bar{x} +\lambda d\right )=f\left ( \bar{x} \right )+\lambda \bigtriangledown f\left ( \bar{x} \right )^T d+\lambda^2d^TH\left ( \bar{x} \right )d+\lambda^2d^TH\left ( \bar{x} \right )d+$
$\lambda^2\left \| d \right \|^2\beta \left ( \bar{x}, \lambda d \right )$
But $\bigtriangledown f\left ( \bar{x} \right )=0$ and $\beta\left ( \bar{x}, \lambda d \right )\rightarrow 0$ as $\lambda \rightarrow 0$
$\Rightarrow f\left ( \bar{x} +\lambda d \right )-f\left ( \bar{x} \right )=\lambda ^2d^TH\left ( \bar{x} \right )d$
Since $\bar{x }$ is a local minima, there exists a $\delta > 0$ such that $f\left ( x \right )\leq f\left ( \bar{x}+\lambda d \right ), \forall \lambda \in \left ( 0,\delta \right )$
Theorem
Let $f:S \rightarrow \mathbb{R}^n$ where $S \subset \mathbb{R}^n$ be twice differentiable over S. If $\bigtriangledown f\left ( x\right )=0$ and $H\left ( \bar{x} \right )$ is positive semi-definite, for all $x \in S$, then $\bar{x}$ is a global optimal solution.
Proof
Since $H\left ( \bar{x} \right )$ is positive semi-definite, f is convex function over S. Since f is differentiable and convex at $\bar{x}$
$\bigtriangledown f\left ( \bar{x} \right )^T \left ( x-\bar{x} \right ) \leq f\left (x\right )-f\left (\bar{x}\right ),\forall x \in S$
Since $\bigtriangledown f\left ( \bar{x} \right )=0, f\left ( x \right )\geq f\left ( \bar{x} \right )$
Hence, $\bar{x}$ is a global optima.
Theorem
Suppose $\bar{x} \in S$ is a local optimal solution to the problem $f:S \rightarrow \mathbb{R}$ where S is a non-empty subset of $\mathbb{R}^n$ and S is convex. $min \:f\left ( x \right )$ where $x \in S$.
Then:
$\bar{x}$ is a global optimal solution.
If either $\bar{x}$ is strictly local minima or f is strictly convex function, then $\bar{x}$ is the unique global optimal solution and is also strong local minima.
Proof
Let $\bar{x}$ be another global optimal solution to the problem such that $x \neq \bar{x}$ and $f\left ( \bar{x} \right )=f\left ( \hat{x} \right )$
Since $\hat{x},\bar{x} \in S$ and S is convex, then $\frac{\hat{x}+\bar{x}}{2} \in S$ and f is strictly convex.
$\Rightarrow f\left ( \frac{\hat{x}+\bar{x}}{2} \right )< \frac{1}{2} f\left (\bar{x}\right )+\frac{1}{2} f\left (\hat{x}\right )=f\left (\hat{x}\right )$
This is contradiction.
Hence, $\hat{x}$ is a unique global optimal solution.
Corollary
Let $f:S \subset \mathbb{R}^n \rightarrow \mathbb{R}$ be a differentiable convex function where $\phi \neq S\subset \mathbb{R}^n$ is a convex set. Consider the problem $min f\left (x\right ),x \in S$,then $\bar{x}$ is an optimal solution if $\bigtriangledown f\left (\bar{x}\right )^T\left (x-\bar{x}\right ) \geq 0,\forall x \in S.$
Proof
Let $\bar{x}$ is an optimal solution, i.e, $f\left (\bar{x}\right )\leq f\left (x\right ),\forall x \in S$
$\Rightarrow f\left (x\right )=f\left (\bar{x}\right )\geq 0$
$f\left (x\right )=f\left (\bar{x}\right )+\bigtriangledown f\left (\bar{x}\right )^T\left (x-\bar{x}\right )+\left \| x-\bar{x} \right \|\alpha \left ( \bar{x},x-\bar{x} \right )$
where $\alpha \left ( \bar{x},x-\bar{x} \right )\rightarrow 0$ as $x \rightarrow \bar{x}$
$\Rightarrow f\left (x\right )-f\left (\bar{x}\right )=\bigtriangledown f\left (\bar{x}\right )^T\left (x-\bar{x}\right )\geq 0$
Corollary
Let f be a differentiable convex function at $\bar{x}$,then $\bar{x}$ is global minimum iff $\bigtriangledown f\left (\bar{x}\right )=0$
Examples
$f\left (x\right )=\left (x^2-1\right )^{3}, x \in \mathbb{R}$.
$\bigtriangledown f\left (x\right )=0 \Rightarrow x= -1,0,1$.
$\bigtriangledown^2f\left (\pm 1 \right )=0, \bigtriangledown^2 f\left (0 \right )=6>0$.
$f\left (\pm 1 \right )=0,f\left (0 \right )=-1$
Hence, $f\left (x \right ) \geq -1=f\left (0 \right )\Rightarrow f\left (0 \right ) \leq f \left (x \right)\forall x \in \mathbb{R}$
$f\left (x \right )=x\log x$ defined on $S=\left \{ x \in \mathbb{R}, x> 0 \right \}$.
${f}'x=1+\log x$
${f}''x=\frac{1}{x}>0$
Thus, this function is strictly convex.
$f \left (x \right )=e^{x},x \in \mathbb{R}$ is strictly convex.