- Convex Optimization Tutorial
- Home
- Introduction
- Linear Programming
- Norm
- Inner Product
- Minima and Maxima
- Convex Set
- Affine Set
- Convex Hull
- Caratheodory Theorem
- Weierstrass Theorem
- Closest Point Theorem
- Fundamental Separation Theorem
- Convex Cones
- Polar Cone
- Conic Combination
- Polyhedral Set
- Extreme point of a convex set
- Direction
- Convex & Concave Function
- Jensen's Inequality
- Differentiable Convex Function
- Sufficient & Necessary Conditions for Global Optima
- Quasiconvex & Quasiconcave functions
- Differentiable Quasiconvex Function
- Strictly Quasiconvex Function
- Strongly Quasiconvex Function
- Pseudoconvex Function
- Convex Programming Problem
- Fritz-John Conditions
- Karush-Kuhn-Tucker Optimality Necessary Conditions
- Algorithms for Convex Problems
- Convex Optimization Resources
- Convex Optimization - Quick Guide
- Convex Optimization - Resources
- Convex Optimization - Discussion
Convex and Concave Function
Let $f:S \rightarrow \mathbb{R}$, where S is non empty convex set in $\mathbb{R}^n$, then $f\left ( x \right )$ is said to be convex on S if $f\left ( \lambda x_1+\left ( 1-\lambda \right )x_2 \right )\leq \lambda f\left ( x_1 \right )+\left ( 1-\lambda \right )f\left ( x_2 \right ), \forall \lambda \in \left ( 0,1 \right )$.
On the other hand, Let $f:S\rightarrow \mathbb{R}$, where S is non empty convex set in $\mathbb{R}^n$, then $f\left ( x \right )$ is said to be concave on S if $f\left ( \lambda x_1+\left ( 1-\lambda \right )x_2 \right )\geq \lambda f\left ( x_1 \right )+\left ( 1-\lambda \right )f\left ( x_2 \right ), \forall \lambda \in \left ( 0, 1 \right )$.
Let $f:S \rightarrow \mathbb{R}$ where S is non empty convex set in $\mathbb{R}^n$, then $f\left ( x\right )$ is said to be strictly convex on S if $f\left ( \lambda x_1+\left ( 1-\lambda \right )x_2 \right )< \lambda f\left ( x_1 \right )+\left ( 1-\lambda \right )f\left ( x_2 \right ), \forall \lambda \in \left ( 0, 1 \right )$.
Let $f:S \rightarrow \mathbb{R}$ where S is non empty convex set in $\mathbb{R}^n$, then $f\left ( x\right )$ is said to be strictly concave on S if $f\left ( \lambda x_1+\left ( 1-\lambda \right )x_2 \right )> \lambda f\left ( x_1 \right )+\left ( 1-\lambda \right )f\left ( x_2 \right ), \forall \lambda \in \left ( 0, 1 \right )$.
Examples
A linear function is both convex and concave.
$f\left ( x \right )=\left | x \right |$ is a convex function.
$f\left ( x \right )= \frac{1}{x}$ is a convex function.
Theorem
Let $f_1,f_2,...,f_k:\mathbb{R}^n \rightarrow \mathbb{R}$ be convex functions. Consider the function $f\left ( x \right )=\displaystyle\sum\limits_{j=1}^k \alpha_jf_j\left ( x \right )$ where $\alpha_j>0,j=1, 2, ...k,$ then $f\left ( x \right )$is a convex function.
Proof
Since $f_1,f_2,...f_k$ are convex functions
Therefore, $f_i\left ( \lambda x_1+\left ( 1-\lambda \right )x_2 \right )\leq \lambda f_i\left ( x_1 \right )+\left ( 1-\lambda \right )f_i\left ( x_2 \right ), \forall \lambda \in \left ( 0, 1 \right )$ and $i=1, 2,....,k$
Consider the function $f\left ( x \right )$.
Therefore,
$ f\left ( \lambda x_1+\left ( 1-\lambda \right )x_2 \right )$
$=\displaystyle\sum\limits_{j=1}^k \alpha_jf_j\left ( \lambda x_1 +1-\lambda \right )x_2\leq \displaystyle\sum\limits_{j=1}^k\alpha_j\lambda f_j\left ( x_1 \right )+\left ( 1-\lambda \right )f_j\left ( x_2 \right )$
$\Rightarrow f\left ( \lambda x_1+\left ( 1-\lambda \right )x_2 \right )\leq \lambda \left ( \displaystyle\sum\limits_{j=1}^k \alpha _jf_j\left ( x_1 \right ) \right )+\left ( \displaystyle\sum\limits_{j=1}^k \alpha _jf_j\left ( x_2 \right ) \right )$
$\Rightarrow f\left ( \lambda x_1+\left ( 1-\lambda \right )x_2 \right )\leq \lambda f\left ( x_2 \right )\leq \left ( 1-\lambda \right )f\left ( x_2 \right )$
Hence, $f\left ( x\right )$ is a convex function.
Theorem
Let $f\left ( x\right )$ be a convex function on a convex set $S\subset \mathbb{R}^n$ then a local minima of $f\left ( x\right )$ on S is a global minima.
Proof
Let $\hat{x}$ be a local minima for $f\left ( x \right )$ and $\hat{x}$ is not global minima.
therefore, $\exists \hat{x} \in S$ such that $f\left ( \bar{x} \right )< f\left ( \hat{x} \right )$
Since $\hat{x}$ is a local minima, there exists neighbourhood $N_\varepsilon \left ( \hat{x} \right )$ such that $f\left ( \hat{x} \right )\leq f\left ( x \right ), \forall x \in N_\varepsilon \left ( \hat{x} \right )\cap S$
But $f\left ( x \right )$ is a convex function on S, therefore for $\lambda \in \left ( 0, 1 \right )$
we have $\lambda \hat{x}+\left ( 1-\lambda \right )\bar{x}\leq \lambda f\left ( \hat{x} \right )+\left ( 1-\lambda \right )f\left ( \bar{x} \right )$
$\Rightarrow \lambda \hat{x}+\left ( 1-\lambda \right )\bar{x}< \lambda f\left ( \hat{x} \right )+\left ( 1-\lambda \right )f\left (\hat{x} \right )$
$\Rightarrow \lambda \hat{x}+\left ( 1-\lambda \right )\bar{x}< f\left (\hat{x} \right ), \forall \lambda \in \left ( 0,1 \right )$
But for some $\lambda<1$ but close to 1, we have
$\lambda \hat{x}+\left ( 1-\lambda \right )\bar{x} \in N_\varepsilon \left ( \hat{x} \right )\cap S$ and $f\left ( \lambda \hat{x}+\left ( 1-\lambda \right )\bar{x} \right )< f\left ( \bar{x} \right )$
which is a contradiction.
Hence, $\bar{x}$ is a global minima.
Epigraph
let S be a non-empty subset in $\mathbb{R}^n$ and let $f:S \rightarrow \mathbb{R}$ then the epigraph of f denoted by epi(f) or $E_f$ is a subset of $\mathbb{R}^n+1$ defined by $E_f=\left \{ \left ( x,\alpha \right ):x \in \mathbb{R}^n, \alpha \in \mathbb{R}, f\left ( x \right )\leq \alpha \right \}$
Hypograph
let S be a non-empty subset in $\mathbb{R}^n$ and let $f:S \rightarrow \mathbb{R}$, then the hypograph of f denoted by hyp(f) or $H_f=\left \{ \left ( x, \alpha \right ):x \in \mathbb{R}^n, \alpha \in \mathbb{R}^n, \alpha \in \mathbb{R}, f\left ( x \right )\geq \alpha \right \}$
Theorem
Let S be a non-empty convex set in $\mathbb{R}^n$ and let $f:S \rightarrow \mathbb{R}^n$, then f is convex if and only if its epigraph $E_f$ is a convex set.
Proof
Let f is a convex function.
To show $E_f$ is a convex set.
Let $\left ( x_1, \alpha_1 \right ),\left ( x_2, \alpha_2 \right ) \in E_f,\lambda \in\left ( 0, 1 \right )$
To show $\lambda \left ( x_1,\alpha_1 \right )+\left ( 1-\lambda \right )\left ( x_2, \alpha_2 \right ) \in E_f$
$\Rightarrow \left [ \lambda x_1+\left ( 1-\lambda \right )x_2, \lambda \alpha_1+\left ( 1-\lambda \right )\alpha_2 \right ]\in E_f$
$f\left ( x_1 \right )\leq \alpha _1, f\left ( x_2\right )\leq \alpha _2$
Therefore, $f\left (\lambda x_1+\left ( 1-\lambda \right )x_2 \right )\leq \lambda f\left ( x_1 \right )+\left ( 1-\lambda \right )f \left ( x_2 \right )$
$\Rightarrow f\left ( \lambda x_1+\left ( 1-\lambda \right )x_2 \right )\leq \lambda \alpha_1+\left ( 1-\lambda \right )\alpha_2$
Converse
Let $E_f$ is a convex set.
To show f is convex.
i.e., to show if $x_1, x_2 \in S,\lambda \left ( 0, 1\right )$
$f\left ( \lambda x_1+\left ( 1-\lambda \right )x_2 \right )\leq \lambda f\left ( x_1 \right )+\left ( 1-\lambda \right )f\left ( x_2 \right )$
Let $x_1,x_2 \in S, \lambda \in \left ( 0, 1 \right ),f\left ( x_1 \right ), f\left ( x_2 \right ) \in \mathbb{R}$
Since $E_f$ is a convex set, $\left ( \lambda x_1+\left ( 1-\lambda \right )x_2, \lambda f\left ( x_1 \right )+\left ( 1-\lambda \right )\right )f\left ( x_2 \right )\in E_f$
Therefore, $f\left ( \lambda x_1+\left ( 1-\lambda \right )x_2 \right )\leq \lambda f\left ( x_1 \right )+\left ( 1-\lambda \right )f\left ( x_2 \right )$