This article is from http://fa.bianp.net/blog/2017/optimization-inequalities-cheatsheet/, just record it.
Most proofs in optimization consist in using inequalities for a particular function class in some creative way. This is a cheatsheet with inequalities that I use most often. It considers class of functions that are convex, strongly convex and
Setting.
∥∇f(y)−∇f(x)∥≤∥x−y∥ |f(x)−f(y)−⟨∇f(x),y−x⟩|≤L2∥y−x∥2 ∥∇2f(x)∥≤L (assuming f is twice differentiable)
f(x)≤f(y)+⟨∇f(x),x−y⟩ 0≤⟨∇f(x)−∇f(y),x−y⟩ f(EX)≤E[f(X)] whereX is a random variable (Jensen’s inequality).
1L∥∇f(x)−∇f(y)∥2≤⟨∇f(x)−∇f(y),x−y⟩ 0≤f(y)−f(x)−⟨∇f(x),y−x⟩≤L2∥x−y∥2 f(x)≤f(y)+⟨∇f(x),x−y⟩−12L∥∇f(x)−∇f(y)∥2
f(x)≤f(y)+⟨∇f(x),x−y⟩−μ2∥x−y∥2 μ2∥x−y∥2≤⟨∇f(x)−∇f(y),x−y⟩ μ2∥x−x∗∥2≤f(x∗)−f(x)
μLμ+L∥x−y∥2+1μ+L∥∇f(x)−∇f(y)∥2≤⟨∇f(x)−∇f(y),x−y⟩
Most of these inequalities appear in the Book: “Introductory lectures on convex optimization: A basic course” by Nesterov, Yurii (2013, Springer Science & Business Media). Another good source (and freely available for download) is the book “Convex Optimization” by
Stephen Boyd and Lieven Vandenberghe.