Stephen Huan

Miscellaneous mathematical observations

  ·  # math

Gradient is orthogonal to contours

Geometric intuition for the gradient.

Definition. For a scalar function f:Rn→Rf: \R^n \to \R we say a contour or level set of ff is a parameterized curve r(t):R→Rn\vec{r}(t): \R \to \R^n such that for every point on the curve, the function value is the same, that is, there is a fixed constant c∈Rc \in \R such that

f(r(t))=c(1) \begin{align*} f(\vec{r}(t)) = c \end{align*} \qquad{(1)}

Claim. The gradient ∇f\nabla f is orthogonal to the contour at every point.

Proof. Take the derivative of both sides of (1) with respect to tt.

dfdt=dfdr1dr1dt+⋯+dfdrndrndtf(r(t))=cdfdt=0 \begin{align*} \hphantom{ \frac{df}{dt} = \frac{df}{d r_1} \frac{d r_1}{dt} + \dotsb + \frac{df}{d r_n} \frac{d r_n}{dt} } \mathllap{f(\vec{r}(t))} &= c \\ \frac{df}{dt} &= 0 \end{align*}

Expanding using the multivariate chain rule,

dfdt=dfdr1dr1dt+⋯+dfdrndrndt=0(dfdr1⋯dfdrn)(dr1dt⋮drndt)=0⟨∇f(t),r′(t)⟩=0 \begin{align*} \frac{df}{dt} = \frac{df}{d r_1} \frac{d r_1}{dt} + \dotsb + \frac{df}{d r_n} \frac{d r_n}{dt} &= 0 \\ \begin{pmatrix} \frac{df}{d r_1} & \cdots & \frac{df}{d r_n} \end{pmatrix} \begin{pmatrix} \frac{d r_1}{dt} \\ \vdots \\ \frac{d r_n}{dt} \end{pmatrix} &= 0 \\ \inner{\nabla f(t)}{\vec{r}'(t)} &= 0 \end{align*}

which is what we wanted to show. □\square

Thus the gradient is orthogonal to every tangent of the contour, so moving along the contour moves orthogonally to the gradient. This should make intuitive sense since the change in the function value after moving in the direction Δx\Delta \vec{x} is approximately (to first order Taylor series approximation)

f(x+Δx)≈f(x)+∇f(t)TΔx \begin{align*} f(\vec{x} + \Delta \vec{x}) \approx f(\vec{x}) + \nabla f(t)^\top \Delta \vec{x} \end{align*}

Therefore if we want the function value to not change (which is the definition of a contour), we want the direction of movement to be orthogonal to the gradient, which is precisely what we have shown happens.

Positive definite implies Cauchy–Schwarz

A quick derivation of the Cauchy–Schwarz inequality.

Theorem. (Cauchy–Schwarz inequality). Every pair of vectors u,vu, v satisfies

⟨u,u⟩⟨v,v⟩≥⟨u,v⟩2 \begin{align*} \inner{u}{u} \inner{v}{v} &\geq \inner{u}{v}^2 \end{align*}

Proof. For vectors u,v∈Rnu, v \in \R^n, let V≔(uv)∈Rn×2V \defeq \begin{pmatrix} u & v \end{pmatrix} \in \R^{n \times 2} be the matrix with uu and $ v$ as columns.

Θ≔VTV=(⟨u,u⟩⟨u,v⟩⟨v,u⟩⟨v,v⟩)det⁡(Θ)=⟨u,u⟩⟨v,v⟩−⟨u,v⟩2≥0  ⟹  ⟨u,u⟩⟨v,v⟩≥⟨u,v⟩2 \begin{align*} \Theta \defeq V^{\top} V = \begin{pmatrix} \inner{u}{u} & \inner{u}{v} \\ \inner{v}{u} & \inner{v}{v} \end{pmatrix} & \\ \det(\Theta) = \inner{u}{u} \inner{v}{v} - \inner{u}{v}^2 &\geq 0 \\ \implies \inner{u}{u} \inner{v}{v} &\geq \inner{u}{v}^2 \end{align*}

where we use that Θ\Theta is symmetric positive-definite so its determinant is positive. □\square

A similar line of reasoning can be used to show that the variance is positive.

Theorem. For any random variable XX, define its variance as

Var[X]≔E[X2]−E[X]2 \begin{align*} \Var{X} \defeq \E{X^2} - \E{X}^2 \end{align*}

Then Var[X]≥0\Var{X} \geq 0.

The usual approach is to show that Var[X]=E[(X−E[X])2]\Var{X} = \E{(X - \E{X})^2} and then argue that the expectation of a nonnegative quantity must be nonnegative. Alternatively, it follows directly from Jensen's inequality on the convex function f(x)=x2f(x) = x^2. We will take a different approach.

Lemma. For any random variable XX, let μi≔E[Xi]\mu_i \defeq \E{X^i} be its ii-th moment. Then the matrices collecting its moments up to order nn

Mr≔(1μ1μ2⋯μrμ1μ2μ3⋯μr+1μ2μ3μ4⋯μr+2⋮⋮⋮⋱⋮μrμr+1μr+2⋯μ2r)(2) \begin{align*} M_r \defeq \begin{pmatrix} 1 & \mu_1 & \mu_2 & \cdots & \mu_r \\ \mu_1 & \mu_2 & \mu_3 & \cdots & \mu_{r + 1} \\ \mu_2 & \mu_3 & \mu_4 & \cdots & \mu_{r + 2} \\ \vdots & \vdots & \vdots & \ddots & \vdots \\ \mu_r & \mu_{r + 1} & \mu_{r + 2} & \cdots & \mu_{2 r} \\ \end{pmatrix} \end{align*} \qquad{(2)}

are all positive semi-definite for r=1,2,…,⌊n/2⌋r = 1, 2, \dotsc, \floor{n/2}.

Proof. Take the expectation of the squared polynomial with coefficients a0,…,ara_0, \dotsc, a_r.

E[(a0+a1X+a2X2+⋯+arXr)2]≥0 \begin{align*} \E{(a_0 + a_1 X + a_2 X^2 + \dotsb + a_r X^r)^2} \geq 0 \end{align*}

Expanding the square directly, we have

E[∑0≤i,j≤raiajXi+j]=∑0≤i,j≤raiajE[Xi+j]=∑0≤i,j≤raiajμi+j \begin{align*} \E*{\sum_{0 \leq i, j \leq r} a_i a_j X^{i + j}} = \sum_{0 \leq i, j \leq r} a_i a_j \E{X^{i + j}} = \sum_{0 \leq i, j \leq r} a_i a_j \mu_{i + j} \end{align*}

where we use the linearity of expectation and the definition of moments. But this is precisely the quadratic form aTMa\vec{a}^{\top} M \vec{a} for a≔(a0,…,ar)\vec{a} \defeq (a_0, \dotsc, a_r) and Mi,j≔μi+jM_{i, j} \defeq \mu_{i + j}, matching the definition in (2). Since aTMa≥0\vec{a}^{\top} M \vec{a} \geq 0 holds for any a∈Rr+1a \in \R^{r + 1}, MM is positive semi-definite by definition. □\square

Now to prove the nonnegativity of variance, we need only to apply the lemma for r=1r = 1.

Proof. Take M2=(1μ1μ1μ2)=(1E[X]E[X]E[X2])M_2 = \left( \begin{smallmatrix} 1 & \mu_1 \\ \mu_1 & \mu_2 \\ \end{smallmatrix} \right) = \left( \begin{smallmatrix} 1 & \E{X} \\ \E{X} & \E{X^2} \\ \end{smallmatrix} \right). Since M2M_2 is positive semi-definite its determinant is nonnegative, so we have det⁡(M2)=E[X2]−E[X]2≥0\det(M_2) = \E{X^2} - \E{X}^2 \geq 0, showing Var[X]≥0\Var{X} \geq 0. □\square

証明は絶対

A joke transcribed from page 9 of 『数学女子: 1』 (著者名：安田まさえ) .

さみしいも、たのしい。Page source. Last updated: .