Singular Values

What are singular values?

Singular values are the square roots of the eigenvalues of the Gram matrix $K=A^{T}A$. In other words, the singular values ${\sigma }_1,...,{\sigma }_r$ are: ${\sigma }_i=\sqrt{{\lambda }_i}>0$. The eigenvectors of the Gram matrix K are called the right singular vectors of A.

What are properties of singular values?

$A^{T}A$ is symmetric and positive semi-definite: ${\lambda }_i\ge 0\to {\sigma }_i\ge 0$. Singular values are always real and non-negative; as convention, we order singular values in decreasing order. Finally, the number of nonzero singular values is the same as the rank of the matrix A.

What is SVD?

When A is symmetric ($A=A^T$) then its singular values are the absolute values of its nonzero eigenvalues. Its singular vectors coincide with its non-null eigenvectors.

Any real matrix of size m x n of rank r > 0 can be factored as:

$A=P\Sigma Q^T$.

where: $P\in {\mathbb{R}}^{m\times r}$: matrix with orthonormal columns $\Sigma =diag({\sigma }_1,...,{\sigma }_r)$: singular values $Q^T\in {\mathbb{R}}^{r\times n}$: matrix with orthonormal rows

We can understand SVD geometrically as decomposing A into three sequential transformations, applied from right to left.

1. $Q^T$ rotates the input space (${\mathbb{R}}^n$)
2. $\Sigma$ scales/compress along the orthogonal directions
3. $P$ rotates the result into the output space (${\mathbb{R}}^m$)

What is the pseudoinverse?

Let A be a nonzero m x n matrix with the SVD: $A=P\Sigma Q^T$ then the pseudoinverse $A^{+}$ is the n x m matrix: $A^{+}=Q{\Sigma }^{-1}{P}^T$.

The nonzero singular values of $A^{+}$ are the reciprocals of the nonzero singular values of A. The zero matrix is the only matrix without a pseudoinverse.

For invertible square matrices, the pseudoinverse is the actual inverse.

When A has linearly independent columns, the pseudoinverse can be calculated as: $A^{+}=(A^{T}A{)}^{-1}{A}^T$.

What are uses of the pseudoinverse?

Say we are solving Ax = b. For any general matrix A, the pseudoinverse $A^{+}$ gives us: $x^{\ast }=A^{+}b$. $x^{\ast }$ is the least squares solution under the Euclidean norm.