• Linear Algebra: #18 Orthogonal Bases

Our vector space V is now assumed to be either Euclidean, or else unitary — that is, it is defined over either the real numbers ℜ, or else the complex numbers ℂ. In either case we have a scalar product <·,·> : V × V → F (here, F = ℜ or ℂ).

As always, we assume that V is finite dimensional, and thus it has a basis {v₁, . . . , v_n}. Thinking about the canonical basis for ℜⁿ or ℂⁿ, and the inner product as our scalar product, we see that it would be nice if we had

• <v_j , v_j> = 1, for all j (that is, the basis vectors are normalized), and furthermore 
• <v_j , v_k> = 0, for all j ≠ k (that is, the basis vectors are an orthogonal set in V). 

That is to say, {v₁, . . . , v_n} is an orthonormal basis of V. Unfortunately, most bases are not orthonormal. But this doesn’t really matter. For, starting from any given basis, we can successively alter the vectors in it, gradually changing it into an orthonormal basis. This process is often called the Gram-Schmidt orthonormalization process. But first, to show you why orthonormal bases are good, we have the following theorem.


Theorem 48
Let V have the orthonormal basis {v₁, . . . , v_n}, and let x ∈ V be arbitrary. Then

That is, the coefficients of x, with respect to the orthonormal basis, are simply the scalar products with the respective basis vectors.

Proof
This follows simply because if x = ∑ a_jv_j, with (j = 1, ... , n), then we have for each k,

So now to the Gram-Schmidt process. To begin with, if a non-zero vector v ∈ V is not normalized — that is, its norm is not one — then it is easy to multiply it by a scalar, changing it into a vector with norm one. For we have <v, v> 0. Therefore ||v|| = √<v, v> > 0 and we have

In other words, we simply multiply the vector by the inverse of its norm.


Theorem 49
Every finite dimensional vector space V which has a scalar product has an orthonormal basis.

Proof
The proof proceeds by constructing an orthonormal basis {u₁, . . . , u_n} from a given, arbitrary basis {v₁, . . . , v_n}. To describe the construction, we use induction on the dimension, n. If n = 1 then there is almost nothing to prove. Any non-zero vector is a basis for V, and as we have seen, it can be normalized by dividing by the norm. (That is, scalar multiplication with the inverse of the norm.)

So now assume that n ≥ 2, and furthermore assume that the Gram-Schmidt process can be constructed for any n−1 dimensional space. Let U ⊂ V be the subspace spanned by the first n − 1 basis vectors {v₁, . . . , v_n−1}. Since U is only n − 1 dimensional, our assumption is that there exists an orthonormal basis {u₁, . . . , u_n−1} for U. Clearly, adding in v_n gives a new basis {u₁, . . . , u_n−1, v_n} for V.

[Since both {v₁, . . . , v_n−1} and {u₁, . . . , u_n−1} are bases for U, we can write each v_j as a linear combination of the u_k’s. Therefore {u₁, . . . , u_n−1, v_n} spans V, and since the dimension is n, it must be a basis.]

Unfortunately, this last vector, v_n, might disturb the nice orthonormal character of the other vectors. Therefore, we replace v_n with the new vector (A linear independent set remains linearly independent if one of the vectors has some linear combination of the other vectors added on to it.)