Steven R. Dunbar
Department of Mathematics
203 Avery Hall
University of Nebraska-Lincoln
Lincoln, NE 68588-0130
http://www.math.unl.edu
Voice: 402-472-3731
Fax: 402-472-8466

Topics in
Probability Theory and Stochastic Processes
Steven R. Dunbar

__________________________________________________________________________

Eigenvalues, Eigenvectors and Normal Forms of Matrices

_______________________________________________________________________

Rating

Rating

Mathematically Mature: may contain mathematics beyond calculus with proofs.

_______________________________________________________________________________________________

QuestionofDay

Question of the Day

What is the definition of an eigenvalue and an eigenvector of a matrix? What is the geometric meaning and interpretation of an eigenvalue and eigenvector for a matrix?

_______________________________________________________________________________________________

Key Concepts

Key Concepts

  1. The eigenvalues of a real symmetric matrix are real numbers.
  2. Eigenvectors of a symmetric matrix corresponding to distinct eigenvalues are orthogonal.
  3. Let A be a matrix of order n with elements from . Then there exists a unitary matrix U such that
    T = UAU

    is upper triangular. The diagonal elements of T are the eigenvalues of A.

  4. Let A be a Hermitian matrix of order n, so A = A. There is a unitary matrix U for which
    UAU = D = diag[λ 1,,λn]

    is a diagonal matrix with diagonal elements which are the eigenvalues λ1,λn. If A is real, then U can be taken as orthogonal.

__________________________________________________________________________

Vocabulary

Vocabulary

  1. A matrix is stochastic if the column sums i=1nP ij = 1 for j = 1n. This is identical to saying that
    1P = 1

  2. Let A be a matrix of order n with elements from . Then there exists a unitary matrix U such that
    T = UAU

    is upper triangular. The diagonal elements of T are the eigenvalues of A. This similar matrix is called the Schur Normal Form of the matrix A.

__________________________________________________________________________

Mathematical Ideas

Mathematical Ideas

This section provides proofs of some standard facts from linear algebra about eigenvalues, eigenvectors and normal forms of matrices. These facts are needed in the section on the fastest mixing Markov chain.

Some basic facts about eigenvalues of symmetric matrices.

Lemma 1. The eigenvalues of a real symmetric matrix are real numbers.

Proof. Let P be a symmetric matrix, so that PT = P. Let λ be an eigenvalue of P with corresponding eigenvector x. Denote the (complex) inner product on n as (x,y) = i=1nx i ̄yi where xj ̄ is the complex conjugate. Note that we use the complex inner product because λ and x are potentially complex. Then

λ(x,x) = (x,λx) = (x,Px) = (PT x,x) = (Px,x) = (λx,x) = λ̄(x,x).

Hence, λ̄ = λ and λ must be real. □

Lemma 2. Let xk and xl be eigenvectors corresponding to distinct eigenvalues λkλl of the symmetric matrix P. Then

(xk,xl) = 0

That is, eigenvectors of a symmetric matrix corresponding to distinct eigenvalues are orthogonal.

Proof.

λl(xk,xl) = (xk,λlxl) = (xk,Pxl) = (PT x k,xl) = (Pxk,x)l = (λkxk,xl) = λk ̄(xk,xl) = λk ̄(xk,xl).

Then

(xk,xl) = 0.

Remark.

  1. These are standard proofs in numerical analysis, linear algebra, operator theory and many other places in mathematics. They are included here for completeness.
  2. The same proofs show that for a Hermitian matrix (P = P) the eigenvalues are real.

Remark. Actually, more is true and the next two theorems prove that a symmetric matrix has a complete set of orthogonal eigenvectors even if some eigenvalues are repeated. We will need this crucial fact in order to prove an essential inequality about the rate of mixing of Markov chains.

Theorem 3 (Schur Normal Form). Let A be a matrix of order n with elements from . Then there exists a unitary matrix U such that

T = UAU

is upper triangular. The diagonal elements of T are the eigenvalues of A.

Proof. The proof is by induction on the order n of A. The result is trivially true for k = 1 using U = (1). We assume the result is true for all matrices of order n k 1, and we will then prove it to be true of all matrices of order n = k.

Let λ1 be an eigenvalue of A and let u(1) be an associated eigenvector with ||u(1)|| 2 = 1. Beginning with u(1), pick an orthonormal basis for k. (Note that this can be done by filling out a basis with the standard unit vectors and then using the Gram-Schmidt orthogonalization procedure.) Call the basis so obtained {u(1),u(2),u(k)}. Define the k × k column matrix

P1 = u(1),u(2),,u(k)

Note that P1P 1 = I, so that P11 = P 1. Define

B1 = P1AP 1

The claim is that

B1 = λ1α2 αk 0 A 2 0

with A2 of order k 1 and α2,α3,,αk some numbers. To prove this, simply multiply:

AP1 = A u(1),u(2),,u(k) = Au(1),Au(2),,Au(k) = λ1u(1),v(2),,v(k) B1 = P1AP 1 = λ1P1u(1),P 1v(2),,P 1v(k) = λ 1e(1),w(2),,w(k)

The last equality follows from the constructed orthonormality of u(1) and the associated construction of P1 and P1, with e = (1, 0,, 0) and w(j) = P 1(j). Note that B1 has the desired form, and the claim is established.

By the induction hypothesis, there exists a unitary matrix P̂2 of order k 1 for which

T̂ = P̂2A 2P̂2

is upper triangular of order k 1. Define

P2 = 10 0 0 P̂2 0

Then P2 is unitary and

P2B 1P2 = λ1γ2 γk 0 P ̂ 2A 2P̂2 0 = λ1γ2 γk 0 T ̂ 0

is an upper triangular matrix. Thus

T = P2B 1P2 = P2P 1AP 1P2 = (P1P2)A(P 1P2)

Define U = P1P2, which is easily seen to be unitary. This completes the induction and the proof. □

Theorem 4 (Principal Axes Theorem). Let A be a Hermitian matrix of order n, so A = A. Then A has n real eigenvalues λ1,,λn, not necessarily distinct, and n corresponding eigenvectors {u(1),u(2),,u(k)} that form an orthogonal basis for n. If A is real, the eigenvectors can be taken as real and they form an orthonormal basis for n. Finally, there is a unitary matrix U for which

UAU = D = diag[λ 1,,λn]

is a diagonal matrix with diagonal elements λ1,λn. If A is real, then U can be taken as orthogonal.

Proof. From the Schur Normal Form theorem, there is a unitary matrix U with

UAU = T

with T upper triangular. Take the conjugate transpose of both sides to obtain

T = (UAU) = UA(U) = UAU = T.

Since T is upper triangular, then T is lower triangular, and since the two are equal, T must be a diagonal matrix

T = diag[λ1,,λn].

Also, T = T involves complex conjugation of all elements of T and thus all diagonal elements of T must be real.

Write U as

U = u(1),u(2),,u(n) .

Then T = UAU implies AU = UT,

A u(1),u(2),,u(n) = u(1),u(2),,u(n) λ1 0 0 λn Au(1),Au(2),,Au(n) = λ 1u(1),λ 2u(2),,λ nu(n)

Hence,

Au(j) = λ ju(j)j = 1,,n

Since the columns of U are orthonormal, and since the dimension of n is n these must form an orthonormal basis for n. □

Lemma 5. If P is a stochastic matrix, then 1 is an eigenvector corresponding to eigenvalue 1.

Proof. To say the matrix is stochastic is to say that the column sums i=1nP ij = 1 for each row j = 1n. But this is identical to saying that

1T P = 1T

which is to say that 1 is an left eigenvector corresponding to eigenvalue 1 of P. □

Remark. Again, this is a completely standard and well-known fact from Markov chain analysis.

Lemma 6. The eigenvalues of a symmetric, stochastic matrix may be arranged in nonincreasing order:

1 = λ1(P) λ2(P) λn(P) 1

Proof. Since the eigenvalues are real, and one eigenvalue has value 1, the proof must show that the remaining eigenvalues are between 1 and 1. This follows from the Gershgorin disk theorem. The Gershgorin disk around the j-th diagonal element of P is

|z pjj| i=1,ijn|p ij| = 1 pjj

The eigenvalues must lie in the union of these disks in the complex plane. However, we know that 0 pjj 1 and further that the eigenvalues must be real, so 1 λk 1. □

Remark. This is a specialized case of well-known facts from Frobenius theory about the eigenvalues and eigenspaces of non-negative and strictly positive matrices.

Sources

This section is completely standard material that can be found in most books on linear algebra or allied subjects such as numerical analysis, advanced calculus and engineering mathematics, or functional analysis.

_______________________________________________________________________________________________

Problems to Work

Problems to Work for Understanding

__________________________________________________________________________

Books

Reading Suggestion:

References

[1]   S. Karlin and H. Taylor. A Second Course in Stochastic Processes. Academic Press, 1981.

__________________________________________________________________________

Links

Outside Readings and Links:

__________________________________________________________________________

I check all the information on each page for correctness and typographical errors. Nevertheless, some errors may occur and I would be grateful if you would alert me to such errors. I make every reasonable effort to present current and accurate information for public use, however I do not guarantee the accuracy or timeliness of information on this website. Your use of the information from this website is strictly voluntary and at your risk.

I have checked the links to external sites for usefulness. Links to external websites are provided as a convenience. I do not endorse, control, monitor, or guarantee the information contained in any external website. I don’t guarantee that the links are active at all times. Use the links here with the same caution as you would all information on the Internet. This website reflects the thoughts, interests and opinions of its author. They do not explicitly represent official positions or policies of my employer.

Information on this website is subject to change without notice.

Steve Dunbar’s Home Page, http://www.math.unl.edu/~sdunbar1

Email to Steve Dunbar, sdunbar1 at unl dot edu

Last modified: Processed from LATEX source on May 28, 2010