Steven R. Dunbar
Department of Mathematics
203 Avery Hall
University of Nebraska-Lincoln
Lincoln, NE 68588-0130
http://www.math.unl.edu
Voice: 402-472-3731
Fax: 402-472-8466

Topics in
Probability Theory and Stochastic Processes
Steven R. Dunbar

__________________________________________________________________________

Smoothed Analysis of Linear Optimization

_______________________________________________________________________

Note: To read these pages properly, you will need the latest version of the Mozilla Firefox browser, with the STIX fonts installed. In a few sections, you will also need the latest Java plug-in, and JavaScript must be enabled. If you use a browser other than Firefox, you should be able to access the pages and run the applets. However, mathematical expressions will probably not display correctly. Firefox is currently the only browser that supports all of the open standards.

_______________________________________________________________________________________________

Rating

Rating

Mathematicians Only: prolonged scenes of intense rigor.

_______________________________________________________________________________________________

QuestionofDay

Question of the Day

__________________________________________________________________________

Key Concepts

Key Concepts

  1. The performance time of an algorithm is usually expressed by its running time, expressed as a function of the input size of the problem it solves.
  2. The performance profiles of algorithms across the landscape of input instances can differ greatly.
  3. Average-case analyses employ distributions with concise mathematical descriptions, such as Gaussian random vectors, uniform vectors, and other standard distributions. The drawback of using such distributions is that the inputs in practice may have little resemblance to the inputs that are likely to be generated.
  4. An alternative is to identify typical properties of real data, define an input model that captures these properties, and then rigorously analyzes the performance of algorithms assuming their inputs have these properties. Smoothed analysis is a step in this direction.

__________________________________________________________________________

Vocabulary

Vocabulary

  1. The worst case measure is defined as
    WC A[n] = max xΩnTA[x].

  2. Suppose S provides a distribution over each Ωn, the average case measure corresponding to S is:
    Ave AS[n] = E T A[x]

    where the expectation is over x SΩn indicating that x is randomly chosen from Ωn according to distribution S.

  3. A Gaussian random vector of variance σ2, centered at the origin in Ωn = n is a vector in which each entry is an independent Gaussian random variable of variance σ2 and mean 0.
  4. The smoothed complexity of A with σ-Gaussian perturbations is given by
    Smoothed Aσ2 [n] = max x[1,1]nE TA(x0 + g)

    where g is a σ2-Gaussian random vector.

__________________________________________________________________________

Mathematical Ideas

Mathematical Ideas

Standard Complexity Measures

The performance time of an algorithm is usually expressed by its running time, expressed as a function of the input size of the problem it solves. The performance profiles of algorithms across the landscape of input instances can differ greatly and can be quite irregular. Some algorithms run in time linear in the input size on all instances, some take quadratic or higher order polynomial time, while some may take an exponential amount of time on some instances. For example, we showed in Worst Case and Average Case Behavior of the Simplex Algorithm. that on the Klee-Minty example in n the Simplex Algorithm with Dantzig’s Rule for pivoting takes 2n 1 steps.

Although we normally evaluate the performance of an algorithm by its running time, other performance parameters are often important. These performance measures include the amount of memory space required, the number of bits of precision required to achieve a given output accuracy, the number of cache misses, the error probability of a decision algorithm, the number of random bits needed in a randomized algorithm, the number of calls to a given subroutine, and the number of examples needed in a learning algorithm.

When A is an algorithm for solving problem P, we let TA[x] denote the running time of algorithm A on input instance x. An input domain Ω of all input instances is usually viewed as the union of a family of subdomains {Ω1, Ω2,, Ωn,.} where Ωn represents all instances in Ω of size n.

The worst case measure is defined as

WC A[n] = max xΩnTA[x].

For example, the Klee-Minty example in Worst Case and Average Case Behavior of the Simplex Algorithm. shows that

WC  Simplex[n] C 2n

where C is some constant measuring the running time at each pivot.

The average case measures have more parameters. In each average-case measure, one first determines a distribution of inputs and then measures the expected performance of the algorithm assuming inputs are drawn from this distribution. Supposing S provides a distributions over each Ωn, the average case measure corresponding to S is:

Ave AS[n] = E T A[x]

where the expectation is over x SΩn indicating that x is randomly chosen from Ωn according to distribution S. One would ideally choose the distribution of inputs that occurs in practice, but it is rare that one can determine or cleanly express these distributions. Furthermore, the distributions can vary greatly from one application to another. Instead, average-case analyses have employed distributions with concise mathematical descriptions, such as Gaussian random vectors, uniform vectors, and other standard distributions. The drawback of using such distributions is that the inputs in practice may have little resemblance to the inputs that are likely to be generated.

Smoothed Analysis Measures

An alternative is to identify typical properties of real data, define an input model that captures these properties, and then rigorously analyzes the performance of algorithms assuming their inputs have these properties. Smoothed analysis is a step in this direction. It is motivated by the observation that real data is often subject to some small degree of noise. For example, in industrial optimization and economic prediction, the input parameters could be obtained by physical measurements, and the measurements usually have some low magnitude uncertainty. At a high level, each input is generated from a two-stage model. In the first stage, an instance of the problem is formulated according to say physical, industrial or economic considerations. In the second stage, the instance from the first stage is slightly perturbed. The perturbed instance is the input to the algorithm.

In smoothed analysis, we assume the input to the algorithm is subject to a slight random perturbation The smoothed measure of an algorithm on an input instance is its expected performance over the perturbations of that instance. Define the smoothed complexity of the algorithm to be the maximum smoothed measure over the input instances.

A Gaussian random vector of variance σ2, centered at the origin in ΩN = n is a vector in which each entry is an independent Gaussian random variable of variance σ2 and mean 0, meaning that the probability density of each entry in the vector is

1 2π σ2ex22σ2

For a vector x0 n, the σ-Gaussian perturbation of x0 is a random vector x = x0 + g where g is a Gaussian random vector of variance σ2.

Definition. Suppose A is an algorithm with Ωn = n. Then the smoothed complexity of A with σ-Gaussian perturbations is given by

Smoothed Aσ2 [n] = max x[1,1]nE TA(x0 + g)

where g is a σ2-Gaussian random vector.

In words, this definition says:

  1. Perturb the original input x0 to obtain the input x0 + g
  2. Feed the perturbed input into the algorithm
  3. For each original input, measure the expected running time of the algorithm A on random perturbations of that input.
  4. Then obtain the smoothed analysis by the expectation under the worst possible input.

By varying σ2 between 0 and infinity, one can use smoothed analysis to interpolate between worst-case and average case analysis. When σ2 = 0, one recovers the ordinary worst-case analysis. As σ2 grows large the random perturbation g dominates the original x0 and one obtains an average-case analysis. We are often interested in the case when σ (the standard deviation, measured in the same units as x) is small relative to x in which case x + g is a slight perturbation of x. Smoothed analysis often demonstrates that a perturbed problem is less time-consuming to solve.

Definition. Algorithm A has polynomial smoothed complexity if there exist positive constants n0, σ0, c, k1 and k2 such that for all n n0, and 0 σ σ0

Smoothed Aσ2 [n] c σk2 nk2 .

Recall Markov’s Inequality: If X is a random variable that takes only nonnegative values, then for any a > 0:

X a E Xa

Therefore, if an algorithm A has smoothed complexity T(n,σ), then

max x0[1,1]n TA[x0 + g] δ1T[n,σ] 1 δ

Proof. Need to work through this. □

This says that if A has polynomial smoothed complexity, then for any x0, with probability at least 1 δ, A can solve a random perturbation of x0 in time polynomial in n, 1σ, and 1δ.

This probabilistic upper bound does not imply that smoothed complexity of A is O(T[n,σ]). Blum, Dunagan, Beier, and Vöcking introduced a relaxation of polynomial smoothed complexity:

Definition. Algorithm A has probably polynomial smoothed complexity if there exist constants n0, σ0, c and α such that for n n0 and 0 σ σ0,

max x[1,1]nE TA[x0 + g]α c σ1 n

They show that some algorithms have probably polynomial smoothed complexity, in spite of the fact that their smoothed complexity is unbounded.

Spielman and Teng considered the smoothed complexity of the simplex algorithm with the shadow-vertex pivot rule developed by Gass and Saaty. They show that the smoothed complexity of the algorithm is polynomial. Vershynin improved their result to obtain a smoothed complexity of

O max n5 (log(m))2,n9 (log(m))4,n3 σ4

Sources

This section is adapted from the article “Smoothed Analysis: An attempt to explain the behavior of algorithms in practice’ by Daniel A. Spielman and Shang-Hua Teng, [1].

_______________________________________________________________________________________________

Problems to Work

Problems to Work for Understanding

__________________________________________________________________________

Books

Reading Suggestion:

References

[1]   Daniel A. Spielman and Shang-Hua Teng. Smoothed analysis: An attempt to explain the behavior of algorithms in practice. Communications of the ACM, 52(10):77–84, October 2009.

__________________________________________________________________________

Links

Outside Readings and Links:

__________________________________________________________________________

I check all the information on each page for correctness and typographical errors. Nevertheless, some errors may occur and I would be grateful if you would alert me to such errors. I make every reasonable effort to present current and accurate information for public use, however I do not guarantee the accuracy or timeliness of information on this website. Your use of the information from this website is strictly voluntary and at your risk.

I have checked the links to external sites for usefulness. Links to external websites are provided as a convenience. I do not endorse, control, monitor, or guarantee the information contained in any external website. I don’t guarantee that the links are active at all times. Use the links here with the same caution as you would all information on the Internet. This website reflects the thoughts, interests and opinions of its author. They do not explicitly represent official positions or policies of my employer.

Information on this website is subject to change without notice.

Steve Dunbar’s Home Page, http://www.math.unl.edu/~sdunbar1

Email to Steve Dunbar, sdunbar1 at unl dot edu

Last modified: Processed from LATEX source on January 27, 2011