Główna zawartość

Analiza matematyczna funkcji wielu zmiennych

Kurs: Analiza matematyczna funkcji wielu zmiennych > Rozdział 3

Lekcja 4: Optymalizacja funkcji wielu zmiennych (artykuły)

Maksima, minima i punkty siodłowe

Google Classroom

Learn what local maxima/minima look like for multivariable function.

Kontekst

Do czego zmierzamy

Intuitively, when you're thinking in terms of graphs, local maxima of multivariable functions are peaks, just as they are with single variable functions.
The gradient of a multivariable function at a maximum point will be the zero vector, which corresponds to the graph having a flat tangent plane.
Formally speaking, a local maximum point is a point in the input space such that all other inputs in a small region near that point produce smaller values when pumped through the multivariable function f.

[A note about planes and hyperplanes.]

Optimizing in higher dimensions

One of the most important applications of calculus is its ability to sniff out the maximum or the minimum of a function.

Perhaps you find yourself running a company, and you've come up with some function to model how much money you can expect to make based on a number of parameters, such as employee salaries, cost of raw materials, etc., and you want to find the right combination of resources that will maximize your revenues.
Maybe you are designing a car, hoping to make it more aerodynamic, and you've come up with a function modelling the total wind resistance as a function of many parameters that define the shape of your car, and you want to find the shape that will minimize the total resistance.
In machine learning and artificial intelligence, the way a computer "learns" how to do something is commonly to minimize some "cost function" that the programmer has specified.

Local maxima and minima, visually

Let's start by thinking about those multivariable functions which we can graph: Those with a two-dimensional input, and a scalar output, like this:

f, left parenthesis, x, comma, y, right parenthesis, equals, cosine, left parenthesis, x, right parenthesis, cosine, left parenthesis, y, right parenthesis, e, start superscript, minus, x, squared, minus, y, squared, end superscript

I chose this function because it has lots of nice little bumps and peaks. We call one of these peaks a local maximum, and the plural is local maxima.

The point left parenthesis, x, start subscript, 0, end subscript, comma, y, start subscript, 0, end subscript, right parenthesis underneath a peak in the input space (which in this case means the x, y-plane) is called a local maximum point.
The output of a function at a local maximum point, which you can visualize as the height of the graph above that point, is the local maximum itself.

The word "local" is used to distinguish these from the global maximum of the function, which is the single greatest value that the function can achieve. If you are on the peak of a mountain, it's a local maximum, but unless that mountain is Mt. Everest, it is not a global peak.

I'll give you the formal definition of a local maximum point at the end of this article. Intuitively, it is a special point in the input space where taking a small step in any direction can only decrease the value of the function.

Similarly, if the graph has an inverted peak at a point, we say the function has a local minimum point at the value left parenthesis, x, comma, y, right parenthesis above/below this point on the x, y-plane, and the value of the function at this point is a local minimum. Intuitively, these are points where stepping in any direction can only increase the value of the function.

Stable points in one variable (review)

You may remember the idea of local maxima/minima from single-variable calculus, where you see many problems like this:

Concept check: For what value x is the function f, left parenthesis, x, right parenthesis, equals, minus, left parenthesis, x, minus, 2, right parenthesis, squared, plus, 5 the greatest? What is the maximum value?

x, equals

The maximum value of f is

In general, local maxima and minima of a function f are studied by looking for input values a where f, prime, left parenthesis, a, right parenthesis, equals, 0. This is because as long as the function is continuous and differentiable, the tangent line at peaks and valleys will flatten out, in that it will have a slope of 0.

Such a point a has various names:

Stable point
Critical point
Stationary point

All of these mean the same thing: f, prime, left parenthesis, a, right parenthesis, equals, 0

The requirement that f be continuous and differentiable is important, for if it was not continuous, a lone point of discontinuity could be a local maximum:

And if f is continuous but not differentiable, a local maximum could look like this:

In either case, talking about tangent lines at these maximum points doesn't really make sense, does it?

However, even when f is continuous and differentiable, it is not enough for the derivative to be 0, since this also happens at inflection points:

This means finding stable points is a good way to start the search for a maximum, but it is not necessarily the end.

Stable points in two variables

The story is very similar for multivariable functions. When the function is continuous and differentiable, all the partial derivatives will be 0 at a local maximum or minimum point.

\begin{aligned} \quad \underbrace{ f_\blueE{x}(x_0, y_0, \dots) }_{\text{Partial with respect to $\blueE{x}$}} &= 0 \\ \underbrace{ f_\redE{y}(x_0, y_0, \dots) }_{\text{Partial with respect to $\redE{y}$}} &= 0 \\ &\vdots \end{aligned}

With respect to the graph of a function, this means its tangent plane will be flat at a local maximum or minimum. For instance, here is a graph with many local extrema and flat tangent planes on each one:

Filmy wideo na Khan Academy

Zobacz transkrypcję filmu

Saying that all the partial derivatives are zero at a point is the same as saying the gradient at that point is the zero vector:

\begin{aligned} &\phantom{=} \nabla f(x_0, y_0, \dots) \\\\ &= \left[ \begin{array}{c} f_\blueE{x}(x_0, y_0, \dots) \\ f_\redE{y}(x_0, y_0, \dots) \\ \vdots \end{array} \right] \\\\ &= \left[ \begin{array}{c} 0 \\ 0 \\ \vdots \end{array} \right] \end{aligned}

People often write this more compactly like this:

del, f, left parenthesis, start bold text, x, end bold text, start subscript, 0, end subscript, right parenthesis, equals, start bold text, 0, end bold text

The convention is that bold variable are vectors. So start bold text, x, end bold text, start subscript, 0, end subscript is a vector of the input values left parenthesis, x, start subscript, 0, end subscript, comma, y, start subscript, 0, end subscript, comma, dots, right parenthesis and start bold text, 0, end bold text is the vector with all zeros.

Such an input start bold text, x, end bold text, start subscript, 0, end subscript goes by the same various names as in the single-variable case:

Stable point
Stationary point
Critical point

The thinking behind the words "stable" and "stationary" is that when you move around slightly near this input, the value of the function doesn't change significantly. The word "critical" always seemed a bit over dramatic to me, as if the function is about to die near those points.

As with single variable functions, it is not enough for the gradient to be zero to ensure that a point is a local maximum or minimum. For one thing, you can still have something similar to an inflection point:

But there is also an entirely new possibility, unique to multivariable functions.

Saddle points

Consider the function f, left parenthesis, x, comma, y, right parenthesis, equals, x, squared, minus, y, squared. Let's make a few observations about what goes on around the origin left parenthesis, 0, comma, 0, right parenthesis

Both partial derivatives are 0 at this point:

\begin{aligned} \dfrac{\partial}{\partial \blueE{x}}(\blueE{x}^2 - y^2) &= 2x \to 2(\blueE{0}) = 0 \\\\ \dfrac{\partial}{\partial \redE{y}}(x^2 - \redE{y}^2) &= -2y \to -2(\redE{0}) = 0 \end{aligned}

Therefore left parenthesis, start color #0c7f99, 0, end color #0c7f99, comma, start color #bc2612, 0, end color #bc2612, right parenthesis is a stable point.

When you just move in the x direction around this point, the function looks like f, left parenthesis, x, comma, 0, right parenthesis, equals, x, squared, minus, 0, squared, equals, x, squared. The single-variable function f, left parenthesis, x, right parenthesis, equals, x, squared has a local minimum at x, equals, 0.
When you just move in the y direction around this point, meaning the function looks like f, left parenthesis, 0, comma, y, right parenthesis, equals, 0, squared, minus, y, squared, equals, minus, y, squared. The single-variable function f, left parenthesis, y, right parenthesis, equals, minus, y, squared has a local maximum at y, equals, 0.

In other words, the x and y directions disagree over whether this input should be a maximum or a minimum point. So even though left parenthesis, 0, comma, 0, right parenthesis is a stable point, and is not an inflection point, it cannot be a local maximum or local minimum!

Here's a video of this graph rotating in space:

Filmy wideo na Khan Academy

Zobacz transkrypcję filmu

Doesn't the region around left parenthesis, 0, comma, 0, comma, 0, right parenthesis kind of have the shape of a horse's saddle?

Well, mathematicians thought so, and they had one of those rare moments of deciding on a good name for something: Saddle points. By definition, these are stable points where the function has a local maximum in one direction, but a local minimum in another direction.

Testing maximality/minimality

"Alright,"

I hear you saying,

"so it's not enough for the gradient to be 0 since you might have an inflection point or a saddle point. But how can you tell if a stable point is a local maximum or minimum?"

I'm glad you asked! This is the topic of the next article on the second partial derivative test. For now, let's finish things off with a formal definition of a local maximum.

Formal definition

I've said this before, but the reason to learn formal definitions, even when you already have an intuition, is to expose yourself to how intuitive mathematical ideas are captured precisely. It's good practice for thinking clearly, and it can also help to understand those times when intuition differs from reality.

In defining a local maximum, let's use vector notation for our input, writing it as start bold text, x, end bold text.

Formal definition of a local maximum: A scalar-valued function f has a local maximum at start bold text, x, end bold text, start subscript, 0, end subscript if there exists some positive number r, is greater than, 0, thought of as a radius, such that the following statement is true:

f, left parenthesis, start bold text, x, end bold text, right parenthesis, is less than or equal to, f, left parenthesis, start bold text, x, end bold text, start subscript, 0, end subscript, right parenthesis for all start bold text, x, end bold text such that vertical bar, vertical bar, start bold text, x, end bold text, minus, start bold text, x, end bold text, start subscript, 0, end subscript, vertical bar, vertical bar, is less than, r

That's a bit of a mouthful, so let's break it down:

Saying "vertical bar, vertical bar, start bold text, x, end bold text, minus, start bold text, x, end bold text, start subscript, 0, end subscript, vertical bar, vertical bar, is less than, r" means the variable start bold text, x, end bold text is within a distance r of the maximum point start bold text, x, end bold text, start subscript, 0, end subscript. When start bold text, x, end bold text is two-dimensional this is the same as saying start bold text, x, end bold text lies inside a circle of radius r centered at the point start bold text, x, end bold text, start subscript, 0, end subscript.

More generally, if start bold text, x, end bold text is n-dimensional, the set of all start bold text, x, end bold text such that vertical bar, vertical bar, start bold text, x, end bold text, minus, start bold text, x, end bold text, start subscript, 0, end subscript, vertical bar, vertical bar, is less than, r forms an n-dimensional ball with radius r centered at start bold text, x, end bold text, start subscript, 0, end subscript.

We can then translate this definition from math-speak to something more closely resembling English as follows:

start bold text, x, end bold text, start subscript, 0, end subscript is a maximum point of f if there is some small (ball-shaped) region in the input space around the point start bold text, x, end bold text, start subscript, 0, end subscript such that the highest possible value you can get for f evaluated on points in that region is achieved at the point start bold text, x, end bold text, start subscript, 0, end subscript.

Test your understanding: Write the formal definition for a local minimum, and think about what each component means as you write it down. (Resist the temptation to just copy down the words in the definition above.)

[Odpowiedź]

Podsumowanie

Intuitively, when you're thinking in terms of graphs, local maxima of multivariable functions are peaks, just as they are with single variable functions.
The gradient of a multivariable function at a maximum point will be the zero vector, which corresponds to the graph having a flat tangent plane.
Formally speaking, a local maximum point is a point in the input space such that all other inputs in a small region near that point produce smaller values when pumped through the multivariable function f.

Chcesz dołączyć do dyskusji?

Zaloguj się

Sortuj według

Na razie brak głosów w dyskusji

Rozumiesz angielski? Kliknij tutaj, aby zobaczyć więcej dyskusji na angielskiej wersji strony Khan Academy.