27 Polynomials
A big part of the high-school algebra curriculum is about polynomials. In some ways, this is appropriate since polynomials played an outsized part in the historical development of mathematical theory. Indeed, the so-called “Fundamental theorem of algebra” is about polynomials.1
For modelers, polynomials are a mixed bag. They are very widely used in modeling. Sometimes this is entirely appropriate, for instance the low-order polynomials that are the subject of Chapter 26. The problems come when high-order polynomials are selected for modeling purposes. Building a reliable model with high-order polynomials requires a deep knowledge of mathematics, and introduces serious potential pitfalls. Modern professional modelers learn the alternatives to high-order polynomials, but newcomers often draw on their experience in high-school and give unwarranted credence to polynomials. This chapter attempts to guide you to the ways you are likely to see polynomials in your future work and to help you avoid them when better alternatives are available.
27.1 Basics of polynomials with one input
A polynomial is a linear combination of a particular class of functions: power-law functions with non-negative, integer exponents: 1, 2, 3, …. The individual functions are called monomials, a word that echoes the construction of chemical polymers out of monomers; for instance, the material polyester is constructed by chaining together a basic chemical unit called an ester.
In one input, say \(x\), the monomials are \(x^1, x^2, x^3\), and so on. (There is also \(x^0\), but that is better thought of as the constant function.) An n-th order polynomial has monomials up to exponent \(n\). For example, the form of a third-order polynomial is \[a_0 + a_1 x^1 + a_2 x^2 + a_3 x^3\]
The domain of polynomials, like the power-law functions they are assembled from, is the real numbers, that is, the entire number line \(-\infty < x < \infty\). But for the purposes of understanding the shape of high-order polynomials, it is helpful to divide the domain into three parts: a wriggly domain at the center and two tail domains to the right and left of the center.
Figure 27.1 shows a 7th order polynomial—that is, the highest-order term is \(x^7\). In one of the tail domains the function value heads off to \(\infty\), in the other to \(-\infty\). This is a necessary feature of all odd-order polynomials: 1, 3, 5, 7, …
In contrast, for even-order polynomials (2, 4, 6, …) the function value in the two tail domains go in the same direction, either both to \(\infty\) (Hands up!) or both to \(-\infty\).
In the wriggly domain in Figure 27.1, there are six argmins or argmaxes.
An \(n\)th-order polynomial can have up to \(n-1\) extrema.
Note that the local polynomial approximations in Chapter 26 are at most 2nd order and so there is at most 1 wriggle: a unique argmax. If the approximation does not include the quadratic terms (\(x^2\) or \(y^2\)) then there is no argmax for the function.
27.2 Multiple inputs?
High-order polynomials are rarely used with multiple inputs. One reason is the proliferation of coefficients. For instance, here is the third-order polynomial in two inputs, \(x\), and \(y\). \[\underbrace{b_0 + b_x x + b_y y}_\text{first-order terms} + \underbrace{b_{xy} x y + b_{xx} x^2 + b_{yy} y^2}_\text{second-order terms} + \underbrace{b_{xxy} x^2 y + b_{xyy} x y^2 + b_{xxx} x^3 + b_{yyy} y^3}_\text{third-order terms}\]
This has 10 coefficients. With so many coefficients it is hard to ascribe meaning to any of them individually. And, insofar as some feature of the function does carry meaning in terms of the modeling situation, that meaning is spread out and hard to quantify.
27.3 High-order approximations
The potential attraction of high-order polynomials is that, with their wriggly interior, they can take on a large number of appearances. This chameleon-like behavior has historically made them the tool of choice for understanding the behavior of approximations. That theory has motivated the use of polynomials for modeling patterns in data, but, paradoxically, has shown that high-order polynomials should not be the tool of choice for modeling data.2
Polynomial functions lend themselves well to calculations, since the output from a polynomial function can be calculated using just the basic arithmetic functions: addition, subtraction, multiplication, and division. To illustrate, consider this polynomial: \[g(x) \equiv x - \frac{1}{6} x^3\] Since the highest-order term is \(x^3\) this is a third-order polynomial. (As you will see, we picked these particular coefficients, 0, 1, 0, -1/6, for a reason.) With such simple coefficients the polynomial is easy to handle by mental arithmetic. For instance, for \(g(x=1)\) is \(5/6\). Similarly, \(g(x=1/2) = 23/48\) and \(g(x=2) = 2/3\). A person of today’s generation would use an electronic calculator for more complicated inputs, but the mathematicians of Newton’s time were accomplished human calculators. It would have been well within their capabilities to calculate, using paper and pencil, \(g(\pi/4) = 0.7046527\).3
Our example polynomial, \(g(x) \equiv x - \frac{1}{6}x^3\), graphed in color in Figure 27.2, does not look exactly like the sinusoid. If we increased the extent of the graphics domain, the disagreement would be even more striking, since the sinusoid’s output is always in \(-1 \leq \sin(x) \leq 1\), while the polynomial’s tails are heading off to \(\infty\) and \(-\infty\). But, for a small interval around \(x=0\), exactly aligns with the sinusoid.
It is clear from the graph that the approximation is excellent near \(x=0\) and gets worse as \(x\) gets larger. The approximation is poor for \(x \approx \pm 2\). We know enough about polynomials to say that the approximation will not get better for larger \(x\); the sine function has a range of \(-1\) to \(1\), while the left and right tails of the polynomial are heading off to \(\infty\) and \(-\infty\) respectively.
One way to measure the quality of the approximation is the error \({\cal E}(x)\) which gives, as a function of \(x\), the difference between the actual sinusoid and the approximation: \[{\cal E}(x) \equiv |\strut\sin(x) - g(x)|\] The absolute value used in defining the error reflects our interest in how far the approximation is from the actual function and not so much in whether the approximation is below or above the actual function. Figure 27.3 shows \({\cal E}(x)\) as a function of \(x\). Since the error is the same on both sides of \(x=0\), only the positive \(x\) domain is shown.
Figure 27.3 shows that for \(x < 0.3\), the error in the polynomial approximation to \(\sin(x)\) is in the 5th decimal place. For instance, \(\sin(0.3) = 0.2955202\) while \(g(0.3) = 0.2955000\).
That the graph of \({\cal E}(x)\) is a straight-line on log-log scales diagnoses \({\cal E}(x)\) as a power law. That is: \({\cal E}(x) = A x^p\). As always for power-law functions, we can estimate the exponent \(p\) from the slope of the graph. It is easy to see that the slope is positive, so \(p\) must also be positive.
The inevitable consequence of \({\cal E}(x)\) being a power-law function with positive \(p\) is that \(\lim_{x\rightarrow 0} {\cal E}(x) = 0\). That is, the polynomial approximation \(x - \frac{1}{6}x^3\) is exact as \(x \rightarrow 0\).
Throughout this book, we’ve been using straight-line approximations to functions around an input \(x_0\). \[g(x) = f(x_0) + \partial_x f(x_0) [x-x_0]\] One way to look at \(g(x)\) is as a straight-line function. Another way is as a first-order polynomial. This raises the question of what a second-order polynomial approximation should be. Rather than the polynomial matching just the slope of \(f(x)\) at \(x_0\), we can arrange things so that the second-order polynomial will also match the curvature of the \(f()\). Since the curvature involves only the first and second derivatives of a function, the polynomial constructed to match both the first and the second derivative will necessarily match the slope and curvature of \(f()\). This can be accomplished by setting the polynomial coefficients appropriately.
Start with a general, second-order polynomial centered around \(x_0\): \[g(x) \equiv a_0 + a_1 [x-x_0] + a_2 [x - x_0]^2\] The first- and second-derivatives, evaluated at \(x=x_0\) are: \[\partial_x g(x)\left.{\Large\strut}\right|_{x=x_0} = a_1 + 2 a_2 [x - x_0] \left.{\Large\strut}\right|_{x=x_0} = a_1\] \[\partial_{xx} g(x)\left.{\Large\strut}\right|_{x=x_0} = 2 a_2\] Notice the 2 in the above expression. When we want to write the coefficient \(a_2\) in terms of the second derivative of \(g()\), we will end up with
\[a_2 = \frac{1}{2} \partial_{xx} g(x)\left.{\Large\strut}\right|_{x=x_0}\]
To make \(g(x)\) approximate \(f(x)\) at \(x=x_0\), we need merely set \[a_1 = \partial_x f(x)\left.{\Large\strut}\right|_{x=x_0}\] and \[a_2 = \frac{1}{2} \partial_{xx} f(x) \left.{\Large\strut}\right|_{x=x_0}\] This logic can also be applied to higher-order polynomials. For instance, to match the third derivative of \(f(x)\) at \(x_0\), set \[a_3 = \frac{1}{6} \partial_{xxx} f(x) \left.{\Large\strut}\right|_{x=x_0}\] Remarkably, each coefficient in the approximating polynomial involves only the corresponding order of derivative. \(a_1\) involves only \(\partial_x f(x) \left.{\Large\strut}\right|_{x=x_0}\); the \(a_2\) coefficient involves only \(\partial_{xx} f(x) \left.{\Large\strut}\right|_{x=x_0}\); the \(a_3\) coefficient involves only \(\partial_{xx} f(x) \left.{\Large\strut}\right|_{x=x_0}\), and so on.
Now we can explain where the polynomial that started this section, \(x - \frac{1}{6} x^3\) came from and why those coefficients make the polynmomial approximate the sinusoid near \(x=0\).
| Order | \(\sin(x)\) derivative | \(x - \frac{1}{6}x^3\) derivative |
|---|---|---|
| 0 | \(\sin(x) \left.{\Large\strut}\right|_{x=0} = 0\) | \(\left( 1 - \frac{1}{6}x^3\right)\left.{\Large\strut}\right|_{x=0} = 0\) |
| 1 | \(\cos(x) \left.{\Large\strut}\right|_{x=0} = 1\) | \(\left(1 - \frac{3}{6} x^2\right) \left.{\Large\strut}\right|_{x=0}= 1\) |
| 2 | \(-\sin(x) \left.{\Large\strut}\right|_{x=0} = 0\) | \(\left(- \frac{6}{6} x\right) \left.{\Large\strut}\right|_{x=0} = 0\) |
| 3 | \(-\cos(x) \left.{\Large\strut}\right|_{x=0} = -1\) | \(- 1\left.{\Large\strut}\right|_{x=0} = -1\) |
| 4 | \(\sin(x) \left.{\Large\strut}\right|_{x=0} = 0\) | \(0\left.{\Large\strut}\right|_{x=0} = 0\) |
The first four derivatives of \(x - \frac{1}{6} x^3\) exactly match, at \(x=0\), the first four derivatives of \(\sin(x)\).
The polynomial constructed by matching successive derivatives of a function \(f(x)\) at some input \(x_0\) is called a Taylor polynomial.
Tip 27.1: Practice: a Taylor polynomial for \(e^x\).
Let’s construct a 3rd-order Taylor polynomial approximation to \(f(x) = e^x\) around \(x=0\).
We know it will be a 3rd order polynomial: \[g_{\exp}(x) \equiv a_0 + a_1 x + a_2 x^2 + a_3 x^3\] The exponential function is particularly nice for examples because the function value and all its derivatives are identical: \(e^x\). So
\[f(x= 0) = 1\]
\[ \partial_x f(x=0) = 1\] \[\partial_{xx} f(x=0) = 1\] \[\partial_{xxx} f(x=0) = 1\] and so on.
The function value and derivatives of \(g_{\exp}(x)\) at \(x=0\) are: \[g_{\exp}(x=0) = a_0\] \[\partial_{x}g_{\exp}(x=0) = a_1\] \[\partial_{xx}g_{\exp}(x=0) = 2 a_2\]
\[\partial_{xxx}g_{\exp}(x=0) = 2\cdot3\cdot a_3 = 6\, a_3\] Matching these to the exponential evaluated at \(x=0\), we get \[a_0 = 1\] \[a_1 = 1\] \[a_2 = \frac{1}{2}\] \[a_3 = \frac{1}{2 \cdot 3} = \frac{1}{6}\]
Result: the 3rd-order Taylor polynomial approximation to the exponential at \(x=0\) is \[g_{\exp}(x) = 1 + x + \frac{1}{2} x^2 + \frac{1}{2\cdot 3} x^3 +\frac{1}{2\cdot 3\cdot 4} x^4\]
Figure 27.4 shows the exponential function \(e^x\) and its 3th-order Taylor polynomial approximation near \(x=0\):
The polynomial is exact at \(x=0\). The error \({\cal E}(x)\) grows with increasing distance from \(x=0\):
The plot of \(\log_{10} {\cal E}(x)\) versus \(\log_{10} | x |\) in Figure 27.5 shows that the error grows from zero at \(x=0\) as a power-law function. Measuring the exponent of the power-law from the slope of the graph on log-log axes give \({\cal E}(x) = a |x-x_0|^5\). This is typical of Taylor polynomials: for a polynomial of degree \(n\), the error will grow as a power-law with exponent \(n+1\). This means that the higher is \(n\), the faster \(\lim_{x\rightarrow x_0}{\cal E}(x) \rightarrow 0\). On the other hand, since \({\cal E}_x\) is a power law function, as \(x\) gets further from \(x_0\) the error grows as \(\left(x-x_0\right)^{n+1}\).
Calculus history—Polynomial models of other functions
Brooke Taylor (1685-1731), a near contemporary of Newton, published his work on approximating polynomials in 1715. Wikipedia reports: “[T]he importance of [this] remained unrecognized until 1772, when Joseph-Louis Lagrange realized its usefulness and termed it ‘the main [theoretical] foundation of differential calculus’.”Source
Due to the importance of Taylor polynomials in the development of calculus, and their prominence in many calculus textbooks, many students assume their use extends to constructing models from data. They also assume that third- and higher-order monomials are a good basis for modeling data. Both these assumptions are wrong. Least squares is the proper foundation for working with data.
Taylor’s work preceded by about a century the development of techniques for working with data. One of the pioneers in these new techniques was Carl Friedrich Gauss (1777-1855), after whom the gaussian function is named. Gauss’s techniques are the foundation of an incredibly important statistical method that is ubiquitous today: least squares. Least squares provides an entirely different way to find the coefficients on approximating polynomials (and an infinite variety of other function forms). The R/mosaic fitModel() function for polishing parameter estimates is based on least squares. In Block 5, we will explore least squares and the mathematics underlying the calculations of least-squares estimates of parameters.
27.4 Indeterminate forms
Let’s return to an issue that has bedeviled calculus students since Newton’s time. The example we will use is the function \[\text{sinc}(x) \equiv \frac{\sin(x)}{x}\]
The sinc() function (pronounced “sink”) is still important today, in part because of its role in converting discrete-time measurements (as in an mp3 recording of sound) into continuous signals.
What is the value of \(\text{sinc}(0)\)? One answer, favored by arithmetic teachers is that \(\text{sinc}(0)\) is meaningless, because it involves division by zero.
On the other hand, \(\sin(0) = 0\) as well, so the sinc function evaluated at zero involves 0/0. This quotient is called an indeterminant form. The logic is this: Suppose we assume that \(0/0 = b\) for some number \(b\). then \(0 = 0 \times b = 0\). So any value of \(b\) would do; the value of \(0/0\) is “indeterminant.”
Still another answer is suggested by plotting out sinc(\(x\)) near \(x=0\) and reading the value off the graph: sinc(0) = 1.
To judge from the output of Active R chunk 27.1, \(\sin(0) / 0 = 1\).
The graph of sinc() looks smooth and the shape makes sense. Even if we zoom in very close to \(x=0\), the graph continues to look smooth. We call such functions well behaved.
Compare the well-behaved sinc() to a very closely related function (which does not seem to be so important in applied work): \(\frac{\sin(x)}{x^3}\).
Both \(\sin(x)/x\) and \(\sin(x) / x^3\), evaluated at \(x=0\) involve a divide by zero. Both are indeterminate forms 0/0 at \(x=0\). But the graph of \(\sin(x) / x^3\) (see Active R chunk 27.2) is not will behaved. \(\sin(x) / x^3\) does not have any particular value at \(x=0\); instead, it has an asymptote.
Active R chunk 27.2 zooms in around the division by zero. Top: The graph of \(\sin(x)/x\) versus \(x\). Bottom: The graph of \(\sin(x)/x^2\). The vertical scales on the two graphs are utterly different.
Since both \(\sin(x)/x\left.{\Large\strut}\right|_{x=0}\) and \(\sin(x)/x^3\left. {\Large\strut}\right|_{x=0}\) involve a divide-by-zero, the answer to the utterly different behavior of the two functions is not to be found at zero. Instead, it is to be found near zero. For any non-zero value of \(x\), the arithmetic to evaluate the functions is straight-forward. Note that \(\sin(x) / x^3\) starts its mis-behavior away from zero. The slope of \(\sin(x) / x^3\) is very large near \(x=0\), while the slope of \(\sin(x) / x\) smoothly approaches zero.
Since we are interested in behavior near \(x=0\), a useful technique is to approximate the numerator and denominator of both functions by polynomial approximations.
- \(\sin(x) \approx x - \frac{1}{6} x^3\) near \(x=0\)
- \(x\) is already a polynomial.
- \(x^3\) is already a polynomial.
Remember, these approximations are exact as \(x\) goes to zero. So sufficiently close to zero,
\[\frac{\sin(x)}{x} = \frac{x - \frac{1}{6} x^3}{x} = 1 + \frac{1}{6} x^2\] Even at \(x=0\), there is nothing indeterminant about \(1 + x^2/6\); it is simply 1.
Compare this to the polynomial approximation to \(\sin(x) / x^3\): \[\frac{\sin(x)}{x^3} = \frac{x - \frac{1}{6} x^3}{x^3} = \frac{1}{x^2} - \frac{1}{6}\]
Evaluating this at \(x=0\) involves division by zero. No wonder it is badly behaved.
The procedure for checking whether a function involving division by zero behaves well or poorly is described in the first-ever calculus textbook, published in 1697. The title (in English) is: The analysis into the infinitely small for the understanding of curved lines. In honor of the author, the Marquis de l’Hospital, the procedure is called l’Hopital’s rule.4
Conventionally, the relationship is written \[\lim_{x\rightarrow x_0} \frac{u(x)}{v(x)} = \lim_{x\rightarrow x_0} \frac{\partial_x u(x)}{\partial_x v(x)}\]
Let’s try this out with our two example functions around \(x=0\):
\[\lim_{x\rightarrow 0} \frac{\sin(x)}{x} = \frac{\lim_{x\rightarrow 0} \cos(x)}{\lim_{x \rightarrow 0} 1} = \frac{1}{1} = 1\]
\[\lim_{x\rightarrow 0} \frac{\sin(x)}{x^3} = \frac{\lim_{x\rightarrow 0} \cos(x)}{\lim_{x \rightarrow 0} 3x^2} = \frac{1}{0} \ \ \ \ \text{Indeterminate}!\]
27.5 Computing with indeterminate forms
In the early days of electronic computers, division by zero would cause a fault in the computer, often signaled by stopping the calculation and printing an error message to some display. This was inconvenient, since programmers did not always forsee division-by-zero situations and avoid them.
As you’ve seen, modern computers have adopted a convention that simplifies programming considerably. Instead of stopping the calculation, the computer just carries on normally, but produces as a result one of two indeterminant forms: Inf and NaN.
Inf is the output for the simple case of dividing zero into a non-zero number, for instance:
NaN, standing for “not a number,” is the output for more challenging cases: dividing zero into zero, or multiplying Inf by zero, or dividing Inf by Inf.
The brilliance of the idea is that any calculation that involves NaN will return a value of NaN. This might seem to get us nowhere. But most programs are built out of other programs, usually written by other people interested in other applications. You can use those programs (mostly) without worrying about the implications of a divide by zero. If it is important to respond in some particularly way, you can always check the result for being NaN in your own programs. (Much the same is true for Inf, although dividing a non-Inf number by Inf will return 0.)
Plotting software will often treat NaN values as “don’t plot this.” that is why it is possible to make a sensible plot of \(\sin(x)/x\) even when the plotting domain includes zero.
The fundamental theorem of algebra says that an order-n polynomial has n roots (including multiplicities).↩︎
The mathematical background needed for those better tools won’t be available to us until Block 5, when we explore linear algebra.↩︎
Unfortunately for these human calculators, pencils weren’t invented until 1795. Prior to the introduction of this advanced, graphite-based computing technology, mathematicians had to use quill and ink.↩︎
In many French words, the sequence “os” has been replaced by a single, accented letter, \(\hat{o}\).↩︎