Chapter 5 Modeling with linear combinations

5.1 Linear algebra

The computations for performing linear algebra operations are among the most important in science. It’s so important that the unit used to measure computer performance for scientific computation is called a “flop”, standing for “floating point operation” and is defined in terms of a linear algebra calculation.

For you, the issue with using the computer to perform linear algebra is mainly how to set up the problem so that the computer can solve it. The notation that we will use has been chosen specifically to relate to the kinds of problems for which you will be using linear algebra: fitting models to data. This means that the notation will be very compact.

The basic linear algebra operations of importance are:

  • Project a single vector onto the space defined by a set of vectors.
  • Make a linear combination of vectors.

In performing these operations, you will use two main functions, project( ) and mat( ), along with the ordinary multiplication * and addition + operations. There is also a new sort of operation that provides a compact description for taking a linear combination: “matrix multiplication,” written %*%.

By the end of this ;ession, you should feel comfortable with those two functions and the new form of multiplication %*%.

To start, consider the sort of linear algebra problem often presented in textbooks in the form of simultaneous linear equations. For example: \[\begin{array}{rcrcr} x & + & 5 y & = &1\\ 2x & + & -2 y & = &1\\ 4x & + & 0 y & = & 1\\ \end{array} .\]

Thinking in terms of vectors, this equation can be re-written as \[ x \left(\begin{array}{r}1\\2\\4\end{array}\right) + y \left(\begin{array}{r}5\\-2\\0\end{array}\right) = \left(\begin{array}{r}1\\1\\1\end{array}\right) .\]

Solving this vector equation involves projecting the vector \(\vec{b} = \left(\begin{array}{r}1\\1\\1\end{array}\right)\) onto the space defined by the two vectors \(\vec{v}_1 = \left(\begin{array}{r}1\\2\\4\end{array}\right)\) and
\(\vec{v}_2 = \left(\begin{array}{r}5\\-2\\0\end{array}\right)\). The solution, \(x\) and \(y\) will be the number of multiples of their respective vectors needed to reach the projected vectors.

When setting this up with the R notation that you will be using, you need to create each of the vectors \(\vec{b}, \vec{v}_1\), and \(\vec{v}_2\). Here’s how:

The projection is accomplished using the project() function:

##         v1         v2 
## 0.32894737 0.09210526

Read this as "project \(\vec{b}\) onto the subspace defined by \(\vec{v}_1\) and \(\vec{v}_1\).

The answer is given in the form of the multiplier on \(\vec{v}_1\) and \(\vec{v}_2\), that is, the values of \(x\) and \(y\) in the original problem. This answer is the “best” in the sense that these particular values for \(x\) and \(y\) are the ones that come the closest to \(\vec{b}\), that is, the linear combination that give the projection of \(\vec{b}\) onto the subspace defined by \(\vec{v}_1\) and \(\vec{v}_2\).

If you want to see what that projection is, just multiply the coefficients by the vectors and add them up. In other words, take the linear combination

## [1] 0.7894737 0.4736842 1.3157895

When there are lots of vectors involved in the linear combination, it’s much easier to be able to refer to all of them by a single object name. The mat( ) function takes the vectors and packages them together into a matrix. It works just like project( ), but doesn’t involve the vector that’s being projected onto the subspace. Like this:

##      v1 v2
## [1,]  1  5
## [2,]  2 -2
## [3,]  4  0

Notice that \(A\) doesn’t have any new information; it’s just the two vectors \(\vec{v}_1\) and \(\vec{v}_2\) placed side by side.

Let’s do the projection again:

##         v1         v2 
## 0.32894737 0.09210526

To get the linear combination of the vectors in \(A\), you matrix-multiply the matrix \(A\) times the solution \(z\):

##           [,1]
## [1,] 0.7894737
## [2,] 0.4736842
## [3,] 1.3157895

Notice, it’s the same answer you got when you did the multiplication “by hand.”

When working with data, statisticians almost always include another vector called the intercept which is simply a vector of all 1s. You can denote the intercept vector with a plain 1 in the mat() or project() function, like this:

##      (Intercept) v1 v2
## [1,]           1  1  5
## [2,]           1  2 -2
## [3,]           1  4  0
## A(Intercept)          Av1          Av2 
## 1.000000e+00 0.000000e+00 2.775558e-17
##      [,1]
## [1,]    1
## [2,]    1
## [3,]    1

Notice that the matrix A has a third vector: the intercept vector. The solution consequently has three coefficients. Notice as well that the linear combination of the three vectors exactly reaches the vector \(\vec{b}\). That’s because now there are three vectors that define the subspace: \(\vec{v}_1\), \(\vec{v}_2\), and the intercept vectors of all ones.

5.1.1 Example: Atomic bomb data.

The data file blastdata.csv contains measurements of the radius of the fireball from an atomic bomb (in meters) versus time (in seconds). In the analysis of these data, it’s appropriate to look for a power-law relationship between radius and time. This will show up as a linear relationship between log-radius and log-time. In other words, we want to find \(m\) and \(b\) in the relationship log-radius \(= m\) log-time \(+ b\). This amounts to the projection

## (Intercept)   log(time) 
##   6.2946893   0.3866425

The parameter \(m\) is the coefficient on log-time, found to be 0.3866.

5.1.2 Exercises

5.1.2.1 Exercise 1

Remember all those “find the line that goes through the points problems” from algebra class. They can be a bit simpler with the proper linear-algebra tools.

Example: “Find the line that goes through the points \((2,3)\) and \((7,-8)\).”

One way to interpret this is that we are looking for a relationship between \(x\) and \(y\) such that \(y = mx + b\). In vector terms, this means that the \(x\)-coordinates of the two points, \(2\) and \(7\), made into a vector \(\left(\begin{array}{c}2\\7\end{array}\right)\) will be scaled by \(m\), and an intercept vector \(\left(\begin{array}{c}1\\1\end{array}\right)\) will be scaled by \(b\).

## (Intercept)           x 
##         7.4        -2.2

Now you know \(m\) and \(b\).

YOUR TASKS: For each of the following, find the line that goes through the two Cartesian points using the project( ) function. Remember, the vectors involved in the projection will have the form \[\vec{x}=\left(\begin{array}{r}x_1\\x_2\end{array}\right) \mbox{and} \ \ \vec{y}=\left(\begin{array}{r}y_1\\y_2\end{array}\right)\] and

  1. Find the line that goes through the two points \((x_1=9, y_1=1)\) and \((x_2=3, y_2=7)\).
  1. \(y = x + 2\)
  2. \(y = -x + 10\)
  3. \(y=x + 0\)
  4. \(y = -x + 0\)
  5. \(y = x - 2\)

ANSWER:

## (Intercept)     c(9, 3) 
##          10          -1
  1. Find the line that goes through the origin \((x_1=0, y_1=0)\) and \((x_2=2,y_2=-2)\).
  1. \(y = x + 2\)
  2. \(y = -x + 10\)
  3. \(y=x + 0\)
  4. \(y = -x + 0\)
  5. \(y = x - 2\)

ANSWER:

##   (Intercept)       c(0, 2) 
## -5.551115e-17 -1.000000e+00
  1. Find the line that goes through \((x_1=1, y_1=3)\) and \((x_2=7, y_2=9)\)
  1. \(y = x + 2\)
  2. \(y = -x + 10\)
  3. \(y=x + 0\)
  4. \(y = -x + 0\)
  5. \(y = x - 2\)

ANSWER:

## (Intercept)     c(1, 7) 
##           2           1

5.1.2.2 Exercise 2

  1. Find \(x\), \(y\), and \(z\) that solve the following: \[ x \left(\begin{array}{r}1\\2\\4\end{array}\right) + y \left(\begin{array}{r}5\\-2\\0\end{array}\right) + z \left(\begin{array}{r}1\\-2\\3\end{array}\right) = \left(\begin{array}{r}1\\1\\1\end{array}\right) .\]

What’s the value of \(x\)?: {-0.2353,0.1617,0.4265,1.3235,1.5739}

ANSWER:

##  c(1, 2, 4) c(5, -2, 0) c(1, -2, 3) 
##   0.4264706   0.1617647  -0.2352941
  1. Find \(x\), \(y\), and \(z\) that solve the following: \[ x \left(\begin{array}{r}1\\2\\4\end{array}\right) + y \left(\begin{array}{r}5\\-2\\0\end{array}\right) + z \left(\begin{array}{r}1\\-2\\3\end{array}\right) = \left(\begin{array}{r}1\\4\\3\end{array}\right) .\]

What’s the value of \(x\)? {-0.2353,0.1617,0.4264,1.3235,1.5739}

ANSWER:

##  c(1, 2, 4) c(5, -2, 0) c(1, -2, 3) 
##  1.32352941  0.08823529 -0.76470588

5.1.2.3 Exercise 3

Using project( ), solve these sets of simultaneous linear equations for \(x\), \(y\), and \(z\):

Two equations in two unknowns: \[\begin{array}{rcrcr} x & + & 2 y & = &1\\ 3 x & + & 2 y & = &7\\ \end{array}\]

  1. \(x=3\) and \(y=-1\)
  2. \(x=1\) and \(y=3\)
  3. \(x=3\) and \(y=3\)

ANSWER:

##  x  y 
##  3 -1

Three equations in three unknowns: \[\begin{array}{rcrcrcr} x & + & 2 y & + & 7 z & = &1\\ 3 x & + & 2 y & + &2 z&= &7\\ -2 x & + & 3 y & + & z&= &7\\ \end{array} \]

  1. \(x=3.1644\), \(y=-0.8767\), \(z=0.8082\)
  2. \(x=-0.8767\),\(y=0.8082\), \(z=3.1644\)
  3. \(x=0.8082\), \(y=3.1644\), \(z=-0.8767\)

ANSWER:

##          x          y          z 
##  0.8082192  3.1643836 -0.8767123

Four equations in four unknowns: \[\begin{array}{rcrcrcrcr} x & + & 2 y & + & 7 z & +& 8 w& = &1\\ 3 x & + & 2 y & + &2 z& +& 2 w& = &7\\ -2 x & + & 3 y & + & z&+& w&= &7\\ x & + & 5 y & + &3 z&+& w&= &3\\ \end{array} \]

\begin{MultipleChoice} a. \(x=5.500\), \(y=-7.356\), \(z=3.6918\), \(w=1.1096\) #. \(x=1.1096\), \(y=3.6918\), \(z=-7.356\), \(w=5.500\) #. \(x=5.500\), \(y=-7.356\), \(z=1.1096\), \(w=3.6918\) #. \(x=1.1096\), \(y=-7.356\), \(z=5.500\), \(w=3.6918\)

ANSWER:

##         x         y         z         w 
##  1.109589  3.691781 -7.356164  5.500000

Three equations in four unknowns: \[\begin{array}{rcrcrcrcr} x & + & 2 y & + & 7 z & +& 8 w& = &1\\ 3 x & + & 2 y & + &2 z& +& 2 w& = &7\\ -2 x & + & 3 y & + & z&+& w&= &7\\ \end{array} \]

  1. There is no solution.
  2. There is a solution.

ANSWER:

##      [,1]
## [1,]    1
## [2,]    7
## [3,]    7

You may hear it said that there is no solution to a problem of three equations in four unknowns. But a more precise statement is that there are many solutions, an infinity of them. Mathematicians tend to use “a solution” to stand for “a unique, exact solution.” In applied work, neither “unique” nor “exact” mean very much.