Neural Networks

ECON526

Paul Schrimpf

University of British Columbia

Table of contents

• Introduction
• Double Descent and Overparameterization
• Software
• Uses for Neural Networks

Introduction

\[ \def\indep{\perp\!\!\!\perp} \def\Er{\mathrm{E}} \def\R{\mathbb{R}} \def\En{{\mathbb{E}_n}} \def\Pr{\mathrm{P}} \newcommand{\norm}[1]{\left\Vert {#1} \right\Vert} \newcommand{\abs}[1]{\left\vert {#1} \right\vert} \def\inprob{{\,{\buildrel p \over \rightarrow}\,}} \def\indist{\,{\buildrel d \over \rightarrow}\,} \DeclareMathOperator*{\plim}{plim} \DeclareMathOperator*{\argmax}{arg\,max} \DeclareMathOperator*{\argmin}{arg\,min} \]

Neural Networks

• Flexible function approximation & regression
• Automated feature engineering
• Exceptional performance on high dimensional data with low noise
  • Text
  • Images
  • Audio
  • Video

Single Layer Perceptron

• \(x_i \in \R^d\), want to approximate some \(f: \R^d \to \R\)
• Approximate by \[ \begin{align*} f(x_i; \mathbf{w},\mathbf{b}) = \psi_1 \left( \sum_{u=1}^m w_{u,1} \psi_0( x_i'w_{u,0} + b_{u,0}) + b_{u,1} \right) \end{align*} \] where
  • Weights: \(w_{u,1} \in \R\), \(w_{u,0} \in \R^d\)
  • Biases: \(b_{u,1}, b_{u,0} \in \R\)
  • Activation functions \(\psi_1, \psi_0: \R \to \R\)
  • Width: \(m\)

Activation functions

import numpy as np
import matplotlib.pyplot as plt
import torch.nn as nn
import torch
# plot 4 activation functions
x = torch.linspace(-5, 5, 100)
fig, ax = plt.subplots(2, 2, figsize=(10, 8))
ax = ax.flatten()
ax[0].plot(x, nn.ReLU()(x))
ax[0].set_title("ReLU")
ax[1].plot(x, nn.Tanh()(x))
ax[1].set_title("Tanh")
ax[2].plot(x, nn.Sigmoid()(x))
ax[2].set_title("Sigmoid")
ax[3].plot(x, nn.Softplus()(x))
ax[3].set_title("Softplus")
fig.tight_layout()
fig.show()

Single Layer Perceptron

import torch.nn as nn
import torch
class SingleLayerPerceptron(nn.Module):
    def __init__(self, d, m, activation=nn.ReLU()):
        super().__init__()
        self.d = d
        self.m = m
        self.activation = activation
        self.w0 = nn.Parameter(torch.randn(d, m))
        self.b0 = nn.Parameter(torch.randn(m))
        self.w1 = nn.Parameter(torch.randn(m))
        self.b1 = nn.Parameter(torch.randn(1))

    def forward(self, x):
        return self.activation(x @ self.w0 + self.b0) @ self.w1 + self.b1

Regression

• Data \(\{y_i, x_i\}_{i=1}^n\), \(x_i \in \R^d\), \(y_i \in \R\)
• Least squares: \[ \hat{\mathbf{w}}, \hat{\mathbf{b}} \in \argmin_{\mathbf{w},\mathbf{b}} \underbrace{\frac{1}{n} \sum_{i=1}^n \left(y_i - f(x_i; \mathbf{w},\mathbf{b}) \right)^2}_{\text{loss function}} \]
• Hard to find global minimum:
  • Loss not convex
  • Parameters very high dimensional

Gradient Descent

• Find approximate minimum instead \[ \overbrace{\hat{\theta}}^{\hat{\mathbf{w}}, \hat{\mathbf{b}}} \approx \argmin_{\theta} \ell(y, f(x;\theta)) \]
• Start with random \(\theta_1\)
• Repeatedly update: \[ \theta_{i+1} = \theta_i - \gamma \underbrace{\nabla \ell(y,f(;\theta))}_{\text{gradient}} \]
• Continue until loss stops improving
  • Optionally modify \(\gamma\) based on progress

Computing Gradients

• Automatic differentiation: automatically use chainrule on each step of computation
• E.g. \(\ell(\theta) = f(g(h(\theta)))\)
  • \(\ell : \R^p \to \R\), \(h: \R^p \to \R^q\), \(g: \R^q \to \R^j\), \(f: \R^j \to \R\) \[ \left(\nabla \ell(\theta)\riight)^T = \underbrace{Df_{g(h(\theta))}}_{1 \times j} \underbrace{Dg_{h(\theta)}}_{j \times q} \underbrace{Dh_\theta}_{q \times p} \]
• Forward mode:
  1. Calculate \(h(\theta)\) and \(D_1=Dh_\theta\)
  2. Calculate \(g(h(\theta))\) and \(Dg_{h(\theta)}\), multiply \(D_2=Dg_{h(\theta)} D_1\) (\(jqp\) scalar products and additions)
  3. Calculate \(f(g(h(\theta)))\) and \(Df_{g(h(\theta))}\), multiply \(Df_{g(h(\theta))} D_2\) (\(1jp\) scalar products and additions)
  • Work to propagate derivative \(\propto jqp + 1jp\)

Computing Gradients

• Reverse mode:
  1. “Forward pass” calculate and save \(h(\theta)\), \(g(h(\theta))\) and \(f(g(h(\theta)))\)
  2. “Back propagation”:
     1. Calculate \(D^r_1 = Df_{g(h(\theta))}\)
     2. Calculate \(D^r_2 = D^r_1 Dg_{h(\theta)}\) (\(1jq\) scalar products and additions)
     3. Calculate \(D^r_3 Dh_{\theta}\) (\(1qp\) scalar products and additions)
  • Work to propagate derivative \(\propto 1jq + 1qp\)
• Reverse mode much less work when \(p\) large

Gradient Descent

n = 100
sigma = 0.1
torch.manual_seed(987)
x = torch.randn(n,1)
f = np.vectorize(lambda x:  np.sin(x)+np.exp(-x*x)/(1+np.abs(x)))
y = torch.tensor(f(x), dtype=torch.float32) + sigma*torch.randn(x.shape)
slp = SingleLayerPerceptron(1, 20, nn.ReLU())
lossfn = nn.MSELoss()
xlin = torch.linspace(-3,3,100)
fig, ax = plt.subplots()
epochs = 1000
stepscale = 0.1
for i in range(epochs):
    if (i % 50) == 0 :
        ax.plot(xlin,slp(xlin.reshape(-1,1)).data, alpha=i/epochs, color='k')
        slp.zero_grad()
    loss = lossfn(y.flatten(),slp(x))
    #print(f"{i}: {loss.item()}")
    # calculate gradient
    loss.backward()
    # access gradient & use to descend
    for p in slp.parameters():
        if p.requires_grad:
            p.data = p.data - stepscale*p.grad.data
            p.grad.data.zero_()
    slp.zero_grad()

ax.plot(xlin, f(xlin), label="f", lw=8)
ax.scatter(x.flatten(),y.flatten())
fig.show()

Multi Layer Perceptron

def multilayer(d,width,depth,activation=nn.ReLU()):
    mlp = nn.Sequential(
        nn.Linear(d,width, bias=True),
        activation
    )
    for i in range(depth-1):
        mlp.add_module(f"layer {i+1}",nn.Linear(width, width))
        mlp.add_module(f"activation {i+1}",activation)


    mlp.add_module("output",nn.Linear(width,1))
    return(mlp)

mlp=multilayer(2, 10, 1, nn.ReLU())

def count_parameters(model):
    return sum(p.numel() for p in model.parameters() if p.requires_grad)

Multi Layer Perceptron

mlp = multilayer(1,4,4,nn.ReLU())
print(count_parameters(mlp))
optimizer = torch.optim.Adam(mlp.parameters(), lr=0.01)
fig, ax = plt.subplots()
epochs = 1000
for i in range(epochs):
    if (i % 50) == 0 :
        ax.plot(xlin,mlp(xlin.reshape(-1,1)).data, alpha=i/epochs, color='k')
        mlp.zero_grad()
    loss = lossfn(y,mlp(x))
    print(f"{i}: {loss.item()}")
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

ax.plot(xlin, f(xlin), label="f", lw=8)
ax.scatter(x.flatten(),y.flatten())
fig.show()

73
0: 0.5855244398117065
1: 0.5788913369178772
2: 0.572568416595459
3: 0.5664961934089661
4: 0.5606109499931335
5: 0.554928719997406
6: 0.5494657754898071
7: 0.5442381501197815
8: 0.5392615795135498
9: 0.5345514416694641
10: 0.5301225781440735
11: 0.5259891748428345
12: 0.5221643447875977
13: 0.518659770488739
14: 0.5154853463172913
15: 0.5126491189002991
16: 0.510155975818634
17: 0.5080077052116394
18: 0.5062022805213928
19: 0.5047329068183899
20: 0.503587543964386
21: 0.5027486681938171
22: 0.5021921396255493
23: 0.5018877983093262
24: 0.5017994046211243
25: 0.5018856525421143
26: 0.5021015405654907
27: 0.502400815486908
28: 0.5027384757995605
29: 0.5030739307403564
30: 0.5033730268478394
31: 0.5036104321479797
32: 0.5037701725959778
33: 0.5038456320762634
34: 0.5038385987281799
35: 0.5037573575973511
36: 0.5036147832870483
37: 0.5034269094467163
38: 0.5032098293304443
39: 0.5029798746109009
40: 0.5027512907981873
41: 0.5025356411933899
42: 0.5023422241210938
43: 0.5021766424179077
44: 0.502042293548584
45: 0.5019398331642151
46: 0.5018678903579712
47: 0.5018234252929688
48: 0.5018025040626526
49: 0.5018004775047302
50: 0.5018125772476196
51: 0.501833975315094
52: 0.5018603205680847
53: 0.5018879771232605
54: 0.5019139647483826
55: 0.5019357800483704
56: 0.5019521713256836
57: 0.5019620656967163
58: 0.5019654631614685
59: 0.5019624829292297
60: 0.5019541382789612
61: 0.5019411444664001
62: 0.5019248127937317
63: 0.5019065737724304
64: 0.5018875598907471
65: 0.5018687844276428
66: 0.5018513202667236
67: 0.5018360018730164
68: 0.5018231868743896
69: 0.5018131732940674
70: 0.5018060207366943
71: 0.501801609992981
72: 0.5017996430397034
73: 0.5017995238304138
74: 0.5018008947372437
75: 0.5018033385276794
76: 0.5018061399459839
77: 0.5018091201782227
78: 0.501811683177948
79: 0.5018137097358704
80: 0.5018149614334106
81: 0.5018154382705688
82: 0.5018151998519897
83: 0.5018142461776733
84: 0.501812756061554
85: 0.5018108487129211
86: 0.5018088817596436
87: 0.5018067955970764
88: 0.5018048286437988
89: 0.5018031597137451
90: 0.5018016695976257
91: 0.5018005967140198
92: 0.5017998218536377
93: 0.501799464225769
94: 0.5017993450164795
95: 0.501799464225769
96: 0.5017997026443481
97: 0.5018000602722168
98: 0.5018004179000854
99: 0.5018007159233093
100: 0.5018010139465332
101: 0.5018011331558228
102: 0.5018011927604675
103: 0.5018011927604675
104: 0.501801073551178
105: 0.5018008947372437
106: 0.5018006563186646
107: 0.5018003582954407
108: 0.5018001794815063
109: 0.5017999410629272
110: 0.5017997622489929
111: 0.5017995238304138
112: 0.5017994046211243
113: 0.5017993450164795
114: 0.5017993450164795
115: 0.5017993450164795
116: 0.5017993450164795
117: 0.5017994046211243
118: 0.5017994046211243
119: 0.501799464225769
120: 0.5017995238304138
121: 0.5017995238304138
122: 0.5017996430397034
123: 0.5017995238304138
124: 0.5017995834350586
125: 0.5017995238304138
126: 0.5017995238304138
127: 0.5017994046211243
128: 0.5017994046211243
129: 0.5017994046211243
130: 0.5017994046211243
131: 0.5017994046211243
132: 0.5017993450164795
133: 0.5017993450164795
134: 0.5017993450164795
135: 0.5017993450164795
136: 0.5017993450164795
137: 0.5017994046211243
138: 0.5017994046211243
139: 0.5017994046211243
140: 0.5017993450164795
141: 0.5017994046211243
142: 0.5017993450164795
143: 0.5017994046211243
144: 0.5017993450164795
145: 0.501799464225769
146: 0.5017993450164795
147: 0.5017994046211243
148: 0.5017993450164795
149: 0.5017993450164795
150: 0.5017994046211243
151: 0.5017993450164795
152: 0.5017993450164795
153: 0.5017994046211243
154: 0.5017993450164795
155: 0.5017993450164795
156: 0.5017993450164795
157: 0.5017993450164795
158: 0.5017993450164795
159: 0.5017993450164795
160: 0.5017994046211243
161: 0.5017994046211243
162: 0.5017993450164795
163: 0.5017993450164795
164: 0.5017993450164795
165: 0.5017993450164795
166: 0.5017993450164795
167: 0.5017994046211243
168: 0.5017994046211243
169: 0.5017994046211243
170: 0.5017994046211243
171: 0.5017993450164795
172: 0.5017994046211243
173: 0.5017993450164795
174: 0.5017993450164795
175: 0.5017993450164795
176: 0.5017993450164795
177: 0.5017993450164795
178: 0.5017993450164795
179: 0.5017993450164795
180: 0.5017994046211243
181: 0.5017993450164795
182: 0.5017993450164795
183: 0.5017994046211243
184: 0.5017993450164795
185: 0.5017994046211243
186: 0.5017993450164795
187: 0.5017993450164795
188: 0.5017993450164795
189: 0.5017993450164795
190: 0.5017994046211243
191: 0.5017993450164795
192: 0.5017993450164795
193: 0.5017992854118347
194: 0.5017993450164795
195: 0.5017993450164795
196: 0.5017994046211243
197: 0.5017993450164795
198: 0.5017993450164795
199: 0.5017994046211243
200: 0.5017993450164795
201: 0.5017993450164795
202: 0.5017993450164795
203: 0.5017993450164795
204: 0.5017993450164795
205: 0.5017993450164795
206: 0.5017993450164795
207: 0.5017993450164795
208: 0.5017994046211243
209: 0.5017993450164795
210: 0.5017993450164795
211: 0.5017993450164795
212: 0.5017993450164795
213: 0.5017993450164795
214: 0.5017993450164795
215: 0.5017994046211243
216: 0.5017993450164795
217: 0.5017993450164795
218: 0.5017994046211243
219: 0.5017993450164795
220: 0.5017993450164795
221: 0.5017993450164795
222: 0.5017993450164795
223: 0.5017993450164795
224: 0.5017993450164795
225: 0.5017993450164795
226: 0.5017993450164795
227: 0.5017993450164795
228: 0.5017993450164795
229: 0.5017993450164795
230: 0.5017993450164795
231: 0.5017993450164795
232: 0.5017994046211243
233: 0.5017993450164795
234: 0.5017993450164795
235: 0.5017994046211243
236: 0.5017993450164795
237: 0.5017993450164795
238: 0.5017993450164795
239: 0.5017993450164795
240: 0.5017993450164795
241: 0.5017993450164795
242: 0.5017993450164795
243: 0.5017993450164795
244: 0.5017993450164795
245: 0.5017993450164795
246: 0.5017993450164795
247: 0.5017993450164795
248: 0.5017993450164795
249: 0.5017994046211243
250: 0.5017994046211243
251: 0.5017993450164795
252: 0.5017994046211243
253: 0.5017993450164795
254: 0.5017993450164795
255: 0.5017993450164795
256: 0.5017993450164795
257: 0.5017993450164795
258: 0.5017993450164795
259: 0.5017993450164795
260: 0.5017993450164795
261: 0.5017993450164795
262: 0.5017993450164795
263: 0.5017993450164795
264: 0.5017993450164795
265: 0.5017993450164795
266: 0.5017993450164795
267: 0.5017993450164795
268: 0.5017993450164795
269: 0.5017993450164795
270: 0.5017994046211243
271: 0.5017993450164795
272: 0.5017993450164795
273: 0.5017993450164795
274: 0.5017993450164795
275: 0.5017993450164795
276: 0.5017993450164795
277: 0.5017993450164795
278: 0.5017993450164795
279: 0.5017994046211243
280: 0.5017994046211243
281: 0.5017993450164795
282: 0.5017993450164795
283: 0.5017993450164795
284: 0.5017993450164795
285: 0.5017993450164795
286: 0.5017993450164795
287: 0.5017993450164795
288: 0.5017993450164795
289: 0.5017993450164795
290: 0.5017993450164795
291: 0.5017993450164795
292: 0.5017993450164795
293: 0.5017993450164795
294: 0.5017993450164795
295: 0.5017993450164795
296: 0.5017993450164795
297: 0.5017993450164795
298: 0.5017993450164795
299: 0.5017993450164795
300: 0.5017993450164795
301: 0.5017993450164795
302: 0.5017993450164795
303: 0.5017993450164795
304: 0.5017993450164795
305: 0.5017993450164795
306: 0.5017993450164795
307: 0.5017993450164795
308: 0.5017993450164795
309: 0.5017993450164795
310: 0.5017993450164795
311: 0.5017993450164795
312: 0.5017993450164795
313: 0.5017993450164795
314: 0.5017993450164795
315: 0.5017993450164795
316: 0.5017993450164795
317: 0.5017993450164795
318: 0.5017993450164795
319: 0.5017993450164795
320: 0.5017993450164795
321: 0.5017993450164795
322: 0.5017993450164795
323: 0.5017993450164795
324: 0.5017993450164795
325: 0.5017993450164795
326: 0.5017993450164795
327: 0.5017993450164795
328: 0.5017993450164795
329: 0.5017993450164795
330: 0.5017993450164795
331: 0.5017993450164795
332: 0.5017993450164795
333: 0.5017993450164795
334: 0.5017993450164795
335: 0.5017993450164795
336: 0.5017993450164795
337: 0.5017993450164795
338: 0.5017993450164795
339: 0.5017993450164795
340: 0.5017993450164795
341: 0.5017993450164795
342: 0.5017993450164795
343: 0.5017993450164795
344: 0.5017993450164795
345: 0.5017993450164795
346: 0.5017993450164795
347: 0.5017993450164795
348: 0.5017993450164795
349: 0.5017993450164795
350: 0.5017993450164795
351: 0.5017993450164795
352: 0.5017993450164795
353: 0.5017993450164795
354: 0.5017993450164795
355: 0.5017993450164795
356: 0.5017993450164795
357: 0.5017993450164795
358: 0.5017993450164795
359: 0.5017993450164795
360: 0.5017993450164795
361: 0.5017993450164795
362: 0.5017993450164795
363: 0.5017993450164795
364: 0.5017993450164795
365: 0.5017993450164795
366: 0.5017993450164795
367: 0.5017993450164795
368: 0.5017993450164795
369: 0.5017993450164795
370: 0.5017993450164795
371: 0.5017993450164795
372: 0.5017993450164795
373: 0.5017993450164795
374: 0.5017993450164795
375: 0.5017993450164795
376: 0.5017993450164795
377: 0.5017993450164795
378: 0.5017993450164795
379: 0.5017993450164795
380: 0.5017993450164795
381: 0.5017993450164795
382: 0.5017993450164795
383: 0.5017993450164795
384: 0.5017993450164795
385: 0.5017993450164795
386: 0.5017993450164795
387: 0.5017993450164795
388: 0.5017993450164795
389: 0.5017993450164795
390: 0.5017993450164795
391: 0.5017993450164795
392: 0.5017993450164795
393: 0.5017993450164795
394: 0.5017993450164795
395: 0.5017993450164795
396: 0.5017993450164795
397: 0.5017993450164795
398: 0.5017993450164795
399: 0.5017993450164795
400: 0.5017993450164795
401: 0.5017993450164795
402: 0.5017993450164795
403: 0.5017993450164795
404: 0.5017993450164795
405: 0.5017993450164795
406: 0.5017993450164795
407: 0.5017993450164795
408: 0.5017993450164795
409: 0.5017993450164795
410: 0.5017993450164795
411: 0.5017993450164795
412: 0.5017993450164795
413: 0.5017993450164795
414: 0.5017993450164795
415: 0.5017993450164795
416: 0.5017993450164795
417: 0.5017993450164795
418: 0.5017993450164795
419: 0.5017993450164795
420: 0.5017993450164795
421: 0.5017993450164795
422: 0.5017993450164795
423: 0.5017993450164795
424: 0.5017993450164795
425: 0.5017993450164795
426: 0.5017993450164795
427: 0.5017993450164795
428: 0.5017993450164795
429: 0.5017993450164795
430: 0.5017993450164795
431: 0.5017993450164795
432: 0.5017993450164795
433: 0.5017993450164795
434: 0.5017993450164795
435: 0.5017993450164795
436: 0.5017993450164795
437: 0.5017993450164795
438: 0.5017993450164795
439: 0.5017993450164795
440: 0.5017993450164795
441: 0.5017993450164795
442: 0.5017993450164795
443: 0.5017993450164795
444: 0.5017993450164795
445: 0.5017993450164795
446: 0.5017993450164795
447: 0.5017993450164795
448: 0.5017993450164795
449: 0.5017993450164795
450: 0.5017993450164795
451: 0.5017993450164795
452: 0.5017993450164795
453: 0.5017993450164795
454: 0.5017993450164795
455: 0.5017993450164795
456: 0.5017993450164795
457: 0.5017993450164795
458: 0.5017993450164795
459: 0.5017993450164795
460: 0.5017993450164795
461: 0.5017993450164795
462: 0.5017993450164795
463: 0.5017993450164795
464: 0.5017993450164795
465: 0.5017993450164795
466: 0.5017993450164795
467: 0.5017993450164795
468: 0.5017993450164795
469: 0.5017993450164795
470: 0.5017993450164795
471: 0.5017993450164795
472: 0.5017993450164795
473: 0.5017993450164795
474: 0.5017993450164795
475: 0.5017993450164795
476: 0.5017993450164795
477: 0.5017993450164795
478: 0.5017993450164795
479: 0.5017993450164795
480: 0.5017993450164795
481: 0.5017993450164795
482: 0.5017993450164795
483: 0.5017993450164795
484: 0.5017993450164795
485: 0.5017993450164795
486: 0.5017993450164795
487: 0.5017993450164795
488: 0.5017993450164795
489: 0.5017993450164795
490: 0.5017993450164795
491: 0.5017993450164795
492: 0.5017993450164795
493: 0.5017993450164795
494: 0.5017993450164795
495: 0.5017993450164795
496: 0.5017993450164795
497: 0.5017993450164795
498: 0.5017993450164795
499: 0.5017993450164795
500: 0.5017993450164795
501: 0.5017993450164795
502: 0.5017993450164795
503: 0.5017993450164795
504: 0.5017993450164795
505: 0.5017993450164795
506: 0.5017993450164795
507: 0.5017993450164795
508: 0.5017993450164795
509: 0.5017993450164795
510: 0.5017993450164795
511: 0.5017993450164795
512: 0.5017993450164795
513: 0.5017993450164795
514: 0.5017993450164795
515: 0.5017993450164795
516: 0.5017993450164795
517: 0.5017993450164795
518: 0.5017993450164795
519: 0.5017993450164795
520: 0.5017993450164795
521: 0.5017993450164795
522: 0.5017993450164795
523: 0.5017993450164795
524: 0.5017993450164795
525: 0.5017993450164795
526: 0.5017993450164795
527: 0.5017993450164795
528: 0.5017993450164795
529: 0.5017993450164795
530: 0.5017993450164795
531: 0.5017993450164795
532: 0.5017993450164795
533: 0.5017993450164795
534: 0.5017993450164795
535: 0.5017993450164795
536: 0.5017993450164795
537: 0.5017993450164795
538: 0.5017993450164795
539: 0.5017993450164795
540: 0.5017993450164795
541: 0.5017993450164795
542: 0.5017993450164795
543: 0.5017993450164795
544: 0.5017993450164795
545: 0.5017993450164795
546: 0.5017993450164795
547: 0.5017993450164795
548: 0.5017993450164795
549: 0.5017993450164795
550: 0.5017993450164795
551: 0.5017993450164795
552: 0.5017993450164795
553: 0.5017993450164795
554: 0.5017993450164795
555: 0.5017993450164795
556: 0.5017993450164795
557: 0.5017993450164795
558: 0.5017993450164795
559: 0.5017993450164795
560: 0.5017993450164795
561: 0.5017993450164795
562: 0.5017993450164795
563: 0.5017993450164795
564: 0.5017993450164795
565: 0.5017993450164795
566: 0.5017993450164795
567: 0.5017993450164795
568: 0.5017993450164795
569: 0.5017993450164795
570: 0.5017993450164795
571: 0.5017993450164795
572: 0.5017993450164795
573: 0.5017993450164795
574: 0.5017993450164795
575: 0.5017993450164795
576: 0.5017993450164795
577: 0.5017993450164795
578: 0.5017993450164795
579: 0.5017993450164795
580: 0.5017993450164795
581: 0.5017993450164795
582: 0.5017993450164795
583: 0.5017993450164795
584: 0.5017993450164795
585: 0.5017993450164795
586: 0.5017993450164795
587: 0.5017993450164795
588: 0.5017993450164795
589: 0.5017993450164795
590: 0.5017993450164795
591: 0.5017993450164795
592: 0.5017993450164795
593: 0.5017993450164795
594: 0.5017993450164795
595: 0.5017993450164795
596: 0.5017993450164795
597: 0.5017993450164795
598: 0.5017993450164795
599: 0.5017993450164795
600: 0.5017993450164795
601: 0.5017993450164795
602: 0.5017993450164795
603: 0.5017993450164795
604: 0.5017993450164795
605: 0.5017993450164795
606: 0.5017993450164795
607: 0.5017993450164795
608: 0.5017993450164795
609: 0.5017993450164795
610: 0.5017993450164795
611: 0.5017993450164795
612: 0.5017993450164795
613: 0.5017993450164795
614: 0.5017993450164795
615: 0.5017993450164795
616: 0.5017993450164795
617: 0.5017993450164795
618: 0.5017993450164795
619: 0.5017993450164795
620: 0.5017993450164795
621: 0.5017993450164795
622: 0.5017993450164795
623: 0.5017993450164795
624: 0.5017993450164795
625: 0.5017993450164795
626: 0.5017993450164795
627: 0.5017993450164795
628: 0.5017993450164795
629: 0.5017993450164795
630: 0.5017993450164795
631: 0.5017993450164795
632: 0.5017993450164795
633: 0.5017993450164795
634: 0.5017993450164795
635: 0.5017993450164795
636: 0.5017993450164795
637: 0.5017993450164795
638: 0.5017993450164795
639: 0.5017993450164795
640: 0.5017993450164795
641: 0.5017993450164795
642: 0.5017993450164795
643: 0.5017993450164795
644: 0.5017993450164795
645: 0.5017993450164795
646: 0.5017993450164795
647: 0.5017993450164795
648: 0.5017993450164795
649: 0.5017993450164795
650: 0.5017993450164795
651: 0.5017993450164795
652: 0.5017993450164795
653: 0.5017993450164795
654: 0.5017993450164795
655: 0.5017993450164795
656: 0.5017993450164795
657: 0.5017993450164795
658: 0.5017993450164795
659: 0.5017993450164795
660: 0.5017993450164795
661: 0.5017993450164795
662: 0.5017993450164795
663: 0.5017993450164795
664: 0.5017993450164795
665: 0.5017993450164795
666: 0.5017993450164795
667: 0.5017993450164795
668: 0.5017993450164795
669: 0.5017993450164795
670: 0.5017993450164795
671: 0.5017993450164795
672: 0.5017993450164795
673: 0.5017993450164795
674: 0.5017993450164795
675: 0.5017993450164795
676: 0.5017993450164795
677: 0.5017993450164795
678: 0.5017993450164795
679: 0.5017993450164795
680: 0.5017993450164795
681: 0.5017993450164795
682: 0.5017993450164795
683: 0.5017993450164795
684: 0.5017993450164795
685: 0.5017993450164795
686: 0.5017993450164795
687: 0.5017993450164795
688: 0.5017993450164795
689: 0.5017993450164795
690: 0.5017993450164795
691: 0.5017993450164795
692: 0.5017993450164795
693: 0.5017993450164795
694: 0.5017993450164795
695: 0.5017993450164795
696: 0.5017993450164795
697: 0.5017993450164795
698: 0.5017993450164795
699: 0.5017993450164795
700: 0.5017993450164795
701: 0.5017993450164795
702: 0.5017993450164795
703: 0.5017993450164795
704: 0.5017993450164795
705: 0.5017993450164795
706: 0.5017993450164795
707: 0.5017993450164795
708: 0.5017993450164795
709: 0.5017993450164795
710: 0.5017993450164795
711: 0.5017993450164795
712: 0.5017993450164795
713: 0.5017993450164795
714: 0.5017993450164795
715: 0.5017993450164795
716: 0.5017993450164795
717: 0.5017993450164795
718: 0.5017993450164795
719: 0.5017993450164795
720: 0.5017993450164795
721: 0.5017993450164795
722: 0.5017993450164795
723: 0.5017993450164795
724: 0.5017993450164795
725: 0.5017993450164795
726: 0.5017993450164795
727: 0.5017993450164795
728: 0.5017993450164795
729: 0.5017993450164795
730: 0.5017993450164795
731: 0.5017993450164795
732: 0.5017993450164795
733: 0.5017993450164795
734: 0.5017993450164795
735: 0.5017993450164795
736: 0.5017993450164795
737: 0.5017993450164795
738: 0.5017993450164795
739: 0.5017993450164795
740: 0.5017993450164795
741: 0.5017993450164795
742: 0.5017993450164795
743: 0.5017993450164795
744: 0.5017993450164795
745: 0.5017993450164795
746: 0.5017993450164795
747: 0.5017993450164795
748: 0.5017993450164795
749: 0.5017993450164795
750: 0.5017993450164795
751: 0.5017993450164795
752: 0.5017993450164795
753: 0.5017993450164795
754: 0.5017993450164795
755: 0.5017993450164795
756: 0.5017993450164795
757: 0.5017993450164795
758: 0.5017993450164795
759: 0.5017993450164795
760: 0.5017993450164795
761: 0.5017993450164795
762: 0.5017993450164795
763: 0.5017993450164795
764: 0.5017993450164795
765: 0.5017993450164795
766: 0.5017993450164795
767: 0.5017993450164795
768: 0.5017993450164795
769: 0.5017993450164795
770: 0.5017993450164795
771: 0.5017993450164795
772: 0.5017993450164795
773: 0.5017993450164795
774: 0.5017993450164795
775: 0.5017993450164795
776: 0.5017993450164795
777: 0.5017993450164795
778: 0.5017993450164795
779: 0.5017993450164795
780: 0.5017993450164795
781: 0.5017993450164795
782: 0.5017993450164795
783: 0.5017993450164795
784: 0.5017993450164795
785: 0.5017993450164795
786: 0.5017993450164795
787: 0.5017993450164795
788: 0.5017993450164795
789: 0.5017993450164795
790: 0.5017993450164795
791: 0.5017993450164795
792: 0.5017993450164795
793: 0.5017993450164795
794: 0.5017993450164795
795: 0.5017993450164795
796: 0.5017993450164795
797: 0.5017993450164795
798: 0.5017993450164795
799: 0.5017993450164795
800: 0.5017993450164795
801: 0.5017993450164795
802: 0.5017993450164795
803: 0.5017993450164795
804: 0.5017993450164795
805: 0.5017993450164795
806: 0.5017993450164795
807: 0.5017993450164795
808: 0.5017993450164795
809: 0.5017993450164795
810: 0.5017993450164795
811: 0.5017993450164795
812: 0.5017993450164795
813: 0.5017993450164795
814: 0.5017993450164795
815: 0.5017993450164795
816: 0.5017993450164795
817: 0.5017993450164795
818: 0.5017993450164795
819: 0.5017993450164795
820: 0.5017993450164795
821: 0.5017993450164795
822: 0.5017993450164795
823: 0.5017993450164795
824: 0.5017993450164795
825: 0.5017993450164795
826: 0.5017993450164795
827: 0.5017993450164795
828: 0.5017993450164795
829: 0.5017993450164795
830: 0.5017993450164795
831: 0.5017993450164795
832: 0.5017993450164795
833: 0.5017993450164795
834: 0.5017993450164795
835: 0.5017993450164795
836: 0.5017993450164795
837: 0.5017993450164795
838: 0.5017993450164795
839: 0.5017993450164795
840: 0.5017993450164795
841: 0.5017993450164795
842: 0.5017993450164795
843: 0.5017993450164795
844: 0.5017993450164795
845: 0.5017993450164795
846: 0.5017993450164795
847: 0.5017993450164795
848: 0.5017993450164795
849: 0.5017993450164795
850: 0.5017993450164795
851: 0.5017993450164795
852: 0.5017993450164795
853: 0.5017993450164795
854: 0.5017993450164795
855: 0.5017993450164795
856: 0.5017993450164795
857: 0.5017993450164795
858: 0.5017993450164795
859: 0.5017993450164795
860: 0.5017993450164795
861: 0.5017993450164795
862: 0.5017993450164795
863: 0.5017993450164795
864: 0.5017993450164795
865: 0.5017993450164795
866: 0.5017993450164795
867: 0.5017993450164795
868: 0.5017993450164795
869: 0.5017993450164795
870: 0.5017993450164795
871: 0.5017993450164795
872: 0.5017993450164795
873: 0.5017993450164795
874: 0.5017993450164795
875: 0.5017993450164795
876: 0.5017993450164795
877: 0.5017993450164795
878: 0.5017993450164795
879: 0.5017993450164795
880: 0.5017993450164795
881: 0.5017993450164795
882: 0.5017993450164795
883: 0.5017993450164795
884: 0.5017993450164795
885: 0.5017993450164795
886: 0.5017993450164795
887: 0.5017993450164795
888: 0.5017993450164795
889: 0.5017993450164795
890: 0.5017993450164795
891: 0.5017993450164795
892: 0.5017993450164795
893: 0.5017993450164795
894: 0.5017993450164795
895: 0.5017993450164795
896: 0.5017993450164795
897: 0.5017993450164795
898: 0.5017993450164795
899: 0.5017993450164795
900: 0.5017993450164795
901: 0.5017993450164795
902: 0.5017993450164795
903: 0.5017993450164795
904: 0.5017993450164795
905: 0.5017993450164795
906: 0.5017993450164795
907: 0.5017993450164795
908: 0.5017993450164795
909: 0.5017993450164795
910: 0.5017993450164795
911: 0.5017993450164795
912: 0.5017993450164795
913: 0.5017993450164795
914: 0.5017993450164795
915: 0.5017993450164795
916: 0.5017993450164795
917: 0.5017993450164795
918: 0.5017993450164795
919: 0.5017993450164795
920: 0.5017993450164795
921: 0.5017993450164795
922: 0.5017993450164795
923: 0.5017993450164795
924: 0.5017993450164795
925: 0.5017993450164795
926: 0.5017993450164795
927: 0.5017993450164795
928: 0.5017993450164795
929: 0.5017993450164795
930: 0.5017993450164795
931: 0.5017993450164795
932: 0.5017993450164795
933: 0.5017993450164795
934: 0.5017993450164795
935: 0.5017993450164795
936: 0.5017993450164795
937: 0.5017993450164795
938: 0.5017993450164795
939: 0.5017993450164795
940: 0.5017993450164795
941: 0.5017993450164795
942: 0.5017993450164795
943: 0.5017993450164795
944: 0.5017993450164795
945: 0.5017993450164795
946: 0.5017993450164795
947: 0.5017993450164795
948: 0.5017993450164795
949: 0.5017993450164795
950: 0.5017993450164795
951: 0.5017993450164795
952: 0.5017993450164795
953: 0.5017993450164795
954: 0.5017993450164795
955: 0.5017993450164795
956: 0.5017993450164795
957: 0.5017993450164795
958: 0.5017993450164795
959: 0.5017993450164795
960: 0.5017993450164795
961: 0.5017993450164795
962: 0.5017993450164795
963: 0.5017993450164795
964: 0.5017993450164795
965: 0.5017993450164795
966: 0.5017993450164795
967: 0.5017993450164795
968: 0.5017993450164795
969: 0.5017993450164795
970: 0.5017993450164795
971: 0.5017993450164795
972: 0.5017993450164795
973: 0.5017993450164795
974: 0.5017993450164795
975: 0.5017993450164795
976: 0.5017993450164795
977: 0.5017993450164795
978: 0.5017993450164795
979: 0.5017993450164795
980: 0.5017993450164795
981: 0.5017993450164795
982: 0.5017993450164795
983: 0.5017993450164795
984: 0.5017993450164795
985: 0.5017993450164795
986: 0.5017993450164795
987: 0.5017993450164795
988: 0.5017993450164795
989: 0.5017993450164795
990: 0.5017993450164795
991: 0.5017993450164795
992: 0.5017993450164795
993: 0.5017993450164795
994: 0.5017993450164795
995: 0.5017993450164795
996: 0.5017993450164795
997: 0.5017993450164795
998: 0.5017993450164795
999: 0.5017993450164795

Double Descent and Overparameterization

Overparameterization

• Classic statistics wisdom about nonparametric regression: as parameters increase:
  • Bias decreases
  • Variance increases
  • Need parameters \(<\) sample size for good performance
• Neural networks: often see best performance when parameters \(>\) sample size

Double Descent

from joblib import Parallel, delayed
#device = 'cuda' if torch.cuda.is_available() else 'cpu'
device = 'cpu' # problem is small, so not much help from cuda
num_epochs = 50
sims = 100
maxw = 10

def doubledescentdemo(x, y, xtest, ytest, f, device=device,
                      num_epochs=num_epochs, sims=sims,
                      maxw=maxw, lr=0.1):
    x=x.to(device)
    y=y.to(device).reshape(x.shape[0],1)
    fx = f(x.T).reshape(y.shape).to(device)
    ytest = ytest.to(device).reshape(xtest.shape[0],1)
    xtest = xtest.to(device)
    loss_fn = nn.MSELoss().to(device)

    losses = np.zeros([maxw,num_epochs,sims])
    nonoise = np.zeros([maxw,num_epochs,sims])

    def dd(w):
        mlp = multilayer(x.shape[1],w+1,1,nn.ReLU()).to(device)
        optimizer = torch.optim.Adam(mlp.parameters(), lr=lr)
        losses = np.zeros(num_epochs)
        nonoise=np.zeros(num_epochs)
        for n in range(num_epochs):
            y_pred = mlp(x)
            loss = loss_fn(y_pred, y)
            optimizer.zero_grad()
            loss.backward()
            optimizer.step()
            losses[n] = loss_fn(mlp(xtest),ytest).item()
            nonoise[n] = loss_fn(y_pred, fx).item()
            mlp.zero_grad()
        return([losses, nonoise])

    for w in range(maxw):
        foo=lambda s: dd(w)
        print(f"width {w}")
        results = Parallel(n_jobs=20)(delayed(foo)(s) for s in range(sims))

        for s in range(sims):
            losses[w,:,s] = results[s][0]
            nonoise[w,:,s] = results[s][1]
    return([losses, nonoise])

Double Descent

f = lambda x: np.exp(x[0]-x[1])
n = 20
torch.manual_seed(1234)
dimx = 5
x = torch.rand(n,dimx)
xtest = torch.rand(n,dimx)
sigma = 1.5
y = f(x.T) + sigma*torch.randn(x.shape[0])
ytest = f(xtest.T) + sigma*torch.randn(xtest.shape[0])
lr = 0.01
dd = doubledescentdemo(x, y, xtest, ytest, f, lr)

width 0

width 1
width 2
width 3
width 4
width 5
width 6
width 7
width 8
width 9

Double Descent

def plotdd(losses, nonoise):
    fig, ax = plt.subplots(2,2)
    ax = ax.flatten()
    ax[0].imshow(losses, cmap='plasma')
    ax[0].set_title('Test Loss')
    ax[0].set_xlabel('epoch')
    ax[0].set_ylabel('width')
    ax[1].imshow(nonoise, cmap='plasma')
    ax[1].set_title('error in f')
    ax[1].set_xlabel('epoch')
    ax[1].set_ylabel('width')

    ws = [0,1, 2, 4, 8]  #10, 12, 19]
    for w in ws:
        ax[2].plot(range(losses.shape[1]),losses[w,:], label=f"width={w+1}")
        ax[3].plot(range(nonoise.shape[1]),nonoise[w,:], label=f"width={w+1}")

    ax[2].set_xlabel('epoch')
    ax[2].set_ylabel('test loss')
    ax[2].legend()
    ax[3].set_xlabel('epoch')
    ax[3].set_ylabel('error in f')
    ax[3].legend()

    fig.tight_layout()

    return(fig)

fig=plotdd(dd[0].mean(axis=2), dd[1].mean(axis=2))
fig.show()

Double Descent: Low Noise

sigma = 0.01
y = f(x.T) + sigma*torch.randn(x.shape[0])
ytest = f(xtest.T) + sigma*torch.randn(xtest.shape[0])
ddlowsig = doubledescentdemo(x,y,xtest,ytest,f, lr=0.05)

width 0
width 1
width 2
width 3
width 4
width 5
width 6
width 7
width 8
width 9

Double Descent: Low Noise

fig=plotdd(ddlowsig[0].mean(axis=2), ddlowsig[1].mean(axis=2))
fig.show()

Double Descent

• Figures show double descent in number of epochs, but has also been demonstrated with respect to number of parameters¹
• These simulations are quite fragile and depend on:
  • Form of \(f\)
  • dimension of \(x\)
  • noise in \(y\)
  • learning rate
  • optimizer

1. If you squint, you can sort of see double descent in parameters in these simulations too.

Software

Software

• PyTorch
  • Used above
  • Very popular
  • Middle ground between convenience and flexibility
  • Could be used via higher level wrapper (e.g. skorch)
• JAX
  • More extensible than pytorch
• Others: Tensorflow, Theano, Keras, and more

Uses for Neural Networks

Uses for Neural Networks

• Flexible regression and/or classification method, in double/debiased learning or elsewhere
  • Consider carefully whether difficulties of fitting are worth benefits
• Flexible function approximation
  • Useful for computing solutions to games, dynamic models, etc
• Text, images, audio
  • Use transfer learning

Transfer Learning

• Fit (or let researchers or company with more resources fit) large model on large, general dataset
• Find specialized data for your task
• Fine tune: take large model and update parameters with your data
• https://huggingface.co/ good source for large models to fine-tune

Other Architectures

• Multi-layer perceptrons / feed forward networks are the simplest neural networks, many extensions and variations exist
• Tricks to help with vanishing gradients and numeric stability:
  • Normalization
  • Residual connections
• Variations motivated by sequential data:
  • Recurrent
  • Transformers
• Variations motivated by images:
  • Convolutions
  • GAN
  • Diffusion¹

1. The motivating idea of diffusion models is different than neural networks, but neural networks are commonly used as one of the parts of diffusion models.

Sources and Further Reading

• QuantEcon Datascience: Regression - Neural Networks

• QuantEcon: Intro to Artificial Neural Nets

• pytorch notebooks from 323

References