Count the Weights

Question

A fully connected network maps 8 inputs through a hidden layer of 16 units, then a hidden layer of 8 units, to a single output. Every layer has biases. How many trainable parameters does the network have?

Accepted Answer

Count layer by layer; a dense layer from n_textin to n_textout units holds n_textin n_textout weights and n_textout biases. - Layer 1: 8 times 16 + 16 = 144 - Layer 2: 16 times 8 + 8 = 136 - Layer 3: 8 times 1 + 1 = 9 Total: 144 + 136 + 9 = mathbf289. Why interviewers bother with this. The count is the model's raw capacity, and the instant sanity check is parameters versus training rows. A 289-parameter net is modest against a million ticks but enormous against 1,500 daily observations of a single asset: at that ratio the network can memorize the sample rather than learn from it, and regularization stops being optional. The habit being tested is reflexively comparing capacity to effective sample size before training anything. A useful corollary: the activation function contributes nothing to the count, and almost all parameters live where consecutive layers are widest, here 144 of 289 sit in the first layer. In trading research the same arithmetic instinct catches bloated embedding tables and feature crosses that quietly multiply capacity far beyond what noisy financial labels can support.

Count the Weights

Problem

Your answer

Hints