Introduction
The previous post ended with a gap. The McCulloch-Pitts neuron could compute any Boolean function, but every threshold and every connection had to be set by hand. For three inputs and a simple logic gate, that is manageable. For a network that needs to recognize a face, read handwriting, or classify anything meaningful from raw data, hand-design is impossible. The model proved that neural computation works. It said nothing about how to make it learn.
In 1958, psychologist Frank Rosenblatt looked at the McCulloch-Pitts model and asked a natural follow-up question: what if the connections could adjust themselves? He took the same mathematical structure (inputs, summation, threshold, binary output) and modified it. He replaced the equal +1 excitatory contributions with variable real-valued weights. He replaced the fixed integer threshold with a learnable bias. And then he added the piece that McCulloch and Pitts never had: a learning rule. An algorithm that takes labeled examples, computes the error, and nudges the weights in the direction that would have produced the correct answer. He called the result the perceptron, described in his paper The Perceptron: A Probabilistic Model for Information Storage and Organization in the Brain.
The perceptron could learn. But it could only learn functions that a single hyperplane can separate. Rosenblatt knew this. In his 1962 book Principles of Neurodynamics, he spent years working on multi-layer architectures that could overcome the limitation. He tried multiple approaches to propagate learning into hidden layers, but none of them worked. His learning rule required targets, and hidden neurons had no targets. When Marvin Minsky and Seymour Papert published their critique in 1969, the single-layer model was formally proven limited and the multi-layer training problem remained unsolved. Rosenblatt died in a boating accident in 1971, never seeing the solution. Funding collapsed, researchers moved on, and neural networks entered a winter that lasted over a decade.
I wanted to put myself in Rosenblatt's shoes. He looked at McCulloch and Pitts' hand-designed neuron and asked: how do I make this learn? I want to ask myself the same question, starting from the M-P model I built in the previous post, and work through the reasoning that leads to the perceptron, its learning rule, its power, and its limits.
From Counting to Weighting
I left the McCulloch-Pitts model with a clear limitation: every connection was equal and every threshold was fixed. If I want the model to learn from data, those fixed elements are the first things that need to change.
The M-P model treats all excitatory inputs equally: each active input contributes +1 to a running count, any active inhibitory input vetoes the output entirely, and the neuron fires if the count meets a fixed threshold. The connections have no strength. They are either on or off, excitatory or inhibitory, all equal.
I need to change three things.
Variable real-valued weights. Instead of every excitatory input contributing +1, what if each input gets its own connection strength? A real number that can be positive, negative, large, small, or zero. A weight of 3.2 means that input has strong excitatory influence. A weight of -1.5 means it has moderate inhibitory influence. A weight near zero means it barely matters. This replaces the binary excitatory/inhibitory distinction with a continuous spectrum of connection strengths.
A bias term. Instead of a fixed integer threshold, I replace it with a learnable bias that shifts the decision boundary. The threshold is no longer a separate parameter set by hand. It is absorbed into the bias and can be adjusted along with the weights.
A learning rule. This is the key piece. Instead of an engineer deciding what the weights should be, the model adjusts its own weights based on its mistakes. When it gets an answer wrong, it changes the weights in the direction that would have produced the correct answer. When it gets an answer right, it leaves the weights alone.
These three changes transform the McCulloch-Pitts neuron from a hand-designed logic gate into a machine that learns from data. This is exactly what Rosenblatt did. Let me build it step by step.
The Perceptron Architecture
With variable weights and a bias, the model takes a vector of real-valued inputs, multiplies each by its corresponding weight, sums the results, adds the bias, and applies a step function:
z = w1*x1 + w2*x2 + ... + wn*xn + b
y = 1 if z >= 0
y = 0 if z < 0
In vector notation:
z = w . x + b
y = step(z)
The step function is the simplest possible activation: output 1 for non-negative inputs, 0 for negative inputs. It preserves the all-or-nothing firing behavior from biology, but now the decision of whether to fire depends on a weighted sum rather than a simple count. I have gone from counting active inputs to computing a weighted combination of them.
Figure 1: The perceptron architecture. Each input is multiplied by its own variable weight (cyan edges), summed with a bias, and passed through a step function. The output is compared to the target. When wrong, the error (red) feeds back to adjust the weights. This feedback loop is the learning mechanism that McCulloch-Pitts lacked.
The Learning Algorithm
Now I have an architecture with adjustable weights, but I still need a rule for adjusting them. The algorithm Rosenblatt devised is remarkably simple. Start with random weights. For each training example, compute the output. If the output is wrong, adjust the weights in the direction that would have produced the correct answer. If the output is right, do nothing. Repeat.
Initialize weights w and bias b to small random values
Set learning rate eta (typically 0.1 to 1.0)
For each training example (x, t) where t is the target:
1. Compute output: y = step(w . x + b)
2. Compute error: e = t - y
3. Update weights: w = w + eta * e * x
4. Update bias: b = b + eta * e
Repeat until all examples are classified correctly
When the output matches the target (e = 0), the weights do not change. When the output is wrong, one of two things happens:
Missed positive (t = 1 but y = 0): The error is e = 1. The update adds eta * x to the weights. This makes the weighted sum larger for this input pattern on the next pass, pushing the output toward 1.
False positive (t = 0 but y = 1): The error is e = -1. The update subtracts eta * x from the weights. This makes the weighted sum smaller for this input pattern, pushing the output toward 0.
The learning rate eta controls step size. Larger values mean faster convergence but risk overshooting. Smaller values converge more smoothly but take longer. For the perceptron, the convergence theorem guarantees convergence regardless of learning rate (as long as it is positive), so the choice affects speed, not correctness.
Worked Example: Learning AND
I want to see this work concretely. The McCulloch-Pitts post required me to hand-design AND with a threshold of 2 and two equal +1 inputs. Now let me see if the perceptron can find a solution on its own.
Training data:
| x1 | x2 | target |
|---|---|---|
| 0 | 0 | 0 |
| 0 | 1 | 0 |
| 1 | 0 | 0 |
| 1 | 1 | 1 |
Initialize: w1 = 0, w2 = 0, b = 0, eta = 1
Epoch 1:
(0,0) target 0: z = 0*0 + 0*0 + 0 = 0 y = 1 e = -1
w1 = 0 + (-1)*0 = 0 w2 = 0 + (-1)*0 = 0 b = 0 + (-1) = -1
(0,1) target 0: z = 0*0 + 0*1 + (-1) = -1 y = 0 e = 0
no update
(1,0) target 0: z = 0*1 + 0*0 + (-1) = -1 y = 0 e = 0
no update
(1,1) target 1: z = 0*1 + 0*1 + (-1) = -1 y = 0 e = 1
w1 = 0 + 1*1 = 1 w2 = 0 + 1*1 = 1 b = -1 + 1 = 0
After epoch 1: w1 = 1, w2 = 1, b = 0. Not converged yet, since (0,0) gives z = 0, which maps to y = 1 (false positive).
Epoch 2:
(0,0) target 0: z = 1*0 + 1*0 + 0 = 0 y = 1 e = -1
w1 = 1 w2 = 1 b = 0 + (-1) = -1
(0,1) target 0: z = 1*0 + 1*1 + (-1) = 0 y = 1 e = -1
w1 = 1 w2 = 1 + (-1)*1 = 0 b = -1 + (-1) = -2
(1,0) target 0: z = 1*1 + 0*0 + (-2) = -1 y = 0 e = 0
no update
(1,1) target 1: z = 1*1 + 0*1 + (-2) = -1 y = 0 e = 1
w1 = 1 + 1 = 2 w2 = 0 + 1 = 1 b = -2 + 1 = -1
After epoch 2: w1 = 2, w2 = 1, b = -1. Still not converged.
I will spare the remaining epochs. After a few more passes, the weights converge to a solution like w1 = 2, w2 = 1, b = -2 (the exact values depend on the learning path). At that point:
(0,0): 2*0 + 1*0 - 2 = -2 < 0 -> y = 0 correct
(0,1): 2*0 + 1*1 - 2 = -1 < 0 -> y = 0 correct
(1,0): 2*1 + 1*0 - 2 = 0 >= 0 -> y = 1 ... wait, that gives 1, not 0
So that particular solution does not work. The algorithm keeps going. The key insight is that it will find a correct solution because AND is linearly separable. One valid solution is w1 = 1, w2 = 1, b = -1.5:
(0,0): 0 + 0 - 1.5 = -1.5 < 0 -> y = 0 correct
(0,1): 0 + 1 - 1.5 = -0.5 < 0 -> y = 0 correct
(1,0): 1 + 0 - 1.5 = -0.5 < 0 -> y = 0 correct
(1,1): 1 + 1 - 1.5 = 0.5 >= 0 -> y = 1 correct
The perceptron finds this (or an equivalent solution) automatically. No hand-design. That is the breakthrough.
The Perceptron Convergence Theorem
The worked example shows the algorithm working, but does it always work? Rosenblatt claimed convergence for linearly separable data, and the formal proof was later provided by Novikoff in 1962:
If the training data is linearly separable, the perceptron learning algorithm is guaranteed to converge to a correct solution in a finite number of steps.
The proof sketch:
- Assume a correct weight vector
w*exists that correctly classifies all training data with some margin. - After each misclassification update, the dot product
w . w*increases by at least a fixed positive amount (progress toward the solution). - After each update, the squared norm
||w||^2increases by at most a bounded amount (the step size is limited). - Since
w . w*grows linearly and||w||grows at most as the square root of the number of updates, the cosine of the angle betweenwandw*eventually reaches 1. But cosine is bounded by 1, so the number of updates must be finite.
The bound on the number of updates is:
number of updates <= (R / gamma)^2
Where R is the maximum norm of any training input and gamma is the margin of the optimal weight vector. Larger margin means fewer updates needed. Smaller margin means the algorithm has to work harder.
The critical condition is linear separability. If the data can be perfectly separated by a hyperplane (a line in 2D, a plane in 3D, a hyperplane in higher dimensions), the perceptron will find it. If the data is not linearly separable, the algorithm never converges. It oscillates forever, adjusting weights back and forth without settling. That condition will become very important shortly.
Geometric Interpretation: Decision Boundaries
To understand what the convergence theorem is really saying, I need to think about what the perceptron computes geometrically.
The perceptron outputs y = step(w . x + b). The boundary between "fire" (y = 1) and "don't fire" (y = 0) is the set of points where w . x + b = 0. In two dimensions, this equation defines a line. In three dimensions, a plane. In general, a hyperplane.
The weight vector w is perpendicular to this hyperplane. It points toward the "positive" side (where y = 1). The bias b shifts the hyperplane toward or away from the origin. So what the perceptron is really doing during training is searching for the orientation and position of a hyperplane that correctly separates the two classes.
Figure 2: The perceptron's decision boundary in 2D. Each weight update rotates the line. For a missed positive, the boundary rotates to include the missed point. For a false positive, it rotates to exclude it. The weight vector (yellow, dashed) is always perpendicular to the boundary, pointing toward the class-1 region. The convergence theorem guarantees that if a separating line exists, the algorithm will find it.
Every time the perceptron makes an error, the weight update rotates and shifts the decision boundary. The convergence theorem says that if a correct boundary exists, the algorithm reaches it in finite steps. But what if no correct boundary exists? What if the problem requires a decision boundary that is not a straight line?
The XOR Problem
This is where the perceptron breaks. Consider the XOR (exclusive or) function.
XOR outputs 1 when exactly one input is 1:
| x1 | x2 | XOR |
|---|---|---|
| 0 | 0 | 0 |
| 0 | 1 | 1 |
| 1 | 0 | 1 |
| 1 | 1 | 0 |
In the McCulloch-Pitts post, I built XOR from a network of hand-designed neurons: AND, OR, and NOT composed together. A single M-P neuron could not compute it either. But now I have a learning rule. Can the perceptron learn XOR? The answer is no, and the proof is geometric. I can see it by plotting the four points:
Figure 3: XOR vs AND. The AND function is linearly separable: a single line cleanly divides the class-1 point (top right) from the class-0 points. XOR is not: the class-1 points (top left, bottom right) sit on opposite corners, and no single line can separate them from the class-0 points (bottom left, top right). The dashed line rotates to show that every possible orientation misclassifies at least one point.
The class-1 points (0,1) and (1,0) sit on opposite corners. The class-0 points (0,0) and (1,1) sit on the other two corners. No single straight line can put the 1s on one side and the 0s on the other.
The algebraic proof is equally clean. For the perceptron to produce the correct outputs, I need:
(0,0) -> 0: w1*0 + w2*0 + b < 0 => b < 0
(0,1) -> 1: w1*0 + w2*1 + b >= 0 => w2 >= -b
(1,0) -> 1: w1*1 + w2*0 + b >= 0 => w1 >= -b
(1,1) -> 0: w1*1 + w2*1 + b < 0 => w1 + w2 < -b
From constraints 2 and 3: w1 >= -b and w2 >= -b. Adding these: w1 + w2 >= -2b. Since b < 0, we know -b > 0, so -2b > 0, and therefore w1 + w2 > 0.
But constraint 4 says w1 + w2 < -b. Since -b > 0, this says w1 + w2 is positive, which is consistent so far. The contradiction: w1 + w2 >= -2b (from constraints 2+3) but also w1 + w2 < -b (constraint 4). This requires -2b <= w1 + w2 < -b, which means -2b < -b, which means -b < 0, which means b > 0. But constraint 1 says b < 0. Contradiction.
No values of w1, w2, and b satisfy all four constraints simultaneously. A single perceptron cannot compute XOR.
The Obvious Fix and the Real Problem
The fix for XOR is conceptually obvious to me: add another layer of neurons. In the previous post, I built XOR from a network of McCulloch-Pitts neurons by composing AND, OR, and NOT gates. The same idea applies here. If I stack perceptrons into layers, with the outputs of one layer feeding as inputs to the next, the network can learn nonlinear decision boundaries. This architecture, multiple layers of perceptrons, is called a multilayer perceptron (MLP).
Rosenblatt knew this too. He did not just suggest the idea in passing. He spent years working on it.
Rosenblatt's Multi-Layer Research
In 1962, Rosenblatt published Principles of Neurodynamics, a comprehensive treatment of perceptron theory that went far beyond the 1958 paper. A large portion of the book is dedicated to multi-layer perceptron architectures and the problem of training them.
He explored several approaches. In his terminology, a multi-layer perceptron had three types of units: S-units (sensory inputs), A-units (association units in the hidden layer), and R-units (response units at the output). The S-units fed into A-units, which fed into R-units. The question was always the same: how do the A-unit connections learn?
His most practical approach was to fix the hidden layer connections randomly and only train the output layer weights. The S-to-A connections were set at initialization and never changed. Only the A-to-R connections (the output layer) were adjusted by the perceptron learning rule. This worked because the output layer had targets. The random hidden layer acted as a fixed feature transformation, projecting the inputs into a different space where the output layer's linear decision boundary might succeed. For some problems, it did. The random projection happened to produce a representation that was linearly separable at the output. For others, it did not. And there was no way to improve the hidden layer when it failed. The A-unit connections were frozen at whatever random values they started with.
Rosenblatt understood the core issue clearly. The perceptron learning rule works by comparing the output to the target and adjusting the weights accordingly. But in a multi-layer network, how do I adjust the weights in the first layer? Those neurons do not have targets. They produce intermediate representations that feed into the next layer, and there is no direct way to know what those intermediate values should be. If the output neuron gets the wrong answer, which hidden neuron is responsible? Should a hidden unit's weight go up or down? By how much? The learning rule has no mechanism for assigning credit (or blame) to neurons that are not directly connected to the output.
This is the credit assignment problem. Rosenblatt could build multi-layer networks. He could not train them. He tried. He spent years trying. He never found a solution that propagated learning into the hidden layers.
Minsky, Papert, and the End of an Era
In 1969, Marvin Minsky and Seymour Papert published Perceptrons, a rigorous mathematical analysis of what single-layer perceptrons can and cannot compute. Their analysis went beyond XOR. They showed that single-layer perceptrons cannot compute:
- Parity functions: determining whether an odd or even number of inputs are active (XOR is the 2-input case)
- Connectedness: determining whether a pattern on a grid forms a single connected region
- Symmetry detection: determining whether a pattern is symmetric
These are not exotic edge cases. Many real-world classification problems require detecting relationships between inputs that cannot be captured by a single hyperplane.
Their book was careful to note that multi-layer networks could solve these problems. But they expressed skepticism that anyone would find a practical learning algorithm for them:
"The perceptron has shown itself worthy of study despite (and even because of) its severe limitations. It has many features to attract attention: its linearity; its intriguing learning theorem; its clear paradigmatic simplicity as a kind of parallel computation. There is no reason to suppose that any of these virtues carry over to the many-layered version."
That skepticism landed on top of a problem Rosenblatt had been working on for over a decade without solving. He had no multi-layer learning algorithm. Minsky and Papert had just proved the single-layer model was fundamentally limited. The combination was devastating.
On July 11, 1971, Frank Rosenblatt died in a boating accident on Chesapeake Bay. He was 43 years old. He never saw the solution to the problem he spent the last decade of his life working on.
Research funding for neural networks collapsed. The U.S. government and military, which had been significant funders of perceptron research, redirected AI spending toward symbolic AI: rule-based systems, expert systems, logical reasoning. Academic researchers moved on. PhD students were advised to avoid neural networks. With the field's most prominent advocate dead and its theoretical limitations formally proven, neural networks entered what is now called the AI winter. The period lasted from roughly 1969 to the mid-1980s.
The cause was not XOR itself. XOR was just the clearest demonstration of a deeper problem: the perceptron learning rule could not train multi-layer networks, and single-layer networks could not solve interesting problems. The field was stuck between an architecture that was too simple and a learning algorithm that could not scale.
The irony is that the solution was already emerging. Paul Werbos described backpropagation in his 1974 PhD thesis Beyond Regression, providing exactly the multi-layer learning algorithm that solved the credit assignment problem. But Werbos's work went largely unnoticed. It took until 1986, when David Rumelhart, Geoffrey Hinton, and Ronald Williams published their landmark paper Learning Representations by Back-Propagating Errors demonstrating backpropagation on multi-layer networks, for the field to revive.
The perceptron was vindicated, not as a complete solution, but as the foundation. Everything that came after builds on the principles Rosenblatt introduced: variable weights, learned representations, error-driven updates. The missing piece was a way to propagate that error backward through multiple layers. That is the next post.
McCulloch-Pitts vs. Perceptron: What Changed
| Property | McCulloch-Pitts (1943) | Perceptron (1958) |
|---|---|---|
| Connection strength | Equal (+1 excitatory, absolute inhibitory veto) | Variable real-valued weights |
| Threshold | Fixed integer, hand-set | Learnable bias term |
| Learning | None (hand-designed) | Perceptron learning rule |
| Input types | Binary (0 or 1) | Real-valued |
| Inhibition | Absolute veto (any active inhibitory input blocks firing) | Negative weight (graded inhibition) |
| Convergence guarantee | N/A | Yes, for linearly separable data |
| Limitation | No learning | Only linearly separable problems |
Starting from the McCulloch-Pitts model, I kept the core structure (inputs, summation, threshold activation, binary output) and replaced every fixed element with a learnable one. That single conceptual shift, from hand-designed to data-driven, is the dividing line between computation and learning. It is exactly what Rosenblatt did.
The Perceptron in Code
Everything I have built reduces to three Python functions. perceptron() computes the forward pass (the model itself), learn() applies the update rule to a single example (the novelty Rosenblatt added), and train() runs the full learning loop:
def perceptron(x, w, b):
"""Rosenblatt's perceptron: weighted sum, step activation."""
z = sum(wi * xi for wi, xi in zip(w, x)) + b
return 1 if z >= 0 else 0
def learn(x, target, w, b, eta=1.0):
"""Single perceptron update. Returns (w, b, error)."""
output = perceptron(x, w, b)
error = target - output
if error != 0:
w = [wi + eta * error * xi for wi, xi in zip(w, x)]
b = b + eta * error
return w, b, error
def train(data, targets, eta=1.0, max_epochs=100):
"""Train a perceptron on data. Returns (w, b)."""
w = [0.0] * len(data[0])
b = 0.0
for epoch in range(max_epochs):
errors = 0
for x, t in zip(data, targets):
w, b, e = learn(x, t, w, b, eta)
if e != 0:
errors += 1
if errors == 0:
print(f"Converged after {epoch + 1} epochs")
break
return w, b
Training on AND:
data = [[0,0], [0,1], [1,0], [1,1]]
w, b = train(data, targets=[0, 0, 0, 1])
Converged after 6 epochs
for x in data:
print(f" {x} -> {perceptron(x, w, b)}")
[0, 0] -> 0
[0, 1] -> 0
[1, 0] -> 0
[1, 1] -> 1
OR also converges:
w, b = train(data, targets=[0, 1, 1, 1])
Converged after 4 epochs
XOR never converges:
w, b = train(data, targets=[0, 1, 1, 0], max_epochs=1000)
(no convergence message -- the loop runs all 1000 epochs)
for x in data:
print(f" {x} -> {perceptron(x, w, b)}")
[0, 0] -> 1
[0, 1] -> 1
[1, 0] -> 0
[1, 1] -> 0
At least one point is always wrong. The algorithm oscillates, adjusting the weights to fix one error only to create another. This is the convergence theorem in reverse: XOR is not linearly separable, so the algorithm cannot settle.
The architecture fix is straightforward. Stack perceptrons into layers:
def mlp(x, hidden_weights, hidden_biases, output_weights, output_bias):
"""Forward pass through a two-layer network."""
hidden = [perceptron(x, w, b) for w, b in zip(hidden_weights, hidden_biases)]
return perceptron(hidden, output_weights, output_bias)
Hand-wired, it solves XOR:
# Hidden layer: neuron 1 computes OR-like, neuron 2 computes NAND-like
hidden_w = [[1, 1], [-1, -1]]
hidden_b = [-0.5, 1.5]
# Output layer: AND-like on hidden outputs
output_w = [1, 1]
output_b = -1.5
for x in data:
print(f" {x} -> {mlp(x, hidden_w, hidden_b, output_w, output_b)}")
[0, 0] -> 0
[0, 1] -> 1
[1, 0] -> 1
[1, 1] -> 0
XOR solved. But I had to set every weight by hand, just like McCulloch-Pitts. The train() function cannot find these weights because learn() computes error = target - output. For the output neuron, the target is known. For the hidden neurons, there is no target. What should the OR-like neuron output for input (1,1)? What should the NAND-like neuron output for input (0,1)? The learning rule has no way to answer these questions. That is the credit assignment problem, expressed in code. The solution is the subject of the next post.
Key Takeaways
I started where the McCulloch-Pitts post left off: a model that computes but cannot learn. Following Rosenblatt's reasoning, I replaced equal contributions with variable weights, the fixed threshold with a learnable bias, and hand-design with an error-driven update rule. The convergence theorem guaranteed that if a solution exists (if the data is linearly separable), the algorithm finds it. But linear separability is a hard constraint. XOR, parity, connectedness, and most real-world problems require decision boundaries that a single hyperplane cannot express. The fix, stacking perceptrons into multiple layers, was architecturally obvious. Rosenblatt knew it. He spent years on it in Principles of Neurodynamics, trying random hidden projections, series-coupled networks, and other multi-layer designs. None of them solved the fundamental problem: his learning rule could not assign credit to neurons that were not directly connected to the output. Minsky and Papert formalized the single-layer limitations. Rosenblatt died in 1971 without solving the multi-layer training problem. The field froze for over a decade. The solution, a way to propagate error backward through every layer of a deep network, is the subject of the next post.