“The hardest problems we have to face do not come from philosophical questions about whether brains are machines or not. There is not the slightest reason to doubt that brains are anything other than machines with enormous numbers of parts that work in perfect accord with physical laws. As far as anyone can tell, our minds are merely complex processes. The serious problems come from our having had so little experience with machines of such complexity that we are not yet prepared to think effectively about them.”
Marvin Minsky, 1986
Few people today know the origin of Machine Learning.
Fewer know it goes back to 1957.
That’s the year that psychologist Frank Rosenblatt was sitting at Cornell with access to an IBM 704 and an idea he couldn’t shake. He wanted to build something that could learn from experience. The way animals do.1
The perceptron.2
The first one was laughably simple by today’s standards. It could look at a small grid of pixels and learn to tell two shapes apart. A circle from a square, light from dark. It had no hidden layers, had no depth, and had no ability to generalize beyond what it had already been shown.
Rosenblatt didn’t call the input layer an “input layer” either. He wanted to make a copy of the brain, using whatever parts a psychologist in 1950s had lying around. Hence, a retina.
Further he wrote, “In the models considered here, an all-or-nothing response will be assumed.” He didn’t invent that either: a biological neuron collects signals through its dendrites. If enough of them fire at once, the combined charge crosses a threshold. The neuron fires a spike down the axon to whoever's next in line.
The mechanism introduced in that 1958 paper may have been archaic by today’s standards, but it was the first step into an entirely new field.
A number of breakthroughs happened over the next few decades, but in my eyes, 2012 was the turning point. Researchers took Rosenblatt’s original bet — weighted inputs, threshold, output, learn from mistakes — and stacked a shit ton of those layers on top of each other to get depth.
They trained the stack on fifteen million labelled images, ran it on two GPUs instead of one CPU. Called it AlexNet.3
ImageNet was a popular benchmark for image recognition with over a million labelled images. Submit your model, get graded. The models’ accuracy improved a percent or two per year. AlexNet absolutely smashed everything that came before it.4
Obsession with depth took over: more layers, more parameters, more data, more compute, more billion-dollar data centres.
Two years later, a team at MIT built a Convolutional Neural Network (CNN) and trained it on exactly one thing: recognize objects in images.5
They checked its internal activity against real recordings from monkey visual cortex and electrodes reading how real neurons grouped objects matched against how the model's own layers grouped them. The model didn’t just have accuracy, it had similar organization.
Maybe that’s all a brain is doing too.
“For reason, in this sense, is nothing but reckoning (that is, adding and subtracting) of the consequences of general names agreed upon for the marking and signifying of our thoughts.”
Thomas Hobbes, 1651
One weekend I built a CNN myself. Matrix multiplication, a training loop, nothing a motivated first year CS student couldn’t do. It took a grid of pixels as input and learned to tell a 3 apart from an 8.
Yet, playing with it, I couldn’t figure out where it stopped being math, and turned into real thinking. A CNN trained on nothing but accuracy ends up organized like a visual cortex. Reinforcement learning runs on the same signal your dopamine system runs on. And here I was, staring at my own code, feeling the same discomfort the people building AI might feel at three in the morning.
Before I get accused of the obvious trick: a thermostat also takes in a signal and changes state accordingly. Nobody believes the thermostat is uncomfortable when it’s cold. So “it responds to input” clearly isn’t sufficient, my five lines of code are not a mind. But how many parameters, and how many layers, would it take to build one?
Ultimately, I was never able to answer that question, even after spending over a year working at an AI company.
Tomorrow I’ll sit down and launch those very same models, like nothing happened. If the discomfort shows up, I’ll call it illogical. Emotional.
It has never stopped the people publishing the papers. It has never stopped the labs racing each other towards Artificial General Intelligence. It has never stopped me from reviewing my next PR using AI.
We jumped straight to the model, forgetting to build a system to check for pain, for consciousness, for feelings. We’ll find out what we’ve built eventually, likely long after the answer would matter.
Geoffrey Hinton left the lab that made him famous and said, publicly, that part of him regrets it. He doesn’t know what he built.6
Rosenblatt taught a machine to tell a circle from a square. I don't know if he'd recognize what it grew into.
Regardless.
They keep building it.
You keep using it.
Hm, it's almost like humans race to extract power and economic value out of new ideas, at the expense of foresight, responsibility, and understanding. That's pretty much the history of science and technology.