How did the neural network?

Over the past 10 years, due to the method of so-called deep learning, we have the best artificial intelligence system — for example, speech recognizers on smartphones or the latest automatic translator Google. Deep learning, in fact, has become a new trend in the already known neural networks, which were in Vogue and went on for more than 70 years. For the first time neural network proposed Warren McCullough and Walter Pitts in 1994, two researchers from the University of Chicago. In 1952 they went to work at the Massachusetts Institute of technology, to lay the groundwork for the first Department of cognitive science.

Neural networks were one of the main directions of studies in neurobiology and computer science until 1969, when, according to legend, killed of mathematics Massachusetts Institute of technology Marvin Minsky and Seymour Papert who in a year became co-managers of the new artificial intelligence lab, MIT.

The Revival of this method has experienced in the 1980s, slightly faded in the first decade of the new century with a fanfare back in the second, on the crest of an incredible development of graphics chips and processing power.

"it Is believed that the idea of science — as the epidemic of the virus," said Tomaso Poggio, a Professor of cognitive science and brain science at MIT. "There is, apparently, five or six major strains of flu viruses, and one of them returns with surprising frequency in 25 years. People get infected, become immune and not get sick the next 25 years. Then there is a new generation, ready to be infected by the same strain of the virus. In science, people fall in love with the idea, it's driving everyone nuts, then beaten to death and become immune to it — I get tired of it. The ideas should be similar to the frequency".

the

Weighty issues

Neural networks are a method of machine learning where the computer learns to perform certain tasks, analyzing the training examples. As a rule, these examples are labeled manually in advance. The system is object recognition, for example, can absorb thousands of labeled images of cars, houses, coffee cups and so forth, and then will be able to find visual images in these images that consistently correlate with specific labels.

A Neural network is often compared to the human brain, which also has such networks, consisting of thousands or millions of simple processing nodes that are closely linked. Most modern neural networks are arranged in layers of nodes, and the data pass through them in only one direction. A separate node may be associated with multiple nodes in the layer beneath it, from which it obtains the data and multiple nodes in the layer above, in which it sends data.

Each of these incoming links node assigns a number — the "weight". When the network is active, the node receives different data sets — different numbers for each of these compounds and multiplies by the corresponding weight. It then sums the results to form a single number. If this number is below a threshold, the node does not transmit data to the next layer. If the number exceeds the threshold, the node is "activated" by sending the number — the sum of the weighted input data on all outgoing connections.

When the neural network trains, the weights and thresholds are initially set at random. Training data is provided in the lower layer — the input — and pass through the next layers, stacking and multiplying in a complex manner, until they finally arrive, already transformed, in the output layer. During training the weights and thresholds are constantly adjusted until the training data with the same label would not give similar conclusions.

the

Minds and machines

Neural networks are described by McCullough and Pitts in 1944, and had the thresholds and weights, but were not organized in layers, and scientists do not ask any specific learning mechanisms. But McCullough and Pitts showed that neural network could, in principle, calculate any function like any digital computer. The result was more of a neuroscience than computer science: it was necessary to assume that the human brain can be viewed as a computing device.

Neural networks continue to be a valuable tool for neurobiological studies. For example, the individual layers of the network rules or settings of the weights and thresholds reproduced the observed features of human neuroanatomy and cognitive functions, and thus touched on how the brain processes information.

The First trainable neural network, "Perceptron" (or "Perceptron"), has been shown by Cornell University psychologist Frank Rosenblatt in 1957. The design of the "Perceptron" was similar to the modern neural network, except that it had one layer with adjustable weights and thresholds between the input and output layers.

"Perceptrons" were actively studied in psychology and computer science until 1959, when Minsky and Papert published a book entitled "Perceptrons", which showed that the work of everyday computing with a perceptron was impractical from the point of view of time expenses.

"of Course, all restrictions would disappear if you make the machines a little more complicated", for example, in two layers," says Poggio. But while the book had an inhibiting effect on the study of neural networks.

"These things should be considered in a historical context," says Poggio. "The proof was built for programming languages such as Lisp. Shortly before this, people used analog computers. It was not clear at that time what will generally result in programming. I think they went a little overboard, but, as always, it is impossible to divide everything in black and white. If we consider it as a contest between analogue computation and digital computation, then they fought because it was necessary."

the

Frequency

1980 years, however, scientists have developed the algorithms to modify neural network weights and thresholds, which were quite effective for networks with more than one layer, eliminating many of the constraints defined Minsky and Papercom. This area experienced a Renaissance.

But with a reasonable point of view in neural networks was lacking something. Long enough exercise could lead to a revision of the network settings until it will start to classify data in a useful way, but what these settings mean? What are the features of the image looks a Discerner of the objects and how he collects parts to form a visual signature cars, houses and cups of coffee? Learning weights of individual compounds does not respond to this question.

In recent years, computer scientists began to think of ingenious methods to determine the analytical strategies adopted neural networks. But in 1980-ies the strategy of these networks was unclear. Therefore, at the turn of the century the neural network were driven vector machines, an alternative approach to machine learning based on clean and elegant mathematics.

The Recent surge of interest in neural networks — the deep learning revolution — bound gaming industry. Complex graphics and fast-paced modern video games requires hardware that can keep up with the trend, which introduces the GPU (graphics processor) with thousands of relatively simple processing cores on a single chip. Very soon scientists realized that the architecture of the GPU is perfectly suited for neural networks.

Modern GPUs has allowed him to build a network of 1960-ies and two - and three-layer networks of the 1980s in bunches of 10, 15 and even 50-layer networks today. Here is responsible for what the word "deep" in "deep learning". The depth of the network. Currently, deep learning is responsible for the most effective system in almost all areas of artificial intelligence research.

the

Under the hood

The opacity of the networks are still concerned about the academics but on this front there is progress. Poggio directs a research program on the topic of the theoretical foundations of intelligence. Not so long ago, Poggio and his colleagues have produced a theoretical study of neural networks in three parts.

The First part which was published last month in the International Journal of Automation and Computing, addressed to the range of computations that can conduct network, deep learning, and that, when the deep web have advantages over the shallow. Parts two and three, which were released in the form of reports addressed to the problems of global optimization, that is, guarantee that a network will find the settings that best fit its training data, as well as cases when the network is so well aware of the specifics of the training data that may not generalize to other manifestations of the same categories.

There is still a lot of theoretical questions, the answers to which will have to give. But there is hope that neural networks can finally break the cycle of generations, which plunge them in hot, then cold.

...