The mountains and rivers are full of doubts_The fastest descent asks the gradient (introduction to deep learning series seven)

Abstract: "The world's martial arts, only fast is not broken". If you want to quickly see the infinite scenery, you must climb dangerous peaks; if you want to quickly reach the valley at the bottom of the mountain, you must roll steep slopes. The reason for this rolling hillside is actually the gradient decreasing strategy, and the gradient decreasing strategy is the "male (ji) person (chu)" behind the success of the BP algorithm. If you want to know why, come and find out!

For more in-depth articles, please pay attention to: https://yq.aliyun.com/cloud

series of articles:
"Deep" is like the sea when you enter Houmen, deep learning deep geometry (one of the deep learning introductory series)
artificial "carbon" Still exhausted, the future of intelligent "silicon" is unknown (Introductory Deep Learning Series 2)
Neural Networks are indescribable, MP models seem to be available (Introductory Deep Learning Series 3)
"Machine Learning" Triple Door, "The Golden Mean" approaching People (Introduction to Deep Learning Series No. 4)
Hello World Perceptron, I will rest in peace if I understand you (Introductory Deep Learning Series Part 5)
Loss function for weight loss, neural network weight adjustment (Introduction to Deep Learning Series 6)

More than a year ago, Dr. Wu Jun wrote a best-selling book "Intelligence Era" [1]. The book mentions that in the field of artificial intelligence, there is a school called "Bird Flying School", also known as "Imitation School". It is said that when people want to learn to fly, the first thing that comes to mind is to fly like a mocking bird.
Many years ago, the Indian poet Tagore published a book "Flying Birds", which has a famous line: "The sky has not left traces of wings, but I have already flown." Some people interpret this as, "Many things have been done in this world, but they are not known, but so what? The important thing is that I have done it and gained a lot from it."
More than two thousand years ago, Sima Qian wrote in "Historical Records • Funny Biography": "If this bird doesn't fly, it will already soar into the sky; It is said that when King Zhuang of Chu chose "dormancy" when he was "not looking after me". Dormancy is just a process of storing power. Sooner or later, it will be ready to go, and "going" will reach the sky.
The emotional intersection of these three makes me think of the protagonist of this chapter, Professor Geoffrey Hinton. In the academic world, he is such an "inspirational" figure!
In 1986, Professor Hinton and his friends redesigned the BP algorithm, imitating the working mechanism of the brain with an "artificial neural network", and "kiss" woke up the "artificial intelligence" princess who had been sleeping for many years.
But "good flowers don't bloom often, and good times don't always come." When the scenery is gone, Hinton and his research direction are gradually forgotten by the world.
Once this "forgotten" cold bench sits, it will be 30 years.
But in these 30 years, Hinton is like a "flying bird", even if he "flyed without a trace", he never gave up. Get up from where you fall. It really doesn't work, even if you change your vest, you will have to live a new life.
Yuru is successful, and the merit is not donated.
Finally, in 2006, Hinton et al. proposed "Deep Belief Nets (DBN)" (this is actually the vest of multi-layer neural networks) [2]. This "deep belief net" was later called "deep learning". Finally, Hinton once again shined in the world of artificial intelligence, and was subsequently named the "Godfather of Deep Learning".
However, careful readers can find that even though Hinton et al. proposed the "deep belief network", in the next 10 years, this concept has developed tepidly (as shown in Figure 1). It was not until the later period (after 2012) that with the rise of big data and big computing (GPU, cloud computing, etc.), deep learning began to become popular, and it became very popular for a while.

473387ba332c29a061579c612ecdcabd36b2d99a
Figure 7-1 Google Trends of Deep Learning
Looking back at Jeffrey Hinton's academic career over the past 40 years, it can be said that he had ups and downs, but he finally achieved positive results. But if we talk about it in detail, this "brute force" has to be blown from 1986.

7.1 The masterpiece in 1986
In October 1986, Jeffrey Hinton was still at Carnegie Mellon University. He and David Rumelhart, a cognitive psychologist at the University of California, San Diego, and others, jointly published in the prestigious academic journal Nature: "Learning Representations by Backpropagation Algorithms (Learning Representations by Back-propagating errors)” paper [3]. In this paper, the application of backpropagation algorithm (BP) in neural network model is systematically and concisely described for the first time. The algorithm reduces the calculation amount of network weight error correction from the original proportional to the square of the number of neurons to only the sum of the The number of neurons is proportional to itself.
At the same time, the background at the time was that in the late 1980s, the development of Intel x86 series of microprocessors and memory technology made the computer's operating speed and data access speed several orders of magnitude faster than 20 years ago. . This time (the amount of calculation decreases) and the other (the calculation speed increases), and the multi-layer neural network can greatly enhance the representation ability of the data features by setting the hidden layer, so as to easily solve the abnormality that the perceptron cannot achieve. The XOR gate problem, these favorable circumstances of "the right place and the right people" greatly eased Minsky's criticism of the neural network at that time.
As a result, the research on artificial neural networks has gradually recovered.
http://click.aliyun.com/m/23532/