source: http://rebooting.ai

“Rebooting AI” notes-3

(a series of posts for each chapter)

<<< Previous post (“Rebooting AI” notes-2)

>>> Next post (“Rebooting AI” notes-4)

In this post I would like to list some quotes from the book which I found them either important or interesting. (Note: some parts of the quotes are paraphrased by myself and some of them are my own insights)

3. Deep Learning, and Beyond

In the early days of AI there wasn’t much data and data wasn’t not a major part of the problem either. Most researchers followed a “knowledge-based” approach (GOFAI — Good Old Fashioned AI): encode the knowledge by hand. Nowadays these techniques has largely been displaced by “Machine Learning” which tries to learn everything from the data.

In 1950s Frank Rosenblatt build a “neural network” that aimed to learn to recognize objects around it without requiring programmers to anticipate every contingency in advance. His camera had only (20X20) 400 pixels and the system was not able to perform well because of lack of sophisticated hardware and large amounts of data. Until big data became widespread , the general consensus in the AI community was that neural network approach was hopeless because it was not working well compared to other methods.

However, there were couple of researchers (Geoffrey Hinton, Yoshua Bengio, Yann LeCun, Jürgen Schmidhuber, etc.) who have been believing that neural networks are the way to go. In 2010s, when big data came out, the neural networks approach gain a huge interest because they were the best performing approach to the data. In 2012, the team of researcher working with Geoffrey Hinton figured out using GPUs (which has been developed for video games for a while) to train neural networks which has increased the power of neural networks. Their model (deep neural network model) won the 2012 ImageNet Object Recognition challenge. This is was the revolution!

It may be important to mention the relationships of the terms: AI, ML, DL.

• Artificial Intelligence consists of the following algorithms: Planning, Reasoning, Search, Knowledge Representation, Machine Learning
• Machine Learning: Decision Trees, Genetice Algorithms, Probabilistic Learning, Deep Learning
• Deep Learning: Machine Learning technique which learn a various representations of the data with multiple layers of neural networks (deep nets)

Deep Learning is based on two fundamental ideas: hierarchical pattern recognition and learning

• Hierarchical pattern recognition. Neuroscientists David Hubel and Trosten Wiesel discovered that different neurons in the visual system responded to visual stimuli in different ways; some responded to very simple stimuli (like lines) some others responded to more complex stimuli (like particular object shape). Their theory was that complex stimuli might be recognized through a hierarchy of increasing abstraction. In 1980s, Japanese neural network pioneer Kunahiko Fukushima build a Neocognitron which is the actual computational realization of Hubel and Wisels’s theory. Neocognitron was made up of a set of layers. Starting from left left to right, the layers were responsible of extracting contrast, edges, and so forth, finally the last layer identifies the category of input image.
• Learning. In neural networks there is a connections between nodes of different layers. These connections (sometimes we call them weights) could be either designed by hand or can be learnt. This is procedure is called “training” a neural network.

Researchers, including Hinton, found a way of training neural network by taking small steps towards the “best” weight configuration (to perform on a particular task) called backpropagation, which is the workhorse of deep learning.

Deep Learning was a game changer in terms of feature engineering; before the deep learning revolution engineers supposed to do a feature engineering a lot, after deep learning showed up it supposed to learn the needed features during the training. Because of auto feature extraction, deep learning achieving superior performance on various fields:

• supervised learning: object recognition (classification, object detection), speech recognition (speech-to-text, speech synthesis, etc)
• unsupervised learning: generative models (creating synthetic art, colorizing old pictures, etc)
• deep reinforcement learning: playing games

Despite all the manifest progress in certain areas (mentioned above), deep learning still is not any kind of universal solvent (and, of course, it is not supposed to be…). There are three core problems: deep learning is greedy, opaque and brittle.

1. Deep Learning is greedy: deep learning requires massive amount of data.
2. Deep Learning is opaque: difficult to understand why deep neural network made a particular decision.
3. Deep Learning is brittle: deep learning can be perfect in one situation and utterly wrong in another.