All about Deep Learning Tutorial

Introduction

In the previous article, we studied Machine Learning. One thing that I believe is that if we can correlate anything with us or our life, there are greater chances of understanding the concept. So I will try to explain everything by relating it to humans.

What is Deep Learning? 

Deep learning is an artificial intelligence function that imitates the workings of the human brain in processing data and creating patterns for use in decision making. Deep learning is a subset of machine learning in artificial intelligence (AI) that has networks capable of learning unsupervised from data that is unstructured or unlabeled. Also known as deep neural learning or deep neural network. 

Imagine a group of detectives investigating complex cases. At first, they are novices and struggle to identify culprits correctly. They analyze crime scenes and evidence, making educated guesses about suspects, but often get things wrong.

To improve their skills, senior detectives mentor them. The senior detectives have extensive experience and can quickly determine if a suspect is guilty or not. They guide the novice detectives by saying things like, "Yes, this person is likely the culprit" or "No, this person is not the culprit." Over time, the novice detectives become more accurate in identifying the real culprits. However, they might not fully understand the exact reasons behind their decisions. Their expertise lies in recognizing patterns, connecting different pieces of evidence, and drawing conclusions.

In a similar manner, a neural network operates. It consists of layers, and each layer represents a deeper level of knowledge or abstraction. A neural network with multiple layers can learn and represent more intricate features compared to a simpler one.

The learning process in a neural network occurs in two phases. In the first phase, the network processes the input data through various nonlinear transformations to create a statistical model as output. Initially, this model may not be accurate, but it serves as an initial estimate.

In the second phase, the neural network improves the model using mathematical techniques, including backpropagation, which utilizes derivatives. These techniques help adjust the model's parameters to minimize errors and improve its accuracy, similar to how the detective's mentors guide them to enhance their skills.

To achieve a satisfactory level of accuracy, the neural network repeats these two phases multiple times, fine-tuning its parameters in each iteration. Just like the detectives' experience grows with each case they solve, the neural network gradually gains a deeper understanding of the patterns in the data, leading to improved performance.

In summary, deep learning involves a process of iteratively refining a statistical model by recognizing patterns and complex features, much like detectives honing their investigative skills with guidance from experienced mentors.

Who invented Deep Learning? 

The history of Deep Learning can be traced back to 1943 when Walter Pitts and Warren McCulloch created a computer model based on the neural networks of the human brain. They used a combination of algorithms and mathematics they called “threshold logic” to mimic the thought process 

Advantages/Features of Deep Learning

1. Features are automatically deduced and optimally tuned for the desired outcome. Features are not required to be extracted ahead of time. This avoids time-consuming machine learning techniques.
2. Robustness to natural variations in the data is automatically learned.
3. The same neural network-based approach can be applied to many different applications and data types.
4. Massive parallel computations can be performed using GPUs and are scalable for large volumes of data. Moreover, it delivers better performance results when the amount of data is huge. 
5. The deep learning architecture is flexible to be adapted to new problems in the future. 

Disadvantages/Shortcomings of Deep Learning 

1. It requires a very large amount of data to perform better than other techniques.
2. It is extremely expensive to train due to complex data models. Moreover, deep learning requires expensive GPUs and hundreds of machines. This increases the cost to the users.
3. There is no standard theory to guide you in selecting the right deep learning tools as it requires knowledge of topology, training method, and other parameters. As a result, it is difficult to be adopted by less skilled people.
4. It is not easy to comprehend output based on mere learning and requires classifiers to do so. Convolutional neural network-based algorithms perform such tasks.

Drivers of Deep Learning

The drivers of Deep Learning are listed below:

• Availability of multi-layered learning network

• Ability to leverage Big Data

• Expanded use of high-performing graphics processing units (GPU)

• An improved scale of data and size of neural networks

• Improved performance of Neural Networks

• Availability of a large amount of labeled data

• The ability of GPUs to perform parallel computing 

Difference Between Machine Learning and Deep Learning?
 

	Machine Learning	Deep Learning
Formal Definition	A computer program is said to learn from experience E concerning some class of tasks T and performance measure P if its performance at tasks in T, as measured by P, improves with experience E	Deep learning is a particular kind of machine learning that achieves great power and flexibility by learning to represent the world as a nested hierarchy of concepts, with each concept defined concerning simpler concepts, and more abstract representations computed in terms of less abstract ones.
Data	When Data is small, machine learning is used	When data is large, deep learning is used,
Hardware Requirements	Machine Learning algorithms can work on CPUs	Deep Learning algorithms require GPUs and TPUs to work
Feature Engineering	In Machine learning, most of the applied features need to be identified by an expert and then hand-coded as per the domain and data type.	Deep learning reduces the task of developing new feature extractors for every problem
Problem Solving Approach	Machine Learning tries to break the problem into parts and solve each part separately	Deep Learning solves the problem in an end-to-end manner
Execution Time	The execution time is comparatively less i.e. a few hours to a week	The execution time is comparatively more i.e up to weeks
Interpretability	Machine Learning algorithms are easy to interpret	Deep Learning algorithms are not that easy to understand, or we can say difficult to impossible
Number of Algorithms present	We currently have a good number of Machine learning algorithms	We currently have very few numbers of Deep Learning algorithms
Training Time	Short	Long

The following image shows a comparison between the popularity of Machine Learning and Deep Learning.

Legend:

Blue: Machine  Learning

Red: Deep Learning 

 

Source: Google Trends 

Python Deep Learning Libraries
 

1. TensorFlow

TensorFlow is a free and open-source software library for dataflow and differentiable programming across a range of tasks. It is a symbolic math library and is also used for machine learning applications such as neural networks. It is used for both research and production at Google.

For a detailed tutorial on TensorFlow, please click 

2. Keras

Keras is a high-level neural networks API, written in Python and capable of running on top of TensorFlow, CNTK, or Theano. It was developed with a focus on enabling fast experimentation. Being able to go from idea to result with the least possible delay is key to doing good research.

3. Apache mxnet

Apache MXNet is an open-source deep learning software framework, used to train, and deploy deep neural networks. It has wide applicability across various OS and using various languages.

4. Caffe

Caffe is a deep learning framework made with expression, speed, and modularity in mind. It is developed by Berkeley AI Research (BAIR) and by community contributors. Yangqing Jia created the project during his Ph.D. at UC Berkeley. Caffe is released under the BSD 2-Clause license.

5. Theano

Theano is a Python library and optimizing compiler for manipulating and evaluating mathematical expressions, especially matrix-valued ones. In Theano, computations are expressed using a NumPy-Esque syntax and compiled to run efficiently on either CPU or GPU architectures.

6. Microsoft Cognitive Toolkit

Azure Cognitive Services are APIs, SDKs, and services available to help developers build intelligent applications without having direct AI or data science skills or knowledge. Azure Cognitive Services enable developers to easily add cognitive features into their applications. The goal of Azure Cognitive Services is to help developers create applications that can see, hear, speak, understand, and even begin to reason. The catalog of services within Azure Cognitive Services can be categorized into five main pillars - Vision, Speech, Language, Web Search, and Decision.

7. PyTorch

PyTorch is an open-source machine learning library based on the Torch library, used for applications such as computer vision and natural language processing. It is primarily developed by Facebook's artificial intelligence research group. It is free and open-source software released under the Modified BSD license.

8. Eclipse DeepLearning4J (DL4J)

Eclipse Deeplearning4j is the first commercial-grade, open-source, distributed deep-learning library written for Java and Scala. Integrated with Hadoop and Apache Spark, DL4J brings AI to business environments for use on distributed GPUs and CPUs.

9. Lasagne

Lasagne is a lightweight library to build and train neural networks in Theano. Its main features are:

• Supports feed-forward networks such as Convolutional Neural Networks (CNNs), recurrent networks including Long Short-Term Memory (LSTM), and any combination thereof
• Allows architectures of multiple inputs and multiple outputs, including auxiliary classifiers
• Many optimization methods including Nesterov momentum, RMSprop, and ADAM
• Freely definable cost function and no need to derive gradients due to Theano's symbolic differentiation
• Transparent support of CPUs and GPUs due to Theano's expression compiler 

10. noLearn

nolearn contains a number of wrappers and abstractions around existing neural network libraries, most notably Lasagne, along with a few machine learning utility modules. All code is written to be compatible with scikit-learn.

11. PyLearn2

Pylearn2 is a machine learning library. Most of its functionality is built on top of Theano. This means you can write Pylearn2 plugins (new models, algorithms, etc) using mathematical expressions, and Theano will optimize and stabilize those expressions for you, and compile them to a backend of your choice (CPU or GPU).

Artificial Neural Networks

A neural network is a series of algorithms that endeavors to recognize underlying relationships in a set of data through a process that mimics the way the human brain operates. In this sense, neural networks refer to systems of neurons, either organic or artificial. Neural networks can adapt to changing input; so the network generates the best possible result without needing to redesign the output criteria. The concept of neural networks, which has its roots in artificial intelligence, is swiftly gaining popularity in the development of trading systems.

Artificial Neural Networks Process

Artificial Neural Networks consist of the following four main parts:

1. Neuron

Artificial Neural Networks contain layers of neurons. A neuron is a computational unit that calculates a piece of information based on weighted input parameters. Inputs accepted by the neuron are separately weighted.

Inputs are summed and passed through a non-linear function to produce output. Each layer of neurons detects some additional information, such as edges of things in a picture or tumors in a human body. Multiple layers of neurons can be used to detect additional information about input parameters.

2. Nodes

Artificial Neural Network is an interconnected group of nodes akin to the vast network of layers of neurons in a brain. Each circular node represents an artificial neuron and an arrow represents a connection from the output of one neuron to the input of another.

3. Inputs

Inputs are passed into the first layer. Individual neurons receive the inputs, with each of them receiving a specific value. After this, an output is produced based on these values.

4. Outputs

The outputs from the first layer are then passed into the second layer to be processed. This continues until the final output is produced. The assumption is that the correct output is predefined.

Each time data is passed through the network, the result is compared with the correct one, and tweaks are made to their values until the network creates the correct final output each time.

Classification of Artificial Neural Networks

1. Shallow neural network

The Shallow neural network has only one hidden layer between the input and output.

2. Deep neural network:

Deep neural networks have more than one layer. For instance, the Google LeNet model for image recognition counts 22 layers.

Types of Deep Learning Networks
 

1. Feed-forward Neural Networks

The simplest type of artificial neural network. With this type of architecture, information flows in only one direction, forward. It means, the information flows starts at the input layer, goes to the "hidden" layers, and ends at the output layer. The network does not have a loop. Information stops at the output layers.

2. Recurrent neural networks (RNNs)

RNN is a multi-layered neural network that can store information in context nodes, allowing it to learn data sequences and output a number or another sequence. In simple words it an Artificial neural network whose connections between neurons include loops. RNNs are well suited for processing sequences of inputs.

For example, if the task is to predict the next word in the sentence "Do you want a…………?

• The RNN neurons will receive a signal that points to the start of the sentence.
• The network receives the word "Do" as an input and produces a vector of the number. This vector is fed back to the neuron to provide a memory to the network. This stage helps the network to remember it received "Do" and it received it in the first position.
• The network will similarly proceed to the next words. It takes the word "you" and "want." The state of the neurons is updated upon receiving each word.
• The final stage occurs after receiving the word "a." The neural network will provide a probability for each English word that can be used to complete the sentence. A well-trained RNN probably assigns a high probability to "café", "drink," "burger," etc.

Common uses of RNN

• Help securities traders to generate analytic reports
• Detect abnormalities in the contract of financial statement
• Detect fraudulent credit-card transaction
• Provide a caption for images
• Power chatbots
• The standard uses of RNN occur when the practitioners are working with time-series data or sequences (e.g., audio recordings or text).

3. Convolutional neural networks (CNN)

CNN is a multi-layered neural network with a unique architecture designed to extract increasingly complex features of the data at each layer to determine the output. CNN's are well suited for perceptual tasks.

CNN is mostly used when there is an unstructured data set (e.g., images) and the practitioners need to extract information from it

For instance, if the task is to predict an image caption:

• The CNN receives an image of let's say a cat, this image, in computer term, is a collection of the pixel. Generally, one layer for the greyscale picture and three layers for a color picture.
• During the feature learning (i.e., hidden layers), the network will identify unique features, for instance, the tail of the cat, the ear, etc.
• When the network thoroughly learned how to recognize a picture, it can provide a probability for each image it knows. The label with the highest probability will become the prediction of the network.

4. Reinforcement Learning

Reinforcement learning is a subfield of machine learning in which systems are trained by receiving virtual "rewards" or "punishments," essentially learning by trial and error.

Google's DeepMind has used reinforcement learning to beat a human champion in the Go games. Reinforcement learning is also used in video games to improve the gaming experience by providing smarter bot.

One of the most famous algorithms are:

• Q-learning
• Deep Q network
• State-Action-Reward-State-Action (SARSA)
• Deep Deterministic Policy Gradient (DDPG)

The following is a list of Neural Networks with their corresponding use cases

Neural Network	Use Case
ANN (Artificial Neural Network)	Computational Neuroscience
CNN (Convolutional Neural Network)	Image Processing
RNN (Recurrent Neural Network)	Speech Recognition
DNN (Deep Neural Network)	Acoustic Modeling
DBN (Deep Brief Network)	Drug Discovery

Computational Graph  

Computational graphs are a nice way to think about mathematical expressions. For example, consider the expression e=(a+b)∗(b+1). There are three operations: two additions and one multiplication. To help us talk about this, let’s introduce two intermediary variables, c, and d so that every function’s output has a variable. We now have:

c=a+b

d=b+1

e=c∗d

To create a computational graph, we make each of these operations, along with the input variables, into nodes. When one node’s value is the input to another node, an arrow goes from one to another.

 

These sorts of graphs come up all the time in computer science, especially in talking about functional programs. They are very closely related to the notions of dependency graphs and call graphs. They’re also the core abstraction behind the popular deep learning framework Theano.

We can evaluate the expression by setting the input variables to certain values and computing nodes up through the graph. For example, let’s set a=2 and b=1 

The output comes out to be 6. 

Python Implementation of Computational Graph

from dask import delayed
import dask

@delayed
def cube(num):
    print("Cube function:", num)
    print()
    return num * num * num


@delayed
def sum_list(args):
    print("Sum_list function:", args)
    return sum(args)


items = [1, 2, 3]
computation_graph = sum_list([cube(i) for i in items])
computation_graph.visualize()

Output  

 

Examples of deep learning applications
 

1. AI in Finance

The financial technology sector has already started using AI to save time, reduce costs, and add value. Deep learning is changing the lending industry by using more robust credit scoring. Credit decision-makers can use AI for robust credit lending applications to achieve faster, more accurate risk assessment, using machine intelligence to factor in the character and capacity of applicants.

Underwrite is a Fintech company providing an AI solution for credit makers company. underwrite.ai uses AI to detect which applicant is more likely to pay back a loan.

2. AI in HR

Under Armour, a sportswear company revolutionizes hiring and modernizes the candidate experience with the help of AI. They had, on average, 30000 resumes a month. Reading all of those applications and begin to start the screening and interview process was taking too long. 

At that time, Under Armour had all of the 'must-have' HR technology in place such as transactional solutions for sourcing, applying, tracking and onboarding but those tools weren't useful enough. Under armor choose HireVue, an AI provider for HR solution, for both on-demand and live interviews. The results were bluffing; they managed to decrease by 35% the time to fill. In return, the hired higher quality staff.

3. AI in Marketing

AI is a valuable tool for customer service management and personalization challenges. Improved speech recognition in call-center management and call routing as a result of the application of AI techniques allows a more seamless experience for customers.

For example, deep-learning analysis of audio allows systems to assess a customer's emotional tone. If the customer is responding poorly to the AI chatbot, the system can be rerouted the conversation to real, human operators that take over the issue.

Conclusion

In this article, we studied deep learning, what is deep learning, who invented deep learning, advantages of deep learning, disadvantages of deep learning, drivers of deep learning, the difference between machine learning and deep learning, python deep learning libraries, artificial neural network, artificial neural network process, classification of artificial neural network, types of deep learning networks, computational graph and examples of deep learning applications . Hope you were able to understand each and everything. For any doubts, please comment on your query.

In the next article, we will learn about Data Science.

Congratulations!!! you have climbed your next step in becoming a successful ML Engineer.

Next Article In this Series >> Data Science