Aniketh Vijesh

Grokking Deep Learning Chapter 2

A brief intro

So the basic idea for this chapter is forward propagation and how a neural network essentially "learns".

The basic paradigm of any model is this:

Predict

Compare

Learn

Repeat

Another rule of thumb when using neural networks is that he minimum amount of data that it should be given is the same amount of data that should be given to a normal human to do the same task, for example, if a person is given a single pixel in a photo and is asked to tell what it is, it is impossible to do so.

So a neural network obviously starts out with just the input fields and some weights.

but wait.. what are weights?

Weights are how data is connected to the overall output. We fine-tune these values to match the output.

Imagine them to be something similar to the amount of ingredients going into the dish.

So how does a neural network learn?

They essentially learn through trial and error. They keep changing the weights around until the best possible fit is found at which we are free to either terminate or proceed to the next stage 

Building our own simple neural networks

Building a neural network with just a single input and an output is easy. Each input will be scaled(multiplied)  by a weight, which is then given out as the output or prediction. Lets go to the next stage, how to handle multiple inputs. Easy, we just have a different weight for each of the inputs. Simply scale them with their respective weight and then add the individual scaled outputs to get the final prediction

Here is a simple example made using python

Now let's make it a bit more complicated, How do you handle multiple outputs?

Let's think of the first case:

A single input and multiple outputs

This is rather easy, it's just getting unadded scaled results. In the last example, we added the scaled inputs to get the final prediction,  in this case, we just skipped the last step of addition.

The next case is multiple inputs and multiple outputs 

This too is rather easy, we just scale the input by different weights for different outputs and at the end just add all of them up. The one thing about this is that it gets complicated to store all that data that's why we use vectors and matrices to store data for easy access and usability.

Here is an example made using NumPy

What is forward Propagation?

It is when a neural network takes input data and makes a prediction. It is called so due to the activations happening or propagating forwards in the network.

What are activations?

An activation is essentially the value generated or output after scaling, there are conditions when some nodes or neurons may not give output or that they are “blocked” from giving a prediction.