Hello World

PyTorch CNN

Module torch.nn

Document

Convolution Layers

class torch.nn.Conv2d(in_channels, out_channels, kernel_size, stride=1, padding=0, dilation=1, groups=1, bias=True)

Parameters:

  • in_channels (int) – Number of channels in the input image
  • out_channels (int)– Number of channels produced by the convolution
  • kernel_size (int or tuple) – Size of the convolving kernel
  • stride (int or tuple, optional) – Stride of the convolution
  • padding (int or tuple, optional) – Zero-padding added to both sides of the input
  • dilation (int or tuple, optional) – Spacing between kernel elements
  • groups (int, optional) – Number of blocked connections from input channels to output channels
  • bias (bool, optional) – If True, adds a learnable bias to the output

Examples:

1
2
3
4
5
6
7
8
>>> # With square kernels and equal stride
>>> m = nn.Conv2d(16, 33, 3, stride=2)
>>> # non-square kernels and unequal stride and with padding
>>> m = nn.Conv2d(16, 33, (3, 5), stride=(2, 1), padding=(4, 2))
>>> # non-square kernels and unequal stride and with padding and dilation
>>> m = nn.Conv2d(16, 33, (3, 5), stride=(2, 1), padding=(4, 2), dilation=(3, 1))
>>> input = autograd.Variable(torch.randn(20, 16, 50, 100))
>>> output = m(input)

Linear Layers

class torch.nn.Linear(in_features, out_features, bias=True)
Applies a linear transformation to the incoming data : $y=Ax+b$

Parameters:

  • in_features – size of each input sample
  • out_features – size of each output sample
  • bias – If set to False, the layer will not learn an additive bias. Default: True

Examples:

1
2
3
4
>>> m = nn.Linear(20, 30)
>>> input = autograd.Variable(torch.randn(128, 20))
>>> output = m(input)
>>> print(output.size())

Containers

class torch.nn.Sequential(*args)

A sequential container. Modules will be added to it in the order they are passed in the constructor. Alternatively, an ordered dict of modules can also be passed in.

Example:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
# Example of using Sequential
model = nn.Sequential(
nn.Conv2d(1,20,5),
nn.ReLU(),
nn.Conv2d(20,64,5),
nn.ReLU()
)

# Example of using Sequential with OrderedDict
model = nn.Sequential(OrderedDict([
('conv1', nn.Conv2d(1,20,5)),
('relu1', nn.ReLU()),
('conv2', nn.Conv2d(20,64,5)),
('relu2', nn.ReLU())
]))

add_module(name, module)
Adds a child module to the current module.
The module can be accessed as an attribute using the given name.

Parameters

class torch.nn.Parameter

A kind of Variable that is to be considered a module parameter.

Parameters are Variable subclasses, that have a very special property when used with Module s - when they’re assigned as Module attributes they are automatically added to the list of its parameters, and will appear e.g. in parameters() iterator. Assigning a Variable doesn’t have such effect. This is because one might want to cache some temporary state, like last hidden state of the RNN, in the model. If there was no such class as Parameter, these temporaries would get registered too.

Another difference is that parameters can’t be volatile and that they require gradient by default.

Parameters:

  • data (Tensor) – parameter tensor.
  • requires_grad (bool, optional) – if the parameter requires gradient. See Excluding subgraphs from backward for more details.

Loss functions

class torch.nn.MSELoss(size_average=True)
Creates a criterion that measures the mean squared error between n elements in the input x and target y:
$$loss(x,y) = \frac{1}{n} \sum{|x_i - y_i|^2}$$

x and y arbitrary shapes with a total of n elements each.

The sum operation still operates over all the elements, and divides by n.

The division by n can be avoided if one sets the internal variable size_average to False.

Examples:

1
2
3
4
5
6
output = net(input)
target = Variable(torch.range(1, 10)) # a dummy target, for example
criterion = nn.MSELoss()

loss = criterion(output, target)
print(loss)

Module torch.nn.functional

Pooling Layers

torch.nn.functional.max_pool2d(input, kernel_size, stride=None, padding=0, dilation=1, ceil_mode=False, return_indices=False)

Non-linear Activations

class torch.nn.ReLU(inplace=False)
Applies the rectified linear unit function element-wise $ReLU(x) = max(0,x)$

Parameters: inplace – can optionally do the operation in-place

Example:

1
2
3
4
>>> m = nn.ReLU()
>>> input = autograd.Variable(torch.randn(2))
>>> print(input)
>>> print(m(input))

Module torch.optim

Taking an optimization step

  1. optimizer.step()
  2. optimizer.step(closure)
    Some optimization algorithms such as Conjugate Gradient and LBFGS need to reevaluate the function multiple times, so you have to pass in a closure that allows them to recompute your model. The closure should clear the gradients, compute the loss, and return it.

Note

  1. torch.nn only supports mini-batches. The entire torch.nn package only supports inputs that are a mini-batch of samples, and not a single sample.
    For example, nn.Conv2d will take in a 4D Tensor of nSamples x nChannels x Height x Width.
    If you have a single sample, just use input.unsqueeze(0) to add a fake batch dimension.

  2. To backpropogate the error all we have to do is to loss.backward(). You need to clear the existing gradients though, else gradients will be accumulated to existing gradients.

  3. To select GPUs, use CUDA_VISIBLE_DEVICES=2,3 python file.py