Weight initialization #204
Conversation
|
Thanks, Michael, this is definitely needed. About 1.5 years ago I started an Initializers PR (#151) but forgot about it. Basically it follows a similar pattern to how activations and optimizers are done in NF, which allows complete customization if specified, and sane defaults (like the ones you have here) if unspecified. Do you think it would work well? |
|
Added it while doing this: https://github.com/OneAdder/neural-fortran/blob/text_classification_example/example/text_classification.f90 |
|
@milancurcic Yes, I think #151 will work! |
| if (& | ||
| self % activation_name == 'relu' & | ||
| .or. self % activation_name == 'leaky_relu' & | ||
| .or. self % activation_name == 'celu' & | ||
| ) then | ||
| call random_he(self % weights, self % input_size) | ||
| elseif (self % activation_name == 'sigmoid' .or. self % activation_name == 'tanhf') then | ||
| call random_xavier(self % weights, self % input_size) |
There was a problem hiding this comment.
Should these be as default? Or should the user be able to choose for another pseudo-random generator?
There was a problem hiding this comment.
I like how it's done here: #151
In the DL Framework of my dreams, I would have an option to pass the algorithm of weights initialization into a layer's constructor. So:
- What I want: Initializers stub #151 with Kaiming weights by default but it requires a lot of refactoring
- Why I made this PR-draft: it is correct from the mathematical standpoint, Xavier for S-shaped and He for .*elu. Will probably resolve CNN training on MNIST does not converge #145 if added to Conv layer
- How Torch does it: Kaiming for everything. Not ideal, but covers vast majority of cases
| @@ -23,4 +23,22 @@ impure elemental subroutine random_normal(x) | |||
| x = sqrt(- 2 * log(u(1))) * cos(2 * pi * u(2)) | |||
| end subroutine random_normal | |||
|
|
|||
| impure elemental subroutine random_he(x, n_prev) | |||
| !! Kaiming weight initialization | |||
| real, intent(in out) :: x | |||
There was a problem hiding this comment.
| real, intent(in out) :: x | |
| real, intent(out) :: x |
|
|
||
| impure elemental subroutine random_xavier(x, n_prev) | ||
| !! Kaiming weight initialization | ||
| real, intent(in out) :: x |
There was a problem hiding this comment.
| real, intent(in out) :: x | |
| real, intent(out) :: x |
| lower = -(1. / sqrt(real(n_prev))) | ||
| upper = 1. / sqrt(real(n_prev)) |
There was a problem hiding this comment.
| lower = -(1. / sqrt(real(n_prev))) | |
| upper = 1. / sqrt(real(n_prev)) | |
| upper = 1. / sqrt(real(n_prev)) | |
| lower = -upper |
| lower = -(1. / sqrt(real(n_prev))) | ||
| upper = 1. / sqrt(real(n_prev)) | ||
| call random_number(x) | ||
| x = lower + x * (upper - lower) |
There was a problem hiding this comment.
Is this correct if lower == -upper?
Weights Initialization
Added functions for Xavier and Kaiming. The rule of thumb here:
tanh,sigmoid, etc.) => Xavierrelu,gelu,silu, etc.) => KaimingFor networks without Layer or Batch Normalization, that simple tweak will significantly increase convergance