Gloon parameters and Blocks naming

API : each parameter or Block in gluon has prefix prefix prefix. The name of Parameters is specified by the user, and the name of Block can be specified by the user or generated automatically.

from __future__ import print_function
import mxnet as mx
from mxnet import gluon

Naming Blocks

You can manually apply a prefix when creating a Block:

mydense = gluon.nn.Dense(100, prefix='mydense_')
print(mydense.prefix)

mydense_

When no prefix is given, gluon automatically generates:

dense0 = gluon.nn.Dense(100)
print(dense0.prefix)

dense0_

When more Blocks of the same type are created, prefix names are automatically incremented to avoid conflicts:

dense1 = gluon.nn.Dense(100)
print(dense1.prefix)

dense1_

Naming Parameters

Parameters in a Block are named by prefixing the parameter name with the Block:

print(dense0.collect_params())


dense0_ (
  Parameter dense0_weight (shape=(100, 0), dtype=<type 'numpy.float32'>)
  Parameter dense0_bias (shape=(100,), dtype=<type 'numpy.float32'>)
)

Name scopes

To manage the names of nested blocks, each Block is appended with a name range name_scope. All blocks created within the name scope are prefixed with the names of their parent blocks.

class Model(gluon.Block):
    def __init__(self, **kwargs):
        super(Model, self).__init__(**kwargs)
        with self.name_scope():
            self.dense0 = gluon.nn.Dense(20)
            self.dense1 = gluon.nn.Dense(20)
            self.mydense = gluon.nn.Dense(20, prefix='mydense_')

    def forward(self, x):
        x = mx.nd.relu(self.dense0(x))
        x = mx.nd.relu(self.dense1(x))
        return mx.nd.relu(self.mydense(x))

Now verify:

model0 = Model()
model0.initialize()
model0(mx.nd.zeros((1, 20)))
print(model0.prefix)
print(model0.dense0.prefix)
print(model0.dense1.prefix)
print(model0.mydense.prefix)


model0_
model0_dense0_
model0_dense1_
model0_mydense_

If the Model is instantiated again, it will also be prefixed incrementally to avoid conflicts:

model1 = Model()
print(model1.prefix)
print(model1.dense0.prefix)
print(model1.dense1.prefix)
print(model1.mydense.prefix)


model1_
model1_dense0_
model1_dense1_
model1_mydense_

It is highly recommended to manually specify a prefix in the Block at the highest level, for example: model = mold (prefix = 'mymodel').

The above principles apply to all container s:

And name_scope can be used in__ init__ Inside, or__ init__ Outside, like this:

with net.name_scope():
    net.add(gluon.nn.Dense(20))
    net.add(gluon.nn.Dense(20))
print(net.prefix)
print(net[0].prefix)
print(net[1].prefix)


sequential0_
sequential0_dense0_
sequential0_dense1_

gluon.model_ The same with zoo:

net = gluon.nn.Sequential()
with net.name_scope():
    net.add(gluon.model_zoo.vision.alexnet(pretrained=True))
    net.add(gluon.model_zoo.vision.alexnet(pretrained=True))
print(net.prefix, net[0].prefix, net[1].prefix)


sequential1_ sequential1_alexnet0_ sequential1_alexnet1_

Saving and loading

Because the prefixes of model0 and model1 defined above are different, the parameter names are also different:

print(model0.collect_params(), '\n')
print(model1.collect_params())

model0_ (

  Parameter model0_dense0_weight (shape

=(20L, 20L), dtype=<type 'numpy.float32'>)
  Parameter model0_dense0_bias (shape=(20L,), dtype=<type 'numpy.float32'>)
  Parameter model0_dense1_weight (shape=(20L, 20L), dtype=<type 'numpy.float32'>)
  Parameter model0_dense1_bias (shape=(20L,), dtype=<type 'numpy.float32'>)
  Parameter model0_mydense_weight (shape=(20L, 20L), dtype=<type 'numpy.float32'>)
  Parameter model0_mydense_bias (shape=(20L,), dtype=<type 'numpy.float32'>)
) 

model1_ (
  Parameter model1_dense0_weight (shape=(20, 0), dtype=<type 'numpy.float32'>)
  Parameter model1_dense0_bias (shape=(20,), dtype=<type 'numpy.float32'>)
  Parameter model1_dense1_weight (shape=(20, 0), dtype=<type 'numpy.float32'>)
  Parameter model1_dense1_bias (shape=(20,), dtype=<type 'numpy.float32'>)
  Parameter model1_mydense_weight (shape=(20, 0), dtype=<type 'numpy.float32'>)
  Parameter model1_mydense_bias (shape=(20,), dtype=<type 'numpy.float32'>)
)

At this time, if the parameters of model0 are saved and you want to load into model1, the load may fail because the names do not match:

model0.collect_params().save('model.params')
try:
    model1.collect_params().load('model.params', mx.cpu())
except Exception as e:
    print(e)

Parameter 'model1_dense0_weight' is missing in file 'model.params', which contains parameters: 'model0_mydense_weight', 'model0_dense1_bias', 'model0_dense1_weight', 'model0_dense0_weight', 'model0_dense0_bias', 'model0_mydense_bias'. Please make sure source and target networks have the same prefix.

To do this, the correct posture is to use save_parameters/load_parameters instead of collect_params and save/load

save_parameters use model structure instead of name matching.

model0.save_parameters('model.params')
model1.load_parameters('model.params')
print(mx.nd.load('model.params').keys())

['dense0.bias', 'mydense.bias', 'dense1.bias', 'dense1.weight', 'dense0.weight', 'mydense.weight']

Replacing Blocks from networks and fine-tuning

The output of the pre training model is changed to class 1000, and to class 100

alexnet = gluon.model_zoo.vision.alexnet(pretrained=True)
print(alexnet.output)
print(alexnet.output.prefix)

Dense(4096 -> 1000, linear)
alexnet0_dense2_

Change classification level:

with alexnet.name_scope():
    alexnet.output = gluon.nn.Dense(100)
alexnet.output.initialize()
print(alexnet.output)
print(alexnet.output.prefix)

Dense(None -> 100, linear)
alexnet0_dense3_

Posted by 121212 on Fri, 05 Jun 2020 21:09:13 -0700

Programmer Group

Gloon parameters and Blocks naming

Name scopes

Saving and loading

Replacing Blocks from networks and fine-tuning

Hot Keywords