*tuple and **dictionary

from /mine_v2/training/BaseTraining.py
self.loss = ug.get_class(self.conf.get_string('network.loss.loss_type'))(**self.conf.get_config('network.loss.properties'))

in a function call

*t means “treat the elements of this iterable as positional arguments to this function call.”

def f(x,y):
    return x+y

>>>t = (1,2)
>>>f(*t)
3

**d means “treat the key-value pairs in the dictionary as additional named arguments to this function call.”

def f(x,y):
    return x+y

>>>d = {'x':1,'y':2}
>>>f(**d)
3

in a function signature

*t means “take all additional positional arguments to this function and pack them into this parameter as a tuple.”

def f(*t):
    print(t)

>>>f(1,2)
(1,2)

**d means “take all additional named arguments to this function and insert them into this parameter as dictionary entries.”

def f(**d):
    print(d)

>>> f(x=1, y=2)
{'y': 2, 'x': 1}

torch.save() & torch.load()

save

step1. define a dictionarty and the path

state = {'epoch': model.epoch, 
'model_state': network.state_dict(), 
'optimizer': optimizer.state_dict()}

dir=(os.path.join(path,subdir)+".pth")

step2. use the torch.save()

torch.save(state, dir)

load

if you want to continue the training runner from the checkpoint

checkpoint = torch.load(dir)
model.load_state_dict(checkpoint['model_state'])
optimizer.load_state_dict(checkpoint['optimizer'])
start_epoch = checkpoint['epoch']+1

model.train()

with torch.no_grad()

There is a question that if we disable gradient calculation in the plot module:

if network.epoch == plot.epoch:
    with torch.no_grad():
        ...


network.train()

then we use .train(), will it restore the funtion of gradient calculation?

and we can call either network.eval() or network.train(mode=False) to tell that we are testing.

Besides, we can also check the network.training flag, it is False when in eval mode. And currently only Dropout and BatchNorm care about that flag. (https://stackoverflow.com/questions/51433378/what-does-model-train-do-in-pytorch)