Optimizer

OptimizerConfig

class matorage.OptimizerConfig(**kwargs)[source]

Optimizer configuration classes. This class overrides StorageConfig.

Parameters
  • endpoint (string, require) – S3 object storage endpoint. or If use NAS setting, NAS folder path.

  • access_key (string, optional, defaults to None) – Access key for the object storage endpoint. (Optional if you need anonymous access).

  • secret_key (string, optional, defaults to None) – Secret key for the object storage endpoint. (Optional if you need anonymous access).

  • secure (boolean, optional, defaults to False) – Set this value to True to enable secure (HTTPS) access. (Optional defaults to False unlike the original MinIO).

  • optimizer_name (string, require) – optimizer name.

  • additional (dict, optional, defaults to {}) – Parameters for additional description of optimizers. The key and value of the dictionay can be specified very freely.

  • compressor (dict, optional, defaults to {"complevel" : 0, "complib" : "zlib"}) –

    Optimizer compressor option. It consists of a dict type that has complevel and complib as keys. For further reference, read pytable’s Filter.

    • complevel (integer, defaults to 0) : compressor level(0~9). The larger the number, the more compressed it is.

    • complib (string, defaults to ‘zlib’) : compressor library. choose in zlib, lzo, bzip2, blosc

Examples:

from matorage import OptimizerConfig

optimizer_config = OptimizerConfig(
    endpoint='127.0.0.1:9000',
    access_key='minio',
    secret_key='miniosecretkey',
    optimizer_name='testoptimizer',
    additional={
        "version" : "1.0.1"
    }
)

optimizer_config.to_json_file('testoptimizer.json')
optimizer_config2 = OptimizerConfig.from_json_file('testoptimizer.json')
to_dict()[source]

Serializes this instance to a Python dictionary.

Returns

Dictionary of all the attributes that make up this configuration instance,

Return type

Dict[str, any]

Manager

class matorage.optimizer.Manager(config, num_worker_threads=4, multipart_upload_size=5242880)[source]
property get_metadata

Get all optimizers according to metadata by step.

Returns

optimizer of metadata

Return type

dict

Examples:

>>> optimizer_manager = OptimizerManager(config=optimizer_config)
>>> optimizer_manager.save(optimizer)
>>> optimizer_manager.get_metadata
{'938':
    {
        'framework': 'pytorch',
        'param_groups': [
            {
                'lr': 0.01, 'betas': [0.9, 0.999], 'eps': 1e-08,
                'weight_decay': 0, 'amsgrad': False,
                'params': [
                    140516594711520, 140516594711760,
                    140517867028384, 140516594711680,
                    140516594693376, 140516594612336
                ]
            }
        ]
    }
}

torch.OptimizerManager

class matorage.torch.OptimizerManager(config, num_worker_threads=4, multipart_upload_size=5242880)[source]

Optimizer Manager Pytorch classes. This class overrides Manager.

Note

Unlike Dataset, optimizer weight is loaded entirely into cpu memory. Therefore, the HDF5_CORE driver using the memory option is default setting.

Parameters
  • config (matorage.OptimizerConfig, require) – A OptimizerConfig instance object

  • num_worker_threads (int, optional, defaults to 4) – Number of backend storage worker to upload or download.

  • multipart_upload_size (integer, optional, defaults to 5 * 1024 * 1024) – size of the incompletely uploaded object. You can sync files faster with multipart upload in MinIO. This is because MinIO clients use multi-threading, which improves IO speed more efficiently regardless of Python’s Global Interpreter Lock(GIL).

Examples:

from matorage.torch import OptimizerManager
optimizer_config = OptimizerConfig(
    endpoint='127.0.0.1:9000',
    access_key='minio',
    secret_key='miniosecretkey',
    optimizer_name='testoptimizer',
    additional={
        "version" : "1.0.1"
    }
)

optimizer_manager = OptimizerManager(config=optimizer_config)
optimizer = optim.Adam(Model().parameters(), lr=0.01)
optimizer_manager.save(optimizer, step=100)
save(optimizer, scheduler=None)[source]

save weight of optimizer

Parameters

optimizer (torch.optim, require) – Pytorch optimizer.

Examples:

>>> model = Model()
>>> optimizer = optim.Adam(model.parameters(), lr=0.01)
# model training...
>>> optimizer_manager.save(optimizer)
load(optimizer, step)[source]

load weight of optimizer

Parameters
  • optimizer (torch.optim, require) – Pytorch optimizer.

  • step (integer, require) – optimizer step.

Examples:

>>> optimizer_manager = OptimizerManager(config=optimizer_config)
>>> optimizer = optim.Adam(model.parameters(), lr=0.01)
>>> optimizer_manager.load(optimizer, step=938)
load_with_scheduler(optimizer, scheduler, step)[source]

load weight of optimizer and scheduler

Parameters
  • optimizer (torch.optim, require) – Pytorch optimizer.

  • scheduler (torch.optim.lr_scheduler, require) – Pytorch scheduler.

  • step (integer, require) – optimizer step.

Examples:

>>> optimizer_manager = OptimizerManager(config=optimizer_config)
>>> optimizer = optim.Adam(model.parameters(), lr=0.01)
>>> scheduler = StepLR(optimizer, step_size=30, gamma=0.1)
>>> optimizer_manager.load_with_scheduler(optimizer, scheduler, step=938)

tensorflow.OptimizerManager

class matorage.tensorflow.OptimizerManager(config, num_worker_threads=4, multipart_upload_size=5242880)[source]

Optimizer Manager Tensorflow classes. This class overrides Manager.

Note

Unlike Dataset, optimizer weight is loaded entirely into cpu memory. Therefore, the HDF5_CORE driver using the memory option is default setting.

Parameters
  • config (matorage.OptimizerConfig, require) – A OptimizerConfig instance object

  • num_worker_threads (int, optional, defaults to 4) – Number of backend storage worker to upload or download.

  • multipart_upload_size (integer, optional, defaults to 5 * 1024 * 1024) –

    size of the incompletely uploaded object. You can sync files faster with multipart upload in MinIO. This is because MinIO clients use multi-threading, which improves IO speed more efficiently regardless of Python’s Global Interpreter Lock(GIL).

Examples:

from matorage.tensorflow import OptimizerManager
optimizer_config = OptimizerConfig(
    endpoint='127.0.0.1:9000',
    access_key='minio',
    secret_key='miniosecretkey',
    optimizer_name='testoptimizer',
    additional={
        "version" : "1.0.1"
    }
)

optimizer_manager = OptimizerManager(config=optimizer_config)
optimizer_manager.save(model.optimizer, step=100)
save(optimizer)[source]

save weight of optimizer

Parameters

optimizer (tf.keras.optimizers.Optimizer, require) – Tensorflow optimizer.

Examples:

>>> model = Model()
# model training...
>>> optimizer_manager.save(model.optimizer)
load(optimizer, step)[source]

load weight of optimizer

Parameters
  • optimizer (tf.keras.optimizers.Optimizer, require) – Tensorflow optimizer.

  • step (integer, require) – optimizer step.

Examples:

>>> model = Model()
>>> optimizer_manager.load(model.optimizer, step=938)