Optimizer¶
OptimizerConfig¶
-
class
matorage.
OptimizerConfig
(**kwargs)[source]¶ Optimizer configuration classes. This class overrides
StorageConfig
.- Parameters
endpoint (
string
, require) – S3 object storage endpoint. or If use NAS setting, NAS folder path.access_key (
string
, optional, defaults to None) – Access key for the object storage endpoint. (Optional if you need anonymous access).secret_key (
string
, optional, defaults to None) – Secret key for the object storage endpoint. (Optional if you need anonymous access).secure (
boolean
, optional, defaults to False) – Set this value to True to enable secure (HTTPS) access. (Optional defaults to False unlike the original MinIO).optimizer_name (
string
, require) – optimizer name.additional (
dict
, optional, defaults to{}
) – Parameters for additional description of optimizers. The key and value of the dictionay can be specified very freely.compressor (
dict
, optional, defaults to{"complevel" : 0, "complib" : "zlib"}
) –Optimizer compressor option. It consists of a dict type that has complevel and complib as keys. For further reference, read pytable’s Filter.
complevel (
integer
, defaults to 0) : compressor level(0~9). The larger the number, the more compressed it is.complib (
string
, defaults to ‘zlib’) : compressor library. choose in zlib, lzo, bzip2, blosc
Examples:
from matorage import OptimizerConfig optimizer_config = OptimizerConfig( endpoint='127.0.0.1:9000', access_key='minio', secret_key='miniosecretkey', optimizer_name='testoptimizer', additional={ "version" : "1.0.1" } ) optimizer_config.to_json_file('testoptimizer.json') optimizer_config2 = OptimizerConfig.from_json_file('testoptimizer.json')
Manager¶
-
class
matorage.optimizer.
Manager
(config, num_worker_threads=4, multipart_upload_size=5242880)[source]¶ -
property
get_metadata
¶ Get all optimizers according to metadata by step.
- Returns
optimizer of metadata
- Return type
dict
Examples:
>>> optimizer_manager = OptimizerManager(config=optimizer_config) >>> optimizer_manager.save(optimizer) >>> optimizer_manager.get_metadata {'938': { 'framework': 'pytorch', 'param_groups': [ { 'lr': 0.01, 'betas': [0.9, 0.999], 'eps': 1e-08, 'weight_decay': 0, 'amsgrad': False, 'params': [ 140516594711520, 140516594711760, 140517867028384, 140516594711680, 140516594693376, 140516594612336 ] } ] } }
-
property
torch.OptimizerManager¶
-
class
matorage.torch.
OptimizerManager
(config, num_worker_threads=4, multipart_upload_size=5242880)[source]¶ Optimizer Manager Pytorch classes. This class overrides
Manager
.Note
Unlike Dataset, optimizer weight is loaded entirely into cpu memory. Therefore, the HDF5_CORE driver using the memory option is default setting.
- Parameters
config (
matorage.OptimizerConfig
, require) – A OptimizerConfig instance objectnum_worker_threads (
int
, optional, defaults to 4) – Number of backend storage worker to upload or download.multipart_upload_size (
integer
, optional, defaults to 5 * 1024 * 1024) – size of the incompletely uploaded object. You can sync files faster with multipart upload in MinIO. This is because MinIO clients use multi-threading, which improves IO speed more efficiently regardless of Python’s Global Interpreter Lock(GIL).
Examples:
from matorage.torch import OptimizerManager optimizer_config = OptimizerConfig( endpoint='127.0.0.1:9000', access_key='minio', secret_key='miniosecretkey', optimizer_name='testoptimizer', additional={ "version" : "1.0.1" } ) optimizer_manager = OptimizerManager(config=optimizer_config) optimizer = optim.Adam(Model().parameters(), lr=0.01) optimizer_manager.save(optimizer, step=100)
-
save
(optimizer, scheduler=None)[source]¶ save weight of optimizer
- Parameters
optimizer (
torch.optim
, require) – Pytorch optimizer.
Examples:
>>> model = Model() >>> optimizer = optim.Adam(model.parameters(), lr=0.01) # model training... >>> optimizer_manager.save(optimizer)
-
load
(optimizer, step)[source]¶ load weight of optimizer
- Parameters
optimizer (
torch.optim
, require) – Pytorch optimizer.step (
integer
, require) – optimizer step.
Examples:
>>> optimizer_manager = OptimizerManager(config=optimizer_config) >>> optimizer = optim.Adam(model.parameters(), lr=0.01) >>> optimizer_manager.load(optimizer, step=938)
-
load_with_scheduler
(optimizer, scheduler, step)[source]¶ load weight of optimizer and scheduler
- Parameters
optimizer (
torch.optim
, require) – Pytorch optimizer.scheduler (
torch.optim.lr_scheduler
, require) – Pytorch scheduler.step (
integer
, require) – optimizer step.
Examples:
>>> optimizer_manager = OptimizerManager(config=optimizer_config) >>> optimizer = optim.Adam(model.parameters(), lr=0.01) >>> scheduler = StepLR(optimizer, step_size=30, gamma=0.1) >>> optimizer_manager.load_with_scheduler(optimizer, scheduler, step=938)
tensorflow.OptimizerManager¶
-
class
matorage.tensorflow.
OptimizerManager
(config, num_worker_threads=4, multipart_upload_size=5242880)[source]¶ Optimizer Manager Tensorflow classes. This class overrides
Manager
.Note
Unlike Dataset, optimizer weight is loaded entirely into cpu memory. Therefore, the HDF5_CORE driver using the memory option is default setting.
- Parameters
config (
matorage.OptimizerConfig
, require) – A OptimizerConfig instance objectnum_worker_threads (
int
, optional, defaults to 4) – Number of backend storage worker to upload or download.multipart_upload_size (
integer
, optional, defaults to 5 * 1024 * 1024) –size of the incompletely uploaded object. You can sync files faster with multipart upload in MinIO. This is because MinIO clients use multi-threading, which improves IO speed more efficiently regardless of Python’s Global Interpreter Lock(GIL).
Examples:
from matorage.tensorflow import OptimizerManager optimizer_config = OptimizerConfig( endpoint='127.0.0.1:9000', access_key='minio', secret_key='miniosecretkey', optimizer_name='testoptimizer', additional={ "version" : "1.0.1" } ) optimizer_manager = OptimizerManager(config=optimizer_config) optimizer_manager.save(model.optimizer, step=100)