Model

ModelConfig

class matorage.ModelConfig(**kwargs)[source]

Model configuration classes. This class overrides StorageConfig.

Parameters

  • endpoint (string, require) – S3 object storage endpoint. or If use NAS setting, NAS folder path.

  • access_key (string, optional, defaults to None) – Access key for the object storage endpoint. (Optional if you need anonymous access).

  • secret_key (string, optional, defaults to None) – Secret key for the object storage endpoint. (Optional if you need anonymous access).

  • secure (boolean, optional, defaults to False) – Set this value to True to enable secure (HTTPS) access. (Optional defaults to False unlike the original MinIO).

  • model_name (string, require) – model name.

  • additional (dict, optional, defaults to {}) – Parameters for additional description of models. The key and value of the dictionay can be specified very freely.

  • compressor (dict, optional, defaults to {"complevel" : 0, "complib" : "zlib"}) –

    Model compressor option. It consists of a dict type that has complevel and complib as keys. For further reference, read pytable’s Filter.

    • complevel (integer, defaults to 0) : compressor level(0~9). The larger the number, the more compressed it is.

    • complib (string, defaults to ‘zlib’) : compressor library. choose in zlib, lzo, bzip2, blosc

Examples:

from matorage import ModelConfig

model_config = ModelConfig(
    endpoint='127.0.0.1:9000',
    access_key='minio',
    secret_key='miniosecretkey',
    model_name='testmodel',
    additional={
        "version" : "1.0.1"
    }
)

model_config.to_json_file('testmodel.json')
model_config2 = ModelConfig.from_json_file('testmodel.json')
to_dict()[source]

Serializes this instance to a Python dictionary.

Returns

Dictionary of all the attributes that make up this configuration instance,

Return type

Dict[str, any]

Manager

class matorage.model.Manager(config, num_worker_threads=4, multipart_upload_size=5242880)[source]
property get_metadata

Get all models according to metadata(ex. step, epoch)

Returns

model of metadata

Return type

dict

Examples:

>>> model_manager.save(model, step=100)
>>> model_manager.save(model, step=200)
>>> model_manager.get_metadata
{
    'additional': {'version': '1.0.1'},
    'compressor': {'complevel': 0, 'complib': 'zlib'},
    'endpoint': '127.0.0.1:9000',
    'model':
    {
        'ad44168f1343bc77b4d9ad6f1fef50b6': {'step': 100},
        'af0677ecf0d15d17d10204be9ff2f9f5': {'step': 200}
    },
    'model_name': 'testmodel'
}

torch.ModelManager

class matorage.torch.ModelManager(config, num_worker_threads=4, multipart_upload_size=5242880)[source]

Model Manager Pytorch classes. This class overrides Manager.

Note

Unlike Dataset, model weight is loaded entirely into cpu memory. Therefore, the HDF5_CORE driver using the memory option is default setting.

Parameters

  • config (matorage.ModelConfig, require) – A ModelConfig instance object

  • num_worker_threads (int, optional, defaults to 4) – Number of backend storage worker to upload or download.

  • multipart_upload_size (integer, optional, defaults to 5 * 1024 * 1024) – size of the incompletely uploaded object. You can sync files faster with multipart upload in MinIO. This is because MinIO clients use multi-threading, which improves IO speed more efficiently regardless of Python’s Global Interpreter Lock(GIL).

Examples:

from matorage import ModelConfig
from matorage.torch import ModelManager

model_config = ModelConfig(
    endpoint='127.0.0.1:9000',
    access_key='minio',
    secret_key='miniosecretkey',
    model_name='testmodel',
    additional={
        "version" : "1.0.1"
    }
)

model_manager = ModelManager(config=model_config)

import torch.nn as nn
class Model(nn.Module):
    def __init__(self):
        super(Model, self).__init__()
        self.net1 = nn.Sequential(
            nn.Linear(5, 10),
            nn.ReLU(),
        )
        self.net2 = nn.Sequential(
            nn.Linear(10, 5),
            nn.ReLU(),
        )
    def forward(self, x):
        x = self.net1(x)
        x = self.net2(x)
        return x

model = Model()
model_manager.save(model, step=100)
save(model, **kwargs)[source]

save weight of model

Parameters

  • model (torch.nn.Module, require) – Pytorch model (torch.nn.Module type)

  • kwargs (**kwargs, optional) – metadata about step or epoch for model.

Examples:

>>> model_manager.save(model, step=0)
>>> model_manager.save(model, epoch=0)
>>> model_manager.save(model, epoch=0, step=0)
load(model, **kwargs)[source]

load weight of model

Parameters

  • model (torch.nn.Module or string, require) – Pytorch model(torch.nn.Module type) or layer name(string type).

  • kwargs (**kwargs, optional) – metadata about step or epoch for model.

Returns

If model is pytorch model, weight is loaded into the model and return None. however, If it is a string type with the name of the layer, it returns the weight of the OrderedDict type.

Return type

None or OrderedDict

Examples:

# Load entire model
>>> model = Model()
>>> model_manager.load(model, step=0)
>>> model
Model(
  (net1): Sequential(
    (0): Linear(in_features=5, out_features=10, bias=True)
    (1): ReLU()
  )
  (net2): Sequential(
    (0): Linear(in_features=10, out_features=5, bias=True)
    (1): ReLU()
  )
)

# Load sub-layer model
>>> class SubModel(nn.Module):
>>>   def __init__(self):
>>>     super(SubModel, self).__init__()
>>>     self.net1 = nn.Sequential(
>>>         nn.Linear(5, 10),
>>>         nn.ReLU(),
>>>     )

>>> submodel = SubModel()
>>> model_manager.load(submodel, step=0)
>>> model
Model(
  (net1): Sequential(
    (0): Linear(in_features=5, out_features=10, bias=True)
    (1): ReLU()
  )
)

# Load from layer name
>>> model_manager.load('net1.0.weight', step=0)
OrderedDict([('net1.0.weight',
tensor([[ 0.2679, -0.2147, -0.1927, -0.3263,  0.0930],
    [ 0.0144,  0.2935,  0.3614, -0.0493, -0.3772],
    [ 0.4101, -0.1864,  0.1076, -0.3900,  0.3613],
    [-0.2831,  0.3692,  0.3367,  0.2491, -0.2971],
    [-0.3019,  0.1682, -0.3951,  0.1528,  0.1778],
    [-0.1593,  0.3315, -0.2286,  0.1294,  0.2087],
    [-0.3394, -0.2706,  0.1515,  0.0357, -0.4252],
    [ 0.2555, -0.4435, -0.3353,  0.2096, -0.3741],
    [ 0.3950, -0.2630, -0.1730,  0.1393,  0.3678],
    [ 0.3065, -0.0095,  0.0988,  0.4294,  0.3338]]))])

tensorflow.ModelManager

class matorage.tensorflow.ModelManager(config, num_worker_threads=4, multipart_upload_size=5242880)[source]

Model Manager Tensorflow classes. This class overrides Manager.

Note

Unlike Dataset, model weight is loaded entirely into cpu memory. Therefore, the HDF5_CORE driver using the memory option is default setting.

Parameters

  • config (matorage.ModelConfig, require) – A ModelConfig instance object

  • num_worker_threads (int, optional, defaults to 4) – Number of backend storage worker to upload or download.

  • multipart_upload_size (integer, optional, defaults to 5 * 1024 * 1024) –

    size of the incompletely uploaded object. You can sync files faster with multipart upload in MinIO. This is because MinIO clients use multi-threading, which improves IO speed more efficiently regardless of Python’s Global Interpreter Lock(GIL).

Examples:

from matorage import ModelConfig
from matorage.tensorflow import ModelManager
from tensorflow.keras import layers, Sequential

model_config = ModelConfig(
    endpoint='127.0.0.1:9000',
    access_key='minio',
    secret_key='miniosecretkey',
    model_name='testmodel',
    additional={
        "version" : "1.0.1"
    }
)

model_manager = ModelManager(config=model_config)

model = Sequential([
    Sequential([
        layers.Dense(10),
        layers.ReLU()
    ]),
    Sequential([
        layers.Dense(10),
        layers.ReLU()
    ])
])
model.build(input_shape=(None, 5))

model_manager.save(model, step=100)
save(model, **kwargs)[source]

save weight of model

Parameters

  • model (tf.keras.Model, require) – Tensorflow model (tf.keras.Model type)

  • kwargs (**kwargs, optional) – metadata about step or epoch for model.

Examples:

>>> model_manager.save(model, step=0)
>>> model_manager.save(model, epoch=0)
>>> model_manager.save(model, epoch=0, step=0)
load(model, **kwargs)[source]

load weight of model

Parameters

  • model (tf.keras.Model or string, require) – Tensorflow model(tf.keras.Model type) or layer name(string type).

  • kwargs (**kwargs, optional) – metadata about step or epoch for model.

Returns

If model is tensorflow model, weight is loaded into the model and return None. however, If it is a string type with the name of the layer, it returns the weight of the OrderedDict type.

Return type

None or OrderedDict

Examples:

# Load entire model
>>> model_manager.load(model, step=0)

# Load sub-layer model
>>> submodel = Sequential([
>>>   Sequential([
>>>     layers.Dense(10),
>>>     layers.ReLU()
>>>   ])
>>> ])
>>> submodel.build(input_shape=(None, 5))
>>> model_manager.load(submodel, step=0)

>>> model_manager.load('sequential/dense/kernel:0', step=0)
OrderedDict([('sequential/dense/kernel:0',
<tf.Tensor: shape=(5, 784), dtype=float32, numpy=
array([[ 0.08160326,  0.00161414, -0.00507049, ...,  0.02965256,
        -0.07447692,  0.02029154],
       [-0.06808375,  0.0112161 , -0.0640984 , ..., -0.05060118,
        -0.03650254,  0.01808494],
       [ 0.00063588,  0.00848304, -0.01014224, ...,  0.0616277 ,
        -0.05507688,  0.02844934],
       [-0.00206905,  0.04553737,  0.03098481, ..., -0.05891491,
         0.0705805 , -0.03912991],
       [ 0.04252511, -0.04907732, -0.07053198, ...,  0.00260394,
         0.07418892, -0.0714546 ]], dtype=float32)>)])