> For the complete documentation index, see [llms.txt](https://campus-rover.gitbook.io/lab-notebook/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://campus-rover.gitbook.io/lab-notebook/fiiva/pretrained-model-deployment.md). # How do I deploy a Pytorch model our cluster? ## Deploying a Pretrained Pytorch Model in Ubuntu 18.04 Virtual Environment By Adam Ring Using a pre-trained deep learning model from a framework such as Pytorch has myriad applications in robotics, from computer vision to speech recognition, and many places inbetween. Sometimes you have a model that you want to train on another system with more powerful hardware, and then deploy the model elsewhere on a less powerful system. For this task, it is extremely useful to be able to transfer the weights of your trained model into another system, such as a virtual machine running Ubuntu 18.04. These methods for model transfer will also run on any machine with pytorch installed. ### Note It is extremely discouraged to mix versions of Pytorch between training and deployment. If you train your model on Pytorch 1.8.9, and then try to load it using Pytorch 1.4.0, you may encounter some errors due to differences in the modules between versions. For this reason it is encouraged that you load your Pytorch model using the same version that is was trained on. ### Saving and loading a trained model Let's assume that you have your model fully trained and loaded with all of the necessary weights. `model = MyModel()` `model.train()` For instructions on how to train a machine learning model, see [this section on training a model](https://campus-rover.gitbook.io/lab-notebook/advanced-topics/computer-vision#training) in the lab notebook. There are multiple ways to save this model, and I will be covering just a few in this tutorial. #### Saving the `state_dict` This is reccommended as the best way to save the weights of your model as its `state_dict`, however it does require some dependencies to work. Once you have your model, you must specify a `PATH` to the directory in which you want to save your model. This is where you can name the file used to store your model. `PATH = "path/to/directory/my_model_state_dict.pt"` or `PATH = "path/to/directory/my_model_state_dict.pth"` You can either specify that the `state_dict` be saved using `.pt` or `.pth` format. Then, to save the model to a path, simply call this line of code. `torch.save(model.state_dict(), PATH)` #### Loading the `state_dict` Download the `my_model_state_dict.pt/pth` into the environment in which you plan on deploying it. Note the path that the state dict is placed in. In order to load the model weights from the `state_dict` file, you must first initialize an untrained istance of your model. `loaded_model = MyModel()` Keep in mind that this step requires you to have your model architecture defined in the environment in which you are deploying your model. Next, you can simply load your model weights from the state dict using this line of code. `loaded_model.load_state_dict(torch.load("path/to/state/dict/my_model_state_dict.pt/pth"))` The trained weights of the model are now loaded into the untrained model, and you are ready to use the model as if it is pre-trained. ### Saving and loading the model using TorchScript TorchScript is a framework built into Pytorch which is used for model deployment in many different types of environments without having the model defined in the deployment environment. The effect of this is that you can save a model using **tracing** and load it from a file generated by tracing it. What tracing does is follow the operations done on an input tensor that is run through your model. Note that if your model has conditionals such as `if` statements or external dependencies, then the tracing will not record these. Your model must only work on tensors as well. #### Saving the trace of a model In order to trace your trained model and save the trace to a file, you may run the following lines of code. `PATH = "path/to/traced/model/traced_model.pt/pth"` `dummy_input = torch.ones(typical_input_size, dtype=dype_of_typical_input)` `traced_model = torch.jit.trace(model, dummy_input)` `torch.jit.save(traced_model, PATH)` The `dummy_input` can simply be a bare tensor that is the same size as a typical input for your model. You may also use one of the training or test inputs. The content of the dummy input does not matter, as long as it is the correct size. #### Loading the trace of a model In order to load the trace of a model, you must download the traced model `.pt` or `.pth` file into your deployment environment and note the path to it. All you need to do to load a traced model for deployment in Pytorch is use the following line of code. `loaded_model = torch.jit.load("path/to/traced/model/traced_model.pt/pth")` Keep in mind that the traced version of your model will only work for torch tensors, and will not mimic the behavior of any conditional statements that you may have in your model. ## Data Annotation Please see the full tutorial in the repo: --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter: ``` GET https://campus-rover.gitbook.io/lab-notebook/fiiva/pretrained-model-deployment.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.