Peltarion Prediction Server

Docker is the most wanted and second most loved developer tool, and helps millions of developers build, share and run any app, anywhere - on-prem or in the cloud.
You can use it to wrap the models you’ve trained on the platform and deploy them easily where you want.

Prerequisites

  • Build and train a model. Check the tutorials to find inspiration and to get started quickly.

  • Download and install Docker to create and deploy the image containing your model.

Containerized model overview

These are the steps to take to run your model as a container:

  1. Download your model, making sure to select Model with Docker container definition.

  2. Extract the downloaded zip which contains the SavedModel file, as well as extra files that define how your model can be deployed.

  3. Use docker build to create an image containing your trained model.

  4. Use docker run to deploy the base image with the downloaded model as parameter.

Create and use a container with your model

File download

When you download your model, make sure you check the Model with Docker container definition option to get all the necessary files.

Extract the archive so that you have all the files available in a folder.

Generate the Docker image

Within the extracted folder, run this command to generate the Docker image:

docker build -t model-export -f Dockerfile .

Run the image

Run the container with the model’s image:

docker run -p 8000:8000 model-export

Get predictions from the deployed model

The API used by the containerized model differs slightly from the Deployment API used on the Peltarion Platform.

The proxy server expects data in the following format:

{
    "instances": [
        {
            "<feature 1 label>": <feature 1 data>,
            "<feature 2 label>": <feature 2 data>,
            ...
        },
        ...
    ]
}

That is, instances is a list of records, and each record is a mapping from feature labels to the feature data.

In addition, there are two endpoints, regress and predict, depending on the model task.

You can use them e.g. in cURL like this:

curl -X POST http://localhost:8000/regress -H "Content-Type: application/json" -d '{"x": 1}'

curl -X POST http://localhost:8000/predict -H "Content-Type: application/json" -d '{"x": 1}'

Sending images

Images need to be base64 encoded. This can be done e.g. with a snippet of code as this one:

import base64

with open("<path to image>", "rb") as f:
    image_bytes = f.read()

data = {
    "instances": [
        {
            "Image": base64.encodebytes(image_bytes).decode("utf-8")
        }
    ]
}

Response format

The response is forwarded directly from TF Serving REST API without post-processing.

The response from the predict endpoint is:

{
  "predictions": <value>|<(nested)list>|<list-of-objects>
}

The response from the classify endpoints is:

{
  "result": [
    // List of class label/score pairs for first Example (in request)
    [ [<label1>, <score1>], [<label2>, <score2>], ... ],

    // List of class label/score pairs for next Example (in request)
    [ [<label1>, <score1>], [<label2>, <score2>], ... ],
    ...
  ]
}

The response from the regress endpoint is:

{
  // One regression value for each example in the request in the same order.
  "result": [ <value1>, <value2>, <value3>, ...]
}
Was this page helpful?
YesNo