Test it on the Peltarion Platform

A platform to build and deploy deep learning projects.

Even if you’re not an AI superstar.

TRY FOR FREE

Data API

The Data API lets you upload data files into your projects. It can replace manually selecting and uploading files on the platform. You can use it to send data to the platform right from your preprocessing code.

Prerequisites

To use the Data API, you need to have:

  • An active token for authentication, referred to as <token>

  • The URL where to send your requests, referred to as <URL>

You manage these values from the Data API dialogue in the datasets view of the project where you want to use the API.

You should also have one or more files compatible with the Platform dataset requirements that you want to upload available on your local drive or online.

How to upload files

There are 4 steps to uploading files with the Data API.
Each step consists of sending a HTTP request to a specific endpoint URL. You get a JSON formatted response at each step indicating if the step succeeded. and sometimes information that you need for the next step.

  1. Create a new Dataset

    • Endpoint: POST <URL>

    • Headers: Authorization: Bearer <token>, Content-Type: application/json

    • Body: A JSON object with name and description properties

    • Response: A JSON object with a datasetWrapperId property, giving the newly created dataset’s id. Save the dataset id since you need to use it in the next step
      Example response:

      {"datasetWrapperId": 123abc456def}

    The new dataset is locked in uploading state until you finalize the upload.

  2. Upload your file(s)

    • Endpoint: POST <URL>/<datasetWrapperId>/upload

    • Headers: Authorization: Bearer <token>, Content-Type: <media type>
      Replace <media type> with one of the following:

      • text/csv when uploading a csv file

      • application/zip when uploading a zip file

      • application/npy when uploading a npy array

    • Body: The raw file data

    • Response: A JSON object with a uploadId property, giving an id for the uploaded file
      Example response:

      {"uploadId": aaa111bbb222}

      You can use this id to check the status of this upload later on.

  3. Get the status of uploads

    • Endpoint: GET <URL>/<datasetWrapperID>/uploads

    • Header: Authorization: Bearer <token>

    • Body: No payload is required

    • Response: A JSON array uploadStatuses, where each element contains uploadId and status, as well as message if the particular array element relates to a failed upload.

      The status can have one of these values:

      • PROCESSING if the upload is being processed

      • SUCCESS if the upload succeeded and was added to your dataset

      • FAILED if a problem happened

      Example response:

      {"uploadStatuses": [{"uploadId": aaa111bbb222, 'status': 'SUCCESS'}, {"uploadId": abcde1234, 'status': 'FAILED', 'message': 'Invalid file format'}]}

    Uploads can be processing even after the transfer is complete. Check that none of your uploads are still PROCESSING before finalizing the upload, or these uploads could go missing from your dataset.

  4. Finalize the upload

    • Endpoint: POST <URL>/<datasetWrapperID>/upload_complete

    • Header: Authorization: Bearer <token>

    • Body: No payload is required

    • Response: There is no JSON response, but the request will return code 200 if successful

Examples

Python

This example shows how to create a dataset and upload a zip file to it in Python, using the requests package.

import requests

# Information from the datasets view
URL = '<URL>'
token = '<token>'

# 1. Create a new dataset
headers = {'Authorization': 'Bearer {}'.format(token), 'Content-Type': 'application/json'}
payload = {'name':'New dataset', 'description':'This dataset was created from the Data API'}

response = requests.request('POST', URL, headers=headers, json=payload)

if response.status_code == 200:           # Code 200 is success
    datasetID = response.json()['datasetWrapperId']

# 2. Upload files
filepath = '/path/to/my/file.zip'
headers = {'Authorization': 'Bearer {}'.format(token), 'Content-Type': 'application/zip'}

with open(filepath, 'rb') as file_data:
    response = requests.request("POST", URL+'/'+str(datasetID)+'/upload', headers=headers, data=file_data.read())

if response.status_code != 200:           # Code 200 is upload success
    print('Upload failed')
else:
    UploadID = response.json()['uploadId']

# 3. Wait until all uploads have been processed
response = requests.request("GET", URL+'/'+str(datasetID)+'/uploads', headers=headers)
statuses = [obj['status'] for obj in response.json()['uploadStatuses']]

while 'PROCESSING' in statuses:    # Wait for all uploads to finish
    response = requests.request("GET", URL+'/'+str(datasetID)+'/uploads', headers=headers)
    statuses = [obj['status'] for obj in response.json()['uploadStatuses']]

for obj in response.json()['uploadStatuses']:
    if obj['status'] == 'FAILED':
        print('Upload {} failed with message: {}'.format(obj['uploadId'], obj['message']))

# 4. Finalize upload
headers = {'Authorization': 'Bearer {}'.format(token)}
response = requests.request("POST", URL+'/'+str(datasetID)+'/upload_complete', headers=headers, data=None)

cURL

curl -X POST -H "Authorization: Bearer <token>" -H "Content-Type: application/json" -d '{"name": "New dataset", "description": "Data uploaded from cURL"}' <URL>

curl -X POST -H "Authorization: Bearer <token>" -H "Content-Type: application/zip" --data-binary @/path/to/file.zip <URL>/<DataID>/upload

curl -X GET -H "Authorization: Bearer <token>" <URL>/<DataID>/uploads

curl -X POST -H "Authorization: Bearer <token>" <URL>/<DataID>/upload_complete