Data warehouse import (beta): Azure, BigQuery

If you have data stored online in a data warehouse like Microsoft Azure Synapse or Google BigQuery, you can import this data easily to the Peltarion Platform and start training models on it.

Data warehouse import buttons

In the Datasets view, click on the BigQuery or Azure Synapse button, depending on the service you want to connect to. A small dialog will open to let you authenticate inside your warehouse, and select the table you want to use as a dataset.

  • We only use your authentication credentials to read the table you want to import, and we don’t store them after the import has finished.

Data warehouse import steps
Figure 1. There are 3 steps to importing data from your data warehouse: Authentication, Source, and Preview.

Azure Synapse import

Azure Synapse import lets you retrieve data stored in an Azure Synapse table, and turns it into a dataset inside your Peltarion Platform project.
This allows you to easily train models on large datasets available to you or your company.

Before you can use the Azure Synapse import, you will need to add the following IP addresses to the Azure Synapse firewall:

35.187.12.247
35.233.88.47
20.50.18.88
52.143.2.9

This will allow the Peltarion Platform to connect to your Azure Synapse instance.

To import data, click on the Azure Synapse Import button and follow these 3 steps:

  1. Authentication
    Type in the Server name and Database name that you want to connect to. The server name should be the full address folowed by a port number, e.g.,
    my-synapse-server.database.windows.net:1433
    You will also need to enter your personal Username and Password.

  2. Source
    Select the Table that you want to import from the list of tables available to your account.
    The platform creates one dataset from a single, entire table. You might need to create a new table in your Azure Synapse environment before importing it on the platform, especially:

    • If you need data spread across several tables

    • If an existing table has more data (columns or rows) than you need

  3. Preview
    The preview lets you check that the table contains the data you expect before importing it.

When everything looks good click on Create, and a new dataset will be created from the Azure Synapse table, ready to be used in your project’s experiments!

BigQuery import

BigQuery import lets you retrieve data stored in a Google BigQuery table, and turns it into a dataset inside your Peltarion Platform project.
This allows you to easily train models on large datasets available to you or your company.

To get started, click on the BigQuery Import button and follow these 3 steps:

  1. Authentication
    Select the Google Account you want to use, and click Allow when asked to allow peltarion.com to view your data in Google BigQuery.
    If you are not already signed in this account, Google will ask you to identify yourself.

    • If you have several Google Accounts, select the one which has access to the BigQuery table you want to use.

    • Peltarion does not store the access permission.
      This means that Peltarion loses all access to your BigQuery information and data as soon as you close the import dialog. This also means that you need to authenticate every time you open the import dialog.

  2. Source
    Google BigQuery charges a fee when you retrieve data from their storage. Select the Billing project that should be charged for the data transfer from BigQuery to the Peltarion Platform. The account you authenticate with needs to have the permission bigquery.jobs.create for the selected Billing project.
    Then find your table by selecting the Project ID, Dataset, and Table from the ones visible to your account.
    The platform creates one dataset from a single, entire BigQuery table. You might need to create a new table in your BigQuery project before importing it on the platform, especially:

    • If you need data spread across several BigQuery tables

    • If an existing table has more data (columns or rows) than you need

  3. Preview
    The preview lets you check that the table contains the data you expect before importing it (there is no transfer cost incurred from showing the preview).

When everything looks good click on Create, and a new dataset will be created from the BigQuery table, ready to be used in your project’s experiments!

Was this page helpful?
YesNo