Exercise 4: Operationalize ML Scoring with Azure ML and Data Factory

Duration: 20 mins

Synopsis: In this exercise, attendees will extend the Data Factory service to operationalize the scoring of data using the previously created ML model.

This exercise has 5 tasks:

• Task 1: Create Azure ML Linked Service
• Task 2: Create Azure ML Input Dataset
• Task 3: Create Azure ML Scored Dataset
• Task 4: Create Azure ML Predictive Pipeline
• Task 5: Monitor Your Pipeline Activities

Get out of Jail Free

If, for whatever reason, you cannot complete this lab whether due to time contraints or if you are not able to troubleshoot an issue, we have created a “get out of jail free” exercise. If you wish to use this exercise at any time, please proceed to Appendix B. Please note that using this exercise will let you surpass all of the Azure Data Factory exercises. After completing Appendix B, you can continue to Exercise 5.

Task 1: Create Azure ML Linked Service

1. Go back to the Azure Data Factory service blade.

2. Click on the Author and deploy from Actions section.

   

3. Click on …More.

   

4. Click on New Compute.

   

5. Select Azure ML from the list.

6. In the new window, be sure change the JSON file to match the following:

   • Back in Exercise 1, Task 9 you noted some values related to your ML web service. The value for mlEndPoint below is your web service’s Batch Requests URL (remembering to remove the query string), and apiKey is the Primary Key of your web service.

     {
       "name": "AzureMLLinkedService",
       "properties": {
           "type": "AzureML",
           "description": "",
           "typeProperties": {
               "mlEndpoint": "<Specify the batch scoring URL>",
               "apiKey": "<Specify the published workspace model's API key>"
           }
       }
     }

7. Click Deploy.

Task 2: Create Azure ML Input Dataset

1. Click on …More.

   

2. To create new dataset that will be copied into Azure Blob storage, click on the New dataset from the top.

   

3. Select Azure Blob storage from the list.

4. In the new window, be sure change the JSON file to match the following or copy the below JSON text and paste into the browser window.

    {
        "name": "AzureBlobDataInPut",
        "properties": {
            "type": "AzureBlob",
            "external": true,
            "linkedServiceName": "OutputLinkedService-AzureBlobStorage",
            "typeProperties": {
                "fileName": "FlightsAndWeather.csv",
                "folderPath": "sparkcontainer",
                "format": {
                    "type": "TextFormat"
                }
            },
            "availability": {
                "frequency": "Hour",
                "interval": 12
            }
        }
    }

5. Click Deploy.

Task 3: Create Azure ML Scored Dataset

1. Click on …More.

   

2. Click on the New dataset.

   

3. Select Azure Blob storage from the list.

4. In the new window, be sure change the JSON file to match the following or copy the below JSON text and paste into the browser window.

    {
        "name": "AzureBlobScoredDataOutPut",
        "properties": {
            "type": "AzureBlob",
            "linkedServiceName": "OutputLinkedService-AzureBlobStorage",
            "typeProperties": {
                "fileName": "Scored_FlightsAndWeather.csv",
                "folderPath": "sparkcontainer",
                "format": {
                    "type": "TextFormat"
                }
            },
            "availability": {
                "frequency": "Hour",
                "interval": 12
            }
        }
    }

5. Click Deploy.

Task 4: Create Azure ML Predictive Pipeline

1. Click on …More.

   

2. Click on the New pipeline.

   

3. In the new window, be sure change the JSON file to match the following or copy the below JSON text and paste into the browser window.

    {
        "name": "PredictivePipeline",
        "properties": {
            "description": "Use AzureML model",
            "activities": [
                {
                    "type": "AzureMLBatchExecution",
                    "typeProperties": {
                        "webServiceInput": "AzureBlobDataInPut",
                        "webServiceOutputs": {
                            "output1": "AzureBlobScoredDataOutPut"
                        },
                        "globalParameters": {}
                    },
                    "inputs": [
                        {
                            "name": "AzureBlobDataInPut"
                        }
                    ],
                    "outputs": [
                        {
                            "name": "AzureBlobScoredDataOutPut"
                        }
                    ],
                    "policy": {
                        "timeout": "02:00:00",
                        "concurrency": 1,
                        "executionPriorityOrder": "NewestFirst",
                        "retry": 1
                    },
                    "name": "MLActivity",
                    "description": "prediction analysis on batch input",
                    "linkedServiceName": "AzureMLLinkedService"
                }
            ],
            "start": "2016-09-14T00:00:00Z",
            "end": "2016-09-15T00:00:00Z"
        }
    }

4. Make sure to change the start to today’s date and end to today + 1 date.
5. Click Deploy.

Task 5: Monitor Your Pipeline Activities

1. Close the current blade by clicking on the X from the top right corner of the blade.
2. Click on the Monitor & Manage from the Actions section.

3. Maximize the new window and you can see the diagram view of the data flow.

   

4. You should start to see Ready status activity listed on the bottom of the new window.
5. Close the Monitor & Manage browser tab.

Next Exercise: Exercise 5 - Summarize Data Using HDInsight Spark