How to Copy Data in Azure VM?

Azure Managed Disks

Azure Managed Disks are managed by Azure from backend created in an Azure Storage Account. You won't be having visibility of Azure Storage Account as it is completely taken care by Microsoft Azure at backend.

Microsoft Azure will completely take care of entire encryption, data recovery plans etc. That means Azure is having the fill control over data.

Note: You cannot create both managed and unmanaged disks on a Virtual Machine.

Azure Managed Disk are of two type: OS Disk and Data Disk.

Copy an Azure Managed OS Disk using PowerShell

Here copy is noting but a sample exact clone of an existing azure managed OS disk.

Note: Temp data will not be get copied while copy the data.

Use this approach to copy an Azure managed OS disk to a newly created Azure Disk.

Step 1: Create a PowerShell file with .ps1 extension. Let's say copy-os-disk.ps1

command: touch copy-os-disk.ps1

Step 2: open a file to code using code command.

command: code copy-os-disk.ps1

Step 3: Copy and paste the below variable to copy-os-disk.ps1 and fill the parameters with required details.

$SubscriptionName = "_add_subscription_name"
Set-AzureRmContext -SubscriptionName "$SubscriptionName" | Out-Null
## Fill up the variable values.
$sourceRG = "_add_source_rg_name_"
$sourceDiskName = "_add_source_datadisk_name_"
$targetDiskName = "_add_target_datadisk_name_"
$targetRG = "_add_target_rg_name_"
$targetLocate = "_add_target_location_"
$targetOS = "add_system_ostype_" #"Windows" or "Linux"
$SKUName = '_add_disk_sku_type_' #Premium_LRS or Standard_LRS
$HyperVGeneration = "_add_vm_gen_" #either V1 or V2

Step 4: Now, copy and paste the entire script below as it is and append it to the copy-os-disk.ps1 powershell file.

Note: Don't make any changes in the below script.

## PowerShell Script to Copy Azure VM Managed OS Disk

$sourceDisk = Get-AzDisk `
-ResourceGroupName $sourceRG `
-DiskName $sourceDiskName

$targetDiskconfig = New-AzDiskConfig `
-SkuName $SKUName `
-osType $targetOS `
-UploadSizeInBytes $($sourceDisk.DiskSizeBytes+512) `
-Location $targetLocate `
-CreateOption 'Upload' `
-HyperVGeneration $HyperVGeneration

$targetDisk = New-AzDisk `
-ResourceGroupName $targetRG `
-DiskName $targetDiskName `
-Disk $targetDiskconfig

$sourceDiskSas = Grant-AzDiskAccess `
-ResourceGroupName $sourceRG `
-DiskName $sourceDiskName `
-DurationInSecond 86400 `
-Access 'Read'

$targetDiskSas = Grant-AzDiskAccess `
-ResourceGroupName $targetRG `
-DiskName $targetDiskName `
-DurationInSecond 86400 `
-Access 'Write'

azcopy copy $sourceDiskSas.AccessSAS $targetDiskSas.AccessSAS `
--blob-type PageBlob

Revoke-AzDiskAccess `
-ResourceGroupName $sourceRG `
-DiskName $sourceDiskName

Revoke-AzDiskAccess `
-ResourceGroupName $targetRG `
-DiskName $targetDiskName

Step 5: Now run the copy-os-disk.ps1 PowerShell file to start copying.

command: ./copy-os-disk.ps1

Azure PowerShell Master Script to Copy Azure VM Managed OS Disk:

<## --- Switch to Azure Subscription --- ##>
$SubscriptionName = "_add_subscription_name"
Set-AzureRmContext -SubscriptionName "$SubscriptionName" | Out-Null

<## --- Assign Variable Values --- ##>
$sourceRG = "_add_source_rg_name_"
$sourceDiskName = "_add_source_datadisk_name_"
$targetDiskName = "_add_target_datadisk_name_"
$targetRG = "_add_target_rg_name_"
$targetLocate = "_add_target_location_"
$targetOS = "add_system_ostype_" #"Windows" or "Linux"
$SKUName = '_add_disk_sku_type_' #Premium_LRS or Standard_LRS
$HyperVGeneration = "_add_vm_gen_" #either V1 or V2

<## --- Exexution Starts from here --- ##>
$sourceDisk = Get-AzDisk -ResourceGroupName $sourceRG -DiskName $sourceDiskName

$targetDiskconfig = New-AzDiskConfig -SkuName $SKUName -osType $targetOS -UploadSizeInBytes $($sourceDisk.DiskSizeBytes+512) -Location $targetLocate -CreateOption 'Upload' -HyperVGeneration $HyperVGeneration

$targetDisk = New-AzDisk -ResourceGroupName $targetRG -DiskName $targetDiskName -Disk $targetDiskconfig

$sourceDiskSas = Grant-AzDiskAccess -ResourceGroupName $sourceRG -DiskName $sourceDiskName -DurationInSecond 86400 -Access 'Read'
$targetDiskSas = Grant-AzDiskAccess -ResourceGroupName $targetRG -DiskName $targetDiskName -DurationInSecond 86400 -Access 'Write'

azcopy copy $sourceDiskSas.AccessSAS $targetDiskSas.AccessSAS --blob-type PageBlob

Revoke-AzDiskAccess -ResourceGroupName $sourceRG -DiskName $sourceDiskName
Revoke-AzDiskAccess -ResourceGroupName $targetRG -DiskName $targetDiskName

Use Case Scenario: Azure VM Migration Copy Azure Managed Disk (Analysis)

Migrate Azure VM (either OS Disk or Data Disk) which has a Disk Encryption with Customer Managed Key(CMK) from an old azure subscription to new azure subscription.

Problem Issue

Managed Azure OS Disk or Azure Data Disk can't be moved from old subscription to new subscription due to server-side encryption (SSE) with a customer-managed key (CMK). There is no disable option to change encryption.

Note: SSE with CMK is server-side encryption with a customer-managed key.

Solutions that Didn't Work!

1. We have taken the snapshot of CMK encrypted managed Disk and tried moving to a new azure subscription from the old azure subscription. It didn't work!

2. Is there an option to disable CMK Encryption?

No, there is no such option to disable CMK Encryption once enabled.

3. Created a new Disk from snapshot and tried moving to other subscriptions. It didn't work!

Problem Solution

Copy Azure VM source managed Disk (either OS Disk or Data Disk) to new Azure managed Disk to resolve the SSE with CMK is server-side encryption with a customer-managed key.

Effective Azure Solution

Use Azure PowerShell Scripts of Copy Azure OS Disk and Copy Azure Data Disk to move the data from source location to destination target.

Limitations or Exceptions while implementing the solution

1. Single User Cannot Run Multiple Copies of managed disk in Parallel due to SAS Conflict.

2. Users cannot copy the data in case of backend movement from one subscription to another azure subscription due to SAS issues.

3. The AzCopy module should be installed if we are using the Windows PowerShell locally. Azure Cloud Shell has pre-installed AzCopy modules, so you can directly run your cmdlets without installing.

Time Analysis: Copy Azure Managed Disk

Execution process will take time depending on the size of the managed disk.

For 50 GB of data, it will be copied in 15-25 minutes for SKU Type: Premium LRS (SSD)

For 50 GB of data, it will be copied in 30-45 minutes for SKU Type: Standard LRS (SSD)

If Standard LRS (HDD), it might even take more time to copy the data.

Copy Disk Estimated Time Analysis Samples

AzServer01 - OS Disk (128 GB Premium LRS) - 7:30 pm - 7: 54 pm

Copy Time: 24 min
Movement (Sub - Sub) : 10 min
Total Time: 34 min

AzServer01 - Data Disk 01 (2048 GB Premium LRS) - 7:55 pm - 9:25 pm

Copy Time: 90 min
Movement (Sub - Sub) : 9 min
Total Time: 99 min

AzServer01 - Data Disk 02 (2048 GB Premium LRS) - 9:08 pm - 11:09 pm

Copy Time: 121 min
Movement (Sub - Sub) : 9 min
Total Time: 130 min

AzServer02 - OS Disk (128 GB Premium LRS) - 9:41 pm - 10:00 pm

Copy Time: 19 min
Movement (Sub - Sub) : 9 min
Total Time: 28 min

AzServer03 - OS Disk (128 GB Premium LRS) - 10:02 pm - 10:25 pm

Copy Time: 23 min
Movement (Sub - Sub) : 9 min
Total Time: 32 min

AzServer03 - Data Disk 01 (64 GB Premium LRS) - 10:32 pm - 10:45 pm

Copy Time: 13 min
Movement (Sub - Sub) : 9 min
Total Time: 22 min

Copy Data from Azure to On-Premise | On-Premise to Azure

Business Requirements:

Copying 10GB of data to Azure from an On-Premise. Need to copy data, files, folders from On-premise Server to Azure Cloud Server or Storage Account Blob/File.
low latency to transfer data from source to destination server.
faster file transfer from on-premise server to azure cloud.

Solution: AzCopy Tool

AzCopy is a command-line utility that helps to copy blobs or files to or from a storage account. Will be using this azcopy tool to transfer data from on-premise server to azure cloud.

AzCopy utility supports data copy between a file system and a storage account, or between storage accounts.
This AzCopy utility tool supports Windows, Linux, and Mac.
If you want to use Cloud Shell then you don't need install azcopy tool as it a preinstalled in Azure Cloud Shell, so you can use it from cloud shell itself to run the azcopy commands.

Using AzCopy, We can upload the files from the source to the destination. It supports the following directions:

local machine<-> Azure Blob (SAS or OAuth authentication)
local machine <-> Azure Files (Share/directory SAS authentication)
Azure Blob (SAS or public) -> Azure Blob (SAS or OAuth authentication)
Azure Blob (SAS or public) -> Azure Files (SAS)
Azure Files (SAS) -> Azure Files (SAS)

Copy Data from On-Premises to Azure Storage Blob with SAS Token

Upload/Copy/Move/Transfer a file/folder from on-premise to azure cloud storage account blob using storage account blob container SAS token.

Basic syntax for azcopy commands: azcopy copy [source] [destination] [flags]

This is how the azcopy syntax looks like: azcopy copy "<Source File>" "<storage_account_name>.<blob>.core.windows.net/<containername>?<SAS token>" --recursive=true

Here, the source file is your on-premise file and destination is azure storage endpoint URL with SAS token access.

Note: Get the list of available commands use azcopy –help

Copy from On-Premises to Azure Storage Blob from Azure Cloud Shell or Local System

Upload/Copy/Move/Transfer a file/folder from on-premise to azure storage account blob using Azure Cloud Shell by logging into Azure Portal from On-premise Server.

Prerequisites:

Access to Azure Portal
Owner or Contributor access on Azure Resource
Requires special RBAC Role: "Storage blob Data Contributor" RBAC role access

First of all you should login to your Azure Tenant using the following command:

command: azcopy login

This command returns the URL of a website and an authentication code to authorize the azure account.

command: azcopy copy "<on-premises file/folder-path>" "https://<storage-account-name>.blob.core.windows.net/<container-name>" --recursive=true

Copy Files from Azure VM to Azure VM

If you want to copy data from one Azure VM to another Azure VM then use the following steps to move data to or from.

Step 1: Download and Install WinSCP Tool on both the Azure VM

Step 2: Now select your Azure Source VM and open WinSCP as administrator.

Step 3: Now, open a new session and Login to destination server with IP

Step 4: Once you have logged in then you can copy data to and from with WinSCP tool.

Step 5: Once you're done with copy then closed the sessions to logout. That's it!

CloudTechIN

Search This Blog