{"id":8827,"date":"2023-08-02T08:39:00","date_gmt":"2023-08-02T06:39:00","guid":{"rendered":"https:\/\/via-internet.de\/blog\/?p=8827"},"modified":"2023-08-26T10:27:44","modified_gmt":"2023-08-26T08:27:44","slug":"daily-download-files-from-website-with-powershell","status":"publish","type":"post","link":"https:\/\/via-internet.de\/blog\/2023\/08\/02\/daily-download-files-from-website-with-powershell\/","title":{"rendered":"Daily Azure: Migrate a Storge Account"},"content":{"rendered":"\n<h2 class=\"wp-block-heading\">TL;DR<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Migration is done via azcopy:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>download souce container to local folder<\/li>\n\n\n\n<li>upload local folder to destination container<\/li>\n<\/ul>\n\n\n\n<h2 class=\"wp-block-heading\">Get AzCopy<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Here is the script <code data-enlighter-language=\"ini\" class=\"EnlighterJSRAW\">install-azcopy.ps1<\/code>:<\/p>\n\n\n\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"powershell\" data-enlighter-theme=\"\" data-enlighter-highlight=\"\" data-enlighter-linenumbers=\"\" data-enlighter-lineoffset=\"\" data-enlighter-title=\"\" data-enlighter-group=\"\"># Download and extract\n\n#\n$URI = \"https:\/\/aka.ms\/downloadazcopy-v10-windows\"\n$DST = \"~\\AppData\\Local\\Programs\\AZCopy\\\"\n\nInvoke-WebRequest -Uri $URI -OutFile AzCopy.zip -UseBasicParsing\nExpand-Archive .\/AzCopy.zip .\/AzCopy -Force\n\n# Move AzCopy\nmkdir ~\\AppData\\Local\\Programs\\AZCopy\nGet-ChildItem .\/AzCopy\/*\/azcopy.exe | Move-Item -Destination \"~\\$DEST\"\n\n# Add AzCopy to PATH\n$userenv = (Get-ItemProperty -Path 'HKCU:\\Environment' -Name Path).path\n$newPath = \"$userenv;\nNew-ItemProperty -Path 'HKCU:\\Environment' -Name Path -Value $newPath -Force\n\n# Clean the kitchen\ndel -Force AzCopy.zip\ndel -Force -Recurse .\\AzCopy\\<\/pre>\n\n\n\n<h2 class=\"wp-block-heading\">Copy Folder<\/h2>\n\n\n\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"generic\" data-enlighter-theme=\"\" data-enlighter-highlight=\"\" data-enlighter-linenumbers=\"\" data-enlighter-lineoffset=\"\" data-enlighter-title=\"\" data-enlighter-group=\"\">param (\n    $FOLDER = \"\",\n\n    [Parameter(Mandatory=$false)]\n    [string]$TYPE   = \"latest\",\n\n    [Parameter(Mandatory=$false)]\n    [switch]$LOGIN\n)\n\nif ($TYPE -eq \"latest\") {\n    $SRC_ROOT=\"&lt;latest-folder>\"\n    $DST_ROOT=\"latest\"\n} else {\n    $SRC_ROOT=\"&lt;history-folder>\"\n    $DST_ROOT=\"history\"\n}\n\n\n$SRC_ACCCOUNT = \"&lt;source storage account>\";\n$DST_ACCCOUNT = \"&lt;destination storage account>\";\n\n$SRC_CONTAINER = \"&lt;source container>\"\n$DST_CONTAINER = \"&lt;destination container>\"\n\n\n$SRC_URL      = \"https:\/\/${SRC_ACCCOUNT}.blob.core.windows.net\/$SRC_CONTAINER\/$SRC_ROOT\/$FOLDER\/\"\n$DST_URL      = \"https:\/\/${DST_ACCCOUNT}.blob.core.windows.net\/$DST_CONTAINER\/$DST_ROOT\/\"\n\n$TMP_FLDR     = \"C:\\TMP\\Downloads\"\n\nWrite-Host  \"== Copy     '$FOLDER'\"\nWrite-Host \"       from  $SRC_URL\"\nWrite-Host  \"        to  $DST_URL\"\n\n#\n\nif ($LOGIN) {\n    $ENV:AZCOPY_CRED_TYPE = \"OAuthToken\";\n    $ENV:AZCOPY_CONCURRENCY_VALUE = \"AUTO\";\n\n    azcopy login\n}\n\nWrite-Host  \"== Download ======================================================\"\nWrite-Host \"       from  $SRC_URL\"\nWrite-Host  \"        to  $TMP_FLDR\\$CONTAINER\"\n\nazcopy copy         $SRC_URL                                                                                      `\n                    $TMP_FLDR                                                                                     `\n                    --trusted-microsoft-suffixes=${SRC_ACCCOUNT}.blob.core.windows.net                            `\n                    --overwrite=true                                                                              `\n                    --check-md5=FailIfDifferent                                                                   `\n                    --from-to=BlobLocal                                                                           `\n                    --recursive                                                                                   `\n                    --log-level=ERROR\n\n# Upload\nWrite-Host  \"== Upload   ======================================================\"\nWrite-Host  \"      from  $TMP_FLDR\\$CONTAINER\"\nWrite-Host  \"        to  $DST_URL\"\n\nazcopy copy         $TMP_FLDR\\$CONTAINER                         `\n                    $DST_URL                                     `\n                    --overwrite=true                             `\n                    --from-to=LocalBlob                          `\n                    --blob-type BlockBlob                        `\n                    --follow-symlinks                            `\n                    --check-length=true                          `\n                    --put-md5                                    `\n                    --follow-symlinks                            `\n                    --disable-auto-decoding=false                `\n                    --recursive                                  `\n                    --log-level=ERROR\n<\/pre>\n\n\n\n<h2 class=\"wp-block-heading\">Call the script<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">First call should use -login to authenticate<\/p>\n\n\n\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"generic\" data-enlighter-theme=\"\" data-enlighter-highlight=\"\" data-enlighter-linenumbers=\"\" data-enlighter-lineoffset=\"\" data-enlighter-title=\"\" data-enlighter-group=\"\">.\\copy.ps1 demo-folder-1 -login<\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Then, following commands dont need the login<\/p>\n\n\n\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"generic\" data-enlighter-theme=\"\" data-enlighter-highlight=\"\" data-enlighter-linenumbers=\"\" data-enlighter-lineoffset=\"\" data-enlighter-title=\"\" data-enlighter-group=\"\">.\\copy.ps1 demo-folder-2<\/pre>\n","protected":false},"excerpt":{"rendered":"<p>TL;DR Migration is done via azcopy: Get AzCopy Here is the script install-azcopy.ps1: Copy Folder Call the script First call should use -login to authenticate Then, following commands dont need the login<\/p>\n","protected":false},"author":1,"featured_media":8902,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_crdt_document":"","_jetpack_memberships_contains_paid_content":false,"footnotes":""},"categories":[9,135],"tags":[133,134],"class_list":["post-8827","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-azure","category-daily","tag-daily","tag-powershell"],"jetpack_featured_media_url":"https:\/\/via-internet.de\/blog\/wp-content\/uploads\/2022\/05\/Header-Daily-Download-files-from-Website-with-PowerShell-1.png","jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/via-internet.de\/blog\/wp-json\/wp\/v2\/posts\/8827","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/via-internet.de\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/via-internet.de\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/via-internet.de\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/via-internet.de\/blog\/wp-json\/wp\/v2\/comments?post=8827"}],"version-history":[{"count":18,"href":"https:\/\/via-internet.de\/blog\/wp-json\/wp\/v2\/posts\/8827\/revisions"}],"predecessor-version":[{"id":9702,"href":"https:\/\/via-internet.de\/blog\/wp-json\/wp\/v2\/posts\/8827\/revisions\/9702"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/via-internet.de\/blog\/wp-json\/wp\/v2\/media\/8902"}],"wp:attachment":[{"href":"https:\/\/via-internet.de\/blog\/wp-json\/wp\/v2\/media?parent=8827"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/via-internet.de\/blog\/wp-json\/wp\/v2\/categories?post=8827"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/via-internet.de\/blog\/wp-json\/wp\/v2\/tags?post=8827"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}