Developer Blog

Tipps und Tricks für Entwickler und IT-Interessierte

Category Azure

Daily Azure: Migrate a Storge Account

TL;DR

.Migration is done via azcopy:

  • download souce container to local folder
  • upload local folder to destination container

Get AzCopy

Here is the script install-azcopy.ps1:

# Download and extract

#
$URI = "https://aka.ms/downloadazcopy-v10-windows"
$DST = "~\AppData\Local\Programs\AZCopy\"

Invoke-WebRequest -Uri $URI -OutFile AzCopy.zip -UseBasicParsing
Expand-Archive ./AzCopy.zip ./AzCopy -Force

# Move AzCopy
mkdir ~\AppData\Local\Programs\AZCopy
Get-ChildItem ./AzCopy/*/azcopy.exe | Move-Item -Destination "~\$DEST"

# Add AzCopy to PATH
$userenv = (Get-ItemProperty -Path 'HKCU:\Environment' -Name Path).path
$newPath = "$userenv;%USERPROFILE%\$DEST;"
New-ItemProperty -Path 'HKCU:\Environment' -Name Path -Value $newPath -Force

# Clean the kitchen
del -Force AzCopy.zip
del -Force -Recurse .\AzCopy\

Copy Folder

param (
    $FOLDER = "",

    [Parameter(Mandatory=$false)]
    [string]$TYPE   = "latest",

    [Parameter(Mandatory=$false)]
    [switch]$LOGIN
)

if ($TYPE -eq "latest") {
    $SRC_ROOT="<latest-folder>"
    $DST_ROOT="latest"
} else {
    $SRC_ROOT="<history-folder>"
    $DST_ROOT="history"
}


$SRC_ACCCOUNT = "<source storage account>";
$DST_ACCCOUNT = "<destination storage account>";

$SRC_CONTAINER = "<source container>"
$DST_CONTAINER = "<destination container>"


$SRC_URL      = "https://${SRC_ACCCOUNT}.blob.core.windows.net/$SRC_CONTAINER/$SRC_ROOT/$FOLDER/"
$DST_URL      = "https://${DST_ACCCOUNT}.blob.core.windows.net/$DST_CONTAINER/$DST_ROOT/"

$TMP_FLDR     = "C:\TMP\Downloads"

Write-Host  "== Copy     '$FOLDER'"
Write-Host "       from  $SRC_URL"
Write-Host  "        to  $DST_URL"

#

if ($LOGIN) {
    $ENV:AZCOPY_CRED_TYPE = "OAuthToken";
    $ENV:AZCOPY_CONCURRENCY_VALUE = "AUTO";

    azcopy login
}

Write-Host  "== Download ======================================================"
Write-Host "       from  $SRC_URL"
Write-Host  "        to  $TMP_FLDR\$CONTAINER"

azcopy copy         $SRC_URL                                                                                      `
                    $TMP_FLDR                                                                                     `
                    --trusted-microsoft-suffixes=${SRC_ACCCOUNT}.blob.core.windows.net                            `
                    --overwrite=true                                                                              `
                    --check-md5=FailIfDifferent                                                                   `
                    --from-to=BlobLocal                                                                           `
                    --recursive                                                                                   `
                    --log-level=ERROR

# Upload
Write-Host  "== Upload   ======================================================"
Write-Host  "      from  $TMP_FLDR\$CONTAINER"
Write-Host  "        to  $DST_URL"

azcopy copy         $TMP_FLDR\$CONTAINER                         `
                    $DST_URL                                     `
                    --overwrite=true                             `
                    --from-to=LocalBlob                          `
                    --blob-type BlockBlob                        `
                    --follow-symlinks                            `
                    --check-length=true                          `
                    --put-md5                                    `
                    --follow-symlinks                            `
                    --disable-auto-decoding=false                `
                    --recursive                                  `
                    --log-level=ERROR

Call the script

First call should use -login to authenticate

.\copy.ps1 demo-folder-1 -login

Then, following commands dont need the login

.\copy.ps1 demo-folder-2
by Ralph Azure, Daily

Azure | Certification Path

Certifications: 1 2

Azure Solutions Architect Expert

Qualification

  • Implementierung von Management- und Sicherheitslösungen
  • Implementieren und Überwachen einer Azure-Infrastruktur
  • Implementieren Sie Lösungen für Apps
  • Implementierung und Verwaltung von Datenplattformen
  • Entwurfsüberwachung
  • Entwurf Identität und Sicherheit
  • Entwerfen eines Datenspeichers
  • Entwicklung von Geschäftskontinuität
  • Infrastruktur entwerfen

Exams

Exams

  • Implementieren und Überwachen einer Azure-Infrastruktur (50-55%)
  • Implementierung von Management- und Sicherheitslösungen (25-30%)
  • Implementieren Sie Lösungen für Apps (10-15%)
  • Implementierung und Verwaltung von Datenplattformen (10-15%)
  • Entwurfsüberwachung (10-15%)
  • Entwurf Identität und Sicherheit (25-30%)
  • Entwerfen eines Datenspeichers (15-20%)
  • Entwicklung von Geschäftskontinuität (10-15%)
  • Infrastruktur entwerfen (25-30%)

Exam Preparation

Readings

Learning Path

AZ-303 / AZ-304

Entwerfen einer Computeinfrastruktur in Azure12
Entwerfen einer Speicherinfrastruktur in Azure9
Entwerfen einer Computeinfrastruktur in Azure9
Entwerfen von Infrastrukturvorgängen in Azure5
Entwerfen einer Datenplattform im Azure10
Entwerfen von Nachrichtenbrokern und serverlosen Anwendungen in Azure8
Entwerfen moderner Anwendungen in Azure8
Entwerfen einer API-Integration in Azure5
Entwerfen von Migration, Geschäftskontinuität und Notfallwiederherstellung in Azure9
by Ralph Azure

Azure Databricks| Working with Unit Tests

Introduction

Problem

Like any other program, Azure Databricks notebooks should be tested automatically to ensure code quality.

Using standard Python Test Tools is not easy because these tools are based on Python files in a file system. And a notebook doesn’t correspond to a Python file.

Solution

To enable automated testing with unittest (documentation), we proceed as follows:

  • Create a test class that contains all the tests you want
  • Execution of all defined tests

Create Notebook with the Code

We will create a simple Notebook for our test.

This notebook will implement a simple calculator, so that we can test the basic calculator operations like add ad multiply.

Create a new Notebook with the name Calculator:

class Calculator:

	def __init__(self, x = 10, y = 8):
		self.x = x
		self.y = y
		
	def add(self, x = None, y = None):
		if x == None: x = self.x
		if y == None: y = self.y			
          
		return x+y

	def subtract(self, x = None, y = None):
		if x == None: x = self.x
		if y == None: y = self.y	
          
		return x-y

	def multiply(self, x = None, y = None):
		if x == None: x = self.x
		if y == None: y = self.y			
          
		return x*y

	def divide(self, x = None, y = None):
		if x == None: x = self.x
		if y == None: y = self.y			
          
		if y == 0:
			raise ValueError('cannot divide by zero')
		else:
			return x/y

The notebook should look like this

To use this class, write the following lines:

c = Calculator()
print(c.add(20, 10), c.subtract(20, 10), c.multiply(20, 10), c.divide(20, 10))

Create Notebook with the Tests

Create a new Notebook in the same folder with the name Calculator.Tests.

The name is not important, but it is convenient to name the test program like the program to be tested with the suffix ‘Tests’.

Create the first command to import the Calculator Notebook

%run "./Calculator"

Create the Test Class

import unittest

class CalculatorTests(unittest.TestCase):
  
  @classmethod
  def setUpClass(cls):
    cls.app = Calculator()

  def setUp(self):
    # print("this is setup for every method")
    pass

  def test_add(self):
    self.assertEqual(self.app.add(10,5), 15, )

  def test_subtract(self):
    self.assertEqual(self.app.subtract(10,5), 5)
    self.assertNotEqual(self.app.subtract(10,2), 4)

  def test_multiply(self):
    self.assertEqual(self.app.multiply(10,5), 50)

  def tearDown(self):
    # print("teardown for every method")
    pass

  @classmethod
  def tearDownClass(cls):
    # print("this is teardown class")
    pass

Create the code to run the tests

suite =  unittest.TestLoader().loadTestsFromTestCase(CalculatorTests)
unittest.TextTestRunner(verbosity=2).run(suite)
by Ralph Azure, Databricks

Azure | Cookbook Databricks

Databricks CLI

Export all Notebooks

databricks workspace list | ForEach { databricks workspace export_dir /$_ $_ }

Troubleshooting
Problem

Error in SQL statement: AnalysisException: Can not create the managed table('`demo`'). The associated location('dbfs:/user/hive/warehouse/demo') already exists.;

Solution

dbutils.fs.rm("dbfs:/user/hive/warehouse/demo/", true)

by Ralph Azure, Databricks

Azure | Working with Widgets

TL;DR

Don’t want to read the post, then explore this Azure Notebook

Requirements

Define needed moduls and functions

from datetime import datetime

import pyspark.sql.functions as F

Create DataFrame for this post:

df = spark.sql("select * from diamonds")
df.show()

Working with Widgets

Default Widgets

dbutils.widgets.removeAll()

dbutils.widgets.text("W1", "1", "Text")
dbutils.widgets.combobox("W2", "3", [str(x) for x in range(1, 10)], "Combobox")
dbutils.widgets.dropdown("W3", "4", [str(x) for x in range(1, 10)], "Dropdown")

Multiselect Widgets

list = [ f"Square of {x} is {x*x}" for x in range(1, 10)]
dbutils.widgets.multiselect("W4", list[0], list, "Multi-Select")

Monitor the changes when selection values

print("Selection: ", dbutils.widgets.get("W4"))
print("Current Time =", datetime.now().strftime("%H:%M:%S"))

Filter Query by widgets

Prepare widgets

dbutils.widgets.removeAll()

df = spark.sql("select * from diamonds")

vals = [ str(x[0]) for x in df.select("cut").orderBy("cut").distinct().collect() ]
dbutils.widgets.dropdown("Cuts", vals[0], vals)

vals = [ str(x[0]) for x in df.select("carat").orderBy("carat").distinct().collect() ]
dbutils.widgets.dropdown("Carat", vals[0], vals)

Now, change some values

filter_cut = dbutils.widgets.get("Cuts")
df=spark.sql(f"select * from diamonds where cut='{filter_cut}'").show()

by Ralph Azure, Databricks, Widgets

Azure Databricks | Cookbook

Reading Data

Create Table from CSV file with SQL

DROP TABLE IF EXISTS quickstart;

CREATE TABLE quickstart
USING csv
OPTIONS (path "/databricks-datasets/data.csv", header "true")

Create Table from CSV file with PySpark

%python
quickstart= spark.read.csv("/databricks-datasets/data.csv", header="true", inferSchema="true")

Analyse Data

Group and Display

%python
from pyspark.sql.functions import avg

display(quickstart.select("color","price").groupBy("color").agg(avg("price")).sort("color"))
by Ralph Azure, Cookbook, Databricks

Python | Working with Azure

First Step: Hello World Sample

The following steps at borrowed from the quick start tutorial.

Download sample repository

$ git clone https://github.com/Azure-Samples/python-docs-hello-world
$ cd python-docs-hello-world

Create virtual environment

$ python3 -m venv venv
$ source venv/bin/activate
$ pip install -r requirements.txt
$ export FLASK_APP=application.py
$ flask run

Login zu Azure

$ az login

Deploy to App Service

$ az webapp up --sku F1 -n azure-toolbox-flask-demo -l westeurope
webapp azure-toolbox-flask-demo doesn't exist
Creating Resource group 'xx_xx_Linux_westeurope' ...
Resource group creation complete
Creating AppServicePlan 'xx_asp_Linux_westeurope_0' ...
Creating webapp 'flask-demo' ...
Configuring default logging for the app, if not already enabled
Creating zip with contents of dir .../Working-with_Python ...
Getting scm site credentials for zip deployment
Starting zip deployment. This operation can take a while to complete ...
Deployment endpoint responded with status code 202
You can launch the app at http://via-internet-flask-demo.azurewebsites.net
{
  "URL": "http:/azure-toolbox-flask-demo.azurewebsites.net",
  "appserviceplan": "xx_asp_Linux_westeurope_0",
  "location": "westeurope",
  "name": "azure-toolbox--flask-demo",
  "os": "Linux",
  "resourcegroup": "xx_xx_Linux_westeurope",
  "runtime_version": "python|3.7",
  "runtime_version_detected": "-",
  "sku": "FREE",
  "src_path": ".../Working-with_Python"
}

Create Django App with PostgreSQL

Installation PostgreSQL on Mac OS

$ brew install postgres
==> Installing dependencies for postgresql: krb5
==> Installing postgresql dependency: krb5
...
==> Installing postgresql
...
==> Caveats
==> krb5
krb5 is keg-only, which means it was not symlinked into /usr/local, because macOS already provides this software and installing another version in
parallel can cause all kinds of trouble.

If you need to have krb5 first in your PATH run:
  echo 'export PATH="/usr/local/opt/krb5/bin:$PATH"' >> ~/.bash_profile
  echo 'export PATH="/usr/local/opt/krb5/sbin:$PATH"' >> ~/.bash_profile

For compilers to find krb5 you may need to set:
  export LDFLAGS="-L/usr/local/opt/krb5/lib"
  export CPPFLAGS="-I/usr/local/opt/krb5/include"

For pkg-config to find krb5 you may need to set:
  export PKG_CONFIG_PATH="/usr/local/opt/krb5/lib/pkgconfig"

==> postgresql
To migrate existing data from a previous major version of PostgreSQL run:
  brew postgresql-upgrade-database

To have launchd start postgresql now and restart at login:
  brew services start postgresql
Or, if you don't want/need a background service you can just run:
  pg_ctl -D /usr/local/var/postgres start

Set user and passwords for postgres database






Create database and user for django app





$ psql postgres
psql (12.1)
Type "help" for help.

postgres=# CREATE DATABASE pollsdb;
CREATE DATABASE
postgres=# CREATE USER manager WITH PASSWORD '########';
CREATE ROLE
postgres=# GRANT ALL PRIVILEGES ON DATABASE pollsdb TO manager;
GRANT





Download sample repository





$ git clone https://github.com/Azure-Samples/djangoapp.git
$ cd djangoapp





Create virtual environment





$ python3 -m venv venv
$ source venv/bin/activate
$ pip install -r requirements.txt





$ cat env.sh
export DBHOST="localhost"
export DBUSER="manager"
export DBNAME="pollsdb"
export DBPASS="supersecretpass"
$ . env.sh





$ python manage.py  makemigrations
No changes detected
$ python manage.py  migrate
Operations to perform:
  Apply all migrations: admin, auth, contenttypes, polls, sessions
Running migrations:
  Applying contenttypes.0001_initial... OK
  Applying auth.0001_initial... OK
  Applying admin.0001_initial... OK
  Applying admin.0002_logentry_remove_auto_add... OK
  Applying admin.0003_logentry_add_action_flag_choices... OK
  Applying contenttypes.0002_remove_content_type_name... OK
  Applying auth.0002_alter_permission_name_max_length... OK
  Applying auth.0003_alter_user_email_max_length... OK
  Applying auth.0004_alter_user_username_opts... OK
  Applying auth.0005_alter_user_last_login_null... OK
  Applying auth.0006_require_contenttypes_0002... OK
  Applying auth.0007_alter_validators_add_error_messages... OK
  Applying auth.0008_alter_user_username_max_length... OK
  Applying auth.0009_alter_user_last_name_max_length... OK
  Applying polls.0001_initial... OK
  Applying sessions.0001_initial... OK





 $ python manage.py createsuperuser
Username (leave blank to use 'user'): admin
Email address: admin@localhost
Password:
Password (again):
Superuser created successfully.





Run server and acccess web page at http://127.0.0.1:8000/





$ python manage.py runserver
Performing system checks...

System check identified no issues (0 silenced).
January 25, 2020 - 16:42:14
Django version 2.1.2, using settings 'azuresite.settings'
Starting development server at http://127.0.0.1:8000/
Quit the server with CONTROL-C.
[25/Jan/2020 16:42:26] "GET / HTTP/1.1" 200 111
[25/Jan/2020 16:42:26] "GET /static/polls/style.css HTTP/1.1" 200 27
Not Found: /favicon.ico
[25/Jan/2020 16:42:26] "GET /favicon.ico HTTP/1.1" 404 2688





Login zu Azure





$ az login





Deploy to App Service





$ az webapp up --sku F1 -n azure-toolbox-django-demo -l westeurope
webapp azure-toolbox-django-demo doesn't exist
Creating Resource group 'xx_xx_Linux_westeurope' ...
Resource group creation complete
Creating AppServicePlan 'xx_asp_Linux_westeurope_0' ...
Creating webapp 'flask-demo' ...
Configuring default logging for the app, if not already enabled
Creating zip with contents of dir .../Working-with_Django ...
Getting scm site credentials for zip deployment
Starting zip deployment. This operation can take a while to complete ...
Deployment endpoint responded with status code 202
You can launch the app at http://via-internet-django-demo.azurewebsites.net
{
  "URL": "http:/azure-toolbox-django-demo.azurewebsites.net",
  "appserviceplan": "xx_asp_Linux_westeurope_0",
  "location": "westeurope",
  "name": "azure-toolbox--django-demo",
  "os": "Linux",
  "resourcegroup": "xx_xx_Linux_westeurope",
  "runtime_version": "python|3.7",
  "runtime_version_detected": "-",
  "sku": "FREE",
  "src_path": ".../Working-with_Django"
}





 





Additional Reading









Installation





Here is the documentation from Microsoft.





Mac OS





Install with Homebrew





$ brew update && brew install azure-cli
$ az login

											

			        										

					    
							by Ralph	
						
												Azure, Python