Ralph – Page 14 – Developer Blog

Entwerfen einer Computeinfrastruktur in Azure	12
Entwerfen einer Speicherinfrastruktur in Azure	9
Entwerfen einer Computeinfrastruktur in Azure	9
Entwerfen von Infrastrukturvorgängen in Azure	5
Entwerfen einer Datenplattform im Azure	10
Entwerfen von Nachrichtenbrokern und serverlosen Anwendungen in Azure	8
Entwerfen moderner Anwendungen in Azure	8
Entwerfen einer API-Integration in Azure	5
Entwerfen von Migration, Geschäftskontinuität und Notfallwiederherstellung in Azure	9

7Sep

NestJS | Getting started – Part 1

by Ralph AngularJS, Ionic, NestJS

Introduction

NestJS (just Nest from here on out), is a Node framework meant to build server-side applications. Not only is it a framework, but it is also a platform to meet many backend application needs, like writing APIs, building microservices, or doing real-time communications through web sockets.

Nest is also heavily influenced by Angular, and you will immediately find its concepts familiar. The creators of Nest strived to make the learning curve as small as possible, while still taking advantage of many higher level concepts such as modules, controllers, and dependency injection.

Installation

Install NodeJS

Download NodeJS from here and install as described here.

For example, on macOS using Homebrew

brew install node

Or download the package

curl "https://nodejs.org/dist/latest/node-${VERSION:-$(wget -qO- https://nodejs.org/dist/latest/ | sed -nE 's|.*>node-(.*)\.pkg</a>.*|\1|p')}.pkg" > "$HOME/Downloads/node-latest.pkg" && sudo installer -store -pkg "$HOME/Downloads/node-latest.pkg" -target "/"

Install NextJS

npm i -g @nestjs/cli

Create server App

Create new server App

nest new demo.server

Start server App

cd demo.server

npm run start:dev

Now open browser on http: localhost:3000

App Structure

main.ts

import { NestFactory } from '@nestjs/core';
import { AppModule } from './app.module';

async function bootstrap() {
  const app = await NestFactory.create(AppModule);
  await app.listen(3000);
}
bootstrap();

app.module.ts

import { Module } from '@nestjs/common';
import { AppController } from './app.controller';
import { AppService } from './app.service';

@Module({
  imports: [],
  controllers: [AppController],
  providers: [AppService],
})
export class AppModule {}

app.controller.ts

import { Controller, Get } from '@nestjs/common';
import { AppService } from './app.service';

@Controller()
export class AppController {
  constructor(private readonly appService: AppService) {}

  @Get()
  getHello(): string {
    return this.appService.getHello();
  }
}

app.service.ts

import { Injectable } from '@nestjs/common';

@Injectable()
export class AppService {
  getHello(): string {
    return 'Hello World!';
  }
}

Add Functionality

Create service and controller

nest g service missions
nest g controller missions

Modify `mission service`

@Injectable()
export class MissionsService {
  missions: Mission[] = [
    { id: 1, title: 'Rescue cat stuck in asteroid', reward: 500, active: true, },
    { id: 2, title: 'Escort Royal Fleet', reward: 5000, active: true, },
    { id: 3, title: 'Pirates attacking the station', reward: 2500, active: false, },
  ];

  async getMissions(): Promise<Mission[]> {
    return this.missions;
  }
}

Modify `mission controller`

@Controller('missions')
export class MissionsController {
  constructor(private missionsService: MissionsService) {}

  @Get()
  getMissions() {
    return this.missionsService.getMissions();
  }
}

Open in browser: http://localhost:3000

Create Frontend App

Create new frontend App

ionic start demo.frontend sidemenu

Working with Database and TypeORM

Create / Sync database with schema

Add command to package.json

"scripts": {
    "typeorm": "ts-node -r tsconfig-paths/register ./node_modules/typeorm/cli.js"
}

npm run typeorm schema:sync

TypeORM Commands

schema:sync
Synchronizes your entities with database schema. It runs schema update queries on all connections you have. To run update queries on a concrete connection use -c option.
schema:log
Shows sql to be executed by schema:sync command. It shows sql log only for your default connection. To run update queries on a concrete connection use -c option.
schema:drop
Drops all tables in the database on your default connection. To drop table of a concrete connection's database use -c option.
query
Executes given SQL query on a default connection. Specify connection name to run query on a specific connection.
entity:create
Generates a new entity.
subscriber:create
Generates a new subscriber.
migration:create
Creates a new migration file. [Aliase: migrations:create]
migration:generate Generates a new migration file with sql needs to be executed to update schema. [Aliase: migrations:generate]
migration:run
Runs all pending migrations. [Aliase: migrations:run]
migration:show
Show all migrations and whether they have been run or not
migration:revert
Reverts last executed migration. [Aliase: migrations:revert]
version
Prints TypeORM version this project uses.
cache:clear
Clears all data stored in query runner cache.
init
Generates initial TypeORM project structure. If name specified then creates files inside directory called as name. If its not specified then creates files inside current directory.

Additional readings

5Sep

Frontend | Toolbox

by Ralph Uncategorized

Installation Overview

NodeJS

Angular

Ionic

Install

npm install -g @ionic/cli

Create App

ionic start Getting-Started tabs --type react

Start App

ionic serve

React / ReactJS

Install

React Native

Stencil

npm init stencil
npm install --save-exact @stencil/core@latest 
npm install --save-dev @types/jest@26.0.12 jest@26.4.2 jest-cli@26.4.2 
npm install --save-dev @types/puppeteer@3.0.1 puppeteer@5.2.1
npm test
npm start

Gatsby

NextJS

NestJS

Usefull Libraries

VideoJS	HTML5 player framework	Video	Home		Github
Animate on Scroll		Animation	Home		michalsnik/aos
ScrollMagic		Animation	Home
ScrollRevealJS		Animation	Home/		jlmakes/scrollreveal
PixiJS		Graphics	Home		pixijs/pixi.jsthub
Anime		Animation	Home		juliangarnier/anime
ThreeJS		Graphics	Home		mrdoob/three.js
animate.css		Animation	Home		animate-css/animate.css
HowlerJS	Audio library	Audio	Home		Github
RevealJS	HTML Presentation Framework	Presentation	Home		Github
ChartJS		Chart	Home		G ithub
anime.js			Home
granim.js	Create fluid and interactive gradient animations	Graphics	Home		sarcadass/granim.js
Multiple.js	Sharing background across multiple elements using CSS		Home		NeXTs/Multiple.js
choreographer-js	A simple library to take care of complicated animations.		Home		christinecha/choreographer-js
cleave.js	Format your <input/> content when you are typing		Home		nosir/cleave.js
premonish			Home		mathisonian/premonish
Splitting		Animation	Home	Codepen	shshaw/splitting/

Introduction

From Wikipedia, the free encyclopedia:

Rust is a multi-paradigm programming language focused on performance and safety, especially safe concurrency. Rust is syntactically similar to C++, and provides memory safety without using garbage collection.

Rust was originally designed by Graydon Hoare at Mozilla Research.
It has gained increasing use in industry and is now Microsoft’s language of choice for secure and safety-critical software components.

Rust has been the “most loved programming language” in the Stack Overflow Developer Survey every year since 2016.

Rust could be used in different areas

Web Assembly
Embedded Programming
Command Line Utilities
Operating System Developmend (see Redox OS)

Installation

Rustup

Download install script and run it

curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh

Modify .bashrc to add Rust path to PATH

source $HOME/.cargo/env

Other way to install Rust on MacOS

brew install rust

Create and run your first App

Create the app

$ cargo new hello_world
$ cd hello_world

Show folder structure

$ tree .
.
├── Cargo.lock
├── Cargo.toml
└── src
    └── main.rs

1 directory, 3 files

Show main source file

fn main() {
    println!("Hello, world!");
}

Build your app

$ cargo build
Compiling hello_world v0.1.0 (.../hello_world)
Finished dev [unoptimized + debuginfo] target(s) in 6.32s

Or build a production ready version

$ cargo build --release<br>Finished release [optimized] target(s) in 0.19s

Run your app

$ cargo run
Finished dev [unoptimized + debuginfo] target(s) in 0.04s
Running `target/debug/hello_world
Hello, world!

Add functionality to your app

Add Dependencies

Let’s add a dependency to our application. You can find all sorts of libraries on crates.io, the package registry for Rust. In Rust, we often refer to packages as “crates.”

In this project, we’ll use a crate called ferris-says.

In our Cargo.toml file we’ll add this information (that we got from the crate page):

[dependencies]
ferris-says = "0.1"

Modify main source

Now let’s write a small application with our new dependency. In our main.rs, add the following code:

use ferris_says::say; // from the previous step
use std::io::{stdout, BufWriter};

fn main() {
    let stdout = stdout();
    let message = String::from("Hello fellow Rustaceans!");
    let width = message.chars().count();

    let mut writer = BufWriter::new(stdout.lock());
    say(message.as_bytes(), width, &mut writer).unwrap();
}

Run App

$ cargo build
    Updating crates.io index
  Downloaded object v0.20.0
  Downloaded textwrap v0.11.0
  Downloaded adler v0.2.3
  Downloaded ansi_term v0.11.0
  Downloaded miniz_oxide v0.4.1
  Downloaded gimli v0.22.0
  Downloaded strsim v0.8.0
  Downloaded error-chain v0.10.0
  Downloaded vec_map v0.8.2
  Downloaded clap v2.33.3
  Downloaded smallvec v0.4.5
  Downloaded ferris-says v0.1.2
  Downloaded backtrace v0.3.50
  Downloaded rustc-demangle v0.1.16
  Downloaded addr2line v0.13.0
  Downloaded 15 crates (1.4 MB) in 1.65s
   Compiling libc v0.2.76
   Compiling bitflags v1.2.1
   Compiling gimli v0.22.0
   Compiling adler v0.2.3
   Compiling rustc-demangle v0.1.16
   Compiling unicode-width v0.1.8
   Compiling object v0.20.0
   Compiling cfg-if v0.1.10
   Compiling strsim v0.8.0
   Compiling vec_map v0.8.2
   Compiling ansi_term v0.11.0
   Compiling smallvec v0.4.5
   Compiling textwrap v0.11.0
   Compiling miniz_oxide v0.4.1
   Compiling addr2line v0.13.0
   Compiling atty v0.2.14
   Compiling backtrace v0.3.50
   Compiling clap v2.33.3
   Compiling error-chain v0.10.0
   Compiling ferris-says v0.1.2
   Compiling hello_world v0.1.0 (.../hello_world)
    Finished dev [unoptimized + debuginfo] target(s) in 14.73s

Run your app

$ cargo run
     Finished dev [unoptimized + debuginfo] target(s) in 0.14s
       Running`target/debug/hello_world`

----------------------------
| Hello fellow Rustaceans! |
----------------------------
              \
               \
                  _~^~^~_
              \) /  o o  \ (/
                '_   -   _'
                / '-----' \

Next steps

Read Getting Started on rust homepage

Explore Learn Rust

Next Readings

https://blog.rust-lang.org
https://blog.rust-lang.org/inside-rust/
https://doc.rust-lang.org/book/
https://doc.rust-lang.org/rust-by-example/

Introduction

Problem

Like any other program, Azure Databricks notebooks should be tested automatically to ensure code quality.

Using standard Python Test Tools is not easy because these tools are based on Python files in a file system. And a notebook doesn’t correspond to a Python file.

Solution

To enable automated testing with unittest (documentation), we proceed as follows:

Create a test class that contains all the tests you want
Execution of all defined tests

Create Notebook with the Code

We will create a simple Notebook for our test.

This notebook will implement a simple calculator, so that we can test the basic calculator operations like add ad multiply.

Create a new Notebook with the name Calculator:

class Calculator:

	def __init__(self, x = 10, y = 8):
		self.x = x
		self.y = y
		
	def add(self, x = None, y = None):
		if x == None: x = self.x
		if y == None: y = self.y			
          
		return x+y

	def subtract(self, x = None, y = None):
		if x == None: x = self.x
		if y == None: y = self.y	
          
		return x-y

	def multiply(self, x = None, y = None):
		if x == None: x = self.x
		if y == None: y = self.y			
          
		return x*y

	def divide(self, x = None, y = None):
		if x == None: x = self.x
		if y == None: y = self.y			
          
		if y == 0:
			raise ValueError('cannot divide by zero')
		else:
			return x/y

The notebook should look like this

To use this class, write the following lines:

c = Calculator()
print(c.add(20, 10), c.subtract(20, 10), c.multiply(20, 10), c.divide(20, 10))

Create Notebook with the Tests

Create a new Notebook in the same folder with the name Calculator.Tests.

The name is not important, but it is convenient to name the test program like the program to be tested with the suffix ‘Tests’.

Create the first command to import the Calculator Notebook




Create the Test Class



import unittest

class CalculatorTests(unittest.TestCase):
  
  @classmethod
  def setUpClass(cls):
    cls.app = Calculator()

  def setUp(self):
    # print("this is setup for every method")
    pass

  def test_add(self):
    self.assertEqual(self.app.add(10,5), 15, )

  def test_subtract(self):
    self.assertEqual(self.app.subtract(10,5), 5)
    self.assertNotEqual(self.app.subtract(10,2), 4)

  def test_multiply(self):
    self.assertEqual(self.app.multiply(10,5), 50)

  def tearDown(self):
    # print("teardown for every method")
    pass

  @classmethod
  def tearDownClass(cls):
    # print("this is teardown class")
    pass
Create the code to run the tests
suite =  unittest.TestLoader().loadTestsFromTestCase(CalculatorTests)
unittest.TextTestRunner(verbosity=2).run(suite)

3Jun

Azure | Cookbook Databricks

by Ralph Azure, Databricks

Databricks CLI

Export all Notebooks

databricks workspace list | ForEach { databricks workspace export_dir /$_ $_ }

Troubleshooting

Problem

Error in SQL statement: AnalysisException: Can not create the managed table('`demo`'). The associated location('dbfs:/user/hive/warehouse/demo') already exists.;

Solution

dbutils.fs.rm("dbfs:/user/hive/warehouse/demo/", true)

Handling Complex Data Scenarios

When working with nested data structures in Databricks, the explode() function is essential but comes with hidden pitfalls. Here are key insights for advanced users:

1. The Null Trap in explode()

The standard explode() function silently drops rows with empty arrays or null values – a common pain point in production pipelines. Consider this dataset:

pythondata = [
    (1, "Luke", ["baseball", "soccer"]),
    (2, "Lucy", None),
    (3, "Eve", [])
]

df = spark.createDataFrame(data, ["id", "name", "likes"])

Standard explode behavior

Output retains only Luke’s exploded rows

df.select("id", "name", explode("likes")).show()

Solution: explode_outer()

Preserves Lucy (null) and Eve (empty array) with null values

from pyspark.sql.functions import explode_outer

df.select("id", "name", explode_outer("likes")).show()

2. Advanced Array Handling

For complex nested structures, combine explode_outer() with struct typing:

pythonfrom pyspark.sql.types import StructType, StructField, StringType

schema = StructType([
    StructField("sport", StringType()),
    StructField("level", StringType())
])

df.withColumn("nested", array(struct(lit("baseball").alias("sport"), 
                                   lit("pro").alias("level")))) \
  .select(explode_outer("nested")) \
  .select("col.*") \
  .show()

3. Z-Order Optimization for Exploded Data

When working with large exploded datasets, optimize Delta Lake storage:

python(df
 .write
 .format("delta")
 .option("delta.optimizeWrite", "true")
 .option("delta.dataSkippingNumIndexedCols", "3")
 .saveAsTable("exploded_data")
)

spark.sql("OPTIMIZE exploded_data ZORDER BY (id, sport)")