Ralph – Page 15 – Developer Blog

Installation

Cobol

brew install gnu-cobol

cobc --version
cobc (GnuCOBOL) 2.2.0
Copyright (C) 2017 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
Written by Keisuke Nishida, Roger While, Ron Norman, Simon Sobisch, Edward Hart
Built     Oct 15 2019 14:14:21
Packaged  Sep 06 2017 18:48:43 UTC
C version "4.2.1 Compatible Apple LLVM 11.0.0 (clang-1100.0.33.8)"

First Steps

Create sample programm

Create Hello World programm hello_world.cob

HELLO * HISTORIC EXAMPLE OF HELLO WORLD IN COBOL
       IDENTIFICATION DIVISION.
       PROGRAM-ID. HELLO.
       PROCEDURE DIVISION.
           DISPLAY "HELLO, WORLD".
           STOP RUN.

Compile programm and build executable program

cobc -x hello_world.cob

Run programm

$ ./helloworld
HELLO, WORLD

19Apr

Power Query | Zeilen gruppenweise pivotieren

by Ralph Power Query

Aufgabenstellung

Werden Daten angeliefert, in denen das Gruppierungsmerkmal in den Zeilen vorhanden ist und somit mehrere Zeilen pro Datensatz vorhanden, wünscht man sich meist eine kompaktere Darstellung.

Für den Datensatz mit dem Wert “Daten 1” werden also vier Zeilen mit unterschiedlichen Werten in GRUPPE und Wert angeliefert.

Gewünscht ist aber eine kompaktere Darstellung mit den vorhandenen Gruppen als Spalten:

Die Aufgabenstellung ist somit die Umwandlung der angelieferten Daten:

Eine Beispieldatei liegt hier. Das Endergebnis liegt hier. Speichern sie beide Datein im Order C: \TMP, dann stimmt der Verweis in Query.xlsx auf die Daten Daten.xlsx.

Schritt 1: Daten vorbereiten

Im ersten Schritt erstellen wir eine neue Excel-Daten und greifen auf die vorbereiteten Daten über Power Query zu.

Wählen Sie dazu im Register Daten den Eintrag Daten abrufen / Aus Datei / Aus Arbeitsmappe und selektieren sie die gewünschte Datei:

Eine Beispieldatei liegt hier.

Ein Klick auf Importieren führt sie zum Navigation

Sie sehen im Navigator 3 verschiedenen Elemente:

DATEN: die intelligente Tabelle im Tabellenblatt. Diese beinhaltet genau die gewünschten Daten
ERGBNIS: die intelligente Tabelle, die das zu erwartende Ergbnis beinhaltet
Beispieldaten: das Tabellenblatt mit den beiden intelligenten Tabellen

Selektieren sie das Element DATEN und klicken sie auf Daten transformieren.

Schritt 2: Spalte pivotieren

Wir wollen die Werte der Spalte GRUPPE als neue Spalten erhalten.

Hier klicken sie auf die Spalte GRUPPE und wählen dein Eintrag Spalte pivotieren im Register Transformieren / Beliebige Spalte:

Die Werte für die neuen Spalten (Gruppe 1, Gruppe 2 , ..) kommen aus der Spalte WERT (Wert 11, Wert 12, ..):

Wir wollen die Werte selbst übernehmen und keine (wie bei Pivottabellen meist üblich) Aggregierungsfunktion verwenden (Summe, Max, Anzahl, ..).

Klicken sie hierzu auf Erweiterte Optionen und selektieren sie den Eintrag Nicht aggregieren:

Anschließen klicken sie auf OK:

Zum Abschluss beenden wir den Power Query Editor:

19Apr

Arbeiten mit dem Header

Schreibweise ändern

Grossschreibung/Kleinschreibung/CamelCase

= Table.TransformColumnNames(RenameColumns, Text.Upper)
= Table.TransformColumnNames(RenameColumns, Text.Lower)
= Table.TransformColumnNames(RenameColumns, Text.Proper)

Bestimmte Zeichen entfernen (z. B. _)

= Table.TransformColumnNames(Source,each Text.Proper(Replacer.ReplaceText( _ , "_", " ")))

Aufteilen in Worte

= Table.TransformColumnNames(#"Removed Other Columns", each Text.Combine(
                    Splitter.SplitTextByCharacterTransition({"a".."z"},{"A".."Z"})(_), " "))

Daten transformieren

Zeilen gruppenweise pivotieren

Aufgabenstellung

Werden Daten angeliefert, in denen das Gruppierungsmerkmal in den Zeilen vorhanden ist und somit mehrere Zeilen pro Datensatz vorhanden, wünscht man sich meist eine kompaktere Darstellung.

Für den Datensatz mit dem Wert “Daten 1” werden also vier Zeilen mit unterschiedlichen Werten in GRUPPE und Wert angeliefert.

Gewünscht ist aber eine kompaktere Darstellung mit den vorhandenen Gruppen als Spalten:

Die Aufgabenstellung ist somit die Umwandlung der angelieferten Daten:

Eine Beispieldatei liegt hier. Das Endergebnis liegt hier. Speichern sie beide Datein im Order C: \TMP, dann stimmt der Verweis in Query.xlsx auf die Daten Daten.xlsx.

Schritt 1: Daten vorbereiten

Im ersten Schritt erstellen wir eine neue Excel-Daten und greifen auf die vorbereiteten Daten über Power Query zu.

Wählen Sie dazu im Register Daten den Eintrag Daten abrufen / Aus Datei / Aus Arbeitsmappe und selektieren sie die gewünschte Datei:

Eine Beispieldatei liegt hier.

Ein Klick auf Importieren führt sie zum Navigation

Sie sehen im Navigator 3 verschiedenen Elemente:

DATEN: die intelligente Tabelle im Tabellenblatt. Diese beinhaltet genau die gewünschten Daten
ERGBNIS: die intelligente Tabelle, die das zu erwartende Ergbnis beinhaltet
Beispieldaten: das Tabellenblatt mit den beiden intelligenten Tabellen

Selektieren sie das Element DATEN und klicken sie auf Daten transformieren.

Schritt 2: Spalte pivotieren

Wir wollen die Werte der Spalte GRUPPE als neue Spalten erhalten.

Hier klicken sie auf die Spalte GRUPPE und wählen dein Eintrag Spalte pivotieren im Register Transformieren / Beliebige Spalte:

Die Werte für die neuen Spalten (Gruppe 1, Gruppe 2 , ..) kommen aus der Spalte WERT (Wert 11, Wert 12, ..):

Wir wollen die Werte selbst übernehmen und keine (wie bei Pivottabellen meist üblich) Aggregierungsfunktion verwenden (Summe, Max, Anzahl, ..).

Klicken sie hierzu auf Erweiterte Optionen und selektieren sie den Eintrag Nicht aggregieren:

Anschließen klicken sie auf OK:

Zum Abschluss beenden wir den Power Query Editor:

16Apr

Reading Data

Create Table from CSV file with SQL

DROP TABLE IF EXISTS quickstart;

CREATE TABLE quickstart
USING csv
OPTIONS (path "/databricks-datasets/data.csv", header "true")

Create Table from CSV file with PySpark

quickstart= spark.read.csv("/databricks-datasets/data.csv", header="true", inferSchema="true")

Analyse Data

Group and Display

from pyspark.sql.functions import avg

display(quickstart.select("color","price").groupBy("color").agg(avg("price")).sort("color"))

16Apr

Scripts

Parameter in a Powershell Script

param (
    [Parameter(Mandatory=$true)]  [string] $folder = "",
    [Parameter(Mandatory=$false)] [string] $type,
    [Parameter(Mandatory=$false)] [switch] $migrate,
    [Parameter(Mandatory=$false)] [switch] $help
)

Call the Script

.\script.ps1 foldername
.\script.ps1 foldername -type=folder

.\script.ps1 foldername -type=folder -migrate

Parameter Debug and Verbose

This parameter could not be defined in a script, because they are already present.

param (
    [Parameter(Mandatory=$false)] [switch] $debug
)

.\using_parameter.ps1 : Ein Parameter mit dem Namen "Debug" wurde mehrfach für den Befehl definiert.
In Zeile:1 Zeichen:1
+ .\using_parameter.ps1
+ ~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : MetadataError: (:) [], MetadataException
    + FullyQualifiedErrorId : ParameterNameAlreadyExistsForCommand

To access their values, use

param (
    [Parameter(Mandatory=$false)] [switch] $help
)

$debug   = $PSBoundParameters.ContainsKey('Debug')
$verbose = $PSBoundParameters.ContainsKey('Verbose')

Write-Host "help    = $help"
Write-Host "debug   = $debug"
Write-Host "verbose = $verbose"

> .\using_parameter.ps1  -debug -help
help    = True
debug   = True
verbose = False
> .\using_parameter.ps1  -debug 
help    = False
debug   = True
verbose = False
> .\using_parameter.ps1  -debug -verbose
help    = False
debug   = True
verbose = True

String

Converting String to TitleCase

function toTitleCase($string) {
    return $string.substring(0,1).toupper()+$string.substring(1).tolower()

}

Filesystem

List of Files in a Folder

Command	Description
`Get-ChildItem`	Folders and Files
`Get-ChildItem -Directory`	Only Folders
`Get-ChildItem -File`	Only Files
`Get-ChildItem -Recurse -Directory`	Only Folders, Recursive
`Get-ChildItem -Recurse -File`	Only Files, Recursive

(ls -r *.txt).fullname

Get-ChildItem -recurse -Filter .editorconfig -path . | 



dir -Path . -Filter ProfileInformationController* -Recurse | 



Counting Files



Command Description
(Get-ChildItem | Measure-Object).Count Count the files and folders
(Get-ChildItem -Directory | Measure-Object).Count Only Folders
(Get-ChildItem -File | Measure-Object).Count Only Files
(Get-ChildItem -Recurse -Directory | Measure-Object).Count Only Folders, Recursive
(Get-ChildItem -Recurse -File | Measure-Object).Count Only Files, Recursive



Using  Scripting.FileSystemObject



$objFSO = New-Object -com  Scripting.FileSystemObject
$objFSO.GetFolder($folder).Files.Count
Find a  Folders within a Folder (not recursive)
(Get-ChildItem --Attributes Directory).Name
Run Command for each Folder
(Get-ChildItem  -Attributes Directory).Name | 



With Enumerator



[IO.Directory]::EnumerateFiles($folder) | ForEach-Object {
    Write-Host $_
}
Find all folders with Pattern
Get-ChildItem -recurse -depth 2 -directory -path ..\packages_flutter | `
Where-Object { $_.FullName -match 'example' }                        | `





Delete files with pattern



Get-ChildItem *.code -recurse | foreach { Remove-Item -Path $_.FullName }
Get-ChildItem -Path C:Temp -Include *.* -File -Recurse | foreach { $_.Delete()}
Files
Filename and Extension of File with Split-Path
Filename
Split-Path -Path <PATH> -Leaf
Folder
Split-Path -Path <PATH>
Extension
Split-Path -Path <PATH> -Extension
Create File
New-Item new.txt -type file
Loop through entries of a textfile
Get-Content list-of-folders | Where-Object {$_ -NotMatch "#.*" } | ForEach-Object { 
    .\check_folder.ps1$_
    # Write-Host "Weiter -->" -NoNewLine
    # $key = $Host.UI.RawUI.ReadKey()
}
Loop through results from Select-String
Here is an example that greps for a string and uses the results in a loop to determine if some action should be taken
$pattern = "powershell"
$files = Select-String -Path "d:\script\*.txt" -Pattern $pattern
foreach ($file in $files) {
   $filename=$file.Filename
   $item = get-item $file.Path
    "File '{0}' matches the pattern '{1}'" -f $item.FullName, $pattern
    "It was last modified on {0}" -f $item.LastWriteTime
 
   $response = Read-Host -Prompt "Set the archive bit on this file?" 
   If ($response -eq 'Y') {
      $item.attributes="Archive"
   }
}
Check if a file exist
Test-Path $PROFILE
Searching in Files
Get-ChildItem -Recurse | Select-String "dummy" -List | Select Path
Get-ChildItem -Recurse *.sql | Select-String "create .*_tab_" | Select-Object -Unique Path
Select-String -path *.txt -pattern PowerShell

Select-String -path *.txt -pattern PowerShell -notmatch
Parsing Files
Get first line of output
$eventResult.Split([Environment]::NewLine) | Select -First 1
Web
Download Web-Page
Invoke-WebRequest -Uri <link>
List all Links
$response = Invoke-WebRequest -Uri <link>
$response.links | Select -Expand href
List all Links with filtering href by RegEx
$response.Links | Where-Object {$_.href -like "*videos*" } | ForEach-Object { $_.href }
List all Links with filtering class name by RegEx
$response.Links | Where-Object {$_.class -eq "page-numbers"} | Format-List innerText, href
Download and Install Visual Studio Code in portable Mode
$FOLDER=Get-Date -Format "yyyy-MM-dd-HH-mm"

Write-Host "Create link for folder $FOLDER"


# Download
# https://code.visualstudio.com/sha/download?build=stable&os=win32-x64-archive
# https://code.visualstudio.com/sha/download?build=insider&os=win32-x64-archive

$LINK="https://code.visualstudio.com/sha/download?build=insider&os=win32-x64-archive"
$FILE="vscode-insider.zip"

if (Test-Path "$FILE") {
	Remove-Item "$FILE"
}

Invoke-WebRequest "$LINK" -OutFile "$FILE"

Expand-Archive "$FILE" "$FOLDER"

if (Test-Path $FOLDER\data)
{
	Remove-Item $FOLDER\data
}


if (Test-Path code) { Remove-Item code }

# Using junction from SysInternalsSuite to create symlinks

junction code $FOLDER
junction code\data data
Environment
Show env variables
gci env:* | Sort-Object Name
Show env variables with name pattern
gci env: | Where name -like '*HOME
Processes
List Processes and Path
Get-Process | Select-Object Path
Show processes using a specific port
Get-Process -Id (Get-NetTCPConnection -LocalPort YourPortNumberHere).OwningProcess
Network
Pipeline
Parse out from command
$REPOSITORY=<URL of Repository>
git branch -r | ForEach-Object { Write-Output "git clone -b $_ $REPOSITORY $_" } | Out-File -FilePath .clone-all-branches
Permissions
Show current policy
Get-ExecutionPolicy
Allow custom scripts to execute
Set-ExecutionPolicy -Scope CurrentUser unrestricted
Security
Self-Sign a script
Step 1: Create your code signing certificate
New-SelfSignedCertificate -DnsName user@via-internet.de `
                          -CertStoreLocation Cert:\currentuser\my  `
                          -Subject "CN=Local Code Signing"  `
                          -KeyAlgorithm RSA  `
                          -KeyLength 2048  `
                          -Provider "Microsoft Enhanced RSA and AES Cryptographic Provider"  `
                          -KeyExportPolicy Exportable  `
                          -KeyUsage DigitalSignature  `
                          -Type CodeSigningCert
Step 2: Open the Certificate Manager for Current User
certmgr /s my
Step 3: Copy the new certificate to the appropriate cert stores
From Personal folder into Trusted Root Certification Authorities and into Trusted Publishers stores.
German: Von Eigene Zertifikate nach Vertrauenswürdige Stammzertifizierungsstellen und Vertrauenswürdige Herausgeber
Step 4: Sign your PowerShell script with the new cert
$CERT=@(Get-ChildItem cert:\CurrentUser\My -CodeSigning)[1]
Set-AuthenticodeSignature .\HelloWorld.ps1 $CERT
Or
❯ Set-AuthenticodeSignature -FilePath .\HelloWorld.ps1 -Certificate (Get-ChildItem -Path Cert:CurrentUserMy -CodeSigningCert)
Final Check
❯ Get-AuthenticodeSignature .\HelloWorld.ps1
Github
Download Repositories
gh repo list <github username> --limit 1000 | 



Profiles



Different PowerShell profiles



Description Path
Current User, Current Host – console $Home[My ]DocumentsWindowsPowerShellProfile.ps1
Current User, All Hosts    $Home[My ]DocumentsProfile.ps1
All Users, Current Host – console    $PsHomeMicrosoft.PowerShell_profile.ps1
All Users, All Hosts       $PsHomeProfile.ps1
Current user, Current Host – ISE $Home[My ]DocumentsWindowsPowerShellMicrosoft.P owerShellISE_profile.ps1
 All users, Current Host – ISE   $PsHomeMicrosoft.PowerShellISE_profile.ps1



Show Path for all profiles



$PROFILE | Format-List * -Force
Create a new profile
New-Item $PROFILE.CurrentUserAllHosts -ItemType file -Force
Customizing
Theming
https://ohmyposh.dev/
Install Posh-Git and Oh-My-Posh.
Install-Module posh-git   -Scope CurrentUser
Install-Module oh-my-posh -Scope CurrentUser
Install-Module -Name PSReadLine -AllowPrerelease -Scope CurrentUser -Force -SkipPublisherCheck
Then run “notepad $PROFILE” and add these lines to the end:
Import-Module posh-git
Import-Module oh-my-posh

Set-Theme Paradox
Set a custom theme
Import-Module posh-git
Import-Module oh-my-posh
Set-Theme Paradox
Show current theme settings
$ThemeSettings
$ThemeSettings.CurrentThemeLocation
Customize Prompt
Show current Path
function prompt
{
    "PS " + $(get-location) + "> "
}
Randor Color
function prompt
{
    $random = new-object random
    $color=[System.ConsoleColor]$random.next(1,16)
    Write-Host ("PS " + $(get-location) +">") -nonewline -foregroundcolor $color
    return " "
}
Display current time at the end of prompt line (this will mess up you console buffer)
function prompt
{
    $oldposition = $host.ui.rawui.CursorPosition
    $Endline = $oldposition
    $Endline.X+=60
    $host.ui.rawui.CursorPosition = $Endline
    Write-Host $(get-date).Tostring("yyyy-MM-dd HH:mm:ss")
    $host.ui.rawui.CursorPosition = $oldposition
    Write-Host ("PS " + $(get-location) +">") -nonewline -foregroundcolor Magenta
    return " "
}
Show current user, host, current line number
$global:CurrentUser = [System.Security.Principal.WindowsIdentity]::GetCurrent()
function prompt
{
    $host.ui.rawui.WindowTitle = $CurrentUser.Name + " " + $Host.Name + " " + $Host.Version + " Line: " + $host.UI.RawUI.CursorPosition.Y
    Write-Host ("PS " + $(get-location) +">") -nonewline -foregroundcolor Magenta
    return " "
}
Weitere Anpassungsmöglichkeiten
https://www.norlunn.net/2019/10/07/powershell-customize-the-prompt/
Security
❯ New-SelfSignedCertificate -DnsName user@via-internet.de -CertStoreLocation Cert:CurrentUserMy -Type CodeSigning

   PSParentPath: Microsoft.PowerShell.SecurityCertificate::CurrentUserMy

Thumbprint                                Subject              EnhancedKeyUsageList
----------                                -------              --------------------
4AED871E6DB5FF3E85EB1625C5369DBDB3E120FD  CN=user@via-interne… Codesignatur
❯ Set-AuthenticodeSignature -FilePath demo.ps1 -Certificate (Get-ChildItem -Path Cert:CurrentUserMy -CodeSigningCert)

    Directory: D:TMP

SignerCertificate                         Status                               StatusMessage                     Path
-----------------                         ------                               -------------                     ----
4AED871E6DB5FF3E85EB1625C5369DBDB3E120FD  Valid                                Signature verified.               demo.ps1
Final Check
❯ Get-AuthenticodeSignature .demo.ps1

    Directory: D:CLOUDEnvironmentsKeycloakKeycloak12.0.1bin

SignerCertificate                         Status                                StatusMessage                    Path
-----------------                         ------                                -------------                    ---
4AED871E6DB5FF3E85EB1625C5369DBDB3E120FD  Valid                                 Signature verified.              demo.ps1
From Bash to Powershell
https://mathieubuisson.github.io/powershell-linux-bash/
Alias for WHICH command
❯ (get-command FILE.EXE).Path
❯ Set-Alias where Get-Command
❯ where FILE.EXE
Snippets
Formatting
0..31 | ForEach-Object { 
    "{0,6}`t{1,6}`t{2,5}`t0x{0:X4}" -f $_,[Convert]::ToString($_,2), [Convert]::ToString($_,8) 
}
Select a Service By Name (Partial Match)
> Get-Service | Select-Object -ExpandProperty Name | Select-String -pattern 'ws'
CURL For Powershell
> (Invoke-WebRequest www.google.com).Content
List Environment Variables
> gci env: | sort name
Get PS Version
> $PSVersionTable
List Installed Programs
> Get-WmiObject -Class Win32_Product
Colored Header in Output
$prefix ="$prefix                   ".Substring(0,6)
$title  ="$title                    ".Substring(0,15)

$line = "$prefix ${title}: ${line}"
$fill = " " * ([Console]::WindowWidth - $line.length)
Write-Host "${line}${fill}"  -ForegroundColor White -BackgroundColor Blue -NoNewline

Command	Description
`(Get-ChildItem \| Measure-Object).Count`	Count the files and folders
(`Get-ChildItem -Directory` \| Measure-Object).Count	Only Folders
(`Get-ChildItem -File` \| Measure-Object).Count	Only Files
(`Get-ChildItem -Recurse -Directory` \| Measure-Object).Count	Only Folders, Recursive
(`Get-ChildItem -Recurse -File` \| Measure-Object).Count	Only Files, Recursive

31Mar

PySpark | Cookbook

by Ralph Apache Spark, Cookbook, PySpark

Websites

The Blaze Ecosystem (Blaze)
Dask: Flexible library for parallel computing in Python.
DataShape: Data layout language for array programming.
Odo: Shapeshifting for your data
It efficiently migrates data from the source to the target through a network of conversions.

Reading Textfiles

from pyspark.sql import SparkSession

spark = SparkSession \
    .builder \
    .appName("Demo") \
    .config("spark.demo.config.option", "demo-value") \
    .getOrCreate()

df = spark.read.csv("input.csv",header=True,sep="|");

from pyspark import SparkContext
from pyspark.sql import SQLContext
import pandas as pd

sc = SparkContext('local','example') 
sql_sc = SQLContext(sc)

p_df= pd.read_csv('file.csv')  # assuming the file contains a header
# p_df = pd.read_csv('file.csv', names = ['column 1','column 2']) # if no header
s_df = sql_sc.createDataFrame(pandas_df)

sc.textFile("file.csv") \
    .map(lambda line: line.split(",")) \
    .filter(lambda line: len(line)>1) \
    .map(lambda line: (line[0],line[1])) \
    .collect()

sc.textFile("input.csv") \
    .map(lambda line: line.split(",")) \
    .filter(lambda line: len(line)<=1) \
    .collect()

spark.read.csv(
    "input.csv", header=True, mode="DROPMALFORMED", schema=schema
)

(spark.read
    .schema(schema)
    .option("header", "true")
    .option("mode", "DROPMALFORMED")
    .csv("input.csv"))

Read CSV file with known structure

from pyspark.sql.types import StructType, StructField
from pyspark.sql.types import DoubleType, IntegerType, StringType

schema = StructType([
    StructField("A", IntegerType()),
    StructField("B", DoubleType()),
    StructField("C", StringType())
])

(sqlContext
    .read
    .format("com.databricks.spark.csv")
    .schema(schema)
    .option("header", "true")
    .option("mode", "DROPMALFORMED")
    .load("input.csv"))

2Mar

Jenkins | Build and Deploy a Groovy App

by Ralph CI/CD, Groovy, Jenkins

Introduction

Using Jenkins as an automation server for your development, you can automate such repeating tasks as testing and deploying your app.

Starting with a sample Groovy App (a simple calculator) with tests, you will learn how to integrate your app in Jenkins and build a pipeline, so that Jenkins runs the desired tasks every time, you change the code.

Prepare the sources

Clone the sample repository from Github.

You should clone the demo repository into you demo account, because you may change some file during this post., and you will not get write permissions for the demo repository.

Also, clone the repository to your local machine to see what our demo app looks like.

$ git clone https://github.com/jenkins-toolbox/SampleApp_GroovyCalculator Cloning into 'SampleApp_GroovyCalculator'... remote: Enumerating objects: 194, done. remote: Counting objects: 100 remote: Compressing objects: 100 remote: Total 194 (delta 44), reused 137 (delta 23), pack-reused 0 Receiving objects: 100 Resolving deltas: 100

Go into the new create folder

$ cd SampleApp_GroovyCalculator/
$ ls
Jenkinsfile      README.md        bin              build.gradle     gradlew          src
Makefile         SampleCalculator build            gradle           settings.gradle

The first task, Jenkins will do in our pipeline: build your app

$ ./gradlew build

Because it’s the first time you start gradlew, the required software will be downloaded:

First: the current Gradle Version (Gradle is the Build Tool used by Groovy Projects)

Downloading https://services.gradle.org/distributions/gradle-6.2.1-bin.zip
………10

Welcome to Gradle 6.2.1!

Here are the highlights of this release:
 - Dependency checksum and signature verification
 - Shareable read-only dependency cache
 - Documentation links in deprecation messages

For more details see https://docs.gradle.org/6.2.1/release-notes.html

Starting a Gradle Daemon, 2 stopped Daemons could not be reused, use --status for details

After this, your app will be tested

> Task :test

Calculator02Spec > two plus two should equal four PASSED

Calculator01Spec > add: 2 + 3 PASSED

Calculator01Spec > subtract: 4 - 3 PASSED

Calculator01Spec > multiply: 2 * 3 PASSED

BUILD SUCCESSFUL in 34s
5 actionable tasks: 5 executed

Perform the build again

No download is required. The build is much quicker.

$ ./gradlew build

BUILD SUCCESSFUL in 1s
5 actionable tasks: 5 up-to-date

Now, test our app:

./gradlew clean test

> Task :test

Calculator02Spec > two plus two should equal four PASSED

Calculator01Spec > add: 2 + 3 PASSED

Calculator01Spec > subtract: 4 - 3 PASSED

Calculator01Spec > multiply: 2 * 3 PASSED

BUILD SUCCESSFUL in 4s
5 actionable tasks: 5 executed

Create a Jenkins Pipeline

Start by clicking on the BlueOcean menu item.

Hint: Blue Ocean is not installed with the default Jenkins installation.

You have to install the corresponding Plugins.

Select Manage Jenkins → Manage Plugins.

Then, select the tab Available and enter in the Filter box: Blue Ocean.

Install all plugins, that will be listed.

Next: Click on the New Pipeline to create your first Pipeline

Use the Item GitHub to specify, where our code is stored

Next, use your GitHub account.

Be sure, that you cloned the demo repository

Next, we select the demo repository SampleApp_GroovyCalculator

Click on Create Pipeline and after a few seconds, the pipeline is created.

Immediately after creating the pipeline, Jenkins is starting the pipeline and all steps included.

If everything went well, you see a positive status

Now, click on the pipeline (e.g. the text master or the status icon) and you will see the pipeline with all steps and their corresponding state.

If you, want to edit the pipeline, for example to add another step, like on the pencil in the header.

Click on Cancel to leave the Pipeline editor.

Hint: If you click on Save, all changes are pushed back to the repository and Jenkins starts the Pipeline again.

Run the Pipeline

If you want to run your pipeline, click on the rerun icon for your pipeline

2Mar

Jenkins | Cookbook

by Ralph CI/CD, Cookbook, Jenkins

Working with VS Code

Validate Jenkins File

Install VS Code Plugin Jenkins Pipeline Linter Connector

Add configuration in .vscode/settings.json

"jenkins.pipeline.linter.connector.crumbUrl": "<JENKINS_URL>/crumbIssuer/api/xml?xpath=concat(//crumbRequestField,
"jenkins.pipeline.linter.connector.user": "<USERNAME>",
"jenkins.pipeline.linter.connector.pass": "<PASSWORD>",
"jenkins.pipeline.linter.connector.url": "<JENKINS_URL>/pipeline-model-converter/validate",

Replace <USERNAME>, <PASSWORD> and <JENKINS_URL> with your values, for example

"jenkins.pipeline.linter.connector.crumbUrl": "http://localhost:8080/crumbIssuer/api/xml?xpath=concat(//crumbRequestField,%22:%22,//crumb)",
"jenkins.pipeline.linter.connector.user": "admin",
"jenkins.pipeline.linter.connector.pass": "secret",
"jenkins.pipeline.linter.connector.url": "http://localhost:8080/pipeline-model-converter/validate",

Working with Jenkins Client (CLI)

Download Client

wget localhost:8080/jnlpJars/jenkins-cli.jar

Working with Plugins

Create aPlugin

mkdir SamplePlugin
cd SamplePlugin

mvn -U archetype:generate -Dfilter="io.jenkins.archetypes:"

mvn -U archetype:generate -Dfilter="io.jenkins.archetypes:global-configuration-plugin"
[INFO] Scanning for projects...
Downloading from central: https://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-metadata.xml
Downloading from central: https://repo.maven.apache.org/maven2/org/codehaus/mojo/maven-metadata.xml
Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-metadata.xml (14 kB at 32 kB/s)
Downloaded from central: https://repo.maven.apache.org/maven2/org/codehaus/mojo/maven-metadata.xml (20 kB at 44 kB/s)
Downloading from central: https://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-archetype-plugin/maven-metadata.xml
Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-archetype-plugin/maven-metadata.xml (918 B at 18 kB/s)
[INFO]
[INFO] ------------------< org.apache.maven:standalone-pom >-------------------
[INFO] Building Maven Stub Project (No POM) 1
[INFO] --------------------------------[ pom ]---------------------------------
[INFO]
[INFO] >>> maven-archetype-plugin:3.1.2:generate (default-cli) > generate-sources @ standalone-pom >>>
[INFO]
[INFO] <<< maven-archetype-plugin:3.1.2:generate (default-cli) < generate-sources @ standalone-pom <<<
[INFO]
[INFO]
[INFO] --- maven-archetype-plugin:3.1.2:generate (default-cli) @ standalone-pom ---
[INFO] Generating project in Interactive mode
[INFO] No archetype defined. Using maven-archetype-quickstart (org.apache.maven.archetypes:maven-archetype-quickstart:1.0)

Choose archetype:
1: remote -> io.jenkins.archetypes:global-configuration-plugin (Skeleton of a Jenkins plugin with a POM and an example piece of global configuration.)
Choose a number or apply filter (format: [groupId:]artifactId, case sensitive contains): : 1
Choose io.jenkins.archetypes:global-configuration-plugin version:
1: 1.2
2: 1.3
3: 1.4
4: 1.5
5: 1.6
Choose a number: 5:
[INFO] Using property: groupId = unused
Define value for property 'artifactId': com.examples.jenkins.plugins
Define value for property 'version' 1.0-SNAPSHOT: :
[INFO] Using property: package = io.jenkins.plugins.sample
Confirm properties configuration:
groupId: unused
artifactId: com.examples.jenkins.plugins
version: 1.0-SNAPSHOT
package: io.jenkins.plugins.sample
 Y: : y

[INFO] ----------------------------------------------------------------------------
[INFO] Using following parameters for creating project from Archetype: global-configuration-plugin:1.6
[INFO] ----------------------------------------------------------------------------
[INFO] Parameter: groupId, Value: unused
[INFO] Parameter: artifactId, Value: com.examples.jenkins.plugins
[INFO] Parameter: version, Value: 1.0-SNAPSHOT
[INFO] Parameter: package, Value: io.jenkins.plugins.sample
[INFO] Parameter: packageInPathFormat, Value: io/jenkins/plugins/sample
[INFO] Parameter: version, Value: 1.0-SNAPSHOT
[INFO] Parameter: package, Value: io.jenkins.plugins.sample
[INFO] Parameter: groupId, Value: unused
[INFO] Parameter: artifactId, Value: com.examples.jenkins.plugins
[INFO] Project created from Archetype in dir: /Users/Shared/CLOUD/Kunde.BSH/workspace/SamplePlugin_Config/com.examples.jenkins.plugins
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  45.525 s
[INFO] Finished at: 2020-03-01T17:28:27+01:00
[INFO] ------------------------------------------------------------------------

Verify Plugin

 cd com.examples.jenkins.plugins
mvn verify

Run Plugin

mvn hpi:run

Working with Groovy Scripts

Include a common groovy script in Jenkins file

1: Create a common.groovy file with function as needed

def mycommoncode() {
}

2: In the main Jenkinfile load the file and use the function as shown below

node{ 
   def common = load “common.groovy”
   common.mycommoncode()
}

Basic example of Loading Groovy scripts

File example.groovy

def example1() {
 println 'Hello from example1' 
}
def example2() {
 println 'Hello from example2'
}

The example.groovy script defines example1 and example2 functions before ending with return this. Note that return this is definitely required and one common mistake is to forget ending the Groovy script with it.Jenkinsfile

def code node('java-agent') { 
    stage('Checkout') { checkout scm } 
    stage('Load') { code = load 'example.groovy' } 
    stage('Execute') { code.example1() }
} 

code.example2()

Processing Github JSON from Groovy

In this demo, we first show how to process JSON response from Github API in Groovy.Processing JSON from Github

String username = System.getenv('GITHUB_USERNAME') 
String password = System.getenv('GITHUB_PASSWORD') 
String GITHUB_API = 'https://api.github.com/repos' String repo = 'groovy' 
String PR_ID = '2' // Pull request ID 
String url = "${GITHUB_API}/${username}/${repo}/pulls/${PR_ID}" 

println "Querying ${url}" 

def text = url.toURL().getText(requestProperties: ['Authorization': "token ${password}"]) 
def json = new JsonSlurper().parseText(text) 

def bodyText = json.body // Check if Pull Request body has certain text if ( bodyText.find('Safari') ) {
     println 'Found Safari user' }

The equivalent bash command for retrieving JSON response from Github API is as follows:Equivalent bash command

// Groovy formatted string 
String cmd = "curl -s -H \"Authorization: token ${password}\" ${url}" 
// Example String 
example = 'curl -s -H "Authorization: token XXX" https://api.github.com/repos/tdongsi/groovy/pulls/2'

Processing Github JSON from Jenkinsfile

Continuing the demo from the last section, we now put the Groovy code into a callable function in a script called “github.groovy”. Then, in our Jenkinsfile, we proceed to load the script and use the function to process JSON response from Github API.github.groovy

import groovy.json.JsonSlurper
def getPrBody(String githubUsername, String githubToken, String repo, String id) {
   String GITHUB_API = 'https://api.github.com/repos' 
   String url = "${GITHUB_API}/${githubUsername}/${repo}/pulls/${id}" 

   println "Querying ${url}"

def text = url.toURL().getText(requestProperties: ['Authorization': "token ${githubToken}"])
def json = new JsonSlurper().parseText(text) 
def bodyText = json.body return bodyText } 
return this

Jenkinsfile

def code node('java-agent') { 
    stage('Checkout') { checkout scm }
    stage('Load') { code = load 'github.groovy' } 
    stage('Execute') {

Best Practice

23Feb

Django | Build a Dashboard with Django and Bootstrap: Part 3

by Ralph Bootstrap, Django

This is Part 3 of Building a Dashboard with Django and Bootstrap:

Part 1: Building a base Django project
Part 2: Prepare for dynamic content
Part 3: Handling navigation and the side menu
Part 4: Deploy Django App to Azure

Introduction

If you follow the first part of this blog topic, you have a running Django dashboard.

But, the content ist still static. Lets review the current state:

Prepare our Django project

Right now, the whole content of our Django project is provided by the dashboard template

dashboard/template/site/base.html

Looking at our web site, you will see the different side menu items. So, intentionally, our web site should display different pages. And each page should provide the dynamic content.

The final goal of this part is to change our web app, so that each side item navigates us to a different page. For this, we have to take care about two things:

Navigation: how to we get to another page in our app
Project Structure: where to place the required components for each page

Basics of Navigation

Navigation usually is the process of getting from one page to another by clicking on a link.

So, we need to things:

the source page, containing the link
the destination page
the link, pointing to the destination page

Let’s take a look into the site template with the side menu:

The corresponding code in the side template is

<div id="collapseTwo" class="collapse" aria-labelledby="headingTwo">
	<div class="bg-white py-2 collapse-inner rounded">
		<h6 class="collapse-header">Custom Components:</h6>
		<a class="collapse-item" href="buttons.html">Buttons</a>
		<a class="collapse-item" href="cards.html">Cards</a>
	</div>
</div>

This is the result:

Linking to a html page is not possible, because Django does not work with html pages. Navigation in Django works with urls (in urls.py) and views in (views.py).

We must replace the html link tag (<a href="buttons.html">) with an Django-conform code. Read here for more details and the basics.

The idea behind the navigation is

Description	Path
Current User, Current Host – console	$Home[My ]DocumentsWindowsPowerShellProfile.ps1
Current User, All Hosts	$Home[My ]DocumentsProfile.ps1
All Users, Current Host – console	$PsHomeMicrosoft.PowerShell_profile.ps1
All Users, All Hosts	$PsHomeProfile.ps1
Current user, Current Host – ISE	$Home[My ]DocumentsWindowsPowerShellMicrosoft.P owerShellISE_profile.ps1
All users, Current Host – ISE	$PsHomeMicrosoft.PowerShellISE_profile.ps1