Daily: Download files from Website with PowerShell

Introduction

Today, i followed my guidelines to learn/refresh one programming language every month. So, i decided to refresh my Prolog knowledge.

The first step in doing this was to set up a development environment. With VS Code and Docker, this was easy. Read here to see how i did this. (I will write this post in the next few days, stay tune :))

The next step was getting some examples. Searching the internet brings me to this interesting site.

Because i want to examine the examples and learn some prolog and not doing a lot of downloads, i decided to write a small script to download all files.

This time, i implemented the script im PowerShell.

The Script

Here is the final script download_files.ps1:

<pre class="EnlighterJSRAW" data-enlighter-group="" data-enlighter-highlight="" data-enlighter-language="powershell" data-enlighter-linenumbers="" data-enlighter-lineoffset="" data-enlighter-theme="" data-enlighter-title="">$ROOT="https://www.ic.unicamp.br/~meidanis/courses/mc336/2009s2/prolog/problemas"

$LINKS=(Invoke-WebRequest $ROOT).Links

$LINKS                                  | `
Where-Object {$_.href -like "*.pl"}     | `
% {
    $LINK=$_.href
    Start-BitsTransfer -Source ${ROOT}/${LINK} -Destination ${LINK}
}

The Explanation

First, i define the starting link

<pre class="EnlighterJSRAW" data-enlighter-group="" data-enlighter-highlight="" data-enlighter-language="generic" data-enlighter-linenumbers="" data-enlighter-lineoffset="" data-enlighter-theme="" data-enlighter-title="">$ROOT="https://www.ic.unicamp.br/~meidanis/courses/mc336/2009s2/prolog/problemas"

Downloading the webpage and extract all the links

Next step is downloading the webpage and extract all the links. the result (list of all links) will be stored in the variable $LINKS

<pre class="EnlighterJSRAW" data-enlighter-group="" data-enlighter-highlight="" data-enlighter-language="generic" data-enlighter-linenumbers="" data-enlighter-lineoffset="" data-enlighter-theme="" data-enlighter-title="">$LINKS=(Invoke-WebRequest $ROOT).Links

Look at two parts of the command: Download and Extraction.

If we separate the two steps, we should write

<pre class="EnlighterJSRAW" data-enlighter-group="" data-enlighter-highlight="" data-enlighter-language="generic" data-enlighter-linenumbers="" data-enlighter-lineoffset="" data-enlighter-theme="" data-enlighter-title="">$PAGE=Invoke-WebRequest $ROOT

<pre class="EnlighterJSRAW" data-enlighter-group="" data-enlighter-highlight="" data-enlighter-language="generic" data-enlighter-linenumbers="" data-enlighter-lineoffset="" data-enlighter-theme="" data-enlighter-title="">$LINKS=$PAGE.Links

Extracting the Destination of the link

Next step is extracting the HREF (Destination of the link) and

<pre class="EnlighterJSRAW" data-enlighter-group="" data-enlighter-highlight="" data-enlighter-language="generic" data-enlighter-linenumbers="" data-enlighter-lineoffset="" data-enlighter-theme="" data-enlighter-title="">Where-Object {$_.href -like "*.pl"}

Initiation of the download of the example file

And the last step is, for each HREF initiation the download of the link (means download the example file)

<pre class="EnlighterJSRAW" data-enlighter-group="" data-enlighter-highlight="" data-enlighter-language="generic" data-enlighter-linenumbers="" data-enlighter-lineoffset="" data-enlighter-theme="" data-enlighter-title="">% {
    $LINK=$_.href
    Start-BitsTransfer -Source ${ROOT}/${LINK} -Destination ${LINK}
}