POWERSHELL FTP MODULE IN A PSAKE BUILD PROCESS

In my effort to learn more about PowerShell I wrote a simple ftp module. I wrote the module primary for using it in a automated deployment process with Psake, GitHub, Jenkins and a target environment with ftp access. I looked around to find a similar module and the best match I found was this script so the module is inspired a lot by that script.

The module publish four methods that does what I needed the module to do for my deployment process so its not really a generic ftp module but I works for some scenarios.

  • Set-FtpConnection
  • Send-ToFtp
  • Get-FromFtp
  • Remove-FromFtp

The build process that I needed to support looks like this. This post focus on the last steps; 6 and 7. Step 1 – 5 requires one or more individual posts.

  1. Setup (clean up last release, create deployment and backup folders)
  2. Compile (.net assemblies, CoffeeSscript to JavaScript and Less to CSS + minifying)
  3. Run unit tests and break deploy if it fails (MSpec)
  4. Copy needed files for running the application at different target environments
  5. Merge/create configurations for different target environments
  6. Backup the current application from the target environment
    • Creating a ftp connection to the target environment
    • Download the complete deployed application from the web root to the backup folders that was created at setup (step 1)
    • Upload the backup to a backup folder at the ftp server
  7. Redeploy the application
    • Creating a ftp connection to the target environment
    • Drop the web root
    • Upload the new release from the deployment folder created at setup (step 1)

The build script (based on psake) looks like this. I have removed code and properties that is not related to this post. If you want to se the complete script you can find it here

import-module .\utilities.psm1  
import-module .\ftp.psm1

properties {  
    $TargetEnvironment = 'Debug'
    $DateLabel = ([DateTime]::Now.ToString("yyyy-MM-dd_HH-mm-ss"))
    $ApplicationBackupRoot = "..\..\Deploy\Backup"
    $ApplicationBackupRootWithDateLabel = "..\..\Deploy\Backup\$DateLabel"
    $BuildOutputDestinationRoot = "..\..\Deploy\Build"

    $StagingFtpUri = 'ftp://127.0.0.1:55/'
    $StagingFtpWwwRoot = "www"
    $StagingFtpBackupRoot = "backup"
    $StagingFtpUsername = 'anton'
    $StagingFtpPassword = 'anton'

    #....
    #code is removed to not loose focus from this post
    #....
}

task Default -depends CopyFiles

task Staging -depends DeployWebToStagingFtp

task DeployWebToStagingFtp -depends BackupWebAtStagingFtp {  
    $fullBuildOutputDestinationRoot = Resolve-Path $BuildOutputDestinationRoot
    Set-FtpConnection $StagingFtpUri $StagingFtpUsername $StagingFtpPassword
    Remove-FromFtp $StagingFtpWwwRoot
    Send-ToFtp $fullBuildOutputDestinationRoot $StagingFtpWwwRoot
}

task BackupWebAtStagingFtp -depends MergeConfiguration {  
    $fullSourcePath = Resolve-Path $ApplicationBackupRootWithDateLabel
    $fullApplicationBackupRootPath = Resolve-Path $ApplicationBackupRoot
    Set-FtpConnection $StagingFtpUri $StagingFtpUsername $StagingFtpPassword
    Get-FromFtp $fullSourcePath $StagingFtpWwwRoot
    Send-ToFtp $fullApplicationBackupRootPath $StagingFtpBackupRoot
}

task MergeConfiguration -depends CopyFiles {  
    #....
    #code is removed to not loose focus from this post
    #....
}

task CopyFiles -depends Test {  
    #....
    #code is removed to not loose focus from this post
    #....
}

task Test -depends Compile, Setup {  
    #....
    #code is removed to not loose focus from this post
    #....
}

task Compile -depends Setup {  
    #....
    #code is removed to not loose focus from this post
    #....
}

task Setup {  
    Add-FolderIfMissing $BuildOutputDestinationRoot
    if (!($TargetEnvironment -ieq 'debug')) {
        Add-FolderIfMissing $ApplicationBackupRoot
        Add-FolderIfMissing $ApplicationBackupRootWithDateLabel
    }
}

As you can see this version of the script only supports deploys to the staging environment but we could quite easily add support to other environments too. The code shows how the ftp module is used in a psake build script. Standalone in a PowerShell session it would something like this.

PS C:\Projects\MvcCiTest\src\buildScripts> Set-ExecutionPolicy RemoteSigned

Execution Policy Change  
The execution policy helps protect you from scripts that you do not trust. Changing the execution policy might expose  
you to the security risks described in the about_Execution_Policies help topic. Do you want to change the execution  
policy?  
[Y] Yes  [N] No  [S] Suspend  [?] Help (default is "Y"):
PS C:\Projects\MvcCiTest\src\buildScripts> Import-Module .\ftp.psm1  
PS C:\Projects\MvcCiTest\src\buildScripts> Set-FtpConnection "ftp://127.0.0.1:21/" "anton" "anton"  
PS C:\Projects\MvcCiTest\src\buildScripts> Send-ToFtp "C:\Users\Anton\Pictures\" "www/pictures"  
uploading cheetah-picture.jpg  
PS C:\Projects\MvcCiTest\src\buildScripts> Get-FromFtp "C:\temp" "www/pictures"  
Downloading ftp://127.0.0.1:21/www/pictures/heetah-picture.jpg ...  
PS C:\Projects\MvcCiTest\src\buildScripts> Remove-FromFtp "www/pictures"  
 deleting ...
Delete status: 250 File deleted successfully

PS C:\Projects\MvcCiTest\src\buildScripts> Remove-Module ftp  

So finally… the first version of my ftp module looks like this. Would appreciate any suggestions on how to improve the module the code is available at GitHub feel free to send me a pull request :)

[string]$script:ftpHost
[string]$script:username
[string]$script:password
[System.Net.NetworkCredential]$script:Credentials

function Set-FtpConnection {  
    param([string]$host, [string]$username, [string]$password)

    $script:Credentials = New-Object System.Net.NetworkCredential($username, $password)
    $script:ftpHost  = $host
    $script:username = $username
    $script:password = $password
}

function Send-ToFtp {  
    param([string]$sourcePath, [string]$ftpFolder)

    foreach($item in Get-ChildItem -recurse $sourcePath){
        $itemName = [System.IO.Path]::GetFullPath($item.FullName).SubString([System.IO.Path]::GetFullPath($sourcePath).Length + 1)
        $fullFtpPath = [System.IO.Path]::Combine($script:ftpHost+"/$ftpFolder/", $itemName)
        if ($item.Attributes -eq "Directory"){
            try{
                $uri = New-Object System.Uri($fullFtpPath)
                $fullFtpPathRequest = [System.Net.WebRequest]::Create($uri)
                $fullFtpPathRequest.Credentials = $script:Credentials
                $fullFtpPathRequest.Method = [System.Net.WebRequestMethods+Ftp]::MakeDirectory
                $fullFtpPathRequest.GetResponse()
            }catch [Net.WebException] {
                Write-Host "$item probably exists ..."
            }
            continue;
        }

        $webClient = New-Object System.Net.WebClient
        $webClient.Credentials = $script:Credentials
        $uri = New-Object System.Uri($fullFtpPath)
        Write-Host "uploading $item"
       $webClient.UploadFile($uri, $item.FullName)
    }
}

function Get-FromFtp {  
    param([string]$sourceFolder, [string]$ftpFolder)

    $fullFtpPath = [System.IO.Path]::Combine($script:ftpHost, $ftpFolder)
    $dirs = Get-FtpDirecoryTree $fullFtpPath
    foreach($dir in $dirs){
       $path = [io.path]::Combine($sourceFolder, $dir)
       if ((Test-Path $path) -eq $false) {
          New-Item -Path $path -ItemType Directory | Out-Null
       }
    }
    $files = Get-FtpFilesTree $fullFtpPath
    foreach($file in $files) {
        $ftpPath = $fullFtpPath + "/" + $file
        $localFilePath = [io.path]::Combine($sourceFolder, $file)
        Write-Host "Downloading $ftpPath ..."
        Get-FtpFile $ftpPath $localFilePath
    }
}

function Remove-FromFtp($ftpFolder) {  
    $fullFtpPath = [System.IO.Path]::Combine($script:ftpHost, $ftpFolder)
    $fileTree = Get-FtpFilesTree $fullFtpPath
    if($fileTree -gt 0){
        foreach($file in $fileTree) {
            $ftpFile = [io.path]::Combine($fullFtpPath, $file)
            Remove-FtpItem $ftpFile "file"
        }
    }
    $dirTree = [array](Get-FtpDirecoryTree $fullFtpPath) | sort -Property @{ Expression = {$_.Split('/').Count} } -Desc
    if($dirTree -gt 0) {
        foreach($dir in $dirTree) {
            $ftpDir = [io.path]::Combine($fullFtpPath, $dir)
            Remove-FtpItem $ftpDir "directory"
        }
    }
}

function Get-FtpDirecoryTree($fullFtpPath) {  
    if($fullFtpPath.EndsWith("/") -eq $false) {
        $fullFtpPath = $fullFtpPath += "/"
    }

    $folderTree = New-Object "System.Collections.Generic.List[string]"
    $folders = New-Object "System.Collections.Generic.Queue[string]"
    $folders.Enqueue($fullFtpPath)
    while($folders.Count -gt 0) {
        $folder = $folders.Dequeue()
        $directoryContent = Get-FtpDirectoryContent $folder
        $dirs = Get-FtpDirectories $folder
        foreach ($line in $dirs){
            $dir = @($directoryContent | Where { $line.EndsWith($_) })[0]
            [void]$directoryContent.Remove($dir)
            $folders.Enqueue($folder + $dir + "/")
            $folderTree.Add($folder.Replace($fullFtpPath, "") + $dir + "/")
        }
    }
    return ,$folderTree
}

function Get-FtpFilesTree($fullFtpPath) {  
    if($fullFtpPath.EndsWith("/") -eq $false) {
        $fullFtpPath = $fullFtpPath += "/"
    }

    $fileTree = New-Object "System.Collections.Generic.List[string]"
    $folders = New-Object "System.Collections.Generic.Queue[string]"
    $folders.Enqueue($fullFtpPath)
    while($folders.Count -gt 0){
        $folder = $folders.Dequeue()
        $directoryContent = Get-FtpDirectoryContent $folder
        $dirs = Get-FtpDirectories $folder
        foreach ($line in $dirs){
            $dir = @($directoryContent | Where { $line.EndsWith($_) })[0]
            [void]$directoryContent.Remove($dir)
            $folders.Enqueue($folder + $dir + "/")
        }
        $directoryContent | ForEach {
            $fileTree.Add($folder.Replace($fullFtpPath, "") + $_)
        }
    }

    return ,$fileTree
}

function Get-FtpDirectories($folder) {  
    $dirs = New-Object "system.collections.generic.list[string]"
    $operation = [System.Net.WebRequestMethods+Ftp]::ListDirectoryDetails
    $reader = Get-Stream $folder $operation
    while (($line = $reader.ReadLine()) -ne $null) {
       if ($line.Trim().ToLower().StartsWith("d") -or $line.Contains(" <DIR> ")) {
            $dirs.Add($line)
        }
    }
    $reader.Dispose();
    return ,$dirs
}

function Get-FtpDirectoryContent($folder) {  
    $files = New-Object "System.Collections.Generic.List[String]"
    $operation = [System.Net.WebRequestMethods+Ftp]::ListDirectory
    $reader = Get-Stream $folder $operation
    while (($line = $reader.ReadLine()) -ne $null) {
       $files.Add($line.Trim())
    }
    $reader.Dispose();
    return ,$files
}

function Get-FtpFile($ftpPath, $localFilePath) {  
    $ftpRequest = [System.Net.FtpWebRequest]::create($ftpPath)
    $ftpRequest.Credentials = $script:Credentials
    $ftpRequest.Method = [System.Net.WebRequestMethods+Ftp]::DownloadFile
    $ftpRequest.UseBinary = $true
    $ftpRequest.KeepAlive = $false
    $ftpResponse = $ftpRequest.GetResponse()
    $responseStream = $ftpResponse.GetResponseStream()

    [byte[]]$readBuffer = New-Object byte[] 1024
    $targetFile = New-Object IO.FileStream ($localFilePath, [IO.FileMode]::Create)
    while ($readLength -ne 0) {
        $readLength = $responseStream.Read($readBuffer,0,1024)
        $targetFile.Write($readBuffer,0,$readLength)
    }

    $targetFile.close()
}

function Remove-FtpItem ($fullFtpPathToItem, [string]$type = "file") {  
    Write-Host " deleting $item..."
    $ftpRequest = [System.Net.FtpWebRequest]::create($fullFtpPathToItem)
    $ftpRequest.Credentials = $script:Credentials
    $ftpRequest.UseBinary = $true
    $ftpRequest.KeepAlive = $false

    if($type -ieq "file") {
        $ftpRequest.Method = [System.Net.WebRequestMethods+Ftp]::DeleteFile
    } else {
        $ftpRequest.Method = [System.Net.WebRequestMethods+Ftp]::RemoveDirectory
    }

    $ftpResponse = $ftpRequest.GetResponse()
    "Delete status: {0}" -f $ftpResponse.StatusDescription
}

function Get-Stream($url, $meth) {  
    $fullFtpPath = [System.Net.WebRequest]::Create($url)
    $fullFtpPath.Credentials = $script:Credentials
    $fullFtpPath.Method = $meth
    $response = $fullFtpPath.GetResponse()
    return New-Object IO.StreamReader $response.GetResponseStream()
}

Export-ModuleMember Set-FtpConnection, Send-ToFtp, Get-FromFtp, Remove-FromFtp  
comments powered by Disqus