Powershell programming help.

[Chuntzu]

I'm trying to create some powershell to distribute video encoding to each of my computers in the house. So far I have done just that. I actually created a function that does great on a single localhost or to another computer just fine using ffmpeg for now. What I would like to do now is take each of these chunks of any array (the array is of my movie files) and have it be a work list for each computer. Since I am pretty new to programing in general I am trying to find a loop,while,foreach that works with this idea and I am stumbling with it. So far here is what I have.


Function Split-Every($List, $count) {
$aggregateList = @()

$blocks = [Math]::Floor($list.Count / $count)
$leftOver = $list.Count % $count
for($i=0; $i -lt $blocks; $i++) {
$end = $count * ($i + 1) - 1

$aggregateList += @(,$list[$start..$end])
$start = $end + 1
}
if($leftOver -gt 0) {
$aggregateList += @(,$list[$start..($end+$leftOver)])
}

$aggregateList
}

$Directories = '\\servername\Videofilder\'
$Dir = get-childitem $Directories -recurse
$extensions = @(".mkv", ".mp4", ".mov")
$FileSize = $Dir | Where-Object {$_.length/1MB -gt 500}
$List = $FileSize | where {$_.extension -in $extensions}
$VideoFileChuck = Split-Every -List $List -count 25

So what this now leaves me with is $VideoFileChunks[0] which has the first 25 movies, $VideoFileChunks[1] has the next 25 movies, etc... On a single computer I would then pipe

$VideoFileChunks[0] | % {& $FFMpeg -y -i $_.fullname -vcodec $vcodec -b:v $vbitrate -acodec $acodec \\servername\Videofilder\ConvertedVideo\ $($_.basename).mp4}

and it would convert the video (I have ffmpeg variables in the function as well and they work fine but will probably rethink that as I develop the idea more and want to add more tools)

Without manually assigning to each computer and running this manually:

Invoke-Command -ComputerName computer01 -ScriptBlock {

Function Split-Every($List, $count) {
$aggregateList = @()

$blocks = [Math]::Floor($list.Count / $count)
$leftOver = $list.Count % $count
for($i=0; $i -lt $blocks; $i++) {
$end = $count * ($i + 1) - 1

$aggregateList += @(,$list[$start..$end])
$start = $end + 1
}
if($leftOver -gt 0) {
$aggregateList += @(,$list[$start..($end+$leftOver)])
}

$aggregateList
}

$Directories = '\\servername\Videofilder\'
$Dir = get-childitem $Directories -recurse
$extensions = @(".mkv", ".mp4", ".mov")
$FileSize = $Dir | Where-Object {$_.length/1MB -gt 500}
$List = $FileSize | where {$_.extension -in $extensions}
$VideoFileChuck = Split-Every -List $List -count 25

$VideoFileChunks[0] | % {& $FFMpeg -y -i $_.fullname -vcodec $vcodec -b:v $vbitrate -acodec $acodec \\servername\Videofilder\ConvertedVideo\ $($_.basename).mp4}
}

I would like to send $VideoFileChunks[0] to something like $ComputerName[0] and loop through $VideoFileChunks[1..50] and each chunk goes to a new $ComputerName[1.25] so as to not need to manually assign and run a new powershell command manually with Invoke-Command -ComputerName computer01 -ScriptBlock {}


Sorry if I wasn't clear with this as I am not so great with programming concepts at this point. I appreciate any advice that can be given, I will attempt to read up on any concepts given to me since I don't expect anyone to write this for me, so I can "learn to fish vs being given a fish".

 

[cheezehead]

If you have a large number of files in your directories you may want to refactor out your "Where" clauses, PowerShell does work with them but performance on large datasets is painfully slow.

$FileSize = foreach($i in $Dir){if($i.length/1MB -gt 500){$i}}

Instead of:

$FileSize = $Dir | Where-Object {$_.length/1MB -gt 500}

To split it across multiple servers, start with an array of servernames:

$myservers = "server1","server2","server3","server4"

Then iterate through your $VideoFileCheck

foreach($i in $VideoFileCheck)
{
$servername= Get-Random $myservers
$i |  % {& $FFMpeg -y -i $_.fullname -vcodec $vcodec -b:v $vbitrate -acodec $acodec \\$servername\Videofilder\ConvertedVideo\ $($_.basename).mp4}
}

Depending on how large your talking, you could also iterate through your encoding farm instead of using a random server. Also depending how deep you want to go, you may want to look into multi-threading the PowerShell process with using multiple runspaces.

 

[Chuntzu]

Hey thanks for the info. I will give it a go tomorrow after work. I have been looking into the invoke-paraellel and foreach parallel and poshrs-job functions from technet gallery/github to help run the commands in run spaces.

The foreach section. You wrote looks like it it only makes reference to the server, or a random server in the output file section of the command, which could be my ignorance showing, but would I need something like "invoke command and then a server name" to indicate which or a server to run the command on?

You made reference to it at the end there but I would like to have all 18-25 servers working on files until all of the applicable files are worked through. There are something like 6800 video files. So each server would work the next available file in the list until all are complete. So keeping each server running jobs is the ideal scenario.

Once I run it with those modifications I will let you know how it works. Thanks again.

Sent from my SM-N920T using Tapatalk

 

[DavidRa]

What you describe is a queue. Architecturally, you want to have a single master process farming out work to each of the computers, and waiting for each to finish before kicking off another one. I would be very surprised if you need much more than base PoSH 4 or 5 for that.

So I'd have something like this:

$CPUIdle = @("PC1", "PC2")
$Jobs = @()

$Files = Get-ChildItem

foreach ($File in $Files) {
  $TargetPC = $CPUIdle[0]
  $CPUIdle.Remove($TargetPC)
  // Execute background job
  while ($CPUIdle.Count -eq 0) -and (all background tasks running) { Sleep 30 }
  // Add finished task computer names to $CPUIdle array
}

That should be enough to start the creative juices...

 

[cesmith9999]

lookup foreach-parallel

Chris

 

[Chuntzu]

What you describe is a queue. Architecturally, you want to have a single master process farming out work to each of the computers, and waiting for each to finish before kicking off another one. I would be very surprised if you need much more than base PoSH 4 or 5 for that.

So I'd have something like this:

$CPUIdle = @("PC1", "PC2")
$Jobs = @()

$Files = Get-ChildItem

foreach ($File in $Files) {
  $TargetPC = $CPUIdle[0]
  $CPUIdle.Remove($TargetPC)
  // Execute background job
  while ($CPUIdle.Count -eq 0) -and (all background tasks running) { Sleep 30 }
  // Add finished task computer names to $CPUIdle array
}

That should be enough to start the creative juices...

I will definitely being trying this out here shortly thank you very much! I like the idle portion as well, since some of the available nodes I will be using g I. The house are used as kodi/plex/mediacenter front ends and don't want the wife getting mad when I start slamming the cpu while she is catching up on her shows.

Sent from my SM-N920T using Tapatalk

 

[Chuntzu]

lookup foreach-parallel

Chris

I actually mentioned that a bit earlier, and poshrs-job which the author of foreach parallel mentioned in a blog post referring to another guy that helped him build the foreach parallel had built. I am very excited to try these out. Hopefully I will have some good/exciting news here shortly.

Sent from my SM-N920T using Tapatalk

 

[cesmith9999]

What you are really looking for is an MSMQ like functionality in distributing jobs. Where you can denote each server as a job taker like a check-out line at Frys. where there is a person telling you which checkout line is available. in your case it is which server is available for the next job.

Chris

 

[Chuntzu]

What you are really looking for is an MSMQ like functionality in distributing jobs. Where you can denote each server as a job taker like a check-out line at Frys. where there is a person telling you which checkout line is available. in your case it is which server is available for the next job.

Chris

This is very true, and as I slowly piece together my "programing knowledge" I am sure I will move over to some real program language instead of doing it with shell scripting. I am also considering windows hpc pack as it may do what I am looking for as well.

I have worked through the examples given to me and I am getting closer. I just figured out, I think, how to properly pass args into invoke-command and run as job functions which I believe has been my issue. So fingers crossed.

Sent from my SM-N920T using Tapatalk