Parallel video encoding

[me-5yx05kfkO/aqeI1yJSURBw at public.gmane.org (Matthew Gillen)]

Kristian Erik Hermansen wrote:
> Let's say I have a box with a multitude of CPU cores on multiple
> physical CPUs and i wanted to utilize mencoder to gain maximal total
> CPU usage.  Any ideas?  I am not sure that mencoder is fully
> multi-threaded

Check the "threads" option (it's codec-specific, ie -<codec>opts threads=8)

>, and even if so, it can only run on one physical CPU at
> a time, right?

What would be the point of having it be multi-threaded then?

> $ time $(for i in $(find . -type f | grep -i foo); do mencoder "$i"; done)
> 
> Can you think of a way to make this exec all in parallel, and if so,
> and even so, would this not be efficient to do because of some
> fundamental issue?  For instance, how would this affect caching in the
> CPUs?  

Shouldn't affect that at all.  In Intel's hyperthreading it would, but the
true dual core chips nowadays have independent caches (a lesson learned from
hyperthreading).

> Considering you know the application is a video encoder, what
> properties can we exploit to gain the fastest CPU time to encode all
> the videos?  Let's say it is an arbitrary amount, but more than the
> number of cores you have.  This is an interesting problem and I should
> probably ask Pixar :-)  But you guys are a cheap start!

Knowing a little bit about how mpeg encoding works, one way to do it would be
to break up the input into "work units" that a single cpu would do.  Each work
unit would start with an I frame, and do all the images leading up to the next
I frame.  So each work unit is completely independent from the others, thus
allowing each CPU to do the bulk of it's work independently.

Then the "master" thread just concatenates all the results in the correct order.

Matt

-- 
This message has been scanned for viruses and
dangerous content by MailScanner, and is
believed to be clean.