Parallel video encoding

[me-5yx05kfkO/aqeI1yJSURBw at public.gmane.org (Matthew Gillen)]

Matthew Gillen wrote:
> Kristian Erik Hermansen wrote:
>> In any event, it doesn't really matter since I wasn't planning on
>> writing a custom app.  I just wanted to know if it would be more
>> efficient to spawn multiple mencoder process at the same time with one
>> thread allocated to the process (up to MAX_CORES or MAX_CORES - 1),
>> spawn mencoder on MAX_CORES - 1 for one file at a time (that is the
>> suggested way, right?), or just invoke mencoder at MAX_FILES_TO_ENCODE
>> times with one thread for each.
>>
>> I bet option #2 works the best, since spawning multiple mencoder
>> processes could be wasteful in terms of caching and memory usage.  I
>> bet one mencoder process working on one file with MAX_CORES - 1 would
>> be ideal.  If you go higher than MAX_CORES, then you are going to have
>> to context switch in and out the threads as they are shared, right?
>> Tell me if my logic is flawed here...
> 
> Well, there's a school of thought on using 'make' that says you should always
> use at least -j(MAX_CORES+1), the idea being that you keep the CPU busier by
> always having an "extra" job ready when one thread needs to wait on disk I/O.
>  Disk I/O will completely eclipse any context-switching penalty, and even
> though video encoding is a cpu-bound task, there's quite a bit of I/O involved
> too.  I'm not sure what the optimal answer is for your case.

And actually, you should also consider that context switching between threads
within a process is generally much faster than heavy-weight process-level
context switching (this doesn't help make, since it uses processes instead of
threads for it's job server).  So you're probably right that option #2 is more
efficient, but I'd use MAX_CORES+2 number of threads (or something like that).

Matt

-- 
This message has been scanned for viruses and
dangerous content by MailScanner, and is
believed to be clean.