Saturday, June 23, 2018

DaVinci Resolve -- Importing Consumer Video

[NOTE: As of the release of DaVinci Resolve 14, some of the details below may have changed.  For example, BlackMagic Fusion is now built into DaVinci Resolve.  However, due to the redesign of the core engine of DaVinci Resolve, my consumer computer (4th Gen. Core i7, 16GB RAM, Radeon R7 370 4GB) cannot use DaVinci Resolve 14, even on existing projects, without the program stopping and/or displaying an error.  As such, it's impossible for me to use this newer version to see what updates to the program may have made the below workarounds unnecessary.  If and when BlackMagic Designs updates DaVinci Resolve to gracefully fall back to use the capabilities of my consumer machine, I will attempt to update this document.  As of the last update to this document, a beta of DaVinci Resolve 15 is available.  When I have time (especially enough time to be able to downgrade back to a functional DaVinci Resolve (12), without losing my existing projects, if the failures still occur), I'll check this out.]



DaVinci Resolve is a wonderful piece of software for editing and processing video, and is available to anyone for free.  However, it is designed for professionals, not (us) amateurs.  So, support for consumer file formats is sparse, and what is there can be picky.  That doesn't mean consumers cannot use DaVinci Resolve with our footage, but it does mean we need to process our video and audio before we can import it into DaVinci Resolve.  Here are my best suggestions for speed and quality.

IMPORTANT NOTE:  Decoding and re-encoding video (or audio) WILL degrade the quality.  Unless you are actually modifying the video, avoid re-compressing your content.  And if a tool says it is converting the video (especially if the resolution, format, or CODEC is being changed, and usually if the tool takes a long time and/or displays frames during the conversion), it is most likely decoding and re-encoding.  This guide provides information on how to process video into formats compatible with DaVinci Resolve without re-encoding.


Tools


I'll mention use of specific tools below.  Here is the list with links:

Video


Before I go any further, you should be familiar with the concept of a CODEC and a container.  Please visit this blog post for a quick discussion.  But if you are already familiar, please read on.

DaVinci Resolve (and Fusion) is primarily designed to work with files containing video that is uncompressed or lightly compressed.  Consumers are unlikely to have even heard of the most common professional formats (DNxHD, ProRes, etc.), let alone used them.  But that's what professionals use, and that is the target (paying) customer base for DaVinci Resolve.

However, us amateurs don't have cameras that produce video in those formats.  Plus, we don't have budgets for the computer systems these formats demand.  Instead, most modern consumer devices use one of the variants of the MPEG-4 compression algorithm.  The camera will use a CODEC to compress the video and audio, and then store these two "streams" into a file on your SD card.

The problem starts here, because DaVinci Resolve doesn't understand all the container formats a consumer camera might use, even though it usually understands the data inside them.  To make matters worse, all the cameras don't package the data into the container file in the same way.  Presumably, all of these files comply with a specification, but DaVinci Resolve wasn't designed for this, so it isn't happy with some variations.

To throw us another curve ball, due to file size limitations of consumer devices (usually the filesystems in use on computer hard drives), longer recordings are typically broken up into multiple files. Unfortunately, I haven't found any of these multi-file recordings that DaVinci Resolve likes as-is.


Repackaging vs Re-encoding


NOTE:  There are lots of software applications out there that can convert videos from one format to another.  But BEWARE: Most of them actually decode the encoded data and re-encode it.  This introduces another, needless layer of loss into the process, degrading your (already consumer-quality) video.  So it's a very good idea to avoid this.  Software that simply repackages the data into a different container is much preferable.


Repackaging software can be difficult to find and cryptic to use.  And all repackaging software isn't created equal.  Unfortunately, DaVinci Resolve still isn't happy with repackaging results from some programs.

I don't claim that the software applications below are the only software that works with DaVinci Resolve, but after trying a bunch, I've found these do work for me.


AVCHD


AVCHD is a format that most consumer FHD (full HD -- i.e. 1920x1080) camcorders use for recording.  On your storage device (usually an SD card) there will be a directory structure similar to the one shown here (that for some bizarre reason starts with a PRIVATE directory):



Please Note:  Software will usually accompany your camera to allow you to import these recordings to your computer.  I'll be honest...I don't ever used that software.  I installed software that came with my very first digital camera, and I will never do so again.  It was old, slow, buggy, cumbersome to use, installed way too many packages that I would never use, and tried to take over my computer (well, it overwrote all my file associations, so just viewing an image, whether it came from the camera or not, always loaded the huge, slow to open package, which felt like it took over my machine).  If you are comfortable with that software (and it works with your operating system), that may be a viable option for you.  If not, read on...

AVCHD uses a more complicated part of the MPEG-4 specification (part 10), requiring more powerful processing by the camera.  The result is better quality with smaller file sizes.

In researching the AVCHD format, and in authoring AVCHD discs myself (the poor man's Blu-Ray, which are normal DVDs, but which contain HD video content that most Blu-Ray players can play), I can tell you that AVCHD can provide a lot of features beyond just storing video.  But in my experience with several camcorders, other than the thumbnails used to show the clips while in the camera, there's only really one feature that the AVCHD directory structure itself provides:

AVCHD video at the bottom of the directory tree shown above, is stored in MTS files inside the STREAM directory.  MTS is one of several file extensions for the MPEG-2 transport stream (others are M2TS, TS, or sometimes VOB, but AVCHD uses MTS).  The MPEG-2 transport stream container can hold a number of different types of streams (and it can hold multiple streams, like video, different language audio tracks, subtitles, etc.), including an MPEG-4 compressed stream.  That's what it holds in an AVCHD recording.  Specifically, the video stream is an MPEG-4 Part 10 stream, also known as h.264 or AVC (Advanced Video Codec).

DaVinci Resolve is happy with MTS files containing h.264, but there are two issues:
  1. MTS files are typically limited to 2 or 4 GB to stay compatible with older filesystems.  (i.e. when a customer copies the files to their computer, if the files are too big, they won't copy, and the customer will be confused and angry).  However, the breaks in MTS files (at least on the Canon and Panasonic camcorders I've tested) are made in mid-stream!  There is some MPEG-4 sub-structure to where the break occurs (it's not at a completely random place), but it does seem to occur in mid-frame.  This means that playing a second, third, or later file will result in some lost frames at the start.  In some cases the later files may fail to play at all.  The latter is the case with DaVinci Resolve.  Each file typically holds 10-20 minutes, so this is something very likely to affect you, if you record longer than this.
  2. The audio format may be incompatible with DaVinci Resolve.  See this blog post for information on that issue.


Multiple Files


So, dealing with issue 1 above basically boils down to two possible solutions:
  1. Use the software that accompanies your camcorder to import the files to your computer.  As I mentioned above, I don't do this, so I can't say how this works, or what format the resulting files will be on your computer hard drive.  However, I will say it's highly unlikely that the software will be compatible with Linux, if that's your operating system of choice.  So...
  2. Join the files yourself (even in Windows), which is what I do.

Joining the files on your computer is extremely simple for MTS files.  Suppose you have copied the following files from the PRIVATE/AVCHD/BDMV/STREAM directory of your SD card:

    00785.MTS      2,044,889,088    Oct ‎20 ‎2015  ‏‎6:35:22PM
    00786.MTS      2,045,165,568    Oct 20 2015  6:52:02PM‏‎
    00787.MTS      2,045,054,976    Oct 20 2015  7:08:34PM
    00788.MTS      2,044,766,208    Oct 20 2015  7:25:04PM
    00789.MTS        579,170,304    Oct 20 2015  ‏‎7:41:26PM‏‎

Windows -- Utilize the command line copy command to concatenate the files:

    copy /b 00785.MTS + 00786.MTS + 00787.MTS + 00788.MTS + 00789.MTS vid.MTS

Linux -- Utilize the command line cat command to concatenate the files:

    cat 00785.MTS 00786.MTS 00787.MTS 00788.MTS 00789.MTS > vid.MTS

MP4

The MPEG-4 format is so flexible (and complicated) that its creators (the Motion Pictures Expert Group) broke it into a number of pieces and parts.  At the time it was published, accomplishing the most innovative and aggressive portions would have been technically and/or financially infeasible, particularly for consumer products (i.e. products amateurs can afford).  Full implementation would require too much CPU power, too much memory, too much speed from the CPU or memory, etc.  So divisions in the specification allowed vendors of devices to create products that followed easier or simpler parts of the MPEG-4 specification.  Later, the more complicated sections could be handled, as hardware capability increased and costs fell.

h.264 (MPEG-4 Part 10) is the most ambitious of the spec (and even it has subdivisions that give some flexibility to vendors), and is what is used in AVCHD.  Early devices couldn't handle h.264 for high resolutions. Some devices could handle h.264 for standard definition, but not HD (1280x720).  Some could handle it for HD, but not FHD.  But, for the most part, FHD video devices today can handle h.264 (to varying quality levels).

However, UHD (ultra high-definition -- 3840x2160), which is 4 times the size of FHD, pushes the processing requirements even higher.  So, many devices today drop down to less complex MPEG-4 for UHD to keep "4K" consumer products affordable.  This results in simpler compression, which is less efficient.  And that gives either higher data rates for similar quality to h.264, or lower quality for similar data rates.  When stored, these files typically use the MPEG-4 container format and the MP4 extension.

DaVinci Resolve is happy with MP4 files containing MP4 compession, but as with h.264, it is not happy with the way long recordings are divided into multiple files.


Multiple Files


Again, the 2 or 4 GB limitation of some consumer file systems is anticipated by many devices, so they break up the recordings into multiple files.  With 4x the number of pixels, and less efficient compression, the individual files tend to contain much less time than MTS files (usually less than 10 minutes).  So it's even more likely you'll see multiple files for a single recording.  But unfortunately, the same technique used to concatenate MTS files does not work for these MP4 files.

To join MP4 files, things get a bit more complicated.  It's still true that the second and later files are not fully independent, so reading them will result in lost frames at the start.  But the method of breaking up the files differs, so raw concatenation does not work.  However, I have found a tool that properly reconstructs the stream (albeit slower than simple concatenating):

mp4box (see tools above) is available for all the major platforms (although as of this writing, Mac support appears to be source only), and would concatenate three MP4 files like this:

    mp4box -add file1.MP4 -cat file2.MP4 -cat file3.MP4 out.MP4


Compatibility


Unfortunately, the above process sometimes isn't all that is needed.  As mentioned above, DaVinci Resolve is picky about what it imports.  I assume the issue lies mostly with the Quicktime CODECs it uses (and I desperately hope BlackMagic Designs will drop Quicktime in favor of ffmpeg CODECs, now that even Apple has dropped support for Quicktime on Windows), but regardless of why, some MP4 files just won't load into DaVinci Resolve (or Fusion).

Repackaging in a container seems to be a black art, and I would have expected that most tools I found would have done a decent job of this, since they are likely to be following the spec.  However, I've only been able to get mp4box, used in the following way, to get these problem MP4 files to be accepted by DaVinci Resolve (and Fusion).

    mp4box -add in.mp4 -raw 1 -new temp
    mp4box -add temp_track1.h264 -new out.mp4

(Note that it is possible to repackage through ffmpeg or mp4box in a single step, but I have found that DaVinci Resolve is NOT happy with the results.  Please let me know if you find a one-step process that is compatible.)

DaVinci Resolve -- Importing Consumer Audio

DaVinci Resolve is a wonderful piece of software for editing and processing video, and is available to anyone for free.  However, it is designed for professionals, not (us) amateurs.  So, support for consumer file formats is sparse, and what is there can be picky.  That doesn't mean consumers cannot use DaVinci Resolve with our footage, but it does mean we need to process our video and audio before we can import it into DaVinci Resolve.  Here are my best suggestions for speed and quality.


Tools


I'll mention use of specific tools below.  Here is the list with links:


Audio


It's not really BlackMagic Design's fault that some audio formats are unsupported.  After all, they are giving away the software for free.  And if you were giving something away for free and someone came up and said that to include support for a particular format in your free product, you'd have to pay a license fee, you'd probably balk at the idea as well.  Well, that's (I'm guessing) what's happened here.  MP3 and AC3 are two common audio formats.  However, these formats are (have been) encumbered by patents that force(d) vendors to pay for licenses to use them in products like DaVinci Resolve.  (Don't get me started on how underhanded it is to manipulate an industry standard so it uses your intellectual property and you can get paid by anyone who wants to be compatible with...well, everyone else.)  So if BlackMagic Designs included support for these formats in DaVinci Resolve, they would have (had) to pay a license fee.  Did I mention that DaVinci Resolve is free?

That said, I do fault BlackMagic Designs for not bothering to tell the user any of this when loading video files.  If you import an MTS file (see this blog post), and it uses AC3 audio (which many camcorders choose instead of the AAC audio format from the MPEG-4 specification), DaVinci Resolve will happily import the video and completely ignore the audio.  No warning.  No error.  No notification of any kind.  But I'm telling you here, what's going on.  It does not support AC3 (Dolby Digital).

[Note that as of the date of the last update to this document, WikiPedia indicates that all AC3 & MP3 patents have expired.  However, see the note at the top of this blog post for why I haven't been able to check whether support for AC3 and/or MP3 has been added in recent updates to DaVinci Resolve.]

So for both types of files, you will need to convert these to a format DaVinci Resolve can handle, and import the results separately.  I prefer RIFF WAV files, which are easy to manipulate under Windows, but you can use any format DaVinci Resolve handles.  Uncompressed formats would be the best choice, because you would not be re-compressing the audio and losing more quality.

ffmpeg is the easiest choice to convert the audio, and its available for all the major platforms:

    ffmpeg -i in.MTS -vn -codec:a pcm_s16le out.wav
    ffmpeg -i in.MP3 -codec:a pcm_s16le out.wav

CODECs vs. Containers

Online, there appears to be a great deal of confusion about the different video (and audio) formats and the files in which they are stored.  The fact that some formats use the same name for containers and the data that is stored within doesn't help.  But here I'll try to clarify this a bit.

Video is made up of pixels (I know you know this, but bear with me).  Storing those pixels directly results in video files that are HUGE.  For example, a 1920x1080@30p video in the RGB24 format (that's 8 bits for each of the red, green, and blue colors for each pixel) produces 186 Mbytes per second of data.  And that's not considering that most high-end cameras capture more than 8 bits for each color!


Now, 186 MB/s may not sound like much, until you consider that a 1 TB drive could only hold about 90 minutes of this type of video.  That's basically one kid's recital...not including the edited copy.


Worse, 186 MB/s is very near or beyond the performance of a typical consumer hard drive.  So you're going to have difficulty being able to play back this video to preview it, let alone edit it (which requires multiple reads and writes at the same time).


But wait, what about 4k (UHD)?  That's 4x the data rate, or 744 MB/s!  At that rate, not only the storage devices have issues, but in many consumer computer systems, even the memory speed isn't sufficient (remember that the CODEC (see below) uses the memory to decode the video, and it requires several memory reads and writes for each pixel).

Using this type of uncompressed data requires a much more professional system, with multiple, better hard drives (e.g. SSD), possibly configured with RAID, faster memory and processors, and of course, that means lots of $$$$$$$ (or your currency of choice).


Then, there is the media in the camera itself.  Even the fastest (and most expensive) SD cards cannot write that much data so quickly.  (Professional cameras use a variety of techniques to capture uncompressed video, again, all of which are expensive.)

Because of all those real-world limitations, consumer cameras don't store video like that.  Instead, they compress the video before storing it, using a CODEC (enCOde/DECode).  That means the size of the data is reduced to something our consumer cameras, storage cards, and computers can handle.  Then CODECS on our computers decompress and compress the video for us as we read, write, and edit the video on our hard drives.


CODEC (a.k.a. Compression Format)

Most modern consumer devices use one of the variants of the MPEG-4 compression algorithm. (e.g. h.264/MPEG-4 Part 10/MPEG-4 AVC, MPEG-4  The camera will use built-in CODECs to compress the video and audio, resulting in two "streams" of data.


Container

After the audio and video emerge from the CODECs, the resulting compressed data streams are packaged into a file and stored on your camera's storage card.  The specific format of that file varies, depending on how the data will be used, and a variety of other reasons (including things like patents).


Confusion

One thing that is confusing is that some compression formats and containers use the same name.  In addition to the compression format, MPEG-4 also has its own file format (severalactually), but the video and audio streams can be stored in other types of files as well.

For example, MPEG-2, a precursor to MPEG-4, also specified a number of file formats for holding streams.  Oddly, some video devices create MPEG-4 compressed video, but store the stream in an MPEG-2 transport stream file.

There are other file formats, which can contain MPEG-4 streams, such as MKV and AVI, but it's unlikely that a consumer product would capture directly to these formats.

The important thing to remember here is that the compressed video and audio data can be stored in a multitude of file formats or containers.  And with the proper tools, the data can be moved from one file format to another WITHOUT MODIFYING THE ENCODED DATA.  That last part is important, because whenever video or audio is decoded and re-encoded, it will suffer a degradation in quality.  And remember that consumer devices start out with video and audio that are already pushing the limit of the balance between quality and cost.  So avoiding further degradation is a good idea.


Confusion (part 2)

Another point of confusion lies in the file extensions.  Some formats have multiple file extensions, and some extensions have multiple file formats associated with them.  For example, MPEG-4 can be seen in files with the MP4 extension, or even the MOV or QT extension.  But files with the MOV and QT extensions can actually contain data that is not MPEG-4 at all.  (Don't shoot the messenger.)  Also, the MPEG-2 transport stream has multiple extensions, including TS, MTS, and M2TS.


Audio

Audio streams, created by audio CODECs are very similar to the information noted above.  However, audio, even uncompressed, is a very small fraction of the amount of data of video.

But, not to be outdone, audio presents a different set of issues for the consumer.  These issues generally revolve around the patents placed on the various audio CODECs.  To make matters worse, some of the aforementioned specifications (e.g. MPEG-2 & MPEG-4) actually recommend (or for some target uses, require) use of specific audio formats.

In any case, the two main compressed audio formats used in consumer video cameras are Dolby Digital (AC3) and the Advanced Audio Coding (AAC).  As above, the data stream produced by their respective CODECs are placed in one of the containers mentioned above, along with the accompanying video.