Wired-Space

In my last blog post I mentioned that creating MP4, or rather Quicktime style files using GStreamer contains a minor problem. The MOOV atom is usually written to the end of the files, but software such as the Flash player needs that atom to be present at the file’s start if it wants to start playing before the whole file is available. This is, for example, the case when you get the video via a slow HTTP connection.

While looking for a solution, I came across Mike Melanson’s qt-faststart tool that does exactly that, moving the MOOV atom to the front of the video file. His tool was written in C and is also found in the FFMPEG source tree by now. I originally had a look at GStreamer because with its many language bindings I was able to do video transcoding in Java without having to call external command line tools. So using qt-faststart in its C written form was of little use to me. To my surprise the process of moving and adjusting the MOOV atom is not that complex and I was able to port Mike’s tool to Java pretty quickly.

And once again was annoyed by Java’s lack of unsigned byte data primitives. :( Really, processing low-level data structures in Java is not nice.

As Mike was kind enough to put his program into the public domain, I think it is just fair to do the same with the Java port. You can get the source code here.  [ Java Quicktime Faststart tool ]

The Java port is pretty straight taken from the C code, so all credit about this handy little tool should go to Mike. I hope somebody can make good use of the Java port somehow.

A few days ago, I began to have a closer look at the GStreamer library, because it seemed to be an interesting alternative to some less-than-optimal solutions for video and audio processing we use at work.

The basic concept behind GStreamer is a framework that allows the developer to assemble a media processing graph by linking elements together, that are either a data source, sink or filter. This approach allows for a wide range of application from a simple MP3 player up to a non-linear video editing application (using the GNonLin Element) .

The data flow is controlled by using capabilities applied to the connections between the pipeline elements. Each element can accept certain capabilities on its input connections and provides capabilities on its output connections. When the pipeline is activated, it negotiates if the specified capabilities can be provided by the used elements.

What really surprised me was the amount of language bindings available for GStreamer. While the library itself is written in C and uses the GLib GObject framework, there are bindings for PHP, Python, Perl, Java and others.

After giving the application development manual a quick read, it was surprisingly easy to set up a pipeline that converts a source video file into a Flash playable MP4 file.

I just had to figure out a few minor things that where not really obvious from the documentation. Or I just didn’t find them :)

The first problem was to retrieve the original video’s characteristics, such as its image dimensions. This was necessary because if I just inserted an image scaling element, the target video was rescaled without taking pixel- or display aspect ratio into any consideration. So to retrieve that information, I had to set up the pipeline and then set it to “PAUSE” state without having target video dimensions set. This forces the pipeline to negotiate all of its capabilities, or to “preroll” as it is called in the GStreamer terminology. Once done you can retrieve the negotiated capabilities from the video decoding element. Those include the width, height and the pixel aspect ratio. Now you determine the correct scaling parameters before setting the pipeline to its “NULL” state, which causes it to discard all negotiated capabilities. Finally, after applying the new scaling parameters, you set the pipeline to its “PLAYING” state to get the conversion done.

The other problem I had was less a problem of GStreamer itself, but of the Java bindings. Those bindings use the Java Native Access (JNA) to call the GStreamer library. As such the pipeline, once having its state set to a new value, runs in a separate thread. This has two implications on how to use it in the Java source code.

For one, the thread does not terminate by itself if it encounters an error or the pipeline reaches the end of its input stream.

And second, the method call that sets the pipeline to a new state is non blocking. So you might run into the end of your main method before the pipeline has finished or maybe even changed its state yet.

The solution to the first problem is to attach a couple of listeners to the pipeline’s message bus and act upon the end of stream or error messages. Otherwise the program might never terminate because the GStreamer thread does not end at all.

The second problem can be tackled by calling the getState() method of the pipeline object. This call blocks until either the pipeline returns its current (new) state or a given timeout expires.

Once I had figured this out, the program works pretty well for most videos. Those that fail are usually quite “exotic”. Yes, when you run the program in a Linux environment, WMV files are to be considered exotic :P

There is one problem I still have to solve.

H.264 encoded videos that are supposed to be played in the Adobe Flash player via HTTP download must be stored in a MP4 container format. The problem with doing a one pass write of that file is that the block containing the meta data, such as duration and seek points, is written at the end of the file. However, if this is the case, the Flash player does not start playing the video until the whole file was downloaded. The proper treatment of this problem is manipulating the MP4 file so that the metadata block is moved to the file’s front and adjust all of its stored offsets. This is called “faststart” in some programs. One MP4 multiplexing plugin of the GStreamer library supported this by writing the video into a temporary file and then move it to the final location after adjusting the metadata block. But it seems the plugin is a bit buggy. For some files the audio track just stopped after a minute or so. The other MP4 plugin does not have that problem, but it does not support faststart at all.

This might be fixed in a future version, or so I hope.

The GStreamer people divide some plugins into three packages that represent their state of quality or legal issues.

The “good” package contains all plugins that meet the developers quality criteria and don’t pose any legal issues.

The “bad” package contains plugins that seem to meet the quality standard but may have questionable legal issues.

The “ugly” package contains all the other plugins that don’t meet quality or legal standards.

Talking about legal issues, there is something that might be of interest to people who worry about the legal issues of using patented technology, such as MP3. Fluendo offers commercially licensed GStreamer plugins so that you can at least for some codecs purchase a license and use it in your product.

Overall I am really impressed with the GStreamer framework. That the plugins’ quality seem to vary is not that surprising, considering that they are provided by several different parties. The framework itself is pretty much all you need for, I’d say, most video or audio related applications. If it can deal with all the media files you throw at it, is only depending on the plugins you have installed, similar to having video codecs registered with DirectX oder Quicktime on other plattforms.

I’m pretty sure I will have some more fun with it :)