CC BY-NC-SA 2.0 by Kevin Lim
The main motivation behind that is that is that I am very tired of the WordPress editor. I am a programmer and spend my days using the same editor1, which means that I have gotten rather efficient using it, or, at least, that’s the way of writing text that is the most comfortable to me.
In a stock WordPress installation, all you could do to edit your blog post in your own editor is to edit it in HTML and then cut and paste your result in the editor and publish. HTML. Eeek! I don’t want to be editing a tag soup, I want to focus on the content. There are better languages than HTML for that, such as reStructuredText or MarkDown. I did try WordPreSt a while ago but wasn’t satisfied with the work flow it provided, and if I remember correctly, I ended up having to correct things manually.
So, reading up a bit on the internets on what’s available, I got seduced by the concept of jekyll and its principle of offline generation of static pages, and the default workflow is very similar to the one I am used to when doing software development. Also, it seems to be actively maintained and used in GitHub Pages, which gives me a feeling that it’s here to stay. The only drawback that I see to it is that commenting cannot be handled by strictly static web pages. Most people using jekyll for blogging seem to resort to using external commenting services, such as Disqus and friends or even facebook, but I’d rather not depend on an external service for that. I’ve found a solution to that that seems decent to me, and I explain this in a later section.
Jekyll in itself is not a blogging system out of the box, but a good tool to make one. Since I didn’t feel like reinventing the wheel, I am using Jekyll-Bootstrap as a base for my blog.
I’d rather leave my old articles online, while still not have to manage a WordPress installation, so I definitely want to migrate my old articles to the new system. Luckily, jekyll comes out of the box with migration tools to make your life easy. I’ve used the tool that gets the content from a Wordpress export file and it went (mostly) like a breeze, as far as I could check. I only had to modify a bit the image caption code in one article. I still have some image files that aren’t where they should be in old posts, but that was already the case with WordPress. I will need to fix these articles at some point.
The issue that I had left to deal with is to be able to have comments, and to import the comments I had in WordPress. I chose to use Jekyll::StaticComments (code here for that purpose. It is a plugin for Jekyll that takes comments in YAML files and renders them at the end of a post. That still left me with two problems: how do I migrate old comments from WordPress, and how do I let people add new comments?
I’ve written a quick and dirty script with BeautifulSoup that gets the comment from a WordPress export file and generates a suitable tree of YAML files for use by Jekyll::StaticComments. I’ve put it online and called it wp2jekyll-comments.
Jekyll:StaticComments comes with a PHP script that allows one to submit comments. Since I’m an awful geek, I don’t like much the idea of having to use PHP here, so I made my own version of it in Python. I’ve published it as jekyll-comments.
]]>I would like to explain a bit the stuff I've been working on recently at Igalia. It is about playing with GStreamer and a sandboxing system to try and make the playback of untrusted media more secure. Hopefully writing this will be an occasion for me to get more distance and understand things better, and for others to give me feedback and ideas. Particularly, even though that is for me a field of interest, I do not claim to have any real expertise in security, therefore comments by people who know better would be gladly welcome.
This story started when I decided to have a look at chromium and its internals. It turns out that one very specific aspect of this application is its sandboxing system. In a nutshell, a sandbox is a virtual container in which untrusted programs can be safely run. In the real world, sandboxes are rarely perfect, but they are a significant security improvement over not using one. Chromium uses a sandbox to run its rendering engine (WebKit), which is basically the part that transforms the code of a web page into the graphical representation of it that you see on your screen. The rationale for running WebKit in a sandbox is not that it is untrusted code in itself, but rather that it is a big and complex project that is bound to have bugs, like all big and complex projects. On top of that, the input given to it is quite often data from untrusted sources, which could potentially be forged so that it exploits security bugs to do bad things to your computer and your beloved files. Now, with WebKit running in a sandbox, if a web page has been forged by an attacker to exploit a vulnerability in WebKit, the attacker will only have access to the sandbox environment, which means that it won't be able to do things like access the data on your computer, install software or connect to remote hosts.
As you might know, I like to play with multimedia things, and have hacked quite a bit on or around GStreamer. Therefore, I quite automatically thought of something else that might be worth running in a sandbox: demuxers and decoders. They are relatively big and complex pieces of software to which we regularly pass a whole bunch of untrusted data, would it be in a web context or a more traditional desktop or mobile context.
Fortunately, Julien Tinnes, a developer of the chromium sandbox for GNU/Linux made a stand alone version of it called setuid-sandbox, which can be used by other projects to easily sandbox any process.
The way setuid-sandbox works is rather straightforward: there is a sandboxme command that needs to be installed setuid root. You run sandboxme my_command and then from inside my_command, you first set up the file descriptors that you will need (being careful not to put there anything that could allow to escape the sandbox, more on that later), and then you call the provided chrootme() function, which will tell the sandboxme process to restrict the privileges that my_command has (e.g. it can still read and write on the fds that it has open, but it cannot open new ones).
Here is how I organised my integration of setuid-sandbox into GStreamer. What I want to do for now is to put what I think are the "most dangerous" parts (demuxing and decoding) in the sandbox, while letting the other components (mainly source and sinks) outside of the sandbox (for now at least). I decided to create a small program (called gst-decoder) that receives the original muxed and encoded video stream and outputs the decoded video and audio buffers. gst-decoder needs 3 channels of communication with the "controlling" process outside the sandbox (which is called the broker):
- one to pass the original stream from the source element in the broker to gst-decoder
- one to pass the video buffers from gst-decoder to the video sink element in the broker
- one to pass the audio buffers from gst-decoder to the audio sink element in the broker
In the future, more channels for subtitle support or other features could be desirable.
Since I am lazy, I wanted to use off the shelf GStreamer elements to handle these communication channels. For the cases explained above, that would be:
- the fdsink element on the broker side, and the fdsrc element in the sandbox
- shmsink (in gst-decoder) and shmsrc (in the broker)
- same elements as above
Since I expect other people to be equally lazy^W^W^Wwant their life to be made easier, my goal is to try and have this reasonably integrated in GStreamer, and easy to integrate in applications. For that, my best idea so far was to make a sandboxedecodebin element that, from the outside, works like decodebin or decodebin2, at least for simple cases: it has a sink pad that can take any format you would throw at decodebin, and it has an audio and a video source pads that output the decoded result. In the future, it might or might not be a good idea to try to integrate the "sandboxed" functionality in decodebin directly.
I implemented sandboxeddecodebin as a subclass of GstBin, and it has the following flow inside it:
fdsink -> [gst-decoder] | -> shmsrc (video) -> gdpdepay | -> shmsrc (audio) -> gdpdepay
Note that gst-decoder is an external (sandboxed) process, and not a GStreamer element like the other entities of this data flow graph. The sink pad of fdsink and the source pads of the two gdpdepay elements are exported by sandboxeddecodebin through ghost pads, which provides a decodebin-like interface.
The gst-decoder program basically runs a pipeline that looks like that:
fdsrc ! decodebin2 name=decoder decoder. ! video/x-raw-yuv;video/x-raw-rgb ! gdppay ! shmsink (video) decoder. ! audio/x-raw-int;audio/x-raw-float ! gdppay ! shmsink (audio)
and it also makes sure to get the privilege dropped at the right time, which is discussed below.
The ordering of operations needs to be thought carefully to combine GStreamer, and these elements in particular, with setuid-sandbox. Each of them brings its own set of conditions.
For setuid-sandbox, inside the sandbox (in gst-decoder):
- before we call chrootme(): we can open new fds and do a lot of nice initialisation, and we don't want to parse any untrusted data
- after we call chrootme(): we can't open new fds any more, or do similar initialisation tasks, but we can work on the data we received.
GStreamer has several states in which an element can be, with some rules on what should be done in which state. From the design documentation, the states are defined as follow:
- NULL: This is the initial state of an element.
- READY: The element should be prepared to go to PAUSED.
- PAUSED: The element should be ready to accept and process data. Sink elements however only accept one buffer and then block.
- PLAYING: The same as PAUSED except for live sources and sinks. Sinks accept and rendering data. Live sources produce data.
In particular, the elements that interest us here behave in the following way:
- shmsink is responsible for the creation and destruction of the shared memory object and the associated control socket and creates them when going from NULL to READY and destroys them when going from READY to NULL. Since shmsink is used from inside the sandbox, this means that the state change NULL to READY needs to happen before chrootme(). This also means that it won't be able to clean up properly the shared memory object and the control socket.
- fdsrc doesn't create nor destroy the fd it uses, so that can be done separately. Moreover, in the case of stdin, we leave that responsibility to the system.
And quite obviously, we want gst-decoder to handle buffers only after it has called chrootme(), so that it is ready to run potentially unsafe operations.
This is relatively easy: all we have to do is, in gst-decoder, to call chrootme() once we are in the READY state and before going to PAUSED.
Another issue with the privilege drop is that we use decodebin2 (things would be the same with decodebin), and it only loads the plugins it needs once it knows what kind of data it will have to decode. That is, it needs to load plugins after it has started to analyse potentially unsafe data. My solution to that is to preload all the installed plugins when gst-decoder starts, so that decodebin2 doesn't need any privilege to have access to the plugins it wants (they are already in memory).
This is obviously suboptimal in memory consumption. I can think of two ways to improve that:
- use a white/black list of plugins to avoid loading plugins we are not likely to need (there are many things we're pretty sure not to need in gst-decoder, such as all sources and sinks or gnonlin)
- use a separate typefinding sandboxed process that will determine what plugins are needed, then have gst-decoder take as argument the plugins that it needs to load before dropping privileges
Another synchronisation issue is that the broker has to wait for the sandboxed process to be ready before interacting with it. As seen before, we have 3 channels through which they interact, and they are of two different types:
- the pipe to which the broker writes, which points to stdin in the sandboxed process
- the shared memory areas, and their associated control sockets created by the two shmsink
The first one is easy to synchronise: as long as the sandboxed process is not ready, it won't read on the pipe, and fdsink on the broker will just wait until it can write.
The second one is more complex: the shared memory areas are announced over the control socket when they are ready, so this part gets done correctly for free by shmsrc. But the control sockets need to exist when shmsrc tries to connect to them (this happens when going from READY to PAUSED). For now, my workaround is to sleep() for 2 seconds when sandboxeddecodebin goes from NULL to READY, after launching the subprocess. With this, the control sockets are very likely to be created when shmsrc goes from READY to PAUSED.
This is obviously very hackish , and I think I would prefer to use GFileMonitors to check when the sockets are created. Also, I don't know if it's better to do that in sandboxeddecodebin (blocking the switch to READY, but using that file monitoring instead of a sleep(), or going to READY asynchronously if that's possible?) or in shmsrc (in which case I think it should be optional and probably make shmsrc go to PAUSED asynchronously).
On the broker side, we have another tricky situation. We typically run a pipeline that contains all of this (the parts between angle brackets are outside of sandboxeddecodebin and given as examples):
<filesrc> ! fdsink (passes data to gst-decoder) shmsrc (gets data from gst-decoder) ! gdpdepay ! <autoaudiosink> shmsrc (gets data from gst-decoder) ! gdpdepay ! <autovideosink>
This pipeline is atypical in that it has a sink that is not really at the downstream end of it (fdsink, which sandboxeddecodebin uses to pass data to gst-decoder). Data would go through it, then through gst-decoder and its own pipeline, and then emerge back in the broker's pipeline in the shmsrc elements.
This is a problem at the preroll phase. Preroll is what usually happens when going to PAUSED: the sinks wait until they have a buffer to render before committing the state to PAUSED. The issue with our pipeline, is that the "real" sinks will only get the data they need to commit to the PAUSED state if fdsink lets the data through, but fdsink only passes data once it is in PLAYING state (apart maybe from one initial buffer). On top of that sandboxeddecodebin is a subclass of GstBin. By default, GstBin only changes to the next state (e.g. PLAYING) once all its elements have reached the previous one (e.g. PAUSED). This gives us a nice deadlock: the final (downstream) sinks are waiting for data to come to them to commit their change to PAUSED, GstBin is waiting for all its elements (including final sinks) to finish their transition to PAUSED before asking them to go to PLAYING, and fdsink is waiting to be asked to switch to PLAYING before it lets the data through (that the final sinks are waiting on). My workaround to solve this deadlock is to manually request fdsink to go to PLAYING when sandboxeddecodebin is switching to PAUSED. That way, fdsink is "one state ahead" of the rest, and lets the data go through. I haven't decided yet if it's a very ugly way of solving that issue or if it's an awesome clever hack. If you have an idea of a cleaner solution, feel free to suggest it in the comments!
Once the privileges have been dropped, the sandboxed process is very limited in what it can do, but it still can use all the fds that it has open, which might be a way for it to escape the limitations we want to put on it. For instance, imagine that the sandboxed process has an open fd on the device that contains your home directory (say, /dev/sda). By reading it, it can access all your data, even though the sandbox is designed not to let it open more files.
This precise example is very unlikely to happen in our case, but some less obvious fds could lead to ways to escape the sandbox. That is why I think it is necessary to analyse the file descriptors that are open in the sandboxed process and to try to understand the risks they bring.
I took a "snapshot" of the open fds of gst-decoder while it was decoding a video, and here is what it looks like:
guijemont@thirtytwo:~$ ls -lv /proc/5860/fd total 0 lr-x------ 1 guijemont guijemont 64 2012-04-18 18:17 0 -> pipe:[8348338] lrwx------ 1 guijemont guijemont 64 2012-04-18 18:17 1 -> /dev/pts/5 lrwx------ 1 guijemont guijemont 64 2012-04-18 18:17 2 -> /dev/pts/5 lrwx------ 1 guijemont guijemont 64 2012-04-18 18:17 3 -> anon_inode:[eventfd] lr-x------ 1 guijemont guijemont 64 2012-04-18 18:17 4 -> pipe:[8348342] l-wx------ 1 guijemont guijemont 64 2012-04-18 18:17 5 -> pipe:[8348342] lrwx------ 1 guijemont guijemont 64 2012-04-18 18:17 6 -> socket:[8358884] lr-x------ 1 guijemont guijemont 64 2012-04-18 18:17 7 -> pipe:[8359036] l-wx------ 1 guijemont guijemont 64 2012-04-18 18:17 8 -> pipe:[8359036] lrwx------ 1 guijemont guijemont 64 2012-04-18 18:17 9 -> anon_inode:[timerfd] lrwx------ 1 guijemont guijemont 64 2012-04-18 18:17 10 -> /run/shm/shmpipe. 5860. 0 lrwx------ 1 guijemont guijemont 64 2012-04-18 18:17 11 -> socket:[8358886] lrwx------ 1 guijemont guijemont 64 2012-04-18 18:17 12 -> socket:[8358887] lrwx------ 1 guijemont guijemont 64 2012-04-18 18:17 13 -> socket:[8358888] lrwx------ 1 guijemont guijemont 64 2012-04-18 18:17 14 -> /run/shm/shmpipe. 5860. 1 lrwx------ 1 guijemont guijemont 64 2012-04-18 18:17 15 -> socket:[8358890] lrwx------ 1 guijemont guijemont 64 2012-04-18 18:17 16 -> socket:[8358891] lrwx------ 1 guijemont guijemont 64 2012-04-18 18:17 17 -> socket:[8358892] lrwx------ 1 guijemont guijemont 64 2012-04-18 18:17 18 -> socket:[8358893] lrwx------ 1 guijemont guijemont 64 2012-04-18 18:17 19 -> socket:[8358894] lrwx------ 1 guijemont guijemont 64 2012-04-18 18:17 20 -> socket:[8358895] lrwx------ 1 guijemont guijemont 64 2012-04-18 18:17 21 -> socket:[8348346] lrwx------ 1 guijemont guijemont 64 2012-04-18 18:17 22 -> socket:[8348347]
I used the "usual suspects" (strace and gdb) to look further into this and understand where each fd comes from, and try to get an idea of how necessary it is and of how much of a risk it brings.
You can check out the code from its github repository, instructions are available here.
I am now back from Prague where I gave a talk on image stabilisation (and my holiday pictures). Hopefully a video of the talk will soon be online. In the meantime, I would like to explain a bit my efforts in written form, with some details slightly updated from the talk (the code progressed a bit since then).
UPDATE: The talk is now online.
I got interested in the issues of image stabilisation through a helium balloon photography project in which I participated. I want to make a nice time lapse video from the pictures I have taken, but they were taken from a camera that was moving, which would make the result very shaky without some kind of postprocessing.
Thankfully, I work at Igalia, which means that on top of my personal time, I could spend on this project some company time (what we call internally our hackfest time, of up to 5 hours per week).
I have around 4h30 of pictures taken from a balloon 100 metres high. The pictures were taken at a rate of one per minute, which makes around 270 pictures. I want to make a nice time lapse out of it. Simply using the frames as is to build a video does not work well. Partly because I would probably be legally required to include a warning to epileptic people at the beginning of the video, but mostly because people actually watching it would wish they were epileptic to have a good excuse not to watch it.
This is due to the huge differences occurring between two consecutive frames.
Here is an example of two consecutive frames in that series:
As you can see, from one frame to the next, a lot of pixels would change. And that does not look pretty. It is also pretty obvious that they are both pictures of the same thing, and could be made to be pretty similar, mainly by rotating one of them, and maybe reprojecting it a bit so that things align properly even though the point of view changed a bit from one frame to the next.
There was no question in my mind that I wanted to use GStreamer for the task, by writing an element or set of elements to do the stabilisation. The two big advantages of this approach are:
Then, after some research, I realised that OpenCV provides a lot of the tools needed for the task as well.
Since I am still in a prototyping/research stage, and I hate to write loads of boilerplate, I am using python for that project, though a later rewrite in C or C++ is not impossible.
I will not present things exactly in the order I researched them, but rather in the order I should have researched them: starting with a simpler problem, then getting into the complications of my balloon problem.
The simpler problem at hand is presented to you by Joe the Hippo:
As you can see, Joe almost looks like he's on a boat. He isn't, but the cameraman is, and the video was taken with a lot of zoom. The movement in that video stream has a particularity that can make things simpler: the position of a feature on the screen does not change much from one frame to the next, because a very short amount of time happens between them. We will see that some potentially very useful algorithms take advantage of that particularity.
As I see it for the moment, there are two basic steps in image stabilisation:
Step 2. is made rather easy by OpenCV with the findHomography() and warpPerspective() functions, so we won't talk much about it here.
For all that matters in this study, we can say that for each frame the optical flow is represented by two lists of point coordinates origins and destinations, such that the feature at the coordinate origins[i] in the previous frame is at the coordinate destinations[i] in the current frame.
Optical flow algorithms can be separated in two classes, depending on whether they provide the flow for all pixels (Dense optical flow algorithms) or only for selected pixels (Sparse optical flow algorithms). Both classes can theoretically provide us with the right data (origins and destinations point lists) to successfully compute the opposite transformation we want to apply using findHomography().
I tried one algorithm of each class, choosing the ones that seemed popular to me after reading a bit of [Bradski2008]. Here is what I managed to do with them.
I tried to use OpenCV's implementation of the Horn-Schunck algorithm [Horn81]. I don't know if I used it incorrectly, or if the algorithm simply cannot be applied to that situation, but this is all I could do to Joe with that:
Now Joe is shaky and flickeryAs you can see, this basically added flickering. Since that, I did not find time to improve this case before I realised that this algorithm is considered as obsolete in OpenCV, and the new python bindings do not include it.
Note that this does not mean that dense optical flow sucks: David Jordan, a Google Summer of Code student, does awesome things with a dense algorithm by Proesmans et al. [Proesmans94].
I played with the Lucas-Kanade algorithm [Lucas81], with the implementation provided by OpenCV. Once I managed to find a good set of parameters (which are now the default in the opticalflowfinder element), I got pretty good results:
Joe enjoys the stability of the river bank, undisturbed by the movements of the water (video)And it is quite fast too. On my laptop (with an i5 processor), I can stabilise Joe the hippo in real time [1] (it is only a 640x480 video, though).
[1] | For those who attended my talk at the Gstreamer Conference 2011: yes, now it is proper real time, I optimised the code a bit. |
As we seen in the previous section, for a shaky hippo video, [Horn81] isn't any help, but [Lucas81] is pretty efficient. But can they be of any use for my balloon problem?
I won't show any video here, because there is nothing much to see. Instead, an explanation in pictures that show how the algorithms rate for the balloon time lapse.
This is what Horn-Schunck can do:
The picture shows two consecutive frames in the time lapse (the older is on the left). Each of the coloured lines goes from a point on the first image to the corresponding point on the second one, according to the algorithm (click on the image to see a larger version where the lines are more visible). Since Horn-Schunck is a dense algorithm, the coloured lines are only displayed for a random subset of points to avoid clutter.
Obviously, these lines show that the algorithm is completely wrong, and could not follow the big rotation happening between the two frames.
Does Lucas-Kanade rate better? Let's see:
This is the same kind of visualisation, except that there is no need to chose a subset, since the algorithm already does that.
As for the result, it might be slightly less wrong than Horn-Schunck, but Lucas-Kanade does not seem to be of any help to us either.
The issue here, as said earlier, is that these two algorithms, like most optical flow algorithms, are making the assumption that a given feature will not move more than a few pixels from one frame to the next (for some value of "a few pixels"). This assumption is very clever for typical video streams taken at 25 or 30 frames per second. Unfortunately, it is obviously wrong in the case of our stream, where the camera has the time to move a lot between two frames (which are captured one minute apart).
Is all hope lost? Of course not!
I found salvation in feature recognition. OpenCV provides a lot of feature recognition algorithms. I have tried only one of them so far, but I hope to find the time to compare it with others in the future.
The one I tried is SURF (for "Speeded Up Robust Features", [Bay06]). It finds "interesting" features in an image and descriptors associated with them. The descriptors it provides are invariant to rotation and scaling, which means that it is in theory possible to find the same descriptors from frame to frame.
To be able to efficiently compare the sets of frame descriptors I get for two consecutive frames, I use FLANN, which is well integrated in OpenCV.
Here is a visualisation of how this method performs:
As you can see, this is obviously much better! There might be a few outliers, but OpenCV's findHomography() can handle them perfectly, and here's a proof video (I am not including it in the article since it is quite high resolution).
Obviously, the result is not perfect yet (especially in the end), but it is quite promising, and I hope to be able to fix the remaining glitches sooner than later.
The code as well as a quick introduction on how to use it is available on github. Bugs and patches should be posted here.
[Bradski2008] | G. Bradski and A. Kaehler , "Learning OpenCV", ISBN 978-0-596-51613-0, 2008. |
[Horn81] | B. K. P. Horn and B. G. Schunck, “Determining optical flow,” Artificial Intelligence 17 (1981): 185–203, 1981. PDF |
[Proesmans94] | M. Proesmans, L. Van Gool, E. Pauwels, A. Oosterlinck, "Determination of optical flow and its discontinuities using non-linear diffusion", J.-O. Eklundh (Ed.), Computer vision -- ECCV '94, Lecture Notes in Comp. Science, Vol. 801, Springer, Berlin, 295–2304, 1994. PDF |
[Lucas81] | B. Lucas and T. Kanade, "An Iterative Image Registration Technique with an Application to Stereo Vision", Proc. of 7th International Joint Conference on Artificial Intelligence (IJCAI), 674–279 PDF |
[Bay06] | H. Bay, T. Tuytelaars and L. Van Gool, “SURF: Speeded Up Robust Features”, 9th European Conference on Computer Vision, 2006. PDF |
C:\Program Files\bazaar\plugins
. I don't remember if I managed to make it work or not (did I say it was a while ago?), but I eventually uninstalled it, as well as any dependency that I might have installed for the occasion. Ever since that day, whenever I ran a bzr command, I had the following output preceding the expected output of said command:
No Python bindings for Subversion installed. See the bzr-svn README for details.
Unable to load plugin u'bzr_svn' from u'C:/Program Files/Bazaar/plugins'
Even though I don't have any bzr_svn plugin in that directory! Yes, I tripled checked!
This had been driving me crazy for months now, and today, by pure chance, I stepped across a directory called C:\Users\guijemont\AppData\Local\VirtualStore
that contained a Program Files
directory. Having a look at it, I discovered, after months of struggling with that awful and undeserved error message that there was a:
C:\Users\guijemont\AppData\Local\VirtualStore\Program Files\bazaar\plugins\bzr_svn
Yeah, really. And it was full of .pyc files. What happened is that, whenever a user program tries to write somewhere where it shouldn't (such as Program Files
), UAC gives it the impression that it succeeded, and write the stuff in that VirtualStore instead of the real place. That is a convenience that might save the day to some programs that don't behave and write in places where they shouldn't.
Enters python. Python writes .pyc files, which are slightly compressed versions of the original .py files (and therefore faster to load). It writes them in the directory where the .py file resides (I guess it makes a lot of things easier to manage).
Then you mix both. Install bzr-svn, use it as a normal user, remove it: it's not removed! Because the .pyc files are still there, hidden in VirtualStore.
And they say that OS is user-friendly? As I user, I find it friendly that a directory is deleted when I delete it...
]]>
let &errorformat .= ',\%+G-version-info\ %*\\d:'
au FileType c compiler gcc
$ ./gobject/test 1000000
n=1000000
stress_set(): 1.440000
stress_get(): 0.480000
stress_get_set(): 1.940000
$
$ python python/test.py 1000000
n= 1000000
stress_set(): 0.29
stress_get(): 0.14
stress_get_set(): 0.34
$
Here, we don't want to precipitate things, so all of these solutions might not be in place in 0.5.0, though the whole project will be architectured with these in mind, so we can add them later in 0.5.x. Anyway, here is the list of solutions we've been thinking of:You can either be part of the solution, or you can be part of the precipitate.
bzr branch http://emont.org/paf/
The only interesting stuff so far is paf.xmi and the stuff in examples/.
Here is what this could look like, in Python, for a GObject image that has a property 'x' defining it's horizontal position:
# Move @image's x position linearly to @dest. Implicit version.
def implicitly_move_image (image, dest):
animator = paf.ImplicitAnimator(image)
animator.x = dest
I have still many things to figure out and to improve, but that's already a start and I plan to actively work on this, since we badly need that for Pigment.
When I start to be happy with the API, I will write the documentation and some empty C/GObject code, so that the output of gtk-doc can be used as a basis for discussion.
Of course, anyone is welcome to participate in this effort, feel free to branch and to send me patch/address of other branches. This will be LGPLv2.1 or later, unless I find good reasons to publish it under LGPLv3.
Please contact me if you have any comments, ideas questions or patches on this blog, on freenode/#pigment or by email (guillaume at fluendo dot com).
]]>Unfortunately, this gadget has a serious flaw: it's quite unstable. My guess is that it is still in an early stage of development and I was given an alpha release. I figured out that the instability is caused by the way the high-tech electronics are positioned. I don't have the necessary hardware skills to change the position of the electronics, but I found a simple solution to make it much more stable: simply pour rice in it.
After doing this, I've found it's much more stable. Yet, the solution is not perfect: it's not able to reach the same speed as before the hack, and I suspect it leaks from time to time, I wonder if valgrind can help me here :).
]]>Effectivement, j'ai posté 2 articles en peu de temps depuis. Il y a aussi que je suis passé à wordpress 2, qui est bien plus utilisable que le 1.5.x]]>
I would like to explain a bit the stuff I've been working on recently at Igalia. It is about playing with GStreamer and a sandboxing system to try and make the playback of untrusted media more secure. Hopefully writing this will be an occasion for me to get more distance and understand things better, and for others to give me feedback and ideas. Particularly, even though that is for me a field of interest, I do not claim to have any real expertise in security, therefore comments by people who know better would be gladly welcome.
This story started when I decided to have a look at chromium and its internals. It turns out that one very specific aspect of this application is its sandboxing system. In a nutshell, a sandbox is a virtual container in which untrusted programs can be safely run. In the real world, sandboxes are rarely perfect, but they are a significant security improvement over not using one. Chromium uses a sandbox to run its rendering engine (WebKit), which is basically the part that transforms the code of a web page into the graphical representation of it that you see on your screen. The rationale for running WebKit in a sandbox is not that it is untrusted code in itself, but rather that it is a big and complex project that is bound to have bugs, like all big and complex projects. On top of that, the input given to it is quite often data from untrusted sources, which could potentially be forged so that it exploits security bugs to do bad things to your computer and your beloved files. Now, with WebKit running in a sandbox, if a web page has been forged by an attacker to exploit a vulnerability in WebKit, the attacker will only have access to the sandbox environment, which means that it won't be able to do things like access the data on your computer, install software or connect to remote hosts.
As you might know, I like to play with multimedia things, and have hacked quite a bit on or around GStreamer. Therefore, I quite automatically thought of something else that might be worth running in a sandbox: demuxers and decoders. They are relatively big and complex pieces of software to which we regularly pass a whole bunch of untrusted data, would it be in a web context or a more traditional desktop or mobile context.
Fortunately, Julien Tinnes, a developer of the chromium sandbox for GNU/Linux made a stand alone version of it called setuid-sandbox, which can be used by other projects to easily sandbox any process.
The way setuid-sandbox works is rather straightforward: there is a sandboxme command that needs to be installed setuid root. You run sandboxme my_command and then from inside my_command, you first set up the file descriptors that you will need (being careful not to put there anything that could allow to escape the sandbox, more on that later), and then you call the provided chrootme() function, which will tell the sandboxme process to restrict the privileges that my_command has (e.g. it can still read and write on the fds that it has open, but it cannot open new ones).
Here is how I organised my integration of setuid-sandbox into GStreamer. What I want to do for now is to put what I think are the "most dangerous" parts (demuxing and decoding) in the sandbox, while letting the other components (mainly source and sinks) outside of the sandbox (for now at least). I decided to create a small program (called gst-decoder) that receives the original muxed and encoded video stream and outputs the decoded video and audio buffers. gst-decoder needs 3 channels of communication with the "controlling" process outside the sandbox (which is called the broker):
- one to pass the original stream from the source element in the broker to gst-decoder
- one to pass the video buffers from gst-decoder to the video sink element in the broker
- one to pass the audio buffers from gst-decoder to the audio sink element in the broker
In the future, more channels for subtitle support or other features could be desirable.
Since I am lazy, I wanted to use off the shelf GStreamer elements to handle these communication channels. For the cases explained above, that would be:
- the fdsink element on the broker side, and the fdsrc element in the sandbox
- shmsink (in gst-decoder) and shmsrc (in the broker)
- same elements as above
Since I expect other people to be equally lazy^W^W^Wwant their life to be made easier, my goal is to try and have this reasonably integrated in GStreamer, and easy to integrate in applications. For that, my best idea so far was to make a sandboxedecodebin element that, from the outside, works like decodebin or decodebin2, at least for simple cases: it has a sink pad that can take any format you would throw at decodebin, and it has an audio and a video source pads that output the decoded result. In the future, it might or might not be a good idea to try to integrate the "sandboxed" functionality in decodebin directly.
I implemented sandboxeddecodebin as a subclass of GstBin, and it has the following flow inside it:
fdsink -> [gst-decoder] | -> shmsrc (video) -> gdpdepay | -> shmsrc (audio) -> gdpdepay
Note that gst-decoder is an external (sandboxed) process, and not a GStreamer element like the other entities of this data flow graph. The sink pad of fdsink and the source pads of the two gdpdepay elements are exported by sandboxeddecodebin through ghost pads, which provides a decodebin-like interface.
The gst-decoder program basically runs a pipeline that looks like that:
fdsrc ! decodebin2 name=decoder decoder. ! video/x-raw-yuv;video/x-raw-rgb ! gdppay ! shmsink (video) decoder. ! audio/x-raw-int;audio/x-raw-float ! gdppay ! shmsink (audio)
and it also makes sure to get the privilege dropped at the right time, which is discussed below.
The ordering of operations needs to be thought carefully to combine GStreamer, and these elements in particular, with setuid-sandbox. Each of them brings its own set of conditions.
For setuid-sandbox, inside the sandbox (in gst-decoder):
- before we call chrootme(): we can open new fds and do a lot of nice initialisation, and we don't want to parse any untrusted data
- after we call chrootme(): we can't open new fds any more, or do similar initialisation tasks, but we can work on the data we received.
GStreamer has several states in which an element can be, with some rules on what should be done in which state. From the design documentation, the states are defined as follow:
- NULL: This is the initial state of an element.
- READY: The element should be prepared to go to PAUSED.
- PAUSED: The element should be ready to accept and process data. Sink elements however only accept one buffer and then block.
- PLAYING: The same as PAUSED except for live sources and sinks. Sinks accept and rendering data. Live sources produce data.
In particular, the elements that interest us here behave in the following way:
- shmsink is responsible for the creation and destruction of the shared memory object and the associated control socket and creates them when going from NULL to READY and destroys them when going from READY to NULL. Since shmsink is used from inside the sandbox, this means that the state change NULL to READY needs to happen before chrootme(). This also means that it won't be able to clean up properly the shared memory object and the control socket.
- fdsrc doesn't create nor destroy the fd it uses, so that can be done separately. Moreover, in the case of stdin, we leave that responsibility to the system.
And quite obviously, we want gst-decoder to handle buffers only after it has called chrootme(), so that it is ready to run potentially unsafe operations.
This is relatively easy: all we have to do is, in gst-decoder, to call chrootme() once we are in the READY state and before going to PAUSED.
Another issue with the privilege drop is that we use decodebin2 (things would be the same with decodebin), and it only loads the plugins it needs once it knows what kind of data it will have to decode. That is, it needs to load plugins after it has started to analyse potentially unsafe data. My solution to that is to preload all the installed plugins when gst-decoder starts, so that decodebin2 doesn't need any privilege to have access to the plugins it wants (they are already in memory).
This is obviously suboptimal in memory consumption. I can think of two ways to improve that:
- use a white/black list of plugins to avoid loading plugins we are not likely to need (there are many things we're pretty sure not to need in gst-decoder, such as all sources and sinks or gnonlin)
- use a separate typefinding sandboxed process that will determine what plugins are needed, then have gst-decoder take as argument the plugins that it needs to load before dropping privileges
Another synchronisation issue is that the broker has to wait for the sandboxed process to be ready before interacting with it. As seen before, we have 3 channels through which they interact, and they are of two different types:
- the pipe to which the broker writes, which points to stdin in the sandboxed process
- the shared memory areas, and their associated control sockets created by the two shmsink
The first one is easy to synchronise: as long as the sandboxed process is not ready, it won't read on the pipe, and fdsink on the broker will just wait until it can write.
The second one is more complex: the shared memory areas are announced over the control socket when they are ready, so this part gets done correctly for free by shmsrc. But the control sockets need to exist when shmsrc tries to connect to them (this happens when going from READY to PAUSED). For now, my workaround is to sleep() for 2 seconds when sandboxeddecodebin goes from NULL to READY, after launching the subprocess. With this, the control sockets are very likely to be created when shmsrc goes from READY to PAUSED.
This is obviously very hackish , and I think I would prefer to use GFileMonitors to check when the sockets are created. Also, I don't know if it's better to do that in sandboxeddecodebin (blocking the switch to READY, but using that file monitoring instead of a sleep(), or going to READY asynchronously if that's possible?) or in shmsrc (in which case I think it should be optional and probably make shmsrc go to PAUSED asynchronously).
On the broker side, we have another tricky situation. We typically run a pipeline that contains all of this (the parts between angle brackets are outside of sandboxeddecodebin and given as examples):
<filesrc> ! fdsink (passes data to gst-decoder) shmsrc (gets data from gst-decoder) ! gdpdepay ! <autoaudiosink> shmsrc (gets data from gst-decoder) ! gdpdepay ! <autovideosink>
This pipeline is atypical in that it has a sink that is not really at the downstream end of it (fdsink, which sandboxeddecodebin uses to pass data to gst-decoder). Data would go through it, then through gst-decoder and its own pipeline, and then emerge back in the broker's pipeline in the shmsrc elements.
This is a problem at the preroll phase. Preroll is what usually happens when going to PAUSED: the sinks wait until they have a buffer to render before committing the state to PAUSED. The issue with our pipeline, is that the "real" sinks will only get the data they need to commit to the PAUSED state if fdsink lets the data through, but fdsink only passes data once it is in PLAYING state (apart maybe from one initial buffer). On top of that sandboxeddecodebin is a subclass of GstBin. By default, GstBin only changes to the next state (e.g. PLAYING) once all its elements have reached the previous one (e.g. PAUSED). This gives us a nice deadlock: the final (downstream) sinks are waiting for data to come to them to commit their change to PAUSED, GstBin is waiting for all its elements (including final sinks) to finish their transition to PAUSED before asking them to go to PLAYING, and fdsink is waiting to be asked to switch to PLAYING before it lets the data through (that the final sinks are waiting on). My workaround to solve this deadlock is to manually request fdsink to go to PLAYING when sandboxeddecodebin is switching to PAUSED. That way, fdsink is "one state ahead" of the rest, and lets the data go through. I haven't decided yet if it's a very ugly way of solving that issue or if it's an awesome clever hack. If you have an idea of a cleaner solution, feel free to suggest it in the comments!
Once the privileges have been dropped, the sandboxed process is very limited in what it can do, but it still can use all the fds that it has open, which might be a way for it to escape the limitations we want to put on it. For instance, imagine that the sandboxed process has an open fd on the device that contains your home directory (say, /dev/sda). By reading it, it can access all your data, even though the sandbox is designed not to let it open more files.
This precise example is very unlikely to happen in our case, but some less obvious fds could lead to ways to escape the sandbox. That is why I think it is necessary to analyse the file descriptors that are open in the sandboxed process and to try to understand the risks they bring.
I took a "snapshot" of the open fds of gst-decoder while it was decoding a video, and here is what it looks like:
guijemont@thirtytwo:~$ ls -lv /proc/5860/fd total 0 lr-x------ 1 guijemont guijemont 64 2012-04-18 18:17 0 -> pipe:[8348338] lrwx------ 1 guijemont guijemont 64 2012-04-18 18:17 1 -> /dev/pts/5 lrwx------ 1 guijemont guijemont 64 2012-04-18 18:17 2 -> /dev/pts/5 lrwx------ 1 guijemont guijemont 64 2012-04-18 18:17 3 -> anon_inode:[eventfd] lr-x------ 1 guijemont guijemont 64 2012-04-18 18:17 4 -> pipe:[8348342] l-wx------ 1 guijemont guijemont 64 2012-04-18 18:17 5 -> pipe:[8348342] lrwx------ 1 guijemont guijemont 64 2012-04-18 18:17 6 -> socket:[8358884] lr-x------ 1 guijemont guijemont 64 2012-04-18 18:17 7 -> pipe:[8359036] l-wx------ 1 guijemont guijemont 64 2012-04-18 18:17 8 -> pipe:[8359036] lrwx------ 1 guijemont guijemont 64 2012-04-18 18:17 9 -> anon_inode:[timerfd] lrwx------ 1 guijemont guijemont 64 2012-04-18 18:17 10 -> /run/shm/shmpipe. 5860. 0 lrwx------ 1 guijemont guijemont 64 2012-04-18 18:17 11 -> socket:[8358886] lrwx------ 1 guijemont guijemont 64 2012-04-18 18:17 12 -> socket:[8358887] lrwx------ 1 guijemont guijemont 64 2012-04-18 18:17 13 -> socket:[8358888] lrwx------ 1 guijemont guijemont 64 2012-04-18 18:17 14 -> /run/shm/shmpipe. 5860. 1 lrwx------ 1 guijemont guijemont 64 2012-04-18 18:17 15 -> socket:[8358890] lrwx------ 1 guijemont guijemont 64 2012-04-18 18:17 16 -> socket:[8358891] lrwx------ 1 guijemont guijemont 64 2012-04-18 18:17 17 -> socket:[8358892] lrwx------ 1 guijemont guijemont 64 2012-04-18 18:17 18 -> socket:[8358893] lrwx------ 1 guijemont guijemont 64 2012-04-18 18:17 19 -> socket:[8358894] lrwx------ 1 guijemont guijemont 64 2012-04-18 18:17 20 -> socket:[8358895] lrwx------ 1 guijemont guijemont 64 2012-04-18 18:17 21 -> socket:[8348346] lrwx------ 1 guijemont guijemont 64 2012-04-18 18:17 22 -> socket:[8348347]
I used the "usual suspects" (strace and gdb) to look further into this and understand where each fd comes from, and try to get an idea of how necessary it is and of how much of a risk it brings.
You can check out the code from its github repository, instructions are available here.
I will give a lightning talk, Saturday at 13:40, about my balloon adventures at Nowhere, with an emphasis on how Free Software made it possible. Hopefully, this will motivate other Free Software hackers to get out of their basement and hack in the “real” world :).
The talk should be streamed live at this url (in the Ferrer room).
Many other Igalians will come (I think there will be 14 of us), and will give talks.
]]>I am now back from Prague where I gave a talk on image stabilisation (and my holiday pictures). Hopefully a video of the talk will soon be online. In the meantime, I would like to explain a bit my efforts in written form, with some details slightly updated from the talk (the code progressed a bit since then).
UPDATE: The talk is now online.
I got interested in the issues of image stabilisation through a helium balloon photography project in which I participated. I want to make a nice time lapse video from the pictures I have taken, but they were taken from a camera that was moving, which would make the result very shaky without some kind of postprocessing.
Thankfully, I work at Igalia, which means that on top of my personal time, I could spend on this project some company time (what we call internally our hackfest time, of up to 5 hours per week).
I have around 4h30 of pictures taken from a balloon 100 metres high. The pictures were taken at a rate of one per minute, which makes around 270 pictures. I want to make a nice time lapse out of it. Simply using the frames as is to build a video does not work well. Partly because I would probably be legally required to include a warning to epileptic people at the beginning of the video, but mostly because people actually watching it would wish they were epileptic to have a good excuse not to watch it.
This is due to the huge differences occurring between two consecutive frames.
Here is an example of two consecutive frames in that series:
As you can see, from one frame to the next, a lot of pixels would change. And that does not look pretty. It is also pretty obvious that they are both pictures of the same thing, and could be made to be pretty similar, mainly by rotating one of them, and maybe reprojecting it a bit so that things align properly even though the point of view changed a bit from one frame to the next.
There was no question in my mind that I wanted to use GStreamer for the task, by writing an element or set of elements to do the stabilisation. The two big advantages of this approach are:
Then, after some research, I realised that OpenCV provides a lot of the tools needed for the task as well.
Since I am still in a prototyping/research stage, and I hate to write loads of boilerplate, I am using python for that project, though a later rewrite in C or C++ is not impossible.
I will not present things exactly in the order I researched them, but rather in the order I should have researched them: starting with a simpler problem, then getting into the complications of my balloon problem.
The simpler problem at hand is presented to you by Joe the Hippo:
As you can see, Joe almost looks like he’s on a boat. He isn’t, but the cameraman is, and the video was taken with a lot of zoom. The movement in that video stream has a particularity that can make things simpler: the position of a feature on the screen does not change much from one frame to the next, because a very short amount of time happens between them. We will see that some potentially very useful algorithms take advantage of that particularity.
As I see it for the moment, there are two basic steps in image stabilisation:
Step 2. is made rather easy by OpenCV with the findHomography() and warpPerspective() functions, so we won’t talk much about it here.
For all that matters in this study, we can say that for each frame the optical flow is represented by two lists of point coordinates origins and destinations, such that the feature at the coordinate origins[i] in the previous frame is at the coordinate destinations[i] in the current frame.
Optical flow algorithms can be separated in two classes, depending on whether they provide the flow for all pixels (Dense optical flow algorithms) or only for selected pixels (Sparse optical flow algorithms). Both classes can theoretically provide us with the right data (origins and destinations point lists) to successfully compute the opposite transformation we want to apply using findHomography().
I tried one algorithm of each class, choosing the ones that seemed popular to me after reading a bit of [Bradski2008]. Here is what I managed to do with them.
I tried to use OpenCV’s implementation of the Horn-Schunck algorithm [Horn81]. I don’t know if I used it incorrectly, or if the algorithm simply cannot be applied to that situation, but this is all I could do to Joe with that:
As you can see, this basically added flickering. Since that, I did not find time to improve this case before I realised that this algorithm is considered as obsolete in OpenCV, and the new python bindings do not include it.
Note that this does not mean that dense optical flow sucks: David Jordan, a Google Summer of Code student, does awesome things with a dense algorithm by Proesmans et al. [Proesmans94].
I played with the Lucas-Kanade algorithm [Lucas81], with the implementation provided by OpenCV. Once I managed to find a good set of parameters (which are now the default in the opticalflowfinder element), I got pretty good results:
Joe enjoys the stability of the river bank, undisturbed by the movements of the water (video)
And it is quite fast too. On my laptop (with an i5 processor), I can stabilise Joe the hippo in real time [1] (it is only a 640×480 video, though).
[1] | For those who attended my talk at the Gstreamer Conference 2011: yes, now it is proper real time, I optimised the code a bit. |
As we seen in the previous section, for a shaky hippo video, [Horn81] isn’t any help, but [Lucas81] is pretty efficient. But can they be of any use for my balloon problem?
I won’t show any video here, because there is nothing much to see. Instead, an explanation in pictures that show how the algorithms rate for the balloon time lapse.
This is what Horn-Schunck can do:
The picture shows two consecutive frames in the time lapse (the older is on the left). Each of the coloured lines goes from a point on the first image to the corresponding point on the second one, according to the algorithm (click on the image to see a larger version where the lines are more visible). Since Horn-Schunck is a dense algorithm, the coloured lines are only displayed for a random subset of points to avoid clutter.
Obviously, these lines show that the algorithm is completely wrong, and could not follow the big rotation happening between the two frames.
Does Lucas-Kanade rate better? Let’s see:
This is the same kind of visualisation, except that there is no need to chose a subset, since the algorithm already does that.
As for the result, it might be slightly less wrong than Horn-Schunck, but Lucas-Kanade does not seem to be of any help to us either.
The issue here, as said earlier, is that these two algorithms, like most optical flow algorithms, are making the assumption that a given feature will not move more than a few pixels from one frame to the next (for some value of “a few pixels”). This assumption is very clever for typical video streams taken at 25 or 30 frames per second. Unfortunately, it is obviously wrong in the case of our stream, where the camera has the time to move a lot between two frames (which are captured one minute apart).
Is all hope lost? Of course not!
I found salvation in feature recognition. OpenCV provides a lot of feature recognition algorithms. I have tried only one of them so far, but I hope to find the time to compare it with others in the future.
The one I tried is SURF (for “Speeded Up Robust Features”, [Bay06]). It finds “interesting” features in an image and descriptors associated with them. The descriptors it provides are invariant to rotation and scaling, which means that it is in theory possible to find the same descriptors from frame to frame.
To be able to efficiently compare the sets of frame descriptors I get for two consecutive frames, I use FLANN, which is well integrated in OpenCV.
Here is a visualisation of how this method performs:
As you can see, this is obviously much better! There might be a few outliers, but OpenCV’s findHomography() can handle them perfectly, and here’s a proof video (I am not including it in the article since it is quite high resolution).
Obviously, the result is not perfect yet (especially in the end), but it is quite promising, and I hope to be able to fix the remaining glitches sooner than later.
The code as well as a quick introduction on how to use it is available on github. Bugs and patches should be posted here.
[Bradski2008] | G. Bradski and A. Kaehler , “Learning OpenCV”, ISBN 978-0-596-51613-0, 2008. |
[Horn81] | B. K. P. Horn and B. G. Schunck, “Determining optical flow,” Artificial Intelligence 17 (1981): 185–203, 1981. PDF |
[Proesmans94] | M. Proesmans, L. Van Gool, E. Pauwels, A. Oosterlinck, “Determination of optical flow and its discontinuities using non-linear diffusion”, J.-O. Eklundh (Ed.), Computer vision — ECCV ’94, Lecture Notes in Comp. Science, Vol. 801, Springer, Berlin, 295–2304, 1994. PDF |
[Lucas81] | B. Lucas and T. Kanade, “An Iterative Image Registration Technique with an Application to Stereo Vision”, Proc. of 7th International Joint Conference on Artificial Intelligence (IJCAI), 674–279 PDF |
[Bay06] | H. Bay, T. Tuytelaars and L. Van Gool, “SURF: Speeded Up Robust Features”, 9th European Conference on Computer Vision, 2006. PDF |
Tomorrow I will fly to Prague, going to the GStreamer Conference, then to LinuxCon Europe and ELCE.
I’m excited to go back there, after having first visited this beautiful city in my last Eurotrip.
I will give a talk at the GStreamer Conference about my work on image stabilisation, related to that balloon project we did with Ugo.
I will go there sponsored by Igalia, with my friends and colleagues Víctor, Philippe and Javi. Víctor will give a talk as well, about the integration of syslink and GStreamer.
If I manage better than some american guys, I will get to Berlin on Friday afternoon for the desktop summit:
I am going there sponsored by the awesome company I work for, Igalia. Many other igalians will be there too.
Grilo will be well represented, with a short talk by Philippe on his integration work of Grilo with the Gnome Shell, and a hacking session with Juan and myself.
I will stay in Berlin until the 15th of August, to attend the GObject Introspection hackfest, where I will also be sponsored by Igalia. I should admit I don’t know much about the internals of GI yet, but I am excited to “learn by hacking” in the company of great hackers!
Then I will continue my Eurotrip, but that is another story.
]]>I hope I managed to do the transition correctly, and that people following me through a feed reader and/or a planet won’t be flooded with a truckload of old articles. If that still happens, I apologise. Also, I know the alert readership that you are noticed that I went back to the default wordpress theme: this is by design (and a bit by laziness). I got tired of the old design, which was ugly anyway (it was hacked together from various bits by a very amateurish designer: me). So, here’s something sober, just like I want it to be.
As for more generic news, I am still happy, enjoying life in Barcelona, loving my job at Igalia, hacking on cool Grilo stuff (and sometime GStreamer stuff as well). Hopefully I will post some things regarding that in the not so distant future.
This might be difficult because most of my spare time these days is taken by what I call the “balloon project”, recently started with a few gifted friends of mine. More on that soon, I promise, as I have plenty of things to tell about this project that involves developments in a lot of domains in which I’d love to know more but am always discovering things, such as physics, electronics or computer science.
]]>
Like a few friends have stated, I am now saying it loud and clear:
And for that I should thank the awesome company for which I work. I am sure great time will be had, as well as great conversations about free software, multimedia, technology, life, the universe and everything.
See you all at FOSDEM!
]]>The two main cool things you can do in python, is define new commands, and new convenience functions. Unfortunately, as you can read in this article, defining a new command requires a lot of boilerplate (and it’s the same for convenience functions).
Since I wanted to write a few commands, I couldn’t be bothered to copy-paste that boilerplate, so I ended up writing a nice magical class that makes registering a function easier. You can find that here.
With this new weapon in hand, I wrote new commands to be able to follow what happens in a GstAdapter. Basically, I wanted to be able to track the position in the original file of each byte that was going out of a given adapter. So I wrote some code to be called by gdb commands when some operations on a GstAdapter are done, doing the necessary calculations and storing, and voilà: I can print, with gdb, the file position of the data obtained with each call to gst_adapter_peek().
All the code and the gdb script can be found over there.
As you can see, a gdb script is still needed. I haven’t yet really tried to play with breakpoints in python, but I have a feeling this is not totally trivial, if at all possible. The other great limitation is speed. It feels like the calls to python stuff from gdb are very costly, and, in the case of GStreamer, don’t expect smooth playing if you use some gdb command implemented in python for every buffer transmitted.
Conclusion: it’s got limits, but overall, this ability to enhance gdb with python allows you to do things you couldn’t do easily otherwise, like store data in complex structures for your debugging, and prototype easily some debugging actions you want to do, where the only alternative is to write it in C and include it in the application/library you are debugging.
]]>At least for starters, and to flex my hacking muscles a bit, I’m going to hack on various gstreamer stuff, which is always fun. Also, I think this will be my first post to appear on planet Igalia, so: HI PLANET READERS!
Oh yeah, as the title says, I’ve just changed flat, am now living in a great flat in a brand new building with cool flatmates, so that’s good news as well! And yes, I’m still in Barcelona, now in Poble Sec, which proves a cool area so far.
]]>