Loud ramblings of a Software Artisan

Friday 20 October 2017

Lightroom exporter/importer

Importing from legacy proprietary software is always important to liberate one's data and help migrate to libre solutions. To that effect, I wrote, in Rust, lrcat-extractor. A library (crate in Rust parlance) with a command line tool to extract and dump Lightroom catalogs, the central part of the user generated data.

Currently only the 2-version-ago Lightroom 4 catalogs are supported as it is the only version I have. Supporting Lightroom 6 is just a matter of me purchasing it and spending some time.

The goal is of course to use this crate in Niepce to import Lightroom 4 catalogs directly.

Head to the lrcat-extractor repository if you are curious.

Also as aside note, I also have the equivalent for the now discontinued Apple Aperture that I started to write almost two years ago as a practical exercise to learn Rust, and that I used as a base. Check out aplib-extractor.

The next step is to implement the actual import, probably filling the gaps in the API.

Update: and before someone says it, the timing of this project is quite coincidental.

Thursday 19 October 2017

Porting to Rust

I have started something crazy a few weeks ago: porting my app from C++ to Rust.

I'll talk about it here, in several parts.

This is Part 1.

Context

Once upon a time I started to write a Free Software photography application for GNOME. That was around 10 years ago.

At the time I chose to use C++ and Gtkmm 2.x. C++ because it provide unprecedented interoperability with existing C code while offering a stronger type system, RAII and template which allow safer programing pattern without a large overhead at runtime. Absolutely no regrets on that.

Over the years, despite never having been released, the code base was updated to C++11 and Gtkm 3.x.

Meanwhile Rust was developed at Mozilla. And Rust provide safety and high performance, but doesn't have the same ease of interoperability with a C codebase - albeit still make it easy.

Why

Why am I porting to Rust?

  • Rust is designed to be a safe language. In this day an age of 0day due to the use of C (and C++) and their unsafe practice, it is something important.
  • I want the application to be multi-threaded (it already is in design), which is one of the design goal of Rust.
  • I started writing other stand alone components in Rust.

The goal is to progressively rewrite the application in Rust, mostly translating the logic from C++ to Rust. It is not the first time I do that across languages. I'll go from the back to the front, the front being the UI written with Gtkmm.

Several steps.

Build system

There have already been a few posts about integrating Rust into an automake based build system, by Federico or by me. The gist of it is to make sure you build your library with Rust and link it to your application.

Building Rust code is trivial using cargo. There is no point using anything else.

Interfaces

When doing an ad-hoc conversion, the biggest problem is calling code write in one language from the other.

You can not call Rust methods or generic functions from C/C++, so you'll need to have some interface in place. Rust has made it relatively easy by using C calling convention and making easy to declare a function with mangling disabled.

Example:

#[no_mangle]
pub extern fn my_function(arg1: &Type1) -> RetType {

}

From C/C++ this would be something like:

extern "C" RetType my_function(const Type1* arg1);

Here we assume that RetType is a POD (Plain Old Datatype, like int, char, float), not a C++ class, nor a struct in Rust.

There is a tool called cbindgen that will generate C/C++ header from the Rust code. You can call it from the Rust build system to do that automatically. More on that later.

Calling C code from Rust is easy provided you have the proper interfaces. It is basically the reverse of the above.

But calling C++ methods on the other hand, you have to use bindgen.

Bindgen

Bindgen will generate from C/C++ header Rust modules to call C++ code.

The tool is pretty amazing to not say magical. Albeit there are some pitfalls.

For example a C++ class with a std::vector<> as a member will a problem if treated as opaque type. To be honest I am not sure if it would work as a non opaque type.

The use of boost might actually generate Rust code that doesn't compile.

Running the Rust test with cargo test will fail. I think it is Issue 1018.

There are a few things I recommend. Most notably, treat types as opaque whenever possible ; whitelist types and functions rather than blacklist. Your mileage may vary but this is what I got to work in my case.

Using these tools I managed to actually rewrite the large parts of the backend in Rust with equal functionality. Progressively more C++ code will be replaced.

I'll get into detail in a future post.

Friday 20 February 2009

Shave

@ebassi: just pushed shave support to Niepce. Does that count as number 2?

Thursday 12 February 2009

Moved to git

Thanks to Owen, Niepce Digital source code repository has been moved to git.

See the cgit interface for the details.

Wednesday 24 December 2008

SVN moved

Niepce Digital SVN has moved from gna.org to GNOME's.

Details in the GNOME wiki. For the long story short, the history has been lost. Thanks to SVN broken design, git-svn broken experince and my impatience combined.

Thursday 21 August 2008

Linux and digital photography

Joel Cornuz posted and excellent summary about Linux and digital photography.

That summarize pretty well my goals with ''Niepce Digital", with one more priority added: Freedom. Not that it is the exhaustive list of what you'll find at first, but it clearly describe what I think needs fixing. Fortunately, with the power of Free Software, lot of thing can be reused and/or shared.

Saturday 26 July 2008

RAW, RAW, RAW

During my demo at GUADEC, I only showed JPEG images. But RAW support is supposed to be central to Niepce Digital. Here we go. Leveraging libopenraw and GEGL, I now have basic RAW demosaicizing working. It needs several adjustments, but it is there, much better than what I achieved before.

Screenshot:

GEGL needs to be build with the "workshop" (see configure). I already have updated my packages for openSUSE. The code will soon be pushed to SVN.

Wednesday 16 July 2008

Introducing...

A while back I posted and teaser, but that was all until GUADEC where, with 3 lightning talk I could present the stack and and application I'm working on, on my copious spare time(tm). Let me introduce you to Niepce Digital. For what Niepce means, there is Wikipedia. For what are the plans, read on.

Niepce Digital is meant to be an application for digital photography that will allow end to end management of the workflow, starting with the acquisition from the camera to the final printing or publishing. For those who chose the easy way it is in the same idea than Adobe Ligthroom or Apple Aperture. I have had some concept written down on paper for over 3 years, before these applications were unleashed, I just didn't manage to make anything out of it. And of course it is Free Software, that goes without saying (licensed under GPLv3 or later).

Design goal

From a development point of view, the design goal is to:

  • try to provide, leverage or reuse infrastructure. In that case this mean using Exempi, libopenraw and GEGL, and making sure the whole processing pipeline can be reused. There is hope with GEGL being used by Gimp that some of the image processing can really be shared. I started Exempi especially with that application in mind, and I'm still developing libopenraw in parallel.
  • try to make it right: don't cut corner when implementing be rather take the time to do the right thing so that in the future it doesn't have to be redone. Of course this means that the initial shot will take more time but in the end the benefit is that things will be easier.
  • try to do things progressively but surely. Better have a well working limited set of feature than a half working extended set of features.
  • try to be modular so that it is easy to implement new modules.

Feature set

The feature set I wish to see implemented is quite extensive, and the application will be divided in "modules". The first one, the library is meant to manage the collection of pictures, add metadata, etc. The second module, darkroom, is meant to perform the image editing. Image editing is not like a full blown version of GIMP, but rather a limited and most used set: image adjustements, crop, straighten, dust removal, etc, all performed in a non-destructive mode. This will be for the first milestone, with maybe a few upload modules to export to you favorite image hosting.

Digital camera RAW files will be first class citizen, and actually they should provide the best result, and not be harder to use than JPEG. Metadata will be centered around XMP for maximum interoperability. And of course, color management will be.

This first milestone is quite ambitious, and having a very good output quality will already an achievement.

As a second milestone will come web gallery generation (static HTML) and printing. That last part might actually be a lot of work, including vertically down the stack.

Now what?

I have setup for a while a project on GNA to host the source code and the repository. You can grab the C++ source code and play with it.

But there won't be a release for a while. See the first milestone above.