As I previously talked about, I started porting libopenraw to Rust. It is now in a state where it has more feature than the original.

When I started writing this post, I didn't have 100% of the code Rust, but since I have removed the last bit of C++, for which I had cut corners to make sure to have a functional API for C.

The only C++ code left is the various utilities and the C++ test suite to validate.

The goal

The goal of the Rust rewrite is to have the digital camera raw parsing library written in Rust instead of C++, while still being available with a C API.

The goal of the updated C API is to be close to the original API. However it's also a good time to do some breaking changes. Notably most of the "objects" are now immutable.

Rewrite

I did the rewrite in a branch, aiming to provide the same level of functionality as the C++ code. One of the first step was to rewrite the test suite to use the same data set. Doing so did allow verifying the consistency in behaviour, a progressively implement all the features and formats.

The C++ code use inheritance a bit to override some of the behaviours based on which file format it was. One of the thing most camera raw file have in common is that they are mostly a TIFF file. However Rust doesn't do object-oriented. It has traits that allow implementing some level of polymorphism, but in that case the concept had to be rethought a little bit.

Benefits

This is where Rust shines.

First there are a lot of things Rust, or the ecosystem around, did for me. In C++ I had re-implemented a stream system to allow IO on buffers and handling endianess. In Rust you can just read from a slice (a fixed size array of data) with the Read trait, and there are built-in functions to read endian-specific values. I still have the memory of trying to figure out which include to add depending on the UNIX flavour, between Linux, macOS and the BSD.

Second, the Debug trait. Just derive Debug (it's a simple macro) and you can "print" an type like that, using string formats. In C++, well, it's a lot of work. And on enum types it will print a human readable value, ie the one you write in code.

Third, the safety. A lot of the safety feature of Rust prevent mistake. And at runtime, the bounds checking catch of a lot of things too. Being required to use Option<> or Result<> to handle cases where in C++ you'd get some null value.

At that point, a few bugs I identified porting / rewriting the code made their way to the 0.3.x series in C++.

Drawbacks

libopenraw requires a Rust compiler (technically it already did, but I know some packagers that did disable Rust¹. This mean that non current architecture might not be supported. To me it's not a big issue, I target modern computers.

Testing

As mentioned I rewrote the test suite with the same data set. This is an essential part of making sure the features work as previously.

I also rewrote ordiag and the dumper ordumper. The dumper allow analyzing the structure of the file, and I didn't have this one in C++ (instead I had a more primitive exifdump). The dumper is heavily inspired from exifprobe that has served me well and that I forked. Really I can see the amount of work I didn't need to do with Rust.

To add to this I wrote libopenraw-viewer, a small Rust application to view the content of camera raw files. This allow much more easily to see the output. This has helped me to find fundamental bugs in some of the parsing that led to some fixes "upstream", namely into the C++ version (0.3.x branch). I should have done that a long time ago. This also allow me to test the Rust API.

C API adaptation

Last but not least I had to provide a C API. This allow using the library.

Rust to C ffi has limitations. You can basically pass numbers, pointers, and eventually simple structures.

The libopenraw API did return several types of "ref" which are just pointers to opaque types. The idea was always to only have opaque types through the API, which internally were C++ instances.

One of the key change is that the only object that can be explicitly be created with the API is ORRawFileRef, because it's the entry point. Very few need to be released, most are held by the containing objects.

Some other constraints of the C API directed some choices in the Rust API.

• Raw files are no longer Box<> but Rc<> due to the need to retain them for the iterator.²
• Metadata::String() contain a Vec<u8> instead of a String to allow for the NUL terminated strings in the C API. They are maybe not NUL terminated but ASCII.

New features

Meanwhile I added a few new features:

• extracting the white balance on most files (saveral Nikon are not yet handled).
• color converting from the camera space to sRGB on rendering.
• a multi stage processing pipeline.

This is still not enough to have a complete processing pipeline, but it's a start. It's going towards the only two issue left in the issue tracker. Not that I don't expect more but it's a nice goal post.

Integration

I submitted a PR for Miniaturo to use the Rust version of the library. This is not ready to be merged, but this actually allowed me to fix a few API bits. The new Rust API is relatively close to the old API that was the Rust to C bindings.

This is the July 2023 update for Niepce.

The importer

Where we left, the workspace tree view didn't display the hierarchy.

First, I discovered bug in the SQL triggers for folder path update. I was missing the AFTER keyword to have it run after the update.

Workspace

Second, I need to specify a parent when adding items to the model, then locate the parent, add to it. The problem is that we get a list of folders in some order (I can order by any of the SQL columns). The problem is adding an item to a parent that is not yet int the model. There are few ways to go about it.

1. Sort the folder list in order of a flattened tree. To be fair my SQL skills are not good enough, and I'm not even sure I can do that in sqlite. This would solve most of the problems.
2. Sort the folder list once received. But this might not always work.
3. Handle the case in the model: add with a placeholder parent and re-parent as needed when the parent is added.

For now I chose 3. The risk is that if the parent doesn't exist then the tree will stay lingering. It's just the view though.

Here is the result:

This is the June 2023 update for Niepce.

The importer

Back where we left, still working on the import.

It looks like there is more work than anticipated to be able to import a tree a files.

If you are looking for a Software Engineer with strong skills in C, C++, Rust and JavaScript, with a strong FLOSS background, I am available to hire.

My résumé

This is the May 2023 update for Niepce.

Life comes at you fast. And hit hard. tl;dr not as much progress as I wished: I had to put this project a bit more on the sideline.

This will be a short update.

The importer

Some small bits:

Some fixing in the metadata processing on import, notably a better handling of raw files. Turns out the previous logic broke getting metadata from video files.

Also fixing some rexiv2 / gexiv2-sys bugs that are mostly memory leaks.

I am pondering directly binding Exiv2 now that 0.28 got rid of auto_ptr<> that had been deprecated for over a decade. cxx should make this easier. This would automatically resolve the problem above, and I don't need to bind all the API, just what I need.

Forecast

To move forward the importer, I need to fix the recursive creation of folders (it currently flatten them to a single level).

Misc

There are always the other fixes.

Fedora 38

I pulled the trigger and updated to Fedora 38. And Niepce failed to build because of a bug with bindgen with clang 16, that was already fixed, triggered by libgphoto2-sys. Submitted the PR for the crate and we are good to go. The short version is that clang 16 sent different data for anonymous enums, that bindgen 0.60 couldn't handle. But at that bindgen 0.65 was fine.

This is the reason why I always check into git the generated bindings and update them as needed, instead of doing it at build time.

Application ID

I have been using org.gnome.Niepce as an application ID. While it is hosted in the main namespace repositories (it was back in the days of Subversion, to which I am thankful), Niepce is not a core app. Policies with the GNOME project do not allow using that namespace for non core apps. So I had a perform a global change rename. Not a big deal, and it doesn't change anything for users since there is no release. I just needed to get this out of the way.

libopenraw

Pushed a bit on the Rust port to implement metadata extraction. This is driven by the goal of not having three different libraries. Still some parity gaps with the C++ code, but it's closing in. I hope to be able to release 0.4.0 based on the Rust code.

Thanks you for reading.

This is the April 2023 update for Niepce. Between outages caused by an unseasonbly ice storm, squirrels chewing cable, I discovered how painful is being off the grid. This is not the excuse, just the events that lead to downtime and April not being very productive.

The importer moved a little bit forward. I tried to use the brand new "sort by date" importer tool, that is used to test and exercise the logic. The improved importer address a few long standing issues, including not using libgphoto2 for USB Mass Storage, including flash card readers. This was a shortcut I had taken and the result was suboptimal. The new approach is to use libgphoto2 to find the device, and then switch to pure filesystem operations. That was issue 26.

I picked up my camera which I haden't done much since the pandemic started. After a firmware upgrade on the Fujifilm X-T3 (it's a 4 year old model, and the firmware is from this year, unlike most smartphones), I notice a new feature that allow setting a picture as favourite. My first question was "how is it stored?". I put that on my list for later.

After taking some pictures of the newly bloomed spring flowers, I tried the importer. The new files cause a bug with metadata making the importer unable to determine the creation date ; I hit this when trying to sort these new pictures.

This is the March 2023 update for Niepce. This is not an April's fool, and this is not the year I can announce a release on April's fool day. Sorry about that.

Continuing with the renderer / previewer cache.

I had to move ncr to Rust. I didn't port it all, but the widget and the main API are now in a Rust crate npc-craw, the fourth one in the workspace. The goal of this crate is to provide the interface to the rendering pipeline. Some of the work included moving away from Cairo surface and using GdkTextures instead. The main pipeline is still the original C++ code using GEGL, it's easier for me to bind the C++ code than to write wrappers for GEGL.

In the same way, I also ported most of the darkroom module to Rust. This module is the one that will allow the image editing and currently only handle displaying the images.

All of this was necessary to finish the render / previewer integration and making it work asynchronously: the image rendering happen in the background without freezing the UI. There are still some issues but it on overall, it works well.

RawTherapee is one of the two major open source RAW photo processing applications, the other is Darktable.

Can I leverage RawTherapee RAW processing code for use in Niepce? Yes I can.

So let's review of I did it.

This is the February 2023 update.

Not as much direct progress as I wished.

One big change is that now the binary is linked from Rust. And at the same time the autotools buid system is gone. The latter came first, as to not have to ignore it when doing the former.

Several challenges presented themselves with linked the app from Rust.

1. We have to invert the dependency. We'd build the Rust code as a library, generate the headers for the cxx bridge and then build the C++ code, and link it. Instead we generate the headers, build the C++ code as libraries and then the Rust code.
2. We have to figure some tricks: the Rust executable needs to link to some system libraries called directly from the C++ code, and also need to link to asan if needed.

Otherwise, for the two previous topics, namely the importer and the previewer.

This will relate my short learning journey in implementing a procedural macro in Rust: i18n-format.

The problem

Building Rust applications means having to deal with localization, ie allowing the UI to be translated in other languages. This is a big deal for users, and thanks to existing tooling, if you use gtk-rs and if you use the rust-gtk-template to boostrap your project you should be all set. The GNOME community has an history of caring about this and most of the infrastructure is here.

However there is one sore point into this: gettext, the standard package used to handle strings localization at runtime doesn't support Rust. For some reason the patch to fix it is stuck in review.

While it mostly works, the lack of support cause one thing: the getttext! macro that allow the use of localized strings as format is ignored by the xgettext command used to extract and strings and generate the PO template. It is the !, a part of the macro invocation syntax in Rust, that is the blocker. Another problem is that in Rust, for safety reasons, you can only pass a string literal to format!, which is why gettext! exists in the first place.

Until xgettext is fixed, either these strings are missed in the localization process, or you end up reimplementing it with less flexibility. That's what some applications do.

What if I implement a macro that would allow using a function syntax and still use the macro underneath? Oh boy, that mean I need to implement a procedural macro, something I have been dreading because it looks like it has a steep learning curve. It's a topic so complicated that it is glanced over by most books about Rust. Turns out it wasn't that difficult, thanks to a great tooling.

The idea is as follow: convince xgettext to recognize the string. It should look like a plain function, while we need to call a macro.