Niepce March 2023 updates

This is the March 2023 update for Niepce. This is not an April's fool, and this is not the year I can announce a release on April's fool day. Sorry about that.

Continuing with the renderer / previewer cache.

I had to move ncr¹ to Rust. I didn't port it all, but the widget and the main API are now in a Rust crate npc-craw², the fourth one in the workspace. The goal of this crate is to provide the interface to the rendering pipeline. Some of the work included moving away from Cairo surface and using GdkTextures instead. The main pipeline is still the original C++ code using GEGL, it's easier for me to bind the C++ code than to write wrappers for GEGL.

In the same way, I also ported most of the darkroom module to Rust. This module is the one that will allow the image editing and currently only handle displaying the images.

All of this was necessary to finish the render / previewer integration and making it work asynchronously: the image rendering happen in the background without freezing the UI. There are still some issues but it on overall, it works well.

Alternative rendering

The initial rendering code for camera raw files was written a long time ago using GEGL, and with GEGL built in camera raw converter. It works, but the result are not satisfying. It's not GEGL's fault it is just that camera raw processing is a complex matter, requires a lot of work, and GEGL here is just a set of operations to build image processing pipelines. It's also more complex now as some cameras require lens correction. There are long term plans for it including using libopenraw, adding lensfun, etc. but this will come later.

I also have a few plans down the road, including compatibility with existing (open source) raw processing, namely with the two most populars, RawTherapee and Darktable. The easiest way to have compatible rendering is to use their code.

RawTherapee

Here we go, I took RawTherapee, added it as a submodule and built a Rust crate, rtengine, around its camera raw engine. I have written up more details on how I did this.

Its integration is light for now, the longer term is to treat the .pp3 on import as sidecars and use them, eventually as the default processor in that case. This is the compatibility I'm talking about. In the long run editing parameters is in the card, it's part of what I consider necessary for the initial release.

But what about Darktable? One thing at a team, but it's in the card.

UI changes

Now as a prelude to change the the rendering parameters (i.e. the adjustement you'll do to the pictures), I have added a simple UI to select ncr or rtengine and this is saved. This is also useful to manually test the rendering cache. The information is displayed in the new "Process" section of the metadata pane.

Rendering cache

I now have a rendering cache that will cache previews on disk, by storing a PNG. This lead to an interesting about space vs time for a cache storage, and I mean the image format. PNG seems like a good compromise for speed but I get 20-30MB files, while JPEG would be more space efficient, but lossy³. This can be investigated later.

This has been more work than anticipated, but not taking some shortcuts lead to some more infrastructure built to provide the core features.

Importer

I moved on to the importer. That's the part that will be used to add images into the catalog. Currently it support importing in place, and a very rough importing from a camera or a flash card. This is very light in functionality.

One of the key feature I want with the importer is being able to copy images from one place to another, and sort they automatically. This is particularly useful when importing from a camera or memory card where the images are all together.

I wrote a shell script not too long ago to do the same from a download of my phone pictures. The file IMG_0123.JPG with the Exif property DateTimeOriginal that is set to 1 April 2023 is copied to the directory 2023/20230401. This is the way I like it, but the importer will be more flexible. The longer term will possibly see a renaming feature.

To implement this image copying workflow, and to help testing, I wrote a command line tool to run the copy. This involves recursively (as an option) walking through the directories, grouping files together (this is called bundle), extracting the date from the metadata, and copying the files. One of the shortcomings of the Rust std::fs::copy is that it doesn't copy the timestamps, so I had to implement this. Also I learned that the file creation date isn't a thing you can change.

Ever felt that you have two pieces of code you need to put together and they don't fit? That's where I'm at now. The copying code needs to be integrated in the UI, and it doesn't fit at all, nor with the existing importer. The whole thing is taking longer than I wished, but it'll get there as I will redesign the import code.

Other

I also updated the Flatpak manifest so I can build a flatpak with the current code.

And last but not least, I have submitted a few PR for RawTherapee, mostly issues triggered by the adresss sanitizer: Issue #6708 - fix overlapping buffer strcpy, Issue #6721 - Fix memory leak in Crop.

Thank you reading.