Loud ramblings of a Software Artisan

That is a strange coincidence. I we were talking about how Safari did better than Konqueror, given that Safari is based on KHTML, the HTML engine of KDE used by Konqueror.

Oh well, I just read that, that and that about the pain of porting or lack thereof Apple changes to KHTML. Apple does not provide full history, and apparently their change are more Apple centric that KHTML centric. They only do the bare minimum to comply with the LGPL: they release the source code of the framework that contain the LGPL code.

And after that people complain when I say that Apple do not support Open Source projects even when they use it. They just use them as a convenience store, to shop what they need.

I found some information: Foveon Inc., and their only known licensee Sigma disclosed the XF3 RAW file format with documentation and some sample images. They are just free for pickup in PDF.

I haven't read it yet, but from a company that try to innovate in the area of sensors, I find that from Foveon it is a really smart move to disclose the file format. That means they don't fear that it will be used to compete with them.

Kudos to Foveon Inc.

On one side you have Adobe, that succeeded is providing a fully-disclosed portable file format, PDF.

On the other side, you have Microsoft, who do not like to not have a dominant position on a market, that decide to run after PDF with Metro, a new XML based file format that will be licensed royalty free and (that) users will be able to open Metro files without a special client according to their CEO. The problem is that Microsoft is notorious for patenting everything and make sure you only get crumbles from the cake, so, despite that we don't have anything yet but some PR, I wonder what they will come up with.

The Metro technology is likely to go head-to-head with Adobe's PostScript technology. "It is a potential Adobe killer," said Richard Doherty

We already heard that a 15 years ago when Microsoft worked on True Image to have a replacement for PostScript. PostScript is still there, and from True Image, there is only one thing left: TrueType that Apple had developped and sold to Microsoft in exchange of the other technology. The only other postive effect of this battle is that Adobe ended up disclosing the Type 1 PostScript font format. But now, there is nothing to disclose as both PostScript and PDF are completely documented and free to use even if still the property of Adobe.

Update: I still have to find any patent issue associated to PDF.

Today I contacted Canon Customer Support to request the documentation of the CR2 RAW file format of my Canon 20D. Here is my request:

Hi,

I would like to obtain the documentation of the Canon CR2 RAW format in order to write a software to decode and convert the pictures I produce.

Thank you.

Hub

I encourage you, as a customer, to do the same, being Canon, Nikon or whoever else. The idea is that if they get numerous requests there may be a change.

Please, once done, post a comment here.

Thanks.

Update: Here are links to Canon customer support.

• Canada
• USA
• UK
• France

Or look for Customer Support, they all have some way to ask a question.

Murray, I have 2 big issues with the GUADEC registration:

• Nobody said GUADEC Registration would require paying this year. Now that everyone has booked, this really looks like a scam. And there is no register-and-pay-when-you-check-in option, which is not nice at all, see below

• You require PayPal. That forbid people from some countries, like say Iran, to register for the GUADEC. And since you have to give your mailing info to PayPal, even if it does not require an account, you just ask everyone to be spammed

Note: I'm not arguing the fact that you need to pay EUR 30 + VAT, just how things are done.

Nikon answer

Nikon posted some bullshit answer to justify they encrypt third-party pictures when shooting in RAW mode on D2X camera. I see people arguing that this is not the image but only the white balance. I must disagree with that, as this being part of the whole result of the shooting, it is part of the picture, hence, it is the photographer property.

Nikon's answer is simply unacceptable. Basically they consider the providing a binary SDK under NDA is a good subsitute for a fully disclosed file format, which is completely wrong. First of all, they limit your market to the one they decided for the SDK as you can't use the SDK on platform they don't support. I don't know which one as I would have to sign a NDA for that, which leads to the second point: they want to limit who can compete with that as they must approve application first. That is not free market.

As stated above, application for the Nikon SDK is possible for bona fide software companies that send Nikon a written application for the SDK. Once approved, the SDK is provided to the developer at no charge and they are authorized to use it.

What can you do ? If you are a Nikon customer, tell them that you want that documentation. This apply to other manufacturers like Canon.

BTW, if they did a Press Release, it is because they feel threatened by this information spreading, and they should be.

OpenRAW initiative

As a result, someone started the OpenRAW initiative, whose goal is to inform about the problem. This is not a new problem. Just that this time it impact a different kind of people.

Source: dpreview

Dave Coffin interview

And to finish, dpreview has an interview of Dave Coffin, dcraw author. Interesting to read.

Not hot news, but News.com has an article about dcraw support of Nikon encrypted data. In short, dcraw support decrypting newer Nikon NEF (RAW) files.

Despite being undocumented by the camera manufacturers, there are several Free Software tools to manipulate RAW files on Linux.

I was looking on how to provide thumbnails to RAW files in Nautilus.

I found 3 methods:

• Using the method described in that Mini-HOWTO for RAW files
• Use Novell package which does do the what as been described above, but providing a nice RPM of SuSE.
• Use gnome-raw-thumbnailer which I'm trying to make work as a package on Ubuntu.

The first two method just wrap dcraw in a shell script, which is quite heavy.

The latest just take dcraw code to convert to thumbnails. It does the right thing as it extract the thumbnails from the RAW file and let GdkPixbuf convert it to the thumbnail in PNG.

I wonder why Nautilus doesn't do that for JPEG files too as most digital cameras comes with a thumbnail inside the EXIF information; falling back to the actual method if needed.

The gphoto project has been running without manufacturer support for at least 6 years now. No manufacturer did help us to write drivers to communicate with their camer, and only a very few have provided documentation in the past. So far, good citizens have been Kodak, who did publish the protocols for almost all their consumer camera, until they switched to PTP and reorganized themself, and Digita, that provided some sort of documentation, even if largely incomplete that was a good basis to write the driver. Other than that, it has only been either total ignorance or even some sort of hostility. One good example is Canon of such hostility, that still so far has to understand that we do their job, supporting camera they no longer provide support.

The dialog with Canon representatives, when we had one, as been useless and a waste of time. It did looked like:

- Hello I'm part of the gphoto project, we write free drivers for digital camera for Linux and other UNIX systems . We would like to support Canon cameras, so we would like to have the proper documentation for that.

- Use the Canon SDK. It is available for Windows and allow you to communicate with Canon cameras

- Mmm. As I said we run on UNIX, not Windows, so it is likely your SDK for Windows is useless to us.

- Mmm. There is a software called gphoto that works on UNIX. It might work for your. Be aware that Canon does not endorse this product at al.

grin

- Can I have access to that SDK ?

- Which company is it for ?

- Sorry I'm an individual.

- Then I'm sorry, the SDK is not available to individuals.

sigh

Yes, they told gphoto developers that just wanted documentation to look at gphoto, and then refused us access to the SDK. But thanks to the obstination of a few smart hackers (that I was NOT part of), reverse engineering has been successfully performed to implement the Canon drivers for almost every cameras.

A couple of years after, Stephen H Westin, on behalf of Cornell university, got access to that SDK, that helped him a bit to complete the reverse engineering process towards provide software support for capturing images with a Canon camera. He admits that the documentation in the SDK, even if just documenting APIs, helped him. The documentation is still under NDA.

Nikon camera support has been somewhat easier. They started with Sierra Imaging firmware, and used this protocol, already reverse engineered by the photopc developers, then they switched to USB Mass Storage, with the advent of USB, that was already in most USB stacks, because it was part of the USB protocol suite. They never provided any assistance.

Now, with the recent Nikon story, I started collecting a bit of information. I went to Adobe user forums and found this:

From Adobe forums:

Thomas Knoll - 11:30am Mar 1, 05 PST (#3 of 18)

Adobe: Can you please document NEF format?

Nikon: No, but you can use our SDK.

Adobe: But the Nikon SDK does not provide what Camera Raw needs to operate, and using it would limit Camera Raw's speed, UI features, and quality of results (e.g. Camera > Raw's special highlight recovery algorithms).

Nikon: Then redesign Camera Raw to work within the limits of our SDK.

Adobe: No, we don't want to cripple Camera Raw. Please document NEF format.

Nikon: No, redesign Camera Raw to work within the limits of our SDK.

etc.

And I thought that with gphoto and other Free Software project, we were the only one to have problem with camera vendors to obtain the documentation and specifications for communication protocols with cameras. But Thomas Knoll, who is one of the original authors of Adobe Photoshop, seems, on behalf of Adobe, to have trouble to obtain some sort of support from Nikon to decode RAW files. Basically Nikon tell Adobe to abandon the flagship features that make Adobe software better than Nikon's (that you also have to pay extra for, shall I remind you), making it useless. Note that Free Software with dcraw did that long before Adobe release their camera RAW plugin, and of course without manufacturer support.

Can someone explain the logic behind these people to hide communication protocol, which is not the major reason to choose a product, and to not disclose file format ? I don't see any. Fortunately, in that market, there is device interroperability, like camera to printers, camera to portable drives, etc. That led to the use of the standard PTP protocol, and before that, the use of USB Mass Storage, thanks Microsoft for making it difficult to write a USB driver, and support it: manufacturers mostly switched for that reason, at least this is what I do believe, because otherwise they had no other interest given theur current attitude.

Even better. Some camera have no support for recent versions of the only OS the manufacturer support, so we end up with some request from users to port gphoto to Windows. We don't have interest in doing it ourself. One more advantage: you are not fooled by the vendor who drops support to sell upgrades.

In my previous post I ranted about the fact that Nikon was making life difficult to decode RAW file from their high-end digital cameras... pointing to article telling that Adobe was mad for that. There have been a couple of insightful comments, that remind me that little ironic story.

Adobe is complaining that Nikon could use DMCA to sue them if they attempted to decrypt these data. The irony is that Adobe using that same law to sue Dmitry Sklyarov for reverse engineering the eBook file format encryption. There are affraid of the weapon they use against other people.

To not be mistaken, I'm not defending Adobe's position at all, I'm just defending photographers, aka customers, rights against an attempt from mafucturer to control the ownership of the pictures photographers take. The only product I have from Adobe is Photoshop Element 2.0 that I was illegally forced to purchase with my camera, and that I will use to see what should be done to improve free software solutions for digital photography, because lot of people claim that it is not good... and I still haven't gone past installing it.

So here is my word: Camera manufacturers, you don't own the pictures people take with your products. So you MUST provide a fully disclosed documentation of all the file formats and protocols used by your products for photographers to be able to plainly use them and exercise their rights on the photographic material they produce.

The unfair side is that Adobe did use the free software dcraw to improve their Camera RAW plugin, but dcraw author cannot use Adobe's software source code to improve his. That is one of the reason I would have released that software as GPL.

...and it happens.

Thanks to Tuomas for the link.

It appears that Nikon has decided to lock down the NEF RAW file format used in their high end cameras. This is very bad. It just goes way beyond the simple dollar issues, as it seems to require people to buy the Nikon software instead of just using third parties, may it be the free dcraw or the pricey Adobe Photoshop.

It is a matter of who own the copyright on images and who own the key to view these images. With that scheme, the photographer depends on Nikon to view the images. It is like having a dependency on some magnifier manufacturer to view your printed pictures or your developed film. This has never happened. There have been some exclusive film processing like Kodachrome, but once processed, the film could be used freely like anything else.

Even worse. It just make these memories fading. Photographic archives are priceless for our history memory. And film has been somewhat appropriate to keep them, to some extent. With the era of digital picture, lot of professionnals have concerns about the perenity of the storage medias. But with Nikon initiative, that reduce it even more. In 100 years, there is absolutely no warranty that one will be able to decode these. Even in 30 years, and maybe not in 10 years.

I would have prefered the adoption of a free and documented common RAW format instead.

Photoshop news has an article.

Adobe and Macromedia are merging. That means there is an even bigger software behemot for graphic design.

But what will happen on the free software side ?

Tim Bray has a post about what could happen to Flash. I just hope he is right. There is really something I'd like to disappear as is is Flash, just because Flash by itself, breaks all the concept put in WWW by breaking navigation, hyper-text and open information into it own world. And releasing the player as Open Source code wouldn't solve that, even if it could help developing alternatives. Dave Shea point of view on the buy out may balance this argument and question about the opportunity for SVG.

And Alan wonder what will happen to Freehand, I'd say it might become orphan like it happened during the Adobe-Aldus merger. BTW, and encourage people to try Inkscape. If they have to change tool because Freehand gets orphaned, that might be a good opportunity.

Beside that, all of this is pure conjecture. Wait and see. But looks like yet another monopoly.

The Register has an article for business oriented readers.

I got a mail from a recruiter this morning. It is for a contracting job in France. Note that I'm not interested at all, it was unsollicited, but I just wanted to underline the level of tech skills they have, or more the lack thereof. This is a basic minimum they just don't have.

In their e-mail they say:

Compétences Techniques de base

Au moins un UNIX parmi : Solaris, Tru 64, AIX, NT4/WIN2K/WIN2003

That means in English:

Basic technical skills required

At least one UNIX in the following : Solaris, Tru 64, AIX, NT4/WIN2K/WIN2003

If on this they are even unable to be accurate in their job postings, then I wonder how they can select applicants (the error is left as an exercise to the reader). The sender of this mail is a "Director" for one of these contracting agency, that try to fullfills customer IT staff requirement. So knowing that should be part of his job.

I simply replied to him that he failed to the basic technical knowledge test.

Canada taxes are lower, French forms are easier.

This is a free adaptation of a similar quote relating to US taxes and swedish forms... that I read a few years ago in some .signature.

Uraeus: I'd really love to see GNOME Office to take over, because I find that OOo is just too much bloat, and it orientation towards Java doesn't make feel more confident. For the presentation tool, I don't know what is going on, but there have been various projects to attempt to write one, and they all failed.... I don't know why, perhaps it is only a curse. Gnumeric is so nice (let alone AbiWord, I'm biased)

The other question I'm asking is about the component APIs. There are lot of rumors going about Bonobo and the fact that it does not evolve, this is not meant to help. If anybody has an answer, I'd love to hear it.

As for UNO, if someone can provide a version that does work independently of OpenOffice, then we might have a candidate. But Gnumeric use Bonobo currently as it is the logical choice.

So Ubuntu Linux has set up some bounties for implementing features. The interesting thing is that they offer a bounty to provide Python scripting in several applications, including in AbiWord.

I have implemented GdkDisplay on MacOS X. It is basically a wrapper around NSScreen class and NSApplication for a couple of things. It is probably not complete, like handling the clipboard.

Next will be GdkScreen, a wrapper the NSScren instances.

I also started to write some unit tests, so that with make check I can verify I'm not crashing, and that it works...

In my test shots of the EF 100mm Macro f/2.8 lens, I ended up taking pictures inside the metro in Montreal.

At the McGill metro station, is there is this ad on the wall, which one may consider as some sort of Art Urbain. I wanted to take a picture of that, when I finally go the framing correct, I had someone in the picture. I waited a few minutes that he goes away to take it again.

Finally, selecting the pictures, I thought that somebody within the frame, like this, made the picture much better. I got lucky.

This is really a picture of the day. It is not a setup, I was walking on Sainte-Catherine (Montreal) and saw that next to a store window.

One of the challenge was that I only had a 100m f/2.8 Macro mounted on the 20D, so it was quite hard to get the proper shooting angle...

I started coding on Gtk MacOS X, and I investigated what they did in XDarwin the rootless XFree X server for MacOS X. I appears that they use Cocoa. So I decided that it my be a solution as I only need event and window management. The graphic layer will be Quartz directly.

Of course see screenshot on the side.

:-) Actually it is one of the "multimedia screen" I saw from outside on the street... crashed.

I have released libgphoto2 and gphoto2 2.1.6-rc1 (Release Candidate). This is the last in the 2.1.x series as we'll start developing 2.2. Thanks to all the contributors, developer and testers, for the work they put it in.

I went Sunday to walk along Canal Lachine in Montreal then back to downtown, and I did not have my usual camera. Here is the gallery, a little bit later than expect because I had to rotate some picture manually... unlike with my Canon 20D.

My favorite shots with the Minolta Dimage Z3:

I went shutterbugging today, walking along Canal Lachine in Montreal. I exchanged my EOS 20D for a Minolta Dimage Z3 and Nikon F70 (35mm film SLR), for the day. I must say I'm disappointed.

With the Nikon I found two problems. One Nikon, one Sigma:

• The shutter trigger is too touchy. On the Canon, you half press for exposure and AF measurements. On the Nikon you just touch it. So I shot accidentally on several occasions
• The zoom ring rotated in the wrong direction. But it was a Sigma lens.

Otherwise, it is convenient and usable. I shot 2 rolls of Black & White.

With the Minolta Dimage Z3, much more problems:

• The view finder is electronic. You must be blind to appreciate. I just find them inconvenient to not say painfull.
• The camera is slow: the preview freeze while autofocusing.
• Electric zooms are slow.
• Auto-white balance seems to be clumsy as sometime I was not getting the right tone, like pale yellow for a yellow road sign. The 20D software is much more accurate.
• 400 ISO has a lot of noise. More than at 1600 ISO on the Canon. This size of the sensor is probably part of the reason.
• The JPEG files don't have the picture orientation encoded. I don't know if it is the camera or that they are encoded in a proprietary way, but this is what is delaying publishing my gallery.

I shot a grand total of 257 pictures. I missed some shots because of the slow zoom and the slow auto-focus.

I don't like Ubuntu Nautilus Spatial mode, then one that makes close the window when you open a subfolder. It is anti-ergonomic, because there are a lot of case when you still want the upper folder (just to open another one). I hate even more the fact that I got that option silently when upgrading the package but not only the first time I run Nautilus. I think that if the option was not there, then it shoudln't be enabled, and if the option is not there, unless you are initializing the whold Nautilus gconf tree. That is the route I'd have taken. Thanks to Gconf, I disabled it, because they forgot to put a UI together for that... that reminds me another GNOME feature where the UI has been forgotten and the expense of user's pain.

For those who don't know, the Nautilus Spatial Mode is that opening a folder opens a new window. This is how the Finder on MacOS was designed back in 1984. On MacOS, pressing the option key (also labelled Alt) close the previous window while opening a new one. In Nautilus, this is with Shift, but the idea is the same.

Ubuntu decided to put that option to close the previous window automatically by default, reverting the effect of the Shift key. There is even a bug 8516 about that, because it is not popular.

I think that what should be written in Nautilus is a "browser" view. By "browser" I mean what Nextstep now MacOS X call browser. Hierchical browsing with adjacent columns... This coupled with a spatial mode and you have a powerful file management. Perhaps should I file a bug if it doesn't exist yet... (or do it, but that is another story)

Today, part of my NDF, I started working on Gtk+ for MacOS X. So I had build a sane environment to get started.

Random caveats:

• Fink only has glib 2.4. So have to build glib and atk as well.
• On MacOS X, it is not LD_LIBRARY_PATH but DYLD_LIBRARY_PATH. Go figure why they do it differently on MacOS X, but this is a noticeable difference.
• Freetype2 package from Fink is broken as it put freetype-config somewhere in /sw/lib/freetype2/bin/freetype-config. And no I can't use freetype-config to find freetype-config path unlike someone suggested, as a joke, on #gtk. Playing with PATH helps. Bug 1179574
• CVS needs insane amount of disk space in /tmp. 70MB was not enough to import. I remove tetex as this machine is now head-less...

Here are the steps:

1. get all the dependencies to be build. glib, atk and pango with Freetype2 backend are done.
2. hack the configure to detect MacOS X. Almost done.
3. port gdk by writing and Carbon and CoreGraphics backend.
4. Have all the samples running.
5. Make it a Framework for nice MacOS X integration.
6. Write an ATSUI backend for Pango.
7. Move to gtk 2.8. This may happen sooner depending on the time line.

Sorry, there is no screenshot.

BitKeeper is not longer free of charge, at least not like it has been so far in the past. Looks live Larry McVoy, BitKeeper head did not like all the attempt to reverse engineer the software to provide free (as in speech) replacements. In their press release, they announce that they refocus on the commercial version of BitKeeper, but the the open source client is still available.

I personally haven't used that free client, but I suspect it does not allow to do everything the free version was allowing, including sharing your tree. I have used BitKeeper at one of my previous job where we did use the free version to get the kernel source-tree and work on it, but didn't really use it as a repository since the boss didn't want us to commit...

I found BK somewhat hard to use (coming from CVS), and with some issues, like not being able to pull a tree with changes not committed, issues probably related to its design. On the other side I really love the concept of having a revisioned tree locally. And BK revision trees like Subversion, unlike CVS.

I don't know what the kernel developers will come up with as a replacement, but Larry just gave the point to people like Andrea Arcangeli who were always arguing about the use of a non-free tool for Linux development, and tried to propose solutions.

Source: jdub who said to Larry that he was not invited to the party.

Update (2005/04/08): the client is open source, not free software, I changed the text above. James Henstridge has an informative post on the subject.

Looks like my rants gets linked indirectly from major sites.

• From Slashdot in the interview of Mark Shuttleworth, he links to Thom May who links to my rant about Fedora dumping AbiWord to extras when talkin about picking your choice when you need to include things in a distribution.

• From Newsforge when talking about the upcoming OpenOffice.org 2.0. They link to Colin Charles argument about the large Java dependency, and he links to my rant about OOo freeness if it depends even more on Java.

Looks like I have to be more careful in what I say here.

For once this is a really a picture of the day.

We went today to Montreal Botanical Garden were there is an exhibition with live butterflies. I did test this EF 100mm Macro f/2.8 that a friend lent me, withmy 20D. It is awesome.

Pictures:

The whole gallery is here.

Some people wonder if I am a switcher. Hell no. I'm in fact and unswitcher, really. I have been use Linux as a workstation at work since march 1997, with a notable exception, and have switched my PowerComputing Mac clone to Linux, permanently, around 2001. My PowerBook G3 that I bought in September 2000, to port AbiWord has moved to Linux in February 2004, because I had access to an iMac to continue AbiWord development.

But what is that MacMini you have ? It is a workstation to maintain AbiWord on MacOS X. I only use it to compile test and use, and beside Project Builder and Terminal, the only other app running all the time is Camino, because it is just needed for Bugzilla. I'm not using Safari, because the bug I reported in 2003 to Apple hasn't been fixed... so it still doesn't work with bugzilla. Everything else including photo work, is done on the Linux machine. And when I need mail, I just use Thunderbird. Why ? Because I can't stand a mailer that can't thread. Apple Mail.app can't (they just do some "grouping") on 10.3.

The other thing that I miss is the Gtk based open source application I'm using, like Dia, Gimp, etc, for which I haven't found a free replacement. And my slide scanner is not supported om MacOS X either by the manufacturer... (it cost me 600 EUR at that time).

So yeah, even though I use MacOS X and do some development on MacOS X it is only for the sole purpose of reaching a wider audience with open source software. My daily personnal use of computer is still on Linux: browsing, e-mai, chat, photo, etc. And I'm not gonna switch back: my freedom is so important.

There is water on Mars !

Source: NASA.

I have been coding a bit on AbiWord lately.

This include fixing a couple of bugs (first time patch in the Piece Table, and I hope not the last), and the Mac build, as now I have a decent build system. 1GB is not too much for the MacMini

I think my priorities will be:

• fix the Cocoa front end
• finish the dialog framework for HEAD
• implement Unit testing
• fix various stuff on my TODO list
• write documentation

As a tradition, today is April's fool.... Here are the Open Source fools:

• So Planet Gnome and Planet KDE have been inverted. Hehe, two competing project work together for the same joke... Who said that competition was not sane...
• AbiWord spatial

More later.

My latest blog about Gtk MacOS X leveraged some questions. I'll try to answer:

Q: Why don't you help GTK-Cocoa guy ?

A: I never said I wouldn't. I'll just investigate that code and see what is good and what can be done. At first sight, two problems. It is a reimplementation of the widget in Cocoa, and it is Gtk+ 1.2.

Q: Why don't you start with Gtk+ 2.8 that has a Cairo backend which has a Quartz back end ?

A: So what do I do with the event management ? AFAIK Cairo does only provide graphic layer. I'd prefer go Quartz straight, with ATSUI for the text.

Corrections:

I took a shortcut saying that Cocoa use Carbon. Sure it does, for menus and to have the same print and file dialogs. But they are both based on CoreFoundation (in fact Cocoa more like interact transparently with CoreFondation), CoreGraphics and ATSUI.

Now, stop talking, start coding.