My digital open darkroom

A photography workflow powered by FOSS

Darktable Digikam Photobook Photography Workflow

After yet another disappointment with Aperture, Photos, Capture One and other proprietary photo development and editing software, I looked for opensource alternatives.

I was looking for something that was open, non-proprietary, and that would allow me to use my data and images without necessarily having to share them on third-party servers, be dependent on a subscriptions-based model, and most importantly give me the ability to be able to use different software without having to lose all the work I have done so far just because my work is locked in an inaccessible, proprietary library.

In the forums and on Reddit I was reading about users using opensource software, and I tried to better understand how to optimize my personal photo workflow to make it completely FOSS, free and open source software.

And so I gradually got into applications like darktable and digiKam. At first I used them on my MacBook Air M1 but with only 8Gb of ram and limited disk space the general disappointment kept growing. I therefore needed a more powerful computer and more space to manage my image archive.

The initial intention was to pair my MacBook Air with a Mac Studio that was a newly released model. However, after much thought, I gave up on the idea of spending money on a machine that was very expensive, not upgradeable, and poorly supported by the opensource community since Silicon processors had just been released by Apple.

And that is how I decided to reuse a CAD workstation I had assembled for my wife during the COVID19 period to turn it into my photoediting workstation.

And it is on this workstation that I built and defined my current workflow, which is based on a local context and without using proprietary platforms or formats.

At the time of writing, the workstation has the following specifications:

• CPU AMD 3900X
• GPU AMD RX 6700 XT with 12gb of memory
• 32gb total memory
• 4Tb SSD Local for operating system and data
• 8Tb HDD local for backup
• an EIZO CG247 monitor

I chose to use Linux OpenSUSE Tumbleweed as my main operating system because it allows me to have recent updates and have the support of a very helpful user community.

Rounding out my configuration is a Synology NAS that I use as a network drive, mainly for backups and to self-host services useful for my work.

My goal was to achieve a totally FOSS photo workflow from the time the RAW photos arrive to be archived, to the printing of a photo book.

The following is my personal experience that I want to share to inspire those who will read it and get suggestions for optimizing my FOSS photography workflow.

The reason I decided to publish and share my photographic workflow is that when I started using opensource software there were not many online resources. The documentation of the various software I use, while comprehensive and thorough, is often written from a technical perspective and lacks a complete overview from start to finish of the process.

Finally, I want to share this story to inspire photographers —professional and non-professional— to find alternatives to commercial software and entice them not to share their personal data and work with large companies that do not always respect user privacy and data.

The photographer’s user journey

Photographing is a complex business, and finding a workflow that works for everything and all types of photographers is far too ambitious a goal since we are talking about something very articulate and personal.

After some research, I found that in general, there are commonalities between different approaches to photography, and I tried to sketch out what the ideal user journey of those involved in photography might look like.

I summarize the main points below:

• Brief: before photographing, photographers need to plan and prepare their activity, for example, a trip. This may involve researching the destination, deciding on a route and itinerary, preparing equipment and supplies, and preparing any necessary permits or authorizations;
• Capturing: when taking pictures, photographers use their cameras and other equipment to capture photos and videos of what they are photographing. This can involve taking many different photos and videos, using different cameras and settings, and capturing a wide range of subjects and scenes;
• Storing: after taking images, photographers transfer photos and videos from their cameras and other devices to a computer or other storage device. They must then organize and classify the assets, using folders, tags, and metadata to create a logical and consistent structure for their library of digital assets;
• Backing-up: to ensure that photos and videos are preserved and protected, photographers need to store and back up their library of digital assets. This may involve storing assets on an external hard drive or a cloud-based storage service, as well as regularly backing up files to prevent loss or damage;
• Curation: is the process of selecting, organizing, and managing images to create a consistent and easily searchable system. This is when photographers add metadata-keywords, descriptions, technical data, copyright-creating a standard for categories and file names, quality management (eliminating duplicates or irrelevant images), and adding historical or narrative context. This work allows the archive to be enhanced, and facilitates quick access. I see this as a key point that enables photographers to increase their creative and professional potential;
• Editing: photographers may want to edit and enhance their photos and videos, using photo editing software or other tools to adjust the color, contrast, and composition of images and to apply filters or other effects to improve their appearance and add their own creative vision to the image.
• Sharing: once photos and videos have been edited and enhanced, photographers may want to share and publish them with others. This may involve creating a portfolio, uploading the assets to a website or social media platform, creating a digital gallery or slideshow, or printing and framing the photos for display in a physical space.

These 7 steps are not meant to be exhaustive, and are very personal, and I will describe them in detail in the following paragraphs.

Photography project planning

Making photography does not mean simply taking pictures. It means telling a story, communicating an idea, making one’s view of the world and reality visible to the viewer through the images we produce. Researching a topic is the first step toward communicating an idea.

Studying, reading, and viewing other work is a start, and wanting to go deeper there are many ways to research and be inspired.

Books and photobooks by other authors, are one of many ways to find inspiration, information and material for your own photographic projects. They can be found free of charge in the most common city libraries or in online libraries that make their material available and which in some cases is also in the public domain.

These libraries in some cases also make available entire archives of digitized images in the public domain, such as the New York Public Library which makes available its archive of images in the public domain. There one can find historical images from the FSA photo campaign —Farm Security Administration— by Walker Evans, Dorothea Lange to Lewis Hine’s Ellis Island immigrant photographs.

Another place where you can find inspiration online is Public domain review a site dedicated to exploring curious and compelling works from the history of art, literature, and ideas-focusing on works that have now fallen into the public domain, the vast trove of out-of-copyright material that everyone is free to enjoy, share, and build upon without restriction.

Finally, I point out a project I personally collaborate on: Landscape stories, is a reference for contemporary photography, useful to take cues and see what other authors are doing.

In this series of notes, I will use photographs from a project whose inspiration came from looking at some minimalist and highly contrasted drawings by Ellsworth Kelly, an American artist and exponent of Hard edge painting and Minimalism.

The images were found by chance just as I was browsing through the archives of the National Gallery of Art.

Ideas come about by chance, and often inspiration comes after seeing other work, whether on the Web, in a museum, in a magazine, or in a book. There is a need to collect notes, whether they are written or visual notes. The images we like can also be saved, captured with the phone or a screenshot. Once collected, the images can be viewed and organized.

In the real world, I like to put all the images on a table or on the floor so that I can look at them from above, touch them, physically move them around, and organize them into a sequence of ideas.

In the digital world the equivalent of my big table is an infinite canvas, which has the great advantage of being able to organize all the images exactly as on my big table.

To organize ideas, cues, and images I use two tools that I consider fundamental in my work:

• Obsidian for written notes but also for sketching mindmaps and visual canvases; is the same tool that I’m using to write these notes and it’s powerful to connect things and ideas;
• Penpot a design tool that I use as an infinite canvas, that allows me to work and organize images in the same way as my table.

I also use my local and personal AI: Open Web UI. In conjunction with local models it became my personal assistant to help me organize ideas as well as keywords, captions, and image metadata. I find it a very useful tool to improve my creativity.

Framing the moment

This is the most fun part.
Go out, observe the world.
Be curious.

Click.

Building memories

After a long day taking pictures, photographers return home and there are two important things to do right away: import the photos into the archive and back them up immediately.

My archive is a folder called Memories and is organized on a file system more or less like this:

Memories contains the following subfolders:

• DigiKam, it’s the digiKam database; I initially configured digiKam to save the database in the same folder as the images, but this is not necessary and can be kept outside;
• export, is the folder I use to export images after they have been processed in darktable or other editing software;
• photobooks, contains the photobooks in PDF format that I created and printed;
• scan, is the archive of the pictures that were captured with my Epson Perfection V850 Pro scanner;
• sessions, the RAW archive organized by year, month and day that contains the equivalent of photo rolls.

In Memories I also added a README.md file that contains information about the archive organization. The reason is simple: the day I’m gone, if any of my family members want to better understand my work, they will have the basic information to navigate and manage it.

To manage my Memories archive I use the opensource software digiKam which offers a complete set of tools to import, manage, edit and share photos and raw files.

I import photos from the camera to the archive, physically copying the photos from the memory card into a sessions folder organized like this:

• YYYY, the year the photos were taken.
  • YYYY-MM, the year and month in which the photos were taken
  • YYYY-MM-DD - Session name, to identify the session with a reference name

In digiKam I rename individual images using this expression:

[date:yyyy-MM-ddThhmmss][meta:Exif.Photo.SubSecTime]-[meta:Exif.Image.Model]{replace:" ",""}.[ext]{lower}

This expression renames the original camera files in a way that is unique, readable and findable even by file system, like:

2020-05-17T121611-X100F.raf

The file uniquely identifies an image taken on 2020-05-17 at 12:16:11 with a Fuji X100F. No other information needs to be added because it is contained in the image metadata readable by digiKam or any image management software.

I chose to use the date in ISO format as the unique identifier for my images along with the camera model because it represents the minimum information to uniquely identify the file. The identifier is critical to make sure we are working with a unique image and to identify any derivative copies, such as the same image exported in a different format.

To be sure of compatible file names on major file systems, for folders and file names I avoid using special characters, and use only the hyphen-minus character - as a separator to improve readability.

In general, I think the important thing about file and folder names is that you can understand what they mean and that you can analyze them programmatically for your own workflow.

In my case, the folder name is chosen from a set of conventional categories that I use to identify the various rolls.

Using conventions, you can use categories as keywords to filter in digiKam all folders/photo rolls that contain, for example, the name “Flowers.”

After importing images into the archive I immediately add basic metadata, the kind that will allow me to find what I need in the future. As optional as it may seem, taking care of the archive is essential for working with my images later.

I also believe that the only way to preserve my work is to rely on a system that has been working now since computers were born: folders and files.

This is the main reason on why my Memories archive is mainly a set of folders and files organized on file systems that allow me to work with different applications and to be able to manage them in a standard way.

I believe that this approach will work and still be valid 100 years from now.

Protecting memories

My photo archive is invaluable to me.

To lose it permanently would be a huge blow. Years of work and memories deserve some attention, which is why backing up is a key part of my workflow.

For my backups, I mainly use restic and rysnc since my archive is a system of folders and files the backup, it is sufficient to make a full backup of the Memories folder.

On the importance of making local and dislocated copies of your work, a million words have been spent and I don’t want to dwell much on this point, and if you are reading these notes and you don’t yet have a backup of your work, do it now.

In my specific case, I back up the Memories folder 5 times:

• locally on my workstation with rsync sun a secondary hard drive; this gives me the ability to access the data directly from file system;
• locally with restic on my workstation’s backup disk, which gives me the ability to make deduplicated and incremental copies (à la Time Machine from macOS for short);
• locally on my NAS with rsync; this allows me to have a copy available on the NAS that I can access even when I’m not at home; thanks to Tailscale I can access my NAS even when away from home;
• off-site with restic using several external hard drives that I physically move when I travel and visit my family in Italy;
• finally, I make regular backups of the important data on the NAS–including the Memories folder–on an additional external hard drive; to automate this process I use backrest a web interface that uses restic at its base.

Considering the original copy, in total there are 6 copies of which a couple are deployed and updated regularly in case of extreme disaster.

As onerous and complicated as this mechanism sounds it is actually very flexible: both restic and rsync are commands from CLI that can be automated in a script (contact me by email if you are curious to see my backup scripts).

Finally, let me add some important recommendations that are often ignored:

• always check your backups particularly the way data is restored; give it a try to retrieve data from your backups; if you are unable to do so and if they are not restored properly, yours is not a backup;
• a single copy on NAS or other device that uses RAID or similar technologies are not enough to ensure a secure copy of your data; these are devices that can fail at any time and that is why you need to make multiple copies;
• ask not if your disks will fail, but when they will fail.

In addition to digital copies of my images, I regularly print photos on photo paper or photo books that are in fact a physical copy that always works even without computers and electricity.

If you then give these copies as gifts to people you love, then you have a displaced backup and happy people.

From chaos to meaning

By the term “curation” of a photo archive, I really mean taking care of the archive and the information in it and especially creating meaningful relationships among the images.

Meticulous selection and management of photographs and metadata can transform a disorganized archive into a strategic asset, enabling photographers to quickly reuse and access their material.

I consider archive curation to be an activity that cuts across the photographer’s entire user journey because curation of information begins even before a photo is taken.

The simplest and most trivial example that comes to mind is setting the time on the camera. A few times I happened to have forgotten to set it correctly, and the consequence was to have incorrect information that affected everything else I did, resulting in wasted time trying to fix it.

Curating a photo archive is not limited to selecting the most representative or significant images, but involves an ongoing process that carefully organizes, labels, and describes each image to ensure easy access and retrieval.

The main activities include:

1. selection and organization: after the images have been taken and archived, photographers must examine each photo and decide which to discard and which to keep; this not only concerns the aesthetic quality of the image, but also its relevance, context, and potential usefulness for future projects; the selected images must be organized in a logical system that can include folders, collections, or thematic albums;
2. addition of metadata: a key element of curation is the addition of metadata; this can include information such as the date and location of the shot, description of the subject, equipment used, keywords (tags), copyright and copyright information, any notes on the context and emotion captured, etc.; good metadata management increases the searchability of images, even years later;
3. keep consistency: it is essential that photographers establish a standard system for metadata and tags; for example, using uniform conventions for file names or image categories (such as landscapes, portraits, events, nature, etc.) facilitates search and management; consistency helps avoid disorganization and overlapping information;
4. quality management: during the curation phase, photographers may decide to delete duplicate images, enhance those that are not perfect, or even make small changes to their appearance or metadata to ensure that the archive remains consistent and useful in the long run;
5. document: adding historical or narrative context can enrich the archive; details about significant events or the creative process behind each image can turn the archive into an important cultural or artistic resource; for example, it might be useful to note the evolution of one’s style or the symbolic significance of a photographic project;
6. findability: a good archive must be easily searchable; the use of image management software that supports advanced searches using keywords, dates, locations, and other metadata is essential; in this way, even among thousands of images, it will be possible to quickly find the one you need; curation, therefore, also involves creating a system that helps to quickly locate the right images at the right time.

Curation represents an investment that allows photographers to maximize the potential of their archive over time. A well-curated archive not only saves time in managing photographic resources, but also becomes an asset that increases in value over time.

Selecting photos for a project is the most important part of the process, and I am very strict when it comes to making selections of my photographs. Often this activity is done at different times, over several days, months and sometimes years.

I do this selection in darktable for the sake of simplicity, but as I will explain in the next section, the rating will be readable by other software that supports reading XMP sidecar files.

For rating my photos, I use the following rating system:

• R - Picture has been rejected
• 0 - To be rejected
• 1 - Imported
• 2 - First Selection
• 3 - Selected for editing
• 4 - Selected for print or photobook
• 5 - Portfolio

By default, all photos imported into darktable have rating 1 which I consider the default rating. After taking and importing the photos, I do an initial round of culling. If I do not like a photograph, or it is technically wrong, and more generally does not give me any emotion, the image does not pass selection, I assign rating 0 and it disappears from my view. These will be 99% images that are later destined to be rejected -rating R- and in the future permanently deleted. I am very strict in this operation, and I do not keep these kinds of images, and archiving them would be unnecessary, and especially costly in terms of noise and archive cost.

Archive – the bin for the indecisive

— Sönke Ahrens, How to take smart notes

During the second round of curation, I make an initial selection, assigning rating 2 to photographs that have creative potential and are part of the initial vision. These are the images that give me emotion and that I consider meaningful.

I always let a few days pass between the different selection stages, because often seeing the same photographs repetitively does not help with selection, it confuses and generates frustration. In the later stages I assign the rating 3 to the images I want to edit, which will most likely be shared and end up in the final project. At this stage there are still similar images but there are still small details that do not help me make a final choice.

Only at the point when I begin to reason with the final vision of the project in mind —for example a photobook— will I assign rating 4 to identify the final images that will be printed and form the final project.

The rating 5 is assigned only and exclusively to the cross-project images that I consider the best in my archive those that will be part of my photographic portfolio.

Crafting the final vision

After taking, storing and backing up photos, photographers have a need to view the photos and be able to develop them.

In the opensource arena there are several software to develop RAW photos but my favorite remains for the moment darktable an application to manage the photographic workflow and development of RAW images. It is both an image library and a virtual darkroom. It manages digital negatives in a database, allows them to be viewed through a zoomable light table, and allows RAW images to be developed and enhanced.

It is a viable open source alternative for RAW photo development for all types of photographers, professional and amateur.

I will not go into much detail on the actual development part of RAW images. This topic is much better covered by other photographers and on YouTube you can find many videos of editing sessions to take inspiration from. My favorite is the channel of Boris Hajdukovic which is always very up-to-date.

On the other hand, I wanted to share a key aspect of darktable that is often overlooked and which allows the archive to be shared interoperably with digiKam.

darktable is a non-destructive image editor and reads RAW files without ever editing them. Everything done in darktable is saved in separate text files in XMP format, the so-called sidecar files.

Information such as rating and captions are saved in these files, but also the complete editing history.

This is an example of how images and sidecar files appear within a folder:

...
-rw-r--r--. 1 fabio fabio  48M May 17  2020 2020-05-17T150537-X100F.raf
-rw-r--r--. 1 fabio fabio  13K Oct  1  2024 2020-05-17T150537-X100F.raf.xmp
-rw-r--r--. 1 fabio fabio  48M May 17  2020 2020-05-17T150547-X100F.raf
-rw-r--r--. 1 fabio fabio  13K Oct  1  2024 2020-05-17T150547-X100F.raf.xmp
-rw-r--r--. 1 fabio fabio  48M May 17  2020 2020-05-17T150602-X100F.raf
-rw-r--r--. 1 fabio fabio  13K Oct  1  2024 2020-05-17T150602-X100F.raf.xmp
-rw-r--r--. 1 fabio fabio  48M May 17  2020 2020-05-17T150603-X100F.raf
-rw-r--r--. 1 fabio fabio  13K Oct  1  2024 2020-05-17T150603-X100F.raf.xmp
...

This is an important point for several reasons:

• the original files in RAW format are saved in a folder on the filesystem and are flanked by the corresponding XMPs that contain all the information and modification history;
• being text files, it is easy to make a backup;
• darktable —but also digiKam— uses a database to quickly show the images in the archive; if the database gets corrupted, it can be rebuilt from the XMP sidecar files.

I stress the importance of this point because in the past I have had to use proprietary software that used a “library” whose proprietary format did not allow me to easily handle this type of operation. In addition, the proprietary format made interoperability between different software difficult.

In my workflow, I can switch from an editing session in darktable to a curation session in digikam by simply re-reading the information saved in the XMP files of the images.

Finally, a personal note on darktable.

I often read that the interface is not intuitive and that the learning curve is steep. As a designer and former user of other operating systems and other photographic software, of course I too had difficulties in the beginning. But I think this applies somewhat to all kinds of complex software from darkable to Blender, from Figma to Photoshop, from Blender to Maya. Fortunately there are many enthusiasts and there are many tutorials to learn, and once you get the basic knowledge, developing photographs in RAW format is done very quickly.

The main darktable community can be found at pixl.us and is very active and willing to help those who have just recently approached this software.

One thing that helped me a lot to understand better how to develop my images was to participate in Play Raw: users post their RAW images and other users develop them by posting only the XMP file. This way anyone can give their interpretation of the same image, but at the same time you can see how others used darktable to get a specific development.

Leaving a trace

At the end of this entire journey, from planning a photography project to editing individual images, photographers obtain images that can be used for other purposes.

In this section I will not cover everything that is possible to do with images, but will simply describe my personal experience in what I think are the most common needs namely online sharing and photo book design.

In the analog world one of the things I’ve always done to choose and organize a series of images is to put all the prints on a big table or on the floor, move them around, match them up, and put them together to build the narrative of the idea, of the project.

In the digital world this approach is enabled by software that uses the concept of an infinite canvas. As a designer I use Penpot not only to collect images to create mood boards but also to organize the images and sequences of my projects.

I used this technique to sketch the sequence of photographs that I included in my personal photobook Memento Floris.

To create photo books the open source software I use has been around for 40 years, and it works so well that I believe there will always be ways in the future I can use it: LaTeX.

Again, using LaTeX is not at all straightforward, and I can already hear the criticism from those who compare it to InDesign or better known publishing editing software that allows you to create a book with graphic mode.

In the case of LaTeX, however, the great advantage besides being opensource and free, is that since it is a code, it can also be compiled in the future without having to convert it as happens with proprietary formats.

I have created a basic template to create Carnet des photos which starting from a folder containing a series of images, automatically paginates them and which I use to print my photobooks in just under a few minutes, just enough time to compile the book with LaTeX.

An excellent visual editor for manipulating files in LaTeX is Kile which offers preview capabilities of LaTeX-generated pdf files for further fine tuning of the final result.

The end of the journey

The main reason in choosing this kind of path using only FOSS software is not cheap to use free software.

On the contrary, my invitation is to support the projects you use by making donations. Don’t wait to receive notifications, and put your hand in your wallet and donate and support your projects financially. And if you cannot offer money, offer your contribution in the form of development, translations or user feedback.

The biggest advantage of this way of working from my perspective is to be sovereign of your own data, your own images, and not be bound to specific non-interoperable formats and photo libraries.

This type of approach also does not require modern hardware and works well on any type of computer allowing —those willing to learn— to create and work on photographic ideas and projects achieving results equal to those obtained using the so-called professional software used by the industry.

Not least, participation in open source communities allows not only to build and contribute to a collective project but is also a way to improve human relations.