My Photography Workflow & Data Backup

cards-400

If you are a seasoned photographer and just want to know how I have my system setup, you can skip down to a Summary HERE. If you are interested in what it is all about, read on.

*    *    *    *    *    *    *   *    *    *    *    *    *    *    *    *    *    *    *    *    *    *    *    *    *    *    *    *

Whether you capture your images on film or shoot straight to digital,  us photographers create large amounts of data these days. No matter which route you take ( Film, Digital or Both), your images will be digitised at some point and thus creating a digital file. Digital files, just like film negatives and any files need to be managed and organised.

As Moore’s Law prevails, we see technology getting better and better. This means that the resolution of scanners and cameras  increases, so to does the amount of data created in the digital process. This data needs to be processed and stored. The data that were are talking about is the digital files ( images) in RAW, JPEG, TIFF, PSD formats etc.  These files have two properties that we are concerned with,  the quantity of them and  their size.

So we are shooting more and more images these days, so what? The amount of images that we shoot concerns us from an organisational point of view. Like anything in life, the more of something that you have, the harder it will be to find the one that you want. Depending on the type of photography you do, you might capture anything from 100 a week to 10,000 images a day. Having so many images results in a need to manage and catalogue them. This requires dedicated software such as Lightroom, Aperture or Bridge.

The size of each image concerns us from a storage capacity and computing power point of view. The bigger the images, the more storage space it needs. Also, the larger the file, the more computer processing power required. Also in-line with Moore’s Law, the cost per GB of memory and processing power is coming down rapidly. So it’s a constant data / computer circle – we creating more date but it’s getting cheaper to story and process it.

Ok so far? Good. 

One of the dilemmas that we face is how to go about to managing all this data as there are many different choices available. What management / library system to go with, how to organise your computer and hard drive space etc.. There are many, many options out there. There are many perfectly acceptable ways of doing things and there are a few bad ways of doing things. There are cheap ways and there are outlandishly expensive ways of doing it. The amount of solutions  available also follows Moore’s Law in that there are always faster computers available and cheaper storage space available. It is a catch 22.

There is no definitive right or wrong way. Below is how I do it currently and will constantly be reviewed. In fact when researching for this blog post I came across a few techniques that I came across that I should also be doing. But more on that later.

Quick Summary

If you have landed here after a Google search and are looking for a quick answer, here it is. ( Click image to enlarge)

workflow-setup

 

Hardware Setup

  1. Mac Book Pro with 1 x SSD and 1 x HDD
  2. The faster SSD is used for the computer OS and current projects to be completed
  3. Once completed  & printed, shoots and projects are moved to the slower HDD along with all other projects that year
  4. At the end of the year, the projects on the HDD are archived to my server , a LaCie 2Big NAS 6TB
  5. The NAS is in Raid-1 (protection / mirror) mode and has a 3rd hot swappable drive changed around monthly

 

Procedure

  1. Capture images onto CF card while also tethering to iPad
  2. Upload CF Card  to project folder on SSD
  3. SSD drives gets backed up automatically to Time Machine
  4. Rename images, Edit the shoot, delete the trash, keep the “ok images” in Adobe Bridge 
  5. Retouch final selection in Photoshop
  6. When project complete, move project folder from SSD to HDD for storage
  7. Monthly copy of HDD projects onto server
  8. Monthly rotation of server 3rd HDD to off site location
  9. On years end, verify HDD copy on server and clear projects from HDD for the coming year

 

The Ins & Outs

So obviously the quick summary has a lot of gaps in it, but it will give most people the gist of it. Here is a more detailed breakdown of my work flow. Starting with the hardware.

My Equipment

macbook-specs

  • 2011 Mac Book Pro – 2.2Ghz i7 with 16GB. It came with a 500Gb HDD and Superdrive (DVD Player)
  • I replace the 500GB HDD with a Crucial M500 480GB SSD
  • The superdrive was removed and  replaced with the old HDD, thus  I have 2x hard drives in the laptop
  • The SSD is the main startup drive. The HDD is only for storage and for my iTunes library
  • I use Lexar USB 3.0  Dual Slot CF & SD Card reader to upload ( Amazon.co.uk Link)
  • Apple 2TB Time Machine for automatic backup of SSD and HDD
  • LeCie 2 Big NAS 6TB  (2 x 3TB drives) for secure archive

Software Workflow

  • Adobe Bridge – > Adobe Camera Raw -> Photoshop

 

1. The Capture

1. capture

When shooting in studio, I am usually tethered to an iPad for instant image viewing. I am still capturing  images onto memory cards which are used to upload to the computer. The iPad is great for quickly seeing images on a big screen, checking colour and focus and also for showing clients what they are doing wrong and what they need to change. I shoot both RAW & small Jpeg and it is the smaller Jpeg that is sent to the iPad. RAW files can be sent also, but it is slower to send and slower to process by the Shutter Snitch software on the iPad. The size of the sJpeg setting is enough for the purpose.

I currently have a large batch of Scandisk 4GB cards. I purposely use 4GB cards as it forces me to change card card often. with a Canon 1D mkIV,  I can fit around 150 images (RAW + sJpeg) on a 4GB card. In my line of work, I rarely shoot more than 150 images per look anyway, so it means that I have a card per look during a portrait / portfolio session.

This means that in the chance of a memory card being corrupt / lost / damaged,  not too many images are lost, one look at most. If I was shooting 64GB cards and captured a whole shoot on one card, it would be high risk to take.

 

2. The Upload

Lexar Card Reader

I use a Lexar USB 3.0  Dual Slot CF card reader. It is a great card reader and very fast. USB 3.0 combined with an SSD drive, means a full card is uploaded very quick.

I reset the camera numbering system at the start of every shoot and then set it to continuous mode. This means that every image from each shoot has a unique filename straight from the camera. This make copying images from the memory card on to the computer hassle free as there will be no duplicate file names. Each camera has a different letter in the filename also to distinguish which images are from wich camera i.e.

Camera A will have filenames: A_0001.CR2 / A_0001.jpg

Camera B will be: B_0001.CR2 / B_0001.jpg etc. This combines with the times of all camera clocks in sync ensures a smooth import process. Vital is shooting an event such as a wedding where you have multiple shooters.

I have my file structure on my SSD drive set up as follows:

Picture Folder / yyyy-mm-dd Project name/

Within this folder is a subfolder called RAW in which all the RAW files from all the memory cards will be copied. Naming the project folder this way means that they are automatically organised by date and then by project name, if there is more than one project on the same date.

To import the images, I do it manually via Finder.

  1. Insert card into the reader
  2. Find DCIM folder with images
  3. I view folders by file type by default
  4. Select only the RAW files and Copy ( CMD + C)
  5. I do not need the Jpeg files as they were only for the iPad tethering
  6. Select the RAW folder in the project folder
  7. Paste ( CMD + V) the images there
  8. Once complete, eject the card and place beside the computer incase I need to re-upload any.

During step 4, its easy here to copy and obvious test images which are usually at the start of a card. It will ultimately save resources on the CF card and SSD drive which have a finite number of read/writes

 

3. Renaming & Organising

Adobe BridgeOnce the files are imported into the RAW folder, renaming and giving them a quick scan through using bridge is my next step. I want to quickly delete and obviously trash images and rename the images by date, project and image number.

Bridge is an image browser as opposed to an image database such as Lightroom or Aperture.. It works great for me because I create a new folder per project on the SSD and import images into them. This way, images are already organised straight away and means I do not need to database system. If I need to find a shoot, I will know the year, so I browse the folder for that year and it will be within the 40-60 folders that are there. You can search, use tags and keywords in Bridge, but I currently don’t need to. I don’t have enough projects or have any problem finding them at the moment to need them.

  1. Open the RAW Folder in Bridge
  2. Make sure that the cache / preview setting is on “Embedded ( faster)” mode, there is no need for a cache at this point
  3. Quickly scan down through the shoot picking out and deleting any obvious trash images, black images with no flash, over exposed images etc. There could be up to 30-40 of these in a shoot and it removing them now will help speed up the preview cache generation
  4. Select all remaining images and rename them via Tools > Batch Rename
  5. I rename files using a date line, text line and a number sequence: yymmdd–project-xxxx.jpg
  6.  xxxx being an incremental number starting at 001.jpg for small shoots or 0001.jpg for larger shoot with over 1,000 images expected
  7. This means that every images that I shoot will have a unique filename based on its date, project and image number captured that day
  8. This system will carry right on through to client delivery
  9. Set preview options to “Always High Quality” and let it generate high quality previews for 10 minutes – Coffee Time

 

bridge-embed
bridge-rename

 

4.Editing

bridge-starOnce Bridge has spent some time generating high quality preview of all the images, it makes the editing process quickly as I can quickly scan through and zoom in on all the images. Being able to zoom in straight away means that focus points can be quickly checked and the images can be discarded if soft. If the Embedded Preview options is used, you will have to wait for Bridge to render each large preview before you can view it, which takes a few seconds.

An average shoot for me is around 600 images. This needs to be edited down an average of 10 final images for retouching. This editing process can take anywhere from 30mins to 2 hours, depending on the shoot and how similar the images are. When editing a shoot, I usually make a few passes:

  1. The first pass is used to rate any obvious bad images, soft images or any images that are defo for the trash. These are rated with a 1 star and deleted straight away. Out of an average shoot of 600 images, there should be at least 200-300 images to be rated 1 star.
  2. The second pass is to select potential images and give them a 3-star rating. An average shoot will usually have around 50-100 images that are potential candidates. This is where you pick images that you know you like and are good without going in to too much detail. The ones that are left behind ( un-rated) are technically ok images, but they are lacking something either in composition or pose
  3. The third pass is to select 5-star images – The Keepers. This is where I select images that I really like and that I would be happy to retouch. There is on average 20 – 30 5-star images at this stage. It is quite common to be tired looking at the same images at this point, so I might usually take a break.
  4. The final step is to demote some of the 5-star images to 4-star. This is general the hardest step and means that images have to be examined closely to be able to choose between two similar images. It is the different between a few inches of a arm here or a knee there than can make or break these images. What is left though is the absolute best images from the shoot, on average 10 but anywhere from 8 – 16 depending on the shoot.

The Review Mode in bridge is great for comparing images for step 4. above. If you select two images and enter review mode ( CMD + B) it brings up the images side by side for comparison. Alternatively, if you click space bar instead, you can alternate between a full screen view of each image quickly. This is very handy for deciding between two images that have only a tiny difference i.e. an eye difference or a piece of hair.

You can view up to 4 images on screen together in review mode, great for step 3. and comparing similar images. But if you select more than 4 images for review more, it brings up a rotating carousel instead and the view is smaller.

I generally don’t use colour labels when editing a shoot. I may use them after if I have to separate images from a shoot into different sections, if there was two models in a shoot or if there is a clash between a client selection and my selection for example. Normally for a client portrait shoot, I do not even edit the shoot. I let the client pick their own photos from an online proofing gallery. I will run through steps 1 and 2 first though to remove any bad images and let them view the rest.

 

5. Retouching

logo_photoshopRetouching is the pixel pushing part of the workflow, when an image is manipulated in order to enhance it, NOT FIX IT! Retouching is generally the bane is most photographers lives these days. It is a complete time drainer and I personally would rather spend time taking pictures… not retouching them, but that’s just me. You could spend 1 minute setting up and taking a picture yet you might have to spend hours working on it afterwards ( depending on your style and effect required of course). I generally like to have an image retouched in under 30mins on average.

Retouching was once a completely separate job to a photographer back in the film days as a different set of skills was required. This should still be true today as retouching is an art form. Unfortunately these days, people expect the photographer also to be the retoucher.

photoshop

I  have two options when it comes to retouching,  I do it myself or I outsource it. For general portfolios and client work, I outsource my retouching to a professional retouching house. For test shoots, special projects and my own retouching. For my own retouching, I use  Photoshop. I am not going into detail on retouching here as it’s a minefield. But a summary is this:

  1. Open bridge and view filter for 5-star images only
  2. Work on one image at a time through Adobe Camera Raw > Photoshop
  3. As I am working on images, I regularly save the PSD file in a new PSD folder within the project folder ( Now there is a RAW & PSD folder).
  4. Once retouching is complete and I have the final image in photoshop, I have an action to flatten the image and save it in multiple folders and sizes i.e. Print, Press, Web-1200px and Web-700px
  5. Due to the nature of PS actions, these images are always saved on the desktop and then moved to the project folder

The Project folder now looks like this/

RAW – Original RAW files and XMP adjustment files from Adobe Bridge
PSD – The multi-layered PSD files ( 100-500mb each)
Print – Large high quality Jpegs for printing ( 5 – 9mb each)
Press – Large – Medium quality Jpegs for emailing to newspapers for printing ( 2-4mb each)
Agency – Jpegs with 1200px on the longest side with no watermark for model agency web usage
Web-12oopx – Jpegs with 1200px on the longest side with watermark
Web-700px – Jpegs with 700px on the longest side and low quality for FaceBook and other social media usage
Instagram – A separate folder for a select view image what I custom square crop for upload onto instagram

bridge-folders

5. Archive & Backup

copy

Once a shoot  has been printed / emailed to the end-user, its is time to archive it. When I archive a shoot, I delete the PSD folder in the project as all I am interested in keeping is the RAW files and the Jpegs. Multi layer PSD files can take up a lot of space and they are really only a midway point from the RAW to the Jpeg, so they can be deleted. 10 PSD files can take more space that all the RAW files from a shoot.

To archive a shoot, I simple COPY the project folder in the SSD and PASTE into the HDD. I do not CUT & Paste and it is safer to copy the folder in case there is any error in the process and the computer has then removed the folder. In the unlikely event of a problem with a folder, it can always be restored with Time Machine. Once the copy has been completed, I check the files to make sure they have the same amount of files in each and are the same size. Only then is the project deleted from the SSD drive and the space regained. The diagram below shows you my current setup.

archive-setup

The Time Machine always has my SSD and HDD covered as it automatically creates hourly backups of these two drives. This means that the current year archive on the HDD is saved on the HDD itself and the Time Machine = Two Locations. Time Machine is a fantastic tool and I have had to use it twice in two years now to recover old images that were deleted or corrupt.

Once a month, the Yearly Folder on the HDD gets copied over to server. I keep a year of photos on the HDD for two main reason:

  1. To keep local copies of recent shoots to be able to send them out to press, instagram, blogging etc
  2. To has as an extra backup

 

Conclusion 

All of the above might give you you a little insight as to how I manage my images. It is not a perfect system, but it currently works for me. I have just found some great other tips for backing up and archiving over on http://pondini.org/TM/27.html . One suggestion is to create a bootable partition on the HDD so I have the option of using the computer if the SDD ever fails.

If you have any questions on anything, or if there are any obviously corrections to make, drop a comment below and let me know 🙂

 

9 comments on “My Photography Workflow & Data Backup

  1. Great read Richie. One of those jobs I keep putting on the long finger is sorting out my backups into something streamlined like yours. Hope my current mish mash of a system doesn’t die on me first!

  2. Cheers Dennis. Hopefully it will help some out in regards to this area… there are so many options out there, its hard to know what way to go.

  3. Very interesting. Using the 4GB cards is a good idea. What’s the name of the device you use to tether to the iPAD?

  4. Hi Dave,

    I use the Canon WFT E2 transmitter to send the images to the ipad via a wifi network.

    I might do a detailed post on it in future if I get time.

  5. Yeah this is a great little tutorial for people starting out that want to know how the pro’s go about there daily back-ups. This method is almost identical to the one I use. I opted for Drobo which is also a really solid, reliable machine. Love the site and your work Richie!

  6. Thanks Gary, I am glad you find it useful.

    An addition that will need to be made soon is that that I have just ordered a LaCie 5Big NAS as an extra system, the 10TB version. I needed more space as the 3TB is nearly full.

    The 5Big drive will become the main drive now and the 2BIG drive will become a extra backup.

Leave a Reply