Capture and Beyond Workflow Lessons That You Cannot Afford to Miss
There are no images to accompany this post. Why? When I first started digital photography in September, 2002 (not 2001 as I have written before…), I did not realize that it was important to save each RAW file. Each RAW file is the digital negative for an image that represents an irreplaceable moment in time. Note that I did know enough at the time to use RAW not JPEG capture. But once I converted the RAW files and optimized them I simply discarded them. Ooops.
I learned how bad the situation in those early years actually was while working on the 13,000+ image file that I mentioned in the Gems Found, the Sunday evening post. I came across many images that needed additional Photoshop work. We do not always realize it, but over the years our optimization skills will increase by leaps and bounds. There were many dozens of images that I opened in Photoshop and worked on for a few to ten minutes using skills that I did not possess back then. In addition, there were more than a few excellent images that screamed for me to start over with the RAW file and process them again using the Photoshop skills that I had honed over the past decade.
Here is the e-mail conversation with Jim Litzenberg that followed:
Jim, Please do a search for the following RAW files: 279T9135, T9J5688, 123F4928, and L8X2290. Thanks, artie
Boss, I found many optimized TIFFs and several JPEGs for each of the file numbers above but there were no RAW files. Jim
Jim, Thanks. That’s what I figgered might happen…. artie
Here are some details on the missing images that I wanted to re-do:
123F4928: Snow Geese, silver yellow sunrise, Bosque Del Apache NWR, San Antonio, NM. December 2, 2002. EOS-1D. This was the image that–with help from Rich Steel– fueled my digital epiphany.
279T9135: Roseate Spoonbill, Lake Martin, LA. April 7, 2003. EOS-1D. This image has helped to sell more than 3300 copies of Digital Basics.
L8X2290: Snow Geese, blast-off pano, Bosque Del Apache NWR, San Antonio, NM. November 19, 2005. EOS-1Ds. A three-layered cropped pano that is one of my very favorite all time Bosque blurs.
All three of the images listed above plus another dozen or so that I e-mailed Jim about would have benefited greatly from being re-converted and re-worked. However, no RAW files, no can do.
Interestingly enough, when I arranged the files by date I learned that the images from the trip to Tanazania that I made in January of 2003 with Patti and the late Wes Ardoin were all labeled January, 1980….
Lessons for you (including those that I have learned the hard way):
1: You will get smarter and you will become better at optimizing your images.
2: Use only RAW capture 100% of the time. If you need JPEGs, simply set your camera to Large RAW plus whatever sized JPEGs you would like. If you set your camera to JPEG capture for whatever frivolous reason or reasons rest assured that you will forget to switch backs to RAW and will encounter a great situation while shooting only JPEGs. Usually small JPEGs….
3: Come up with a workflow that allows you to save and archive all of your RAW files even after they are optimized.
4: Back up your images daily while you are in the field and once a week once you have the images on your home computer.
5: Be sure to set the date on your cameras properly and to check them regularly. And that goes double when you travel to a different time zone.
Those who would like to improve their Photoshop skills more quickly than I did should get themselves a copy of our Digital Basics. This PDF is sent via e-mail. It includes my complete digital workflow, dozens and dozens of great Photoshop tips, and the personalized keyboard shortcuts that I use every day (as well as instructions on how to create them).
I must admit that for the first time about 10 weeks ago I received an e-mail from a guy complaining that Digital Basics was poorly written and was so complex that he could not follow any of the instructions. With more than 3,300 sold a single complaint is nothing to get too upset about, but needless to say I was somewhat mystified as I pride myself in taking great care with all of my writing. Especially with my how-to stuff. I answered the guy as politely as possible urging him to have another go at it and suggesting that he use Digital Basics exactly as he would a cookbook. About a month ago I got another e-mail from the same guy thanking me profusely for creating such a helpful, easy to read, and easy to follow guide. 🙂
As I say often, you gotta love it.
HC, DNGs, whether accepted by contests or not, do have all of the information that a “RAW” file has whether it is a NEF or a CR2. I cannot answer why contests are not run in the 21st Century; I do understand contests wanting to see the original file in some form.
Jay. This remark, “I do understand contests wanting to see the original file in some form” makes zero sense to me; if a contest does not require the submission of the RAW file it becomes a digital free for all with anything goes, adding birds and creating composites… artie
ps: You missed a question that I asked of you below.
All of these posts are so the helpful….I’m still struggling with using a consistent workflow and being able to find all of my images; I have Digital Basics and I’m using a MAC, Lightrrom3 and PhotoshopCS4 so my workflow is a little different than in Digital basics and until now, I’ve been converting to DNG when I download from my CF Cards but I’ll start saving my RAW files…I thought that the DNG’s had all of the info as my RAW files but in a more universal format… thanks for the tips, (I want to believe lesson #1 but for me it might be magical thinking!
Looking around, there’s also the OptiBay, which allows to replace the optical disk reader with another hard disk (or SSD). Actually, I haven’t been using the disk reader for months (and when I need it, I can connect an external box). This would allow me to have two disks in my laptop, one for business and one for photos… Thinking of it.
Fabrizio, you might be interested in my new system. I have the programs all on an 128 gb SSD, a second SSD for the Photoshop scratch disc and a 2TB HD for my data. I can easily replace the HD when it is full without having to reinstall my programs since they are on a separate drive. The PC is attached to a Drobo which scans the HD each night and automatically uploads a copy of new files to the Drobo (copy and backup on separate HDs). I use external HDs (2 for a copy and a cuplicate) for downloading on trips and do not store anything on my little notebook.
Thanks Antonio. Indeed I do have the setup you recommend: an identical copy of my internal hard disk is weekly kept up to date in case of failure. The problem is *upgrading*. E.g. to replace my current 640GB internal hard disk with a 1TB hard disk. There’s no cloning that you can do, in this case, but I’d have to reinstall all the operating systems I have on the disk. That’s why I try to avoid this operation.
For what concerns keeping the images on an external disk, it’s what I did some years ago. But I discovered that external disks were failing with a worrying rate. Probably related to the bumps they got while in the bag. So at the moment I don’t trust external disks to hold valuable data any longer. SSD could be an interesting option, but still too expensive (512GB cost about 500 euros).
Probably the mistake is the desire to keep everything (job + photos) on the same gear.
Fabrizio,
You mentioned a full day or so of work to setup your laptop’s HD, in case you need to. You may want to consider a piece of software that clones (not just backs up) your laptop’s HD to an external drive, or to a partition of an external drive. That way, if disaster strikes, you can EITHER replace your defective HD in your laptop and just restore everything you had from that clone, OR you can boot up from that clone if you can’t immediately replace your internal one. In summary, you’re always up and running.
So, as Art suggested, keep your pics in an external drive. Keep a backup of this in another external drive and, either in a partition of one of those drives or on both or on a 3rd external drive, keep a clone of your internal HD.
For cloning on a Mac, I’d suggest Carbon Copy Cloner, a shareware.
Like Mike, I have lost side car files. They are separate from RAW files so losing them is a definite risk. If you process thousands of files, it is going to happen eventually. No such problem with DNG. It is just simpler and more secure to use DNG for that reason. I cannot conceive of why a contest would prohibit DNGs except for ignorabce as to the DNG concept. However, I cannot conceive of why a contest would prohibit cropping and selective editing either. ;-0
Jay, The Valley Land Fund Photo Contests out of McAllen, TX (when they existed) had such a requirement. These contests also prohibited any cropping, the addition or removal of anything other dust spots, and only global corrections could be made (i.e.: no selections of any kind).
Thanks again Mike. There are many contests including the BBC and my very own BIRDS AS ART 1ST International Bird Photography Competition that require that RAW files (not DNGs) be submitted for winning images in all categories (except Digital Creations). Bottom line: you should be saving your intact RAW files (at least for your very best images. artie
About the use of DNG, there are pros and cons. For me, I prefer to keep sidecars (in short, I keep stuff as NEF and then I postprocess in Lightroom; the information about postprocessing stay both in the Lightroom database and in XMP files). I much prefer this approach because NEF are never touched, so I can safely backup them once and for all; any further postprocessing will only modify XMP, which I can backup separately.
Artie, just not to be misunderstood: sure I know that you as other professionals are driven by the desire of improving too! But amateurs like me have to learn much more stuff because we start from a point far behind pros.
Fabrizio, Pros are not born with any innate knowledge of Phostoshop or Lightroom. They start from zero just like amateurs do. In 2002 I knew nothing about the right way to optimize images. Folks today have a huge advantage in that there is so much great information around from a variety of sources including but in no way limited to Digital Basics and Bird Photographers.Net. And the same thing is true for those wishing to learn to create better images or to learn exposure theory. When I started there was nothing around that can compare to what is out there today…. artie
[…] post: Capture and Beyond Workflow Lessons That You Cannot Afford to … /* */ /* */ window.fbAsyncInit = function() { […]
Mike, I cannot image any contest judge indicating that they will not accept a DNG. My understanding is that if you make any changes to the original, e.g., in Lightroom and then save/resave as a DNG the fact of the change would be shown in the metadata. Gurus like Scott Kelby and Julieanne Kost all suggest the use of DNGs; much smaller file and no .xmp. Artie, this would be an interesting subject to obtain a comment from Tim Grey. Cheers.
Jay, what does this mean: “I cannot image any contest judge indicating that they will not accept a DNG.” artie
ps: As for what the gurus think and advise, please see my comments to Mike’s last post below.
Artie, with respect to the comment asking do you save CR2s with side cars or DNGs, If it is the file that came from your camera it is a CR2 (as I am sure you know). The metadata are stored in a companion sidecar file. I save two versions of my RAW files. One is an unaltered, other than name change, CR2 one an external hard drive with the sidecar file. The other is a DNG that I save on another external hard drive. My reasoning is that I have entered some contests that want to see the unadjusted RAW that came out of the camera. Hence, the unadjusted CR2. However, one of the aspects of DNGs that I like is that the metadata are stored within the file so I do not have to be concerned with moving files and loosing the metadata. (For reasons I do not understand, I have moved files only to later discovered that the sidecar file did not move with the image.) As a result, I do all of my ARC adjustments on the DNG. Another point: I rename my CR2s before I convert them to DNGs. This way, any file originating from a given CR2 will have the original’s name (i.e.: The DNG and all of the processed files).
Thanks for the info Mike. Being an old dog I like to keep things simple. I select my keepers from a given folder whether it be a morning’s take or an entire trip. I back up the RAWs as they come out of the camera. I process maybe 10% of what I keep in Photoshop saving the optimized images as TIFFs. I try to back up every day or so when I am on the road. I don’t know from sidecars or DNGs or XMP files. Being an old dog I like to keep things simple. :). It has been working for me this way and with storage so cheap I am happy in my ignorance! artie
Anyone needing more storage should check out the Drobo. I love mine. Jay points out something else. No side car files with DNG which is a signifigant advantage IMO.
Also, Art, I found that going over and using your Digital Basics was much easier the second time around with at least a year of serious shooting under my belt. It was not your writing at all, but there is so much to learn and digest when one gets serious about digital I just got overwhelmed. And the patch tool really is the “bomb” as you like to say, so easy. Thanks
Thanks a stack Stan. And to think that less than ten years ago the word “Photoshop” instilled fear in me…. artie
Keith, do you find that the Topaz denoise is that much better than Lightroom 3? I’ve been thinking of purchasing that or noise ninja.
Thanks to any one responding.
TIFFs have a lot more information in them than a JPEG, and are essentially lossless. Although Keith is right, you can’t benefit from the improved RAW convertors. Still, those TIFFs should still be quite editable.
Bill, I am confused by your comment… What was Keith right about? Who cannot benefit from improved RAW converters? And TIFFs can always be edited…. Please explain. artie
Artie, are you saving as CR2s with an .xmp sidecar file that you also have to keep track of or are you converting and saving as DNGs?
Jay, I do not know the answer to your first question. I simply save the RAW files as they came out of the camera (with the embedded JPEG intact). After converting my RAW files I optimize them and save them as TIFFs so the answer to your second question would be “No.” artie
Artie: Just to refresh your memory, your trip to Tanzania in Jan, 2003 was made with Todd Gustafson, John Dupps, John Bryant, Nick (?) from Hawaii, and myself. I am sure you will recall your electrical problems on our arrival night! Regards, Jim
I remember that night. Brand new laptop did not work!!! I was there the year before with Patti and Wes and more folks. John Dupps still travels with me and I am regularly in touch with John Bryant. And Toddi and I are still good friends though I sort of quit going to Africa. Good to know that you are still breathing. artie
Yes, disks are cheaper and cheaper, but to replace the hard disk on my laptop I need more than 1 day of work (the setup for my job is complex). This costs much more than the hard disk itself. Probably I should keep two separate laptops.
For the rest, I think that a pro can map decisions on revenues, as it’s logic. So you can decide not to reprocess many photos because you know that you wouldn’t sell them better. As an amateur, I’m primarily driven by the desire of improving, and I’ve very wide margins of things to learn in postprocessing.
There is not much need to keep your best images on your laptop…. I only keep the most recent ones on my laptop plus tons of 1400 X 1050 JPEGs for slide programs. Your collection should be on external HDs. That way your laptop will run much faster…. BTW, as a professional I am driven by the desire to improve in all areas of photography. In that respect I am no different from you…. artie
HD is really cheap. I just got one with 3TB for $130. After reading your Digital Basics months ago, I have been shooting only raw. I agree that we all should keep a copy of raw files just in case new technology comes along and we can reprocess some of them. Because of your Digital Basics, I am also able to save many images that I once considered garbage. Please do let us know when you have an update on Digital Basics. Thanks.
Hi Nick, Thanks for your purchase and your kind words. Not sure when the next update will be but everyone will know about it :). artie
I second the post and the comments. As a drawback (but, I mean, every good thing has got drawbacks) you need lots of disk room (I prefer to have all my photos on my laptop, because often I find some time to process them during job breaks, and considering that I’m a software engineer and I need large disk room for my job, I always have the desire of a larger disk) and lots of time, because when you face with a dramatic software improvement you feel the need of reprocessing the whole archive…
Fabrizio, Storage is so cheap today as to barely be a factor: 1 tb drives for $100…. Reprocessing the whole archive would be a great plan if I have say 10 or 20 extra lifetimes…. As that will not happen soon I will be opting as above to work on a few select images. Another factor is that image sales are down miles from a decade ago making it hardly worth the effort to re-work anything…. artie
I will second what Keith said. I started with Photoshop Elements and Photoshop 7 with no plug ins. I have learned a lot over the years but Photoshop has also made remarkable strides. I also am now converting to DNG so I save a bit of space. However, storage is now cheap so there is no reason to not save the RAW files.
Agree 100% with your storage comments. Robert O’Toole is now saving every image the he creates without deleting anything. His reasoning is that HDs are so cheap that he does not want to risk deleting anything good by accident. artie
A lot of good information. Thanks.
Not only that Art, but both RAW converters and image editors get better all the time too – I’ve breathed new life into old images time and time again simply because (say) Lightroom 3 provides far cleaner, more detailed conversions than Lightroom 2; or because some unwanted feature in an old image that I could never quite remove to my satisfaction becomes an easy job because of Content-Aware Fill in CS5.
And my life has been tranformed (well not quite, but you get the idea!) by the fantastic Topaz Denoise 5 plugin, which makes it ridiculously easy to deal with images that were once consigned to the doldrums simply because they were too noisy for the software I was using at the time to deal with.
Thanks Keith. You are 100% on the improved conversion engines. I will have to check out Denoise. artie