What You Should Know Before Reading This Post
This document is intended for runners wishing to enter wrangling. It assumes no prior experience with wrangling, ingesting, data management or other related tasks. It is helpful if you are comfortable with OS X. You should know how to get to System Preferences, use Spotlight (⌘CMD+spacebar) and navigate quickly through Finder.
In this guide we will be using OS X Yosemite, Shotput Pro 5 or 6, VLC and Black Magic Disk Speed Test.
What is a Data Wrangler (Digital Media Technician)?
The wrangler secures and copies original camera and sound media. Depending on the country, genre and specific show, their tasks may also include: transcoding, general runner duties, driving, pulling stills for the DP/producers, troubleshooting, general IT tasks.
They work closely with camera assistants, onset sound, edit assistants. Their director supervisor is typically the director of photography in film, and the production manager in factual entertainment.
What is a DIT? How is it Different?
The Digital Imaging Technician works with the director of photography to establish the look of show. He does the data wrangler’s job of copying and securing the media, and in addition acts as a mini-colourist on set, doing light colour correction and/or grading either on the camera truck or doing a live grade onset with the director of photography at his side. His looks are passed to the dailies colourist (or edit assistant if the show does not have one) who ensures everything is consistent and matching, and that is how the show will look until picture lock when it will be handed to the colourist.
DITs come from various backgrounds, including colourists, director of photography, dailies colourist, assistant DIT, and sometimes wrangling. It’s probably worth noting that they show up to set with between £15k and £50k worth of kit.
General Tips for Wrangling
Your primary mission is to protect original camera and audio media; it is the end result of everyone’s hard work, and a mistake by you can cost everyone else a day of production. It is important to understand your workflow completely, automate as much as you can, identify spots where it is possible for you (or others) to make a mistake, and then create a step in your workflow to check if a mistake has been made.
Be as organized as possible, with both your hardware and your software. In an ideal world, your work is so organized and clearly labeled that another data wrangler can step in at any moment and pick up where you left off. If this is not possible, document any unusual steps that you have taken and list the reasons why. This helps anyone who touches your work afterward: other data wranglers, edit assistants, assistant editors, storyline producers etc. It also helps you should you be asked to revisit the work after your job has ended.
Never rename any original camera footage. Always copy camera cards, the entire card with all metadata folders. Don’t rename any of the folders. If the camera card names are not unique, devise a naming scheme for them, create a new folder with the name, and dump the entire camera card contents into this. Always consult with others before renaming anything; generally the next person to handle your work, which is likely going to be an edit assistant, dailies operator or technical runner.
Anything you name should be as descriptive as possible. Spending an extra few minutes to think of a good, unique, self-explanatory name will end up saving many more minutes in the future. Version your filenames by adding either a version number (ie 2.4, 2.5) or alternatively the date (can be more difficult to read, and presents problems if you have multiple versions for one date).
The matrix final final.mov
Good naming conventions should be applied to file names, folders, email titles, hard drives, in both work and personal life. It’s just a nice habit that will save you and others time.
General Tips for Troubleshooting
One of the first things you should be doing when setting up on a job is establishing some sort of internet connection; if production is not able to obtain one for you, you may be forced to tether off your phone. The net is your lifeline to solutions if something goes wrong: anything from software failure, hard disks failing to mount, to full laptop meltdown. Google and a second laptop that you trust (typically your own) will end up being all you need to fix most problems that you run into.
So you have some type of problem. It’s okay. Don’t panic. This is a great opportunity to learn. If it’s a software problem, often the software itself will tell you what the problem is, by generating an error report. Read the error report. If you don’t understand what the error report says, this is also okay. Save it somewhere or take a screenshot of it (CMD+OPT+3). Try googling what the problem is about. Restart the computer (turn it off, then turning it on again, as opposed to selecting “restart”, often works better). Make a note of each thing that you’ve tried.
If it’s a hardware problem, try to identify exactly what is failing. Let’s say a hard drive isn’t mounting. What happens if you change a cable? What happens if you change the USB port on the laptop? What happens if you try to plug into a different laptop? What happens if you try to view the drive in Disk Utility instead of looking for it on Finder? What happens if you look for it in Terminal (not sure how to do that? The answer is just a quick Google search away!). What happens if you plug it into a Windows machine instead of OS X?
If you’ve tried searching around and can’t find an immediate solution to your problem, try asking the next person involved, such as the camera assistant, another wrangler, studio applications support, techzone. The person who understands your workflow the best, and is the most able to help you, is the edit assistant in the post-production studio, however that person may not always be available, in the case of doing producers’ audition tours, you won’t have an edit assistant at all. You can also try posting your problem on a relevant interest group, such as Creative Cow, Adobe’s tech forums, Stack Overflow, etc. Many people will not know the answer, and some people may know the answer but not have time to help you. If your support network can’t figure it out, you can take it to your direct supervisor and explain what the problem is, what you have attempted to resolve it, and then ask how best you should proceed. The show goes on whether you’re having technical problems or not.
Cool Stuff to Make You Look Cool
At bare minimum, your kit should include a laptop, surge protector, card readers, USB hub, and camera tape in red and green. It is recommended that you also have a UPS (uninterruptible power supply), RAID 5 or 6, external monitor, small control surface (even a Tangent Ripple would be okay) USB hub, label maker and peripherals, and pelican cases for transport.
A fancier setup would include a workstation (iMac 5k, Mac Pro, HP z840 being good choices), broadcast monitor, and more purpose-built colour control surface – this would be for if you’re intending to transition to DIT or colourist.
Software & User Preferences
When transcoding the most important component is typically the CPU, but for working in Resolve, Premiere or other NLEs, your RAM and graphics card will also play a role. Go through machine(s) and make sure that you have the correct programs installed, which means: ShotPutPro (hopefully ShotPutPro 6, but ShotPutPro 5 is also fine), VLC, DaVinci Resolve 12.5, and Black Magic Disk Speed Test (available for free in the App Store, or in the Downloads section of Black Magic’s site). If you’re on a Premiere workflow, then you should also have Premiere, Media Encoder, and Adobe Creative Cloud. Keep Resolve on your system, it may be helpful for troubleshooting.
Disable Sleeping –> Can cause problems with transcoding, working in Resolve/Premiere in general, and computer putting itself or limiting itself when you leave a transcoding job overnight
System Preferences > Energy Saver > Battery > uncheck “Put hard disks to sleep when possible”
System Preferences > Energy Saver > Power Adaptor > check “Prevent computer from sleeping automatically when display is off”
System Preferences > Energy Saver > Power Adaptor > uncheck “Put hard disks to sleep when possible”
System Preferences > Energy Saver > check “Show battery status in the menu bar”
Show Filename Extensions –> You always want to quickly see what type of file you’re working with.
Finder> Preferences> Advanced> Show Filename Extensions
Set up Finder –> Show information about the folders you’re working in.
Finder > View > Customize > Show Icon And Text
Finder > View > Show Path name
Finder > View > Show Tab bar
Finder > Advanced > Show Hard disks + External
Trackpad Settings –> Faster is better
System Preferences > Trackpad > Tap to click
Tracking Speed > fast
Click > light
Disable Mac Function Keys –> Certain programs require function keys for important shortcuts
System Preferences > Keyboard > Keyboard > Check “use all F1, F2 etc keys as std function”
Show login name top right –> Show which user you are logged into
System Preferences > Users & Groups > Login Options > Enable Show Fast user switching menu
Customize Dock –> You shouldn’t be using Dock to access programs, use Spotlight, it’s faster (⌘CMD + space)
Minimize Dock / disable bouncing dock
magnification 0, size towards small
System Prefs > Dock > check “automatically hide and show Dock”
System Prefs > Dock > uncheck “animate opening programs”
Why Use ShotputPro?
The purpose of copying media in ShotputPro is threefold:
- Automating the workflow. Shotput allows you to copy footage onto any number of drives that you like, meaning you can create a double or triple backup without having to remember what has been copied.
- Creating proof of successful copy. Useful if in the future footage has become corrupted or lost: Shotput automatically creates records of what you copied and to where.
- Checksum verification re-examines the data after you copied it, basically a check to make sure that everything is really and truly copied. If Shotput finds a problem, it will notify you so that you can re-copy the data.
To use Shotput is to know peace of mind as a wrangler. You should never work on any production, even if it’s just a student thing, without it (or something similar like Silverstack). It has a one-time license of around £80 so there’s little reason not to use it.
Handling Camera Cards
The most likely place for something to go wrong is not during the copy process itself; ShotputPro works really well the ensure everything that starts to get copied, finishes being copied. The place where a mistake has the most potential is the time in between the card is ejected from the camera and before it is inserted into your computer’s card reader. You must have a system in place. Every wrangler will ultimately develop their own system; the important thing is that you have a system, you always follow it, and everyone who touches camera cards (camera assistants, operators) is aware of the system.
I prefer a positive-tape-colour system; when the card has been copied and ejected from my computer, I wrap the tape in green tape to show that this card is ready to be reformatted. I don’t format cards myself, and leave it to the the camera assistants to format cards. They are instructed to only format cards with green tape.
In an ideal world, the camera assistants will mark exposed cards with red tape or exposed tape. However, the assistants are very busy and are multitasking, so it’s possible for them to forget to mark a card as exposed. That’s why you should not rely on negative-tape-colours alone; because cards that have a mistake (no red tape) look like fresh cards.
Take a bin or box and label it “ingest” and instruct all camera assistants to leave their exposed cards and logs there. Always keep the exposed cards and copied cards in separate areas. When you pick up an exposed card, always insert it directly into the reader and copy it into Shotput right away. If you need to put the card down, if someone is asking you something, never hold the card in your hand; finish your work or put it back in the ingest box. You always want to have supreme confidence about what has been copied and what is not copied yet. Once the card is finished copying, immediately wrap it in green tape once it comes out of the reader. If something interrupts you, you take the card out, set it down, then come back to work, you can’t just wrap it in green tape now even if you’re confident in what just happened; you must check the card first.
Create a Preset by going to the + sign in the lower left. Think of a good name, like Matrix Day 31, or Matrix Day 31 July 3 (an example of a bad name would be: “My Preset”). Convention > Card Name. Under Offload Destinations, click the + icon to add each drive that you want to copy to, and navigate to the correct folder path. For something that’s not meant to go to broadcast you are likely making two copies, and if the show is for broadcast you are likely to be making three copies. One copy will be going onto the USB-powered drive (called the shuttle drive) and this is what is given to the post house. The other drive(s) are externally-powered and stay with you. Check with your direct supervisor, and if he/she is unsure, check with the assistant editor.
When the day changes, or if you do a drive change, you will need to edit your preset to route to the updated drive and folder path (make sure to edit your title too to reflect a date change etc!). When you first create the preset, you’ll want to do a test to make sure that the cards are named as you expect them, and everything is showing up in the place you expect it to be. After you have tested the preset convention once, you don’t need to check it anymore.
With your preset selected, you can now drop the contents of the camera card into Shotput Pro; it is added into a queue. Press the “play” or “begin” button to start copying files. There will be a popup notification when each card completes.
Verifying Your Work
We know, once we put a card into ShotputPro, that it’s going to be copied. But Shotput isn’t going to tell you if the camera failed to record audio, if the camera is in 720p instead of 4k, or if there’s six dead pixels in the middle of talent’s face. As data wrangler, there’s a limit to how much quality control you can reasonably do on set, and errors will slip by you. It’s okay. Try to catch as much as you can.
Particularly on the first day, you want to make sure that the camera is recording what you expect it to, in the right format, and so on. Bring your media into either Resolve or Premiere, check the metadata, have a listen to the clips, make sure they are playing properly. Once you are satisfied with that, you can skim camera cards after you have copied them in Quicktime or VLC. When skimming a clip, you’re mainly checking if it plays; a corrupted clip may play at some parts and stop at others. So you want to hit play at the beginning, scrub through to one or two random spots in the middle, then play out the end. You’ll find that when a clip corrupts, it’s often going to be somewhere at the very end, so make sure you’re checking for that.
So now you know that your cards that you copied play correctly – but how do you know if you have all the cards? On a two-camera drama feature, this isn’t typically an issue. But on a large multicam action show, or 17-cam entertainment show, you might be running through cards left and right. Your card handling system should clearly identify what cards need processing and what cards are finished, but under heavy load you want a secondary check. This usually means cross-referencing with the logger or camera assistant reports that are handed to you with each camera card to make sure that you have the right amount of clips and the right amount of cards.
ShotputPro Settings & Tips
If your camera card doesn’t have a unique name (for instance if they are all named something unhelpful such as Canon XF), you can rename the camera card to something useful, such as B1003, B2001 (where B1003 = B1 room, the third camera card of the day, and B2001 = B2 room, the first camera clip of the day). Consult with the camera crew to change the card names in-camera to match your naming scheme if possible. This doesn’t break the rule of not renaming things because essentially what we’re doing is having Shotput create a folder and dumping the unedited camera contents into that. Creating top level folders is okay; modifying existing camera folders is not okay.
I like the enable “auto-eject” in ShotputPro but this is an optional setting; ShotputPro by default will eject a camera or audio card when it is finished copying, therefore you know when you see a mounted camera card in Finder, it’s something that hasn’t been copied yet. ShotputPro > Automatically Eject Card. The checksum verification should be MD5 Checksum; it is the default option, chosen because it is a good blend of performance + security; other checks may be more secure but take longer to complete.
Hard Drives, Thunderbolt vs USB, RAIDs: An Overview
How long should it take to complete a copy in ShotputPro? It depends of the size of your media, the amount of drives, and the disk speed of your drives. A lot of the time, the wrangler will be handled two types of drive: a USB-powered rugged and hopefully 7200rpm drive (Lacie is a popular manufacturer), and an externally powered 7200rpm single drive (G-Tech is popular). If you check the drives in the recommended helper program Black Magic Disk Speed Test, you will see that the USB powered drive read/writes at around 80-90 MB/s, and the externally powered should be writing at 170-220MB/s. USB-powered drives are typically much slower, as well as being more prone to data corruption and other issues, than externally powered drives.
Working with data, drive speeds are a really important thing to understand.
USB-powered single 5400rpm drive – 25-30 MB/s
USB-powered single 7200rpm drive – 80-90 MB/s
externally powered single 5400rpm drive – 100-130 MB/s
externally powered single 7200rpm drive – 150-200 MB/s
2-bay raid 0 – 240-290 MB/s
6-bay raid 5 – 700-800 MB/s
single SSD – 240-1500 MB/s
12-bay raid 0 – 1200 MB/s
You’ll notice I didn’t write the connection types. That’s because USB 3 has a capacity of 5 GB/s, so until you start getting into a raided SSD, it doesn’t matter if you’re on USB 3 or Thunderbolt; your limiting factor is the disks themselves, not your connection type. That doesn’t mean Thunderbolt is a waste of money on drives… the connector itself is physically less prone to damage, and devices can be daisy-chained (you can plug a thunderbolt drive into your computer, then a second thunderbolt drive into the first one, then a third thunderbolt drive in). So it’s a nice thing to have, but it doesn’t magically make anything “faster”, don’t buy into the hype!
What is a RAID drive? It’s something you’ll often be using as a data wrangler. It means two or more drives that are working together. G-Tech, Lacie, Promise, Western Digital and Areca are examples of hard drive companies that offer RAID setups. RAID 0 means that data is split up and written across two or more drives at the same time. This ends up being faster than writing to a single drive, but as most things in life it comes at a cost: because the data is split across two drives, now you are twice as likely to encounter a hard drive failure. If one drive goes down, the other drive only contains fragments of what you need. That’s pretty scary, which is why we have Raid 1: data is written in full to two or more drives at the same time. Each drive is an exact copy of every other drive. This ends up being incredibly slow, which is why we have Raid 5: it works like Raid 0, so that data is back to getting split up again (“striping”), but then there is also a reserve drive set aside (called the parity drive), which holds enough information about the data that any one of those drives can fail and the parity drive will be able to allow reconstruction of the data. The parity drive itself can also fail and then you’re left with the original data, effectively functioning as a Raid 0 configuration. Raid 6 works similarly to Raid 5, except you can lose two hard drives, and it’s a little slower. A 5-bay 20TB Lacie “5 Big” in Raid 0 will write at 910 MB/s; in Raid 5 it writes at 710 MB/s.
The other magic number you want to be aware of is your NLE (in this case Resolve or Premiere) is very sensitive to the speed of the drive you’re using. If you’re working in 1080p footage, you want to be read/writing at around 160-200 MB/s as a minimum. If you use a slower drive, you’ll experience a significant drop in performance. Camera cards can read/write as slow as 15 MB/s, which is why it is not recommended to work off of these (it can also damage the card, corrupt data, and cause problems relinking in your NLE).
Formatting a Hard Drive
Eventually someone’s going to hand you a fresh hard drive or perhaps broken drive and expect you to format it. Don’t worry, it’s super easy and fun.
- First check that the hard drive is blank or contains things you know you can delete. If you’re not sure, ask.
- CMD+space + type “Disk Utility”
- Select the hard drive. Make sure it’s the correct one. It’s not a terrible idea to eject all other hard drives temporarily to prevent making a horrible mistake.
- Partition > Partition Layout “2 Partitions”
- Select the first partition, name it something short, useful and unique. Under Size type in 90% of the size of the hard drive; so for a 1 TB drive we would write 0.9. If you’re expecting the drive to only be played back on Macs, use the default format Mac OS Extended. If you’re not sure what it will be played back on, choose ExFAT.
- The second partition is automatically sized to the remaining space. Select it, and for format, choose “free space”. This locks off a section of free space on the hard drive that no one can write to, and it’s done to protect the drive from being accidentally overfilled. Hard drives perform better while staying inside a 90-95% buffer zone.
- Maybe do one last check to make sure that you indeed have the right drive selected 🙂
Hit “Apply” – you are now reformatted.
Issue: Spotlight/Finder is not able to locate media in a specific location that you know to exist.
Fix: Re-index the applicable folder or drive. System Preferences > Spotlight > Privacy. Drag the glitchy folder into the list, wait briefly, then remove it from the list. This forces a re-index.
Issue: USB-powered drives are spontaneously unmounting and re-mounting themselves.
Fix: Try switching cables; try switching to a less troublesome drive; try switching laptops; try switching to Thunderbolt.
Where Else Can I Learn Cool Stuff?
If you had fun reading this guide, I recommend checking out:
Data Management, Backup and Archive for Media Professionals
Marc M. Batschkus, 2014 (free on iTunes!)
Digital Imaging Technician: A Very Practical Guide To On-Set Management Of Digital Media
Robert L Trim, 2013
The Filmmaker’s Guide to Digital Imaging
Blain Brown, 2015
Modern Post: Workflows and Techniques for Digital Filmmakers
Scott Arundale and Tashi Trieu, 2014
Make the Cut: A Guide to Becoming a Successful Assistant Editor in Film and TV
Lori Coleman and Diana Friedberg, 2010
Color Correction Handbook: Professional Techniques for Video and Cinema
Alexis Van Hurkman, 2013
The Avid Assistant Editor’s Handbook
Kyra Coffie, 2011