Similar presentations:
Challenges of digital media preservation
1. Challenges of Digital Media Preservation
Karen Cariani, DirectorMedia Library and Archives
Dave MacCarn, Chief Technologist
2. Who we are: WGBH Media Library and Archives
23. Transition challenges (Analog to Digital)
Preservation needs are morecomplicated
— New and changing content formats
— Network connections
— Software
— Storage media
— Hardware
Access expectations challenging
— Faster access
— Anywhere, anytime
3
4. Content formats
4H D Acq u i sit ion Cod e cs
V id e o Sa m p le
Au d io
1920
1080p
1920
1280
960
1280
M bps
1080i
1440
720p
Sa m p le
√
√
√
√
3:1:1
4:2:2
4:2:2
4:2:2
4:2:2
4:2:2
4:2:2
4:2:2
√
RGB or 4:2:2
√
4:2:0
4:2:0
4:2:2
4:2:0
4:2:0
4:2:2
4:2:2
2
4
8
2
2
4
2
MP1
16
24
16
16
16
16
4:2:2
4
16
4:2:2*
4:2:0
4:2:0
4:2:0
4:2:0**
2
2
2
2
2
AC3
AC3
AC3
16
16
1 0 Bit
#
Bit s
√
√
√
4
8
8
8
4
4
8
8
20
16
16
16
16
16
24
24
12
4
24
Co n t a in e r
D CT
[1]
HDCam
DVCProHD@24p
DVCProHD-720p
DVCProHD-1080i
Avid DNxHD
Avid DNxHD
Apple ProRes
Apple ProRes HQ
140
40
100
100
145
220
145
220
√
√
√
√
√
√
√
√
√
√
√
√
W a v e le t
Red
224-336
√
M PEG2 Lo n g GOP
HDV
XDCamHD
XDCam422
XDCamEX
XDCamEX
GFCam
Canon C300
25
18-35
50
25
35
50
50
√
√
√
√
√
√
√
100
√
√
√
√
√
M PEG2 I - f r a m e
GFCam
H .2 6 4
AVCHD PS
AVCHD
AVCHD@24
Canon 5DMKII
Nikon D800
28
24
24
38
24
H.2 6 4 I - fram e
AVCIntra
AVCIntra
50
100
√
√
√
√
√
√
√
√
√
√
√
√
√
440
√
√
880
√
√
√
Tape
DV-AVI, DV-DIF, MXF, QuickTime & Tape
DV-AVI, DV-DIF, MXF, QuickTime & Tape
DV-AVI, DV-DIF, MXF, QuickTime & Tape
MXF & QuickTime
MXF & QuickTime
QuickTime
QuickTime
REDCODE & QuickTime
M2T, MXF, QuickTime & Tape
DV-AVI, MP4, MXF & QuickTime
DV-AVI, MP4, MXF & QuickTime
DV-AVI, MP4, MXF & QuickTime
DV-AVI, MP4, MXF & QuickTime
MXF
MXF
MXF
MTS, MP4
MTS, MP4 & QuickTime
MTS, MP4 & QuickTime
QuickTime
QuickTime
4:2:0
4:2:2
√
√
2
2
16
16
MXF
4:2:2/4:4:4
√
12
24
DPX, Tape
4:2:2/4:4:4
√
12
24
DPX, Tape
MXF
M PEG4 St u d io Pr o f ile
[2]
HDCamSR
[2]
HDCamSR-HQ
√
*Sony FS100 HDMI output
** 4:2:2 HDMI output
[1] Tape format for comparison
[2] Tape with DPX file out
D. MacCarn, WGBH
5. Storage and retrieval
How do we:Capture the audio and video generated by myriad cameras
Store the project information to allow potential re-edit
Store files with rich, meaningful metadata
Store born-digital materials
Display and retrieve born-digital materials
5
6. Access: Organizational Issues
MetadataDescriptive metadata
— Need description for video to be useful, findable
— How to capture that
— How to make sure it is linked to video files
6
7. Folder Structure
7Create folders by card
— Assign unique number
— Continue numbers
— Add description
— Place ENTIRE card
contents into this
folder!!
8. Original footage
© 2011 WGBH8
9. Proposed tapeless workflow
Create a mapping document between filemaker andDAM
Used to generate an xml stylesheet
Video is ingested simultaneously with the metadata
from filemaker using the xml stylesheet
Technical metadata is ingested simultaneously with the
video and production data using the xml generated by
the source digital files
9
10. Challenges - again
Access issues— File size
— Formats – to playback
— Useable — Search/findable
Metadata
Organize files
Preservation issues
— Copies
— Formats – for migration
— Being able to play again later
— Speed of access (big file size) – to use/process
— Migration ease
10
11. Software /Network
File management— Where are the files?
Needed for access to files
— Large preservation files
— Smaller access, proxy files
Network speed
— Larger files, need faster
network to meet speed
expectations
11
12. Issues with current file mgmt systems/software
Preservation not a priorityInterface issues
— Access vs. Preservation
IT relationship
— Tech support
— Vendor reliance issues
— Need library based system for Archivist needs rather than traditional
IT company needs
Expense
— License cost
— Development
— Customizations
12
13. Access
13Can find
Can view
Can select
Can get out of
system
Can reuse in
editing system
14. Preservation Needs
Multiple CopiesValidity
Bit quality checks
Long lasting storage
Regular migration
Persistence
14
15. Challenges of preservation and access
For preservation— Want to capture as close to original as possible
— Originals may be many different formats
— Will need to make sure you can export and use different formats in
future
— File format issues
— Fixity check big files
For access
— Want one consistent format for playback/access
— Needs to be easy to migrate, use
15
16. What makes video different?
Preservation files are large— Uncompressed
— Slow to move around
Need proxy files for viewing
— Smaller size for quick transport
over network
Complicated formats
— Not just one file type
— Codecs, wrappers, frame speed,
etc
16
17.
Technology Mix:17
18. Hydra project
Combine preservation system with access systemBetter interface
Flexible design
Easy to evolve
18
19. Insert graphic
Blacklight Hydra headsHydra mgmt layer
Fedora repository
HSM storage system
19
20. Fundamental Assumption #1
No single system can provide the full range ofrepository-based solutions for a given institution’s needs,
…yet sustainable solutions require a
common repository infrastructure.
20
21. Fundamental Assumption #2
No single institution can resource thedevelopment of a full range of solutions on its
own,
—…yet each needs the flexibility to tailor
solutions to local demands and workflows.
21
22. Hydra Philosophy -- Community
• An open architecture, with many contributors to acommon core
• Collaboratively built “solution bundles” that can be
adapted and modified to suit local needs
• A community of developers and adopters extending
and enhancing the core
• “If you want to go fast, go alone. If you want to go far,
go together.”
22
23.
CRUD in RepositoriesCreate/Submit/Edit
(CUD)
Search/View
(R)
Repository/
Persistent Storage
24.
Major Hydra Componentshydra-head
Rails Plugin
(CUD)
Blacklight
(Read
(R)
Only)
Solrizer
Fedora
Solr
25. Hardware/Storage media: HSM
Access— Online
XX bytes Spinning disk
— Offline
— Nearline
Preservation (offline)
— Robotic tape library system
— LT04 data tapes
— 2 copies
— One stored off site
Migration needs 3-5 years
— Both tape migration to newer formats
— Technology migration
26. New Storage Types and Costs
Need hierarchical storage (HSM)— Video files are large
— Spinning disks are expensive
— Tape can help save cost
— Tape copies/migration can be automated
26
27. New Storage Types and Costs
But HSM has licensing issues— Some systems cost by gigabyte managed
— Need Open source alternative
27
28. Q & A
Q&AKaren: [email protected]
Dave: [email protected]
28