SIGGRAPH 2010

SIGGRAPH 2010 Dmitry Andreev (AND) [email protected] Real-time Frame Rate Up-conversion for Video Games or how to get from 30 to 60 fps for “free” Agenda

• Stories, Ideas, Possibilities Excitement, Inspiration Better Games Happy Players…

• Technical Details are Simplified– Only current real-time implementation is discussed Real-time DEMO Frame Rate Up-conversion

• Natural Motion of Objects

• In Software– AVISynth + MSU FRC– WinDVD (Trimension)

• In Hardware– Display devices (120Hz HDTVs) Motion Estimation

• Block-matching– Diamond Search

• Optical Flow

• Phase Correlation

• MAP/MRF or take it from the MPEG stream :)AB Motion Vectors Video Frame Interpolation

• Build Inner frames from outer frames– Using Motion Vectors– High-res estimation is required

• Scaling, rotation, transparency, dynamic lighting ???ABA1A2A3 Motion Vectors vs Velocities Estimated vs Rendered Motion Vectors Velocity Buffer No Need for Motion Estimation

• The Source of Motion is Known– Objects– Camera

• Handle Special Cases Differently– Transparency– Shadows, reflections, refractions– Characters ? Running at 30 fps

• Game– Extra 16.6 ms– More advanced rendering techniques

• Deferred / Screen-space, etc…– Less strict production process (easier to make) Running at 30 fps

• Game– Extra 16.6 ms– More advanced rendering techniques

• Deferred / Screen-space, etc…– Less strict production process (easier to make)

• User– Exposed to the same frame Twice Motion Eye-tracking (60 fps) Observed @ 60Hz

• Display A, B, C, ...

• Temporal Functions– Display–Eye 01/60 s Motion Eye-tracking (30 fps) Observed @ 60Hz

• Display A, A, B, B, C, C, ...

• Temporal Functions– Display–Eye0 1/60 s 1/30 s0 1/60 s 1/30 s Motion Blur

• Helps to Remedy “Flickering”, but…

• Happens in the Eye…– Not the camera (virtual)– Should not be used at all

• At eye-trackable speeds Observed @ 60Hz Motion Blur In Games The Ideais …

• Render Velocity BufferP n-½ The Idea is simple …

• Render Velocity Buffer

• Interpolate the Middle FramePn-1PnP n-½ The Idea is very simple …

• Render Velocity Buffer

• Interpolate the Middle Frame

• Present at the Right Moment in TimePn-1Pn VBlankVBlankVBlank 1/60 s1/60 s Ideally (to be “free”)

• Reuse as Much as Possible– Velocity buffers– Previous frames @ lower-res– Any other data available (e.g.

animations)

• No Extra– Latency– Memory / performance hitPn-1Pn Typical Rendering Pipeline

• Depth Buffer ( ~2-3 ms)

• Shadows, Lighting (deferred), SSAO

• Main Color Pass & Alpha Pass

• Post Effects (HDR, DOF, Motion blur, DLAA , etc...)PnP n-½ Modified Rendering Pipeline

• Depth Buffer ( ~2-3 ms)

• Render Velocity/ Interpolate/ Schedule to swap

• Do the Rest of the Current FramePn GPU is working onPn-1 One-frame Based Solution

• Pros– No extra latency– Little extra memory

• Cons– Issues with shadows and reflections

• Manageable– Harder to fix interpolation errors

• But possible ;) INTERPOLATION Very Basic Interpolation•P n-½=f(Pn-1 ,-½·Vn)– Sample half way backwards– Previous framePn-1 Original I Interpolated Very basic interpolation II Very Basic InterpolationIII

• Based on Current VelocityP n-½ =f(Pn-1, -½·Vn )– Sample half way backwards– Previous framePn-1

• Two Types of Issues– Static geometry tails– Dynamic geometry tails Camera Motions (TFU2)I•3rd Person Camera Translation Camera Motions (TFU2)II•3rd Person Camera TranslationRotation Artifact Minimization

• Static Geometry– Velocity buffer filtering

• Dynamic Geometry (Characters)– Render separately, re-render completely or ...

• Use Previous Color & Velocity Frames– To fix-up invalid regions– Two-frame solution works best Two-frame Based Solution

• Pros– Smoother shadows, alpha, reflections– More real data to handle overlaps

• Cons– Extra frame of latency– More memory and slower Modified Velocity Rendering

• Re-render Characters– At interpolated positions

• Use other Channels– To help with interpolation and artifact detection– B – 8bit depth buffer– A – 8bit id masks (e.g.

characters) Velocity Merging

• Very Simple and Efficient•V merged =Vn-1 if (Zn-1 + ∂ ) <Zn elseVn–Z is embedded inV– Works well for background objects– Assuming continuity of motion Original I Interpolated Velocity Merging II Character RemovalI

• Visual System’s Blind Spot Inspired Character RemovalI

• Visual System’s Blind Spot Inspired

• Synthesize the Interior

• Assuming That– Horizontal remains horizontal– Vertical remains vertical– Texture is preserved– Shading changes gradually Character Removal II

• Generate Mask Character Removal II

• Generate Mask

• Leak Neighboring Image Patches Character Removal II

• Generate Mask

• Leak Neighboring Image Patches– Duplicate and offsetUp Character Removal II

• Generate Mask

• Leak Neighboring Image Patches– Duplicate and offsetUp Down Character Removal II

• Generate Mask

• Leak Neighboring Image Patches– Duplicate and offsetUp Down Left Character Removal II

• Generate Mask

• Leak Neighboring Image Patches– Duplicate and offsetUp Down Left Right– Merge all layers together Character Removal III

• Repeat N Times

• Blur the Interior Slightly original mask pass 1 pass 2 pass 3 finalize Character Removal IV Original Synthesized 3 passes @ 640x360 XBox360 ~ 0.4ms PS3 ~ 0.3ms (5 SPUs) Improved Interpolation

• Based onV merged– Sample half way backwards– Previous real framePn-1– Synthesized frameP'n-1

• Based on Masks inVn andP'n-1– SelectP'n-1 , if it had a character whereVn does not– Else selectPn-1 Original I Interpolated Velocity Mergingand Character Removal II Next Original In case of One-frame based Interpolation it is Unknown III Typical Flipping

• Flip Normal Buffer– As soon as VBlank is received– Immediately , if VBlank passed (frame rate drops) wait render wait render wait render Modified Flipping

• Flip Interpolated Frame Based on Time– Normal conditions– Frame rate drops wait renderP Custom flipping on XBox360

• Bypass Direct3D Present with...

• Asynchronous Swaps– Multiple front buffers– SwapCallback, VerticalBlankCallback

• Manual Hardware Manipulation– D1-GRPH_PRIMARY_SURFACE_ADDRESS 0x7FC86110 (undocumented) Real-time Implementation

• One-frame Based

• Interpolation of Dynamic Objects– Character removal

• Performance @ 1280x720– XBox360 ~ 1.5 ms– PS3 (5 SPUs)~ 1.2 ms Motion Blur In TFU2

• Current Solution– Alpha “friendly”, simple edge condition– Better than plain 30 fps– But not as good as 60 fps

• Performance @ 1280x720– XBox360~ 2.2 ± 0.4 ms (5-11 samples)– PS3 (5 SPUs)~ 1.4 ± 0.5 ms (16 samples) Future Work

• Transparency Customizations– Re-render lightsabers, HUD– Multi-layer interpolation

• Up-scale from Lower Frame Rates– E.g.

render environment at 20 or 15 fps

• Interpolate Shadows and Reflections (or blur)

• Predictive Performance Adjustment Acknowledgments Szymon Swistun Ruslan Abdikeev Cory Bloyd Cedrick Collomb Axel Wefers Words of Wisdom Thank You