raskar Mocap

1.

Mitsubishi Electric Research Laboratories
Special Effects in the Real World
Raskar 2006
Prakash: Lighting-Aware Motion Capture Using
Photosensing Markers and Multiplexed Illuminators
R Raskar, H Nii, B de Decker, Y Hashimoto, J Summet, D
Moore, Y Zhao, J Westhues, P Dietz, M Inami, S Nayar, J
Barnwell, M Noland, P Bekaert, V Branzoi, E Bruns
Mitsubishi Electric Research Labs (MERL)
Cambridge, MA

2.

Mitsubishi Electric Research Laboratories
Special Effects in the Real World
Raskar 2006
Vicon
Motion Capture
Medical Rehabilitation
Athlete Analysis
High-speed
IR Camera
Performance Capture
Biomechanical Analysis

3.

Mitsubishi Electric Research Laboratories
Hidden
Marker Tags
Outdoors
Unique Id
Special Effects in the Real World
Raskar 2006

4. Inverse Optical Mo-Cap

Mitsubishi Electric Research Laboratories
Special Effects in the Real World
Inverse Optical Mo-Cap
Raskar 2006
High Speed Camera
High Speed Projector
Reflecting/Emitting Marker
Photosensing Marker
Only Location
Location, Orientation, Illum

5.

Mitsubishi Electric Research Laboratories
Imperceptible Tags
Special Effects in the Real World
Location
Raskar 2006

6.

Mitsubishi Electric Research Laboratories
Location
Special Effects in the Real World
Orientation
Raskar 2006

7.

Mitsubishi Electric Research Laboratories
3D Overlay
Special Effects in the Real World
Orientation
Raskar 2006

8.

Mitsubishi Electric Research Laboratories
Imperceptible Tags
Special Effects in the Real World
Raskar 2006
Incident Illumination

9. Prakash: Lighting-aware Mo-Cap

Mitsubishi Electric Research Laboratories
Special Effects in the Real World
Raskar 2006
Prakash: Lighting-aware Mo-Cap
• Geometry via Space Labeling
– Binary
• High speed LED projector
• 3D location
– Analog
• Bright Beacons
• Orientation

10. Photosensing Marker Tag

Mitsubishi Electric Research Laboratories
Special Effects in the Real World
Raskar 2006
Photosensing Marker Tag
Scene

11. Labeling Space (Indoor GPS)

Mitsubishi Electric Research Laboratories
Special Effects in the Real World
Raskar 2006
Projector
x=0
x=255
Each location
receives a unique
temporal code
Tag
But 60Hz
video projector
is too slow
Time

12.

Mitsubishi Electric Research Laboratories
Special Effects in the Real World
Binary Gray-codes
Pattern
MSB
Pattern
MSB-1
Pattern
LSB
Raskar 2006

13.

Mitsubishi Electric Research Laboratories
Special Effects in the Real World
Raskar 2006
Binary Gray-codes
Pattern
MSB
0
Pattern
MSB-1
1
Pattern
LSB
1
0
0
X=12
For each tag
a. From projected pattern sequence, decode x coordinate
b. From projected pattern sequence, decode y coordinate
c. Transmit back ( Id, x, y )

14. Fast Pattern Projector?

Mitsubishi Electric Research Laboratories
Special Effects in the Real World
Fast Pattern Projector?
Projector
Pos=0
Tags
Pos=255
60 Hz => 3 location/sec
10,000 Hz <= 500 locations/sec !
Raskar 2006

15. Fast Switching using Non-colocated Emitters for Structured Light

Mitsubishi Electric Research Laboratories
Special Effects in the Real World
Raskar 2006
Fast Switching using
Non-colocated Emitters for Structured Light
Fixed Masks + Blinking LEDs

16. Fast Switching using Non-colocated Emitters for Structured Light

Mitsubishi Electric Research Laboratories
Special Effects in the Real World
Raskar 2006
Fast Switching using
Non-colocated Emitters for Structured Light
Tag
Fixed Masks + Blinking LEDs

17.

Mitsubishi Electric Research Laboratories
Special Effects in the Real World
Raskar 2006

18.

Mitsubishi Electric Research Laboratories
Special Effects in the Real World
Raskar 2006

19. 1D location tracking

Mitsubishi Electric Research Laboratories
Special Effects in the Real World
Raskar 2006
1D location tracking
Projector
IR LED
Scene
GrayCode
Mask
X-Axis

20. Inside of Projector

Mitsubishi Electric Research Laboratories
Special Effects in the Real World
Inside of Projector
Raskar 2006

21.

Mitsubishi Electric Research Laboratories
Special Effects in the Real World
Raskar 2006

22.

Mitsubishi Electric Research Laboratories
2D Location
X data
Special Effects in the Real World
Raskar 2006
3D Location
X2 data
X data
Y data
Y data

23. Prakash: Lighting-aware Mo-Cap

Mitsubishi Electric Research Laboratories
Special Effects in the Real World
Raskar 2006
Prakash: Lighting-aware Mo-Cap
• Geometry via Space Labeling
– Binary
• High speed LED projector
• 3D location
– Analog
• Bright Beacons
• Orientation

24. Analog Space Labeling

Mitsubishi Electric Research Laboratories
Special Effects in the Real World
Raskar 2006
Y projector
X2 Projector
X Projector
IR Beacons
for Orientation

25.

Mitsubishi Electric Research Laboratories
Special Effects in the Real World
Orientation
IR Beacons
Photosensor
On Marker Tag
Raskar 2006

26. Orientation

Mitsubishi Electric Research Laboratories
Special Effects in the Real World
Raskar 2006
Analog Space Labeling
Beacon1
N?
Projectors
Beacon2
Beacon3
Tag

27. Analog Space Labeling

Mitsubishi Electric Research Laboratories
Special Effects in the Real World
Raskar 2006
Analog Space Labeling
Beacon1
N?
d1
V1
Projectors
V2
Beacon2
d2
Tag
Bi
I Vi N 2
di
Beacon3
Cosine fall-off

28. Analog Space Labeling

Mitsubishi Electric Research Laboratories
Special Effects in the Real World
Incident Illumination Color
On-set MoCap:
Location + Orientation + Incident Illumination
Raskar 2006

29.

Mitsubishi Electric Research Laboratories
Special Effects in the Real World
Limitations
• Occlusions and Accuracy
– Multiple surround transmitters
• Strong Ambient Light
– High frequency (455Khz) Modulation
• Inter-reflections
– Binary optical communication
• Wires on Tags
– Batteries + cables
– Limited Wireless Bandwidth
• Compression on aggregate data
• Very Fast Motion
– Simultaneity assumption
Raskar 2006

30. Limitations

Mitsubishi Electric Research Laboratories
Special Effects in the Real World
Tracking Specular Object
Tracking behind Coca Cola
Raskar 2006

31.

Mitsubishi Electric Research Laboratories
Special Effects in the Real World
System Specifications
• Location tracking in 2D: 500 Hz
• Location+Orientation+Illum: 124 Hz
• Location:
– FoV: 27 degrees
– Angular resolution 0.1 degrees
– Accuracy ~5mm at 3 meters
• Orientation:
– Resolution ~1 degree at 3 meters
Raskar 2006

32. System Specifications

Mitsubishi Electric Research Laboratories
Special Effects in the Real World
• Emerging Technology Demo
–
–
–
–
200,000 Hz Multi-LED Projector
Location precision 100 micrometer
Camera tracking with imperceptible markers
All week
• Sketch (Later today)
– More details
– Rendering techniques
– Tuesday 3:30pm, ‘Is it real’, Room 1AB
Raskar 2006

33.

Mitsubishi Electric Research Laboratories
Special Effects in the Real World
Raskar 2006
Prakash: Lighting Aware Optical Mo-Cap
Device
Projector + Photosensor
Camera + Markers
Params
Location, Orientation, Illum
Location
Settings
Natural Light +
Hidden Tags
Controlled Light +
Visible, High contrast Markers
# of Tags
Unlimited + Unique Id
Limited + MarkerSwap
Speed
Unlimited with Optical
comm comps
Limited with camera fps
Low
Cost
Open-loop projectors
Current: Projector/Tag=$100
High
High bandwidth camera
Current Camera: $10K

34. Prakash: Lighting Aware Optical Mo-Cap

Mitsubishi Electric Research Laboratories
Special Effects in the Real World
END
Raskar 2006

35. END

Mitsubishi Electric Research Laboratories
Special Effects in the Real World
Raskar 2006
Related Work
• Camera Based systems
[ViconPeak 2006; Optotrak 2007;
PhaseSpace2005]
– Expensive high speed cameras
• Systems using Photosensors
[Indoor GPS 2004; HiBall 1997]
– Low framerate or challenging setup
• Magnetic/Acoustic Based Systems
[F.H. Raab et al. 1979]
– Interference, drift
• Inertial Tracking (Gyro / Compass)
– Drift
[G. Welch 1995]

36. Related Work

Inverse
Optical
Mo-Cap
Mitsubishi Electric Research Laboratories
Special Effects in the Real World
Device
High Speed Projector +
Photosensing Markers
Params
Location, Orientation, Illum
Natural Settings
Ambient Light
Settings
Outdoors, Stage lighting
Imperceptible tags
Raskar 2006
High Speed Camera +
Reflecting/Emitting Markers
Location
Controlled Lighting
Visible,
High contrast Markers
Hidden under wardrobe
Unlimited
#of Tags
Speed
Space Labeling
Unique Id
Virtually unlimited
Optical comm comps
Low
Cost
Open-loop projectors
Current: Projector/Tag=$100
Limited
No Unique Id
Marker swapping
Limited
Special high fps camera
High
High bandwidth camera
Current Camera: $10K

37. Inverse Optical Mo-Cap

Mitsubishi Electric Research Laboratories
Special Effects in the Real World
Raskar 2006
Inside of Projector
Focusing Optics
Condensing Optics
Light Source
Gray code Slide
The Gray code pattern

38. Inside of Projector

Mitsubishi Electric Research Laboratories
Special Effects in the Real World
Raskar 2006

39.

Mitsubishi Electric Research Laboratories
Special Effects in the Real World
Raskar 2006
Optical Mo-Cap
• High-speed Cameras
– Controlled Settings
• Visible retroreflective markers/LEDs
• High contrast clothing
• Controlled lighting
– No unique Id
• Marker-swapping/reacquisition issues
• Cleanup software
– Expensive
• High bandwidth
• Limited by camera frame rate
• Non-optical Mo-Cap
• Drift/Global distortions
• Difficult for video overlay
1M at 500fps

40. Optical Mo-Cap

Mitsubishi Electric Research Laboratories
Special Effects in the Real World
Raskar 2006
Labeling Space
LED
Optics
GrayCode Mask
Screen
pos=0
1 LED for 1 Bit pattern
pos=15