You are previewing Start Here!™ Learn Microsoft® Kinect API.
O'Reilly logo
Start Here!™ Learn Microsoft® Kinect API

Book Description

Ready to learn Kinect programming? Start Here!

Learn the fundamentals of programming with the Kinect API—and begin building apps that use motion tracking, voice recognition, and more. If you have experience programming with C#—simply start here! This book introduces must-know concepts and techniques through easy-to-follow explanations, examples, and exercises.

Here’s where you start learning Kinect

  • Build an application to display Kinect video on your PC

  • Have Kinect take photographs when it detects movement

  • Draw on a computer screen by moving your finger in the air

  • Track your body gestures and use them to control a program

  • Make a program that understands your speech and talks back to you

  • Play a part in your own augmented reality game

  • Create an "air piano" using Kinect with a MIDI device

  • Table of Contents

    1. Start Here!™ Learn Microsoft® Kinect API
    2. Dedication
    3. A Note Regarding Supplemental Files
    4. Introduction
      1. Who Should Read This Book
        1. Assumptions
      2. Who Should Not Read This Book
      3. Organization of This Book
      4. Conventions and Features in This Book
      5. System Requirements
      6. Code Samples
        1. Installing the Code Samples
        2. Using the Code Samples
      7. Acknowledgments
      8. Errata and Book Support
      9. We Want to Hear from You
      10. Stay in Touch
    5. I. Getting Started
      1. 1. An Introduction to Kinect
        1. The Kinect Sensor
          1. Getting Inside a Kinect Sensor
            1. The Depth Sensor
            2. The Kinect Microphones
        2. Recognizing People with Kinect
        3. Programming the Kinect
        4. Kinect for Xbox and Kinect for Windows
        5. Summary
      2. 2. Getting Started with Kinect
        1. Kinect for Windows SDK Prerequisites
          1. Kinect Device
          2. Visual Studio
          3. DirectX Studio
        2. Installing the Kinect for Windows SDK
          1. Installing the Kinect SDK
            1. Installing the Kinect SDK
              1. Installing the Kinect SDK
        3. Connecting the Kinect Sensor Bar
          1. Powering the Kinect Sensor
          2. Installing the Kinect Sensor USB Drivers
        4. Testing the Kinect Sensor Bar
          1. The Kinect SDK Sample Browser
        5. Troubleshooting Your Kinect Installation
          1. Remove Old SDK Installations
          2. Ensure That Visual Studio 2010 Is Installed but Not Running During Installation
          3. Ensure That There Are No Windows Updates in Progress
          4. Ensure That the Kinect Is Powered Correctly
          5. Remove Any Old USB Drivers
        6. Summary
      3. 3. Writing Software for Kinect
        1. Making a Kinect Video Camera
          1. Creating a New Visual Studio Project for Kinect
            1. Creating a new Kinect Program
              1. Creating a new Kinect Program
          2. Getting the Kinect Sensor Working
            1. Using the KinectSensor Class to Control the Kinect Sensor
              1. Using the KinectSensor Class to Control the Kinect Sensor
          3. Displaying a Video Frame
            1. Creating a WPF Image Display Element for the Kinect Camera
              1. Adding an Image Display Element to the WPF Window
            2. Displaying the Kinect Camera Picture
              1. Obtaining the Kinect Color Camera Video Data
            3. Testing the Kinect Camera
          4. Adding Error Handling
            1. Detecting the Presence of a Kinect
            2. Handling Setup Errors
        2. Summary
    6. II. Using the Kinect Sensor
      1. 4. Your First Kinect Application—Video Snapshots
        1. Image Storage in Computers
          1. Getting the Kinect Image Data onto the Screen
          2. Controlling the Color of the Pixels
        2. Creating a Color Adjustment Program
          1. Improving the Speed by Writing Unsafe Code
            1. Managed Code and Unsafe Code
            2. References and Pointers
            3. Pointers and Fixed Memory Locations
            4. Using 32-Bit Integer Pointers
          2. Saving the Image to a File
          3. Improving Video Quality
        3. Improving Performance by Waiting for Each Kinect Frame
          1. Creating a Video Display Thread
          2. Updating the Image from a Different Thread
          3. Stopping the Background Thread
        4. Summary
      2. 5. Moving Pictures
        1. Detecting Movement in Video Images
          1. Storing a Video Image in Program Memory
          2. Detecting Changes in Video Images
            1. Dealing with Noise
            2. Using Change Thresholds to Filter Out Noise
          3. Sounding the Alarm
            1. Adding Sound Playback to a Project
              1. Adding Sound Playback to a Project
          4. A Complete Alarm Program
          5. Switching to Black and White
            1. Triggering Pictures with Motion Capture
            2. Capturing Multiple Frames
        2. Summary
      3. 6. Fun with the Depth Sensor
        1. Visualizing Kinect Depth Information
          1. The Kinect Depth Sensor
          2. Obtaining Depth Information from the Sensor
            1. Depth Data Values
          3. Visualizing Depth Information
        2. Using the Depth Information to Detect Intruders
          1. Using the Depth and Video Sensors at the Same Time
        3. Drawing in the Air
        4. Detecting Objects
          1. Counting Depth Values
          2. Making You into the Controller
          3. Using the Kinect Sensor with an XNA Game
            1. Getting the Control Value from the Kinect
            2. Drawing the Kinect Depth Image in XNA
        5. Summary
      4. 7. Fun with the Sound Sensor
        1. Capturing Sound Using Kinect
          1. Sound and Computers
          2. Receiving Sound Signals from Kinect
          3. Playing Sound Using XNA
            1. Stopping the Program
          4. Sound Signals and Latency
        2. Visualizing a Sound Signal in XNA
        3. Storing Sound Data in a File and Replaying It
          1. Creating a WAV File
          2. Playing a Recorded Sound
            1. Triggering the Sound Playback
            2. Sound Playback Management
        4. Summary
    7. III. Creating Advanced User Interfaces
      1. 8. Body Tracking with Kinect
        1. Kinect Body Tracking
          1. Kinect Skeleton Information
            1. Kinect Joint Positions
        2. A Head Tracking Program
          1. The Joints Collection and C# Dictionaries
          2. Using Format Strings to Build a Message
          3. Skeleton Information Quality
          4. Joint Tracking State
        3. Drawing a Skeleton
          1. Drawing Lines in WPF
          2. Converting Joint Positions to Image Coordinates
          3. Clearing the Canvas
            1. Turning Yourself into a Paintbrush
          4. Drawing a Complete Skeleton
        4. Detecting Gestures
          1. Calculating the Distance Between Two Points in Space
          2. Using a Gesture to Trigger an Action
          3. Biometric Recognition with Kinect
        5. Creating a “Kiss-Detecting” Program
          1. Finding Two Skeletons That Are Being Tracked
        6. Summary
      2. 9. Voice Control with Kinect
        1. Using the Microsoft Speech Platform
          1. Testing Voice Recognition
          2. Creating a Program That Recognizes Color Names
          3. Adding the Speech Platform SDK Assemblies to a Project
          4. Creating a Speech Recognition Engine
          5. Building the Commands
          6. Creating a Grammar
          7. Getting Audio into the Speech Recognizer
          8. Responding to Recognized Words
        2. Creating a Voice-Controlled Painting Program
          1. Speech Commands
          2. Drawing a Skeleton Cursor
          3. Drawing Using the Artist’s Hand
          4. Saving the Drawing Canvas to a File
            1. Rendering a Canvas to a Bitmap
            2. Saving a Bitmap into a Named File
          5. Tidying Up When the Program Ends
          6. Improving the Drawing Program
        3. Adding Speech Output to Programs
        4. Feedback Problems
        5. Summary
      3. 10. Augmented Reality with Kinect
        1. An Augmented-Reality Game
          1. Creating Sprites
            1. Creating Sprite Graphics
            2. Adding Images to a Project
            3. Drawing Sprite Images
            4. Setting the Sprite Position
            5. Making a Sprite Move
            6. Putting the Bug Back on the Top
          2. Creating Augmented Reality
            1. Displaying Computer Graphics on Top of the Video Image
            2. Adding a Mallet for the Player
            3. Matching Together Screen and Depth Coordinates
            4. Drawing the Mallet
            5. Splatting Bugs
          3. Isolating the Player Image from the Background
            1. Using a Display Mask
            2. Finding the Player Pixels
            3. Using Depth Information to Make a Mask
            4. Drawing the Mask Image
            5. Setting up the Game Image
          4. Putting the Whole Game Together
          5. The Kinect Manager Class
            1. Creating a KinectManager Instance
            2. Using Kinect Data
            3. Displaying the Kinect Status
            4. Polling the Sensor
            5. Starting and Stopping the KinectManager Sensor
            6. Using the KinectManager Class in Your Programs
        2. Improving the Game
        3. Summary
    8. VI. Kinect in the Real World
      1. 11. Real-World Control with Kinect
        1. Controlling MIDI Devices with Kinect
          1. The MIDI Protocol
          2. Creating a Class to Manage a MIDI Connection
            1. Using MIDI in Windows PC Programs
          3. Constructing a MIDI Connection Class
          4. Creating a MIDIControl Instance
          5. Creating MIDI Messages
            1. The MIDI Command Byte
            2. The Note Value
            3. The Velocity Value
          6. Sending MIDI Messages
            1. Playing MIDI Notes
            2. Releasing MIDI Notes
            3. Creating the MIDI Connection
            4. Sending Note Messages
          7. Making a Multi-Note Piano
            1. Creating the Keyboard Display Elements
            2. Responding to Keyboard Events
          8. Playing a Proper Scale
          9. Creating a Human MIDI Keyboard
            1. Creating a List of MIDI Note Keys
            2. Controlling Note Playback
            3. Using the Player Position to Control Notes
          10. Developing the MIDI Program
        2. Using the Kinect with a Serial Port
          1. Serial Ports and Devices
            1. Serial Ports and Devices
            2. The Robot Slave
          2. Linking a Kinect Program to a Serial Port
            1. Creating a Serial Port Connection
            2. Sending Messages to the Robot Using the Serial Port
            3. Creating Command Touch Areas
            4. Receiving Messages from the Robot Using the Serial Port
        3. Summary
      2. 12. Taking Kinect Further
        1. Adjusting the Sensor Angle
        2. Using Kinect to Track Multiple People
          1. Identifying Particular People in a Scene
          2. Combining Skeleton and Person Depth Information
        3. Sound Location with the Kinect Microphone Array
        4. Using Kinect with the Microsoft Robotics Development Studio
          1. Mobile Autonomous Reference Using Kinect
          2. Emulating a Robot Environment
          3. Robots and Kinect in the Future
        5. Taking Kinect Further
          1. Mount the Sensor in Different Orientations
          2. Use Multiple Sensors
          3. Move the Sensor Around
          4. Use Skeleton Tracking to Measure Things
          5. Investigate TransformSmoothParameters
          6. Use Voice Response to Do Anything
          7. Have Fun Playing with Video
          8. Make More of MIDI
        6. Good Luck and Have Fun!
        7. Summary
    9. Index
    10. About the Author
    11. Copyright