The way we interact with machines is always changing. As technology evolves, new ways of interacting with computers become available to us, one innovative breakthrough after the next. If we go back 10 years, RIM was just starting to implement phone capabilities into their line of Blackberry mobile devices. Now we have touch screens capable of delivering a full computing experience in the palm of our hands. Voice recognition software is finally becoming mainstream with the introduction of Siri on the iPhone 4S. We are rapidly entering an age in which being tethered to an accessory or peripheral, such as a mouse or keyboard, will be considered an archaic way of getting things done.
Touch screen interfaces are all the rage right now, but the next true evolution in human/computer interaction won't require you to physically touch a thing. You've seen it before in a number of sci-fi films: some guy in a futuristic get up is waving his hands around, barking orders at a computer that seamlessly tracks his movement and executes every command with flawless accuracy. The proper name for this type of computer interaction is called a Natural User Interface (NUI), which describes the ability to issue commands to a device using nothing more than your own natural body movements, gestures, and voice. You'll no longer need to clean those germ-infested keyboards and touch screens or pick up new batteries for your wireless mouse or gaming controller. The possibilities are truly endless, and we're already starting to see deployments of NUI interactivity all over the world. Soon you'll see NUIs in store windows, bus stations, malls, as well as many other places that could benefit from adding natural human interaction to the process of selling and providing information.
One great piece of tech that has ushered in this new wave of creative development is the Microsoft Kinect. The Kinect has introduced itself to the masses at the best possible time. In its short existence, it has proven to be a great option for those looking to spice up the way we work with computers. For starters, the Kinect is a fraction of the cost of most professional 3D depth sensing cameras. A wide assortment of programs allow you to use Kinect 3D depth data with Windows, Linux, and Mac machines. Whether you're an innovative professional interested in bleeding edge tech, or a part time hobbyist with a great new idea, the Kinect is easy to set up, it's cheap, and it's loaded with potential to change the way we interact with machines.
My intention from the start was to write a book that everyone could take something from. Whether you're completely new to the scene or have been tinkering around with the Kinect since its debut, there's something in here for everyone. Some of the hacks range from basic installation of tool sets or programs to detailed code write-ups relating to a wide range of available IDEs such as Processing and openFrameworks.
There are gesture-based solutions that work with a quite a few diverse applications in this book. For music related application, there is information on how to set up Ableton Live with Kinectar. If mocap is your thing, there are tips and tricks for using Blender or Animata with NI mate. HTML5 and JavaScript integration is included using Zigfu's ZDK along with full-on 3D object and scene recreation written in C++ using PCL and OpenCV. If visual effects are your thing, be sure to check out all the diverse Kinect hacks using open source IDEs such as Processing and openFrameworks.
Whether you're looking for some fun projects to work on in your spare time or finding the perfect jump off spot to get started on your next big project, I believe you'll be able to find something in here that will appeal to you. Whatever is your proverbial cup of tea, there's definitely something in here for everyone to enjoy working on.
Writing this book was one of the best experiences of my life, hands down. I had a pretty good idea of what was out there and how to set things up, but what I learned going through the amount of diverse hacks that would eventually compose the content of this book truly made me aware of the great work people are doing with the Kinect. I got to know many of the talented people involved in the scene from around the world and learned an incredible amount in the process.
I started off by writing the guides necessary to get you started on your preferred OS. The great thing about the Kinect scene is that it was born out of the open source community. This ensured that drivers would be released across all platforms. From there, various libraries were released allowing people to start playing around with Kinect data in other open source IDEs such as Processing, openFrameworks, and Cinder. Getting the Kinect up and running and installing the proper libraries for your IDE of choice is the first step. After that, the real fun begins.
After completing the first two chapters, things really started to get interesting. A wide assortment of new tools became available to play around with. For example, working with Chris Vik’s amazing Kinectar app, which allows you to be your very own hands-free composer and musician, and Stephan Howell’s Kinect2Scratch, which uses the Scratch IDE, was an unbelievable experience. I had a blast settings things up and then playing around with the end results. I can honestly tell you that I loved working on this book from beginning to end.
A big huge juicy thank you to everyone at O’Reilly Media for taking a chance on this lowly blogger. I never thought in my wildest dreams that putting a bit of work into a niche scene like Kinect hacking would result in something like this. In my opinion, they took a huge chance with me and I hope I was able to deliver a great product to add to their impeccable track record in the world of technical writing.
Even though O’Reilly Media made this thing you’re holding in your hands a reality, I would have never had the opportunity if it wasn’t for Greg Mackenzie and the Dashhacks network. His passion for hacking and modding electronics eventually produced the best network of hacking sites on the Internet. The journey seemed short in hindsight, but what I got out of it has changed my life forever. Mad props for letting me do my own thing, GregCube.
Last but certainly not least, the amazing folks who took time out of their lives to develop such incredible projects using Kinect’s 3D depth sensing capabilities. You are an inspiration and also responsible for making us think beyond the keyboard and mouse, ushering in a new age of interacting with machines. In no particular order, here is my "I’m not worthy!" list of incredible contributors to both this book and the scene in general. Without the efforts of each and all of you—I can’t complete this sentence because I don’t even want to think about it!
Thanks to Daniel Shiffman, Joshua Blake, Kyle McDonald, Nicolas Burrus, Shawn Wallace, Brian Jepson, Ryan Challinor, Chris Vik, Julius Tuomisto, Peter Nash, RJ Durran, Ning Ma, Kris Temmerman, Mike Newell, Andrew Berg, Felix Endres, Stephen Howell, Jordi Llobet Torrens, Anna Fusté Lleixà, Jeremy Archer, Taylor Veltrop, Takashi Nishibayashi, Daniel Ho, Javier Graciá Carpio, Theodore Blackman, Amir Hirsch, and Stefan Stegmueller.
The following typographical conventions are used in this book:
- Italic
Indicates new terms, URLs, email addresses, filenames, and file extensions.
Constant widthUsed for program listings, as well as within paragraphs to refer to program elements such as variable or function names, databases, data types, environment variables, statements, and keywords.
Constant width boldShows commands or other text that should be typed literally by the user.
Constant width italicShows text that should be replaced with user-supplied values or by values determined by context.
Tip
These lines signify a tip, suggestion, warning, caution, or general note.
This book is here to help you get your job done. In general, you may use the code in this book in your programs and documentation. You do not need to contact us for permission unless you’re reproducing a significant portion of the code. For example, writing a program that uses several chunks of code from this book does not require permission. Selling or distributing a CD-ROM of examples from O’Reilly books does require permission. Answering a question by citing this book and quoting example code does not require permission. Incorporating a significant amount of example code from this book into your product’s documentation does require permission.
We appreciate, but do not require, attribution. An attribution usually includes the title, author, publisher, and ISBN. For example: “Kinect Hacks by Jared St. Jean (O’Reilly). Copyright 2013 Jared St. Jean, 978-1-449-31520-7.”
If you feel your use of code examples falls outside fair use or the permission given above, feel free to contact us at permissions@oreilly.com.
Note
Safari Books Online is an on-demand digital library that lets you easily search over 7,500 technology and creative reference books and videos to find the answers you need quickly.
With a subscription, you can read any page and watch any video from our library online. Read books on your cell phone and mobile devices. Access new titles before they are available for print, and get exclusive access to manuscripts in development and post feedback for the authors. Copy and paste code samples, organize your favorites, download chapters, bookmark key sections, create notes, print out pages, and benefit from tons of other time-saving features.
O’Reilly Media has uploaded this book to the Safari Books Online service. To have full digital access to this book and others on similar topics from O’Reilly and other publishers, sign up for free at http://my.safaribooksonline.com.
Please address comments and questions concerning this book to the publisher:
| O’Reilly Media, Inc. |
| 1005 Gravenstein Highway North |
| Sebastopol, CA 95472 |
| 800-998-9938 (in the United States or Canada) |
| 707-829-0515 (international or local) |
| 707-829-0104 (fax) |
We have a web page for this book, where we list errata, examples, and any additional information. You can access this page at:
| http://oreil.ly/kinect_hacks |
To comment or ask technical questions about this book, send email to:
| bookquestions@oreilly.com |
For more information about our books, courses, conferences, and news, see our website at http://www.oreilly.com.
Find us on Facebook: http://facebook.com/oreilly
Follow us on Twitter: http://twitter.com/oreillymedia
Watch us on YouTube: http://www.youtube.com/oreillymedia
Anna Fusté Lleixà (hi@annafuste.com) is a Multimedia Engineering student at La Salle Barcelona (Ramon Llull University), an Audiovisual Communication Graduate (Pompeu Fabra University), and an Intern at la Salle HCI Department. Find out more at http://www.annafuste.com.
Peter Nash is a creative technologist based in London. He thrives on variety, including taking on hardcore academic challenges (he achieved "top of the year" for researching Nanotechnology in College) and experimenting with interactive installations, programming mobile apps and web, and working with big high street retailers to keep with it. He is currently pioneering a start-up dedicated to making the lives of smartphone app developers and designers easier. Peter contributed Draw in the Air with Processing.
Reach him via LinkedIn, Twitter, or his personal website.
RJ Duran is a Computational Artist & Engineer exploring the inherent properties and aesthetics of biological and emergent pattern formation through Media Arts & Technology. His fascination with natural systems, engineering, traditional art forms such as music and architecture, and philosophy inform and guide his artistic pursuits. His mission as a creator is to understand the "deep complexity" embedded within systems in order to develop engaging, interactive and experiential audio, visual, and physical tools and experiences for education, exploration, and enlightenment.
He holds a BS in Electrical Engineering from Colorado State University, a certificate in Audio Engineering from The Conservatory of Recording Arts & Sciences, and a certificate in Digital Culture & Creative Technology from Boulder Digital Works. He is currently a Graduate Student Researcher in the Media Arts & Technology program at the University of California Santa Barbara. RJ contributed Create a Gravitational Particle Effect. He can be reached at his website or via email at rjduranjr@gmail.com.
Born in China, Ning Ma now lives in Germany and has received his Master’s Degree in Computational Engineering in Faculty Civil and Environmental Engineering from Ruhr-Bochum University. He is currently a PhD candidate.
Apart from his studies, he is very interested in digital graphics. He is self-taught to work with design tools, playing around with images and animations. His recent experiments with Kinect extends the field of his hobby. He loves actually interacting with graphics by programming is fun and exciting. Ning contributed Make Objects Follow a Tracked User’s Hand and Dynamically Change Colors and Background Perspective.
Mike Newell works for Goodby Silverstein & Partners. Mike contributed Create a Live 3D Rendered Effect with Processing.
He can be reached via his personal website, via email at mike@iwearshorts.com, on Twitter at @newshorts, or on Google+.
Stephen Howell is a Computing Lecturer in the Computing Dept. at the Institute of Technology Tallaght, Dublin. He lectures on Kinect software development in modules called Interactive Media Design & Interactive Media Development. Visit his website at http://scratch.saorog.com. Stephen contributed Create a Basic Tennis Game Using Scratch, Build a Defender-Style Shooter Game with Scratch, and Build a Kinect Space Shooter Game in Processing.
Stephen can be contacted by email at stephen.r.howell@gmail.com.
Jordi Llobet (jordi.llobet@me.com) is a Multimedia Engineering student at La Salle Barcelona (Ramon Llull University). He’s also a team member at the Interactive Prototype Design Group (La Salle HCI Department). Find out more at http://www.jordillobet.es. Jordi contributed Build a Processing Fridge Magnet Game.
Jeremy Archer (https://github.com/fatlotus) is a student at the University of Chicago. He works on various open-source projects, primarily for scientific computing, at Chicago’s Computation Institute (http://ci.uchicago.edu/). He is also co-founder of Carbonless Community, a green energy software company based in Chicago. Jeremy contributed Control a Robotic Arm.
Taylor Veltrop grew up near Chicago surrounded by Lego, Meccano, and computers. His first humanoid robot was made from wood (mostly by his father) at the age of 6. He studied Japanese and computer science at the engineering school in the University of Colorado and eventually found himself in Japan chasing robot dreams. There he came across Kinect-based teleoperation while exploring solutions for robot hand-eye coordination and navigation. Taylor currently resides in Paris working with Aldebaran Robotics. Taylor contributed Control a Robot’s Navigation and Use Candescent NUI.
Taylor can be reached via email at taylor@veltrop.com, at his personal website, or on YouTube.
Stefan Stegmueller is a professional Software Architect/Developer from Switzerland (Master of Science in Computer Science). Candescent.ch is his private development project. I runs candescent.ch to publish programs which he writes mainly for himself but could be useful for others too.
One of his current fields of interest is natural user interaction. Stefan believes that after the Command Line Interface (CLI) and the Graphical User Interface (GUI) the Natural User Interface (NUI) is the next big thing in human-machine interaction. Stephan contributed Use Kinect Jesture for Mac.
Stefan can be reached via email at info@candescent.ch, at his website, blog, and on Twitter at @CandescentCH.
Winect is a final year project by Daniel Ho in National University of Singapore, School of Computing. Since graduation, he had continued this project during his free time at http://ixorastudios.com/software/winect/. Daniel contributed Use Winect for Windows.
Kris Temmerman contributed Create a Hairy Effect: Create a Hairy Effect.
Andrew Berg contributed Create Hand Tracking Trails Using Cinder: Create Hand Tracking Trails Using Cinder.
Chris Vik contributed Install Kinectar: Install Kinectar; Map Parameters in Ableton Live with Kinectar: Map Parameters in Ableton Live With Kinectar; Set Up a Drum Kit with Kinectar: Set Up a Drum Kit with Kinectar; and Create a Dubstep Wobble Bassline with Kinectar: Create a Dubstep Wobble Bassline with Kinectar.
Javier Graciá Carpio contributed Use Processing to Create a 3D Scanner with Mesh Viewer: Use Processing to Create a 3D Scanner with Mesh Viewer.
Felix Endres contributed Set Up PCL and OpenCV: Set Up PCL and OpenCV; Display a Colored Point Cloud: Display a Colored Point Cloud; Use Features to Track Camera Image Motion: Use Features to Track Camera Image Motion; Fuse Point Clouds into a Consistent 3D Model: Fuse Point Clouds into a Consistent 3D Model; Add Convenience Functionality to a 3D Model: Add Convenience Functionality to 3D Model; and Next Steps: SLAM, OctoMaps, Surface Reconstruction: Next Steps; SLAM, OctoMaps, Surface Reconstruction.
Takashi Nishibayashi contributed Use Linux Gesture-Based Mouse Control: Use Kinect Justure for Mac.
Get Kinect Hacks now with the O’Reilly learning platform.
O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.
