The Future of Gestures and a Niche For BB10 or QNX

I'm hoping to start a discussion on the future of gestures in mobile computing, and the potential for BB10/QNX to find develop a strong niche.

The potential for simple gestures to interact with a hub workflow is going to be a strong point for BB10 devices. However, much focus with the z10 and x10 is on the relationship between gestures and a touch screen. Has anyone else been following the development of LEAP motion? https://leapmotion.com/

The Leapmotion device is pretty simple -- it is similar to Microsoft's Kinect. It's designed for users to be able to control a screen without physical interaction with a device.. It is much more precise than kinect, and runs off of infra-red -- so it can track what each finger is doing with incredible accuracy. The real innovation seems to be in the algorithms for interpreting gestures. It is incredible really -- and there is a lot of potential. Imagine sitting in front of your screen or a projection and just be able to control it with your fingers -- no keyboard, mouse, or even touch screen.Specialists in all fields will be able to interact with programs without 'switching' to the action of computing. Surgeons will be able manipulate a program without taking their gloves off -- oil workers while doing messy mechanical work -- policemen arresting a violent person -- bratty teenagers in classes who have been told to turn off their devices. Leapmotion costs a mere 70$, and you can plug it into your computer with USB until the next wave will operate wirelessly. Developers are ensuring it will work with all operating systems before it is released -- but it feels like there is a lot of potential in an augmented reality future to wed a device like this to QNX.

Could something like this be integrated into a BB10 device? It would involve including a small infra-red scanner. Imagine if your phone could not only read the gestures that you make on the screen, but also those you make with your hands in the air. A simple voice command could tell your Z10 to switch on IR gestures -- and you just start interacting with your device. With the phone in your pocket, and your hands by your side, you could be 'typing' a message while standing in an elevator, or holding the wheel of a car.

Imagine if when you get to the office, you could just put your z10 down in front of a screen or a projection, and start 'computing' using gestures in the air -- surfing through pages or drafting images.

BB10 is already great -- I'm banking on it being a success.. But I think that if this sort of capability could exist with BB10, it would genuinely give it the edge that it needs to outpace Apple and Android. Very soon, everything is going to be 'smart'. It won't just be our phones, or porche cars. It will be our homes, toasters, elevators -- and gestures are going to be our way of navigating. RIM should be anticipating this. Remember the comment about 'nobody will watch a video on a phone'? Rim has clearly caught up -- and has created a better OS... however this could be the next step in creating a niche.

Nice first post on CrackBerry. Welcome! I love to see conversations like this in our forums.

I've been keeping tabs on LeapMotion since it was first announced. Definitely cool stuff there.

One thing to note too is the ability to do gestures with the built in camera. Technically, you can already do kinect-style gestures on the platform. Short on time to dig it up right now, but there's a cook book app for the PlayBook that allows you to turn the pages just by waving your hand over the device. Perfect for when you're in the kitchen with dirty hands and need to turn the page without mucking up your tablet. Not sure exactly how much or little you can do via the front facing camera, but I know I've seen this at least. Will do a bit more digging into it... been a while since I've thought about that.

Like Kevin said, this is a great first post!
What you are talking about is very interesting, however, I'm not sure if it will come to phones any time soon.
Right now, people still like buttons on phones, if BlackBerry 10 is a success it might help further implement a button free phone environment...
I think that sensor may be the future with TV's and computer's but, a smartphone screen may just be a bit too small for that sort of thing.
I could be mistaken though.

Thanks!

Yeah, I agree the screen on a phone itself would be a little small for some of the more exciting things you can do with a device like Leap. But I was thinking that because of the mobility of the phone, it could act as a controller for all of those devices (TV, Computer) etc. It would just communicate with them like a wireless/bluetooth input device. I'm sure there are already android apps that allow you to control those larger devices through the touch screen.

Very interesting ideas. I would have to wonder about how intuitive the gesturing would be, and how standard they would need to be between the various platforms. Case in point is the QWERTY keyboard, which I had to take classes in in highschool a long time ago to master (not sure if this is still taught?). The keyboard layout already provides a finger jesture matrix, however it's very flat and one dimensional, I still haven't mastered all that it can do. Bringing this concept off of and away from the a keyboard would be a challange, making it easy to learn and master would be a second challenge, or would there be somekind of integration of the two technologies, effectively just replacing the mouse with hand jestures (which I think would be a huge improvement) but still using a keyboard for typing?

MEh too gimmicky the only market i see for this is home fitness and medical check ups MS might have another lucretaive oppertunity if they look into it.That leap stuff wont work on phones its more tablet and tv orientated problem is tv and tablet folk could remove there little gizmo so they should consider software licensing for better future..

Wouldn't have to clean the screen as often......LOL

Seriously though, it makes sense this where we go, coupled with voice recognition.

There are a ton of applications here, from 3D imaging and modeling design to simple kids applications, and everything in between. I have been playing around with NUI Group - Natural User Interface Group this type of natural user interface concept for some time. (this group is not mine, just one i peruse from time to time) I am building (well I have designed, havent had a chance to build yet) a tabletop display used for digital photographs to replace the age old photo albums. Allowing people to really interact with family photos, make prints directly from it, enlarge, "pass" them back and forth, watch videos, etc. There are a multitude of ways to achieve this, just finding the time and materials would be the issue hah. LEAP looks quite interesting, and I very much look forward to how this progresses.

Thanks, Kevin! Quite the honour!

I've seen the cookbook app -- it's pretty good. But like Kinect you kind of have to 'flail' a bit, and it is limited by the camera's point of view. But the advantage to the Leap (I'm sure it runs on I.R.) is that it is truly measuring 3d motion. Even simple gestures -- a casual flick of the finger etc., are picked up and interpreted.

The Leap devices actually have a 3d box in which they sense motion. Based on the prototypes, I'm sure it is something that could eventually fit into a camera. I guessing that it would be possible for something like that to read your gestures, even if it was sitting in your pocket.

Yeah - that sounds like it would be pretty complex. I myself prefer the QWERTY because I just seem to be too clumsy with typing on a touch screen, even after owning one for a year. I can touch type no problem on a larger keyboard, but typing on something so tiny takes all my attention.

It looks to me like you would be able to just hold your hands at your sides, and make the motions of typing, which it would be able to read. You'd be able to effectively 'type' while walking etc, paying attention to other things. You can see in this video that Leap can track all 10 fingers with incredible precision in 3d. As a QWERTY user, I think I'd be able to type on that much easier than a touch screen.

Leap 3D Preview - YouTube

Originally Posted by mw12341

Thanks, Kevin! Quite the honour!

I've seen the cookbook app -- it's pretty good. But like Kinect you kind of have to 'flail' a bit, and it is limited by the camera's point of view. But the advantage to the Leap (I'm sure it runs on I.R.) is that it is truly measuring 3d motion. Even simple gestures -- a casual flick of the finger etc., are picked up and interpreted.

The Leap devices actually have a 3d box in which they sense motion. Based on the prototypes, I'm sure it is something that could eventually fit into a camera. I guessing that it would be possible for something like that to read your gestures, even if it was sitting in your pocket.

IR would be aproblem when walking outside etc. Direct sunlight can play havoc on anything sensing IR. Or how about in a car when you have the heat on (or for that matter any strong heat source). The Leap product is indeed an excellent thing, but needs to be worked on in anything other than a controlled environment.

Originally Posted by mw12341

Thanks!

Yeah, I agree the screen on a phone itself would be a little small for some of the more exciting things you can do with a device like Leap. But I was thinking that because of the mobility of the phone, it could act as a controller for all of those devices (TV, Computer) etc. It would just communicate with them like a wireless/bluetooth input device. I'm sure there are already android apps that allow you to control those larger devices through the touch screen.

Ok, I'm understanding more now.
That would be great, if a phone could be used as a wireless controller!


Sent from my BlackBerry 10 device, in the near future!

Originally Posted by Kevin Michaluk

Short on time to dig it up right now, but there's a cook book app for the PlayBook that allows you to turn the pages just by waving your hand over the device. Perfect for when you're in the kitchen with dirty hands and need to turn the page without mucking up your tablet. Not sure exactly how much or little you can do via the front facing camera, but I know I've seen this at least. Will do a bit more digging into it... been a while since I've thought about that.

That was the brilliant Cooklet app: Buy Cooklet - Download Cooklet - Buy Apps from BlackBerry App World by Greg Trubilowicz

I remember it partly because I judged it with Kevin and others during the first BlackBerry Jam in Orlando, where we unanimously gave it the win in its category for the App Circus. ;-)

Originally Posted by mw12341

Thanks, Kevin! Quite the honour!

I've seen the cookbook app -- it's pretty good. But like Kinect you kind of have to 'flail' a bit, and it is limited by the camera's point of view. But the advantage to the Leap (I'm sure it runs on I.R.) is that it is truly measuring 3d motion. Even simple gestures -- a casual flick of the finger etc., are picked up and interpreted.

The Leap devices actually have a 3d box in which they sense motion. Based on the prototypes, I'm sure it is something that could eventually fit into a camera. I guessing that it would be possible for something like that to read your gestures, even if it was sitting in your pocket.

What is the cookbook app called? I would like to check this thing out! dont have to touch the screen to flip pages sounds fun to play with!! Is it Cooklet or something else?

Whoops, I should have read to the end before asking...someone posted it! it is Cooklet! thanks @Peter9477

Do you think RIM should buy a company like this?

That's the one.