Gil Maman, Dani Rubim
The object of the project is to be the ultimate fruit ninja player with the hand robot on “XBOX Kinect Fruit Ninja”,
or in simple words, play fruit ninja using the robotic arm in the RDB lab using the Kinect device on the XBOX 360.
In this project, we aimed in creating a working proof of concept of the fruit ninja player,
while making a solid workspace and easy to use code for following projects to continue our ninja/play other games with its algorithms, and improving it, to finally be the worlds best fruit ninja.
Our solution and workflow
1. The game is played in classic mode – no bombs are allowed.
Our solution to playing fruit ninja is as follows:
We first connected to our XBOX 360 an easy-cap hauppauge device to capture its view and connected it into our Windows computer, in that computer we have installed easyDevice which turns the easy-cap device video into a available video device in Open CV.
in our Client – the computer which is connected to the EasyCap, we have created an algorithm to detect the fruits with image
processing and created an algorithm to make a good slice of them, with sockets we send each time the best slice to the Linux computer
which is connected to the robotic arm, and it makes the move and asks for a nice slice, and that continues until the game is over, we have made the hand wrist keep rotating, so in the time it goes to slice fruits, it will slice all the fruits surrounding it.
In the end, we have made the kinecket device recognize the robot by dressing it with a jacket and a fake head, it took about 5 seconds each game for the kinecket to recognize the robot after it started moving.
We have first created our fruit detecting algorithm through image processing on static images, and then on videos to see if it
works, then we have created a basic algorithm which slices the fruits, and then we have made it better with real time
support, after that we have worked on the hand movement on the Robot, then worked on getting the video from the XBOX device, and
then connected our detection and slice algorithm on the real video of the game, we connected using sockets our windows
client which gives the slices to our linux computer which controls the arm and tells it where to move.
and in the end calibrated the robot and dressed it with a jacket and a fake head so the kinecket device will recognize it, and made the wrist turn and the real robot movements so they will move accordingly to the coordinates that the computer generates.
reconstructing our project – set up
Devices and softwares used
Windows & Linux computer
Yaskawa Robotic arm
Softwares & libraries
any c++ editor
How everything is connected:
A short clip showing how all the devices are connected:
Running the programs
First open the linux computer handling the arm and start up the server sockets code.
after that you will need to open a game of fruit ninja on the xbox while all the devices are connected as explained above – and the
kinecket device pointing start to the robotic arm, on the windows client computer you will need to open easyDevice and connect to the
easy cap hauppauge device, after doing that start the client program and the hand will start playing.
Please notice to put the current IP of the Linux computer to the Client windows computer so the sockets will work correctly, you can find it with the ifconfig command on the linux bash.
Detecting the Fruits
The algorithm was created based on Research done and error and success.:
decrement from the current image an background image with no fruits (save from the start)
by that you get mostly only the fruits
delete lives and time from the image by changing those pixels to black
so this wont interrupt getting the fruits
gray scale and blur the image
easier to detect area with a lot of pixels this way and contours
do the gradient of x,y and add the weights of it to the picture
this gives more strength to the edges of the fruits
do the algorithm of canny on it,
this helps us recognize the fruits edges
find contours of the image
find smallest circle which fits on all the points of the contours – this circles each fruit
by that we get circles around each fruit
Now with this information, we have center (x,y) and radius, and we can pass it to our best slice algorithm
Best slice algorithm
The algorithm gets all the circles -center(x,y) and radius of current frame:
create all lines from current robot hand position to the circles center
calculate how many circles each line cuts
take the maximum line which cuts the most
update the current hand position the where it needs to cut
now we can pass the coordinate to the hand so it can slice to there.
Future Improvements/Projects suggestions based on the project
Currently, we have transformed the moment from the computer to the real world using a simple algorithm on the movement data, this works fine,
but to make it work better and more accurate we can use Matrix multiplication and transform the moment from the 2d dimension to the 3d dimension very accurately and by that it will slice better.
Other fruit ninja modes:
In this project we have decided to focus on a game with no bombs – “zen mode”, there are a lot of modes available with different fruits and bombs that we do not support and can be done for following projects.
best cut algorithm:
our current cutting algorithm is not optimal, a better cutting algorithm can be made done with mathematical and physics calculations and taking into consideration the current height/speed/angle of the fruits.
transform to other games:
our Image processing algorithm can be taken to a lot of more games where we have moving parts on a static background, so future projects can be done on the kinecket with our algorithm and some adjustments such as “cut the rope”, “angry birds” and etc.
Gil Maman: firstname.lastname@example.org
Dani Rubin: Sbh.email@example.com