Kivy - Gesture

The Kivy framework is capable of recording and identifying the gestures. Gesture is a sequence of touches generated by mouse pointer or fingers on a multitouch device. The kivy.gesture module defines Gesture class whose object is obtained by the (x,y) coordinated of the successive touch events captured on Kivy canvas. Gesture in Kicy is a python implementation of a gesture recognition algorithm by Oleg Dopertchouk.

The same module also has GestureDatabase class. You can store one more Gesture objects in the gesture database, and find out if a certain gesture matches with any of the gestures already stored in the database.

To create a Gesture object, you need a list of x,y coordinates. For example −

from kivy.gesture import Gesture
g = Gesture()
g.add_stroke(point_list=[(1,1), (3,4), (2,1)])
g.normalize()

The add_stroke() method constructs a gesture object from the pairs of coordinates. The normalize() method is needed to run the gesture normalization algorithm and calculates the dot product with self.

Such Gesture objects are stored in GestureDatabase.

from kivy.gesture import Gesture, GestureDatabase

# Create a gesture
gdb = GestureDatabase()
gdb.add_gesture(g)

Against the objects stored in this database, you can compare a certain other object and find if any of the gestures in the database match.

g2 = Gesture()
# ...
gdb.find(g2)

The kivy.gesture module defines following methods in Gesture class −

  • add_stroke() − constructs a stroke from a list of touch points to the gesture and returns the Stroke instance.

  • normalize() − Runs the gesture normalization algorithm and calculates the dot product with self.

  • get_score() − When a gesture is matched with another, this method returns the matching score.

The GestureDatabase class has following important methods −

  • add_gesture() − Add a new gesture to the database.

  • find() − Find a matching gesture in the database. You can define the precision of find with min_score parameter. It should be between 0 to 1.

  • gesture_to_str(gesture) − Convert a gesture into a unique string.

  • str_to_gesture(data) − Convert a unique string to a gesture.

Example

We define the touch_down(), touch_move() and touch_up() handlers to capture the touch points and draw a pattern from them. All the points with their (touch.x and touch.y) coordinates are collected in a List.

When the Add button is pressed, the points List is used to construct a Gesture. The gesture_to_string() method returns a binary string.

if instance.text=='Add':
   g = Gesture()
   g.add_stroke(point_list=Drawgesture.points)
   g.normalize()
   print (self.d.gdb.gesture_to_str(g))
   self.d.gdb.add_gesture(g)
   print (str(g))

An example of the gesture string might look like this −

b'eNprYJmayc4ABj082ZlllXrpqcUlpUWpU3rY3aGsyVM0G6fUTtHoYS3PTCnJmOLuYO9kuU766IwetozUzPSMEqCIC9NEhiUOGj38UO3xBUX5KaXJICmhWZ/F3Pse9LAXlxTlZ6cWT4mdksHQwws1PRgsiLCDrSA/M68EpEgDqIoHqioAJIhQxFgxxX3/LdkuHrnEhh7Gyinu9g9vmvlOTnlRmpQhCFGTIQJXkSHqbn9/U85stZMXcMrfxiZ/TfZI/b2QH8TIXydH/pLsv8/zPDJA8pfJkT9jU3RuT/kBYuTPp4ACaAGq/AmbtU412Qo45Q/YKmn+CRIAyR+nUP4wWD4BVX5DtZ7Sj8IHIPltJ4EeUHdAlY9n/VPH/4ABJL92MtAAvwaS5O3n8Z6ZJZ8Gkt9fDLK/hwGn/CJQ8G1E078eZP5TB5D8RlDyunEAp/xOkPxNNPO3N3WGd3CD/Lf/AND4TTlo5u9vEingUAHLnwDLo4aP/eED54+4yH3AKX/8wNSAFu0JIPkzYHnU8Lc/fSDqzhELUPzuvwBynpkBqvz5AwqZLC4LQPJXwPKo4W9/8f6nX4s0iJK/hk3+6v0dbY9MNUDyNyiUvwNzf2oPT3FyUWpqHqKccHdIcNSwvsgQ4+5QGrZn4XqNnLYpyGJOuwTWtWijiultr197/w2qGNum2DXTs8FiE3XfGfUrYRcrubfWerXfa7DYQ+MFU2RfAsW2rZBcxQZWl2hoGfR1zXocYn2Lvq/Y+wosFmmjo1YijCq20vFeB9NNoFja3KvLS7NQxYJmuyy7qAkWu+iyfccpW6CY3YzNy3Qgen+6T3g5cQFQTGua0tKOVSCxJE9fZ2+FdKCY2OSJS55kgsUKA2Sqn59ydyh+15e/ePZLVLFb3fcWfV8JFpsJcrIuUOxYp++i4ExUsU1toIAGix0MPXe3bCJQbF6L9kKuF2CxlxEr+Gy/AMXK6jnnH8oAiSULRjfas4ajilnGReWf2Q0US6qpmC+nDhZLTAQGqhxQzK/y+bzKF6hiVuVhc6+uAIt1pvBcjG4EiqmVHJ1rmA4W25j2jEnpKQ4xoSKTOb3qKGJF/4BB8OI5SCyFMWdG8sbVOMRe5QrNdlmOKnYtq3HWArAdKZr5hVMq+ZHEUkuTEns4S/JzUosS85JTgTXUzpkgMKs0SQ8Ayq8zuw=='

You can add as many gestures as you want.

Kivy Gesture

Then, draw a pattern on the canvas and try to find out if any of the gestures matches with this one. The find() method does this work when the Find button is pressed.

if instance.text=='Find':
   g=Gesture()
   g.add_stroke(point_list=Drawgesture.points)
   g.normalize()
   g1=self.d.gdb.find(g, 0.65)
   print (g1)

Here is the complete code for gesture recognition exercise −

from kivy.app import App
from kivy.graphics import *
from kivy.uix.floatlayout import FloatLayout
from kivy.gesture import Gesture, GestureDatabase
from kivy.uix.button import Button
from kivy.uix.widget import Widget
from random import random
from kivy.core.window import Window

Window.size = (720, 400)

class Drawgesture(Widget):
   points = []
   
   def __init__(self, *args, **kwargs):
      super(Drawgesture, self).__init__()
      self.gdb = GestureDatabase()

   def on_touch_down(self, touch):
      with self.canvas:
         self.col = (random(), random(), random())
         Color(self.col)
         touch.ud["line"] = Line(points=(touch.x, touch.y), width=3)
         Drawgesture.points.append((touch.x, touch.y))

   def on_touch_move(self, touch):
      with self.canvas:
         Color(self.col)
         touch.ud["line"].points += (touch.x, touch.y)
         Drawgesture.points.append((touch.x, touch.y))
         
   def on_touch_up(self, touch):
      print('touch up')

class gestureApp(App):
   def pressed(self, instance):
      if instance.text == 'Add':
         g = Gesture()
         g.add_stroke(point_list=Drawgesture.points)
         g.normalize()
         print(self.d.gdb.gesture_to_str(g))
         self.d.gdb.add_gesture(g)
         print(str(g))
         
      if instance.text == 'Find':
         g = Gesture()
         g.add_stroke(point_list=Drawgesture.points)
         g.normalize()
         g1 = self.d.gdb.find(g, 0.65)
         print(g1)

      if instance.text == 'Clear':
         self.d.canvas.clear()
         
   def build(self):
      f = FloatLayout()
      self.d = Drawgesture()
      f.add_widget(self.d)
      b1 = Button(
         text='Add', pos_hint={'x': 0, 'y': 0},
         size_hint=(None, None)
      )
      f.add_widget(b1)
      b2 = Button(
         text='Find', pos_hint={'x': .2, 'y': 0}, 
         size_hint=(None, None)
      )
      f.add_widget(b2)
      b3 = Button(
         text='Clear', pos_hint={'x': .4, 'y': 0},
         size_hint=(None, None)
      )
      f.add_widget(b3)
      b1.bind(on_press=self.pressed)
      b2.bind(on_press=self.pressed)
      b3.bind(on_press=self.pressed)
      return f
      
gestureApp().run()

Output

If the match store is greater than or equal to the min_score parameter, you get the following result −

(0.7093289348205829, <kivy.gesture.Gesture object at 0x000001B817C70490>)
Advertisements