筆跡入力用新規デバイスの開発 Novel Device for Inputting Handwriting Trajectory 佐藤 康弘* 新行内 充** 古田 俊之* Yasuhiro SATO Mitsuru SHINGYOUUCHI Toshiyuki FURUTA 要 別府 智彦* Tomohiko BEPPU 旨 図形や記号，文字をコンピュータへ入力するためのペン型入力装置を開発した．ペンに内蔵し た複数の慣性力検出センサにより筆記面上での2次元の筆記軌跡を再現する．センサとしては3個 の加速度センサと3個のジャイロを用いた．ペン上の各軸方向の加速度と各軸周りの角速度が検 出でき，これら6自由度の物理量よりペン先の筆記軌跡を計算できる．問題となる積分誤差は， 速度の次元で補正処理するアルゴリズムで低減できることを見出した．実際のシステムを作製し， 評価することで筆記入力装置としての可能性を示すことができた． ABSTRACT A pen-shaped input apparatus for inputting drawings, symbols, characters into a data processing device, such as a computer, is developed. It is able to trace a handwritten two-dimensional trajectory using built-in inertial sensors. As the sensor, it has three accelerometers and three gyroscopes. By detecting accelerations about three axes and angular rates around them as six degrees of freedom, it can represent the two-dimensional trajectory of the pen tip after additional calculation. A velocity correction method enables precise trajectories to be represented with a small integration error. Experimental results demonstrate the feasibility of this device as a handwritten input apparatus. * ** 研究開発本部 オフィスシステム研究所 Office System Research and Development Center, Research and Development Group αタスクフォース 商品開発室 Product Development Department, α Task Force Ricoh Technical Report No.27 52 NOVEMBER, 2001 accelerometers is studied, too.5-7) However the gravity influence 1．Introduction can’t be eliminated completely in these researches, therefore it is There are 2 applications of a handwriting input apparatus for a difficult to represent the precise trajectory. 1) 2) We chose to study the first approach using such a inertial and character recognition for a PDA (Personal Digital Assistant). sensors. This system needs nothing more than the pen itself, so it The other is to input the written trajectory itself. In the former, it is the ultimate small appliance. If inertial sensors can detect the is not necessary to represent the trajectory; it can be verified or writing motion with 6DOF, then the two-dimensional writing 3) recognized directly by sensing the pen motion data, for example. trajectory can be represented after some calculation of coordinate The latter application is used to input the actual trajectory of transformation and double integration.8) Here we think that the written drawings, symbols, and characters. Here, we call this kind calculating method how eliminates the integration error is much of appliance, which retains the written trajectory as digital data, worth researching. computer. One is signature verification for security purposes Public telephone “Digital Ink”. Office telephone Cellular phone A tablet that has a sensor to detect the trajectory of a pen tip has been widely applied as a “Digital Ink” appliance for personal 4 computers (PCs). As its method can detect the pen tip position Network directly, the handwritten trajectory is represented with high and Im ag e Co mm un Company ica tio n reliable accuracy. However, the tablet is large, so it is difficult to apply to mobile or office applications where the workspace is limited. IrDA, Bluetooth Connection e I m a g n ic a t io n u Co mm What is needed is to shrink the sensing device and peripheral PC circuits. Our target is mobile and office applications based on new Notebook small “Digital Ink” appliances. The future scenario of our “Digital Fig.1 Ink” appliance is shown in Fig.1. Writing tools are connected to Future application using our “Digital Ink” appliance. various networked instruments (public telephone, office telephone, cellular phone, and PC) and send/receive data to/from each other. 2．Detection of Pen Motion in Handwriting Recently, there have been several studies related to systems of this kind in some research organizations and venture companies. There are two main approaches to shrinking “Digital 2-1 Ink” appliances: i) using inertial sensors to detect acceleration, Frequency of Ordinary Handwriting angular rate, tilting, and so on and ii) detecting the distance The FFT (Fast Fourier Transformation) power spectrum of between the pen tip and a reference point by sending and handwriting acceleration on paper with a ball-point pen is shown receiving ultrasonic waves or infrared rays, and calculating the in Fig.2. Its profile consists of two frequency levels. The lower pen tip position by triangulation. And the apparatus for inputting one below 10 Hz is the genuine motion signal, while the higher the written trajectory is researched, which is to detect three one above 100 Hz corresponds to the interaction between the accelerations and three angular rates with six degrees of freedom pen tip and paper surface5) 9). These signals can easily be (6DOF).4) This method based on INS (Inertial Navigation System) separated by ordinary filtering. The extracted higher one is of the navigation system for an aircraft and an automobile. And available to correct some error. another apparatus for inputting the written trajectory by means of Ricoh Technical Report No.27 53 NOVEMBER, 2001 fabrication techniques10) for gyroscopes is in progress and there Intensity(-) 1.0 are many desirable applications, so we can expect smaller ones soon. 0.5 2-3 0.0 Fig.2 Definition of Coordinate Systems Two coordinate systems are defined, as shown in Fig.3. The 1 1000 10 100 Frequency(Hz) Laboratory frame is the coordinate system (XG, YG, ZG), where the ZG axis corresponds to the direction in which gravity is acting. Power spectrum of handwriting acceleration. The Pen frame is the coordinate system (XS, YS, ZS), where the ZS axis corresponds to the axis of the pen core. Rotation vector is also defined to denote the rotation of the 2-2 Sensing Devices Pen frame relative to the Laboratory frame. The sensing devices consist of three accelerometers, three gyroscopes, and a pen touch switch. The layout of each sensor Zs-axis acceleration Ys-axis angular rate except the pen touch switch and coordinate systems are shown in Fig.3. A pair of accelerometers set at a distance L1 from the pen tip detects the Xs- and Ys-axis accelerations Zs-axis angular rate Xs-axis acceleration XG and another L2 accelerometer set at a distance L2 from the pen tip detects the Ys-axis acceleration Xs Zs-axis acceleration . The actual values of L1 and L2 in our Xs-axis angular rate L1 prototype are 41 and 160 mm, respectively. There are also three Ys gyroscopes set along the Xs, Ys, and Zs axes approximately at YG the center of the pen. The specifications of these sensors are Zs shown in Table 1. ZG Table 1 Sensor specifications Accelerometer Gyroscope Maker IC Sensors, Inc. Tokin Cooperation Principle Piezoresistive Piezoelectric Model 3021-002P CG-16D Sensitivity ※ 1.53 mV/(m/s2) 1.1 mV/(deg./s) Fig.3 Sensor layout and coordinate systems. 3．Algorithm for Calculating Pen Tip Trajectory The fundamental working process of the system is shown in Fig.5. It mainly consists of seven processes that are described ※ typical value below in 3.1. to 3.7. The accelerometers are fabricated by MEMS (Micro Electrical 3-1 Mechatronics Systems) technology, so they will become smaller Filtering and smaller during the next few years. The gyroscope was First, both the acceleration and angular rate signals pass developed for an image stabilizer in consumer cameras and through the filtering stage. The acceleration signals pass through binoculars. both a high pass filter (HPF) and a low pass filter (LPF); the Currently, the gyroscope is rather large to fit into a pen-like angular rate signals pass through only an LPF. These filters may device, but a lot of research and development of micro- be either hardware with electrical circuits or software using digital Ricoh Technical Report No.27 54 NOVEMBER, 2001 filtering. 3-3 Start Acceleration 3.1. High-pass filter Now the rotation vector is defined as the vector that Pen frame is rotating to Laboratory frame to the direction of rotating Angular rate 3.1. Low-pass filter First Rotation Vector axis of φ0 . The relationship between each coordinate system is shown in Fig.6. 3.1. Low-pass filter XS N 3.2. Stationariness ? XG (on pen touch) Y 3.3. Calculation of first rotation vector in laboratory frame Integration (to angle) X&& S 0 3.4. Coordinate transformation (acceleration (pen frame → laboratory frame)) YG 3.5. Integration (to velocity in laboratory Frame) Y&&S 0 Z&&S 0 YS 3.2. Velocity judgement and storage of result G (gravity vector) 3.5. Velocity correction ZS 3.6. Integration (to Displacement) ZG 3.7 Separation of pen touch movement (pen touch switch) Fig.6 The relationship between each coordinate system. End The gravity signal detected by accelerometers gives the Fig.4 absolute rotation vector value φ0 . This value is calculated when the pen tip is stationary. The value of e X 0 , e Y 0 shows the unit Fundamental working process of the system. vector of each axis. Here it is assuming that the direction of normal line on a paper 3-2 Velocity Judgement is corresponding to gravity direction. The small subscripts “0” The velocity judgement using HPF acceleration signal is shown mean initial value of each. φ0e X 0 φ0 = φ0e Y 0 ･････････････････････････････････ (1) in Fig.4. The pen is judged to be stationary when the acceleration falls below a threshold level. This knowledge is effective for correcting the integration error that occurs in velocity. The later integration sequence, so the result is stored. HPF signal (acceleration) Fig.5 Here φ0 , e X 0 , e Y 0 are given by number of stationary points in writing a character is used in the Velocity judgement 0 φ0 = Threshold levels Time ( ) && − X && 2 + Y && 2 Z 1 S0 S0 cos −1 S0 2 ･･････････ (1-a) 2 2 X && & & & & 2 + + Y Z S0 S0 S0 e X0 = && −Y S0 ･･･････････････････････ (1-b) && 2 + Y && 2 X S0 S0 Velocity judgement using HPF acceleration signal. Ricoh Technical Report No.27 55 NOVEMBER, 2001 && −X S0 e Y0 = 3-4 (d) ･･･････････････････････ (1-c) 2 && && X +Y S0 S0 Coordinate Transformation .. .. .. The pen tip acceleration (X 0 G , Y 0 G , Z 0 G ) 2 in the Laboratory frame can be acquired as && && X X S − ω& ySL1 − ω zSω xS L1 0 && oG S && & YoG = C G YS − ω xS L1 − − ω ySω zS L1 + 0 && && Z Z 2 2 S 0 ω xS + ω yS L 2 g oG Coordinate Transformation ) ( Using the LPF acceleration signals, the velocity judgement ･･･････････････････････････････････････････････ (5) result, and the integration value (angle) of angular rate, the .. .. .. ( X S , Y S , ZS ) Here accelerations in the Pen frame are transformed into ones in the is the detected acceleration, Laboratory frame. This calculation also corrects the positions of (ω XS , ω YS , ω Zs ) and (ω XS , ω YS ) are the detected angular the built-in accelerometers. rate and calculated angular acceleration of each, and “g” is the acceleration of gravity (9.81 m/s2). (a) Calculating the Rotation Vector φ (φ X , φ Y , φ Z )φ 1 φ n = φ n −1 + ωS + φ n −1 × ωS ∆t ･････････････ (2) 2 3-5 ( ω s : Angular rate vector ω XS , ω YS , ω ZS ) Velocity error caused by sensor signal noise and drift. To reduce the integration error, we correct the velocity by utilizing the velocity ( Calculating the Eulerian Parameter χ , ρ X , ρ Y , ρ Z 2 and give a velocity. However, their values include a lot of integration ∆t : Sampling time χ = cos Velocity The accelerations in the Laboratory frame are integrated to φ n −1 : Rotation vector on “n-1” sample φ to Correction φ n : Rotation vector on “n” sample (b) Integration judgement result. This can prevent the divergence of integration. ) 3-6 Integration to Displacement ･･････････････････････････････････ (3-a) A simple integration such as Simpson’s law is performed on the velocity values. This can yield an accurate displacement value with small divergence. ρX = φX φ φ φ sin , ρ Y = Y 2 φ φ φ sin , ρ Z = Z 2 φ φ ･････ (3-b) sin 2 3-7 φ = φ ･･････････････････････････････････････ (3-c) Distinguishing Pen Touch Movement Distinguishing whether the pen is touched or untouched enables complicated multi-stroke data to be represented. (c) Calculating the Coordinate Transformation Matrix CSG χ 2 + ρ 2 − ρ 2 − ρ 2 X Y Z 2 ρ X ρ Y + χρ Z = 2 ρ X ρ Z − χρ Y ( ( ) ) ( 2 ρX ρY − χρ Z ) χ 2 − ρX 2 + ρ Y 2 − ρ Z2 ( 2 ρY ρ Z + χρX ) ( ( 2 2 ρ Y ρ Z − χρX 2 χ − ρX − ρ Y 2 ) ) 2 + ρZ 2 ρX ρ Z + χρ Y 4 ． Correction Method to Achieve Accurate Trajectory Representation ･･･････････････････････････････････････････････ (4) : Coordinate transformation matrix from Pen Detection of the pen tip stationary point and velocity frame to Laboratory frame correction is effective to achieve accurate trajectory representation. And particulars about these processes are described below. Ricoh Technical Report No.27 56 NOVEMBER, 2001 Acceleration (m/sec2) 4-1 Pen Tip Stationary Points on Handwriting Our analysis of high-frequency acceleration in natural handwriting (Table 2) showed that it occurs quite often when the pen tip velocity becomes almost zero. The average number of Velocity (m/sec) information can be used for corrections of integration error. 4-2 Average number of stationary points Letters 4.1 Numerals 3.4 Kanji (Japanese) 12.5 Hiragana (Japanese) 5.1 Katakana (Japanese) 4.8 Velocity (m/sec) Average stationary points. (total 1145 characters written by one person) Character type 2.0 A B C 0.0 -2.0 0.0 Time (sec) 1.0 (a) HPF acceleration signal in Pen frame stationary points is almost more than three points, so this Table 2 Velocity judgement 0.1 Reference 0.0 -0.1 a b c Experiment -0.2 Time (sec) 0.0 1.0 (b) LPF velocity signals in Laboratory frame (No correction) 0.1 Reference 0.0 Experiment 0.1 0.0 Time (sec) 1.0 (c) LPF velocity signals in Laboratory frame (Correction) Velocity Correction Fig.7 The data for processing are shown in Fig.7. The HPF signal (a) Velocity judgement and correction. shows when the velocity becomes zero in A, B, C. The uncorrected velocity data (b) leads to a bigger and bigger error as 5．Prototype and Experiments the integration error diverges. Correction using the gradient among zero-velocity points (a, b and b, c) brings the 5-1 experimental velocity closer to the reference. Prototype Unit Our prototype unit is shown in Fig.8. It consists of sensors and their analog electrical circuits. An amplifier is provided for differential amplification of the accelerometers. Circuits for adjusting the offset voltage value and amplifying the voltage are provided for the gyroscopes. There is an interface for connecting external appliances by wiring, but in future a wireless interface and memory will be provided for practical use. What is designing smart layout of each sensor can make it small as usual pen. Ricoh Technical Report No.27 57 NOVEMBER, 2001 (b) shows the representation data obtained by our prototype. The 30mm written characters were each about 10 mm square. The bold points in the figures indicate the stationary points used for recognition. Some of the stationary points used by the prototype unit differed from those used by the tablet. These results demonstrate the effectiveness of our prototype Y (mm) Y (mm) 160mm system. X (mm) (a) Tablet data Fig.8 Fig.10 Handwriting trajectory (1). Y (mm) Experimental System Y (mm) 5-2 Prototype unit. X (mm) (b) Representing data The experimental system is schematically shown in Fig.9. An electromagnetic tablet was used to acquire the ideal true writing trajectory. The tablet was connected to an RS232C serial X (mm) (a) Tablet data interface on a PC. The analog output signals of the prototype unit were input to an ordinary I/O board having a 12-bit A/D X (mm) (b) Representing data Fig.11 Handwriting trajectory (2). converter. Almost simultaneous sampling between the tablet and prototype was achieved, so the representation result is a quantitative evaluation. Tablet data 6．Conclusion A/D converter Representing data Prototype pen unit Our 6DOF prototype handwriting input apparatus can detect Synchronous acquisition RS232C accelerations and angular rates around three axes. By using a velocity correction method to calculate the accurate pen tip trajectory, we obtained several good representation results, PC Fig.9 demonstrating its feasibility. Electromagnetic inductive tablet A future goal is to be making practicable system as described Experimental system to evaluate the represented trajectory. in our future scenario. Reference) 5-3 Experimental Results 1) Typical results for trajectories of Japanese writing are shown in Writing Tool, International on Micro Machine and Human Science, Figs. 10 and 11. Fig.10(a) shows the tablet data for reference and Ricoh Technical Report No.27 Dominiek Reynaerts, et al. : Design of an Advanced Computer IEEE, Sixth, (1995) pp. 229-234. 58 NOVEMBER, 2001 2) Dirk Diddens, et al. : Design of a ring-shaped three-axis micro force/torque sensor, Sensors and Actuators, A 46-47, (1995) pp. 225-232. 3) Furuta, et al. : A Study of Character Recognition by Detecting Pen Motion, PROCEEDINGS OF THE 1997 ENGINEERING SOCIETY CONFERENCE OF IECE, (1997) pp. 190. 4) Christopher Verplaetse : Can A Pen Remember What It Has Written Using Inertial Navigation? : An Evaluation Accelerometer Of Current Technology, http://xenia.media.mit.edu/~verp/projects/smartpen/, (1995) 5) Ishikawa, et al. : Extraction of Character Information from Acceleration of Handwriting Pen, THE TRANSACTIONS OF THE INSTITUTE OF ELECTRONICS, INFORMATION AND COMMUNICATION ENGINEERS, VOL.J76-D-ⅠNo.10, (1993) pp. 541-543. 6) Ishikawa, et al. :Reproduction of character pattern from pen acceleration in handwriting, Proceedings of the 24th Joint Conference on Imaging Technology, (1993) pp. 25-28. 7) Miyagawa, et al. : Character Pattern Reproduction from Handwriting Pen Acceleration, THE TRANSACTIONS OF THE INSTITUTE OF ELECTRONICS, INFORMATION AND COMMUNICATION ENGINEERS, VOL. J83-D-ⅡNo.2, (2000) pp. 671-679. 8) Sato et al. : A PEN-SHAPED INPUT APPARATUS, US. Patent, (1999)USRP.5902968 9) Sato et al. : A Study of Handwriting Input Apparatus by Inertial Sensing, 2000 JSME Conference on Robotics and Mechatronics (ROBOMEC’00), (2000)1P1-02-002 10) S.Sassen et al. : Tuning fork silicon angular rate sensor with enhanced performance for automotive applications, Sensors and Actuators A 83, (2000) pp.80-83. Ricoh Technical Report No.27 59 NOVEMBER, 2001 Ricoh Technical Report No.27 60 NOVEMBER, 2001