CE1.1 After graduation, I was always actively involved in undertaking research and development (R&D) projects. I supervised many final year projects. In year 2011, one of such activity was related to Automatic Patient Monitoring system which was the FYP of Sir Syed University of Engineering and Technology Students of Biomedical Engineering Department.They started the project from Feb 2011 & completed in December 2011. I was part of their project in the capacity of external supervisor as I was working with Aga Khan Development Network eHealth Resource Centre.Project name was patient monitoring and activity recognition system. I myself was student of Sir Syed University of Engineering & Technology having Roll number: BM-057-2005.
Nature of the overall engineering project:
CE1.2 According to a recent World Health Statistics, there are only 13 nurses for a population of 10,000 in the South East Asian countries. There is an alarming shortage and requisites multifaceted responses from all possible quarters. Capacity building and empowering of available human resource with cutting edged technological biomedical tools is what an engineer can deliver in this regard. We need to append nursing staff in a way that a maximum utilization of human resource should be made possible with least efforts.
CE1.3 ICU and old home are places which are a challenge for nursing. Higher standards of vigilance, focus attention and diligence is mandatory for nurses working there. However, due to scarcity of nursing staff, large number of patients and many other factors nursing staff is not able to maintain the standard of work. There exists a serious gap in requirement and availability of required tools, which generate a significant room of improvement and technological development. There is a need for system which can assist nursing staff in performing their duty and enable them to provide timely and required medical assistance.
CE1.4 My project was focused on development anautomatic system for patient monitoring and their specific activity recognition. My project was focused to have some important feature such as easily installation and use of commercial of the shelf (COTS) items to have quick deployment.
CE1.5 Objectives of the project: Designing of a system for patient monitoring which is able to provide assistance to nursing staff in critical place such as ICU and old homes
CE1.6 The nature of particular work area: In execution of the project design work, besides development of the algorithm, I was also responsible for testing, troubleshooting, debugging and implementation of the software on the designated hardware. The developed algorithm was to take input from KINECT® sensor, an accessory which is used with commercially available
CE1.7 Gaming consoles. Data from the Kinect sensor was analyzed by computer using my designed algorithm.
Chart of the organizational structure highlighting position:
CE1.8 As anexternal Supervisor my responsibility was to keep the project on track as per my scope furthermore to guide students where they lack technical expertise.
CE1.9 As a research engineer with an innovation driven organization, I undertook the project in which we developed a hardware cum software. It was able to detect different motion attributes of individual. Main aim was to make use of commercially available hardware for quick solution development and easy realization. Kinect sensor, which was available with Sony Xbox, was considered suitable for the desired application. I offered the idea a group of students of biomedical department who were to provide hardware implementation of the developed algorithm.
CE1.10 As project supervisor, I was the focal person and I was involved in almost all the activities related to the project. However, following are some the specific activities which I undertook:
a. Interface of the Kinect sensor was required for the project data collection. I decided that the student should work in Matlab environment, Kinect driver was required to port data from hardware cache to computer. In the older version of Matlab such driver was not available. I carried out study of the newer version and interacted with Mathworks office. Based on which I established that the problem can be addressed by using newer version of Matlab.
b. The students were tasked to collect data. However the system was giving error message that local cache is full and system resources are exhausted. I checked their setting and it revealed that they are sampling at a frequency of 30 kHz, which much higher. Keeping in view requirements of the desired task, I suggested them to reduce sampling frequency to 10 kHz and sampling time was also reduced to as the make optimal use of the memory available. It resolved the issue of warnings and system hang were addressed.
c. I tasked the student to select features in frequency domain as well as in time frequency domain. The students were not conversant with the time frequency domain conversion techniques. I wrote the Matlab for students and explained them the generic and intuitive understanding about Wavelet distribution as to provide them better insight about the project.
d. Selection of discriminative feature is challenging tasking and for undergraduate student is was even difficult. I provided them with the desire figure of merit (FOM) for selection of feature and also helped them in developing code. As it was classification of features which had a greater overlap, I asked them to use BhatacharyyaDistance instead of commonly Euclidean disk
e. Undergraduate students generally don’t study the course of “Classification and Machine learning”. Therefore, classifiers are not within their knowledge ambit. Being project supervisor and design engineer, it was my job to select suitable classifier among the plethora of machine learning techniques. I selected Linear Discriminant Classifier (LDC) and Support Vector Machine (SVM) to be implemented. The selection was based on literature review of state of the art classifiers available. SVM is considered as the most robust and with best accuracy despite low discriminative features, and LDC being linear is most easy to implement and is computationally efficient. In the final demonstration of the project, I asked students to present LDC as it was easy to understand and explain.
f. I can state with confidence that I did the task assignment and division of work among the students, selection of the required tools, techniques and software, a good part of coding, and finally editing and proof reading of the report.
Personal Engineering Activity:
CE1.11 I undertook many activities related to the project. I was the prime coordinator of the project and undertook planning and costing of the project. It was followed by tasking to different students who were assigned tasks in such a manner that all of them were working in parallel without mutual dependence. I, myself carried out algorithm development as it was beyond the understanding level of students, who were primarily tasked to collect design of experiment, fabrication of required accessories, data Collection and initial signal processing.
CE1.12 As already mentioned, the data was collected using Kinect sensor. Kinect has three lenses. It performed denoising of collected data and initial signal processing called preprocessing. Preprocessing involves obtaining depth data using IR sensor. Kinect data from depth sensor tracks and calibrates position using a tracking software, which was developed by myself in Matlab. I used technique of differential gradient to identify relative change of different joint, along with the associated speed (i.e. rate of change of the position). Jerk in the joint position as considered as falling or unusual activity. More specifically, it was based on histogram of oriented gradient.
CE1.13 In the developed algorithm, each person was assigned an ID which gives 3D position of joint position in real world and pixel coordinates. Position data required for activity detection and monitoring was filtered for minor errors in movements and saved for further processing.
CE1.14 One method to smooth the time series position data was to substitute each value of the series with a new value which was acquired from a polynomial fit of 2n+1 neighboring points counting the point to be smoothed.
CE1.15 My developed algorithm can be represented with the following block diagram which shows different steps:
CE1.16 Feature extraction modulewas the prime activity which I performed for motion detection. Certain distinctive measurable inputs were acquired which were useful to classify image, action or a signal. Success of any classification and recognition system relies on the selection and ability to develop unique and robust features. Another job of this module was to process data and converts it so that it becomes suitable for your system.
CE1.17 Natural human motion can be explained with concepts of position, velocity and acceleration known as kinematics. These concepts can be extended to jerk. My algorithm provided interpretation of single and multi – limb movements performed by a subject and wereclassified as regular or jerking movement. 3D position of one limb joint was saved at each instant. Within a given frame, an object’s position can be described by a displacement vector from the origin, which in this case was Kinect mounted in room, to the object from Kinect. Position was calculated in three-dimensional Euclidean space and further processed for jerking motion detection.
There were many issues which were tackled while algorithm development:
CE1.18 First problem that arises is “occlusion” due to blocking of subject by another object. In 2Ddata analysis, this problem worsens. Proposed way to solve this was to use Kinect to acquire 3D image information in place of 2D to get as much information out of an image as possible in hope of increasing classification efficiency.
CE1.19 Second problem addressed was “controlled environment”. In facial recognition systems, we have a database of a static picture of human face with white background. In activity recognition, this constraint could not be achieved. It was addressed by use of depth information acquired from 3D images from kinect’s IR sensors. Kinects range was from 400 to 8000 mm and objects and humans closer appear dark and as we moved away from the camera, image keep moving towards lighter intensities.
CE1.20 Besides, all goods, Kinect also had few limitations. It gave skeleton of whole body This problem was solved by mixing information gathered from depth sensor and RGB camera using image processing algorithms in order to get a skeleton of body for a more concise classification of activity recognition.
CE1.21 I normalized the data in order to make is independent of magnitude of the acquired signal:
CE1.22 I used histogram of oriented gradient on 3-dimensional time-space trajectory data, x-axis of histogram has orientation angles for a single frame of activity and magnitude was calculated and added to corresponding angle bin. I applied it to each activity matrix after passing it through pre-processing stages. After histogram formation, linear discriminate Analysis was applied as a dimensionality reduction tool. I checked System efficiency by applying support vector machine (SVM)as a classifier and a result comparison was carried out. Histogram of 3 dimensional directional derivatives are used as features. Histogram of directional derivative is given as
CE1.23 The above equation gave the shortest angle between two 3-dimensional vectors. In order to get theta from 0 to 360 degrees, I used direction cosines to get angle from 0 to 360 degrees by taking an xy,xz or yx plane as a reference for counter clockwise measurement as shown in Fig. Since z-axis shows minimal overall movement, I have chosen xy-plane to determine the counter clockwise angle between the gradient vectors. XY-plane in Fig is represented by no grid lines. If gradient vector between J1 and J2 in maximum gradient vector, ? was the angle between the maximum gradient vector formed by joints J1 and J2 and position vector formed by joints J1 and J3.
CE1.24 Using the gradient vector, classification of different human activities was made. Using the algorithm which I developed, activity recognition of accuracy upto 84% was achieved which were considered as very high.
CE1.25 Project is very complicated and to avoid any confusion we created an email group for coordination and for any meeting to plan, so I use google calendar to intimate any meeting.I force everyone to write email for every meeting or for small discussion, this help us to track what we have discussed. In the end it helped me a lot when I was drafting my section of the report for reasoning of any problem.
CE1.26 Project is very complicated and to avoid any confusion we created an email group for coordination and for any meeting to plan, so I use google calendar to intimate any meeting.I force everyone to write email for every meeting or for small discussion, this help us to track what we have discussed. In the end it helped me a lot when I was drafting my section of the report for reasoning of any problem.
CE1.27 All the object such as electric power supply Kinect sensor & computer labs software, each followed safety measure. When I use power supply I make sure that plug is fully insert in the power outlet. It was instructed that do not place any have object on KINECT sensor otherwise the desire result cannot be obtained.
CE1.28 During the execution of the project, there were many aspects of learning. I was able to carry out a real world problem based project. I was able to plan activities in an efficient manner ensuring parallel tasking of team members, which was a main reason of timely completion of the project. The algorithm I developed was able to produce results with desired level of accuracy. The same was cross validated using simulated data. Overall, it was a good experience to undertake supervision of undergraduate project with respect learning of engineering as well as project management.