When a hearing impaired student attends class, one of the measures used to communicate to the student what the teacher is saying is a technique called summary transcription. Summary transcription is a task in which a summary scribe sits next to a student with a hearing impairment and writes down a summary of what the teacher says or enters it in a PC and shows it to the student. However, it is often the case that there are few competent people available to perform summary transcription satisfactorily, leaving the students unable to receive this support. At NEC System Technologies, R&D of a remote summary transcription system is underway that aims to achieve a solution capable of presenting opportunities for more students with hearing impairment issues to receive summary transcription support and for people with summary transcription skills who are unable attend class to act as remote summary scribes.

1. Introduction

NEC System Technologies has established a remote support technology via the Internet that enables real-time summary transcription with a view to increasing the opportunities for students with hearing impairments. By applying this technology the system allows class attendance while receiving the summary transcription service remotely. In the latter half of FY 2012, we conducted validation experiments in which students with hearing impairment attended classes at two colleges in Okayama Prefecture.

This paper discusses the R&D background to this technology and goes on to introduce the features and mechanisms of the remote summary transcription system that was developed as well as the results of the validation experiment that was conducted subsequently.

2. Background

2.1 Circumstances Regarding the Development of the Remote Summary Transcription System

It was originally an idea of a co-worker who was the parent of a child with a hearing impairment that spurred the R&D of the support technology that led to the present remote summary transcription system. As we proceeded with our research into the environmental conditions that currently surround students with hearing impairments, we found issues such as that there were few people who had acquired the skill of summary transcription and their numbers varied from area to area. Therefore, it was difficult to maintain the support structure when no students with hearing impairments remained at school, which made it difficult to pass on the expertise. Although there were people who would like to offer support as summary scribes, they could not get easily to the relevant locality. As a framework to solving these issues, we aimed at achieving a supporting technology that would enable these people to perform summary transcription remotely via the Internet. Our aim was to develop a remote summary transcription system that would utilize such a technology.

2.2 Present Conditions Surrounding Students with Hearing Impairments

People with hearing impairments number approximately 340,000 in Japan; among them, students under the age of 18, account for approximately 17,000 (Fig. 1). On the other hand, the number of summary scribes registered in Japan is only about 10,000.

Usually, several summary scribes (2 or 3) sit next to a student with hearing impairment and perform summary transcription service. Since the number of available summary scribes is limited however, there are many students who are unable to receive the summary transcription service. Moreover, at colleges and other higher educational institutions, the ordinary summary scribes sometimes cannot take charge due to the need of specialist knowledge. There are colleges that have built supporting structures using their current students and graduates in order to deal with this situation. Nevertheless, the summary scribes themselves have their own classes and jobs to attend, so the summary transcription service cannot be provided in every class that a student with hearing impairment attends. What is more, there are cases where the supporting structure cannot be maintained once students with hearing impairment graduate, thereby creating the problem of being able to pass on expertise to successors.

2.3 Present Status of Summary Transcription

There are two procedures of summary transcription: one is called manual summary transcription in which notes are handwritten on a sheet of paper or a transparency for an overhead projector and the other is called PC-based summary transcription. When a PC is used, an ordinary text editor or free application such as IPTalk is used to perform summary transcription. IPTalk is a free application developed and improved based on requests from summary scribes that is popular due to its various functions. However, it has become evident that there are users who think that simpler functions than these would be sufficient.

3. Features and Mechanisms of the Remote Summary Transcription System

3.1 Features

The present remote summary transcription system can support transcription effectively via the Internet in real time and from a remote location as long as there is an available environment for ordinary web conferencing (Windows PCs, webcams, and headsets). Moreover, the required preparation merely comprises three steps - (1) starting up the application, (2) connecting (logging in) to the server, and (3) starting the class. While the settings of the environment are restricted to minimally required items in order to allow anyone to use the system easily. From the viewpoint of management there is a "matching function" that facilitates class allocations and thereby reduces the burden on the staff who manage the assignment of the summary scribes.

3.2 Architecture

The remote summary transcription system consists of the client for users (the hearing-impaired), the client for the summary scribes, and the administration server (Fig. 2).

There are three main functions as follows.

(1) Advance reservation

Matching is made by giving consideration to the requests for support from schools, guardians and students, as well as to the schedules and skills of the summary scribes.

(2) Transmission and storage

The classroom and the remote location are connected via the Internet to support remote summary transcription, and the transcribed data is automatically saved in the server.

(3) Subsequent support

Based on the transcribed data stored as mentioned above, the users (the hearing-impaired) and the summary scribes that were in charge can review the transcribed data and correct it if necessary.

3.2.1 Advance Reservation

First of all, a guardian or student registers information on the classes for which the summary transcription service will be needed as well as the required number of the summary scribes. Then after coordination is performed while the details of the registered request for the service and the schedules and skills of the available summary scribes are taken into consideration. The request is then notified to the summary scribes who are capable of handling the classes under the request. Now if the time and dates of the classes and their subjects are compatible, the summary scribes register their acceptance of the request (Fig. 3).

3.2.2 Transmission and Storage

1) Transmission of transcribed data from a remote location

Using the connection via the Internet between classrooms and summary scribes at remote locations who listen to the voice of the teacher transmitted from the classroom, the transcribed data is sent to the PC of the student with hearing impairment in the classroom (Fig. 4).

2) Automatic recognition of speech intervals to support summary transcription tasks

The system automatically recognizes the breathing pauses of the teacher's utterance and judges the speech intervals in order to instruct the switching of the summary scribes. If the summary scribe fails to hear the utterance of the teacher from a remote location, the utterance can be automatically played back from the last speech interval by giving a direction to the system to play back repeatedly. Helped by this function, the summary scribes can easily tack back from where they missed and confirm what they have entered.

3.2.3 Subsequent Support

The video, audio and transcribed data are recorded in the library on the server. After the classes are finished, either the summary scribes or the lecturers can make an addition to or correction of the data.

Besides, even if a suitable summary scribe cannot be allocated to the class when an assignment is offered, transcribed data can be generated later on from the record. Moreover, the student with hearing impairment can take the recorded class again and again. (Fig. 5)

4. Example of an Assignment at Kibi International University in Okayama Prefecture

Introduced here is an experiment conducted under the supervision of Professor Tadshi Sato at Kibi International University in Okayama Prefecture, where joint experiments were carried out with us in the latter half of FY 2012 (Photo). The hearing-impaired student attended the class having a webcam-mounted Windows tablet PC while the professor wore a wireless headset. Sitting in a separate room (e.g., a small seminar room), the students who performed the summary transcription tasks transcribed the professor's words transmitted from the hearing-impaired student's PC via the remote summary transcription system and then transmitted the transcribed data to the hearing-impaired student's PC in the classroom.

There was a comment from the student who performed the summary transcription tasks saying that the system was easy to set up and took very little time to start up, which made him wish to use it again. Moreover, the student with hearing impairment commented that it was easy to use and he was happy because he now had fewer places to look at.

5. Conclusion

We are optimistic that the promotion of the usage and dissemination of the remote summary transcription system will allow more students with hearing impairments to receive support. They will also be given more opportunities to receive higher education, thereby leading to a significant social contribution.

Additionally, the remote summary transcription system was developed and went through a validation assessment under a grant from the National Institute of Information and Communications Technology, an Independent Administrative Agency, in FY 2011.

We would like to express our gratitude to Professor Tadashi Sato at Kibi International University for carrying out the joint experiment and for the cooperation of the summary scribes' association as well as the hearing-impaired participants and their guardians.

Authors' Profiles