Figure 1 : MIPAD's first card and MIPAD's calendar card
"People have had notebook computers for a long time, but they don't usually bring them to meetings, in part because it's considered impolite to be typing away while someone's talking," said Brad Myers. "But people do bring in pen-based PDA(Personal Digital Assistants) to take notes, so we're trying to take advantage of what they're already using."
Demands on ubiquitous computing devices, such as PDAs, are increasing. The limited capability of the interfaces, however, is an obstacle that hinders their progress. As a PDA has restricted, various size of screen and inconvenient input devices, study of user interface for PDA is more important and practical than any other devices. This technotes presents features of PDA interface designing and development. It explores some interesting projects that demonstrate interaction of PDAs. Moreover it introduces a useful development tool kit for PDAs, especially for PocketPC, along with some illustration examples on how this toolkit can be used in developing an application for PDAs.
1. MIPAD : Tap and Talk interface (Speech Technology Group, Microsoft)
MiPad integrates continuous speech recognition (CSR) and spoken language understanding (SLU) to enable users to
accomplish many common tasks using a multimodal interface and wireless technologies. As a user taps the screen or uses a build-in roller
to navigate, the tapping action narrows the number of possible instructions for spoken language processing.
| Pen | Speech |
| Direct manipulation | Hands/eyes free manipulation |
| Simple actions | Complex actions |
| Visual feedback | No visual feedback |
| No reference ambiguity | Reference ambiguity |
| not affected by noise | Affected by noise |
| Secrecy | Privacy is of concern |
| Restricted by limitted space | space independent |
As shown in Table 1, pen and speech can be complementary and they can be used very effectively for handheld devices. The advantage of pen is typically the weakness of speech and vice versa. This implied that user interface performance and acceptance could increase by combining both.
MIPAD has a Tap & Talk field that is always present on the screen as illustrated in start page in Figure 1. The user can give spontaneous commands by tapping the Tap & Talk field and talking to it. The system recognizes and parses the command, such as showing a new appointment form as illustrated in, The appointment form shown on MIPAD's display is similar to the underlying semantic objects. The user can have conversation by tapping and talking to any sub-field as well. By tapping to the attendees field in the calendar card shown in Figure 1, for example, the semantic information related to potential attendees is used to constrain both CSR and SLU, leading to a significantly reduced error rate and dramatically improved throughput.
2. Pebbles project : Collaboration using multiple PDAs connected to PCs (HCI Institute, CMU)
Many environments, such as offices, meeting rooms and classrooms, already contain computers, and the smart homes of the future will have ubiquitous embedded computation. When the user enters one of these environments carrying a PDA or wearable computer, how will that computer interact with the environment?
The Pebbles project is exploring the many ways that small handheld Personal Digital Assistants (PDAs) such as 3Com Palm Pilots or PocketPC devices can serve as a useful adjunct to the "fixed" computers in those situations.
The main components in the Pebbles architecture are client programs running on (one or more) PDAs, server programs running on the PC and PebblesPC, a PC program that mediates between clients and servers. (Figure 2)
The first application from Pebbles project is the Remote Commander. This allows strokes on the main display area of PDA to control the PC's mouse cursor, and for Graffiti input to emulate the PC's keyboard input. Another application of this is PebblesDraw. PebblesDraw is a multi-cursor drawing program that is analogous to other "shared whiteboard" application. One important different is that all the users share the same display. Therefore with PebblesDraw, multiple users can concurrently be annotating and setting up the map as Figure 3.
Microsoft Embbeded Visual Tools are offered from Microsoft for free, and can be downloaded from the Microsoft website. If you don't want to wait for the 400MB download to complete, you can get it from a MSDN subscription or order the CD's.
1. Microsoft Embedded Visual Tools 3.0 : http://www.microsoft.com/mobile/downloads/emvt30.aspYou need to install all of above softwares in this order. The way to install and system requirement is specified in EVT installation guide site.
Before starting the porting process, we should take a minute going through all the stuff that we will not port to Pocket MP3 Player, simply because it is not possible, or it requires too much work. Many of the things that we take for granted when doing Windows development with MFC are not implemented in MFC for the Pocket Windows platform. The most prominent features which are included in the PC version of MP3 Player but not be in Pocket Windows version are:
In this technotes, I will show you a simple example of development using Embedded Visual C++ (included in Embedded Visual Tools). The software I will make for this technotes is a MP3 player for language learners. When students study English for TOEFL(Test of English as a Foreign Language) test or TSE they need a software that plays audio files like mp3 and wav to improve their listening skill. Until now, during the analog age, a cassette player had played this role. But a normal cassette player, a sequential media access device, isn't suitable for listening practice. Because language learners need to listen same section repeatedly, skip some track, and sometimes jump to next chapter. A MP3 player is a best device for these needs. In this example I didn't implement a MP3 player engine or MP3 decoder. I used the core code of the xaudio mp3 player. Xaudio is a open source project and its source codes for Windows, Linux and UNIX are downloadable from the web. In this example I just added an user interface to Xaudio for language learners who want to use their PDAs for listening practice.
Before start we mush have the following:
Programming with Embedded Visual C++ is exactly same as Visual C++ Integrated Development Environment. One difference is user can select CPU type of aiming device as Figure 5. If I select "Win32 [WCE x86em] Debug" or "Win32 [WCE x86em] Release" for CPU type, compiler build for x86 emulator. So I can run it on WindowsCE emulator without PocketPC.
If our project is compiled for emulator and run by pressing F7, WindowsCE emulator will be appear on the screen like Figure 6. But I don't recommend using WindowsCE emulator. Because emulator is very different from real WindowsCE, especially PocketPC, you had better test it on real PocketPC.
Embedded Visual C++ provides visual tools, resource editor, to make user edit resources.
So I can easily layout widgets like buttons, progress bars and comboboxes.
If you are comfortable with Visual Studio you will feel no difference. To make an interface of MP3 player for language learners, I made a dialog as shown in Figure 7 and
edited source file related with view like below source code.
These functions are called when button is clicked or slide bar is moved.
BOOL jPlayerCore::Play()
{
BOOL bForceRefill=FALSE;
if(outputModule.GetAppropriateDecoder())
{
bForceRefill = TRUE;
SetThreadPriority(GetCurrentThread(), THREAD_PRIORITY_TIME_CRITICAL);
// If we don't have an output stage - initialise one now
if(outputModule.IsWaveOutActive() == FALSE)
{
ReadyToPlay();
}
outputModule.SetPause(FALSE);
NotifyMessage(JNWM_PLAYERSTATE, (WPARAM)jpsPlaying);
TRACE(_T("JNWM_PLAYERSTATE, (WPARAM)jpsPlaying\n"));
iLastSentTime_Secs = -1;
UpdateProgress();
}
return (bForceRefill)?RefillAndSoundOut():TRUE;
}
BOOL jPlayerCore::Stop()
{
if(outputModule.GetAppropriateDecoder())
{
outputModule.GetAppropriateDecoder()->CloseFile();
outputModule.SetDecoder(NULL);
}
outputModule.Uninitialize();
UpdateProgress();
NotifyMessage(JNWM_PLAYERSTATE, (WPARAM)jpsStopped);
TRACE(_T("JNWM_PLAYERSTATE, (WPARAM)jpsStopped\n"));
return TRUE;
}
BOOL jPlayerCore::Pause()
{
if(outputModule.IsWaveOutActive() == TRUE)
outputModule.SetPause(TRUE);
UpdateProgress();
NotifyMessage(JNWM_PLAYERSTATE, (WPARAM)jpsPaused);
TRACE(_T("JNWM_PLAYERSTATE, (WPARAM)jpsPaused\n"));
return TRUE;
}
BOOL jPlayerCore::Seek( int iNumerator, int iDenominator)
{
BOOL ret=FALSE;
if(outputModule.IsWaveOutActive() == TRUE)
{
outputModule.Flush();
if(outputModule.GetAppropriateDecoder())
outputModule.GetAppropriateDecoder()->Seek((int)iNumerator, (int)iDenominator);
ret = TRUE;
}
return ret;
}
Below is the text version of dialog I made in Figure 7. This text version, needless to say, is generated automatically. But you can edit it directly if you want to.
// Dialog
STYLE DS_MODALFRAME | WS_POPUP | WS_CAPTION | WS_SYSMENU
FONT 10, "System"
BEGIN
DEFPUSHBUTTON "OK",IDOK,15,77,50,14
PUSHBUTTON "Cancel",IDCANCEL,76,77,50,14
COMBOBOX IDC_SCR_TOOGLE,7,20,84,30,CBS_DROPDOWNLIST | CBS_SORT |
WS_VSCROLL | WS_TABSTOP
END
IDD_JP1_DIALOG DIALOGEX 0, 0, 141, 154
STYLE WS_POPUP | WS_VISIBLE | WS_CAPTION
EXSTYLE WS_EX_APPWINDOW | 0x80000000L
CAPTION "MP3 Player"
FONT 8, "System"
BEGIN
CONTROL "Slider1",IDC_TIME_SLIDER,"msctls_trackbar32",TBS_TOP |
TBS_NOTICKS | WS_TABSTOP,0,71,135,18
LTEXT "TITLE",IDC_TITLE,5,26,80,11
LTEXT "total sec",IDC_TOTAL_SEC,80,41,55,11
LTEXT "current sec",IDC_CURR_TIME,5,41,55,11
PUSHBUTTON "PLAY/PAUSE",IDC_PLAYPAUSE,5,95,80,20,NOT WS_TABSTOP
PUSHBUTTON "<<",IDC_PREV_TRACK,5,56,25,15,NOT WS_TABSTOP
PUSHBUTTON ">>",IDC_NEXT_TRACK,35,56,25,15,NOT WS_TABSTOP
PUSHBUTTON "<--",IDC_REWIND,5,120,60,20,NOT WS_TABSTOP
PUSHBUTTON "-->",IDC_FASTFOWARD,75,120,60,20,NOT WS_TABSTOP
PUSHBUTTON "Open",IDC_OPEN,65,56,30,15,NOT WS_TABSTOP
COMBOBOX IDC_REWIND_SEC,101,56,35,70,CBS_DROPDOWNLIST |
WS_VSCROLL
END
In this technotes, I presented interesting projects connected with PDA interaction. And I introduced Microsoft Embedded Visual Tools. And I show a simple developing process using Embedded Visual Tools. Microsoft provides consistent visual tools for development of PocketPC software. Therefore a user familiar with Windows programming can develop PocketPC software without any extra effort.
1. Brad A. Myers, Using Handhelds and PCs Together, Communication of the ACM, Nov.2001, Vol 44, Issue 11
2. Y.Ayatsuka, N.Matsushita, J.Rekimoto, HyperPalette: a Hybrid Computing Environment for Small Computing Devices, CHI 2000
3. Brad A. Myers, Herb Stiel, Robert Cargiulo, Collaboration Using Multiple PDAs Connected to a PC, Proceeding CSCW98 ACM conference on Computer-Supported Cooperative Work
4. Xerox PARCTAB technote : http://sandbox.parc.xerox.com/parctab/csl9501/node4.html
5. Speech Technology Group of Microsoft Research : http://research.microsoft.com/stg
6. Pebble Project, CMU : http://www-2.cs.cmu.edu/~pebbles/
7. The Dasher Project, Cambridge UK : http://www.inference.phy.cam.ac.uk/dasher/
8. Xaudio MP3 player open source : http://www.xaudio.com
9. PDA at a glance : http://www.microsoft.com/mobile/pocketpc/pdainfo.asp