1. Introduction

SITPLUS is a free software framework whose main goal is to provide recreational activities for people with multiple disabilities. It offers new forms of interaction based on computer vision, voice and other peripherals to produce a result in the form of image and sound. Inspired by the cause and effect applications, SITPLUS provides a tool for continuous and remote interaction, attainable to the majority of people with multiple disabilities.

2. How to install

2.1. Requirements

2.2. Installation process

You can download the latest SITPLUS version from SITPLUS website. A Windows installation package is provided. For GNU/Linux you might need to fetch the sources and compile them.

  1. Before installing SITPLUS first uninstall any previous version.

  2. Install Pure Data (pd).

    • Windows. Download and install with the default options Pd-extended

    • GNU/Linux. Major distributions provide pre-compiled pd packages. Unfortunately, most pd packages do not provide all required components (externals in pd jargon) which are required by SITPLUS and thus voice based activities are currently unsupported.

  3. Install SITPLUS.

3. Running SITPLUS

Before running SITPLUS make sure that the required devices such as sound input (i.e. a microphone) and output, web camera, Wii Remote, etc. are already installed and work properly. Set also the system sound volume (both for recording and playback) accordingly.

Once started check the configuration options under the 'Configuration' menu. To use a Wii Remote you must pair it with the computer, check Wii Remote Configuration for further details. Check also how to configure Pure Data. Pure Data Configuration.

Currently, activities can be started either by choosing 'Start…' or 'Load file…' under the 'Activity' menu. The former option allows to select which input channels to use (camera, voice and/or Wii Remote) before starting one of the built-in the activities. The later option allows to load a .sp file (some .sp files are provided under sp/ folder).

3.1. Built-in activities

Although SITPLUS is conceived as a framework which allows for several ways of customization, it currently provides some built-in activities which combine different input methods (camera tracker, voice input and Wii Remote based motion sensing) with real-time MIDI sound generation and a graphical collage.

Activity screenshot
Figure: Activity screenshot

The screenshot depicts an activity which combines the use of the three input channels is shown. Using this activity the user can play a MIDI musical score though the motion picked up using the camera and/or a wiimote. Motion is also use to make the graphical collage evolve. At the same time, the user's voice is also picked up, processed using the chosen effects and played it back to the loudspeakers. Moreover, user's voice also makes the collage evolve.

The window is organized in three main areas:

4. Configuration

4.1. Wii Remote Configuration

To be able to connect a Wii Remote device (aka wiimote) - currently the standard Wii Remote (also with Motion Plus) and the Balance Board are supported - you need a bluetooth adapter. The configuration of such device is beyond the scope of this manual.

On Windows you first need to pair you wiimote with the computer, see Pairing the wiimote.

To check if the wiimote is properly detected access Configuration → Wii Remotes and press the “Reconnect” button (on GNU/Linux you might need to simultaneously press buttons “1” and “2” on the wiimote before reconnecting). Once connected the wiimote will rumble and the dialogue will be updated.

Wii remote screenshot
Figure: Wii remotes configuration screenshot

4.1.1. Pairing the wiimote (Windows only)

The basic steps are:

  1. Open the Bluetooth Device configuration. Depending on your Windows version you will find the Bluetooth icon in the Windows control panel or in the task bar. A list containing the currently paired devices will appear.

  2. Remove any previously paired device.

  3. Select “Add new device” to start searching for available Bluetooth devices nearby.

  4. Simultaneously press buttons “1” and “2” on the wiimote (LEDs will start flashing, the number of flashing LEDs also tells you the state of the batteries).

  5. Select the newly detected device and pair it without using codes.

If the process succeeded, LEDs will keep flashing. Note that after a system reboot you need to repeat this process again.

Sometimes (if the wiimote was not properly shut down) you might need to reset it by removing the batteries and pressing “1” and “2” before putting them into again.

4.2. Pure Data Configuration

  1. Open the audio configuration dialogue (Configuration → Pure Data).

  2. Once opened you should hear a continuous sound. If no sound is present make sure you have previously closed other applications dealing with sound (e.g. media players, web browsers, etc.) and that system sound settings are properly set. NOTE: on Windows Vista/7 it is advised to not to use ASIO drivers on Pure Data, otherwise MIDI sound won't work. For further information about Pure Data check Pd documentation.

  3. Set “Delay” to the minimum possible value before you get sound defects. You can also check if the microphone works properly by selecting “Microphone”.

PD configuration dialogue
Figure: PD configuration dialogue screenshot

5. Customization

SITPLUS allows different ways of customization ranging from creating new musical scores or graphical collages to developing your own components.

What can be customized?

6. Information for developers

SITPLUS is built around a core C++ library (spcore) which provides basic services such as module loading, component instantiation, basic types and internalization among others, and base classes to develop new components.

The basic building block is the component (IComponent interface). A component is a processing unit which provides input and/or output pins and an optional UI panel. Components can be arranged in a composite and its pins can be connected to build data-flow oriented applications.

Along with with the core, two helper libraries are also provided:

Finally, several modules are also provided:

One of the goals of SITPLUS is to become GUI toolkit agnostic. Currently it uses wxWidgets and most modules provide UI panels based on this toolkit, but the core itself has not an strong dependency (component's method GetGUI expects wxWindows types though this is easily changeable by just defining a neutral window type, the core itself provides methods such as InitGUISupport, RunMessageLoop and CleanupGUISupport which are based on wx).

(This section is a draft and needs to be extended).

7. Project origin

SITPLUS was born as a research project in the Cerebral Palsy Centre APPC of Tarragona (Spain) in collaboration with CREA Software Systems and others. SITPLUS is the result of more than three years of research, development and direct intervention with people with cerebral palsy in the APPC. Inspired by the cause and effect applications, SITPLUS provides a tool for continuous and remote interaction, attainable to the majority of people with multiple disabilities. It was conceived in the APPC in late 2007 to try to provide a tool to the group of people whose combination of motor, cognitive and even sensory limitations prevented them from accessing other commonly used tools. Experimental sessions, conducted in the laboratory that the APPC has devoted to this project, show that this tool not only promotes participation, engagement and play for many people with moderate to severe impairments, but is also very motivating for users with mild impairments.

SITPLUS laboratory in the APPC

Although SITPLUS focuses on people with cerebral palsy, we believe that it could benefit other people with cognitive disabilities. Moreover, it can also be useful as a platform for game or interactive music and visual arts research. Indeed, SITPLUS is conceived as an application framework in which several activities using resources run. We call resources things like input sound analysis, computer vision based motion tracking, drawing and sound generation algorithms, and so on. We call activities these pieces of software that employ resources to do something useful (e.g. let someone interact in some way). The idea behind this model is to allow reusing those components to provide rich interactive scenarios whilst keeping simple its further development and usage.