SITPLUS is a free software framework whose main goal is to provide recreational activities for people with multiple disabilities. It offers new forms of interaction based on computer vision, voice and other peripherals to produce a result in the form of image and sound. Inspired by the cause and effect applications, SITPLUS provides a tool for continuous and remote interaction, attainable to the majority of people with multiple disabilities.
Dual core processor running at 1.5Ghz or higher.
Sound card and speakers.
Microphone (optional, for voice based interaction).
Web camera (high quality webcam recommended).
Wii Remote + Bluetooth adapter (optional, for Wii Remote based interaction).
(Optional but recommended.) Good quality sound system and secondary display or projector.
Pure Data (pd)
Supported operating systems:
Microsoft Windows XP SP3/Vista/7
GNU/Linux
You can download the latest SITPLUS version from SITPLUS website. A Windows installation package is provided. For GNU/Linux you might need to fetch the sources and compile them.
Before installing SITPLUS first uninstall any previous version.
Install Pure Data (pd).
Windows. Download and install with the default options Pd-extended
GNU/Linux. Major distributions provide pre-compiled pd packages. Unfortunately, most pd packages do not provide all required components (externals in pd jargon) which are required by SITPLUS and thus voice based activities are currently unsupported.
Install SITPLUS.
Before running SITPLUS make sure that the required devices such as sound input (i.e. a microphone) and output, web camera, Wii Remote, etc. are already installed and work properly. Set also the system sound volume (both for recording and playback) accordingly.
Once started check the configuration options under the 'Configuration' menu. To use a Wii Remote you must pair it with the computer, check Wii Remote Configuration for further details. Check also how to configure Pure Data. Pure Data Configuration.
Currently, activities can be started either by choosing 'Start…' or 'Load file…' under the 'Activity' menu. The former option allows to select which input channels to use (camera, voice and/or Wii Remote) before starting one of the built-in the activities. The later option allows to load a .sp file (some .sp files are provided under sp/ folder).
Although SITPLUS is conceived as a framework which allows for several ways of customization, it currently provides some built-in activities which combine different input methods (camera tracker, voice input and Wii Remote based motion sensing) with real-time MIDI sound generation and a graphical collage.
The screenshot depicts an activity which combines the use of the three input channels is shown. Using this activity the user can play a MIDI musical score though the motion picked up using the camera and/or a wiimote. Motion is also use to make the graphical collage evolve. At the same time, the user's voice is also picked up, processed using the chosen effects and played it back to the loudspeakers. Moreover, user's voice also makes the collage evolve.
The window is organized in three main areas:
The left area holds input channel options organized in a notebook.
The central area shows the MIDI sequencer.
The right area holds the graphical output controls.
To be able to connect a Wii Remote device (aka wiimote) - currently the standard Wii Remote (also with Motion Plus) and the Balance Board are supported - you need a bluetooth adapter. The configuration of such device is beyond the scope of this manual.
On Windows you first need to pair you wiimote with the computer, see Pairing the wiimote.
To check if the wiimote is properly detected access Configuration → Wii Remotes and press the “Reconnect” button (on GNU/Linux you might need to simultaneously press buttons “1” and “2” on the wiimote before reconnecting). Once connected the wiimote will rumble and the dialogue will be updated.
The basic steps are:
Open the Bluetooth Device configuration. Depending on your Windows version you will find the Bluetooth icon in the Windows control panel or in the task bar. A list containing the currently paired devices will appear.
Remove any previously paired device.
Select “Add new device” to start searching for available Bluetooth devices nearby.
Simultaneously press buttons “1” and “2” on the wiimote (LEDs will start flashing, the number of flashing LEDs also tells you the state of the batteries).
Select the newly detected device and pair it without using codes.
If the process succeeded, LEDs will keep flashing. Note that after a system reboot you need to repeat this process again.
Sometimes (if the wiimote was not properly shut down) you might need to reset it by removing the batteries and pressing “1” and “2” before putting them into again.
Open the audio configuration dialogue (Configuration → Pure Data).
Once opened you should hear a continuous sound. If no sound is present make sure you have previously closed other applications dealing with sound (e.g. media players, web browsers, etc.) and that system sound settings are properly set. NOTE: on Windows Vista/7 it is advised to not to use ASIO drivers on Pure Data, otherwise MIDI sound won't work. For further information about Pure Data check Pd documentation.
Set “Delay” to the minimum possible value before you get sound defects. You can also check if the microphone works properly by selecting “Microphone”.
SITPLUS allows different ways of customization ranging from creating new musical scores or graphical collages to developing your own components.
What can be customized?
Musical scores. The scores/ directory contains musical scores that the currently provided activities use. To add new scores simply copy the file containing the new score to this directory. The score file format is quite simple. It consist on a text file where each line contains up to 4 notes (each note is represented by its MIDI number). Optionally the first line might contain the word percussion to select the percussion kit.
Graphical collages. The graphics/mod_collage_xml/ directory contains a collection of collage definition files. These are XML files which describe the behaviour and contents of the graphical collages. See Structure of XML file for mod_collage for a detailed description. Under graphics/mod_collage_xml/ there are two folders:
activities/: it holds the activities which will be draw in the foreground.
backgrounds/: contains the background definitions.
Activities. The activity definitions can be found under the sp/ directory. It contains scripts whose format is specified in SITPLUS script syntax.
SITPLUS is built around a core C++ library (spcore) which provides basic services such as module loading, component instantiation, basic types and internalization among others, and base classes to develop new components.
The basic building block is the component (IComponent interface). A component is a processing unit which provides input and/or output pins and an optional UI panel. Components can be arranged in a composite and its pins can be connected to build data-flow oriented applications.
Along with with the core, two helper libraries are also provided:
sphost: provides some helper functions and classes to develop final applications. It provides, for instance, a script parser to ease the creation of component composites.
widgets_base: provides UI base classes
Finally, several modules are also provided:
mod_camera: components and types to deal with camera capture.
mod_io: components for file I/O.
mod_midi: components and types for MIDI output.
mod_sdl: components and types to draw on SDL surfaces.
mod_vision: components for computer vision.
mod_widgets: UI components such as buttons, combo boxes, etc.
mod_wiimotes: components and types to use Wii remotes
One of the goals of SITPLUS is to become GUI toolkit agnostic. Currently it uses wxWidgets and most modules provide UI panels based on this toolkit, but the core itself has not an strong dependency (component's method GetGUI expects wxWindows types though this is easily changeable by just defining a neutral window type, the core itself provides methods such as InitGUISupport, RunMessageLoop and CleanupGUISupport which are based on wx).
(This section is a draft and needs to be extended).
SITPLUS was born as a research project in the Cerebral Palsy Centre APPC of Tarragona (Spain) in collaboration with CREA Software Systems and others. SITPLUS is the result of more than three years of research, development and direct intervention with people with cerebral palsy in the APPC. Inspired by the cause and effect applications, SITPLUS provides a tool for continuous and remote interaction, attainable to the majority of people with multiple disabilities. It was conceived in the APPC in late 2007 to try to provide a tool to the group of people whose combination of motor, cognitive and even sensory limitations prevented them from accessing other commonly used tools. Experimental sessions, conducted in the laboratory that the APPC has devoted to this project, show that this tool not only promotes participation, engagement and play for many people with moderate to severe impairments, but is also very motivating for users with mild impairments.
Although SITPLUS focuses on people with cerebral palsy, we believe that it could benefit other people with cognitive disabilities. Moreover, it can also be useful as a platform for game or interactive music and visual arts research. Indeed, SITPLUS is conceived as an application framework in which several activities using resources run. We call resources things like input sound analysis, computer vision based motion tracking, drawing and sound generation algorithms, and so on. We call activities these pieces of software that employ resources to do something useful (e.g. let someone interact in some way). The idea behind this model is to allow reusing those components to provide rich interactive scenarios whilst keeping simple its further development and usage.