Academic Year 2019-2020

AUDITORY AND TACTILE INTERACTIONS

Teachers

Federico Fontana
Unit Credits
6
Teaching Period
Second Period
Course Type
Caratterizzante
Prerequisites. Useful for attending the course is a knowledge of human-machine interaction, basic math (especially, logarithms, exponential, derivatives, integrals, arithmetic and geometric sequences, functions), basic physics (inertial systems, wave propagation) and basic computer science (computation, algorithm, basic constructs of programming).
Teaching Methods. The course includes:

– didactic activity aimed at teaching the psychology of auditory and tactile perception, spectral analysis, the systems for the synthesis of sound and vibration;

– laboratory activities aimed at developing the students’ ability to apply their knowledge on real application cases regarding the auditory and tactile phenomena, the definition and inspection of discrete signals, the Fourier transform, the implementation and evaluation of convolutions, the realization of filters and the use of standard computer functions for the sound.

Concepts are exposed from the blackboard, and by projecting the activity at the computer in the case of the laboratory lesson. The computer is also used to show text pages and other sources related to the topic on display. Teaching excludes the presentation of commented transparencies by the teacher.

Students can choose to take the examination with emphasis given to the aspects related to the creation of interfaces for sound and vibrations. In that case, they are guided to develop a theme linked to the program in independence. In particular the teacher during the course presents research in which he is involved, and suggests fonts to start from when the assignment falls within this research.

Verification of Learning. If a student does not complete the assignment, the exam consists of an oral in which he/she is asked to sketch an audio-tactile interface project; during the sketch, the student is posed further questions depending on how the sketching activity is developing. If the student completes an assignment, the exam consists of an evaluation of a written work and/or prototype followed by a discussion on its main aspects. In any case all students are required to know the course content.
More Information. Teaching takes place in two locations linked by a distance learning platform. The teacher balances his presence in the two locations and transmits the lessons from remotely to either site.

The educational material and video recordings of classroom lectures are made available on the E-learning platform of the University of Udine. Such materials are reserved for students enrolled in the course. Through the platform, the teacher provides students also laboratory exercises and sources for assignment.

The dissertation activities connected with the teaching normally provide to assist the teacher in the research on the interaction on acousto-haptic and multi-modal interfaces, around an educational theme and with difficulty proportional to the time and resources of the student.

Objectives
After completing the course, students will master some simple sound and vibration processing methods, with specific regard to real time synthesis. In particular, they will:

– have learned the fundamentals of psychology of auditory and tactile perception;

– know the nature of continuous-time and discrete signals and systems;

– recognize and evaluate the spectrum of a discrete-time signal;

– be aware of conventional techniques for sound synthesis;

– have sufficient knowledge to evaluate and design simple software and hardware systems for the synthesis of sounds and vibrations;

– have understood the basic principles of the operation of the software for the synthesis of the sound in real time, on which they will elaborate examples of realizations of models for the synthesis of the sound and vibration.

The expected learning outcomes are, as reflected in the following Dublin Descriptors.

(Skills related to the disciplines)

– Knowledge and understanding: Students acquire basic knowledge about sound and vibrations in their fundamental components of perception, mathematics, and operation. The same knowledge are applied to real-time synthesis on the computer.

– Ability to apply knowledge and understanding: Through the solution of a sufficient number of exercises, the students understand the difficulty of designing and subsequently realize software architectures and hardware for the

generation of sound and vibrations in real-time applications.

(Side skills / soft skills)

– Making judgments: Due to sound processing as part of the broader signals and systems theory field, the student at the end of the course will have acquired a deeper autonomy in judging the design and performance quality of the real-time signal processing software fundamental components.

– Communication skills: At the end of the course students will have knowledge of the most important results related to the audio-tactile perception of the signals, and simultaneously give a meaning to the terminology related to the techniques and technologies for the rendering of auditory and tactile signals, used a lot also in communication engineering: decibels, spectrum, bandwidth, frequency response just to mention a few terms. Consequently, the student will have future opportunities to interact with diverse professionals, typically possessing creative or technical skills, who are active in the realization of non-visual interface component.

– Learning ability: Based on the learned concepts, students will be able to explore a variety of issues in the psychophysics of auditory and tactile perception, the design of audio-tactile interfaces, and the design of software for rendering of sound and vibration.

Contents
The course includes a series of lectures accompanied by practical and specific application examples on the main issues related to the synthesis of sound and vibration.

1. Basic Design – Fundamentals. Background. Methodologies. The non-visual interface. Synestesia

2. Psychology of auditory perception – The auditory system. Intensity; Loudness. Psychophysical maps. Fechner’s hypothesis, Weber’s law. Pitch. Masking; Critical bands. Spatial perception.

3. Psychology of tactile perception – The somatosensory system: kinesthetic perception and touch; types of receptors and associated sensitivities. Perception of vibration: summation phenomena, masking, adaptation, enhancement. Bone conduction.

4. Introduction to Spectral Analysis – Elements of trigonometry, algebra of complex numbers, integral functions. Continuous-time signals. Meaning of the Fourier transform. Sampling. Discrete-Time Fourier Transform: spectrum of a signal and its interpretation. Sampling, windowing, temporal discretization, frequency bins. Discrete Fourier transform: definition, resolution in time and frequency; invertibility of the Fourier matrix. Fast Fourier Transform (FFT). Spectrograms.

5. Discrete time systems (qualitative aspects) – linearity, time-invariance, impulse response, convolution. Stability. Transfer Function (TF): frequency response,

amplitude response. Notion of numerical filter. Composition of series and parallel filters. Delay lines. TF characteristics. Filter order. Interpretation of the amplitude response. FIR filters: example of the impulse response of finite duration; characteristics of the FIR filter transfer. IIR filters: example of the impulse response of infinite duration; characteristics of the IIR filter transfer. Computationand evaluation of the amplitude and phase response of simple FIR and IIR filters in practice.

6. Synthesis of sound and vibration (qualitative aspects) – IIR Resonators: characteristics; formalization of TF’s; normalization of the response; amplitude response; selectivity; fading of the transient. “comb” FIR and IIR filters: characteristics and use. Allpass comb-filter. Acoustic reverberation: linear model; physical and perceptive approach; measurement of characteristic of acoustic parameters: first reflections, late reflections, decay; modal description; statistical description. Realization of oscillators and digital

wavetable. Additive synthesis, ADSR envelope. Subtractive synthesis. Realization of filters in the Puredata and Matlab environment. Synthesis of vibrations:

typical patterns; resonance control.

7. State of the art technologies for the presentation of sound and vibrations – Loudspeakers, piezoelectric transducers and their use in diffusing or focalizing the acousto-tactile response of a non-visual interface.

During the course laboratory moments guided by the teacher will be planned.

1. Introduction to the Puredata environment – Understanding visual patches for the reproduction and listening of perceptual auditory phenomena.

2. Introduction to the Matlab audio environment – Understanding scripts for the reproduction and listening of perceptual auditory phenomena.

3. Introduction to the Unity audio environment – Understanding of sound modules and their use for vibration pattern generation.

4. Free software in Linux environment – Use of Jack for routing multichannel audio streams in the simultaneous use of Linux audio applications such as Puredata, Audacity, spectral analyzers, synthesizers, MIDI sequencers.

Texts
Davide Rocchesso, Introduction to Sound Processing, ed. Mondo Estremo, 2003, ISBN 8890112611. Matjaž Mihelj, Janez Podobnik, Haptics for Virtual Reality and Teleoperation, Springer, 2012, ISBN 9400757174. Selection of scientifis papers. Web manuals for Puredata and Unity.