NAME

gensai - generate stabilised auditory image

CONTENTS

Synopsis/syntax
Description
Strobed Temporal Integration
Options
     I. Display Options For The Auditory Image
     Ii. Storage Options For The Auditory Image
     Iii. Options For The Auditory Image
Examples
References
Files
See Also
Copyright
Acknowledgements

gensai [ option=value | -option ] filename

Periodic sounds give rise to static, rather than oscillating,perceptions indicating that temporal integration is applied to the NAPin the production of our initial perception of a sound -- our auditoryimage (Patterson et al., 1992b). Traditionally, auditory temporalintegration is represented by a simple leaky integration process, andAIM provides a bank of lowpass filters to enable the user to generateauditory spectra, excitation patterns (see Patterson, 1994a; genasaand genepn), and auditory spectrograms (see Patterson et al., 1992a,1993; gensgm and gencgm). However, leaky integrators remove thephase-locked fine structure observed in the NAP, and this conflictswith perceptual data indicating that the fine structure plays animportant role in determining sound quality and source identification(Patterson, 1994b; Yost et al., 1996; Patterson et al., 1996). As aresult, AIM includes two modules which preserve much of thetime-interval information in the NAP during temporal integration, andwhich produce a better representation of our auditory images. Thefunctional version of AIM uses a form of Strobed Temporal Integration(STI) (Patterson et al., 1992a,b), and this is the primary topic ofthis manual entry.

In the physiological version of AIM, the auditory image is constructedwith a bank of autocorrelators and the multi-channel result isreferred to as a ’correlogram’ (Lyon, 1984; Slaney and Lyon, 1990;Meddis and Hewitt, 1991). The correlogram module is an aimTool ratherthan an integral part of the main program ’gen’. The name of the toolis ’acgram’ and there are man pages for all the tools. An extendedexample involving correlograms is presented in the script ’gtmdcg_pk’.The correlogram module extracts periodicity information and preservesintra-period fine structure by autocorrelating the function in eachchannel of the NAP. It was originally introduced as a model of pitchperception by Licklider (1951). It is not yet known whether STI orautocorrelation is more realistic, or more efficient, as a means ofsimulating our perceived auditory images. At present, the purpose isto provide a software package that can be used to compare theseauditory representations in a way not previously possible.

There are a large number of Silent Options associated with printingAIM displays (see docs/aimSilentOptions). They are particularly usefulwhen printing correlograms, summary correlograms, and summary auditoryimages, where some of the default axes and labels are incorrect.Examples of how to use these silent options are presented indocs/aimR8demo.

STROBED TEMPORAL INTEGRATION

                                                                .

In strobed temporal integration (STI), a bank of delay lines is usedto form a buffer store for the NAP, one delay line per channel. Asthe NAP proceeds along the buffer, it decays linearly with time, atabout 2.5 %/ms, so there is no activity beyond about 40 ms in the NAPbuffer. Each channel of the buffer is assigned a strobe unit whichmonitors activity in that channel looking for local maxima in thestream of NAP pulses. When one is found, the unit initiates temporalintegration in that channel; that is, it transfers a copy of theentire NAP function in that channel at that instant, to thecorresponding channel of an image buffer, where it adds the NAPfunction point-for-point with whatever is already in that channel ofthe image buffer. The local maximum itself is mapped to the 0-mspoint in the image buffer. The multi-channel version of this STIprocess is AIM’s representation of our auditory image of a sound.Periodic and quasi-periodic sounds typically produce a single localmaximum per cycle, per channel of the NAP. In any given channel, thisleads to regular strobing and the transfer into the auditory image ofa sequence of NAP functions which are all virtually identical. As aresult, the auditory images of periodic sounds lead to static auditoryimages, and quasi-periodic sounds lead to nearly static images. Theseimages, however, have the same temporal resolution as the NAP.Dynamic sounds are represented as a sequence of auditory imageframes. If the rate of change in a sound is not too rapid, as isdiphthongs, features are seen to move smoothly as the sound proceeds,much as objects move smoothly in animated cartoons.

The primary difference between the auditory images produced with STIand autocorrelation is that STI preserves much more of the temporalasymmetry that sounds generate in the NAP (Allerhand and Patterson,1992). Indeed the features that appear in correlograms are virtuallysymmetric. The preservation of asymmetry in STI is heavily dependenton the criterion that the stobe mechanism uses to identify points atwhich to initiate temporal integration. If it successfully identifieslocal maxima as the integration points, asymmetry is preserved(Allerhand and Patterson, 1992). If it initiates temporal integrationon every pulse in the NAP, the features lose their asymmetry and theresulting image is quite similar to the corresponding correlogram. Adetailed description of the STI process is presented in/docs/aimStrobeCriterion with examples provided by a companion script(/scripts/aimStrobeCriterion). The discussion of strobe criteriabegins with the simplest criterion for initiating temporal integration-- strobe on every non-zero NAP point. This converts sharp,asymmetric NAP features into rough-edged, largely-symmetric featuresin the auditory image. From there the discussion proceeds to morerestrictive criteria which gradually reduce the rough edges andrestore the asymmetry of the features in the auditory image. Thedegree of restriction in the strobe criterion is an option in AIM(stcrit_ai) which enables the user to experiment with the relationshipbetween STI and autocorrellation.

It is important to emphasise, that the strobing in a given channel isindependent of that in all other channels so far as the mechanismitself is concerned. It is this aspect of the strobing process, andthe fact that the local maximum is mapped to 0 ms in the auditoryimage, that causes the alignment of channels in the auditory image.This passive alignment of channels in turn enables AIM to explainmonaural phase perception as set out in Patterson (1987).

The auditory image has the same vertical dimension as the neuralactivity pattern (filter centre frequency). The continuous timedimension of the neural activity pattern becomes a local,time-interval dimension in the auditory image; specifically, it is"the time interval between a given pulse and the succeeding strobepulse". In order to preserve the direction of asymmetry of featuresthat appear in the NAP, the time-interval origin is plotted towardsthe right-hand edge of the image, with increasing, positive timeintervals proceeding to towards the left.

                                                                .

                                                                .

The options that control the positioning of the window in which theauditory image appears are the same as those used to set up theearlier windows (i.e. those with the suffix _win), as are the optionsthat control the level of the image within the display (top, bottomand magnitude). In addition, there are three new options that arerequired to present this new auditory representation. The options arefrstep_aid, pwidth_aid, and nwidth_aid; the suffix "_aid" means "auditoryimage display". These options are described here before the optionsthat control the image construction process itself, as they occurfirst in the options list. There are also three extra display optionsfor presenting the auditory image in its spiral form; these optionshave the suffix "_spd" for "spiral display"; they are described in themanual entry for ’genspl’.

frstep_aid	The frame step interval, or the update interval for the auditory image display Default units: ms. Default value: 16 ms. Conceptually, the auditory image exists continuously in time. Thesimulation of the image produced by AIM is not continuous, however;rather it is like an animated cartoon. Frames of the cartoon arecalculated at discrete points in time, and then the sequence of framesis replayed to reveal the dynamics of the sound, or the lack ofdynamics in the case of periodic sounds. When the sound is changingat a rate where we hear smooth glides, the structures in the simulatedauditory image move much like objects in a cartoon. frstep_aiddetermines the time interval between frames of the auditory imagecartoon. Frames are calculated at time zero and integer multiples offrstep_aid. The default value (16 ms) is reasonable for musical sounds and speechsounds. For a detailed examination of the development of the image ofbrief transient sounds frstep_aid should be decreased to 4 or even 2ms.
pwidth_aid	The maximum positive time interval presented in the display of theauditory image (to the left of 0 ms). Default units: ms. Default value: 35 ms.
nwidth_aid	The maximum negative time interval presented in the display of theauditory image (to the right of 0 ms). Default units: ms. Default value: -5 ms. Note that the minus sign is required when entering nwidth_aid.
animate	Present the frames of the simulated auditory image as a cartoon. Switch. Default off. With reasonable resolution and a reasonable frame rate, the auditorycartoon for a second of sound will require on the order of 1 Mbyte ofstorage. As a result, auditory cartoons are only stored at thespecific request of the user. When the animate flag is set to ‘on’,the bit maps that constitute the frames of the auditory cartoon arestored in computer memory. They can then be replayed as an auditorycartoon by pressing ‘carriage return’. To exit the instruction, type"q" for ‘quit’ or "control c". The bit maps are discarded unlessoption bitmap is set to on. There is a silent option ’review’ associated with animate. Whenreview=on, it which causes AIM to pause between the frames of thecartoon and wait for a ’return’.

                                                                .

A record of the auditory image can be stored in two ways depending onthe purpose for which it is stored. The actual numerical values ofthe auditory image can be stored as previously, by setting output=on.In this case, a file with a .sai suffix will be created in accordancewith the conventions of the software. These values can be recalledfor further processing with the aimTools. In this regard the SAImodule is like any previous module.

It is also possible to store the bit maps which are displayed on thescreen for the auditory image cartoon. The bit maps require lessstorage space and reload more quickly, so this is the preferred modeof storage when one simply wants to view the auditory image.

bitmap

Produce a bit-map storage file Switch. Default value: off. When the bitmap option is set to ‘on’, the bit maps are stored in afile with the suffix .ctn. The bitmaps are reloaded into memory usingthe instructions review, or xreview, followed by the file name withoutthe suffix .ctn. The auditory image can then be replayed, as withanimate, by typing ‘carriage return’. xreview is the newer andpreferred display routine. It enables the user to select subsets ofthe cartoon and to change the rate of play via a convenient controlwindow. It does, however, require an ANSI C compiler like gcc.

                                                                .

There are six options with the suffix "_ai", short for ’auditoryimage’. The first option, napdecay_ai, controls the decay rate forthe NAP while it flows down the NAP buffer and before it istransferred to the auditory image. In point of fact, then, napdecay_aiis a NAP option rather than and auditory image option. But its effectsare only observed in the auditory image and so it is grouped with theother _ai options. The next four options control the STI process --stdecay_ai, stcrit_ai, stlag_ai and decay_ai. The final option,stinfo_ai, is a switch that causes the software to produce informationabout the current STI analysis for demonstration or diagnosticpurposes.

The strobe mechanism is conceptually simple. An adaptive strobethreshold is set up for each channel of the NAP and its value is setto the height of the first NAP pulse when it occurs. Thereafter, thestrobe threshold decays exponentially with time in the absence ofsuprathreshold NAP pulses. The rate of decay is an option, decay_ai,which is specified as the half life of the image; the default halflife is 30 ms. When a NAP pulse next exceeds the strobe threshold,the level of the strobe threshold is reset to the height of the peakof the NAP pulse, and the time of the peak of the NAP pulse isrecorded as a potential temporal integration time. There is then ashort lag before integration, to see if another larger NAP pulse isabout to occur, since the production of a stable images depends onidentifying local maxima in the NAP. The strobe lag is an option,stlag_ai, whose default value is 5 ms. If a larger NAP pulse occurswithin stlag_ai ms, its peak time becomes the new potentialintegration time, the strobe threshold level is reset to the heightof the peak of the new NAP pulse, and the strobe lag is reset tostlag_ai. In the event of a continuous stream of rising NAP pulses,the total strobe lag after the occurance of the first suprathresholdNAP pulse is limited to twice stlag_ai.

napdecay_ai	Decay rate for the neural activity pattern (NAP) Default units: %/ms. Default value 2.5 %/ms. napdecay_ai determines the rate at which the information in the neuralactivity pattern decays as it proceeds along the auditory buffer thatstores the NAP prior to temporal integration.
stdecay_ai	Strobe threshold decay rate Default units: %/ms. Default value: 5 %/ms. stdecay_sai determines the rate at which the strobe threshold decays.At 5 %/ms, strobe threshold returns to zero from a NAP peak of anyheight in 40 ms, in the absence of further suprathreshold NAP pulses.Note that, in absolute terms, strobe threshold decays faster afterlarge NAP peaks than after small NAP peaks.

This Section presents a pair of examples intended to illustrate thepredominant forms of motion that dynamic sounds produce in theauditory image, and the fact that structures and features can betracked across the image provided the rate of change is not excessive.The first example is a pitch glide for a note with fixed timbre; itproduces predominantly horizontal motion in the auditory image. Thesecond example is a timbre glide for a note with fixed pitch; itproduces predominantly vertical motion in the auditory image.

To this point, the discussion has focussed on how to convert a NAPfrom a periodic sound with a repeating pattern into a stabilisedauditory image without smearing the fine structure of the NAP pattern.The mechanism is not, however, limited to periodic sounds. The soundfile cegc contains a set of click trains that produce four musicalnotes referred to as C3, E3, G3, and C4, along with glides from onenote to the next. The notes are relatively long (300 ms) and thepitch glides are relatively slow (300 ms for 3-5 semitones). As aresult, each note forms a stabilised auditory image and there issmooth motion in the image over the 300-ms interval as the soundglides from one note to the next. The pitch of musical notes isdetermined by the lower harmonics when they are present and so thefrequency range is limited to 2000 Hz. The demonstration is generatedand stored with the instruction

> gensai channels=40 max=2000 input=cegc bitmap=on

It can then be replayed at will with either ’review cegc’ or ’xreviewcegc’. (Click on the image with the middle mouse button to pull upthe control window for xreview.) For brevity, the example can belimited to the transition from C to E near the start of the sound usingthe instruction

> gensai channels=40 max=2000 start=150 length=600 input=cegc

(In point of fact, the click train associated with the first note hasa period of 8 ms; so this "C" is actually a little below the musicalnote C3.)

When the note comes on, a stable image of the first note forms overthe first 4-6 cycles of the note. The vertical structure that repeatsfour times across the image is the time-interval pattern thatidentifies a click-train sound. When the transition begins, in thelower channels associated with the first and second harmonic, theindividual SAI pulses move from left to right. At thesame time, they move up in frequency as these resolved harmonicsmove up into filters with higher centre frequencies. In these lowchannels the motion is relatively smooth because the SAI pulses have aduration which is a significant proportion of the period of the sound.As the pitch rises and the periods get shorter, each new NAP cyclecontributes a NAP pulse which is shifted a little to the rightrelative to the corresponding SAI pulse. This increases the leadingedge of the SAI pulse without contributing to the lagging edge. As aresult, the leading edge builds, the lagging edge decays, and the SAIpulse moves to the right. The SAI pulses are asymmetric during themotion, with the trailing edge more shallow than the leading edge, andthe effect is greater towards the left edge of the image because thediscrepancies over four cycles are larger than the discrepancies overone cycle. The effects are larger for the second harmonic than forthe first harmonic because the width of the pulses of the secondharmonic are a smaller proportion of the period. During the pitchglide the SAI pulses have a reduced peak height because the activityis distributed over more channels and over time intervals.

The SAI pulses associated with the higher harmonics are relativelynarrow with respect to the changes in period during the pitch glide.As a result there is more blurring of the image during the glide inthe higher channels. Towards the right-hand edge, for the column thatshows correlations over one cycle, the blurring is minimal. Towardsthe left-hand edge the details of the pattern are blurred and we seemainly activity moving in broad vertical bands from left to right.When the glide terminates the fine structure reforms from right toleft across the image and the stationary image for the note E appears.

The details of the motion are more readily observed when the image isplayed in slow motion, using review or xreview and one of the ’slowdown’ options.

The vowels of speech are quasi-periodic sounds and the period for theaverage male speaker is on the order of 8ms. As the articulatorschange the shape of the vocal tract during speech, formants appear inthe auditory image and move about. The position and motion of theformants is an important part of the information conveyed by thevoiced parts of speech. When the speaker uses a monotone voice, thepitch remains relatively steady and the motion of the formants isessentially in the vertical dimension. An example of monotone voicedspeech is provided in the file leo which is the acoustic waveform ofthe word ’leo’. The auditory image of leo can be produced and viewedwith the instruction

> gensai input=leo bitmap=on animate=on

It can be replayed under user control with either

> review leo    or

> xreview leo

The dominant impression on first observing the auditory image ofleo is the motion in the formation of the "e" sound, thetransition from "e" to "o", and the formation of the "o" sound.

The vocal chords come on at the start of the "l" sound but the tip ofthe tongue is pressed against the roof of the mouth just behind theteeth and so it restricts the air flow and the start of the "l" doesnot contain much energy. As a result, in the auditory image, thepresence of the "l" is primarily observed in the transition from the"l" to the "e". That is, as the three formants in the auditory imageof the "e" come on and grow stronger, the second formant glides intoits "e" position from below, indicating that the second formant wasrecently at a lower frequency for the previous sound. The details ofthe motion are more readily observed when the image is played in slowmotion, using review or xreview and one of the ’slow down’ options.

In the "e", the first formant is low, centred on the thirdharmonic at the bottom of the auditory image. The second formantis high, up near the third formant. The lower portion of thefourth formant shows along the upper edge of the image. Recognition systems that ignore temporal fine structure oftenhave difficulty determining whether a high frequencyconcentration of energy is a single broad formant or a pair ofnarrower formants close together. This makes it more difficultto distinguish "e". In the auditory image, information about thepulsing of the vocal chords is maintained and the temporalfluctuation of the formant shapes makes it easier to distinguishthat there are two overlapping formants rather than a singlelarge formant.

As the "e" changes into the "o", the second formant moves backdown onto the eighth harmonic and the first formant moves up toa position between the third and fourth harmonics. The third andfourth formants remain relatively fixed in frequency but theybecome softer as the "o" takes over. During the transition, thesecond formant becomes fuzzy as it moves down the verticalridges of the glottal pulse.

FILES

.gensairc	The options file for gensai.

genspl, acgram

Copyright (c) Applied Psychology Unit, Medical Research Council, 1995

Permission to use, copy, modify, and distribute this software withoutfee is hereby granted for research purposes, provided that thiscopyright notice appears in all copies and in all supportingdocumentation, and that the software is not redistributed for any fee(except for a nominal shipping charge). Anyone wanting to incorporateall or part of this software in a commercial product must obtain alicense from the Medical Research Council.

The MRC makes no representations about the suitability of thissoftware for any purpose. It is provided "as is" without express orimplied warranty.

THE MRC DISCLAIMS ALL WARRANTIES WITH REGARD TO THIS SOFTWARE, INCLUDINGALL IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS, IN NO EVENT SHALLTHE A.P.U. BE LIABLE FOR ANY SPECIAL, INDIRECT OR CONSEQUENTIAL DAMAGESOR ANY DAMAGES WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS,WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION,ARISING OUT OF OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THISSOFTWARE.

The AIM software was developed for Unix workstations by JohnHoldsworth and Mike Allerhand of the MRC APU, under the direction ofRoy Patterson. The physiological version of AIM was developed byChristian Giguere. The options handler is by Paul Manson. The revisedSAI module is by Jay Datta. Michael Akeroyd extended the postscriptfacilites and developed the xreview routine for auditory imagecartoons.

The project was supported by the MRC and grants from the U.K. DefenseResearch Agency, Farnborough (Research Contract 2239); the EEC EspritBR Porgramme, Project ACTS (3207); and the U.K. Hearing Research Trust.