A website devoted to exploring accessibility at the intersection of technology and rhetoric. What began in 2008 as a space to explore accessible podcasting has turned into an ongoing reflection on the rhetoric of closed captioning. I welcome your feedback.
Sound effects can be tricky to caption. There are at least three challenges. First, there’s the problem of deciding which sounds need to be captioned. Distinguishing significant from insignificant sounds is not always simple. Only significant sounds should be captioned because there’s just not enough space or time to caption every sound. Captioning insignificant sounds — that is, sounds in the background or incidental to the plot — can confuse, distract, or annoy viewers who rely on captions for meaning. Hearing viewers are able to compartmentalize background sounds so that they stay in the background of their awareness. Caption viewers don’t have the same luxury when every sound on the caption track, regardless of whether it came from the foreground or background, is seen as equally “loud.”
For those sounds deemed significant, a second challenge is to determine which word(s) will sufficiently describe the sound in closed caption form. Does the captioner use a description of the sound (e.g. “[exclaiming in pain]“) or a more direct phonetic transcription (e.g. “[ahhhh]“). While Captioning Key suggests using both “description and onomatopoeia” (“If the presentation rate permits”), the typical approach in DVD captioning is to include one or the other for any given sound, but never both at once. (Captioning Key uses “[engine idling]/rrrrr” as an example of both at once.)
Third, sound effects need to be captioned from a position of deep awareness of the entire movie or TV series, not in isolation. If a sound-theme is repeated throughout a show or series, it may need to be captioned the same way for each occurrence, especially if it’s doing the work of building thematic resonances. Caption the series, not the episode is a rallying cry I’ve used recently to suggest the importance of having a deep knowledge of those thematic sounds that cross episodic boundaries or link scenes together.
The hurt knee gag and series awareness
Running gags usually have both visual and sonic elements. Hearing viewers see and hear them. The question — and one I’m still grappling with — is whether a series gag that depends on repetitive visual and sonic elements, and spans multiple episodes, should be captioned the same way each time it occurs. Even though instances of the gag will not be identical to each other, they will still share recurring elements, including recurring sounds. If the gag depends on repetitive sounds, those sounds may need to be captioned in the same way. If the content producers took pains to duplicate visual elements and sequences, the same pains must be taken at the caption layer.
Consider the Peter Hurts His Knee gag on Family Guy. According to the Family Guy Wikia, the gag has to date appeared five times over four episodes. What interests me is how each instance of the knee gag is captioned. For this analysis, I gathered together the gag examples from the DVD releases of these four episodes. For each episode, I looked at the bitmap and text captions (since the DVDs contained both types). If there were differences, I included them in my review:
Placing the five instances of the gag side-by-side allows us not only to compare the choices made in each instance but to ask questions about those choices and the approaches to captioning that inform them. When describing the repetitive, two-part hurt knee sound, which words are most appropriate: Exclaiming, Howling, Groaning, Whimpers, Wincing, or something else? Should a more phonetic substitute be used instead, such as “Ahh!” or even “Ssssssss”? Does every instance of the sound need to be captioned, or is one caption sufficient? When do captions need to cite the cultural touchstones or “prior texts” that inform them (e.g. the name of the song that links the gag to Willy Wonka and the Chocolate Factory)?
These are hard questions. What’s clear is that the level of attention paid to the gag’s visual and audio layers needs to be extended to the caption layer. Captioning scenes in isolation results in major inconsistencies and obvious errors (e.g. poorly captioned music). My analysis suggests that a set of standards might begin to address the major differences revealed in the side-by-side comparisons.
The hurt knee meme and alternative captions
The hurt knee gag has become something of a minor Internet meme. The gag has inspired captioned responses to real photos using some form of “ssssss ahhhhh!” These photo captions don’t seem to be influenced by traditional closed captions at all. Rather, the Internet meme is motivated by an alternative, purely onomatopoetic style of describing paralinguistic sounds.
This alt.style of captioning seems to be more expressive, recognizable, and memorable than [groan] or [exclaim]. But I am not suggesting that paralinguistic sounds necessarily need to be captioned phonetically. Still, the phonetic approach (“ssssss ahhhhh”) does seem to address the problem of standardization that plagues the formally captioned versions of the hurt knee gag. Alternative captions also offer us an opportunity to discuss non-standard approaches to describing non-speech sounds. The question of which word to use — exclaim, groan, wince, whimper, howl — is resolved by an alternative approach that more closely mimics the sounds themselves (rather than their functions). In an alt.style, the sound’s function is available visually. We don’t need a caption to tell us that Peter is exclaiming “in pain,” because we can see the pain on his face.
Replace “ssssss ahhhhh” with “groan” or “whimper,” and it’s obvious why an alt. approach is more effective as a channel for carrying the Internet meme.
Lakers basketball player Kobe Bryant lays on the court in pain.
Oklahoma quarterback Sam Bradford lays on the turf grabbing his shoulder in pain.
Grabbing his knee in pain, Peter is inserted into a photo of soccer players on the field.
Soccer player sits on the grass, leaning back, grabbing his thigh with a slight pained look on his face.
A large brown bear lays on his back in a green clearing.
A leg tattoo featuring Peter from Family Guy grabbing his knee in pain. The caption on the site where the image was found: Ahhhh sssss...Ahhhhh ssss...Ahhhh hhee gotta love Peter Griffin!
One woman writing on the arm of another while the first woman winces. The caption on the site where the image was found: sssssss.......ahhhh