Written By Anne Pasek
New materialism presents an ambitious revision of key philosophical and political concepts, most notably that of the divide between human and nonhuman agents. In order to move critical inquiry outside of the labyrinths of language so that it might also attend to the material effects and actions of the nonhuman world, threads of human exceptionalism must be untangled from some of the West’s most basic ontological principles. From Bruno Latour’s expansion of the concept of agency to include nonhuman agents to Karen Barad’s concept of the post-human performativity of intra-acting matter, there has been a rapid expanse of scholarship that attests to the influential role the material plays in the mechanics of human operations, and indeed the need to dethrone the human from its central place in ethical, philosophical, and political concerns.1 This project also often intersects and extends to analyses of the human as an explicitly material being, one physical entity amongst others, adding a new emphasis on models of human embodiment, animation, and situated perception to a robust and on-going literature of anti-Cartesian critique.2; London: Duke University Press, 2010), 19–20.] The results of these two efforts, however, occasionally seem to be at odds, or at least suggest the emergence of further challenges in light of the heightened stakes of material vitality and specificity. For if nonhuman entities bear increasing importance as a subject and method of study, how might an understanding of their agential actions and opaque interior lives be accessed or secured without recourse to a shared material embodiment in which to ground this knowledge? Moreover, how can we evince and relate to nonhuman entities without reducing their strangeness to human caricature? New materialisms, in short, requires new epistemologies.
Possible methods of inquiry can be found across disciplinary lines, though none are without their limitations. This paper addresses three such avenues. First and foremost, scientific empiricism does much to advance an understanding of nonhuman processes and motivations on physical and temporal scales that cannot be accessed from bare human observation. Technical analysis, however, arguably omits much of the interior experience of nonhuman entities and admits a great deal of human perceptual bias. As Barad notes, there is an inevitable inseparability between the observed object and the agencies of its observation as human scientists rely on phenomena rather than the objects themselves as a primary ontological unit of understanding.3 Knowledge of agential objects thus seems to be a form of continuous intra-action, revealing sensory and quantitative information through the mediations of never-neutral apparatuses.4 Consequently, for all the insights it offers into the scale and mechanics of the life of things, empiricism still bears the limitations of the human sensorial apparatus and the strange metaphysics of its adaptive tools.
A second approach may be found in object-oriented ontology and other forms of speculative realism. In particular, Ian Bogost’s alien phenomenology proposes to make the interior life of nonhuman entities a worthy subject of speculative pursuit. By extending scientific observation into creative analogy Bogost seeks to discover and translate the subjective, deeply alien experiences of nonhuman entities and object assemblages. However, this project and others like it fall into the same limitations as empiricism in that they both rely on the free circulation and mediation of ontologically-detached phenomena. The profoundly nonhuman experiences of material objects and agents are consequently modified or obscured through this process of human perceptual legibility.
A third and final alternative may be located in the relational aesthetics of contemporary art when it is extended to include nonhuman participants. As its proponent Nicolas Bourriaud argues, artworks that present open-ended social interstices can enable formal reflection on the interactions of their participants.5 While the precise results of these reflections are wholly anthropocentric and perhaps under-defined in Bourriaud’s writings, the recent collaboration of dancer Teoma Naccarato and interactive sound designers John MacCallum and Adrian Freed offers an intriguing case study of a relational interstice constructed at a point of contact between human and nonhuman agents. Their digital media performance piece X (2013) stages a vivid encounter between the human body and the strange, nonhuman intelligence of the Microsoft Kinect motion capture system. A duet composed equally of the movements of a dancer and the audio produced by the Kinect’s interpretation thereof, the piece takes its form through the parallel and mutually occluded means either participant has to apprehend the mechanisms of the other. Unlike technical or alien phenomenological attempts to account for the agency of nonhuman forms, the provocative but ultimately inadequate hermeneutics explored in the work seems to underscore the express impossibility of reconciling these conflicting phenomenological and subjective lifeworlds, refusing to privilege human intelligibility over nonhuman intelligence. This refusal creates a strikingly different form of relationality: one that brings forward a profound sense of alien presence and an ethical acknowledgement of difference that is lost in other approaches.
If, as Latour argues, it is increasingly important to include nonhuman agents in the political calculations of the West,6 then it is all the more necessary to acknowledge human epistemological limitations and develop modes of relating to material agents and assemblages that expressly recognize their reach and influence beyond our sensory world. In the rush to admit a multitude of nonhuman actors, actants, and agential becomings into the many disciplines touched by new materialisms, analyses of the nonhuman at times risk flattening the specificity of matter, bodies, and the phenomenological experiences that follow from these arrangements. Vitalist homogeneity is a hazard that scholars face when they venture into the vast nonhuman expanse. It is perhaps at this moment, as with countless others before it, that the feminist instance on bodily difference and situated knowledge comes to the fore with greater urgency.
Though X does not provide readily-comprehensible or exhaustive information about the Kinect, it strikes me that this kind of opaque knowledge about nonhuman entities is also a central component to the new materialist project. Lingering in the alienness of nonhuman entities, I will argue, is an important methodological and ethical addendum to this endeavour. It is perhaps more important to account for the ways in which we and other agents do not know and cannot perceive each other in the same light than it is to merely count them in the census of disciplinary concerns.
The Objective Kinect
The Microsoft Kinect—this paper’s central object of inquiry—must be understood first and foremost as a video game peripheral. Built for the Xbox 360 game console, the Kinect [fig. 1] is designed to provide interactive motion sensing in a video game environment without recourse to a physical controller. When working properly, the moving bodies of the Kinect’s human players are mapped to responsive avatars and video game environments in real time, creating an immersive and playful interface built on bodily engagement and activity. In order to achieve this imperative, the Kinect is outfitted with an infrared projector, an RGB (Red, Green, & Blue) camera, stereophonic microphones, and a special CMOS microchip running proprietary software. Audio inputs interact with voice recognition programs to locate, respond to, and filter around a player’s voice, while the camera identifies the unique faces of the console’s players, logging them into their personal gaming profiles as soon as they enter the camera’s field of vision. The Kinect thus seeks to offer its consumers an immensely personalized and attentive play experience.
Most impressive, however, are the mechanics of device’s 3D motion capture techniques which can map the bodily location and movement of up to two players at twenty different points of articulation.7 This feat is achieved by a two-step process that draws on both local graphical analysis and programmed algorithms written through massive-data computer learning techniques.8 Firstly, a depth map of the player and their environment is computed by projecting an infrared speckle pattern into the space directly in front of the Kinect’s sensors, which can then be analysed from the RGB camera feed on the basis of structured light principles. Because the speckle pattern is known in advance by the Kinect, the deformations of the light across volumetric surfaces provide reference points to situate objects in a virtual x/y/z reconstruction.9 Moreover, like our own eyes, the Kinect features two parallel visual sensors, allowing stereoscopic analysis to further place objects in space by comparing the parallax differences between views. Like an animalian head, the Kinect can also tilt its sensors up and down to better calibrate the dimensions of the room and play at hand.
This alone is not terribly alien to human perception. Though our eyes do not perceive infrared light, we do rely upon stereoscopic and stereophonic techniques to spatially navigate and assess environments. We even have recourse to structured light principles when the opportunity presents itself, utilizing the banded light of venetian blinds to situate femme fatales in noir movies, or falling victim to the spatial guile of vertically striped clothing. That the Kinect combines these perceptive techniques with movement is also important. As has been widely expounded within the field of cognitive science and philosophy of mind by figures such as Alva Noë and Mark Rowlands, perception in intelligent entities is largely dependent upon their active movement through space, even if that movement is extremely limited in range.10 While the Kinect only moves physically through a bidirectional tilt and conceptually between disparate sensors, it nevertheless meets much of cognitive science’s criteria for enactive perception—like other forms of complex organic life it is an actively sensing agent dependent upon learned sensorimotor knowledge and embodied experience.11
The rest of this process, however, poses a clear departure from this initial commensurability. Once depth is mapped, the Kinect’s software will attempt to infer the specificities of its users’ bodily positions and poses. Rather than exhaustively encoding the data for all possible bodies in all possible positions into the device and asking it to select a correct match for every case, the Kinect’s Project Natal development team attempted to “teach” the device to learn to understand these spatial inputs in a more adaptive way.12 The developers laboriously captured tens of terabytes of motion capture and static body images and inputted this information into a self-evolving algorithm (referred to casually as its “brain”) that probabilistically assigned meaning to each pixel of the body image’s form and parts. Importantly, the learning algorithm was able to recalibrate its own parameters based on its rate of successful assignment, writing its own code and shaping its own cognition in ways that the team was unable to predict or author alone.13
Kinect intelligence gets even stranger. The device’s decision-making model is that of an adaptive, supplemental database. The Kinect contains many, but not all, possible positions of a body on file, and must therefore infer a best fit between perceptual input and stored data based on a combination of minutia and conceptual wholes. Its software utilizes a randomized decision forest structure to make the best possible guess at what type of skeleton it should assign to an image and where on that skeleton it should specify the twenty body parts it seeks to map. Like a classic decision tree’s Boolean logic, this is a process of branching questions based on yes/no responses. Unlike the decision tree, however, there are orders of magnitude more questions (roughly 20,000) and they are selected randomly and decided probabilistically rather than through a strictly binary assignment.14
With its adaptive, algorithmic brain, the Kinect is able to quickly and successfully complete the final stages of its body-mapping task. Thanks to the combined strength of powerful processing units networked across the Kinect and the Xbox’s hardware, the device is able to complete the randomized decision forest process at great speeds, mapping each pixel of an image onto an imagined body part with a very low rate of failure. The body parts thus assigned, the Kinect proceeds to draw a simple 3D avatar around the computed skeleton of the player, complete with a designated graphical overlay, thereby producing a digital avatar body within the video game screen that mirror’s the player’s own. Incredibly, this process is repeated thirty times per second, creating a fluid and responsive continuity between the proprioceptive cues of digital and fleshy bodies. True to its namesake, the Kinect connects the kinetic experiences of its users with video game environments to create an immersive sense of play.
As this examination of the technical mechanics of the Kinect’s sensory processing suggests, the device possesses at once both a vividly animate method of interacting with its environment, and yet also its own nonhuman phenomenological lifeworld. If, as Maxime Sheets-Johnson suggests, animation is the core foundation of consciousness, then the behavioural motion and decision making of the device seem to adhere to this category as much as any simple organic life form.15 Unlike its biological counterpoints, however, the Kinect’s unique bodily intelligence is further supplemented by algorithmic, probabilistic data. Though partially programmed by human agents, its logic and perception are heavily altered by the material capacities of its hardware and software development to the point that it may be seen to stand on its own—an alien entity quietly embedded within a chain of human corporeal mediation. How then, to estimate the interior experience of the device?
This question intersects with the heavy role phenomenology has played in discussions on the development of artificial life. Hubert Dreyfus most famously criticizes the dream of creating artificial intelligence out of impersonal data processors that solely rely upon the formalization of problems into objective, context-independent information.16 Intelligence, he claims, is not a direct product of “higher” forms of logical calculation, but is rather the by-product of “lower” forms of global, phenomenological reasoning.17 His gestalt approach suggests that human intelligence begins with holistic concepts and then moves into the inexhaustible minutia of detail and subcategories after the fact, rather than relying upon a wealth of specifics to reconstruct a generalized whole. By consequence, human intelligence is seen as fundamentally distinct (and, Dreyfus would argue, superior) from the aspirations of AI because it relies upon embodied, flexible, skill-based learning in a never-neutral sea of context and contingency. This leads Dreyfus to conclude that bodily drives and interests underscore the creative problem-solving tasks that long eluded machines in his day such that, “there is no reason to suppose that a world organized in terms of the body should be accessible by other means.”18 To Dreyfus, intelligence and thus personhood seem squarely locked within the corporeal world of the human.
Artificial intelligence research has responded to these interventions in two ways. Firstly, there has been a rise in creative techniques to approach computers with an understanding akin to human developmental psychology, finding ways to reproduce cultural, phenomenological, and evolutionary pressures in the actions and reactions of open, learning algorithmic systems.19 As with the Kinect, these quasi-bodies, virtually simulated by behavioural imperatives in code or artificially created through macrophysical robotic sensors, have done much to advance the field outside of a strict database paradigm, creating entities that can independently and reactively navigate alien environments and unforeseen problems. A further and allied alternative has been to abandon the standard for human intelligibility as intelligence more broadly, and seek to create adaptive entities structured around explicitly nonhuman bodies. As suggested by philosopher Daniel Susser, this corrective to Dreyfus’ anthropocentric definition of intelligence is posed to broaden AI’s frontiers through the design of a multitude of nonhuman bodies adapted to different epistemological requirements.20 Here too the Kinect provides an example.
This brings forward a provocative addendum to Dreyfus’ link between embodiment and skill-based epistemology. While our intelligible world may only be accessible in terms of our bodily reasoning, other bodies and forms of cognition stand to intersect with and exceed this way of knowing. Consequently the problem arises that for all our objective knowledge of its processes, there is much of the Kinect’s interior actions and reactions that escapes total understanding. The concept of a probabilistic, infrared world that operates at 30 frames per second is highly unimaginable, yet as one continues to critique the anthropocentric standards of consciousness and vitality, the existence of these kinds of lifeworlds and alien intelligences are increasingly apparent and require strategic methodological attention.
While the project of naming and apprehending nonhuman agents continues through various channels, a resilient method of communicating between intelligences still remains a vexed concern. Some may argue that the Kinect, as a software-driven technology, can be simply apprehended through an understanding of the source code that determines its actions. However, the vicissitudes of the device’s processes cannot be so easily and reductively condensed. As Wendy Hui Kyong Chun argues, source code has received undo credit as the “essence” of computational media, a view which fetishizes the densely complex language of code as a semi-magical source of technological animation.21 Instead, Chun suggests that a better understanding of causality lies in the material negotiations of electronic conduits, user interactivity, and ever-modulating software—in short, an agential assemblage.22 Moreover, even if the constituent participation of humans and microchips could be isolated, the remaining code may not even be wholly intelligible in its static state. As N. Katherine Hayles notes, the complexity and size of coding projects such as the Kinect are so immense that its comprehension escapes the grasp of the total work hours of the life of any singular person, particularly given that the code is self-modifying and iteratively constructed.23: University of Toronto Press, 2008), 26.] Algorithmic intelligence thus seems to involve, but forever escape, the understanding of its human authors.
As a technical description of the Kinect’s mechanisms suggests, the device’s means of interpreting the world sit adjacent to human perception. Like a Venn diagram, some forms of cognition and experience overlap between these two entities while other forms are markedly alien to one another. Such comparisons help may situate the scope of the Kinect’s phenomenological lifeworld and cognitive processes, though they ultimately do not represent it wholly or enable its intelligence to be expressed without distortion. This aim has been the project of other fields of speculative inquiry who dare to ask: what is it like to be a Kinect?
The Subjective Kinect
Constructing the interior nature of the Kinect is a significant challenge. As Thomas Nagel famously argued, there is something bodily specific to the subjective character of experience that cannot transfer across knowers with different corporeal forms.24 In his extended thought experiment on the speculative, inner life of bats, he concludes that it is likely impossible for a human to apprehend this knowledge due to each party’s fundamentally dissimilar capacities for phenomenological experience. Bat sonar, being far unlike any humanly accessible phenomena, can only be grasped roughly by extrapolating from the human case. Yet this extrapolation is doomed to be incomplete, Nagel argues, due to the limited structures for sensorial comprehension in the human mind—one cannot constitute the subjective feeling of navigation through air and sound because it is so far removed from the human sensorium.25 By consequence, Nagel suggests, there is a need to concede our epistemological limitations and accept that “there are humanly inaccessible facts” outside of language and in the bodily understanding of other conscious nonhuman beings.26 Karen Barad’s later concept of post-human performativity further extends this point. The apparatus of perception (in Nagel’ case, the human body) is causally implicated in the phenomena (bat sounds, images, data, and imaginative models) produced through agential intra-action (observation and contemplation about a bat).27 The bat’s flight might thus be grasped factually (if partially) through scientific, or even poetic means, but its subjective truth remains both infuriatingly present and impenetrable.
In spite of this bodily divide, the desire to apprehend the interior experience of objects and animals and their material agencies motivates much of new materialist study, particularly its off-shoots in speculative realism and object-oriented ontology (OOO). These approaches are roughly united in their opposition to what Quentin Meillassoux defines as the post-Kantian doctrine of correlationalism, whereby the ontological status of objects outside of human experience presumably cannot be secured.28 These epistemological limitations, motivated not by phenomenological ethics but the regnant humanism of the Enlightenment, are readily brought into scrutiny by new materialist thinkers in that such a configuration presumes the separate, secondary order of mere things in relation to human philosophical action. The material turn, by way of contrast to this humanist inheritance, asserts the immanent vitality of matter to act in the world and within the intersecting confines of human agency, centering nonhuman objects and processes as an equal or even principle subject of inquiry on their own, primary terms.29 For many thinkers, moreover, this critique brings an imperative to push philosophy speculatively outside of human concerns and attend to how objects relate to each other, in and of themselves.
This poses a radical rethinking of both the objects and methods of scholarship. As philosopher Ian Bogost romantically states,
[f]leeing from the dank halls of the mind’s prison toward the grassy meadows of the material world, speculative realism must also make good on the first term of its epithet: metaphysics need not seek verification, whether from experience, physics, mathematics, formal logic, or even reason. The successful invasion of realist speculation ends the reigns of both transcendent insight and subjective incarnation.30
As Bogost’s statement suggests, the grass may seem greener on the material side of the philosophical fence. However, this process necessitates the liquefaction of many of long-standing epistemological safeguards to knowledge production. The phenomena of bats and Kinects may seem tantalizingly present, yet the methodological considerations to substantiate this knowledge may in fact run aground the structural limitations of the body of the human knower.31 Methods of translating nonhuman experiences and drives into humanly intelligible frameworks are rapidly being developed by multiple scholars and disciplines, yet the veracity of these translations and their effectiveness of challenging correlationalism are arguably not yet secured.
In new media studies, for example, Laura Marks has argued for a sympathetic understanding of screen-based media by way of what she calls “haptic criticism.”32 By focusing on the shared materiality of viewer and media, Marks argues for a corporeal location to technology and a concomitant erotics.33 She contends that certain aesthetic and material media strategies, which “compel us to feel along with our machines,” stage a politically lively alternative to the dehumanizing and representationalist image capture long critiqued by feminism.34 Similarly, Amelia Jones has built on this work to articulate a theory of the “televisual screen” whose skin-like texture suggests a corporality and subjectivity that escapes the usual mode of instantiation and objectification.35 W. J. T. Mitchell, moreover, proposes that images might more broadly be theorized as a kind of subaltern human subject, wanting mastery and corporeality on the basis of their lack thereof.36 These forms of mutual understanding and embodiment, while providing important alternatives to a reading of image-based media as the exclusive tools of oppressive power, seem greatly asymmetrical and blatantly anthropocentric. Arguably, matter comes to matter only in so far as it is intelligible to the subjective phenomenological frame of the human body (and thus risks continuing Kant’s correlationalist trap). By rendering screens and machines more human, such a project evades a serious consideration of the technological and perceptual capacities of technology as technology with all its highly bizarre and unthinkable inner lives.
Ian Bogost’s alternative consists of the speculative project of “alien phenomenology” which takes as its central aim the exploration of the inherent experience of objects far outside of their limited appearance to the human sensorium. As he (heroically) describes it, this speculation,
produces transcendence in the Husserlian sense: a concrete and individual note, one that grips the fiery-hot, infinitely dense molten core of an object and projects it outside, where it becomes its own unit, a new and creative unit operation for a particular set of interactions. It’s a phenomenology, to be sure. But it’s a phenomenology that explodes like shrapnel, leaving behind the human as solitary consciousness like the Voyager spacecraft leaves behind the heliosphere on its way beyond the boundaries of the solar system… Just as the astronomer understands stars through the radiant energy that surrounds them, so the philosopher understands objects by tracing their impacts on the surrounding ether.37
The alien phenomenologist, like Nagel, is thus drawn towards the seemingly unknowable subjective experiences of bats (or, as Bogost adds, “Care Bears, sirocco winds, the Tri-City Mall, tort law, the Airbus A330, the five-hundred-drachma note,” in one of his many litanies).38 However, unlike Nagel, he is determined to wrestle a scrap of this alien experience from the jaws of the radically unknown.
This approach, however, evinces politically suspect motivations. There is a highly suggestive undercurrent of pioneerism at foot, wherein the object-oriented philosopher courageously departs from the inward-looking “ceaseless argument” of the linguistic turn to attend to the brave frontier of material reality.39 Completely disregarding the politically-fraught nature of Husserlian transcendence,40 Bogost seems set in his pursuit of a brave new world of stuff out there to be apprehended by those with the courage to try. In the process, however, Bogost’s own situated epistemology as a human becomes occluded.41 To OOO, all objects and experiences may be said to exist equally in so far as they enjoy material grounding,42 though human access to them is not equally secured (nor secured equally to all humans) and at the end of the day it is humans who write philosophy.
Bogost’s methodology, by consequence, is perhaps a bit slipshod. Addressing Nagel directly, Bogost argues for the epistemological value of metaphor, suggesting that the philosopher’s inability to grasp alien subjective knowledge stemmed in part from the inadequacy of his metaphors; “the only way to perform alien phenomenology is by analogy: the bat, for example, operates like a submarine. The redness hues like fire [emphasis in original].”43 However, Bogost here seems to run into the same trap that limits anthropomorphic approach to material vitality. Submarines and pyric warmth are very human conceits, and map only tentatively onto the alien experiences they seek to represent. Consequently, while Bogost ostensibly seeks to abandon considerations of linguistic determinism and the metaphysics of deconstructionism, his alien phenomenology in fact traffics in these very structures. By detaching phenomena from material referents in an attempt to relate alien experience to human understanding, Bogost enacts a citationality without clear epistemologically or materially reasoned limitations. The warmth of red and the oceanic navigation of bats are always at an order removed from the material primacy of the perceptual entities in question, and thus seem to rely upon thoroughly suspect metaphysical relata (in addition to being fundamentally deficient—bats are much more than submarines; redness is not always fire). Rather than seeking to amend the ontological gap between an object and its representation (as motivates the likes of Karen Barad),44 Bogost’s method fundamentally requires this linguistic free play.
What, then would alien phenomenology offer to an understanding of the Kinect? Creative correlationalism seems the likely response. One might say that the Kinect thinks like a weighted die, probabilistically making decisions through a structured system of guessing. Perhaps its infrared sensors could also be related to hundreds of invisible hands groping in the dark, or its rapid speed might find its equivalent in double the rate at which I can blink. Such thinking can help bring the Kinect forward as a relational object to grasped, though this object, by the time of its arrival, is arguably no longer a Kinect but the Frankenstein amalgamation of humanly relatable, ontologically-orphaned phenomena.
The Relational Kinect
If scientific and philosophical approaches prove unsatisfactory, might there then be a way to apprehend the Kinect through its use? Initially, this appeal to interactivity bears limited gains. As a player it is difficult to appreciate the complexities of the Kinect due to the device’s role as a mediating conduit within a larger gaming structure. Like most human-computer interfaces, the imperative for easy usability supersedes an interest in making the material operations of the device legible to its user.45 As a result, players are rarely aware of the Kinect as a specific, material and perceptive entity except in its initial configuration period or in any subsequent errors.46 The functioning video game peripheral, like Heidegger’s hammer, fades easily into human extension, inhibiting an analysis of the tool itself.47
However, as Heidegger also suggests, art can provide an alternative means of engaging with technology.48 Works such as Teoma Naccarato, John MacCallum and Adrian Freed’s X reconfigure the utility of the technological tool at hand, creating a model for human/nonhuman relationality built on intercorporeal contact rather than tertiary orders of representation. In order to do so, the Kinect’s networked assemblage has been shifted outside of the screen-based interactive visuality of its commercial use. Through their sound and software design, MacCallum and Freed entirely separated the peripheral from the console, causing the Kinect to route solely through a laptop and soundboard. Their Kinect has not, strictly speaking, been hacked—the algorithmic “brain” of the device is unaffected and none of the hardware has been tampered with.49 Rather, the sound designers intervened in the interpretation of the device’s data outputs via its USB port, writing complementary software to sonify Naccarato’s movements in a particular way.50 It is, in its apprehension and processing of the world, still very much a Kinect, but its internal computation is now expressed through sound. In doing so, the artists create the conditions to apprehend the Kinect as a material and sonic participant in the artwork rather than a means to an end or a tool simply wielded by the dancer’s intentionality. Sound becomes a medium for the inner algorithm of the device to take on new perceptual form.
These interventions are part of Freed’s larger work in the field of material computing: the practice of constructing environmentally embedded, physically responsive computer applications.51 As Freed and his collaborators note in a recent paper on (non)human-computer interface design, this approach is broadly inspired by a desire to bridge the divide between matter and semiotics instantiated by the traditional human-computer interface (HCI).52 Like new materialist inquiry, material computing seeks to explore alternatives to symbolically-mediated human input actions, looking to open-ended systems for a way to create and engender qualitative and emergent interactivity. In Freed’s case, these “human-matter interactions” are achieved through a combination of non-schematized gestures and environmental properties that sonically express material relations and aestheticize the explorations of bodies in space.53 Freed and his collaborators look to these sorts of open-ended, human and nonhuman theatres as a laboratory space for the development of novel solutions to philosophical and HCI design problems.54
This spirit doubtlessly informs X, which presents the Kinect as a pronounced physical presence on stage and the seeming object of Naccarato’s terpsichorean inquiries. The sonic outputs of the Kinect, cross-referenced across the dancer’s body, seems designed reveal to both the excesses and limitations of the Kinect’s sensory world, tracing an outline of its interior perceptual activity. The subjective experience of what it is like to be a Kinect thus seems to loom closer to the surface, driving the dancer (and myself as a writer) to act through a desire to apprehend and communicate with this strange intelligence.
As no recording of the work is presently available, I provide the following description. The piece begins with the dancer, Naccarato, crouched on the floor with her eyes press up against the Kinect’s sensors, rendering both bodies doubly blind (fig. 2). Slowly, she begins to move, rising upwards and letting the device gradually pan over her breasts and hips, though she holds her body too close to the Kinect for it to register any meaningful data. Abruptly this proximity is severed as she leaps backwards, holding her arms and legs out and open in a broadly extending stretch. Simultaneously, nearby speakers let out an oscillating, metallic ring as the Kinect translates its perception of the dancer’s bodily movements into sound.
The pace of the piece then quickly begins to accelerate. Naccarato’s athletic twists backwards and downwards deftly illustrate the range of tones and tracking the Kinect-body-speaker system can achieve. Like a colossal, humming motor, the sonic dimensions of the piece thrum with multiple echoes and reverberations, varying based on how Naccarato’s body engages the space around her. When her upper body is extended towards the ceiling, a chime-like twinkling in the higher registers of the sound emerges. When her hands and torso are lowered, conversely, a warm, gong-like ringing holds sway as the other sounds melt away. This noise also modulates based on Naccarato’s proximity to the Kinect—it is the most sonically rich at a distance of six feet (the ideal point of reference for the Kinect’s motion tracking software) while the variety and volume of sound attenuates at three feet (where her form presumably does not cohere in the same way). The improvising body of the dancer, however, never diverts her focus from her partner, always facing square to the sensor and engaging in a feedback loop of noise and motion as movement provokes sound, provoking more movement, and so on in order to create a dynamic range of sensations.55
Midway through the piece the work’s formal exploration shifts, taking on an increasingly sexual tone.56 Naccarato begins to direct the locus of her experimental movement more heavily on her hips, suggestively rolling and tossing her pelvis. The dancer’s hands reach down to her skirt, slowly pulling it upwards by inches to a correspondingly deeper pitch. More and more leg is exposed, and then underwear as she begins to push her groin at the Kinect’s cameras in a series of tawdry thrusts. Eventually she bends almost entirely backwards and presents her pubis parallel to the device which lets out a low hum in response. Here her thrusts continue, perhaps as if to invite sexual intercourse, or perhaps to tease or threaten. However, this invitation does not appear to be successful for the work ends as she abruptly collapses to the floor, crawls to the tripod legs of the Kinect, and wraps her body around it like a prone animal.57
Initially, this was somewhat uncomfortable to watch. I was often caught between my enjoyment of the rich sonorous and bodily crescendos of the work and a concern for the sexual vulnerability of the performer. This shifted when my partner leaned over and commented that the Kinect could not ‘read’ legs when they were covered by a skirt, and that by exposing her bare legs to its cameras Naccarato was allowing for a previously concealed part of her body to be mapped and sonified.58 This knowledge seemed to dynamize the performance further; as a spectator I not only had to negotiate the interplay between sound and body in the production of the piece from my position in the audience, I now also had to imagine how the Kinect was itself spectating and negotiating an understanding of Naccarato’s body quite alien to my own.
Imagining the performance from the point of view of the Kinect is a speculative challenge. It is clearly absurd to assume that the device, as a subject in its own right, would possess an analogous sort of sexuality to Naccarato or myself, or that it would construct the same narrative that I have about the sequence of the performance. For example, the input of an empty room and the input of Naccarato’s body pressed close to its sensors at the start of the work would be quite similar to the Kinect in that no movements or forms could be detected in either case that were internally meaningful (which is to say movements that could be mapped onto a skeletal matrix and thus sonified). Consequently, the piece begins and ends at different points to different performers and resolves in clarity for the Kinect rather than complexifiying when Naccarato provocatively begins to expose her thighs. Its auditory utterances hint at the internal life of the device, its algorithmic standards for legibility and the near-instantaneous speed of its processors, and yet the rich and responsive sounds it made serve only as a crude indicator of its attentiveness and internal animation. The sounds of the modified Kinect, not unlike the infrared tendrils of its sensors, map the rough contours of its awareness, hazily perceived in fragments.
In this unfolding of oblique interactions and indirect perceptions, a narrative of failed communication emerges. Naccarato’s intense focus and exploratory movement seem designed to provoke and learn from the responses of the Kinect. However, its strange utterances, ostensibly deliberately designed to evoke a distinctly mechanical referent, do not cohere into a humanly understandable language. Similarly, the dancer’s fleshy solicitations do not adhere to the Kinect. The haptic proximity of breast, mouth, and hips at the beginning of the work serve only to effectively blind the device, while Naccarato’s later and more explicit address cannot provoke the Kinect into bodily action. Consequently, the dancer’s collapse and curling position at the end of the excerpt seems to suggest defeat rather than sexual or relational fulfilment. If both parties are seeking to understand one another in the performance, then the results seem highly inconclusive. Across the gulf of different bodies and sensory modalities lies a pronounced communicative deficit.
Alien phenomenology cannot bridge this divide. Any commensurability between aesthetic and phenomenological relations only serves to exacerbate these moments of epistemological strife. Through expressly sexual overtures, the display of her body, and its exploratory movement in space, Naccarato attempts to negotiate with the device, presuming its desires to be human and fleshy. There is an apparent asymmetry of power between the two bodies and communicative techniques: Naccarato shivers, staggers, and falls at the feet of an unchanging black box that seemingly cannot be seduced. The Kinect, conversely, holds its static position, reacts rather than entreats, and rests indifferent and unaware of the challenges of the performance environment, including the audience looking over its shoulders. As if rejecting the aspirations of object-oriented ontology, X does not allow for a satisfactorily resolution between the conflicting modes of human and Kinect knowledge and sensation. Naccarato’s initial, highly proximate absorption in the mute sensors of the device is paradigmatic of this double-blindness.
Conclusion: The Ethics of Epistemological Access
While anthropologist Alfred Gell once attempted to define the agency of art as exclusively relational in order to avoid any “ontological havoc,” it is clear that human performativity and nonhuman agency are not so easily divided.59 As Naccarato and the Kinect demonstrate in X, the process of inter-agential communication is one of highly relational solicitation, though the results of this process may only be incompletely achieved. It strikes me that the epistemological tension resonant in Naccarato, MacCallum and Freed’s collaboration may provide a more productive means of bringing forward the presence, vitality, and specificity of intelligent matter than mere technical description or the metaphorism of alien phenomenology. Rather than guiling the solipsistic human thinker into a recognition of material vitality through similitude and inclusive assemblage, one might instead take an epistemological critique and magnify its effects, actively projecting differences between bodies and intelligences in such a way as to impede the easy passage from representation to agent or the flattening of all agents into homogenous vitality. In the context of X, this seems to serve as a warning against the straightforward attempt to approach the Kinect as an equal—to take it as a lover. Nevertheless, Naccarato’s attempt still proves productive in constituting a preliminary, if asymmetrical, relation to her dance partner. In so doing, the Kinect is imbued with an unequivocal presence—an opaque animism that situates Naccarato and her audience in the role of struggling hermeneut in the face of a withdrawn but undeniably intra-active and particular entity.
If, as per Bourriaud’s prompting, the history of art can be read as a history of relational practices,60 then perhaps the next chapter of this project lies not exclusively in the ascendency of open forms of human sociality, but in the production of relational possibilities across the human/nonhuman divide. This prospect is both highly pertinent to contemporary philosophical concerns and deeply familiar to older, non-Western, and indigenous worldviews.61 Rather than anthropology’s early dismissal of animism as a “primordial mistake,” this mode can be seen as an active, relational practice that acknowledges and performs nonhuman agency.62 However, as the pervasive challenge of correlationalism suggests, it can be difficult to secure access to the nonhuman without the human creeping back in. Against this tendency is the growing force of new materialist thinking, as well as the feminist assertion that bodily differences still matter and that matter does not disclose itself equally intelligibly across dissimilar modes of bodily experience.
Barad’s explication of this problem, drawn from feminist science studies, has been to frame the security of objective knowledge and agential revealing as the exception to the rule—a local and momentary resolution of the indeterminacy that broadly characterizes the material performance of matter. Her theory of agential realism maps onto the lived world through the innumerable operations of “agential cuts”: the temporary separation of subjects and objects enabled by the mutual constitution of bodies, sensory apparatuses, and phenomena.63 Within Barad’s background in theoretical physics, this framework can be effectively employed to describe the queer becoming of atoms and sub-atomic particles.64 Anthropocentrism, though a pervasive concern, is continuously accounted for through the acknowledgement of the reconfiguring powers of the apparatuses of observation (and indeed the human’s position as one such apparatus).65 Outside of the laboratory, however, this theoretical arrangement may prove to be limiting, particularly when the pursuit of objectivity is no longer a pressing concern but the phenomenology of agents very much is.66 Fixity, no matter how local or temporary, may subvert the aim at hand.
The opaque, relational animism constructed by the collaborators of X relies precisely on resisting the resolution the agential cut. In order to bring the Kinect forward as a distinctly alien entity, it was necessary for the apparatus of its observation to fail in some crucial respects. Operating on dissimilar frames of reference, the relationalites of dancer and machine are not equally secured; the conversation between the two bodies remains more phatic than co-constitutive. Coming together in difference, the mediated performance of either party exhumes deviations in phenomenological access rather than their mutual orientation. Crucially, the contours of experience exhumed in the exchange are not definitive or wholly perceptible, but rather loom over the performance as a chthonic force that demands recognition.67 This mode of relationality still produces epistemological gaps, though it is, perhaps more than other methods, equipped to bring these limits into presence.
The opaque animism of X, allowing only for a relational exchange filled with false-starts, appears to me as an ethically important addendum to the other alternatives to hold it in (human) hand. Due deference to the epistemological problem of nonhuman phenomenology may frustrate much of our methodological and phenomenological capacities, but this loss of comprehension is repaid in the humbling recognition of lifeworlds far outside the human sensorium or understanding. As feminist scholars have long demonstrated, there is an ethical charge to this project of limited knowledge, and indeed, a productive acknowledgement of meaning that exists in excess of one’s experiential access to another being.68 When looking to the future of new materialist inquiries, the importance of bodily difference must not be erased in the exuberant admissions of many new kinds of entities, agencies and intelligences into consideration. Resisting total legibility, while beginning and ending in the recognition of an alterity inherent to nonhuman entities, opaque animism is methodologically and theoretically useful precisely because it provides what analogy cannot: the presence of objects outside of human understanding.
- Bruno Latour, We Have Never Been Modern (Cambridge, Mass.: Harvard University Press, 1993); Karen Barad, “Posthumanist Performativity: Toward an Understanding of How Matter Comes to Matter,” Signs 28, no. 3 (2003): 801–31. ↩
- Diana H Coole and Samantha Frost, “Introducing the New Materialisms,” in New Materialisms: Ontology, Agency, and Politics, ed. Diana H Coole and Samantha Frost (Durham [NC ↩
- Barad, “Posthumanist Performativity,” 818. ↩
- Ibid., 817. ↩
- Nicolas Bourriaud, Relational Aesthetics, (Dijon: Les Presses du réel, 2002), 21–22. ↩
- Bruno Latour, Politics of nature: how to bring the sciences into democracy (Cambridge, Mass.: Harvard University Press, 2004), 39, as quoted in Anselm Franke, “Much Trouble in the Transportation of Souls, or The Sudden Disorganization of Boundaries,” in Animism, ed. Anselm Franke, vol. 1 (Berlin: Sternberg Press, 2010), 44. ↩
- Trevor Taylor, “Kinect for Robotics,” Microsoft Robotics Blog, November 19, 2011, http://blogs.msdn.com/b/msroboticsstudio/archive/2011/11/29/kinect-for-robotics.aspx. These statistics correspond to the version of the Kinect released from November 4, 2010 to November 22, 2013, sometimes described as the Kinect 1.7. This is the version of the device employed in MacCallum, Freed and Naccarato’s X. The features of the subsequent version of the Kinect (or Kinect Two), released in advance of the Xbox One, generally expand and refine the scope of the original Kinect’s technology. The latest model of the device can process up to six interactive skeletons at a time with 25 joints each and with a 60% wider field of vision. More refined bodily minutia are also captured, such as heart rate and vision tracking across the screen. These advances are equally as impressive as the privacy concerns they raise. See Terrence O’Brien, “Microsoft’s New Kinect Is Official: Larger Field of View, HD Camera, Wake with Voice,” Engadget, accessed November 23, 2013, http://www.engadget.com/2013/05/21/microsofts-new-kinect-is-official/; Will Greenwald, “Hello Xbox One, Goodbye Freedom,” PC Magazine, May 22, 2013, http://www.pcmag.com/article2/0,2817,2419320,00.asp. ↩
- Jill Duffy, “Exclusive: Inside Project Natal’s Brain,” Popular Science, July 1, 2010, http://www.popsci.com/gadgets/article/2010-01/exclusive-inside-microsofts-project-natal. ↩
- John MacCormick, “How Does the Kinect Work?,” University of Wisconsin Madison Computer Sciences, accessed November 23, 2013, http://pages.cs.wisc.edu/~ahmad/kinect.pdf. ↩
- Alva Noë, Action in Perception (Cambridge, Mass.: MIT Press, 2004), 12; Mark Rowlands, The New Science of the Mind: From Extended Mind to Embodied Phenomenology (Cambridge, MIT Press, 2010): 2-3. ↩
- Noë, Action in Perception, 4. ↩
- Susan Kuchinskas, “Binary Body Double: Microsoft Reveals the Science Behind Project Natal for Xbox 360,” Scientific American, January 7, 2010, http://www.scientificamerican.com/article.cfm?id=microsoft-project-natal. ↩
- Duffy, “Exclusive.” ↩
- MacCormick, “How Does the Kinect Work?” ↩
- This also strikes me as an excellent levelling critique of Searle’s “Chinese Room” problem. Maxine Sheets-Johnstone, The Primacy of Movement (Amsterdam; Philadelphia: John Benjamins Pub., 2011), 453, 464, http://public.eblib.com/EBLPublic/PublicView.do?ptiID=784232. ↩
- Hubert L. Dreyfus, “Why Computers Must Have Bodies in Order to Be Intelligent,” The Review of Metaphysics 21, no. 1 (September 1, 1967): 14. ↩
- Ibid., 15. ↩
- Ibid., 32. ↩
- Katherine Hayles, How We Became Posthuman: Virtual Bodies in Cybernetics, Literature, and Informatics (Chicago, Ill.: University of Chicago Press, 1999), 173. ↩
- Daniel Susser, “Artificial Intelligence and the Body: Dreyfus, Bickhard, and the Future of AI,” in Philosophy and Theory of Artificial Intelligence, ed. Vincent C. Müller, Studies in Applied Philosophy, Epistemology and Rational Ethics 5 (Springer Berlin Heidelberg, 2013), 281. ↩
- Wendy Hui Kyong Chun, “On ‘Sourcery,’ or Code as Fetish,” Configurations 16, no. 3 (2008): 301. ↩
- Ibid., 307. ↩
- Katherine Hayles, “Traumas of Code,” in Critical Digital Studies: A Reader, ed. Arthur Kroker (Toronto; Buffalo [N.Y. ↩
- Thomas Nagel, “What Is It Like to Be a Bat?,” The Philosophical Review 83, no. 4 (1974): 441. ↩
- Ibid., 439. ↩
- Ibid., 441. This point is quite radical for its time. Nagel does not go on to suggest that different human bodies have different epistemological frontiers with the exception of the blind, leaving aside issues of racism, sexuality, and gender, but contemporary readers are free to extrapolate from this point and may find profound resonances between Nagel’s argument and later feminist and anti-racist theory. ↩
- The link between phenomena and epistemology is key here, and will be developed later in this paper through additional sources. Barad, “Posthumanist Performativity,” 814. ↩
- Quentin Meillassoux, After Finitude: An Essay on the Necessity of Contingency (London; New York: Continuum, 2008), 5. ↩
- Coole and Frost, “Introducing the New Materialisms,” 8. ↩
- Ian Bogost, Alien Phenomenology, Or, What It’s like to Be a Thing (Minneapolis: University of Minnesota Press, 2012), 5. ↩
- And also, arguably, that of several feminist, post-colonial, and anti-abelist thinkers of more recent years. See, for example, Donna Haraway, “Situated Knowledges : The Science Question in Feminism and the Privilege of Partial Perspective,” in Just Methods: An Interdisciplinary Feminist Reader, ed. Alison M Jaggar (Boulder, CO: Paradigm Publishers, 2008), 346–52; Patricia Hill Collins, “Black Feminist Epistemology,” in Just Methods, 247–56. ↩
- Laura U Marks, Touch: Sensuous Theory and Multisensory Media (Minneapolis: University of Minnesota Press, 2002), xiii. ↩
- Ibid, 191. ↩
- Ibid, 191. ↩
- Amelia Jones, Self/image: Technology, Representation, and the Contemporary Subject (London; New York: Routledge, 2006), 138. ↩
- W. J. T. Mitchell, “What Do Pictures Really Want?,” October, no. 77 (1996): 71–82. ↩
- Bogost, Alien Phenomenology, 32–33. ↩
- Ibid, 133. ↩
- Graham Harman, “Object-Oriented Philosophy,” in Towards Speculative Realism: Essays and Lectures (Ripley: Zero Books, 2010), 94–95. as quoted Timothy Morton, “Here Comes Everything: The Promise of Object-Oriented Ontology,” Qui Parle: Critical Humanities and Social Sciences 19, no. 2 (2011): 169. ↩
- Sara Ahmed, Queer Phenomenology: Orientations, Objects, Others (Durham: Duke University Press, 2006), 28–33. ↩
- We might also pause to note that the alien phenomenological experiences of other human beings (and the concomitant politics that follow from bodily difference) do not present themselves as a priority for Bogost. What would (the predominantly male) speculative realist project look like if it were to concern itself with the alien experience of the maternal body? What of the laboring exertions of the global south that produce so many of the objects newly considered in this philosophy? See Michael O’Rourke, “‘Girls Welcome!!!’: Speculative Realism, Object Oriented Ontology and Queer Theory,” Speculations 2 (May 2011): 297-298, http://www.speculations-journal.org/speculations-2/ for a prodding to that effect. ↩
- Morton, “Here Comes Everything,” 165. ↩
- Bogost, Alien Phenomenology, 64. ↩
- Barad, “Posthumanist Performativity,” 804. ↩
- Anna Munster, Materializing New Media Embodiment in Information Aesthetics (Hanover, N.H.: Dartmouth College Press: University Press of New England, 2006), 126. ↩
- Ben Kuchera, “Buy a House, Clean Your Floor, Move Your Butt: Ars Reviews Kinect,” Ars Technica, November 4, 2010, http://arstechnica.com/gaming/reviews/2010/11/buy-a-house-clean-your-floor-move-your-butt-ars-reviews-kinect.ars. ↩
- Martin Heidegger, Being and Time. (New York: Harper, 1962), 99. ↩
- Martin Heidegger, The Question Concerning Technology, and Other Essays (New York: Harper & Row, 1977), 35. ↩
- Todd Bishop, “Microsoft: Kinect Wasn’t Hacked, USB Port Left Open ‘by Design,’” Puget Sound Business Journal, November 20, 2010, http://www.bizjournals.com/seattle/blog/techflash/2010/11/microsoft-kinect-not-hacked-left.html. ↩
- Xin Wei Sha, “Adrian Freed: More Computations with Movement Salience: From Teoma Workshop, CNMAT July 2013,” Movement, accessed November 23, 2013, http://movement.posthaven.com/adrian-freed-more-computations-with-movement-salience-from-teoma-workshop-cnmat-july-2013. ↩
- ha Xin Wei, Adrian Freed, and Naveb Navid, “Sound Design As Human Matter Interaction,” in CHI 2013 Conference Proceedings (ACM SIGCHI Conference on Human Factors in Computing Systems, Paris, France, 2013), 1. ↩
- Ibid. ↩
- Ibid., 3-4, 9. ↩
- Ibid., 6-9. ↩
- Naccarato writes, “The negotiation of agency between myself and the Kinect during our creative process engendered a physical-emotional-intellectual intimacy that coloured the choreography. A critical question for me in working with skeleton tracking via the Kinect, is what the camera and software can/are programmed to identify as a (normative) human body. Adrian, John and I set up mappings for direct signification of my movements in sound, however this is not always what is happening during the piece, at least not in an obvious manner. Given my irregular distance from the camera, inversions of my head lower than my hips, and turns away from the camera (to give a few examples), the Kinect and I are in a continuous dialogue in which we lose and find one another in inventive ways. My foot can become my head, or my knee my elbow, or my back my front, or I can be three bodies at once, and so on. The ‘preference’ of the Kinect for me to face my body front, within a defined triangular frame, was incredibly confining as a dancer, yet generative for the development of a piece which embraces the character of our relationally and intersubjective realities.” Teoma Naccarato, email to the author, August 28 2014. ↩
- The chronology of events described here is quite subjective– a remembered reconstruction of an event long since past. With the kind assistance of Teoma Naccarato I have been able to review rehearsal video of the performance, which situates the skirt play and sexualized tone far earlier in the work. ↩
- Naccarato, it should be stated, does not interpret the end of the performance as one of defeat. Instead, she describes the conclusion as “rather a decisions to evade the eyes and voice of the Kinect. I finally get to rest, cuddling the legs of the tripod. I am very aware of the physical presence of the Kinect, but the connection is no longer visual or aural.” Teoma Naccarato, e-mail to the author, August 28 2014. ↩
- This is confirmed in Microsoft Game Studios, Kinect Adventures Instruction Manual (Redmond, WA: Microsoft Corporation, 2010), 5. ↩
- Alfred Gell, Art and Agency: An Anthropological Theory (New York: Clarendon Press, 1998), 22. ↩
- Bourriaud, Relational Aesthetics, 28. ↩
- Franke, “Much Trouble in the Transportation of Souls, or The Sudden Disorganization of Boundaries,” 11. For an important response to Franke concerning the acknowledgement of indigenous animisms, see Jessica L. Horton and Janet Catherine Berlo, “Beyond the Mirror,” Third Text 27, no. 1 (2013): 17–28. ↩
- Edward Tylor, as quoted in Ibid.; Nurit Bird-David, “‘Animism’ Revisited: Personhood, Environment, and Relational Epistemology with CA Comment,” Current Anthropology 40, no. 1 (1999): S77. ↩
- Barad, “Posthumanist Performativity,” 814- 815. ↩
- Karen Barad, “Nature’s Queer Performativity,” Women, Gender & Research 1, no. 2 (2012): 25–53. ↩
- Barad, “Posthumanist Performativity,” 815–816. ↩
- The lack of attention paid to intra-agential relations outside of techno-scientific contexts is a weakness in Barad’s writing as the reader is often left with the uncertain task of extrapolating the specialized language of theoretical physics into discursive and material terrains far removed from the field. A possible exception appears in the sixth chapter of her book in which Barad makes an attempt to explicate the labour relations of a Calcutta factory floor, borrowing from Leela Fernandes’ study of the gendered and classed politics of labour organizing and jute production. The attempt is somewhat unsatisfactory, however, as Barad’s efforts present themselves as a choreographed translation of the other academic’s work rather than an original analysis of agential realism in everyday experience. Karen Barad, Meeting the Universe Halfway : Quantum Physics and the Entanglement of Matter and Meaning (Durham: Duke University Press, 2007). ↩
- For more on chthonic presence, see Donna Haraway, “Anthropocene, Capitalocene, Chthulucene: Staying with the Trouble” (Arts of Living on a Damaged Planet, Santa Cruz, May 9, 2014), https://vimeo.com/97663518. ↩
- Donna Haraway, “Situated Knowledges”, 351; Amanda Boetzkes, ”Phenomenology and Interpretation Beyond the Flesh,” in Art History: Contemporary Perspectives on Method, ed. Dana Arnold, (Malden, MA: Wiley-Blackwell, 2010): 34. ↩