Abstract
This essay proposes "Cultural Computing" as a form of cultural translation that uses scientific methods to represent essential aspects of culture. Using visuals that have not been the focus of digital practice -- such as images representing Eastern thought and Buddhism, sansui painting (landscape ink painting), as well as poetry and kimonos that evoke these images -- we projected the style of communication developed by Zen schools over hundreds of years into a virtual world that users can explore, an Eastern sansui world. Through encounters with Zen koans and haiku poetry, users are constantly forced to reflect on their "self-consciousness."
Consciousness and the Self
Within the context of contemporary science and philosophy, it becomes difficult to determine how consciousness and information are related. One cannot even say that this connection has been accurately described. On the contrary, the relationship between consciousness and the self involves significant methodological problems and has rarely been the subject of research when it comes to the various realizations of these relationships that are occurring in people's everyday lives, such as art, religion, or even sports.
There are very interesting border zones stretching between definitions of consciousness and the self. "Information sensations" such as dreams, which can be taken as a level of consciousness -- although they do not necessarily seem to be stored in memory -- are one of these border zones. Other examples are the "waves of emotion" experienced in encounters with literature or music, or the expanded sense of self perceived through activities such as religious training or meditation.
In terms of more mundane experiences, there are the difficult to define feelings of "self-projection" gained from philosophical thought or from the development of a moving story; the "superego sensations" brought about by sports games, training or running; feelings such as happiness, disappointment, and uncertainty that deepen through experiences such as romance, accidents, or illness; the sublime state of quasi-consciousness that feels as though one has entered into the kind of fantastic time-space that paintings and movies create. The exact nature of these feelings remains largely unknown. We also do not fully understand why we are able to process virtual images from concepts that are expressible only in language, like "round square," or why certain short phrases, sounds and scenes -- such as lines from poetry and haiku, a song melody, or a single scene from a television commercial -- implant themselves deeply in the human consciousness.
In order to approach ideas such as a "unified consciousness" (the fusion of the consciousness that determines interactions in daily life with subconscious processes) and the "information self," it is necessary to turn our attention to the diverse border zones stretching between definitions of consciousness and the self. In the research discussed here, the authors hypothesized in the planning stages whether it might be possible to peek at the flow of at least part of human consciousness on the basis of people's interactions with a computer system.
One might think that this approach could at best shed some light on the realm of the "subconscious," but this is not necessarily true. By preparing a system where not everything has been carefully calculated and "jostling around" the user's consciousness, one might actually get closer to a natural flow of human consciousness.
Cultural Computing
With various kinds of media technologies increasingly becoming part of our everyday lives, we have built a sphere of communication that potentially reaches all parts of the globe. On the other hand, there seems to be an increasing danger that, as the communication network expands, the level of interpersonal communication is suffering.
Within this context, we decided to develop a project exploring the possibilities of a communication medium that incorporates a new kind of interactivity -- with "editorial engineering" [1] and "art and technology" [2] as a foundation -- and includes research on both the operations in which interactions between multiple cultures occur and the "intelligence" that emerges between a user and the system.
Human communication is normally cultivated in an environment that comprises localities, customs, and language. Any culture obviously has strong roots in its unique histories. The mass media that conveyed these peculiarities across cultures originally were "communication media" such as writing, music, and film. With the advent of the "global digital society," computers take on the role of vehicles for intercommunication between local and global cultures. It is up to those using and developing computer technology to bring to life local cultural characteristics. The authors and developers of this project focused on the cultural roots of their native country, Japan. These cultural roots include Buddhism, kanji, haiku and other Japanese poetry, music and traditional Japanese clothing (kimono). We decided to mine the unique communication spaces and imagery developed in Zen Buddhism and landscape ink painting (sansui painting), in particular. [3]
Fig. 1. ZENetic Computer at SIGGRAPH 2004,
Emerging Technologies  The role that the computer has played with regard to traditional, local culture has been mostly that of a tool for the preservation of culture for future generations, for example through the restoration of artifacts and the recreation of lost relics through simulation. However, we were more interested in the possibilities of "computing" the unquantifiable essence of culture carried within people, such as culturally formed subjectivity, emotions, and personality. [4]
With this research project, we offer a concept of "cultural computing" and describe -- through the realization of an actual interactive system -- computer-based strategies that reflect differences in the formation of emotion, consciousness, and memory, which we see as essential for the communication capacities of computers.
The result of our research is ZENetic Computer, a system that invites various applications -- among them its use in the field of digital storytelling -- and functions as an assessment of the current state and future possibilities of "cultural computing." [Fig. 1]
Concept
The following explains the scenario for a user experience within the ZENetic Computer environment. First, the user builds a three-dimensional sansui ink painting of a landscape on the display, constructing her own virtual space by using an intuitive and user-friendly interface.
The images convey the natural world characteristic of the East and Japan as well as philosophical concepts related to it, providing the user with an experience very different from the one created by images of contemporary life. [5] Through this introduction, the system brings about a kind of "awakening" within users' "information selves," and strives to stimulate their unconscious imagination. As the system "classifies" the user's personality based on the composition of her sansui landscape design, it generates a story for the user, drawing her into an alternate world through the display.
The story inorporates mechanisms that are meant to stir the user's self-consciousness and have been developed on the basis of haiku poetry and Zen riddles (koan). The story built from these elements is not a linear one, like the narrative of a traditional movie or novel, but rather a nonlinear collection of short story fragments. A user experiencing these inconclusive story fragments is faced with a degree of uncertainty creating an expectation and a desire to achieve closure by connecting the fragments. The user is asked questions by the system and, given her uncertainty and desire for closure, may hesitate but cannot help trying to answer these questions.
Through several of these triggers, the user connects the stories and builds her own unique narrative. As she uses a virtual brush, a rake for the system's rock garden interface (see below), images appearing on the screen, and clapping of her hands in response to questions that are posed by the system via images and voice, the user begins to realize that the system is demanding that she meet with it "face-to-face." Through this interaction, the user becomes increasingly conscious of her process. As her desire to connect story fragments intersects with the architecture of the interface, the distance between the standardized responses that characterize exchanges in daily life and her unconscious feelings begins to diminish.
In the process of fusing these two "selves," Ma interaction plays an important role. Ma is a Japanese concept that places high value on ephemeral occurrences -- the moments of the here-and-now -- within every experience. Having traveled through the stages and scenes of the system, the user finally encounters and interacts with a version of the "Ten Bulls Story," which is a metaphor for expressing one's "true self" in Zen. Through the dialogue, the user is meant to experience a process in which the part of the self defined by norms of everyday behavior and subconscious levels of self begin to merge in a more unified self-consciousness.
Fig. 2 An example of a user's sansui painting 
Technologies
The main technologies used in constructing the system are the following:
1. The three-dimensional sansui landscape allows users to generate a virtual world within the landscape that they can enter and explore.
2. The neural network engine identifies the user's personality on the basis of the sansui painting she created, and classifies it according to categories based on the Buddhist goun system. [Goun are the five basic physical and mental elements that make up the core of the world in Buddist thought.]
3. The dynamic chaos engine, on the basis of the goun categorization and the user's other actions, generates a sansui world as well as a narrative context tailored to each user. [Fig. 2]
Interaction Flow
The system processes its interactions with the user by means of the following methods:
1. The user places icons based on symbolic images on the screen and thus constructs a three-dimensional, virtual sansui world.
2. The system classifies the user's personality and places it into one of five categories based on goun.
3. The user enters into the sansui world and the story begins. Selected according to the user's profile from a list of choices, the haiku poems that the user encounters in the sansui world draw her into various fragmental stories.
4. Within each interaction, the user's goun state changes based on the chaos engine and the results of the user's interaction.
5. The user experiences a customized Zen dialogue based on her changing goun state (4.).
6. The system expresses the user's goun history (step 2. to 5.) through kimono designs of the Yuzen brand. (The kimono designs change based on the user's interaction history.)
7. Finally, the user experiences an original version of the "Ten Bulls Story" and views a flashback of her goun interaction history.
Fig. 3. The san-en method of sansui painting 
3D Ink Painting System
The main elements of the sansui ink painting consist of twelve symbols (rock, mountain, moon, traveler, bridge, bird, tree, house, flower, wise man, cloud and water) that are turned into hieroglyphic icons and placed in the upper area of the display as seen in Figure 2. Perspective in the three-dimensional sansui painting is realized through 3D computer graphics coordinates based on the unique san-en perspective used in Chinese sansui ink painting. (Kou-en is a distant perspective from below, as if one is looking up at the objects; hei-en is a parallel, straight-on view; and shin-en is a closer perspective from above, as if one is looking down at the objects.) The san-en perspective is determined within the three-dimensional space based on where the user places the icons. The sansui painting space also has a temporal component: the scenery changes as the day passes from morning to evening. [Fig. 3]
In analyzing the process through which the user builds her preferred sansui painting, the system uses a neural network that classifies the user's personality, which corresponds to a position within the goun space. The interactive system applies the five basic physical and mental elements of goun in the classification of personality:
Fig. 4. Determining a user's personality on the basis of the composition of their painting 
In addition to this classification system, other user-related data influences the course of interaction between the user and the system and is analyzed in the recognition of the user's "virtual personality" based on goun. Depending on the user's responses during the various stories, there may be instances where the virtual personality and the user's subconscious processes diverge. This difference is reflected in the content of the story built by the dynamic chaos engine. [Fig. 4]
Fig. 5. Rock Garden Interface 
Story Generated from Symbols and Allegories
The sansui landscape icons chosen by the user determine the context of the story. The user enters the sansui space, a metaphorical representation of her state of mind, and, using the rock garden interface [Fig. 5], travels through the Zen world. The system is designed so that the symbols located in the framed display get linked when the user approaches them and story fragments relating to the framed scene are presented as haiku or Zen dialogues.
Fig. 6. Zen Dialogue "Daruma Anjin" Interaction  Story Generated by the
Chaos Engine
The dynamic chaos engine's function is to generate a complex context and story based on the difference between the user's virtual personality (created by the system) and the user's subconscious processes and "personality." The chaos engine consists of three dynamic components or agents. The three agents are User Agent, Target Agent, and Master Agent. The agents move through goun space, acting in accordance with the internal dynamics of the chaos engine. The integration of the agents' motion into the goun space creates an interplay between the different agents and allows for synchronization to develop based on the internal dynamics. The dynamic movements of the chaos engine are sampled into the system as data and are used to create the sounds and images experienced by the user and to control the construction of the story, including the Zen dialogues. [Fig. 6] In the 2004 version of ZENetic Computer, the User Agent corresponds to the user's virtual personality as determined by previous story fragments (Virtual Personality 1) and the Target Agent corresponds to a personality determined by the most recent interaction result (Virtual Personality 2).
The User Agent starts at the position of Virtual Personality 1 and tries to approach the position of Virtual Personality 2. If there is no interference from the Master Agent, synchronization between the User Agent and the Target Agent develops, and the User Agent moves toward the Target Agent until they merge and become identical.
On the other hand, if there is interference from the Master, it becomes difficult for the User Agent to synchronize with the Target Agent, and the former has a hard time reaching the latter. In this research, we focused on this kind of space between "decision" and "indecision," in particular.
User Response
The implementation of real-time interaction between individual users' conscious and unconscious processes and computers has been a dream of computer science for a long time. One could say that the field of interactive storytelling is at the forefront of the research that tries to make this dream a reality.
While this dream is still a distant goal, the ZENetic Computer system aims at realizing a dynamic interaction that can express the spaces in between a user's conscious and subconscious processes through the use of a chaos engine.
In response to a survey of people who experienced ZENetic Computer at an exhibition at the MIT Museum in 2003, 90% found the project to be a relaxing and spiritual experience unlike anything they had encountered before. In order to determine the basic reasons for this response, we looked at the computer-generated user interaction history in order to investigate at which interaction point psychological changes occurred or became active.
Figure 7 shows the distribution of answers to a question regarding users' interests, which was posed to visitors during exhibitions at ACM SIGGRAPH 2004 (blue) and the Virtual Reality Society of Japan (red).
Looking at Figure 7, one can see that Americans and Europeans were most interested in ink painting technique and interaction technologies, while Japanese users found the Zen dialogues and ink painting technique to be the most interesting aspects. One could argue that this data shows a difference between a Japanese disposition to turn inwards and a Western tendency to focus on the logic of ink painting expression (which was perceived as mysterious).
Fig. 7. Elements users found interesting after finishing the interaction
Outlook to the Future: Cultural Computing Innovation and Spiritualizing Bits
At the 32nd UNESCO General Conference, the meaning of culture was defined (once again) as encompassing all customs, images, expressions, knowledge, skills, as well as related tools, goods, artificial objects and cultural spaces. Included in this definition of culture are not only physical cultural relics, but also the information exchange systems and communal, spiritual, and philosophical systems on which creations are based.
In 2004, ZENetic Computer won second place in the UNESCO-sponsored Digital Storytelling Competition of Intangible Heritage. One could speculate that at some point in the future, when the processing power of computers, high-quality displays, and input devices approach the limits of human perception, high technology might enter the spiritual domain. For most of Western culture, Japanese Zen is an old and mysterious philosophy. While there are books such as Zen for Dummies, it would be hard to say that it is possible to get an experience of Zen by reading one of these books. ZENetic Computer tries to convey the spirit of a foreign culture through experiences such as participating in Zen dialogues, listening to haiku, and exploring kimono patterns.
In the future, there will probably be a stronger interest in the concept and design of cultural computing for establishing cross-cultural connections. The exploration of this concept was the main intent of ZENetic Computer and the authors made use of advanced game design, graphics, and interactive displays in the process. We believe that the methods used in ZENetic Computer have broader applications in the field of education and can contribute to experience-based, cross-cultural understanding.
References
[1] Seigow Matsuoka, The Editorial Engineering of Knowledge (Asahi Shimbun, 2001). Japanese.
[2] Naoko Tosa, "Affective Minds" in ELSEVIER (2000), pp. 183-201
[3] Seigow Matsuoka, The Science of the Beauties of Nature (Chuko-bunko, 2004). Japanese.
[4] Naoko Tosa, "A Method of Story Generation based on Unconscious Information" in Journal of the Institute of Image Information and Television Engineers Vol.57 No.4 (2003)
[5] Seigow Matsuoka, Sansui Thought (Gogatsu shobo, 2003). Japanese.
[6] Y. Liu and P. Davis, "Dual synchronization of chaos" in Physical Review E 61 (2000), pp. R2176-R2179
|
