IMAGINE performing a traditional dance while wearing a motion capture suit and a headpiece with multiple sensors to track every movement of your body.
A slight extension of the fingers, a turn of the head or the shifting of a foot is captured as digital data, allowing a performance that exists only in the physical world to be recreated in a virtual environment.
That is the execution for the idea behind the Digital Heritage Sandbox (DHX), an initiative by the National Academy of Arts, Culture and Heritage (Aswara) under the Tourism, Arts and Culture Ministry (Motac) to digitise aspects of Malaysia’s intangible cultural heritage.
“The video recording and the motion data we capture here will become part of a database in our Digital Heritage Sandbox project, where we aim to keep intangible assets like cultural dances alive and accessible for future use,” says Dr Jazmi Jamal, director of the Future Creative School at Aswara’s Faculty of Animation and Multimedia.
Since 2024, the team has been experimenting with technologies, including motion capture and volumetric 3D scanning to record both the movements of performers and the physical details of their traditional attire.
During an interview in Cyberjaya, Jazmi demonstrates the 3D scanning process by standing still on a turntable for about 30 seconds. As the turntable rotates, a vertical 3D scanner with four cameras captures him from different angles before the software stitches the images together to create a 3D representation of his likeness. The technology was also used to capture the traditional attire worn by dancers before they performed their motion-capture sequences.
But digitising traditional clothing comes with its own technical challenges.
“We had a female Ngajat Iban dancer from Sarawak with this stunning costume, which included a belt and headpiece made of silver coins. The problem was that the 3D scanner couldn’t quite capture the details because of the reflective surfaces, so they appeared blurry in the scan,” says Jazmi.
The solution was surprisingly low-tech. “So, what do we do? We ended up using some powder to cover the reflective surfaces. It’s an old-school method, but it works,” he says.
Capturing movement, meanwhile, can take longer, as dancers may have to perform multiple takes to ensure the desired movements are recorded accurately.
But Jazmi says the bigger challenge comes after the movement has been captured: teaching a computer what that movement actually means.
Uploading culture and context
To a person familiar with a particular dance, an instruction such as “langkah ke hadapan” may seem straightforward. But translating that instruction into something a computer can understand is less simple.
“Does it mean taking one step forward? Does the dancer then move to the right? And does the same instruction describe the same movement across different dances or contexts? This is where the vocabulary of traditional dance becomes important,” says Jazmi.
For the team, preserving the original terminology is therefore part of preserving the movement itself.
Jazmi explains that rather than reducing a traditional dance to a series of generic instructions, the aim is to retain its original vocabulary and context so that the data can eventually be understood by digital systems, including AI.
This means the project is not simply creating digital copies of dancers. It is also building a structured body of knowledge around the movements.
“That includes identifying the fundamental steps and recurring movement patterns demonstrated by dancers, while accounting for improvisations or variations that may have been introduced by individual practitioners,” says Jazmi.
In terms of selecting dances to be included in the DHX, Jazmi says the team works closely with other local organisations and also looks at dances that have been documented by Unesco before investigating their histories and traditions in greater depth.
“This is quite fragmented because there are different camps of culture. So we need to identify dances that have a long lineage of practice,” he says.
The team consults academics at Aswara, as well as adiguru or master practitioners who can help establish the roots and lineage of a particular dance.
While academic research provides citations and historical records, practitioners contribute knowledge passed down through generations. Where possible, the team traces dances back to their earliest available documentation and historical roots, including traditions associated with the Malay sultanates.
Jazmi says the dances are then classified according to their context, such as ritualistic dances, palace performances and tarian rakyat or folk dances. All of this information becomes part of the digital record.
“In other words, a digitised dance is more than a video or 3D model. It needs metadata describing where the dance comes from, its lineage, the people associated with it, the context in which it is performed and the vocabulary used to describe its movements,” says Jazmi.
Mapping movement
There is another question the team is still working through: who gets to access and use the digital version of Malaysia’s cultural heritage? As the DHX is still in development, Jazmi says that his team is still configuring access rights for the digital assets.
By the end of the year, the plan is to have the platform ready for limited access before opening it to a wider audience.
For Jazmi, this is an important part of putting culture into the digital space.
“How do you promote various aspects of culture in the digital space while making sure others understand the meaning and the origin? For our culture to exist and thrive in that space, we have to establish its ownership and roots in Malaysia,” he says.
That could mean determining whether a digital dance can be freely viewed, who can access its underlying data and whether permission is needed to use it for research, education or commercial purposes such as developing characters in video games.
“Have you seen how some video games feature South-East Asian settings without identifying them specifically? It becomes a mish-mash of different cultures. That’s something we hope to change by encouraging developers and creators to attribute cultural elements used in the digital space to their specific countries,” says Jazmi.
For now, the project is working towards documenting dances with adult performers from all states in Malaysia. So far, it has included Ngajat Iban from Sarawak, Sumazau from Sabah, Tarian Piring from Negeri Sembilan and Mek Mulung from Kedah.
But the longer-term ambition goes beyond creating a digital archive.
Jazmi envisions a kind of museum of dances; one where people can explore not only how a traditional dance looks and moves, but also where it came from, how it evolved and the practitioners who shaped it.
“This scholarly work is important because the aim is not simply to recreate dances digitally, but to preserve the context behind them. Ultimately, we want to create museums of dances. You can see the roots, the interventions and how the dance has evolved,” he says.
In that future, Jazmi believes technology does not replace the dancer or the tradition. Instead, it becomes another way of passing the movement from one generation to the next, and perhaps, eventually from humans to machines.
“There is a post-humanism consideration here. Do we let aspects of culture transcend or fade away? Or maybe they can continue to exist in a different realm thanks to technology.”
When Jazmi reached out to some master practitioners about Aswara’s plan to digitise traditional dances as movement data, he expected to face some hesitation.
Master practitioners have spent years perfecting their art and Jazmi says they are widely regarded as the custodians of knowledge that has been passed down through generations.
“Surprisingly, I hear that they are open to the idea of the project. In the future, we may also start documenting the adiguru’s movements as data for DHX. I was told that one adiguru said just one word when asked why they accepted the project. I don’t know how true the story is, but the word was... ‘immortality’.”
