Imagine looking at a portrait of someone, and it begins to talk to you. Startling? Maybe. Attention grabbing? Very. But interactive portraits are very much becoming a reality thanks to an Auckland start-up and Mahuki accelerator graduate called Vaka Interactiv.

CEO Jesse Armstrong says the idea came while preparing for the 2017 Mahuki accelerator, which is based out of Te Papa Museum in Wellington and helps develop digital businesses for the cultural and heritage sector.

To see what pain points people were experiencing with museums, the Vaka Interactiv team spent some time hanging out in Auckland Museum observing people’s behaviour throughout different exhibitions. They noticed foreign visitors would flock to the marae in the museum in awe because they’d never seen anything like it before, but once they got inside, they didn’t stay long.

The co-founding Vaka Interactiv team at National Pacific Business Trust Awards earlier this year. L-R: Jordan Tupuola, Kaveinga Lisati, Andrea Fua and Jesse Armstrong

“Because it has carvings all over the place and all this symbolism that tells these stories, we found it interesting that people were only spending 30 seconds to a minute max in there,” Armstrong says. “And as they’d leave, they’d say the craftsmanship is amazing, but to them, that’s all they saw. They didn’t get the deeper understanding and we realised there was a lack of storytelling and presence inside that whare (house).

“We took a step back and were brainstorming, and noticed right next to it there was paintings, and we thought what if we just brought the paintings to life and had them tell the culture? We researched the tech and found it was totally possible.”

None of Vaka Interactiv’s co-founders come from a background in tech, so they had to bring in game developers and animators to help them pull it off.

“What we’re trying to build from a technological standpoint is a game, so game development is very useful,” Armstrong explains.

The digital portraits have been dubbed ‘Culture Lens’ by Vaka Interactiv and aim to connect the audience with the storytellers themselves – something that’s particular poignant for Maori and Pacific communities, who have traditionally passed these stories down orally from generation to generation.

The first one to be debuted to the public is a collaboration with Vaka and Dunedin-based artist Greg McDonald at his gallery in Invercargill, the Chiaroni Gallery.

The painting is of his niece and is named Descendant X, and using a high definition image of my painting on a screen, inside a frame, the portrait speaks to the viewer, telling them about her story and her whakapapa (genealogy). A hidden camera in the portrait frame detects when people’s face within a certain distance to know when to start the storytelling spiel, or timers can be set.

The result is an eerie and engaging experience, not unlike in Harry Potter where portraits can talk to observers. Armstrong calls it “magic, art, culture and tech” all rolled into one.

“A lot of inspiration did come from Harry Potter, but this magic is useful, not just for the entertainment factor but because when people for a split second forget where they are and interact with the portrait, it’s compelling enough to make people stay and want to check it out and find out more,” he says.

“That’s one of the major barriers inside the museum sector: to get people to stay long enough for the story of a particular object. They glance at signs, but wouldn’t invest a lot of time.”

As for what’s next, Descendant X was the very first iteration of the interactive portrait technology, so the Vaka team want to hone that technology first before rolling out other products, but there many exciting paths the company could head down.

Armstrong says they are interested in adding in elements of AI, but also don’t want to push the boundaries of tech so far that people feel terrified by a talking portrait.

“Our initial idea was to have people engage with portraits in two-way conversations, but because some people don’t like the sound of that, we don’t want people to be freaked out by thinking this thing has a mind of its own,” he says.

“We’ve found a way to avoid that that’s early in development using a concept from gaming – an AI tree – where you used constrained choices in the script, so the character might have introductory dialogue and say, ‘Would you like you to learn about A B C or D?’ So it’s only trained to listen for constrained responses – you can’t ask it any question you want.”

Another line of interest is wayfinding where a character can move between multiple portrait frames, just like on the aforementioned Harry Potter.

Interest is brewing both here and abroad from the public and private sector about where the team could take this technology, Armstrong says. But what he wants Vaka Interactiv to achieve the most is the emotional impact.

“In these museums, there are signs around the objects presenting information and while that’s helpful, what is lost is the emotional impact – or the kind of impact that creates unforgettable experiences. Culture lens is all about bringing these stories to life themselves.”