An NVIDIA employee built the MonsterHouse Halloween installation with the Nemotron voice agent

An employee of NVIDIA created a Halloween interactive installation called MonsterHouse – a garage door with an animated mouth that reacts to visitors with the voice of a grumpy pirate ship. According to the author, the character backs down from its grumpiness if a visitor says "please," and bids a warm farewell after "thank you."

The installation runs on the Nemotron Voice Agent pipeline from NVIDIA, which combines several models: the Nemotron ASR model converts visitors' speech to text, responses are generated by the open small language model Nemotron 3.5 Bolt E4B (in 4-bit quantization, the so-called "4GB" version), the voice is synthesized by the Magpie TTS Zeroshot model, and speaking is animated by the Audio2Face-3D tool in Unreal Engine 5. For costume recognition, the author used the Nemotron 3 Nano Omni model, and for tracking and counting visitors, the YOLO and ByteTrack tools. The entire system runs offline on a single workstation with RTX PRO 6000 Blackwell Max-Q and RTX PRO 4000 Blackwell cards.

The author created the character's personality using a LoRA adapter trained on more than 5000 dialogue conversations. According to the author, more than ten models were tested, and the Bolt E4B model was chosen for its lower memory requirements compared to the Qwen3 30B-A3B and Qwen 3.6 35B-A3B models, to which the author had access even before their public release. Details can be found in the source article.