2024 | Visualization Research Centre, HKBU
Project Overview
PanoGen redefines audience engagement by inviting participants to become active co-creators in a shared storytelling experience. In this 3D immersive environment, users can contribute to the narrative in real time by inputting text descriptions that translate directly into dynamic visual elements. This interactive process transforms traditional passive viewing into a collaborative adventure, allowing participants to shape both the storyline and visuals within a panoramic, 360-degree virtual space. Here, audience members engage not only with the environment but also with one another, collectively evolving the shared experience.
PanoGen project aligns with core FCS objectives by encouraging personal expression, promoting group decision-making, and fostering the co-creation of a visual narrative.
Technical Narrative
The evolving display of PanoGen operates through an advanced text-to-image generator that allows users to input scene descriptions, character ideas, or object details on handheld devices. These inputs are instantly transformed into vivid, panoramic visuals displayed on a shared virtual canvas. With support for multiple participants, the system enables real-time collaboration, where users can add, modify, or influence each other's contributions. This results in an evolving visual environment shaped by both human creativity and AI-generated visuals.
To accommodate high user traffic, PanoGen is scaled across additional servers with a streamlined communication framework to ensure smooth performance and minimal latency. AI agents collaborate with human participants to co-create and adapt the storyline dynamically, balancing human input with algorithmic processing to support a responsive and flexible storytelling experience.
第一部分:項目概述
PanoGen 重新定義了觀眾參與的方式,邀請參與者成為共同創作的一部分,塑造出共享的敘事體驗。在這個 3D 沉浸式環境中,用戶可以即時輸入文字描述,並直接將其轉化為動態的視覺元素。這種互動過程將傳統的被動觀看轉變為一場協作冒險,使參與者能夠在全景 360 度的虛擬空間中共同塑造故事和視覺內容。觀眾在這裡不僅與環境互動,還能彼此交流,集體發展出一個共享的沉浸式體驗。
PanoGen 項目符合 FCS 的核心目標,通過鼓勵個人表達、促進小組決策,並支持共創視覺敘事。
第二部分:技術敘述
PanoGen 系統透過先進的文字生成圖像技術運作,讓用戶可以在手持設備上輸入場景描述、角色設計或物品細節,這些內容會即時轉化為栩栩如生的全景視覺,並顯示在共享的虛擬畫布上。系統支持多位參與者同時協作,使用戶可以實時添加、修改或互相影響彼此的貢獻,從而形成由人類創意和 AI 視覺生成共同塑造的動態視覺環境。
為了支持高流量的用戶數,PanoGen 在多台伺服器上進行了擴展,並設置了精簡的通訊框架,以確保流暢的性能和低延遲。AI 代理與人類參與者合作,共同創建並動態適應故事線,讓人類輸入和算法處理完美結合,實現了一種靈活且響應迅速的敘事體驗。
Principal Investigator: Sarah KENDERDINE
Collaborated institution: The École Polytechnique Fédérale de Lausanne, Switzerland
Partner: …
Date: 2024
Team: eM+ (programmer, software developer), Vack CHEUNG (Installation plug-in)
Hardware: nViz with 29.6 surround sound system, Mobile device
Software: Stable Diffusion, Unreal
Innovation and Technology Fund, HKSAR