Imagine a world where a seasoned mechanic, trained on the inner workings of combustion engines, seamlessly transitions to fixing a Tesla, thanks to a trusty augmented-reality “thinking cap” that offers real-time feedback, pinpoints missteps and answers questions on the fly.

That's the ambitious vision behind MILLY, a cutting-edge artificial intelligence (“AI”) research program set to transform task guidance as we know it.  This will be the subject of a September 20, 2023, webinar presented by the Stevens Institute's IT Torchbearers, which will feature MILLY's co-investigator, Dr. Enrique Dunn, an Associate Professor of Computer Science at Stevens Institute of Technology in Hoboken, NJ.

The scope of MILLY—a nickname inspired by the project's formal title, “Multi-Directional Loosely-Linked Archetype Models for Perceptually-Enabled Task Guidance”—goes beyond many familiar AI programs.  Those other programs use reams of data to recognize complex patterns, such as those of modern language and all its associated nuance, and then reproduce those patterns when called on by users.

MILLY, however, extends that foundational AI functionality to extracting procedural knowledge from visual and linguistic cues. This allows the program to adaptably learn from diverse sources, including instructional videos, manuals and checklists. Armed with both its own knowledge and training history, MILLY then dynamically customizes training for its user, which means accounting for (and constantly updating) the user's present knowledge and skill level.

So, it would be able to “know” the baseline skill set of the user and then customize its guidance to what the user needs – not just providing a “one-size-fits-all” approach.

In doing so, MILLY bridges the gap between expert and non-expert, facilitating the execution of intricate tasks. As Dr. Dunn aptly put it in a press release announcing the project's receipt of a sizeable DARPA grant in 2022,

“Task guidance is a complex multi-modal problem. The goal of automating the entirety of this process is an ambitious challenge that expands the frontiers of current research in AI and augmented reality (“AR”), which our entire interdisciplinary team is excited to undertake.”

The potential applications of AI-guided task execution extend well beyond mechanics and engineers, of course. An augmented reality headset could be used by surgeons doing complex procedures, with MILLY guiding them through critical steps, identifying anatomical structures and offering immediate suggestions based on changing conditions of the patient. Non-professional users can also benefit from tailored coaching.   For example, amateur chefs and culinary students can wear AR glasses that offer step-by-step guidance during the cooking process.

This technology could also be combined with virtual reality (“VR”) settings to provide fully controlled training environments. Using its own training data—both the factual scenarios and its procedural knowledge—MILLY can create the immersive experience and then guide the trainee with its usual real-time feedback, all customized to the precise level needed for each individual trainee.

As technology continues to shape the landscape of education and professional development, MILLY stands as a testament to the boundless possibilities that AI brings. At Crowley Law, we understand the transformative potential of AI in various sectors, and we're here to guide you through the legal aspects of implementing such groundbreaking technologies. Whether you're in healthcare, engineering or any field seeking to harness the potential of AI-driven training, our experienced team is ready to provide expert legal counsel.

Contact us at 844-256-5891 or email at [email protected] to explore how we can help you embark on a journey towards a smarter, more efficient future.