You can direct the camera of your phone to various pianos and almost instantly create an AR pianist who can play works from different performers, while sheet music of the corresponding notes scrolls across the bottom in real-time.
The app uses machine learning to detect the piano, and then figures out the exact 3D position and orientation of your piano in a 3D space down to 1 cm accuracy. It does this just by looking at a single frame and without the use of 3D depth cameras.
This AR application works with any model or brand of piano under any lighting conditions, without the need to use special targets, image markers or the need to scan the piano model in advance.
This application will help many users in music education. Any user will be able to slow down the song up to 100 times without losing information on the movement of the hands, which is impossible with a traditional recorded video.
The full version of the application will also use the microphone of your smartphone to listen to you while you practice, and through machine learning, give you instant feedback on your playing; plus use the AR pianist to offer you tips on how to exactly move your hands during your performance.
© Live Animations. All rights reserved
