The idea
Leave your phone on the music stand and hit record. The app listens to the whole practice session, then writes the log for you: forty-one minutes of playing, eleven of them scales, a stretch of one passage repeated nineteen times at gradually increasing tempo, and a clean run of the piece at the end. You get a session summary you didn't write and a history you can actually look back on months later.
Everything runs on the device. No audio leaves the phone unless you explicitly export a take.
Why build this
Every teacher tells students to keep a practice journal, and almost nobody does. The friction is in the wrong place: you have to stop playing, pick up a phone, and describe what you just did — at exactly the moment your hands are warm and you want to keep going. The result is that most people have no idea whether they practiced twenty minutes or ninety, or whether that hard passage is actually getting better.
The signal is right there in the audio, and it has been extractable for a while. Onset detection, pitch tracking, and tempo estimation are solved well enough to run in real time on a mid-range phone. Small neural pitch models like CREPE-tiny now fit comfortably in TensorFlow Lite. What has been missing is somebody pointing them at the boring, useful problem of "what did I just do for the last hour" instead of at automatic transcription, which is much harder and mostly unnecessary here.
Adult returners and parents of student musicians are the sweet spot. They are motivated, they pay for lessons, and they have no evidence that the money is doing anything.
Stack sketch
- Native mobile first — Swift on iOS with AVAudioEngine tapping the mic at 16 kHz mono. Android second, via Oboe.
- On-device pitch tracking with CREPE-tiny converted to TensorFlow Lite; aubio (compiled in via C) for onset detection and tempo estimation.
- Segmentation as classic DSP, not ML: RMS energy plus onset density separates playing from silence, talking, and page turns. Repetition detection via self-similarity over chroma features — a passage played five times shows up as a bright off-diagonal band.
- SQLite through GRDB for sessions, segments, and derived metrics. Everything local, with an optional iCloud/Drive file backup of the database.
- Charts with Swift Charts; no server, no account for v1.
Scope for v1
In:
- Record a session, detect playing versus not-playing, report time actually spent making sound.
- Detect repeated passages and show tempo drift across repetitions — the single most convincing feature.
- Tag a session with a piece name from a short user-maintained list.
- A week and month view: minutes practiced, streak, tempo progress on tagged passages.
Out:
- Note-level transcription or sheet-music alignment.
- Any cloud sync, account system, or sharing.
- Multi-instrument or ensemble separation. Ship for one player, one instrument, one mic.
Where it could go
The obvious next step is score alignment. If a user imports a MusicXML or MIDI file for the piece they are working on, you can align the audio to it with dynamic time warping and start reporting per-bar accuracy: which four bars they consistently stumble on, and whether last week's stumbles moved. That turns a log into a coach, and it is where a subscription becomes defensible.
After that, the teacher side. A student shares a read-only weekly summary with their teacher before the lesson, so the lesson starts from evidence rather than from "how did the week go?" That is a two-sided product, but it grows out of a single-player app that already works — which is the right order. A third path is instrument-specific tuning of the analysis: drums need onset and timing variance, not pitch; singers want a pitch-stability metric. Each is a separate detection profile over the same session pipeline.
Watch out for
Always-on microphone recording is the trust cliff — be loudly local-only, show a visible recording state, and never upload audio by default, or the App Store review and the first privacy-minded reviewer will both punish you. The other risk is over-promising accuracy: repetition and tempo detection are reliable, note-level correctness is not, so keep the UI honest about which is which.