How we test AI music: our method and results

We evaluate AI songs by whether the intended words can be understood, whether the sound fits the musical brief and whether the finished take is useful for its purpose. We judge each question separately. Clear pronunciation cannot rescue an unsuitable arrangement, and polished backing cannot replace a missing chorus line.

This page separates two records. Our team reported a blind listening comparison of three generation options using English briefs. Our October 14, 2026 launch check covers five original recordings in our supported languages. These records answer different questions, so we keep their results separate.

Why a blind listening test helps

Knowing a product name or expecting a particular version to win can influence what a listener hears. A blind comparison hides those identities while the listener judges the recording against the supplied words and musical direction.

Our team reports using anonymous clips, shuffled playback order and three scores recorded before revealing the results. English-speaking reviewers assessed the words against the supplied lyrics. Each generation option received the same brief, so the comparison focused on how the sampled outputs handled the same request.

Listeners need the brief in front of them. A pleasant recording can still miss the requested language, omit words or ignore the musical direction.

The three quality criteria

The English listening comparison uses pronunciation, production and overall quality scores from 1 to 5. Pronunciation covers the six vocal briefs. Production and overall cover all seven briefs, including the instrumental. Our earlier five-language launch check did not retain numerical ratings, so we do not assign retrospective scores to it.

The scale runs from 1 to 5 and records a listening judgment. It does not measure an objective percentage of musical quality. Keep concrete notes beside the number so a later reader can understand the judgment.

Pronunciation also needs language competence. When a listener cannot assess the words confidently, leave that criterion unassessed. Do not guess from how fluent the singing sounds, or substitute the production score for a language judgment.

Comparing three outputs fairly

Give each option the same lyric sheet, musical direction and intended duration within a brief. Where input controls differ, record those differences rather than treating matching words as proof that every setting was identical.

Use neutral playback labels and shuffle them independently for each brief. A, B and C should not stay attached to one option throughout the listening session. Hide the output identities and averages until scoring is complete.

Use one playback setup at a comfortable level. If one clip is noticeably louder, compare at a similar perceived level and record that observation. Loudness alone should not decide which arrangement is better.

Keep unsuccessful requests separate from completed recordings. A technical failure, a skipped score and a completed take the listener dislikes are three different outcomes. Do not quietly discard a poor take and replace it with a favorite.

This comparison uses three generation options, rather than three repeat takes from the same setup. It compares the sampled outputs and does not establish a winner across every product, language or genre.

English listening comparison

Our team reported testing three generation options available through fal.ai. The comparison used six English vocal briefs and one instrumental, with the same seven briefs sent to each option. That setup produces 21 clips. English-speaking reviewers assessed the recordings with option identities hidden and playback order shuffled.

The table contains the averages reported by our team. We keep the generation options anonymous and do not name the individual model providers.

Pronunciation averages cover the six vocal briefs; production and overall cover all seven. Overall is a separate listening judgment, not an arithmetic average of pronunciation and production. Option 1 has the highest reported average on all three criteria in this comparison.

The test date and reviewer count are not recorded here. We also do not map these anonymous options to our current generator. Those gaps limit what the averages establish, particularly for anyone trying to reproduce the test or compare results across model versions.

Five-language launch inputs

Our launch check used original verse-and-chorus lyrics in English, Spanish, German, French and Brazilian Portuguese. The English sheet contained two verses with returning choruses; the other sheets contained a verse and chorus each.

The language sheets describe ordinary scenes and relationships. They are different original lyrics, rather than literal translations of one song. This original chorus appears in the English sheet:

Come home slow, take your time
There is a light at the end of the line
Come home slow, let it shine
This little ordinary life is mine

The different words and durations matter when interpreting the results. These inputs check several working language requests; they do not create a controlled comparison of which language performs best.

Our prompt library offers other musical directions. A fast rap passage, a large choir or a dense electronic arrangement would require different listening material before extending conclusions to those uses.

Our completed launch results

On October 14, 2026, all five of our requested recordings completed. There was no hidden product identity or shuffled scoring sheet in this check. One reviewer listened to and accepted the set for launch, without retaining separate numerical pronunciation, production or overall scores.

Recorded generation time is the elapsed generation-job time, including waiting and processing. It excludes preparing the words, reviewing the recording and downloading it. These observations are dated measurements, not a speed guarantee.

The audio durations were close to the requested lengths in this batch. That does not establish that every chorus arrived at a particular second or that every word was preserved. Those questions require listening to the recording against the lyric sheet.

The limits of these records

Our launch check has five recordings and one review. It does not measure a typical failure rate, a typical number of attempts or a preference shared by all listeners. The English recording is longer than the other four, so timing differences cannot be read as a ranking between languages.

The English comparison covers seven briefs and one vocal language. The reviewer count and test date are not documented on this page, and the averages have not been checked against a clip-level score sheet. These scores cannot establish performance in Spanish, German, French or Brazilian Portuguese, or predict the quality of every new recording.

Neither record is a controlled comparison of Words into Music against Suno or Udio. Our comparison guide separates product research from listening results. A published price or feature can be checked without pretending that it establishes better sound.

How findings inform our product

The launch review supported offering our current five language choices and a workflow based on an idea or supplied lyrics, a separate sound description and a recording you can review. It did not establish perfect lyric adherence or control over every note.

The quality criteria help make a revision specific. If an important phrase disappears, review the lyric density and section order. If the words are correct but difficult to hear, simplify the accompaniment or request a more restrained vocal. Save the take you like before asking for another performance.

A new generation can change the melody, voice and arrangement. For your own songs, follow the audio with the lyric sheet and check the important lines before sharing. Our lyrics-to-song guide explains that process. When you are ready, make a song with a brief you can evaluate in the same way.