Current-status note — September 4, 2026: This article restores a proof of concept originally published on August 3, 2020. Amazon SageMaker A2I is no longer open to new customers; existing customers can continue using it, but AWS does not plan to introduce new features. The sample-notebook repository was archived on May 21, 2025 and is read-only. Treat the notebook as historical reference code. For new SageMaker notebook instances, follow the current platform guidance, including use of the supported Amazon Linux 2023 environment.
Businesses use audio and video for advertising, customer service, media production, employee training, and education. Accurate transcripts make this content easier to search, organize, repurpose, and present accessibly.
Amazon Transcribe is an automatic speech-recognition service. When recordings contain specialized terminology, its custom-vocabulary feature can help the service recognize names, acronyms, and domain-specific expressions that a general speech model may render incorrectly.
This walkthrough demonstrates a human-in-the-loop method for discovering those terms. It selects words with low transcription-confidence scores, sends the surrounding audio to reviewers through Amazon Augmented AI (Amazon A2I), extracts vocabulary candidates from their corrections, reruns Amazon Transcribe with a custom vocabulary, and evaluates the result. The complete historical implementation is in A2I-Video-Transcription-with-Amazon-Transcribe.ipynb.
The motivating example was the phrase “an EC2 instance,” which the baseline transcript rendered as “Annecy two instance.” After EC2 was represented in the custom vocabulary, the new transcript rendered the phrase correctly.
Although the example uses MP4 video, the same design can process audio files. Upload the audio to Amazon S3, supply it as the transcription input, and revise the worker instructions so reviewers are told to listen to an audio clip.
Solution overview
1. Perform an initial transcription. Run Amazon Transcribe without a custom vocabulary, retain word timestamps and confidence scores, and establish a baseline transcript.
2. Create a human-review workflow. Select low-confidence words and send short surrounding clips to reviewers through an Amazon A2I custom task.
3. Build a custom vocabulary. Parse the corrected text, identify likely domain terms, and manually vet the candidates before formatting them as an Amazon Transcribe vocabulary table.
4. Transcribe and evaluate again. Repeat transcription with the custom vocabulary and compare the output with a hand-prepared reference transcript using word error rate and term-specific accuracy.
Prerequisites
The 2020 walkthrough used an AWS account, an Amazon S3 bucket, an IAM execution role, a SageMaker notebook instance, and a private human-review work team. The S3 input and output buckets and the private workforce must be in the same AWS Region as the A2I workflow. A Transcribe custom vocabulary must likewise be created in the same Region as the transcription that uses it.
The original notebook attached the AWS-managed AmazonAugmentedAIFullAccess and AmazonTranscribeFullAccess policies and granted access to the relevant S3 bucket. Those policies still exist, but full-access policies are broad. For production use, follow the current IAM least-privilege guidance and scope permissions to the required actions, roles, and S3 resources.
Existing A2I customers also need a private work team and its ARN. For this demonstration, the authors added themselves to the team so they could inspect the worker interface and complete their own review tasks.
Getting started
The notebook used four introductory SageMaker videos. Video 1, Fully-Managed Notebook Instances with Amazon SageMaker — a Deep Dive, supplied the baseline demonstration and vocabulary-building data. Video 3, Bring Your Own Custom ML Models with Amazon SageMaker, also contributed vocabulary candidates. These were the in-sample videos.
Video 2, Built-in Machine Learning Algorithms with Amazon SageMaker — a Deep Dive, and video 4, Train Your ML Models Accurately with Amazon SageMaker, were held out while the vocabulary was built. They served as out-of-sample tests of whether the vocabulary also helped on previously unseen recordings from the same subject area.
Step 1: Perform the initial transcription
The first pass establishes performance without a custom vocabulary. In the notebook, the helper call was:
transcribe(job_names[0], folder_path + all_videos[0], BUCKET)
After the job reaches COMPLETED, the notebook parses the result JSON into transcript text, word timestamps, and confidence scores. It also writes the baseline transcript to a text file for later comparison.
The authors plotted the confidence-score distribution and then selected words below a threshold. In the first demonstration, a threshold of 0.4 identified 16 words. The reported number is specific to the 2020 model output, recording, and notebook run; it should not be treated as a general property of Amazon Transcribe.
THRESHOLD = 0.4
all_bad_scores = [score for score in flat_scores_list if score < THRESHOLD]
A threshold is a triage policy, not a guarantee that every selected word is wrong or that every error has low confidence. Raising it may capture more transcription errors, but it also creates more review tasks and may admit more false positives. Threshold choice therefore has to balance coverage, review cost, and the value of the errors being corrected.
Step 2: Create the Amazon A2I review workflow
The workflow requires a workforce, a worker task template, a flow definition, and one or more human loops. With an A2I custom task, the application itself decides when to call StartHumanLoop. Every such call produces a review task; confidence-based activation conditions are not configured inside the flow definition for custom tasks.
Each task lets a reviewer replay a short section of the recording, compare it with the original machine transcript, enter a correction, and submit the result. The application sends the S3 media path, original words, and clip boundaries as task input.
Restored worker-template fields
The archived copy lost several Liquid substitutions and the response field, leaving its displayed template incomplete. The accompanying notebook preserves the required elements. The essential expressions are:
<source src=“{{ task.input.filePath | grant_read_access }}” type=“audio/mp4”>
The original transcript is “{{ task.input.original_words }}”.
<crowd-text-area name=“transcription” rows=“2” label=“Your transcription” placeholder=“Please enter the transcribed text.”></crowd-text-area>
video.currentTime = {{ task.input.start_time }};
if (video.currentTime >= {{ task.input.end_time }}) { video.pause(); }
The grant_read_access filter produces worker access to the S3 object. The start and end values constrain playback to the relevant subsection, while crowd-text-area records the reviewer’s answer. These mechanisms remain documented in the current A2I custom-template reference.
To provide context, the notebook selects up to three words on either side of each low-confidence word and uses the first and last selected words’ timestamps as clip boundaries. This helps reviewers correct phrases rather than judging isolated tokens.
In the first video, the 0.4 threshold produced 15 review tasks from 16 low-confidence words because the task-generation logic operates on the selected sequences. Among the reported machine outputs were “every version of Annecy two instance is” and “I started using Boto three but I.” These examples show why reviewers need both audio and neighboring words.
Completed human-loop results are written to the configured S3 output location. A2I workflows can also be monitored through the A2I Runtime API and Amazon EventBridge, formerly known as CloudWatch Events.
Step 3: Extract terms and create a custom vocabulary
The notebook reads each completed task’s transcription answer and splits the corrected text into words. It then compares those words with the NLTK English words corpus. Words absent from that corpus become vocabulary candidates.
This is only a discovery heuristic. A corpus can omit legitimate ordinary words, while unusual words are not necessarily technical terms. The example output included both useful candidates—such as Boto3, ECR, EBS, Jupyter, S3, SDK, SageMaker, IAM, VPC, EC2, and BlazingText—and ordinary forms such as “including,” “started,” and “using.” Human vetting remains necessary.
The authors built their final vocabulary from corrections to videos 1 and 3, using a 0.5 confidence threshold for that larger experiment. The table represented how a phrase should be recognized, optionally how it sounds, and how it should appear in the final transcript. Examples included pronunciations or display forms for Amazon, Boto3, T3, ECR, EBS, Jupyter, /opt/ml, S3, SDK, SageMaker, IAM, VPC, EC2, and BlazingText.
Amazon Transcribe vocabulary tables use the fields Phrase, SoundsLike, IPA, and DisplayAs. Multiword entries in the Phrase field use hyphens, and pronounced acronym letters are separated with periods. As of September 2026, AWS identifies the table format as preferred and says that list-format custom vocabularies are being deprecated.
After saving the table to a text file and uploading it to S3, the notebook calls CreateVocabulary with a unique vocabulary name, the language code en-US, and the S3 URI. Processing must reach READY before the vocabulary can be used. A vocabulary’s language must match the transcription language, and its S3 file must be in the Region required by the Transcribe request.
Step 4: Transcribe with the custom vocabulary
The notebook starts a second transcription job and supplies the custom-vocabulary name:
transcribe(job_name_custom_vid_0, folder_path + all_videos[0], BUCKET, vocab_name=vocab_improved)
After the job completes, the new transcripts are written to separate text files. This preserves the baseline and customized outputs for comparison with the hand-prepared reference transcripts.
Results and analysis
The following figures are the results reported in the original 2020 AWS post. They have been restored as historical experimental results, not independently reproduced against the current Amazon Transcribe service.
Word error rate, or WER, compares a hypothesis transcript with a reference transcript. It is calculated as (S + D + I) / N, where S is the number of substitutions, D deletions, I insertions, and N words in the reference. This edit-based approach is documented in the speech-recognition evaluation literature, including work published by NIST. The notebook used the JiWER Python package.
In-sample video 1: WER fell from 5.18% to 2.62%, reported as a 49.4% relative reduction.
In-sample video 3: WER fell from 11.94% to 7.84%, reported as a 34.4% relative reduction.
Out-of-sample video 2: WER fell from 7.55% to 6.56%, reported as a 13.1% relative reduction.
Out-of-sample video 4: WER fell from 10.91% to 8.98%, reported as a 17.6% relative reduction.
The direction of change was favorable in all four recordings, including the two held-out videos. The larger reductions on the in-sample recordings are consistent with the fact that those recordings supplied the terms used to construct the vocabulary.
Overall WER can hide errors in rare but important words. A transcript may correctly capture common connective language while repeatedly missing product names, acronyms, or technical expressions that matter for search and content organization. The authors therefore also counted correct occurrences of selected technical terms.
For the terms presented in the original comparison tables, the authors reported increases of at least 80 percentage points in correct transcription, with many selected terms reaching 100% in this four-video experiment. These figures should be read as sample-specific observations rather than a general accuracy guarantee.
What the experiment establishes
The experiment supports a practical pattern: when a model exposes useful uncertainty estimates, uncertain cases can be routed to targeted human review; reviewer corrections can then be converted into domain knowledge for a later pass. The held-out videos provide a more meaningful test than evaluating only on recordings used to discover the vocabulary.
It does not establish that a fixed confidence threshold will work for every domain, that all terminology errors have low confidence, or that a vocabulary built from a few recordings will generalize indefinitely. Teams should measure review volume, false positives, term recall, WER, and held-out performance with their own recordings.
Cleaning up
Delete resources that are no longer required to avoid continuing charges or retaining unnecessary data. Depending on what you created, this may include transcription jobs, custom vocabularies, A2I human loops and flow definitions, task templates, S3 objects or buckets, private-workforce resources, and the SageMaker notebook instance.
Conclusion
This 2020 proof of concept combined Amazon Transcribe confidence scores, Amazon A2I human review, and custom vocabularies to improve recognition of specialized terminology. Its durable lesson is architectural rather than service-specific: use uncertainty to focus human attention, convert verified corrections into reusable domain knowledge, and test the result on data that did not produce that knowledge.
Amazon Transcribe custom vocabularies remain available, but this particular A2I implementation is now primarily relevant to existing A2I customers and to readers studying historical human-in-the-loop designs. New implementations should confirm current service availability, security requirements, supported notebook environments, and vocabulary formatting before adapting the archived code.
About the authors
At publication in 2020, Jasper Huang was a technical-writing intern at AWS and a computer-science student at the University of Pennsylvania. His interests included cloud computing and machine learning.
At publication in 2020, Talia Chopra was an AWS technical writer specializing in machine learning and artificial intelligence, including work involving Amazon SageMaker, MXNet, and AutoGluon.
Responses