Do these 3 things before closing this tab:
1Scan for outdated or missing drivers - takes under a minute2Clear out junk files and repair common Windows errors3Fix the driver behind crashes, sound loss and screen glitchesTo train a model to predict masked words in Keras, choose between building a compact BERT-like encoder yourself and using KerasHub’s BertMaskedLM task with a BERT preset. The first path makes the mechanics visible; the second streamlines preprocessing and model setup. Both train an MLM objective, but neither path should be mistaken for a guarantee of reproducing every part of original BERT pretraining.
What masked language modeling trains
Masked language modeling (MLM) is a self-supervised objective: select token positions, hide or otherwise corrupt the inputs at those positions, and train the model to predict the original token IDs. For example, a sentence might be presented with one token replaced by a mask token, while the target label at that position is the token that was there before corruption.
As an Amazon Associate I earn from qualifying purchases.
Keep the tokenizer and model’s representations aligned throughout the pipeline. The vocabulary, special-token IDs, sequence length, padding mask, selected mask positions, and target labels must all refer to the same tokenization. If they do not, the model can be asked to predict the wrong IDs or treat padding as meaningful content.
Free tools Windows power users keep installed
One-click scans. No signup required.
MLM is not the whole of original BERT pretraining. Google Research’s BERT repository describes masking 15% of input words and predicting only those masked words, as part of a workflow that also included next sentence prediction: Google Research BERT repository. KerasHub’s BertMaskedLM, by contrast, documents an MLM task; using that task does not automatically add next sentence prediction.
#1 Best Overall
Choose your Keras workflow
| Path | Best for | What it provides |
|---|---|---|
| Compact from-scratch encoder | Learning how the embeddings, encoder, and prediction head fit together | A small BERT-like model and an MLM training example; configuration is educational, not a full-scale BERT reproduction. |
| KerasHub preset and task API | Using an existing BERT configuration and weights with a supported MLM workflow | BertMaskedLM and, by default when constructed from a preset, preprocessing that can tokenize and dynamically mask raw text during fitting and evaluation. |
Path 1: build a compact BERT-like model from scratch
The Keras example End-to-end Masked Language Modeling with BERT demonstrates a small model built with TextVectorization and Keras attention layers. It trains with an MLM objective on IMDB reviews and later illustrates downstream sentiment fine-tuning. That downstream example is a separate use of the learned representation; it does not turn the sample into a full-scale BERT pretraining run.
Understand the sample configuration
The example’s settings are teaching values, not BERT-base specifications or universal production recommendations:
Rank #2
| Setting | Example value in the Keras tutorial | How to interpret it |
|---|---|---|
| Maximum sequence length | 256 | Tokens beyond the configured length cannot be represented as part of the same input sequence. |
| Batch size | 32 | A tutorial setting; feasible batch size depends on the model, sequence length, and hardware. |
| Learning rate | 0.001 | The tutorial’s example value, not a generally validated setting for other models or datasets. |
| Vocabulary size | 30,000 | Must match the vectorizer’s token-ID space and the prediction head’s output classes. |
| Embedding dimension | 128 | A compact example dimension, not a BERT-base specification. |
| Attention heads | 8 | The tutorial’s encoder setting. |
| Feed-forward dimension | 128 | The tutorial’s encoder setting. |
| Encoder layers | 1 | A single-layer teaching model, not a full-scale BERT stack. |
The tutorial page lists a creation date of 2020-09-18 and a last-modified date of 2024-03-15. It mentions a tf-nightly setup, but that line should not be treated as a current version-compatibility matrix. Check the current example and the installed TensorFlow/Keras packages before adapting code.
PC Slower Than It Used to Be?
A free scan shows the junk files, broken settings and background clutter dragging Windows down - then fixes them in one click.Free scan · Windows 10 & 11Outdated Drivers Are Slowing You Down
One free scan finds every outdated or missing driver and matches the right update for your exact hardware.Free scan · exact hardware matchKeep the targets consistent with masking
In a from-scratch pipeline, preprocessing must preserve the original token IDs for the selected positions so they can serve as labels after the input has been corrupted. The output layer must predict across the same vocabulary used by the tokenizer. Make sure padding positions are excluded from attention and from the loss, and that labels correspond only to positions selected for prediction.
Masking proportions are recipe choices, not fixed Keras defaults. Google Research’s original BERT repository describes masking 15% of input words. A separate KerasHub pretraining guide uses a 25% mask rate in its sample configuration. These figures describe different source-specific recipes, not a single required value for all models.
Path 2: use KerasHub’s BERT masked-LM task
KerasHub defines keras_hub.models.BertMaskedLM(backbone, preprocessor=None, **kwargs). Its BertMaskedLM API documentation shows loading a BERT preset with from_preset(). With preset preprocessing enabled by default, the task accepts raw strings and tokenizes and dynamically masks them during fitting and evaluation.
Rank #4
import keras_hub
masked_lm = keras_hub.models.BertMaskedLM.from_preset(
"bert_base_en_uncased",
)
# text_features is a collection of raw text strings.
masked_lm.fit(x=text_features, batch_size=batch_size)This is a streamlined starting point when the goal is to use the preset’s BERT-backed MLM workflow rather than implement the encoder and masking machinery yourself. Verify that the preset and API are available in the KerasHub version installed in your environment.
Recommended Free Tools
Use explicit preprocessed features when you need control
The API also supports explicit preprocessed input. Its example feature mapping includes token_ids, padding_mask, mask_positions, and segment_ids; labels are the original tokens at the masked positions. This route gives you control over tokenization, batching, and masking, but means you are responsible for keeping every tensor consistent with the tokenizer and preprocessor conventions.
The API example uses zero as the mask token ID for its illustrative input. Do not assume zero is the correct mask ID for another vocabulary: obtain special-token IDs from the tokenizer or preprocessor you actually use.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Build a custom KerasHub pretraining pipeline
For more control than the preset’s raw-text path, the Keras team’s Transformer pretraining with KerasHub guide describes a pipeline based on WordPiece tokenization and MaskedLMMaskGenerator. The masking step can be mapped over a tf.data input pipeline so selected positions are generated as batches are iterated. The model encodes token IDs, and MaskedLMHead gathers the encodings at selected positions and projects them to vocabulary predictions.
The guide’s sample uses sequence length 128, MASK_RATE = 0.25, and PREDICTIONS_PER_SEQ = 32. Those are configuration choices in that guide’s example, not measured performance results or defaults required by KerasHub. The original BERT repository advises setting the maximum predictions per sequence to roughly the maximum sequence length multiplied by the masked-LM probability, and using that value consistently in data generation and training.
The KerasHub guide’s example compiles with sparse categorical cross-entropy, AdamW, and weighted sparse categorical accuracy. Sparse categorical loss is appropriate when labels are integer token IDs; the weighting is relevant because only selected masked positions contribute to the MLM objective. Match the loss input shape, label shape, and sample weights to the outputs produced by your own preprocessing pipeline.
Check the pipeline before training
- Vocabulary: the tokenizer’s token IDs and the model’s output vocabulary dimension agree.
- Special tokens: mask, padding, and any separator or classification token IDs come from the active tokenizer or preprocessor.
- Sequence and padding: sequences use the intended maximum length, and the padding mask marks actual tokens rather than padded positions.
- Mask positions: each selected position points to a token that was corrupted in the input.
- Labels: each label is the original token ID for its selected position, not the mask ID or a shifted token.
- Loss scope: padding and unselected positions do not accidentally contribute as ordinary prediction targets.
Plan for compute and scope
Transformer pretraining can be computationally intensive. Training cost varies with model size, dataset, sequence length, batch size, and hardware, so there is no generic runtime or minimum-hardware promise that applies to both a compact example and a BERT preset. A small from-scratch model is useful for understanding the objective; it should not be presented as reproducing full-scale BERT pretraining.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
