The Tool Desk
Outbyte PC Repair FREERepair Windows errors before they cause bigger problemsFix Now →Outbyte Driver Updater FREEFix the driver behind crashes, sound loss and screen glitchesFind Drivers →A Keras dual-encoder image search system turns a sentence into a vector, then retrieves images whose vectors are close to it. Khalid Salama’s Keras example shows the approach with an Xception image encoder, a BERT text encoder and a shared embedding space. It is a useful architectural demonstration, but it is an older implementation dated January 30, 2021—not current installation guidance.
What a dual encoder does
A dual encoder, also called a two-tower model, has one neural network for images and another for text. Training teaches both encoders to represent related images and captions as vectors in the same embedding space. Once trained, the system can encode a natural-language query and compare it directly with image vectors prepared in advance.
As an Amazon Associate I earn from qualifying purchases.
The Keras example is described as inspired by CLIP. Its training objective uses pairwise caption–image dot-product similarities and cross-entropy; target similarities also account for caption–caption and image–image relationships. The combined model is used for training, while retrieval uses the separately fine-tuned image and text encoders.
Free tools Windows power users keep installed
One-click scans. No signup required.
What the Keras example uses
| Part | Implementation in the tutorial |
|---|---|
| Image encoder | ImageNet-pretrained Xception without its classification head, using average pooling. It takes 299 × 299 RGB images, applies Xception preprocessing and feeds its representation through projection layers. |
| Text encoder | An uncased small BERT model and its preprocessing, loaded through TensorFlow Hub. The pooled BERT output passes through projection layers. |
| Shared representation | Projection heads map both encoders’ outputs to the same dimensionality so text and image vectors can be compared. |
| Base-model training | The example freezes the base encoders by default. |
For its MS-COCO example, the tutorial describes a collection of more than 82,000 images, each with at least five captions. Its configuration samples 30,000 training images and two captions per image, producing 60,000 caption–image pairs. The page reports a 13 GB compressed image archive for that dataset; this is not a general storage estimate for other collections.
#1 Best Overall
- Use scikit-learn to track an example ML project end to end
- Explore several models, including support vector machines, decision trees, random forests, and ensemble methods
- Exploit unsupervised learning techniques such as dimensionality reduction, clustering, and anomaly detection
- Dive into neural net architectures, including convolutional nets, recurrent nets, generative adversarial networks, autoencoders, diffusion models, and transformers
- Use TensorFlow and Keras to build and train neural nets for computer vision, natural language processing, generative models, and deep reinforcement learning
How image retrieval works
- Build the image index: Run the trained vision encoder over each image and retain its vector alongside the image path or another identifier.
- Encode the query: Send a sentence such as “a family standing next to the ocean on a sandy beach with a surf board” through the text encoder.
- Compare vectors: Normalize the query and image embeddings in the tutorial’s retrieval function, then compute dot products between the query vector and image vectors.
- Return results: Select the top-k scores, map their indices back to image paths and display those images.
The tutorial’s other example prompts include “a plate of healthy food,” “a woman wearing a hat is walking down a sidewalk,” “a bird sits near to the water” and “wild animals are standing in a field.” They illustrate the kind of sentence the model accepts; they do not establish what people search for most often.
Exact search versus approximate search
The demonstration computes exact dot-product matches against the image vectors. This is straightforward, but a large collection can make searching every vector for every query costly. The tutorial names ScaNN, Annoy and Faiss as approximate-matching options for real-time search at larger scale. It does not benchmark or rank them, so the right choice depends on measured latency and retrieval quality for the target system.
Rank #2
Generating vectors for a large collection can also be parallelized; the page mentions Apache Spark and Apache Beam as possible frameworks. In a production design, account for the cost and throughput of initial embedding generation, updates when images change, and the extra complexity of maintaining an index. These are implementation considerations, not performance results reported by the tutorial.
Quick wins for a faster PC:
Clear out junk files and repair common Windows errorsFree Scan →Scan for outdated or missing drivers - takes under a minuteDriver Scan →Repair Windows errors before they cause bigger problemsFix Now →What the reported evaluation means
The tutorial reports 6.235% evaluation top-k accuracy for its run. Its evaluation uses captions against images outside the training sample and counts a hit when the associated image appears within the top-k results; the printed metric uses k=100. This figure describes that tutorial’s dataset, configuration and evaluation, not a general benchmark or a predicted result for another collection.
To assess a new system, use representative held-out queries and state both the retrieval metric and the chosen top-k. Also measure query latency and indexing cost at the intended collection size. A result from one historical setup cannot answer whether another model will be accurate enough or fast enough for a particular application.
Improving results and adapting the example
Salama suggests increasing the training sample and training for more epochs, trying other image and text encoders, unfreezing the base models, and tuning hyperparameters—especially the loss temperature. These are proposed avenues to explore, not improvements demonstrated by a controlled comparison in the tutorial. Evaluate each change against a held-out set that reflects the queries and images the finished search tool will handle.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Version and compatibility caveats
The Keras page was created and last modified on January 30, 2021. It specifies TensorFlow 2.4 or higher and lists TensorFlow Hub, TensorFlow Text and TensorFlow Addons as dependencies of that period. Those references document the example’s original setup; they do not verify that the same installation steps work with current package releases. Check compatibility among TensorFlow, Keras, Hub, Text and Addons before attempting to reproduce it.
The accompanying Hugging Face model card says its TF-Keras loading path requires keras<3.x or tf_keras. It also describes a reproduction trained with 30,000 images and states on that page that the model is not deployed by an Inference Provider. That statement is specific to the card and does not mean that independent third parties cannot run the model. For the documented architecture, data setup and original code, see Keras: Natural language image search with a Dual Encoder.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
