What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
Some links on this page are affiliate links: if you buy through them we may earn a commission, at no extra cost to you.
The most practical way to implement Lucas–Kanade optical flow in Python is to detect strong corners with cv2.goodFeaturesToTrack(), track them between grayscale video frames with cv2.calcOpticalFlowPyrLK(), discard failed tracks, and visualize the resulting displacement vectors or trajectories.
This is sparse optical flow: it estimates motion only at selected feature points, not at every pixel. The method is fast and useful for camera tracking, stabilization, robotics, and motion analysis, provided that frame-to-frame motion is moderate and the scene contains reliable texture.
What Lucas–Kanade optical flow measures
Optical flow estimates the apparent two-dimensional movement of image structures between consecutive frames. For a point at (x, y), its motion is commonly represented by:
(u, v)uis horizontal displacement in pixels.vis vertical displacement in pixels.
This is image motion, not automatically the true three-dimensional velocity of an object. Camera movement, depth, lighting changes, reflections, occlusion, and independently moving objects all influence the measured flow. The OpenCV tutorial describes Lucas–Kanade as a sparse optical-flow method that tracks supplied feature points rather than producing a vector for every pixel.
#1 Best Overall
Read OpenCV’s optical-flow documentation.
Install the dependencies
For a desktop environment with OpenCV display support:
python -m pip install opencv-python numpyFor a server without GUI support, choose the headless package instead. Do not install both OpenCV distributions in the same environment:
python -m pip install opencv-python-headless numpyRecord the environment when reproducing an example:
python --version
python -m pip show opencv-python numpyThe example below expects an input file named input.mp4. A camera can be used instead by passing a camera index such as 0 to cv2.VideoCapture().
How Lucas–Kanade works
Brightness constancy
Lucas–Kanade assumes that the brightness of a moving image point remains approximately constant between frames:
I(x, y, t) ≈ I(x + u, y + v, t + Δt)Applying a first-order Taylor expansion gives the optical-flow constraint equation:
Iₓu + Iᵧv + Iₜ = 0Iₓis the horizontal image gradient.Iᵧis the vertical image gradient.Iₜis the temporal intensity change.uandvare the unknown motion components.
A single pixel provides one equation with two unknowns. Lucas–Kanade resolves this ambiguity by assuming that nearby pixels in a small window share approximately the same motion.
Why corners are selected
A flat region has little gradient information. An edge usually reveals motion only perpendicular to the edge, which is the aperture problem. A corner has meaningful intensity variation in two directions and therefore provides a better-conditioned estimate.
goodFeaturesToTrack() is the feature detector; calcOpticalFlowPyrLK() is the tracker. Lucas–Kanade does not automatically decide which image points are good enough to track, so the two functions are commonly used together. OpenCV’s detector uses the Shi–Tomasi corner criterion.
The local least-squares system
For the pixels in a feature window, the equations form:
[ Ix₁ Iy₁ ] [u] [−It₁]
[ Ix₂ Iy₂ ] [v] = [−It₂]
[ ⋮ ] [ ⋮ ]
[ Ixₙ Iyₙ ] [−Itₙ]Writing this as A d = b, the least-squares displacement is:
Free tools Windows power users keep installed
One-click scans. No signup required.
d = (AᵀA)⁻¹AᵀbIn practice, a weak or nearly singular AᵀA means that the feature is poorly constrained. OpenCV can reject such points using minEigThreshold, which relates to the minimum eigenvalue of the local gradient matrix. A production implementation should not blindly invert an ill-conditioned matrix.
Why pyramidal Lucas–Kanade is better for larger motion
Single-scale Lucas–Kanade assumes that motion is small relative to its tracking window. If a point moves too far, the local linear approximation may fail.
Pyramidal Lucas–Kanade builds reduced-resolution versions of both frames. It estimates motion at a coarse level, propagates that estimate to a finer level, and refines it iteratively. A large displacement in the original image becomes smaller at a lower resolution.
- Build an image pyramid for the previous and current frames.
- Estimate a coarse displacement at the smallest level.
- Upscale that estimate at the next level.
- Refine it using the local Lucas–Kanade equations.
Pyramids extend the useful motion range, but they do not make the method unlimited. Very large frame gaps, severe blur, occlusion, or major appearance changes can still cause failure. See Bouguet’s description of the pyramidal Lucas–Kanade tracker.
Complete Python implementation
This example validates the video and frames, filters invalid tracks, draws motion trails, and redetects features when too few remain.
from pathlib import Path
import cv2
import numpy as np
VIDEO_PATH = Path("input.mp4")
FEATURE_PARAMS = {
"maxCorners": 200,
"qualityLevel": 0.3,
"minDistance": 7,
"blockSize": 7,
}
LK_PARAMS = {
"winSize": (21, 21),
"maxLevel": 3,
"criteria": (
cv2.TERM_CRITERIA_EPS | cv2.TERM_CRITERIA_COUNT,
30,
0.01,
),
}
def main() -> None:
cap = cv2.VideoCapture(str(VIDEO_PATH))
if not cap.isOpened():
raise RuntimeError(f"Could not open video: {VIDEO_PATH}")
ok, first_frame = cap.read()
if not ok or first_frame is None:
raise RuntimeError("Could not read the first video frame")
previous_gray = cv2.cvtColor(first_frame, cv2.COLOR_BGR2GRAY)
previous_points = cv2.goodFeaturesToTrack(
previous_gray,
mask=None,
**FEATURE_PARAMS,
)
if previous_points is None:
raise RuntimeError("No suitable features were detected")
trail = np.zeros_like(first_frame)
colors = np.random.default_rng(0).integers(
0, 255, size=(FEATURE_PARAMS["maxCorners"], 3)
)
while True:
ok, frame = cap.read()
if not ok or frame is None:
break
current_gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)
current_points, status, error = cv2.calcOpticalFlowPyrLK(
previous_gray,
current_gray,
previous_points,
None,
**LK_PARAMS,
)
if current_points is None or status is None:
break
valid = status.reshape(-1) == 1
old_valid = previous_points.reshape(-1, 2)[valid]
new_valid = current_points.reshape(-1, 2)[valid]
for i, (old, new) in enumerate(zip(old_valid, new_valid)):
old_x, old_y = np.round(old).astype(int)
new_x, new_y = np.round(new).astype(int)
color = tuple(int(value) for value in colors[i % len(colors)])
cv2.line(
trail,
(old_x, old_y),
(new_x, new_y),
color,
thickness=2,
)
cv2.circle(
frame,
(new_x, new_y),
radius=4,
color=color,
thickness=-1,
)
output = cv2.add(frame, trail)
cv2.imshow("Lucas-Kanade optical flow", output)
key = cv2.waitKey(30) & 0xFF
if key == 27 or key == ord("q"):
break
if len(new_valid) < 10:
replacement_points = cv2.goodFeaturesToTrack(
current_gray,
mask=None,
**FEATURE_PARAMS,
)
if replacement_points is None:
break
previous_points = replacement_points
trail = np.zeros_like(frame)
else:
previous_points = new_valid.reshape(-1, 1, 2)
previous_gray = current_gray
cap.release()
cv2.destroyAllWindows()
if __name__ == "__main__":
main()The central sequence matches OpenCV’s official Python sample: read a frame, detect corners, track them in the next frame, retain points with a valid status, draw the tracks, and update the previous image and point set. The sample is a concise demonstration, not a complete solution for every production failure mode. See the official OpenCV sample.
Understanding the OpenCV outputs
next_points, status, error = cv2.calcOpticalFlowPyrLK(
previous_gray,
current_gray,
previous_points,
None,
**LK_PARAMS,
)next_points: estimated locations in the current frame.status: one value per input point. A value of1means OpenCV found a usable result according to its internal criteria; it is not proof that the correspondence is physically correct.error: a tracking-error measure. Treat it as implementation-specific rather than as a universal probability or confidence score.
The displacement vectors are calculated by subtracting the old coordinates from the new coordinates:
flow = new_valid - old_valid
dx = flow[:, 0]
dy = flow[:, 1]
speed_in_pixels = np.linalg.norm(flow, axis=1)
mean_motion = flow.mean(axis=0)These values describe displacement per processed frame. They are not automatically pixels per second. If the footage is processed at its intended frame rate:
Recommended Free Tools
pixels_per_second = speed_in_pixels * fpsParameter tuning
Shi–Tomasi feature parameters
| Parameter | Purpose | Practical effect |
|---|---|---|
maxCorners |
Maximum number of features | More points improve coverage only when they are reliable and well distributed. |
qualityLevel |
Relative corner-quality threshold | Increase it for fewer, stronger points; decrease it when too few features are found. |
minDistance |
Minimum spacing between points | Increase it to prevent clusters; decrease it when features are too sparse. |
blockSize |
Neighborhood used for quality evaluation | Larger neighborhoods use more surrounding image context. |
A smaller set of strong, distributed points is usually more useful than hundreds of weak points concentrated in one textured area.
Lucas–Kanade parameters
| Parameter | Purpose | Trade-off |
|---|---|---|
winSize |
Local search and update window | Larger windows tolerate more motion but can combine different motions across boundaries. |
maxLevel |
Highest pyramid level | Higher values can handle larger motion but cost more and may lose fine detail. |
criteria |
Iteration stopping rule | COUNT limits iterations; EPS stops when updates become small. |
minEigThreshold |
Minimum feature conditioning threshold | Rejects windows with insufficient gradient information. |
Increasing every parameter is not a reliable tuning strategy. For large motion, try a shorter frame interval, better image quality, a suitable window, and additional pyramid levels together.
Handling common failures
The video does not open
Check the path, codec support, camera permissions, and camera index. Always test:
if not cap.isOpened():
raise RuntimeError("Could not open video")On a headless server, cv2.imshow() may also fail. Save output frames or use opencv-python-headless when a GUI is unnecessary.
The first frame is empty
Check the return value before converting it:
ok, frame = cap.read()
if not ok or frame is None:
raise RuntimeError("Could not read frame")Do not pass None to cv2.cvtColor().
No corners are detected
The scene may be dark, blurred, textureless, or outside the selected region. Try improving illumination, lowering qualityLevel, reducing minDistance, using an ROI, or supplying a mask. Redetect after a scene change rather than continuing with an empty point set.
Rank #4
Most points disappear
Possible causes include motion blur, defocus, occlusion, points leaving the image, lighting changes, or motion larger than the configured search range. Try a smaller frame interval, better shutter speed, a carefully larger winSize, or a higher maxLevel. These changes cannot compensate for severe image degradation.
Tracks drift
A point can remain marked valid while gradually moving away from the physical feature. Useful defenses include:
- Track forward and then backward, rejecting points whose return position differs too much.
- Use affine or homography estimation with RANSAC and retain geometric inliers.
- Redetect features periodically.
- Limit track age.
- Reject points near image borders.
- Maintain spatially distributed features.
Point shapes cause errors
OpenCV commonly represents points as (N, 1, 2). After filtering, NumPy may produce (N, 2). Reshape them before passing them back:
Do these 3 things before closing this tab:
1Scan for outdated or missing drivers - takes under a minute2Clear out junk files and repair common Windows errors3Fix the driver behind crashes, sound loss and screen glitchesvalid = status.reshape(-1) == 1
points = current_points.reshape(-1, 2)[valid]
points = points.reshape(-1, 1, 2)Flow coordinates are floating-point values. Round them before drawing, and check image bounds before using them for array indexing.
Inconsistent image input
Convert both frames in the same way, normally to 8-bit grayscale:
previous_gray = cv2.cvtColor(previous_frame, cv2.COLOR_BGR2GRAY)
current_gray = cv2.cvtColor(current_frame, cv2.COLOR_BGR2GRAY)Making the tracker more reliable
Redetect features periodically
Features are temporary observations, not permanent object identities. Redetect when the valid count becomes low, after a scene cut, after substantial camera movement, or at a fixed interval. If trajectory continuity matters, maintain track IDs rather than simply replacing the entire point set.
Distribute points spatially
Good corners can cluster around one object. For camera-motion estimation, divide the image into grid cells and keep only a limited number per cell. This prevents one local object from dominating the global estimate.
Quick wins for a faster PC:
Scan for outdated or missing drivers - takes under a minuteDriver Scan →Repair Windows errors before they cause bigger problemsFix Now →Use forward–backward validation
Track a point from frame A to frame B, then track the result from B back to A. Reject it when the returned position is too far from the original. This catches many false correspondences that a forward status value alone cannot detect.
Best Value
Estimate global motion robustly
If the goal is camera motion rather than individual object motion:
- Track points between frames.
- Remove invalid tracks.
- Estimate an affine transform or homography with a robust method such as RANSAC.
- Use only geometric inliers for stabilization or camera-motion analysis.
Do not assume that the average displacement of all points equals camera motion. Moving objects can bias that average substantially.
Improve throughput
- Resize very large frames when full resolution is unnecessary.
- Limit the number of tracked points.
- Avoid unnecessary color conversions.
- Separate drawing from measurement when visualization is not required.
- Choose a frame interval that matches the expected motion.
- Benchmark with representative footage.
When sparse Lucas–Kanade is the wrong choice
| Requirement | Better direction |
|---|---|
| Track selected corners or feature trajectories | Pyramidal Lucas–Kanade |
| Estimate motion at most or all pixels | Dense optical flow, such as Farneback |
| Track a known object region | Lucas–Kanade with ROI and feature management |
| Estimate camera motion | Lucas–Kanade tracks followed by robust affine or homography fitting |
| Handle severe appearance changes | Feature matching or learned optical-flow methods |
| Handle substantial deformation | A method designed for nonrigid motion or dense flow |
Use sparse Lucas–Kanade when selected trajectories are enough, latency matters, and the scene contains textured points with moderate frame-to-frame motion. It is a poor fit when dense per-pixel motion, severe deformation, frequent occlusion, or major illumination changes are central to the task.
The Tool Desk
Outbyte Driver Updater FREEFix the driver behind crashes, sound loss and screen glitchesFind Drivers →Outbyte PC Repair FREEClear out junk files and repair common Windows errorsFree Scan →Farneback is not simply a “better” Lucas–Kanade implementation. It answers a different question by estimating dense rather than sparse motion.
Implementing Lucas–Kanade from scratch
Use OpenCV for an application; implement the equations manually when the goal is learning. A teaching implementation should:
- Convert frames to grayscale.
- Compute
Iₓ,Iᵧ, andIₜ. - Extract a window around each point.
- Build the matrix
Aand vectorb. - Solve the least-squares displacement.
- Iterate using the updated displacement and image interpolation.
- Reject poorly conditioned windows.
- Add an image pyramid for larger motion.
A compact educational solver for one window is:
def solve_lucas_kanade(ix, iy, it):
A = np.column_stack((ix.ravel(), iy.ravel()))
b = -it.ravel()
normal_matrix = A.T @ A
if np.linalg.det(normal_matrix) < 1e-6:
return None
displacement, *_ = np.linalg.lstsq(A, b, rcond=None)
return displacementThis is not equivalent to OpenCV’s full implementation. It omits interpolation, border handling, iterative warping, robust weighting, careful conditioning checks, and pyramid construction.
Summary
A dependable Lucas–Kanade pipeline is:
- Open and validate the video.
- Convert consecutive frames to grayscale.
- Detect Shi–Tomasi corners with
goodFeaturesToTrack(). - Track them with pyramidal
calcOpticalFlowPyrLK(). - Filter using
statusand stronger checks when needed. - Visualize or analyze
new_points - old_points. - Redetect features as points disappear or coverage deteriorates.
The method is fast and practical, but its result is apparent image displacement. Validate tracks, use robust model fitting for global motion, and choose dense or more advanced methods when the scene exceeds sparse corner tracking’s assumptions.
Windows Errors? Fix Them Before They Spread
Repair common Windows errors and clear accumulated junk for a smoother, more stable PC - no reinstall needed.Free scan · no reinstallOutdated Drivers Are Slowing You Down
One free scan finds every outdated or missing driver and matches the right update for your exact hardware.Free scan · exact hardware matchQuick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
