What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
Turning traces into a training dataset takes more than exporting logs: select traces that match a clearly defined task, correct or label the behavior you want a model to learn, protect sensitive information, and transform the examples into the format required by your training method. Keep evaluation data separate when possible, so you can check whether a change actually improves the system.
First decide what the dataset is for
A trace is evidence of what an application or agent did, not automatically a good example to train on. Decide whether you are building supervised fine-tuning data, an evaluation set, a preference dataset, or separate datasets for more than one purpose.
As an Amazon Associate I earn from qualifying purchases.
For training, each example needs a target that represents the behavior you want. For evaluation, each example needs a way to judge performance, such as an expected answer, required facts, tool-use constraints, or a scoring rubric. A dataset used to measure behavior can be rerun against different model, prompt, or agent versions; mixing those examples into training can make the evaluation less useful as an independent check.
Write down the behavior and success signal before collecting examples. For instance, a support assistant might need to answer a defined class of questions accurately, use a particular tool when required, or follow a response format. More traces will not resolve an unclear target.
#1 Best Overall
- Use scikit-learn to track an example ML project end to end
- Explore several models, including support vector machines, decision trees, random forests, and ensemble methods
- Exploit unsupervised learning techniques such as dimensionality reduction, clustering, and anomaly detection
- Dive into neural net architectures, including convolutional nets, recurrent nets, generative adversarial networks, autoencoders, diffusion models, and transformers
- Use TensorFlow and Keras to build and train neural nets for computer vision, natural language processing, generative models, and deep reinforcement learning
Capture and select relevant traces
A trace may contain several linked events or spans: the user request, model call, retrieval activity, tool arguments and results, and final response. Which fields are available depends on your instrumentation and trace platform. OpenTelemetry provides tracing instrumentation, collection, and export primitives; it does not label examples or prescribe a fine-tuning recipe (OpenTelemetry .NET traces).
Use the trace store or platform to select records by scenario, time window, outcome, or recorded attributes. Microsoft Foundry documents selecting an agent and time range for trace-based dataset generation; MLflow documents selecting records in its UI or querying them with its SDK (Microsoft Foundry trace-dataset workflow; MLflow evaluation datasets).
Then curate rather than simply maximizing row count. Exclude empty, malformed, irrelevant, or low-signal records; control near-duplicates so frequent requests do not overwhelm less common cases; and include meaningful failures when they help define the desired behavior. Review examples themselves, not just metadata or automated scores.
Rank #2
Foundry documents an automated sampling workflow that filters low-intent traffic and uses MinHash to select diverse examples, with handling for sensitive content including personal data. That is a capability of this specific Foundry workflow, not a property to assume of other trace platforms. MLflow offers a more explicitly hands-on approach: filter by tags or properties, sort or query records, and inspect poor outputs, edge cases, missing context, and faulty reasoning. Automated selection can reduce review effort, but it cannot establish that an example is fit for your task.
Make the target trustworthy
For supervised fine-tuning
Choose the desired response or behavior for each input. Do not copy a production answer into the target merely because it appears in the trace: if it is incorrect, unsafe, incomplete, or inconsistent with the intended behavior, that example can teach the wrong thing. Correct it, annotate it appropriately, or leave it out.
For evaluation
Define what a successful result looks like. Depending on the task, that may be an expected answer, a set of facts that must appear, constraints the answer must follow, required tool use, or a rubric. MLflow supports logging expectations on traces and adding those records to evaluation datasets, including expected answers (MLflow evaluation datasets).
Protect sensitive information and keep provenance
Review user prompts, model responses, retrieved material, tool arguments, and metadata for personal, confidential, or otherwise restricted information before reuse. Minimize fields that are not needed, apply the access and retention rules that govern your application, and preserve a source trace ID or equivalent provenance so a questionable row can be reviewed, corrected, or removed.
Recommended Free Tools
Vendor features do not settle your organization’s privacy or compliance obligations. Foundry says its sampling handles sensitive content, including personal data. OpenAI’s platform data-controls documentation says API data is not used to train or improve OpenAI models unless a customer opts in; it also describes retention and application-state behavior that varies by endpoint and settings. Check the current controls for the specific endpoint and account before sending or storing trace-derived data (Foundry trace-dataset workflow; OpenAI platform data controls).
Map traces to the destination format
There is no universal trace-to-dataset row format. Define a mapping from the fields you retained to the fields required by your selected model and training or evaluation method. A conceptual record might include conversation messages, relevant context, a target or expectation, scenario tags, quality labels, and source provenance, but that is not a vendor-specific schema.
Rank #4
Microsoft Foundry says its evaluation datasets typically use JSONL, with one JSON object per line and a messages field for model or agent interactions. If completed responses are included, Foundry can evaluate those responses directly; when evaluating against a live model or agent, it generates a new response for evaluation (Evaluation datasets in Microsoft Foundry).
OpenAI’s fine-tuning API requires a JSONL training file, but the expected contents differ for chat, completions, and preference methods. Transform traces for the particular target and method rather than assuming raw trace exports can be uploaded unchanged (OpenAI fine-tuning API reference).
Do these 3 things before closing this tab:
1Fix the driver behind crashes, sound loss and screen glitches2Repair Windows errors before they cause bigger problems3Scan for outdated or missing drivers - takes under a minuteValidate, version, and test the dataset
Before training or evaluation, inspect a sample of rows and check that:
Best Value
- Every record parses and includes the fields required by the destination schema.
- Conversation turns and tool interactions are ordered and represented consistently.
- Targets or expectations are nonempty and appropriate for the task.
- Duplicates are controlled and sensitive fields are handled as intended.
- Labels and corrections have a known source or review process.
Record the dataset version, source time window, transformation code or version, selection criteria, and label provenance. Foundry allows users to preview generated rows, download them, or delete the dataset; MLflow supports reusable evaluation datasets, expectations, and source-type provenance. These features aid review, but do not guarantee that the data is correct (Foundry trace-dataset workflow; MLflow evaluation datasets).
Keep a held-out evaluation set separate from training examples where possible. Run the model or agent on that set, examine aggregate measures and individual failures, and compare versions. If results regress, use provenance to find the relevant examples and revise the data or curation pipeline. A successful training run by itself is not evidence of improvement.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Choose a workflow that fits your trace setup
| Approach | Documented capabilities | Trade-offs to plan for |
|---|---|---|
| Microsoft Foundry | Select an agent and date range, generate trace-derived datasets in the portal or SDK, preview rows, and use the result for evaluation or fine-tuning; intelligent sampling is documented. | The trace-to-dataset feature is marked preview, and Microsoft says preview features may have constrained support and are not recommended for production workloads. Confirm current status, supported regions, SDK version, and permissions. Source |
| MLflow | Select traces through the UI or SDK, filter and inspect them, add expectations, and merge them into reusable evaluation datasets. | MLflow’s current documentation requires an MLflow Tracking Server with a SQL backend for evaluation datasets. Its workflow puts curation more directly in the user’s hands. Source |
| Custom pipeline | Export from an existing trace store or telemetry system, transform the data, and validate against the chosen destination. OpenTelemetry provides instrumentation and export primitives; OpenAI documents JSONL fine-tuning file requirements. | You own filtering, deduplication, privacy handling, labels, provenance, schema changes, and validation. OpenTelemetry; OpenAI fine-tuning API |
Compare workflows by trace selection and export control, labeling support, schema flexibility, provenance and versioning, privacy and retention controls, model compatibility, operational maturity, and the amount of custom pipeline work. The right choice depends on those needs; the documentation alone does not establish that one path produces better models or better data.
Free tools Windows power users keep installed
One-click scans. No signup required.
When production traces are not enough
Production traces show behavior and user scenarios that have occurred, but cannot cover every situation you need to test. Microsoft Learn describes production traces as representative of how an agent behaves with real users, and characterizes trace-based and synthetic generation as complementary: traces reflect observed user behavior, while synthetic generation can cover prelaunch scenarios and edge cases (Microsoft Foundry trace-dataset workflow). Use that distinction to identify gaps rather than treating observed traffic as complete coverage.
In Foundry’s documented dataset-creation flow, Microsoft recommends setting at least 15 samples. This is a product-specific setup recommendation, not a universal minimum for a useful training dataset.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
