diff --git a/docs/docs/static/img/mlflow-tracing-rag.png b/docs/docs/static/img/mlflow-tracing-rag.png new file mode 100644 index 000000000..5c1c302dc Binary files /dev/null and b/docs/docs/static/img/mlflow-tracing-rag.png differ diff --git a/docs/docs/tutorials/agents/index.ipynb b/docs/docs/tutorials/agents/index.ipynb index fbf55e6a1..c8870e17a 100644 --- a/docs/docs/tutorials/agents/index.ipynb +++ b/docs/docs/tutorials/agents/index.ipynb @@ -8,7 +8,46 @@ "\n", "Let's walk through a quick example of setting up a `dspy.ReAct` agent with a couple of tools and optimizing it to conduct advanced browsing for multi-hop search.\n", "\n", - "Install the latest DSPy via `pip install -U dspy` and follow along." + "Install the latest DSPy via `pip install -U dspy` and follow along.\n", + "\n", + "
\n", + "Optional: Set up MLflow Tracing to understand what's happening under the hood.\n", + "\n", + "### MLflow DSPy Integration\n", + "\n", + "MLflow is an LLMOps tool that natively integrates with DSPy and offer explainability and experiment tracking. In this tutorial, you can use MLflow to visualize prompts and optimization progress as traces to understand the DSPy's behavior better. You can set up MLflow easily by following the four steps below.\n", + "\n", + "![MLflow Trace](./mlflow-tracing-agent.png)\n", + "\n", + "1. Install MLflow\n", + "\n", + "```bash\n", + "%pip install mlflow>=2.20\n", + "```\n", + "\n", + "2. Start MLflow UI in a separate terminal\n", + "```bash\n", + "mlflow ui --port 5000\n", + "```\n", + "\n", + "3. Connect the notebook to MLflow\n", + "```python\n", + "import mlflow\n", + "\n", + "mlflow.set_tracking_uri(\"http://localhost:5000\")\n", + "mlflow.set_experiment(\"DSPy\")\n", + "```\n", + "\n", + "4. Enabling tracing.\n", + "```python\n", + "mlflow.dspy.autolog()\n", + "```\n", + "\n", + "Once you have completed the steps above, you can see traces for each program execution on the notebook. They provide great visibility into the model's behavior and helps you understand the DSPy's concepts better throughout the tutorial.\n", + "\n", + "To kearn more about the integration, visit [MLflow DSPy Documentation](https://mlflow.org/docs/latest/llms/dspy/index.html) as well.\n", + "\n", + "
" ] }, { @@ -440,6 +479,54 @@ "evaluate(safe_react)" ] }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "
\n", + "Tracking Evaluation Results in MLflow Experiment\n", + "\n", + "
\n", + "\n", + "To track and visualize the evaluation results over time, you can record the results in MLflow Experiment.\n", + "\n", + "\n", + "```python\n", + "import mlflow\n", + "\n", + "with mlflow.start_run(run_name=\"agent_evaluation\"):\n", + " evaluate = dspy.Evaluate(\n", + " devset=devset,\n", + " metric=top5_recall,\n", + " num_threads=16,\n", + " display_progress=True,\n", + " # To record the outputs and detailed scores to MLflow\n", + " return_all_scores=True,\n", + " return_outputs=True,\n", + " )\n", + "\n", + " # Evaluate the program as usual\n", + " aggregated_score, outputs, all_scores = evaluate(cot)\n", + "\n", + " # Log the aggregated score\n", + " mlflow.log_metric(\"top5_recall\", aggregated_score)\n", + " # Log the detailed evaluation results as a table\n", + " mlflow.log_table(\n", + " {\n", + " \"Claim\": [example.claim for example in eval_set],\n", + " \"Expected Titles\": [example.titles for example in eval_set],\n", + " \"Predicted Titles\": outputs,\n", + " \"Top 5 Recall\": all_scores,\n", + " },\n", + " artifact_file=\"eval_results.json\",\n", + " )\n", + "```\n", + "\n", + "To learn more about the integration, visit [MLflow DSPy Documentation](https://mlflow.org/docs/latest/llms/dspy/index.html) as well.\n", + "\n", + "
" + ] + }, { "cell_type": "markdown", "metadata": {}, @@ -662,7 +749,7 @@ "source": [ "Awesome. It looks like the system improved drastically from 8% recall to around 40% recall. That was a pretty straightforward approach, but DSPy gives you many tools to continue iterating on this from here.\n", "\n", - "Next, let's inspect the optimized prompts to understand what it has learned. We'll run one query and then inspect the last two prompts, which will show us the prompts used for both ReAct sub-modules, the one that does the agentic loop and the other than prepares the final results." + "Next, let's inspect the optimized prompts to understand what it has learned. We'll run one query and then inspect the last two prompts, which will show us the prompts used for both ReAct sub-modules, the one that does the agentic loop and the other than prepares the final results. (Alternatively, if you enabled MLflow Tracing following the instructions above, you can see all steps done by the agent including LLM calls, prompts, tool execution, in a rich tree-view.)" ] }, { @@ -1350,6 +1437,42 @@ "\n", "loaded_react(claim=\"The author of the 1960s unproduced script written for The Beatles, Up Against It, and Bernard-Marie Koltès are both playwrights.\").titles" ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "
\n", + "Saving programs in MLflow Experiment\n", + "\n", + "
\n", + "\n", + "Instead of saving the program to a local file, you can track it in MLflow for better reproducibility and collaboration.\n", + "\n", + "1. **Dependency Management**: MLflow automatically save the frozen environment metadata along with the program to ensure reproducibility.\n", + "2. **Experiment Tracking**: With MLflow, you can track the program's performance and cost along with the program itself.\n", + "3. **Collaboration**: You can share the program and results with your team members by sharing the MLflow experiment.\n", + "\n", + "To save the program in MLflow, run the following code:\n", + "\n", + "```python\n", + "import mlflow\n", + "\n", + "# Start an MLflow Run and save the program\n", + "with mlflow.start_run(run_name=\"optimized_rag\"):\n", + " model_info = mlflow.dspy.log_model(\n", + " optimized_react,\n", + " artifact_path=\"model\", # Any name to save the program in MLflow\n", + " )\n", + "\n", + "# Load the program back from MLflow\n", + "loaded = mlflow.dspy.load_model(model_info.model_uri)\n", + "```\n", + "\n", + "To learn more about the integration, visit [MLflow DSPy Documentation](https://mlflow.org/docs/latest/llms/dspy/index.html) as well.\n", + "\n", + "
" + ] } ], "metadata": { diff --git a/docs/docs/tutorials/agents/mlflow-tracing-agent.png b/docs/docs/tutorials/agents/mlflow-tracing-agent.png new file mode 100644 index 000000000..f2beb707d Binary files /dev/null and b/docs/docs/tutorials/agents/mlflow-tracing-agent.png differ diff --git a/docs/docs/tutorials/classification_finetuning/index.ipynb b/docs/docs/tutorials/classification_finetuning/index.ipynb index 0e103e9b1..fbc798d65 100644 --- a/docs/docs/tutorials/classification_finetuning/index.ipynb +++ b/docs/docs/tutorials/classification_finetuning/index.ipynb @@ -24,6 +24,48 @@ "```" ] }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "
\n", + "Optional: Set up MLflow Tracing to understand what's happening under the hood.\n", + "\n", + "### MLflow DSPy Integration\n", + "\n", + "MLflow is an LLMOps tool that natively integrates with DSPy and offer explainability and experiment tracking. In this tutorial, you can use MLflow to visualize prompts and optimization progress as traces to understand the DSPy's behavior better. You can set up MLflow easily by following the four steps below.\n", + "\n", + "![MLflow Trace](./mlflow-tracing-classification.png)\n", + "\n", + "1. Install MLflow\n", + "\n", + "```bash\n", + "%pip install mlflow>=2.20\n", + "```\n", + "\n", + "2. Start MLflow UI in a separate terminal\n", + "```bash\n", + "mlflow ui --port 5000\n", + "```\n", + "\n", + "3. Connect the notebook to MLflow\n", + "```python\n", + "import mlflow\n", + "\n", + "mlflow.set_tracking_uri(\"http://localhost:5000\")\n", + "mlflow.set_experiment(\"DSPy\")\n", + "```\n", + "\n", + "4. Enabling tracing.\n", + "```python\n", + "mlflow.dspy.autolog()\n", + "```\n", + "\n", + "\n", + "To learn more about the integration, visit [MLflow DSPy Documentation](https://mlflow.org/docs/latest/llms/dspy/index.html) as well.\n", + "
" + ] + }, { "cell_type": "markdown", "metadata": {}, @@ -472,6 +514,54 @@ "evaluate(classify_ft)" ] }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "
\n", + "Tracking Evaluation Results in MLflow Experiment\n", + "\n", + "
\n", + "\n", + "To track and visualize the evaluation results over time, you can record the results in MLflow Experiment.\n", + "\n", + "\n", + "```python\n", + "import mlflow\n", + "\n", + "with mlflow.start_run(run_name=\"classifier_evaluation\"):\n", + " evaluate_correctness = dspy.Evaluate(\n", + " devset=devset,\n", + " metric=extraction_correctness_metric,\n", + " num_threads=16,\n", + " display_progress=True,\n", + " # To record the outputs and detailed scores to MLflow\n", + " return_all_scores=True,\n", + " return_outputs=True,\n", + " )\n", + "\n", + " # Evaluate the program as usual\n", + " aggregated_score, outputs, all_scores = evaluate_correctness(people_extractor)\n", + "\n", + " # Log the aggregated score\n", + " mlflow.log_metric(\"exact_match\", aggregated_score)\n", + " # Log the detailed evaluation results as a table\n", + " mlflow.log_table(\n", + " {\n", + " \"Text\": [example.text for example in devset],\n", + " \"Expected\": [example.example_label for example in devset],\n", + " \"Predicted\": outputs,\n", + " \"Exact match\": all_scores,\n", + " },\n", + " artifact_file=\"eval_results.json\",\n", + " )\n", + "```\n", + "\n", + "To learn more about the integration, visit [MLflow DSPy Documentation](https://mlflow.org/docs/latest/llms/dspy/index.html) as well.\n", + "\n", + "
" + ] + }, { "cell_type": "markdown", "metadata": {}, @@ -944,6 +1034,42 @@ "classify_ft(text=\"why hasnt my card come in yet?\")\n", "dspy.inspect_history()" ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "
\n", + "Saving fine-tuned programs in MLflow Experiment\n", + "\n", + "
\n", + "\n", + "To deploy the fine-tuned program in production or share it with your team, you can save it in MLflow Experiment. Compared to simply saving it to a local file, MLflow offers the following benefits:\n", + "\n", + "1. **Dependency Management**: MLflow automatically save the frozen environment metadata along with the program to ensure reproducibility.\n", + "2. **Experiment Tracking**: With MLflow, you can track the program's performance and cost along with the program itself.\n", + "3. **Collaboration**: You can share the program and results with your team members by sharing the MLflow experiment.\n", + "\n", + "To save the program in MLflow, run the following code:\n", + "\n", + "```python\n", + "import mlflow\n", + "\n", + "# Start an MLflow Run and save the program\n", + "with mlflow.start_run(run_name=\"optimized_classifier\"):\n", + " model_info = mlflow.dspy.log_model(\n", + " classify_ft,\n", + " artifact_path=\"model\", # Any name to save the program in MLflow\n", + " )\n", + "\n", + "# Load the program back from MLflow\n", + "loaded = mlflow.dspy.load_model(model_info.model_uri)\n", + "```\n", + "\n", + "To learn more about the integration, visit [MLflow DSPy Documentation](https://mlflow.org/docs/latest/llms/dspy/index.html) as well.\n", + "\n", + "
" + ] } ], "metadata": { diff --git a/docs/docs/tutorials/classification_finetuning/mlflow-tracing-classification.png b/docs/docs/tutorials/classification_finetuning/mlflow-tracing-classification.png new file mode 100644 index 000000000..27707cc0c Binary files /dev/null and b/docs/docs/tutorials/classification_finetuning/mlflow-tracing-classification.png differ diff --git a/docs/docs/tutorials/entity_extraction/index.ipynb b/docs/docs/tutorials/entity_extraction/index.ipynb index f298add18..8f14c4a4b 100644 --- a/docs/docs/tutorials/entity_extraction/index.ipynb +++ b/docs/docs/tutorials/entity_extraction/index.ipynb @@ -29,6 +29,47 @@ "%pip install datasets" ] }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "
\n", + "Optional: Set up MLflow Tracing to understand what's happening under the hood.\n", + "\n", + "### MLflow DSPy Integration\n", + "\n", + "MLflow is an LLMOps tool that natively integrates with DSPy and offer explainability and experiment tracking. In this tutorial, you can use MLflow to visualize prompts and optimization progress as traces to understand the DSPy's behavior better. You can set up MLflow easily by following the four steps below.\n", + "\n", + "![MLflow Trace](./mlflow-tracing-entity-extraction.png)\n", + "\n", + "1. Install MLflow\n", + "\n", + "```bash\n", + "%pip install mlflow>=2.20\n", + "```\n", + "\n", + "2. Start MLflow UI in a separate terminal\n", + "```bash\n", + "mlflow ui --port 5000\n", + "```\n", + "\n", + "3. Connect the notebook to MLflow\n", + "```python\n", + "import mlflow\n", + "\n", + "mlflow.set_tracking_uri(\"http://localhost:5000\")\n", + "mlflow.set_experiment(\"DSPy\")\n", + "```\n", + "\n", + "4. Enabling tracing.\n", + "```python\n", + "mlflow.dspy.autolog()\n", + "```\n", + "\n", + "To learn more about the integration, visit [MLflow DSPy Documentation](https://mlflow.org/docs/latest/llms/dspy/index.html) as well.\n", + "
" + ] + }, { "cell_type": "markdown", "metadata": {}, @@ -452,6 +493,54 @@ "evaluate_correctness(people_extractor, devset=test_set)" ] }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "
\n", + "Tracking Evaluation Results in MLflow Experiment\n", + "\n", + "
\n", + "\n", + "To track and visualize the evaluation results over time, you can record the results in MLflow Experiment.\n", + "\n", + "\n", + "```python\n", + "import mlflow\n", + "\n", + "with mlflow.start_run(run_name=\"extractor_evaluation\"):\n", + " evaluate_correctness = dspy.Evaluate(\n", + " devset=test_set,\n", + " metric=extraction_correctness_metric,\n", + " num_threads=24,\n", + " display_progress=True,\n", + " # To record the outputs and detailed scores to MLflow\n", + " return_all_scores=True,\n", + " return_outputs=True,\n", + " )\n", + "\n", + " # Evaluate the program as usual\n", + " aggregated_score, outputs, all_scores = evaluate_correctness(people_extractor)\n", + "\n", + " # Log the aggregated score\n", + " mlflow.log_metric(\"exact_match\", aggregated_score)\n", + " # Log the detailed evaluation results as a table\n", + " mlflow.log_table(\n", + " {\n", + " \"Tokens\": [example.tokens for example in test_set],\n", + " \"Expected\": [example.expected_extracted_people for example in test_set],\n", + " \"Predicted\": outputs,\n", + " \"Exact match\": all_scores,\n", + " },\n", + " artifact_file=\"eval_results.json\",\n", + " )\n", + "```\n", + "\n", + "To learn more about the integration, visit [MLflow DSPy Documentation](https://mlflow.org/docs/latest/llms/dspy/index.html) as well.\n", + "\n", + "
" + ] + }, { "cell_type": "markdown", "metadata": {}, @@ -946,6 +1035,42 @@ "loaded_people_extractor(tokens=[\"Italy\", \"recalled\", \"Marcello\", \"Cuttitta\"]).extracted_people" ] }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "
\n", + "Saving programs in MLflow Experiment\n", + "\n", + "
\n", + "\n", + "Instead of saving the program to a local file, you can track it in MLflow for better reproducibility and collaboration.\n", + "\n", + "1. **Dependency Management**: MLflow automatically save the frozen environment metadata along with the program to ensure reproducibility.\n", + "2. **Experiment Tracking**: With MLflow, you can track the program's performance and cost along with the program itself.\n", + "3. **Collaboration**: You can share the program and results with your team members by sharing the MLflow experiment.\n", + "\n", + "To save the program in MLflow, run the following code:\n", + "\n", + "```python\n", + "import mlflow\n", + "\n", + "# Start an MLflow Run and save the program\n", + "with mlflow.start_run(run_name=\"optimized_extractor\"):\n", + " model_info = mlflow.dspy.log_model(\n", + " optimized_people_extractor,\n", + " artifact_path=\"model\", # Any name to save the program in MLflow\n", + " )\n", + "\n", + "# Load the program back from MLflow\n", + "loaded = mlflow.dspy.load_model(model_info.model_uri)\n", + "```\n", + "\n", + "To learn more about the integration, visit [MLflow DSPy Documentation](https://mlflow.org/docs/latest/llms/dspy/index.html) as well.\n", + "\n", + "
" + ] + }, { "cell_type": "markdown", "metadata": {}, diff --git a/docs/docs/tutorials/entity_extraction/mlflow-tracing-entity-extraction.png b/docs/docs/tutorials/entity_extraction/mlflow-tracing-entity-extraction.png new file mode 100644 index 000000000..27707cc0c Binary files /dev/null and b/docs/docs/tutorials/entity_extraction/mlflow-tracing-entity-extraction.png differ diff --git a/docs/docs/tutorials/games/index.ipynb b/docs/docs/tutorials/games/index.ipynb index e6b794353..2cb5f4369 100644 --- a/docs/docs/tutorials/games/index.ipynb +++ b/docs/docs/tutorials/games/index.ipynb @@ -20,6 +20,48 @@ "```" ] }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "
\n", + "Optional: Setup MLflow Tracing for learning what's happening under the hood\n", + "\n", + "### MLflow DSPy Integration\n", + "\n", + "MLflow is an LLMOps tool that natively integrates with DSPy and offer explainability and experiment tracking. In this tutorial, you can use MLflow to visualize prompts and optimization progress as traces to understand the DSPy's behavior better. You can set up MLflow easily by following the four steps below.\n", + "\n", + "![MLflow Trace](./mlflow-tracing-agent.png)\n", + "\n", + "1. Install MLflow\n", + "\n", + "```bash\n", + "%pip install mlflow>=2.20\n", + "```\n", + "\n", + "2. Start MLflow UI in a separate terminal\n", + "```bash\n", + "mlflow ui --port 5000\n", + "```\n", + "\n", + "3. Connect the notebook to MLflow\n", + "```python\n", + "import mlflow\n", + "\n", + "mlflow.set_tracking_uri(\"http://localhost:5000\")\n", + "mlflow.set_experiment(\"DSPy\")\n", + "```\n", + "\n", + "4. Enabling tracing.\n", + "```python\n", + "mlflow.dspy.autolog()\n", + "```\n", + "\n", + "\n", + "To learn more about the integration, visit [MLflow DSPy Documentation](https://mlflow.org/docs/latest/llms/dspy/index.html) as well.\n", + "
" + ] + }, { "cell_type": "markdown", "metadata": {}, @@ -369,6 +411,53 @@ "evaluate = dspy.Evaluate(devset=devset, metric=metric, display_progress=True, num_threads=16)" ] }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "
\n", + "Tracking Evaluation Results in MLflow Experiment\n", + "\n", + "
\n", + "\n", + "To track and visualize the evaluation results over time, you can record the results in MLflow Experiment.\n", + "\n", + "\n", + "```python\n", + "import mlflow\n", + "\n", + "with mlflow.start_run(run_name=\"agent_evaluation\"):\n", + " evaluate = dspy.Evaluate(\n", + " devset=devset,\n", + " metric=metric,\n", + " num_threads=16,\n", + " display_progress=True,\n", + " # To record the outputs and detailed scores to MLflow\n", + " return_all_scores=True,\n", + " return_outputs=True,\n", + " )\n", + "\n", + " # Evaluate the program as usual\n", + " aggregated_score, outputs, all_scores = evaluate(cot)\n", + "\n", + " # Log the aggregated score\n", + " mlflow.log_metric(\"success_rate\", aggregated_score)\n", + " # Log the detailed evaluation results as a table\n", + " mlflow.log_table(\n", + " {\n", + " \"Idx\": [example.idx for example in eval_set],\n", + " \"Result\": outputs,\n", + " \"Success\": all_scores,\n", + " },\n", + " artifact_file=\"eval_results.json\",\n", + " )\n", + "```\n", + "\n", + "To learn more about the integration, visit [MLflow DSPy Documentation](https://mlflow.org/docs/latest/llms/dspy/index.html) as well.\n", + "\n", + "
" + ] + }, { "cell_type": "code", "execution_count": 7, @@ -556,6 +645,42 @@ "finetuned_4o_mini.save('finetuned_4o_mini_001.pkl')" ] }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "
\n", + "Saving programs in MLflow Experiment\n", + "\n", + "
\n", + "\n", + "Instead of saving the program to a local file, you can track it in MLflow for better reproducibility and collaboration.\n", + "\n", + "1. **Dependency Management**: MLflow automatically save the frozen environment metadata along with the program to ensure reproducibility.\n", + "2. **Experiment Tracking**: With MLflow, you can track the program's performance and cost along with the program itself.\n", + "3. **Collaboration**: You can share the program and results with your team members by sharing the MLflow experiment.\n", + "\n", + "To save the program in MLflow, run the following code:\n", + "\n", + "```python\n", + "import mlflow\n", + "\n", + "# Start an MLflow Run and save the program\n", + "with mlflow.start_run(run_name=\"optimized\"):\n", + " model_info = mlflow.dspy.log_model(\n", + " finetuned_4o_mini,\n", + " artifact_path=\"model\", # Any name to save the program in MLflow\n", + " )\n", + "\n", + "# Load the program back from MLflow\n", + "loaded = mlflow.dspy.load_model(model_info.model_uri)\n", + "```\n", + "\n", + "To learn more about the integration, visit [MLflow DSPy Documentation](https://mlflow.org/docs/latest/llms/dspy/index.html) as well.\n", + "\n", + "
" + ] + }, { "cell_type": "markdown", "metadata": {}, diff --git a/docs/docs/tutorials/games/mlflow-tracing-agent.png b/docs/docs/tutorials/games/mlflow-tracing-agent.png new file mode 100644 index 000000000..f2beb707d Binary files /dev/null and b/docs/docs/tutorials/games/mlflow-tracing-agent.png differ diff --git a/docs/docs/tutorials/math/index.ipynb b/docs/docs/tutorials/math/index.ipynb index 3e6c55894..8e7e562cc 100644 --- a/docs/docs/tutorials/math/index.ipynb +++ b/docs/docs/tutorials/math/index.ipynb @@ -8,7 +8,46 @@ "\n", "Let's walk through a quick example of setting up a `dspy.ChainOfThought` module and optimizing it for answering algebra questions.\n", "\n", - "Install the latest DSPy via `pip install -U dspy` and follow along." + "Install the latest DSPy via `pip install -U dspy` and follow along.\n", + "\n", + "
\n", + "Optional: Set up MLflow Tracing to understand what's happening under the hood.\n", + "\n", + "### MLflow DSPy Integration\n", + "\n", + "MLflow is an LLMOps tool that natively integrates with DSPy and offer explainability and experiment tracking. In this tutorial, you can use MLflow to visualize prompts and optimization progress as traces to understand the DSPy's behavior better. You can set up MLflow easily by following the four steps below.\n", + "\n", + "1. Install MLflow\n", + "\n", + "```bash\n", + "%pip install mlflow>=2.20\n", + "```\n", + "\n", + "2. Start MLflow UI in a separate terminal\n", + "```bash\n", + "mlflow ui --port 5000\n", + "```\n", + "\n", + "3. Connect the notebook to MLflow\n", + "```python\n", + "import mlflow\n", + "\n", + "mlflow.set_tracking_uri(\"http://localhost:5000\")\n", + "mlflow.set_experiment(\"DSPy\")\n", + "```\n", + "\n", + "4. Enabling tracing.\n", + "```python\n", + "mlflow.dspy.autolog()\n", + "```\n", + "\n", + "Once you have completed the steps above, you can see traces for each program execution on the notebook. They provide great visibility into the model's behavior and helps you understand the DSPy's concepts better throughout the tutorial.\n", + "\n", + "![MLflow Trace](./mlflow-tracing-math.png)\n", + "\n", + "To learn more about the integration, visit [MLflow DSPy Documentation](https://mlflow.org/docs/latest/llms/dspy/index.html) as well.\n", + "\n", + "
" ] }, { @@ -313,6 +352,48 @@ "evaluate(module)" ] }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "
\n", + "Tracking Evaluation Results in MLflow Experiment\n", + "\n", + "
\n", + "\n", + "To track and visualize the evaluation results over time, you can record the results in MLflow Experiment.\n", + "\n", + "\n", + "```python\n", + "import mlflow\n", + "\n", + "# Start an MLflow Run to record the evaluation\n", + "with mlflow.start_run(run_name=\"math_evaluation\"):\n", + " kwargs = dict(num_threads=THREADS, display_progress=True, return_all_scores=True, return_outputs=True)\n", + " evaluate = dspy.Evaluate(devset=dataset.dev, metric=dataset.metric, **kwargs)\n", + "\n", + " # Evaluate the program as usual\n", + " aggregated_score, outputs, all_scores = evaluate(module)\n", + "\n", + " # Log the aggregated score\n", + " mlflow.log_metric(\"correctness\", aggregated_score)\n", + " # Log the detailed evaluation results as a table\n", + " mlflow.log_table(\n", + " {\n", + " \"Question\": [example.question for example in dataset.dev],\n", + " \"Gold Answer\": [example.answer for example in dataset.dev],\n", + " \"Predicted Answer\": outputs,\n", + " \"Correctness\": all_scores,\n", + " },\n", + " artifact_file=\"eval_results.json\",\n", + " )\n", + "```\n", + "\n", + "To learn more about the integration, visit [MLflow DSPy Documentation](https://mlflow.org/docs/latest/llms/dspy/index.html) as well.\n", + "\n", + "
" + ] + }, { "cell_type": "markdown", "metadata": {}, @@ -535,7 +616,7 @@ "metadata": {}, "source": [ "\n", - "Just to understand what changed, let's view the prompt after optimization." + "Just to understand what changed, let's view the prompt after optimization. Alternatively, if you enabled MLflow tracing following above instructions, you can compare the prompts before and after optimization in the rich trace UI." ] }, { diff --git a/docs/docs/tutorials/math/mlflow-tracing-math.png b/docs/docs/tutorials/math/mlflow-tracing-math.png new file mode 100644 index 000000000..6cbb61721 Binary files /dev/null and b/docs/docs/tutorials/math/mlflow-tracing-math.png differ diff --git a/docs/docs/tutorials/multihop_search/index.ipynb b/docs/docs/tutorials/multihop_search/index.ipynb index 87313d196..f22920597 100644 --- a/docs/docs/tutorials/multihop_search/index.ipynb +++ b/docs/docs/tutorials/multihop_search/index.ipynb @@ -11,6 +11,48 @@ "Install the latest DSPy via `pip install -U dspy` and follow along." ] }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "
\n", + "Optional: Set up MLflow Tracing to understand what's happening under the hood.\n", + "\n", + "### MLflow DSPy Integration\n", + "\n", + "MLflow is an LLMOps tool that natively integrates with DSPy and offer explainability and experiment tracking. In this tutorial, you can use MLflow to visualize prompts and optimization progress as traces to understand the DSPy's behavior better. You can set up MLflow easily by following the four steps below.\n", + "\n", + "1. Install MLflow\n", + "\n", + "```bash\n", + "%pip install mlflow>=2.20\n", + "```\n", + "\n", + "2. Start MLflow UI in a separate terminal\n", + "```bash\n", + "mlflow ui --port 5000\n", + "```\n", + "\n", + "3. Connect the notebook to MLflow\n", + "```python\n", + "import mlflow\n", + "\n", + "mlflow.set_tracking_uri(\"http://localhost:5000\")\n", + "mlflow.set_experiment(\"DSPy\")\n", + "```\n", + "\n", + "4. Enabling tracing.\n", + "```python\n", + "mlflow.dspy.autolog()\n", + "```\n", + "\n", + "![MLflow Trace](./mlflow-tracing-multi-hop.png)\n", + "\n", + "\n", + "To learn more about the integration, visit [MLflow DSPy Documentation](https://mlflow.org/docs/latest/llms/dspy/index.html) as well.\n", + "
" + ] + }, { "cell_type": "markdown", "metadata": {}, @@ -471,6 +513,54 @@ "evaluate(Hop())" ] }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "
\n", + "Tracking Evaluation Results in MLflow Experiment\n", + "\n", + "
\n", + "\n", + "To track and visualize the evaluation results over time, you can record the results in MLflow Experiment.\n", + "\n", + "\n", + "```python\n", + "import mlflow\n", + "\n", + "with mlflow.start_run(run_name=\"hop_evaluation\"):\n", + " evaluate = dspy.Evaluate(\n", + " devset=devset,\n", + " metric=top5_recall,\n", + " num_threads=16,\n", + " display_progress=True,\n", + " # To record the outputs and detailed scores to MLflow\n", + " return_all_scores=True,\n", + " return_outputs=True,\n", + " )\n", + "\n", + " # Evaluate the program as usual\n", + " aggregated_score, outputs, all_scores = evaluate(Hop())\n", + "\n", + " # Log the aggregated score\n", + " mlflow.log_metric(\"top5_recall\", aggregated_score)\n", + " # Log the detailed evaluation results as a table\n", + " mlflow.log_table(\n", + " {\n", + " \"Claim\": [example.claim for example in eval_set],\n", + " \"Expected Titles\": [example.titles for example in eval_set],\n", + " \"Predicted Titles\": outputs,\n", + " \"Top 5 Recall\": all_scores,\n", + " },\n", + " artifact_file=\"eval_results.json\",\n", + " )\n", + "```\n", + "\n", + "To learn more about the integration, visit [MLflow DSPy Documentation](https://mlflow.org/docs/latest/llms/dspy/index.html) as well.\n", + "\n", + "
" + ] + }, { "cell_type": "markdown", "metadata": {}, @@ -699,7 +789,7 @@ "source": [ "Awesome. It looks like the system improved drastically from around 30% recall to a little below 60% recall. That was a pretty straightforward approach, but DSPy gives you many tools to continue iterating on this from here.\n", "\n", - "Next, let's inspect the optimized prompts to understand what it has learned. We'll run one query and then inspect the last two prompts, which will show us the prompts used for both sub-modules, in the later iteration inside the `Hop()` program." + "Next, let's inspect the optimized prompts to understand what it has learned. We'll run one query and then inspect the last two prompts, which will show us the prompts used for both sub-modules, in the later iteration inside the `Hop()` program. (Alternatively, if you enabled MLflow Tracing following the instructions above, you can see all steps done by the agent including LLM calls, prompts, tool execution, in a rich tree-view.)" ] }, { @@ -1102,6 +1192,42 @@ "\n", "loaded_program(claim=\"The author of the 1960s unproduced script written for The Beatles, Up Against It, and Bernard-Marie Koltès are both playwrights.\").titles" ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "
\n", + "Saving programs in MLflow Experiment\n", + "\n", + "
\n", + "\n", + "Instead of saving the program to a local file, you can track it in MLflow for better reproducibility and collaboration.\n", + "\n", + "1. **Dependency Management**: MLflow automatically save the frozen environment metadata along with the program to ensure reproducibility.\n", + "2. **Experiment Tracking**: With MLflow, you can track the program's performance and cost along with the program itself.\n", + "3. **Collaboration**: You can share the program and results with your team members by sharing the MLflow experiment.\n", + "\n", + "To save the program in MLflow, run the following code:\n", + "\n", + "```python\n", + "import mlflow\n", + "\n", + "# Start an MLflow Run and save the program\n", + "with mlflow.start_run(run_name=\"optimized\"):\n", + " model_info = mlflow.dspy.log_model(\n", + " optimized,\n", + " artifact_path=\"model\", # Any name to save the program in MLflow\n", + " )\n", + "\n", + "# Load the program back from MLflow\n", + "loaded = mlflow.dspy.load_model(model_info.model_uri)\n", + "```\n", + "\n", + "To learn more about the integration, visit [MLflow DSPy Documentation](https://mlflow.org/docs/latest/llms/dspy/index.html) as well.\n", + "\n", + "
" + ] } ], "metadata": { diff --git a/docs/docs/tutorials/multihop_search/mlflow-tracing-multi-hop.png b/docs/docs/tutorials/multihop_search/mlflow-tracing-multi-hop.png new file mode 100644 index 000000000..218d77beb Binary files /dev/null and b/docs/docs/tutorials/multihop_search/mlflow-tracing-multi-hop.png differ diff --git a/docs/docs/tutorials/rag/index.ipynb b/docs/docs/tutorials/rag/index.ipynb index d8b6cd927..0d42ef57b 100644 --- a/docs/docs/tutorials/rag/index.ipynb +++ b/docs/docs/tutorials/rag/index.ipynb @@ -21,6 +21,50 @@ "Let's tell DSPy that we will use OpenAI's `gpt-4o-mini` in our modules. To authenticate, DSPy will look into your `OPENAI_API_KEY`. You can easily swap this out for [other providers or local models](https://github.com/stanfordnlp/dspy/blob/main/examples/migration.ipynb).\n" ] }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "
\n", + "Optional: Set up MLflow Tracing to understand what's happening under the hood.\n", + "\n", + "### MLflow DSPy Integration\n", + "\n", + "MLflow is an LLMOps tool that natively integrates with DSPy and offer explainability and experiment tracking. In this tutorial, you can use MLflow to visualize prompts and optimization progress as traces to understand the DSPy's behavior better. You can set up MLflow easily by following the four steps below.\n", + "\n", + "![MLflow Trace](./mlflow-tracing-rag.png)\n", + "\n", + "1. Install MLflow\n", + "\n", + "```bash\n", + "%pip install mlflow>=2.20\n", + "```\n", + "\n", + "2. Start MLflow UI in a separate terminal\n", + "```bash\n", + "mlflow ui --port 5000\n", + "```\n", + "\n", + "3. Connect the notebook to MLflow\n", + "```python\n", + "import mlflow\n", + "\n", + "mlflow.set_tracking_uri(\"http://localhost:5000\")\n", + "mlflow.set_experiment(\"DSPy\")\n", + "```\n", + "\n", + "4. Enabling tracing.\n", + "```python\n", + "mlflow.dspy.autolog()\n", + "```\n", + "\n", + "Once you have completed the steps above, you can see traces for each program execution on the notebook. They provide great visibility into the model's behavior and helps you understand the DSPy's concepts better throughout the tutorial.\n", + "\n", + "To kearn more about the integration, visit [MLflow DSPy Documentation](https://mlflow.org/docs/latest/llms/dspy/index.html) as well.\n", + "\n", + "
" + ] + }, { "cell_type": "code", "execution_count": 1, @@ -78,7 +122,7 @@ "\n", "Now, what did DSPy do to build this `qa` module? Nothing fancy in this example, yet. The module passed your signature, LM, and inputs to an Adapter, which is a layer that handles structuring the inputs and parsing structured outputs to fit your signature.\n", "\n", - "Let's see it directly. You can inspect the `n` last prompts sent by DSPy easily.\n" + "Let's see it directly. You can inspect the `n` last prompts sent by DSPy easily. Alternatively, if you enabled MLflow Tracing above, you can see the full LLM interactions for each program execution in a tree view.\n" ] }, { @@ -666,6 +710,55 @@ "evaluate(cot)" ] }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "
\n", + "Tracking Evaluation Results in MLflow Experiment\n", + "\n", + "
\n", + "\n", + "To track and visualize the evaluation results over time, you can record the results in MLflow Experiment.\n", + "\n", + "\n", + "```python\n", + "import mlflow\n", + "\n", + "with mlflow.start_run(run_name=\"rag_evaluation\"):\n", + " evaluate = dspy.Evaluate(\n", + " devset=devset,\n", + " metric=metric,\n", + " num_threads=24,\n", + " display_progress=True,\n", + " # To record the outputs and detailed scores to MLflow\n", + " return_all_scores=True,\n", + " return_outputs=True,\n", + " )\n", + "\n", + " # Evaluate the program as usual\n", + " aggregated_score, outputs, all_scores = evaluate(cot)\n", + "\n", + "\n", + " # Log the aggregated score\n", + " mlflow.log_metric(\"semantic_f1_score\", aggregated_score)\n", + " # Log the detailed evaluation results as a table\n", + " mlflow.log_table(\n", + " {\n", + " \"Question\": [example.question for example in eval_set],\n", + " \"Gold Response\": [example.response for example in eval_set],\n", + " \"Predicted Response\": outputs,\n", + " \"Semantic F1 Score\": all_scores,\n", + " },\n", + " artifact_file=\"eval_results.json\",\n", + " )\n", + "```\n", + "\n", + "To learn more about the integration, visit [MLflow DSPy Documentation](https://mlflow.org/docs/latest/llms/dspy/index.html) as well.\n", + "\n", + "
" + ] + }, { "cell_type": "markdown", "metadata": {}, @@ -1320,6 +1413,42 @@ "loaded_rag(question=\"cmd+tab does not work on hidden or minimized windows\")" ] }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "
\n", + "Saving programs in MLflow Experiment\n", + "\n", + "
\n", + "\n", + "Instead of saving the program to a local file, you can track it in MLflow for better reproducibility and collaboration.\n", + "\n", + "1. **Dependency Management**: MLflow automatically save the frozen environment metadata along with the program to ensure reproducibility.\n", + "2. **Experiment Tracking**: With MLflow, you can track the program's performance and cost along with the program itself.\n", + "3. **Collaboration**: You can share the program and results with your team members by sharing the MLflow experiment.\n", + "\n", + "To save the program in MLflow, run the following code:\n", + "\n", + "```python\n", + "import mlflow\n", + "\n", + "# Start an MLflow Run and save the program\n", + "with mlflow.start_run(run_name=\"optimized_rag\"):\n", + " model_info = mlflow.dspy.log_model(\n", + " optimized_rag,\n", + " artifact_path=\"model\", # Any name to save the program in MLflow\n", + " )\n", + "\n", + "# Load the program back from MLflow\n", + "loaded = mlflow.dspy.load_model(model_info.model_uri)\n", + "```\n", + "\n", + "To learn more about the integration, visit [MLflow DSPy Documentation](https://mlflow.org/docs/latest/llms/dspy/index.html) as well.\n", + "\n", + "
" + ] + }, { "cell_type": "markdown", "metadata": {}, diff --git a/docs/docs/tutorials/rag/mlflow-tracing-rag.png b/docs/docs/tutorials/rag/mlflow-tracing-rag.png new file mode 100644 index 000000000..27a703067 Binary files /dev/null and b/docs/docs/tutorials/rag/mlflow-tracing-rag.png differ