notebooks/1.7-python-basics-working-with-files.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Working with Files\n",
    "\n",
    "Lesson 11 - txt files python , open read, store contents to file, extract information with LLMs in bullet points to txt file, structure prompts to extract desired data from txt file"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In Python it is extremely simple to work with files like .txt or .md files for example."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "from ai_tools import ask_ai\n",
    "from IPython.display import Markdown"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "https://arxiv.org/pdf/2412.14161\n",
      "\n",
      "The paper titled \"TheAgent Company: Benchmarking LLM Agents on Consequential Real World Tasks\" introduces a new benchmark, TheAgentCompany, aimed at evaluating the efficacy of large language model (LLM)-powered AI agents in completing real-world professional tasks in a simulated software company environment. The research is conducted by a collaborative team, mainly from Carnegie Mellon University and other institutions, and emphasizes the growing presence of AI in work settings.\n",
      "\n",
      "### Key Points from the Paper:\n",
      "\n",
      "1. **Motivation**:\n",
      "   - The rapid advancements in LLMs are prompting questions about AI's potential to automate or assist in various work-related tasks.\n",
      "   - Understanding AI agents’ capabilities is crucial for businesses considering AI integration and for policymakers assessing AI’s impact on employment.\n",
      "\n",
      "2. **Benchmark Overview**:\n",
      "   - TheAgentCompany simulates a software company environment with 175 diverse professional tasks spanning categories like software engineering, project management, and finance.\n",
      "   - The benchmark allows agents to interact through web browsing, coding, and colleague communication, providing a realistic testing framework.\n",
      "\n",
      "3. **Performance Findings**:\n",
      "   - Experiments conducted with several LLMs, including closed (like OpenAI's GPT-4o and Claude) and open-weight models (like Llama), reveal that the top-performing model, Claude-3.5-Sonnet, achieved 24% task completion autonomously, with a score of 34.4% when accounting for partial completions.\n",
      "   - Despite these advancements, LLM agents struggle significantly with longer, more complex tasks, especially those requiring social interaction and navigation of intricate user interfaces.\n",
      "\n",
      "4. **Framework and Design**:\n",
      "   - TheAgentCompany provides a self-hosted and reproducible environment utilizing open-source software.\n",
      "   - Tasks are structured into parts with defined checkpoints, allowing agents to receive partial credit for incomplete tasks.\n",
      "   - Evaluators for tasks are tailored to assess not just the success of task completion but also the quality of interactions with simulated colleagues.\n",
      "\n",
      "5. **Interaction and Collaboration**:\n",
      "   - A significant component of the benchmark involves the ability of agents to communicate effectively with simulated colleagues within the environment, enhancing the realism and complexity of tasks.\n",
      "\n",
      "6. **Future Directions**:\n",
      "   - The paper suggests that while TheAgentCompany provides a foundational step for understanding LLM capabilities in professional settings, there is a need to expand the tasks covered and include more creative or less straightforward tasks.\n",
      "   - Continuous improvements in LLMs are expected, highlighting their potential for increased efficiency and performance across various domains.\n",
      "\n",
      "7. **Conclusions**:\n",
      "   - The research underscores the current limitations of LLM agents in effectively automating diverse professional tasks.\n",
      "   - The results serve as a litmus test for future developments, pointing towards areas where LLM technology must improve, particularly in tasks involving human-like social interactions and complex decision-making.\n",
      "\n",
      "In summary, TheAgentCompany represents a significant effort to quantify AI agents' performance in real-world applications and to chart a course for further research in this rapidly evolving field.\n",
      "append thisnew data\n"
     ]
    }
   ],
   "source": [
    "# To open a file\n",
    "\n",
    "with open(\"./file.txt\", \"r\") as f:\n",
    "    data = f.read()\n",
    "\n",
    "print(data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this particular file we read a summary of a paper called: [\"TheAgentCompany: Benchmarking LLM Agents on Clnsequential Real World Tasks\"](https://arxiv.org/pdf/2412.14161)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also create files easily in Python:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "content = \"This is a file\"\n",
    "with open(\"summary-notes.txt\", \"w\") as f:\n",
    "    f.write(content)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "This is a file"
     ]
    }
   ],
   "source": [
    "# in the cmd below we print the contents of an existing file in the current directory\n",
    "!cat ./summary-notes.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Below are more examples for the different modes of reading and writing files available via the built-in `open()` method:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Common file modes in Python's open() function:\n",
    "\n",
    "# \"a\" - Append - Opens file for appending, creates new file if not exists\n",
    "with open(\"summary-notes.txt\", \"a\") as f:\n",
    "    f.write(\"append this\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "This is a fileappend this"
     ]
    }
   ],
   "source": [
    "!cat summary-notes.txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "ename": "FileExistsError",
     "evalue": "[Errno 17] File exists: 'file.txt'",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mFileExistsError\u001b[0m                           Traceback (most recent call last)",
      "Cell \u001b[0;32mIn[8], line 3\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[38;5;66;03m# \"x\" - Exclusive creation - Opens for writing, fails if file exists\u001b[39;00m\n\u001b[0;32m----> 3\u001b[0m \u001b[38;5;28;01mwith\u001b[39;00m \u001b[38;5;28;43mopen\u001b[39;49m\u001b[43m(\u001b[49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[38;5;124;43mfile.txt\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[38;5;124;43mx\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[43m)\u001b[49m \u001b[38;5;28;01mas\u001b[39;00m f:\n\u001b[1;32m      4\u001b[0m     f\u001b[38;5;241m.\u001b[39mwrite(\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mnew file content\u001b[39m\u001b[38;5;124m\"\u001b[39m)\n",
      "File \u001b[0;32m~/miniconda3/envs/oreilly-automate-tasks/lib/python3.11/site-packages/IPython/core/interactiveshell.py:324\u001b[0m, in \u001b[0;36m_modified_open\u001b[0;34m(file, *args, **kwargs)\u001b[0m\n\u001b[1;32m    317\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m file \u001b[38;5;129;01min\u001b[39;00m {\u001b[38;5;241m0\u001b[39m, \u001b[38;5;241m1\u001b[39m, \u001b[38;5;241m2\u001b[39m}:\n\u001b[1;32m    318\u001b[0m     \u001b[38;5;28;01mraise\u001b[39;00m \u001b[38;5;167;01mValueError\u001b[39;00m(\n\u001b[1;32m    319\u001b[0m         \u001b[38;5;124mf\u001b[39m\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mIPython won\u001b[39m\u001b[38;5;124m'\u001b[39m\u001b[38;5;124mt let you open fd=\u001b[39m\u001b[38;5;132;01m{\u001b[39;00mfile\u001b[38;5;132;01m}\u001b[39;00m\u001b[38;5;124m by default \u001b[39m\u001b[38;5;124m\"\u001b[39m\n\u001b[1;32m    320\u001b[0m         \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mas it is likely to crash IPython. If you know what you are doing, \u001b[39m\u001b[38;5;124m\"\u001b[39m\n\u001b[1;32m    321\u001b[0m         \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124myou can use builtins\u001b[39m\u001b[38;5;124m'\u001b[39m\u001b[38;5;124m open.\u001b[39m\u001b[38;5;124m\"\u001b[39m\n\u001b[1;32m    322\u001b[0m     )\n\u001b[0;32m--> 324\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[43mio_open\u001b[49m\u001b[43m(\u001b[49m\u001b[43mfile\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43margs\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43mkwargs\u001b[49m\u001b[43m)\u001b[49m\n",
      "\u001b[0;31mFileExistsError\u001b[0m: [Errno 17] File exists: 'file.txt'"
     ]
    }
   ],
   "source": [
    "# \"x\" - Exclusive creation - Opens for writing, fails if file exists\n",
    "\n",
    "with open(\"file.txt\", \"x\") as f:\n",
    "    f.write(\"new file content\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "# \"+\" - Read and write mode\n",
    "with open(\"file.txt\", \"r+\") as f:  # Open for both reading and writing\n",
    "    data = f.read()\n",
    "    f.write(\"new data\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The cool stuff about being able to do this is that we can connect our ability of generating summaries of information with AI, along with our ability to read and write files in Python to create super powerful workflows.\n",
    "\n",
    "For example, below we will write single sentence summaries for multiple files containing information about different papers. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "# Paper Summaries\n",
       "\n",
       "## Paper 1\n",
       "\n",
       "The paper titled \"TheAgent Company: Benchmarking LLM Agents on Consequential Real World Tasks\" introduces a new benchmark, TheAgentCompany, aimed at evaluating the efficacy of large language model (LLM)-powered AI agents in completing real-world professional tasks in a simulated software company environment. The research is conducted by a collaborative team, mainly from Carnegie Mellon University and other institutions, and emphasizes the growing presence of AI in work settings.\n",
       "\n",
       "### Key Points from the Paper:\n",
       "\n",
       "1. **Motivation**:\n",
       "   - The rapid advancements in LLMs are prompting questions about AI's potential to automate or assist in various work-related tasks.\n",
       "   - Understanding AI agents’ capabilities is crucial for businesses considering AI integration and for policymakers assessing AI’s impact on employment.\n",
       "\n",
       "2. **Benchmark Overview**:\n",
       "   - TheAgentCompany simulates a software company environment with 175 diverse professional tasks spanning categories like software engineering, project management, and finance.\n",
       "   - The benchmark allows agents to interact through web browsing, coding, and colleague communication, providing a realistic testing framework.\n",
       "\n",
       "3. **Performance Findings**:\n",
       "   - Experiments conducted with several LLMs, including closed (like OpenAI's GPT-4o and Claude) and open-weight models (like Llama), reveal that the top-performing model, Claude-3.5-Sonnet, achieved 24% task completion autonomously, with a score of 34.4% when accounting for partial completions.\n",
       "   - Despite these advancements, LLM agents struggle significantly with longer, more complex tasks, especially those requiring social interaction and navigation of intricate user interfaces.\n",
       "\n",
       "4. **Framework and Design**:\n",
       "   - TheAgentCompany provides a self-hosted and reproducible environment utilizing open-source software.\n",
       "   - Tasks are structured into parts with defined checkpoints, allowing agents to receive partial credit for incomplete tasks.\n",
       "   - Evaluators for tasks are tailored to assess not just the success of task completion but also the quality of interactions with simulated colleagues.\n",
       "\n",
       "5. **Interaction and Collaboration**:\n",
       "   - A significant component of the benchmark involves the ability of agents to communicate effectively with simulated colleagues within the environment, enhancing the realism and complexity of tasks.\n",
       "\n",
       "6. **Future Directions**:\n",
       "   - The paper suggests that while TheAgentCompany provides a foundational step for understanding LLM capabilities in professional settings, there is a need to expand the tasks covered and include more creative or less straightforward tasks.\n",
       "   - Continuous improvements in LLMs are expected, highlighting their potential for increased efficiency and performance across various domains.\n",
       "\n",
       "7. **Conclusions**:\n",
       "   - The research underscores the current limitations of LLM agents in effectively automating diverse professional tasks.\n",
       "   - The results serve as a litmus test for future developments, pointing towards areas where LLM technology must improve, particularly in tasks involving human-like social interactions and complex decision-making.\n",
       "\n",
       "In summary, TheAgentCompany represents a significant effort to quantify AI agents' performance in real-world applications and to chart a course for further research in this rapidly evolving field. The study emphasizes the necessity of enhancing both the complexity of tasks and the agents' social interaction capabilities, urging further exploration into LLM agent performance to better align with real-world professional needs.\n",
       "\n",
       "## Paper 2\n",
       "\n",
       "The paper presents a novel research area called Automated Design of Agentic Systems (ADAS), which aims to automatically create and optimize agentic systems using Foundation Models. The authors introduce an innovative algorithm called Meta Agent Search, wherein a \"meta\" agent is tasked with iteratively designing new agents by leveraging code-based representations. Through extensive experiments across various domains (coding, reading comprehension, math), the results demonstrate that agents developed via this automation significantly outperform state-of-the-art hand-designed agents, showcasing robustness across tasks and the potential for ADAS in advancing artificial intelligence research while highlighting the importance of safety in this burgeoning field.\n",
       "\n",
       "## Paper 3\n",
       "\n",
       "The paper titled \"Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences\" introduces EvalGen, a mixed-initiative system designed to enhance the alignment of evaluation metrics generated by Large Language Models (LLMs) with user preferences during the evaluation of LLM outputs. The authors emphasize the challenges posed by the subjective nature of evaluation and the common phenomenon of \"criteria drift,\" where users refine their criteria based on their experiences grading outputs.\n",
       "\n",
       "EvalGen enables users to generate evaluation criteria and corresponding assertions (either LLM-based or code-based), while also allowing for iterative feedback through user grading. The system utilizes this feedback to select the most aligned assertions, thereby improving the evaluation process. A qualitative study involving industry practitioners highlighted both the positive reception of EvalGen's capabilities and the difficulties posed by criteria drift, indicating that users often need to redefine their evaluation criteria as they interact with outputs.\n",
       "\n",
       "Key findings suggest that criteria are dynamically dependent on the specific outputs being evaluated, and this highlights the necessity for an iterative design in future LLM evaluation assistants. The authors propose directions for designing these tools, emphasizing the importance of accommodating evolving user standards in interactive settings as well as considering the implications of deploying both code-based and LLM-based assertions in practice. The paper ultimately calls into question the notion of fixed evaluation criteria and encourages ongoing adjustment and refinement throughout the evaluation process.\n",
       "\n"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "folder_with_papers = \"./assets-resources/papers/\"\n",
    "file_names = [\"paper1.txt\", \"paper2.txt\", \"paper3.txt\"]\n",
    "\n",
    "def summarize_this_paper(paper_contents):\n",
    "    summary_prompt = f\"Summarize this paper\\n\\n: {paper_contents} in a couple of sentences.\"\n",
    "    output_summary = ask_ai(summary_prompt)\n",
    "    \n",
    "    return output_summary\n",
    "\n",
    "paper_summaries_list = []\n",
    "for file_name in file_names:\n",
    "    file_path = folder_with_papers + file_name\n",
    "    with open(file_path, \"r\") as f:\n",
    "        contents_of_the_paper = f.read()\n",
    "    \n",
    "    paper_summary = summarize_this_paper(contents_of_the_paper)\n",
    "    paper_summaries_list.append(paper_summary)\n",
    "            \n",
    "\n",
    "# Display the markdown content in the notebook\n",
    "    \n",
    "markdown_content = \"# Paper Summaries\\n\\n\"\n",
    "for i, summary in enumerate(paper_summaries_list, 1):\n",
    "    markdown_content += f\"## Paper {i}\\n\\n\"\n",
    "    markdown_content += f\"{summary}\\n\\n\"\n",
    "        \n",
    "Markdown(markdown_content)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can do similar things to extract specific information from documents, imagine you have a bunch of differently formatted invoices from which you would like to organize the information extracting things like the amounts and dates."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Key Takeaways\n",
    "\n",
    "- Always use `with` statements when working with files to ensure proper closure\n",
    "- Different file modes serve different purposes:\n",
    "  - `\"r\"` for reading\n",
    "  - `\"w\"` for writing (creates new/overwrites)\n",
    "  - `\"a\"` for appending\n",
    "  - `\"r+\"` for reading and writing\n",
    "- Always handle potential file-related exceptions\n",
    "- File operations can be combined with data processing for powerful automation\n",
    "- Consider creating helper functions for common file operations\n",
    "- Remember to close files properly (using `with` statements)\n",
    "- Be careful with file paths and permissions\n",
    "\n",
    "In the next lesson, we'll explore how to work with different file formats like CSV and JSON!"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "oreilly-automate-tasks",
   "language": "python",
   "name": "oreilly-automate-tasks"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}