esm2_preserving_ph.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Comparing Motif Persistent Homology of ESM-2 Models\n",
    "\n",
    "Here are some comments and explanations to make the notebook more readable:\n",
    "\n",
    "### Overall Goal\n",
    "\n",
    "This notebook generates a set of random protein sequences containing a common motif, computes contextual embeddings for the sequences using ESM models, and compares the embeddings using persistent homology and Wasserstein distance.\n",
    "\n",
    "### Code Walkthrough\n",
    "\n",
    "- Generate 100 random protein sequences containing the motif 'KLGFMNPTQY'\n",
    "  - Use amino_acids string to sample from\n",
    "  - Generate random length and random motif position\n",
    "  - Insert motif at random position\n",
    "  \n",
    "- Load ESM-2 models of different sizes \n",
    "  - Smaller 6 layer model \n",
    "  - Larger 12 layer model\n",
    "  \n",
    "- Get hidden states for a given sequence\n",
    "  - Helpful utility function to extract states from a given layer\n",
    "  \n",
    "- Compute persistent homology on phrase embeddings\n",
    "  - Calculate distance matrix between motif vector embeddings\n",
    "  - Generate persistent homology from distance matrix\n",
    "  - Capture high-level topological structure\n",
    "  \n",
    "- Compute Wasserstein distance between two persistence diagrams\n",
    "  - Quantify difference in topological structure\n",
    "  - Use as distance metric between sequences\n",
    "  \n",
    "- Compare models by looking at difference in distances\n",
    "  - Larger model gives larger distances\n",
    "  - Majority are increased, some are decreased\n",
    "  - Outliers likely due to instability for small model\n",
    "\n",
    "### Conclusion\n",
    "\n",
    "The notebook provides a workflow to compare protein sequence embeddings using topological data analysis. Key findings:\n",
    "\n",
    "- Persistent homology captures topological structure of motif embeddings \n",
    "- Wasserstein distance quantifies difference between sequences\n",
    "- Larger ESM model leads to greater distinction between sequences\n",
    "- Outliers show instability for small model on some sequences"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['RSWLLKTGCKEMMSEWYKKLGFMNPTQYPHPAVEFVYYCWTKLDHNEFSDVPPR',\n",
       " 'QWGPKLGFMNPTQYDYTGYLHMTYRNGKVREHMWLGWYPGHYMFGLTDHMRENVTQHHCMQNLAFGC',\n",
       " 'NTKPPTMQQEPISRFCMVEGWMAFNHVTSKLGFMNPTQYSPQNECELSELDHSLNGFDANWR',\n",
       " 'PFRCKIVQISSACMTSMLFTVWCDFQWFVFASFLYEWKQKLGFMNPTQYNNWGSF',\n",
       " 'CILDKVWFNVESRFCGPNGLMLSTDYYQKDWYVPTKFHENERYQKLGFMNPTQYWNYQQSFHGKACSEQS',\n",
       " 'YQGRLRLFNTQGQYEEYISVMNLNIATYREHKLGFMNPTQYIRIRTANMAHNYCPWDFRYNGKWVQEKQYV',\n",
       " 'FVFGESYIHLARHKLGFMNPTQYHGMMKSQVRYSELYVYGHFFQHHVAHETPYEFPSDFQDS',\n",
       " 'TSAMSEICIAFVGSFYLLVQMGANLDMNYCNTLMVFVFMKLGFMNPTQYSMNKH',\n",
       " 'RRFWDCQQEEYCKETSWPMTIGHMHWYNYENRGVCVPLQIQMWMKLGFMNPTQYPQD',\n",
       " 'QKCDMPFMTIMKYFRNCVAIFEFERHREILFNNTPDFINWFVIFNWQPKDSERFQQHANVKLGFMNPTQYSM',\n",
       " 'AIMLALMWIESMTQLIGRELYPDKKAFTNGVSEMERWKMPAGQQRKLGFMNPTQYKKVDMFTDEKQKGER',\n",
       " 'LFFGNVHTIGKLGFMNPTQYYTICNWETLKLFVCSHHCTRSSCMEWMPAWAENKYDTN',\n",
       " 'HPFYLRFTKLGFMNPTQYHTAGHEHQWVDLMTYVYDAQKHDIQMWNTMESKD',\n",
       " 'PWMREPTKDENVKLGFMNPTQYPWKMAYFFRWIIFLEGIEADVGDPKPVEFSNRFVFYMRGICCPQYTP',\n",
       " 'TYGDIHVIAVPGISDFMQLTICLLTHEAHWHDPPVSHWHPEPSVQKLGFMNPTQY',\n",
       " 'GYSLEFVVYWDPLFRTRIPSMQHFNIIKLGFMNPTQYLRSECAMNDTSMSDLHWIK',\n",
       " 'RFKLGFMNPTQYHNACAVFCFCTSTGRIVQFLHIWHFVDGGLGHVFVSEIMEFA',\n",
       " 'AKSLAYPSGRWMSVMGRNPSRWKMIMKLGFMNPTQYGRQDYFEHTYSRAHRPE',\n",
       " 'PYVKLGFMNPTQYIIFWAQVIFYFTHARVLFNCKVFTMFAMPLTLALCPAM',\n",
       " 'RMIMTFWALGCKIALCCTVHQKEKLGFMNPTQYFQPGWVMYFCCDWKYWCWHFQEKFANYDHAEYYA',\n",
       " 'GTFHCYGYRGCHHHQALSNERLGIWPNIPIQRLNNFTTGGFACNIARRNKLGFMNPTQYGVL',\n",
       " 'PAIKLGFMNPTQYAVYVFSPYALLRLKMWYECGYIRWCVKPNMFFNYCHFAPW',\n",
       " 'YDYPFHNLILKLGFMNPTQYDKMCCKDPWRILICVTPKDYFYYNNCNFKLDEDGFKYHSRKH',\n",
       " 'HGRYEPIPLEEQLYCEIMDNYPKDWLNTSARSLKLFQIHFDNEDLFVNQKLGFMNPTQYHG',\n",
       " 'TQYNDPCHWRLHKLGFMNPTQYHMLGRMEAVHGCGVMDYSDFAVYGPPGEHEYWMQCFR',\n",
       " 'SGNSMRCLAKLGFMNPTQYDFYTQNVDPEKEWYVCGAANTKSYHAGHIDFTMFESGQVLYVTCVTWCVYSIRCQK',\n",
       " 'MLPHVYRVPKWHPKFMRCVDKLGFMNPTQYKQELVILHRKDRVELNSQTDFFPINCYAMGHAGRPNNP',\n",
       " 'PRKFMTKVKNWTINYEEDTILPIEFTDHLMKRIALRTGCGVAGHFLTWNESNLKVKLGFMNPTQYWQK',\n",
       " 'YVCRDYPLNVDHPFWVRDTNDFCCFRQCRIFQNSHIGNVKLGFMNPTQYWRPPIK',\n",
       " 'YREYQNDTFCGGKLGFMNPTQYVCYMCWMYDGAQFIRWSAEIESCWEVTWG',\n",
       " 'WNCSVCIHQLNQRVKKPFSYDICWQSCWNTKKLGFMNPTQYCHVFQIFCHLIHTDKSVDWKEPP',\n",
       " 'RIYADGMKLGFMNPTQYQFSGLYKMSWSDNMDHFCCFCGQTRCKNQLPNVHHRWPYQDT',\n",
       " 'LSMCIDSQSEVVNHREEPWVPVKADYCHYWTLHFGYFVVLISPTSESVFDKLGFMNPTQYRGR',\n",
       " 'WPCHIMEYRCRIWETQALGYQCSKYHITTNVPEMHGPLNPICMKLGFMNPTQYQMAT',\n",
       " 'MEEWCCQCFPMNGNEHWTFWWDCVWNDKDRKLGFMNPTQYGGAPGLSQNGRTLSWFDQF',\n",
       " 'WWLNSAQAAVWKTTSFEIHTLLNISNEYQWTMTRQECKLGFMNPTQYRISADFPADRPKCTGEYVHECVRH',\n",
       " 'FKKIDVRTQLEGKVPAQLLKLGFMNPTQYIQCLFMCPVWDACLGCLNGCIDIVQMGKYEETAQCCTYQG',\n",
       " 'PYHQVTEEEDEMACKLGFMNPTQYYNSWHSQYRTKTFPHYMINDRSMTIEYNICRHSVGAENATPNTH',\n",
       " 'SQIFSEAYELETFERNVHCWMWWNDLMLSSDAIGVQKLGFMNPTQYCCLDRPGERHTHMLQNWVSN',\n",
       " 'WGFMWRFDDTNVPMCNDFYVNSKRKGHWGRHDTSSYHDFFMECVEDVKLGFMNPTQYFFPASILKIN',\n",
       " 'RNCLVAASTPHNKLGFMNPTQYHAGMCANDIDDDKMRTFCWNDWESACGFWAVYDGAWCHGAWKHKCIIKQHWKQ',\n",
       " 'RKKAKNNWHKLGFMNPTQYEHVNPKHGMCDKLACEHHKPAGFAYPICTLCARFIGISICTHDDQGV',\n",
       " 'VEVAGPLTYCQGPSIRMNIDSTTFIDEVWGMSLQVYKCGPIPGGEQLNYHCVFIPPYPCWKLGFMNPTQYVQI',\n",
       " 'MTEVWVQMEPSINQWRRVFWECHGHIFEPAKLGFMNPTQYERMIDSAGVCDWDPGTKMIWANTD',\n",
       " 'PQYAHFWWAMWVLRMSPWSWPSDPSSAQELWYAHTSMKLGFMNPTQYNYCAELTCCAF',\n",
       " 'AQFMDYWHKLGFMNPTQYFSSFEKRVDLQGFCGFFNQPQYEWYGWVTPIV',\n",
       " 'HIQGRYSHMWFKLGFMNPTQYLKRGHVCDHAHIWCMNVWWNRCKMYDYWEC',\n",
       " 'SMTMMHAKQQVFCHHRHDTCSKLGFMNPTQYLYPTRNIVKEDPTHTFDWKAEICYMN',\n",
       " 'FNSWGVNHWNACDWAMSHVQCIDKGTDFKCMHWNVQNDFDWQDNWAVKLGFMNPTQYIGPDRLGPYNQNQHI',\n",
       " 'KDQVNIDMWYLPFSRRGITCRYVGRWTVFWPMCKCILWSYQQCSCMNKLGFMNPTQYGNYEIRKMNMEMHM',\n",
       " 'HGMQSQHYTGQYVKWIPNDWYQQNYYYHQTQQQKKKLGFMNPTQYLWATYSFWQGQQRTWFHIFNFRYHGD',\n",
       " 'GNATLTDSYFHSHVNQHLSIHTEAVNGMWSYDISMCPRHGCMSGKLGFMNPTQYR',\n",
       " 'WEYTYRPVIVQYDHGHCKLLWAPIKLEFKLGFMNPTQYYDYNKFQNYQVL',\n",
       " 'WGHTGEYWCLKLGFMNPTQYDSDCWNVRDDQDAHVEFPIWQRHNDADRTRIETQFCQ',\n",
       " 'GYTKMTHSEEEKNEPDIRGKHYFVLEKLGFMNPTQYVYNVCKWQGGATIQ',\n",
       " 'KDWLWSSHVHRTTLWFDIKCCCYDVPLELEGQPRVHYWKVKWHWEAQEGEKLGFMNPTQY',\n",
       " 'YSACGVPHAMFVWRPSGKYWSMSPAQANSWYYHKFHMFVDQTFTKLGFMNPTQYFCRSYNLGDYNE',\n",
       " 'KLGFMNPTQYCGWILMEHPGKDCKECEINDSRQIHHNPGNHWEGDSLHAMRTDKYPSV',\n",
       " 'KARLVEFFCHHANWFYYVNKGFMLVPTHAAVMQPSQKLGFMNPTQYAFIRCLSIKNPPTSIW',\n",
       " 'QMNCQRPCKIMCVFWWHRDTPDSTKPDKISPKLGFMNPTQYMHYVGPAHFYTYTAWLNYHNN',\n",
       " 'FSLRSWFDKRHQEHAAKNCSANCSEPQAVAEAYQWKWQWQQAKRRKLGFMNPTQYYPNWNNPWCF',\n",
       " 'MNGDSNPYLGSSKYQIMYIDNWVCRLKMPCQTRMKLGFMNPTQYLKAVRCCVHNLWTWGWDYDPDVLAGMCLMTC',\n",
       " 'QEEDAVLNAKLVVRICMNKLGFMNPTQYGVDCAHCYMDNWAGWWSYTCLEWWYQFMSTAQQWPIKGK',\n",
       " 'GSDAKLGFMNPTQYWMPVDTLYAFHMKFLALNYPGTDYPFYVFVALIIHKGVNWVWITHFMWT',\n",
       " 'LDFITLDINNFAQGKGWIVPMKLGFMNPTQYKLITRYNWDSHCHIYHCHPHKVVQLHLLCFIYM',\n",
       " 'VQHQLSNGEVSYMNNANMDVFLDQKLGFMNPTQYMQIDAPFLITREEKRCVTLLFHWCGRNHNDKKNDRYPSGS',\n",
       " 'SQWGHPGAWWNRQDFDRQRHYRLTSACLSELTQNLPTCETEKNGGCQGIPWVCHHCEFKKLGFMNPTQYMYY',\n",
       " 'KLYCQFYYWCVEKLGFMNPTQYQQYAKMMCSGAHVDMYLTCEQRMNFTITGGDAACNQHYTACDKQYS',\n",
       " 'QWEHFHKAAAQYSSLTAQDMVQPQHKSMKQDFRFSLQYSAFRPPDAKLGFMNPTQY',\n",
       " 'MEYAHWRHIRWGKLGFMNPTQYVFNISEITKPFMRCGAKLNPLVRAQIDKLVYGNYWSEADEHWFYAWIV',\n",
       " 'KKLGFMNPTQYNWRFILMSLPQRGDQGHISQVTDMWFAWCFKIDQPLPCYATVDFSMTDYYSQHAFPNVIWDKVR',\n",
       " 'TDQIDFLKCQHDNEALKQNNWQVPKLGFMNPTQYITSFQNAWTVGYNFYCK',\n",
       " 'NPSVMKQSWYYPGCTTVMDGVSFNFQPWAEEPWITPDLIFNKLGFMNPTQYMCHRNTLANMSSTHC',\n",
       " 'RDAGVSFFVQHLYWKEWNEGEPYGKTNFNKLGFMNPTQYEHCFAGWSQHRSYWAET',\n",
       " 'KCCRSDKETNQHHRNRLKMVFDPGADWKADMETRQPWEQVKLGFMNPTQYMFAITYRE',\n",
       " 'KLGFMNPTQYLESMSKGYRHAWSWNAMQPMDTMQRFASCKCKEQIKFARGDAQKANNNEQCDLTHNMLQ',\n",
       " 'LPIHDKLGFMNPTQYQTISATRMFKRLMVPWMHEVIMAILAMNTHDHVTNCIYCKKHT',\n",
       " 'ADNMNSQEYCVGRHANPICQYKLGFMNPTQYIVQGMTFYHDGDWTGHWYGVYQDDFCDKDKVQWSQEEPNM',\n",
       " 'LKLGFMNPTQYELLEYYWVNDVPFYHGRSQHNVKLFEKAIGTYWMPEVHRNARFADEICYVHVAHANAQVSE',\n",
       " 'VPREGVFGFIQKLGFMNPTQYVDGIQEHQRKPMLCQIWGWAKWEEVDRRIQHDAYNLMLHGHDQDYMEWEIRWAK',\n",
       " 'IKSDYMRTFGTGFVHLYNISMFGSEVWGNRSNQAGYLTCQKLGFMNPTQYLMLLEPNW',\n",
       " 'EFQHHMYHTYMEALFHDTFCATNWIYMCGAWDWTFNMNDHWNCGWQEQEIRRCAKYGTDNGKKLGFMNPTQYN',\n",
       " 'IGEPNHMDDNYIKNVPVRPQRQATSDTLGIHGTTASFYMSNLWLARDPGAHEYMKLGFMNPTQYYEFLNHQSPQ',\n",
       " 'PYDMTRRCVCHMMTYCDKCTIMKWFWRWIWTYEFAMLFQMKLGFMNPTQYHDIDTFQDEKSP',\n",
       " 'VRRCCLTWIPWDAQKLGFMNPTQYWYIMQSVSTCDLMVEHVHFFQTTMHYKLGRSG',\n",
       " 'GGTLIWNDCLLDFEANMTKTIIGACRIEMFDKLEWHVNHGKISERALGYKSEPSYVIWKLGFMNPTQYGYPNG',\n",
       " 'PPLQLIWAFMDFEQQRKKQFMMKMGFTGITHCTMQRPCITIAVPCNYEGNKVFRIMQKLGFMNPTQYEC',\n",
       " 'KYVYWYELSYNIMNTEAFGPVMLVWSWDQHYVMLFRKLGFMNPTQYIYYLFIFEMRMYEYSSKTPPCHY',\n",
       " 'CEHWKFYSDVESSNCARDMCWLFPECAVCRIKDTMDAKLGFMNPTQYVCADTHVEYSSYGRPVYGGKECQSTEI',\n",
       " 'ASVQIHAVKTVGSMSFTTVSPYPKGMTCMASMPMNQALASGLFPQSFHPTIVPELKLGFMNPTQYPKVPYEMMC',\n",
       " 'SKQPRMWLNSVELCFMKYRCFCPITRVDWCKSSMNCQTIRECYFFFHRADCVDKLGFMNPTQYAYHSRFLI',\n",
       " 'HGVNAVWLCMEPEGYSNYYTVYLHQIYDKHMGTWESHFRYSRLDKMAYCAKCMNVQHTTKLGFMNPTQYKTD',\n",
       " 'LVTWRQKLMQVLYRGACYHNTYCVWAGNSNFYATKLGFMNPTQYCITLVPEWPNSHTMMMMMEINW',\n",
       " 'YVFRHHNMYFEHKLGFMNPTQYRVWDISDFFFKKCGRAAAMNTVRLMSIALGMVVGSYVCPNG',\n",
       " 'SHKYRYPPQEPESFNPRLKTPGCNWMKLGFMNPTQYALNPRICCIHEWWTRKLYH',\n",
       " 'QSLLTAVTAERRKVTDGWFMKCWPCYVIGKLGFMNPTQYTWGQLYKNCIFIKKCKMGDKTWLECATIWG',\n",
       " 'FTYDIKLVNQVWPKLGFMNPTQYNWIMIPEMKALYRFKWASKICLVMPLGCFKWQNHCACYDDHP',\n",
       " 'CINVHRMPQDKLGFMNPTQYEPQWAQKCMIVRVLMPNFMRFILDYIQGKKIQWCANKGIKGIMRSSFYIDKH',\n",
       " 'MAFHMYTRVCADEDECKNEYEHRDEFMAVMDKLGFMNPTQYKQISLMHWMIPMWM',\n",
       " 'VFPQASCRSRPVDGASYAWGNHTHWLFKLGFMNPTQYLWAWFKTGKGWSMSGSFYFRTENSCSP']"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import random\n",
    "\n",
    "amino_acids = 'ACDEFGHIKLMNPQRSTVWY'\n",
    "motif = 'KLGFMNPTQY'\n",
    "sequences = []\n",
    "\n",
    "for _ in range(100):\n",
    "    seq_length = random.randint(50, 75)\n",
    "    motif_position = random.randint(0, seq_length - len(motif))\n",
    "    sequence = ''.join(random.choice(amino_acids) for _ in range(seq_length))\n",
    "    sequence = sequence[:motif_position] + motif + sequence[motif_position + len(motif):]\n",
    "    sequences.append(sequence)\n",
    "\n",
    "# Print the list of protein sequences\n",
    "sequences\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "\n",
    "def get_hidden_states(tokenizer, model, sentence, layer):\n",
    "    # Load pre-trained model\n",
    "    tokenizer = tokenizer\n",
    "    model = model\n",
    "\n",
    "    # Tokenize input and convert to tensor\n",
    "    inputs = tokenizer(sentence, return_tensors=\"pt\")\n",
    "\n",
    "    # Forward pass\n",
    "    # Specify `output_hidden_states=True` when calling the model\n",
    "    outputs = model(**inputs, output_attentions=True, output_hidden_states=True)\n",
    "\n",
    "    # Obtain the hidden states for the specific layer\n",
    "    hidden_states = outputs.hidden_states[layer]\n",
    "\n",
    "    return hidden_states\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Some weights of the model checkpoint at facebook/esm2_t6_8M_UR50D were not used when initializing EsmModel: ['lm_head.dense.bias', 'lm_head.layer_norm.weight', 'lm_head.layer_norm.bias', 'lm_head.bias', 'lm_head.dense.weight']\n",
      "- This IS expected if you are initializing EsmModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
      "- This IS NOT expected if you are initializing EsmModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
      "Some weights of EsmModel were not initialized from the model checkpoint at facebook/esm2_t6_8M_UR50D and are newly initialized: ['esm.pooler.dense.weight', 'esm.pooler.dense.bias']\n",
      "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n"
     ]
    }
   ],
   "source": [
    "from transformers import AutoTokenizer, EsmModel\n",
    "tokenizer = AutoTokenizer.from_pretrained(\"facebook/esm2_t6_8M_UR50D\")\n",
    "model = EsmModel.from_pretrained(\"facebook/esm2_t6_8M_UR50D\")\n",
    "\n",
    "# Set the layer and head to use for computation\n",
    "layer = 5\n",
    "\n",
    "# Compute the context vectors for each text in the corpus\n",
    "context = [get_hidden_states(tokenizer, model, t, layer) for t in sequences]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[[ 9.6350e-01,  3.3161e+00,  2.3348e+00,  ...,  6.6520e-01,\n",
       "           1.7391e+00, -4.6667e+00],\n",
       "         [ 2.7638e-01,  2.9199e-01,  2.3925e+00,  ...,  2.1819e+00,\n",
       "           1.0009e+00, -3.5376e-01],\n",
       "         [-1.4134e+00, -1.8429e+00,  6.6259e-01,  ...,  7.7934e-01,\n",
       "           8.1615e-01,  2.7464e+00],\n",
       "         ...,\n",
       "         [-9.6685e-01, -1.4864e+00, -1.3682e+00,  ..., -2.9277e-01,\n",
       "           7.7223e-01, -1.4342e+00],\n",
       "         [-2.8106e+00, -1.4600e+00,  7.1654e-01,  ..., -5.5445e-01,\n",
       "           3.5375e-03, -9.1865e-01],\n",
       "         [-6.8736e-01, -5.0814e-01, -3.8937e-01,  ..., -3.6261e-01,\n",
       "          -4.4213e+00, -2.6621e+00]]], grad_fn=<AsStridedBackward0>)"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "context[len(sequences)-1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "from scipy.spatial import distance_matrix\n",
    "import gudhi as gd\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "from transformers import AutoTokenizer, EsmModel\n",
    "\n",
    "\n",
    "def compute_phrase_distances_and_homology(tokenizer, context_vectors, sentence, phrase):\n",
    "    # Initialize the tokenizer\n",
    "    tokenizer = tokenizer\n",
    "\n",
    "    # Tokenize the sentence and the phrase\n",
    "    sentence_tokens = tokenizer.encode(sentence, add_special_tokens=False)\n",
    "    phrase_tokens = tokenizer.encode(phrase, add_special_tokens=False)\n",
    "\n",
    "    # Find the indices of the phrase tokens in the sentence\n",
    "    phrase_indices = []\n",
    "    phrase_length = len(phrase_tokens)\n",
    "    for i in range(len(sentence_tokens) - phrase_length + 1):\n",
    "        if sentence_tokens[i:i+phrase_length] == phrase_tokens:\n",
    "            phrase_indices.extend(range(i, i+phrase_length))\n",
    "            break\n",
    "\n",
    "    # Extract the context vectors for the phrase\n",
    "    phrase_context_vectors = context_vectors[0, phrase_indices]\n",
    "\n",
    "    # Detach the tensor and convert to numpy array\n",
    "    phrase_context_vectors_np = phrase_context_vectors.detach().numpy()\n",
    "\n",
    "    # Compute the pairwise Euclidean distances among the phrase context vectors\n",
    "    distances = distance_matrix(phrase_context_vectors_np, phrase_context_vectors_np)\n",
    "\n",
    "    # Compute the persistent homology of the distance matrix\n",
    "    rips_complex = gd.RipsComplex(distance_matrix=distances, max_edge_length=np.max(distances))\n",
    "    simplex_tree = rips_complex.create_simplex_tree(max_dimension=2)\n",
    "    persistent_homology = simplex_tree.persistence(min_persistence=0.001)\n",
    "\n",
    "    return persistent_homology"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Wasserstein distance: = 9.52\n"
     ]
    }
   ],
   "source": [
    "from gudhi.hera import wasserstein_distance\n",
    "import numpy as np\n",
    "from transformers import AutoTokenizer, EsmModel\n",
    "tokenizer = AutoTokenizer.from_pretrained(\"facebook/esm2_t6_8M_UR50D\")\n",
    "\n",
    "persistent_homology_1 = compute_phrase_distances_and_homology(tokenizer, context[0], sequences[0], \"KLGFMNPTQY\")\n",
    "\n",
    "persistent_homology_2 = compute_phrase_distances_and_homology(tokenizer, context[1], sequences[1], \"KLGFMNPTQY\")\n",
    "\n",
    "dimension = 0  # set the dimension you're interested in\n",
    "\n",
    "persistent_homology_np_1 = np.array([[pt[1][0], pt[1][1]] for pt in persistent_homology_1 if pt[1][1] != float('inf') and pt[0] == dimension])\n",
    "\n",
    "persistent_homology_np_2 = np.array([[pt[1][0], pt[1][1]] for pt in persistent_homology_2 if pt[1][1] != float('inf') and pt[0] == dimension])\n",
    "\n",
    "\n",
    "print(f\"Wasserstein distance: = {wasserstein_distance(persistent_homology_np_1, persistent_homology_np_2, order=1., internal_p=2.):.2f}\")\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[ 0.          9.52312771 15.05591117 ... 14.36145408 40.78931892\n",
      "   6.31245712]\n",
      " [ 9.52312771  0.         20.85794811 ... 21.16120551 50.31024142\n",
      "  13.28347528]\n",
      " [15.05591117 20.85794811  0.         ...  2.52152111 29.45229331\n",
      "   9.66645219]\n",
      " ...\n",
      " [14.36145408 21.16120551  2.52152111 ...  0.         29.14903591\n",
      "  10.21494934]\n",
      " [40.78931892 50.31024142 29.45229331 ... 29.14903591  0.\n",
      "  37.02676614]\n",
      " [ 6.31245712 13.28347528  9.66645219 ... 10.21494934 37.02676614\n",
      "   0.        ]]\n"
     ]
    }
   ],
   "source": [
    "from gudhi.hera import wasserstein_distance\n",
    "import numpy as np\n",
    "from transformers import AutoTokenizer, EsmModel\n",
    "\n",
    "# Load the tokenizer\n",
    "tokenizer = AutoTokenizer.from_pretrained(\"facebook/esm2_t6_8M_UR50D\")\n",
    "\n",
    "# Function to convert persistent homology to numpy array\n",
    "def ph_to_np(persistent_homology, dimension):\n",
    "    return np.array([[pt[1][0], pt[1][1]] for pt in persistent_homology if pt[1][1] != float('inf') and pt[0] == dimension])\n",
    "\n",
    "# Function to compute Wasserstein distance between two sequences\n",
    "def compute_wd(tokenizer, seq1, seq2, context1, context2, dimension=0):\n",
    "    ph1 = compute_phrase_distances_and_homology(tokenizer, context1, seq1, \"KLGFMNPTQY\")\n",
    "    ph2 = compute_phrase_distances_and_homology(tokenizer, context2, seq2, \"KLGFMNPTQY\")\n",
    "    ph_np_1 = ph_to_np(ph1, dimension)\n",
    "    ph_np_2 = ph_to_np(ph2, dimension)\n",
    "    wd = wasserstein_distance(ph_np_1, ph_np_2, order=1., internal_p=2.)\n",
    "    return wd\n",
    "\n",
    "# Compute pairwise Wasserstein distances\n",
    "n = len(sequences)\n",
    "wd_matrix = np.zeros((n, n))\n",
    "for i in range(n):\n",
    "    for j in range(i+1, n):  # no need to compute for j < i due to symmetry\n",
    "        wd_matrix[i, j] = compute_wd(tokenizer, sequences[i], sequences[j], context[i], context[j])\n",
    "        wd_matrix[j, i] = wd_matrix[i, j]  # the distance is symmetric\n",
    "\n",
    "print(wd_matrix)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Some weights of the model checkpoint at facebook/esm2_t12_35M_UR50D were not used when initializing EsmModel: ['lm_head.dense.bias', 'lm_head.layer_norm.weight', 'lm_head.layer_norm.bias', 'lm_head.bias', 'lm_head.dense.weight']\n",
      "- This IS expected if you are initializing EsmModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
      "- This IS NOT expected if you are initializing EsmModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
      "Some weights of EsmModel were not initialized from the model checkpoint at facebook/esm2_t12_35M_UR50D and are newly initialized: ['esm.pooler.dense.weight', 'esm.pooler.dense.bias']\n",
      "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n"
     ]
    }
   ],
   "source": [
    "from transformers import AutoTokenizer, EsmModel\n",
    "tokenizer_2 = AutoTokenizer.from_pretrained(\"facebook/esm2_t12_35M_UR50D\")\n",
    "model_2 = EsmModel.from_pretrained(\"facebook/esm2_t12_35M_UR50D\")\n",
    "\n",
    "# Set the layer and head to use for computation\n",
    "layer = 11\n",
    "\n",
    "# Compute the context vectors for each text in the corpus\n",
    "context_2 = [get_hidden_states(tokenizer_2, model_2, t, layer) for t in sequences]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Wasserstein distance: = 37.39\n"
     ]
    }
   ],
   "source": [
    "from gudhi.hera import wasserstein_distance\n",
    "import numpy as np\n",
    "from transformers import AutoTokenizer, EsmModel\n",
    "tokenizer_2 = AutoTokenizer.from_pretrained(\"facebook/esm2_t12_35M_UR50D\")\n",
    "\n",
    "persistent_homology_3 = compute_phrase_distances_and_homology(tokenizer_2, context_2[0], sequences[0], \"KLGFMNPTQY\")\n",
    "\n",
    "persistent_homology_4 = compute_phrase_distances_and_homology(tokenizer_2, context_2[1], sequences[1], \"KLGFMNPTQY\")\n",
    "\n",
    "dimension = 0  # set the dimension you're interested in\n",
    "\n",
    "persistent_homology_np_3 = np.array([[pt[1][0], pt[1][1]] for pt in persistent_homology_3 if pt[1][1] != float('inf') and pt[0] == dimension])\n",
    "\n",
    "persistent_homology_np_4 = np.array([[pt[1][0], pt[1][1]] for pt in persistent_homology_4 if pt[1][1] != float('inf') and pt[0] == dimension])\n",
    "\n",
    "\n",
    "print(f\"Wasserstein distance: = {wasserstein_distance(persistent_homology_np_3, persistent_homology_np_4, order=1., internal_p=2.):.2f}\")\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[  0.          37.38918053  56.12622933 ...  90.61542573 127.75986032\n",
      "   25.22901981]\n",
      " [ 37.38918053   0.          62.14173315 ...  60.29814263  97.44257723\n",
      "   45.81406345]\n",
      " [ 56.12622933  62.14173315   0.         ...  34.4891964   71.63363099\n",
      "   71.62300969]\n",
      " ...\n",
      " [ 90.61542573  60.29814263  34.4891964  ...   0.          37.1444346\n",
      "  106.11220608]\n",
      " [127.75986032  97.44257723  71.63363099 ...  37.1444346    0.\n",
      "  143.25664068]\n",
      " [ 25.22901981  45.81406345  71.62300969 ... 106.11220608 143.25664068\n",
      "    0.        ]]\n"
     ]
    }
   ],
   "source": [
    "# Load the tokenizer\n",
    "tokenizer_2 = AutoTokenizer.from_pretrained(\"facebook/esm2_t12_35M_UR50D\")\n",
    "\n",
    "# Compute pairwise Wasserstein distances\n",
    "n = len(sequences)\n",
    "wd_matrix_2 = np.zeros((n, n))\n",
    "for i in range(n):\n",
    "    for j in range(i+1, n):  # no need to compute for j < i due to symmetry\n",
    "        wd_matrix_2[i, j] = compute_wd(tokenizer_2, sequences[i], sequences[j], context_2[i], context_2[j])\n",
    "        wd_matrix_2[j, i] = wd_matrix_2[i, j]  # the distance is symmetric\n",
    "\n",
    "print(wd_matrix_2)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[  0.        ,  27.86605282,  41.07031815, ...,  76.25397164,\n",
       "         86.9705414 ,  18.9165627 ],\n",
       "       [ 27.86605282,   0.        ,  41.28378504, ...,  39.13693712,\n",
       "         47.13233581,  32.53058817],\n",
       "       [ 41.07031815,  41.28378504,   0.        , ...,  31.96767529,\n",
       "         42.18133768,  61.95655749],\n",
       "       ...,\n",
       "       [ 76.25397164,  39.13693712,  31.96767529, ...,   0.        ,\n",
       "          7.99539869,  95.89725675],\n",
       "       [ 86.9705414 ,  47.13233581,  42.18133768, ...,   7.99539869,\n",
       "          0.        , 106.22987455],\n",
       "       [ 18.9165627 ,  32.53058817,  61.95655749, ...,  95.89725675,\n",
       "        106.22987455,   0.        ]])"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "wd_matrix_2 - wd_matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(94.94949494949495,\n",
       " 205.9887035027673,\n",
       " -33.50869924775133,\n",
       " 38.63009773096543,\n",
       " 42.63264214354211,\n",
       " 31.582733299496553,\n",
       " array([ 90.09785646,  91.44264837,  92.84893095, 100.48104425,\n",
       "        101.94778474,  -6.64630103,  -6.04887921, 100.41971326,\n",
       "        114.4672457 , 102.20613192, 118.31902012,  96.10189332,\n",
       "        -16.19022699,  -7.69609209,  -9.53037177, -12.58779501,\n",
       "         90.0313116 ,  96.00217512, -11.70717053, -18.57745759,\n",
       "         -6.49036944,  -7.35313098, -17.85245186, -16.16993629,\n",
       "        -10.68902272, -23.65699194,  -8.15293604,  -7.76608573,\n",
       "        -24.44866859, -13.9143451 ,  -5.17645417,  91.74962401,\n",
       "         93.53432691, 110.97155856,  -9.35218583,  92.62566071,\n",
       "        100.2419612 ,  97.79473112, 127.64931065, 130.42938076,\n",
       "        101.90047228, 108.21898218, 115.92006279, 147.08085034,\n",
       "        113.29608362,  96.00666536,  91.67193596,  97.93437913,\n",
       "        104.4616999 , 124.88281222, 105.45047555, 121.50629781,\n",
       "         95.78307604, 125.63765558, 129.07511395,  99.34969807,\n",
       "        106.2073271 , 114.69926404, 145.18800214, 111.70982287,\n",
       "         93.99501028,  95.92272405, 102.45004482, 122.87115714,\n",
       "        103.43882048, 119.49464273,  93.6999517 ,  92.81034584,\n",
       "        120.93068082,  97.9650116 ,  -9.04412292,  95.67531034,\n",
       "        107.75258   ,  95.83334743, 129.6375958 , 126.05570415,\n",
       "        106.25663604, 113.75579542, 143.36151516,  90.91165027,\n",
       "        108.96626747,  95.21514928,  93.33499813, 108.72531153,\n",
       "         97.22692115, 103.75424191,  97.82625784, 123.56485732,\n",
       "        104.74301757, 120.79883983,  -8.17303854,  -4.9869342 ,\n",
       "         99.65395911, 104.49731763,  97.47826809, 133.37795473,\n",
       "        131.36629965,  96.12441422, 132.67049675,  -7.26828178,\n",
       "         -8.7673879 ,  98.12668454, -15.37268554, 127.00732164,\n",
       "        124.99566657, 125.22573016, 114.37108508, 113.57460494,\n",
       "        109.78608902, -12.31755253, -12.91434393,  -5.63174838,\n",
       "        -10.05118549, -15.79470514,  -5.89258848, 100.14116886,\n",
       "         92.53631809, -11.40933861,  97.5093324 ,  91.79902639,\n",
       "        107.91191458,  94.09021209,  -8.10689237, -12.05046358,\n",
       "         94.35669446, -11.51230827, -10.78853122,  -8.83541119,\n",
       "        123.75691116, 122.84573024, 114.70857428,  -8.72043008,\n",
       "        106.60973483, 108.20669197, 106.34580815, 120.63025382,\n",
       "         91.49308099, 116.59085414, 116.02384534, 107.92510788,\n",
       "         93.17294461, 119.95180446, 121.94918979,  92.68439728,\n",
       "        107.96812835, 133.90147552,  98.07580052,  -5.98792729,\n",
       "         91.42089123,  93.98058791,  93.59575839, 106.59124133,\n",
       "        -10.13905788,  -5.85249878,  -9.24984442, -15.81335342,\n",
       "        -11.74186702,  94.11374007,  -8.41041223, -29.41156654,\n",
       "         97.42540509, -13.31589016, -11.17505233,  95.89263816,\n",
       "        -24.96501706, 101.37719692,  -8.13645784,  90.29819465,\n",
       "        -20.13420762,  93.3058466 ,  99.8525176 ,  -9.84562048,\n",
       "         -6.15259395, -12.7928891 ,  98.27889551,  -9.09771956,\n",
       "         98.61424819, -13.25194136, -12.98886969, -17.14371075,\n",
       "         -9.62020144, -19.29289386, -19.23946546, -21.74947108,\n",
       "        -23.14270009, -28.79323654, -15.81135206,  -5.2385625 ,\n",
       "         -9.56305151, -11.36103604,  -5.99896373, -14.99008939,\n",
       "        -13.58226744,  -5.57345684,  -5.25010797, -11.662068  ,\n",
       "         -6.57399355, -26.77078954,  -5.09794174,  -7.99158589,\n",
       "        -26.64330385,  -7.59924446, -11.61751308,  96.8284693 ,\n",
       "         -7.91738406,  94.92576351, 110.97494568,  -5.32846214,\n",
       "         -5.69594932, 102.44057841, 105.87803678,  91.50218687,\n",
       "        121.99092497,  99.67407997,  96.29756556, 108.16922248,\n",
       "        101.7985894 ,  90.37752777, 103.0367654 ,  96.18498124,\n",
       "        -11.12005977,  -6.94087455, -30.29917228, 152.9831761 ,\n",
       "        113.14497051, 107.89414796, 151.24782104, 150.68709   ,\n",
       "        157.30766511, 104.08823122,  97.56956787, 137.50025621,\n",
       "        136.92115209, 146.01348778, 160.43299516, 149.26393786,\n",
       "        135.76720862, 135.11973216,  92.26722556, 136.49527661,\n",
       "        109.98203815, 114.79114373, 160.73683103,  99.73423921,\n",
       "        177.1080664 , 139.05357441, 115.56056569, 118.82851613,\n",
       "        152.73894329, 148.82035789, 127.83242985, 119.66001972,\n",
       "         98.7212888 , 170.08901686, 205.9887035 , 125.88424115,\n",
       "        203.97704843, 111.14433189, 173.90583089, 205.28124552,\n",
       "        134.64753407, 152.53446059,  90.40244348, 133.9291716 ,\n",
       "        144.86572714, 151.32536437, 160.03078361, 166.70096087,\n",
       "        121.70539743,  97.25657264, 105.85981858, 122.62480387,\n",
       "        120.17377375, 119.91163061, 119.59810319, 147.78856143,\n",
       "        110.14326826, 140.51173527, 159.11925993, 124.34088184,\n",
       "        129.76927486, 147.7446136 , 122.29655142, 136.09139914,\n",
       "        123.98559715, 180.77997126, 116.53180223,  -6.1110685 ,\n",
       "         -6.5664273 , 115.41889687, 113.67278586, 111.2702411 ,\n",
       "         -4.85157088,  90.47570869,  90.30426257,  90.77322747,\n",
       "         93.04489199,  -7.42871082,  96.25778623, 151.83393828,\n",
       "        -10.0086709 , -11.51212995, 137.63476289, -13.7854101 ,\n",
       "        -13.72686553, -19.32734489,  -5.72693502, -33.50869925,\n",
       "         -7.99495292, 117.74574176, 114.28184421, 114.89199304,\n",
       "         96.41813051, 123.13260112,  96.34531444,  -9.44600106,\n",
       "        -10.82288642,  97.10465294, 101.06856427, -12.02838592,\n",
       "        147.52205333, 124.3218288 , -13.17457126, 149.11716908,\n",
       "         96.61399725,  95.74522551,  93.73357043,  93.48783376,\n",
       "        111.78434735, 109.39761924, -22.61429618,  91.92421887,\n",
       "        -10.12984224, 141.30941953, 114.42303485, 112.68767979,\n",
       "        108.07622971, 118.74752386,  94.65951624,  98.36101084,\n",
       "        107.45334652, 121.8728539 , 110.70379661,  97.20706737,\n",
       "         96.27789986, 100.99137021, 122.17668977, 136.52830026,\n",
       "         99.08525386, 115.18343608, 110.26021663, 118.0217935 ,\n",
       "        131.5288756 , 167.44672775,  90.73627538, 165.41690717,\n",
       "        136.55231249, 163.84727047,  96.47764181, 114.75792919,\n",
       "        113.0850949 , 111.70666769, 112.76522312, 121.47064236,\n",
       "        128.14081961,  90.39053802,  98.18405472, 109.22842017,\n",
       "        101.95159402, 120.55911867, 108.19253705,  93.87799675,\n",
       "        109.18447235,  97.53125788, 142.21983   , 111.21562474,\n",
       "         99.07462163, 108.96191208, 110.55702783, 102.74927828,\n",
       "         -6.81015349,  -9.64562522,  -5.78596941, 119.90216834,\n",
       "        103.3086651 , 116.06241953, 110.87980937,  90.04296549,\n",
       "         96.50395946, 126.99269756,  93.51451829,  97.69706715,\n",
       "         90.91044523, 104.67585256, 101.29933815, -10.24472633,\n",
       "        113.17099507, 106.80036199,  95.37930036, 111.56828889,\n",
       "        108.20490865, 106.19853814, 121.54697829,  93.01606506,\n",
       "         93.36999234,  93.21962081,  98.42364312, 185.78174384,\n",
       "         95.47748128, 117.27443431,  98.20917107, 147.22160259,\n",
       "         95.28218397,  -5.9062409 ,  -7.56369194,  -8.83427832,\n",
       "         97.46677695,  -6.99973693,  93.85263391,  -6.99918308,\n",
       "        -17.84333145, 169.22208398, 135.06143556, 109.31017656,\n",
       "        107.29852149, 106.84489142,  -9.293572  ,  94.4635026 ,\n",
       "         95.41159665,  98.84905503, 114.96194322,  92.64509822,\n",
       "        101.14024073,  94.76960765,  97.06724844,  90.67353593,\n",
       "        173.7509895 , 135.19084825,  -7.29489292,  -8.40583443,\n",
       "        150.85393109, 112.29378984,  -8.60494767,  93.77674788,\n",
       "         -7.11012217, 144.3291396 , 102.79812989, 106.66152311,\n",
       "        109.22121979,  90.70371221, 101.74669269, 121.83187321,\n",
       "        106.36689083, 108.25669694, -19.50089285,  90.13419565,\n",
       "        -11.42430228, 139.98355355, 101.4234123 , 113.63432716,\n",
       "         -6.09060616, 113.09051487, 108.91708883, 119.71280634,\n",
       "         95.60876981,  -9.97095744,  99.71271602,  90.01103477,\n",
       "        -10.36349164,  90.36773151, 119.99049093, 103.10003301,\n",
       "         -6.97937932,  96.08098347, 131.98067012, 129.96901504,\n",
       "        131.01700092,  92.69292748, 106.77193787, 111.77371046,\n",
       "         99.74295612,  91.29503041,  94.42036046,  -5.44992212,\n",
       "         94.72419633, 111.0954317 ,  94.2379192 , 104.07638216,\n",
       "        139.97606881, 137.96441373, 115.3285257 , 139.26861082,\n",
       "         94.01814891, 100.68832617,  93.10662523, 114.76733656,\n",
       "        119.76910915, 107.60894212, 107.73835481,  93.90311639,\n",
       "        107.99691107,  -7.99306453,  -9.69837797,  99.63176047,\n",
       "        -16.3166507 , 171.61157947, 133.68236799,  95.89725675,\n",
       "        106.22987455]))"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import numpy as np\n",
    "from scipy import stats\n",
    "\n",
    "matrix = wd_matrix_2 - wd_matrix\n",
    "\n",
    "# Compute the lower triangular part of the matrix, excluding the diagonal\n",
    "lower_triangular = matrix[np.tril_indices(matrix.shape[0], k=-1)]\n",
    "\n",
    "# Compute the statistics\n",
    "percentage_non_negative = np.mean(lower_triangular >= 0) * 100\n",
    "max_value = np.max(lower_triangular)\n",
    "min_value = np.min(lower_triangular)\n",
    "median = np.median(lower_triangular)\n",
    "mean = np.mean(lower_triangular)\n",
    "std_dev = np.std(lower_triangular)\n",
    "\n",
    "# Compute outliers using Z-score\n",
    "z_scores = stats.zscore(lower_triangular)\n",
    "outliers = lower_triangular[np.abs(z_scores) > 1.5]\n",
    "\n",
    "percentage_non_negative, max_value, min_value, median, mean, std_dev, outliers\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "env_1",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.11"
  },
  "orig_nbformat": 4
 },
 "nbformat": 4,
 "nbformat_minor": 2
}