diary.html

<!doctype html>
<html lang="en">
  <head>
    <meta charset="utf-8">
    <meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">

<link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/4.0.0/css/bootstrap.min.css" integrity="sha384-Gn5384xqQ1aoWXA+058RXPxPg6fy4IWvTNh0E263XmFcJlSAwiGgFAW/dAiS6JXm" crossorigin="anonymous">
<link rel="stylesheet" href="style.css" type="text/css">

<script src="https://code.jquery.com/jquery-3.2.1.slim.min.js" integrity="sha384-KJ3o2DKtIkvYIK3UENzmM7KCkRr/rE9/Qpg6aAZGJwFDMVNA/GpGFF93hXpG5KkN" crossorigin="anonymous"></script>
<script src="https://cdnjs.cloudflare.com/ajax/libs/popper.js/1.12.9/umd/popper.min.js" integrity="sha384-ApNbgh9B+Y1QKtv3Rn7W3mgPxhU9K/ScQsAP7hUibX39j7fakFPskvXusvfa0b4Q" crossorigin="anonymous"></script>
<script src="https://maxcdn.bootstrapcdn.com/bootstrap/4.0.0/js/bootstrap.min.js" integrity="sha384-JZR6Spejh4U02d8jOt6vLEHfe/JQGiRRSQQxSfFWpi1MquVdAyjUar5+76PVCmYl" crossorigin="anonymous"></script>

    <title>Thesis Repository</title>
  </head>


  <body>
                 <nav class="navbar navbar-expand-lg navbar-light navbar-custom ">
  <button class="navbar-toggler" type="button" data-toggle="collapse" data-target="#navbarSupportedContent" aria-controls="navbarSupportedContent" aria-expanded="false" aria-label="Toggle navigation">
    <span class="navbar-toggler-icon"></span>
  </button>

  <div class="collapse navbar-collapse" id="navbarSupportedContent">
    <ul class="navbar-nav mr-auto">
      <li class="nav-item active">
        <a class="nav-link" href="index.html">Home <span class="sr-only">(current)</span></a>
      </li>
      <li class="nav-item">
        <a class="nav-link" href="diary.html">Diary</a>
      </li>
      <li class="nav-item dropdown">
        <a class="nav-link dropdown-toggle" href="phases.html" id="navbarDropdown" role="button" data-toggle="dropdown" aria-haspopup="true" aria-expanded="false">
          Phases
        </a>
        <div class="dropdown-menu" aria-labelledby="navbarDropdown">
          <a class="dropdown-item" href="phase1.html">Phase 1</a>
          <a class="dropdown-item" href="phase2.html">Phase 2</a>
          <a class="dropdown-item" href="phase3.html">Phase 3</a>
        </div>
      </li>
      <li class="nav-item">
        <a class="nav-link" href="meetings.html">Meetings</a>
      </li>
    </ul>
  </div>
</nav>


    <h1>Diary</h1>
      <div id="accordion">

  <div class="card">
    <div class="card-header" id="headingOne">
      <h5 class="mb-0">
        <button class="btn btn-link" data-toggle="collapse" data-target="#collapseOne" aria-expanded="true" aria-controls="collapseOne">
          Week #1 (01-14-20)
        </button>
      </h5>
    </div>

    <div id="collapseOne" class="collapse show" aria-labelledby="headingOne" data-parent="#accordion">
      <div class="card-body">
        <h2>UnitTest Introduction</h2>
        <p>UnitTest is a test runner built into a python library, containing both a testing framework and a test runner.</p>
        <p><em>Requirements:</em></p>
        <ul>
          <li>tests must be included into <em>classes</em> as <em>methods</em> </li>
          <li>instead of the built-in statement "assert", the <em>special assertion methods in unittest.TestCase class </em> must be used </li>
        </ul>
      
      <h2>Example 1</h2>
      <p>file test_sum_2.py</p>
      <div class="codebg">
        <code>
          <pre>
def test_sum():
    assert sum([1, 2, 3]) == 6, "Should be 6"

def test_sum_tuple():
    assert sum((1, 2, 2)) == 6, "Should be 6"

if __name__ == "__main__":
    test_sum()
    test_sum_tuple()
    print("Everything passed")
          </pre>
        </code>
      </div>
  <p>Steps of the procedure to convert the aforementioned example into a unittest case:</p>
  <ol>
    <li> <em>import unittest</em> from the library</li>
    <li>create a <em>TestSum class</em>, inherited from TestCase class</li>
    <li>use <em>self</em> as first argument, so that functions are turned into methods</li>
    <li>change assertions, so to use <em>self.assertEqual() method from the TestCase class</em></li>
    <li>change the command-line entry point to call unittest.main()</li>
  </ol>
  <div class="container">
    <img src="unittest1.png" alt="unittest1">

  </div>
    <div class="info"><p><span class="remind">assert</span><br>The assert keyword lets you test if a condition in your code returns True, if not, the program will raise an AssertionError.

You can write a message to be written if the code returns False</p>
      <div class="codebg">
        <code><pre>
x = "hello"

#if condition returns True, then nothing happens:
assert x == "hello"

#if condition returns False, AssertionError is raised:
assert x == "goodbye"
        </pre></code></div>
        <p>Source: <a href="https://www.w3schools.com/python/ref_keyword_assert.asp#:~:text=The%20assert%20keyword%20is%20used,False%2C%20check%20the%20example%20below.">w3school.com</a></p>
      </div>
    <p>Source:<a href="https://realpython.com/python-testing/">realpython.com</a></p>
    <h2>Example 2</h2>
    <div class="codebg">
      <code>
        <pre>
          import unittest

          def add_fish_to_aquarium(fish_list):
              if len(fish_list) > 10:
                  raise ValueError("A maximum of 10 fish can be added to the aquarium")
              return {"tank_a": fish_list}


          class TestAddFishToAquarium(unittest.TestCase):
              def test_add_fish_to_aquarium_success(self):
                  actual = add_fish_to_aquarium(fish_list=["shark", "tuna"])
                  expected = {"tank_a": ["shark", "tuna"]}
                  self.assertEqual(actual, expected)
        </pre>
      </code>
    </div>
    <p>Procedure:</p>
    <ol>
      <li>Import unittest</li>
      <li>Define the function to test, defining also cases in which the function raises errors</li>
      <li>Define a class as a subclass of unittest.TestCase</li>
      <li>Define one or more methods of the defined class (with self argument). This test function calls the function to test with a specific input and verifies whether the actual returned value matches the expected one</li>
      <li>Execute the test (from command line: python -m unittest test_add_fish_to_aquarium.py)</li>
    </ol>
    <p>Source <a href="https://www.digitalocean.com/community/tutorials/how-to-use-unittest-to-write-a-test-case-for-a-function-in-python">https://www.digitalocean.com/</a></p>
    <h2>Example 3</h2>
    <div class="codebg">
      <code>
        <pre>
          import unittest

          class TestStringMethods(unittest.TestCase):

              def test_upper(self):
                  self.assertEqual('foo'.upper(), 'FOO')

              def test_isupper(self):
                  self.assertTrue('FOO'.isupper())
                  self.assertFalse('Foo'.isupper())

              def test_split(self):
                  s = 'hello world'
                  self.assertEqual(s.split(), ['hello', 'world'])
                  # check that s.split fails when the separator is not a string
                  with self.assertRaises(TypeError):
                      s.split(2)

          if __name__ == '__main__':
              unittest.main()
        </pre>
      </code>
    </div>
    <p>Procedure:</p>
    <ul>
      <li>import unittest</li>
      <li>a testcase is created by subclassing unittest.TestCase.</li>
      <li>individual tests are defined with methods whose names start with the letters test, in order to inform the test runner about which methods represent tests.</li>
      <li>each test is a call to assertEqual() to check for an expected result; assertTrue() or assertFalse() to verify a condition; or assertRaises() to verify that a specific exception gets raised. Differently from the simple assert statement, these methods allow the test runner to accumulate all test results and produce a report.</li>
      <li>setUp() and tearDown() methods allow you to define instructions that will be executed before and after each test method.</li>
      <li>unittest.main() is a simple way to run the tests. Provides a command-line interface to the test script. </li>
      <li>Passing the -v option to your test script will instruct unittest.main() to enable a higher level of verbosity, so that each test's outcome can be seen individually.</li>
      <div class="container">
        <img src="prompt.png" alt="promptscreenshot">
      </div>
    </ul>
    <p>Source <a href="https://docs.python.org/3/library/unittest.html">unittest documentation</a></p>

    <h2>Classes and functions</h2>
    <div class="info"><span class="remind">tearDown()</span><br>
      <p>Method called immediately after the test method has been called and the result recorded. This is called even if the test method raised an exception, so the implementation in subclasses may need to be particularly careful about checking internal state. This method will only be called if the setUp() succeeds, regardless of the outcome of the test method.</p>
      </div>

    <div class="info"><span class="remind">setUp()</span><br>
      <p>Method called to prepare the test fixture. This is called immediately before calling the test method; other than AssertionError or SkipTest, any exception raised by this method will be considered an error rather than a test failure. </p>
      </div>

    <div class="info"><span class="remind">assertEqual(first, second, msg=None)</span><br>
      <p>Test that first and second are not equal. If the values do compare equal, the test will fail. </p>
      </div>

    <div class="info"><span class="remind">assertNotEqual(first, second, msg=None)</span><br>
      <p>Test that first and second are not equal. If the values do compare equal, the test will fail. </p>
      </div>

    <div class="info"><span class="remind">assertTrue(expr, second, msg=None) & assertFalse(expr, second, msg=None)</span><br>
      <p>Test that expr is true (or false). </p>
      </div>

    <div class="info"><span class="remind">assertIs(first, second, msg=None) & assertIsNot(first, second, msg=None)</span><br>
      <p>Test that first and second are (or are not) the same object. </p>
      </div>

    <div class="info"><span class="remind">assertIsNone(expr, msg=None) & assertIsNotNone(expr, msg=None)</span><br>
      <p>Test that expr is (or is not) None </p>
    </div>

    <div class="info"><span class="remind">assertIn(member, container, msg=None) & assertNotIn(member, container, msg=None)</span><br>
      <p>Test that member is (or is not) in container.</p>
      </div>

    <div class="info"><span class="remind">assertIsInstance(obj, cls, msg=None) & assertNotIsInstance(obj, cls, msg=None)</span><br>
      <p>Test that obj is (or is not) an instance of cls (which can be a class or a tuple of classes, as supported by isinstance()). To check for the exact type, use assertIs(type(obj), cls). </p>
      </div>

    <div class="info"><span class="remind">assertCountEqual(first, second, msg=None)</span><br>
      <p>Test that sequence first contains the same elements as second, regardless of their order.  </p>
      </div>

    <div class="info"><span class="remind">assertMultiLineEqual(first, second, msg=None)</span><br>
      <p>Test that the multiline string first is equal to the string second. </p>
      </div>

    <div class="info"><span class="remind">assertListEqual(first, second, msg=None) & assertTupleEqual(first, second, msg=None)</span><br>
      <p>Tests that two lists or tuples are equal. If not, an error message is constructed that shows only the differences between the two.</p>
      </div>

    <div class="info"><span class="remind">assertSetEqual(first, second, msg=None)</span><br>
      <p>Tests that two sets are equal. If not, an error message is constructed that lists the differences between the sets. This method is used by default when comparing sets or frozensets with assertEqual().</p>
      </div>

    <div class="info"><span class="remind">assertDictEqual(first, second, msg=None)</span><br>
      <p>Test that two dictionaries are equal. If not, an error message is constructed that shows the differences in the dictionaries. This method will be used by default to compare dictionaries in calls to assertEqual().</p>
      </div>

    <div class="info"><span class="remind">assertListEqual(first, second, msg=None) & assertTupleEqual(first, second, msg=None)</span><br>
      <p>Tests that two lists or tuples are equal. If not, an error message is constructed that shows only the differences between the two.</p>
      </div>

    <p>Source <a href="https://docs.python.org/3/library/unittest.html">unittest documentation</a></p>

    <h2>Organizing test code</h2>
    <ul>
      <li>In unittest, test cases are unittest.TestCase instances.</li>
      <li>The testing code of a TestCase instance should be entirely self contained, in order to be runnable either in isolation or in combination</li>
      <li>In order to test, the assert*() methods provided by the TestCase base class are used. When the test fails, an exception is raised with an explanatory message, and unittest identifies the test case as a failure.</li>
      <li>When many tests with repetitive set ups have to be executed, setUP() method can be automatically called for every test run. </li>
      <li>tearDown() method, similarly, tidies up after the test method has been run</li>
      <li>Such a working environment for the testing code is called a test fixture. A new TestCase instance is created as a unique test fixture used to execute each individual test method. Thus setUp(), tearDown(), and __init__() will be called once per test.</li>
      <li>Test suite, represented by unittest’s TestSuite class, is a mechanism to group TestCase implementations according to the features they test.</li>
    </ul>
    <p>Source <a href="https://docs.python.org/3/library/unittest.html">unittest documentation</a></p>

    <h2>Reusing old test code</h2>
    <p>FunctionTestCase class: TestCase subclass to wrap an existing test function. </p>
    <div class="codebg">
      <code>
        <pre>
          testcase = unittest.FunctionTestCase(testSomething,
                                     setUp=makeSomethingDB,
                                     tearDown=deleteSomethingDB)
        </pre>
      </code>

    </div>
    <p>Source <a href="https://docs.python.org/3/library/unittest.html">unittest documentation</a></p>

    <h2>index/test/09_croci.py/</h2>
    <div class="codebg">
      <code>
        <pre>
          import unittest
          from index.coci.glob import process
          from os import sep, makedirs
          from os.path import exists
          from shutil import rmtree
          from index.storer.csvmanager import CSVManager
          from index.croci.crowdsourcedcitationsource import CrowdsourcedCitationSource
          from csv import DictReader


          class CROCITest(unittest.TestCase):

              def setUp(self):
                  self.input_file = "index%stest_data%scroci_dump%ssource.csv" % (sep, sep, sep)
                  self.citations = "index%stest_data%scroci_dump%scitations.csv" % (sep, sep, sep)

              def test_citation_source(self):
                  ccs = CrowdsourcedCitationSource(self.input_file)
                  new = []
                  cit = ccs.get_next_citation_data()
                  while cit is not None:
                      citing, cited, creation, timespan, journal_sc, author_sc = cit
                      new.append({
                          "citing": citing,
                          "cited": cited,
                          "creation": "" if creation is None else creation,
                          "timespan": "" if timespan is None else timespan,
                          "journal_sc": "no" if journal_sc is None else journal_sc,
                          "author_sc": "no" if author_sc is None else author_sc
                      })
                      cit = ccs.get_next_citation_data()

                  with open(self.citations) as f:
                      old = list(DictReader(f))

                  self.assertEqual(new, old)
        </pre>
      </code>
    </div>
    <div class="info"><span class="remind">from index.coci.glob import process</span><br>
      <div class="codebg">
        <code>
          <pre>
            def process(input_dir, output_dir):
    if not exists(output_dir):
        makedirs(output_dir)
          </pre>
        </code>
      </div>
      </div>

      <div class="info"><span class="remind">from os import sep, makedirs</span><br>
        <ol>
          <li> <b>os.sep</b>: The character used by the operating system to separate pathname components. This is '/' for POSIX and '\\' for Windows. Note that knowing this is not sufficient to be able to parse or concatenate pathnames — use os.path.split() and os.path.join() — but it is occasionally useful. Also available via os.path. </li>
          <li> <b>os.makedirs</b>: os.makedirs(name, mode=0o777, exist_ok=False) is a recursive directory creation function. Like mkdir(), but makes all intermediate-level directories needed to contain the leaf directory. </li>
        </ol>
        </div>

        <div class="info"><span class="remind">from os.path import exists</span><br>
          <p> <b>os.path.exists</b>:os.path.exists(path)
return True if path refers to an existing path or an open file descriptor. Returns False for broken symbolic links. On some platforms, this function may return False if permission is not granted to execute os.stat() on the requested file, even if the path physically exists. In more recent versions (Python 3.3 and 3.6) path can be an integer (True is returned if it is an open file descriptor, False otherwise) and paths like objects are now accepted.  </p>
          </div>

          <div class="info"><span class="remind">from shutil import rmtree</span><br>
            <p> <b>shutil.rmtree</b>: shutil.rmtree(path, ignore_errors=False, onerror=None)
delete an entire directory tree; path must point to a directory (but not a symbolic link to a directory). If ignore_errors is true, errors resulting from failed removals will be ignored; if false or omitted, such errors are handled by calling a handler specified by onerror or, if that is omitted, they raise an exception. </p>
            </div>

            <div class="info"><span class="remind">from index.storer.csvmanager import CSVManager</span><br>
              <div class="codebg">
                <code>
                  <pre>
                    class CSVManager(object):
                        """This class is able to load a simple CSV composed by two fields, 'id' and
                        'value', and then to index all its items in a structured form so as to be
                        easily queried. In addition, it allows one to store new information in the CSV,
                        if needed."""

                        def __init__(self, csv_path=None, line_threshold=10000, store_new=True):
                            self.csv_path = csv_path
                            self.data = {}
                            self.store_new = store_new

                            if csv_path is not None and exists(csv_path):
                                CSVManager.__load_all_csv_files([csv_path], self.__load_csv, line_threshold=line_threshold)

                        @staticmethod
                        def load_csv_column_as_set(file_or_dir_path, key, line_threshold=10000):
                            result = set()

                            if exists(file_or_dir_path):
                                file_to_process = []
                                if isdir(file_or_dir_path):
                                    for cur_dir, cur_subdir, cur_files in walk(file_or_dir_path):
                                        for cur_file in cur_files:
                                            if cur_file.endswith(".csv"):
                                                file_to_process.append(cur_dir + sep + cur_file)
                                else:
                                    file_to_process.append(file_or_dir_path)

                                for item in CSVManager.__load_all_csv_files(file_to_process, CSVManager.__load_csv_by_key,
                                                                            line_threshold=line_threshold, key=key):
                                    result.update(item)

                            return result

                        @staticmethod
                        def __load_csv_by_key(csv_string, key):
                            result = set()

                            csv_metadata = DictReader(StringIO(csv_string), delimiter=',')
                            for row in csv_metadata:
                                result.add(row[key])

                            return result

                        @staticmethod
                        def __load_all_csv_files(list_of_csv_files, fun, line_threshold, **params):
                            result = []
                            header = None

                            for csv_path in list_of_csv_files:
                                with open(csv_path, encoding="utf-8") as f:
                                    csv_content = ""
                                    for idx, line in enumerate(f.readlines()):
                                        if header is None:
                                            header = line
                                            csv_content = header
                                        else:
                                            if idx % line_threshold == 0:
                                                result.append(fun(csv_content, **params))
                                                csv_content = header
                                            csv_content += line

                                result.append(fun(csv_content, **params))

                            return result

                        def get_value(self, id_string):
                            """It returns the set of values associated to the input 'id_string',
                            or None if 'id_string' is not included in the CSV."""
                            if id_string in self.data:
                                return set(self.data[id_string])

                        def add_value(self, id_string, value):
                            """It adds the value specified in the set of values associated to 'id_string'.
                            If the object was created with the option of storing also the data in a CSV
                            ('store_new' = True, default behaviour), then it also add new data in the CSV."""
                            if id_string not in self.data:
                                self.data[id_string] = set()

                            if value not in self.data[id_string]:
                                self.data[id_string].add(value)

                                if self.csv_path is not None and self.store_new:
                                    if not exists(self.csv_path):
                                        with open(self.csv_path, "w") as f:
                                            f.write('"id","value"\n')

                                    with open(self.csv_path, "a") as f:
                                        f.write('"%s","%s"\n' % (id_string.replace('"', '""'),
                                                                 value.replace('"', '""')))

                        def __load_csv(self, csv_string):
                            csv_metadata = DictReader(StringIO(csv_string), delimiter=',')
                            for row in csv_metadata:
                                cur_id = row["id"]
                                if cur_id not in self.data:
                                    self.data[cur_id] = set()

                                self.data[cur_id].add(row["value"])
                  </pre>
                </code>
              </div>
              </div>

              <div class="info"><span class="remind">from index.croci.crowdsourcedcitationsource import CrowdsourcedCitationSource</span><br>
                <div class="codebg">
                  <code>
                    <pre>
                      class CrowdsourcedCitationSource(CSVFileCitationSource):
                          def __init__(self, src, local_name=""):
                              self.doi = DOIManager()
                              super(CrowdsourcedCitationSource, self).__init__(src, local_name)

                          def get_next_citation_data(self):
                              row = self._get_next_in_file()

                              while row is not None:
                                  citing = self.doi.normalise(row.get("citing_id"))
                                  cited = self.doi.normalise(row.get("cited_id"))

                                  if citing is not None and cited is not None:
                                      created = row.get("citing_publication_date")
                                      if not created:
                                          created = None

                                      cited_pub_date = row.get("cited_publication_date")
                                      if not cited_pub_date:
                                          timespan = None
                                      else:
                                          c = Citation(None, None, created, None, cited_pub_date, None, None, None, None, "", None, None, None, None, None)
                                          timespan = c.duration

                                      self.update_status_file()
                                      return citing, cited, created, timespan, None, None

                                  self.update_status_file()
                                  row = self._get_next_in_file()

                              remove(self.status_file)
                    </pre>
                  </code>
                </div>
                </div>

                <div class="info"><span class="remind">from csv import DictReader</span><br>
                  <p> <b>class csv.DictReader</b>(f, fieldnames=None, restkey=None, restval=None, dialect='excel', *args, **kwds)
create an object that operates like a regular reader but maps the information in each row to a dict whose keys are given by the optional fieldnames parameter.
The fieldnames parameter is a sequence. If fieldnames is omitted, the values in the first row of file f will be used as the fieldnames. Regardless of how the fieldnames are determined, the dictionary preserves their original ordering.
If a row has more fields than fieldnames, the remaining data is put in a list and stored with the fieldname specified by restkey (which defaults to None). If a non-blank row has fewer fields than fieldnames, the missing values are filled-in with the value of restval (which defaults to None). </p>
<p>Example:</p>
              <div class="codebg">
                    <code>
                      <pre>
                        >>> import csv
                        >>> with open('names.csv', newline='') as csvfile:
                        ...     reader = csv.DictReader(csvfile)
                        ...     for row in reader:
                        ...         print(row['first_name'], row['last_name'])
                        ...
                        Eric Idle
                        John Cleese

                        >>> print(row)
                        {'first_name': 'John', 'last_name': 'Cleese'}
                      </pre>
                    </code>
                  </div>
                  </div>

<p>Sources:<a href="https://docs.python.org/3/library/">The Python Standard Library</a> and <a href="https://github.com/opencitations/index">OpenCitations Index</a> </p>



    <h2>Exercises</h2>
    <h3>Beginners 1</h3>
    <div class="codebg">
      <pre>
        <code>
import unittest

def t(x, y):
    return x + y - 2

class TestT(unittest.TestCase):
    def test_t(self):
        actual = t(x=5, y=6)
        expected = 5+6-2
        self.assertEqual(actual, expected)
        </code>
      </pre>
    </div>
<p>In this way it works</p>
    <div class="codebg">
      <pre>
        <code>
        import unittest

        class TestT(unittest.TestCase):
            def test_t(self):
                result = t(x, y)
                expected = x + y -2
                self.assertEqual(result, expected)

        def t(x, y):
            return x + y - 2
        </code>
      </pre>
    </div>
<p>((In this way it doesn't. I think I know why but I'd like to clarify this point)) </p>
<div class="codebg">
  <code>
    <pre>
      def t(x, y):
          return x + y - 2

      import unittest

      class TestT(unittest.TestCase):
          def test_t(self):
              self.assertEqual(t(5,9), 12)


      if __name__ == "__main__":
          unittest.main()
    </pre>
  </code>
</div>
<p>In this way it works and I also have the result displayed in Pycharm console</p>
<h3>Beginners 2</h3>
<div class="codebg">
  <code>
    <pre>
def f(s1, s2):
    if len(s1) > len(s2):
        return -1
    elif len(s1) < len(s2):
        return 1
    else:
        return 0

import unittest

class TestF(unittest.TestCase):
    def test_f(self):
        self.assertEqual(f("dog","cat"), 0)
        self.assertEqual(f("dog","monkey"), 1)
        self.assertEqual(f("dog", "my"), -1)


if __name__ == "__main__":
    unittest.main()
    </pre>
  </code>
</div>
<h3>Beginners 3</h3>
<div class="codebg">
  <code>
    <pre>
      def f(s1, s2):
          result = set()

          for c in s1:
              if c in s2:
                  result.add(c)

          return result

      import unittest

      class TestF(unittest.TestCase):
          def test_f(self):
              self.assertEqual(f("123","435"), {"3"})
              self.assertEqual(f("126","435"), set()) #if I use {} it doesn't work
              self.assertEqual(f("brother","sister"), {"r","t","e"})
              self.assertEqual(f("cat","catsitter"), {"c","a","t"})

      if __name__ == "__main__":
          unittest.main()
    </pre>
  </code>
</div>
<p>Original codes source: <a href="https://comp-think.github.io/">The CTP Book</a> </p>

    <h2>Issues</h2>
    <ol>
      <li># check that s.split fails when the separator is not a string \\
      with self.assertRaises(TypeError):\\
          s.split(2)\\?</li>
      <li>Why is unittest more efficient then other testing approaches?</li>
      <li>In practice, when and how do we use assertRaises(TypeError)? </li>
      <li>if __name__ == '__main__':\\
          unittest.main()</li>
      <li>In many of the tutorials I followed the code is launched through the command prompt. Is it necessary or suggested? Can I work with Pycharm only?</li>
      <li>Is TestSuit necessary in our case? "In most cases, calling unittest.main() will do the right thing and collect all the module’s test cases for you and execute them."</li>
      <li>Does the assertEqual have more or less the same function of the print part of the test case that we used in CTP course? (since -in order to make the test work- some values have to be assigned to the arguments of the tested function) </li>
      <li>For now I'm only using assertEqual() since I have some troubles in understanding how all the other methods could be applied in tests (how can I do something more than testing if the result of the function matches the expected one? In OpenCitation which other ones are mainly used?).</li>
      <li>SetUp(): real contexts of use (examples)? In particular setUp in CrociTest <br> self.input_file = "index%stest_data%scroci_dump%ssource.csv" % (sep, sep, sep) <br>
        self.citations = "index%stest_data%scroci_dump%scitations.csv" % (sep, sep, sep)</li>
    </ol>
      </div>
    </div>
  </div>

  <div class="card">
    <div class="card-header" id="headingTwo">
      <h5 class="mb-0">
        <button class="btn btn-link collapsed" data-toggle="collapse" data-target="#collapseTwo" aria-expanded="false" aria-controls="collapseTwo">
          Week #2 (01-21-21)
        </button>
      </h5>
    </div>
    <div id="collapseTwo" class="collapse" aria-labelledby="headingTwo" data-parent="#accordion">
      <div class="card-body">
          <h2>NIH library data</h2>
          <h3>Ten examples</h3>
          <p>I randomly extracted ten samples from the open_citation_collection.csv and I stored the data in a new csv file (open_citation_collection_sample.csv, then renamed source.csv).</p>
          <div class="codebg"><code><pre>C:\Users\arimoretti\Desktop\TESI\icite>pip install subsample
Collecting subsample
  Downloading https://files.pythonhosted.org/packages/21/86/ff247222b81baa8b47a9970c76e6f930662d7b9336e669f7e4b39857e674/subsample-0.0.6.tar.gz
Installing collected packages: subsample
  Running setup.py install for subsample ... done
Successfully installed subsample-0.0.6
WARNING: You are using pip version 19.2.3, however version 21.0.1 is available.
You should consider upgrading via the 'python -m pip install --upgrade pip' command.

C:\Users\arimoretti\Desktop\TESI\icite>subsample -n 10 open_citation_collection.csv > open_citation_collection_sample.csv
LOG 03:21 > Data begins at 0

C:\Users\arimoretti\Desktop\TESI\icite></pre></code></div>
          <ol>
              <li>2140506,2942070</li>
              <li>1523579,7097569</li>
              <li>1509982,6501574</li>
              <li>1968312,13673087</li>
              <li>2330868,3958380</li>
              <li>1854174,3037997</li>
              <li>2038824,2494239</li>
              <li>2373284,7189714</li>
              <li>3591292,4092853</li>
              <li>2368927,355650</li>
          </ol>
          <h3>Notes on format</h3>
          <p>Before extracting data, we expected these to be either PMCID or  PMID. The pure numeric format denotes that we are handling PMIDs.</p>
          <p>OC process is based on the use of a <strong>tuple of six values</strong> derived from the citation source(citing, cited, created, timespan, journal_sc, author_sc). However, here we just have two of these values, which are also the only indispensable ones (i.e. citing + referenced/cited).</p>
          
          <h2>Data mapping</h2>
          <p>The 2 primary identifiers are not DOIs, but PubMedID. Since until now only DOIs have been managed by our process, some of these citations may already be present in Open Citations (COCI: doi to doi only). </p>
          <p>In NIH dataset some articles probably have a doi, but exposed differently. So our options for the mapping are: </p>
          <ol>
              <li>use article additional data provided by icite</li>
              <li>use external mapping datasets</li>
          </ol>
          <h3>External mapping tools</h3>
          <h4>Results obtained with <a href="https://www.pmid2cite.com/pmid-to-doi-converter">www.pmid2cite.com</a></h4>
          <table style="width: 117px;">
<tbody>
<tr style="height: 22px;">
<td style="height: 22px; width: 15.2px;">&nbsp;</td>
<td style="height: 22px; width: 64.8px;">PMID citing</td>
<td style="height: 22px; width: 36px;">&nbsp;DOI citing</td>
<td style="height: 22px; width: 36px;">PMID referenced</td>
<td style="height: 22px; width: 36px;">DOI referenced</td>
</tr>
<tr style="height: 22px;">
<td style="height: 22px; width: 15.2px;">1</td>
<td style="height: 22px; width: 64.8px;">&nbsp;2140506</td>
<td style="height: 22px; width: 36px;"><a title="Link by DOI for the given PMID" href="https://doi.org/10.1111/j.1445-5994.1990.tb01289.x" target="_blank">10.1111/j.1445-5994.1990.tb01289.x</a></td>
<td style="height: 22px; width: 36px;">2942070</td>
<td style="height: 22px; width: 36px;"><a title="Link by DOI for the given PMID" href="https://doi.org/10.7326/0003-4819-105-2-173" target="_blank">10.7326/0003-4819-105-2-173</a></td>
</tr>
<tr style="height: 22px;">
<td style="height: 22px; width: 15.2px;">2</td>
<td style="height: 22px; width: 64.8px;">&nbsp;1523579</td>
<td style="height: 22px; width: 36px;"><a title="Link by DOI for the given PMID" href="https://doi.org/10.1002/tera.1420460305" target="_blank">10.1002/tera.1420460305</a></td>
<td style="height: 22px; width: 36px;">7097569</td>
<td style="height: 22px; width: 36px;">Unable to resolve the DOI&nbsp;</td>
</tr>
<tr style="height: 22px;">
<td style="height: 22px; width: 15.2px;">3</td>
<td style="height: 22px; width: 64.8px;">&nbsp;1509982</td>
<td style="height: 22px; width: 36px;"><a title="Link by DOI for the given PMID" href="https://doi.org/10.1007/BF01990957" target="_blank">10.1007/BF01990957</a></td>
<td style="height: 22px; width: 36px;">6501574</td>
<td style="height: 22px; width: 36px;"><a title="Link by DOI for the given PMID" href="https://doi.org/10.1172/JCI111604" target="_blank">10.1172/JCI111604</a></td>
</tr>
<tr style="height: 22px;">
<td style="height: 22px; width: 15.2px;">4</td>
<td style="height: 22px; width: 64.8px;">&nbsp;1968312</td>
<td style="height: 22px; width: 36px;"><a title="Link by DOI for the given PMID" href="https://doi.org/10.1016/0002-9149(90)90817-k" target="_blank">10.1016/0002-9149(90)90817-k</a></td>
<td style="height: 22px; width: 36px;">13673087</td>
<td style="height: 22px; width: 36px;"><a title="Link by DOI for the given PMID" href="https://doi.org/10.1172/JCI103906" target="_blank">10.1172/JCI103906</a></td>
</tr>
<tr style="height: 22px;">
<td style="height: 22px; width: 15.2px;">5</td>
<td style="height: 22px; width: 64.8px;">&nbsp;2330868</td>
<td style="height: 22px; width: 36px;"><a title="Link by DOI for the given PMID" href="https://doi.org/10.1016/s0002-8703(05)80240-6" target="_blank">10.1016/s0002-8703(05)80240-6</a></td>
<td style="height: 22px; width: 36px;">3958380</td>
<td style="height: 22px; width: 36px;"><a title="Link by DOI for the given PMID" href="https://doi.org/10.1016/s0735-1097(86)80214-5" target="_blank">10.1016/s0735-1097(86)80214-5</a></td>
</tr>
<tr style="height: 22px;">
<td style="height: 22px; width: 15.2px;">6</td>
<td style="height: 22px; width: 64.8px;">&nbsp;1854174</td>
<td style="height: 22px; width: 36px;"><a title="Link by DOI for the given PMID" href="https://doi.org/10.1128/aac.35.5.929" target="_blank">10.1128/aac.35.5.929</a></td>
<td style="height: 22px; width: 36px;">3037997</td>
<td style="height: 22px; width: 36px;">
<div id="buttonHide" class="out">
<p class="output"><a title="Link by DOI for the given PMID" href="https://doi.org/10.1128/aac.31.4.497" target="_blank">10.1128/aac.31.4.497</a></p>
</div>
</td>
</tr>
<tr style="height: 22px;">
<td style="height: 22px; width: 15.2px;">7</td>
<td style="height: 22px; width: 64.8px;">&nbsp;2038824</td>
<td style="height: 22px; width: 36px;"><a title="Link by DOI for the given PMID" href="https://doi.org/10.1016/0165-2427(91)90031-7" target="_blank">10.1016/0165-2427(91)90031-7</a></td>
<td style="height: 22px; width: 36px;">2494239</td>
<td style="height: 22px; width: 36px;"><a title="Link by DOI for the given PMID" href="https://doi.org/10.3168/jds.S0022-0302(89)79089-5" target="_blank">10.3168/jds.S0022-0302(89)79089-5</a></td>
</tr>
<tr style="height: 22.6px;">
<td style="height: 22.6px; width: 15.2px;">8</td>
<td style="height: 22.6px; width: 64.8px;">&nbsp;2373284</td>
<td style="height: 22.6px; width: 36px;"><a title="Link by DOI for the given PMID" href="https://doi.org/10.1111/j.1432-0436.1990.tb00437.x" target="_blank">10.1111/j.1432-0436.1990.tb00437.x</a></td>
<td style="height: 22.6px; width: 36px;">7189714</td>
<td style="height: 22.6px; width: 36px;"><a title="Link by DOI for the given PMID" href="https://doi.org/10.1016/0014-4827(80)90437-1" target="_blank">10.1016/0014-4827(80)90437-1</a></td>
</tr>
<tr style="height: 22px;">
<td style="height: 22px; width: 15.2px;">9</td>
<td style="height: 22px; width: 64.8px;">&nbsp;3591292</td>
<td style="height: 22px; width: 36px;"><a title="Link by DOI for the given PMID" href="https://doi.org/10.1111/j.1651-2227.1987.tb10456.x" target="_blank">10.1111/j.1651-2227.1987.tb10456.x</a></td>
<td style="height: 22px; width: 36px;">4092853</td>
<td style="height: 22px; width: 36px;"><a title="Link by DOI for the given PMID" href="https://doi.org/10.1111/j.1469-8749.1985.tb03805.x" target="_blank">10.1111/j.1469-8749.1985.tb03805.x</a></td>
</tr>
<tr style="height: 22px;">
<td style="height: 22px; width: 15.2px;">10</td>
<td style="height: 22px; width: 64.8px;">2368927</td>
<td style="height: 22px; width: 36px;"><a title="Link by DOI for the given PMID" href="https://doi.org/10.1002/ar.1092270109" target="_blank">10.1002/ar.1092270109</a></td>
<td style="height: 22px; width: 36px;">355650</td>
<td style="height: 22px; width: 36px;">Unable to resolve the DOI&nbsp;</td>
</tr>
</tbody>
</table>

<h4>Results obtained with <a href="https://www.ncbi.nlm.nih.gov/pmc/pmctopmid/#api">/www.ncbi.nlm.nih.gov</a></h4>
          
          <table style="width: 117px;">
<tbody>
<tr style="height: 22px;">
<td style="height: 22px; width: 15.2px;">&nbsp;</td>
<td style="height: 22px; width: 64.8px;">PMID citing</td>
<td style="height: 22px; width: 36px;">&nbsp;DOI citing</td>
<td style="height: 22px; width: 36px;">PMID referenced</td>
<td style="height: 22px; width: 36px;">DOI referenced</td>
</tr>
<tr style="height: 22px;">
<td style="height: 22px; width: 15.2px;">1</td>
<td style="height: 22px; width: 64.8px;">&nbsp;2140506</td>
<td style="height: 22px; width: 36px;">""</td>
<td style="height: 22px; width: 36px;">2942070</td>
<td style="height: 22px; width: 36px;">""</td>
</tr>
<tr style="height: 22px;">
<td style="height: 22px; width: 15.2px;">2</td>
<td style="height: 22px; width: 64.8px;">&nbsp;1523579</td>
<td style="height: 22px; width: 36px;">""</td>
<td style="height: 22px; width: 36px;">7097569</td>
<td style="height: 22px; width: 36px;">""</td>
</tr>
<tr style="height: 22px;">
<td style="height: 22px; width: 15.2px;">3</td>
<td style="height: 22px; width: 64.8px;">&nbsp;1509982</td>
<td style="height: 22px; width: 36px;">""</td>
<td style="height: 22px; width: 36px;">6501574</td>
<td style="height: 22px; width: 36px;"><a title="Link by DOI for the given PMID" href="https://doi.org/10.1172/JCI111604" target="_blank">10.1172/JCI111604</a></td>
</tr>
<tr style="height: 22px;">
<td style="height: 22px; width: 15.2px;">4</td>
<td style="height: 22px; width: 64.8px;">&nbsp;1968312</td>
<td style="height: 22px; width: 36px;">""</td>
<td style="height: 22px; width: 36px;">13673087</td>
<td style="height: 22px; width: 36px;"><a title="Link by DOI for the given PMID" href="https://doi.org/10.1172/JCI103906" target="_blank">10.1172/JCI103906</a></td>
</tr>
<tr style="height: 22px;">
<td style="height: 22px; width: 15.2px;">5</td>
<td style="height: 22px; width: 64.8px;">&nbsp;2330868</td>
<td style="height: 22px; width: 36px;">""</td>
<td style="height: 22px; width: 36px;">3958380</td>
<td style="height: 22px; width: 36px;">""</td>
</tr>
<tr style="height: 22px;">
<td style="height: 22px; width: 15.2px;">6</td>
<td style="height: 22px; width: 64.8px;">&nbsp;1854174</td>
<td style="height: 22px; width: 36px;"><a title="Link by DOI for the given PMID" href="https://doi.org/10.1128/aac.35.5.929" target="_blank">10.1128/aac.35.5.929</a></td>
<td style="height: 22px; width: 36px;">3037997</td>
<td style="height: 22px; width: 36px;">
<div id="buttonHide" class="out">
<p class="output"><a title="Link by DOI for the given PMID" href="https://doi.org/10.1128/aac.31.4.497" target="_blank">10.1128/aac.31.4.497</a></p>
</div>
</td>
</tr>
<tr style="height: 22px;">
<td style="height: 22px; width: 15.2px;">7</td>
<td style="height: 22px; width: 64.8px;">&nbsp;2038824</td>
<td style="height: 22px; width: 36px;">""</td>
<td style="height: 22px; width: 36px;">2494239</td>
<td style="height: 22px; width: 36px;">""</td>
</tr>
<tr style="height: 22.6px;">
<td style="height: 22.6px; width: 15.2px;">8</td>
<td style="height: 22.6px; width: 64.8px;">&nbsp;2373284</td>
<td style="height: 22.6px; width: 36px;">""</td>
<td style="height: 22.6px; width: 36px;">7189714</td>
<td style="height: 22.6px; width: 36px;">""</td>
</tr>
<tr style="height: 22px;">
<td style="height: 22px; width: 15.2px;">9</td>
<td style="height: 22px; width: 64.8px;">&nbsp;3591292</td>
<td style="height: 22px; width: 36px;">""</td>
<td style="height: 22px; width: 36px;">4092853</td>
<td style="height: 22px; width: 36px;">""</td>
</tr>
<tr style="height: 22px;">
<td style="height: 22px; width: 15.2px;">10</td>
<td style="height: 22px; width: 64.8px;">2368927</td>
<td style="height: 22px; width: 36px;">""</td>
<td style="height: 22px; width: 36px;">355650</td>
<td style="height: 22px; width: 36px;">""</td>
</tr>
</tbody>
</tbody>
</table>
        
<p>As we can see, results obtained by using different tools are not the same. It is evident that the first tool obtains better results, but - most importantly - neither the one nor the other is able to match 100% of PMID-DOI couples. So, the mapping is incomplete.</p>
          
          <h2>Notes on expected structure of the main function</h2> 
        <p>In order to understand what kind of input and output to expect, we can start from the model provided by CROCI.</p>
          <h2>Test Case development</h2>
        <p>Reference Test Case:</p>
        <h3>index/test/09_croci.py</h3>
        <div class="codebg"><code><pre>          import unittest
          from index.coci.glob import process
          from os import sep, makedirs
          from os.path import exists
          from shutil import rmtree
          from index.storer.csvmanager import CSVManager
          from index.croci.crowdsourcedcitationsource import CrowdsourcedCitationSource
          from csv import DictReader


          class CROCITest(unittest.TestCase):

              def setUp(self):
                  self.input_file = "index%stest_data%scroci_dump%ssource.csv" % (sep, sep, sep)
                  self.citations = "index%stest_data%scroci_dump%scitations.csv" % (sep, sep, sep)

              def test_citation_source(self):
                  ccs = CrowdsourcedCitationSource(self.input_file)
                  new = []
                  cit = ccs.get_next_citation_data()
                  while cit is not None:
                      citing, cited, creation, timespan, journal_sc, author_sc = cit
                      new.append({
                          "citing": citing,
                          "cited": cited,
                          "creation": "" if creation is None else creation,
                          "timespan": "" if timespan is None else timespan,
                          "journal_sc": "no" if journal_sc is None else journal_sc,
                          "author_sc": "no" if author_sc is None else author_sc
                      })
                      cit = ccs.get_next_citation_data()

                  with open(self.citations) as f:
                      old = list(DictReader(f))

                  self.assertEqual(new, old)</pre></code></div>
        
        <h3>On this base, a basic idea of index/test/(n?)_noci.py</h3>
        
        <iframe src="https://trinket.io/embed/python/221e5803ec?toggleCode=true" width="100%" height="356" frameborder="0" marginwidth="0" marginheight="0" allowfullscreen></iframe>
        
        
        <div class="codebg">
            <code>
                <pre>
        
import unittest
from index.coci.glob import process #something to be developed to perform the same task but with pmids
from os import sep, makedirs  #Recursive directory creation function. Like mkdir(), but makes all intermediate-level directories needed to contain the leaf directory.
from os.path import exists  #used to check whether the specified path exists or not.
from shutil import rmtree #used to delete an entire directory tree, path must point to a directory (but not a symbolic link to a directory)
from index.storer.csvmanager import CSVManager #This class is able to load a simple CSV composed by two fields, 'id' and'value', and then to index all its items in a structured form so as to be easily queried. In addition, it allows one to store new information in the CSV, if needed.
from index.noci.nationalinstituteofhealthsource import NationalInstitutHealthSource #crowdsourcedcitationsource sostituito
from csv import DictReader


class NOCITest(unittest.TestCase):

    def setUp(self): #The self parameter is a reference to the current instance of the class, and is used to access variables that belongs to the class.
        self.input_file = "index%stest_data%snoci_dump%ssource.csv" % (sep, sep, sep) #without placeholders for the tuple format, data as provided by NIH
        self.citations = "index%stest_data%snoci_dump%scitations.csv" % (sep, sep, sep) #adaptation for tuple format, with all 6 fields
    def test_citation_source(self):
        ns = NationalInstitutHealthSource(self.input_file) #class of nationalinstituteofhealthsource, where a version of get next citation handling pmid data will be defined.
        new = []
        cit = ns.get_next_citation_data_pmidversion() #will extract citational data from our input file, which is the source one, with just citing and cited
        while cit is not None: #(which is: until we have citational data)
            citing, cited, creation, timespan, journal_sc, author_sc = cit #related to the csv format (?) does the comma separator define the fields, so that cit becomes che full ciation divided by fields?
            #we append each citational data to the new list (which will be a list of dictionaries), in the format required for the 6-elements tuple.
            new.append({
                "citing": citing,
                "cited": cited,
                "creation": "" if creation is None else creation, #in all our cases the last four fields will be "","","no","no"
                "timespan": "" if timespan is None else timespan,
                "journal_sc": "no" if journal_sc is None else journal_sc,
                "author_sc": "no" if author_sc is None else author_sc
            })
            cit = ns.get_next_citation_data_pmidversion() #cit variable is assigned the value of the subsequent citational datum from input_file (source.csv)

        with open(self.citations) as f: #open the file as a parameter of a function, this time the csv with the data compiled in the 6-elements tuple format
            old = list(DictReader(f)) #a list of dictionaries is derived from data in citations.


        self.assertEqual(new, old) #check that the lists of dictionaries derived respectively from source and from citations are the equal.
                </pre>
            </code>
        </div>
          <h2>Prompt: basic commands</h2>
        <h3>General</h3>
        <ul>
            <li>Lists Installed Drivers (<strong>driverquery</strong>): get a full list of installed drivers in your pc (driverquery -v to obtain more information). </li>
            <li><strong>ipconfig</strong>: provides your ip address + your local network.</li>
            <li><strong>systeminfo</strong>:  to know very basic information about your pc’s hardware, like – motherboard, processor, ram</li>
            <li><strong>ping</strong>:sends packets of data to a specific IP address (or domain) on a network and then lets you know how long it took to transmit that data and get a response (ping + ip or domain).</li>
            <li><strong>sfc /scannow</strong>scan and repare windown system files. But you must be run the console as an administrator</li>
            <li><strong>tasklist</strong>: gets a list of all tasks running on your pc (tasklist -v). </li>
            <li><strong>cd</strong>: </li>
            <ol>
                <li><strong>cd\</strong> to go to the top of the directory tree</li>
                <li><strong>CD Folder</strong>to go to a specific folder from this drive. The subfolders must be separated by a backslash character: \.</li>
                <li><strong>cd..</strong>to go one folder up.</li>
            </ol>
            <li>To change the drive from “C:” to “D:”, type d:</li>
            <li><strong>mkdir</strong>:  (Make Directory) command. mkdir Folder creates a folder.</li>
            <li><strong>cls</strong>: clear screen, to clear the command prompt screen.</li>
        </ul>
        <p>Source: <a href="https://dev.to/iamprogrammmer/command-prompt-basic-commands-you-should-know-cmd-4aj">dev.to</a></p>
        <h3>Unittest</h3>
        <p> Unittest from command line in order to make tests on modules, classes or specific methods</p>
        <ul>
            <li>python -m unittest test_module1 test_module2</li>
            <li>python -m unittest test_module.TestClass</li>
            <li>python -m unittest test_module.TestClass.test_method</li>
        </ul>
        <p>Test modules can be specified by file path, e.g.: python -m unittest tests/test_something.py</p>
        <p>For additional information, we can obtain tests with more detail (higher verbosity) by passing in the -v flag: python -m unittest -v test_module</p>
        <p>List of all command line options: python -m unittest -h</p>
        <div class="codebg">
            <code>
                <pre>
                C:\Users\arimoretti>python -m unittest -h
usage: python.exe -m unittest [-h] [-v] [-q] [--locals] [-f] [-c] [-b] [-k TESTNAMEPATTERNS] [tests [tests ...]]

positional arguments:
  tests                a list of any number of test modules, classes and test methods.

optional arguments:
  -h, --help           show this help message and exit
  -v, --verbose        Verbose output
  -q, --quiet          Quiet output
  --locals             Show local variables in tracebacks
  -f, --failfast       Stop on first fail or error
  -c, --catch          Catch Ctrl-C and display results so far
  -b, --buffer         Buffer stdout and stderr during tests
  -k TESTNAMEPATTERNS  Only run tests which match the given substring

Examples:
  python.exe -m unittest test_module               - run tests from test_module
  python.exe -m unittest module.TestClass          - run tests from module.TestClass
  python.exe -m unittest module.Class.test_method  - run specified test method
  python.exe -m unittest path/to/test_file.py      - run tests from test_file.py

usage: python.exe -m unittest discover [-h] [-v] [-q] [--locals] [-f] [-c] [-b] [-k TESTNAMEPATTERNS] [-s START]
                                       [-p PATTERN] [-t TOP]

optional arguments:
  -h, --help            show this help message and exit
  -v, --verbose         Verbose output
  -q, --quiet           Quiet output
  --locals              Show local variables in tracebacks
  -f, --failfast        Stop on first fail or error
  -c, --catch           Catch Ctrl-C and display results so far
  -b, --buffer          Buffer stdout and stderr during tests
  -k TESTNAMEPATTERNS   Only run tests which match the given substring
  -s START, --start-directory START
                        Directory to start discovery ('.' default)
  -p PATTERN, --pattern PATTERN
                        Pattern to match tests ('test*.py' default)
  -t TOP, --top-level-directory TOP
                        Top level directory of project (defaults to start directory)

For test discovery (python -m unittest) all test modules must be importable from the top level directory of the project.

C:\Users\arimoretti>
                </pre>
            </code>

        </div>
<p>Source: <a href="https://docs.python.org/3/library/unittest.html">docs.python.org</a></p>

          <h2>Clone locally OC index so to run tests.</h2>
        <p> I tried to clone OC index locally but something went wrong.</p>
        <div class="codebg"><code><pre>
        Cloning into 'C:\Users\arimoretti\Documents\GitHub\index'...
remote: Enumerating objects: 1040, done.        
remote: Counting objects: 100% (1040/1040), done.        
remote: Compressing objects: 100% (481/481), done.        
remote: Total 1040 (delta 373), reused 913 (delta 254), pack-reused 0        
Receiving objects: 100% (1040/1040), 172.79 KiB | 875.00 KiB/s, done.
Resolving deltas: 100% (373/373), done.
error: invalid path 'test_data/tmp_load/data/csv/2020/05/2020-05-18T11:01:36_1.csv'
fatal: unable to checkout working tree
warning: Clone succeeded, but checkout failed.
You can inspect what was checked out with 'git status'
and retry with 'git restore --source=HEAD :/'
        </pre></code></div>
        <p>Then I tried to fix it on my own but I didn't succeed.</p>
        <div class="codebg"><code><pre>
        
C:\Users\arimoretti\Documents\GitHub\index> git status
On branch master
Your branch is up to date with 'origin/master'.

Changes to be committed:
  (use "git restore --staged <file>..." to unstage)
        deleted:    .gitignore
        deleted:    README.md
        deleted:    __init__.py
        deleted:    citation/__init__.py
        deleted:    citation/citationsource.py
        deleted:    citation/oci.py
        deleted:    cnc.py
        deleted:    coci/checkmetadata.py
        deleted:    coci/crossrefcitationsource.py
        deleted:    coci/glob.py
        deleted:    coci/trimdump.py
        deleted:    croci/crowdsourcedcitationsource.py
        deleted:    finder/__init__.py
        deleted:    finder/crossrefresourcefinder.py
        deleted:    finder/dataciteresourcefinder.py
        deleted:    finder/orcidresourcefinder.py
        deleted:    finder/resourcefinder.py
        deleted:    identifier/doimanager.py
        deleted:    identifier/identifiermanager.py
        deleted:    identifier/issnmanager.py
        deleted:    identifier/orcidmanager.py
        deleted:    storer/__init__.py
        deleted:    storer/citationstorer.py
        deleted:    storer/csvmanager.py
        deleted:    storer/updatetp.py
        deleted:    support/__init__.py
        deleted:    support/csv.py
        deleted:    support/dictionary.py
        deleted:    support/stats.py
        deleted:    test/01_csvmanager.py
        deleted:    test/02_identifiermanager.py
        deleted:    test/03_resourcefinder.py
        deleted:    test/04_oci.py
        deleted:    test/05_citationstorer.py
        deleted:    test/06_citationsource.py
        deleted:    test/07_cnc.py
        deleted:    test/08_coci.py
        deleted:    test/09_croci.py
        deleted:    test/__init__.py
        deleted:    test/test.py
        deleted:    test_data/additional_data.csv
        deleted:    test_data/citations_data.csv
        deleted:    test_data/citations_data.ttl
        deleted:    test_data/citations_data_prov.scholix
        deleted:    test_data/citations_partial.csv
        deleted:    test_data/citations_prov.csv
        deleted:    test_data/citations_prov.ttl
        deleted:    test_data/cnc_id_date.csv
        deleted:    test_data/cnc_id_issn.csv
        deleted:    test_data/cnc_id_orcid.csv
        deleted:    test_data/cnc_valid_doi.csv
        deleted:    test_data/croci_dump/citations.csv
        deleted:    test_data/croci_dump/source.csv
        deleted:    test_data/crossref_dump/1.json
        deleted:    test_data/crossref_dump/2.json
        deleted:    test_data/crossref_dump/citations.csv
        deleted:    test_data/crossref_dump/id_date.csv
        deleted:    test_data/crossref_dump/id_issn.csv
        deleted:    test_data/crossref_dump/id_orcid.csv
        deleted:    test_data/crossref_dump/valid_doi.csv
        deleted:    test_data/crossref_glob/id_date.csv
        deleted:    test_data/crossref_glob/id_issn.csv
        deleted:    test_data/crossref_glob/id_orcid.csv
        deleted:    test_data/crossref_glob/valid_doi.csv
        deleted:    test_data/id_date.csv
        deleted:    test_data/id_issn.csv
        deleted:    test_data/id_orcid.csv
        deleted:    test_data/initial_data.csv
        deleted:    test_data/lookup_full.csv
        deleted:    test_data/lookup_new.csv
        deleted:    test_data/tmp_load/data/csv/2020/05/2020-05-18T11:01:36_1.csv
        deleted:    test_data/tmp_load/data/rdf/2020/05/2020-05-18T11:01:36_1.ttl
        deleted:    test_data/tmp_load/data/slx/2020/05/2020-05-18T11:01:36_1.scholix
        deleted:    test_data/tmp_load/prov/csv/2020/05/2020-05-18T11:01:36_1.csv
        deleted:    test_data/tmp_load/prov/rdf/2020/05/2020-05-18T11:01:36_1.ttl
        deleted:    test_data/tmp_store/data/csv/2020/05/2020-05-18T11:01:36_1.csv
        deleted:    test_data/tmp_store/data/csv/2020/05/2020-05-18T11:01:36_2.csv
        deleted:    test_data/tmp_store/data/csv/2020/05/2020-05-18T11:01:36_3.csv
        deleted:    test_data/tmp_store/data/rdf/2020/05/2020-05-18T11:01:36_1.ttl
        deleted:    test_data/tmp_store/data/rdf/2020/05/2020-05-18T11:01:36_2.ttl
        deleted:    test_data/tmp_store/data/rdf/2020/05/2020-05-18T11:01:36_3.ttl
        deleted:    test_data/tmp_store/data/rdf/2020/05/2020-05-18T11:01:36_4.ttl
        deleted:    test_data/tmp_store/data/rdf/2020/05/2020-05-18T11:01:36_5.ttl
        deleted:    test_data/tmp_store/data/rdf/2020/05/2020-05-18T11:01:36_6.ttl
        deleted:    test_data/tmp_store/data/slx/2020/05/2020-05-18T11:01:36_1.scholix
        deleted:    test_data/tmp_store/data/slx/2020/05/2020-05-18T11:01:36_2.scholix
        deleted:    test_data/tmp_store/data/slx/2020/05/2020-05-18T11:01:36_3.scholix
        deleted:    test_data/tmp_store/data/slx/2020/05/2020-05-18T11:01:36_4.scholix
        deleted:    test_data/tmp_store/prov/csv/2020/05/2020-05-18T11:01:36_1.csv
        deleted:    test_data/tmp_store/prov/csv/2020/05/2020-05-18T11:01:36_2.csv
        deleted:    test_data/tmp_store/prov/csv/2020/05/2020-05-18T11:01:36_3.csv
        deleted:    test_data/tmp_store/prov/rdf/2020/05/2020-05-18T11:01:36_1.ttl
        deleted:    test_data/tmp_store/prov/rdf/2020/05/2020-05-18T11:01:36_2.ttl
        deleted:    test_data/tmp_store/prov/rdf/2020/05/2020-05-18T11:01:36_3.ttl
        deleted:    test_data/tmp_store/prov/rdf/2020/05/2020-05-18T11:01:36_4.ttl
        deleted:    test_data/tmp_store/prov/rdf/2020/05/2020-05-18T11:01:36_5.ttl
        deleted:    test_data/tmp_store/prov/rdf/2020/05/2020-05-18T11:01:36_6.ttl
        deleted:    test_data/tmp_workflow/data/csv/2020/05/2020-05-18T11:01:49_1.csv
        deleted:    test_data/tmp_workflow/data/rdf/2020/05/2020-05-18T11:01:49_1.ttl
        deleted:    test_data/tmp_workflow/data/slx/2020/05/2020-05-18T11:01:49_1.scholix
        deleted:    test_data/tmp_workflow/prov/csv/2020/05/2020-05-18T11:01:49_1.csv
        deleted:    test_data/tmp_workflow/prov/rdf/2020/05/2020-05-18T11:01:49_1.ttl
        deleted:    test_data/valid_doi.csv


C:\Users\arimoretti\Documents\GitHub\index>git restore --staged test_data/tmp_load/data/csv/2020/05/2020-05-18T11:01:36_1.csv
error: invalid path 'test_data/tmp_load/data/csv/2020/05/2020-05-18T11:01:36_1.csv'
error: pathspec 'test_data/tmp_load/data/csv/2020/05/2020-05-18T11:01:36_1.csv' did not match any file(s) known to git

C:\Users\arimoretti\Documents\GitHub\index>git restore --source=HEAD :/
error: invalid path 'test_data/tmp_load/data/csv/2020/05/2020-05-18T11:01:36_1.csv'
error: invalid path 'test_data/tmp_load/data/rdf/2020/05/2020-05-18T11:01:36_1.ttl'
error: invalid path 'test_data/tmp_load/data/slx/2020/05/2020-05-18T11:01:36_1.scholix'
error: invalid path 'test_data/tmp_load/prov/csv/2020/05/2020-05-18T11:01:36_1.csv'
error: invalid path 'test_data/tmp_load/prov/rdf/2020/05/2020-05-18T11:01:36_1.ttl'
error: invalid path 'test_data/tmp_store/data/csv/2020/05/2020-05-18T11:01:36_1.csv'
error: invalid path 'test_data/tmp_store/data/csv/2020/05/2020-05-18T11:01:36_2.csv'
error: invalid path 'test_data/tmp_store/data/csv/2020/05/2020-05-18T11:01:36_3.csv'
error: invalid path 'test_data/tmp_store/data/rdf/2020/05/2020-05-18T11:01:36_1.ttl'
error: invalid path 'test_data/tmp_store/data/rdf/2020/05/2020-05-18T11:01:36_2.ttl'
error: invalid path 'test_data/tmp_store/data/rdf/2020/05/2020-05-18T11:01:36_3.ttl'
error: invalid path 'test_data/tmp_store/data/rdf/2020/05/2020-05-18T11:01:36_4.ttl'
error: invalid path 'test_data/tmp_store/data/rdf/2020/05/2020-05-18T11:01:36_5.ttl'
error: invalid path 'test_data/tmp_store/data/rdf/2020/05/2020-05-18T11:01:36_6.ttl'
error: invalid path 'test_data/tmp_store/data/slx/2020/05/2020-05-18T11:01:36_1.scholix'
error: invalid path 'test_data/tmp_store/data/slx/2020/05/2020-05-18T11:01:36_2.scholix'
error: invalid path 'test_data/tmp_store/data/slx/2020/05/2020-05-18T11:01:36_3.scholix'
error: invalid path 'test_data/tmp_store/data/slx/2020/05/2020-05-18T11:01:36_4.scholix'
error: invalid path 'test_data/tmp_store/prov/csv/2020/05/2020-05-18T11:01:36_1.csv'
error: invalid path 'test_data/tmp_store/prov/csv/2020/05/2020-05-18T11:01:36_2.csv'
error: invalid path 'test_data/tmp_store/prov/csv/2020/05/2020-05-18T11:01:36_3.csv'
error: invalid path 'test_data/tmp_store/prov/rdf/2020/05/2020-05-18T11:01:36_1.ttl'
error: invalid path 'test_data/tmp_store/prov/rdf/2020/05/2020-05-18T11:01:36_2.ttl'
error: invalid path 'test_data/tmp_store/prov/rdf/2020/05/2020-05-18T11:01:36_3.ttl'
error: invalid path 'test_data/tmp_store/prov/rdf/2020/05/2020-05-18T11:01:36_4.ttl'
error: invalid path 'test_data/tmp_store/prov/rdf/2020/05/2020-05-18T11:01:36_5.ttl'
error: invalid path 'test_data/tmp_store/prov/rdf/2020/05/2020-05-18T11:01:36_6.ttl'
error: invalid path 'test_data/tmp_workflow/data/csv/2020/05/2020-05-18T11:01:49_1.csv'
error: invalid path 'test_data/tmp_workflow/data/rdf/2020/05/2020-05-18T11:01:49_1.ttl'
error: invalid path 'test_data/tmp_workflow/data/slx/2020/05/2020-05-18T11:01:49_1.scholix'
error: invalid path 'test_data/tmp_workflow/prov/csv/2020/05/2020-05-18T11:01:49_1.csv'
error: invalid path 'test_data/tmp_workflow/prov/rdf/2020/05/2020-05-18T11:01:49_1.ttl'

C:\Users\arimoretti\Documents\GitHub\index>git restore --source=HEAD :/

        </pre></code></div>
            <h3>Structure</h3>
            <p>However, I created some folders on my pc, so to integrate them later, when I'll be able to clone the whole repository locally (an index folder containing 3 folders: noci (with nationalinstituteofhealthsource.py), test (with 10_noci.py) and test_data (with noci_dump, containing citations.csv and source.csv))</p>

            
          <h2>Self Citation</h2>
          <p> In COCI, 29,755,045 (6.7%) citations are journal self-citations, while 250,991 (0.06%)are author self-citations. In fact, the two cases in which we define a self-citation are:</p>
          <ul>
              <li>Citing and Cited articles belong to the same journal</li>
              <li>There is at least one author in common between the group of citing authors and the group of cited authors</li>
          </ul>
              
        <p>However, the second case is more difficult to handle, because of the problem of the sparsity of data about ORCID author identifers in Crossref database. </p>
          <p>Source: <a href="Heibi2019_Article_SoftwareReviewCOCITheOpenCitat.pdf">Software review: COCI, the OpenCitations Index of Crossref open DOI‑to‑DOI citations</a></p>
          <h2>Issues</h2>
          <ol>
              <li>From my notes in "Meetings": The <strong>DOI</strong> may be (and probably is) present; however, we can't rely on this + In this dataset we will probably have the <strong>"selfcitation" info</strong>. --> Since in the open_citation_collection.csv we only have citing and referenced, are those data supposed to be in icite_metadata.zip?</li>
              <li>I tried to use a couple of mapping tools, one of which should be the one we discussed in the last meeting (they basically perform the same task, however).</li>
              <ul>
                  <li>For some PMID,e.g. 4151790, I was able to find the correspondent DOI, i.e. 10.1083/jcb.61.3.688, both with <a href="https://www.pmid2cite.com/pmid-to-doi-converter">www.pmid2cite.com/pmid-to-doi-converter</a> and <a href="https://www.ncbi.nlm.nih.gov/pmc/pmctopmid/#api">www.ncbi.nlm.nih.gov/pmc/pmctopmid</a> coverter, (with this latter allowing the download of a generated file in the specified format -in my case I selected CSV*). However, I was not able to use the API service (it's the very first time I've tried to use an API service, so I really don't know where did I go wrong). I typed "service-root?ids=4145504" as required, and got back an error message. Maybe It's not necesary for me in this phase to learn how to use this tool, but I would like to understand why I can't make it work.</li>
                  <li>Is it possible/normal that pmid2cite tools performs better than NIH official tool? If so, why?</li>
              </ul>
              <li>I still don't get fully the % separator issue. I tried reading the os documentation and analysing OC correspondent files and I still don't understand (a) why we use % specifying it as separator (instead of the simple slash) and (b)why we add an "s" after it in the path name. I know I asked a similar question last time, but I was pretty sure I understood, while I'm realizing I didn't. </li>
              <li>I have some doubts concerning the general settings of the test function I should develop: It's like it stays really similar to the croci one, and that the only parts that have to be changed are called functions that are external to the test case itself. But maybe the point was just focusing on them, in order to uderstand which parts of the overall process I won't be able to reuse, since they were implementes for DOIs only. In this latter case, I'm not so far from understanding. Otherwise I'll try to study the function better and properly prepare it for next time.</li>
              <li>I didn't succeed in locally cloning the oc index from github.</li>
              <li> In the csv files I created i changed "referenced" with "cited" for uniformity purposes. Is it correct or should I keep the original label by NIH?</li>
              <li> I have some doubts about this line of code in 09_croci.py : citing, cited, creation, timespan, journal_sc, author_sc = cit . Is it like defining the labels of the columns in the csv file? i.e. : is it related to the csv format, so that the comma separator define the fields, and cit becomes che full ciation divided by fields?</li>
              <li>I don't get properly the value of the variable c in this part of the function of the class CrowdsourcedCitationSource in crowdsourcedcitationsource.py:</li>
              <div class="codebg">
                  <code>
                      <pre>
                      cited_pub_date = row.get("cited_publication_date")
                if not cited_pub_date:
                    timespan = None
                else:
                    c = Citation(None, None, created, None, cited_pub_date, None, None, None, None, "", None, None, None, None, None)
                    timespan = c.duration
                      </pre>
                  </code>
              </div>
          </ol>
          <p>*</p>
          <div class="codebg">          <code><pre>"PMID","PMCID","DOI","Version","MID","IsCurrent","IsLive","ReleaseDate","Msg"
"4151790","PMC2109316","10.1083/jcb.61.3.688","","","",1,"",""</pre></code></div>

      </div>
    </div>
  </div>
  <div class="card">
    <div class="card-header" id="headingThree">
      <h5 class="mb-0">
        <button class="btn btn-link collapsed" data-toggle="collapse" data-target="#collapseThree" aria-expanded="false" aria-controls="collapseThree">
          Week #3 (02-5-21)
        </button>
      </h5>
    </div>
    <div id="collapseThree" class="collapse" aria-labelledby="headingThree" data-parent="#accordion">
      <div class="card-body">
          <h2>Download Metadata File</h2>
          <p>sample</p>
          <div class="codebg">
              <code>
                  <pre>
                  "pmid","doi","title","authors","year","journal","is_research_article","citation_count","field_citation_rate","expected_citations_per_year","citations_per_year","relative_citation_ratio","nih_percentile","human","animal","molecular_cellular","x_coord","y_coord","apt","is_clinical","cited_by_clin","cited_by","references","provisional"
20961202,"10.1515/CCLM.2010.324","Thyroid hormones are stable even during prolonged frozen storage.","Tuija Männistö, Eila Suvanto, Heljä-Marja Surcel, Aimo Ruokonen",2010,"Clin. Chem. Lab. Med.",FALSE,7,4.22240468196911,2.40345243945313,0.7,0.29,15.3,0.75,0,0.25,-0.21650635094611,0.625,0.75,FALSE,"","32632723 30734656 31876038 26033776 22323002 27599301 30616423","17954505 20014958","No"
28140346,"10.1088/1758-5090/aa5c1c","Increased lipid accumulation and adipogenic gene expression of adipocytes in 3D bioprinted nanocellulose scaffolds.","I Henriksson, P Gatenholm, D A Hägg",2017,"Biofabrication",TRUE,17,4.05346355438924,2.23977268910794,5.66666666666667,2.53,81.8,0,0.25,0.75,-0.433012701892219,-0.5,0.05,FALSE,"","31992046 32045053 30844234 32159140 31108877 32259809 31273962 30699947 30940007 32365578 30471044 31719887 30741518 31715587 28829453 30271688 31949436","19416711 9442874 17286602 26492473 19402786 20153520 22275210 19398967 26478282 2559938 26017717 24887553 25617132 10875926 15009947 25806996 20170951 24188635 21902468 25411113 22029845 26202781 18038435 18767968 24038000 15084517 26056727 16997371 24912145 17544502 15766899 25384685 23242478 24094166 24866945 23017116 21358040 24464765 26210285 21247363 24192056 25093879 12867492 24819827 26797605 23517589","No"
20961211,"10.3109/10520295.2010.494474","Sensitivity and specificity of cytodiagnosis of body fluids in a laboratory of urgencies.","A E Rocher, F Guerra, J Rofrano, A Angeleri, O E Canessa, G R Mendeluk, L A Palaoro",2011,"Biotech Histochem",TRUE,2,3.22327656896745,1.94254995780862,0.222222222222222,0.11,5.7,1,0,0,0,1,0.05,FALSE,"","27069641 27785319","7819512 7942600 1511123 11349500 19191294 11284303 12589641 10352907 16079317 8693893 17250601 15690338 9237183 8960030 16786726 2430142 1294046 8038428","No"
20961185,"10.1515/CCLM.2011.005","Determination of daptomycin in human plasma by liquid chromatography-tandem mass spectrometry. Clinical application.","Marie-Clémence Verdier, Danièle Bentué-Ferrer, Olivier Tribut, Nicolas Collet, Matthieu Revest, Eric Bellissant",2011,"Clin. Chem. Lab. Med.",TRUE,20,2.97732333867758,1.8402221288327,2.22222222222222,1.21,57.5,0.67,0,0.33,-0.288675134594813,0.5,0.75,FALSE,"28399872 30086083","30295740 28207237 27960546 23545525 21596511 23569376 28399872 31061317 24798278 25979773 32378527 24036034 26020161 30086083 26937555 26512995 23384531 24148450 24037956 26184288","17005801 16455920 1324637 18408238 15762934 12654665 19898996 18838314 9484768 19715980 14972382 17124423 15273084 9061492 17940348 17629567 15705644 17965030 11751107 17284073 1318678 19939597 16455939 16882289 17065618 18829721 19364845 19179125 14693519","No"
25855742,"10.1128/JVI.03156-14","Interaction between TIM-1 and NPC1 Is Important for Cellular Entry of Ebola Virus.","Makoto Kuroda, Daisuke Fujikura, Asuka Nanbo, Andrea Marzi, Osamu Noyori, Masahiro Kajihara, Junki Maruyama, Keita Matsuno, Hiroko Miyamoto, Reiko Yoshida, Heinz Feldmann, Ayato Takada",2015,"J. Virol.",TRUE,29,6.88112871136308,3.49174836367486,5.8,1.66,69.1,0.17,0.5,0.33,0.144337567297406,-0.25,0.25,FALSE,"","32109150 29338048 27147086 31638994 28045014 30441759 31732320 30893855 28483690 32522856 28167539 31648236 29511990 33052961 26509109 29593155 26861827 26875443 26003830 26487564 29411209 27640102 28710692 30615471 32541946 30010949 26923960 31810353 27406444","23084921 9990080 16571833 24478428 22039362 21501828 15220407 11967302 19641222 10092649 15007635 23702199 23233528 21866101 9405687 23555248 14585362 22238307 24131711 20219911 18082433 9525641 22395071 21987776 24122154 23698310 21987779 23288419 16051304 20862315 15383160 15958209 18566430 23035224 14638512 21866103 18414680 20886108 7844558 14990712 15831716 17442946 10479477 11160735 17368819 24738640 12050398 12502850 21536871 15479853","No"
20961186,"10.1515/CCLM.2011.003","Harmonization of free thyroid hormone tests: a mission impossible?","Giorgio Iervasi, Aldo Clerico",2011,"Clin. Chem. Lab. Med.",TRUE,8,3.67191697570346,2.12920495216694,0.888888888888889,0.42,22.5,1,0,0,0,1,0.75,FALSE,"","24937843 28189596 31912542 28245185 27333057 27329994 21679128 31526200","1307685 16840583 17126310 10487939 11468222 19114271 14725937 11444163 20395622 17439341 8448866 7923779 645767 19110971 20395624 1623599 2180687 20395623 19942151 20378766 15737023 12858311 1597014 19147729 11522272 16595820 10620577 17456290","No"
20961170,"10.1162/jocn.2010.21581","Spatial attention determines the nature of nonverbal number representation.","Daniel C Hyde, Justin N Wood",2011,"J Cogn Neurosci",TRUE,36,3.78186268954311,2.17494741380959,4,1.84,72.5,1,0,0,0,1,0.5,FALSE,"24267592","23167969 24039340 23420691 27879322 26689808 24267592 23933254 28197086 21909934 32038201 24639374 24353478 22266261 26385418 30618975 33000437 32152929 29549663 23355830 25463351 27683275 30095174 29132016 31696296 26439926 28261078 24632675 30778314 26447575 22144955 25224181 27747998 24198803 27423447 25883563 22727938","3153671 15504333 18578852 9636237 10890585 16359648 11933999 19380739 18254657 17958487 16866741 18385672 12893126 17803921 12535996 18164282 11848588 15720375 6460833 16139585 19899916 17227194 18399893 16771804 9448241 1511586 11245838 10594312 7803199 21399717 15242690 17214890 16683927 21223875 11697933 17678639 23964915 8121961 17761158 15486303 9784133 18752403 16260263 12485738 11689021 10652523 11340926 11426231 12930467 23972235 6886634","No"
25855731,"10.1128/JVI.00315-15","Molecular chaperone Hsp90 is a therapeutic target for noroviruses.","Surender Vashist, Luis Urena, Mariam B Gonzalez-Hernandez, Jayoung Choi, Alexis de Rougemont, Joana Rocha-Pereira, Johan Neyts, Seungmin Hwang, Christiane E Wobus, Ian Goodfellow",2015,"J. Virol.",TRUE,22,5.68254498875044,2.98477154678445,4.4,1.47,64.9,0.14,0.43,0.43,0,-0.285714285714286,0.05,FALSE,"","30483236 28990809 31751365 29875467 29686409 27264433 28374783 27187154 32661235 32060956 26881878 30584800 31621968 26258852 29530841 27736665 28901254 29212943 28780632 27957385 26945041 28602838","20637238 18680645 22520467 22154817 21295323 21173246 24696493 24922570 21505273 17121807 19176631 19740508 18528491 22951568 20575662 24583027 23209418 22626565 22915796 6710869 21871502 20224775 20864038 22787222 16698991 10945979 19864676 21068151 15688069 15761153 16465162 20651736 24319055 10655441 20832499 22792064 20375172 11707594 20027183 21228240 20979457 16394098 18789755 22222203 22933270 12176997 23602469 24199805 22183253 21698775 22535777 15562321 8627664 17350731 25378626 19150985 24225499 21516251 21994656 23876403 24243731 24227937 12857886 20053745 21840346 18783811 22558448 22071521 24706778 19697319 19073655 22654666 22002165 20161407 24316032 9815206 22939624 12624267 12948373 23986582 15209518 17622609 9581772 21144866 21945180 16860662 16835235 24789596 22150036 17234885","No"
25855727,"10.1128/JVI.00589-15","Temperature-Sensitive Mutants in the Influenza A Virus RNA Polymerase: Alterations in the PA Linker Reduce Nuclear Targeting of the PB1-PA Dimer and Result in Viral Attenuation.","Bruno Da Costa, Alix Sausset, Sandie Munier, Alexandre Ghounaris, Nadia Naffakh, Ronan Le Goffic, Bernard Delmas",2015,"J. Virol.",TRUE,11,5.67084723962221,2.97982363408889,2.2,0.74,39.2,0.12,0.38,0.5,-0.108253175473055,-0.3125,0.05,FALSE,"","26792748 30326610 28866238 32776651 27440882 30695536 27798704 30839917 27803181 32867106 30353004","12186883 25409151 22621130 19194459 20962084 19194458 17081640 12620793 1484872 17005651 19264657 6950380 15613301 15308710 20061134 17494067 9135141 21901097 8760421 25071209 22936969 8150274 23469251 15956611 18615018 22449422 16306596 19906916 20538599 7966557 21562121 11483758 22745253 7133998 2555175 9224927 22127214 18660801 18987140 8107244 12008925 20844191 22325937 23816991 10516084 25620561 25409142","No"
20961158,"10.1021/jp1028855","Ultrafast relaxation dynamics observed through time-resolved photoelectron angular distributions.","Julien Lecointre, Gareth M Roberts, Daniel A Horke, Jan R R Verlet",2010,"J Phys Chem A",TRUE,34,4.0214198484127,2.32211017467769,3.4,1.46,64.6,0,0,1,-0.866025403784439,-0.5,0.05,FALSE,"","28451374 25301059 26980306 24955934 24204191 21682320 28948280 25686152 23968065 28471670 26030180 29461530 22463527 22614441 27367260 28937696 24092279 27792360 24734261 31663558 29560245 22755609 26274076 21947027 23881504 23642262 30221282 21971531 27809535 24006992 32233436 26263111 32499507 26286406","","No"

                  </pre>
              </code>
          </div>
          <h2>Correct the test function + restore original format of data source</h2>
          <h3>Previous modified version</h3>
          <div class="codebg">
              <code>
                  <pre>
                  citing_id,cited_id
                    2140506,2942070
                    1523579,7097569
                    1509982,6501574
                    1968312,13673087
                    2330868,3958380
                    1854174,3037997
                    2038824,2494239
                    2373284,7189714
                    3591292,4092853
                    2368927,355650
                  </pre>
              </code>
 
          </div>
          <h3>Restored original version</h3>
          <div class="codebg">
          <code>
              <pre>
                    citing,referenced
                    2140506,2942070
                    1523579,7097569
                    1509982,6501574
                    1968312,13673087
                    2330868,3958380
                    1854174,3037997
                    2038824,2494239
                    2373284,7189714
                    3591292,4092853
                    2368927,355650
                  </pre>
              </code>
          </div>
          <h2>Test Function with corrections</h2>
          <div class="codebg">
              <code>
                  <pre>
                  import unittest
from index.coci.glob import process 
from os import sep, makedirs 
from os.path import exists  
from shutil import rmtree
from index.storer.csvmanager import CSVManager 
from index.noci.nationalinstituteofhealthsource import NationalInstitutHealthSource 
from csv import DictReader


class NOCITest(unittest.TestCase):

    def setUp(self): 
        self.input_file = "index%stest_data%snoci_dump%ssource.csv" % (sep, sep, sep) 
        self.citations = "index%stest_data%snoci_dump%scitations.csv" % (sep, sep, sep) 
    def test_citation_source(self):
        ns = NationalInstitutHealthSource(self.input_file) 
        new = []
        cit = ns.get_next_citation_data()
        while cit is not None:
            citing, cited, creation, timespan, journal_sc, author_sc = cit 
            new.append({
                "citing": citing,
                "cited": cited,
                "creation": "" if creation is None else creation, 
                "timespan": "" if timespan is None else timespan,
                "journal_sc": "no" if journal_sc is None else journal_sc,
                "author_sc": "no" if author_sc is None else author_sc
            })
            cit = ns.get_next_citation_data() 
        with open(self.citations) as f: 
            old = list(DictReader(f)) 

        self.assertEqual(new, old) 
                  </pre>
              </code>
          </div>
          
          <p></p>
          <h2>Develop the main function</h2>
          <h3>nationalinstitutehealthsource.py</h3>
          <div class="codebg">
              <code>
                  <pre>
                  from index.identifier.identifiermanager import IdentifierManager
from re import sub, match

class PMIDManager(IdentifierManager):
    def __init__(self):
        self.p="pmid:"
        super(PMIDManager, self).__init__()

    def is_valid(self, id_string):
        pmid = self.normalise(id_string)
        return pmid is not None and match("^[0-9]{1,7}$", pmid) and PMIDManager.__check_digit(pmid) #Ancore ^ $ per definire inizio e fine della stringa
    #exact function of check_digit?
    def normalise(self, id_string, include_prefix=False):
        try:
            pmid_string = sub("[^X0-9]", "", id_string.upper()) #re.sub(pattern, repl, string, count=0, flags=0) -- Return the string obtained by replacing the leftmost non-overlapping occurrences of pattern in string by the replacement repl. If the pattern isn’t found, string is returned unchanged.
            return "%s%s" % (self.p if include_prefix else "", pmid_string[:])  #is it required to use the separator here?
        except:  # Any error in processing the PMID will return None
            return None

@staticmethod
    def __check_digit(pmid):
    #don't know exaclty how to proceed: I only know that the format of the pmid is from 1 to 7 digits
                  
                  </pre>
              </code>
          </div>
          
          <iframe src="https://trinket.io/embed/python/e9f4db5ca5" width="100%" height="356" frameborder="0" marginwidth="0" marginheight="0" allowfullscreen></iframe>
          
          
          <h3>pmidmanager</h3>
          <div class="codebg">
              <code>
                  <pre>
                  from index.identifier.identifiermanager import IdentifierManager
from re import sub, match

class PMIDManager(IdentifierManager):
    def __init__(self):
        self.p="pmid:"
        super(PMIDManager, self).__init__()

    def is_valid(self, id_string):
        pmid = self.normalise(id_string)
        return pmid is not None and match("^[0-9]{1,7}$", pmid) and PMIDManager.__check_digit(pmid) #Ancore ^ $ per definire inizio e fine della stringa
    #exact function of check_digit?
    def normalise(self, id_string, include_prefix=False):
        try:
            pmid_string = sub("[^X0-9]", "", id_string.upper()) #re.sub(pattern, repl, string, count=0, flags=0) -- Return the string obtained by replacing the leftmost non-overlapping occurrences of pattern in string by the replacement repl. If the pattern isn’t found, string is returned unchanged.
            return "%s%s" % (self.p if include_prefix else "", pmid_string[:])  #is it required to use the separator here?
        except:  # Any error in processing the PMID will return None
            return None

@staticmethod
    def __check_digit(pmid):
    #don't know exaclty how to proceed: I only know that the format of the pmid is from 1 to 7 digits
                  </pre>
              </code>
          </div>
          
          <iframe src="https://trinket.io/embed/python/2ec315fe53" width="100%" height="356" frameborder="0" marginwidth="0" marginheight="0" allowfullscreen></iframe>
          
          
          <h2>Locally clone and launch OC index</h2>
          <p>OC Index locally cloned + NOCI files added.</p>
          <h2>Read Mail about mapping</h2>
          <h3>Kevin Boyack</h3>
          <p>The DOI-PMID mapping is not in iCite. (FALSE --> is in metadata --> forse l'hannoe estesa nel frattempo)</p>
          <p> Boyack built a mapping file from:</p> 
          <ol>
              <li><strong>PMC Europe's DOI-PMID-PMCID (csv) downloadable mapping file</strong>. Porblems: DOIs for old papers mapped to newer PMID.</li>
              <li>DOI-PMID pairs from PubMed (prob icite usa questo, essendo in nih che è responsabile di pubmed)</li>
          </ol>
          <p> Then, he identified by logic the best DOI-PMID matches. </p>
          <ol>
              <li>pairs that appear in both sources are the most reliable.</li>
              <li>for the remaining pairs, use first the ones from PubMed (that seem to be more reliable than PMC Europe's ones; then the ones from PMC-E. </li>
          </ol>
          <p>According to Boyack, <strong>once a DOI or PMID has been used, it is excluded from being used again</strong>: if a DOI-PMID pair exists in both datasets, any remaining pairs in either source containing those DOIs or PMIDs are removed. </p>
          <p>Errori nei due dataset di partenza, ci sono occorrenze di coppie molteplici. trovando una doppia coppia, cancella tutte le altre, dovendo essere un matching uno a uno</p>
          <p>N.B.:Several hundred thousand pairs are left over, as they are not unique to DOI or PMID: these are excluded entirely because there is no way to know which is correct. </p>
          <p> Boyack's DOI-PMID file: (<a href="https://www.dropbox.com/s/rl63f1pfhym8l3r/doi_pmid.txt.gz?dl=0"> https://www.dropbox.com/s/rl63f1pfhym8l3r/doi_pmid.txt.gz?dl=0</a>) –  The third column denotes the source of the DOI-PMID pair.</p>
          
          <h2>Issues</h2>
          <ol>
              <li><p>According to Boyack, <strong>once a DOI or PMID has been used, it is excluded from being used again</strong>: if a DOI-PMID pair exists in both datasets, any remaining pairs in either source containing those DOIs or PMIDs are removed. --> I don't fully get the sense of this sentence.</li>
              <li>Issues related to pmid manager (@staticmethod def __check_digit(pmid):)</li>
              <li>pmid manager: pmid_string = sub("[^X0-9]", "", id_string.upper()) #re.sub(pattern, repl, string, count=0, flags=0) -- Return the string obtained by replacing the leftmost non-overlapping occurrences of pattern in string by the replacement repl. If the pattern isn’t found, string is returned unchanged.
            return "%s%s" % (self.p if include_prefix else "", pmid_string[:])  #is it required to use the separator here?</li>
              <li></li>
          </ol>


      </div>
    </div>
  </div>
  <div class="card">
    <div class="card-header" id="headingFour">
      <h5 class="mb-0">
        <button class="btn btn-link collapsed" data-toggle="collapse" data-target="#collapseFour" aria-expanded="false" aria-controls="collapseFour">
          Week #4 (02-11-21)
        </button>
      </h5>
    </div>
    <div id="collapseFour" class="collapse" aria-labelledby="headingFour" data-parent="#accordion">
      <div class="card-body">
          
        <h2>PMIDmanager</h2>
        <h3>Fix the second function developed last time following the DOIs example</h3>
          <div class="codebg">
              <code>
                  <pre>
from index.identifier.identifiermanager import IdentifierManager
from re import sub, match
from urllib.parse import unquote, quote #This module defines a standard interface to break Uniform Resource Locator (URL)
# strings up in components (addressing scheme, network location, path etc.), to combine the components back into a URL
# string, and to convert a “relative URL” to an absolute URL given a “base URL.”
from requests import get
from json import loads
from index.storer.csvmanager import CSVManager
from requests import ReadTimeout
from requests.exceptions import ConnectionError
from time import sleep

class PMIDManager(IdentifierManager):
    def __init__(self, valid_pmid=None, use_api_service=True):
        if valid_pmid is None:
            valid_pmid = CSVManager(store_new=False)

        self.api = "https://pubmed.ncbi.nlm.nih.gov/"
        self.valid_pmid = valid_pmid
        self.use_api_service = use_api_service
        self.p = "pmid:"
        super(PMIDManager, self).__init__()

    def set_valid(self, id_string):
        pmid = self.normalise(id_string, include_prefix=True)
        if self.valid_pmid.get_value(pmid) is None:
            self.valid_pmid.add_value(pmid, "v")

    def is_valid(self, id_string): #nothing here can self-validate the id
        pmid = self.normalise(id_string, include_prefix=True) #it calls the normaliser and checks whether the pmid is none or it doesn't match (so it returns false)
        if pmid is None or match("^[1-9]\d*$", pmid) is None: #anchors (^ $) to define start and stop of the string;  "is none" --> it isn't present in my database (Boolean answer)
            return False
        else: #after having checked the local database handling the class (in the case we already got the valid/invalid information for this id
            if self.valid_pmid.get_value(pmid) is None: #In the case it returns None, it means we don't have it yet, so we have to understand if it is valid or not.
                if self.__pmid_exists(pmid): #pmid exist (API call)
                    self.valid_pmid.add_value(pmid, "v")
                else:
                    self.valid_pmid.add_value(pmid, "i")
            return "v" in self.valid_pmid.get_value(pmid)



    def normalise(self, id_string, include_prefix=False):
        try:
            pmid_string = sub("\0+", "", (sub("[^\d+]", "", id_string))) #is it correct?
            return "%s%s" % (self.p if include_prefix else "", pmid_string.strip()[:]) #[:] --> è necessary?
        except:  # Any error in processing the PMID will return None --> when the id can't be recognised as a pubmed
            return None


    def __pmid_exists(self, pmid_full):  # I try and use the api to check the existence, otherwise it returns "none" or False
        pmid = self.normalise(pmid_full)
        if self.use_api_service:
            tentative = 3
            while tentative:
                tentative -= 1
                try:
                    r = get(self.api + quote(pmid) + "/?format=pmid", headers=self.headers, timeout=30) #/?format=pmid --> to avoid articles research
                    #urllib.parse.quote(string, safe='/', encoding=None, errors=None)
                    # Replace special characters in string using the %xx escape. Letters, digits, and the characters '_.-~'
                    # are never quoted. By default, this function is intended for quoting the path section of a URL.
                    if r.status_code == 200: #the request was successful
                        r.encoding = "utf-8"
                        json_res = loads(r.text)
                        return json_res.get("responseCode") == 1
                except ReadTimeout:
                    pass  # Do nothing, just try again
                except ConnectionError:
                    sleep(5)  # Sleep 5 seconds, then try again

        return False
                  </pre>
              </code>
          </div>
          
          <iframe src="https://trinket.io/embed/python/93720538c0" width="100%" height="356" frameborder="0" marginwidth="0" marginheight="0" allowfullscreen></iframe>
        <h4>Notes</h4>
          <ul>
              <li><strong>https://pubmed.ncbi.nlm.nih.gov/</strong>: is the correct url for the request?</li>
              <li>r = get(self.api + quote(pmid) + <strong>"/?format=pmid"</strong>, headers=self.headers, timeout=30) #/?format=pmid --> to avoid articles research</li>
          </ul>
          
          <h2>API Requests</h2>
          <p><strong>API Requests: Get</strong>When we make a request, the response from the API comes with a response code which tells us whether our request was successful. Response codes are important because they immediately tell us if something went wrong. To make a ‘GET’ request, we’ll use the requests.get() function, which requires one argument — the URL we want to make the request to.</p>
          <p>The get() function returns a response object. We can use the response.status_code attribute to receive the status code for our request.</p>
          <h3>API Status Codes</h3>
          <ol>
              <li>200: Everything went okay, and the result has been returned (if any).</li>
              <li>301: The server is redirecting you to a different endpoint. This can happen when a company switches domain names, or an endpoint name is changed.</li>
              <li>400: The server thinks you made a bad request. This can happen when you don’t send along the right data, among other things.</li>
              <li>401: The server thinks you’re not authenticated. Many APIs require login ccredentials, so this happens when you don’t send the right credentials to access an API.</li>
              <li>403: The resource you’re trying to access is forbidden: you don’t have the right permissions to see it.</li>
              <li>404: The resource you tried to access wasn’t found on the server.</li>
              <li>503: The server is not ready to handle the request.</li>
          </ol>

          
        <h2>PMIDmanager Test Case (02_identifiermanager.py)</h2>
        <h3>Develop a test case for this latter function</h3>

          <div>
              <code>
                  <pre>               
import unittest
from os import sep
from index.identifier.doimanager import DOIManager
from index.identifier.issnmanager import ISSNManager
from index.identifier.orcidmanager import ORCIDManager
#pmid extension
from index.identifier.pmidmanager import PMIDManager
from index.storer.csvmanager import CSVManager


class IdentifierManagerTest(unittest.TestCase):
    """This class aim at testing the methods of the class CSVManager."""

    def setUp(self):
        self.valid_doi_1 = "10.1108/jd-12-2013-0166"
        self.valid_doi_2 = "10.1130/2015.2513(00)"
        self.invalid_doi_1 = "10.1108/12-2013-0166"
        self.invalid_doi_2 = "10.1371"
        self.valid_doi_path = "index%stest_data%svalid_doi.csv" % (sep, sep)

        self.valid_issn_1 = "2376-5992"
        self.valid_issn_2 = "1474-175X"
        self.invalid_issn_1 = "2376-599C"
        self.invalid_issn_2 = "2376-5995"
        self.invalid_issn_3 = "2376-599"

        self.valid_orcid_1 = "0000-0003-0530-4305"
        self.valid_orcid_2 = "0000-0001-5506-523X"
        self.invalid_orcid_1 = "0000-0003-0530-430C"
        self.invalid_orcid_2 = "0000-0001-5506-5232"
        self.invalid_orcid_3 = "0000-0001-5506-523"
        self.invalid_orcid_4 = "1-5506-5232"

#class extension for pubmedid
        self.valid_pmid_1 = "2942070"
        self.valid_pmid_2 = "1509982"
        self.valid_pmid_3 = "7189714"
        self.invalid_pmid_1 = "00673087"
        self.invalid_pmid_2 = "pmid:174"
        self.invalid_pmid_3 = "0000092"
        self.valid_pmid_path = "index%stest_data%svalid_pmid.csv" % (sep, sep)


    def test_doi_normalise(self):
        dm = DOIManager()
        self.assertEqual(self.valid_doi_1, dm.normalise(self.valid_doi_1.upper().replace("10.", "doi: 10. ")))
        self.assertEqual(self.valid_doi_1, dm.normalise(self.valid_doi_1.upper().replace("10.", "doi:10.")))
        self.assertEqual(self.valid_doi_1, dm.normalise(self.valid_doi_1.upper().replace("10.", "https://doi.org/10.")))

    def test_doi_is_valid(self):
        dm_nofile = DOIManager()
        self.assertTrue(dm_nofile.is_valid(self.valid_doi_1))
        self.assertTrue(dm_nofile.is_valid(self.valid_doi_2))
        self.assertFalse(dm_nofile.is_valid(self.invalid_doi_1))
        self.assertFalse(dm_nofile.is_valid(self.invalid_doi_2))

        valid_doi = CSVManager(self.valid_doi_path)
        dm_file = DOIManager(valid_doi=valid_doi, use_api_service=False)
        self.assertTrue(dm_file.is_valid(self.valid_doi_1))
        self.assertFalse(dm_file.is_valid(self.invalid_doi_1))

        dm_nofile_noapi = DOIManager(use_api_service=False)
        self.assertFalse(dm_nofile_noapi.is_valid(self.valid_doi_1))
        self.assertFalse(dm_nofile_noapi.is_valid(self.invalid_doi_1))

    def test_issn_normalise(self):
        im = ISSNManager()
        self.assertEqual(self.valid_issn_1, im.normalise(self.valid_issn_1.replace("-", "  ")))
        self.assertEqual(self.valid_issn_2, im.normalise(self.valid_issn_2.replace("-", "  ")))
        self.assertEqual(self.invalid_issn_3, im.normalise(self.invalid_issn_3.replace("-", "  ")))

    def test_issn_is_valid(self):
        im = ISSNManager()
        self.assertTrue(im.is_valid(self.valid_issn_1))
        self.assertTrue(im.is_valid(self.valid_issn_2))
        self.assertFalse(im.is_valid(self.invalid_issn_1))
        self.assertFalse(im.is_valid(self.invalid_issn_2))
        self.assertFalse(im.is_valid(self.invalid_issn_3))

    def test_orcid_normalise(self):
        om = ORCIDManager()
        self.assertEqual(self.valid_orcid_1, om.normalise(self.valid_orcid_1.replace("-", "  ")))
        self.assertEqual(self.valid_orcid_1, om.normalise("https://orcid.org/" + self.valid_orcid_1))
        self.assertEqual(self.valid_orcid_2, om.normalise(self.valid_orcid_2.replace("-", "  ")))
        self.assertEqual(self.invalid_orcid_3, om.normalise(self.invalid_orcid_3.replace("-", "  ")))

    def test_orcid_is_valid(self):
        om = ORCIDManager()
        self.assertTrue(om.is_valid(self.valid_orcid_1))
        self.assertTrue(om.is_valid(self.valid_orcid_2))
        self.assertFalse(om.is_valid(self.invalid_orcid_1))
        self.assertFalse(om.is_valid(self.invalid_orcid_2))
        self.assertFalse(om.is_valid(self.invalid_orcid_3))
        self.assertFalse(om.is_valid(self.invalid_orcid_4))


#class extension for pubmedid
    def test_pmid_normalise(self):
        pm = PMIDManager()
        self.assertEqual(self.valid_pmid_1, pm.normalise(self.valid_pmid_1.replace("", "pmid:"))) #check other characters exclusion
        self.assertEqual(self.valid_pmid_1, pm.normalise(self.valid_pmid_1.replace("", " "))) #check other characters (spaces) exclusion
        self.assertEqual(self.valid_pmid_1, pm.normalise("https://pubmed.ncbi.nlm.nih.gov/"+self.valid_pmid_1)) #check other characters (url) exclusion
        self.assertEqual(self.valid_pmid_2, pm.normalise("000"+self.valid_pmid_2)) #check initial 0s exclusion

    def test_pmid_is_valid(self):
        pm_nofile = PMIDManager()
        self.assertTrue(pm_nofile.is_valid(self.valid_pmid_1))
        self.assertTrue(pm_nofile.is_valid(self.valid_pmid_2))
        self.assertTrue(pm_nofile.is_valid(self.valid_pmid_3))
        self.assertFalse(pm_nofile.is_valid(self.invalid_pmid_1))
        self.assertFalse(pm_nofile.is_valid(self.invalid_pmid_2))
        self.assertFalse(pm_nofile.is_valid(self.invalid_pmid_3))

        valid_pmid = CSVManager(self.valid_pmid_path)
        pm_file = PMIDManager(valid_pmid=valid_pmid, use_api_service=False)
        self.assertTrue(pm_file.is_valid(self.valid_pmid_1))
        self.assertFalse(pm_file.is_valid(self.invalid_pmid_1))

        pm_nofile_noapi = PMIDManager(use_api_service=False)
        self.assertFalse(pm_nofile_noapi.is_valid(self.valid_pmid_1)) #Why not assert True?
        self.assertFalse(pm_nofile_noapi.is_valid(self.invalid_pmid_1))
                  </pre>
              </code>
          </div>
        <iframe src="https://trinket.io/embed/python/a7977c33ce" width="100%" height="356" frameborder="0" marginwidth="0" marginheight="0" allowfullscreen></iframe>
        <h4>Notes</h4>
          <ol>
        pm_nofile_noapi = PMIDManager(use_api_service=False) <br>self.assertFalse(pm_nofile_noapi.is_valid(self.valid_pmid_1)) #Why not assert True?<br>
        self.assertFalse(pm_nofile_noapi.is_valid(self.invalid_pmid_1))
          </ol>

        <h2>Run Tests</h2>
        <h3>Check the correct functioning of the developed functions using their test cases</h3>
        <p></p>
        <h2>Overall Process</h2>
        <h3>Where do we need additional code to manage pubmedIDs?</h3>
        <p></p>
        <h2>Issues</h2>
          <p>See "Notes"</p>
          <ul>
              <li>I definitely have problems in running tests, not only mine but tests in general. </li>
              <ul>
                  <li>When I try to run croci test, for example, I get this message:  C:\Users\arimoretti\AppData\Local\Programs\Python\Python38-32\python.exe C:/Users/arimoretti/Documents/GitHub/index/test/09_croci.py ----
Traceback (most recent call last): ---
  File "C:/Users/arimoretti/Documents/GitHub/index/test/09_croci.py", line 18, in module ----
    from index.coci.glob import process ----
ModuleNotFoundError: No module named 'index'---- 

Process finished with exit code 1 </li>
              </ul>
          </ul>
<p>Running Noci test:</p>
          <div class="codebg">
              <code>
                  <pre>
                  Testing started at 08:01 ...
C:\Users\arimoretti\AppData\Local\Programs\Python\Python38-32\python.exe "C:\Program Files\JetBrains\PyCharm Community Edition 2019.2.3\helpers\pycharm\_jb_unittest_runner.py" --path C:/Users/arimoretti/Documents/GitHub/index/test/10_noci.py
Launching unittests with arguments python -m unittest C:/Users/arimoretti/Documents/GitHub/index/test/10_noci.py in C:\Users\arimoretti\Documents\GitHub\index\test



Ran 1 test in 0.097s

FAILED (errors=1)

Error
Traceback (most recent call last):
  File "C:\Users\arimoretti\AppData\Local\Programs\Python\Python38-32\lib\unittest\case.py", line 60, in testPartExecutor
    yield
  File "C:\Users\arimoretti\AppData\Local\Programs\Python\Python38-32\lib\unittest\case.py", line 676, in run
    self._callTestMethod(testMethod)
  File "C:\Users\arimoretti\AppData\Local\Programs\Python\Python38-32\lib\unittest\case.py", line 633, in _callTestMethod
    method()
  File "C:\Users\arimoretti\AppData\Local\Programs\Python\Python38-32\lib\unittest\loader.py", line 34, in testFailure
    raise self._exception
ImportError: Failed to import test module: 10_noci
Traceback (most recent call last):
  File "C:\Users\arimoretti\AppData\Local\Programs\Python\Python38-32\lib\unittest\loader.py", line 154, in loadTestsFromName
    module = __import__(module_name)
  File "C:\Users\arimoretti\Documents\GitHub\index\test\10_noci.py", line 2, in <module>
    from index.coci.glob import process #something to be developed to perform the same task but with pmids
ModuleNotFoundError: No module named 'index'



Assertion failed

Assertion failed

Process finished with exit code 1

Assertion failed

Assertion failed

                  </pre>
              </code>
          </div>
          
      </div>
    </div>
  </div>
  <div class="card">
    <div class="card-header" id="headingFive">
      <h5 class="mb-0">
        <button class="btn btn-link collapsed" data-toggle="collapse" data-target="#collapseFive" aria-expanded="false" aria-controls="collapseFive">
          Week #5 (02-18-21)
        </button>
      </h5>
    </div>
    <div id="collapseFive" class="collapse" aria-labelledby="headingFive" data-parent="#accordion">
      <div class="card-body">
          <h2>Run all tests and correct the functions</h2>
          <h3>How to run tests</h3>
          <img src="img/LANCIARETERMINAL.PNG" class="myimg">
          <p>All tests'launch</p>
          <div class="codebg">
              <code>
                  <pre>

C:\Users\arimoretti\Documents\GitHub>python -m unittest index.test.02_resourcefinder
E
======================================================================
ERROR: 02_resourcefinder (unittest.loader._FailedTest)
----------------------------------------------------------------------
ImportError: Failed to import test module: 02_resourcefinder
Traceback (most recent call last):
  File "C:\Users\arimoretti\AppData\Local\Programs\Python\Python38-32\lib\unittest\loader.py", line 154, in loadTestsFromName
    module = __import__(module_name)
ModuleNotFoundError: No module named 'index.test.02_resourcefinder'


----------------------------------------------------------------------
Ran 1 test in 0.000s

FAILED (errors=1)

C:\Users\arimoretti\Documents\GitHub>python -m unittest index.test.02_identifiermanager
......FF
======================================================================
FAIL: test_pmid_is_valid (index.test.02_identifiermanager.IdentifierManagerTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\Users\arimoretti\Documents\GitHub\index\test\02_identifiermanager.py", line 126, in test_pmid_is_valid
    self.assertTrue(pm_nofile.is_valid(self.valid_pmid_1))
AssertionError: False is not true

======================================================================
FAIL: test_pmid_normalise (index.test.02_identifiermanager.IdentifierManagerTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\Users\arimoretti\Documents\GitHub\index\test\02_identifiermanager.py", line 122, in test_pmid_normalise
    self.assertEqual(self.valid_pmid_2, pm.normalise("000"+self.valid_pmid_2)) #check initial 0s exclusion
AssertionError: '1509982' != '0001509982'
- 1509982
+ 0001509982
? +++


----------------------------------------------------------------------
Ran 8 tests in 1.348s

FAILED (failures=2)

C:\Users\arimoretti\Documents\GitHub>python -m unittest index.test.03_resourcefinder
...E......
======================================================================
ERROR: test_datacite_get_issn (index.test.03_resourcefinder.ResourceFinderTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\Users\arimoretti\Documents\GitHub\index\test\03_resourcefinder.py", line 88, in test_datacite_get_issn
    self.assertIn("2197-6775", df_1.get_container_issn("10.14763/2019.1.1389"))
  File "C:\Users\arimoretti\AppData\Local\Programs\Python\Python38-32\lib\unittest\case.py", line 1176, in assertIn
    if member not in container:
TypeError: argument of type 'NoneType' is not iterable

----------------------------------------------------------------------
Ran 10 tests in 19.386s

FAILED (errors=1)

C:\Users\arimoretti\Documents\GitHub>python -m unittest index.test.04_oci
.......E
======================================================================
ERROR: test_lookup (index.test.04_oci.CitationTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\Users\arimoretti\Documents\GitHub\index\test\04_oci.py", line 286, in test_lookup
    oci_man = OCIManager(lookup_file="index%stest_data%slookup_full.csv" % (sep, sep))
  File "C:\Users\arimoretti\Documents\GitHub\index\citation\oci.py", line 553, in __init__
    for row in lookupcsv_reader:
  File "C:\Users\arimoretti\AppData\Local\Programs\Python\Python38-32\lib\csv.py", line 110, in __next__
    self.fieldnames
  File "C:\Users\arimoretti\AppData\Local\Programs\Python\Python38-32\lib\csv.py", line 97, in fieldnames
    self._fieldnames = next(self.reader)
  File "C:\Users\arimoretti\AppData\Local\Programs\Python\Python38-32\lib\encodings\cp1252.py", line 23, in decode
    return codecs.charmap_decode(input,self.errors,decoding_table)[0]
UnicodeDecodeError: 'charmap' codec can't decode byte 0x90 in position 905: character maps to <undefined>

----------------------------------------------------------------------
Ran 8 tests in 0.066s

FAILED (errors=1)

C:\Users\arimoretti\Documents\GitHub>python -m unittest index.test.05_citationstorer
EEEE
======================================================================
ERROR: test_load_citations_csv (index.test.05_citationstorer.CitationStorerTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\Users\arimoretti\Documents\GitHub\index\test\05_citationstorer.py", line 75, in test_load_citations_csv
    origin_citation_list, stored_citation_list = self.load_and_store_citations(
  File "C:\Users\arimoretti\Documents\GitHub\index\test\05_citationstorer.py", line 67, in load_and_store_citations
    cs.store_citation(citation)
  File "C:\Users\arimoretti\Documents\GitHub\index\storer\citationstorer.py", line 291, in store_citation
    CitationStorer.__store_csv_on_file(data_csv_f_path, Citation.header_citation_data,
  File "C:\Users\arimoretti\Documents\GitHub\index\storer\citationstorer.py", line 148, in __store_csv_on_file
    with open(f_path, "a") as f:
OSError: [Errno 22] Invalid argument: 'index\\test_data\\tmp_load\\data\\csv\\2021\\02\\2021-02-24T22:58:32_1.csv'

======================================================================
ERROR: test_load_citations_rdf (index.test.05_citationstorer.CitationStorerTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\Users\arimoretti\Documents\GitHub\index\test\05_citationstorer.py", line 87, in test_load_citations_rdf
    origin_citation_list, stored_citation_list = self.load_and_store_citations(
  File "C:\Users\arimoretti\Documents\GitHub\index\test\05_citationstorer.py", line 67, in load_and_store_citations
    cs.store_citation(citation)
  File "C:\Users\arimoretti\Documents\GitHub\index\storer\citationstorer.py", line 291, in store_citation
    CitationStorer.__store_csv_on_file(data_csv_f_path, Citation.header_citation_data,
  File "C:\Users\arimoretti\Documents\GitHub\index\storer\citationstorer.py", line 148, in __store_csv_on_file
    with open(f_path, "a") as f:
OSError: [Errno 22] Invalid argument: 'index\\test_data\\tmp_load\\data\\csv\\2021\\02\\2021-02-24T22:58:32_1.csv'

======================================================================
ERROR: test_load_citations_slx (index.test.05_citationstorer.CitationStorerTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\Users\arimoretti\Documents\GitHub\index\test\05_citationstorer.py", line 107, in test_load_citations_slx
    origin_citation_list, stored_citation_list = self.load_and_store_citations(
  File "C:\Users\arimoretti\Documents\GitHub\index\test\05_citationstorer.py", line 67, in load_and_store_citations
    cs.store_citation(citation)
  File "C:\Users\arimoretti\Documents\GitHub\index\storer\citationstorer.py", line 291, in store_citation
    CitationStorer.__store_csv_on_file(data_csv_f_path, Citation.header_citation_data,
  File "C:\Users\arimoretti\Documents\GitHub\index\storer\citationstorer.py", line 148, in __store_csv_on_file
    with open(f_path, "a") as f:
OSError: [Errno 22] Invalid argument: 'index\\test_data\\tmp_load\\data\\csv\\2021\\02\\2021-02-24T22:58:32_1.csv'

======================================================================
ERROR: test_store_citation (index.test.05_citationstorer.CitationStorerTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\Users\arimoretti\Documents\GitHub\index\test\05_citationstorer.py", line 133, in test_store_citation
    cs.store_citation(citation)
  File "C:\Users\arimoretti\Documents\GitHub\index\storer\citationstorer.py", line 291, in store_citation
    CitationStorer.__store_csv_on_file(data_csv_f_path, Citation.header_citation_data,
  File "C:\Users\arimoretti\Documents\GitHub\index\storer\citationstorer.py", line 148, in __store_csv_on_file
    with open(f_path, "a") as f:
OSError: [Errno 22] Invalid argument: 'index\\test_data\\tmp_store\\data\\csv\\2021\\02\\2021-02-24T22:58:32_1.csv'

----------------------------------------------------------------------
Ran 4 tests in 0.098s

FAILED (errors=4)

C:\Users\arimoretti\Documents\GitHub>python -m unittest index.test.06_citationsource
E
======================================================================
ERROR: test_get_next_citation_data (index.test.06_citationsource.CitationSourceTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\Users\arimoretti\Documents\GitHub\index\test\06_citationsource.py", line 33, in setUp
    self.oci = OCIManager(lookup_file="index%stest_data%slookup_full.csv" % (sep, sep))
  File "C:\Users\arimoretti\Documents\GitHub\index\citation\oci.py", line 553, in __init__
    for row in lookupcsv_reader:
  File "C:\Users\arimoretti\AppData\Local\Programs\Python\Python38-32\lib\csv.py", line 110, in __next__
    self.fieldnames
  File "C:\Users\arimoretti\AppData\Local\Programs\Python\Python38-32\lib\csv.py", line 97, in fieldnames
    self._fieldnames = next(self.reader)
  File "C:\Users\arimoretti\AppData\Local\Programs\Python\Python38-32\lib\encodings\cp1252.py", line 23, in decode
    return codecs.charmap_decode(input,self.errors,decoding_table)[0]
UnicodeDecodeError: 'charmap' codec can't decode byte 0x90 in position 905: character maps to <undefined>

----------------------------------------------------------------------
Ran 1 test in 0.014s

FAILED (errors=1)

C:\Users\arimoretti\Documents\GitHub>python -m unittest index.test.07_cnc
E
======================================================================
ERROR: test_execute_workflow (index.test.07_cnc.CreateNewCitationsTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\Users\arimoretti\Documents\GitHub\index\test\07_cnc.py", line 79, in test_execute_workflow
    execute_workflow(self.idbaseurl, self.baseurl, self.python, self.pclass, self.input, self.doi_file,
  File "C:\Users\arimoretti\Documents\GitHub\index\cnc.py", line 69, in execute_workflow
    return extract_citations(idbaseurl, baseurl, python, pclass, input, lookup, data, prefix,
  File "C:\Users\arimoretti\Documents\GitHub\index\cnc.py", line 78, in extract_citations
    oci_manager = OCIManager(lookup_file=lookup)
  File "C:\Users\arimoretti\Documents\GitHub\index\citation\oci.py", line 553, in __init__
    for row in lookupcsv_reader:
  File "C:\Users\arimoretti\AppData\Local\Programs\Python\Python38-32\lib\csv.py", line 110, in __next__
    self.fieldnames
  File "C:\Users\arimoretti\AppData\Local\Programs\Python\Python38-32\lib\csv.py", line 97, in fieldnames
    self._fieldnames = next(self.reader)
  File "C:\Users\arimoretti\AppData\Local\Programs\Python\Python38-32\lib\encodings\cp1252.py", line 23, in decode
    return codecs.charmap_decode(input,self.errors,decoding_table)[0]
UnicodeDecodeError: 'charmap' codec can't decode byte 0x90 in position 905: character maps to <undefined>

----------------------------------------------------------------------
Ran 1 test in 0.031s

FAILED (errors=1)

C:\Users\arimoretti\Documents\GitHub>python -m unittest index.test.09_coci
E
======================================================================
ERROR: 09_coci (unittest.loader._FailedTest)
----------------------------------------------------------------------
ImportError: Failed to import test module: 09_coci
Traceback (most recent call last):
  File "C:\Users\arimoretti\AppData\Local\Programs\Python\Python38-32\lib\unittest\loader.py", line 154, in loadTestsFromName
    module = __import__(module_name)
ModuleNotFoundError: No module named 'index.test.09_coci'


----------------------------------------------------------------------
Ran 1 test in 0.000s

FAILED (errors=1)

C:\Users\arimoretti\Documents\GitHub>python -m unittest index.test.08_coci
.

# Add valid DOIs from Crossref metadata
Open file 1 of 2
Open file 2 of 2


# Check cited DOIs from Crossref reference field
Open file 1 of 2
Open file 2 of 2
.
----------------------------------------------------------------------
Ran 2 tests in 11.409s

OK

C:\Users\arimoretti\Documents\GitHub>python -m unittest index.test.09_croci
.
----------------------------------------------------------------------
Ran 1 test in 0.019s

OK

C:\Users\arimoretti\Documents\GitHub>python -m unittest index.test.10_noci
Traceback (most recent call last):
  File "C:\Users\arimoretti\AppData\Local\Programs\Python\Python38-32\lib\runpy.py", line 192, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Users\arimoretti\AppData\Local\Programs\Python\Python38-32\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\Users\arimoretti\AppData\Local\Programs\Python\Python38-32\lib\unittest\__main__.py", line 18, in <module>
    main(module=None)
  File "C:\Users\arimoretti\AppData\Local\Programs\Python\Python38-32\lib\unittest\main.py", line 100, in __init__
    self.parseArgs(argv)
  File "C:\Users\arimoretti\AppData\Local\Programs\Python\Python38-32\lib\unittest\main.py", line 147, in parseArgs
    self.createTests()
  File "C:\Users\arimoretti\AppData\Local\Programs\Python\Python38-32\lib\unittest\main.py", line 158, in createTests
    self.test = self.testLoader.loadTestsFromNames(self.testNames,
  File "C:\Users\arimoretti\AppData\Local\Programs\Python\Python38-32\lib\unittest\loader.py", line 220, in loadTestsFromNames
    suites = [self.loadTestsFromName(name, module) for name in names]
  File "C:\Users\arimoretti\AppData\Local\Programs\Python\Python38-32\lib\unittest\loader.py", line 220, in <listcomp>
    suites = [self.loadTestsFromName(name, module) for name in names]
  File "C:\Users\arimoretti\AppData\Local\Programs\Python\Python38-32\lib\unittest\loader.py", line 154, in loadTestsFromName
    module = __import__(module_name)
  File "C:\Users\arimoretti\Documents\GitHub\index\test\10_noci.py", line 7, in <module>
    from index.noci.nationalinstituteofhealthsource import NationalInstituteHealthSource #crowdsourcedcitationsource sostituito
  File "C:\Users\arimoretti\Documents\GitHub\index\noci\nationalinstituteofhealthsource.py", line 30
    else:
        ^
IndentationError: unindent does not match any outer indentation level

C:\Users\arimoretti\Documents\GitHub>python -m unittest index.test.test
Traceback (most recent call last):
  File "C:\Users\arimoretti\AppData\Local\Programs\Python\Python38-32\lib\runpy.py", line 192, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Users\arimoretti\AppData\Local\Programs\Python\Python38-32\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\Users\arimoretti\AppData\Local\Programs\Python\Python38-32\lib\unittest\__main__.py", line 18, in <module>
    main(module=None)
  File "C:\Users\arimoretti\AppData\Local\Programs\Python\Python38-32\lib\unittest\main.py", line 100, in __init__
    self.parseArgs(argv)
  File "C:\Users\arimoretti\AppData\Local\Programs\Python\Python38-32\lib\unittest\main.py", line 147, in parseArgs
    self.createTests()
  File "C:\Users\arimoretti\AppData\Local\Programs\Python\Python38-32\lib\unittest\main.py", line 158, in createTests
    self.test = self.testLoader.loadTestsFromNames(self.testNames,
  File "C:\Users\arimoretti\AppData\Local\Programs\Python\Python38-32\lib\unittest\loader.py", line 220, in loadTestsFromNames
    suites = [self.loadTestsFromName(name, module) for name in names]
  File "C:\Users\arimoretti\AppData\Local\Programs\Python\Python38-32\lib\unittest\loader.py", line 220, in <listcomp>
    suites = [self.loadTestsFromName(name, module) for name in names]
  File "C:\Users\arimoretti\AppData\Local\Programs\Python\Python38-32\lib\unittest\loader.py", line 154, in loadTestsFromName
    module = __import__(module_name)
  File "C:\Users\arimoretti\Documents\GitHub\index\test\test.py", line 15, in <module>
    with open("/srv/data/glob/id_date.csv") as f:
FileNotFoundError: [Errno 2] No such file or directory: '/srv/data/glob/id_date.csv'

                  </pre>
              </code>
          </div>
          
          <h3>Functions' corrections</h3>
          <h4>identifier.pmidmanager</h4>
          <div class="codebg">
            <code>
                <pre>
from index.identifier.identifiermanager import IdentifierManager
from re import sub, match
from urllib.parse import unquote, quote #This module defines a standard interface to break Uniform Resource Locator (URL)
# strings up in components (addressing scheme, network location, path etc.), to combine the components back into a URL
# string, and to convert a “relative URL” to an absolute URL given a “base URL.”
from requests import get
from json import loads
from index.storer.csvmanager import CSVManager
from requests import ReadTimeout
from requests.exceptions import ConnectionError
from time import sleep

class PMIDManager(IdentifierManager):
    def __init__(self, valid_pmid=None, use_api_service=True):
        if valid_pmid is None:
            valid_pmid = CSVManager(store_new=False) #crea csv come struttura locale senza salvare nulla sul file

        self.api = "https://pubmed.ncbi.nlm.nih.gov/"
        self.valid_pmid = valid_pmid
        self.use_api_service = use_api_service
        self.p = "pmid:"
        super(PMIDManager, self).__init__()

    def set_valid(self, id_string):
        pmid = self.normalise(id_string, include_prefix=True)

        if self.valid_pmid.get_value(pmid) is None:
            self.valid_pmid.add_value(pmid, "v")

    def is_valid(self, id_string): #non ho nessuna cosa che dovrebbe autovalidare l'identificativo
        pmid = self.normalise(id_string, include_prefix=True) #it calls the normaliser and checks whether the pmid is none or it doesn't match (so it returns false)
        if pmid is None or match("^[1-9]\d*$", pmid) is None: #anchors (^ $) to define start and stop of the string;  "is none" --> it isn't present in my database (Boolean answer)
            return False
        else: #after having checked the local database handling the class (in the case we already got the valid/invalid information for this id
            if self.valid_pmid.get_value(pmid) is None: #In the case it returns None, it means we don't have it yet, so we have to understand if it is valid or not.
                if self.__pmid_exists(pmid): #pmid exist (API call) --> non avendo informazione di api
                    self.valid_pmid.add_value(pmid, "v")
                else:
                    self.valid_pmid.add_value(pmid, "i") #aggiunge al dizionario di invalid. se è anche esistente ma non c'è possibilità di verificarlo è false
            return "v" in self.valid_pmid.get_value(pmid)



    def normalise(self, id_string, include_prefix=False):
        try:
            pmid_string = sub("^0+", "", sub("\0+", "", (sub("[^\d+]", "", id_string)))) #tolto lettere, null, tolgo ^0+  --> tutto qullo che inizia con uno o più zero, non metto $ perché riguarda fine stringa
            return "%s%s" % (self.p if include_prefix else "", pmid_string)
        except:  # Any error in processing the PMID will return None --> when the id can't be recognised as a pubmed
            return None


    def __pmid_exists(self, pmid_full):  # I try and use the api to check the existence, otherwise it returns "none" or False
        pmid = self.normalise(pmid_full) #normalise per sicurezza
        if self.use_api_service:
            tentative = 3
            while tentative:
                tentative -= 1
                try:
                    r = get(self.api + quote(pmid) + "/?format=pmid", headers=self.headers, timeout=30) #/?format=pmid --> to avoid articles research
                    #urllib.parse.quote(string, safe='/', encoding=None, errors=None)
                    # Replace special characters in string using the %xx escape. Letters, digits, and the characters '_.-~'
                    # are never quoted. By default, this function is intended for quoting the path section of a URL.

#dato che mi restituisce l'id devo assicurarmi che quello che mi restituisce lui sia uguale a quello che ho passato io (700000 =! 70) ---> deve superare status code + quello che restituisce uguale a input
                    #funziona ma restituisce sempre un'html file
                    if r == pmid and r.status_code == 200: #id identificato è valido. 404 -> errore, corretto r=pmid? ovvero, r è risultato della ricerca?
                        r.encoding = "utf-8"
                        json_res = loads(r.text)
                        return json_res.get("responseCode") == 1
                except ReadTimeout:
                    pass  # Do nothing, just try again
                except ConnectionError:
                    sleep(5)  # Sleep 5 seconds, then try again

        return False
                </pre>
            </code>
          </div>
          
          <iframe src="https://trinket.io/embed/python/3f7dc20c98" width="100%" height="356" frameborder="0" marginwidth="0" marginheight="0" allowfullscreen></iframe>
          
          
          <p>Notes:</p>
          <ul>
            <li>Is it correct to check that input == output like this --> r == pmid?</li>
            <li>Is it correct this normalisation: pmid_string = sub("^0+", "", sub("\0+", "", (sub("[^\d+]", "", id_string)))) ?</li>
          </ul>
          
          <h4>noci.nationalinstituteofhealthsource</h4>
          <div class="codebg">
            <code>
                <pre>
                from os import walk, sep, remove
from os.path import isdir
from json import load
from csv import DictWriter
from index.citation.citationsource import CSVFileCitationSource
from index.identifier.pmidmanager import PMIDManager
from index.citation.oci import Citation


class NationalInstituteHealthSource(CSVFileCitationSource):
    def __init__(self, src, local_name=""):
        self.pmid = PMIDManager()
        super(NationalInstituteHealthSource, self).__init__(src, local_name)

    def get_next_citation_data(self):
        row = self._get_next_in_file()

        while row is not None:
            citing = self.pmid.normalise(row.get("citing")) #substituted with NIH labels
            cited = self.pmid.normalise(row.get("referenced"))

            if citing is not None and cited is not None:
                #created = row.get("citing_publication_date")
                # if not created:
                    created = None

                #cited_pub_date = row.get("cited_publication_date")
                #if not cited_pub_date:
                    timespan = None
                else:
                    c = Citation(None, None, created, None, cited_pub_date, None, None, None, None, "", None, None, None, None, None)
                    timespan = c.duration

                self.update_status_file()
                return citing, cited, created, timespan, None, None

            self.update_status_file()
            row = self._get_next_in_file()

        remove(self.status_file)
                </pre>
            </code>
          </div>
          <iframe src="https://trinket.io/embed/python/5b49f453d6" width="100%" height="356" frameborder="0" marginwidth="0" marginheight="0" allowfullscreen></iframe>
          
          <h4>test.10_noci</h4>
          <div class="codebg">
            <code>
                <pre>
import unittest
from index.coci.glob import process #something to be developed to perform the same task but with pmids
from os import sep, makedirs  #Recursive directory creation function. Like mkdir(), but makes all intermediate-level directories needed to contain the leaf directory.
from os.path import exists  #used to check whether the specified path exists or not.
from shutil import rmtree #used to delete an entire directory tree, path must point to a directory (but not a symbolic link to a directory)
from index.storer.csvmanager import CSVManager #This class is able to load a simple CSV composed by two fields, 'id' and'value', and then to index all its items in a structured form so as to be easily queried. In addition, it allows one to store new information in the CSV, if needed.
from index.noci.nationalinstituteofhealthsource import NationalInstituteHealthSource #crowdsourcedcitationsource sostituito
from csv import DictReader


class NOCITest(unittest.TestCase):

    def setUp(self): #The self parameter is a reference to the current instance of the class, and is used to access variables that belongs to the class.
        self.input_file = "index%stest_data%snoci_dump%ssource.csv" % (sep, sep, sep) #without placeholders for the tuple format, data as provided by NIH
        self.citations = "index%stest_data%snoci_dump%scitations.csv" % (sep, sep, sep) #adaptation for tuple format, with all 6 fields
    def test_citation_source(self):
        ns = NationalInstituteHealthSource(self.input_file) #class of nationalinstituteofhealthsource, where a version of get next citation handling pmid data will be defined.
        new = []
        cit = ns.get_next_citation_data() #will extract citational data from our input file, which is the source one, with just citing and cited
        while cit is not None: #(which is: until we have citational data)
            citing, cited, creation, timespan, journal_sc, author_sc = cit
            #we append each citational data to the new list (which will be a list of dictionaries), in the format required for the 6-elements tuple.
            new.append({
                "citing": citing,
                "cited": cited,
                "creation": "" if creation is None else creation, #in all our cases the last four fields will be "","","no","no"
                "timespan": "" if timespan is None else timespan,
                "journal_sc": "no" if journal_sc is None else journal_sc,
                "author_sc": "no" if author_sc is None else author_sc
            })
            cit = ns.get_next_citation_data() #cit variable is assigned the value of the subsequent citational datum from input_file (source.csv)

        with open(self.citations) as f: #open the file as a parameter of a function, this time the csv with the data compiled in the 6-elements tuple format
            old = list(DictReader(f)) #a list of dictionaries is derived from data in citations.


        self.assertEqual(new, old) #check that the lists of dictionaries derived respectively from source and from citations are the equal.
                </pre>
            </code>
          </div>
          <p>Notes:</p>
          <ul>
            <li>There was a typing error in the name of the imported class</li>
            <li>Running the test, I got an "indentation error", even if it is indented in the exact same way as 09_croci, which instead works.</li>
          </ul>
          <div class="codebg">
            <code>
                <pre>
                Microsoft Windows [Versione 10.0.18363.1379]
(c) 2019 Microsoft Corporation. Tutti i diritti sono riservati.

C:\Users\arimoretti>cd Documents

C:\Users\arimoretti\Documents>cd GitHub

C:\Users\arimoretti\Documents\GitHub>python -m unittest index.test.10_noci
Traceback (most recent call last):
  File "C:\Users\arimoretti\AppData\Local\Programs\Python\Python38-32\lib\runpy.py", line 192, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Users\arimoretti\AppData\Local\Programs\Python\Python38-32\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\Users\arimoretti\AppData\Local\Programs\Python\Python38-32\lib\unittest\__main__.py", line 18, in <module>
    main(module=None)
  File "C:\Users\arimoretti\AppData\Local\Programs\Python\Python38-32\lib\unittest\main.py", line 100, in __init__
    self.parseArgs(argv)
  File "C:\Users\arimoretti\AppData\Local\Programs\Python\Python38-32\lib\unittest\main.py", line 147, in parseArgs
    self.createTests()
  File "C:\Users\arimoretti\AppData\Local\Programs\Python\Python38-32\lib\unittest\main.py", line 158, in createTests
    self.test = self.testLoader.loadTestsFromNames(self.testNames,
  File "C:\Users\arimoretti\AppData\Local\Programs\Python\Python38-32\lib\unittest\loader.py", line 220, in loadTestsFromNames
    suites = [self.loadTestsFromName(name, module) for name in names]
  File "C:\Users\arimoretti\AppData\Local\Programs\Python\Python38-32\lib\unittest\loader.py", line 220, in <listcomp>
    suites = [self.loadTestsFromName(name, module) for name in names]
  File "C:\Users\arimoretti\AppData\Local\Programs\Python\Python38-32\lib\unittest\loader.py", line 154, in loadTestsFromName
    module = __import__(module_name)
  File "C:\Users\arimoretti\Documents\GitHub\index\test\10_noci.py", line 7, in <module>
    from index.noci.nationalinstituteofhealthsource import NationalInstituteHealthSource
  File "C:\Users\arimoretti\Documents\GitHub\index\noci\nationalinstituteofhealthsource.py", line 25
    else:
        ^
IndentationError: unindent does not match any outer indentation level

                </pre>
            </code>
          </div>
          <h4>index.test.02_identifiermanager</h4>
          <div>
            <code>
                <pre>
                
C:\Users\arimoretti\Documents\GitHub>python -m unittest index.test.02_identifiermanager
......F.
======================================================================
FAIL: test_pmid_is_valid (index.test.02_identifiermanager.IdentifierManagerTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\Users\arimoretti\Documents\GitHub\index\test\02_identifiermanager.py", line 126, in test_pmid_is_valid
    self.assertTrue(pm_nofile.is_valid(self.valid_pmid_1))
AssertionError: False is not true

----------------------------------------------------------------------
Ran 8 tests in 2.049s

FAILED (failures=1)

                </pre>
            </code>
          </div>
          <h2>Study the execution process</h2>
          <h3>Required integrations</h3>
          <ol>
            <li>citation</li>
                <ul>
                    <li>citationsource.py: contains CitationSource class, that must be implemented for any source providing citations to be included in an OpenCitations Index (Coci, Croci, Noxi) -- OK</li>
                    <li>oci.py: contains class Citation-- self.citing_pub_date = Citation.check_date(citing_pub_date[:10] if citing_pub_date else citing_pub_date)?? -> why :10? --        if "D" in duration or Citation.contains_days(creation_date):
            cut = 10 // elif "M" in duration or Citation.contains_months(creation_date):
            cut = 7 //else:
            cut = 4 --? --OK</li>
                </ul>
            <li>coci</li>
                <ul>
                    <li>checkmetadata.py: --is it required also for noci?</li><li>crossrefcitationsource.py: -- works with doi only-- OK</li>
                    <li>glob.py:-- NEEDS INTEGRATION (manages DOI, ISSN and ORCID only)</li>
                    <li>trimdump.py: -- OK</li>
                </ul>
            <li>croci</li>
                <ul>
                    <li>crowdsourcedcitationsource.py: works with doi only --OK</li>
                </ul>
            <li>finder</li>
                <ul>
                    <li>crossrefresourcefinder.py: works with orcid + issn + doi -- NEEDS EXTENSION?</li>
                    <li>dataciterefresourcefinder.py: " --NEEDS EXTENSION?</li>
                    <li>orcidrefresourcefinder.py: " --NEEDS EXTENSION?</li>
                    <li>resourcefinder.py:" --NEEDS EXTENSION?</li>
                </ul>
            <li>identifier</li>
                <ul>
                    <li>doimanager.py - OK</li>
                    <li>identifiermanager.py -- general -OK</li>
                    <li>issnmanager.py -OK</li>
                    <li>orcidmanager.py -OK</li>
                    <li>pmidmanager.py - OK (EXTENDED)</li>
                </ul>
            <li>storer</li>
                <ul>
                    <li>citationstorer.py :  @staticmethod
    def load_citations_from_file(data_f_path, prov_f_path=None, oci="04201-04201", baseurl="",
                                 service_name="CitationStorer", id_type="doi",
                                 id_shape="http://dx.doi.org/([[XXX__decode]])",
                                 citation_type="", agent="", source=""): ---> TO BE SPECIFIED?</li>
                    <li>csvmanager.py : This class is able to load a simple CSV composed by two fields, 'id' and
    'value', and then to index all its items in a structured form so as to be
    easily queried. In addition, it allows one to store new information in the CSV,
    if needed.-- general - OK</li>
                    <li>updatetp.py -- general - OK</li>
                </ul>
            <li>support</li>
            <li>test</li>
            <li>test_data</li>
            <li>__init__.py</li>
            <li>cnc.py: Cnc.py for now works with dois only. It neither checks whether the id is a doi or not. We need to make it work with pmid too, so the nature of the id will have to be specified.</li>
          </ol>
          
          
          <h2>Dynamic requests for croci</h2>
          <p>To call CROCI, here it is the command (to be launched from the directory **containing** the folder of the GitHub repo "index"):</p>
          <div class="codebg">
              <code>
                  <pre>
                  python -m index.cnc -ib "http://dx.doi.org/" -b "https://w3id.org/oc/index/croci/" -p "index/croci/crowdsourcedcitationsource.py" -c "CrowdsourcedCitationSource" -i "[PATH-ASSOLUTO-DATI-CSV-DI-PARTENZA]" -l "[PATH-ASSOLUTO-LOOKUP-TABLE]" -d "[PATH-ASSOLUTO-DOVE-SALVARE-IL-RISULTATO]" -px "050" -a "https://orcid.org/0000-0003-0530-4305" -s "https://doi.org/10.5281/zenodo.3832935" -sv "OpenCitations Index: CROCI" -o "[CHIAVE-PER-API-ORCID]" -v
                  </pre>
              </code>
          </div>
          <p>I tried with:</p>
          <div class="codebg">
              <code>
                  <pre>
                  C:\Users\arimoretti\Documents\GitHub>python -m index.cnc -ib "http://dx.doi.org/" -b "https://w3id.org/oc/index/croci/" -p "index/croci/crowdsourcedcitationsource.py" -c "CrowdsourcedCitationSource" -i "C:\Users\arimoretti\Documents\GitHub\index\test_data\croci_dump\source.csv" -l "C:\Users\arimoretti\Documents\GitHub\index\test_data\tmp_store\lookup_full.csv" -d " C:\Users\arimoretti\Documents" -px "050" -a "https://orcid.org/0000-0003-0530-4305" -s "https://doi.org/10.5281/zenodo.3832935" -sv "OpenCitations Index: CROCI" -o "0000-0003-0530-4305" -v
                  </pre>
              </code>
          </div>
          <p>Notes:</p>
          <ul>
          <li>Why a specific doi and Orcid are already specified?</li>
          <li>What does ""[CHIAVE-PER-API-ORCID]"" correspond to? i tried with the orcid specified in the request but it didn't work, so I imagine I misunderstood its meaning</li>
          </ul>
          
          
          <p>In particular:</p>
          <ul>
              <li>[PATH-ASSOLUTO-DATI-CSV-DI-PARTENZA] è il csv che contiene i dati iniziali nella tabella a quattro colone che il processo di CROCI si aspetta in input (vedi, per esempio, il file "index/test_data/croci_dump/source.csv"</li>
              <li>[PATH-ASSOLUTO-LOOKUP-TABLE] punta al file della lookup table (puoi usare il file "index/test_data/lookup_full.csv")</li>
              <li>[PATH-ASSOLUTO-DOVE-SALVARE-IL-RISULTATO] lo devi specificare tu</li>
          </ul>

          <h2>Information provided specifying "?format=pubmed" in the api request</h2> 
          <p>e.g.: <strong>https://pubmed.ncbi.nlm.nih.gov/47/?format=pubmed</strong></p>
          <p>We are interested in Publication Date and in IS</p>
            <div class="codebg">
              <code>
                  <pre>
PMID- 47
OWN - NLM
STAT- MEDLINE
DCOM- 19760209
LR  - 20190612
IS  - 0006-2960 (Print)
IS  - 0006-2960 (Linking)
VI  - 14
IP  - 24
DP  - 1975 Dec 2
TI  - The influence of pH on the interaction of inhibitors with triosephosphate isomerase 
      and determination of the pKa of the active-site carboxyl group.
PG  - 5274-9
AB  - Ionization effects on the binding of the potential transition state analogues 
      2-phosphoglycolate and 2-phosphoglycolohydroxamate appear to be attributable to the 
      changing state of ionization of the ligands themselves, therefore it is unnecessary 
      to postulate the additional involvement of an ionizing residue at the active site of 
      triosephosphate isomerase to explain the influence of changing pH on Ki in the 
      neutral range. The binding of the competitive inhibitor inorganic sulfate is 
      insensitive to changing pH in the neutral range. 3-Chloroacetol sulfate, synthesized 
      as an active-site-specific reagent for triosephosphate isomerase, is used to provide 
      an indication of the pKa of the essential carboxyl group of this enzyme. Previously 
      described active-site-specific reagents for the isomerase were phosphate esters, and 
      their changing state of ionization (accompanied by possible changes in their 
      affinity for the active site) may have complicated earlier attempts to determine the 
      pKa of the essential carboxyl group from the pH dependence of the rate of 
      inactivation. Being a strong monoprotic acid, chloroacetol sulfate is better suited 
      to the determination of the pKa of the carboxyl group. Chloroacetol sulfate 
      inactivates triosephosphate isomerase by the selective esterification of the same 
      carboxyl group as that which is esterified by the phosphate esters described 
      earlier. From the pH dependence of the rate of inactivation of yeast triosephosphate 
      isomerase, the apparent pKa of the active-site carboxyl group is estimated as 3.9 
      +/- 0.1.
FAU - Hartman, F C
AU  - Hartman FC
FAU - LaMuraglia, G M
AU  - LaMuraglia GM
FAU - Tomozawa, Y
AU  - Tomozawa Y
FAU - Wolfenden, R
AU  - Wolfenden R
LA  - eng
PT  - Journal Article
PT  - Research Support, U.S. Gov't, Non-P.H.S.
PT  - Research Support, U.S. Gov't, P.H.S.
PL  - United States
TA  - Biochemistry
JT  - Biochemistry
JID - 0370623
RN  - 0 (Hydroxymercuribenzoates)
RN  - 9BZQ3U62JX (Dithionitrobenzoic Acid)
RN  - EC 5.1.3.- (Carbohydrate Epimerases)
RN  - EC 5.3.1.1 (Triose-Phosphate Isomerase)
SB  - IM
MH  - Animals
MH  - Binding Sites
MH  - Carbohydrate Epimerases/*metabolism
MH  - Dithionitrobenzoic Acid/pharmacology
MH  - Hydrogen-Ion Concentration
MH  - Hydroxymercuribenzoates/pharmacology
MH  - Kinetics
MH  - Muscles/enzymology
MH  - Protein Binding
MH  - Rabbits
MH  - Saccharomyces cerevisiae/enzymology
MH  - Triose-Phosphate Isomerase/*metabolism
EDAT- 1975/12/02 00:00
MHDA- 1975/12/02 00:01
CRDT- 1975/12/02 00:00
PHST- 1975/12/02 00:00 [pubmed]
PHST- 1975/12/02 00:01 [medline]
PHST- 1975/12/02 00:00 [entrez]
AID - 10.1021/bi00695a007 [doi]
PST - ppublish
SO  - Biochemistry. 1975 Dec 2;14(24):5274-9. doi: 10.1021/bi00695a007.
                  </pre>
              </code>
          </div>
          <p>Another example:</p>
          <div class="codebg">
              <code>
                  <pre>
PMID- 4700
OWN - NLM
STAT- MEDLINE
DCOM- 19760706
LR  - 20080226
IS  - 0093-7061 (Print)
IS  - 0093-7061 (Linking)
VI  - 5
IP  - 3
DP  - 1976 Mar
TI  - The issues have become legitimate.
PG  - 104, 103
FAU - Cunningham, R M Jr
AU  - Cunningham RM Jr
LA  - eng
PT  - Journal Article
PL  - United States
TA  - Mod Healthc (Short Term Care)
JT  - Modern healthcare. [Short-term care ed.]
JID - 0417653
SB  - IM
MH  - *Delivery of Health Care
MH  - Health Education
MH  - Home Nursing
MH  - Preventive Medicine
MH  - United States
EDAT- 1976/03/01 00:00
MHDA- 2001/03/28 10:01
CRDT- 1976/03/01 00:00
PHST- 1976/03/01 00:00 [pubmed]
PHST- 2001/03/28 10:01 [medline]
PHST- 1976/03/01 00:00 [entrez]
PST - ppublish
SO  - Mod Healthc (Short Term Care). 1976 Mar;5(3):104, 103.
                  </pre>
              </code> 
          </div>
    <h2>?format=pmid: various cases</h2>
          <h4>https://pubmed.ncbi.nlm.nih.gov/47/?format=pmid</h4>
          <p><strong>Case 1</strong>: The pmid exists and is in its normal format</p>
          <div class="codebg">
              <code>
                  <pre>
                  47
                  </pre>
              </code>
          </div>
          <h4>https://pubmed.ncbi.nlm.nih.gov/0047/?format=pmid</h4>
          <p><strong>Case 2</strong>: The pmid exists but it is preceeded by 0s</p>
          <div class="codebg">
              <code>
                  <pre>
                  0047
                  </pre>
              </code>
          </div>
          <p>The API gives back the number in the same format, but the request https://pubmed.ncbi.nlm.nih.gov/0047/ ends in "PMID 0047 is not available", which means the format of the number is not controlled. The api can give back an answer even if no article is related to the specified identifier. So, the format of the number must be checked and normalised before the request is done.</p>
          
          <h4>https://pubmed.ncbi.nlm.nih.gov/470d67/?format=pmid</h4>
          <p><strong>Case 3</strong>: The pmid contains characters which are not digits</p>
          <p>The requests resolves to https://pubmed.ncbi.nlm.nih.gov/470/, which means the id is broken before the first non digit character, and only the first part of the id is considered. Moreover, for the api request, /?format=pmid should be specified again. So, extra characted should be left off from the id before the api request is done.</p>
      </div>
    </div>
  </div>
  <div class="card">
    <div class="card-header" id="headingSix">
      <h5 class="mb-0">
        <button class="btn btn-link collapsed" data-toggle="collapse" data-target="#collapseSix" aria-expanded="false" aria-controls="collapseSix">
          Week #6 (02-25-21)
        </button>
      </h5>
    </div>
    <div id="collapseSix" class="collapse" aria-labelledby="headingSix" data-parent="#accordion">
      <div class="card-body">
        <h2>os.makedirs()local tries</h2>
        <h3>Create directories</h3>
        <div class="codebg">
            <code>
                <pre>
                """"
Suppose we want to create directory ‘ihritik’ but Directory ‘GeeksForGeeks’ and ‘Authors’
are unavailable in the path. Then os.makedirs() method will create all unavailable/missing
directory in the specified path. ‘GeeksForGeeks’ and ‘Authors’ will be created first
then ‘ihritik’ directory will be created.
"""

# Python program to explain os.makedirs() method

# importing os module
import os

# Leaf directory
directory = "ihritik"

# Parent Directories
parent_dir = "C:/Users/arimoretti/mkdirstry/GeeksForGeeks/Authors"

# Path
path = os.path.join(parent_dir, directory)

# Create the directory
# 'ihritik'
os.makedirs(path)
print("Directory '%s' created" %directory)

# Directory 'GeeksForGeeks' and 'Authors' will
# be created too
# if it does not exists



# Leaf directory
directory = "c"

# Parent Directories
parent_dir = "C:/Users/arimoretti/mkdirstry/GeeksforGeeks/a/b"

# mode
mode = 0o666

path = os.path.join(parent_dir, directory)

# Create the directory
# 'c'

os.makedirs(path, mode)
print("Directory '%s' created" %directory)


# 'GeeksForGeeks', 'a', and 'b'
# will also be created if
# it does not exists

# If any of the intermediate level
# directory is missing
# os.makedirs() method will
# create them

# os.makedirs() method can be
# used to create a directory tree
                </pre>
            </code>
        </div>
        <p>Output: Directory 'ihritik' created, Directory 'c' created
        </p>
        <p>Note: the filepath is to be specified with separators in the opposite direction (not the one suggested by windows)</p>
        
        <h3>Error raised when creating already existent directories</h3>

        <div class="codebg">
            <code>
                <pre>
# importing os module
import os

# os.makedirs() method will raise
# an OSError if the directory
# to be created already exists


# Directory
directory = "ihritik"

# Parent Directory path
parent_dir = "C:/Users/arimoretti/mkdirstry/GeeksforGeeks"

# Path
path = os.path.join( parent_dir, directory )

# Create the directory
# 'ihritik'
os.makedirs( path )
print( "Directory '%s' created" % directory )

# Python program to explain os.makedirs() method
                </pre>
            </code>
         </div>
         <p>Output: FileExistsError: [WinError 183] Impossibile creare un file, se il file esiste già: 'C:/Users/arimoretti/mkdirstry/GeeksForGeeks/Authors\\ihritik'</p>
         <p>N.B.: The path is specified with mixed separators, but the error code is exactly the expected one. So, probably, this is not determinant</p>
         
        <h3>Handling errors with: os.makedirs( path, exist_ok=True )</h3>
        
        <div class="codebg">
            <code>
                <pre>
                # importing os module
import os

# os.makedirs() method will raise
# an OSError if the directory
# to be created already exists
# But It can be suppressed by
# setting the value of a parameter
# exist_ok as True

# Directory
directory = "ihritik"

# Parent Directory path
parent_dir = "C:/Users/arimoretti/mkdirstry/GeeksForGeeks"

# Path
path = os.path.join( parent_dir, directory )

# Create the directory
# 'ihritik'
try:
    os.makedirs( path, exist_ok=True )
    print( "Directory '%s' created successfully" % directory )
except OSError as error:
    print( "Directory '%s' can not be created" )

# By setting exist_ok as True
# error caused due already
# existing directory can be suppressed
# but other OSError may be raised
# due to other error like
# invalid path name

                </pre>
            </code>
        </div>
        
        <p>Output: "C:\Program Files\Python36\python.exe" C:/Users/arimoretti/Desktop/mkdirstry/try.py
Directory 'ihritik' created successfully

Process finished with exit code 0 </p>
        <p>Notes: It seems that the separator issue doesn't cause any particular problem</p>

         
        <p>Source: <a href="https://www.geeksforgeeks.org/python-os-makedirs-method/">https://www.geeksforgeeks.org/python-os-makedirs-method/</a></p>
        <h2>Other dynamic requests(croci)</h2>
        <div class="codebg">
            <code>
                <pre>
                C:\Users\arimoretti\Documents\GitHub>python -m index.cnc -ib "http://dx.doi.org/" -b "https://w3id.org/oc/index/croci/" -p "C:\Users\arimoretti\Documents\GitHub\index\croci\crowdsourcedcitationsource.py" -c "CrowdsourcedCitationSource" -i "C:\Users\arimoretti\Documents\GitHub\index\test_data\croci_dump\source.csv" -l "C:\Users\arimoretti\Documents\GitHub\index\test_data\tmp_store\lookup_full.csv" -d "C:\Users\arimoretti\Documents\prova2" -px "050" -a "https://orcid.org/0000-0003-0530-4305" -s "https://doi.org/10.5281/zenodo.2582584" -sv "OpenCitations Index: CROCI" -v
Create citation data for 'oci:0500001010203040506070201080909-05000000110031106070201011207001213' between DOI '10.1002/asi.20755' and DOI '10.1109/wi.2006.164'
Create citation data for 'oci:0500001010203040506070201080909-05009020001031415071600000600000700130017' between DOI '10.1002/asi.20755' and DOI '10.5210/fm.v11i11.1413'
Create citation data for 'oci:0500001010203040506070201080909-050090200010314150716180600020700000118' between DOI '10.1002/asi.20755' and DOI '10.5210/fm.v8i12.1108'
Create citation data for 'oci:0500001010203040506070201080909-050090200010314150716000006100700130101' between DOI '10.1002/asi.20755' and DOI '10.5210/fm.v11i9.1400'
Create citation data for 'oci:0500001010203040506070201080909-050000117180313171810010104' between DOI '10.1002/asi.20755' and DOI '10.1038/438900a'
Create citation data for 'oci:0500001010203040506070201080909-050021701080302090210170001' between DOI '10.1002/asi.20755' and DOI '10.2307/2529310'
Create citation data for 'oci:0500001010203040506070201080909-050021701080313131812011202' between DOI '10.1002/asi.20755' and DOI '10.2307/4486062'
Create citation data for 'oci:0500001010203040506070201080909-050090200010314150716000206130700081217' between DOI '10.1002/asi.20755' and DOI '10.5210/fm.v12i4.1763'
Create citation data for 'oci:0500001010203040506070201080909-050000013090309011717081207090117130912' between DOI '10.1002/asi.20755' and DOI '10.1145/503376.503456'
Create citation data for 'oci:0500001010203040506070201080909-0500000130203100818101800020801000902081901010110' between DOI '10.1002/asi.20755' and DOI '10.1142/9789812701527_0009'
Create citation data for 'oci:0500001010203040506070201080909-0500000130903000901001317130700090100131309' between DOI '10.1002/asi.20755' and DOI '10.1145/1501434.1501445'
Create citation data for 'oci:0500001010203040506070201080909-05000010108030000181710091210191709' between DOI '10.1002/asi.20755' and DOI '10.1007/11839569_35'
Create citation data for 'oci:0500001010203040506070201080909-050021701080300091202021308' between DOI '10.1002/asi.20755' and DOI '10.2307/1562247'

# Summary
Number of new citations added to the OpenCitations Index: 13
Number of citations already present in the OpenCitations Index: 0
Number of citations with invalid DOIs: 0
                </pre>
            </code>
        </div>
        <h2>Request my own orcid-api key</h2>
        <p>I don't receive the confirmation e-mail for my orcid id, so I can't proceed with the process for asking the orcid-api key.</p>
        
        <h2>Tests Run and errors detection/fixation</h2>
        <h3>test.02_identifiermanager.py</h3>
        <h4>Extension for pubmedid</h4>
        <p>There is a problem related to the is_valid function</p>
        <div class="codebg">
            <code>
                <pre>
print(pm_nofile.normalise(self.valid_pmid_1, include_prefix=True )) # pmid:2942070
print(pm_nofile.is_valid("pmid:2942070")) # False
                </pre>
            </code>
        </div>
        <p>Notes:</p>
        <ul>
            <li>It does not depend on "if pmid is None or match("^pmid:[1-9]\d*$", pmid) is None" in is_valid in pmidmanager.py (I rerun the code both with the first part of the condition only and with the second part only, and both gave the same fail outcome)</li>
            <li>It does not depend on the normalise, since the print for the normalise step results as expected and I also tried to run the test using directly an id in its correct and expected format</li>
            <li>It does not depend on the specific id: I tried with all the three used as valid_id samples</li>
            <li>It does not depend on the prefix: the pmid comes without any prefix and the prefix is then added in the normalise step (prefix =True) at the second code line of is_valid </li>
        </ul>
        <p><strong>Deductions:</strong>Since print(pm_nofile.is_valid("pmid:2942070")) # False, the is_valid perceives that the pmid:number is none and/or doesn't match the regex (which, however, is correct, since the normalise step adds the prefix). 
        </p>
        <p>More in detail, in this way everything works:</p>
        
        <div class="codebg">
            <code>
                <pre>
#class extension for pubmedid
    def test_pmid_normalise(self):
        pm = PMIDManager()
        self.assertEqual(self.valid_pmid_1, pm.normalise(self.valid_pmid_1.replace("", "pmid:"))) #check other characters exclusion
        self.assertEqual(self.valid_pmid_1, pm.normalise(self.valid_pmid_1.replace("", " "))) #check other characters (spaces) exclusion
        self.assertEqual(self.valid_pmid_1, pm.normalise("https://pubmed.ncbi.nlm.nih.gov/"+self.valid_pmid_1)) #check other characters (url) exclusion --- aggiungi pmid
        self.assertEqual(self.valid_pmid_2, pm.normalise("000"+self.valid_pmid_2)) #check initial 0s exclusion

    def test_pmid_is_valid(self):
        pm_nofile = PMIDManager()
        print(pm_nofile.normalise(self.valid_pmid_1, include_prefix=True ))
        print(pm_nofile.is_valid(self.valid_pmid_1))
        self.assertFalse(pm_nofile.is_valid(self.invalid_pmid_1))
        self.assertFalse(pm_nofile.is_valid(self.invalid_pmid_2))
        self.assertFalse(pm_nofile.is_valid(self.invalid_pmid_3))

        valid_pmid = CSVManager(self.valid_pmid_path)
        pm_file = PMIDManager(valid_pmid=valid_pmid, use_api_service=False)
        self.assertFalse(pm_file.is_valid(self.invalid_pmid_1))

        pm_nofile_noapi = PMIDManager(use_api_service=False)
        self.assertFalse(pm_nofile_noapi.is_valid(self.valid_pmid_1)) #Why not assert True?
        self.assertFalse(pm_nofile_noapi.is_valid(self.invalid_pmid_1))


                </pre>
            </code>
        </div>
        
        <p>Part which is not working:</p>
        <div class="codebg">
            <code>
                <pre>
    def test_pmid_is_valid(self):

    self.assertTrue(pm_nofile.is_valid(self.valid_pmid_1))
    self.assertTrue(pm_nofile.is_valid(self.valid_pmid_2))
    self.assertTrue(pm_nofile.is_valid(self.valid_pmid_3))

    valid_pmid = CSVManager(self.valid_pmid_path)
    pm_file = PMIDManager(valid_pmid=valid_pmid, use_api_service=False)
    self.assertTrue(pm_file.is_valid(self.valid_pmid_1)) #in input un csv manager che eventualmente punta a quel file.
    self.assertFalse(pm_file.is_valid(self.invalid_pmid_1))

                </pre>
            </code>
        </div>
        
      <h3>test.03_resourcefinder.py fail</h3> 
      <h4>There may be a problem with the API request</h4>
      <div class="codebg">
        <code>
            <pre>
# Do not use support files, only APIs
df_1 = DataCiteResourceFinder()
self.assertIn("2197-6775", df_1.get_container_issn("10.14763/2019.1.1389"))
self.assertNotIn("1588-2861", df_1.get_container_issn("10.14763/2019.1.1389"))

            </pre>
        </code>
      </div>
      <h3>test.04_oci.py </h3>
      <h4>UnicodeDecodeError</h4>
      <p>'charmap' codec can't decode byte 0x90 in position 905: character maps to undefined </p>

<h4>Solved:</h4>
      <div class="codebg">
        <code>
            <pre>
with open(self.lookup_file, 'r', encoding="utf8") as f:
#(550)
with open(self.lookup_file, 'w', encoding="utf8") as f:

#(559)
with open(conf_file, encoding="utf8") as f:
#(565)
with open(csv_path, 'a', newline='', encoding="utf8") as csvfile:
#(599)

            </pre>
        </code>
      </div>
      
      <h3>test.05_citationstorer.py</h3>
      <h4>self.assertTrue(isomorphic(g1, g2)) --> true is not false</h4>
      
      <div class="codebg">
        <code>
            <pre>
def citations_rdf(self, origin_citation_list, stored_citation_list):
    g1 = ConjunctiveGraph()
    g2 = ConjunctiveGraph()

    for idx, cit in enumerate(origin_citation_list):
        for s, p, o, g in cit.get_citation_rdf(
                self.baseurl, False, False, True).quads((None, None, None, None)):
            g1.add((s, p, o, g))
        for s, p, o, g in stored_citation_list[idx].get_citation_rdf(
                self.baseurl, False, False, True).quads((None, None, None, None)):
            g2.add((s, p, o, g))

    self.assertTrue(isomorphic(g1, g2))
            </pre>
        </code>
      </div>
      <p>However, it should be correct, since enumerating a list we expect list[index] (stored_citation_list[idx]) to "refer" to the value (cit) </p>
      
      <h3>oci.py and pmid</h3>
      <h4>Needed integrations</h4>
      <p>From line 570</p>
      <div class="codebg">
        <code>
            <pre>
self.oci = oci_string.lower().strip()
elif doi_1 and doi_2:
    self.oci = self.get_oci(doi_1, doi_2, prefix)
else:
    self.oci = None
    self.add_message("__init__", W, "No OCI specified!")

            </pre>
        </code>
      </div>      
      
      <p>From line 617</p>
      <div class="codebg">
        <code>
            <pre>
def __decode(self, s):
    result = []

    for code in findall("(9*[0-8][0-9])", s):
        if code in self.lookup:
            result.append(self.lookup[code])
        else:
            result.append(code)

    return "10." + "".join(result)

def __decode_inverse(self, doi):
    return self.__match_str_to_lookup(doi.replace("10.", ""))

def get_oci(self, doi_1, doi_2, prefix):
    self.oci = "oci:%s%s-%s%s" % (prefix, self.__decode_inverse(doi_1), prefix, self.__decode_inverse(doi_2))
    return self.oci
            </pre>
        </code>
      </div>
      
      <h4>Related problems in noci.nationalinstituteofhealthsource.py</h4>
      <div class="codebg">
        <code>
            <pre>
c = Citation(None, None, created, None, cited_pub_date, None, None, None, None, "", None, None, None, None, None)
timespan = c.duration

#------------------------------------------------
self.update_status_file()
return citing, cited, created, timespan, None, None

#-------------------------------------------------
    self.update_status_file()
    row = self._get_next_in_file()
remove(self.status_file)

            </pre>
        </code>
      <div/>
      
      <h3>test.test.py</h3>
      <div class="codebg">
        <code>
            <pre>
            with open("/srv/data/glob/id_date.csv") as f:
            </pre>
        </code>
      </div>
      <p>Output: FileNotFoundError: [Errno 2] No such file or directory: '/srv/data/glob/id_date.csv'</p>
      
      <h4>Correction</h4>
      <div class="codebg">
        <code>
            <pre>
            with open("index/test_data/crossref_glob/id_date.csv") as f:            
            </pre>
        </code>
      </div>
      
      </div>
    </div>
  </div>
  
  </div>
  <div class="card">
    <div class="card-header" id="headingSeven">
      <h5 class="mb-0">
        <button class="btn btn-link collapsed" data-toggle="collapse" data-target="#collapseSeven" aria-expanded="false" aria-controls="collapseThree">
          Week #7 (03-4-21)
        </button>
      </h5>
    </div>
    <div id="collapseSeven" class="collapse" aria-labelledby="headingSeven" data-parent="#accordion">
      <div class="card-body">
        <h2>GitHub Issues</h2>
        <p>Make issues on GitHub (UnicodeDecodeError, Test 05_citationstorer fail)</p>
        
        <h2>ocy.py extension</h2>
        <p>Check where ocy has to be extended, so to manage pmids</p>
        
        <h2>nationalinstituteofhealthsource.py</h2>
        <p>Correct the indentation error</p>
        <div class="codebg">
            <code>
                <pre>
from os import walk, sep, remove
from os.path import isdir
from json import load
from csv import DictWriter
from index.citation.citationsource import CSVFileCitationSource
from index.identifier.pmidmanager import PMIDManager
from index.citation.oci import Citation


class NationalInstituteHealthSource(CSVFileCitationSource):
    def __init__(self, src, local_name=""):
        self.pmid = PMIDManager()
        super(NationalInstituteHealthSource, self).__init__(src, local_name)

    def get_next_citation_data(self):
        row = self._get_next_in_file()

        while row is not None:
            citing = self.pmid.normalise(row.get("citing"))
            cited = self.pmid.normalise(row.get("referenced"))

            if citing is not None and cited is not None:
                created = row.get("citing_publication_date")
                if not created:
                    created = None

                cited_pub_date = row.get("cited_publication_date")
                if not cited_pub_date:
                    timespan = None

                else:
                    c = Citation(None, None, created, None, cited_pub_date, None, None, None, None, "", None, None, None, None, None)
                    timespan = c.duration

                self.update_status_file()
                return citing, cited, created, timespan, None, None

            self.update_status_file()
            row = self._get_next_in_file()

        remove(self.status_file)
                </pre>
            </code>
        </div>
        <p>Test passed:</p>
        <div class="codebg">
            <code>
                <pre>
C:\Users\arimoretti\Documents\GitHub>python -m unittest index.test.10_noci
.
----------------------------------------------------------------------
Ran 1 test in 0.008s

OK

                </pre>
            </code>
        </div>
        
        
        
        
        
        <h2>Graphs serialization (g1 and g2 in 05_citationstorer)</h2>
        <p>Use n-triple to print the serialization</p>
        <h3>Plugin serializers</h3>
        <p>These serializers are available in default RDFLib, you can use them by passing the name to a graph’s serialize() method:</p>
        <div class="codebg">
            <code>
                <pre>
                graph.serialize(format='nt')
                </pre>
            </code>
        </div>
        <h3>rdflib.compare.isomorphic(graph1, graph2)</h3>
        <p>Compare graph for equality. Uses an algorithm to compute unique hashes which takes bnodes into account. Examples:</p>
        <div class="bgcode">
            <code>
                <pre>
                >>> g1 = Graph().parse(format='n3', data='''
...     @prefix : <http://example.org/ns#> .
...     <http://example.org> :rel <http://example.org/a> .
...     <http://example.org> :rel <http://example.org/b> .
...     <http://example.org> :rel [ :label "A bnode." ] .
... ''')
>>> g2 = Graph().parse(format='n3', data='''
...     @prefix ns: <http://example.org/ns#> .
...     <http://example.org> ns:rel [ ns:label "A bnode." ] .
...     <http://example.org> ns:rel <http://example.org/b>,
...             <http://example.org/a> .
... ''')
>>> isomorphic(g1, g2)
True

>>> g3 = Graph().parse(format='n3', data='''
...     @prefix : <http://example.org/ns#> .
...     <http://example.org> :rel <http://example.org/a> .
...     <http://example.org> :rel <http://example.org/b> .
...     <http://example.org> :rel <http://example.org/c> .
... ''')
>>> isomorphic(g1, g3)
False
                </pre>
            </code>
        </div>
        
        
        <p>Sources: <a href="https://rdflib.readthedocs.io/en/stable/apidocs/rdflib.html#rdflib.compare.IsomorphicGraph">https://rdflib.readthedocs.io/en/stable/apidocs/rdflib.html#rdflib.compare.IsomorphicGrap</a>, <a href="https://rdflib.readthedocs.io/en/stable/plugin_serializers.html">https://rdflib.readthedocs.io/en/stable/plugin_serializers.html</a></p>
        
        
        
        <h2>pmidmanager.py</h2>
        <p>Correct the function: api gives back an html and not a string</p>
        <div class="codebg">
            <code>
                <pre>
                
from index.identifier.identifiermanager import IdentifierManager
from re import sub, match
from urllib.parse import unquote, quote
from requests import get
from json import loads
from index.storer.csvmanager import CSVManager
from requests import ReadTimeout
from requests.exceptions import ConnectionError
from time import sleep
from bs4 import BeautifulSoup
from tempfile import NamedTemporaryFile
import shutil
import csv


class PMIDManager(IdentifierManager):
    def __init__(self, valid_pmid=None, use_api_service=True):
        if valid_pmid is None:
            valid_pmid = CSVManager(store_new=False)

        self.api = "https://pubmed.ncbi.nlm.nih.gov/"
        self.valid_pmid = valid_pmid
        self.use_api_service = use_api_service
        self.p = "pmid:"
        super(PMIDManager, self).__init__()

    def set_valid(self, id_string):
        pmid = self.normalise(id_string, include_prefix=True)

        if self.valid_pmid.get_value(pmid) is None:
            self.valid_pmid.add_value(pmid, "v")

    def is_valid(self, id_string):
        pmid = self.normalise(id_string, include_prefix=True)
        if pmid is None or match("^pmid:[1-9]\d*$", pmid) is None:
            return False
        else:
            if self.valid_pmid.get_value(pmid) is None:
                if self.__pmid_exists(pmid): #potrebbe essere qui
                    self.valid_pmid.add_value(pmid, "v")
                else:
                    self.valid_pmid.add_value(pmid, "i")
            return "v" in self.valid_pmid.get_value(pmid) #potrebbe restituire false anche qui
        #probabilmente ..
        #if 33 vede se il valore è giù incluso (potrebbe esserci già. se già incluso salta e va direttamente al return)
        #il problema probabilmente è tra 34 34



    def normalise(self, id_string, include_prefix=False):
        try:
            pmid_string = sub("^0+", "", sub("\0+", "", (sub("[^\d+]", "", id_string))))
            return "%s%s" % (self.p if include_prefix else "", pmid_string)
        except:
            return None


    def __pmid_exists(self, pmid_full):
        pmid = self.normalise(pmid_full)
        if self.use_api_service:
            print("usa API?")
            tentative = 3
            while tentative:
                tentative -= 1
                try:
                    print(self.api + quote(pmid) + "/?format=pmid")
                    r = get(self.api + quote(pmid) + "/?format=pmid", headers=self.headers, timeout=30)
                    if r.status_code == 200:
                        r.encoding = "utf-8"
                        print("Ho trovato l'ID?", r.text, pmid)

                        #inserire
                        soup = BeautifulSoup(r.content)
                        print("THIS IS SOUP", soup)
                        for i in soup.find_all("meta", {"name" : "uid"}):
                            id = i["content"]
                            print("THIS IS ID", id)
                            print("THIS IS PMID", pmid)
                            if id == pmid:
                                """

                                filename = 'C:/Users/arimoretti/Documents/GitHub/index/test_data/valid_pmid.csv'
                                tempfile = NamedTemporaryFile(mode='w', delete=False)

                                fields = ['id', 'value']

                                with open(filename, 'r') as csvfile, tempfile:
                                    reader = csv.DictReader(csvfile, fieldnames=fields)
                                    writer = csv.DictWriter(tempfile, fieldnames=fields)
                                    for row in reader:
                                        if row['id'] == pmid:
                                            row['id'], row['value'] = pmid, "v"
                                        row = {'id': row['id'], 'value': row['value']}
                                        writer.writerow(row)

                                shutil.move(tempfile.name, filename)
                                
                                """
                                return True

                except ReadTimeout:
                    pass
                except ConnectionError:
                    sleep(5)

        return False


                
                
                
                </pre>
            </code>
        </div>
        
        
        <h3>Notes</h3>
        <ul>
            <li>I had to correct the 3 invalid pmids: the ones taken as samples before were actually correct pmid after normalisation. </li>
            <li> However, there is still a problem related to the fact that all pmid are registered in the csv as "i", invalid. I checked that changing manually all the values in the csv file the test are passed. By the way, I can't understand why also correct pmids are registered as invalid. I tried to fix the probem by adding the part of code commented but a) It doesn't work as expected, b) I don't think it is the optimal solution</li>
        </ul>
        <p>The error occurs at:</p>
        <div class="codebg">
            <code>
                <pre>
valid_pmid = CSVManager(self.valid_pmid_path)
pm_file = PMIDManager(valid_pmid=valid_pmid, use_api_service=False)
#Any pmid is saved as "invalid" in the csv file. why?
self.assertTrue(pm_file.is_valid(self.valid_pmid_1)) #HERE THE ERROR!! in input a csv manager that can point at that file
self.assertFalse(pm_file.is_valid(self.invalid_pmid_1))

======================================================================
FAIL: test_pmid_is_valid (index.test.02_identifiermanager.IdentifierManagerTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\Users\arimoretti\Documents\GitHub\index\test\02_identifiermanager.py", line 142, in test_pmid_is_valid
    self.assertTrue(pm_file.is_valid(self.valid_pmid_1)) #in input un csv manager che eventualmente punta a quel file.
AssertionError: False is not true

----------------------------------------------------------------------
Ran 8 tests in 7.377s

FAILED (failures=1)


                </pre>
            </code>
        </div>
      </div>
    </div>
  </div>
  
  <div class="card">
    <div class="card-header" id="headingEight">
      <h5 class="mb-0">
        <button class="btn btn-link collapsed" data-toggle="collapse" data-target="#collapseEight" aria-expanded="false" aria-controls="collapseEight">
          Week #8 (03-11-21)
        </button>
      </h5>
    </div>
    <div id="collapseEight" class="collapse" aria-labelledby="headingEight" data-parent="#accordion">
      <div class="card-body">
      
        <h2>To do:</h2>
          <ul>
              <li>Correct the csv file manually so that it can be used in case api can't be used</li>
              <li>Check isomorphism between the two graph, and try to understand why do they differ</li>
              <li>Manage oci extension in order to manage pmid</li>
          </ul>
          
        <h2>oci.py extension</h2>
        <p>Extend oci so to handle pmids (for now only dois are). Since pmids are already numeric strings, both encoding and decoding functions have to be skipped.</p>
        
        <div class="codebg">
            <code>
                <pre>
    def __decode(self, s):
        if match( "^[1-9]\d*$", s) is not None:
            return s
        else:
            result = []

            for code in findall("(9*[0-8][0-9])", s):
                if code in self.lookup:
                    result.append(self.lookup[code])
                else:
                    result.append(code)
            return "10." + "".join(result)

    def __decode_inverse(self, doi):
        if match( "^[1-9]\d*$", doi) is not None: #pmid:?
            return doi
        else:
            return self.__match_str_to_lookup(doi.replace("10.", ""))
                </pre>
            </code>
        </div>
        
        <h2>05_citationstorer.py - Check isomorphism between the two graph</h2>
        <h3>Save serializations of the graphs in nt format</h3>
        <div class="codebg">
            <code>
                <pre>
        g1.serialize("rdf1.nt", format='nt')
        g2.serialize("rdf2.nt", format='nt')
                </pre>
            </code>
        </div>
        
        <h3>Sort lines of the two files</h3>
        <p>sortcsv.py</p>
        <div class="codebg">
            <code>
                <pre>
f= open("rdf1.txt","w+")
with open("rdf1.nt", 'r') as r:
    for line in sorted(r):
        print(line, end='')
        f.write(line)

f= open("rdf2.txt","w+")
with open("rdf2.nt", 'r') as r:
    for line in sorted(r):
        print(line, end='')
        f.write(line)
                </pre>
            </code>
        </div>
        
        <h3>Compare the two text file to see in which point the triples differ</h3>
        <p>Using <a href="https://text-compare.com/">https://text-compare.com/</a></p>
        <p>The following triple is not present in the second generted text file:</p>
                
                <p> https://w3id.org/oc/index/coci/ci/02001000002361927283705040000-02001000002361927283705030002  http://purl.org/spar/cito/hasCitationTimeSpan "P6M"^^ http://www.w3.org/2001/XMLSchema#duration .
                
                </p>


        <h2>valid_pmid.csv + 02_identifiermanager.py correction</h2>
        <h3>02_identifiermanager.py (pmid extension)</h3>
        <div class="codebg">
            <code>
                <pre>
                
import unittest
from os import sep
from index.identifier.doimanager import DOIManager
from index.identifier.issnmanager import ISSNManager
from index.identifier.orcidmanager import ORCIDManager
#pmid extension
from index.identifier.pmidmanager import PMIDManager
from index.storer.csvmanager import CSVManager


class IdentifierManagerTest(unittest.TestCase):
    """This class aim at testing the methods of the class CSVManager."""

    def setUp(self):
        self.valid_doi_1 = "10.1108/jd-12-2013-0166"
        self.valid_doi_2 = "10.1130/2015.2513(00)"
        self.invalid_doi_1 = "10.1108/12-2013-0166"
        self.invalid_doi_2 = "10.1371"
        self.valid_doi_path = "index%stest_data%svalid_doi.csv" % (sep, sep)

        self.valid_issn_1 = "2376-5992"
        self.valid_issn_2 = "1474-175X"
        self.invalid_issn_1 = "2376-599C"
        self.invalid_issn_2 = "2376-5995"
        self.invalid_issn_3 = "2376-599"

        self.valid_orcid_1 = "0000-0003-0530-4305"
        self.valid_orcid_2 = "0000-0001-5506-523X"
        self.invalid_orcid_1 = "0000-0003-0530-430C"
        self.invalid_orcid_2 = "0000-0001-5506-5232"
        self.invalid_orcid_3 = "0000-0001-5506-523"
        self.invalid_orcid_4 = "1-5506-5232"

#class extension for pubmedid
        self.valid_pmid_1 = "2942070"
        self.valid_pmid_2 = "1509982"
        self.valid_pmid_3 = "7189714"
        self.invalid_pmid_1 = "0067308798798"
        self.invalid_pmid_2 = "pmid:174777777777"
        self.invalid_pmid_3 = "000009265465465465"
        self.valid_pmid_path = "index%stest_data%svalid_pmid.csv" % (sep, sep)

    def test_doi_normalise(self):
        dm = DOIManager()
        self.assertEqual(self.valid_doi_1, dm.normalise(self.valid_doi_1.upper().replace("10.", "doi: 10. ")))
        self.assertEqual(self.valid_doi_1, dm.normalise(self.valid_doi_1.upper().replace("10.", "doi:10.")))
        self.assertEqual(self.valid_doi_1, dm.normalise(self.valid_doi_1.upper().replace("10.", "https://doi.org/10.")))

    def test_doi_is_valid(self):
        dm_nofile = DOIManager()
        self.assertTrue(dm_nofile.is_valid(self.valid_doi_1))
        self.assertTrue(dm_nofile.is_valid(self.valid_doi_2))
        self.assertFalse(dm_nofile.is_valid(self.invalid_doi_1))
        self.assertFalse(dm_nofile.is_valid(self.invalid_doi_2))

        valid_doi = CSVManager(self.valid_doi_path)
        dm_file = DOIManager(valid_doi=valid_doi, use_api_service=False)
        self.assertTrue(dm_file.is_valid(self.valid_doi_1))
        self.assertFalse(dm_file.is_valid(self.invalid_doi_1))

        dm_nofile_noapi = DOIManager(use_api_service=False)
        self.assertFalse(dm_nofile_noapi.is_valid(self.valid_doi_1))
        self.assertFalse(dm_nofile_noapi.is_valid(self.invalid_doi_1))

    def test_issn_normalise(self):
        im = ISSNManager()
        self.assertEqual(self.valid_issn_1, im.normalise(self.valid_issn_1.replace("-", "  ")))
        self.assertEqual(self.valid_issn_2, im.normalise(self.valid_issn_2.replace("-", "  ")))
        self.assertEqual(self.invalid_issn_3, im.normalise(self.invalid_issn_3.replace("-", "  ")))

    def test_issn_is_valid(self):
        im = ISSNManager()
        self.assertTrue(im.is_valid(self.valid_issn_1))
        self.assertTrue(im.is_valid(self.valid_issn_2))
        self.assertFalse(im.is_valid(self.invalid_issn_1))
        self.assertFalse(im.is_valid(self.invalid_issn_2))
        self.assertFalse(im.is_valid(self.invalid_issn_3))

    def test_orcid_normalise(self):
        om = ORCIDManager()
        self.assertEqual(self.valid_orcid_1, om.normalise(self.valid_orcid_1.replace("-", "  ")))
        self.assertEqual(self.valid_orcid_1, om.normalise("https://orcid.org/" + self.valid_orcid_1))
        self.assertEqual(self.valid_orcid_2, om.normalise(self.valid_orcid_2.replace("-", "  ")))
        self.assertEqual(self.invalid_orcid_3, om.normalise(self.invalid_orcid_3.replace("-", "  ")))

    def test_orcid_is_valid(self):
        om = ORCIDManager()
        self.assertTrue(om.is_valid(self.valid_orcid_1))
        self.assertTrue(om.is_valid(self.valid_orcid_2))
        self.assertFalse(om.is_valid(self.invalid_orcid_1))
        self.assertFalse(om.is_valid(self.invalid_orcid_2))
        self.assertFalse(om.is_valid(self.invalid_orcid_3))
        self.assertFalse(om.is_valid(self.invalid_orcid_4))


#class extension for pubmedid
    def test_pmid_normalise(self):
        pm = PMIDManager()
        self.assertEqual(self.valid_pmid_1, pm.normalise(self.valid_pmid_1.replace("", "pmid:"))) #check other characters exclusion
        self.assertEqual(self.valid_pmid_1, pm.normalise(self.valid_pmid_1.replace("", " "))) #check other characters (spaces) exclusion
        self.assertEqual(self.valid_pmid_1, pm.normalise("https://pubmed.ncbi.nlm.nih.gov/"+self.valid_pmid_1)) #check other characters (url) exclusion --- aggiungi pmid
        self.assertEqual(self.valid_pmid_2, pm.normalise("000"+self.valid_pmid_2)) #check initial 0s exclusion

    def test_pmid_is_valid(self):
        pm_nofile = PMIDManager()
        print(pm_nofile.normalise(self.valid_pmid_1, include_prefix=True ))
        print(pm_nofile.is_valid(self.valid_pmid_1))
        self.assertTrue(pm_nofile.is_valid(self.valid_pmid_1))
        self.assertTrue(pm_nofile.is_valid(self.valid_pmid_2))
        self.assertTrue(pm_nofile.is_valid(self.valid_pmid_3))
        self.assertFalse(pm_nofile.is_valid(self.invalid_pmid_1))
        self.assertFalse(pm_nofile.is_valid(self.invalid_pmid_2))
        self.assertFalse(pm_nofile.is_valid(self.invalid_pmid_3))

        valid_pmid = CSVManager(self.valid_pmid_path)
        pm_file = PMIDManager(valid_pmid=valid_pmid, use_api_service=False)
        self.assertTrue(pm_file.is_valid(self.valid_pmid_1))
        self.assertFalse(pm_file.is_valid(self.invalid_pmid_1))

        pm_nofile_noapi = PMIDManager(use_api_service=False)
        self.assertFalse(pm_nofile_noapi.is_valid(self.valid_pmid_1))
        self.assertFalse(pm_nofile_noapi.is_valid(self.invalid_pmid_1))
        self.assertFalse(pm_nofile_noapi.is_valid(self.valid_pmid_2))
        self.assertFalse(pm_nofile_noapi.is_valid(self.invalid_pmid_2))
        self.assertFalse(pm_nofile_noapi.is_valid(self.valid_pmid_3))
        self.assertFalse(pm_nofile_noapi.is_valid(self.invalid_pmid_3))
                </pre>
            </code>
        </div>
        
        <h3>valid_pmid.csv correction (manual updating)</h3>
        <div class="codebg">
            <code>
                <pre>
        "id","value"
        "pmid:2942070","v"
        "pmid:1509982","v"
        "pmid:7189714","v"
        "pmid:67308798798","i"
        "pmid:9265465465465","i"
        "pmid:174777777777","i"
                </pre>
            </code>
        </div>
        
        <h2>Issues</h2>
        <ul>
            <li>Is "encode function" == __decode_inverse?</li>
            <li>__decode_inverse(self, doi) --> Should "doi" parameter be left as it is or its name should be changed to be generally referable to both pmids and dois?</li>
        </ul>
      </div>
    </div>
  </div>
  
  
    <div class="card">
    <div class="card-header" id="headingNine">
      <h5 class="mb-0">
        <button class="btn btn-link collapsed" data-toggle="collapse" data-target="#collapseNine" aria-expanded="false" aria-controls="collapseNine">
          Week #9 (03-18-21)
        </button>
      </h5>
    </div>
    <div id="collapseNine" class="collapse" aria-labelledby="headingNine" data-parent="#accordion">
      <div class="card-body">
      
        <h2>To do:</h2>
          <ul>
              <li>read the paper about oci</li>
              <li>have a look at the conversion table in opencitations/oci/blob/master/lookup.csv</li>
              <li>update issue about Test 05_citationstorer fail</li>
              <li>chek up files and notes on them -- check (and fix): cnc 05_citationstorer + 04_oci</li>
              <li>start analysing cnc and try to understand how does cnc work and what is used in this part of the software, in order to figure out whether some parts of the code can be in contrast with the PMID structure to be integrated</li>
          </ul>
        
        <h2>Conversion Table</h2>
        <h3><a href="https://github.com/opencitations/oci/blob/master/lookup.csv"></a>lookup.csv</h3>
        <p><strong>https://github.com/opencitations/oci/blob/master/lookup.csv</strong></p>
        <p>Here there is a table created in order to map the whole character conversion oci-doi (for OCIs coming from DOIs). The structure presents on the left a character and on the right the code identifying that character. All the codes are composed of at least 2 digits. This is the case of the most common characters, which are the alphanumeric ones. Then, the more the characters gets particular, the more "9" digits we find at the beginning of their mapping codes. The aim is to have an unique numeric identifier for each character. In the case of DOIs, which are not composed of digits only, we need such a conversion, which instead is not necessary in the case citing and cited ids are already expressed as sequences of only digits (which is PMIDs case). </p>
        
        <h2>Read paper</h2>
        <a href="documenti/open%20citation%20+%20open%20citation%20identifier%20.docx">Summary "Open Citation" + "Open Citation Identifier"</a>
        <p>In each OCI, the first identifier is the identifier for the citing bibliographic resource, while
the second identifier is the identifier for the cited bibliographic resource.</p>
        <p>E.g.: <strong>0200102030405-0200102030406</strong> -> in both parts "0200" is a prefix. </p>
        <p>Considering a citations database, each OCI referring to one of the citations within that database must contain:</p>

        <ul>
            <li><strong>prefix</strong>: specifying the supplier’s database where the citation is recorded</li>
            <li><strong>numerical identifiers of the database</strong>, or the encoded numerical equivalent of
(part of) the identifiers used in that database, uniquely identifying citing and cited entity.</li>
        </ul>

        
        <h2>Update Issue on Open Citations</h2>
        <p>Issue updated: <a href="https://github.com/opencitations/index/issues/7">https://github.com/opencitations/index/issues/7</a></p>
        
        
        <h2>oci.py extension</h2>
        <p>Extend oci so to handle pmids (and correct previous wrong extension). Since pmids are already numeric strings, both encoding and decoding functions have to be skipped (they both are specific for dois).</p>
        
        <h3>APIManager class</h3>
        <div class="codebg">
            <code>
                <pre>
class APIManager(object):
    doi_type = "doi"
    pmid_type = "pmid"
                </pre>
            </code>
        </div>
        
        <div class="codebg">
            <code>
                <pre>
    def __decode(self, s):
        result = []

        for code in findall("(9*[0-8][0-9])", s):
            if code in self.lookup:
                result.append(self.lookup[code])
            else:
                result.append(code)

        return "10." + "".join(result)

    def __decode_inverse(self, doi):
        return self.__match_str_to_lookup(doi.replace("10.", ""))

    def get_oci(self, doi_1, doi_2, prefix, id_type):
        if id_type == APIManager.doi_type:
            self.oci = "oci:%s%s-%s%s" % (prefix, self.__decode_inverse(doi_1), prefix, self.__decode_inverse(doi_2)) 
            return self.oci
        elif id_type == APIManager.pmid_type:
            self.oci = "oci:%s%s-%s%s" % (prefix, doi_1, doi_2)
            return self.oci
        else:
            raise Exception("Only DOI or PMID can be handled")

                </pre>
            </code>
        </div>
        
        <p>Note: I'm not sure doi_type and pmid_type were to be defined inside Citation class or not. However, reading the notes of last meeting I thought I was supposed to make a new class, defining the two class variables in it.</p>
        <p>A second point which is not clear to me is the necessity to assign a string to the two variables, and the API function in this specific case. Should I import is_valid from pmidmanager (and the correspective from doimanager) or something similar to check the validity of what is passed as parameter in get_oci or the function of the id_type is conceived as a pattern matching (i.e. to verify the presence of the prefix string to understand if the user passed a doi or a pmid? ) </p>
        <p>Last question about this issue: get_oci parameters should be changed into something more neutral like "id_1", "id_2"?</p>
        
                <div class="codebg">
            <code>
                <pre>
    class OCIManager(object):
    def __init__(self, oci_string=None, lookup_file=None, conf_file=None, doi_1=None, doi_2=None, prefix=""):
        self.is_valid = None
        self.messages = []
        self.f = {
            "decode": self.__decode,
            "encode": quote,
            "join": OCIManager.__join,
            "shape": OCIManager.__shape,
            "remove": OCIManager.__remove,
            "normdate": OCIManager.__normdate,
            "datestrings": OCIManager.__datestrings,
            "api": OCIManager.__call_api,
            "avoid_prefix_removal": OCIManager.__avoid_prefix_removal
        }
        self.lookup = {}
        self.inverse_lookup = {}
        self.lookup_file = lookup_file
        self.lookup_code = -1
        if self.lookup_file is not None:
            if exists(self.lookup_file):
                with open(self.lookup_file, 'r', encoding="utf8") as f:
                    lookupcsv_reader = DictReader(f)
                    code = -1
                    for row in lookupcsv_reader:
                        self.lookup[row["code"]] = row["c"]
                        self.inverse_lookup[row["c"]] = row["code"]
                        code = int(row['code'])
                    self.lookup_code = code
            else:
                with open(self.lookup_file, 'w', encoding="utf8") as f:
                    f.write('"c","code"')
        else:
            self.add_message("__init__", W, "No lookup file has been found (path: '%s')." % lookup_file)
        self.conf = None
        if conf_file is not None and exists(conf_file):
            with open(conf_file, encoding="utf8") as f:
                self.conf = load(f)
        else:
            self.add_message("__init__", W, "No configuration file has been found (path: '%s')." % conf_file)

        if oci_string:
            self.oci = oci_string.lower().strip()
        elif doi_1 and doi_2:
            self.oci = self.get_oci(doi_1, doi_2, prefix, id_type) #HERE!
        else:
            self.oci = None
            self.add_message("__init__", W, "No OCI specified!")
                </pre>
            </code>
        </div>
        
        <p>I added the id_type parameter also in the get_oci call inside the OCIManager class, was it correct?</p>
        
        
        <h2>cnc.py integrations and issues</h2>
        <h3>APIManager import</h3>
        
        <div class="codebg">
            <code>
                <pre>
                from index.citation.oci import OCIManager, Citation, APIManager
                </pre>
            </code>
        </div>
        <h3>Prefix issue</h3>
        <p>I have a doubt about this part of the code: prefix is a necessary part of the oci structure, but at the same time all PMIDs should come from the same provider. The answer should be "they will have all the same prefix" but just to be sure I ask you if it is necessary that also pmids get a prefix (since in the opposite case also this part of cnc should be modified to handle the case in which the prefix is null or something similar) </p>

      </div>
    </div>
  </div>
  
  
  
  
  
    <div class="card">
    <div class="card-header" id="headingTen">
      <h5 class="mb-0">
        <button class="btn btn-link collapsed" data-toggle="collapse" data-target="#collapseTen" aria-expanded="false" aria-controls="collapseTen">
          Week #10 (04-01-21)
        </button>
      </h5>
    </div>
    <div id="collapseTen" class="collapse" aria-labelledby="headingTen" data-parent="#accordion">
      <div class="card-body">
          
          <h2>oci.py : get_oci(self, doi_1, doi_2, prefix, id_type)</h2>
          <h3>Change argument name to "id_1" and "id_2" instead of doi_1 and doi_2</h3>
          <p>Since no more only dois are handled, correct the arguments names</p>

          <h3>Create the 2 class variables doi_type and pmid_type in OCIManager</h3>
          <p>Since no more only dois are handled, create the two class variables to define the type of id</p>

          <div class="codebg">
            <code>
                <pre>
                
class OCIManager(object):
    doi_type = "doi"
    pmid_type = "pmid"
    def __init__(self, oci_string=None, lookup_file=None, conf_file=None, id_1=None, id_2=None, prefix="", id_type=""): 
        self.is_valid = None
        self.messages = []
        self.f = {
            "decode": self.__decode,
            "encode": quote,
            "join": OCIManager.__join,
            "shape": OCIManager.__shape,
            "remove": OCIManager.__remove,
            "normdate": OCIManager.__normdate,
            "datestrings": OCIManager.__datestrings,
            "api": OCIManager.__call_api,
            "avoid_prefix_removal": OCIManager.__avoid_prefix_removal
        }
        self.lookup = {}
        self.inverse_lookup = {}
        self.lookup_file = lookup_file
        self.lookup_code = -1
        if self.lookup_file is not None:
            if exists(self.lookup_file):
                with open(self.lookup_file, 'r', encoding="utf8") as f:
                    lookupcsv_reader = DictReader(f)
                    code = -1
                    for row in lookupcsv_reader:
                        self.lookup[row["code"]] = row["c"]
                        self.inverse_lookup[row["c"]] = row["code"]
                        code = int(row['code'])
                    self.lookup_code = code
            else:
                with open(self.lookup_file, 'w', encoding="utf8") as f:
                    f.write('"c","code"')
        else:
            self.add_message("__init__", W, "No lookup file has been found (path: '%s')." % lookup_file)
        self.conf = None
        if conf_file is not None and exists(conf_file):
            with open(conf_file, encoding="utf8") as f:
                self.conf = load(f)
        else:
            self.add_message("__init__", W, "No configuration file has been found (path: '%s')." % conf_file)

        if oci_string:
            self.oci = oci_string.lower().strip()
        elif id_1 and id_2:
            self.oci = self.get_oci(id_1, id_2, prefix, id_type)
        else:
            self.oci = None
            self.add_message("__init__", W, "No OCI specified!")

     
     (...)
     
     
     def get_oci(self, id_1, id_2, prefix, id_type):
        if id_type == OCIManager.doi_type:
            self.oci = "oci:%s%s-%s%s" % (prefix, self.__decode_inverse(id_1), prefix, self.__decode_inverse(id_2))
            return self.oci
        elif id_type == OCIManager.pmid_type:
            self.oci = "oci:%s%s-%s%s" % (prefix, (id_1), prefix, (id_2))
        else:
            raise Exception("Only DOI or PMID can be handled")
    
    (...)
                
                </pre>
            </code>
          </div>
          
          <h2>04_oci test: Add the missing id_type argument where needed</h2>
          
          <h3>oci.py</h3>
          
          <div class="codebg">
            <code>
                <pre>
                
                    def __init__(self,
                 oci, citing_url, citing_pub_date,
                 cited_url, cited_pub_date,
                 creation, timespan,
                 prov_entity_number, prov_agent_url, source, prov_date,
                 service_name, id_type, id_shape, citation_type,
                 journal_sc=False, author_sc=False,
                 prov_inv_date=None, prov_description=None, prov_update=None):
        self.oci = oci
        self.id_type = "doi" if OCIManager.doi_type else "pmid" #qui?? ininfluente?
        self.citing_url = citing_url
        self.cited_url = cited_url
        self.duration = Citation.check_duration(timespan)
        self.creation_date = Citation.check_date(creation[:10] if creation else creation)
        self.author_sc = "yes" if author_sc else "no"
        self.journal_sc = "yes" if journal_sc else "no"
        self.citing_pub_date = Citation.check_date(citing_pub_date[:10] if citing_pub_date else citing_pub_date)
        self.cited_pub_date = Citation.check_date(cited_pub_date[:10] if cited_pub_date else cited_pub_date)

        self.citation_type = citation_type if citation_type in CITATION_TYPES else DEFAULT_CITATION_TYPE

        # Set uniformly all the time-related data in a citation
        if self.citing_pub_date is None and self.creation_date is not None:
            self.citing_pub_date = self.creation_date
        if self.cited_pub_date is None and self.creation_date is not None and self.duration:
            self.cited_pub_date = Citation.check_date(Citation.get_date(self.creation_date, self.duration))
        if self.cited_pub_date is None:
            self.duration = None

        if self.contains_years(self.citing_pub_date):
            self.creation_date = self.citing_pub_date[:10]

            if self.contains_years(self.cited_pub_date):
                citing_contains_months = Citation.contains_months(self.citing_pub_date)
                cited_contains_months = Citation.contains_months(self.cited_pub_date)
                citing_contains_days = Citation.contains_days(self.citing_pub_date)
                cited_contains_days = Citation.contains_days(self.cited_pub_date)

                # Handling incomplete dates
                citing_complete_pub_date = self.creation_date
                cited_complete_pub_date = self.cited_pub_date[:10]
                if citing_contains_months and not cited_contains_months:
                    cited_complete_pub_date += self.citing_pub_date[4:7]
                elif not citing_contains_months and cited_contains_months:
                    citing_complete_pub_date += self.cited_pub_date[4:7]
                if citing_contains_days and not cited_contains_days:
                    cited_complete_pub_date += self.citing_pub_date[7:]
                elif not citing_contains_days and cited_contains_days:
                    citing_complete_pub_date += self.cited_pub_date[7:]

                try:
                    citing_pub_datetime = parse(citing_complete_pub_date, default=DEFAULT_DATE)
                except ValueError:  # It is not a leap year
                    citing_pub_datetime = parse(citing_complete_pub_date[:7] + "-28", default=DEFAULT_DATE)
                try:
                    cited_pub_datetime = parse(cited_complete_pub_date, default=DEFAULT_DATE)
                except ValueError:  # It is not a leap year
                    cited_pub_datetime = parse(cited_complete_pub_date[:7] + "-28", default=DEFAULT_DATE)

                delta = relativedelta(citing_pub_datetime, cited_pub_datetime)
                self.duration = Citation.get_duration(
                    delta,
                    citing_contains_months and cited_contains_months,
                    citing_contains_days and cited_contains_days)

        self.prov_entity_number = prov_entity_number
        self.prov_agent_url = prov_agent_url
        self.prov_date = Citation.check_datetime(prov_date)
        self.service_name = service_name
        self.prov_inv_date = Citation.check_datetime(prov_inv_date)
        self.prov_description = Citation.check_string(prov_description)
        self.prov_update = Citation.check_string(prov_update)

        self.id_type = id_type #qui ininfluente? entrano in contrasto?
        self.id_shape = id_shape

        self.source = source
        if "[[citing]]" in self.source:
            self.source = self.source.replace("[[citing]]", quote(self.get_id(citing_url)))
        elif "[[cited]]" in self.source:
            self.source = self.source.replace("[[cited]]", quote(self.get_id(cited_url)))
        
        (...)
                </pre>
            </code>
          </div>
          
          <p><strong>NOTES:</strong> There is a double definition of id_type in init, could they contrast each other? One was defined before my additions. Further, the whether the id_type is defined or not in this part of the code seems to be ininfluent, as long as it is defined in 07_cnc CreateNewCitationsTest.setUp</p>
          
          <div class="codebg">
            <code>
                <pre>
                
            "normdate": OCIManager.__normdate,
            "datestrings": OCIManager.__datestrings,
            "api": OCIManager.__call_api,
            "avoid_prefix_removal": OCIManager.__avoid_prefix_removal
        }
        self.lookup = {}
        self.inverse_lookup = {}
        self.lookup_file = lookup_file
        self.lookup_code = -1
        if self.lookup_file is not None:
            if exists(self.lookup_file):
                with open(self.lookup_file, 'r', encoding="utf8") as f:
                    lookupcsv_reader = DictReader(f)
                    code = -1
                    for row in lookupcsv_reader:
                        self.lookup[row["code"]] = row["c"]
                        self.inverse_lookup[row["c"]] = row["code"]
                        code = int(row['code'])
                    self.lookup_code = code
            else:
                with open(self.lookup_file, 'w', encoding="utf8") as f:
                    f.write('"c","code"')
        else:
            self.add_message("__init__", W, "No lookup file has been found (path: '%s')." % lookup_file)
        self.conf = None
        if conf_file is not None and exists(conf_file):
            with open(conf_file, encoding="utf8") as f:
                self.conf = load(f)
        else:
            self.add_message("__init__", W, "No configuration file has been found (path: '%s')." % conf_file)

        if oci_string:
            self.oci = oci_string.lower().strip()
        elif id_1 and id_2:
            self.oci = self.get_oci(id_1, id_2, prefix, id_type)
        else:
            self.oci = None
            self.add_message("__init__", W, "No OCI specified!")
                </pre>
            </code>
          </div>
          
          
          <div class="codebg">
            <code>
                <pre>
    def get_oci(self, id_1, id_2, prefix, id_type):
        if id_type == OCIManager.doi_type:
            self.oci = "oci:%s%s-%s%s" % (prefix, self.__decode_inverse(id_1), prefix, self.__decode_inverse(id_2))
            return self.oci
        elif id_type == OCIManager.pmid_type:
            self.oci = "oci:%s%s-%s%s" % (prefix, (id_1), prefix, (id_2))
        else:
            raise Exception("Only DOI or PMID can be handled")
                </pre>
            </code>
          </div>
          
          <div class="codebg">
            <code>
                <pre>
    def __execute_query(self, citing_entity, cited_entity):
        result = None

        if self.conf is None:
            self.add_message("__execute_query", E, "No citations can be retrieved since no configuration "
                                                   "file has been specified.")
        else:
            try:
                i = iter(self.conf["services"])
                while result is None:
                    item = next(i)
                    name, query, api, tp, use_it, preprocess, prefix, id_type, id_shape, citation_type = \
                        item.get("name"), item.get("query"), item.get("api"), item.get("tp"), item.get("use_it"), \
                        item["preprocess"] if "preprocess" in item else [], \
                        item["prefix"] if "prefix" in item else [], item.get("id_type"), item.get("id_shape"), \
                        item["citation_type"] if "citation_type" in item else DEFAULT_CITATION_TYPE

                    if use_it == "yes" and all(sub("^(%s).+$" % PREFIX_REGEX, "\\1", p) in prefix
                                               for p in (citing_entity, cited_entity)):
                        citing = sub("^%s(.+)$" % PREFIX_REGEX, "\\1", citing_entity)
                        cited = sub("^%s(.+)$" % PREFIX_REGEX, "\\1", cited_entity)

                        for f_name in preprocess:
                            citing = self.f[f_name](citing)
                            cited = self.f[f_name](cited)

                        if tp is None:
                            rest_query = api.replace("[[CITING]]", quote(citing)).replace("[[CITED]]", quote(cited))
                            structured_res, type_res = OCIManager.__call_api(rest_query)
                            if structured_res:
                                result = self.__read_api_data(structured_res, type_res, query.get("citing"),
                                                              citing, cited, api), \
                                         self.__read_api_data(structured_res, type_res, query.get("cited"),
                                                              citing, cited, api), \
                                         self.__read_api_data(structured_res, type_res, query.get("citing_date"),
                                                              citing, cited, api), \
                                         self.__read_api_data(structured_res, type_res, query.get("cited_date"),
                                                              citing, cited, api), \
                                         self.__read_api_data(structured_res, type_res, query.get("creation"),
                                                              citing, cited, api), \
                                         self.__read_api_data(structured_res, type_res, query.get("timespan"),
                                                              citing, cited, api), \
                                         rest_query, name, id_type, id_shape, citation_type
                        else:
                            sparql = SPARQLWrapper(tp)
                            sparql_query = sub("\\[\\[CITED\\]\\]", cited, sub("\\[\\[CITING\\]\\]", citing, query))

                            sparql.setQuery(sparql_query)
                            sparql.setReturnFormat(JSON)
                            q_res = sparql.query().convert()["results"]["bindings"]
                            if len(q_res) > 0:
                                answer = q_res[0]
                                result = answer["citing"]["value"], \
                                         answer["cited"]["value"], \
                                         answer["citing_date"]["value"] if "citing_date" in answer else None, \
                                         answer["cited_date"]["value"] if "cited_date" in answer else None, \
                                         answer["creation"]["value"] if "creation" in answer else None, \
                                         answer["timespan"]["value"] if "timespan" in answer else None, \
                                         tp + "?query=" + quote(sparql_query), name, id_type, id_shape, citation_type

            except StopIteration:
                pass  # No nothing

        return result
                </pre>
            </code>
          </div>
          
          <p><strong>NOTE:</strong>This part also refers to the other id_type variable (the originally defined one, which actually I think had more or less the same meaning of the one defined now to differentiate pmids from dois). May it be a problem?</p>
          
          
          <div class="codebg">
            <code>
                <pre>
    def get_citation_object(self):
        if self.validate():
            citing_entity_local_id = sub("^oci:([0-9]+)-([0-9]+)$", "\\1", self.oci)
            cited_entity_local_id = sub("^oci:([0-9]+)-([0-9]+)$", "\\2", self.oci)

            res = self.__execute_query(citing_entity_local_id, cited_entity_local_id)
            if res is not None:
                citing_url, cited_url, full_citing_pub_date, full_cited_pub_date, \
                creation, timespan, sparql_query_url, name, id_type, id_shape, citation_type = res

                citation = Citation(self.oci,
                                    citing_url, full_citing_pub_date,
                                    cited_url, full_cited_pub_date,
                                    creation, timespan,
                                    URL, sparql_query_url,
                                    datetime.now().strftime('%Y-%m-%dT%H:%M:%S'),
                                    name, id_type, id_shape, citation_type)

                return citation
            else:
                self.add_message("get_citation_object", I, "No citation data have been found for the OCI '%s'. "
                                                           "While the OCI specified is syntactically valid, "
                                                           "it is possible that it does not identify any "
                                                           "citation at all." % self.oci)
        else:
            self.add_message("get_citation_object", E, "No citation data can be returned since the OCI specified is "
                                                       "not valid.")

                </pre>
            </code>
          </div>
          
          <h3>cnc.py</h3>
          
          <div class="codebg">
            <code>
                <pre>
def execute_workflow(idbaseurl, baseurl, python, pclass, input, doi_file, date_file,
                     orcid_file, issn_file, orcid, lookup, data, prefix, id_type, agent, source, service, verbose, no_api): #qui?
    # Create the support file for handling information about bibliographic resources
    valid_doi, id_date, id_orcid, id_issn = create_csv(doi_file, date_file, orcid_file, issn_file)

    doi_manager = DOIManager(valid_doi, use_api_service=not no_api)
    crossref_rf = CrossrefResourceFinder(
        date=id_date, orcid=id_orcid, issn=id_issn, doi=valid_doi, use_api_service=not no_api)
    datacite_rf = DataCiteResourceFinder(
        date=id_date, orcid=id_orcid, issn=id_issn, doi=valid_doi, use_api_service=not no_api)
    orcid_rf = ORCIDResourceFinder(
        date=id_date, orcid=id_orcid, issn=id_issn, doi=valid_doi,
        use_api_service=True if orcid is not None and not no_api else False, key=orcid)

    rf_handler = ResourceFinderHandler([crossref_rf, datacite_rf, orcid_rf])
    return extract_citations(idbaseurl, baseurl, python, pclass, input, lookup, data, prefix, id_type,
                             agent, source, service, verbose, doi_manager, rf_handler) #qui?
                
                </pre>
            </code>
          </div>
          
          
          
          
          <div class="codebg">
            <code>
                <pre>
def extract_citations(idbaseurl, baseurl, python, pclass, input, lookup, data, prefix, id_type,
                      agent, source, service, verbose, doi_manager, rf_handler, oci_to_do=None): #qui???
    BASE_URL = idbaseurl
    DATASET_URL = baseurl + "/" if not baseurl.endswith("/") else baseurl

    oci_manager = OCIManager(lookup_file=lookup)
    exi_ocis = CSVManager.load_csv_column_as_set(data + sep + "data", "oci")  # TODO: we need to specify carefully the dir, eg by adding an additional flag to distinguish between the files belonging to a particular process, and it should be aligned with the storer.
    if oci_to_do is not None:
        oci_to_do.difference_update(exi_ocis)
    cit_storer = CitationStorer(data, DATASET_URL)

    citations_already_present = 0
    new_citations_added = 0
    error_in_dois_existence = 0

    cs = import_citation_source(python, pclass, input)
    next_citation = cs.get_next_citation_data()

    while next_citation is not None:
        citing, cited, created, timespan, journal_sc, author_sc = next_citation
        print("the id type is:", id_type) #for now it works with doi only
        oci = oci_manager.get_oci(citing, cited, prefix, id_type)
        oci_noprefix = oci.replace("oci:", "")
        if oci_noprefix not in exi_ocis and (oci_to_do is None or oci_noprefix in oci_to_do):
            if doi_manager.is_valid(citing) and doi_manager.is_valid(cited):
                if created is None:
                    citing_date = rf_handler.get_date(citing)
                else:
                    citing_date = created
                cited_date = rf_handler.get_date(cited)
                if journal_sc is None or type(journal_sc) is not bool:
                    journal_sc = rf_handler.share_issn(citing, cited)
                if author_sc is None or type(author_sc) is not bool:
                    author_sc = rf_handler.share_orcid(citing, cited)

                if created is not None and timespan is not None:
                    cit = Citation(oci,#primo parametro passato
                                   BASE_URL + quote(citing), None,
                                   BASE_URL + quote(cited), None,
                                   created, timespan,
                                   1, agent, source, datetime.now().strftime('%Y-%m-%dT%H:%M:%S'),
                                   service, id_type, BASE_URL + "([[XXX__decode]])", "reference",
                                   journal_sc, author_sc,
                                   None, "Creation of the citation", None) #NOTA: qui ho sostituito il precedente "doi" con id_type
                else:
                    cit = Citation(oci,
                                   BASE_URL + quote(citing), citing_date,
                                   BASE_URL + quote(cited), cited_date,
                                   None, None,
                                   1, agent, source, datetime.now().strftime('%Y-%m-%dT%H:%M:%S'),
                                   service, "doi", BASE_URL + "([[XXX__decode]])", "reference",
                                   journal_sc, author_sc,
                                   None, "Creation of the citation", None)

                cit_storer.store_citation(cit)

                if verbose:
                    print("Create citation data for '%s' between ID '%s' and ID '%s'" % (oci, citing, cited)) #changed DOI in ID
                new_citations_added += 1
                exi_ocis.add(oci_noprefix)
            else:
                if verbose:
                    print("WARNING: some IDs, among '%s' and '%s', do not exist" % (citing, cited)) #changed DOI in ID
                error_in_dois_existence += 1
            if oci_to_do is not None:
                oci_to_do.remove(oci_noprefix)
        else:
            if verbose:
                print("WARNING: the citation between ID '%s' and ID '%s' has been already processed" % #changed DOI in ID
                      (citing, cited))
            citations_already_present += 1

        next_citation = cs.get_next_citation_data()

    return new_citations_added, citations_already_present, error_in_dois_existence


if __name__ == "__main__":
    arg_parser = ArgumentParser("cnc.py (Create New Citations",
                                description="This tool allows one to take a series of entity-to-entity"
                                            "citation data, and to store it according to CSV used by"
                                            "the OpenCitations Indexes so as to be added to an Index. It uses"
                                            "several online services to check several things to create the"
                                            "final CSV/TTL/Scholix files.")

    arg_parser.add_argument("-p", "--python", required=True,
                            help="The input Python file implementing the class index.citation.CitationSource "
                                 "which is responsible for parsing and passing all the input entity-to-entity"
                                 "citations.")
    arg_parser.add_argument("-c", "--pclass", required=True,
                            help="The name of the class implementing the class index.citation.CitationSource "
                                 "which is responsible for parsing and passing all the input entity-to-entity"
                                 "citations.")
    arg_parser.add_argument("-i", "--input", required=True,
                            help="The input file/directory to provide as input of the specified input "
                                 "Python file (using -p).")
    arg_parser.add_argument("-d", "--data", required=True,
                            help="The directory containing all the CSV files already added in the Index, "
                                 "including data and provenance files.")
    arg_parser.add_argument("-o", "--orcid", default=None,
                            help="ORCID API key to be used to query the ORCID API.")
    arg_parser.add_argument("-l", "--lookup", required=True,
                            help="The lookup table that must be used to produce OCIs.")
    arg_parser.add_argument("-b", "--baseurl", required=True, default="",
                            help="The base URL of the dataset")
    arg_parser.add_argument("-ib", "--idbaseurl", required=True, default="",
                            help="The base URL of the identifier of citing and cited entities, if any")
    arg_parser.add_argument("-doi", "--doi_file", default=None,
                            help="The file where the valid and invalid DOIs are stored.") #how do I handle this?
    arg_parser.add_argument("-date", "--date_file", default=None,
                            help="The file that maps id of bibliographic resources with their publication date.")
    arg_parser.add_argument("-orcid", "--orcid_file", default=None,
                            help="The file that maps id of bibliographic resources with the ORCID of its authors.")
    arg_parser.add_argument("-issn", "--issn_file", default=None,
                            help="The file that maps id of bibliographic resources with the ISSN of the journal "
                                 "they have been published in.")
    arg_parser.add_argument("-px", "--prefix", default="",
                            help="The '0xxx0' prefix to use for creating the OCIs.")
    arg_parser.add_argument("-a", "--agent", required=True, default="https://w3id.org/oc/index/prov/pa/1",
                            help="The URL of the agent providing or processing the citation data.")
    arg_parser.add_argument("-s", "--source", required=True,
                            help="The URL of the source from where the citation data have been extracted.")
    arg_parser.add_argument("-sv", "--service", required=True,
                            help="The name of the service that will made available the citation data.")
    arg_parser.add_argument("-v", "--verbose", action="store_true", default=False,
                            help="Print the messages on screen.")
    arg_parser.add_argument("-na", "--no_api", action="store_true", default=False,
                            help="Tell the tool explicitly not to use the APIs of the various finders.")
    arg_parser.add_argument("-pn", "--process_number", default=1, type=int,
                            help="The number of parallel process to run for working on the creation of citations.")

    args = arg_parser.parse_args()
    n_processes = args.process_number
    if n_processes minore = 1: #nb I changed the symbol in "minore" in order to avoid breaking the code in brackets 
        new_citations_added, citations_already_present, error_in_dois_existence = \
            execute_workflow(args.idbaseurl, args.baseurl, args.python, args.pclass, args.input, args.doi_file,
                             args.date_file, args.orcid_file, args.issn_file, args.orcid, args.lookup, args.data,
                             args.prefix, args.id_type, args.agent, args.source, args.service, args.verbose, args.no_api) #qui?
    else:  # Run in parallel
        pass  # TODO: do things

    print("\n# Summary\n"
          "Number of new citations added to the OpenCitations Index: %s\n"
          "Number of citations already present in the OpenCitations Index: %s\n"
          "Number of citations with invalid DOIs: %s" %
          (new_citations_added, citations_already_present, error_in_dois_existence))

                </pre>
            </code>
          </div>
          
          <h2>Add type information to the six (now seven) elements tuple returned by get_next_citation_data</h2>
          
          <p> I modified the returned tuple in NationalInstituteHealthSource, CrossrefCitationSource and CrowdSourcedCitationSource, defining in the first case the variable id_type = OCIManager.pmid_type and in the other two cases id_type = OCIManager.doi_type and then adding id_type as seventh and last element of the returned tuple.</p>
          
          <h2>Extend 07_cnc</h2>
          <p>The extension is to be implemented, I just started checking 07_cnc and making some consideration to be discussed. I commented my consideration in several points of the code</p>
          
          <div class="codebg">
            <code>
                <pre>
import unittest
from os import sep, remove
from index.cnc import execute_workflow
from os.path import exists
from index.citation.citationsource import CSVFileCitationSource
from index.citation.oci import Citation, OCIManager
from urllib.parse import quote
from index.storer.citationstorer import CitationStorer
from glob import glob
from shutil import rmtree


class CreateNewCitationsTest(unittest.TestCase):

    def setUp(self):
        self.idbaseurl = "http://dx.doi.org/"
        self.baseurl = "https://w3id.org/oc/index/coci/"
        self.python = "index%scitation%scitationsource.py" % (sep, sep)
        self.pclass = "CSVFileCitationSource"
        self.input = "index%stest_data%scitations_partial.csv" % (sep, sep)
        self.doi_file = "index%stest_data%scnc_valid_doi.csv" % (sep, sep)
        self.pmid_file = "index%stest_data%scnc_valid_pmid.csv" % (sep, sep) #here extension for pmid
        self.date_file = "index%stest_data%scnc_id_date.csv" % (sep, sep)
        self.orcid_file = "index%stest_data%scnc_id_orcid.csv" % (sep, sep)
        self.issn_file = "index%stest_data%scnc_id_issn.csv" % (sep, sep)
        self.orcid = None
        self.lookup = "index%stest_data%slookup_full.csv" % (sep, sep)
        self.data = "index%stest_data%stmp_workflow" % (sep, sep)
        self.prefix = "020" #here the prefix is for dois only.
        # How can I specify an alternative prefix? maybe using where required an if-else structure to
        # identify the id type and then using the doi prefix where needed and pmid prefix in other cases?
        # All this specifying two distict prefix variables.
        self.id_type = "doi" if OCIManager.doi_type else "pmid" #Tests are passed but for dois only. does it
        #depend on this part of the code or on the lack of a further specification of id_type in other parts of the code?
        #i.e.: is  --"doi" if OCIManager.doi_type else "pmid"-- the problem or should I look for it somewhere else?
        #hint: deve passare solo doi perché qui si gestiscono solo doi
        self.agent = "https://w3id.org/oc/index/prov/ra/1"
        self.source = "https://api.crossref.org/works/[[citing]]"
        self.service = "OpenCitations Index: COCI" #Should I modify this part too?
        self.verbose = True
        self.no_api = False

        self.citation_list = self.__load_citations("index%stest_data%scitations_data.csv" % (sep, sep),
                                                   "index%stest_data%scitations_prov.csv" % (sep, sep))
        self.data_path = self.data + sep + "data" + sep + "**" + sep + "*.csv"
        self.prov_path = self.data + sep + "prov" + sep + "**" + sep + "*.csv"

        if exists(self.data):
            rmtree(self.data)

    def __load_citations(self, data, prov):
        return CitationStorer.load_citations_from_file(data, prov, baseurl="http://dx.doi.org/", #NOTE: The base url is no more this one only!
            service_name=self.service, id_type="doi",
            id_shape="http://dx.doi.org/([[XXX__decode]])", citation_type=None) #NOTE: discuss id_shape

    def __citations_csv(self, origin_citation_list, stored_citation_list):
        l1 = [cit.get_citation_csv() for cit in origin_citation_list]
        l2 = [cit.get_citation_csv() for cit in stored_citation_list]
        self.assertEqual(len(l1), len(l2))
        self.assertEqual(set(l1), set(l2))

    def __test_citations(self):
        data_csv = glob(self.data_path, recursive=True)
        prov_csv = glob(self.prov_path, recursive=True)
        self.assertEqual(len(data_csv), 1)
        self.assertEqual(len(prov_csv), 1)
        self.__citations_csv(self.citation_list, self.__load_citations(data_csv[0], prov_csv[0]))

    def test_execute_workflow(self): #Probably an alternative version for PMIDs is to be implemented. In case
        #only one function differentiating between id types is mantained, the argument self.id_type is ok, otherwise
        #ot is probably to be specified in one case "doi" and "pmid" in the other. Same for doi_file/pmid_file
        new_citations_added, citations_already_present, error_in_dois_existence = \
            execute_workflow(self.idbaseurl, self.baseurl, self.python, self.pclass, self.input, self.doi_file,
                             self.date_file, self.orcid_file, self.issn_file, self.orcid, self.lookup, self.data,
                             self.prefix, self.id_type, self.agent, self.source, self.service, self.verbose, self.no_api)#qui???
        self.assertEqual(new_citations_added, 6)
        self.assertEqual(citations_already_present, 0)
        self.assertEqual(error_in_dois_existence, 0)
        self.__test_citations()

        new_citations_added, citations_already_present, error_in_dois_existence = \
            execute_workflow(self.idbaseurl, self.baseurl, self.python, self.pclass, self.input, self.doi_file,
                             self.date_file, self.orcid_file, self.issn_file, self.orcid, self.lookup, self.data,
                             self.prefix, self.id_type, self.agent, self.source, self.service, self.verbose, self.no_api) #qui?
        self.assertEqual(new_citations_added, 0)
        self.assertEqual(citations_already_present, 6)
        self.assertEqual(error_in_dois_existence, 0)
        self.__test_citations()

                
                </pre>
            </code>
          </div>
          
          <h2>Issues</h2>
          <ul>

              <li>Discuss cnc.py execute_workflow: it works with doi only. Discuss the format of the support csv file built in this step</li>
              <li>Discuss cnc.py arg_parser.add_argument("-doi", "--doi_file"... and how to manage the pmid version</li>
              <li>07_cnc. Discuss how to test the cases in which the id is not a doi. </li>
          </ul>

      </div>
    </div>
  </div>
  
  
      <div class="card">
    <div class="card-header" id="headingEleven">
      <h5 class="mb-0">
        <button class="btn btn-link collapsed" data-toggle="collapse" data-target="#collapseEleven" aria-expanded="false" aria-controls="collapseEleven">
          Week #11 (04-15-21)
        </button>
      </h5>
    </div>
    <div id="collapseEleven" class="collapse" aria-labelledby="headingEleven" data-parent="#accordion">
      <div class="card-body">
          
          <h2>To do:</h2>
          <ul>
              <li>Id type specification in Tests of: CrossrefCitationSource, CrowdsourcedCitationSource, NationalInstituteOfHealthSource</li>
              <li>Study Class and Instance Variables and Inheritance</li>
              <li>Integrate a test for get_oci so to test also pmid type --- which is the pmid prefix?</li>
              <li>Add type information to the six (now seven) elements tuple which is returned by get_next_citation_data(self). Remember that this information is given, since NationalInstituteHealthSource manages pmids only, while CrossrefCitationSource and CrowdSourcedCitationSource dois only </li>
          </ul>
          
          <h2>Id type specification in Tests of: CrossrefCitationSource, CrowdsourcedCitationSource, NationalInstituteOfHealthSource </h2>
          
          <h3>CROCI</h3>
          <div class="codebg">
            <code>
                <pre>
class CrowdsourcedCitationSource(CSVFileCitationSource):
    def __init__(self, src, local_name=""):
        super(CrowdsourcedCitationSource, self).__init__(src, local_name, id_type=OCIManager.doi_type)
                </pre>
            </code>
          </div>
          
          <h3>COCI</h3>
          <div class="codebg">
            <code>
                <pre>
class CrossrefCitationSource(DirCitationSource):
    def __init__(self, src, local_name=""):
        self.last_ref = -1
        super(CrossrefCitationSource, self).__init__(src, local_name, id_type=OCIManager.doi_type)
                </pre>
            </code>
          </div>
          
          
          <h3>NOCI</h3>
          <div class="codebg">
            <code>
                <pre>
class NationalInstituteHealthSource(CSVFileCitationSource):
    def __init__(self, src, local_name=""):
        super(NationalInstituteHealthSource, self).__init__(src, local_name, id_type=OCIManager.pmid_type)
                </pre>
            </code>
          </div>
          
          <p><strong>Note:</strong> self.doi = DOIManager() and self.pmid = PMIDManager() vere replaced by id_type=OCIManager.pmid_type and id_type=OCIManager.doi_type in __init__</p>
          
          
          <h2>Class and Instance Variables + Inheritance</h2>
          <p>I studied these two chapters from How To code in Python e-book, available <a href="https://www.digitalocean.com/community/books/digitalocean-ebook-how-to-code-in-python">here</a></p>
          
          <h2>cnc</h2>
          <p>Cnc is to be modified in order to handle pmid too. I commented the passages I focused on, but before including them in the code, I'd prefer to discuss them. </p>
          
          <div class="codebg">
            <code>
                <pre>
                
from argparse import ArgumentParser
from urllib.parse import quote
from datetime import datetime
from os.path import abspath, dirname, basename
from os import sep
from sys import path
from importlib import import_module
from index.storer.csvmanager import CSVManager
from index.identifier.doimanager import DOIManager
form index.identifier.pmidmanager import PMIDManager #added PMIDManager
from index.finder.orcidresourcefinder import ORCIDResourceFinder
from index.finder.dataciteresourcefinder import DataCiteResourceFinder
from index.finder.crossrefresourcefinder import CrossrefResourceFinder
from index.finder.resourcefinder import ResourceFinderHandler
from index.citation.oci import OCIManager, Citation
from index.storer.citationstorer import CitationStorer


def execute_workflow_parallel():
def execute_workflow_parallel():
    pass  # TODO


def create_csv(doi_file, date_file, orcid_file, issn_file): #only file?
    valid_doi = CSVManager(csv_path=doi_file) #how can I
    id_date = CSVManager(csv_path=date_file)
    id_orcid = CSVManager(csv_path=orcid_file)
    id_issn = CSVManager(csv_path=issn_file)

    return valid_doi, id_date, id_orcid, id_issn


def import_citation_source(python, pclass, input):
    addon_abspath = abspath(python)
    path.append(dirname(addon_abspath))
    addon = import_module(basename(addon_abspath).replace(".py", ""))
    return getattr(addon, pclass)(input)


def execute_workflow(idbaseurl, baseurl, python, pclass, input, doi_file, date_file,
                     orcid_file, issn_file, orcid, lookup, data, prefix, id_type, agent, source, service, verbose, no_api): #qui?
    # Create the support file for handling information about bibliographic resources
    valid_doi, id_date, id_orcid, id_issn = create_csv(doi_file, date_file, orcid_file, issn_file)

    doi_manager = DOIManager(valid_doi, use_api_service=not no_api)
    crossref_rf = CrossrefResourceFinder(
        date=id_date, orcid=id_orcid, issn=id_issn, doi=valid_doi, use_api_service=not no_api)
    datacite_rf = DataCiteResourceFinder(
        date=id_date, orcid=id_orcid, issn=id_issn, doi=valid_doi, use_api_service=not no_api)
    orcid_rf = ORCIDResourceFinder(
        date=id_date, orcid=id_orcid, issn=id_issn, doi=valid_doi,
        use_api_service=True if orcid is not None and not no_api else False, key=orcid)

    rf_handler = ResourceFinderHandler([crossref_rf, datacite_rf, orcid_rf])
    return extract_citations(idbaseurl, baseurl, python, pclass, input, lookup, data, prefix, id_type,
                             agent, source, service, verbose, doi_manager, rf_handler) #qui?

"""
def execute_workflow(idbaseurl, baseurl, python, pclass, input, file, date_file,
                     orcid_file, issn_file, orcid, lookup, data, prefix, id_type, agent, source, service, verbose, no_api):
    # Create the support file for handling information about bibliographic resources
    if id_type == "doi":
        valid_doi, id_date, id_orcid, id_issn = create_csv(doi_file, date_file, orcid_file, issn_file)
        doi_manager = DOIManager(valid_doi, use_api_service=not no_api)
        crossref_rf = CrossrefResourceFinder(
            date=id_date, orcid=id_orcid, issn=id_issn, doi=valid_doi, use_api_service=not no_api)
        datacite_rf = DataCiteResourceFinder(
            date=id_date, orcid=id_orcid, issn=id_issn, doi=valid_doi, use_api_service=not no_api)
        orcid_rf = ORCIDResourceFinder(
            date=id_date, orcid=id_orcid, issn=id_issn, doi=valid_doi,
            use_api_service=True if orcid is not None and not no_api else False, key=orcid)
        rf_handler = ResourceFinderHandler([crossref_rf, datacite_rf, orcid_rf])
        return extract_citations(idbaseurl, baseurl, python, pclass, input, lookup, data, prefix, id_type,
                             agent, source, service, verbose, doi_manager, rf_handler) #qui?
    elif id_type == "pmid": ..... 
        
"""


def extract_citations(idbaseurl, baseurl, python, pclass, input, lookup, data, prefix, id_type,
                      agent, source, service, verbose, doi_manager, rf_handler, oci_to_do=None):
    """Is it possible to leave out doi_manager from the arguments and specify it as a variable which changes value in accordance with id_type value?"""
    BASE_URL = idbaseurl
    DATASET_URL = baseurl + "/" if not baseurl.endswith("/") else baseurl

    oci_manager = OCIManager(lookup_file=lookup)
    exi_ocis = CSVManager.load_csv_column_as_set(data + sep + "data", "oci")  # TODO: we need to specify carefully the dir, eg by adding an additional flag to distinguish between the files belonging to a particular process, and it should be aligned with the storer.
    if oci_to_do is not None:
        oci_to_do.difference_update(exi_ocis)
    cit_storer = CitationStorer(data, DATASET_URL)

    citations_already_present = 0
    new_citations_added = 0
    error_in_dois_existence = 0

    cs = import_citation_source(python, pclass, input)
    next_citation = cs.get_next_citation_data()

    while next_citation is not None:
        citing, cited, created, timespan, journal_sc, author_sc = next_citation
        print("the id type is:", id_type) #for now it works with doi only
        oci = oci_manager.get_oci(citing, cited, prefix, id_type)
        oci_noprefix = oci.replace("oci:", "")
        if oci_noprefix not in exi_ocis and (oci_to_do is None or oci_noprefix in oci_to_do):
            """
            if id_type == "doi":
                manager == doi_manager
            elif id_type == "pmid":
                manager == pmid_manager
            if manager.is_valid(citing) and manager.is_valid(cited): ...
            """
            if doi_manager.is_valid(citing) and doi_manager.is_valid(cited):
                if created is None:
                    citing_date = rf_handler.get_date(citing)
                else:
                    citing_date = created
                cited_date = rf_handler.get_date(cited)
                if journal_sc is None or type(journal_sc) is not bool:
                    journal_sc = rf_handler.share_issn(citing, cited)
                if author_sc is None or type(author_sc) is not bool:
                    author_sc = rf_handler.share_orcid(citing, cited)

                if created is not None and timespan is not None:
                    cit = Citation(oci,#primo parametro passato
                                   BASE_URL + quote(citing), None,
                                   BASE_URL + quote(cited), None,
                                   created, timespan,
                                   1, agent, source, datetime.now().strftime('%Y-%m-%dT%H:%M:%S'),
                                   service, id_type, BASE_URL + "([[XXX__decode]])", "reference",
                                   journal_sc, author_sc,
                                   None, "Creation of the citation", None) #id_type here substituted the previous "doi", since it is one of the parameters of extract_citation
                else:
                    cit = Citation(oci,
                                   BASE_URL + quote(citing), citing_date,
                                   BASE_URL + quote(cited), cited_date,
                                   None, None,
                                   1, agent, source, datetime.now().strftime('%Y-%m-%dT%H:%M:%S'),
                                   service, id_type, BASE_URL + "([[XXX__decode]])", "reference",
                                   journal_sc, author_sc,
                                   None, "Creation of the citation", None) #id_type here substituted the previous "doi", since it is one of the parameters of extract_citation

                cit_storer.store_citation(cit)

                if verbose:
                    print("Create citation data for '%s' between ID '%s' and ID '%s'" % (oci, citing, cited)) #changed DOI in ID
                new_citations_added += 1
                exi_ocis.add(oci_noprefix)
            else:
                if verbose:
                    print("WARNING: some IDs, among '%s' and '%s', do not exist" % (citing, cited)) #changed DOI in ID
                error_in_dois_existence += 1
            if oci_to_do is not None:
                oci_to_do.remove(oci_noprefix)
        else:
            if verbose:
                print("WARNING: the citation between ID '%s' and ID '%s' has been already processed" % #changed DOI in ID
                      (citing, cited))
            citations_already_present += 1

        next_citation = cs.get_next_citation_data()

    return new_citations_added, citations_already_present, error_in_dois_existence


if __name__ == "__main__": #since this if is outside citations, I have to specify somehow the different behaviour in case the id type is pmid
    arg_parser = ArgumentParser("cnc.py (Create New Citations",
                                description="This tool allows one to take a series of entity-to-entity"
                                            "citation data, and to store it according to CSV used by"
                                            "the OpenCitations Indexes so as to be added to an Index. It uses"
                                            "several online services to check several things to create the"
                                            "final CSV/TTL/Scholix files.")

    arg_parser.add_argument("-p", "--python", required=True,
                            help="The input Python file implementing the class index.citation.CitationSource "
                                 "which is responsible for parsing and passing all the input entity-to-entity"
                                 "citations.")
    arg_parser.add_argument("-c", "--pclass", required=True,
                            help="The name of the class implementing the class index.citation.CitationSource "
                                 "which is responsible for parsing and passing all the input entity-to-entity"
                                 "citations.")
    arg_parser.add_argument("-i", "--input", required=True,
                            help="The input file/directory to provide as input of the specified input "
                                 "Python file (using -p).")
    arg_parser.add_argument("-d", "--data", required=True,
                            help="The directory containing all the CSV files already added in the Index, "
                                 "including data and provenance files.")
    arg_parser.add_argument("-o", "--orcid", default=None,
                            help="ORCID API key to be used to query the ORCID API.")
    arg_parser.add_argument("-l", "--lookup", required=True,
                            help="The lookup table that must be used to produce OCIs.")
    arg_parser.add_argument("-b", "--baseurl", required=True, default="",
                            help="The base URL of the dataset")
    arg_parser.add_argument("-ib", "--idbaseurl", required=True, default="",
                            help="The base URL of the identifier of citing and cited entities, if any")
    """
    if id_type == "doi":
        arg_parser.add_argument("-doi", "--doi_file", default=None,
                            help="The file where the valid and invalid DOIs are stored.")
    elif id_type == "pmid":
        arg_parser.add_argument("-pmid", "--pmid_file", default=None,
                            help="The file where the valid and invalid PMIDs are stored.")
    """
    #this solution, anyway, is not suitable, since id_type isn't specified anywhere. 
    
    arg_parser.add_argument("-doi", "--doi_file", default=None,
                            help="The file where the valid and invalid DOIs are stored.") 
    arg_parser.add_argument("-date", "--date_file", default=None,
                            help="The file that maps id of bibliographic resources with their publication date.")
    arg_parser.add_argument("-orcid", "--orcid_file", default=None,
                            help="The file that maps id of bibliographic resources with the ORCID of its authors.")
    arg_parser.add_argument("-issn", "--issn_file", default=None,
                            help="The file that maps id of bibliographic resources with the ISSN of the journal "
                                 "they have been published in.")
    arg_parser.add_argument("-px", "--prefix", default="",
                            help="The '0xxx0' prefix to use for creating the OCIs.")
    arg_parser.add_argument("-a", "--agent", required=True, default="https://w3id.org/oc/index/prov/pa/1",
                            help="The URL of the agent providing or processing the citation data.")
    arg_parser.add_argument("-s", "--source", required=True,
                            help="The URL of the source from where the citation data have been extracted.")
    arg_parser.add_argument("-sv", "--service", required=True,
                            help="The name of the service that will made available the citation data.")
    arg_parser.add_argument("-v", "--verbose", action="store_true", default=False,
                            help="Print the messages on screen.")
    arg_parser.add_argument("-na", "--no_api", action="store_true", default=False,
                            help="Tell the tool explicitly not to use the APIs of the various finders.")
    arg_parser.add_argument("-pn", "--process_number", default=1, type=int,
                            help="The number of parallel process to run for working on the creation of citations.")

    args = arg_parser.parse_args()
    n_processes = args.process_number
    if n_processes min= 1: #HERE I SUBSTITUTE THE SYMBOL WITH THE WORD
        new_citations_added, citations_already_present, error_in_dois_existence = \
            execute_workflow(args.idbaseurl, args.baseurl, args.python, args.pclass, args.input, args.doi_file,
                             args.date_file, args.orcid_file, args.issn_file, args.orcid, args.lookup, args.data,
                             args.prefix, args.id_type, args.agent, args.source, args.service, args.verbose, args.no_api)
        """
        I had some problems with this if __name__ == "__main__":
        - How can I handle this part:
         arg_parser.add_argument("-doi", "--doi_file", default=None,
                            help="The file where the valid and invalid DOIs are stored.")
        - Does it makes sense to keep args.id_type?
        - Is there a way to make an argument change in accordance to the value assumed by another one? Id est: can I make args.doi_file turn into 
        args.pmid_file if id_type is pmid?
        
        """
    else:  # Run in parallel
        pass  # TODO: do things

    print("\n# Summary\n"
          "Number of new citations added to the OpenCitations Index: %s\n"
          "Number of citations already present in the OpenCitations Index: %s\n"
          "Number of citations with invalid DOIs: %s" %
          (new_citations_added, citations_already_present, error_in_dois_existence))

# How to call the service (e.g. for COCI)
# python -m index.cnc -ib "http://dx.doi.org/" -b "https://w3id.org/oc/index/coci/" -p "index/citation/citationsource.py" -c "CSVFileCitationSource" -i "index/test_data/citations_partial.csv" -doi "index/coci_test/doi.csv" -orcid "index/coci_test/orcid.csv" -date "index/coci_test/date.csv" -issn "index/coci_test/issn.csv" -l "index/test_data/lookup_full.csv" -d "index/coci_test" -px "020" -a "https://w3id.org/oc/index/prov/pa/1" -s "https://api.crossref.org/works/[[citing]]" -sv "OpenCitations Index: COCI" -v

                </pre>
            </code>
          </div>
          
          <p>How can I call the service for NOCI? How should I generate my test files?</p>
          
          <h2>07_cnc</h2>
          <p> I implemented the needed class for managing pmid. However, I didn't understand how to generate files I should use as input.</p>
          <p> I took for granted that test data for pmid are going to be stored in a distinct files, i.e.: citations_prov_pmid.csv, citations_data_pmid.csv and cnc_valid_pmid.csv. I'd need to discuss how to create or generate these files.</p>
          <p>What I have for now is just an initial idea which still needs to be corrected and perfectioned a lot. </p>
          <div class="codebg">
            <code>
                <pre>
                class CreateNewCitationsTestPmid(unittest.TestCase):
    def setUp(self):
        self.idbaseurl = "https://pubmed.ncbi.nlm.nih.gov/" #serve specificare in qualche modo /?format=pmid dopo l'id?
        self.baseurl = "https://w3id.org/oc/index/noci/" #ripprta a opencitations, ora non esiste
        self.python = "index%scitation%scitationsource.py" % (sep, sep)
        self.pclass = "CSVFileCitationSource" #permetto all'utente di passare file python e classe che devo usare. python ha una caratteristica
        #che è la reflection. crea un oggetto di una classe dinamicamente. posso dire al mio softw di istanziarmi un oggetto di qualsiasi classe
        #mi interessi passando quanto detto dalla linea di comando. devono essere conformi alla stessa struttura. noci, coci e croci seguono questo.
        #IL problema è che estendendo la logica dei vari oggetti che fanno settuple, bisogna andare a modificare anche in citation source in modo che ritorno
        #dappertutto delle settuple. solo che questo csvfilecitationsource è generico.
        self.input = "index%stest_data%scitations_partial_pmid.csv" % (sep, sep) #come faccio a generare l'oci per il test^
        self.pmid_file = "index%stest_data%scnc_valid_pmid.csv" % (sep, sep)
        self.date_file = "index%stest_data%scnc_id_date.csv" % (sep, sep)
        self.orcid_file = "index%stest_data%scnc_id_orcid.csv" % (sep, sep)
        self.issn_file = "index%stest_data%scnc_id_issn.csv" % (sep, sep)
        self.orcid = None
        self.lookup = "index%stest_data%slookup_full.csv" % (sep, sep)
        self.data = "index%stest_data%stmp_workflow" % (sep, sep)
        self.prefix = "0160"
        self.agent = "https://w3id.org/oc/index/prov/ra/1"
        self.source = "https://doi.org/10.35092/yhjc.c.4586573.v16/[[citing]]" #la sorgente da cambiare. è il dataset del NIH quindi un file. passa il doi url del dataset
        #ho passato iCite Database Snapshots (NIH Open Citation Collection)
        self.service = "OpenCitations Index: NOCI"
        self.verbose = True
        self.no_api = False

        self.citation_list = self.__load_citations("index%stest_data%scitations_data_pmid.csv" % (sep, sep),
                                                   "index%stest_data%scitations_prov_pmid.csv" % (sep, sep))
        self.data_path = self.data + sep + "data" + sep + "**" + sep + "*.csv"
        self.prov_path = self.data + sep + "prov" + sep + "**" + sep + "*.csv"

        if exists(self.data):
            rmtree(self.data)

    def __load_citations(self, data, prov):
        return CitationStorer.load_citations_from_file(data, prov, baseurl="https://pubmed.ncbi.nlm.nih.gov/",
            service_name=self.service, id_type="pmid",
            id_shape="https://pubmed.ncbi.nlm.nih.gov/([[XXX__decode]])", citation_type=None) #NOTE: discuss id_shape

    def __citations_csv(self, origin_citation_list, stored_citation_list):
        l1 = [cit.get_citation_csv() for cit in origin_citation_list]
        l2 = [cit.get_citation_csv() for cit in stored_citation_list]
        self.assertEqual(len(l1), len(l2))
        self.assertEqual(set(l1), set(l2))

    def __test_citations(self):
        data_csv = glob(self.data_path, recursive=True)
        prov_csv = glob(self.prov_path, recursive=True)
        self.assertEqual(len(data_csv), 1)
        self.assertEqual(len(prov_csv), 1)
        self.__citations_csv(self.citation_list, self.__load_citations(data_csv[0], prov_csv[0]))

    def test_execute_workflow(self):
        new_citations_added, citations_already_present, error_in_dois_existence = \
            execute_workflow(self.idbaseurl, self.baseurl, self.python, self.pclass, self.input, self.pmid_file,
                             self.date_file, self.orcid_file, self.issn_file, self.orcid, self.lookup, self.data,
                             self.prefix, OCIManager.pmid_type, self.agent, self.source, self.service, self.verbose, self.no_api)
        self.assertEqual(new_citations_added, 6)
        self.assertEqual(citations_already_present, 0)
        self.assertEqual(error_in_dois_existence, 0)
        self.__test_citations()

        new_citations_added, citations_already_present, error_in_dois_existence = \
            execute_workflow(self.idbaseurl, self.baseurl, self.python, self.pclass, self.input, self.pmid_file,
                             self.date_file, self.orcid_file, self.issn_file, self.orcid, self.lookup, self.data,
                             self.prefix, OCIManager.pmid_type, self.agent, self.source, self.service, self.verbose, self.no_api) #qui?
        self.assertEqual(new_citations_added, 0)
        self.assertEqual(citations_already_present, 6)
        self.assertEqual(error_in_dois_existence, 0)
        self.__test_citations()
                </pre>
            </code>
          </div>

      </div>
    </div>
  </div>
  
  <div class="card">
    <div class="card-header" id="headingTwelve">
      <h5 class="mb-0">
        <button class="btn btn-link collapsed" data-toggle="collapse" data-target="#collapseTwelve" aria-expanded="false" aria-controls="collapseTwelve">
          Week #12 (04-22-21)
        </button>
      </h5>
    </div>
    <div id="collapseTwelve" class="collapse" aria-labelledby="headingTwelve" data-parent="#accordion">
      <div class="card-body">
          
          <h2>citationsource.py</h2>
          <ul>
            <li><strong>class CSVFileCitationSource(DirCitationSource):</strong> When it is instantiated in the test, it is necessary to pass in input also the id type related to the CSV in question. It is necessary to create a csv file also for PMID. </li>
            <li><strong>return citing, cited, created, timespan, journal_sc, author_sc, self.id_type</strong>: It is necessary to pass the self.id_type correctly; the function will return it as it was passed by the user.</li>
          </ul>
          <p>During the last meeting, the function __init__(self, src, id_type=None) in CitationSource was extended so to let the user specify the id_type of the citation data used in the source file: </p>
          <div class="codebg">
            <code>
                <pre>
    def __init__(self, src, id_type=None):
        self.src = src
        self.id_type = id_type
                </pre>
            </code>
          </div>
          <p>Accordingly, I extended the comment to the function __init__(self, src, id_type=None) in CitationSource by adding an explanation about the use of the id_type variable, similar to the one describing the variable src: <i>"""The constructor allows to associate to the variable 'src' a kind of source,
        that will be used to retrieve citation data, and to the variable 'id_type' the
        type of identifier used in the citation data, i.e. either DOIs or PMIDs."""</i> </p>
        <p><strong>Note</strong>:  CitationSource(object) is the generic class that all the citationsources have to implement. It defines a constructor and a method which is not implemented, since it is to be implemented by each specific CitationSource (i.e.: the one for COCI, CROCI and NOCI).</p>
        
        <h3>Issues</h3>
          <ul>
            <li>The reason why the id_type is initialised in this way is to handle the possibility that the user doesn't specify it, is it correct?</li>
            <li>I was asked to create a staticmethod to give the possibility to overwrite the value of the variable id_type. This was the request: "provide a staic method to change the value of that variable: set_id_type, which takes in input self, id_type (was it self.id_type??), so to overwrite the initial value. The class needs a mechanism to specify the specific type of id." However, the problem is that, first of all, I haven't a clear idea of why is this staticmethod strictly necessary: Isn't it enough to pass in input the correct id type? (i.e. id_type=OCIManager.doi_type or OCIManager.pmid_type when needed?, second, what is supposed to return this staticmethod? how should it work? In which occasions a staticmethod is necessary?)</li>
            <li>Discuss the function of the following line of code:                 self.status_file = cur_dir + sep + ".dir_citation_source" + local_name #?
</li>
          </ul>
          
          
          <h2>06_citationsource.py</h2>
          <ul>
            <li>check the file and transpose the notes.</li>
          </ul>
          <p>First of all, I divided the function test_get_next_citation_data(self) in two versions: test_get_next_citation_data_doi(self) and test_get_next_citation_data_pmid(self), accordingly with the changes to the class CitationSource(object) in citationsource.py. Now, since I am instantiating the tests for two functions of the class CSVFileCitationSource(DirCitationSource), I'll have to pass in input also the id_type. </p>
          
          <h3>Issues</h3>
          <ul>
            <li>The reason why I am passing OCIManager.doi_type and OCIManager.pmid_type as parameters of CSVFileCitationSource in the two test functions for get_next_citation_data is that the class CSVFileCitationSource(DirCitationSource) takes in input DirCitationSource(CitationSource), which takes in input CitationSource(object), where the __init__(self, src, id_type=None) function takes in input src and the id_type. Am I correct?</li>
            <li> For now we have two versions of test_get_next_citation_data only. May it make sense the creation of two whole separate classes for the unittest? (one for pmids and one for dois, since the most of the functions should otherwise handle many conditional statements, splitting the process for dois and pmid). In this way the citations would be also stored separately. </li>
            <li>__create_citation : Discuss how does it work in detail and understand where does it store the citations (tmp_store?, and what about the inner folder?).</li>
          </ul>
          
          <h2>cnc.py</h2>
          <ul>
            <li>check the file and transpose the the notes.</li>
          </ul>
          <p> I generalised the variables' names where necessary and added an if-else structure in execute_workflow. </p>
          
          <div class = "codebg">
            <code>
                <pre>
                
from argparse import ArgumentParser
from urllib.parse import quote
from datetime import datetime
from os.path import abspath, dirname, basename
from os import sep
from sys import path
from importlib import import_module
from index.storer.csvmanager import CSVManager
from index.identifier.doimanager import DOIManager
form index.identifier.pmidmanager import PMIDManager
from index.finder.orcidresourcefinder import ORCIDResourceFinder
from index.finder.dataciteresourcefinder import DataCiteResourceFinder
from index.finder.crossrefresourcefinder import CrossrefResourceFinder
from index.finder.nationalinstituteofhealthresourcefinder import NationalInstituteOfHealthResourceFinder #imported
from index.finder.resourcefinder import ResourceFinderHandler
from index.citation.oci import OCIManager, Citation
from index.storer.citationstorer import CitationStorer


def execute_workflow_parallel():
def execute_workflow_parallel():
    pass  # TODO


def create_csv(id_file, date_file, orcid_file, issn_file): #id_file instead of doi_file
    valid_id = CSVManager(csv_path=id_file)
    id_date = CSVManager(csv_path=date_file)
    id_orcid = CSVManager(csv_path=orcid_file)
    id_issn = CSVManager(csv_path=issn_file)

    return valid_id, id_date, id_orcid, id_issn


def import_citation_source(python, pclass, input):
    addon_abspath = abspath(python)
    path.append(dirname(addon_abspath))
    addon = import_module(basename(addon_abspath).replace(".py", ""))
    return getattr(addon, pclass)(input)


def execute_workflow(idbaseurl, baseurl, python, pclass, input, id_file, date_file,
                     orcid_file, issn_file, orcid, lookup, data, prefix, id_type, agent, source, service, verbose, no_api): #how to specify the id_type?
    #is it something to be extracted automatically (so which should be present in the files) or which can be specified manually?
    # Create the support file for handling information about bibliographic resources: do they share the same journal or the same publisher?
    valid_id, id_date, id_orcid, id_issn = create_csv(id_file, date_file, orcid_file, issn_file)

    if id_type == OCIManager.doi_type:
        doi_manager = DOIManager(valid_id, use_api_service=not no_api)
        crossref_rf = CrossrefResourceFinder(
            date=id_date, orcid=id_orcid, issn=id_issn, doi=valid_id, use_api_service=not no_api)
        datacite_rf = DataCiteResourceFinder(
            date=id_date, orcid=id_orcid, issn=id_issn, doi=valid_id, use_api_service=not no_api)
        orcid_rf = ORCIDResourceFinder(
            date=id_date, orcid=id_orcid, issn=id_issn, doi=valid_id,
            use_api_service=True if orcid is not None and not no_api else False, key=orcid)

        rf_handler = ResourceFinderHandler([crossref_rf, datacite_rf, orcid_rf])
        return extract_citations( idbaseurl, baseurl, python, pclass, input, lookup, data, prefix, id_type,
                                  agent, source, service, verbose, doi_manager, rf_handler )

    elif id_type == OCIManager.pmid_type:
        pmid_manager = PMIDManager(valid_id, use_api_service=not no_api)
        nationalinstituteofhealth_rf = NationalInstituteOfHealthResourceFinder(
            date=id_date, orcid=id_orcid, issn=id_issn, pmid=valid_id, use_api_service=not no_api)

        rf_handler = ResourceFinderHandler([nationalinstituteofhealth_rf])
        return extract_citations( idbaseurl, baseurl, python, pclass, input, lookup, data, prefix, id_type,
                                  agent, source, service, verbose, pmid_manager, rf_handler )

def extract_citations(idbaseurl, baseurl, python, pclass, input, lookup, data, prefix, id_type,
                      agent, source, service, verbose, id_manager, rf_handler, oci_to_do=None): ##id_manager instead of doi_manager
    """Is it possible to leave out doi_manager from the arguments and specify it as a variable which changes value in accordance with id_type value?"""
    BASE_URL = idbaseurl
    DATASET_URL = baseurl + "/" if not baseurl.endswith("/") else baseurl

    oci_manager = OCIManager(lookup_file=lookup)
    exi_ocis = CSVManager.load_csv_column_as_set(data + sep + "data", "oci")  # TODO: we need to specify carefully the dir, eg by adding an additional flag to distinguish between the files belonging to a particular process, and it should be aligned with the storer.
    if oci_to_do is not None:
        oci_to_do.difference_update(exi_ocis)
    cit_storer = CitationStorer(data, DATASET_URL)

    citations_already_present = 0
    new_citations_added = 0
    error_in_ids_existence = 0 #error_in_ids_existence

    cs = import_citation_source(python, pclass, input)
    next_citation = cs.get_next_citation_data()

    while next_citation is not None:
        citing, cited, created, timespan, journal_sc, author_sc = next_citation
        print("the id type is:", id_type) 
        oci = oci_manager.get_oci(citing, cited, prefix, id_type)
        oci_noprefix = oci.replace("oci:", "")
        if oci_noprefix not in exi_ocis and (oci_to_do is None or oci_noprefix in oci_to_do):

            if id_manager.is_valid(citing) and id_manager.is_valid(cited):
                if created is None:
                    citing_date = rf_handler.get_date(citing)
                else:
                    citing_date = created
                cited_date = rf_handler.get_date(cited)
                if journal_sc is None or type(journal_sc) is not bool:
                    journal_sc = rf_handler.share_issn(citing, cited)
                if author_sc is None or type(author_sc) is not bool:
                    author_sc = rf_handler.share_orcid(citing, cited)

                if created is not None and timespan is not None:
                    cit = Citation(oci,
                                   BASE_URL + quote(citing), None,
                                   BASE_URL + quote(cited), None,
                                   created, timespan,
                                   1, agent, source, datetime.now().strftime('%Y-%m-%dT%H:%M:%S'),
                                   service, id_type, BASE_URL + "([[XXX__decode]])", "reference",
                                   journal_sc, author_sc,
                                   None, "Creation of the citation", None)
                else:
                    cit = Citation(oci,
                                   BASE_URL + quote(citing), citing_date,
                                   BASE_URL + quote(cited), cited_date,
                                   None, None,
                                   1, agent, source, datetime.now().strftime('%Y-%m-%dT%H:%M:%S'),
                                   service, id_type, BASE_URL + "([[XXX__decode]])", "reference",
                                   journal_sc, author_sc,
                                   None, "Creation of the citation", None)

                cit_storer.store_citation(cit)

                if verbose:
                    print("Create citation data for '%s' between ID '%s' and ID '%s'" % (oci, citing, cited)) #changed DOI in ID
                new_citations_added += 1
                exi_ocis.add(oci_noprefix)
            else:
                if verbose:
                    print("WARNING: some IDs, among '%s' and '%s', do not exist" % (citing, cited)) #changed DOI in ID
                error_in_ids_existence += 1 #error_in_ids_existence
            if oci_to_do is not None:
                oci_to_do.remove(oci_noprefix)
        else:
            if verbose:
                print("WARNING: the citation between ID '%s' and ID '%s' has been already processed" % #changed DOI in ID
                      (citing, cited))
            citations_already_present += 1

        next_citation = cs.get_next_citation_data()

    return new_citations_added, citations_already_present, error_in_ids_existence #error_in_ids_existence


if __name__ == "__main__":
    arg_parser = ArgumentParser("cnc.py (Create New Citations",
                                description="This tool allows one to take a series of entity-to-entity"
                                            "citation data, and to store it according to CSV used by"
                                            "the OpenCitations Indexes so as to be added to an Index. It uses"
                                            "several online services to check several things to create the"
                                            "final CSV/TTL/Scholix files.")

    arg_parser.add_argument("-p", "--python", required=True,
                            help="The input Python file implementing the class index.citation.CitationSource "
                                 "which is responsible for parsing and passing all the input entity-to-entity"
                                 "citations.")
    arg_parser.add_argument("-c", "--pclass", required=True,
                            help="The name of the class implementing the class index.citation.CitationSource "
                                 "which is responsible for parsing and passing all the input entity-to-entity"
                                 "citations.")
    arg_parser.add_argument("-i", "--input", required=True,
                            help="The input file/directory to provide as input of the specified input "
                                 "Python file (using -p).")
    arg_parser.add_argument("-d", "--data", required=True,
                            help="The directory containing all the CSV files already added in the Index, "
                                 "including data and provenance files.")
    arg_parser.add_argument("-o", "--orcid", default=None,
                            help="ORCID API key to be used to query the ORCID API.")
    arg_parser.add_argument("-l", "--lookup", required=True,
                            help="The lookup table that must be used to produce OCIs.")
    arg_parser.add_argument("-b", "--baseurl", required=True, default="",
                            help="The base URL of the dataset")
    arg_parser.add_argument("-ib", "--idbaseurl", required=True, default="",
                            help="The base URL of the identifier of citing and cited entities, if any")
    arg_parser.add_argument("-id", "--id_file", default=None,
                            help="The file where the valid and invalid IDs are stored.")  #id_file
    arg_parser.add_argument("-date", "--date_file", default=None,
                            help="The file that maps id of bibliographic resources with their publication date.")
    arg_parser.add_argument("-orcid", "--orcid_file", default=None,
                            help="The file that maps id of bibliographic resources with the ORCID of its authors.")
    arg_parser.add_argument("-issn", "--issn_file", default=None,
                            help="The file that maps id of bibliographic resources with the ISSN of the journal "
                                 "they have been published in.")
    arg_parser.add_argument("-px", "--prefix", default="",
                            help="The '0xxx0' prefix to use for creating the OCIs.")
    arg_parser.add_argument("-a", "--agent", required=True, default="https://w3id.org/oc/index/prov/pa/1",
                            help="The URL of the agent providing or processing the citation data.")
    arg_parser.add_argument("-s", "--source", required=True,
                            help="The URL of the source from where the citation data have been extracted.")
    arg_parser.add_argument("-sv", "--service", required=True,
                            help="The name of the service that will made available the citation data.")
    arg_parser.add_argument("-v", "--verbose", action="store_true", default=False,
                            help="Print the messages on screen.")
    arg_parser.add_argument("-na", "--no_api", action="store_true", default=False,
                            help="Tell the tool explicitly not to use the APIs of the various finders.")
    arg_parser.add_argument("-pn", "--process_number", default=1, type=int,
                            help="The number of parallel process to run for working on the creation of citations.")

    args = arg_parser.parse_args()
    n_processes = args.process_number
    if n_processes (#min or equal to#)) 1:
        new_citations_added, citations_already_present, error_in_ids_existence = \
            execute_workflow(args.idbaseurl, args.baseurl, args.python, args.pclass, args.input, args.id_file,
                             args.date_file, args.orcid_file, args.issn_file, args.orcid, args.lookup, args.data,
                             args.prefix, args.id_type, args.agent, args.source, args.service, args.verbose, args.no_api) # ID_FILE, error_in_ids_existence

    else:  # Run in parallel
        pass  # TODO: do things

    print("\n# Summary\n"
          "Number of new citations added to the OpenCitations Index: %s\n"
          "Number of citations already present in the OpenCitations Index: %s\n"
          "Number of citations with invalid DOIs: %s" %
          (new_citations_added, citations_already_present, error_in_ids_existence)) #error_in_ids_existence

# How to call the service (e.g. for COCI)
# python -m index.cnc -ib "http://dx.doi.org/" -b "https://w3id.org/oc/index/coci/" -p "index/citation/citationsource.py" -c "CSVFileCitationSource" -i "index/test_data/citations_partial.csv" -doi "index/coci_test/doi.csv" -orcid "index/coci_test/orcid.csv" -date "index/coci_test/date.csv" -issn "index/coci_test/issn.csv" -l "index/test_data/lookup_full.csv" -d "index/coci_test" -px "020" -a "https://w3id.org/oc/index/prov/pa/1" -s "https://api.crossref.org/works/[[citing]]" -sv "OpenCitations Index: COCI" -v

                </pre>
            </code>
          </div>
          
          <h2>07_cnc.py</h2>
          <ul>
            <li>check the file and transpose the the notes.</li>
            <li>create the required support files.</li>
          </ul>
          <p> I manually created the following files: citations_data_pmid.csv, citations_partial_pmid.csv, citations_prov_pmid.csv, cnc_id_date_pmid.csv, cnc_id_issn_pmid.csv, cnc_valid_pmid.csv, valid_pmid.csv. Accordingly, I updated the two testing classes. </p>
          <p>I'd need to discuss the functioning of <strong>citationstorer</strong> and understand how does it create and fill the folders with citation files.</p>
          
          <h2>oci.py</h2>
          <ul>
            <li>check the file and transpose the notes.</li>
          </ul>
          
          <h2>resourcefinder.py</h2>
          <ul>
            <li>read the file so to understand how to implement the version for NIH.</li>
          </ul>
          <p> Also in this case, how to handle PMID? I implemented a class APIPMIDResourceFinder, but its just a temporary solution to be discussed. Indeed, the problem is in the ResourceFinder, since it does not handle the possibility of having an identfier which is not a doi (can it be handled like in citationsource?).</p>
          <h2>nationalinstituteofhealthresourcefinder.py</h2>
          <ul>
            <li>create the file.</li>
            <li>Use as a model the crossrefresourcefinder.py (emulate its functions)</li>
          </ul>
          <p> I created the file with the class NationalInstituteOfHealthResourceFinder, where  i specified "https://pubmed.ncbi.nlm.nih.gov/" as API. I then took notes on where to intervene for the changes dependent on the resourcefinder updates to be implemented.</p>
          
          <h2>03_resourcefinder.py</h2>
          <ul>
            <li>update the file by adding the piece of code to test nationalinstituteofhealthresourcefinder.py</li>
          </ul>
          <p> In order to handle also pmid, should I extend the files id_issn.csv, id_orcid.csv and id_date.csv or should I create new versions of the same files for pmid only? Further, since in the dataset we imported the citational data from this information is not provided, am I supposed to invent them? or should I just assign empty strings as values to the identifiers?</p>
          <p> I prepared the structure of the functions test_nationalinstititeofhealth_get_orcid(self), test_nationalinstititeofhealth_get_issn(self), test_nationalinstititeofhealth_get_pub_date(self), that I'll implement when I'll know better how to structure the support files.</p>
          
          
          
          
          <h2>Transcribe OpenCitations commits</h2>
          <ul>
            <li>Update the code in accordance with the commits in OpenCitations</li>
          </ul>
          <p>The code was updated: now my local version of the software reflects the latest updates of the official Software of OpenCitations (commit :"added encoding set to utf8 everywhere")</p>
          
          <h2>DARCH intership request</h2>
          <ul>
            <li>Send the request for the intership</li>
          </ul>
          <p>The intership request was sent, now pending approval. Details: Intership request n. 565309; notes: "Richiesta di tirocinio per lavoro di tesi già avviato, all'interno del progetto OpenCitations. L'attività si incentra sull'importazione di dati citazionali provenienti dal National Institute of Health, tra pubblicazioni identificate da PMIDs. Il  lavoro del tirocinio, in particolare, riguarda l'analisi del software esistente per la creazione di dati citazionali in OpenCitations e la proposta di possibili estensioni per gestire le citazioni che intercorrono tra articoli identificati da PubMed Ids".</p>

      </div>
    </div>
  </div>
  
      <div class="card">
    <div class="card-header" id="headingThirteen">
      <h5 class="mb-0">
        <button class="btn btn-link collapsed" data-toggle="collapse" data-target="#collapseThirteen" aria-expanded="false" aria-controls="collapseThirteen">
          Week #13 (05-13-21)
        </button>
      </h5>
    </div>
    <div id="collapseThirteen" class="collapse" aria-labelledby="headingThirteen" data-parent="#accordion">
      <div class="card-body">
          
          <h2>resourcefinder.py generalisation</h2>
          <p>The file was modified so to handle either the possibility that the ID is a DOI or a PMID. In particular, the name of the class ApiDOIResourceFinder(ResourceFinder) was changed in ApiIDResourceFinder(ResourceFinder), all the names of the variables containing "doi" were generalised, and the argument "id_type" was added in all the functions were it was needed an if-else statement in order to split the process in accordance with the type of id. </p>
          
          <div class="codebg">
            <code>
                <pre>
from index.storer.csvmanager import CSVManager
from index.identifier.orcidmanager import ORCIDManager
from index.identifier.doimanager import DOIManager
from index.identifier.pmidmanager import PMIDManager
from index.identifier.issnmanager import ISSNManager
from index.citation.oci import OCIManager

from collections import deque
# TODO: For multiprocessing purposes
# from multiprocessing.managers import BaseManager


class ResourceFinder(object):
    """This is the abstract class that must be implemented by any resource finder
    for a particular service (Crossref, DataCite, ORCiD, etc.). It provides
    the signatures of the methods that should be implemented, and a basic
    constructor."""

    def __init__(self, date=None, orcid=None, issn=None, id=None, id_type=None, **params): #vedere se crea problemi l'aggiunta di id_type
        if date is None:
            date = CSVManager(store_new=False)
        if orcid is None:
            orcid = CSVManager(store_new=False)
        if issn is None:
            issn = CSVManager(store_new=False)
        if id is None:
            id = CSVManager(store_new=False)

        for key in params:
            setattr(self, key, params[key])

        self.issn = issn
        self.date = date
        self.orcid = orcid
        self.id_type = id_type
        if hasattr(self, 'use_api_service'):
            if id_type is OCIManager.doi_type:
                self.dm = DOIManager(id, self.use_api_service)
            elif id_type is OCIManager.pmid_type:
                self.pm = PMIDManager(id, self.use_api_service)
            else:
                print("The id_type specified is not compliant")
        self.im = ISSNManager()
        self.om = ORCIDManager()

        self.headers = {
            "User-Agent": "ResourceFinder / OpenCitations Indexes "
                          "(http://opencitations.net; mailto:contact@opencitations.net)"
        }

        # TODO: For multiprocessing purposes
        # c_type = type(self)
        # BaseManager.register(c_type.__name__, c_type)

    def get_orcid(self, id_string):
        pass

    def get_pub_date(self, id_string):
        pass

    def get_container_issn(self, id_string):
        pass

    def is_valid(self, id_string):
        pass

    def normalise(self, id_string):
        pass

class ApiIDResourceFinder(ResourceFinder): #The name of the class was changed
    """This is the abstract class that must be implemented by any resource finder
        for a particular service which is based on DOI retrieving via HTTP REST APIs
        (Crossref, DataCite). It provides basic methods that are be used for
        implementing the main methods of the ResourceFinder abstract class."""

    # The following four methods are those ones that should be implemented in
    # the concrete subclasses of this abstract class.
    def _get_date(self, json_obj):
        pass

    def _get_issn(self, json_obj):
        return set()

    def _get_orcid(self, json_obj):
        return set()

    def _call_api(self, id_full):
        pass

    # The implementation of the following methods is strictly dependent on the actual
    # implementation of the previous three methods, since they strictly reuse them
    # for returning the result.
    def get_orcid(self, id_string):
        return self._get_item(id_string, self.orcid)

    def get_pub_date(self, id_string):
        return self._get_item(id_string, self.date)

    def get_container_issn(self, id_string):
        return self._get_item(id_string, self.issn)

    def is_valid(self, id_string, id_type):
        if id_type is OCIManager.doi_type:
            return self.dm.is_valid(id_string)
        elif id_type is OCIManager.pmid_type:
            return self.pm.is_valid(id_string)
        else:
            print("The id_type specified is not compliant")



    def normalise(self, id_string, id_type):
        if id_type is OCIManager.doi_type:
            return self.dm.normalise(id_string, include_prefix=True)
        elif id_type is OCIManager.pmid_type:
            return self.pm.normalise(id_string, include_prefix=True)
        else:
            print("The id_type specified is not compliant")


    def _get_item(self, id_entity, csv_manager):
        if self.is_valid(id_entity):
            id = self.normalise(id_entity)

            if csv_manager.get_value(id) is None:
                json_obj = self._call_api(id)

                if json_obj is not None:
                    for issn in self._get_issn(json_obj):
                        self.issn.add_value(id, issn)

                    if self.date.get_value(id) is None:
                        pub_date = self._get_date(json_obj)
                        if pub_date is not None:
                            self.date.add_value(id, pub_date)

                    for orcid in self._get_orcid(json_obj):
                        self.orcid.add_value(id, orcid)

            return csv_manager.get_value(id)


class ResourceFinderHandler(object):
    """This class allows one to use multiple resource finders at the same time
    so as to find the information needed for the creation of the citations to
    include in the index."""

    def __init__(self, resource_finders):
        self.resource_finders = resource_finders

    def get_date(self, id_string):
        result = None
        finders = deque(self.resource_finders)

        while result is None and finders:
            finder = finders.popleft()
            result_set = finder.get_pub_date(id_string)
            if result_set:
                result = result_set.pop()

        if result is None:  # Add the empty value in all the finders
            for finder in self.resource_finders:
                if finder.is_valid(id_string):
                    finder.date.add_value(finder.normalise(id_string), "")

        return result

    def share_issn(self, id_string_1, id_string_2):
        return self.__share_data(id_string_1, id_string_2, "get_container_issn")

    def share_orcid(self, id_string_1, id_string_2):
        return self.__share_data(id_string_1, id_string_2, "get_orcid")

    def __share_data(self, id_string_1, id_string_2, method):
        result = False
        finders = deque(self.resource_finders)
        set_1 = set()
        set_2 = set()

        while not result and finders:
            finder = finders.popleft()

            result_set_1 = getattr(finder, method)(id_string_1)
            if result_set_1:
                set_1.update(result_set_1)

            result_set_2 = getattr(finder, method)(id_string_2)
            if result_set_2:
                set_2.update(result_set_2)

            result = len(set_1.intersection(set_2)) > 0

        return result

                </pre>
            </code>
          </div>
          
          <h2>citationstorer.py</h2>
          <p>The overall structure seems to be neutral and thus reusable also for pmid. However,  load_citations_from_file is initialised with the following arguments: (data_f_path, prov_f_path=None, oci="04201-04201", baseurl="",
                                 service_name="CitationStorer", id_type="doi",
                                 id_shape="http://dx.doi.org/([[XXX__decode]])",
                                 citation_type="", agent="", source=""). Should I implement an alternative static method to call when it's up to PMIDs or just generalise these parameters? </p>
                                 
         <h2>05_citationstorer.py</h2>
         <p>How to handle the test? If there is the necessity to create a separate class? The procedure is pretty much neutral (except for some passages in which it would be useful to specify the id_type among the parameters), However, if the paths for doi data and pmid data have to stay separated, in which way should the SetUp be handled? </p>


      </div>
    </div>
  </div>
  
  <div class="card">
    <div class="card-header" id="headingFourteen">
      <h5 class="mb-0">
        <button class="btn btn-link collapsed" data-toggle="collapse" data-target="#collapseFourteen" aria-expanded="false" aria-controls="collapseFourteen">
          Week #14 (05-20-21)
        </button>
      </h5>
    </div>
    <div id="collapseFourteen" class="collapse" aria-labelledby="headingFourteen" data-parent="#accordion">
      <div class="card-body">
          
          <h2>nihresourcefinder.py</h2>
          <h3>Documentation on PubMed Format</h3>
          <p> Information on the acronyms used in PubMed Format is provided at <a href="https://pubmed.ncbi.nlm.nih.gov/help/#understanding-docsum">https://pubmed.ncbi.nlm.nih.gov/help/#understanding-docsum</a> </p>
          <p><i>The PubMed Format tags table defines the data tags that compose the PubMed format. The tags are presented in alphabetical order. Some of the tags (e.g., CIN) are not mandatory and therefore will not be found in every record. Other tags (e.g., AU, MH, and RN) may occur multiple times in one record. You can download records in PubMed format as a text file (.txt) or as an .nbib file for exporting into citation management software programs.</i></p>
          
          <table tabindex="0">
      <tbody><tr>
        <th scope="colgroup" colspan="3">PubMed Format tags</th>
      </tr>
      <tr>
        <th scope="col">Tag</th>
        <th scope="col">Name</th>
        <th scope="col">Description</th>
      </tr>
      <tr>
        <td>AB</td>
        <td>Abstract</td>
        <td>English language abstract taken directly from the published article</td>
      </tr>
      <tr>
        <td>AD</td>
        <td>Affiliation</td>
        <td>Author or corporate author addresses</td>
      </tr>
      <tr>
        <td>AID</td>
        <td>Article Identifier</td>
        <td>Article ID values supplied by the publisher may include the pii (controlled publisher identifier), doi (digital object identifier), or book accession</td>
      </tr>
      <tr>
        <td>AU</td>
        <td>Author</td>
        <td>Authors</td>
      </tr>
      <tr>
        <td>BTI</td>
        <td>Book Title</td>
        <td>Book Title</td>
      </tr>
      <tr>
        <td>CI</td>
        <td>Copyright Information</td>
        <td>Copyright statement provided by the publisher</td>
      </tr>
      <tr>
        <td>CIN</td>
        <td>Comment In</td>
        <td>Reference containing a comment about the article</td>
      </tr>
      <tr>
        <td>CN</td>
        <td>Corporate Author</td>
        <td>Corporate author or group names with authorship responsibility</td>
      </tr>
      <tr>
        <td>COI</td>
        <td>Conflict of Interest</td>
        <td>Conflict of interest statement</td>
      </tr>
      <tr>
        <td>CON</td>
        <td>Comment On</td>
        <td>Reference upon which the article comments</td>
      </tr>
      <tr>
        <td>CP</td>
        <td>Chapter</td>
        <td>Book chapter</td>
      </tr>
      <tr>
        <td>CRDT</td>
        <td>Create Date</td>
        <td>The date the citation record was first created</td>
      </tr>
      <tr>
        <td>CRF</td>
        <td>Corrected and republished from</td>
        <td>Final, correct version of an article</td>
      </tr>
      <tr>
        <td>CRI</td>
        <td>Corrected and republished in</td>
        <td>Original article that was republished in corrected form</td>
      </tr>
      <tr>
        <td>CTDT</td>
        <td>Contribution Date</td>
        <td>Book contribution date</td>
      </tr>
      <tr>
        <td>CTI</td>
        <td>Collection Title</td>
        <td>Collection Title</td>
      </tr>
      <tr>
        <td>DCOM</td>
        <td>Completion Date</td>
        <td>NLM internal processing completion date</td>
      </tr>
      <tr>
        <td>DDIN</td>
        <td>Dataset described in</td>
        <td>Citation for the primary article resulting from a dataset</td>
      </tr>
      <tr>
        <td>DRIN</td>
        <td>Dataset use reported in</td>
        <td>Citation for an article that uses a dataset from another scientific article</td>
      </tr>
      <tr>
        <td>DEP</td>
        <td>Date of Electronic Publication</td>
        <td>Electronic publication date</td>
      </tr>
      <tr>
        <td>DP</td>
        <td>Publication Date</td>
        <td>The date the article was published</td>
      </tr>
      <tr>
        <td>DRDT</td>
        <td>Date Revised</td>
        <td>Book Revision Date</td>
      </tr>
      <tr>
        <td>ECF</td>
        <td>Expression of Concern For</td>
        <td>Reference containing an expression of concern for an article</td>
      </tr>
      <tr>
        <td>ECI</td>
        <td>Expression of Concern In</td>
        <td>Cites the original article for which there is an expression of concern</td>
      </tr>
      <tr>
        <td>EDAT</td>
        <td>Entrez Date</td>
        <td>The date the citation was added to PubMed; the date is set to the publication date if added more than 1 year after the date published</td>
      </tr>
      <tr>
        <td>EFR</td>
        <td>Erratum For</td>
        <td>Cites the original article needing the correction</td>
      </tr>
      <tr>
        <td>EIN</td>
        <td>Erratum In</td>
        <td>Reference containing a published erratum to the article</td>
      </tr>
      <tr>
        <td>ED</td>
        <td>Editor</td>
        <td>Book editors</td>
      </tr>
      <tr>
        <td>EN</td>
        <td>Edition</td>
        <td>Book edition</td>
      </tr>
      <tr>
        <td>FAU</td>
        <td>Full Author Name</td>
        <td>Full Author Names</td>
      </tr>
      <tr>
        <td>FED</td>
        <td>Full Editor Name</td>
        <td>Full Editor Names</td>
      </tr>
      <tr>
        <td>FIR</td>
        <td>Full Investigator</td>
        <td>Full investigator or collaborator name</td>
      </tr>
      <tr>
        <td>FPS</td>
        <td>Full Personal Name as Subject</td>
        <td>Full Personal Name of the subject of the article</td>
      </tr>
      <tr>
        <td>GN</td>
        <td>General Note</td>
        <td>Supplemental or descriptive information related to the document</td>
      </tr>
      <tr>
        <td>GR</td>
        <td>Grant Number</td>
        <td>Research grant numbers, contract numbers, or both that designate financial support by any agency of the US PHS or other funding agencies</td>
      </tr>
      <tr>
        <td>GS</td>
        <td>Gene Symbol</td>
        <td>Abbreviated gene names (used 1991 through 1996)</td>
      </tr>
      <tr>
        <td>IP</td>
        <td>Issue</td>
        <td>The number of the issue, part, or supplement of the journal in which the article was published</td>
      </tr>
      <tr>
        <td>IR</td>
        <td>Investigator</td>
        <td>Investigator or collaborator</td>
      </tr>
      <tr>
        <td>IRAD</td>
        <td>Investigator Affiliation</td>
        <td>Investigator or collaborator addresses</td>
      </tr>
      <tr>
        <td>IS</td>
        <td>ISSN</td>
        <td>International Standard Serial Number of the journal</td>
      </tr>
      <tr>
        <td>ISBN</td>
        <td>ISBN</td>
        <td>International Standard Book Number</td>
      </tr>
      <tr>
        <td>JID</td>
        <td>NLM Unique ID</td>
        <td>Unique journal ID in the NLM catalog of books, journals, and audiovisuals</td>
      </tr>
      <tr>
        <td>JT</td>
        <td>Full Journal Title</td>
        <td>Full journal title from NLM cataloging data</td>
      </tr>
      <tr>
        <td>LA</td>
        <td>Language</td>
        <td>The language in which the article was published</td>
      </tr>
      <tr>
        <td>LID</td>
        <td>Location ID</td>
        <td>The pii or doi that serves the role of pagination</td>
      </tr>
      <tr>
        <td>LR</td>
        <td>Modification Date</td>
        <td>Citation last revision date</td>
      </tr>
      <tr>
        <td>MH</td>
        <td>MeSH Terms</td>
        <td>NLM Medical Subject Headings (MeSH) controlled vocabulary</td>
      </tr>
      <tr>
        <td>MHDA</td>
        <td>MeSH Date</td>
        <td>The date MeSH terms were added to the citation. The MeSH date is the same as the Entrez date until MeSH are added</td>
      </tr>
      <tr>
        <td>OAB</td>
        <td>Other Abstract</td>
        <td>Abstract supplied by an NLM collaborating organization</td>
      </tr>
      <tr>
        <td>OABL</td>
        <td>Other Abstract Language</td>
        <td>Language of an abstract available from the publisher</td>
      </tr>
      <tr>
        <td>OCI</td>
        <td>Other Copyright Information</td>
        <td>Copyright owner</td>
      </tr>
      <tr>
        <td>OID</td>
        <td>Other ID</td>
        <td>Identification numbers provided by organizations supplying citation data</td>
      </tr>
      <tr>
        <td>ORI</td>
        <td>Original Report In</td>
        <td>Cites the original article associated with the patient summary</td>
      </tr>
      <tr>
        <td>OT</td>
        <td>Other Term</td>
        <td>Non-MeSH subject terms (keywords) either assigned by an organization identified by the Other Term Owner, or generated by the author and submitted by the publisher</td>
      </tr>
      <tr>
        <td>OTO</td>
        <td>Other Term Owner</td>
        <td>Organization that may have provided the Other Term data</td>
      </tr>
      <tr>
        <td>OWN</td>
        <td>Owner</td>
        <td>Organization acronym that supplied citation data</td>
      </tr>
      <tr>
        <td>PB</td>
        <td>Publisher</td>
        <td>Publishers of Books &amp; Documents citations</td>
      </tr>
      <tr>
        <td>PG</td>
        <td>Pagination</td>
        <td>The full pagination of the article</td>
      </tr>
      <tr>
        <td>PHST</td>
        <td>Publication History Status Date</td>
        <td>Publisher supplied dates regarding the article publishing process 
          and PubMed date stamps:
          <ul>
            <li>
              received: manuscript received for review
            </li>
            <li>
              revised: manuscript revised by publisher or author
            </li>
            <li>
              accepted: manuscript accepted for publication
            </li>
            <li>
              aheadofprint: published electronically prior to final publication
            </li>            
            <li>
              entrez: PubMed Create Date [crdt]
            </li>
            <li>
              pubmed: PubMed Entry Date [edat]
            </li>
            <li>
              medline: PubMed MeSH Date [mhda]
            </li>            
          </ul>
        </td>
      </tr>
      <tr>
        <td>PL</td>
        <td>Place of Publication</td>
        <td>Journal's (country only) or book’s place of publication</td>
      </tr>
      <tr>
        <td>PMCR</td>
        <td>PMC Release</td>
        <td>Availability of PMC article</td>
      </tr>
      <tr>
        <td>PMID</td>
        <td>PubMed Unique Identifier</td>
        <td>Unique number assigned to each PubMed citation</td>
      </tr>
      <tr>
        <td>PRIN</td>
        <td>Partial Retraction In</td>
        <td>Partial retraction of the article</td>
      </tr>
      <tr>
        <td>PROF</td>
        <td>Partial Retraction Of</td>
        <td>Article being partially retracted</td>
      </tr>
      <tr>
        <td>PS</td>
        <td>Personal Name as Subject</td>
        <td>Individual is the subject of the article</td>
      </tr>
      <tr>
        <td>PST</td>
        <td>Publication Status</td>
        <td>Publication status</td>
      </tr>
      <tr>
        <td>PT</td>
        <td>Publication Type</td>
        <td>The type of material the article represents</td>
      </tr>
      <tr>
        <td>RF</td>
        <td>Number of References</td>
        <td>Number of bibliographic references for Review articles</td>
      </tr>
      <tr>
        <td>RIN</td>
        <td>Retraction In</td>
        <td>Retraction of the article</td>
      </tr>
      <tr>
        <td>RN</td>
        <td>EC/RN Number</td>
        <td>Includes chemical, protocol or disease terms. May also a number assigned by the Enzyme Commission or by the Chemical Abstracts Service.</td>
      </tr>
      <tr>
        <td>ROF</td>
        <td>Retraction Of</td>
        <td>Article being retracted</td>
      </tr>
      <tr>
        <td>RPF</td>
        <td>Republished From</td>
        <td>Article being cited has been republished or reprinted in either full or abridged form from another source</td>
      </tr>
      <tr>
        <td>RPI</td>
        <td>Republished In</td>
        <td>Article being cited also appears in another source in either full or abridged form</td>
      </tr>
      <tr>
        <td>RRI</td>
        <td>Retracted and Republished In</td>
        <td>Final, republished version of an article</td>
      </tr>
      <tr>
        <td>RRF</td>
        <td>Retracted and Republished From</td>
        <td>Original article that was retracted and republished</td>
      </tr>
      <tr>
        <td>SB</td>
        <td>Subset</td>
        <td>Journal or citation subset values representing specialized topics</td>
      </tr>
      <tr>
        <td>SFM</td>
        <td>Space Flight Mission</td>
        <td>NASA-supplied data space flight/mission name and/or number</td>
      </tr>
      <tr>
        <td>SI</td>
        <td>Secondary Source Identifier</td>
        <td>Identifies secondary source databanks and accession numbers of molecular sequences discussed in articles</td>
      </tr>
      <tr>
        <td>SO</td>
        <td>Source</td>
        <td>Composite field containing bibliographic information</td>
      </tr>
      <tr>
        <td>SPIN</td>
        <td>Summary For Patients In</td>
        <td>Cites a patient summary article</td>
      </tr>
      <tr>
        <td>STAT</td>
        <td>Status Tag</td>
        <td>Used for internal processing at NLM</td>
      </tr>
      <tr>
        <td>TA</td>
        <td>Journal Title Abbreviation</td>
        <td>Standard journal title abbreviation</td>
      </tr>
      <tr>
        <td>TI</td>
        <td>Title</td>
        <td>The title of the article</td>
      </tr>
      <tr>
        <td>TT</td>
        <td>Transliterated Title</td>
        <td>Title of the article originally published in a non-English language, in that language</td>
      </tr>
      <tr>
        <td>UIN</td>
        <td>Update In</td>
        <td>Update to the article</td>
      </tr>
      <tr>
        <td>UOF</td>
        <td>Update Of</td>
        <td>The article being updated</td>
      </tr>
      <tr>
        <td>VI</td>
        <td>Volume</td>
        <td>Volume number of the journal</td>
      </tr>
      <tr>
        <td>VTI</td>
        <td>Volume Title</td>
        <td>Book Volume Title</td>
      </tr>
    </tbody></table>
    
    
    
    <h3>Date of Publication</h3>
    <p>The information about dates:</p>
    <ul>
        <li>CRDT: Create Date, The date the citation record was first created</li>
        <li>DCOM: Completion Date, NLM internal processing completion date</li>
        <li>DEP: Date of Electronic Publication, Electronic publication date</li>
        <li>DP: Publication Date, The date the article was published</li>
        <li>DRDT: Date Revised, Book Revision Date</li>
        <li>EDAT: Entrez Date, The date the citation was added to PubMed; the date is set to the publication date if added more than 1 year after the date published</li>
        <li>MHDA: MeSH Date, The date MeSH terms were added to the citation. The MeSH date is the same as the Entrez date until MeSH are added</li>
    </ul>
    <p>According to the documentation, DP (Date of publication) seems to be the most appropriate datum. It comes in the following format:</p>
    <ul>
        <li>Year: Four digits</li>
        <li>Month: three charactes representing the month, where the first letter is capital</li>
        <li>Day: One or two digits</li>
    </ul>
    <p>In order to mantain a coherent format for the dates (so that the ones retrieved from NIH keept the same style of the ones retrieved form other sources, e.g.: Crossref), I handled the data contained in DP in this way:</p>
    <div class="codebg">
     <code>
        <pre>
    def _get_date(self, txt_obj):
        date = re.findall("^DP  - \d{4}\s+(Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec)\s+(([0-9])|([0-2][0-9])|([3][0-1]))$", txt_obj)
        if date:
            datetime_object = datetime.strptime(date, '%Y-%m-%d')
            return datetime_object
        else:
            date = re.findall("^DP  - \d{4}\s+(Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec)$", txt_obj)
            if date:
                datetime_object = datetime.strptime(date, '%Y-%m')
                return datetime_object
            else:
                date = re.findall("^DP  - \d{4}$", txt_obj)
                if date:
                    datetime_object = datetime.strptime(date, '%Y')
                    return datetime_object
                else:
                    return None
        </pre>
     </code>
    </div>
    
    <h3>Api call output</h3>
    <p>The other resource finders handle data in json format. However, in NIH the correspondant pieces of information can't be retrieved in the same way. For this reason I restructured the call_api(self, pmid_full) function, whose output is supposed to be the input for the other functions of the class (Is it correct?) </p>
    <div class="codebg">
        <code>
            <pre>
    def _call_api(self, pmid_full):
        if not self.use_api_service:
            return
        pmid = self.pm.normalise(pmid_full)
        r = get(self.api + quote(pmid) + "/?format=pubmed", headers=self.headers, timeout=30)
        if r.status_code == 200:
            r.encoding = "utf-8"
            html = open(self.baseurl + pmid + "/?format=pubmed").read()
            soup = BeautifulSoup(html)
            mdata = soup.find(id="article-details")
            return mdata
            </pre>
        </code>
    </div>
    
    <h3>nihresourcefinder.py current version (Full)</h3>
    <p> Current version of the complete structure of nihresourcefinder.py:</p>
    <div class="codebg">
        <code>
            <pre>
from index.finder.resourcefinder import ApiIDResourceFinder
from requests import get
from urllib.parse import quote
from datetime import datetime
from bs4 import BeautifulSoup
import re

class NIHResourceFinder(ApiIDResourceFinder):
    def __init__(self, date=None, orcid=None, issn=None, pmid=None, use_api_service=True):
        self.use_api_service = use_api_service
        self.api = "https://pubmed.ncbi.nlm.nih.gov/"
        self.baseurl = "https://pubmed.ncbi.nlm.nih.gov/"
        super(NIHResourceFinder, self).__init__(date=date, orcid=orcid, issn=issn, pmid=pmid,
                                                     use_api_service=use_api_service)

    def _get_orcid(self, txt_obj):
        result = set()
        return result


    def _get_issn(self, txt_obj):
        result = set()
        issns = re.findall("^IS  - \d{4}-\d{4}", txt_obj)
        for issn in issns:
            result.add(issn)
        return result

    def _get_date(self, txt_obj):
        date = re.findall("^DP  - \d{4}\s+(Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec)\s+(([0-9])|([0-2][0-9])|([3][0-1]))$", txt_obj)
        if date:
            datetime_object = datetime.strptime(date, '%Y-%m-%d')
            return datetime_object
        else:
            date = re.findall("^DP  - \d{4}\s+(Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec)$", txt_obj)
            if date:
                datetime_object = datetime.strptime(date, '%Y-%m')
                return datetime_object
            else:
                date = re.findall("^DP  - \d{4}$", txt_obj)
                if date:
                    datetime_object = datetime.strptime(date, '%Y')
                    return datetime_object
                else:
                    return None


    def _call_api(self, pmid_full):
        if not self.use_api_service:
            return
        pmid = self.pm.normalise(pmid_full)
        r = get(self.api + quote(pmid) + "/?format=pubmed", headers=self.headers, timeout=30)
        if r.status_code == 200:
            r.encoding = "utf-8"
            html = open(self.baseurl + pmid + "/?format=pubmed").read()
            soup = BeautifulSoup(html)
            mdata = soup.find(id="article-details")
            return mdata


            </pre>
        </code>
    </div>
    
    <h2>03_resourcefinder.py</h2>
    <p>I created the support files: id_issn_pmid.csv, id_date_pmid.csv, id_orcid_pmid.csv. However, I'm not sure about the way id_orcid_pmid.csv was supposed to be structured, since I had no real id-orcid match to use as example. In the end I used empty strings as orcid values for all the pmids of the support file. Would it be better to use None as value for the missing orcids? </p>
    <h3>Support files:</h3>
    <h4>id_issn_pmid.csv</h4>
    <div class="codebg">
        <code>
            <pre>
"id","value"
"pmid:2942070","0003-4819"
"pmid:1509982","0065-4299"
"pmid:7189714","0014-4827"
"pmid:9689714","0920-9964"
            </pre>
        </code>
    </div>
    <h4>id_date_pmid.csv</h4>
    <div class="codebg">
        <code>
            <pre>
"id","value"
"pmid:2942070","1986-08"
"pmid:7189714","1980-06"
"pmid:9689714","1998-05-25"
            </pre>
        </code>
    </div>
    <h4>id_orcid_pmid.csv</h4>
    <div class="codebg">
        <code>
            <pre>
"id","value"
"pmid:2942070",""
"pmid:1509982",""
"pmid:7189714",""
"pmid:9689714",""
            </pre>
        </code>
    </div>
    
    <h3>Added/modified pieces of code in 03_resourcefinder.py:</h3>
    <div class="codebg">
        <code>
            <pre>
class ResourceFinderTest(unittest.TestCase):
    """This class aim at testing the methods of the class CSVManager."""

    def setUp(self):
        self.date_path = "index%stest_data%sid_date.csv" % (sep, sep)
        self.date_path_pmid = "index%stest_data%sid_date_pmid.csv" % (sep, sep)
        self.orcid_path = "index%stest_data%sid_orcid.csv" % (sep, sep)
        self.orcid_path_pmid = "index%stest_data%sid_orcid_pmid.csv" % (sep, sep)
        self.issn_path = "index%stest_data%sid_issn.csv" % (sep, sep)
        self.issn_path_pmid = "index%stest_data%sid_issn_pmid.csv" % (sep, sep)
        self.doi_path = "index%stest_data%svalid_doi.csv" % (sep, sep)
        self.pmid_path = "index%stest_data%svalid_pmid.csv" % (sep, sep)
            </pre>
        </code>
    </div>
    
    <div class="codebg">
        <code>
            <pre>
    # IMPLEMENTED THE TEST FOR  NIH finder
    def test_nationalinstititeofhealth_get_orcid(self):
        # Do not use support files, only APIs
        nf_1 = NIHResourceFinder()
        self.assertIsNone(nf_1.get_orcid("7189714"))
        self.assertIsNone(nf_1.get_orcid("1509982"))

        # Do use support files, but avoid using APIs
        nf_2 = NIHResourceFinder(orcid=CSVManager(self.orcid_path_pmid),
                                      pmid=CSVManager(self.pmid_path), use_api_service=False)
        self.assertIn("", nf_2.get_orcid("7189714"))
        self.assertNotIn("", nf_2.get_orcid("1509982"))

        # Do not use support files neither APIs
        nf_3 = NIHResourceFinder(use_api_service=False)
        self.assertIsNone(nf_3.get_orcid("7189714"))

    def test_nationalinstititeofhealth_get_issn(self):
        # Do not use support files, only APIs
        nf_1 = NIHResourceFinder()
        self.assertIn("0003-4819", nf_1.get_container_issn("2942070"))
        self.assertNotIn("0003-0000", nf_1.get_container_issn("2942070"))

        # Do use support files, but avoid using APIs
        nf_2 = NIHResourceFinder(date=CSVManager(self.date_path_pmid),
                                      pmid=CSVManager(self.pmid_path), use_api_service=False)
        self.assertIn("0065-4299", nf_2.get_container_issn("1509982"))
        self.assertNotIn("0065-4444", nf_2.get_container_issn("1509982"))

        # Do not use support files neither APIs
        nf_3 = NIHResourceFinder(use_api_service=False)
        self.assertIsNone(nf_3.get_container_issn("7189714"))

    def test_nationalinstititeofhealth_get_pub_date(self):
        # Do not use support files, only APIs
        nf_1 = NIHResourceFinder()
        self.assertIn("1998-05-25", nf_1.get_pub_date("9689714"))
        self.assertNotEqual("1998", nf_1.get_pub_date("9689714"))

        # Do not use support files, only APIs
        nf_2 = NIHResourceFinder(date=CSVManager(self.date_path_pmid),
                                      pmid=CSVManager(self.pmid_path), use_api_service=False)
        self.assertIn("1980-06", nf_2.get_pub_date("7189714"))
        self.assertNotEqual("1980-06-22", nf_2.get_pub_date("7189714"))

        # Do not use support files neither APIs
        nf_3 = NIHResourceFinder(use_api_service=False)
        self.assertIsNone(nf_3.get_pub_date("2942070"))

            </pre>
        </code>
    </div>
    
    <p>However, the test failed:</p>
    <div class="codebg">
        <code>
            <pre>
C:\Users\arimoretti\Documents\GitHub>python -m unittest index.test.03_resourcefinder
The id_type specified is not compliant
The id_type specified is not compliant
EThe id_type specified is not compliant
The id_type specified is not compliant
EThe id_type specified is not compliant
The id_type specified is not compliant
EThe id_type specified is not compliant
The id_type specified is not compliant
EThe id_type specified is not compliant
The id_type specified is not compliant
EThe id_type specified is not compliant
The id_type specified is not compliant
EThe id_type specified is not compliant
The id_type specified is not compliant
The id_type specified is not compliant
The id_type specified is not compliant
The id_type specified is not compliant
The id_type specified is not compliant
The id_type specified is not compliant
The id_type specified is not compliant
The id_type specified is not compliant
The id_type specified is not compliant
The id_type specified is not compliant
The id_type specified is not compliant
FThe id_type specified is not compliant
The id_type specified is not compliant
The id_type specified is not compliant
The id_type specified is not compliant
The id_type specified is not compliant
The id_type specified is not compliant
The id_type specified is not compliant
The id_type specified is not compliant
The id_type specified is not compliant
The id_type specified is not compliant
The id_type specified is not compliant
The id_type specified is not compliant
FThe id_type specified is not compliant
The id_type specified is not compliant
The id_type specified is not compliant
The id_type specified is not compliant
The id_type specified is not compliant
The id_type specified is not compliant
The id_type specified is not compliant
The id_type specified is not compliant
The id_type specified is not compliant
The id_type specified is not compliant
The id_type specified is not compliant
The id_type specified is not compliant
FThe id_type specified is not compliant
The id_type specified is not compliant
EThe id_type specified is not compliant
The id_type specified is not compliant
The id_type specified is not compliant
The id_type specified is not compliant
The id_type specified is not compliant
EThe id_type specified is not compliant
The id_type specified is not compliant
EThe id_type specified is not compliant
The id_type specified is not compliant
E
======================================================================
ERROR: test_crossref_get_issn (index.test.03_resourcefinder.ResourceFinderTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\Users\arimoretti\Documents\GitHub\index\test\03_resourcefinder.py", line 146, in test_crossref_get_issn
    self.assertIn("0138-9130", cf_1.get_container_issn("10.1007/s11192-018-2988-z"))
  File "C:\Program Files\Python36\lib\unittest\case.py", line 1085, in assertIn
    if member not in container:
TypeError: argument of type 'NoneType' is not iterable

======================================================================
ERROR: test_crossref_get_orcid (index.test.03_resourcefinder.ResourceFinderTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\Users\arimoretti\Documents\GitHub\index\test\03_resourcefinder.py", line 130, in test_crossref_get_orcid
    self.assertIn("0000-0003-0530-4305", cf_1.get_orcid("10.1007/s11192-018-2988-z"))
  File "C:\Program Files\Python36\lib\unittest\case.py", line 1085, in assertIn
    if member not in container:
TypeError: argument of type 'NoneType' is not iterable

======================================================================
ERROR: test_crossref_get_pub_date (index.test.03_resourcefinder.ResourceFinderTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\Users\arimoretti\Documents\GitHub\index\test\03_resourcefinder.py", line 162, in test_crossref_get_pub_date
    self.assertIn("2019-01-02", cf_1.get_pub_date("10.1007/s11192-018-2988-z"))
  File "C:\Program Files\Python36\lib\unittest\case.py", line 1085, in assertIn
    if member not in container:
TypeError: argument of type 'NoneType' is not iterable

======================================================================
ERROR: test_datacite_get_issn (index.test.03_resourcefinder.ResourceFinderTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\Users\arimoretti\Documents\GitHub\index\test\03_resourcefinder.py", line 95, in test_datacite_get_issn
    self.assertIn("2197-6775", df_1.get_container_issn("10.14763/2019.1.1389"))
  File "C:\Program Files\Python36\lib\unittest\case.py", line 1085, in assertIn
    if member not in container:
TypeError: argument of type 'NoneType' is not iterable

======================================================================
ERROR: test_datacite_get_orcid (index.test.03_resourcefinder.ResourceFinderTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\Users\arimoretti\Documents\GitHub\index\test\03_resourcefinder.py", line 78, in test_datacite_get_orcid
    self.assertIn("0000-0001-7734-8388", df_1.get_orcid("10.5065/d6b8565d"))
  File "C:\Program Files\Python36\lib\unittest\case.py", line 1085, in assertIn
    if member not in container:
TypeError: argument of type 'NoneType' is not iterable

======================================================================
ERROR: test_datacite_get_pub_date (index.test.03_resourcefinder.ResourceFinderTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\Users\arimoretti\Documents\GitHub\index\test\03_resourcefinder.py", line 112, in test_datacite_get_pub_date
    self.assertIn("2019-05-27", df_1.get_pub_date("10.6092/issn.2532-8816/8555"))
  File "C:\Program Files\Python36\lib\unittest\case.py", line 1085, in assertIn
    if member not in container:
TypeError: argument of type 'NoneType' is not iterable

======================================================================
ERROR: test_nationalinstititeofhealth_get_issn (index.test.03_resourcefinder.ResourceFinderTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\Users\arimoretti\Documents\GitHub\index\test\03_resourcefinder.py", line 196, in test_nationalinstititeofhealth_get_issn
    self.assertIn("0003-4819", nf_1.get_container_issn("2942070"))
  File "C:\Program Files\Python36\lib\unittest\case.py", line 1085, in assertIn
    if member not in container:
TypeError: argument of type 'NoneType' is not iterable

======================================================================
ERROR: test_nationalinstititeofhealth_get_orcid (index.test.03_resourcefinder.ResourceFinderTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\Users\arimoretti\Documents\GitHub\index\test\03_resourcefinder.py", line 186, in test_nationalinstititeofhealth_get_orcid
    self.assertIn("", nf_2.get_orcid("7189714"))
  File "C:\Program Files\Python36\lib\unittest\case.py", line 1085, in assertIn
    if member not in container:
TypeError: argument of type 'NoneType' is not iterable

======================================================================
ERROR: test_nationalinstititeofhealth_get_pub_date (index.test.03_resourcefinder.ResourceFinderTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\Users\arimoretti\Documents\GitHub\index\test\03_resourcefinder.py", line 212, in test_nationalinstititeofhealth_get_pub_date
    self.assertIn("1998-05-25", nf_1.get_pub_date("9689714"))
  File "C:\Program Files\Python36\lib\unittest\case.py", line 1085, in assertIn
    if member not in container:
TypeError: argument of type 'NoneType' is not iterable

======================================================================
ERROR: test_orcid_get_orcid (index.test.03_resourcefinder.ResourceFinderTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\Users\arimoretti\Documents\GitHub\index\test\03_resourcefinder.py", line 62, in test_orcid_get_orcid
    self.assertIn("0000-0003-0530-4305", of_1.get_orcid("10.1108/jd-12-2013-0166"))
  File "C:\Program Files\Python36\lib\unittest\case.py", line 1085, in assertIn
    if member not in container:
TypeError: argument of type 'NoneType' is not iterable

======================================================================
FAIL: test_handler_get_date (index.test.03_resourcefinder.ResourceFinderTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\Users\arimoretti\Documents\GitHub\index\test\03_resourcefinder.py", line 44, in test_handler_get_date
    self.assertEqual("2019-05-27", handler.get_date("10.6092/issn.2532-8816/8555"))
AssertionError: '2019-05-27' != None

======================================================================
FAIL: test_handler_share_issn (index.test.03_resourcefinder.ResourceFinderTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\Users\arimoretti\Documents\GitHub\index\test\03_resourcefinder.py", line 50, in test_handler_share_issn
    self.assertTrue(handler.share_issn("10.1007/s11192-018-2988-z", "10.1007/s11192-015-1565-y"))
AssertionError: False is not true

======================================================================
FAIL: test_handler_share_orcid (index.test.03_resourcefinder.ResourceFinderTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\Users\arimoretti\Documents\GitHub\index\test\03_resourcefinder.py", line 56, in test_handler_share_orcid
    self.assertTrue(handler.share_orcid("10.1007/s11192-018-2988-z", "10.5281/zenodo.3344898"))
AssertionError: False is not true

----------------------------------------------------------------------
Ran 13 tests in 0.027s

FAILED (failures=3, errors=10)


            </pre>
        </code>
    </div>
    
    
    <h2>crossrefresourcefinder.py, dataciteresourcefinder.py, orcidresourcefinder.py</h2>
    <p>After testing 03_resourcefinder.py I refactored ApiDOIResourceFinder to ApiIDResourceFinder.</p>


          


      </div>
    </div>
  </div>
  
  <div class="card">
    <div class="card-header" id="headingFifteen">
      <h5 class="mb-0">
        <button class="btn btn-link collapsed" data-toggle="collapse" data-target="#collapseFifteen" aria-expanded="false" aria-controls="collapseFifteen">
          Week #15 (05-25-21)
        </button>
      </h5>
    </div>
    <div id="collapseFifteen" class="collapse" aria-labelledby="headingFifteen" data-parent="#accordion">
      <div class="card-body">
          
          <h2>Points to be discussed:</h2>
          <ul>
            <li><strong>Import format</strong>: e.g.:from index.finder.nihresourcefinder import NIHResourceFinder. In all the other projects I developed, I used to call the import from inside the project folder. I.e.: If the file importing the element and the file containing the element are both in index, why do we need to specify "index." when we import? Further, by default, the IDE I use suggests not to use the full path for the import statement. I can't understand if it can be a problem itself or if it just appears for collateral errors in the code I developed. After having fixed a factual bug related to the specification of the id_type, I started getting this error: ModuleNotFoundError: No module named 'index'.</li>
            <li><strong>Debugging</strong>: I started studying how to debug and I'd like to understand better the aspect related to the order of the presented errors after "Traceback (most recent call last)". Does it means that the first problem to be addressed is always the last one showed? If so, why?</li>
            <li><strong>Project Evolution</strong>: Until now I worked locally following the test driven development, so that the steps of the thesis project were locally checked by implementing a test for them. However, since the amount of edited/added files has been recently growing, I think that it could be resonable to try and finding a way to keep track of the evolutions of the project. Indeed, I'm currently meeting some difficulties in identifying specific passages I modified. Further, in the moment in which a test is not passed anymore, I can't go back to a previous version of the code. In conclusion, I think it maybe useful to start working on this aspect, for example by forking the original software or also by finding another way which in any case should allow me to see the code history. In order to do that, I thought about two possible alternative ways to address the issue: (1) Stopping the code development for a week or two, and repeating all the development step by step, so to keep track of it and document it properly; (2) Keeping working at the code development and in the meantime starting to transpose in a code fork only the passages I'm already sure about, progressively.  </li>
            <li>Discuss the <strong>get_item</strong> function.</li>
          </ul>
          
          <h2>id_type Issue</h2>
          <p>The addition of the id_type argument to all the finders allowed the fixation of the error related to the id_type compliance.</p>
          
          <h2>Regex improvement</h2>
          
          <p>I slightly modified the code of the finder so to make the regex more efficient. The updated version of the code is as follows:</p>
          <div class="codebg">
              <code>
                  <pre>
class NIHResourceFinder(ApiIDResourceFinder):
    def __init__(self, date=None, orcid=None, issn=None, pmid=None, id_type=None, use_api_service=True):
        self.use_api_service = use_api_service
        self.api = "https://pubmed.ncbi.nlm.nih.gov/"
        self.baseurl = "https://pubmed.ncbi.nlm.nih.gov/"
        super(NIHResourceFinder, self).__init__(date=date, orcid=orcid, issn=issn, pmid=pmid, id_type=OCIManager.pmid_type,
                                                     use_api_service=use_api_service)

#quando esegui un metodo in una classe, se non è definito nella classe, lo cerca nella superclasse. get orcid è già definito in Apiidresourcefinder


    def _get_issn(self, txt_obj):
        result = set()
        issns = re.findall("^IS\s+-\s+\d{4}-\d{4}", txt_obj)
        print("HEY THIS OBJECT IS ISSNS, risultato di re.findall", issns)
        for i in issns:
            issn = re.search("\d{4}-\d{4}", i)
            print("HEY THIS OBJECT IS ISSN, risultato di re.search", issn)
            result.add(issn)
        return result

    def _get_date(self, txt_obj):
        date = re.findall("^DP\s+-\s+\d{4}(\s?(Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec))?(\s?(([0]?[1-9])|([1-2][0-9])|(3[0-1])))?$", txt_obj)
        result = re.search("(\d{4})\s+(Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec)\s+(([0-9])|([0-2][0-9])|([3][0-1]))", date[0])
        if result:
            datetime_object = datetime.strptime(result, '%Y-%m-%d')
            return datetime_object
        else:
            result = re.search("(\d{4})\s+(Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec)", date[0])
            if result:
                datetime_object = datetime.strptime(result, '%Y-%m')
                return datetime_object
            else:
                result = re.search("(\d{4})", date[0])
                if result:
                    datetime_object = datetime.strptime(result, '%Y')
                    return datetime_object
                else:
                    return None


    def _call_api(self, pmid_full):
        if self.use_api_service:
            pmid = self.pm.normalise(pmid_full)
            r = get(self.api + quote(pmid), headers=self.headers, timeout=30)
            if r.status_code == 200:
                r.encoding = "utf-8"
                soup = BeautifulSoup( r.text )
                mdata = soup.find(id="article-details")
                return mdata
                  </pre>
              </code>    
      </div>
      <p><strong>Note</strong>: the get_orcid function implemented in this subclass was deleted, since from the moment in which we don't have a way to retrieve the orcid of the publication from the resources offered by NIH it doesn't make sense to have an implementation of the get orcid function specific for the NIH.</p>
      
    <h3>Orcid support files</h3>
    <p>As suggested, I added an orcid for the tested identifiers in id_orcid_pmid.csv, because in the case I left empty spaces for all the pmid for which NIH does not provide an identifier, the system would recognise all the identifiers as sharing the same publisher. For now they are just examples to check the correct functioning of the finders with support files.</p>
             
    </div>
  </div>

</div>

<div class="card">
    <div class="card-header" id="headingSixteen">
      <h5 class="mb-0">
        <button class="btn btn-link collapsed" data-toggle="collapse" data-target="#collapseSixteen" aria-expanded="false" aria-controls="collapseSixteen">
          Week #16 (06-03-21)
        </button>
      </h5>
    </div>
    <div id="collapseSixteen" class="collapse" aria-labelledby="headingSixteen" data-parent="#accordion">
      <div class="card-body">
      
      <h2>Source folder and reorganization of the project</h2>
      <p>I forked OpenCitations code.
I moved the folder index to another path in my personal computer, so that I could keep working on my modified code, while starting updating the fork with pieced of code which already work properly. </p>
      <p><strong>Note:</strong> During last meeting, we realized that, in order to make the code work properly, it is necessary that the folder to be opened as project is the one containing the whole code (so the folder index). This is very important, because in all the other cases the import statements stop working.</p>
      
     <h2>Finders tests and path problems</h2>
     <p>Trying to debug the finders' tests, it turned out a problem -that I wasn't able to fix- related to the csv manager: at a certain point is like the support file is not identified nor found anymore. I also implemented a separate function to check the existence and the correctness of the paths of thr support files specified in the SetUp of the finders' test file, and everything seems to work. Anyway, for this reason, none of the finder tests involving the support files seems to work. </p>
     
     <h2>Finders tests and path problems</h2>
     <p>While debugging, a some problems related to the specific structure of nihresourcefinder.py emerged. I consequently modified some parts of the functions:</p>

<div class="codebg">
    <code>
        <pre>


def _get_issn(self, txt_obj):
    result = set()
    issns = re.findall("IS\s+-\s+\d{4}-\d{4}", txt_obj)
    print("HEY THIS OBJECT IS ISSNS, risultato di re.findall", issns)
    for i in issns:
        issn = re.search("\d{4}-\d{4}", i).group(0)
        print("HEY THIS OBJECT IS ISSN, risultato di re.search", issn)
        result.add(issn)
    return result

def _get_date(self, txt_obj):
    date = re.search("DP\s+-\s+\d{4}(\s?(Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec))?(\s?((3[0-1])|([1-2][0-9])|([0]?[1-9])))?", txt_obj).group(0)
    result = re.search("(\d{4})\s+(Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec)\s+((3[0-1])|([1-2][0-9])|([0]?[1-9]))", date).group(0)
    if result:
        datetime_object = datetime.strptime(result, '%Y %b %d')
        return datetime.strftime(datetime_object, '%Y-%m-%d')
    else:
        result = re.search("(\d{4})\s+(Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec)", date).group(0)
        if result:
            datetime_object = datetime.strptime(result, '%Y %b')
            return datetime.strftime(datetime_object, '%Y-%m')
        else:
            result = re.search("(\d{4})", date).group(0)
            if result:
                datetime_object = datetime.strptime(result, '%Y')
                return datetime.strftime(datetime_object, '%Y')
            else:
                return None

def _call_api(self, pmid_full):
    if self.use_api_service:
        pmid = self.pm.normalise(pmid_full)
        r = get(self.api + quote(pmid) + "/?format=pubmed", headers=self.headers, timeout=30)
        if r.status_code == 200:
            r.encoding = "utf-8"
            soup = BeautifulSoup( r.text )
            mdata = str(soup.find(id="article-details"))
            return mdata
        
        </pre>
    </code>
</div>

<p>In particular:</p>
<ol>
    <li>The missing format information ("/?format=pubmed") in _call_api led to the wrong URL.</li>
    <li>The returned output of _call_api was not a text object, but a tag object, so I turned it into a string. </li>
    <li>In _get_pub_date and _get_issn the regex didn’t need the starting and ending delimiters.</li>
    <li>In _get_pub_date the regex for the date was not correct: considering first the possibility to match a number between 1 and 9 led to a mismatch for days numbers with more than one digit, so I inverted the order of the 3 alternative patterns to match the day number (from 1 to 31).  </li>
    <li>I substituted re.search to re.findall in the pub date function, since there should be just one DP in the text object. I used group(0) to select the matched term from the match object, so to obtain a string.</li>
    <li>Something still doesn’t work as it should, since the tests for issn and orcid seems to fail for problems related to the get pub date function (whose test is instead passed). </li>
    <li>I also modified some details in the read and written datetime object: I used "%b" to let the literal abbreviation of the month to be correctly individuated, and I changed the overall format structure in the strptime parameters. The previously used parameter format ('%Y-%m-%d') is keept instead in the strftime function, which is supposed to return the date in the format compliant with the rest of the OC project. </li>
</ol>       
    </div>
  </div>

</div>



<div class="card">
    <div class="card-header" id="headingSeventeen">
      <h5 class="mb-0">
        <button class="btn btn-link collapsed" data-toggle="collapse" data-target="#collapseSeventeen" aria-expanded="false" aria-controls="collapseSeventeen">
          Week #17 (06-17-21)
        </button>
      </h5>
    </div>
    <div id="collapseSeventeen" class="collapse" aria-labelledby="headingSeventeen" data-parent="#accordion">
      <div class="card-body">
          <h2>03_resourcefinder</h2>
          <div class="codebg">
            <code>
                <pre>
                def get_orcid(self, id_string):
    print("this is id_string and it is ok if...", id_string)
    print("this is self.orcid and it is ok if...", self.orcid) #index.storer.csvmanager.CSVManager object at 0x0000016A3A62BAC8
    print("here self._get_item(id_string, self.orcid)", self._get_item(id_string, self.orcid))
    print("here self._get_item(doi:10.1108/jd-12-2013-0166, 0000-0003-0530-4305)", self._get_item("doi:10.1108/jd-12-2013-0166", "0000-0003-0530-4305"))
    print("here self._get_item(10.1108/jd-12-2013-0166, 0000-0003-0530-4305)", self._get_item("10.1108/jd-12-2013-0166", "0000-0003-0530-4305"))
    return self._get_item(id_string, self.orcid)


                </pre>
            </code>
          </div>
          
          <p>Terminal</p>
          <div class="codebg">
            <code>
                <pre>
this is id_string and it is ok if... 10.1108/jd-12-2013-0166
this is self.orcid and it is ok if... index.storer.csvmanager.CSVManager object at 0x0000014BB8E565C0
here self._get_item(id_string, self.orcid) None
here self._get_item(doi:10.1108/jd-12-2013-0166, 0000-0003-0530-4305) None
here self._get_item(10.1108/jd-12-2013-0166, 0000-0003-0530-4305) None
E
                </pre>
            </code>
          </div>
          
          <p>I stepped into get_orcid from the test to see its functioning. I supposed that the error could be related to self.orcid (index.storer.csvmanager.CSVManager object at 0x0000014BB8E565C0), but, instead, it seems that any combination doesn’t work as well, as I checked with some print attempts. </p>
          <p>I proceed now analysing the process in the original software, in order to see at which point the two versions start working differently, and – consequently – at which point the extended version of the software stops working properly. 
this is self._get_item(id_string, self.orcid) {'0000-0003-0530-4305'} --> self.get_item function should get in input an id string without prefix and an object like index.storer.csvmanager.CSVManager object at 0x0000025C2E12C278, and return a set containing the string(s) of the orcid related to the input identifier. </p>
        <p>THE PROBLEM MUST BE RELATED TO THE FUNCTIONING OF THE self._get_item(id_string, self.orcid) in the extended version</p>
        
        <h2>csvmanager.py</h2>
        <p>In my version of the software the function update_value is still present, but I noticed that it was deleted from the OC software in the meantime. Should I delete it too?</p>
        
        <h2>Current version of my files</h2>
        
        <h3>resourcefinder.py</h3>
        <div class="codebg">
            <code>
                <pre>

from index.storer.csvmanager import CSVManager
from index.identifier.orcidmanager import ORCIDManager
from index.identifier.doimanager import DOIManager
from index.identifier.pmidmanager import PMIDManager
from index.identifier.issnmanager import ISSNManager
from index.citation.oci import OCIManager

from collections import deque
# TODO: For multiprocessing purposes
# from multiprocessing.managers import BaseManager


class ResourceFinder(object):
    """This is the abstract class that must be implemented by any resource finder
    for a particular service (Crossref, DataCite, ORCiD, etc.). It provides
    the signatures of the methods that should be implemented, and a basic
    constructor."""

    def __init__(self, date=None, orcid=None, issn=None, id=None, id_type=None, **params): #vedere se crea problemi l'aggiunta di id_type
        if date is None:
            date = CSVManager(store_new=False)
        if orcid is None:
            orcid = CSVManager(store_new=False)
        if issn is None:
            issn = CSVManager(store_new=False)
        if id is None:
            id = CSVManager(store_new=False)

        for key in params:
            setattr(self, key, params[key])

        self.issn = issn
        self.date = date
        self.orcid = orcid
        self.id_type = id_type
        if hasattr(self, 'use_api_service'):
            if id_type is OCIManager.doi_type:
                self.dm = DOIManager(id, self.use_api_service)
            elif id_type is OCIManager.pmid_type:
                self.pm = PMIDManager(id, self.use_api_service)
            else:
                print("The id_type specified is not compliant")
        else:
            if id_type is OCIManager.doi_type:
                self.dm = DOIManager(id)
            elif id_type is OCIManager.pmid_type:
                self.pm = PMIDManager(id)
            else:
                print("The id_type specified is not compliant")

        self.im = ISSNManager()
        self.om = ORCIDManager()

        self.headers = {
            "User-Agent": "ResourceFinder / OpenCitations Indexes "
                          "(http://opencitations.net; mailto:contact@opencitations.net)"
        }

        # TODO: For multiprocessing purposes
        # c_type = type(self)
        # BaseManager.register(c_type.__name__, c_type)

    def get_orcid(self, id_string):
        pass

    def get_pub_date(self, id_string):
        pass

    def get_container_issn(self, id_string):
        pass

    def is_valid(self, id_string):
        pass

    def normalise(self, id_string):
        pass

class ApiIDResourceFinder(ResourceFinder): #The name of the class was changed
    """This is the abstract class that must be implemented by any resource finder
        for a particular service which is based on DOI retrieving via HTTP REST APIs
        (Crossref, DataCite). It provides basic methods that are be used for
        implementing the main methods of the ResourceFinder abstract class."""

    # The following four methods are those ones that should be implemented in
    # the concrete subclasses of this abstract class.
    def _get_date(self, json_obj):
        pass

    def _get_issn(self, json_obj):
        return set()

    def _get_orcid(self, json_obj):
        return set()

    def _call_api(self, id_full):
        pass

    # The implementation of the following methods is strictly dependent on the actual
    # implementation of the previous three methods, since they strictly reuse them
    # for returning the result.
    def get_orcid(self, id_string):
        print("this is id_string and it is ok if...", id_string)
        print("this is self.orcid and it is ok if...", self.orcid) # index.storer.csvmanager.CSVManager object at 0x0000016A3A62BAC8>
        print("here self._get_item(id_string, self.orcid)", self._get_item(id_string, self.orcid))
        print("here self._get_item(doi:10.1108/jd-12-2013-0166, 0000-0003-0530-4305)", self._get_item("doi:10.1108/jd-12-2013-0166", "0000-0003-0530-4305"))
        print("here self._get_item(10.1108/jd-12-2013-0166, 0000-0003-0530-4305)", self._get_item("10.1108/jd-12-2013-0166", "0000-0003-0530-4305"))
        return self._get_item(id_string, self.orcid)

    def get_pub_date(self, id_string):
        return self._get_item(id_string, self.date)

    def get_container_issn(self, id_string):
        return self._get_item(id_string, self.issn)

    def is_valid(self, id_string): #FAI IN TUTTI COSì
        if self.id_type is OCIManager.doi_type:
            return self.dm.is_valid(id_string)
        elif self.id_type is OCIManager.pmid_type:
            return self.pm.is_valid(id_string)
        else:
            print("The id_type specified is not compliant")

    def normalise(self, id_string):
        if self.id_type is OCIManager.doi_type:
            return self.dm.normalise(id_string, include_prefix=True)
        elif self.id_type is OCIManager.pmid_type:
            return self.pm.normalise(id_string, include_prefix=True)
        else:
            print("The id_type specified is not compliant")


    def _get_item(self, id_entity, csv_manager):
        if self.is_valid(id_entity):
            print("self.is_valid(id_entity) in _get_item is ok if....", self.is_valid(id_entity))
            id = self.normalise(id_entity) #!!!!!!!!!!!!!!!!!!!!
            print ("self.normalise(id_entity) in _get_item is ok if....", self.normalise(id_entity))

            if csv_manager.get_value(id) is None:
                json_obj = self._call_api(id) #la prima volta c'erano solo json. quello che restituirà sarà sicuramente compliant. Non è un problema è solo il nome della variabile

                if json_obj is not None:
                    for issn in self._get_issn(json_obj):
                        self.issn.add_value(id, issn)

                    if self.date.get_value(id) is None:
                        pub_date = self._get_date(json_obj)
                        if pub_date is not None:
                            self.date.add_value(id, pub_date)

                    for orcid in self._get_orcid(json_obj):
                        self.orcid.add_value(id, orcid)

            return csv_manager.get_value(id)


class ResourceFinderHandler(object):
    """This class allows one to use multiple resource finders at the same time
    so as to find the information needed for the creation of the citations to
    include in the index."""

    def __init__(self, resource_finders):
        self.resource_finders = resource_finders

    def get_date(self, id_string):
        result = None
        finders = deque(self.resource_finders)

        while result is None and finders:
            finder = finders.popleft()
            result_set = finder.get_pub_date(id_string)
            if result_set:
                result = result_set.pop()

        if result is None:  # Add the empty value in all the finders
            for finder in self.resource_finders:
                if finder.is_valid(id_string):
                    finder.date.add_value(finder.normalise(id_string), "")

        return result

    def share_issn(self, id_string_1, id_string_2):
        return self.__share_data(id_string_1, id_string_2, "get_container_issn")

    def share_orcid(self, id_string_1, id_string_2):
        return self.__share_data(id_string_1, id_string_2, "get_orcid")

    def __share_data(self, id_string_1, id_string_2, method):
        result = False
        finders = deque(self.resource_finders)
        set_1 = set()
        set_2 = set()

        while not result and finders:
            finder = finders.popleft()

            result_set_1 = getattr(finder, method)(id_string_1)
            if result_set_1:
                set_1.update(result_set_1)

            result_set_2 = getattr(finder, method)(id_string_2)
            if result_set_2:
                set_2.update(result_set_2)

            result = len(set_1.intersection(set_2)) > 0

        return result

                </pre>
            </code>
        </div>
        
        <div class="codebg">
            <code>
            <pre>

from csv import DictReader
from io import StringIO
from os.path import exists, isdir
from os import walk, sep


class CSVManager(object):
    """This class is able to load a simple CSV composed by two fields, 'id' and
    'value', and then to index all its items in a structured form so as to be
    easily queried. In addition, it allows one to store new information in the CSV,
    if needed."""

    def __init__(self, csv_path=None, line_threshold=10000, store_new=True):
        self.csv_path = csv_path
        self.data = {}
        self.store_new = store_new

        if csv_path is not None and exists(csv_path):
            CSVManager.__load_all_csv_files([csv_path], self.__load_csv, line_threshold=line_threshold)

    @staticmethod
    def load_csv_column_as_set(file_or_dir_path, key, line_threshold=10000):
        result = set()

        if exists(file_or_dir_path):
            file_to_process = []
            if isdir(file_or_dir_path):
                for cur_dir, cur_subdir, cur_files in walk(file_or_dir_path):
                    for cur_file in cur_files:
                        if cur_file.endswith(".csv"):
                            file_to_process.append(cur_dir + sep + cur_file)
            else:
                file_to_process.append(file_or_dir_path)

            for item in CSVManager.__load_all_csv_files(file_to_process, CSVManager.__load_csv_by_key,
                                                        line_threshold=line_threshold, key=key):
                result.update(item)

        return result

    @staticmethod
    def __load_csv_by_key(csv_string, key):
        result = set()

        csv_metadata = DictReader(StringIO(csv_string), delimiter=',')
        for row in csv_metadata:
            result.add(row[key])

        return result

    @staticmethod
    def __load_all_csv_files(list_of_csv_files, fun, line_threshold, **params):
        result = []
        header = None

        for csv_path in list_of_csv_files:
            with open(csv_path, encoding="utf-8") as f:
                csv_content = ""
                for idx, line in enumerate(f.readlines()):
                    if header is None:
                        header = line
                        csv_content = header
                    else:
                        if idx % line_threshold == 0:
                            result.append(fun(csv_content, **params))
                            csv_content = header
                        csv_content += line

            result.append(fun(csv_content, **params))

        return result

    def get_value(self, id_string):
        """It returns the set of values associated to the input 'id_string',
        or None if 'id_string' is not included in the CSV."""
        if id_string in self.data:
            return set(self.data[id_string])

    def update_value(self,id_string, value):
        if id_string not in self.data:
            self.data[id_string] = value
        else:
            if self.data[id_string] != value:
                self.data[id_string] = value

    def add_value(self, id_string, value):
        """It adds the value specified in the set of values associated to 'id_string'.
        If the object was created with the option of storing also the data in a CSV
        ('store_new' = True, default behaviour), then it also add new data in the CSV."""
        if id_string not in self.data:
            self.data[id_string] = set()

        if value not in self.data[id_string]:
            self.data[id_string].add(value)

            if self.csv_path is not None and self.store_new:
                if not exists(self.csv_path):
                    with open(self.csv_path, "w", encoding="utf8") as f:
                        f.write('"id","value"\n')

                with open(self.csv_path, "a", encoding="utf8") as f:
                    f.write('"%s","%s"\n' % (id_string.replace('"', '""'),
                                             value.replace('"', '""')))

    def __load_csv(self, csv_string):
        csv_metadata = DictReader(StringIO(csv_string), delimiter=',')
        for row in csv_metadata:
            cur_id = row["id"]
            if cur_id not in self.data:
                self.data[cur_id] = set()

            self.data[cur_id].add(row["value"])

            </pre>
            </code>
        </div>
        
        <p>Terminal after test run</p>
        <div class="codebg">
            <code>
                <pre>

E:\tesi>python -m unittest index.test.03_resourcefinder
this is id_string and it is ok if... 10.1108/jd-12-2013-0166
this is self.orcid and it is ok if... index.storer.csvmanager.CSVManager object at 0x000001D9883F55C0>
here self._get_item(id_string, self.orcid) None
here self._get_item(doi:10.1108/jd-12-2013-0166, 0000-0003-0530-4305) None
here self._get_item(10.1108/jd-12-2013-0166, 0000-0003-0530-4305) None
of_2.get_orcid(10.1108/jd-12-2013-0166) None
this is id_string and it is ok if... 10.1108/jd-12-2013-0166
this is self.orcid and it is ok if... index.storer.csvmanager.CSVManager object at 0x000001D9883F55C0>
here self._get_item(id_string, self.orcid) None
here self._get_item(doi:10.1108/jd-12-2013-0166, 0000-0003-0530-4305) None
here self._get_item(10.1108/jd-12-2013-0166, 0000-0003-0530-4305) None
E
======================================================================
ERROR: test_orcid_get_orcid (index.test.03_resourcefinder.ResourceFinderTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "E:\tesi\index\test\03_resourcefinder.py", line 82, in test_orcid_get_orcid
    self.assertIn("0000-0003-0530-4305", of_2.get_orcid("10.1108/jd-12-2013-0166"))
  File "C:\Program Files\Python36\lib\unittest\case.py", line 1085, in assertIn
    if member not in container:
TypeError: argument of type 'NoneType' is not iterable

----------------------------------------------------------------------
Ran 1 test in 0.077s

FAILED (errors=1)

E:\tesi>

                </pre>
            </code>
        </div>
         
         
          
             
    </div>
  </div>

</div>

    <div class="card">
    <div class="card-header" id="headingEighteen">
      <h5 class="mb-0">
        <button class="btn btn-link collapsed" data-toggle="collapse" data-target="#collapseEighteen" aria-expanded="false" aria-controls="collapseEighteen">
          Week #18 (07-07-21)
        </button>
      </h5>
    </div>
    <div id="collapseEighteen" class="collapse" aria-labelledby="headingEighteen" data-parent="#accordion">
      <div class="card-body">
          <h2>03_resourcefinder, spotted mistake in orcidresourcefinder.py</h2>
          <h3>orcidresourcefinder.py</h3>
          <p> The whole file loading issue was related to a wrong parameter name in <strong> super(ORCIDResourceFinder, self).__init__(date=date, orcid=orcid, issn=issn, id=doi, id_type=OCIManager.doi_type,
                                                  use_api_service=use_api_service) </strong>, inside the init of ORCIDResourceFinder(ApiIDResourceFinder) class: </p>
          <div class="codebg">
            <code>
                <pre>
class ORCIDResourceFinder(ApiIDResourceFinder):
    def __init__(self, date=None, orcid=None, issn=None, doi=None, use_api_service=True, key=None):
        self.key = key
        self.use_api_service = use_api_service
        self.api = "https://pub.orcid.org/v2.1/search?q="
        super(ORCIDResourceFinder, self).__init__(date=date, orcid=orcid, issn=issn, id=doi, id_type=OCIManager.doi_type,
                                                  use_api_service=use_api_service) # HERE doi=doi was changed into id=doi

    def _get_orcid(self, json_obj):
        result = set()
        print("IN REALTA' SONO QUI")
        if json_obj is not None:
            for item in json_obj:
                orcid = item.get("orcid-identifier")
                if orcid is not None:
                    orcid_norm = self.om.normalise(orcid["path"])
                    if orcid_norm is not None:
                        result.add(orcid_norm)

        return result

    def _call_api(self, doi_full):
        if self.use_api_service:
            if self.key is not None:
                self.headers["Authorization"] = "Bearer %s" % self.key
            self.headers["Content-Type"] = "application/json"

            doi = self.dm.normalise(doi_full)
            r = get(self.api + quote("doi-self:\"%s\" OR doi-self:\"%s\"" % (doi, doi.upper())),
                    headers=self.headers, timeout=30)
            if r.status_code == 200:
                r.encoding = "utf-8"
                json_res = loads(r.text)
                return json_res.get("result")

                </pre>
            </code>
          </div>
          <h3>Issue</h3>
          <p> Why the wrong parameter name in orcidresourcefinder.py made the whole file loading process fail? The spotted error led to have an empty container in the following test. I don't fully understand the reason.</p>
          <div class="codebg">
            <code>
                <pre>
of_2 = ORCIDResourceFinder(orcid=CSVManager(self.orcid_path), doi=CSVManager(self.doi_path), use_api_service=False)
container = of_2.get_orcid("10.1108/jd-12-2013-0166")
print("IS CONTAINER NONE?", container)
self.assertIn("0000-0003-0530-4305", container)
                </pre>
            </code>
          </div>
          
          <h2>03_resourcefinder, spotted mistake in crossrefresourcefinder.py</h2>
          <p>The very same mistake was corrected also in the class CrossrefResourceFinder(ApiIDResourceFinder)</p>
          <div class="codebg">
          <code>
          <pre>
class CrossrefResourceFinder(ApiIDResourceFinder):
    def __init__(self, date=None, orcid=None, issn=None, doi=None, use_api_service=True):
        self.use_api_service = use_api_service
        self.api = "https://api.crossref.org/works/"
        super(CrossrefResourceFinder, self).__init__(date=date, orcid=orcid, issn=issn, id=doi, id_type=OCIManager.doi_type,
                                                     use_api_service=use_api_service)
          </pre>
          </code>
          </div>
          
          <h2>03_resourcefinder, spotted mistake in dataciteresourcefinder.py</h2>
          <p>The very same mistake was corrected also in the class DataCiteResourceFinder(ApiIDResourceFinder)</p>
          <div class="codebg">
          <code>
          <pre>
class DataCiteResourceFinder(ApiIDResourceFinder):
    def __init__(self, date=None, orcid=None, issn=None, doi=None, use_api_service=True):
        self.api = "https://api.datacite.org/dois/"
        self.use_api_service = use_api_service
        super(DataCiteResourceFinder, self).__init__(date=date, orcid=orcid, issn=issn, id=doi, id_type=OCIManager.doi_type,
                                                     use_api_service=use_api_service) #doi=doi corrected: id=doi
          </pre>
          </code>
          </div>
          
          <h2>03_resourcefinder, spotted mistake in nihresourcefinder.py</h2>
          <p>The very same mistake was corrected also in the class NIHResourceFinder(ApiIDResourceFinder)</p>
          <div class="codebg">
          <code>
          <pre>
class NIHResourceFinder(ApiIDResourceFinder):
    def __init__(self, date=None, orcid=None, issn=None, pmid=None, use_api_service=True):
        self.use_api_service = use_api_service
        self.api = "https://pubmed.ncbi.nlm.nih.gov/"
        self.baseurl = "https://pubmed.ncbi.nlm.nih.gov/"
        super(NIHResourceFinder, self).__init__(date=date, orcid=orcid, issn=issn, id=pmid, id_type=OCIManager.pmid_type,
                                                     use_api_service=use_api_service)
          </pre>
          </code>
          </div>
          <h2>Further issues</h2>
          <ul>
          <li> Some similar problems still occur for test_datacite_get_issn(self) and for test_handler_share_issn(self), but not for all tests involving issns in general.</li>
          <li> Other issues are related to NIH resource finder only, and in particular to the regex structure. Focus on this before the next meeting.</li>
          </ul>
          

          
          <h2>Material revision and contents index creation (first Five Weeks)</h2>
          <ol>
          <li><strong>Week 1</strong>:UnitTest Introduction, Test code organization and hierarchies</li>
          <li><strong>Week 2</strong>:NIH library data, open_citation_collection.csv sample, stored as source.csv for NIH in noci_dump in test_data folder (+ correspondent citations.csv ), Data Mapping with external mapping tools (www.pmid2cite.com and www.ncbi.nlm.nih.gov ), test case development : 10_noci.py  on the basis of 09_croci.py, prompt basic commands,  Unittest launch from command line, local OC index cloning, “self citation” concept.  </li>
          <li><strong>Week 3</strong>:Metadata file download, Correction of 10_noci and source.csv (headers names must be “citing” and “referenced”), nationalinstitutehealthsource.py creation ( PMIDManager(IdentifierManager), Kevin Boyack mapping with related link</li>
          <li><strong>Week 4</strong>:PMIDManager fix, API request via (https://pubmed.ncbi.nlm.nih.gov/), Status codes, PMIDmanager Test Case (02_identifiermanager.py extension)</li>
          <li><strong>Week 5</strong>:identifier.pmidmanager.py fix, noci. nationalinstitutehealthsource.py fix, test.10_noci fix, Execution process study (overview: Important for all the workflow), Dynamic requests for CROCI, Analysis of the information retrieved by specifying “? format=pubmed” in the API request. </li>
          </ol>
          
          <h2>Fork update</h2>
          <p>All the material produced during the first five weeks of work was checked, tested and published.</p>
          <p>In particular:</p>
          <ol>
          <li><strong>citation\oci.py</strong>:  the class OCIManager(object) was added the two variables doi_type ="doi" and pmid_type="pmid"</li>
          <li><strong>identifier\pmidmanager.py</strong>: file creation</li>
          <li><strong>noci\nationalinstituteofhealthsource.py</strong>: file creation</li>
          <li><strong>storer\citationstorer.py</strong>: strftime from self.cur_time = datetime.now().strftime('%Y-%m-%dT%H:%M:%S') to self.cur_time = datetime.now().strftime('%Y-%m-%dT%H%M%S')</li>
          <li><strong>test\02_identifiermanager.py</strong>: extension to handle pmid</li>
          <li><strong>test\03_resourcefinder.py</strong>: temporary update --> much of the tests were commented, so to focus only on one of them at time, i.e. : test_orcid_get_orcid(self). Second weekly update: all tests were uncommented and all of them were passed successfully </li>
          <li><strong>test\10_noci.py</strong>: file creation</li>
          <li><strong>test_data\noci_dump\citations.csv</strong>: file creation</li>
          <li><strong>test_data\noci_dump\source.csv</strong>: file creation</li>
          <li><strong>test_data\valid_pmid.csv</strong> : file creation and manual update</li>
          </ol>
          <p>Available at: <a href="https://github.com/ariannamorettj/index">https://github.com/ariannamorettj/index</a></p>
    </div>
  </div>

</div>


    <div class="card">
    <div class="card-header" id="headingNineteen">
      <h5 class="mb-0">
        <button class="btn btn-link collapsed" data-toggle="collapse" data-target="#collapseNineteen" aria-expanded="false" aria-controls="collapseNineteen">
          Week #19 (07-27-21)
        </button>
      </h5>
    </div>
    <div id="collapseNineteen" class="collapse" aria-labelledby="headingNineteen" data-parent="#accordion">
      <div class="card-body">

          <h2>Revise at least five meetings (6-10) and update the fork accordingly</h2>
          
          <ol>
          <li><strong>Week 6</strong>: os.makedirs() use (applications and examples), croci dynamic request (successful), attempt of orcid-api key request, test.02_identifiermanager.py fix, test.03_resourcefinder.py fail, test.04_oci.py : UnicodeDecodeError detected and solved, test.05_citationstorer.py: isomorphic graphs error spotted (but not fixed), some hints about possible needed integrations in oci.py and contrasts with noci.nationalinstituteofhealthsource.py, test.test.py: extra test spotted. </li>
          <li><strong>Week 7</strong>: creation of GitHub issues for UnicodeDecodeError, Test 05_citationstorer fail, correction of indentation error on nationalinstituteofhealthsource.py (test.10_noci.py passed), Graphs serialization (g1 and g2 in 05_citationstorer) using n-triple with graph.serialize(format='nt') and rdflib.compare.isomorphic(graph1, graph2),  pmidmanager.py fix (related to the API response format, which is an html and not a string) and correction of the sample invalid pmids used for the tests.</li>
          <li><strong>Week 8</strong>: oci.py extension to handle pmids (skipping encoding and decoding functions since PMIDs are already numeric strings), isomorphism check between graphs in 05_citationstorer.py by serializing them in nt format and sorting their lines in view of a comparison, valid_pmid.csv correction (manual update), 02_identifiermanager.py extension for pmid.</li>
          <li><strong>Week 9</strong>: study of the conversion table for doi-oci (opencitations/oci/blob/master/lookup.csv), issue update of 05_citationstorer.py fail, oci.py extension (correction of previous wrong extension, APIManager class update, id_type addition in OCIManager class), cnc.py integration and issues (APIManager import) </li>
          <li><strong>Week 10</strong>: oci.py update (get_oci(self, doi_1, doi_2, prefix, id_type) : Change argument name to "id_1" and "id_2" instead of doi_1 and doi_2, Create the 2 class variables doi_type and pmid_type in OCIManager), 04_oci test update (add the missing id_type argument where needed), cnc.py update, addition of type information to the six (now seven) elements tuple returned by get_next_citation_data in NationalInstituteHealthSource, CrossrefCitationSource and CrowdSourcedCitationSource, extension of 07_cnc.</li>
          </ol>
          
          <h2>Update the fork with OC changes and the weekly scheduled updates</h2>
          <p>Here below the list of all the updates which were done in the fork this week:</p>
          <ol>
          <li>creation of  finder\nihresourcefinder.py</li>
          <li>creation of all required csv files containing sample data - in order to run the tests (test_data\cnc_valid_pmid.csv, test_data\id_date_pmid.csv, test_data\id_issn_pmid.csv, test_data\id_orcid_pmid.csv)</li>
          <li>update of test_data\valid_pmid.csv (with a new pmid included for testing purposes)</li>
          <li>finder\crossrefresourcefinder.py refactoring of ApiDOIResourceFinder with its new and generalised version: ApiIDResourceFinder, meant to manage pmid too; specification of id_type parameter in  super(CrossrefResourceFinder, self).__init__(date=date, orcid=orcid, issn=issn, id=doi, id_type=OCIManager.doi_type,                            use_api_service=use_api_service)</li>
          <li>finder\dataciteresourcefinder.py: same updates</li>
          <li>finder\orcidresourcefinder.py: same updates</li>
          <li>finder\resourcefinder.py: updates in import, in __init__ parameters, in some variables names, in the name of the class ApiDOIResourceFinder (now ApiIDResourceFinder) and in  __init__ if else structures nested in the main if-else structure starting with if hasattr(self, 'use_api_service'), in order to manage pmid too.</li>
          <li>identifier\doimanager.py: some prints were removed, since the tests they were added for are now being passed</li>
          <li>storer\csvmanager.py: addition of some prints for testing purposes</li>
          <li>test\03_resourcefinder.py: addition of NIHResourceFinder() among the parameters of ResourceFinderHandler, addition of the tests for nih resource finder</li>
          </ol>
          
          <h2>Fix issn issue in resourcefinder tests</h2>
          <p>The issue was due to some typos, such as the wrong name of a parameter specified in one of the tests ("date" instead of "issn", probably due to the wrong reuse of a an already developed test for get_date functions)</p>
          <p>The issue is now fixed and all the tests involving issns pass successfully.</p>
          
          <h2>Fix NIH issue in resourcefinder tests</h2>
          <p>The previous structure of the code implicitly gave for granted that the first pattern matching implemented with regex for the dates in their extended format (day, month and year) was always successful, which was a mistake. The current structure works properly: it was tested successfully.</p>
          
          <div class="codebg">
          <code>
          <pre>
          
    def _get_date(self, txt_obj):
        date = re.search("DP\s+-\s+(\d{4}(\s?(Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec))?(\s?((3[0-1])|([1-2][0-9])|([0]?[1-9])))?)", txt_obj).group(1)
        print("this is date", date)
        re_search = re.search("(\d{4})\s+(Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec)\s+((3[0-1])|([1-2][0-9])|([0]?[1-9]))", date)
        if re_search is not None:
            print("FULL DATE!")
            result = re_search.group(0)
            datetime_object = datetime.strptime(result, '%Y %b %d')
            return datetime.strftime(datetime_object, '%Y-%m-%d')
        else:
            print("NO! DAY MISSING....")
            re_search = re.search("(\d{4})\s+(Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec)", date)
            if re_search is not None:
                result = re_search.group(0)
                datetime_object = datetime.strptime(result, '%Y %b')
                return datetime.strftime(datetime_object, '%Y-%m')
            else:
                print( "NO! MONTH AND DAY MISSING...." )
                re_search = re.search("(\d{4})", date)
                if re_search is not None:
                    result = re.search("(\d{4})", date).group(0)
                    datetime_object = datetime.strptime(result, '%Y')
                    return datetime.strftime(datetime_object, '%Y')
                else:
                    return None
          </pre>
          </code>
          </div>
          
          <h2>Issues</h2>
          <ol>
          <li>Discuss the functioning of the test for shared orcid, issn, dates</li>
          <li>Discuss how to go on with the work concerning cnc and oci</li>
          <li>Discuss the tests failure with the new setting of the code structure</li>
          </ol>
          <p><strong>Share Orcid Test</strong></p>
          <div class="codebg">
          <code>
          <pre>
  def test_handler_share_orcid(self):
        handler = ResourceFinderHandler(
            [CrossrefResourceFinder(), DataCiteResourceFinder(), ORCIDResourceFinder()])
        self.assertTrue(handler.share_orcid("10.1007/s11192-018-2988-z", "10.5281/zenodo.3344898"))
        self.assertFalse(handler.share_issn("10.1007/s11192-018-2988-z", "10.1007/s11192-015-1565-y5"))
          </pre>
          </code>
          </div>
          <p>Is <strong>.share_issn</strong> voluntary?</p>

    </div>
  </div>
</div>

    <div class="card">
    <div class="card-header" id="headingTwenty">
      <h5 class="mb-0">
        <button class="btn btn-link collapsed" data-toggle="collapse" data-target="#collapseTwenty" aria-expanded="false" aria-controls="collapseTwenty">
          Week #20 (08-05-21)
        </button>
      </h5>
    </div>
    <div id="collapseTwenty" class="collapse" aria-labelledby="headingTwenty" data-parent="#accordion">
      <div class="card-body">
      <h2>Citation Storer and Test extension</h2>
      <h3>05_citationstorer.py</h3>
      <div class="codebg">
        <code>
            <pre>
class CitationStorerTest(unittest.TestCase):
    """This class aim at testing the methods of the class CitationStorer."""

    def setUp(self):
        self.citation_data_csv_path = "index%stest_data%scitations_data.csv" % (sep, sep)
        self.citation_prov_csv_path = "index%stest_data%scitations_prov.csv" % (sep, sep)
        self.citation_data_ttl_path = "index%stest_data%scitations_data.ttl" % (sep, sep)
        self.citation_prov_ttl_path = "index%stest_data%scitations_prov.ttl" % (sep, sep)
        self.citation_data_prov_scholix_path = "index%stest_data%scitations_data_prov.scholix" % (sep, sep)
        self.tmp_path = "index%stest_data%stmp" % (sep, sep)
        self.baseurl = "https://w3id.org/oc/index/coci/"

        self.citation_data_csv_path_pmid = "index%stest_data%scitations_pmid_data.csv" % (sep, sep)
        self.citation_prov_csv_path_pmid = "index%stest_data%scitations_pmid_prov.csv" % (sep, sep)
        self.citation_data_ttl_path_pmid = "index%stest_data%scitations_pmid_data.ttl" % (sep, sep)
        self.citation_prov_ttl_path_pmid = "index%stest_data%scitations_pmid_prov.ttl" % (sep, sep)
        self.citation_data_prov_scholix_path_pmid = "index%stest_data%scitations_pmid_data_prov.scholix" % (sep, sep)
        self.tmp_path_pmid = "index%stest_data%stmp_pmid" % (sep, sep)
        self.baseurl_pmid = "https://w3id.org/oc/index/noci/"

        self.ext_local_dir = {
            "ttl": "rdf",
            "scholix": "slx",
            "csv": "csv"
        }

        # Hack for correct handling of date datatypes
        if XSD.gYear in _toPythonMapping:
            _toPythonMapping.pop(XSD.gYear)
        if XSD.gYearMonth in _toPythonMapping:
            _toPythonMapping.pop(XSD.gYearMonth)

    def load_and_store_citations(self, data_path, prov_path, ext):
        tmp_path = self.tmp_path + "_load"

        if exists(tmp_path):
            rmtree(tmp_path)

        origin_citation_list = CitationStorer.load_citations_from_file(
            data_path, prov_path, baseurl="http://dx.doi.org/",
            service_name="OpenCitations Index: COCI", id_type="doi",
            id_shape="http://dx.doi.org/([[XXX__decode]])", citation_type=None)

        cs = CitationStorer(tmp_path, self.baseurl)
        for citation in origin_citation_list:
            cs.store_citation(citation)

        stored_citation_list = CitationStorerTest.get_stored_citation_list(
            tmp_path + sep + "data" + sep + self.ext_local_dir[ext] + sep, ext)

        return origin_citation_list, stored_citation_list

    def load_and_store_citations_pmid(self, data_path, prov_path, ext):
        tmp_path = self.tmp_path_pmid + "_load"

        if exists(tmp_path):
            rmtree(tmp_path)

        origin_citation_list = CitationStorer.load_citations_from_file(
            data_path, prov_path, baseurl="https://pubmed.ncbi.nlm.nih.gov/",
            service_name="OpenCitations Index: NOCI", id_type="pmid",
            id_shape="https://pubmed.ncbi.nlm.nih.gov/([[XXX__decode]])", citation_type=None)

        cs = CitationStorer(tmp_path, self.baseurl_pmid)
        for citation in origin_citation_list:
            cs.store_citation(citation)

        stored_citation_list = CitationStorerTest.get_stored_citation_list_pmid(
            tmp_path + sep + "data" + sep + self.ext_local_dir[ext] + sep, ext)

        return origin_citation_list, stored_citation_list

    def test_load_citations_csv(self):
        origin_citation_list, stored_citation_list = self.load_and_store_citations(
            self.citation_data_csv_path, self.citation_prov_csv_path, "csv")

        self.citations_csv(origin_citation_list, stored_citation_list)

    def test_load_citations_csv_pmid(self):
        origin_citation_list, stored_citation_list = self.load_and_store_citations_pmid(
            self.citation_data_csv_path_pmid, self.citation_prov_csv_path_pmid, "csv")

        self.citations_csv(origin_citation_list, stored_citation_list)

    def citations_csv(self, origin_citation_list, stored_citation_list):
        l1 = [cit.get_citation_csv() for cit in origin_citation_list]
        l2 = [cit.get_citation_csv() for cit in stored_citation_list]
        self.assertEqual(len(l1), len(l2))
        self.assertEqual(set(l1), set(l2))

    def test_load_citations_rdf(self):
        origin_citation_list, stored_citation_list = self.load_and_store_citations(
            self.citation_data_ttl_path, self.citation_prov_ttl_path, "ttl")

        self.citations_rdf(origin_citation_list, stored_citation_list)

    def test_load_citations_rdf_pmid(self):
        origin_citation_list, stored_citation_list = self.load_and_store_citations_pmid(
            self.citation_data_ttl_path_pmid, self.citation_prov_ttl_path_pmid, "ttl")

        self.citations_rdf(origin_citation_list, stored_citation_list)

    def citations_rdf(self, origin_citation_list, stored_citation_list):
        g1 = ConjunctiveGraph()
        g2 = ConjunctiveGraph()

        for idx, cit in enumerate(origin_citation_list):
            for s, p, o, g in cit.get_citation_rdf(
                    self.baseurl, False, False, True).quads((None, None, None, None)):
                g1.add((s, p, o, g))
            for s, p, o, g in stored_citation_list[idx].get_citation_rdf(
                    self.baseurl, False, False, True).quads((None, None, None, None)):
                g2.add((s, p, o, g))
        
        s1 = "\n".join(sorted(g1.serialize(
            format="nt11", encoding="utf-8").decode("utf-8").split("\n")))
        s2 = "\n".join(sorted(g1.serialize(
            format="nt11", encoding="utf-8").decode("utf-8").split("\n")))

        self.assertEqual(s1, s2)

    def citations_rdf_pmid(self, origin_citation_list, stored_citation_list):
        g1 = ConjunctiveGraph()
        g2 = ConjunctiveGraph()

        for idx, cit in enumerate(origin_citation_list):
            for s, p, o, g in cit.get_citation_rdf(
                    self.baseurl_pmid, False, False, True).quads((None, None, None, None)):
                g1.add((s, p, o, g))
            for s, p, o, g in stored_citation_list[idx].get_citation_rdf(
                    self.baseurl_pmid, False, False, True).quads((None, None, None, None)):
                g2.add((s, p, o, g))

        s1 = "\n".join(sorted(g1.serialize(
            format="nt11", encoding="utf-8").decode("utf-8").split("\n")))
        s2 = "\n".join(sorted(g1.serialize(
            format="nt11", encoding="utf-8").decode("utf-8").split("\n")))

        self.assertEqual(s1, s2)

    def test_load_citations_slx(self):
        origin_citation_list, stored_citation_list = self.load_and_store_citations(
            self.citation_data_prov_scholix_path, None, "scholix")

        self.citations_slx(origin_citation_list, stored_citation_list)

    def test_load_citations_slx_pmid(self):
        origin_citation_list, stored_citation_list = self.load_and_store_citations_pmid(
            self.citation_data_prov_scholix_path_pmid, None, "scholix")

        self.citations_slx(origin_citation_list, stored_citation_list)

    def citations_slx(self, origin_citation_list, stored_citation_list):
        l1 = [cit.get_citation_scholix() for cit in origin_citation_list]
        l2 = [cit.get_citation_scholix() for cit in stored_citation_list]
        self.assertEqual(len(l1), len(l2))
        self.assertEqual(set(l1), set(l2))

    def test_store_citation(self):
        tmp_subpath = "_store"
        tmp_path = self.tmp_path + tmp_subpath

        if exists(tmp_path):
            rmtree(tmp_path)

        origin_citation_list = CitationStorer.load_citations_from_file(
            self.citation_data_csv_path, self.citation_prov_csv_path, baseurl="http://dx.doi.org/",
            service_name="OpenCitations Index: COCI", id_type="doi",
            id_shape="http://dx.doi.org/([[XXX__decode]])", citation_type=None)

        cs = CitationStorer(tmp_path, self.baseurl,
                            n_citations_csv_file=4, n_citations_rdf_file=2, n_citations_slx_file=3)
        for citation in origin_citation_list:
            cs.store_citation(citation)

        data_path = tmp_path + sep + "data"
        csv_data_path = data_path + sep + "csv" + sep
        rdf_data_path = data_path + sep + "rdf" + sep
        slx_data_path = data_path + sep + "slx" + sep

        prov_path = tmp_path + sep + "prov" + sep
        csv_prov_path = prov_path + sep + "csv" + sep
        rdf_prov_path = prov_path + sep + "rdf" + sep

        # Check if directories exist
        self.assertTrue(all([exists(p) for p in
                             [csv_data_path, rdf_data_path, slx_data_path, csv_prov_path, rdf_prov_path]]))

        # Check if files exist
        self.assertEqual(len([f for f in glob(csv_data_path + "**" + sep + "*.csv", recursive=True)]), 2)
        self.assertEqual(len([f for f in glob(csv_prov_path + "**" + sep + "*.csv", recursive=True)]), 2)
        self.assertEqual(len([f for f in glob(rdf_data_path + "**" + sep + "*.ttl", recursive=True)]), 3)
        self.assertEqual(len([f for f in glob(rdf_prov_path + "**" + sep + "*.ttl", recursive=True)]), 3)
        self.assertEqual(len([f for f in glob(slx_data_path + "**" + sep + "*.scholix", recursive=True)]), 2)

        # Check if the new stored files contains the same citations of the original one
        stored_citation_list_csv = CitationStorerTest.get_stored_citation_list(csv_data_path, "csv")
        self.citations_csv(origin_citation_list, stored_citation_list_csv)
        stored_citation_list_rdf = CitationStorerTest.get_stored_citation_list(rdf_data_path, "ttl")
        self.citations_rdf(origin_citation_list, stored_citation_list_rdf)
        stored_citation_list_slx = CitationStorerTest.get_stored_citation_list(slx_data_path, "scholix")
        self.citations_slx(origin_citation_list, stored_citation_list_slx)

        # Store again all citations previously stored and checked in they are correctly
        # added to the existing files
        for citation in origin_citation_list:
            cs.store_citation(citation)
        self.assertEqual(len([f for f in glob(csv_data_path + "**" + sep + "*.csv", recursive=True)]), 3)
        self.assertEqual(len([f for f in glob(csv_prov_path + "**" + sep + "*.csv", recursive=True)]), 3)
        self.assertEqual(len([f for f in glob(rdf_data_path + "**" + sep + "*.ttl", recursive=True)]), 6)
        self.assertEqual(len([f for f in glob(rdf_prov_path + "**" + sep + "*.ttl", recursive=True)]), 6)
        self.assertEqual(len([f for f in glob(slx_data_path + "**" + sep + "*.scholix", recursive=True)]), 4)


    def test_store_citation_pmid(self):
        tmp_subpath = "_store"
        tmp_path = self.tmp_path_pmid + tmp_subpath

        if exists(tmp_path):
            rmtree(tmp_path)

        origin_citation_list = CitationStorer.load_citations_from_file(
            self.citation_data_csv_path_pmid, self.citation_prov_csv_path_pmid, baseurl="https://pubmed.ncbi.nlm.nih.gov/",
            service_name="OpenCitations Index: NOCI", id_type="pmid",
            id_shape="https://pubmed.ncbi.nlm.nih.gov/([[XXX__decode]])", citation_type=None)

        cs = CitationStorer(tmp_path, self.baseurl_pmid,
                            n_citations_csv_file=4, n_citations_rdf_file=2, n_citations_slx_file=3)
        for citation in origin_citation_list:
            cs.store_citation(citation)

        data_path = tmp_path + sep + "data"
        csv_data_path = data_path + sep + "csv" + sep
        rdf_data_path = data_path + sep + "rdf" + sep
        slx_data_path = data_path + sep + "slx" + sep

        prov_path = tmp_path + sep + "prov" + sep
        csv_prov_path = prov_path + sep + "csv" + sep
        rdf_prov_path = prov_path + sep + "rdf" + sep

        # Check if directories exist
        self.assertTrue(all([exists(p) for p in
                             [csv_data_path, rdf_data_path, slx_data_path, csv_prov_path, rdf_prov_path]]))

        # Check if files exist
        self.assertEqual(len([f for f in glob(csv_data_path + "**" + sep + "*.csv", recursive=True)]), 2)
        self.assertEqual(len([f for f in glob(csv_prov_path + "**" + sep + "*.csv", recursive=True)]), 2)
        self.assertEqual(len([f for f in glob(rdf_data_path + "**" + sep + "*.ttl", recursive=True)]), 3)
        self.assertEqual(len([f for f in glob(rdf_prov_path + "**" + sep + "*.ttl", recursive=True)]), 3)
        self.assertEqual(len([f for f in glob(slx_data_path + "**" + sep + "*.scholix", recursive=True)]), 2)

        # Check if the new stored files contains the same citations of the original one
        stored_citation_list_csv = CitationStorerTest.get_stored_citation_list_pmid(csv_data_path, "csv")
        self.citations_csv(origin_citation_list, stored_citation_list_csv)
        stored_citation_list_rdf = CitationStorerTest.get_stored_citation_list_pmid(rdf_data_path, "ttl")
        self.citations_rdf_pmid(origin_citation_list, stored_citation_list_rdf)
        stored_citation_list_slx = CitationStorerTest.get_stored_citation_list_pmid(slx_data_path, "scholix")
        self.citations_slx(origin_citation_list, stored_citation_list_slx)

        # Store again all citations previously stored and checked in they are correctly
        # added to the existing files
        for citation in origin_citation_list:
            cs.store_citation(citation)
        self.assertEqual(len([f for f in glob(csv_data_path + "**" + sep + "*.csv", recursive=True)]), 3)
        self.assertEqual(len([f for f in glob(csv_prov_path + "**" + sep + "*.csv", recursive=True)]), 3)
        self.assertEqual(len([f for f in glob(rdf_data_path + "**" + sep + "*.ttl", recursive=True)]), 6)
        self.assertEqual(len([f for f in glob(rdf_prov_path + "**" + sep + "*.ttl", recursive=True)]), 6)
        self.assertEqual(len([f for f in glob(slx_data_path + "**" + sep + "*.scholix", recursive=True)]), 4)

    @staticmethod
    def get_stored_citation_list(data_path, ext):
        stored_citation_list = []

        for f in [f for f in glob(data_path + "**/*." + ext, recursive=True)]:
            stored_citation_list.extend(CitationStorer.load_citations_from_file(
                f, f.replace("%sdata%s" % (sep, sep), "%sprov%s" % (sep, sep)),
                baseurl="http://dx.doi.org/", service_name="OpenCitations Index: COCI", id_type="doi",
                id_shape="http://dx.doi.org/([[XXX__decode]])", citation_type=None))

        return stored_citation_list\


    @staticmethod
    def get_stored_citation_list_pmid(data_path, ext):
        stored_citation_list = []

        for f in [f for f in glob(data_path + "**/*." + ext, recursive=True)]:
            stored_citation_list.extend(CitationStorer.load_citations_from_file(
                f, f.replace("%sdata%s" % (sep, sep), "%sprov%s" % (sep, sep)),
                baseurl="https://pubmed.ncbi.nlm.nih.gov/", service_name="OpenCitations Index: NOCI", id_type="pmid",
                id_shape="https://pubmed.ncbi.nlm.nih.gov/([[XXX__decode]])", citation_type=None))

        return stored_citation_list
            </pre>
        </code>
      </div>
      <p>Extra functions were added where needed, while the generic ones were reused. Further, all the required support files with test data were created where necessary and the SetUp was extended accordingly.</p>
      <p><strong>Note:</strong> The structure of the added functions for now is quite basic and mimics for the most the reference functions. Should they be integrated with if-else structures differentiating the process for pmids from the one for dois where necessary or is it enough to leave them as they are?</p>
      
      <h2>Citation Source and Test</h2>
      <h3>06_citationsource</h3>

    <div class="codebg">
        <code>
            <pre>
class CitationSourceTest(unittest.TestCase):
    """This class aim at testing the methods of the class CSVManager."""

    def setUp(self):
        info_file_path = "index%stest_data%stmp_store%sdata%s.dir_citation_source" % (sep, sep, sep, sep)
        if exists(info_file_path):
            remove(info_file_path)
        info_file_path_pmid = "index%stest_data%stmp_store_pmid%sdata%s.dir_citation_source" % (sep, sep, sep, sep)
        if exists(info_file_path_pmid):
            remove(info_file_path_pmid)
        self.oci = OCIManager(lookup_file="index%stest_data%slookup_full.csv" % (sep, sep))
        self.citation_list = CitationStorer.load_citations_from_file(
            "index%stest_data%scitations_data.csv" % (sep, sep),
            "index%stest_data%scitations_prov.csv" % (sep, sep),
            baseurl="http://dx.doi.org/",
            service_name="OpenCitations Index: COCI", id_type="doi",
            id_shape="http://dx.doi.org/([[XXX__decode]])", citation_type=None)
        #quale id shape per pmid?
        self.citation_list_pmid = CitationStorer.load_citations_from_file(
            "index%stest_data%scitations_pmid_data.csv" % (sep, sep),
            "index%stest_data%scitations_pmid_prov.csv" % (sep, sep),
            baseurl="https://pubmed.ncbi.nlm.nih.gov/",
            service_name="OpenCitations Index: NOCI", id_type="pmid",
            id_shape="https://pubmed.ncbi.nlm.nih.gov/([[XXX__decode]])", citation_type=None)

    def __create_citation(self, citing, cited, created, timespan, journal_sc, author_sc, id_type):
        if id_type == OCIManager.doi_type:
            return Citation(
                self.oci.get_oci(citing, cited, "020"),
                "http://dx.doi.org/" + quote(citing), None,
                "http://dx.doi.org/" + quote(cited), None,
                created, timespan,
                1, "https://w3id.org/oc/index/prov/ra/1",
                "https://api.crossref.org/works/" + quote(citing), "2018-01-01T00:00:00",
                "OpenCitations Index: COCI", "doi", "http://dx.doi.org/([[XXX__decode]])", None,
                journal_sc, author_sc, prov_description="Creation of the citation")
        elif id_type == OCIManager.pmid_type: #RICONTROLLARE QUESTA PARTE CON DATI NECESSARI PER I TEST
            #dubbi su: "https://doi.org/10.35092/yhjc.c.4586573.v16" + quote(citing) : così non esce chiaramente niente perché la mia risorsa è un file, come faccio?
            #dubbi su: cosa sia il corrispettivo dell'id_shape per pmid, ovvero di "http://dx.doi.org/([[XXX__decode]])"
            return Citation(
                self.oci.get_oci(citing, cited, "0160"),
                "https://pubmed.ncbi.nlm.nih.gov/" + quote(citing), None,
                "https://pubmed.ncbi.nlm.nih.gov/" + quote(cited), None,
                created, timespan,
                1, "https://w3id.org/oc/index/prov/ra/1",
                "https://doi.org/10.35092/yhjc.c.4586573.v16" + quote(citing), "2018-01-01T00:00:00",
                "OpenCitations Index: NOCI", "pmid", "https://pubmed.ncbi.nlm.nih.gov/([[XXX__decode]])", None,
                journal_sc, author_sc, prov_description="Creation of the citation")
        else:
            print("the specified id_type is not compliant")

    def __citations_csv(self, origin_citation_list, stored_citation_list):
        l1 = [cit.get_citation_csv() for cit in origin_citation_list]
        l2 = [cit.get_citation_csv() for cit in stored_citation_list]
        self.assertEqual(len(l1), len(l2))
        self.assertEqual(set(l1), set(l2))

    def test_get_next_citation_data(self):
        cs = CSVFileCitationSource("index%stest_data%stmp_store%sdata" % (sep, sep, sep))
        citation_1 = self.__create_citation(*cs.get_next_citation_data(), "doi")
        citation_2 = self.__create_citation(*cs.get_next_citation_data(), "doi")
        self.__citations_csv(self.citation_list[:2], [citation_1, citation_2])
        cs = CSVFileCitationSource( "index%stest_data%stmp_store%sdata" % (sep, sep, sep) )
        citation_3 = self.__create_citation( *cs.get_next_citation_data(), "doi" )
        citation_4 = self.__create_citation( *cs.get_next_citation_data(), "doi" )
        citation_5 = self.__create_citation( *cs.get_next_citation_data(), "doi" )
        citation_6 = self.__create_citation( *cs.get_next_citation_data(), "doi" )
        self.__citations_csv( self.citation_list[2:6], [citation_3, citation_4, citation_5, citation_6] )

        idx = 0
        while cs.get_next_citation_data() is not None:
            idx += 1
        self.assertEqual( idx, 6 )

    def test_get_next_citation_data_pmid(self):
        cs = CSVFileCitationSource("index%stest_data%stmp_pmid_store%sdata" % (sep, sep, sep))
        citation_1 = self.__create_citation(*cs.get_next_citation_data(), "pmid")
        citation_2 = self.__create_citation(*cs.get_next_citation_data(), "pmid")
        self.__citations_csv(self.citation_list_pmid[:2], [citation_1, citation_2])
        cs = CSVFileCitationSource("index%stest_data%stmp_pmid_store%sdata" % (sep, sep, sep) )
        citation_3 = self.__create_citation( *cs.get_next_citation_data(), "pmid" )
        citation_4 = self.__create_citation( *cs.get_next_citation_data(), "pmid" )
        citation_5 = self.__create_citation( *cs.get_next_citation_data(), "pmid" )
        citation_6 = self.__create_citation( *cs.get_next_citation_data(), "pmid" )
        self.__citations_csv( self.citation_list_pmid[2:6], [citation_3, citation_4, citation_5, citation_6] )

        idx = 0
        while cs.get_next_citation_data() is not None:
            idx += 1
        self.assertEqual( idx, 6 )
            </pre>
        </code>
      </div>
      <p>SetUp extension and differentiation of the process for pmid where required.</p>
      <p>Id_shape issue should be discussed together.</p>
      
      <h2>Cnc, Data Handler and Test</h2>
      
      <h3>cnc.py</h3>
      <div class="codebg">
        <code>
            <pre>
from argparse import ArgumentParser
from urllib.parse import quote
from datetime import datetime
from os import sep
from index.citation.oci import Citation
from index.storer.citationstorer import CitationStorer
from index.storer.datahandler import FileDataHandler
import ray


@ray.remote
class ParallelFileDataHandler( FileDataHandler ):
    """This class makes the FileDataHandler class as a remote object"""
    pass


def execute_workflow(idbaseurl, baseurl, pclass, inp, doi_file, date_file,
                     orcid_file, issn_file, orcid, lookup, data, prefix, agent,
                     source, service, verbose, no_api, process_number, id_type):
    if process_number > 1:  # Run process in parallel via RAY
        ray.init( num_cpus=process_number )
        p_handler = ParallelFileDataHandler.remote( pclass, inp, lookup )
        print( "[parallel] Initialising the data handler" )
        id_remote_init = p_handler.init.remote( data, doi_file, date_file, orcid_file, issn_file, orcid, no_api, id_type)
        ray.wait( [id_remote_init] )  # wait until the handler is not ready
        print( "[parallel] The data handler has been initialised, and the extraction of citations begins" )
        futures = [_parallel_extract_citations.remote( data, idbaseurl, baseurl, prefix, agent, source,
                                                       service, verbose, p_handler, str( i ) )
                   for i in range( process_number - 1 )]
        ray.get( futures )
        return ray.get( p_handler.get_values.remote() )
    else:
        p_handler = FileDataHandler( pclass, inp, lookup )
        print( "[standalone] Initialising the data handler" )
        p_handler.init( data, doi_file, date_file, orcid_file, issn_file, orcid, no_api, id_type)
        print( "[standalone] The data handler has been initialised, and the extraction of citations begins" )
        _extract_citations( data, idbaseurl, baseurl, prefix, agent, source, service, verbose, p_handler )
        return p_handler.get_values()


@ray.remote
def _parallel_extract_citations(data, idbaseurl, baseurl, prefix, agent,
                                source, service, verbose, p_handler, suffix):
    return _extract_citations( data, idbaseurl, baseurl, prefix, agent,
                               source, service, verbose, p_handler, suffix, True )


def _extract_citations(data, idbaseurl, baseurl, prefix, agent,
                       source, service, verbose, p_handler, suffix="", parallel=False):
    h_get_next_citation_data = lambda h: \
        ray.get( h.get_next_citation_data.remote() ) if parallel \
            else h.get_next_citation_data()
    h_get_oci = lambda h, citing, cited, prefix: \
        ray.get( h.get_oci.remote( citing, cited, prefix ) ) if parallel \
            else h.get_oci( citing, cited, prefix )
    h_oci_exists = lambda h, oci: \
        ray.get( h.oci_exists.remote( oci ) ) if parallel \
            else h.oci_exists( oci )
    h_are_valid = lambda h, citing, cited: \
        ray.get( h.are_valid.remote( citing, cited ) ) if parallel \
            else h.are_valid( citing, cited )
    h_get_date = lambda h, id_string: \
        ray.get( h.get_date.remote( id_string ) ) if parallel \
            else h.get_date( id_string )
    h_share_issn = lambda h, citing, cited: \
        ray.get( h.share_issn.remote( citing, cited ) ) if parallel \
            else h.share_issn( citing, cited )
    h_share_orcid = lambda h, citing, cited: \
        ray.get( h.share_orcid.remote( citing, cited ) ) if parallel \
            else h.share_orcid( citing, cited )

    BASE_URL = idbaseurl
    DATASET_URL = baseurl + "/" if not baseurl.endswith( "/" ) else baseurl

    cit_storer = CitationStorer( data, DATASET_URL, suffix=suffix )
    next_citation = h_get_next_citation_data( p_handler )

    while next_citation is not None:
        citing, cited, created, timespan, journal_sc, author_sc = next_citation
        oci = h_get_oci( p_handler, citing, cited, prefix )
        oci_noprefix = oci.replace( "oci:", "" )

        if not h_oci_exists( p_handler, oci_noprefix ):
            if h_are_valid( p_handler, citing, cited ):
                if created is None:
                    citing_date = h_get_date( p_handler, citing )
                else:
                    citing_date = created
                cited_date = h_get_date( p_handler, cited )
                if journal_sc is None or type( journal_sc ) is not bool:
                    journal_sc = h_share_issn( p_handler, citing, cited )
                if author_sc is None or type( author_sc ) is not bool:
                    author_sc = h_share_orcid( p_handler, citing, cited )

                if created is not None and timespan is not None:
                    cit = Citation( oci,
                                    BASE_URL + quote( citing ), None,
                                    BASE_URL + quote( cited ), None,
                                    created, timespan,
                                    1, agent, source, datetime.now().strftime( '%Y-%m-%dT%H:%M:%S' ),
                                    service, "doi", BASE_URL + "([[XXX__decode]])", "reference",
                                    journal_sc, author_sc,
                                    None, "Creation of the citation", None )
                else:
                    cit = Citation( oci,
                                    BASE_URL + quote( citing ), citing_date,
                                    BASE_URL + quote( cited ), cited_date,
                                    None, None,
                                    1, agent, source, datetime.now().strftime( '%Y-%m-%dT%H:%M:%S' ),
                                    service, "doi", BASE_URL + "([[XXX__decode]])", "reference",
                                    journal_sc, author_sc,
                                    None, "Creation of the citation", None )

                cit_storer.store_citation( cit )

                if verbose:
                    print( "Create citation data for '%s' between ID '%s' and ID '%s'" % (oci, citing, cited) )
            else:
                if verbose:
                    print( "WARNING: some IDs, among '%s' and '%s', do not exist" % (citing, cited) )
        else:
            if verbose:
                print( "WARNING: the citation between ID '%s' and ID '%s' has been already processed" %
                       (citing, cited) )

        next_citation = h_get_next_citation_data( p_handler )


if __name__ == "__main__":
    arg_parser = ArgumentParser( "cnc.py (Create New Citations",
                                 description="This tool allows one to take a series of entity-to-entity"
                                             "citation data, and to store it according to CSV used by"
                                             "the OpenCitations Indexes so as to be added to an Index. It uses"
                                             "several online services to check several things to create the"
                                             "final CSV/TTL/Scholix files." )

    arg_parser.add_argument( "-c", "--pclass", required=True,
                             help="The name of the class of data source to use to process citatation data.",
                             choices=['csv', 'crossref', 'croci'] )
    arg_parser.add_argument( "-i", "--input", required=True,
                             help="The input file/directory to provide as input of the specified input "
                                  "Python file (using -p)." )
    arg_parser.add_argument( "-d", "--data", required=True,
                             help="The directory containing all the CSV files already added in the Index, "
                                  "including data and provenance files." )
    arg_parser.add_argument( "-o", "--orcid", default=None,
                             help="ORCID API key to be used to query the ORCID API." )
    arg_parser.add_argument( "-l", "--lookup", required=True,
                             help="The lookup table that must be used to produce OCIs." )
    arg_parser.add_argument( "-b", "--baseurl", required=True, default="",
                             help="The base URL of the dataset" )
    arg_parser.add_argument( "-ib", "--idbaseurl", required=True, default="",
                             help="The base URL of the identifier of citing and cited entities, if any" )
    arg_parser.add_argument( "-doi", "--doi_file", default=None,
                             help="The file where the valid and invalid DOIs are stored." )
    arg_parser.add_argument( "-date", "--date_file", default=None,
                             help="The file that maps id of bibliographic resources with their publication date." )
    arg_parser.add_argument( "-orcid", "--orcid_file", default=None,
                             help="The file that maps id of bibliographic resources with the ORCID of its authors." )
    arg_parser.add_argument( "-issn", "--issn_file", default=None,
                             help="The file that maps id of bibliographic resources with the ISSN of the journal "
                                  "they have been published in." )
    arg_parser.add_argument( "-px", "--prefix", default="",
                             help="The '0xxx0' prefix to use for creating the OCIs." )
    arg_parser.add_argument( "-a", "--agent", required=True, default="https://w3id.org/oc/index/prov/pa/1",
                             help="The URL of the agent providing or processing the citation data." )
    arg_parser.add_argument( "-s", "--source", required=True,
                             help="The URL of the source from where the citation data have been extracted." )
    arg_parser.add_argument( "-sv", "--service", required=True,
                             help="The name of the service that will made available the citation data." )
    arg_parser.add_argument( "-v", "--verbose", action="store_true", default=False,
                             help="Print the messages on screen." )
    arg_parser.add_argument( "-na", "--no_api", action="store_true", default=False,
                             help="Tell the tool explicitly not to use the APIs of the various finders." )
    arg_parser.add_argument( "-pn", "--process_number", default=1, type=int,
                             help="The number of parallel process to run for working on the creation of citations." )
    arg_parser.add_argument( "-type", "--id_type", default=None,
                             help="The type of the specified id, either doi or pmid." )

    args = arg_parser.parse_args()

    new_citations_added, citations_already_present, error_in_dois_existence = \
        execute_workflow( args.idbaseurl, args.baseurl, args.pclass,
                          args.input, args.doi_file, args.date_file, args.orcid_file,
                          args.issn_file, args.orcid, args.lookup, args.data,
                          args.prefix, args.agent, args.source, args.service,
                          args.verbose, args.no_api, args.process_number, args.id_type)

    print( "\n# Summary\n"
           "Number of new citations added to the OpenCitations Index: %s\n"
           "Number of citations already present in the OpenCitations Index: %s\n"
           "Number of citations with invalid DOIs: %s" %
           (new_citations_added, citations_already_present, error_in_dois_existence) )
            </pre>
        </code>
      </div>
      <p>id_type parameter was added where necessary. Some further changes should be done for the strings which only show messages referring to "dois", where "ids" would be more appropriate. However, for now I left the changes which were just formal and that therefore could be done later. </p>
      
      <h3>datahandler.py</h3>
      <div class="codebg">
        <code>
            <pre>
from abc import abstractmethod
from index.citation.citationsource import CSVFileCitationSource
from index.coci.crossrefcitationsource import CrossrefCitationSource
from index.croci.crowdsourcedcitationsource import CrowdsourcedCitationSource
from index.noci.nationalinstituteofhealthsource import NIHCitationSource
from index.storer.csvmanager import CSVManager
from index.identifier.doimanager import DOIManager
from index.identifier.pmidmanager import PMIDManager
from index.finder.orcidresourcefinder import ORCIDResourceFinder
from index.finder.dataciteresourcefinder import DataCiteResourceFinder
from index.finder.crossrefresourcefinder import CrossrefResourceFinder
from index.finder.nihresourcefinder import NIHResourceFinder
from index.finder.resourcefinder import ResourceFinderHandler
from index.citation.oci import OCIManager
from os import sep


class DataHandler(object):
    """A class acting as a proxy for accessing specific data useful to create citations."""
    _source_classes = {
        "csv": CSVFileCitationSource,
        "crossref": CrossrefCitationSource,
        "croci": CrowdsourcedCitationSource,
        "noci" : NIHCitationSource,
    }

    def __init__(self, pclass, inp, lookup):
        # print("pclass", pclass)
        # print("inp", inp)
        # print("lookup", lookup)
        # print("_source_classes[pclass]", self._source_classes[pclass])
        self.cs = self._source_classes[pclass](inp)
        self.oci_manager = OCIManager(lookup_file=lookup)

        self.new_citations_added = 0
        self.citations_already_present = 0
        self.error_in_ids_existence = 0

    def get_values(self):
        """It returns the number of citations added, already present, and with id errors."""
        return self.new_citations_added, self.citations_already_present, self.error_in_ids_existence

    def get_oci(self, citing, cited, prefix):
        """It returns an OCI computed considering the identifiers of the citing and cited
        entities and a prefix."""
        return self.oci_manager.get_oci(citing, cited, prefix)

    def get_next_citation_data(self):
        """It returns the next available citation data in the citation data source specified."""
        return self.cs.get_next_citation_data()

    @abstractmethod
    def init(self, *params):
        """This method allows one to initialise all internal variable to make
        the data handler works corretly. It must be implemented in each particular
        subclass."""
        pass

    @abstractmethod
    def are_valid(self, citing, cited):
        """This method checks if the identifiers of the citing and cited entities
        are both valid (it returns True in this case, otherwise it returns False)."""
        pass

    @abstractmethod
    def share_orcid(self, citing, cited):
        """This method checks if the citing and cited entities share at least
        one ORCID (it returns True in this case, otherwise it returns False)."""
        pass

    @abstractmethod
    def share_issn(self, citing, cited):
        """This method checks if the citing and cited entities share at least
        one ISSN (it returns True in this case, otherwise it returns False)."""
        pass

    @abstractmethod
    def get_date(self, id_string):
        """This method retrives the date of publication of the entity indetified by
        the input string."""
        pass

    @abstractmethod
    def oci_exists(self, oci):
        """This method checks if the OCI in input has been already added to a
        database (it returns True in this case, otherwise it returns False)."""
        pass


class FileDataHandler(DataHandler):
    @staticmethod
    def _create_csv(doi_file, date_file, orcid_file, issn_file):
        valid_doi = CSVManager(csv_path=doi_file)
        id_date = CSVManager(csv_path=date_file)
        id_orcid = CSVManager(csv_path=orcid_file)
        id_issn = CSVManager(csv_path=issn_file)

        return valid_doi, id_date, id_orcid, id_issn

    def init(self, data, doi_file, date_file, orcid_file, issn_file, orcid, no_api, id_type):
        valid_doi, id_date, id_orcid, id_issn = \
            FileDataHandler._create_csv(doi_file, date_file, orcid_file, issn_file)

        if id_type == OCIManager.doi_type:
            self.id_manager = DOIManager(valid_doi, use_api_service=not no_api)
        elif id_type == OCIManager.pmid_type:
            self.id_manager = PMIDManager(valid_doi, use_api_service=not no_api)
        else:
            print("the id_type specified is not compliant")

        crossref_rf = CrossrefResourceFinder(
            date=id_date, orcid=id_orcid, issn=id_issn, doi=valid_doi, use_api_service=not no_api)
        datacite_rf = DataCiteResourceFinder(
            date=id_date, orcid=id_orcid, issn=id_issn, doi=valid_doi, use_api_service=not no_api)
        orcid_rf = ORCIDResourceFinder(
            date=id_date, orcid=id_orcid, issn=id_issn, doi=valid_doi,
            use_api_service=True if orcid is not None and not no_api else False, key=orcid)
        nih_rf = NIHResourceFinder(
            date=id_date, orcid=id_orcid, issn=id_issn, pmid=valid_doi, use_api_service=not no_api)

        self.rf_handler = ResourceFinderHandler([crossref_rf, datacite_rf, orcid_rf, nih_rf])

        self.exi_ocis = CSVManager.load_csv_column_as_set(
            data + sep + "data", "oci")

    def are_valid(self, citing, cited):
        result = self.id_manager.is_valid(citing) and \
                 self.id_manager.is_valid(cited)
        if result:
            self.new_citations_added += 1
        else:
            self.error_in_ids_existence += 1

        return result

    def share_orcid(self, citing, cited):
        return self.rf_handler.share_orcid(citing, cited)

    def share_issn(self, citing, cited):
        return self.rf_handler.share_issn(citing, cited)

    def get_date(self, id_string):
        return self.rf_handler.get_date(id_string)

    def oci_exists(self, oci):
        result = oci in self.exi_ocis
        if result:
            self.citations_already_present += 1
        else:
            self.exi_ocis.add(oci)
        return result
            </pre>
        </code>
      </div>
      <p>NIH Resource Finder and Citation Storer were integrated in the process and the process was differentiated according to the type of id were necessary</p>
      
      <h3>test_cnc.py</h3>
      <div class="codebg">
        <code>
            <pre>
class CreateNewCitationsTestPmid(unittest.TestCase):
    def setUp(self):
        self.idbaseurl = "https://pubmed.ncbi.nlm.nih.gov/"
        self.baseurl = "https://w3id.org/oc/index/noci/"#per ora non porta a niente
        self.python = "index%scitation%scitationsource.py" % (sep, sep)
        self.pclass = "csv"
        self.input = "index%stest_data%scitations_partial_pmid.csv" % (sep, sep)
        self.pmid_file = "index%stest_data%scnc_valid_pmid.csv" % (sep, sep)
        self.date_file = "index%stest_data%scnc_id_date_pmid.csv" % (sep, sep)
        self.orcid_file = "index%stest_data%scnc_id_orcid_pmid.csv" % (sep, sep)
        self.issn_file = "index%stest_data%scnc_id_issn_pmid.csv" % (sep, sep)
        self.orcid = None
        self.lookup = "index%stest_data%slookup_full.csv" % (sep, sep)
        self.data = "index%stest_data%stmp_workflow_pmid" % (sep, sep)
        self.prefix = "0160"
        self.agent = "https://w3id.org/oc/index/prov/ra/1"
        self.source = "https://doi.org/10.35092/yhjc.c.4586573.v16"
        self.service = "OpenCitations Index: NOCI"
        self.verbose = True
        self.no_api = False
        self.id_type = OCIManager.pmid_type


        self.citation_list = self.__load_citations("index%stest_data%scitations_pmid_data.csv" % (sep, sep),
                                                   "index%stest_data%scitations_pmid_prov.csv" % (sep, sep))
        self.data_path = self.data + sep + "data" + sep + "**" + sep + "*.csv"
        self.prov_path = self.data + sep + "prov" + sep + "**" + sep + "*.csv"

        if exists(self.data):
            rmtree(self.data)

    def __load_citations(self, data, prov):
        return CitationStorer.load_citations_from_file(data, prov, baseurl="https://pubmed.ncbi.nlm.nih.gov/",
            service_name=self.service, id_type="pmid",
            id_shape="https://pubmed.ncbi.nlm.nih.gov/([[XXX]])", citation_type=None) #NOTE: discuss id_shape -- il doi può arrivare con caratteri strani,, dvo decodarlo
#xxx sostutuisce con l'identificativo di quella specifica citazione

    def __citations_csv(self, origin_citation_list, stored_citation_list):
        l1 = [cit.get_citation_csv() for cit in origin_citation_list]
        l2 = [cit.get_citation_csv() for cit in stored_citation_list]
        self.assertEqual(len(l1), len(l2))
        self.assertEqual(set(l1), set(l2))

    def __test_citations(self):
        data_csv = glob(self.data_path, recursive=True)
        prov_csv = glob(self.prov_path, recursive=True)
        self.assertEqual(len(data_csv), 1)
        self.assertEqual(len(prov_csv), 1)
        self.__citations_csv(self.citation_list, self.__load_citations(data_csv[0], prov_csv[0]))

    def test_execute_workflow(self):
        new_citations_added, citations_already_present, error_in_pmids_existence = \
            execute_workflow( self.idbaseurl, self.baseurl, self.pclass, self.input, self.pmid_file,
                              self.date_file, self.orcid_file, self.issn_file, self.orcid, self.lookup, self.data,
                              self.prefix, self.agent, self.source, self.service, self.verbose, self.no_api, 1, self.id_type)

        # la barra serve per andare a capo, restituisce tre numeri
        self.assertEqual(new_citations_added, 6)
        self.assertEqual(citations_already_present, 0)
        self.assertEqual(error_in_pmids_existence, 0)
        self.__test_citations()

        new_citations_added, citations_already_present, error_in_pmids_existence = \
            execute_workflow( self.idbaseurl, self.baseurl, self.pclass, self.input, self.pmid_file,
                              self.date_file, self.orcid_file, self.issn_file, self.orcid, self.lookup, self.data,
                              self.prefix, self.agent, self.source, self.service, self.verbose, self.no_api, 1, self.id_type)
        self.assertEqual(new_citations_added, 0)
        self.assertEqual(citations_already_present, 6)
        self.assertEqual(error_in_pmids_existence, 0)
        self.__test_citations()




if __name__ == '__main__':
    unittest.main()  # Launch with "python text_cnc.py"
            </pre>
        </code>
      </div>
      <p>The test was extended: SetUp was duplicated so to manage separately pmids and dois and the needed test data were created for pmids also. The folder tmp_workflow_pmid stores now the data processed for pmids.</p>
      
      <h2>Issues</h2>
      <ol>
        <li>Even if every test is passed now and the testing data that I created work properly, not al of them are real. In some cases I just used invented data in order to check the functioning of the process. Further, running test_cnc I realised that I should check also the inner coherence of the data I created initially.</li>
        <li>cnc parallel process seems not to be tested, so I tried to modify one parameter in the test for the regular process in oreder to see if the extension worked also for the parallel process. However, I got this message: <em>RuntimeError: Maybe you called ray.init twice by accident? This error can be suppressed by passing in 'ignore_reinit_error=True' or by calling 'ray.shutdown()' prior to 'ray.init()'.</em> </li>
        <li>GitHub Desktop can not locate the repository of my fork, which is strage, since I didn't move it at all. For this reason I could not update the fork properly and further I can not see clearly the changes and the extensions I did this week, which would be necessary in order to comment them in detail. For now I copied the files in an external hard disk, but I really need to solve this problem as soon as possible.</li>
      </ol>
      
      <h2>Service call example for NOCI</h2>
      <p>How to call cnc for NOCI from the terminal</p>
      <div class="codebg">
        <code>
            <pre>
            # python cnc.py -ib "https://pubmed.ncbi.nlm.nih.gov/" -b "https://w3id.org/oc/index/noci/" -c "csv" -i "index/test_data/citations_partial_pmid.csv" -doi "index/noci_test/pmid.csv" -orcid "index/noci_test/orcid.csv" -date "index/noci_test/date.csv" -issn "index/noci_test/issn.csv" -l "index/test_data/lookup_full.csv" -d "index/noci_test" -px "0160" -a "https://w3id.org/oc/index/prov/ra/1" -s "https://doi.org/10.35092/yhjc.c.4586573.v16" -sv "OpenCitations Index: NOCI" -type "pmid" -v
            </pre>
        </code>
      </div>
          
    </div>
  </div>
</div>

<div class="card">
    <div class="card-header" id="headingTwenty-one">
      <h5 class="mb-0">
        <button class="btn btn-link collapsed" data-toggle="collapse" data-target="#collapseTwenty-one" aria-expanded="false" aria-controls="collapseTwenty-one">
          Week #21 (08-12-21)
        </button>
      </h5>
    </div>
    <div id="collapseTwenty-one" class="collapse" aria-labelledby="headingTwenty-one" data-parent="#accordion">
      <div class="card-body">
          <h2>COCI glob.py study </h2>
          <p>Analysis of COCI glob, so to understand how to recreate a glob with the same functions for pmids</p>
          <div class="codebg">
                  <iframe src="https://trinket.io/embed/python/861045261a" width="100%" height="356" frameborder="0" marginwidth="0" marginheight="0" allowfullscreen></iframe>
          </div>
          
          <h2> Analysis of the structure of the two files downloaded from iCite</h2>
          <h3>open_citation_collection</h3>
          <div class="codebg">
            <code>
                <pre>
citing,referenced
1,4972128
1,4332837
1,13672941
1,14203183
1,14161139
1,14323813
1,4990046
1,4988806
2,4150960
2,4356257
2,4846745
2,4357832
2,4683494
2,4414857
2,4337195
2,4747846
2,4266012
2,4147828
2,1111570
2,4366080
2,1133382
2,4364802
2,4709237
2,4733399
2,4379057
2,17742850
3,4632671
3,4977579
3,4992430
3,4958988
3,14417215
3,4992429
3,4621826
3,804171
3,4263471
3,4625501
3,4399045
3,4987292
3,4628675
                </pre>
            </code>
          </div>
          
          <h3>icite_metadata</h3>
          <div class="codebg">
          <code>
            <pre>
"pmid","doi","title","authors","year","journal","is_research_article","citation_count","field_citation_rate","expected_citations_per_year","citations_per_year","relative_citation_ratio","nih_percentile","human","animal","molecular_cellular","x_coord","y_coord","apt","is_clinical","cited_by_clin","cited_by","references","provisional"
1,"10.1016/0006-2944(75)90147-7","Formate assay in body fluids: application in methanol poisoning.","A B Makar, K E McMartin, M Palese, T R Tephly",1975,"Biochem Med",TRUE,83,5.05644116019705,,1.80434782608696,,,0.17,0.33,0.5,-0.144337567297406,-0.25,0.25,FALSE,"","27354968 6430597 27548239 7055965 30849241 21923939 6525992 7004236 33554654 34013366 34176544 34122605 6875695 109089 6615550 7396890 31264500 518695 33134728 27574557 30612633 28159467 33825562 2920026 24058668 2219121 2566887 6915 2733395 2334642 21569229 21912457 32727301 8461035 7265415 18553624 17016 20870 941156 6767446 28002644 32820233 19454010 3112558 33872434 24589977 6383751 7361809 7471469 9750739 34142875 28851427 29460824 31371922 33561370 30186270 3754027 7205659 99844 405471 24004664 31123410 115004 6815832 8040258 23350017 25489175 32660571 3537623 728186 20184691 7165305 32733622 7230276 214088 3789485 6968503 3426949 27555514 6549198 31544580 6634838 32765117","4972128 4332837 13672941 14203183 14161139 14323813 4990046 4988806","No"
2,"10.1016/0006-291x(75)90482-9","Delineation of the intimate details of the backbone conformation of pyridine nucleotide coenzymes in aqueous solution.","K S Bose, R H Sarma",1975,"Biochem Biophys Res Commun",TRUE,15,5.56685044103907,,0.326086956521739,,,0,0,1,-0.866025403784439,-0.5,0.05,FALSE,"21542697","6267127 26376 29458872 25548608 190032 22558138 26259654 31435170 28302598 21542697 26140007 890065 31544580 990401 39671","4150960 4356257 4846745 4357832 4683494 4414857 4337195 4747846 4266012 4147828 1111570 4366080 1133382 4364802 4709237 4733399 4379057 17742850","No"
3,"10.1016/0006-291x(75)90498-2","Metal substitutions incarbonic anhydrase: a halide ion probe study.","R J Smith, R G Bryant",1975,"Biochem Biophys Res Commun",TRUE,13,5.13560082085996,,0.282608695652174,,,0.14,0.29,0.57,-0.247435829652697,-0.285714285714286,0.25,FALSE,"","25624746 33776281 32427033 28053241 22558138 24349293 7022113 24775716 27767123 29897055 13818 23643052 12463","4632671 4977579 4992430 4958988 14417215 4992429 4621826 804171 4263471 4625501 4399045 4987292 4628675","No"
            </pre>
          </code>
          </div>
          <h2>NOCI glob.py creation</h2>
          <p>Development of a glob for NOCI</p>
          <div class="codebg">
                  <iframe src="https://trinket.io/embed/python/44f62995de" width="100%" height="356" frameborder="0" marginwidth="0" marginheight="0" allowfullscreen></iframe>
          </div>
          
          
          
          <h2>Notes</h2>
          <ul>
              <li> I couldn't download the .tar.gz file from figshare. The alert message is: "Operazione non riuscita. Vietato"</li>
              <li> Discuss the possibility of a mapping pmids-dois, so that the ids already present in OC as DOIs are not processed twice. Alternatively, it could be interesting to integrate the information already present with the ones retrieved from NIH.</li>
              <li> Discuss if it is possible to make an API request using OC existent code in a glob, or if the missing information is just to leave empty</li>
              <li> Discuss the support file journal_name_issn. Its function is that of avoiding an extra API call in case the name of the jurnal has already been associated to some ISSN codes.</li>
              <li><strong>Files to download:</strong> NIH files in iCite Database, available on <a href="https://nih.figshare.com/articles/dataset/iCite_Database_Snapshot_2021-07/15148737?file=29101575">figshare</a></li>
          </ul>

    </div>
  </div>
</div>

      
<div class="card">
    <div class="card-header" id="headingTwenty-two">
      <h5 class="mb-0">
        <button class="btn btn-link collapsed" data-toggle="collapse" data-target="#collapseTwenty-two" aria-expanded="false" aria-controls="collapseTwenty-two">
          Week #22 (09-19-21)
        </button>
      </h5>
    </div>
    <div id="collapseTwenty-two" class="collapse" aria-labelledby="headingTwenty-two" data-parent="#accordion">
      <div class="card-body">
          <h2>noci/glob1.py</h2>
          <ol>
              <li>Two new functions (issn_data_recover(directory), issn_data_to_cache(name_issn_dict, directory)) were developed to manage the storage of the data concerning the mapping between journals shortnames and ISSNs. The first function is aimed at recreating a local dictionary of the mapping from the JSON file stored and periodically updated, so to avoid the repetition of API calls for journal names which have already been encountered before an hypotetical run break. The second one is aimed at storing the new information processed periodically, each n rows of csv, where n is a parameter specified by the user. For this reason, the parameter n was added (arg_parser.add_argument( "-n", "--num_lines", dest="n", required=True, help="Number of lines after which the data stored in the dictionary for the mapping between a Journal name and the related issns are passed into a JSON cache file" )). </li>
              <li>A new iteration for all the rows of all the source file was added, so to check the validity of the referenced pmids of each publication, exploiting the information provided in the field "references" of the csv metadata file by iCite.</li>
              <li>Thank to the chunksize parameter, the memory overload issue which occurred when processing a very large csv file with Pandas was overcome.</li>
              <li>In order to check the correct functioning of the glob, it was run locally using the icite_metadata.csv file as input, and it worked properly.</li>
            
          </ol>
          
          <iframe src="https://trinket.io/embed/python/a5210b640d" width="100%" height="356" frameborder="0" marginwidth="0" marginheight="0" allowfullscreen></iframe>
          
          <h2>Notes</h2>
          <ul>
              <li><strong>if type(row["references"]) is str and row["references"] != "" and row["references"] != " ":</strong> : I'm not sure that it is correct, but without this specification i got an issue with "float" type when the "reference" string was empty</li>
              <li>For the tar.gz file download problem, I'm trying to download the file again after the log-in, as suggested at <a href="https://support.google.com/chrome/answer/2898334?hl=it#zippy=%2Cvietato-oppure-operazione-non-riuscita---vietato"> Google Chrome assistance page</a> about download errors correction.</li>
          </ul>

    </div>
  </div>
</div>


<div class="card">
    <div class="card-header" id="headingTwenty-three">
      <h5 class="mb-0">
        <button class="btn btn-link collapsed" data-toggle="collapse" data-target="#collapseTwenty-three" aria-expanded="false" aria-controls="collapseTwenty-three">
          Week #23 (09-14-21)
        </button>
      </h5>
    </div>
    <div id="collapseTwenty-three" class="collapse" aria-labelledby="headingTwenty-three" data-parent="#accordion">
      <div class="card-body">
      <h2>Multiple cnc tests</h2>
      
      <h3>Test 1</h3>
      <p>Cnc is called on a citations source dataset (which must exist), but without all the support files. These latter shouldn't exist: you have to use names of not existing files. In this way cnc will create the named files.</p>   
      <div class="codebg">
        <code>
            <pre>
C:\Users\arimoretti\Documents\GitHub\index>python cnc.py -ib "https://pubmed.ncbi.nlm.nih.gov/" -b "https://w3id.org/oc/index/noci/" -c "csv" -i "index/test_data/citations_partial_pmid.csv" -doi "index/noci_test/p
mid.csv" -orcid "index/noci_test/orcid.csv" -date "index/noci_test/date.csv" -issn "index/noci_test/issn.csv" -l "index/test_data/lookup_full.csv" -d "index/noci_test" -px "0160" -a "https://w3id.org/oc/index/prov
/ra/1" -s "https://doi.org/10.35092/yhjc.c.4586573.v16" -sv "OpenCitations Index: NOCI" -type "pmid" -v
            </pre>
        </code>
      </div>
      <h4>Comments:</h4>
      <p>I deleted the existent files generated by a previous run of cnc and then run the code using API.</p>
      
      <h3>Test 2</h3>
      <p>The same call is repeated, but adding the "na" parameter, which implies not calling the API, using the just created files. The result should be the same.</p>
            <div class="codebg">
        <code>
            <pre>
C:\Users\arimoretti\Documents\GitHub\index>python cnc.py -ib "https://pubmed.ncbi.nlm.nih.gov/" -b "https://w3id.org/oc/index/noci/" -c "csv" -i "index/test_data/citations_partial_pmid.csv" -doi "index/noci_test/p
mid.csv" -orcid "index/noci_test/orcid.csv" -date "index/noci_test/date.csv" -issn "index/noci_test/issn.csv" -l "index/test_data/lookup_full.csv" -d "index/noci_test" -px "0160" -a "https://w3id.org/oc/index/prov
/ra/1" -s "https://doi.org/10.35092/yhjc.c.4586573.v16" -sv "OpenCitations Index: NOCI" -type "pmid" -v -na
            </pre>
        </code>
      </div>
      <h4>Comments:</h4>
      <p>I repeated the very same test, with the only addition of the last parameter, "-na". Before I run the test, I deleted all the files generated by the previous test but the support ones.</p>
      
      <h3>Test 3</h3>
      <p>The same call is repeated, this time using API and already generated files.</p>
      
      <div class="codebg">
        <code>
            <pre>
C:\Users\arimoretti\Documents\GitHub\index>python cnc.py -ib "https://pubmed.ncbi.nlm.nih.gov/" -b "https://w3id.org/oc/index/noci/" -c "csv" -i "index/test_data/citations_partial_pmid.csv" -doi "index/noci_test/p
mid.csv" -orcid "index/noci_test/orcid.csv" -date "index/noci_test/date.csv" -issn "index/noci_test/issn.csv" -l "index/test_data/lookup_full.csv" -d "index/noci_test" -px "0160" -a "https://w3id.org/oc/index/prov
/ra/1" -s "https://doi.org/10.35092/yhjc.c.4586573.v16" -sv "OpenCitations Index: NOCI" -type "pmid" -v
            </pre>
        </code>
      </div>
      <h4>Comments:</h4>
      <p>I repeated the first call, with the only difference that now the support file exist. Before I run the test, I deleted all the files generated by the previous test but the support ones.</p>
      
      <h3>Test 4</h3>
      <p>The same call is repeated, this time using "na" and empty files: in this case we expect to obtain a different result (with only citing and cited) obtain the only different result among the four attempts.</p>
      <div class="codebg">
        <code>
            <pre>
C:\Users\arimoretti\Documents\GitHub\index>python cnc.py -ib "https://pubmed.ncbi.nlm.nih.gov/" -b "https://w3id.org/oc/index/noci/" -c "csv" -i "index/test_data/citations_partial_pmid.csv" -doi "index/noci_test/p
mid.csv" -orcid "index/noci_test/orcid.csv" -date "index/noci_test/date.csv" -issn "index/noci_test/issn.csv" -l "index/test_data/lookup_full.csv" -d "index/noci_test" -px "0160" -a "https://w3id.org/oc/index/prov
/ra/1" -s "https://doi.org/10.35092/yhjc.c.4586573.v16" -sv "OpenCitations Index: NOCI" -type "pmid" -v -na
            </pre>
        </code>
      </div>
      <h4>Comments:</h4>
      <p>In order to run this test, I deleted all the files generated in the previous calls, included the supported ones, and I added the "-na" parameter. In this case, the only generated file is pmid.csv, containing the following values:</p>
      
      <div class="codebg">
        <code>
            <pre>
"id","value"
"pmid:2140506","i"
"pmid:1523579","i"
"pmid:1509982","i"
"pmid:1968312","i"
"pmid:2330868","i"
"pmid:1854174","i"
            </pre>
        </code>
      </div>
      
      <h3>Conclusions</h3>
      <p>All the files were checked with <a href="https://text-compare.com/"> https://text-compare.com/ </a></p>
      <ul>
        <li>In general, slx and csv results of the first three tests differ only for the date of creation of the Citation processed.</li>
        <li>The three output csv in the folder "data" (oci,citing,cited,creation,timespan,journal_sc,author_sc) are exactly the same</li>
        <li>The three versions of ttl formats differ both in "data" and in "prov" folders.</li>
      </ul>



      <h2>Empty spaces in referenced pmids (in noci glob) </h2>
      <p>To do:</p>
      <ul>
        <li>remove possible spaces at the beginning and at the end of the string</li>
        <li>using a regex, make sure that no repeated spaces are placed as separators between a pmid and another</li>
      </ul>
      <div class="codebg">
        <iframe src="https://trinket.io/embed/python/e492a1e3bc" width="100%" height="356" frameborder="0" marginwidth="0" marginheight="0" allowfullscreen></iframe>
      </div>
      <p>In particular, the piece of code modified is:</p>
      <div class="codebg">
        <code>
            <pre>
    for file_idx, file in enumerate( all_files, 1 ):
        df = pd.DataFrame()
        for chunk in pd.read_csv( file, chunksize=1000 ):
            f = pd.concat( [df, chunk], ignore_index=True )
            print( "Open file %s of %s" % (file_idx, len_all_files) )
            for index, row in f.iterrows():
                if type(row["references"]) is str and row["references"] != "":
                    ref_string = row["references"].strip()
                    ref_string_norm = re.sub("\s\s+", " ", ref_string)
                else:
                    print("the type of row reference is", type(row["references"]))

                cited_pmids = set(ref_string_norm.split(" "))
                for cited_pmid in cited_pmids:
                    if pmid_manager.is_valid(cited_pmid):
                        pmid_manager.set_valid(cited_pmid) #is it necessary?
                        print("valid cited pmid added:", cited_pmid)
                    else:
                        print("invalid cited pmid discarded:", cited_pmid)
            </pre>
        </code>
      </div>

      <h2>Repeated citations issue</h2>
      <p>Different indexes can share repeated citations. The problem is to be fixed with <strong>URLs of meta ids</strong>. Below, the study of the steps through which this process will be handled computationally:</p>
      
      <h3>Meta ids creation</h3>
      <p>Each metaid should have the format: ________________ + an assigned number from 1 on</p>
      
      <h3>Pmid - Meta ids association</h3>
      <p>Then, maybe with a csv, each pmid from cnc output should be associated with a metaid.</p>
      
      <h3>Pmid - Doi association</h3>
      <p>A structure similar to the previous one should be created to associate pmid to doi. In order to do that, the data provided by NIH could be exploited. </p>
      
      <h3>Pmid-doi-metaid association</h3>
      <p>Consider the possibility to associate the pmid, dois and metaid into a unique structure (which could be a dictionary which uses as keys the metaids and as values other dictionaries with keys "pmid" and "doi", having as values lists with the pmid and dois associated.</p>
      
      <h3>Blazegraph, Triplestore, Rdflib</h3>
      
      <p>Use blazegraph to create a triplestore with only one property, which could be dcterms:relations. The data in the resulting ntriple file should have this format: <strong> metaid - dcterms:relation - [format generated by cnc]</strong>. Look at the ttl formats for this.</p>
      <p>The result is to be queried with SPARQL. Study triplestore and rdflib and use as initlial data cnc output.</p>


      <h2>Notes</h2>
      <ul>
          <li>Ask which format a meta id should have</li>
          <li>Will the same rdf have as subjects both the pmids and dois? In this case, will the association be done by querying with SPARQL the triplestore in order to check which subjects have the same object?</li>
          <li>Do all doi need a metaid? Or only the ones with a pmid?</li>
          <li>Does it make sense this procedure? a) associate a metaid to a pmid processed by cnc. b) check if the association pmid - doi is present in the metadata provided by NIH. c) In case the association is already present, search in the csv (or other structure, like a dictionary) the pmid and  assign to the doi the same metaid of the related pmid; In case the association is not present assign to it a new metaid (the last assigned one +1). (Should I try to check the association also in other ways? For example exploiting doi API services?). d) then, when I have a data structure associating each pmid to its own metaid (and in case also to its doi), i turn it into an rdf. </li>
      </ul>

    </div>
  </div>
</div>

<div class="card">
    <div class="card-header" id="headingTwenty-four">
      <h5 class="mb-0">
        <button class="btn btn-link collapsed" data-toggle="collapse" data-target="#collapseTwenty-four" aria-expanded="false" aria-controls="collapseTwenty-four">
          Week #24 (09-30-21)
        </button>
      </h5>
    </div>
    <div id="collapseTwenty-four" class="collapse" aria-labelledby="headingTwenty-four" data-parent="#accordion">
      <div class="card-body">
          <h2>Noci Mapping</h2>
          <p>This file exploits the metadata from NIH to map pmids and doi</p>
          <iframe src="https://trinket.io/embed/python/446245ed54" width="100%" height="356" frameborder="0" marginwidth="0" marginheight="0" allowfullscreen></iframe>
          <p>It generates output csv files looking like this:</p>
          <div class="codebg">
            <code>
                <pre>
"id","value"
"pmid:3","doi:10.1016/0006-291x(75)90498-2"
"pmid:9790254","doi:10.1507/endocrj.45.suppl_s159"
"pmid:23183539","doi:10.1016/j.nbt.2012.11.016"
"pmid:1","doi:10.1016/0006-2944(75)90147-7"
"pmid:19768734","doi:10.1002/mrc.2517"
"pmid:1854174","doi:10.1002/mrc.2517"
                </pre>
            </code>
          </div>
          
          <h2>Metaid Mapping</h2>
          <p>This file takes in input mapping files, a metaid mapping csv (if it doesn't exist already it generates it), and a cnc output file containing citations. For now, it gives in output an updated version of the input csv file. Later it will give in output also the completed rdf file containing triples.</p>
          <p>This part of the software considers each id (either cited or citing) and maps it to a metaid. In the case a new mapping file allows in a later moment to assert that two ids already present in the metaid mapping file point to the same publication, the software reassign to the id with he highest metaid assigned the id of the other correspondant id, and allows the deleted metaid to be reassigned. The csv metaid mapping is updated accordingly.</p>
          <iframe src="https://trinket.io/embed/python/b67a1db282" width="100%" height="356" frameborder="0" marginwidth="0" marginheight="0" allowfullscreen></iframe>
          <p>It generates output csv files looking like this:</p>
          <div class="codebg">
            <code>
                <pre>
id,value
doi:10.1002/mrc.2517,1
pmid:2140506,1
pmid:2942070,2
pmid:1523579,3
pmid:7097569,4
pmid:1509982,1
pmid:6501574,5
pmid:1968312,1
pmid:13673087,6
pmid:2330868,3
pmid:3958380,3
pmid:1854174,1
pmid:3037997,9
                </pre>
            </code>
          </div>
          <p><strong>Note:</strong> More than one pmid is assigned the same metaid because the proper functioning of the updating process was tested with fictitious mapping files associating two pmids. </p>
          
          <h2>CsvManager extension</h2>
          <p>In order to handle the possibility to reassign metaid to ids already present in the input csv, I extended the CsvManager with a function called "substitute_value", which changes the value associated to an id in case the previous value is not the expected one (for example in the case a new mapping file reveas that two ids already in the metaid mapping csv point to the same publication, and then must be associated to the same id)</p>
          <iframe src="https://trinket.io/embed/python/4f32a065de" width="100%" height="356" frameborder="0" marginwidth="0" marginheight="0" allowfullscreen></iframe>

    </div>
  </div>
</div>

<div class="card">
    <div class="card-header" id="headingTwenty-five">
      <h5 class="mb-0">
        <button class="btn btn-link collapsed" data-toggle="collapse" data-target="#collapseTwenty-five" aria-expanded="false" aria-controls="collapseTwenty-five">
          Week #25 (10-07-21)
        </button>
      </h5>
    </div>
    <div id="collapseTwenty-five" class="collapse" aria-labelledby="headingTwenty-five" data-parent="#accordion">
      <div class="card-body">
          <h2>Previous fixes:</h2>
          <ol>
              <li>Pandas nan to empty string in NOCI Glob: <strong>(df.fillna('', inplace=True))</strong></li>
              <li>Successful content comparison of ttl cnc output files. I developed one function to re-order lexicographically the content of each ttl file, deleting also empty lines. Then, the result is stored in a textual file. Another function was then developed in order to check if the content of the resulting textual file is the same, or - in case it is not - to show which triple(s) differ. All the material is temporarily stored in a folder named "comparer" insider the folder "support". </li>
          </ol>

    </div>
  </div>
</div>

<div class="card">
    <div class="card-header" id="headingTwenty-six">
      <h5 class="mb-0">
        <button class="btn btn-link collapsed" data-toggle="collapse" data-target="#collapseTwenty-six" aria-expanded="false" aria-controls="collapseTwenty-six">
          Week #26 (10-14-21)
        </button>
      </h5>
    </div>
    <div id="collapseTwenty-six" class="collapse" aria-labelledby="headingTwenty-six" data-parent="#accordion">
      <div class="card-body">          
          <h2>Most recent version of mapping.py and overall description of its overall functioning</h2>
          <iframe src="https://trinket.io/embed/python/1fab62ca69" width="100%" height="356" frameborder="0" marginwidth="0" marginheight="0" allowfullscreen></iframe>
          <h3>Notes</h3>
          <ul>
          <li>Creation of create_rdf_from_csv(csvfile)</li>
          <li>Neceassary revision step by step of the overall procedure: required fix of the metaid re-assignation mechanism.</li>
          <li>General code corrections and re-organization of files and functions</li>
          </ul>
          
          <h2>Ramose preparatory study:</h2>
          <ul>
            <li><a href="http://www.semantic-web-journal.net/system/files/swj2752.pdf">Article on Ramose</a></li>
            <li><a href="https://github.com/opencitations/ramose/blob/master/test/ramose.py">Ramose Python code</a></li>
            <li><a href="https://github.com/opencitations/api/blob/master/coci_v1.hf">Coci APIconfigurations</a></li>
          </ul>

    </div>
  </div>
</div>

<div class="card">
    <div class="card-header" id="headingTwenty-seven">
      <h5 class="mb-0">
        <button class="btn btn-link collapsed" data-toggle="collapse" data-target="#collapseTwenty-seven" aria-expanded="false" aria-controls="collapseTwenty-seven">
          Week #27 (10-20-21)
        </button>
      </h5>
    </div>
    <div id="collapseTwenty-seven" class="collapse" aria-labelledby="headingTwenty-seven" data-parent="#accordion">
      <div class="card-body">          
          <h2>Most recent version of mapping.py and overall description of its overall functioning</h2>
          <iframe src="https://trinket.io/embed/python/c7b2528dd2" width="100%" height="356" frameborder="0" marginwidth="0" marginheight="0" allowfullscreen></iframe>
          <h3>Notes</h3>
          <ul>
              <li>Fix in create_rdf_from_csv(csvfile_add, csvfile_delete): now the function takes in input both the csv files at the same time, so that the triples which should be added and deleted immediatly afterwards are not added at all.</li>
              <li>Fix of the metaid re-assignation mechanism: A bug in the management of the mapping files for the cited element of the citation led to several and repeated metaid assignation errors. Now the bug is fixed and the procedure works as expected.</li>
              <li>Fix of metaid association mechanism for already encountered ids: in the previous version of mapping.py, if the code was run with the same input data but with new mapping files, also the ids which shouldn't be reassigned a metaid were given a new one. Now the bug is fixed and only the ids whose metaid should be changed according to the information retrieved from new mapping files are actually updated.</li>
          </ul>
          
          <h2>Ramose preparatory study:</h2>
            <ol>
              <li>fix mapping.py code, so to obtain your own triplestore.</li>
              <li>make your blazegraph with your example data and query it.  After that, it will be possible to understand where to modify the code for coci in order to develop NOCI version. But, in the meantime:</li>
              <li>get acquainted with ramose. a simplified version of the test was developed for testing purposes and to let new users to get acquainted with ramose. The file name is “m0”, and it allows Ramose execution in shell, and also the creation of a webserver for making requests on browser. M0 queries wikidata, so that it is possible to try it without a local triplestore. This example is in the new updated and fixed version of ramose. Download the most recent one and run python ramose.py -s test/test_m0.hf -w localhost:8080.</li>
              <li>Use of the existing files. Until now, m2 has been the simplest version. However, it needs some extra functions in external python files. The reason why it is so simple is that it has one implemented operation only. Another option is using COCI APIs and its configuration file (the one previously linked) and run RAMOSE with: : python ramose.py -s ../api/coci_v1.hf -w localhost:8080. Note that “-s” specifies the source of the data used by the API, while “-w” specifies the localhost. Localhost 8080 allows you to use ramose in your browser. So, after having copied and pasted the example after the url, you can call the triplestore and manage the request locally. By pressing enter, without any further specification, the default format of the output will be a JSON. When we want to change something, we intervene on the configuration file. The aim is understanding how does ramose work with an existing triplestore: the endpoint has to be specified at the beginning of the configuration file.</li>
              <li>Once we have NOCI rdf data on a local triplestore, we replace the OC endpoint with that of our local blazegraph. The various operations must be modified because for now it only received DOIs in input. The idea is that we need to change the input shape, since it is a regex made for matching DOIs instead of PMIDs. Since in the pre and post processing phases there are some encoding/decoding operations specifically meant for DOIs, in the case of PMIDs we can probably skip both passages. The test will then be done by specifying another endpoint (specific for NOCI, to be tested in localhost 8080).  In order to do that, it is necessary to change the base on COCI file (http://.. etc) with: #base http://localhost:8080, before running ramose. All the execution links are clickable and they go to the specific software. </li>
              <li>In order to realize the main operations for NOCI (all of them except for metadata), simple SPARQL queries will be enough, without any pre and post processing.  The only operation which will require external functions is metadata. We will save metadata of the links by pubmed. The metadata acquisition will be done on the fly, by making requests to PubMed Central. The procedure will be the same performed when we had to get data such as ISSN and so on, from the txt output given by the APIs. citation, reference, citation count, reference count, citations are the base operations which won’t require external functions. Metadata will be the last one to be realised. </li>
          </ol>
          
          <h2>Issues:</h2>
          <ol>
            <li>How to use blazegraph? Also, I read in a stackoverflow post that "Blazegraph is now deprecated and the develops have joined Amazon and became Amazon Neptune". Should I use Blazegraph anyway?. In case, can I follow this <a href ="https://stackoverflow.com/questions/33368741/docker-container-with-blazegraph-triple-store-not-working-possibly-due-to-networ">stackoverflow post</a> structure?</li>
            <li>I could not correctly install all the requirements for ramose: ERROR: Cannot install -r Documents\GitHub\ramose\requirements.txt (line 12) and urllib3==1.26.5 because these package versions have conflicting dependencies.</li>
            <li>Discuss in detail api/blob/master/coci_v1.hf structure</li>
          </ol>
    </div>
  </div>
</div>

<div class="card">
    <div class="card-header" id="headingTwenty-eight">
      <h5 class="mb-0">
        <button class="btn btn-link collapsed" data-toggle="collapse" data-target="#collapseTwenty-eight" aria-expanded="false" aria-controls="collapseTwenty-eight">
          Week #28 (10-28-21)
        </button>
      </h5>
    </div>
    <div id="collapseTwenty-eight" class="collapse" aria-labelledby="headingTwenty-eight" data-parent="#accordion">
      <div class="card-body">
      <h2>Last metaid file, structure to correct</h2>
      <p>The part preceding the metaid number must be deleted, since another part of the code is structured so to work with files containing the number only.</p>
      <p>Fixed:</p>
      <img src="img/last_metaid_1.png" class="myimg">
      <img src="img/last_metaid_2.png" class="myimg">
      
      <h2>Test the code</h2>
      <p>Create a class which tests all the functionalities of this part of the code. Add a new test file to the directory "tests". For now test all the existing cases. Then, later, it will be possible to simplify the code and eventually leave out the unnecessary parts (the if-else cases which represent non factual possibilities).</p>
      
      <h3>nocimapping.py test: 11_nocimapping.py</h3>
      <iframe src="https://trinket.io/embed/python/885e0e4878" width="100%" height="356" frameborder="0" marginwidth="0" marginheight="0" allowfullscreen></iframe>
      <p>This script tests:</p>
      <ol>
        <li>The functioning of get_all_files.</li>
        <li>That process function correctly creates necessary directories and subdirectories.</li>
        <li>That process function correctly creates a mapping csv file where it associate a pmid to its correspondant doi.</li>
      </ol>
      
      <h3>mapping.py test: 12_nocimapping.py</h3>
      
      <iframe src="https://trinket.io/embed/python/b02ed7762d" width="100%" height="356" frameborder="0" marginwidth="0" marginheight="0" allowfullscreen></iframe>
      
      <h4>Note:</h4>
      <p>All tests are passed and during the execution they work as expected. However, at the end of the run, the output files are not the ones I expected. Discuss the functioning of a unittest (consequentiality among the functions)</p>
      
      <h3>Issues:</h3>
      <ul>
        <li>urllib.error.HTTPError: HTTP Error 503: Service Unavailable, when running python -m index.storer.updatetp -s "http://localhost:3001/blazegraph/sparql" -i (path cartella contenente ttl di dati citazionali) -g "https://w3id.org/oc/index/noci/" with the files containing my triples to upload. I also tried change the files extension but it didn't solve the issue.</li>
        <li>La pagina non funziona192.168.1.112 non ha inviato dati. ERR_EMPTY_RESPONSE, when running java -server -Xmx1g -Dbigdata.propertyFile=index.properties -Djetty.port=3001 -Djetty.host=127.0.0.1 -jar blazegraph.jar. I think I had problems with the blazegraph.jar file.</li>
      </ul>


    </div>
  </div>
</div>
      
<div class="card">
    <div class="card-header" id="headingTwenty-nine">
      <h5 class="mb-0">
        <button class="btn btn-link collapsed" data-toggle="collapse" data-target="#collapseTwenty-nine" aria-expanded="false" aria-controls="collapseTwenty-nine">
          Week #29 (11-11-21)
        </button>
      </h5>
    </div>
    <div id="collapseTwenty-nine" class="collapse" aria-labelledby="headingTwenty-nine" data-parent="#accordion">
      <div class="card-body">
      <p>http://127.0.0.1:3001/blazegraph/#splash</p>
      
      
      <h2>Correct the blazegraph update error: ExecutionException: org.openrdf.query.MalformedQueryException </h2>
      <p>Try: os.path.join(path, *paths) --> <strong>failed</strong> </p>
      <p>Try: (abspath(f_n.__str__())) e (abspath(f_n)).__str__() --> <strong>failed</strong> </p>
      <p>Try: normpath(abspath(f_n)) e abspath(normpath(f_n)) --> <strong>failed</strong> </p>
      <p>Try: split("\\") e split("/") e os.path.join() --> <strong>failed</strong> </p>
      <p>NOTE: It properly works when the triples file is uploaded from blazegraph interface, with the online path of the file, and also when the local file is uploaded manually.</p>
      
      <h2>Make independent each test for mapping.py</h2>
      <iframe src="https://trinket.io/embed/python/475f03f9c3" width="100%" height="356" frameborder="0" marginwidth="0" marginheight="0" allowfullscreen></iframe>
      
      <h2>Issues</h2>
      <ul>
      <li>Why does serviceURL: http://10.201.10.177:3001 does not work, but serviceURL: http://127.0.0.1:3001 does</li>
      <li>Apparently, I can manually update files on my triplestore on blazegraph and I can then query the triplestore. However, I didn't understand where to see the full triplestore. </li>
      </ul>


    </div>
  </div>
</div>

<div class="card">
    <div class="card-header" id="headingThirty">
      <h5 class="mb-0">
        <button class="btn btn-link collapsed" data-toggle="collapse" data-target="#collapseThirty" aria-expanded="false" aria-controls="collapseThirty">
          Week #30 (19-11-21)
        </button>
      </h5>
    </div>
    <div id="collapseThirty" class="collapse" aria-labelledby="headingThirty" data-parent="#accordion">
      <div class="card-body">
      
      <h2>Split of the output folders for the triples to add and the ones to remove</h2>
      <p>Split the output folders for the csv to rdf function (in mapping.py), so that it becomes easier to upload and remove triples, since the update function takes in input a folder and not a file.</p>
      <h3>mapping.py, fixed code</h3>
      <p>A new argument is passed in input, which is the output folder which is going to contain both of the subfolders.</p>
      <iframe src="https://trinket.io/embed/python/cb4548112b" width="100%" height="356" frameborder="0" marginwidth="0" marginheight="0" allowfullscreen></iframe>
      <h4>Issue:</h4>
      <p>Ask whether the subfolders for the storage of the files of the triples to add and to remove should be emptied or not before each run</p>
      
      <h3>12_mapping.py (test), fixed code</h3>
      <p>The code of the two functions affected by the new organization in subfolders of the files containing the triples to upload and remove.</p>
      <iframe src="https://trinket.io/embed/python/2b9245eb18" width="100%" height="356" frameborder="0" marginwidth="0" marginheight="0" allowfullscreen></iframe>
      
      <iframe src="https://trinket.io/embed/python/11aecfa1a6" width="100%" height="356" frameborder="0" marginwidth="0" marginheight="0" allowfullscreen></iframe>
      
    
      <h2>Avoid extra triples addition in the remove file</h2>
      <p>Check that - in the case a triple should be uploaded and then removed during the same updating cycle - the extra triple is neither in the upload file nor in the remotion one.</p>
      <ul>
      <li>The problem was handled with the creation of a list of the triples which are contained in both the addition and in the deletion csv file, and - at the moment of the creation of the graphs which are going to be stored in the ttl files - the triples which were present in both of the csv files aren't added to none of the output ttl files (see create_rdf_from_csv(csvfile_add, csvfile_delete, output_dir) in mapping.py).  </li>
      <li>This functionality is then tested in the function test_create_rdf_from_csv(self), in 12_mapping.py </li>
      </ul>
      
      <h2>Glob test file</h2>
      <p>glob1.py (glob for noci) should be revised and tested, so to check that it works as it should.</p>
      <iframe src="https://trinket.io/embed/python/c7a778f35b" width="100%" height="356" frameborder="0" marginwidth="0" marginheight="0" allowfullscreen></iframe>
      <h3>Notes:</h3>
      <ul>
      <li>All the functions are tested individually and their input/output material is stored separately</li>
      <li>Functionalities related to orcid are still to be tested (with another example of publication)</li>
      <li>Ask if we deliberately chose to avoid validating doi mapped with pmids in pmid glob function.</li>
      </ul>


      <h2>update.py extension to remove triples</h2>
      <p>Suggestion: Update update.py with the function for removing triples. If not, ask if you should do it. We need a query sparql which does the opposite of loading. There is the possibility that the only way to do that is to use "delete data", specifying the triples to remove after having read the file. The query may be too big to be handled in input. So, it may be necessary to split the query in more queries.</p>
      <h3>UNLOAD - for Neptune only (?)</h3>
      <p>Execution: There is the possibility to perform the LOAD opposite action with "UNLOAD INTO GRAPH", but it seems to be for NEPTUNE only. So I parsed the ttl file containing the triples to delete into a graph and I deleted each of them with DELETE DATA. I added a new argument (which is the folder containing the files with the triples to remove) and added a part of code in order to handle properly the newly added "remove" function.</p>
      
      <iframe src="https://trinket.io/embed/python/b55b07f8d5" width="100%" height="356" frameborder="0" marginwidth="0" marginheight="0" allowfullscreen></iframe>
 

    </div>
  </div>
</div>

<div class="card">
    <div class="card-header" id="headingThirty-One">
      <h5 class="mb-0">
        <button class="btn btn-link collapsed" data-toggle="collapse" data-target="#collapseThirty-One" aria-expanded="false" aria-controls="collapseThirty-One">
          Week #31 (3-12-21)
        </button>
      </h5>
    </div>
    <div id="collapseThirty-One" class="collapse" aria-labelledby="headingThirty-One" data-parent="#accordion">
      <div class="card-body">

      
      <h2>Ramose revision</h2>
      <h3>Notes and Questions:</h3>
      <ul>
      <li>call /references/10.1186/1756-8722-6-59?</li>
      <li>encode/decode is to be left out or just escaped?</li>
      <li>discuss difference test and configuration file</li>
      </ul>
      
      <h2>nocimapping.py fix</h2>
      <p>Previous version</p>
      <iframe src="https://trinket.io/embed/python/efccce304a" width="100%" height="356" frameborder="0" marginwidth="0" marginheight="0" allowfullscreen></iframe>
      <p>Current fixed version, exploiting the normalise function for both pmids and dois, setting the parameter include_prefix to True</p>
      <iframe src="https://trinket.io/embed/python/4576d1b133" width="100%" height="356" frameborder="0" marginwidth="0" marginheight="0" allowfullscreen></iframe>
      
      <h2>13_glob.py</h2>
      <p>Last functionality tested: association publication id - author's orcid</p>
      <iframe src="https://trinket.io/embed/python/00b9e85ca5" width="100%" height="356" frameborder="0" marginwidth="0" marginheight="0" allowfullscreen></iframe>
      
      <h2>update.py: query concatenation</h2>
      <p>Since having access to the triplestore database can be time consuming, it is convenient to update a group of triples each time, and avoiding updating triple by triple. A new parameter was added, so to give the possibility to choose after how many triples the trascription to the triplestore is performed.</p>
      
      <iframe src="https://trinket.io/embed/python/ee9065a789" width="100%" height="356" frameborder="0" marginwidth="0" marginheight="0" allowfullscreen></iframe>
      
      <h2>update.py: query concatenation fix</h2>
      <p>Fixed code: Now, in the case the number of triples to delete can be divided by the integer in input, no empty operation is performed (since there is the possibility that an operation trying to delete no triples  causes some issues).</p>
      <iframe src="https://trinket.io/embed/python/a9f1cf329a" width="100%" height="356" frameborder="0" marginwidth="0" marginheight="0" allowfullscreen></iframe>
      
      <h2>Ramose and Configuration File</h2>
      <h3>Preliminary steps:</h3>
      <ol>
      <li>get acquainted with ramose. a simplified version of the test was developed for testing purposes and to let new users to get acquainted with ramose. The file name is “m0”, and it allows Ramose execution in shell, and also the creation of a webserver for making requests on browser. M0 queries wikidata, so that it is possible to try it without a local triplestore. This example is in the new updated and fixed version of ramose. Download the most recent one and run <strong>python ramose.py -s test/test_m0.hf -w localhost:8080</strong></li>
      <li>COCI APIs + its configuration file to run RAMOSE with: <strong>python ramose.py -s ../api/coci_v1.hf -w localhost:8080</strong>. <strong>“-s”</strong>  specifies the source of the data used by the API, <strong>“-w” </strong>specifies the localhost. Triplestore is defined in the configuration file. Localhost 8080 allows you to use ramose in your browser. So, after having copied and pasted the example after the url, you can call the triplestore and manage the request locally. <strong>By pressing enter, without any further specification, the default format of the output will be a JSON. When we want to change something, we intervene on the configuration file. The endpoint has to be specified at the beginning of the configuration file.</strong></li>
      <li>For PMIDs, we need to <strong>change the input shape</strong>, since it is a regex made for matching DOIs. Since in the pre and post processing phases there are some <strong>encoding/decoding</strong> operations specifically meant for DOIs, in the case of PMIDs we can probably <strong>skip both passages</strong>. The test will then be done by specifying another endpoint (specific for NOCI, to be tested in localhost 8080). In order to do that, it is necessary to change the base on COCI file (http://.. etc) with: #base http://localhost:8080, before running ramose. <strong>All the execution links are clickable and they go to the specific software.</strong></li>
      <li>To manage addition and remotion of triples from the triplestore: <strong>python -m index.storer.updatetp -s "http://localhost:3001/blazegraph/sparql" -i "index/test_data/mapping_test_output_1/triples_to_add" -i_r "index/test_data/mapping_test_output_1/triples_to_remove" -g "https://w3id.org/oc/index/noci/" -n 3</strong>. Where <strong>-s</strong> is the source data (triplestore used by the API),  <strong>-i</strong> is the folder containing the file with the triples to add,  <strong>- i_r</strong> the folder containing the file with the triples to delete,  <strong>-n</strong> the number of triples after which the deletion operation is performed.</li>
      <li>To manage the triplestore: <strong>C:\Users\arimoretti\Documents\GitHub\triplestore-index>java -server -Xmx1g -Dbigdata.propertyFile=index.properties -Djetty.port=3001 -Djetty.host=127.0.0.1 -jar blazegraph.jar</strong>, Then click on: <strong><a href="http://127.0.0.1:3001/blazegraph">http://127.0.0.1:3001/blazegraph</a></strong>.</li>
      <li>What is Ramose? OC software for all the APIs. It is a kind of <strong>tool which works as a proxy between SPARQL and a normal request (http).</strong> It is a Rest API. There is a file for the textual interpretation about how this aspect is handled: We need a configuration file for NOCI (noci_v1.hf) which allows the APIs creation. In order to do that, it is necessary to have a triplestore somewhere, with the data within: the local blazegraph: a small amount of data is enough. Ramose will allow to test that everything works correctly, when properly configured.</li>
      </ol>

      
      <h2>update.py: query concatenation</h2>
      <p>Since having access to the triplestore database can be time consuming, it is convenient to update a group of triples each time, and avoiding updating triple by triple. A new parameter was added, so to give the possibility to choose after how many triples the trascription to the triplestore is performed.</p>
      
      <iframe src="https://trinket.io/embed/python/ee9065a789" width="100%" height="356" frameborder="0" marginwidth="0" marginheight="0" allowfullscreen></iframe>


    </div>
  </div>
</div>

<div class="card">
    <div class="card-header" id="headingThirty-Two">
      <h5 class="mb-0">
        <button class="btn btn-link collapsed" data-toggle="collapse" data-target="#collapseThirty-Two" aria-expanded="false" aria-controls="collapseThirty-Two">
          Week #32 (10-12-21)
        </button>
      </h5>
    </div>
    <div id="collapseThirty-Two" class="collapse" aria-labelledby="headingThirty-Two" data-parent="#accordion">
      <div class="card-body">
      
      <h2>update.py: query concatenation fix</h2>
      <p>Fixed code: Now, in the case the number of triples to delete can be divided by the integer in input, no empty operation is performed (since there is the possibility that an operation trying to delete no triples  causes some issues).</p>
      <iframe src="https://trinket.io/embed/python/a9f1cf329a" width="100%" height="356" frameborder="0" marginwidth="0" marginheight="0" allowfullscreen></iframe>
      
      <h2>Ramose and Configuration File</h2>
      <h3>Preliminary steps:</h3>
      <ol>
      <li>get acquainted with ramose. a simplified version of the test was developed for testing purposes and to let new users to get acquainted with ramose. The file name is “m0”, and it allows Ramose execution in shell, and also the creation of a webserver for making requests on browser. M0 queries wikidata, so that it is possible to try it without a local triplestore. This example is in the new updated and fixed version of ramose. Download the most recent one and run <strong>python ramose.py -s test/test_m0.hf -w localhost:8080</strong></li>
      <li>COCI APIs + its configuration file to run RAMOSE with: <strong>python ramose.py -s ../api/coci_v1.hf -w localhost:8080</strong>. <strong>“-s”</strong>  specifies the source of the data used by the API, <strong>“-w” </strong>specifies the localhost. Triplestore is defined in the configuration file. Localhost 8080 allows you to use ramose in your browser. So, after having copied and pasted the example after the url, you can call the triplestore and manage the request locally. <strong>By pressing enter, without any further specification, the default format of the output will be a JSON. When we want to change something, we intervene on the configuration file. The endpoint has to be specified at the beginning of the configuration file.</strong></li>
      <li>For PMIDs, we need to <strong>change the input shape</strong>, since it is a regex made for matching DOIs. Since in the pre and post processing phases there are some <strong>encoding/decoding</strong> operations specifically meant for DOIs, in the case of PMIDs we can probably <strong>skip both passages</strong>. The test will then be done by specifying another endpoint (specific for NOCI, to be tested in localhost 8080). In order to do that, it is necessary to change the base on COCI file (http://.. etc) with: #base http://localhost:8080, before running ramose. <strong>All the execution links are clickable and they go to the specific software.</strong></li>
      <li>To manage addition and remotion of triples from the triplestore: <strong>python -m index.storer.updatetp -s "http://localhost:3001/blazegraph/sparql" -i "index/test_data/mapping_test_output_1/triples_to_add" -i_r "index/test_data/mapping_test_output_1/triples_to_remove" -g "https://w3id.org/oc/index/noci/" -n 3</strong>. Where <strong>-s</strong> is the source data (triplestore used by the API),  <strong>-i</strong> is the folder containing the file with the triples to add,  <strong>- i_r</strong> the folder containing the file with the triples to delete,  <strong>-n</strong> the number of triples after which the deletion operation is performed.</li>
      <li>To manage the triplestore: <strong>C:\Users\arimoretti\Documents\GitHub\triplestore-index>java -server -Xmx1g -Dbigdata.propertyFile=index.properties -Djetty.port=3001 -Djetty.host=127.0.0.1 -jar blazegraph.jar</strong>, Then click on: <strong><a href="http://127.0.0.1:3001/blazegraph">http://127.0.0.1:3001/blazegraph</a></strong>.</li>
      <li>What is Ramose? OC software for all the APIs. It is a kind of <strong>tool which works as a proxy between SPARQL and a normal request (http).</strong> It is a Rest API. There is a file for the textual interpretation about how this aspect is handled: We need a configuration file for NOCI (noci_v1.hf) which allows the APIs creation. In order to do that, it is necessary to have a triplestore somewhere, with the data within: the local blazegraph: a small amount of data is enough. Ramose will allow to test that everything works correctly, when properly configured.</li>
      </ol>

      
      <h2>update.py: query concatenation</h2>
      <p>Since having access to the triplestore database can be time consuming, it is convenient to update a group of triples each time, and avoiding updating triple by triple. A new parameter was added, so to give the possibility to choose after how many triples the trascription to the triplestore is performed.</p>
      
      <iframe src="https://trinket.io/embed/python/ee9065a789" width="100%" height="356" frameborder="0" marginwidth="0" marginheight="0" allowfullscreen></iframe>


    </div>
  </div>
</div>

  </body>
</html>