TODO.txt

Road to Production Ready
========================


Needed Features
----------------

  * Dmedia must clearly convey to user when they *should* panic, which currently
    include four scenarios:

    1. Insufficient storage - the user's Dmedia library doesn't have enough
       (recently) attached storage in order to maintain 3 copies of all user
       created files (aka irreplaceable data); there will be a persistent
       indicator shown till the user corrects the situation by provisioning
       more storage

    2. Disconnected drive - when the drive assumed to contain the only known
       copy of one or more files is disconnected, Dmedia can't fix the situation
       without help from the user; as above, there will be a persistent
       indicator shown till the user has taken the appropriate action; there is
       some trickiness in conveying this as the "disconnected" drive might be
       an internal drive in, say, a laptop that is currently off, in which case
       the correct action is to boot the laptop, or it might be, say, an
       external USB drive, in which case the correct action is to plug the USB
       drive into on of the currently powered-on Dmedia peers

    3. Drive failure - when Dmedia suspects a drive is failing (currently this
       means corrupt files were found, but eventually we might consider SMART
       also), we must *always* convey this to the user, even if Dmedia has
       sufficient storage and durability minus the failed drive; this is
       important because we need to warn the user against using this drive for
       other data storage outside of Dmedia; UX wise this is a touch dangerous
       because this will be one of those reflexive click OK to make it go away
       interactions (indicator state with menu item that opens a window with
       details), but note that the danger is only in terms of the user
       mistakenly (or foolishly) continuing to use the drive for other data
       storage... Dmedia itself will stop trusting the drive automatically

    4. Low durability - when any user created file have fewer than 2 known good
       copies, we'll convey this to the user with a persistent indicator; there
       is no user action needed because of this alone, this is more a way to
       tell the user that Dmedia is working, to hint to the user that they might
       want to wait longer before, say, shutting down their laptop or
       disconnected a drive

  * Dmedia must do runtime schema and consistency checks - basically, this
    means reworking MetaStore.check_schema() to do something useful and modern;
    it must periodically check the schema of all docs, and for all dmedia/file
    docs, it must also check the cryptographic relationship between the file ID,
    file size, and leaf hashes; failures must be logged via standard Python
    logging, and saved in 'log-1'

  * MetaStore must save data-safety related events in 'log-1', which will make
    it easy to do runtime checks, postmortem analysis, and give outside test
    harnesses a library-wide view of these critical events; three types of
    events come to mind:

    1. When a store is downgraded (store atime is too old)

    2. When a store is purged (store atime is too old or store doc is missing)

    3. When a corrupt file is found

  * MetaStore must automatically downgrade a store when a corrupt file is found;
    we also need a flag in the dmedia/store doc so that Dmedia wont create new
    copies there, even though existing copies will still be tracked

  * Dmedia must immediately purge the corresponding store when a drive is
    connected but is missing the FileStore we expect to find (based on either
    the drive serial or partition UUID); although Dmedia will otherwise do this
    based on the drive atime (downgrade after 3 days, purge after 8 days), this
    is an easy scenario where we can adjust Dmedia's metadata immediately rather
    than waiting; this is also a quite plausible scenario, for example, the user
    is using a removable drive between two computers, one running an OS that
    can't read ext4, which then prompts the user to format it

  * Note to self: `dmedia.httpd` seems to have a memory leak, possibly due to a
    leak in `ssl` module or `socket.socket` instances not being freed


Integration Test Harness
------------------------