storing log files and metadata

[bl-young]

In the new branch, the validation log files are not being generated. We have found this to be a valuable way to track and QA FBS generation.

Similarly the metadata needs to be generated and stored.

[matthewlchambers]

I moved some things that were being sent to vLog to the regular log, so that's part of it; I also probably left out some vLog messages that I didn't realize the value of. There's a validation logger already set up in flowsa_log, called vlog (note lowercase "l") which can be (imported and) used for validation logging.

[bl-young]

Yes and I also think this could be a good opportunity to revisit what is most useful in these log files.

[catherinebirney]

I've been thinking about merging the two log files produced into a single validation log file because I rarely look at the non-validation log file. So I concur that we should reevaluate the information saved to log files

[bl-young]

The things I scan through the most are for the comparisons of final FBS to the original FBA to make sure they are the same. I wonder if the new additional columns in the FBS that track the activity set and allocation source that have been done in this branch and on DataVis reduce the need for some of the other validation reporting.

[catherinebirney]

Ah - yes, including those columns could simplify the validation log. I'd like to also keep the printouts of allocation ratios when they are out of the tolerance range because problems with the ratios help pinpoint errors

[matthewlchambers]

If we want to add a column for the attribution source name in this branch, that would be pretty straightforward, I think.

[matthewlchambers]

Slightly tangential, but after thinking about adding to flowsa_log a method for printing a "warning and error report" to the console after generating an FBS (which I ended up doing but then reverting), I realized that if you want the console to only show warning and error messages (for example, if you're going to do what I do and start generating one or more SEEA FBSs, then work on something else for half an hour), you can just include the following code in your script (or run these lines in your console):

import logging
from flowsa import flowsa_log

flowsa_log.console_handler.setLevel(logging.WARNING)

Then the info level logging messages will go to the log file but not to the console, for the duration of your python session. (You can switch it back by running flowsa_log.console_handler.setLevel(logging.INFO).)

[matthewlchambers]

It's also worth pointing out that due to an oversight on my part, the FBS log files aren't being saved under the name of the FBS method at all, currently.