Merge branch 'develop' of github.com:rfcx/arbimon2 into develop

CHANGELOG.md

@@ -1,24 +1,28 @@
 # Arbimon Release Notes
 
-## v3.0.32 - June XX, 2021
+## v3.0.33 - June 01, 2021
 
-Resolved issues:
+Performance improvements:
 
-- CE-1046 The synchronization to work in transactions between Arbimon and RFCx is updated
+- CE-988 The datime_local column is removed
+
+## v3.0.32 - June 26, 2021
+
+New feature:
+
+- CE-1110 User can scale spectrogram in Visualizer to standard frequencies
 
 Performance improvements:
 
 - CE-1065 Patten Matching list loads quicker
 - CE-1068 CNN list page loads quicker
 - CE-1069 CNN job details loads quicker
 - CE-323 When user creates/edits/deletes a project role for another user in Arbimon it's synced with Core API in transactions
+- CE-1046 The synchronization to work in transactions between Arbimon and RFCx is updated
 
 Resolved issues:
 
 - CE-1160 The opacity of PM' results is decreased on the Visualizer page
-
-Resolved issues:
-
 - CE-1004 Fixed issue with pixelated ROI boxes in Safari browser
 
 ## v3.0.31 - July 14, 2021
@@ -53,16 +57,15 @@ Performance improvements:
 - CE-1062 Load only one site per page for "Best per Site, Day" filter
 - CE-1064 Update Pattern Matching query for Citizen Scientist PM list page
 
+## v3.0.30 - June XX, 2021
+
 Other:
 
 - The columns order is changed on the recording page
 - Show "Unknown" if the recording hasn't in the file name data
 - CE-940 Update the text for the Recordings data in the Summary page
 - CE-792 Save validated rois in the recording validations table
 
-
-## v3.0.30 - June 18, 2021
-
 Resolved issues:
 
 - CE-689 Map should default back to showing all pins for all sites
@@ -74,10 +77,6 @@ Resolved issues:
 - CE-751 Display selected recording if the site has many recordings with the similar datetime
 - Hotfix displaying the original filename in the export recordings report
 
-Other:
-
-- CE-880 Beta tag on CNN, AE & clustering
-
 ## v3.0.29-hotfix.0 - June 16, 2021
 
 Performance improvements:

DEPLOYMENT_NOTES.md

@@ -1,5 +1,10 @@
 # Arbimon Deployment Notes
 
+## v.3.0.32
+
+- Update the "upload" lambda to not save datetime_local
+- Run sql migration 018-remove-datetime-local-column-from-recordings-table.sql
+
 ## v3.0.25
 
 - Run migration 014-projects-table-add-image-column.sql

TEST_NOTES.md

@@ -33,5 +33,11 @@ Test Notes are used to list what pages / components / features / user flows are
   - Patten Matching details page: When site from sites list (button with map pin icon) is selected, it loads results for selected site
 - CE-1064 Update Pattern Matching query for Citizen Scientist PM list page
   - Citizen Scientist section: Patten Matching page: the list with pattern matchings should load relatively quick
+- CE-1110 User can scale spectrogram in Visualizer to standard frequencies
+  - Visualizer page: ability of choosing the original Y-scale or 24 Khz scale
+  - When the user is switching between frequencies detection boxes are still shown correctly
+  - When the user reloads the page it keeps the selected Y-scale so the user does not have to change it on every recording
+  - When the recording frequency is 24 kHz or more it shows original Y-scale and the one button with the active state
+  - When the user selects 24 Khz scale and the recording frequencies are less than 24 kHz spectrogram is resized and the background has a gray-darker color from the Arbimon color scheme
 - CE-323 When user creates/edits/deletes a project role for another user in Arbimon it's synced with Core API in transactions
   - Settings section: Users page: When the user creates/edits/deleted a project role for another user and the issue (authentication, validation issues, etc) is happened it is not change the db data the action in the Arbimon/Core API side

app/model/recordings.js

@@ -226,7 +226,6 @@ var Recordings = {
             "R.uri, \n"+
             "R.datetime, \n"+
             "R.datetime_utc, \n"+
-            "R.datetime_local, \n"+
             "R.mic, \n"+
             "R.recorder, \n"+
             "R.version, \n"+
@@ -300,7 +299,6 @@ var Recordings = {
                         "R.uri, \n"+
                         "R.datetime, \n"+
                         "R.datetime_utc, \n"+
-                        "R.datetime_local, \n"+
                         "R.mic, \n"+
                         "R.recorder, \n"+
                         "R.version, \n"+
@@ -917,7 +915,6 @@ var Recordings = {
             bit_rate:        joi.string(),
             sample_encoding: joi.string(),
             upload_time:     joi.date(),
-            datetime_local:  joi.date(),
             datetime_utc:    joi.date(),
             meta:  joi.optional(),
         };
@@ -927,10 +924,10 @@ var Recordings = {
 
             queryHandler('INSERT INTO recordings (\n' +
                 '`site_id`, `uri`, `datetime`, `mic`, `recorder`, `version`, `sample_rate`, \n'+
-                '`precision`, `duration`, `samples`, `file_size`, `bit_rate`, `sample_encoding`, `upload_time`, `datetime_local`, `datetime_utc`, `meta`\n' +
-            ') VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?);', [
+                '`precision`, `duration`, `samples`, `file_size`, `bit_rate`, `sample_encoding`, `upload_time`, `datetime_utc`, `meta`\n' +
+            ') VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?);', [
                 rec.site_id, rec.uri, rec.datetime, rec.mic || '(not specified)', rec.recorder || '(not specified)', rec.version || '(not specified)', rec.sample_rate,
-                rec.precision, rec.duration, rec.samples, rec.file_size, rec.bit_rate, rec.sample_encoding, rec.upload_time, rec.datetime_local, rec.datetime_utc, rec.meta
+                rec.precision, rec.duration, rec.samples, rec.file_size, rec.bit_rate, rec.sample_encoding, rec.upload_time, rec.datetime_utc, rec.meta
             ], callback);
         });
     },
@@ -1526,7 +1523,7 @@ var Recordings = {
                         if (classes && classes.length) {
                             // Divide the results of the validations if the count more than 50 tables.
                             let classesArray = classes.reduce((resultArray, item, index) => {
-                                const chunkIndex = Math.floor(index/50);
+                                const chunkIndex = Math.floor(index/20);
                                 if (!resultArray[chunkIndex]) {
                                     resultArray[chunkIndex] = [];
                                 }
@@ -1643,12 +1640,13 @@ var Recordings = {
                         typeCast: sqlutil.parseUtcDatetime,
                     })
                     if (projection.grouped) {
-                        results = [...results, ...queryResult];
-                        return results;
+                        results = results.concat([...new Set(queryResult)]);
+                    }
+                    else {
+                        results = [...new Set(queryResult)];
                     }
-                    results = [...new Set(queryResult)];
                 }
-                return Q.all(results);
+                return projection.grouped? results : Q.all(results);
             })
     },
 

app/model/uploads.js

@@ -24,7 +24,6 @@ module.exports = {
                 sver: Joi.string().required()
             }),
             datetime: Joi.date().required(),
-            datetime_local: Joi.date(),
             channels: Joi.number().required()
         };
         Joi.validate(uploadData, schema, function(err, upload) {

app/routes/data-api/ingest.js

@@ -103,8 +103,6 @@ router.post('/recordings/create', verifyToken(), hasRole(['systemUser']), async
       }
       const datetimeLocal = await model.recordings.calculateLocalTimeAsync(recordingData.site_id, recordingData.datetime_utc);
       recordingData.datetime = datetimeLocal ? datetimeLocal : moment.utc(recordingData.datetime_utc).format('YYYY-MM-DD HH:mm:ss');
-      // TODO: check that we can remove it and remove
-      recordingData.datetime_local = datetimeLocal ? datetimeLocal : moment.utc(recordingData.datetime_utc).format('YYYY-MM-DD HH:mm:ss');
       await model.recordings.insertAsync(recordingData);
     }
     res.sendStatus(201);

app/routes/data-api/project/recordings.js

@@ -64,7 +64,7 @@ router.get('/species-count', function(req, res, next) {
                 species.push(s.species)
             }
         })
-        
+
         res.json({count: species.length});
     });
 });
@@ -193,22 +193,25 @@ processFiltersData = async function(req, res, next) {
     if (projectionFilter && projectionFilter.grouped && projectionFilter.validation && !projectionFilter.species) {
         return model.recordings.exportRecordingData(projectionFilter, filters).then(async function(results) {
             let gKey = projectionFilter.grouped;
-            let fields = [gKey];
-            fields.push(...Object.keys(results[0]).filter(f => f !== gKey));
+            let fields = [];
+            results.forEach(result => {
+                fields.push(...Object.keys(result).filter(f => f !== gKey && !fields.includes(f)))
+            });
             let data = {};
-            results.forEach((r) => {
+            results.forEach((r, i) => {
                 const s = r[gKey]
+                // Create empty row for each grouped key.
                 if (!data[s]) {
                     data[s] = {};
                     data[s][gKey] = s;
                 };
-                fields
-                    .filter(f => f !== gKey)
-                    .forEach((f) => {
-                        if (data[s][f] === undefined) { data[s][f] = 0 };
-                        data[s][f] += r[f] === '---' ? 0 : +r[f];
-                    })
+                fields.forEach((f) => {
+                    // Fill for each cell 0 as default value.
+                    if (data[s][f] === undefined) { data[s][f] = 0 };
+                    data[s][f] += r[f] === '---' || r[f] === undefined ? 0 : +r[f];
+                })
             });
+            fields.unshift(gKey);
             let datastream = new stream.Readable({objectMode: true});
                 let streamArray = Object.values(data);
                 if (gKey === 'hour') {

app/utils/upload-queue.js

@@ -92,7 +92,6 @@ module.exports = {
                 model.recordings.calculateLocalTime(upload_row.site_id, upload_row.datetime, callback);
             },
             function(result, callback) {
-                upload_row.datetime_local = result? result : upload_row.datetime;
                 if (upload.info && upload.info.isUTC) {
                     upload_row.datetime = result? result : upload_row.datetime;
                 }

assets/app/app/visualizer/spectrogram-layer/base-image-layer.html

@@ -1,13 +1,13 @@
 <a2-img
-    ng-repeat="tile in visobject.tiles.set track by $index" 
-    a2-src="{{tile.src | a2UpdatedUrl}}" 
+    ng-repeat="tile in visobject.tiles.set track by $index"
+    a2-src="{{tile.src | a2UpdatedUrl}}"
     style="position:absolute;z-index:-1"
     ng-class="tile.crisp ? 'crisp-image' : ''"
     ng-style="{
-            left: Math.floor(tile.s * layout.scale.sec2px) + 'px', 
-            bottom: Math.floor(layout.spectrogram.height - tile.hz * layout.scale.hz2px) + 'px', 
-            height: Math.ceil(tile.dhz * layout.scale.hz2px + 1) + 'px', 
-            width: Math.ceil(tile.ds * layout.scale.sec2px + 1) + 'px' 
+            left: Math.floor(tile.s * layout.scale.sec2px) + 'px',
+            bottom: (layout.scale.originalScale ? Math.floor(layout.spectrogram.height - tile.hz * layout.scale.hz2px) : 0) + 'px',
+            height: Math.ceil(tile.dhz * layout.scale.hz2px + 1) + 'px',
+            width: Math.ceil(tile.ds * layout.scale.sec2px + 1) + 'px'
         }"
 ></a2-img>
 <div a2-visualizer-spectrogram-affixed>

assets/app/app/visualizer/spectrogram.js

@@ -293,6 +293,10 @@ angular.module('visualizer-spectrogram', [
                 $scope.layout.apply($element, $scope, $element.width(), $element.height(), true);
             });
 
+            $scope.$watch('layout.scale.originalScale', function (newValue, oldValue) {
+                $scope.layout.apply($element, $scope, $element.width(), $element.height(), true);
+            });
+
             $scope.layout.apply($element, $scope, $element.width(), $element.height());
             $scope.onScrolling();
         }

assets/app/app/visualizer/visobjects/recording.js

@@ -10,16 +10,24 @@ angular.module('a2.visobjects.recording', [
         }]
     });
 })
-.service('VisualizerObjectRecordingTypeLoader', function ($q, Project) {
+.service('VisualizerObjectRecordingTypeLoader', function ($q, Project, $localStorage) {
     var khz_format = function(v){return (v/1000) | 0; };
 
+    var getSelectedFrequencyCache = function() {
+        try {
+            return JSON.parse($localStorage.getItem('visuilizer.frequencies.cache')) || {originalScale: true};
+        } catch(e){
+            return {originalScale: true};
+        }
+    };
+    var scaleCache = getSelectedFrequencyCache();
+
     var recording = function(data, extra){
         for(var i in data){ this[i] = data[i]; }
         this.sampling_rate = this.sample_rate;
         this.extra  = extra;
-        // fix up some stuff
         this.max_freq = this.sampling_rate / 2;
-        // setup the domains
+        this.span = scaleCache && scaleCache.originalScale ? this.max_freq : (this.max_freq > 24000 ? this.max_freq : 24000);
         this.domain = {
             x : {
                 from : 0,
@@ -30,8 +38,8 @@ angular.module('a2.visobjects.recording', [
             },
             y : {
                 from : 0,
-                to   : this.max_freq,
-                span : this.max_freq,
+                to   : this.span,
+                span : this.span,
                 unit : 'Frequency ( kHz )',
                 tick_format : khz_format
             }

assets/app/app/visualizer/visualizer-spectrogram.html

@@ -6,30 +6,30 @@
             Please select a {{visobject_type}}.
         </div>
         <div class="loader" ng-if="loading_visobject">
-            <loader class="blocky big" 
+            <loader class="blocky big"
             text="Loading {{loading_visobject}}"
-            ></loader>        
+            ></loader>
         </div>
     </div>
-    
-    <div class="zoom-control-group ghost-like" 
-        a2-visualizer-spectrogram-affixed="{top:3, right:3, container:'viewport'}" 
+
+    <div class="zoom-control-group ghost-like"
+        a2-visualizer-spectrogram-affixed="{top:3, right:3, container:'viewport'}"
         ng-if="visobject && visobject.zoomable"
     >
         <a2-zoom-control level="layout.scale.zoom.x" horizontal="true"></a2-zoom-control>
         <a2-zoom-control level="layout.scale.zoom.y"></a2-zoom-control>
         <div class="btns">
-            <button class="btn btn-default btn-xs" 
+            <button class="btn btn-default btn-xs"
                 ng-class="click2zoom.active ? 'active' : ''"
                 a2-global-keyup="($event.which == 90) && visobject.zoomable && click2zoom.toggle_active()"
                 ng-click="click2zoom.toggle_active('in')">
                     <i class="fa fa-search-plus"></i>
-            </button>                
-            <button class="btn btn-default btn-xs top-right-2" 
+            </button>
+            <button class="btn btn-default btn-xs top-right-2"
                 ng-click="click2zoom.zoom_out()">
                     <i class="fa fa-search-minus"></i>
             </button>
-            <button class="btn btn-default btn-xs top-right-2" 
+            <button class="btn btn-default btn-xs top-right-2"
                 a2-global-keyup="($event.which == 82) && visobject.zoomable && click2zoom.reset()"
                 ng-click="click2zoom.zoom_reset()">
                 <span class="fa-stack fa-xs">
@@ -39,22 +39,27 @@
             </button>
         </div>
     </div>
-    
-    <div class="spectrogram-container" ng-show="visobject" >
-        <a2-visualizer-spectrogram-layer 
-            ng-repeat="layer in layers | filter: canDisplayInSpectrogram" 
-            ng-show="isSpectrogramVisible(layer)" 
+
+    <div class="spectrogram-container" ng-show="visobject" ng-class="{'container-color': convertedScale < 24}">
+        <a2-visualizer-spectrogram-layer
+            ng-repeat="layer in layers | filter: canDisplayInSpectrogram"
+            ng-show="isSpectrogramVisible(layer)"
         ></a2-visualizer-spectrogram-layer>
-        
+
         <div ng-show="visobject.type=='recording'"
-            class="spectrogram-layer playback-head" 
+            class="spectrogram-layer playback-head"
             ng-style="{left:((audio_player.getCurrentTime() * layout.scale.sec2px) | round)+'px'}"
         ></div>
-        
+
     </div>
 
     <div class="mask-panel" a2-visualizer-spectrogram-affixed="{container:'root', bottom:0, }" ng-style="{height : layout.root.height - (layout.viewport.top + layout.viewport.height), left : layout.spectrogram.left , width  : layout.root.width }"></div>
-
+    <div class="btn-group btn-group-xs pt-1 pl-1" role="group" aria-label="Option to resize the y-axis" ng-show="visobject.domain.y" ng-style="{top: layout.spectrogram.top, left : layout.spectrogram.left }" style="position: absolute; z-index: 100;">
+        <button class="btn btn-sm btn-default" ng-repeat="item in yAxisOptions" ng-show="convertedScale >= 24 ? item.active : item"
+            ng-class="{'active': item.active, 'default-borders': convertedScale >= 24}" ng-click="resizeYScale(item)">
+            {{ item.title }}
+        </button>
+    </div>
     <svg class="axis-y" ng-show="visobject.domain.y" ></svg>
     <svg class="axis-x" ng-show="visobject.domain.x" ></svg>
     <svg class="legend" ng-show="visobject && has_legend" ></svg>