Merge pull request #466 from telefonicaid/release/2.3.0

Release/2.3.0

CHANGES_NEXT_RELEASE

@@ -1,18 +1,5 @@
-- Fix precedence in merging device and configuration attributes.
-- Merge configuration information in the retrieveDevice() function (#405)
-- Expose the singleConfigurationMode attribute using environment variables (#404)
-- Devices provisioned with no configuration are not found in HTTP measure reporting (#124)
-- In MongoDB32 no group is found if undefined resource is passed (#410)
-- Extract common middleware to a new module (#415)
-- Add an administrative operation to get the log level (#417)
-- Add the 'comp' field to the IoTA's log (#419)
-- Fix Static attributes not shown in device group updates (#423)
-- Fix IoTAgent returns non-NGSI error as a response to an error in the Context Providers (#422)
-- Fix Documentation typos and aspects to fix (#426)
-- Add the names of the DB fields to the documentation (#427)
-- FIX IOTAM Registrations won't send static attributes in group information #430
-- Make the IoTA path in the IoTAM registration request configurable
-- Document TimeInstant entity attribute and metadata (#435)
-- FIX Update device operation ignores static attributes (#438)
-- FIX Logger modules not being singleton cause logging inconsistencies (#442)
-- Remove Device ID Index (#445)
+- Add polling command support for the Library #455
+- FIX Entity name used instead of id in command identification
+- Add expiration time for Polling commands (#460)
+- FIX It is not possible to create two devices with the same ID (#462)
+- ADD At service start-up retry if dependencies (mongo, orion, etc.) are not available #357

README.md

@@ -142,6 +142,29 @@ mapped to the command's utility attributes `_info` and `_result` leaving alone t
 values for this attributes are stored locally in the Context Broker (instead of being redirected with the Context
 Provider operations).
 
+There are two types of commands:
+* **Push commands**: when a command of this type arrives to the IoTAgent, the IoTAgent will immediately forward the command
+request to the device, translating the request to the proper protocol (that will depend on the type of IoTAgent). The
+library implement this kind of commands by offering a set functions that can be used to set an IoTAgent-specific handler
+for incoming commands. In order for this type of commands to work properly, the devices must be preprovisioned with an
+endpoint of the proper protocol, where it can be accessed by the IoTAgent who pushes de commits.
+
+* **Poll commands**: polling commands are meant to be used on those cases where the device can't be online the whole time
+waiting for commands. In this case, the IoTAgents must store the received commands, offering a way for the device to
+retrieve the pending commands upon connection. To enable this feature, the Library offers a set of functions to manage
+command storage, and a mechanism to automatically store incoming commands for those devices marked as 'polling devices'.
+
+The distinction between push and poll commands will be made based on the presence of a `polling` flag in the device
+provisioning data. The default option (with the flag with value `false` or not present) is to use push commands (as they
+were the only ones available until the latest versions).
+
+Polling commands could be subjected to expiration: two configuration properties pollingExpiration` and `pollingDaemonFrequency`
+can be set to start a daemon that will remove expired commands from the DB if the device is taking too much to pick them
+up. See the configuration section for details.
+
+The library does not deal with protocol transformation or South Bound communications for neither of the command types
+(that's the task for those specific IoTAgents using the library).
+
 #### Active attributes
 Whenever a device proactively sends a message to the IoT Agent, it should tranform its data to the appropriate NGSI
 format, and send it to the Context Broker as an `updateContext` request.
@@ -183,7 +206,7 @@ to a command update to the device.
 
 ![General ](https://raw.github.com/dmoranj/iotagent-node-lib/develop/img/iotAgentLib.png "Architecture Overview")
 
-### The ´TimeInstant´ element
+### <a name="TimeInstant"/>The ´TimeInstant´ element
 
 As part of the device to entity mapping process the IoT Agent creates and updates automatically a special timestamp.
 This timestamp is represented as two different properties of the mapped entity::
@@ -1014,12 +1037,17 @@ These are the parameters that can be configured in the global section:
 }
 ```
 * **mongodb**: configures the MongoDB driver for those repositories with 'mongodb' type. If the `host` parameter is a list of comma-separated IPs, they will
-be considered to be part of a Replica Set. In that case, the optional property `replicaSet` should contain the Replica Set name. E.g.:
+be considered to be part of a Replica Set. In that case, the optional property `replicaSet` should contain the Replica Set name. The MongoBD driver will
+retry the connection at startup time `retries` times, waiting `retryTime` seconds between attempts, if those attributes are present (default values
+are 5 and 5 respectively). E.g.:
 ```
 {
   host: 'localhost',
   port: '27017',
-  db: 'iotagent'
+  db: 'iotagent',
+  retries: 5,
+  retryTime: 5
+
 }
 ```
 * **iotManager**: configures all the information needed to register the IoT Agent in the IoTManager. If this section is
@@ -1051,6 +1079,10 @@ have implications in the use of attributes with Context Providers, so this flag
 * **defaultResource**: default string to use as resource for the registration of new Configurations (if no resource is provided).
 * **defaultKey**: default string to use as API Key for devices that do not belong to a particular Configuration.
 * **componentName**: default string identifying the component name for this IoT Agent in the logs.
+* **pollingExpiration**: expiration time for commands waiting in the polling queue in miliseconds. If a command has been in the queue for this amount of time without
+being collected by the device, the expiration daemon will reclaim it. This attribute is optional (if it doesn't exist, commands won't expire).
+* **pollingDaemonFrequency**: time between collection of expired commands in milliseconds. This attribute is optional
+(if this parameter doesn't exist the polling daemon won't be started).
 
 ### Configuration using environment variables
 Some of the configuration parameters can be overriden with environment variables, to ease the use of those parameters with
@@ -1077,8 +1109,13 @@ The following table shows the accepted environment variables, as well as the con
 | IOTA_MONGO_HOST           | mongodb.host                        |
 | IOTA_MONGO_PORT           | mongodb.port                        |
 | IOTA_MONGO_DB             | mongodb.db                          |
+| IOTA_MONGO_REPLICASET     | mongodb.replicaSet                  |
+| IOTA_MONGO_RETRIES        | mongodb.retries                     |
+| IOTA_MONGO_RETRY_TIME     | mongodb.retryTime                   |
 | IOTA_SINGLE_MODE          | singleConfigurationMode             |
 | IOTA_APPEND_MODE          | appendMode                          |
+| IOTA_POLLING_EXPIRATION   | pollingExpiration                   |
+| IOTA_POLLING_DAEMON_FREQ  | pollingDaemonFrequency              |
 
 ## <a name="aboutapi"/> About API
 The library provides a simple operation to retrieve information about the library and the IoTA using it. A GET request

doc/Contribution.md

@@ -21,32 +21,29 @@ repository, it does not have to be iotagent-node-lib, although we will use it in
 git remote add iotagent-node-lib https://github.com/telefonicaid/iotagent-node-lib.git
 </pre>
 
-Before starting your contribution, remember to synchronize the `develop` branch in your forked repository with the `develop`
+Before starting your contribution, remember to synchronize the `master` branch in your forked repository with the `master`
 branch in the main iotagent-node-lib repository, by following this steps
 
-1. Change to your local `develop` branch (in case you are not in it already):
+1. Change to your local `master` branch (in case you are not in it already):
 <pre>
-  git checkout develop
+  git checkout master
 </pre>
 2. Fetch the remote changes:
 <pre>
   git fetch iotagent-node-lib
 </pre>
 3. Merge them:
 <pre>
-  git rebase iotagent-node-lib/develop
+  git rebase iotagent-node-lib/master
 </pre>
 
-Contributions following these guidelines will be added to the `develop` branch, and released in the next version. The
+Contributions following these guidelines will be added to the `master` branch, and released in the next version. The
 release process is explaind in the *Releasing* section below.
 
 ## Branching model
-There are two special branches in the repository:
+There are one special branch in the repository:
 
-* `master`: holds the code for the last stable version of the project. It is only updated when a new version is released,
-and its always updated with the current state of `develop`.
-
-* `develop`: contains the last stable development code. New features and bug fixes are always merged to `develop`.
+* `master`: contains the last stable development code. New features and bug fixes are always merged to `master`.
 
 In order to start developing a new feature or refactoring, a new branch should be created with one of the following names:
 
@@ -56,14 +53,14 @@ In order to start developing a new feature or refactoring, a new branch should b
 
 depending on the kind of work.
 
-This branch must be created from the current version of the `develop` branch. Once the new functionality has been
-completed, a Pull Request will be created from the feature branch to `develop`. Remember to check both the linters
+This branch must be created from the current version of the `master` branch. Once the new functionality has been
+completed, a Pull Request will be created from the feature branch to `master`. Remember to check both the linters
 and the tests before creating the Pull Request.
 
 Bug fixes work the same way as other tasks, with the exception of the branch name, that should be called `bug/<bugName>`.
 
 In order to contribute to the repository, these same scheme should be replicated in the forked repositories, so the
-new features or fixes should all come from the current version of `develop` and end up in `develop` again.
+new features or fixes should all come from the current version of `master` and end up in `master` again.
 
 All the `task/*` and `bug/*` branches are temporary, and should be removed once they have been merged.
 
@@ -72,7 +69,7 @@ point to each of the released versions of the project, they are permanent and th
 
 ## Changelog
 The project contains a version changelog, called CHANGES_NEXT_RELEASE, that can be found in the root of the project.
-Whenever a new feature or bug fix is going to be merged with `develop`, a new entry should be added to this changelog.
+Whenever a new feature or bug fix is going to be merged with `master`, a new entry should be added to this changelog.
 The new entry should contain the reference number of the issue it is solving (if any).
 
 When a new version is released, the changelog is frozen, and CHANGES_NEXT_RELEASE remains fixed in the last commit of
@@ -83,12 +80,11 @@ copied to the %changelog section in the .spec file for the component.
 ## Releasing
 The process of making a release consists of the following steps:
 1. Create a new task branch changing the development version number in the package.json (with a sufix `-next`), to the
-new target version (without any sufix), and PR into `develop`. Also, in this task, the contents of the Changelog are
+new target version (without any sufix), and PR into `master`. Also, in this task, the contents of the Changelog are
 copied to the RPM spec.
-2. Create a tag from the last version of `develop` named with the version number and push it to the repository.
+2. Create a tag from the last version of `master` named with the version number and push it to the repository.
 3. Create the release in Github, from the created tag. In the description, add the contents of the Changelog.
-4. Create a release branch from the last version of `develop` named with the version number.
-5. Merge `develop` into `master`.
+4. Create a release branch from the last version of `master` named with the version number.
 6. Create a new task for preparing the next release, adding the sufix `-next` to the current version number (to signal
 this as the development version), and flush the contents of the CHANGES_NEXT_RELEASE file.
 
@@ -102,17 +98,17 @@ very important changes in the feature set of the component. If X changes, Y is s
 have been released, but the component is just an improved version of the current major release.
 * *Z* will be reserved for bugfixes inside the releases.
 
-Between releases, the version number in the develop branch will be `X.Y.Z-next` (where `X.Y.Z` is the last stable
+Between releases, the version number in the master branch will be `X.Y.Z-next` (where `X.Y.Z` is the last stable
 release), indicating that its a development version.
 
 Concerning the branching model:
-* The develop branch will always have a development version, that will change in every release.
+* The master branch will always have a development version, that will change in every release.
 * The release *tag* will always have the X.Y.0 version number corresponding to the release.
 * The release *branch* will contain the X.Y.Z version number correspoding to the last bugfix in this release.
 
 ## Bugfixing in releases
 When a bug is found affecting a release, a branch will be created *from that release* to create the patch. As a part
 of the patch, the release version must be increased in its last number (Z). The patch then will be merged (via PR)
-to the release branch. If the same bug may affect the main develop branch, the bugfixing branch should also be merged
-to develop (or a new branch should be created from develop with the cherry-picked commits).
+to the release branch. If the same bug may affect the main master branch, the bugfixing branch should also be merged
+to master (or a new branch should be created from master with the cherry-picked commits).
 

lib/commonConfig.js

@@ -26,7 +26,8 @@
 var config = {},
     logger = require('logops'),
     registry,
-    groupRegistry;
+    groupRegistry,
+    commandRegistry;
 
 function anyIsSet(variableSet) {
     for (var i = 0; i < variableSet.length; i++) {
@@ -170,13 +171,30 @@ function processEnvironmentVariables() {
         config.mongodb.replicaSet = process.env.IOTA_MONGO_REPLICASET;
     }
 
+    if (process.env.IOTA_MONGO_RETRIES) {
+        config.mongodb.retries = process.env.IOTA_MONGO_RETRIES;
+    }
+
+    if (process.env.IOTA_MONGO_RETRY_TIME) {
+        config.mongodb.retryTime = process.env.IOTA_MONGO_RETRY_TIME;
+    }
+
     if (process.env.IOTA_SINGLE_MODE) {
         config.singleConfigurationMode = process.env.IOTA_SINGLE_MODE;
     }
 
     if (process.env.IOTA_APPEND_MODE) {
         config.appendMode = process.env.IOTA_APPEND_MODE;
     }
+
+    if (process.env.IOTA_POLLING_EXPIRATION) {
+        config.pollingExpiration = process.env.IOTA_POLLING_EXPIRATION;
+    }
+
+    if (process.env.IOTA_POLLING_DAEMON_FREQ) {
+        config.pollingDaemonFrequency = process.env.IOTA_POLLING_DAEMON_FREQ;
+    }
+
 }
 
 function setConfig(newConfig) {
@@ -209,9 +227,19 @@ function getGroupRegistry() {
     return groupRegistry;
 }
 
+function setCommandRegistry(newCommandRegistry) {
+    commandRegistry = newCommandRegistry;
+}
+
+function getCommandRegistry() {
+    return commandRegistry;
+}
+
 exports.setConfig = setConfig;
 exports.getConfig = getConfig;
 exports.setRegistry = setRegistry;
 exports.getRegistry = getRegistry;
 exports.setGroupRegistry = setGroupRegistry;
 exports.getGroupRegistry = getGroupRegistry;
+exports.setCommandRegistry = setCommandRegistry;
+exports.getCommandRegistry = getCommandRegistry;

lib/constants.js

@@ -34,5 +34,8 @@ module.exports = {
     COMMAND_STATUS: 'commandStatus',
     COMMAND_RESULT: 'commandResult',
 
-    DEFAULT_RESOURCE: '/iot/d'
+    DEFAULT_RESOURCE: '/iot/d',
+
+    DEFAULT_MONGODB_RETRIES: 5,
+    DEFAULT_MONGODB_RETRY_TIME: 5
 };

lib/fiware-iotagent-lib.js

@@ -34,6 +34,7 @@ var async = require('async'),
     domainUtils = require('./services/common/domain'),
     deviceService = require('./services/devices/deviceService'),
     groupConfig = require('./services/groups/groupService'),
+    commands = require('./services/commands/commandService'),
     iotManager = require('./services/common/iotManagerService'),
     contextServer = require('./services/northBound/northboundServer'),
     errors = require('./errors'),
@@ -75,7 +76,8 @@ function globalErrorHandler(err) {
  */
 function doActivate(newConfig, callback) {
     var registry,
-        groupRegistry;
+        groupRegistry,
+        commandRegistry;
 
     logger.format = logger.formatters.pipe;
 
@@ -104,17 +106,20 @@ function doActivate(newConfig, callback) {
 
         registry = require('./services/devices/deviceRegistryMongoDB');
         groupRegistry = require('./services/groups/groupRegistryMongoDB');
+        commandRegistry = require('./services/commands/commandRegistryMongoDB');
     } else {
         logger.info(context, 'Falling back to Transient Memory registry for NGSI Library');
 
         registry = require('./services/devices/deviceRegistryMemory');
         groupRegistry = require('./services/groups/groupRegistryMemory');
+        commandRegistry = require('./services/commands/commandRegistryMemory');
     }
 
     exports.clearAll = function(callback) {
         async.series([
             registry.clear,
             groupRegistry.clear,
+            commandRegistry.clear,
             ngsi.resetMiddlewares
         ], callback);
     };
@@ -125,6 +130,9 @@ function doActivate(newConfig, callback) {
 
     config.setRegistry(registry);
     config.setGroupRegistry(groupRegistry);
+    config.setCommandRegistry(commandRegistry);
+
+    commands.start();
 
     async.series([
         db.configureDb,
@@ -163,6 +171,7 @@ function activate(newConfig, callback) {
  */
 function deactivate(callback) {
     process.removeListener('uncaughtException', globalErrorHandler);
+    commands.stop();
     contextServer.stop(callback);
 }
 
@@ -198,6 +207,11 @@ exports.unsubscribe = subscriptions.unsubscribe;
 exports.getEffectiveApiKey = groupConfig.getEffectiveApiKey;
 exports.intoTrans = intoTrans;
 
+exports.commandQueue = commands.list;
+exports.addCommand = commands.add;
+exports.updateCommand = commands.update;
+exports.removeCommand = commands.remove;
+
 exports.ensureSouthboundDomain = domainUtils.ensureSouthboundDomain;
 exports.finishSouthBoundTransaction = domainUtils.finishSouthBoundTransaction;
 exports.requestDomain = domainUtils.requestDomain;

lib/model/Command.js

@@ -0,0 +1,43 @@
+/*
+ * Copyright 2016 Telefonica Investigación y Desarrollo, S.A.U
+ *
+ * This file is part of fiware-iotagent-lib
+ *
+ * fiware-iotagent-lib is free software: you can redistribute it and/or
+ * modify it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the License,
+ * or (at your option) any later version.
+ *
+ * fiware-iotagent-lib is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
+ * See the GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public
+ * License along with fiware-iotagent-lib.
+ * If not, seehttp://www.gnu.org/licenses/.
+ *
+ * For those usages not covered by the GNU Affero General Public License
+ * please contact with::daniel.moranjimenez@telefonica.com
+ */
+'use strict';
+
+var mongoose = require('mongoose'),
+    Schema = mongoose.Schema;
+
+var Command = new Schema({
+    deviceId: String,
+    type: String,
+    name: String,
+    value: String,
+    service: String,
+    subservice: String,
+    creationDate: { type: Date, default: Date.now }
+});
+
+function load(db) {
+    module.exports.model = db.model('Command', Command);
+    module.exports.internalSchema = Command;
+}
+
+module.exports.load = load;