Merge pull request #29 from kevgliss/sources

Adding ability to define sources for lemur to sync with

docs/administration/index.rst

@@ -150,9 +150,6 @@ Lemur supports sending certification expiration notifications through SES and SM
         LEMUR_SECURITY_TEAM_EMAIL = ['security@example.com']
 
 
-.. data::
-
-
 Authority Options
 -----------------
 
@@ -505,11 +502,19 @@ All commands default to `~/.lemur/lemur.conf.py` if a configuration is not speci
 
 .. data:: sync
 
-    Sync attempts to discover certificates in the environment that were not created by Lemur. There
+    Sync attempts to discover certificates in the environment that were not created by Lemur. If you wish to only sync
+    a few sources you can pass a comma delimited list of sources to sync
+
+    ::
+
+        lemur sync source1,source2
+
+
+    Additionally you can also list the available sources that Lemur can sync
 
     ::
 
-        lemur sync --all
+        lemur sync -list
 
 
 Identity and Access Management

docs/developer/internals/lemur.certificates.rst

@@ -28,15 +28,6 @@ certificates Package
     :undoc-members:
     :show-inheritance:
 
-:mod:`sync` Module
-------------------
-
-.. automodule:: lemur.certificates.sync
-    :noindex:
-    :members:
-    :undoc-members:
-    :show-inheritance:
-
 :mod:`verify` Module
 --------------------
 

docs/developer/plugins/index.rst

@@ -215,9 +215,9 @@ certificate Lemur does not know about and adding the certificate to it's invento
 The `SourcePlugin` object has one default option of `pollRate`. This controls the number of seconds which to get new certificates.
 
  .. warning::
-    Lemur currently has a very basic polling system of running a cron job every 15min to see which source plugins need to be run.
-    This means special consideration needs to be taken such that running all `SourcePlugins` does not take >15min to run. It also means
-    that the minimum resolution of a source plugin poll rate is effectively 15min.
+    Lemur currently has a very basic polling system of running a cron job every 15min to see which source plugins need to be run. A lock file is generated to guarentee that ]
+    only one sync is running at a time. It also means that the minimum resolution of a source plugin poll rate is effectively 15min. You can always specify a faster cron
+    job if you need a higher resolution sync job.
 
 
 The `SourcePlugin` object requires implementation of one function::

docs/quickstart/index.rst

@@ -42,13 +42,13 @@ Finally, activate your virtualenv::
 Installing build dependencies
 -----------------------------
 
-If installing Lemur on true bare Ubuntu OS you will need to grab the following packages so that Lemur can correctly build it's
-dependencies.
+If installing Lemur on truely bare Ubuntu OS you will need to grab the following packages so that Lemur can correctly build it's
+dependencies::
 
     $ sudo apt-get update
     $ sudo apt-get install nodejs-legacy python-pip libpq-dev python-dev build-essential libssl-dev libffi-dev nginx git supervisor
 
-And optionally if your database is going to be on the same host as the webserver.
+And optionally if your database is going to be on the same host as the webserver::
 
     $ sudo apt-get install postgres
 
@@ -110,7 +110,7 @@ Update your configuration
 Once created you will need to update the configuration file with information about your environment,
 such as which database to talk to, where keys are stores etc..
 
-.. Note:: If you are unVfamiliar with with the SQLALCHEMY_DATABASE_URI string it can be broken up like so:
+.. Note:: If you are unfamiliar with with the SQLALCHEMY_DATABASE_URI string it can be broken up like so:
       postgresql://userame:password@databasefqdn:databaseport/databasename
 
 Setup Postgres
@@ -119,7 +119,7 @@ Setup Postgres
 For production a dedicated database is recommended, for this guide we will assume postgres has been installed and is on
 the same machine that Lemur is installed on.
 
-First, set a password for the postgres user.  For this guide, we will use **lemur** as an example but you should use the database password generated for by Lemur.::
+First, set a password for the postgres user.  For this guide, we will use **lemur** as an example but you should use the database password generated for by Lemur::
 
      $ sudo -u postgres psql postgres
      # \password postgres
@@ -139,10 +139,17 @@ Initializing Lemur
 
 Lemur provides a helpful command that will initialize your database for you. It creates a default user (lemur) that is
 used by Lemur to help associate certificates that do not currently have an owner. This is most commonly the case when
-Lemur has discovered certificates from a third party resource. This is also a default user that can be used to
+Lemur has discovered certificates from a third party source. This is also a default user that can be used to
 administer Lemur.
 
-**Make note of the password used as this will be use to first login to the Lemur UI**
+In addition to create a new User, Lemur also creates a few default email notifications. These notifications are based
+on a few configuration options such as `LEMUR_SECURITY_TEAM_EMAIL` they basically garentee that every cerificate within
+Lemur will send one expiration notification to the security team.
+
+Additional notifications can be created through the UI or API.
+See :ref:`Creating Notifications <CreatingNotifications>` and :ref:`Command Line Interface <CommandLineInterface>` for details.
+
+**Make note of the password used as this will be used during first login to the Lemur UI**
 
 .. code-block:: bash
 

lemur/__init__.py

@@ -14,28 +14,27 @@
 from lemur.roles.views import mod as roles_bp
 from lemur.auth.views import mod as auth_bp
 from lemur.domains.views import mod as domains_bp
-from lemur.elbs.views import mod as elbs_bp
 from lemur.destinations.views import mod as destinations_bp
 from lemur.authorities.views import mod as authorities_bp
-from lemur.listeners.views import mod as listeners_bp
 from lemur.certificates.views import mod as certificates_bp
 from lemur.status.views import mod as status_bp
 from lemur.plugins.views import mod as plugins_bp
 from lemur.notifications.views import mod as notifications_bp
+from lemur.sources.views import mod as sources_bp
+
 
 LEMUR_BLUEPRINTS = (
     users_bp,
     roles_bp,
     auth_bp,
     domains_bp,
-    elbs_bp,
     destinations_bp,
     authorities_bp,
-    listeners_bp,
     certificates_bp,
     status_bp,
     plugins_bp,
     notifications_bp,
+    sources_bp
 )
 
 

lemur/authorities/service.py

@@ -9,10 +9,12 @@
 
 """
 from flask import g
+from flask import current_app
 
 from lemur import database
 from lemur.authorities.models import Authority
 from lemur.roles import service as role_service
+from lemur.notifications import service as notification_service
 
 from lemur.roles.models import Role
 from lemur.certificates.models import Certificate
@@ -56,9 +58,15 @@ def create(kwargs):
     cert.description = "This is the ROOT certificate for the {0} certificate authority".format(kwargs.get('caName'))
     cert.user = g.current_user
 
+    cert.notifications = notification_service.create_default_expiration_notifications(
+        'DEFAULT_SECURITY',
+        current_app.config.get('LEMUR_SECURITY_TEAM_EMAIL')
+    )
+
     # we create and attach any roles that the issuer gives us
     role_objs = []
     for r in issuer_roles:
+
         role = role_service.create(
             r['name'],
             password=r['password'],

lemur/certificates/models.py

@@ -23,7 +23,9 @@
 from lemur.domains.models import Domain
 
 from lemur.constants import SAN_NAMING_TEMPLATE, DEFAULT_NAMING_TEMPLATE
-from lemur.models import certificate_associations, certificate_destination_associations, certificate_notification_associations
+
+from lemur.models import certificate_associations, certificate_source_associations, \
+    certificate_destination_associations, certificate_notification_associations
 
 
 def create_name(issuer, not_before, not_after, subject, san):
@@ -222,8 +224,8 @@ class Certificate(db.Model):
     authority_id = Column(Integer, ForeignKey('authorities.id'))
     notifications = relationship("Notification", secondary=certificate_notification_associations, backref='certificate')
     destinations = relationship("Destination", secondary=certificate_destination_associations, backref='certificate')
+    sources = relationship("Source", secondary=certificate_source_associations, backref='certificate')
     domains = relationship("Domain", secondary=certificate_associations, backref="certificate")
-    elb_listeners = relationship("Listener", lazy='dynamic', backref='certificate')
 
     def __init__(self, body, private_key=None, chain=None):
         self.body = body
@@ -277,5 +279,4 @@ def as_dict(self):
 @event.listens_for(Certificate.destinations, 'append')
 def update_destinations(target, value, initiator):
     destination_plugin = plugins.get(value.plugin_name)
-
-    destination_plugin.upload(target.body, target.private_key, target.chain, value.options)
+    destination_plugin.upload(target.name, target.body, target.private_key, target.chain, value.options)

lemur/certificates/service.py

@@ -134,8 +134,11 @@ def import_certificate(**kwargs):
     :param kwargs:
     """
     from lemur.users import service as user_service
-    cert = Certificate(kwargs['public_certificate'])
-    cert.owner = kwargs.get('owner', current_app.config.get('LEMUR_SECURITY_TEAM_EMAIL'))
+    from lemur.notifications import service as notification_service
+    cert = Certificate(kwargs['public_certificate'], chain=kwargs['intermediate_certificate'])
+
+    # TODO future source plugins might have a better understanding of who the 'owner' is we should support this
+    cert.owner = kwargs.get('owner', current_app.config.get('LEMUR_SECURITY_TEAM_EMAIL')[0])
     cert.creator = kwargs.get('creator', user_service.get_by_email('lemur@nobody'))
 
     # NOTE existing certs may not follow our naming standard we will
@@ -146,7 +149,9 @@ def import_certificate(**kwargs):
     if kwargs.get('user'):
         cert.user = kwargs.get('user')
 
-    database.update_list(cert, 'notifications', Notification, kwargs.get('notifications'))
+    notification_name = 'DEFAULT_SECURITY'
+    notifications = notification_service.create_default_expiration_notifications(notification_name, current_app.config.get('LEMUR_SECURITY_TEAM_EMAIL'))
+    cert.notifications = notifications
 
     cert = database.create(cert)
     return cert
@@ -156,39 +161,68 @@ def upload(**kwargs):
     """
     Allows for pre-made certificates to be imported into Lemur.
     """
+    from lemur.notifications import service as notification_service
     cert = Certificate(
         kwargs.get('public_cert'),
         kwargs.get('private_key'),
         kwargs.get('intermediate_cert'),
     )
 
-    database.update_list(cert, 'destinations', Destination, kwargs.get('destinations'))
-    database.update_list(cert, 'notifications', Notification, kwargs.get('notifications'))
+    cert.description = kwargs.get('description')
 
     cert.owner = kwargs['owner']
     cert = database.create(cert)
+
     g.user.certificates.append(cert)
+
+    database.update_list(cert, 'destinations', Destination, kwargs.get('destinations'))
+
+    database.update_list(cert, 'notifications', Notification, kwargs.get('notifications'))
+
+    # create default notifications for this certificate if none are provided
+    notifications = []
+    if not kwargs.get('notifications'):
+        notification_name = "DEFAULT_{0}".format(cert.owner.split('@')[0].upper())
+        notifications += notification_service.create_default_expiration_notifications(notification_name, [cert.owner])
+
+    notification_name = 'DEFAULT_SECURITY'
+    notifications += notification_service.create_default_expiration_notifications(notification_name, current_app.config.get('LEMUR_SECURITY_TEAM_EMAIL'))
+    cert.notifications = notifications
+
+    database.update(cert)
     return cert
 
 
 def create(**kwargs):
     """
     Creates a new certificate.
     """
+    from lemur.notifications import service as notification_service
     cert, private_key, cert_chain = mint(kwargs)
 
     cert.owner = kwargs['owner']
 
-    database.update_list(cert, 'destinations', Destination, kwargs.get('destinations'))
-
     database.create(cert)
     cert.description = kwargs['description']
     g.user.certificates.append(cert)
     database.update(g.user)
 
     # do this after the certificate has already been created because if it fails to upload to the third party
     # we do not want to lose the certificate information.
+    database.update_list(cert, 'destinations', Destination, kwargs.get('destinations'))
+
     database.update_list(cert, 'notifications', Notification, kwargs.get('notifications'))
+
+    # create default notifications for this certificate if none are provided
+    notifications = []
+    if not kwargs.get('notifications'):
+        notification_name = "DEFAULT_{0}".format(cert.owner.split('@')[0].upper())
+        notifications += notification_service.create_default_expiration_notifications(notification_name, [cert.owner])
+
+    notification_name = 'DEFAULT_SECURITY'
+    notifications += notification_service.create_default_expiration_notifications(notification_name, current_app.config.get('LEMUR_SECURITY_TEAM_EMAIL'))
+    cert.notifications = notifications
+
     database.update(cert)
     return cert
 
@@ -297,7 +331,7 @@ def create_csr(csr_config):
                 x509.SubjectAlternativeName(general_names), critical=True
             )
 
-    # TODO support more CSR options, none of the authorities support these atm
+    # TODO support more CSR options, none of the authority plugins currently support these options
     #    builder.add_extension(
     #        x509.KeyUsage(
     #            digital_signature=digital_signature,
@@ -365,14 +399,6 @@ def stats(**kwargs):
     :param kwargs:
     :return:
     """
-    query = database.session_query(Certificate)
-
-    if kwargs.get('active') == 'true':
-        query = query.filter(Certificate.elb_listeners.any())
-
-    if kwargs.get('destination_id'):
-        query = query.filter(Certificate.destinations.any(Destination.id == kwargs.get('destination_id')))
-
     if kwargs.get('metric') == 'not_after':
         start = arrow.utcnow()
         end = start.replace(weeks=+32)
@@ -385,10 +411,6 @@ def stats(**kwargs):
         attr = getattr(Certificate, kwargs.get('metric'))
         query = database.db.session.query(attr, func.count(attr))
 
-        # TODO this could be cleaned up
-        if kwargs.get('active') == 'true':
-            query = query.filter(Certificate.elb_listeners.any())
-
         items = query.group_by(attr).all()
 
     keys = []

lemur/certificates/views.py

@@ -369,6 +369,7 @@ def post(self):
            :statuscode 403: unauthenticated
            :statuscode 200: no error
         """
+        self.reqparse.add_argument('description', type=str, location='json')
         self.reqparse.add_argument('owner', type=str, required=True, location='json')
         self.reqparse.add_argument('publicCert', type=pem_str, required=True, dest='public_cert', location='json')
         self.reqparse.add_argument('destinations', type=list, default=[], dest='destinations', location='json')