0.5 Release (#750)

* Upgrade dependency marshmallow-sqlalchemy to ==0.13.0

* Upgrade dependency marshmallow-sqlalchemy to ==0.13.1

.coveragerc

@@ -1,2 +1,4 @@
 [report]
 include = lemur/*.py
+omit = lemur/migrations/*
+

.pre-commit-config.yaml

@@ -3,3 +3,8 @@
     hooks:
     -   id: trailing-whitespace
     -   id: flake8
+    -   id: check-merge-conflict
+-   repo: git://github.com/pre-commit/mirrors-jshint
+    sha: e72140112bdd29b18b0c8257956c896c4c3cebcb
+    hooks:
+    -   id: jshint

.travis.yml

@@ -3,15 +3,13 @@ sudo: required
 dist: trusty
 
 node_js:
-  - "4.2"
+  - "6.2.0"
 
 addons:
   postgresql: "9.4"
 
 matrix:
   include:
-    - python: "2.7"
-      env: TOXENV=py27
     - python: "3.5"
       env: TOXENV=py35
 

CHANGELOG.rst

@@ -1,14 +1,46 @@
 Changelog
 =========
 
-0.5 - `master`
-~~~~~~~~~~~~~~
+0.5 - `2016-04-08`
+~~~~~~~~~~~~~~~~~~
 
-.. note:: This version is not yet released and is under active development
+This release is most notable for dropping support for python2.7. All Lemur versions >0.4 will now support python3.5 only.
+
+Big thanks to neilschelly for quite a lot of improvements to the `lemur-cryptography` plugin.
+
+Other Highlights:
+
+* Closed `#501 <https://github.com/Netflix/lemur/issues/501>`_ - Endpoint resource as now kept in sync via an
+expiration mechanism. Such that non-existant endpoints gracefully fall out of Lemur. Certificates are never
+removed from Lemur.
+
+* Closed `#551 <https://github.com/Netflix/lemur/pull/551>`_ - Added the ability to create a 4096 bit key during certificate
+creation. Closed `#528 <https://github.com/Netflix/lemur/pull/528>`_ to ensure that issuer plugins supported the new 4096 bit keys.
+
+* Closed `#566 <https://github.com/Netflix/lemur/issues/566>`_ - Fixed an issue changing the notification status for  certificates
+without private keys.
+
+* Closed `#594 <https://github.com/Netflix/lemur/issues/594>`_ - Added `replaced` field indicating if a certificate has been superseded.
+
+* Closed `#602 <https://github.com/Netflix/lemur/issues/602>`_ - AWS plugin added support for ALBs for endpoint tracking.
 
 
-0.4 - ``
-~~~~~~~~
+Special thanks to all who helped with with this release, notably:
+
+- RcRonco
+- harmw
+- jeremyguarini
+
+See the full list of issues closed in `0.5 <https://github.com/Netflix/lemur/milestone/4>`_.
+
+Upgrading
+---------
+
+.. note:: This release will need a slight migration change. Please follow the `documentation <https://lemur.readthedocs.io/en/latest/administration.html#upgrading-lemur>`_ to upgrade Lemur.
+
+
+0.4 - `2016-11-17`
+~~~~~~~~~~~~~~~~~~
 
 There have been quite a few issues closed in this release. Some notables:
 
@@ -24,11 +56,11 @@ Several fixes/tweaks to Lemurs python3 support (thanks chadhendrie!)
 This will most likely be the last release to support python2.7 moving Lemur to target python3 exclusively. Please comment
 on issue #340 if this negatively affects your usage of Lemur.
 
+See the full list of issues closed in `0.4 <https://github.com/Netflix/lemur/milestone/3>`_.
+
 Upgrading
 ---------
 
-See the full list of issues closed in `0.4 <https://github.com/Netflix/lemur/milestone/3>`_.
-
 .. note:: This release will need a slight migration change. Please follow the `documentation <https://lemur.readthedocs.io/en/latest/administration.html#upgrading-lemur>`_ to upgrade Lemur.
 
 

Makefile

@@ -17,7 +17,22 @@ endif
 	pip install "file://`pwd`#egg=lemur[dev]"
 	pip install "file://`pwd`#egg=lemur[tests]"
 	node_modules/.bin/gulp build
-	node_modules/.bin/gulp package
+	node_modules/.bin/gulp package --urlContextPath=$(urlContextPath)
+	@echo ""
+
+release:
+	@echo "--> Installing dependencies"
+ifeq ($(USER), root)
+	@echo "WARNING: It looks like you are installing Lemur as root. This is not generally advised."
+	npm install --unsafe-perm
+else
+	npm install
+endif
+	pip install "setuptools>=0.9.8"
+	# order matters here, base package must install first
+	pip install -e .
+	node_modules/.bin/gulp build
+	node_modules/.bin/gulp package --urlContextPath=$(urlContextPath)
 	@echo ""
 
 dev-docs:
@@ -89,4 +104,4 @@ coverage: develop
 publish:
 	python setup.py sdist bdist_wheel upload
 
-.PHONY: develop dev-postgres dev-docs setup-git build clean update-submodules test testloop test-cli test-js test-python lint lint-python lint-js coverage publish
+.PHONY: develop dev-postgres dev-docs setup-git build clean update-submodules test testloop test-cli test-js test-python lint lint-python lint-js coverage publish release

README.rst

@@ -18,7 +18,7 @@ Lemur
 Lemur manages TLS certificate creation. While not able to issue certificates itself, Lemur acts as a broker between CAs
 and environments providing a central portal for developers to issue TLS certificates with 'sane' defaults.
 
-It works on CPython 2.7, 3.3, 3.4. We deploy on Ubuntu and develop on OS X.
+It works on CPython 3.5. We deploy on Ubuntu and develop on OS X.
 
 
 Project resources

bower.json

@@ -20,7 +20,6 @@
     "angular-loading-bar": "~0.8.0",
     "angular-moment": "~0.10.3",
     "moment-range": "~2.1.0",
-    "angular-spinkit": "~0.3.3",
     "angular-clipboard": "~1.3.0",
     "angularjs-toaster": "~1.0.0",
     "angular-chart.js": "~0.8.8",
@@ -37,7 +36,7 @@
     "angular-underscore": "^0.5.0",
     "angular-translate": "^2.9.0",
     "angular-ui-switch": "~0.1.0",
-    "angular-sanitize": "^1.5.0",
+    "angular-sanitize": "~1.5.0",
     "angular-file-saver": "~1.0.1",
     "angular-ui-select": "~0.17.1",
     "d3": "^3.5.17"

docs/administration.rst

@@ -164,6 +164,14 @@ and are used when Lemur creates the CSR for your certificates.
         LEMUR_DEFAULT_ISSUER_PLUGIN = "verisign-issuer"
 
 
+.. data:: LEMUR_DEFAULT_AUTHORITY
+    :noindex:
+
+    ::
+
+        LEMUR_DEFAULT_AUTHORITY = "verisign"
+
+
 Notification Options
 --------------------
 
@@ -236,7 +244,7 @@ For more information about how to use social logins, see: `Satellizer <https://g
 
         ::
 
-            ACTIVE_PROVIDERS = ["ping", "google"]
+            ACTIVE_PROVIDERS = ["ping", "google", "oauth2"]
 
 .. data:: PING_SECRET
     :noindex:
@@ -295,6 +303,63 @@ For more information about how to use social logins, see: `Satellizer <https://g
 
             PING_AUTH_ENDPOINT = "https://<yourpingserver>/oauth2/authorize"
 
+.. data:: OAUTH2_SECRET
+    :noindex:
+
+        ::
+
+            OAUTH2_SECRET = 'somethingsecret'
+
+.. data:: OAUTH2_ACCESS_TOKEN_URL
+    :noindex:
+
+        ::
+
+            OAUTH2_ACCESS_TOKEN_URL = "https://<youroauthserver> /oauth2/v1/authorize"
+
+
+.. data:: OAUTH2_USER_API_URL
+    :noindex:
+
+        ::
+
+            OAUTH2_USER_API_URL = "https://<youroauthserver>/oauth2/v1/userinfo"
+
+.. data:: OAUTH2_JWKS_URL
+    :noindex:
+
+        ::
+
+            OAUTH2_JWKS_URL = "https://<youroauthserver>/oauth2/v1/keys"
+
+.. data:: OAUTH2_NAME
+    :noindex:
+
+        ::
+
+            OAUTH2_NAME = "Example Oauth2 Provider"
+
+.. data:: OAUTH2_CLIENT_ID
+    :noindex:
+
+        ::
+
+            OAUTH2_CLIENT_ID = "client-id"
+
+.. data:: OAUTH2_REDIRECT_URI
+    :noindex:
+
+        ::
+
+            OAUTH2_REDIRECT_URI = "https://<yourlemurserver>/api/1/auth/oauth2"
+
+.. data:: OAUTH2_AUTH_ENDPOINT
+    :noindex:
+
+        ::
+
+            OAUTH2_AUTH_ENDPOINT = "https://<youroauthserver>/oauth2/v1/authorize"
+
 .. data:: GOOGLE_CLIENT_ID
     :noindex:
 
@@ -407,7 +472,7 @@ The following configuration properties are required to use the Digicert issuer p
 CFSSL Issuer Plugin
 ^^^^^^^^^^^^^^^^^^^
 
-The following configuration properties are required to use the the CFSSL issuer plugin.
+The following configuration properties are required to use the CFSSL issuer plugin.
 
 .. data:: CFSSL_URL
     :noindex:
@@ -481,7 +546,7 @@ STS-AssumeRole
 
 
 
-Next we will create the the Lemur IAM role.
+Next we will create the Lemur IAM role.
 
 .. note::
 
@@ -731,7 +796,7 @@ and to get help on sub-commands
 Upgrading Lemur
 ===============
 
-To upgrade Lemur to the newest release you will need to ensure you have the lastest code and have run any needed
+To upgrade Lemur to the newest release you will need to ensure you have the latest code and have run any needed
 database migrations.
 
 To get the latest code from github run
@@ -939,7 +1004,7 @@ Identity and Access Management
 
 Lemur uses a Role Based Access Control (RBAC) mechanism to control which users have access to which resources. When a
 user is first created in Lemur they can be assigned one or more roles. These roles are typically dynamically created
-depending on a external identity provider (Google, LDAP, etc.,) or are hardcoded within Lemur and associated with special
+depending on an external identity provider (Google, LDAP, etc.), or are hardcoded within Lemur and associated with special
 meaning.
 
 Within Lemur there are three main permissions: AdminPermission, CreatorPermission, OwnerPermission. Sub-permissions such

docs/developer/index.rst

@@ -48,7 +48,7 @@ of Lemur. You'll want to make sure you have a few things on your local system fi
 * pip
 * virtualenv (ideally virtualenvwrapper)
 * node.js (for npm and building css/javascript)
-* (Optional) Potgresql
+* (Optional) PostgreSQL
 
 Once you've got all that, the rest is simple:
 
@@ -86,7 +86,7 @@ You'll likely want to make some changes to the default configuration (we recomme
 	lemur upgrade
 
 
-.. note:: The ``upgrade`` shortcut is simply a shorcut to Alembic's upgrade command.
+.. note:: The ``upgrade`` shortcut is simply a shortcut to Alembic's upgrade command.
 
 
 Coding Standards
@@ -113,6 +113,12 @@ HTML:
   2 Spaces
 
 
+Git hooks
+~~~~~~~~~
+
+To help developers maintain the above standards, Lemur includes a configuration file for Yelp's `pre-commit <http://pre-commit.com/>`_. This is an optional dependency and is not required in order to contribute to Lemur.
+
+
 Running the Test Suite
 ----------------------
 
@@ -156,7 +162,7 @@ This is accomplished with a Gulp task:
 The gulp task compiles all the JS/CSS/HTML files and opens the Lemur welcome page in your default browsers. Additionally any changes to made to the JS/CSS/HTML with be reloaded in your browsers.
 
 Developing with Flask
-----------------------
+---------------------
 
 Because Lemur is just Flask, you can use all of the standard Flask functionality. The only difference is you'll be accessing commands that would normally go through manage.py using the ``lemur`` CLI helper instead.
 
@@ -175,7 +181,7 @@ Schema changes should always introduce the new schema in a commit, and then intr
 
 Removing columns and tables requires a slightly more painful flow, and should resemble the follow multi-commit flow:
 
-- Remove all references to the column or table (but dont remove the Model itself)
+- Remove all references to the column or table (but don't remove the Model itself)
 - Remove the model code
 - Remove the table or column
 
@@ -279,4 +285,3 @@ Internals
     :maxdepth: 2
 
     internals/lemur
-

docs/developer/plugins/index.rst

@@ -25,7 +25,7 @@ if you want to pull the version using pkg_resources (which is what we recommend)
     try:
         VERSION = __import__('pkg_resources') \
             .get_distribution(__name__).version
-    except Exception, e:
+    except Exception as e:
         VERSION = 'unknown'
 
 Inside of ``plugin.py``, you'll declare your Plugin class::
@@ -70,10 +70,18 @@ at multiple plugins within your package::
         },
     )
 
+Once your plugin files are in place and the ``/www/lemur/setup.py`` file has been modified, you can load your plugin into your instance by reinstalling lemur:
+::
+
+    (lemur)$cd /www/lemur
+    (lemur)$pip install -e .
+
 That's it! Users will be able to install your plugin via ``pip install <package name>``.
 
 .. SeeAlso:: For more information about python packages see `Python Packaging <https://packaging.python.org/en/latest/distributing.html>`_
 
+.. SeeAlso:: For an example of a plugin operation outside of Lemur's core, see `lemur-digicert <https://github.com/opendns/lemur-digicert>`_
+
 .. _PluginInterfaces:
 
 Plugin Interfaces
@@ -94,7 +102,7 @@ it can treat any issuer plugin as both a source of creating new certificates as
 
 The `IssuerPlugin` exposes two functions::
 
-    def create_certificate(self, options):
+    def create_certificate(self, csr, issuer_options):
         # requests.get('a third party')
 
 Lemur will pass a dictionary of all possible options for certificate creation. Including a valid CSR, and the raw options associated with the request.
@@ -145,7 +153,7 @@ in the plugins base class like so::
 
 The DestinationPlugin requires only one function to be implemented::
 
-    def upload(self, cert, private_key, cert_chain, options, **kwargs):
+    def upload(self, name, body, private_key, cert_chain, options, **kwargs):
         # request.post('a third party')
 
 Additionally the DestinationPlugin allows the plugin author to add additional options
@@ -154,25 +162,25 @@ that can be used to help define sub-destinations.
 For example, if we look at the aws-destination plugin we can see that it defines an `accountNumber` option::
 
     options = [
-      {
-          'name': 'accountNumber',
-          'type': 'int',
-          'required': True,
-          'validation': '/^[0-9]{12,12}$/',
-          'helpMessage': 'Must be a valid AWS account number!',
-      }
+        {
+            'name': 'accountNumber',
+            'type': 'int',
+            'required': True,
+            'validation': '/^[0-9]{12,12}$/',
+            'helpMessage': 'Must be a valid AWS account number!',
+        }
     ]
 
 By defining an `accountNumber` we can make this plugin handle many N number of AWS accounts instead of just one.
 
 The schema for defining plugin options are pretty straightforward:
 
-  - **Name**: name of the variable you wish to present the user, snake case (snakeCase) is preferrred as Lemur
+  - **Name**: name of the variable you wish to present the user, snake case (snakeCase) is preferred as Lemur
     will parse these and create pretty variable titles
   - **Type** there are currently four supported variable types
       - **Int** creates an html integer box for the user to enter integers into
       - **Str** creates a html text input box
-      - **Boolean** creates a checkbox for the user to signify truithyness
+      - **Boolean** creates a checkbox for the user to signify truthiness
       - **Select** creates a select box that gives the user a list of options
           - When used a `available` key must be provided with a list of selectable options
   - **Required** determines if this option is required, this **must be a boolean value**
@@ -188,7 +196,7 @@ Notification
 ------------
 
 Lemur includes the ability to create Email notifications by **default**. These notifications
-currently come in the form of expiration noticies. Lemur periodically checks certifications expiration dates and
+currently come in the form of expiration notices. Lemur periodically checks certifications expiration dates and
 determines if a given certificate is eligible for notification. There are currently only two parameters used to
 determine if a certificate is eligible; validity expiration (date the certificate is no longer valid) and the number
 of days the current date (UTC) is from that expiration date.
@@ -199,10 +207,10 @@ are trying to create a new notification type (audit, failed logins, etc.) this w
 You would also then need to build additional code to trigger the new notification type.
 
 The second is `ExpirationNotificationPlugin`, this object inherits from `NotificationPlugin` object.
-You will most likely want to base your plugin on, if you want to add new channels for expiration notices (Slack, Hipcat, Jira, etc.). It adds default options that are required by
-by all expiration notifications (interval, unit). This interface expects for the child to define the following function::
+You will most likely want to base your plugin on, if you want to add new channels for expiration notices (Slack, HipChat, Jira, etc.). It adds default options that are required by
+all expiration notifications (interval, unit). This interface expects for the child to define the following function::
 
-    def send(self):
+    def send(self, notification_type, message, targets, options, **kwargs):
         #  request.post("some alerting infrastructure")
 
 
@@ -210,10 +218,10 @@ Source
 ------
 
 When building Lemur we realized that although it would be nice if every certificate went through Lemur to get issued, but this is not
-always be the case. Often times there are third parties that will issue certificates on your behalf and these can get deployed
+always be the case. Oftentimes there are third parties that will issue certificates on your behalf and these can get deployed
 to infrastructure without any interaction with Lemur. In an attempt to combat this and try to track every certificate, Lemur has a notion of
 certificate **Sources**. Lemur will contact the source at periodic intervals and attempt to **sync** against the source. This means downloading or discovering any
-certificate Lemur does not know about and adding the certificate to it's inventory to be tracked and alerted on.
+certificate Lemur does not know about and adding the certificate to its inventory to be tracked and alerted on.
 
 The `SourcePlugin` object has one default option of `pollRate`. This controls the number of seconds which to get new certificates.
 
@@ -225,12 +233,12 @@ The `SourcePlugin` object has one default option of `pollRate`. This controls th
 
 The `SourcePlugin` object requires implementation of one function::
 
-      def get_certificates(self, **kwargs):
+      def get_certificates(self, options, **kwargs):
           #  request.get("some source of certificates")
 
 
 .. note::
-    Often times to facilitate code re-use it makes sense put source and destination plugins into one package.
+    Oftentimes to facilitate code re-use it makes sense put source and destination plugins into one package.
 
 
 Export
@@ -270,9 +278,9 @@ Augment your setup.py to ensure at least the following:
 
    setup(
        # ...
-      install_requires=[
+       install_requires=[
           'lemur',
-      ]
+       ]
    )
 
 

docs/guide/index.rst

@@ -18,7 +18,7 @@ that Lemur can then manage.
 
 .. figure:: create_authority.png
 
-    Enter a authority name and short description about the authority. Enter an owner,
+    Enter an authority name and short description about the authority. Enter an owner,
     and certificate common name. Depending on the authority and the authority/issuer plugin
     these values may or may not be used.
 
@@ -56,7 +56,7 @@ Import an Existing Certificate
 
 .. figure:: upload_certificate.png
 
-    Enter a owner, short description and public certificate. If there are intermediates and private keys
+    Enter an owner, short description and public certificate. If there are intermediates and private keys
     Lemur will track them just as it does if the certificate were created through Lemur. Lemur generates
     a certificate name but you can override that by passing a value to the `Custom Name` field.
 

docs/index.rst

@@ -54,7 +54,7 @@ Doing a Release
     doing-a-release
 
 FAQ
-----
+---
 
 .. toctree::
     :maxdepth: 1

docs/production/index.rst

@@ -37,20 +37,20 @@ Entropy
 -------
 
 Lemur generates private keys for the certificates it creates. This means that it is vitally important that Lemur has enough entropy to draw from. To generate private keys Lemur uses the python library `Cryptography <https://cryptography.io>`_. In turn Cryptography uses OpenSSL bindings to generate
-keys just like you might from the OpenSSL command line. OpenSSL draws it's initial entropy from system during startup and uses PRNGs to generate a stream of random bytes (as output by /dev/urandom) whenever it needs to do a cryptographic operation.
+keys just like you might from the OpenSSL command line. OpenSSL draws its initial entropy from system during startup and uses PRNGs to generate a stream of random bytes (as output by /dev/urandom) whenever it needs to do a cryptographic operation.
 
 What does all this mean? Well in order for the keys
 that Lemur generates to be strong, the system needs to interact with the outside world. This is typically accomplished through the systems hardware (thermal, sound, video user-input, etc.) since the physical world is much more "random" than the computer world.
 
-If you are running Lemur on its own server with its own hardware "bare metal" then the entropy of the system is typically "good enough" for generating keys. If however you are using an VM on shared hardware there is a potential that your initial seed data (data that was initially
-fed to the PRNG) is not very good. What's more VMs have been known to be unable to inject more entropy into the system once it has been started. This is because there is typically very little interaction with the server once it has been started.
+If you are running Lemur on its own server with its own hardware "bare metal" then the entropy of the system is typically "good enough" for generating keys. If however you are using a VM on shared hardware there is a potential that your initial seed data (data that was initially
+fed to the PRNG) is not very good. What's more, VMs have been known to be unable to inject more entropy into the system once it has been started. This is because there is typically very little interaction with the server once it has been started.
 
 The amount of effort you wish to expend ensuring that Lemur has good entropy to draw from is up to your specific risk tolerance and how Lemur is configured.
 
 If you wish to generate more entropy for your system we would suggest you take a look at the following resources:
 
 - `WES-entropy-client <https://github.com/WhitewoodCrypto/WES-entropy-client>`_
-- `haveaged <http://www.issihosts.com/haveged/>`_
+- `haveged <http://www.issihosts.com/haveged/>`_
 
 For additional information about OpenSSL entropy issues:
 
@@ -72,7 +72,7 @@ Nginx is a very popular choice to serve a Python project:
 Nginx doesn't run any Python process, it only serves requests from outside to
 the Python server.
 
-Therefore there are two steps:
+Therefore, there are two steps:
 
 - Run the Python process.
 - Run Nginx.
@@ -223,7 +223,7 @@ Also included in the configurations above are several best practices when it com
 HSTS, disabling vulnerable ciphers are all good ideas when it comes to deploying Lemur into a production environment.
 
 .. note::
-    This is a rather incomplete apache config for running Lemur (needs mod_wsgi etc.,), if you have a working apache config please let us know!
+    This is a rather incomplete apache config for running Lemur (needs mod_wsgi etc.), if you have a working apache config please let us know!
 
 .. seealso::
     `Mozilla SSL Configuration Generator <https://mozilla.github.io/server-side-tls/ssl-config-generator/>`_
@@ -240,10 +240,10 @@ most of the time), but here is a quick overview on how to use it.
 Create a configuration file named supervisor.ini::
 
     [unix_http_server]
-    file=/tmp/supervisor.sock;
+    file=/tmp/supervisor.sock
 
     [supervisorctl]
-    serverurl=unix:///tmp/supervisor.sock;
+    serverurl=unix:///tmp/supervisor.sock
 
     [rpcinterface:supervisor]
     supervisor.rpcinterface_factory=supervisor.rpcinterface:make_main_rpcinterface
@@ -316,4 +316,4 @@ Example cron entries::
 
     0 22 * * * lemuruser export LEMUR_CONF=/Users/me/.lemur/lemur.conf.py; /www/lemur/bin/lemur notify
     */15 * * * * lemuruser export LEMUR_CONF=/Users/me/.lemur/lemur.conf.py; /www/lemur/bin/lemur sync -s all
-    0 22 * * * lemuruser export LEMUR_CONF=/Users/me/.lemur/lemur.conf.py; /www/lemur/bin/lemur check_revoked
+    0 22 * * * lemuruser export LEMUR_CONF=/Users/me/.lemur/lemur.conf.py; /www/lemur/bin/lemur check_revoked

docs/quickstart/index.rst

@@ -12,11 +12,11 @@ Dependencies
 Some basic prerequisites which you'll need in order to run Lemur:
 
 * A UNIX-based operating system (we test on Ubuntu, develop on OS X)
-* Python 2.7
-* PostgreSQL
+* Python 3.5 or greater
+* PostgreSQL 9.4 or greater
 * Nginx
 
-.. note:: Lemur was built with in AWS in mind. This means that things such as databases (RDS), mail (SES), and TLS (ELB), are largely handled for us.  Lemur does **not** require AWS to function. Our guides and documentation try to be be as generic as possible and are not intended to document every step of launching Lemur into a given environment.
+.. note:: Lemur was built with in AWS in mind. This means that things such as databases (RDS), mail (SES), and TLS (ELB), are largely handled for us.  Lemur does **not** require AWS to function. Our guides and documentation try to be as generic as possible and are not intended to document every step of launching Lemur into a given environment.
 
 
 Installing Build Dependencies
@@ -27,7 +27,7 @@ If installing Lemur on a bare Ubuntu OS you will need to grab the following pack
 .. code-block:: bash
 
     $ sudo apt-get update
-    $ sudo apt-get install nodejs-legacy python-pip python-dev libpq-dev build-essential libssl-dev libffi-dev nginx git supervisor npm postgresql
+    $ sudo apt-get install nodejs-legacy python-pip python-dev python3-dev libpq-dev build-essential libssl-dev libffi-dev nginx git supervisor npm postgresql
 
 .. note:: PostgreSQL is only required if your database is going to be on the same host as the webserver.  npm is needed if you're installing Lemur from the source (e.g., from git).
 
@@ -52,6 +52,10 @@ Clone Lemur inside the just created directory and give yourself write permission
 
 .. code-block:: bash
 
+    $ sudo useradd lemur
+    $ sudo passwd lemur
+    $ sudo mkdir /home/lemur
+    $ sudo chown lemur:lemur /home/lemur
     $ sudo git clone https://github.com/Netflix/lemur
     $ sudo chown -R lemur lemur/
 
@@ -59,7 +63,8 @@ Create the virtual environment, activate it and enter the Lemur's directory:
 
 .. code-block:: bash
 
-    $ virtualenv lemur
+    $ su lemur
+    $ virtualenv -p python3 lemur
     $ source /www/lemur/bin/activate
     $ cd lemur
 
@@ -79,11 +84,23 @@ And then run:
 
 .. code-block:: bash
 
-  $ make develop
+  $ make release
 
 .. note:: This command will install npm dependencies as well as compile static assets.
 
 
+You may also run with the urlContextPath variable set. If this is set it will add the desired context path for subsequent calls back to lemur.
+::
+
+  Example:
+    urlContextPath=lemur
+    /api/1/auth/providers -> /lemur/api/1/auth/providers
+
+.. code-block:: bash
+
+  $ make release urlContextPath={desired context path}
+
+
 Creating a configuration
 ------------------------
 
@@ -105,9 +122,24 @@ Update your configuration
 
 Once created, you will need to update the configuration file with information about your environment, such as which database to talk to, where keys are stored etc.
 
-.. note:: If you are unfamiliar with with the SQLALCHEMY_DATABASE_URI string it can be broken up like so:
+.. code-block:: bash
+
+    $ vi ~/.lemur/lemur.conf.py
+
+.. note:: If you are unfamiliar with the SQLALCHEMY_DATABASE_URI string it can be broken up like so:
       ``postgresql://userame:password@<database-fqdn>:<database-port>/<database-name>``
 
+Before Lemur will run you need to fill in a few required variables in the configuration file:
+
+.. code-block:: bash
+
+    LEMUR_SECURITY_TEAM_EMAIL
+    #/the e-mail address needs to be enclosed in quotes
+    LEMUR_DEFAULT_COUNTRY
+    LEMUR_DEFAULT_STATE
+    LEMUR_DEFAULT_LOCATION
+    LEMUR_DEFAUTL_ORGANIZATION
+    LEMUR_DEFAULT_ORGANIZATIONAL_UNIT
 
 Setup Postgres
 --------------
@@ -134,7 +166,10 @@ Next, we will create our new database:
 .. _InitializingLemur:
 
 .. note::
-    For this guide we assume you will use the `postgres` user to connect to your database, when deploying to a VM or container this is often all you will need. If you have a shared database it is recommend you give Lemur it's own user.
+    For this guide we assume you will use the `postgres` user to connect to your database, when deploying to a VM or container this is often all you will need. If you have a shared database it is recommend you give Lemur its own user.
+
+.. note::
+    Postgres 9.4 or greater is required as Lemur relies advanced data columns (e.g. JSON Column type)
 
 Initializing Lemur
 ------------------

docs/security.rst

@@ -60,7 +60,7 @@ and public disclosure may be shortened considerably.
 
 The list of people and organizations who receives advanced notification of
 security issues is not, and will not, be made public. This list generally
-consists of high profile downstream distributors and is entirely at the
+consists of high-profile downstream distributors and is entirely at the
 discretion of the ``lemur`` team.
 
 .. _`master`: https://github.com/Netflix/lemur

gulp/build.js

@@ -1,13 +1,12 @@
+'use strict';
+
 var gulp = require('gulp'),
   minifycss = require('gulp-minify-css'),
   concat = require('gulp-concat'),
   less = require('gulp-less'),
   gulpif = require('gulp-if'),
-  order = require('gulp-order'),
   gutil = require('gulp-util'),
-  rename = require('gulp-rename'),
   foreach = require('gulp-foreach'),
-  debug = require('gulp-debug'),
   path =require('path'),
   merge = require('merge-stream'),
   del = require('del'),
@@ -27,7 +26,8 @@ var gulp = require('gulp'),
   minifyHtml = require('gulp-minify-html'),
   bowerFiles = require('main-bower-files'),
   karma = require('karma'),
-  replace = require('gulp-replace');
+  replace = require('gulp-replace'),
+  argv = require('yargs').argv;
 
 gulp.task('default', ['clean'], function () {
   gulp.start('fonts', 'styles');
@@ -89,9 +89,9 @@ gulp.task('dev:styles', function () {
     .pipe(gulpif(isBootswatchFile, foreach(function (stream, file) {
       var themeName = path.basename(path.dirname(file.path)),
         content = replaceAll(baseContent, '$theme$', themeName),
-        file = string_src('bootstrap-' +  themeName + '.less', content);
+        file2 = string_src('bootstrap-' +  themeName + '.less', content);
 
-      return file;
+      return file2;
     })))
     .pipe(less())
     .pipe(gulpif(isBootstrapFile, foreach(function (stream, file) {
@@ -101,7 +101,7 @@ gulp.task('dev:styles', function () {
       // http://stackoverflow.com/questions/21719833/gulp-how-to-add-src-files-in-the-middle-of-a-pipe
       // https://github.com/gulpjs/gulp/blob/master/docs/recipes/using-multiple-sources-in-one-task.md
       return merge(stream, gulp.src(['.tmp/styles/font-awesome.css', '.tmp/styles/lemur.css']))
-        .pipe(concat('style-' + themeName + ".css"));
+        .pipe(concat('style-' + themeName + '.css'));
     })))
     .pipe(plumber())
     .pipe(concat('styles.css'))
@@ -113,7 +113,7 @@ gulp.task('dev:styles', function () {
 
 // http://stackoverflow.com/questions/1144783/replacing-all-occurrences-of-a-string-in-javascript
 function escapeRegExp(string) {
-  return string.replace(/([.*+?^=!:${}()|\[\]\/\\])/g, "\\$1");
+  return string.replace(/([.*+?^=!:${}()|\[\]\/\\])/g, '\\$1');
 }
 
 function replaceAll(string, find, replace) {
@@ -123,7 +123,7 @@ function replaceAll(string, find, replace) {
 function string_src(filename, string) {
   var src = require('stream').Readable({ objectMode: true });
   src._read = function () {
-    this.push(new gutil.File({ cwd: "", base: "", path: filename, contents: new Buffer(string) }));
+    this.push(new gutil.File({ cwd: '', base: '', path: filename, contents: new Buffer(string) }));
     this.push(null);
   };
   return src;
@@ -144,26 +144,18 @@ gulp.task('build:extras', function () {
 function injectHtml(isDev) {
   return gulp.src('lemur/static/app/index.html')
     .pipe(
-    inject(gulp.src(bowerFiles({ base: 'app' }), {
-      read: false
-    }), {
+    inject(gulp.src(bowerFiles({ base: 'app' })), {
       starttag: '<!-- inject:bower:{{ext}} -->',
       addRootSlash: false,
       ignorePath: isDev ? ['lemur/static/app/', '.tmp/'] : null
     })
   )
-    .pipe(inject(gulp.src(['lemur/static/app/angular/**/*.js'], {
-      read: false
-    }), {
-      read: false,
+    .pipe(inject(gulp.src(['lemur/static/app/angular/**/*.js']), {
       starttag: '<!-- inject:{{ext}} -->',
       addRootSlash: false,
       ignorePath: isDev ? ['lemur/static/app/', '.tmp/'] : null
     }))
-    .pipe(inject(gulp.src(['.tmp/styles/**/*.css'], {
-      read: false
-    }), {
-      read: false,
+    .pipe(inject(gulp.src(['.tmp/styles/**/*.css']), {
       starttag: '<!-- inject:{{ext}} -->',
       addRootSlash: false,
       ignorePath: isDev ? ['lemur/static/app/', '.tmp/'] : null
@@ -171,13 +163,11 @@ function injectHtml(isDev) {
     .pipe(
     gulpif(!isDev,
       inject(gulp.src('lemur/static/dist/ngviews/ngviews.min.js'), {
-        read: false,
         starttag: '<!-- inject:ngviews -->',
         addRootSlash: false
       })
     )
-  )
-    .pipe(gulp.dest('.tmp/'));
+  ).pipe(gulp.dest('.tmp/'));
 }
 
 gulp.task('dev:inject', ['dev:styles', 'dev:scripts'], function () {
@@ -200,23 +190,17 @@ gulp.task('build:ngviews', function () {
 });
 
 gulp.task('build:html', ['dev:styles', 'dev:scripts', 'build:ngviews', 'build:inject'], function () {
-  var jsFilter = filter('**/*.js');
-  var cssFilter = filter('**/*.css');
-
-  var assets = useref.assets();
+  var jsFilter = filter(['**/*.js'], {'restore': true});
+  var cssFilter = filter(['**/*.css'], {'restore': true});
 
   return gulp.src('.tmp/index.html')
-    .pipe(assets)
-    .pipe(rev())
     .pipe(jsFilter)
     .pipe(ngAnnotate())
-    .pipe(jsFilter.restore())
+    .pipe(jsFilter.restore)
     .pipe(cssFilter)
     .pipe(csso())
-    .pipe(cssFilter.restore())
-    .pipe(assets.restore())
+    .pipe(cssFilter.restore)
     .pipe(useref())
-    .pipe(revReplace())
     .pipe(gulp.dest('lemur/static/dist'))
     .pipe(size());
 });
@@ -242,10 +226,34 @@ gulp.task('package:strip', function () {
     .pipe(replace('http:\/\/localhost:3000', ''))
     .pipe(replace('http:\/\/localhost:8000', ''))
     .pipe(useref())
-    .pipe(revReplace())
     .pipe(gulp.dest('lemur/static/dist/scripts'))
     .pipe(size());
 });
 
+gulp.task('addUrlContextPath',['addUrlContextPath:revreplace'], function(){
+  var urlContextPathExists = argv.urlContextPath ? true : false;
+  return gulp.src('lemur/static/dist/scripts/main*.js')
+    .pipe(gulpif(urlContextPathExists, replace('api/', argv.urlContextPath + '/api/')))
+    .pipe(gulpif(urlContextPathExists, replace('angular/', argv.urlContextPath + '/angular/')))
+    .pipe(gulp.dest('lemur/static/dist/scripts'))
+});
+
+gulp.task('addUrlContextPath:revision', function(){
+  return gulp.src(['lemur/static/dist/**/*.css','lemur/static/dist/**/*.js'])
+    .pipe(rev())
+    .pipe(gulp.dest('lemur/static/dist'))
+    .pipe(rev.manifest())
+    .pipe(gulp.dest('lemur/static/dist'))
+})
+
+gulp.task('addUrlContextPath:revreplace', ['addUrlContextPath:revision'], function(){
+  var manifest = gulp.src("lemur/static/dist/rev-manifest.json");
+  var urlContextPathExists = argv.urlContextPath ? true : false;
+  return gulp.src( "lemur/static/dist/index.html")
+    .pipe(gulpif(urlContextPathExists, revReplace({prefix: argv.urlContextPath + '/', manifest: manifest}, revReplace({manifest: manifest}))))
+    .pipe(gulp.dest('lemur/static/dist'));
+})
+
+
 gulp.task('build', ['build:ngviews', 'build:inject', 'build:images', 'build:fonts', 'build:html', 'build:extras']);
-gulp.task('package', ['package:strip']);
+gulp.task('package', ['addUrlContextPath', 'package:strip']);

hooks/pre-commit

@@ -1,45 +0,0 @@
-#!/usr/bin/env python
-
-import glob
-import os
-import sys
-
-os.environ['PYFLAKES_NODOCTEST'] = '1'
-
-# pep8.py uses sys.argv to find setup.cfg
-sys.argv = [os.path.join(os.path.dirname(__file__), os.pardir, os.pardir)]
-
-# git usurbs your bin path for hooks and will always run system python
-if 'VIRTUAL_ENV' in os.environ:
-    site_packages = glob.glob(
-        '%s/lib/*/site-packages' % os.environ['VIRTUAL_ENV'])[0]
-    sys.path.insert(0, site_packages)
-
-
-def py_lint(files_modified):
-    from flake8.engine import get_style_guide
-
-    # remove non-py files and files which no longer exist
-    files_modified = filter(lambda x: x.endswith('.py'), files_modified)
-
-    flake8_style = get_style_guide(parse_argv=True)
-    report = flake8_style.check_files(files_modified)
-
-    return report.total_errors != 0
-
-
-def main():
-    from flake8.hooks import run
-
-    gitcmd = "git diff-index --cached --name-only HEAD"
-
-    _, files_modified, _ = run(gitcmd)
-
-    files_modified = filter(lambda x: os.path.exists(x), files_modified)
-
-    if py_lint(files_modified):
-        return 1
-    return 0
-
-if __name__ == '__main__':
-    sys.exit(main())

lemur/__about__.py

@@ -9,10 +9,10 @@ __title__ = "lemur"
 __summary__ = ("Certificate management and orchestration service")
 __uri__ = "https://github.com/Netflix/lemur"
 
-__version__ = "0.4.0"
+__version__ = "0.5.0"
 
 __author__ = "The Lemur developers"
 __email__ = "security@netflix.com"
 
 __license__ = "Apache License, Version 2.0"
-__copyright__ = "Copyright 2015 {0}".format(__author__)
+__copyright__ = "Copyright 2016 {0}".format(__author__)

lemur/__init__.py

@@ -25,6 +25,7 @@ from lemur.plugins.views import mod as plugins_bp
 from lemur.notifications.views import mod as notifications_bp
 from lemur.sources.views import mod as sources_bp
 from lemur.endpoints.views import mod as endpoints_bp
+from lemur.logs.views import mod as logs_bp
 
 from lemur.__about__ import (
     __author__, __copyright__, __email__, __license__, __summary__, __title__,
@@ -49,7 +50,8 @@ LEMUR_BLUEPRINTS = (
     plugins_bp,
     notifications_bp,
     sources_bp,
-    endpoints_bp
+    endpoints_bp,
+    logs_bp
 )
 
 
@@ -66,7 +68,7 @@ def configure_hook(app):
     :return:
     """
     from flask import jsonify
-    from werkzeug.exceptions import default_exceptions
+    from werkzeug.exceptions import HTTPException
     from lemur.decorators import crossdomain
     if app.config.get('CORS'):
         @app.after_request
@@ -74,13 +76,16 @@ def configure_hook(app):
         def after(response):
             return response
 
-    def make_json_handler(code):
-        def json_handler(error):
-            metrics.send('{}_status_code'.format(code), 'counter', 1)
-            response = jsonify(message=str(error))
-            response.status_code = code
-            return response
-        return json_handler
+    @app.after_request
+    def log_status(response):
+        metrics.send('status_code_{}'.format(response.status_code), 'counter', 1)
+        return response
 
-    for code, value in default_exceptions.items():
-        app.error_handler_spec[None][code] = make_json_handler(code)
+    @app.errorhandler(Exception)
+    def handle_error(e):
+        code = 500
+        if isinstance(e, HTTPException):
+            code = e.code
+
+        app.logger.exception(e)
+        return jsonify(error=str(e)), code

lemur/analyze/service.py

@@ -1,64 +0,0 @@
-"""
-.. module: lemur.analyze.service
-    :copyright: (c) 2015 by Netflix Inc., see AUTHORS for more
-    :license: Apache, see LICENSE for more details.
-.. moduleauthor:: Kevin Glisson <kglisson@netflix.com>
-"""
-# def analyze(endpoints, truststores):
-#    results = {"headings": ["Endpoint"],
-#               "results": [],
-#               "time": datetime.now().strftime("#Y%m%d %H:%M:%S")}
-#
-#    for store in truststores:
-#        results['headings'].append(os.path.basename(store))
-#
-#    for endpoint in endpoints:
-#        result_row = [endpoint]
-#        for store in truststores:
-#            result = {'details': []}
-#
-#            tests = []
-#            for region, ip in REGIONS.items():
-#                try:
-#                    domain = dns.name.from_text(endpoint)
-#                    if not domain.is_absolute():
-#                        domain = domain.concatenate(dns.name.root)
-#
-#                    my_resolver = dns.resolver.Resolver()
-#                    my_resolver.nameservers = [ip]
-#                    answer = my_resolver.query(domain)
-#
-#                    #force the testing of regional enpoints by changing the dns server
-#                    response = requests.get('https://' + str(answer[0]), verify=store)
-#                    tests.append('pass')
-#                    result['details'].append("{}: SSL testing completed without errors".format(region))
-#
-#                except SSLError as e:
-#                    log.debug(e)
-#                    if 'hostname' in str(e):
-#                        tests.append('pass')
-#                        result['details'].append(
-#                              "{}: This test passed ssl negotiation but failed hostname verification because \
-#                                   the hostname is not included in the certificate".format(region))
-#                    elif 'certificate verify failed' in str(e):
-#                        tests.append('fail')
-#                        result['details'].append("{}: This test failed to verify the SSL certificate".format(region))
-#                    else:
-#                        tests.append('fail')
-#                        result['details'].append("{}: {}".format(region, str(e)))
-#
-#                except Exception as e:
-#                    log.debug(e)
-#                    tests.append('fail')
-#                    result['details'].append("{}: {}".format(region, str(e)))
-#
-#            #any failing tests fails the whole endpoint
-#            if 'fail' in tests:
-#                result['test'] = 'fail'
-#            else:
-#                result['test'] = 'pass'
-#
-#            result_row.append(result)
-#        results['results'].append(result_row)
-#    return results
-#

lemur/auth/permissions.py

@@ -9,15 +9,12 @@
 from functools import partial
 from collections import namedtuple
 
-from flask.ext.principal import Permission, RoleNeed
+from flask_principal import Permission, RoleNeed
 
 # Permissions
 operator_permission = Permission(RoleNeed('operator'))
 admin_permission = Permission(RoleNeed('admin'))
 
-CertificateCreator = namedtuple('certificate', ['method', 'value'])
-CertificateCreatorNeed = partial(CertificateCreator, 'key')
-
 CertificateOwner = namedtuple('certificate', ['method', 'value'])
 CertificateOwnerNeed = partial(CertificateOwner, 'role')
 
@@ -28,8 +25,8 @@ class SensitiveDomainPermission(Permission):
 
 
 class CertificatePermission(Permission):
-    def __init__(self, certificate_id, owner, roles):
-        needs = [RoleNeed('admin'), CertificateCreatorNeed(certificate_id), RoleNeed(owner)]
+    def __init__(self, owner, roles):
+        needs = [RoleNeed('admin'), RoleNeed(owner), RoleNeed('creator')]
         for r in roles:
             needs.append(CertificateOwnerNeed(str(r)))
 

lemur/auth/service.py

@@ -8,7 +8,6 @@
 .. moduleauthor:: Kevin Glisson <kglisson@netflix.com>
 
 """
-import sys
 import jwt
 import json
 import binascii
@@ -18,18 +17,17 @@ from datetime import datetime, timedelta
 
 from flask import g, current_app, jsonify, request
 
-from flask.ext.restful import Resource
-from flask.ext.principal import identity_loaded, RoleNeed, UserNeed
+from flask_restful import Resource
+from flask_principal import identity_loaded, RoleNeed, UserNeed
 
-from flask.ext.principal import Identity, identity_changed
+from flask_principal import Identity, identity_changed
 
 from cryptography.hazmat.backends import default_backend
 from cryptography.hazmat.primitives import serialization
 from cryptography.hazmat.primitives.asymmetric.rsa import RSAPublicNumbers
 
 from lemur.users import service as user_service
-from lemur.auth.permissions import CertificateCreatorNeed, \
-    AuthorityCreatorNeed, RoleMemberNeed
+from lemur.auth.permissions import AuthorityCreatorNeed, RoleMemberNeed
 
 
 def get_rsa_public_key(n, e):
@@ -40,12 +38,8 @@ def get_rsa_public_key(n, e):
     :param e:
     :return: a RSA Public Key in PEM format
     """
-    if sys.version_info >= (3, 0):
-        n = int(binascii.hexlify(jwt.utils.base64url_decode(bytes(n, 'utf-8'))), 16)
-        e = int(binascii.hexlify(jwt.utils.base64url_decode(bytes(e, 'utf-8'))), 16)
-    else:
-        n = int(binascii.hexlify(jwt.utils.base64url_decode(str(n))), 16)
-        e = int(binascii.hexlify(jwt.utils.base64url_decode(str(e))), 16)
+    n = int(binascii.hexlify(jwt.utils.base64url_decode(bytes(n, 'utf-8'))), 16)
+    e = int(binascii.hexlify(jwt.utils.base64url_decode(bytes(e, 'utf-8'))), 16)
 
     pub = RSAPublicNumbers(e, n).public_key(default_backend())
     return pub.public_bytes(
@@ -74,7 +68,7 @@ def create_token(user):
 
 def login_required(f):
     """
-    Validates the JWT and ensures that is has not expired.
+    Validates the JWT and ensures that is has not expired and the user is still active.
 
     :param f:
     :return:
@@ -100,7 +94,12 @@ def login_required(f):
         except jwt.InvalidTokenError:
             return dict(message='Token is invalid'), 403
 
-        g.current_user = user_service.get(payload['sub'])
+        user = user_service.get(payload['sub'])
+
+        if not user.active:
+            return dict(message='User is not currently active'), 403
+
+        g.current_user = user
 
         if not g.current_user:
             return dict(message='You are not logged in'), 403
@@ -128,10 +127,7 @@ def fetch_token_header(token):
         raise jwt.DecodeError('Not enough segments')
 
     try:
-        if sys.version_info >= (3, 0):
-            return json.loads(jwt.utils.base64url_decode(header_segment).decode('utf-8'))
-        else:
-            return json.loads(jwt.utils.base64url_decode(header_segment))
+        return json.loads(jwt.utils.base64url_decode(header_segment).decode('utf-8'))
     except TypeError as e:
         current_app.logger.exception(e)
         raise jwt.DecodeError('Invalid header padding')
@@ -163,11 +159,6 @@ def on_identity_loaded(sender, identity):
         for authority in user.authorities:
             identity.provides.add(AuthorityCreatorNeed(authority.id))
 
-    # apply ownership of certificates
-    if hasattr(user, 'certificates'):
-        for certificate in user.certificates:
-            identity.provides.add(CertificateCreatorNeed(certificate.id))
-
     g.user = user
 
 

lemur/auth/views.py

@@ -5,15 +5,15 @@
     :license: Apache, see LICENSE for more details.
 .. moduleauthor:: Kevin Glisson <kglisson@netflix.com>
 """
-import sys
 import jwt
 import base64
+import sys
 import requests
 
 from flask import Blueprint, current_app
 
-from flask.ext.restful import reqparse, Resource, Api
-from flask.ext.principal import Identity, identity_changed
+from flask_restful import reqparse, Resource, Api
+from flask_principal import Identity, identity_changed
 
 from lemur.extensions import metrics
 from lemur.common.utils import get_psuedo_random_string
@@ -94,7 +94,7 @@ class Login(Resource):
         else:
             user = user_service.get_by_username(args['username'])
 
-        if user and user.check_password(args['password']):
+        if user and user.check_password(args['password']) and user.active:
             # Tell Flask-Principal the identity changed
             identity_changed.send(current_app._get_current_object(),
                                   identity=Identity(user.id))
@@ -109,7 +109,7 @@ class Login(Resource):
 class Ping(Resource):
     """
     This class serves as an example of how one might implement an SSO provider for use with Lemur. In
-    this example we use a OpenIDConnect authentication flow, that is essentially OAuth2 underneath. If you have an
+    this example we use an OpenIDConnect authentication flow, that is essentially OAuth2 underneath. If you have an
     OAuth2 provider you want to use Lemur there would be two steps:
 
     1. Define your own class that inherits from :class:`flask.ext.restful.Resource` and create the HTTP methods the \
@@ -143,12 +143,8 @@ class Ping(Resource):
         # the secret and cliendId will be given to you when you signup for the provider
         token = '{0}:{1}'.format(args['clientId'], current_app.config.get("PING_SECRET"))
 
-        if sys.version_info >= (3, 0):
-            basic = base64.b64encode(bytes(token, 'utf-8'))
-            headers = {'authorization': 'basic {0}'.format(basic.decode('utf-8'))}
-        else:
-            basic = base64.b64encode(token, 'utf-8')
-            headers = {'authorization': 'basic {0}'.format(basic)}
+        basic = base64.b64encode(bytes(token, 'utf-8'))
+        headers = {'authorization': 'basic {0}'.format(basic.decode('utf-8'))}
 
         # exchange authorization code for access token.
 
@@ -172,10 +168,7 @@ class Ping(Resource):
 
         # validate your token based on the key it was signed with
         try:
-            if sys.version_info >= (3, 0):
-                jwt.decode(id_token, secret.decode('utf-8'), algorithms=[algo], audience=args['clientId'])
-            else:
-                jwt.decode(id_token, secret, algorithms=[algo], audience=args['clientId'])
+            jwt.decode(id_token, secret.decode('utf-8'), algorithms=[algo], audience=args['clientId'])
         except jwt.DecodeError:
             return dict(message='Token is invalid'), 403
         except jwt.ExpiredSignatureError:
@@ -202,6 +195,7 @@ class Ping(Resource):
             roles.append(role)
 
         role = role_service.get_by_name(profile['email'])
+
         if not role:
             role = role_service.create(profile['email'], description='This is a user specific role')
         roles.append(role)
@@ -239,6 +233,133 @@ class Ping(Resource):
                 roles
             )
 
+        if not user.active:
+            metrics.send('invalid_login', 'counter', 1)
+            return dict(message='The supplied credentials are invalid'), 403
+
+        # Tell Flask-Principal the identity changed
+        identity_changed.send(current_app._get_current_object(), identity=Identity(user.id))
+
+        metrics.send('successful_login', 'counter', 1)
+        return dict(token=create_token(user))
+
+
+class OAuth2(Resource):
+    def __init__(self):
+        self.reqparse = reqparse.RequestParser()
+        super(OAuth2, self).__init__()
+
+    def post(self):
+        self.reqparse.add_argument('clientId', type=str, required=True, location='json')
+        self.reqparse.add_argument('redirectUri', type=str, required=True, location='json')
+        self.reqparse.add_argument('code', type=str, required=True, location='json')
+
+        args = self.reqparse.parse_args()
+
+        # take the information we have received from the provider to create a new request
+        params = {
+            'grant_type': 'authorization_code',
+            'scope': 'openid email profile groups',
+            'redirect_uri': args['redirectUri'],
+            'code': args['code'],
+        }
+
+        # you can either discover these dynamically or simply configure them
+        access_token_url = current_app.config.get('OAUTH2_ACCESS_TOKEN_URL')
+        user_api_url = current_app.config.get('OAUTH2_USER_API_URL')
+
+        # the secret and cliendId will be given to you when you signup for the provider
+        token = '{0}:{1}'.format(args['clientId'], current_app.config.get("OAUTH2_SECRET"))
+
+        basic = base64.b64encode(bytes(token, 'utf-8'))
+
+        headers = {
+            'Content-Type': 'application/x-www-form-urlencoded',
+            'authorization': 'basic {0}'.format(basic.decode('utf-8'))
+        }
+
+        # exchange authorization code for access token.
+        r = requests.post(access_token_url, headers=headers, params=params)
+        id_token = r.json()['id_token']
+        access_token = r.json()['access_token']
+
+        # fetch token public key
+        header_data = fetch_token_header(id_token)
+        jwks_url = current_app.config.get('OAUTH2_JWKS_URL')
+
+        # retrieve the key material as specified by the token header
+        r = requests.get(jwks_url)
+        for key in r.json()['keys']:
+            if key['kid'] == header_data['kid']:
+                secret = get_rsa_public_key(key['n'], key['e'])
+                algo = header_data['alg']
+                break
+        else:
+            return dict(message='Key not found'), 403
+
+        # validate your token based on the key it was signed with
+        try:
+            if sys.version_info >= (3, 0):
+                jwt.decode(id_token, secret.decode('utf-8'), algorithms=[algo], audience=args['clientId'])
+            else:
+                jwt.decode(id_token, secret, algorithms=[algo], audience=args['clientId'])
+        except jwt.DecodeError:
+            return dict(message='Token is invalid'), 403
+        except jwt.ExpiredSignatureError:
+            return dict(message='Token has expired'), 403
+        except jwt.InvalidTokenError:
+            return dict(message='Token is invalid'), 403
+
+        headers = {'authorization': 'Bearer {0}'.format(access_token)}
+
+        # retrieve information about the current user.
+        r = requests.get(user_api_url, headers=headers)
+        profile = r.json()
+
+        user = user_service.get_by_email(profile['email'])
+        metrics.send('successful_login', 'counter', 1)
+
+        # update their google 'roles'
+        roles = []
+
+        role = role_service.get_by_name(profile['email'])
+        if not role:
+            role = role_service.create(profile['email'], description='This is a user specific role')
+        roles.append(role)
+
+        # if we get an sso user create them an account
+        if not user:
+            # every user is an operator (tied to a default role)
+            if current_app.config.get('LEMUR_DEFAULT_ROLE'):
+                v = role_service.get_by_name(current_app.config.get('LEMUR_DEFAULT_ROLE'))
+                if v:
+                    roles.append(v)
+
+            user = user_service.create(
+                profile['name'],
+                get_psuedo_random_string(),
+                profile['email'],
+                True,
+                profile.get('thumbnailPhotoUrl'),
+                roles
+            )
+
+        else:
+            # we add 'lemur' specific roles, so they do not get marked as removed
+            for ur in user.roles:
+                if ur.authority_id:
+                    roles.append(ur)
+
+            # update any changes to the user
+            user_service.update(
+                user.id,
+                profile['name'],
+                profile['email'],
+                True,
+                profile.get('thumbnailPhotoUrl'),  # incase profile isn't google+ enabled
+                roles
+            )
+
         # Tell Flask-Principal the identity changed
         identity_changed.send(current_app._get_current_object(), identity=Identity(user.id))
 
@@ -280,10 +401,16 @@ class Google(Resource):
 
         user = user_service.get_by_email(profile['email'])
 
+        if not user.active:
+            metrics.send('invalid_login', 'counter', 1)
+            return dict(message='The supplied credentials are invalid.'), 401
+
         if user:
             metrics.send('successful_login', 'counter', 1)
             return dict(token=create_token(user))
 
+        metrics.send('invalid_login', 'counter', 1)
+
 
 class Providers(Resource):
     def get(self):
@@ -313,10 +440,27 @@ class Providers(Resource):
                     'type': '2.0'
                 })
 
+            elif provider == "oauth2":
+                active_providers.append({
+                    'name': current_app.config.get("OAUTH2_NAME"),
+                    'url': current_app.config.get('OAUTH2_REDIRECT_URI'),
+                    'redirectUri': current_app.config.get("OAUTH2_REDIRECT_URI"),
+                    'clientId': current_app.config.get("OAUTH2_CLIENT_ID"),
+                    'responseType': 'code',
+                    'scope': ['openid', 'email', 'profile', 'groups'],
+                    'scopeDelimiter': ' ',
+                    'authorizationEndpoint': current_app.config.get("OAUTH2_AUTH_ENDPOINT"),
+                    'requiredUrlParams': ['scope', 'state', 'nonce'],
+                    'state': 'STATE',
+                    'nonce': get_psuedo_random_string(),
+                    'type': '2.0'
+                })
+
         return active_providers
 
 
 api.add_resource(Login, '/auth/login', endpoint='login')
 api.add_resource(Ping, '/auth/ping', endpoint='ping')
 api.add_resource(Google, '/auth/google', endpoint='google')
+api.add_resource(OAuth2, '/auth/oauth2', endpoint='oauth2')
 api.add_resource(Providers, '/auth/providers', endpoint='providers')

lemur/authorities/models.py

@@ -1,7 +1,7 @@
 """
 .. module: lemur.authorities.models
     :platform: unix
-    :synopsis: This module contains all of the models need to create a authority within Lemur.
+    :synopsis: This module contains all of the models need to create an authority within Lemur.
     :copyright: (c) 2015 by Netflix Inc., see AUTHORS for more
     :license: Apache, see LICENSE for more details.
 .. moduleauthor:: Kevin Glisson <kglisson@netflix.com>

lemur/authorities/schemas.py

@@ -60,7 +60,7 @@ class AuthorityInputSchema(LemurInputSchema):
     def validate_subca(self, data):
         if data['type'] == 'subca':
             if not data.get('parent'):
-                raise ValidationError("If generating a subca parent 'authority' must be specified.")
+                raise ValidationError("If generating a subca, parent 'authority' must be specified.")
 
     @pre_load
     def ensure_dates(self, data):
@@ -70,7 +70,7 @@ class AuthorityInputSchema(LemurInputSchema):
 class AuthorityUpdateSchema(LemurInputSchema):
     owner = fields.Email(required=True)
     description = fields.String()
-    active = fields.Boolean()
+    active = fields.Boolean(missing=True)
     roles = fields.Nested(AssociatedRoleSchema(many=True))
 
 

lemur/authorities/service.py

@@ -8,8 +8,6 @@
 .. moduleauthor:: Kevin Glisson <kglisson@netflix.com>
 
 """
-from flask import g
-
 from lemur import database
 from lemur.extensions import metrics
 from lemur.authorities.models import Authority
@@ -20,7 +18,7 @@ from lemur.certificates.service import upload
 
 def update(authority_id, description=None, owner=None, active=None, roles=None):
     """
-    Update a an authority with new values.
+    Update an authority with new values.
 
     :param authority_id:
     :param roles: roles that are allowed to use this authority
@@ -31,9 +29,7 @@ def update(authority_id, description=None, owner=None, active=None, roles=None):
     if roles:
         authority.roles = roles
 
-    if active:
-        authority.active = active
-
+    authority.active = active
     authority.description = description
     authority.owner = owner
     return database.update(authority)
@@ -53,13 +49,14 @@ def mint(**kwargs):
     elif len(values) == 4:
         body, private_key, chain, roles = values
 
-    roles = create_authority_roles(roles, kwargs['owner'], kwargs['plugin']['plugin_object'].title)
+    roles = create_authority_roles(roles, kwargs['owner'], kwargs['plugin']['plugin_object'].title, kwargs['creator'])
     return body, private_key, chain, roles
 
 
-def create_authority_roles(roles, owner, plugin_title):
+def create_authority_roles(roles, owner, plugin_title, creator):
     """
     Creates all of the necessary authority roles.
+    :param creator:
     :param roles:
     :return:
     """
@@ -75,7 +72,7 @@ def create_authority_roles(roles, owner, plugin_title):
 
         # the user creating the authority should be able to administer it
         if role.username == 'admin':
-            g.current_user.roles.append(role)
+            creator.roles.append(role)
 
         role_objs.append(role)
 
@@ -95,10 +92,9 @@ def create(**kwargs):
     """
     Creates a new authority.
     """
-    kwargs['creator'] = g.user.email
     body, private_key, chain, roles = mint(**kwargs)
 
-    g.user.roles = list(set(list(g.user.roles) + roles))
+    kwargs['creator'].roles = list(set(list(kwargs['creator'].roles) + roles))
 
     kwargs['body'] = body
     kwargs['private_key'] = private_key
@@ -114,7 +110,7 @@ def create(**kwargs):
 
     authority = Authority(**kwargs)
     authority = database.create(authority)
-    g.user.authorities.append(authority)
+    kwargs['creator'].authorities.append(authority)
 
     metrics.send('authority_created', 'counter', 1, metric_tags=dict(owner=authority.owner))
     return authority
@@ -151,17 +147,17 @@ def get_by_name(authority_name):
     return database.get(Authority, authority_name, field='name')
 
 
-def get_authority_role(ca_name):
+def get_authority_role(ca_name, creator=None):
     """
     Attempts to get the authority role for a given ca uses current_user
     as a basis for accomplishing that.
 
     :param ca_name:
     """
-    if g.current_user.is_admin:
-        return role_service.get_by_name("{0}_admin".format(ca_name))
-    else:
-        return role_service.get_by_name("{0}_operator".format(ca_name))
+    if creator:
+        if creator.is_admin:
+            return role_service.get_by_name("{0}_admin".format(ca_name))
+    return role_service.get_by_name("{0}_operator".format(ca_name))
 
 
 def render(args):
@@ -180,13 +176,13 @@ def render(args):
         else:
             query = database.filter(query, Authority, terms)
 
-    # we make sure that a user can only use an authority they either own are are a member of - admins can see all
-    if not g.current_user.is_admin:
+    # we make sure that a user can only use an authority they either own are a member of - admins can see all
+    if not args['user'].is_admin:
         authority_ids = []
-        for authority in g.current_user.authorities:
+        for authority in args['user'].authorities:
             authority_ids.append(authority.id)
 
-        for role in g.current_user.roles:
+        for role in args['user'].roles:
             for authority in role.authorities:
                 authority_ids.append(authority.id)
         query = query.filter(Authority.id.in_(authority_ids))

lemur/authorities/views.py

@@ -5,8 +5,8 @@
     :license: Apache, see LICENSE for more details.
 .. moduleauthor:: Kevin Glisson <kglisson@netflix.com>
 """
-from flask import Blueprint
-from flask.ext.restful import reqparse, Api
+from flask import Blueprint, g
+from flask_restful import reqparse, Api
 
 from lemur.common.utils import paginated_parser
 from lemur.common.schema import validate_schema
@@ -95,7 +95,7 @@ class AuthoritiesList(AuthenticatedResource):
               }
 
            :query sortBy: field to sort on
-           :query sortDir: acs or desc
+           :query sortDir: asc or desc
            :query page: int default is 1
            :query filter: key value pair. format is k;v
            :query count: count number default is 10
@@ -107,6 +107,7 @@ class AuthoritiesList(AuthenticatedResource):
         """
         parser = paginated_parser.copy()
         args = parser.parse_args()
+        args['user'] = g.current_user
         return service.render(args)
 
     @validate_schema(authority_input_schema, authority_output_schema)
@@ -218,6 +219,7 @@ class AuthoritiesList(AuthenticatedResource):
            :statuscode 403: unauthenticated
            :statuscode 200: no error
         """
+        data['creator'] = g.current_user
         return service.create(**data)
 
 
@@ -283,7 +285,7 @@ class Authorities(AuthenticatedResource):
         """
         .. http:put:: /authorities/1
 
-           Update a authority
+           Update an authority
 
            **Example request**:
 
@@ -503,6 +505,7 @@ class AuthorityVisualizations(AuthenticatedResource):
         authority = service.get(authority_id)
         return dict(name=authority.name, children=[{"name": c.name} for c in authority.certificates])
 
+
 api.add_resource(AuthoritiesList, '/authorities', endpoint='authorities')
 api.add_resource(Authorities, '/authorities/<int:authority_id>', endpoint='authority')
 api.add_resource(AuthorityVisualizations, '/authorities/<int:authority_id>/visualize', endpoint='authority_visualizations')

lemur/certificates/cli.py

@@ -0,0 +1,236 @@
+"""
+.. module: lemur.certificate.cli
+    :platform: Unix
+    :copyright: (c) 2015 by Netflix Inc., see AUTHORS for more
+    :license: Apache, see LICENSE for more details.
+.. moduleauthor:: Kevin Glisson <kglisson@netflix.com>
+"""
+import sys
+
+from flask import current_app
+
+from flask_script import Manager
+
+from lemur import database
+from lemur.extensions import metrics
+from lemur.deployment import service as deployment_service
+from lemur.endpoints import service as endpoint_service
+from lemur.notifications.messaging import send_rotation_notification
+from lemur.certificates.service import reissue_certificate, get_certificate_primitives, get_all_pending_reissue, get_by_name, get_all_certs
+
+from lemur.certificates.verify import verify_string
+
+manager = Manager(usage="Handles all certificate related tasks.")
+
+
+def print_certificate_details(details):
+    """
+    Print the certificate details with formatting.
+    :param details:
+    :return:
+    """
+    print("[+] Re-issuing certificate with the following details: ")
+    print(
+        "\t[+] Common Name: {common_name}\n"
+        "\t[+] Subject Alternate Names: {sans}\n"
+        "\t[+] Authority: {authority_name}\n"
+        "\t[+] Validity Start: {validity_start}\n"
+        "\t[+] Validity End: {validity_end}\n"
+        "\t[+] Organization: {organization}\n"
+        "\t[+] Organizational Unit: {organizational_unit}\n"
+        "\t[+] Country: {country}\n"
+        "\t[+] State: {state}\n"
+        "\t[+] Location: {location}".format(
+            common_name=details['common_name'],
+            sans=",".join(x['value'] for x in details['extensions']['sub_alt_names']['names']) or None,
+            authority_name=details['authority'].name,
+            validity_start=details['validity_start'].isoformat(),
+            validity_end=details['validity_end'].isoformat(),
+            organization=details['organization'],
+            organizational_unit=details['organizational_unit'],
+            country=details['country'],
+            state=details['state'],
+            location=details['location']
+        )
+    )
+
+
+def validate_certificate(certificate_name):
+    """
+    Ensuring that the specified certificate exists.
+    :param certificate_name:
+    :return:
+    """
+    if certificate_name:
+        cert = get_by_name(certificate_name)
+
+        if not cert:
+            print("[-] No certificate found with name: {0}".format(certificate_name))
+            sys.exit(1)
+
+        return cert
+
+
+def validate_endpoint(endpoint_name):
+    """
+    Ensuring that the specified endpoint exists.
+    :param endpoint_name:
+    :return:
+    """
+    if endpoint_name:
+        endpoint = endpoint_service.get_by_name(endpoint_name)
+
+        if not endpoint:
+            print("[-] No endpoint found with name: {0}".format(endpoint_name))
+            sys.exit(1)
+
+        return endpoint
+
+
+def request_rotation(endpoint, certificate, message, commit):
+    """
+    Rotates a certificate and handles any exceptions during
+    execution.
+    :param endpoint:
+    :param certificate:
+    :param message:
+    :param commit:
+    :return:
+    """
+    if commit:
+        try:
+            deployment_service.rotate_certificate(endpoint, certificate)
+            metrics.send('endpoint_rotation_success', 'counter', 1)
+
+            if message:
+                send_rotation_notification(certificate)
+
+        except Exception as e:
+            metrics.send('endpoint_rotation_failure', 'counter', 1)
+            print(
+                "[!] Failed to rotate endpoint {0} to certificate {1} reason: {2}".format(
+                    endpoint.name,
+                    certificate.name,
+                    e
+                )
+            )
+
+
+def request_reissue(certificate, commit):
+    """
+    Reissuing certificate and handles any exceptions.
+    :param certificate:
+    :param commit:
+    :return:
+    """
+    details = get_certificate_primitives(certificate)
+
+    print_certificate_details(details)
+    if commit:
+        try:
+            new_cert = reissue_certificate(certificate, replace=True)
+            metrics.send('certificate_reissue_success', 'counter', 1)
+            print("[+] New certificate named: {0}".format(new_cert.name))
+        except Exception as e:
+            metrics.send('certificate_reissue_failure', 'counter', 1)
+            print(
+                "[!] Failed to reissue certificate {1} reason: {2}".format(
+                    certificate.name,
+                    e
+                )
+            )
+
+
+@manager.option('-e', '--endpoint', dest='endpoint_name', help='Name of the endpoint you wish to rotate.')
+@manager.option('-n', '--new-certificate', dest='new_certificate_name', help='Name of the certificate you wish to rotate to.')
+@manager.option('-o', '--old-certificate', dest='old_certificate_name', help='Name of the certificate you wish to rotate.')
+@manager.option('-a', '--notify', dest='message', action='store_true', help='Send a rotation notification to the certificates owner.')
+@manager.option('-c', '--commit', dest='commit', action='store_true', default=False, help='Persist changes.')
+def rotate(endpoint_name, new_certificate_name, old_certificate_name, message, commit):
+    """
+    Rotates an endpoint and reissues it if it has not already been replaced. If it has
+    been replaced, will use the replacement certificate for the rotation.
+    """
+    if commit:
+        print("[!] Running in COMMIT mode.")
+
+    print("[+] Starting endpoint rotation.")
+
+    old_cert = validate_certificate(old_certificate_name)
+    new_cert = validate_certificate(new_certificate_name)
+    endpoint = validate_endpoint(endpoint_name)
+
+    if endpoint and new_cert:
+        print("[+] Rotating endpoint: {0} to certificate {1}".format(endpoint.name, new_cert.name))
+        request_rotation(endpoint, new_cert, message, commit)
+
+    elif old_cert and new_cert:
+        print("[+] Rotating all endpoints from {0} to {1}".format(old_cert.name, new_cert.name))
+
+        for endpoint in old_cert.endpoints:
+            print("[+] Rotating {0}".format(endpoint.name))
+            request_rotation(endpoint, new_cert, message, commit)
+
+    else:
+        print("[+] Rotating all endpoints that have new certificates available")
+        for endpoint in endpoint_service.get_all_pending_rotation():
+            if len(endpoint.certificate.replaced) == 1:
+                print("[+] Rotating {0} to {1}".format(endpoint.name, endpoint.certificate.replaced[0].name))
+                request_rotation(endpoint, endpoint.certificate.replaced[0], message, commit)
+            else:
+                metrics.send('endpoint_rotation_failure', 'counter', 1)
+                print("[!] Failed to rotate endpoint {0} reason: Multiple replacement certificates found.".format(
+                    endpoint.name
+                ))
+
+    print("[+] Done!")
+
+
+@manager.option('-o', '--old-certificate', dest='old_certificate_name', help='Name of the certificate you wish to reissue.')
+@manager.option('-c', '--commit', dest='commit', action='store_true', default=False, help='Persist changes.')
+def reissue(old_certificate_name, commit):
+    """
+    Reissues certificate with the same parameters as it was originally issued with.
+    If not time period is provided, reissues certificate as valid from today to
+    today + length of original.
+    """
+    if commit:
+        print("[!] Running in COMMIT mode.")
+
+    print("[+] Starting certificate re-issuance.")
+
+    old_cert = validate_certificate(old_certificate_name)
+
+    if not old_cert:
+        for certificate in get_all_pending_reissue():
+            print("[+] {0} is eligible for re-issuance".format(certificate.name))
+            request_reissue(certificate, commit)
+    else:
+        request_reissue(old_cert, commit)
+
+    print("[+] Done!")
+
+
+@manager.command
+def check_revoked():
+    """
+    Function attempts to update Lemur's internal cache with revoked
+    certificates. This is called periodically by Lemur. It checks both
+    CRLs and OCSP to see if a certificate is revoked. If Lemur is unable
+    encounters an issue with verification it marks the certificate status
+    as `unknown`.
+    """
+    for cert in get_all_certs():
+        try:
+            if cert.chain:
+                status = verify_string(cert.body, cert.chain)
+            else:
+                status = verify_string(cert.body, "")
+
+            cert.status = 'valid' if status else 'invalid'
+
+        except Exception as e:
+            current_app.logger.exception(e)
+            cert.status = 'unknown'
+
+        database.update(cert)

lemur/certificates/exceptions.py

@@ -1,88 +0,0 @@
-"""
-.. module: lemur.certificates.exceptions
-    :synopsis: Defines all monterey specific exceptions
-    :copyright: (c) 2015 by Netflix Inc., see AUTHORS for more
-    :license: Apache, see LICENSE for more details.
-.. moduleauthor:: Kevin Glisson <kglisson@netflix.com>
-"""
-from flask import current_app
-from lemur.exceptions import LemurException
-
-
-class UnknownAuthority(LemurException):
-    def __init__(self, authority):
-        self.code = 404
-        self.authority = authority
-        self.data = {"message": "The authority specified '{}' is not a valid authority".format(self.authority)}
-
-        current_app.logger.warning(self)
-
-    def __str__(self):
-        return repr(self.data['message'])
-
-
-class InsufficientDomains(LemurException):
-    def __init__(self):
-        self.code = 400
-        self.data = {"message": "Need at least one domain specified in order create a certificate"}
-
-        current_app.logger.warning(self)
-
-    def __str__(self):
-        return repr(self.data['message'])
-
-
-class InvalidCertificate(LemurException):
-    def __init__(self):
-        self.code = 400
-        self.data = {"message": "Need at least one domain specified in order create a certificate"}
-
-        current_app.logger.warning(self)
-
-    def __str__(self):
-        return repr(self.data['message'])
-
-
-class UnableToCreateCSR(LemurException):
-    def __init__(self):
-        self.code = 500
-        self.data = {"message": "Unable to generate CSR"}
-
-        current_app.logger.error(self)
-
-    def __str__(self):
-        return repr(self.data['message'])
-
-
-class UnableToCreatePrivateKey(LemurException):
-    def __init__(self):
-        self.code = 500
-        self.data = {"message": "Unable to generate Private Key"}
-
-        current_app.logger.error(self)
-
-    def __str__(self):
-        return repr(self.data['message'])
-
-
-class MissingFiles(LemurException):
-    def __init__(self, path):
-        self.code = 500
-        self.path = path
-        self.data = {"path": self.path, "message": "Expecting missing files"}
-
-        current_app.logger.error(self)
-
-    def __str__(self):
-        return repr(self.data['message'])
-
-
-class NoPersistanceFound(LemurException):
-    def __init__(self):
-        self.code = 500
-        self.data = {"code": 500, "message": "No peristence method found, Lemur cannot persist sensitive information"}
-
-        current_app.logger.error(self)
-
-    def __str__(self):
-        return repr(self.data['message'])

lemur/certificates/models.py

@@ -5,35 +5,74 @@
     :license: Apache, see LICENSE for more details.
 .. moduleauthor:: Kevin Glisson <kglisson@netflix.com>
 """
-import datetime
+import arrow
 
-import lemur.common.utils
 from flask import current_app
 
+from cryptography import x509
+from cryptography.hazmat.primitives.asymmetric import rsa
+
+from idna.core import InvalidCodepoint
+
 from sqlalchemy.orm import relationship
 from sqlalchemy.sql.expression import case
 from sqlalchemy.ext.hybrid import hybrid_property
-from sqlalchemy import event, Integer, ForeignKey, String, DateTime, PassiveDefault, func, Column, Text, Boolean
+from sqlalchemy import event, Integer, ForeignKey, String, PassiveDefault, func, Column, Text, Boolean
+
+from sqlalchemy_utils.types.arrow import ArrowType
+
+import lemur.common.utils
 
 from lemur.database import db
+
+from lemur.utils import Vault
+from lemur.common import defaults
+
+from lemur.plugins.base import plugins
+
+from lemur.extensions import metrics
+
 from lemur.models import certificate_associations, certificate_source_associations, \
     certificate_destination_associations, certificate_notification_associations, \
     certificate_replacement_associations, roles_certificates
-from lemur.plugins.base import plugins
-from lemur.utils import Vault
 
-from lemur.common import defaults
 from lemur.domains.models import Domain
 
 
+def get_sequence(name):
+    if '-' not in name:
+        return name, None
+
+    parts = name.split('-')
+    end = parts.pop(-1)
+    root = '-'.join(parts)
+
+    if len(end) == 8:
+        return root + '-' + end, None
+
+    try:
+        end = int(end)
+    except ValueError:
+        end = None
+
+    return root, end
+
+
 def get_or_increase_name(name):
     name = '-'.join(name.strip().split(' '))
-    count = Certificate.query.filter(Certificate.name.ilike('{0}%'.format(name))).count()
+    certificates = Certificate.query.filter(Certificate.name.ilike('{0}%'.format(name))).all()
 
-    if count >= 1:
-        return name + '-' + str(count)
+    if not certificates:
+        return name
 
-    return name
+    ends = [0]
+    root, end = get_sequence(name)
+    for cert in certificates:
+        root, end = get_sequence(cert.name)
+        if end:
+            ends.append(end)
+
+    return '{0}-{1}'.format(root, max(ends) + 1)
 
 
 class Certificate(db.Model):
@@ -53,31 +92,34 @@ class Certificate(db.Model):
     cn = Column(String(128))
     deleted = Column(Boolean, index=True)
 
-    not_before = Column(DateTime)
-    not_after = Column(DateTime)
-    date_created = Column(DateTime, PassiveDefault(func.now()), nullable=False)
+    not_before = Column(ArrowType)
+    not_after = Column(ArrowType)
+    date_created = Column(ArrowType, PassiveDefault(func.now()), nullable=False)
 
     signing_algorithm = Column(String(128))
     status = Column(String(128))
     bits = Column(Integer())
     san = Column(String(1024))  # TODO this should be migrated to boolean
 
+    rotation = Column(Boolean, default=False)
+
     user_id = Column(Integer, ForeignKey('users.id'))
     authority_id = Column(Integer, ForeignKey('authorities.id', ondelete="CASCADE"))
     root_authority_id = Column(Integer, ForeignKey('authorities.id', ondelete="CASCADE"))
 
-    notifications = relationship("Notification", secondary=certificate_notification_associations, backref='certificate')
-    destinations = relationship("Destination", secondary=certificate_destination_associations, backref='certificate')
-    sources = relationship("Source", secondary=certificate_source_associations, backref='certificate')
-    domains = relationship("Domain", secondary=certificate_associations, backref="certificate")
-    roles = relationship("Role", secondary=roles_certificates, backref="certificate")
-    replaces = relationship("Certificate",
+    notifications = relationship('Notification', secondary=certificate_notification_associations, backref='certificate')
+    destinations = relationship('Destination', secondary=certificate_destination_associations, backref='certificate')
+    sources = relationship('Source', secondary=certificate_source_associations, backref='certificate')
+    domains = relationship('Domain', secondary=certificate_associations, backref='certificate')
+    roles = relationship('Role', secondary=roles_certificates, backref='certificate')
+    replaces = relationship('Certificate',
                             secondary=certificate_replacement_associations,
                             primaryjoin=id == certificate_replacement_associations.c.certificate_id,  # noqa
                             secondaryjoin=id == certificate_replacement_associations.c.replaced_certificate_id,  # noqa
                             backref='replaced')
 
-    endpoints = relationship("Endpoint", backref='certificate')
+    logs = relationship('Log', backref='certificate')
+    endpoints = relationship('Endpoint', backref='certificate')
 
     def __init__(self, **kwargs):
         cert = lemur.common.utils.parse_certificate(kwargs['body'])
@@ -108,7 +150,8 @@ class Certificate(db.Model):
         self.notifications = kwargs.get('notifications', [])
         self.description = kwargs.get('description')
         self.roles = list(set(kwargs.get('roles', [])))
-        self.replaces = kwargs.get('replacements', [])
+        self.replaces = kwargs.get('replaces', [])
+        self.rotation = kwargs.get('rotation')
         self.signing_algorithm = defaults.signing_algorithm(cert)
         self.bits = defaults.bitstrength(cert)
         self.serial = defaults.serial(cert)
@@ -120,16 +163,65 @@ class Certificate(db.Model):
     def active(self):
         return self.notify
 
+    @property
+    def organization(self):
+        cert = lemur.common.utils.parse_certificate(self.body)
+        return defaults.organization(cert)
+
+    @property
+    def organizational_unit(self):
+        cert = lemur.common.utils.parse_certificate(self.body)
+        return defaults.organizational_unit(cert)
+
+    @property
+    def country(self):
+        cert = lemur.common.utils.parse_certificate(self.body)
+        return defaults.country(cert)
+
+    @property
+    def state(self):
+        cert = lemur.common.utils.parse_certificate(self.body)
+        return defaults.state(cert)
+
+    @property
+    def location(self):
+        cert = lemur.common.utils.parse_certificate(self.body)
+        return defaults.location(cert)
+
+    @property
+    def key_type(self):
+        cert = lemur.common.utils.parse_certificate(self.body)
+        if isinstance(cert.public_key(), rsa.RSAPublicKey):
+            return 'RSA{key_size}'.format(key_size=cert.public_key().key_size)
+
+    @property
+    def validity_remaining(self):
+        return abs(self.not_after - arrow.utcnow())
+
+    @property
+    def validity_range(self):
+        return self.not_after - self.not_before
+
+    @property
+    def subject(self):
+        cert = lemur.common.utils.parse_certificate(self.body)
+        return cert.subject
+
+    @property
+    def public_key(self):
+        cert = lemur.common.utils.parse_certificate(self.body)
+        return cert.public_key()
+
     @hybrid_property
     def expired(self):
-        if self.not_after <= datetime.datetime.now():
+        if self.not_after <= arrow.utcnow():
             return True
 
     @expired.expression
     def expired(cls):
         return case(
             [
-                (cls.now_after <= datetime.datetime.now(), True)
+                (cls.not_after <= arrow.utcnow(), True)
             ],
             else_=False
         )
@@ -148,6 +240,61 @@ class Certificate(db.Model):
             else_=False
         )
 
+    @property
+    def extensions(self):
+        # setup default values
+        return_extensions = {
+            'sub_alt_names': {'names': []}
+        }
+
+        try:
+            cert = lemur.common.utils.parse_certificate(self.body)
+            for extension in cert.extensions:
+                value = extension.value
+                if isinstance(value, x509.BasicConstraints):
+                    return_extensions['basic_constraints'] = value
+
+                elif isinstance(value, x509.SubjectAlternativeName):
+                    return_extensions['sub_alt_names']['names'] = value
+
+                elif isinstance(value, x509.ExtendedKeyUsage):
+                    return_extensions['extended_key_usage'] = value
+
+                elif isinstance(value, x509.KeyUsage):
+                    return_extensions['key_usage'] = value
+
+                elif isinstance(value, x509.SubjectKeyIdentifier):
+                    return_extensions['subject_key_identifier'] = {'include_ski': True}
+
+                elif isinstance(value, x509.AuthorityInformationAccess):
+                    return_extensions['certificate_info_access'] = {'include_aia': True}
+
+                elif isinstance(value, x509.AuthorityKeyIdentifier):
+                    aki = {
+                        'use_key_identifier': False,
+                        'use_authority_cert': False
+                    }
+
+                    if value.key_identifier:
+                        aki['use_key_identifier'] = True
+
+                    if value.authority_cert_issuer:
+                        aki['use_authority_cert'] = True
+
+                    return_extensions['authority_key_identifier'] = aki
+
+                # TODO: Don't support CRLDistributionPoints yet https://github.com/Netflix/lemur/issues/662
+                elif isinstance(value, x509.CRLDistributionPoints):
+                    current_app.logger.warning('CRLDistributionPoints not yet supported for clone operation.')
+
+                # TODO: Not supporting custom OIDs yet. https://github.com/Netflix/lemur/issues/665
+                else:
+                    current_app.logger.warning('Custom OIDs not yet supported for clone operation.')
+        except InvalidCodepoint as e:
+            current_app.logger.warning('Unable to parse extensions due to underscore in dns name')
+
+        return return_extensions
+
     def get_arn(self, account_number):
         """
         Generate a valid AWS IAM arn
@@ -165,7 +312,7 @@ class Certificate(db.Model):
 @event.listens_for(Certificate.destinations, 'append')
 def update_destinations(target, value, initiator):
     """
-    Attempt to upload the new certificate to the new destination
+    Attempt to upload certificate to the new destination
 
     :param target:
     :param value:
@@ -175,9 +322,11 @@ def update_destinations(target, value, initiator):
     destination_plugin = plugins.get(value.plugin_name)
 
     try:
-        destination_plugin.upload(target.name, target.body, target.private_key, target.chain, value.options)
+        if target.private_key:
+            destination_plugin.upload(target.name, target.body, target.private_key, target.chain, value.options)
     except Exception as e:
         current_app.logger.exception(e)
+        metrics.send('destination_upload_failure', 'counter', 1, metric_tags={'certificate': target.name, 'destination': value.label})
 
 
 @event.listens_for(Certificate.replaces, 'append')
@@ -191,20 +340,3 @@ def update_replacement(target, value, initiator):
     :return:
     """
     value.notify = False
-
-
-# @event.listens_for(Certificate, 'before_update')
-# def protect_active(mapper, connection, target):
-#    """
-#     When a certificate has a replacement do not allow it to be marked as 'active'
-#
-#     :param connection:
-#     :param mapper:
-#     :param target:
-#     :return:
-#     """
-#     if target.active:
-#         if not target.notify:
-#             raise Exception(
-#                 "Cannot silence notification for a certificate Lemur has been found to be currently deployed onto endpoints"
-#             )

lemur/certificates/schemas.py

@@ -6,7 +6,7 @@
 .. moduleauthor:: Kevin Glisson <kglisson@netflix.com>
 """
 from flask import current_app
-from marshmallow import fields, validates_schema, post_load, pre_load
+from marshmallow import fields, validate, validates_schema, post_load, pre_load
 from marshmallow.exceptions import ValidationError
 
 from lemur.schemas import AssociatedAuthoritySchema, AssociatedDestinationSchema, AssociatedCertificateSchema, \
@@ -54,12 +54,15 @@ class CertificateInputSchema(CertificateCreationSchema):
 
     destinations = fields.Nested(AssociatedDestinationSchema, missing=[], many=True)
     notifications = fields.Nested(AssociatedNotificationSchema, missing=[], many=True)
-    replacements = fields.Nested(AssociatedCertificateSchema, missing=[], many=True)
+    replaces = fields.Nested(AssociatedCertificateSchema, missing=[], many=True)
+    replacements = fields.Nested(AssociatedCertificateSchema, missing=[], many=True)  # deprecated
     roles = fields.Nested(AssociatedRoleSchema, missing=[], many=True)
 
     csr = fields.String(validate=validators.csr)
+    key_type = fields.String(validate=validate.OneOf(['RSA2048', 'RSA4096']), missing='RSA2048')
 
     notify = fields.Boolean(default=True)
+    rotation = fields.Boolean()
 
     # certificate body fields
     organizational_unit = fields.String(missing=lambda: current_app.config.get('LEMUR_DEFAULT_ORGANIZATIONAL_UNIT'))
@@ -75,18 +78,30 @@ class CertificateInputSchema(CertificateCreationSchema):
         validators.dates(data)
 
     @pre_load
-    def ensure_dates(self, data):
+    def load_data(self, data):
+        if data.get('replacements'):
+            data['replaces'] = data['replacements']  # TODO remove when field is deprecated
         return missing.convert_validity_years(data)
 
 
 class CertificateEditInputSchema(CertificateSchema):
-    notify = fields.Boolean()
     owner = fields.String()
+
+    notify = fields.Boolean()
+    rotation = fields.Boolean()
+
     destinations = fields.Nested(AssociatedDestinationSchema, missing=[], many=True)
     notifications = fields.Nested(AssociatedNotificationSchema, missing=[], many=True)
-    replacements = fields.Nested(AssociatedCertificateSchema, missing=[], many=True)
+    replaces = fields.Nested(AssociatedCertificateSchema, missing=[], many=True)
+    replacements = fields.Nested(AssociatedCertificateSchema, missing=[], many=True)  # deprecated
     roles = fields.Nested(AssociatedRoleSchema, missing=[], many=True)
 
+    @pre_load
+    def load_data(self, data):
+        if data.get('replacements'):
+            data['replaces'] = data['replacements']  # TODO remove when field is deprecated
+        return data
+
     @post_load
     def enforce_notifications(self, data):
         """
@@ -104,18 +119,31 @@ class CertificateEditInputSchema(CertificateSchema):
 class CertificateNestedOutputSchema(LemurOutputSchema):
     __envelope__ = False
     id = fields.Integer()
-    active = fields.Boolean()
+    name = fields.String()
+    owner = fields.Email()
+    creator = fields.Nested(UserNestedOutputSchema)
+    description = fields.String()
+
+    status = fields.Boolean()
+
     bits = fields.Integer()
     body = fields.String()
     chain = fields.String()
-    description = fields.String()
-    name = fields.String()
-    cn = fields.String()
-    not_after = fields.DateTime()
-    not_before = fields.DateTime()
-    owner = fields.Email()
-    status = fields.Boolean()
-    creator = fields.Nested(UserNestedOutputSchema)
+    active = fields.Boolean()
+
+    rotation = fields.Boolean()
+    notify = fields.Boolean()
+
+    # Note aliasing  is the first step in deprecating these fields.
+    cn = fields.String()  # deprecated
+    common_name = fields.String(attribute='cn')
+
+    not_after = fields.DateTime()  # deprecated
+    validity_end = ArrowDateTime(attribute='not_after')
+
+    not_before = fields.DateTime()  # deprecated
+    validity_start = ArrowDateTime(attribute='not_before')
+
     issuer = fields.Nested(AuthorityNestedOutputSchema)
 
 
@@ -127,8 +155,6 @@ class CertificateCloneSchema(LemurOutputSchema):
 
 class CertificateOutputSchema(LemurOutputSchema):
     id = fields.Integer()
-    active = fields.Boolean()
-    notify = fields.Boolean()
     bits = fields.Integer()
     body = fields.String()
     chain = fields.String()
@@ -136,15 +162,33 @@ class CertificateOutputSchema(LemurOutputSchema):
     description = fields.String()
     issuer = fields.String()
     name = fields.String()
+
+    rotation = fields.Boolean()
+
+    # Note aliasing  is the first step in deprecating these fields.
+    notify = fields.Boolean()
+    active = fields.Boolean(attribute='notify')
+
     cn = fields.String()
+    common_name = fields.String(attribute='cn')
+
     not_after = fields.DateTime()
+    validity_end = ArrowDateTime(attribute='not_after')
+
     not_before = fields.DateTime()
+    validity_start = ArrowDateTime(attribute='not_before')
+
     owner = fields.Email()
     san = fields.Boolean()
     serial = fields.String()
     signing_algorithm = fields.String()
+
     status = fields.Boolean()
     user = fields.Nested(UserNestedOutputSchema)
+
+    extensions = fields.Nested(ExtensionSchema)
+
+    # associated objects
     domains = fields.Nested(DomainNestedOutputSchema, many=True)
     destinations = fields.Nested(DestinationNestedOutputSchema, many=True)
     notifications = fields.Nested(NotificationNestedOutputSchema, many=True)
@@ -152,6 +196,7 @@ class CertificateOutputSchema(LemurOutputSchema):
     authority = fields.Nested(AuthorityNestedOutputSchema)
     roles = fields.Nested(RoleNestedOutputSchema, many=True)
     endpoints = fields.Nested(EndpointNestedOutputSchema, many=True, missing=[])
+    replaced_by = fields.Nested(CertificateNestedOutputSchema, many=True, attribute='replaced')
 
 
 class CertificateUploadInputSchema(CertificateCreationSchema):
@@ -160,11 +205,11 @@ class CertificateUploadInputSchema(CertificateCreationSchema):
 
     private_key = fields.String(validate=validators.private_key)
     body = fields.String(required=True, validate=validators.public_certificate)
-    chain = fields.String(validate=validators.public_certificate)  # TODO this could be multiple certificates
+    chain = fields.String(validate=validators.public_certificate, missing=None, allow_none=True)  # TODO this could be multiple certificates
 
     destinations = fields.Nested(AssociatedDestinationSchema, missing=[], many=True)
     notifications = fields.Nested(AssociatedNotificationSchema, missing=[], many=True)
-    replacements = fields.Nested(AssociatedCertificateSchema, missing=[], many=True)
+    replaces = fields.Nested(AssociatedCertificateSchema, missing=[], many=True)
     roles = fields.Nested(AssociatedRoleSchema, missing=[], many=True)
 
     @validates_schema
@@ -178,9 +223,21 @@ class CertificateExportInputSchema(LemurInputSchema):
     plugin = fields.Nested(PluginInputSchema)
 
 
+class CertificateNotificationOutputSchema(LemurOutputSchema):
+    description = fields.String()
+    issuer = fields.String()
+    name = fields.String()
+    owner = fields.Email()
+    user = fields.Nested(UserNestedOutputSchema)
+    validity_end = ArrowDateTime(attribute='not_after')
+    replaced_by = fields.Nested(CertificateNestedOutputSchema, many=True, attribute='replaced')
+    endpoints = fields.Nested(EndpointNestedOutputSchema, many=True, missing=[])
+
+
 certificate_input_schema = CertificateInputSchema()
 certificate_output_schema = CertificateOutputSchema()
 certificates_output_schema = CertificateOutputSchema(many=True)
 certificate_upload_input_schema = CertificateUploadInputSchema()
 certificate_export_input_schema = CertificateExportInputSchema()
 certificate_edit_input_schema = CertificateEditInputSchema()
+certificate_notification_output_schema = CertificateNotificationOutputSchema()

lemur/certificates/service.py

@@ -1,37 +1,40 @@
 """
-.. module: service
+.. module: lemur.certificate.service
     :platform: Unix
     :copyright: (c) 2015 by Netflix Inc., see AUTHORS for more
     :license: Apache, see LICENSE for more details.
 .. moduleauthor:: Kevin Glisson <kglisson@netflix.com>
 """
 import arrow
+from datetime import timedelta
 
-from sqlalchemy import func, or_
-from flask import g, current_app
-
-from lemur import database
-from lemur.extensions import metrics
-from lemur.plugins.base import plugins
-from lemur.certificates.models import Certificate
-
-from lemur.destinations.models import Destination
-from lemur.notifications.models import Notification
-from lemur.authorities.models import Authority
-from lemur.domains.models import Domain
-
-from lemur.roles.models import Role
-from lemur.roles import service as role_service
+from flask import current_app
+from sqlalchemy import func, or_, not_, cast, Boolean, Integer
 
 from cryptography import x509
 from cryptography.hazmat.backends import default_backend
 from cryptography.hazmat.primitives import hashes, serialization
-from cryptography.hazmat.primitives.asymmetric import rsa
+
+from lemur import database
+from lemur.extensions import metrics
+from lemur.plugins.base import plugins
+from lemur.common.utils import generate_private_key
+
+from lemur.roles.models import Role
+from lemur.domains.models import Domain
+from lemur.authorities.models import Authority
+from lemur.destinations.models import Destination
+from lemur.certificates.models import Certificate
+from lemur.notifications.models import Notification
+
+from lemur.certificates.schemas import CertificateOutputSchema, CertificateInputSchema
+
+from lemur.roles import service as role_service
 
 
 def get(cert_id):
     """
-    Retrieves certificate by it's ID.
+    Retrieves certificate by its ID.
 
     :param cert_id:
     :return:
@@ -41,7 +44,7 @@ def get(cert_id):
 
 def get_by_name(name):
     """
-    Retrieves certificate by it's Name.
+    Retrieves certificate by its Name.
 
     :param name:
     :return:
@@ -67,14 +70,35 @@ def get_all_certs():
     return Certificate.query.all()
 
 
-def get_by_source(source_label):
+def get_all_pending_cleaning(source):
     """
-    Retrieves all certificates from a given source.
+    Retrieves all certificates that are available for cleaning.
 
-    :param source_label:
+    :param source:
     :return:
     """
-    return Certificate.query.filter(Certificate.sources.any(label=source_label))
+    return Certificate.query.filter(Certificate.sources.any(id=source.id))\
+                            .filter(not_(Certificate.endpoints.any())).all()
+
+
+def get_all_pending_reissue():
+    """
+    Retrieves all certificates that need to be rotated.
+
+    Must be X days from expiration, uses `LEMUR_DEFAULT_ROTATION_INTERVAL`
+    to determine how many days from expiration the certificate must be
+    for rotation to be pending.
+
+    :return:
+    """
+    now = arrow.utcnow()
+    interval = current_app.config.get('LEMUR_DEFAULT_ROTATION_INTERVAL', 30)
+    end = now + timedelta(days=interval)
+
+    return Certificate.query.filter(Certificate.rotation == True)\
+        .filter(Certificate.endpoints.any())\
+        .filter(not_(Certificate.replaced.any()))\
+        .filter(Certificate.not_after <= end.format('YYYY-MM-DD')).all()  # noqa
 
 
 def find_duplicates(cert):
@@ -86,7 +110,10 @@ def find_duplicates(cert):
     :param cert:
     :return:
     """
-    return Certificate.query.filter_by(body=cert['body'].strip(), chain=cert['chain'].strip()).all()
+    if cert['chain']:
+        return Certificate.query.filter_by(body=cert['body'].strip(), chain=cert['chain'].strip()).all()
+    else:
+        return Certificate.query.filter_by(body=cert['body'].strip(), chain=None).all()
 
 
 def export(cert, export_plugin):
@@ -102,26 +129,16 @@ def export(cert, export_plugin):
     return plugin.export(cert.body, cert.chain, cert.private_key, export_plugin['pluginOptions'])
 
 
-def update(cert_id, owner, description, notify, destinations, notifications, replaces, roles):
+def update(cert_id, **kwargs):
     """
     Updates a certificate
     :param cert_id:
-    :param owner:
-    :param description:
-    :param notify:
-    :param destinations:
-    :param notifications:
-    :param replaces:
     :return:
     """
     cert = get(cert_id)
-    cert.notify = notify
-    cert.description = description
-    cert.destinations = destinations
-    cert.notifications = notifications
-    cert.roles = roles
-    cert.replaces = replaces
-    cert.owner = owner
+
+    for key, value in kwargs.items():
+        setattr(cert, key, value)
 
     return database.update(cert)
 
@@ -129,12 +146,18 @@ def update(cert_id, owner, description, notify, destinations, notifications, rep
 def create_certificate_roles(**kwargs):
     # create an role for the owner and assign it
     owner_role = role_service.get_by_name(kwargs['owner'])
+
     if not owner_role:
         owner_role = role_service.create(
             kwargs['owner'],
             description="Auto generated role based on owner: {0}".format(kwargs['owner'])
         )
 
+    # ensure that the authority's owner is also associated with the certificate
+    if kwargs.get('authority'):
+        authority_owner_role = role_service.get_by_name(kwargs['authority'].owner)
+        return [owner_role, authority_owner_role]
+
     return [owner_role]
 
 
@@ -198,11 +221,7 @@ def upload(**kwargs):
 
     cert = database.create(cert)
 
-    try:
-        g.user.certificates.append(cert)
-    except AttributeError:
-        current_app.logger.debug("No user to associate uploaded certificate to.")
-
+    kwargs['creator'].certificates.append(cert)
     return database.update(cert)
 
 
@@ -210,7 +229,6 @@ def create(**kwargs):
     """
     Creates a new certificate.
     """
-    kwargs['creator'] = g.user.email
     cert_body, private_key, cert_chain = mint(**kwargs)
     kwargs['body'] = cert_body
     kwargs['private_key'] = private_key
@@ -225,7 +243,7 @@ def create(**kwargs):
 
     cert = Certificate(**kwargs)
 
-    g.user.certificates.append(cert)
+    kwargs['creator'].certificates.append(cert)
     cert.authority = kwargs['authority']
     database.commit()
 
@@ -270,6 +288,8 @@ def render(args):
 
         elif 'destination' in terms:
             query = query.filter(Certificate.destinations.any(Destination.id == terms[1]))
+        elif 'notify' in filt:
+            query = query.filter(Certificate.notify == cast(terms[1], Boolean))
         elif 'active' in filt:
             query = query.filter(Certificate.active == terms[1])
         elif 'cn' in terms:
@@ -279,14 +299,16 @@ def render(args):
                     Certificate.domains.any(Domain.name.ilike('%{0}%'.format(terms[1])))
                 )
             )
+        elif 'id' in terms:
+            query = query.filter(Certificate.id == cast(terms[1], Integer))
         else:
             query = database.filter(query, Certificate, terms)
 
     if show:
-        sub_query = database.session_query(Role.name).filter(Role.user_id == g.user.id).subquery()
+        sub_query = database.session_query(Role.name).filter(Role.user_id == args['user'].id).subquery()
         query = query.filter(
             or_(
-                Certificate.user_id == g.user.id,
+                Certificate.user_id == args['user'].id,
                 Certificate.owner.in_(sub_query)
             )
         )
@@ -312,83 +334,45 @@ def create_csr(**csr_config):
 
     :param csr_config:
     """
-    private_key = rsa.generate_private_key(
-        public_exponent=65537,
-        key_size=2048,
-        backend=default_backend()
-    )
+    private_key = generate_private_key(csr_config.get('key_type'))
 
-    # TODO When we figure out a better way to validate these options they should be parsed as str
     builder = x509.CertificateSigningRequestBuilder()
-    builder = builder.subject_name(x509.Name([
-        x509.NameAttribute(x509.OID_COMMON_NAME, csr_config['common_name']),
-        x509.NameAttribute(x509.OID_ORGANIZATION_NAME, csr_config['organization']),
-        x509.NameAttribute(x509.OID_ORGANIZATIONAL_UNIT_NAME, csr_config['organizational_unit']),
-        x509.NameAttribute(x509.OID_COUNTRY_NAME, csr_config['country']),
-        x509.NameAttribute(x509.OID_STATE_OR_PROVINCE_NAME, csr_config['state']),
-        x509.NameAttribute(x509.OID_LOCALITY_NAME, csr_config['location']),
-        x509.NameAttribute(x509.OID_EMAIL_ADDRESS, csr_config['owner'])
-    ]))
+    name_list = [x509.NameAttribute(x509.OID_COMMON_NAME, csr_config['common_name']),
+                 x509.NameAttribute(x509.OID_EMAIL_ADDRESS, csr_config['owner'])]
+    if 'organization' in csr_config and csr_config['organization'].strip():
+        name_list.append(x509.NameAttribute(x509.OID_ORGANIZATION_NAME, csr_config['organization']))
+    if 'organizational_unit' in csr_config and csr_config['organizational_unit'].strip():
+        name_list.append(x509.NameAttribute(x509.OID_ORGANIZATIONAL_UNIT_NAME, csr_config['organizational_unit']))
+    if 'country' in csr_config and csr_config['country'].strip():
+        name_list.append(x509.NameAttribute(x509.OID_COUNTRY_NAME, csr_config['country']))
+    if 'state' in csr_config and csr_config['state'].strip():
+        name_list.append(x509.NameAttribute(x509.OID_STATE_OR_PROVINCE_NAME, csr_config['state']))
+    if 'location' in csr_config and csr_config['location'].strip():
+        name_list.append(x509.NameAttribute(x509.OID_LOCALITY_NAME, csr_config['location']))
+    builder = builder.subject_name(x509.Name(name_list))
 
-    builder = builder.add_extension(
-        x509.BasicConstraints(ca=False, path_length=None), critical=True,
-    )
+    extensions = csr_config.get('extensions', {})
+    critical_extensions = ['basic_constraints', 'sub_alt_names', 'key_usage']
+    noncritical_extensions = ['extended_key_usage']
+    for k, v in extensions.items():
+        if v:
+            if k in critical_extensions:
+                current_app.logger.debug('Adding Critical Extension: {0} {1}'.format(k, v))
+                if k == 'sub_alt_names':
+                    builder = builder.add_extension(v['names'], critical=True)
+                else:
+                    builder = builder.add_extension(v, critical=True)
 
-    if csr_config.get('extensions'):
-        for k, v in csr_config.get('extensions', {}).items():
-            if k == 'sub_alt_names':
-                # map types to their x509 objects
-                general_names = []
-                for name in v['names']:
-                    if name['name_type'] == 'DNSName':
-                        general_names.append(x509.DNSName(name['value']))
+            if k in noncritical_extensions:
+                current_app.logger.debug('Adding Extension: {0} {1}'.format(k, v))
+                builder = builder.add_extension(v, critical=False)
 
-                builder = builder.add_extension(
-                    x509.SubjectAlternativeName(general_names), critical=True
-                )
-
-    # TODO support more CSR options, none of the authority plugins currently support these options
-    #    builder.add_extension(
-    #        x509.KeyUsage(
-    #            digital_signature=digital_signature,
-    #            content_commitment=content_commitment,
-    #            key_encipherment=key_enipherment,
-    #            data_encipherment=data_encipherment,
-    #            key_agreement=key_agreement,
-    #            key_cert_sign=key_cert_sign,
-    #            crl_sign=crl_sign,
-    #            encipher_only=enchipher_only,
-    #            decipher_only=decipher_only
-    #        ), critical=True
-    #    )
-    #
-    #    # we must maintain our own list of OIDs here
-    #    builder.add_extension(
-    #        x509.ExtendedKeyUsage(
-    #            server_authentication=server_authentication,
-    #            email=
-    #        )
-    #    )
-    #
-    #    builder.add_extension(
-    #        x509.AuthorityInformationAccess()
-    #    )
-    #
-    #    builder.add_extension(
-    #        x509.AuthorityKeyIdentifier()
-    #    )
-    #
-    #    builder.add_extension(
-    #        x509.SubjectKeyIdentifier()
-    #    )
-    #
-    #    builder.add_extension(
-    #        x509.CRLDistributionPoints()
-    #    )
-    #
-    #    builder.add_extension(
-    #        x509.ObjectIdentifier(oid)
-    #    )
+    ski = extensions.get('subject_key_identifier', {})
+    if ski.get('include_ski', False):
+        builder = builder.add_extension(
+            x509.SubjectKeyIdentifier.from_public_key(private_key.public_key()),
+            critical=False
+        )
 
     request = builder.sign(
         private_key, hashes.SHA256(), default_backend()
@@ -406,7 +390,7 @@ def create_csr(**csr_config):
 
     csr = request.public_bytes(
         encoding=serialization.Encoding.PEM
-    )
+    ).decode('utf-8')
 
     return csr, private_key
 
@@ -470,13 +454,12 @@ def calculate_reissue_range(start, end):
     """
     span = end - start
 
-    new_start = arrow.utcnow().date()
+    new_start = arrow.utcnow()
     new_end = new_start + span
 
-    return new_start, new_end
+    return new_start, arrow.get(new_end)
 
 
-# TODO pull the OU, O, CN, etc + other extensions.
 def get_certificate_primitives(certificate):
     """
     Retrieve key primitive from a certificate such that the certificate
@@ -486,22 +469,36 @@ def get_certificate_primitives(certificate):
     certificate via `create`.
     """
     start, end = calculate_reissue_range(certificate.not_before, certificate.not_after)
-    names = [{'name_type': 'DNSName', 'value': x.name} for x in certificate.domains]
+    data = CertificateInputSchema().load(CertificateOutputSchema().dump(certificate).data).data
 
-    extensions = {
-        'sub_alt_names': {
-            'names': names
-        }
-    }
+    # we can't quite tell if we are using a custom name, as this is an automated process (typically)
+    # we will rely on the Lemur generated name
+    data.pop('name', None)
 
-    return dict(
-        authority=certificate.authority,
-        common_name=certificate.cn,
-        description=certificate.description,
-        validity_start=start,
-        validity_end=end,
-        destinations=certificate.destinations,
-        roles=certificate.roles,
-        extensions=extensions,
-        owner=certificate.owner
-    )
+    data['validity_start'] = start
+    data['validity_end'] = end
+    return data
+
+
+def reissue_certificate(certificate, replace=None, user=None):
+    """
+    Reissue certificate with the same properties of the given certificate.
+    :param certificate:
+    :param replace:
+    :param user:
+    :return:
+    """
+    primitives = get_certificate_primitives(certificate)
+
+    if not user:
+        primitives['creator'] = certificate.user
+
+    else:
+        primitives['creator'] = user
+
+    if replace:
+        primitives['replaces'] = [certificate]
+
+    new_cert = create(**primitives)
+
+    return new_cert

lemur/certificates/verify.py

@@ -7,12 +7,12 @@
 """
 import requests
 import subprocess
-from OpenSSL import crypto
+from requests.exceptions import ConnectionError
 from cryptography import x509
 from cryptography.hazmat.backends import default_backend
 
-from flask import current_app
 from lemur.utils import mktempfile
+from lemur.common.utils import parse_certificate
 
 
 def ocsp_verify(cert_path, issuer_chain_path):
@@ -33,13 +33,16 @@ def ocsp_verify(cert_path, issuer_chain_path):
                            '-cert', cert_path, "-url", url.strip()], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
 
     message, err = p2.communicate()
-    if 'error' in message or 'Error' in message:
+
+    p_message = message.decode('utf-8')
+
+    if 'error' in p_message or 'Error' in p_message:
         raise Exception("Got error when parsing OCSP url")
 
-    elif 'revoked' in message:
+    elif 'revoked' in p_message:
         return
 
-    elif 'good' not in message:
+    elif 'good' not in p_message:
         raise Exception("Did not receive a valid response")
 
     return True
@@ -54,17 +57,27 @@ def crl_verify(cert_path):
     :raise Exception: If certificate does not have CRL
     """
     with open(cert_path, 'rt') as c:
-        cert = x509.load_pem_x509_certificate(c.read(), default_backend())
+        cert = parse_certificate(c.read())
 
     distribution_points = cert.extensions.get_extension_for_oid(x509.OID_CRL_DISTRIBUTION_POINTS).value
+
     for p in distribution_points:
         point = p.full_name[0].value
-        response = requests.get(point)
-        crl = crypto.load_crl(crypto.FILETYPE_ASN1, response.content)  # TODO this should be switched to cryptography when support exists
-        revoked = crl.get_revoked()
-        for r in revoked:
-            if cert.serial == r.get_serial():
+
+        try:
+            response = requests.get(point)
+
+            if response.status_code != 200:
+                raise Exception("Unable to retrieve CRL: {0}".format(point))
+        except ConnectionError:
+            raise Exception("Unable to retrieve CRL: {0}".format(point))
+
+        crl = x509.load_der_x509_crl(response.content, backend=default_backend())
+
+        for r in crl:
+            if cert.serial == r.serial_number:
                 return
+
     return True
 
 
@@ -81,13 +94,10 @@ def verify(cert_path, issuer_chain_path):
     try:
         return ocsp_verify(cert_path, issuer_chain_path)
     except Exception as e:
-        current_app.logger.debug("Could not use OCSP: {0}".format(e))
         try:
             return crl_verify(cert_path)
         except Exception as e:
-            current_app.logger.debug("Could not use CRL: {0}".format(e))
             raise Exception("Failed to verify")
-        raise Exception("Failed to verify")
 
 
 def verify_string(cert_string, issuer_string):

lemur/certificates/views.py

@@ -8,8 +8,8 @@
 import base64
 from builtins import str
 
-from flask import Blueprint, make_response, jsonify
-from flask.ext.restful import reqparse, Api
+from flask import Blueprint, make_response, jsonify, g
+from flask_restful import reqparse, Api
 
 from lemur.common.schema import validate_schema
 from lemur.common.utils import paginated_parser
@@ -22,6 +22,7 @@ from lemur.certificates.schemas import certificate_input_schema, certificate_out
     certificate_upload_input_schema, certificates_output_schema, certificate_export_input_schema, certificate_edit_input_schema
 
 from lemur.roles import service as role_service
+from lemur.logs import service as log_service
 
 
 mod = Blueprint('certificates', __name__)
@@ -98,6 +99,7 @@ class CertificatesList(AuthenticatedResource):
                         "name": "*.test.example.net"
                     }],
                     "replaces": [],
+                    "replaced": [],
                     "name": "WILDCARD.test.example.net-SymantecCorporation-20160603-20180112",
                     "roles": [{
                         "id": 464,
@@ -110,7 +112,7 @@ class CertificatesList(AuthenticatedResource):
               }
 
            :query sortBy: field to sort on
-           :query sortDir: acs or desc
+           :query sortDir: asc or desc
            :query page: int. default is 1
            :query filter: key value pair format is k;v
            :query count: count number. default is 10
@@ -129,6 +131,7 @@ class CertificatesList(AuthenticatedResource):
         parser.add_argument('show', type=str, location='args')
 
         args = parser.parse_args()
+        args['user'] = g.user
         return service.render(args)
 
     @validate_schema(certificate_input_schema, certificate_output_schema)
@@ -265,6 +268,7 @@ class CertificatesList(AuthenticatedResource):
         authority_permission = AuthorityPermission(data['authority'].id, roles)
 
         if authority_permission.can():
+            data['creator'] = g.user
             return service.create(**data)
 
         return dict(message="You are not authorized to use {0}".format(data['authority'].name)), 403
@@ -293,10 +297,10 @@ class CertificatesUpload(AuthenticatedResource):
               Accept: application/json, text/javascript
 
               {
-                 "owner": "joe@exmaple.com",
-                 "publicCert": "---Begin Public...",
-                 "intermediateCert": "---Begin Public...",
-                 "privateKey": "---Begin Private..."
+                 "owner": "joe@example.com",
+                 "publicCert": "-----BEGIN CERTIFICATE-----...",
+                 "intermediateCert": "-----BEGIN CERTIFICATE-----...",
+                 "privateKey": "-----BEGIN RSA PRIVATE KEY-----..."
                  "destinations": [],
                  "notifications": [],
                  "replacements": [],
@@ -369,6 +373,7 @@ class CertificatesUpload(AuthenticatedResource):
            :statuscode 200: no error
 
         """
+        data['creator'] = g.user
         if data.get('destinations'):
             if data.get('private_key'):
                 return service.upload(**data)
@@ -423,7 +428,7 @@ class CertificatePrivateKey(AuthenticatedResource):
               Content-Type: text/javascript
 
               {
-                 "key": "----Begin ...",
+                 "key": "-----BEGIN ...",
               }
 
            :reqheader Authorization: OAuth token to authenticate
@@ -434,16 +439,19 @@ class CertificatePrivateKey(AuthenticatedResource):
         if not cert:
             return dict(message="Cannot find specified certificate"), 404
 
-        owner_role = role_service.get_by_name(cert.owner)
-        permission = CertificatePermission(cert.id, owner_role, [x.name for x in cert.roles])
+        # allow creators
+        if g.current_user != cert.user:
+            owner_role = role_service.get_by_name(cert.owner)
+            permission = CertificatePermission(owner_role, [x.name for x in cert.roles])
 
-        if permission.can():
-            response = make_response(jsonify(key=cert.private_key), 200)
-            response.headers['cache-control'] = 'private, max-age=0, no-cache, no-store'
-            response.headers['pragma'] = 'no-cache'
-            return response
+            if not permission.can():
+                return dict(message='You are not authorized to view this key'), 403
 
-        return dict(message='You are not authorized to view this key'), 403
+        log_service.create(g.current_user, 'key_view', certificate=cert)
+        response = make_response(jsonify(key=cert.private_key), 200)
+        response.headers['cache-control'] = 'private, max-age=0, no-cache, no-store'
+        response.headers['pragma'] = 'no-cache'
+        return response
 
 
 class Certificates(AuthenticatedResource):
@@ -513,6 +521,7 @@ class Certificates(AuthenticatedResource):
                     "name": "*.test.example.net"
                 }],
                 "replaces": [],
+                "replaced": [],
                 "name": "WILDCARD.test.example.net-SymantecCorporation-20160603-20180112",
                 "roles": [{
                     "id": 464,
@@ -615,27 +624,27 @@ class Certificates(AuthenticatedResource):
         """
         cert = service.get(certificate_id)
 
-        owner_role = role_service.get_by_name(cert.owner)
-        permission = CertificatePermission(cert.id, owner_role, [x.name for x in cert.roles])
+        if not cert:
+            return dict(message="Cannot find specified certificate"), 404
 
-        if permission.can():
-            for destination in data['destinations']:
-                if destination.plugin.requires_key:
-                    if not cert.private_key:
-                        return dict('Unable to add destination: {0}. Certificate does not have required private key.'.format(destination.label))
+        # allow creators
+        if g.current_user != cert.user:
+            owner_role = role_service.get_by_name(cert.owner)
+            permission = CertificatePermission(owner_role, [x.name for x in cert.roles])
 
-            return service.update(
-                certificate_id,
-                data['owner'],
-                data['description'],
-                data['notify'],
-                data['destinations'],
-                data['notifications'],
-                data['replacements'],
-                data['roles']
-            )
+            if not permission.can():
+                return dict(message='You are not authorized to update this certificate'), 403
 
-        return dict(message='You are not authorized to update this certificate'), 403
+        for destination in data['destinations']:
+            if destination.plugin.requires_key:
+                if not cert.private_key:
+                    return dict(
+                        message='Unable to add destination: {0}. Certificate does not have required private key.'.format(
+                            destination.label
+                        )
+                    ), 400
+
+        return service.update(certificate_id, **data)
 
 
 class NotificationCertificatesList(AuthenticatedResource):
@@ -708,6 +717,7 @@ class NotificationCertificatesList(AuthenticatedResource):
                         "name": "*.test.example.net"
                     }],
                     "replaces": [],
+                    "replaced": [],
                     "name": "WILDCARD.test.example.net-SymantecCorporation-20160603-20180112",
                     "roles": [{
                         "id": 464,
@@ -720,7 +730,7 @@ class NotificationCertificatesList(AuthenticatedResource):
               }
 
            :query sortBy: field to sort on
-           :query sortDir: acs or desc
+           :query sortDir: asc or desc
            :query page: int default is 1
            :query filter: key value pair format is k;v
            :query count: count number default is 10
@@ -740,6 +750,7 @@ class NotificationCertificatesList(AuthenticatedResource):
 
         args = parser.parse_args()
         args['notification_id'] = notification_id
+        args['user'] = g.current_user
         return service.render(args)
 
 
@@ -811,6 +822,7 @@ class CertificatesReplacementsList(AuthenticatedResource):
                         "name": "*.test.example.net"
                     }],
                     "replaces": [],
+                    "replaced": [],
                     "name": "WILDCARD.test.example.net-SymantecCorporation-20160603-20180112",
                     "roles": [{
                         "id": 464,
@@ -904,45 +916,41 @@ class CertificateExport(AuthenticatedResource):
         """
         cert = service.get(certificate_id)
 
-        owner_role = role_service.get_by_name(cert.owner)
-        permission = CertificatePermission(cert.id, owner_role, [x.name for x in cert.roles])
+        if not cert:
+            return dict(message="Cannot find specified certificate"), 404
 
-        options = data['plugin']['plugin_options']
         plugin = data['plugin']['plugin_object']
 
         if plugin.requires_key:
-            if cert.private_key:
-                if permission.can():
-                    extension, passphrase, data = plugin.export(cert.body, cert.chain, cert.private_key, options)
-                else:
-                    return dict(message='You are not authorized to export this certificate.'), 403
+            if not cert.private_key:
+                return dict(
+                    message='Unable to export certificate, plugin: {0} requires a private key but no key was found.'.format(
+                        plugin.slug)), 400
+
             else:
-                return dict(message='Unable to export certificate, plugin: {0} requires a private key but no key was found.'.format(plugin.slug))
-        else:
-            extension, passphrase, data = plugin.export(cert.body, cert.chain, cert.private_key, options)
+                # allow creators
+                if g.current_user != cert.user:
+                    owner_role = role_service.get_by_name(cert.owner)
+                    permission = CertificatePermission(owner_role, [x.name for x in cert.roles])
+
+                    if not permission.can():
+                        return dict(message='You are not authorized to export this certificate.'), 403
+
+        options = data['plugin']['plugin_options']
+
+        log_service.create(g.current_user, 'key_view', certificate=cert)
+        extension, passphrase, data = plugin.export(cert.body, cert.chain, cert.private_key, options)
 
         # we take a hit in message size when b64 encoding
         return dict(extension=extension, passphrase=passphrase, data=base64.b64encode(data).decode('utf-8'))
 
 
-class CertificateClone(AuthenticatedResource):
-    def __init__(self):
-        self.reqparse = reqparse.RequestParser()
-        super(CertificateExport, self).__init__()
-
-    @validate_schema(None, certificate_output_schema)
-    def get(self, certificate_id):
-
-        pass
-
-
 api.add_resource(CertificatesList, '/certificates', endpoint='certificates')
 api.add_resource(Certificates, '/certificates/<int:certificate_id>', endpoint='certificate')
 api.add_resource(CertificatesStats, '/certificates/stats', endpoint='certificateStats')
 api.add_resource(CertificatesUpload, '/certificates/upload', endpoint='certificateUpload')
 api.add_resource(CertificatePrivateKey, '/certificates/<int:certificate_id>/key', endpoint='privateKeyCertificates')
 api.add_resource(CertificateExport, '/certificates/<int:certificate_id>/export', endpoint='exportCertificate')
-api.add_resource(CertificateClone, '/certificates/<int:certificate_id>/clone', endpoint='cloneCertificate')
 api.add_resource(NotificationCertificatesList, '/notifications/<int:notification_id>/certificates',
                  endpoint='notificationCertificates')
 api.add_resource(CertificatesReplacementsList, '/certificates/<int:certificate_id>/replacements',

lemur/common/defaults.py

@@ -53,9 +53,82 @@ def common_name(cert):
     :param cert:
     :return: Common name or None
     """
-    return cert.subject.get_attributes_for_oid(
-        x509.OID_COMMON_NAME
-    )[0].value.strip()
+    try:
+        return cert.subject.get_attributes_for_oid(
+            x509.OID_COMMON_NAME
+        )[0].value.strip()
+    except Exception as e:
+        current_app.logger.error("Unable to get common name! {0}".format(e))
+
+
+def organization(cert):
+    """
+    Attempt to get the organization name from a given certificate.
+    :param cert:
+    :return:
+    """
+    try:
+        return cert.subject.get_attributes_for_oid(
+            x509.OID_ORGANIZATION_NAME
+        )[0].value.strip()
+    except Exception as e:
+        current_app.logger.error("Unable to get organization! {0}".format(e))
+
+
+def organizational_unit(cert):
+    """
+    Attempt to get the organization unit from a given certificate.
+    :param cert:
+    :return:
+    """
+    try:
+        return cert.subject.get_attributes_for_oid(
+            x509.OID_ORGANIZATIONAL_UNIT_NAME
+        )[0].value.strip()
+    except Exception as e:
+        current_app.logger.error("Unable to get organizational unit! {0}".format(e))
+
+
+def country(cert):
+    """
+    Attempt to get the country from a given certificate.
+    :param cert:
+    :return:
+    """
+    try:
+        return cert.subject.get_attributes_for_oid(
+            x509.OID_COUNTRY_NAME
+        )[0].value.strip()
+    except Exception as e:
+        current_app.logger.error("Unable to get country! {0}".format(e))
+
+
+def state(cert):
+    """
+    Attempt to get the from a given certificate.
+    :param cert:
+    :return:
+    """
+    try:
+        return cert.subject.get_attributes_for_oid(
+            x509.OID_STATE_OR_PROVINCE_NAME
+        )[0].value.strip()
+    except Exception as e:
+        current_app.logger.error("Unable to get state! {0}".format(e))
+
+
+def location(cert):
+    """
+    Attempt to get the location name from a given certificate.
+    :param cert:
+    :return:
+    """
+    try:
+        return cert.subject.get_attributes_for_oid(
+            x509.OID_LOCALITY_NAME
+        )[0].value.strip()
+    except Exception as e:
+        current_app.logger.error("Unable to get location! {0}".format(e))
 
 
 def domains(cert):
@@ -74,7 +147,7 @@ def domains(cert):
         for entry in entries:
             domains.append(entry)
     except Exception as e:
-        current_app.logger.warning("Failed to get SubjectAltName: {0}".format(e))
+        pass
 
     return domains
 
@@ -131,19 +204,23 @@ def bitstrength(cert):
 
 def issuer(cert):
     """
-    Gets a sane issuer from a given certificate.
+    Gets a sane issuer name from a given certificate.
 
     :param cert:
     :return: Issuer
     """
     delchars = ''.join(c for c in map(chr, range(256)) if not c.isalnum())
     try:
-        issuer = str(cert.issuer.get_attributes_for_oid(x509.OID_ORGANIZATION_NAME)[0].value)
+        # Try organization name or fall back to CN
+        issuer = (cert.issuer.get_attributes_for_oid(x509.OID_ORGANIZATION_NAME)
+                  or cert.issuer.get_attributes_for_oid(x509.OID_COMMON_NAME))
+        issuer = str(issuer[0].value)
         for c in delchars:
             issuer = issuer.replace(c, "")
         return issuer
     except Exception as e:
         current_app.logger.error("Unable to get issuer! {0}".format(e))
+        return "Unknown"
 
 
 def not_before(cert):

lemur/common/fields.py

@@ -1,8 +1,22 @@
+"""
+.. module: lemur.common.fields
+    :platform: Unix
+    :copyright: (c) 2015 by Netflix Inc., see AUTHORS for more
+    :license: Apache, see LICENSE for more details.
+.. moduleauthor:: Kevin Glisson <kglisson@netflix.com>
+"""
 import arrow
 import warnings
+import ipaddress
+
+from flask import current_app
 from datetime import datetime as dt
-from marshmallow.fields import Field
+
+from cryptography import x509
+
 from marshmallow import utils
+from marshmallow.fields import Field
+from marshmallow.exceptions import ValidationError
 
 
 class ArrowDateTime(Field):
@@ -91,3 +105,292 @@ class ArrowDateTime(Field):
             warnings.warn('It is recommended that you install python-dateutil '
                           'for improved datetime deserialization.')
             raise self.fail('invalid')
+
+
+class KeyUsageExtension(Field):
+    """An x509.KeyUsage ExtensionType object
+
+    Dict of KeyUsage names/values are deserialized into an x509.KeyUsage object
+    and back.
+
+    :param kwargs: The same keyword arguments that :class:`Field` receives.
+
+    """
+
+    def _serialize(self, value, attr, obj):
+        return {
+            'useDigitalSignature': value.digital_signature,
+            'useNonRepudiation': value.content_commitment,
+            'useKeyEncipherment': value.key_encipherment,
+            'useDataEncipherment': value.data_encipherment,
+            'useKeyAgreement': value.key_agreement,
+            'useKeyCertSign': value.key_cert_sign,
+            'useCRLSign': value.crl_sign,
+            'useEncipherOnly': value._encipher_only,
+            'useDecipherOnly': value._decipher_only
+        }
+
+    def _deserialize(self, value, attr, data):
+        keyusages = {
+            'digital_signature': False,
+            'content_commitment': False,
+            'key_encipherment': False,
+            'data_encipherment': False,
+            'key_agreement': False,
+            'key_cert_sign': False,
+            'crl_sign': False,
+            'encipher_only': False,
+            'decipher_only': False
+        }
+
+        for k, v in value.items():
+            if k == 'useDigitalSignature':
+                keyusages['digital_signature'] = v
+
+            elif k == 'useNonRepudiation':
+                keyusages['content_commitment'] = v
+
+            elif k == 'useKeyEncipherment':
+                keyusages['key_encipherment'] = v
+
+            elif k == 'useDataEncipherment':
+                keyusages['data_encipherment'] = v
+
+            elif k == 'useKeyCertSign':
+                keyusages['key_cert_sign'] = v
+
+            elif k == 'useCRLSign':
+                keyusages['crl_sign'] = v
+
+            elif k == 'useKeyAgreement':
+                keyusages['key_agreement'] = v
+
+            elif k == 'useEncipherOnly' and v:
+                keyusages['encipher_only'] = True
+                keyusages['key_agreement'] = True
+
+            elif k == 'useDecipherOnly' and v:
+                keyusages['decipher_only'] = True
+                keyusages['key_agreement'] = True
+
+        if keyusages['encipher_only'] and keyusages['decipher_only']:
+            raise ValidationError('A certificate cannot have both Encipher Only and Decipher Only Extended Key Usages.')
+
+        return x509.KeyUsage(
+            digital_signature=keyusages['digital_signature'],
+            content_commitment=keyusages['content_commitment'],
+            key_encipherment=keyusages['key_encipherment'],
+            data_encipherment=keyusages['data_encipherment'],
+            key_agreement=keyusages['key_agreement'],
+            key_cert_sign=keyusages['key_cert_sign'],
+            crl_sign=keyusages['crl_sign'],
+            encipher_only=keyusages['encipher_only'],
+            decipher_only=keyusages['decipher_only']
+        )
+
+
+class ExtendedKeyUsageExtension(Field):
+    """An x509.ExtendedKeyUsage ExtensionType object
+
+    Dict of ExtendedKeyUsage names/values are deserialized into an x509.ExtendedKeyUsage object
+    and back.
+
+    :param kwargs: The same keyword arguments that :class:`Field` receives.
+
+    """
+
+    def _serialize(self, value, attr, obj):
+        usages = value._usages
+        usage_list = {}
+        for usage in usages:
+            if usage == x509.oid.ExtendedKeyUsageOID.CLIENT_AUTH:
+                usage_list['useClientAuthentication'] = True
+
+            elif usage == x509.oid.ExtendedKeyUsageOID.SERVER_AUTH:
+                usage_list['useServerAuthentication'] = True
+
+            elif usage == x509.oid.ExtendedKeyUsageOID.CODE_SIGNING:
+                usage_list['useCodeSigning'] = True
+
+            elif usage == x509.oid.ExtendedKeyUsageOID.EMAIL_PROTECTION:
+                usage_list['useEmailProtection'] = True
+
+            elif usage == x509.oid.ExtendedKeyUsageOID.TIME_STAMPING:
+                usage_list['useTimestamping'] = True
+
+            elif usage == x509.oid.ExtendedKeyUsageOID.OCSP_SIGNING:
+                usage_list['useOCSPSigning'] = True
+
+            elif usage.dotted_string == '1.3.6.1.5.5.7.3.14':
+                usage_list['useEapOverLAN'] = True
+
+            elif usage.dotted_string == '1.3.6.1.5.5.7.3.13':
+                usage_list['useEapOverPPP'] = True
+
+            elif usage.dotted_string == '1.3.6.1.4.1.311.20.2.2':
+                usage_list['useSmartCardLogon'] = True
+
+            else:
+                current_app.logger.warning('Unable to serialize ExtendedKeyUsage with OID: {usage}'.format(usage=usage.dotted_string))
+
+        return usage_list
+
+    def _deserialize(self, value, attr, data):
+        usage_oids = []
+        for k, v in value.items():
+            if k == 'useClientAuthentication' and v:
+                usage_oids.append(x509.oid.ExtendedKeyUsageOID.CLIENT_AUTH)
+
+            elif k == 'useServerAuthentication' and v:
+                usage_oids.append(x509.oid.ExtendedKeyUsageOID.SERVER_AUTH)
+
+            elif k == 'useCodeSigning' and v:
+                usage_oids.append(x509.oid.ExtendedKeyUsageOID.CODE_SIGNING)
+
+            elif k == 'useEmailProtection' and v:
+                usage_oids.append(x509.oid.ExtendedKeyUsageOID.EMAIL_PROTECTION)
+
+            elif k == 'useTimestamping' and v:
+                usage_oids.append(x509.oid.ExtendedKeyUsageOID.TIME_STAMPING)
+
+            elif k == 'useOCSPSigning' and v:
+                usage_oids.append(x509.oid.ExtendedKeyUsageOID.OCSP_SIGNING)
+
+            elif k == 'useEapOverLAN' and v:
+                usage_oids.append(x509.oid.ObjectIdentifier("1.3.6.1.5.5.7.3.14"))
+
+            elif k == 'useEapOverPPP' and v:
+                usage_oids.append(x509.oid.ObjectIdentifier("1.3.6.1.5.5.7.3.13"))
+
+            elif k == 'useSmartCardLogon' and v:
+                usage_oids.append(x509.oid.ObjectIdentifier("1.3.6.1.4.1.311.20.2.2"))
+
+            else:
+                current_app.logger.warning('Unable to deserialize ExtendedKeyUsage with name: {key}'.format(key=k))
+
+        return x509.ExtendedKeyUsage(usage_oids)
+
+
+class BasicConstraintsExtension(Field):
+    """An x509.BasicConstraints ExtensionType object
+
+    Dict of CA boolean and a path_length integer names/values are deserialized into an x509.BasicConstraints object
+    and back.
+
+    :param kwargs: The same keyword arguments that :class:`Field` receives.
+
+    """
+
+    def _serialize(self, value, attr, obj):
+        return {'ca': value.ca, 'path_length': value.path_length}
+
+    def _deserialize(self, value, attr, data):
+        ca = value.get('ca', False)
+        path_length = value.get('path_length', None)
+
+        if ca:
+            if not isinstance(path_length, (type(None), int)):
+                raise ValidationError('A CA certificate path_length (for BasicConstraints) must be None or an integer.')
+            return x509.BasicConstraints(ca=True, path_length=path_length)
+        else:
+            return x509.BasicConstraints(ca=False, path_length=None)
+
+
+class SubjectAlternativeNameExtension(Field):
+    """An x509.SubjectAlternativeName ExtensionType object
+
+    Dict of CA boolean and a path_length integer names/values are deserialized into an x509.BasicConstraints object
+    and back.
+
+    :param kwargs: The same keyword arguments that :class:`Field` receives.
+
+    """
+    def _serialize(self, value, attr, obj):
+        general_names = []
+        name_type = None
+
+        if value:
+            for name in value._general_names:
+                value = name.value
+
+                if isinstance(name, x509.DNSName):
+                    name_type = 'DNSName'
+
+                elif isinstance(name, x509.IPAddress):
+                    name_type = 'IPAddress'
+
+                elif isinstance(name, x509.UniformResourceIdentifier):
+                    name_type = 'uniformResourceIdentifier'
+
+                elif isinstance(name, x509.DirectoryName):
+                    name_type = 'directoryName'
+
+                elif isinstance(name, x509.RFC822Name):
+                    name_type = 'rfc822Name'
+
+                elif isinstance(name, x509.RegisteredID):
+                    name_type = 'registeredID'
+                    value = value.dotted_string
+                else:
+                    current_app.logger.warning('Unknown SubAltName type: {name}'.format(name=name))
+
+                general_names.append({'nameType': name_type, 'value': value})
+
+        return general_names
+
+    def _deserialize(self, value, attr, data):
+        general_names = []
+        for name in value:
+            if name['nameType'] == 'DNSName':
+                general_names.append(x509.DNSName(name['value']))
+
+            elif name['nameType'] == 'IPAddress':
+                general_names.append(x509.IPAddress(ipaddress.ip_address(name['value'])))
+
+            elif name['nameType'] == 'IPNetwork':
+                general_names.append(x509.IPAddress(ipaddress.ip_network(name['value'])))
+
+            elif name['nameType'] == 'uniformResourceIdentifier':
+                general_names.append(x509.UniformResourceIdentifier(name['value']))
+
+            elif name['nameType'] == 'directoryName':
+                # TODO: Need to parse a string in name['value'] like:
+                # 'CN=Common Name, O=Org Name, OU=OrgUnit Name, C=US, ST=ST, L=City/emailAddress=person@example.com'
+                # or
+                # 'CN=Common Name/O=Org Name/OU=OrgUnit Name/C=US/ST=NH/L=City/emailAddress=person@example.com'
+                # and turn it into something like:
+                # x509.Name([
+                #     x509.NameAttribute(x509.OID_COMMON_NAME, "Common Name"),
+                #     x509.NameAttribute(x509.OID_ORGANIZATION_NAME, "Org Name"),
+                #     x509.NameAttribute(x509.OID_ORGANIZATIONAL_UNIT_NAME, "OrgUnit Name"),
+                #     x509.NameAttribute(x509.OID_COUNTRY_NAME, "US"),
+                #     x509.NameAttribute(x509.OID_STATE_OR_PROVINCE_NAME, "NH"),
+                #     x509.NameAttribute(x509.OID_LOCALITY_NAME, "City"),
+                #     x509.NameAttribute(x509.OID_EMAIL_ADDRESS, "person@example.com")
+                # ]
+                # general_names.append(x509.DirectoryName(x509.Name(BLAH))))
+                pass
+
+            elif name['nameType'] == 'rfc822Name':
+                general_names.append(x509.RFC822Name(name['value']))
+
+            elif name['nameType'] == 'registeredID':
+                general_names.append(x509.RegisteredID(x509.ObjectIdentifier(name['value'])))
+
+            elif name['nameType'] == 'otherName':
+                # This has two inputs (type and value), so it doesn't fit the mold of the rest of these GeneralName entities.
+                # general_names.append(x509.OtherName(name['type'], bytes(name['value']), 'utf-8'))
+                pass
+
+            elif name['nameType'] == 'x400Address':
+                # The Python Cryptography library doesn't support x400Address types (yet?)
+                pass
+
+            elif name['nameType'] == 'EDIPartyName':
+                # The Python Cryptography library doesn't support EDIPartyName types (yet?)
+                pass
+
+            else:
+                current_app.logger.warning('Unable to deserialize SubAltName with type: {name_type}'.format(name_type=name['nameType']))
+
+        return x509.SubjectAlternativeName(general_names)

lemur/common/managers.py

@@ -8,6 +8,8 @@
 """
 from flask import current_app
 
+from lemur.exceptions import InvalidConfiguration
+
 
 # inspired by https://github.com/getsentry/sentry
 class InstanceManager(object):
@@ -58,9 +60,14 @@ class InstanceManager(object):
                     results.append(cls())
                 else:
                     results.append(cls)
+
+            except InvalidConfiguration as e:
+                current_app.logger.warning("Plugin '{0}' may not work correctly. {1}".format(class_name, e))
+
             except Exception as e:
-                current_app.logger.exception('Unable to import %s. Reason: %s', cls_path, e)
+                current_app.logger.exception("Unable to import {0}. Reason: {1}".format(cls_path, e))
                 continue
+
         self.cache = results
 
         return results

lemur/common/schema.py

@@ -13,7 +13,7 @@ from flask import request, current_app
 from sqlalchemy.orm.collections import InstrumentedList
 
 from inflection import camelize, underscore
-from marshmallow import Schema, post_dump, pre_load, pre_dump
+from marshmallow import Schema, post_dump, pre_load
 
 
 class LemurSchema(Schema):
@@ -68,10 +68,9 @@ class LemurOutputSchema(LemurSchema):
             data = self.unwrap_envelope(data, many)
         return self.under(data, many=many)
 
-    @pre_dump(pass_many=True)
     def unwrap_envelope(self, data, many):
         if many:
-            if data:
+            if data['items']:
                 if isinstance(data, InstrumentedList) or isinstance(data, list):
                     self.context['total'] = len(data)
                     return data
@@ -115,6 +114,29 @@ def wrap_errors(messages):
     return errors
 
 
+def unwrap_pagination(data, output_schema):
+    if not output_schema:
+        return data
+
+    if isinstance(data, dict):
+        if 'total' in data.keys():
+            if data.get('total') == 0:
+                return data
+
+            marshaled_data = {'total': data['total']}
+            marshaled_data['items'] = output_schema.dump(data['items'], many=True).data
+            return marshaled_data
+
+        return output_schema.dump(data).data
+
+    elif isinstance(data, list):
+        marshaled_data = {'total': len(data)}
+        marshaled_data['items'] = output_schema.dump(data, many=True).data
+        return marshaled_data
+
+    return output_schema.dump(data).data
+
+
 def validate_schema(input_schema, output_schema):
     def decorator(f):
         @wraps(f)
@@ -144,10 +166,7 @@ def validate_schema(input_schema, output_schema):
             if not resp:
                 return dict(message="No data found"), 404
 
-            if output_schema:
-                data = output_schema.dump(resp)
-                return data.data, 200
-            return resp, 200
+            return unwrap_pagination(resp, output_schema), 200
 
         return decorated_function
     return decorator

lemur/common/utils.py

@@ -6,15 +6,19 @@
 
 .. moduleauthor:: Kevin Glisson <kglisson@netflix.com>
 """
-import six
-import sys
 import string
 import random
 
+import sqlalchemy
+from sqlalchemy import and_, func
+
 from cryptography import x509
 from cryptography.hazmat.backends import default_backend
+from cryptography.hazmat.primitives.asymmetric import rsa
 
-from flask.ext.restful.reqparse import RequestParser
+from flask_restful.reqparse import RequestParser
+
+from lemur.exceptions import InvalidConfiguration
 
 paginated_parser = RequestParser()
 
@@ -37,15 +41,44 @@ def get_psuedo_random_string():
 
 
 def parse_certificate(body):
-    if sys.version_info[0] <= 2:
-        return x509.load_pem_x509_certificate(bytes(body), default_backend())
+    """
+    Helper function that parses a PEM certificate.
 
-    if isinstance(body, six.string_types):
+    :param body:
+    :return:
+    """
+    if isinstance(body, str):
         body = body.encode('utf-8')
 
     return x509.load_pem_x509_certificate(body, default_backend())
 
 
+def generate_private_key(key_type):
+    """
+    Generates a new private key based on key_type.
+
+    Valid key types: RSA2048, RSA4096
+
+    :param key_type:
+    :return:
+    """
+    valid_key_types = ['RSA2048', 'RSA4096']
+
+    if key_type not in valid_key_types:
+        raise Exception("Invalid key type: {key_type}. Supported key types: {choices}".format(
+            key_type=key_type,
+            choices=",".join(valid_key_types)
+        ))
+
+    if 'RSA' in key_type:
+        key_size = int(key_type[3:])
+        return rsa.generate_private_key(
+            public_exponent=65537,
+            key_size=key_size,
+            backend=default_backend()
+        )
+
+
 def is_weekend(date):
     """
     Determines if a given date is on a weekend.
@@ -55,3 +88,69 @@ def is_weekend(date):
     """
     if date.weekday() > 5:
         return True
+
+
+def validate_conf(app, required_vars):
+    """
+    Ensures that the given fields are set in the applications conf.
+
+    :param app:
+    :param required_vars: list
+    """
+    for var in required_vars:
+        if not app.config.get(var):
+            raise InvalidConfiguration("Required variable '{var}' is not set in Lemur's conf.".format(var=var))
+
+
+# https://bitbucket.org/zzzeek/sqlalchemy/wiki/UsageRecipes/WindowedRangeQuery
+def column_windows(session, column, windowsize):
+    """Return a series of WHERE clauses against
+    a given column that break it into windows.
+
+    Result is an iterable of tuples, consisting of
+    ((start, end), whereclause), where (start, end) are the ids.
+
+    Requires a database that supports window functions,
+    i.e. Postgresql, SQL Server, Oracle.
+
+    Enhance this yourself !  Add a "where" argument
+    so that windows of just a subset of rows can
+    be computed.
+
+    """
+    def int_for_range(start_id, end_id):
+        if end_id:
+            return and_(
+                column >= start_id,
+                column < end_id
+            )
+        else:
+            return column >= start_id
+
+    q = session.query(
+        column,
+        func.row_number().over(order_by=column).label('rownum')
+    ).from_self(column)
+
+    if windowsize > 1:
+        q = q.filter(sqlalchemy.text("rownum %% %d=1" % windowsize))
+
+    intervals = [id for id, in q]
+
+    while intervals:
+        start = intervals.pop(0)
+        if intervals:
+            end = intervals[0]
+        else:
+            end = None
+        yield int_for_range(start, end)
+
+
+def windowed_query(q, column, windowsize):
+    """"Break a Query into windows on a given column."""
+
+    for whereclause in column_windows(
+            q.session,
+            column, windowsize):
+        for row in q.filter(whereclause).order_by(column):
+            yield row

lemur/common/validators.py

@@ -20,7 +20,8 @@ def public_certificate(body):
     """
     try:
         parse_certificate(body)
-    except Exception:
+    except Exception as e:
+        current_app.logger.exception(e)
         raise ValidationError('Public certificate presented is not valid.')
 
 
@@ -88,7 +89,7 @@ def csr(data):
     :return:
     """
     try:
-        x509.load_pem_x509_csr(bytes(data), default_backend())
+        x509.load_pem_x509_csr(data.encode('utf-8'), default_backend())
     except Exception:
         raise ValidationError('CSR presented is not valid.')
 

lemur/database.py

@@ -274,6 +274,9 @@ def sort_and_page(query, model, args):
     page = args.pop('page')
     count = args.pop('count')
 
+    if args.get('user'):
+        user = args.pop('user')
+
     query = find_all(query, model, args)
 
     if sort_by and sort_dir:

lemur/defaults/schemas.py

@@ -0,0 +1,23 @@
+"""
+.. module: lemur.defaults.schemas
+    :platform: unix
+    :copyright: (c) 2015 by Netflix Inc., see AUTHORS for more
+    :license: Apache, see LICENSE for more details.
+.. moduleauthor:: Kevin Glisson <kglisson@netflix.com>
+"""
+from marshmallow import fields
+from lemur.common.schema import LemurOutputSchema
+from lemur.authorities.schemas import AuthorityNestedOutputSchema
+
+
+class DefaultOutputSchema(LemurOutputSchema):
+    authority = fields.Nested(AuthorityNestedOutputSchema)
+    country = fields.String()
+    state = fields.String()
+    location = fields.String()
+    organization = fields.String()
+    organizational_unit = fields.String()
+    issuer_plugin = fields.String()
+
+
+default_output_schema = DefaultOutputSchema()

lemur/defaults/views.py

@@ -1,13 +1,17 @@
 """
-.. module: lemur.status.views
+.. module: lemur.defaults.views
     :copyright: (c) 2015 by Netflix Inc., see AUTHORS for more
     :license: Apache, see LICENSE for more details.
 """
 from flask import current_app, Blueprint
-from flask.ext.restful import Api
+from flask_restful import Api
 
+from lemur.common.schema import validate_schema
+from lemur.authorities.service import get_by_name
 from lemur.auth.service import AuthenticatedResource
 
+from lemur.defaults.schemas import default_output_schema
+
 
 mod = Blueprint('default', __name__)
 api = Api(mod)
@@ -18,6 +22,7 @@ class LemurDefaults(AuthenticatedResource):
     def __init__(self):
         super(LemurDefaults)
 
+    @validate_schema(None, default_output_schema)
     def get(self):
         """
         .. http:get:: /defaults
@@ -52,13 +57,18 @@ class LemurDefaults(AuthenticatedResource):
            :statuscode 200: no error
            :statuscode 403: unauthenticated
         """
+
+        default_authority = get_by_name(current_app.config.get('LEMUR_DEFAULT_AUTHORITY'))
+
         return dict(
             country=current_app.config.get('LEMUR_DEFAULT_COUNTRY'),
             state=current_app.config.get('LEMUR_DEFAULT_STATE'),
             location=current_app.config.get('LEMUR_DEFAULT_LOCATION'),
             organization=current_app.config.get('LEMUR_DEFAULT_ORGANIZATION'),
-            organizationalUnit=current_app.config.get('LEMUR_DEFAULT_ORGANIZATIONAL_UNIT'),
-            issuerPlugin=current_app.config.get('LEMUR_DEFAULT_ISSUER_PLUGIN')
+            organizational_unit=current_app.config.get('LEMUR_DEFAULT_ORGANIZATIONAL_UNIT'),
+            issuer_plugin=current_app.config.get('LEMUR_DEFAULT_ISSUER_PLUGIN'),
+            authority=default_authority
         )
 
+
 api.add_resource(LemurDefaults, '/defaults', endpoint='default')

lemur/deployment/service.py

@@ -0,0 +1,16 @@
+from lemur import database
+
+
+def rotate_certificate(endpoint, new_cert):
+    """
+    Rotates a certificate on a given endpoint.
+
+    :param endpoint:
+    :param new_cert:
+    :return:
+    """
+    # ensure that certificate is available for rotation
+
+    endpoint.source.plugin.update_endpoint(endpoint, new_cert)
+    endpoint.certificate = new_cert
+    database.update(endpoint)

lemur/destinations/schemas.py

@@ -29,7 +29,8 @@ class DestinationOutputSchema(LemurOutputSchema):
 
     @post_dump
     def fill_object(self, data):
-        data['plugin']['pluginOptions'] = data['options']
+        if data:
+            data['plugin']['pluginOptions'] = data['options']
         return data
 
 

lemur/destinations/service.py

@@ -56,7 +56,7 @@ def delete(destination_id):
 
 def get(destination_id):
     """
-    Retrieves an destination by it's lemur assigned ID.
+    Retrieves an destination by its lemur assigned ID.
 
     :param destination_id: Lemur assigned ID
     :rtype : Destination
@@ -67,7 +67,7 @@ def get(destination_id):
 
 def get_by_label(label):
     """
-    Retrieves a destination by it's label
+    Retrieves a destination by its label
 
     :param label:
     :return:

lemur/destinations/views.py

@@ -7,7 +7,7 @@
 .. moduleauthor:: Kevin Glisson <kglisson@netflix.com>
 """
 from flask import Blueprint
-from flask.ext.restful import Api, reqparse
+from flask_restful import Api, reqparse
 from lemur.destinations import service
 
 from lemur.auth.service import AuthenticatedResource
@@ -82,7 +82,7 @@ class DestinationsList(AuthenticatedResource):
               }
 
            :query sortBy: field to sort on
-           :query sortDir: acs or desc
+           :query sortDir: asc or desc
            :query page: int. default is 1
            :query filter: key value pair format is k;v
            :query count: count number default is 10
@@ -392,7 +392,7 @@ class CertificateDestinations(AuthenticatedResource):
               }
 
            :query sortBy: field to sort on
-           :query sortDir: acs or desc
+           :query sortDir: asc or desc
            :query page: int default is 1
            :query filter: key value pair format is k;v
            :query count: count number default is 10

lemur/domains/service.py

@@ -34,7 +34,7 @@ def get_all():
 
 def get_by_name(name):
     """
-    Fetches domain by it's name
+    Fetches domain by its name
 
     :param name:
     :return:

lemur/domains/views.py

@@ -8,7 +8,7 @@
 
 """
 from flask import Blueprint
-from flask.ext.restful import reqparse, Api
+from flask_restful import reqparse, Api
 
 from lemur.domains import service
 from lemur.auth.service import AuthenticatedResource
@@ -68,7 +68,7 @@ class DomainsList(AuthenticatedResource):
               }
 
            :query sortBy: field to sort on
-           :query sortDir: acs or desc
+           :query sortDir: asc or desc
            :query page: int default is 1
            :query filter: key value pair format is k;v
            :query count: count number. default is 10
@@ -115,7 +115,7 @@ class DomainsList(AuthenticatedResource):
               }
 
            :query sortBy: field to sort on
-           :query sortDir: acs or desc
+           :query sortDir: asc or desc
            :query page: int default is 1
            :query filter: key value pair format is k;v
            :query count: count number default is 10
@@ -255,7 +255,7 @@ class CertificateDomains(AuthenticatedResource):
               }
 
            :query sortBy: field to sort on
-           :query sortDir: acs or desc
+           :query sortDir: asc or desc
            :query page: int default is 1
            :query filter: key value pair format is k;v
            :query count: count number default is 10

lemur/endpoints/cli.py

@@ -0,0 +1,39 @@
+"""
+.. module: lemur.endpoints.cli
+    :platform: Unix
+    :copyright: (c) 2015 by Netflix Inc., see AUTHORS for more
+    :license: Apache, see LICENSE for more details.
+.. moduleauthor:: Kevin Glisson <kglisson@netflix.com>
+"""
+from flask_script import Manager
+
+import arrow
+from datetime import timedelta
+
+from sqlalchemy import cast
+from sqlalchemy_utils import ArrowType
+
+from lemur import database
+from lemur.extensions import metrics
+from lemur.endpoints.models import Endpoint
+
+
+manager = Manager(usage="Handles all endpoint related tasks.")
+
+
+@manager.option('-ttl', '--time-to-live', type=int, dest='ttl', default=2, help='Time in hours, which endpoint has not been refreshed to remove the endpoint.')
+def expire(ttl):
+    """
+    Removed all endpoints that have not been recently updated.
+    """
+    print("[+] Staring expiration of old endpoints.")
+    now = arrow.utcnow()
+    expiration = now - timedelta(hours=ttl)
+    endpoints = database.session_query(Endpoint).filter(cast(Endpoint.last_updated, ArrowType) <= expiration)
+
+    for endpoint in endpoints:
+        print("[!] Expiring endpoint: {name} Last Updated: {last_updated}".format(name=endpoint.name, last_updated=endpoint.last_updated))
+        database.delete(endpoint)
+        metrics.send('endpoint_expired', 'counter', 1)
+
+    print("[+] Finished expiration.")

lemur/endpoints/models.py

@@ -1,16 +1,20 @@
 """
 .. module: lemur.endpoints.models
     :platform: unix
-    :synopsis: This module contains all of the models need to create a authority within Lemur.
+    :synopsis: This module contains all of the models need to create an authority within Lemur.
     :copyright: (c) 2015 by Netflix Inc., see AUTHORS for more
     :license: Apache, see LICENSE for more details.
 .. moduleauthor:: Kevin Glisson <kglisson@netflix.com>
 """
+import arrow
 from sqlalchemy.orm import relationship
-from sqlalchemy import Column, Integer, String, func, DateTime, PassiveDefault, Boolean, ForeignKey
+from sqlalchemy.ext.associationproxy import association_proxy
+from sqlalchemy import Column, Integer, String, Boolean, ForeignKey
 from sqlalchemy.ext.hybrid import hybrid_property
 from sqlalchemy.sql.expression import case
 
+from sqlalchemy_utils import ArrowType
+
 from lemur.database import db
 
 from lemur.models import policies_ciphers
@@ -58,13 +62,16 @@ class Endpoint(db.Model):
     type = Column(String(128))
     active = Column(Boolean, default=True)
     port = Column(Integer)
-    date_created = Column(DateTime, PassiveDefault(func.now()), nullable=False)
     policy_id = Column(Integer, ForeignKey('policy.id'))
     policy = relationship('Policy', backref='endpoint')
     certificate_id = Column(Integer, ForeignKey('certificates.id'))
     source_id = Column(Integer, ForeignKey('sources.id'))
     sensitive = Column(Boolean, default=False)
     source = relationship('Source', back_populates='endpoints')
+    last_updated = Column(ArrowType, default=arrow.utcnow, nullable=False)
+    date_created = Column(ArrowType, default=arrow.utcnow, onupdate=arrow.utcnow, nullable=False)
+
+    replaced = association_proxy('certificate', 'replaced')
 
     @property
     def issues(self):

lemur/endpoints/schemas.py

@@ -39,5 +39,6 @@ class EndpointOutputSchema(LemurOutputSchema):
 
     issues = fields.List(fields.Dict())
 
+
 endpoint_output_schema = EndpointOutputSchema()
 endpoints_output_schema = EndpointOutputSchema(many=True)

lemur/endpoints/service.py

@@ -8,12 +8,16 @@
 .. moduleauthor:: Kevin Glisson <kglisson@netflix.com>
 
 """
-from lemur import database
-from lemur.extensions import metrics
-from lemur.endpoints.models import Endpoint, Policy, Cipher
+import arrow
+
+from flask import current_app
 
 from sqlalchemy import func
 
+from lemur import database
+from lemur.endpoints.models import Endpoint, Policy, Cipher
+from lemur.extensions import metrics
+
 
 def get_all():
     """
@@ -36,14 +40,24 @@ def get(endpoint_id):
     return database.get(Endpoint, endpoint_id)
 
 
-def get_by_dnsname(endpoint_dnsname):
+def get_by_name(name):
     """
     Retrieves an endpoint given it's name.
 
-    :param endpoint_dnsname:
+    :param name:
     :return:
     """
-    return database.get(Endpoint, endpoint_dnsname, field='dnsname')
+    return database.get(Endpoint, name, field='name')
+
+
+def get_by_dnsname(dnsname):
+    """
+    Retrieves an endpoint given it's name.
+
+    :param dnsname:
+    :return:
+    """
+    return database.get(Endpoint, dnsname, field='dnsname')
 
 
 def get_by_source(source_label):
@@ -55,6 +69,15 @@ def get_by_source(source_label):
     return Endpoint.query.filter(Endpoint.source.label == source_label).all()  # noqa
 
 
+def get_all_pending_rotation():
+    """
+    Retrieves all endpoints which have certificates deployed
+    that have been replaced.
+    :return:
+    """
+    return Endpoint.query.filter(Endpoint.replaced.any()).all()
+
+
 def create(**kwargs):
     """
     Creates a new endpoint.
@@ -63,7 +86,7 @@ def create(**kwargs):
     """
     endpoint = Endpoint(**kwargs)
     database.create(endpoint)
-    metrics.send('endpoint_added', 'counter', 1)
+    metrics.send('endpoint_added', 'counter', 1, metric_tags={'source': endpoint.source.label})
     return endpoint
 
 
@@ -92,10 +115,26 @@ def update(endpoint_id, **kwargs):
 
     endpoint.policy = kwargs['policy']
     endpoint.certificate = kwargs['certificate']
+    endpoint.source = kwargs['source']
+    endpoint.last_updated = arrow.utcnow()
+    metrics.send('endpoint_updated', 'counter', 1, metric_tags={'source': endpoint.source.label})
     database.update(endpoint)
     return endpoint
 
 
+def rotate_certificate(endpoint, new_cert):
+    """Rotates a certificate on a given endpoint."""
+    try:
+        endpoint.source.plugin.update_endpoint(endpoint, new_cert)
+        endpoint.certificate = new_cert
+        database.update(endpoint)
+        metrics.send('certificate_rotate_success', 'counter', 1, metric_tags={'endpoint': endpoint.name, 'source': endpoint.source.label})
+    except Exception as e:
+        metrics.send('certificate_rotate_failure', 'counter', 1, metric_tags={'endpoint': endpoint.name})
+        current_app.logger.exception(e)
+        raise e
+
+
 def render(args):
     """
     Helper that helps us render the REST Api responses.

lemur/endpoints/views.py

@@ -5,8 +5,8 @@
     :license: Apache, see LICENSE for more details.
 .. moduleauthor:: Kevin Glisson <kglisson@netflix.com>
 """
-from flask import Blueprint
-from flask.ext.restful import reqparse, Api
+from flask import Blueprint, g
+from flask_restful import reqparse, Api
 
 from lemur.common.utils import paginated_parser
 from lemur.common.schema import validate_schema
@@ -51,7 +51,7 @@ class EndpointsList(AuthenticatedResource):
 
 
            :query sortBy: field to sort on
-           :query sortDir: acs or desc
+           :query sortDir: asc or desc
            :query page: int default is 1
            :query filter: key value pair. format is k;v
            :query limit: limit number default is 10
@@ -63,6 +63,7 @@ class EndpointsList(AuthenticatedResource):
         """
         parser = paginated_parser.copy()
         args = parser.parse_args()
+        args['user'] = g.current_user
         return service.render(args)
 
 

lemur/exceptions.py

@@ -7,8 +7,8 @@ from flask import current_app
 
 
 class LemurException(Exception):
-    def __init__(self):
-        current_app.logger.error(self)
+    def __init__(self, *args, **kwargs):
+        current_app.logger.exception(self)
 
 
 class DuplicateError(LemurException):
@@ -19,41 +19,11 @@ class DuplicateError(LemurException):
         return repr("Duplicate found! Could not create: {0}".format(self.key))
 
 
-class AuthenticationFailedException(LemurException):
-    def __init__(self, remote_ip, user_agent):
-        self.remote_ip = remote_ip
-        self.user_agent = user_agent
-
-    def __str__(self):
-        return repr("Failed login from: {} {}".format(self.remote_ip, self.user_agent))
-
-
-class IntegrityError(LemurException):
-    def __init__(self, message):
-        self.message = message
-
-    def __str__(self):
-        return repr(self.message)
-
-
-class AssociatedObjectNotFound(LemurException):
-    def __init__(self, message):
-        self.message = message
-
-    def __str__(self):
-        return repr(self.message)
-
-
 class InvalidListener(LemurException):
     def __str__(self):
         return repr("Invalid listener, ensure you select a certificate if you are using a secure protocol")
 
 
-class CertificateUnavailable(LemurException):
-    def __str__(self):
-        return repr("The certificate requested is not available")
-
-
 class AttrNotFound(LemurException):
     def __init__(self, field):
         self.field = field
@@ -62,16 +32,5 @@ class AttrNotFound(LemurException):
         return repr("The field '{0}' is not sortable".format(self.field))
 
 
-class NoPersistanceFound(Exception):
-    def __str__(self):
-        return repr("No peristence method found, Lemur cannot persist sensitive information")
-
-
-class NoEncryptionKeyFound(Exception):
-    def __str__(self):
-        return repr("Aborting... Lemur cannot locate db encryption key, is ENCRYPTION_KEY set?")
-
-
-class InvalidToken(Exception):
-    def __str__(self):
-        return repr("Invalid token")
+class InvalidConfiguration(Exception):
+    pass

lemur/factory.py

@@ -92,16 +92,20 @@ def configure_app(app, config=None):
     """
     # respect the config first
     if config and config != 'None':
+        app.config['CONFIG_PATH'] = config
         app.config.from_object(from_file(config))
+    else:
+        try:
+            app.config.from_envvar("LEMUR_CONF")
+        except RuntimeError:
+            # look in default paths
+            if os.path.isfile(os.path.expanduser("~/.lemur/lemur.conf.py")):
+                app.config.from_object(from_file(os.path.expanduser("~/.lemur/lemur.conf.py")))
+            else:
+                app.config.from_object(from_file(os.path.join(os.path.dirname(os.path.realpath(__file__)), 'default.conf.py')))
 
-    try:
-        app.config.from_envvar("LEMUR_CONF")
-    except RuntimeError:
-        # look in default paths
-        if os.path.isfile(os.path.expanduser("~/.lemur/lemur.conf.py")):
-            app.config.from_object(from_file(os.path.expanduser("~/.lemur/lemur.conf.py")))
-        else:
-            app.config.from_object(from_file(os.path.join(os.path.dirname(os.path.realpath(__file__)), 'default.conf.py')))
+    # we don't use this
+    app.config['SQLALCHEMY_TRACK_MODIFICATIONS'] = False
 
 
 def configure_extensions(app):

lemur/logs/models.py

@@ -0,0 +1,23 @@
+"""
+.. module: lemur.logs.models
+    :platform: unix
+    :synopsis: This module contains all of the models related private key audit log.
+    :copyright: (c) 2015 by Netflix Inc., see AUTHORS for more
+    :license: Apache, see LICENSE for more details.
+
+.. moduleauthor:: Kevin Glisson <kglisson@netflix.com>
+"""
+from sqlalchemy import Column, Integer, ForeignKey, PassiveDefault, func, Enum
+
+from sqlalchemy_utils.types.arrow import ArrowType
+
+from lemur.database import db
+
+
+class Log(db.Model):
+    __tablename__ = 'logs'
+    id = Column(Integer, primary_key=True)
+    certificate_id = Column(Integer, ForeignKey('certificates.id'))
+    log_type = Column(Enum('key_view', name='log_type'), nullable=False)
+    logged_at = Column(ArrowType(), PassiveDefault(func.now()), nullable=False)
+    user_id = Column(Integer, ForeignKey('users.id'), nullable=False)

lemur/logs/schemas.py

@@ -0,0 +1,23 @@
+"""
+.. module: lemur.logs.schemas
+    :platform: unix
+    :copyright: (c) 2015 by Netflix Inc., see AUTHORS for more
+    :license: Apache, see LICENSE for more details.
+.. moduleauthor:: Kevin Glisson <kglisson@netflix.com>
+"""
+from marshmallow import fields
+
+from lemur.common.schema import LemurOutputSchema
+from lemur.certificates.schemas import CertificateNestedOutputSchema
+from lemur.users.schemas import UserNestedOutputSchema
+
+
+class LogOutputSchema(LemurOutputSchema):
+    id = fields.Integer()
+    certificate = fields.Nested(CertificateNestedOutputSchema)
+    user = fields.Nested(UserNestedOutputSchema)
+    logged_at = fields.DateTime()
+    log_type = fields.String()
+
+
+logs_output_schema = LogOutputSchema(many=True)

lemur/logs/service.py

@@ -0,0 +1,54 @@
+"""
+.. module: lemur.logs.service
+    :platform: Unix
+    :synopsis: This module contains all of the services level functions used to
+    administer logs in Lemur
+    :copyright: (c) 2015 by Netflix Inc., see AUTHORS for more
+    :license: Apache, see LICENSE for more details.
+.. moduleauthor:: Kevin Glisson <kglisson@netflix.com>
+"""
+from lemur import database
+from lemur.logs.models import Log
+
+
+def create(user, type, certificate=None):
+    """
+    Creates logs a given action.
+
+    :param user:
+    :param type:
+    :param certificate:
+    :return:
+    """
+    view = Log(user_id=user.id, log_type=type, certificate_id=certificate.id)
+    database.add(view)
+    database.commit()
+
+
+def get_all():
+    """
+    Retrieve all logs from the database.
+
+    :return:
+    """
+    query = database.session_query(Log)
+    return database.find_all(query, Log, {}).all()
+
+
+def render(args):
+    """
+    Helper that paginates and filters data when requested
+    through the REST Api
+
+    :param args:
+    :return:
+    """
+    query = database.session_query(Log)
+
+    filt = args.pop('filter')
+
+    if filt:
+        terms = filt.split(';')
+        query = database.filter(query, Log, terms)
+
+    return database.sort_and_page(query, Log, args)

lemur/logs/views.py

@@ -0,0 +1,74 @@
+"""
+.. module: lemur.logs.views
+    :platform: Unix
+    :copyright: (c) 2015 by Netflix Inc., see AUTHORS for more
+    :license: Apache, see LICENSE for more details.
+.. moduleauthor:: Kevin Glisson <kglisson@netflix.com>
+"""
+from flask import Blueprint
+from flask_restful import reqparse, Api
+
+from lemur.common.schema import validate_schema
+from lemur.common.utils import paginated_parser
+
+from lemur.auth.service import AuthenticatedResource
+from lemur.logs.schemas import logs_output_schema
+
+from lemur.logs import service
+
+
+mod = Blueprint('logs', __name__)
+api = Api(mod)
+
+
+class LogsList(AuthenticatedResource):
+    """ Defines the 'logs' endpoint """
+    def __init__(self):
+        self.reqparse = reqparse.RequestParser()
+        super(LogsList, self).__init__()
+
+    @validate_schema(None, logs_output_schema)
+    def get(self):
+        """
+        .. http:get:: /logs
+
+           The current log list
+
+           **Example request**:
+
+           .. sourcecode:: http
+
+              GET /logs HTTP/1.1
+              Host: example.com
+              Accept: application/json, text/javascript
+
+           **Example response**:
+
+           .. sourcecode:: http
+
+              HTTP/1.1 200 OK
+              Vary: Accept
+              Content-Type: text/javascript
+
+              {
+                "items": [
+                  ]
+                "total": 2
+              }
+
+           :query sortBy: field to sort on
+           :query sortDir: asc or desc
+           :query page: int default is 1
+           :query filter: key value pair format is k;v
+           :query count: count number default is 10
+           :reqheader Authorization: OAuth token to authenticate
+           :statuscode 200: no error
+        """
+        parser = paginated_parser.copy()
+        parser.add_argument('owner', type=str, location='args')
+        parser.add_argument('id', type=str, location='args')
+        args = parser.parse_args()
+        return service.render(args)
+
+
+api.add_resource(LogsList, '/logs', endpoint='logs')

lemur/manage.py

@@ -1,39 +1,32 @@
 from __future__ import unicode_literals    # at top of module
 
-from datetime import datetime, timedelta
-from collections import Counter
-
 import os
 import sys
 import base64
-import time
 import requests
 import json
 
-from tabulate import tabulate
 from gunicorn.config import make_settings
 
 from cryptography.fernet import Fernet
 
 from flask import current_app
-from flask.ext.script import Manager, Command, Option, prompt_pass
-from flask.ext.migrate import Migrate, MigrateCommand, stamp
+from flask_script import Manager, Command, Option, prompt_pass
+from flask_migrate import Migrate, MigrateCommand, stamp
 from flask_script.commands import ShowUrls, Clean, Server
 
+from lemur.sources.cli import manager as source_manager
+from lemur.certificates.cli import manager as certificate_manager
+from lemur.notifications.cli import manager as notification_manager
+from lemur.endpoints.cli import manager as endpoint_manager
+from lemur.reporting.cli import manager as report_manager
 from lemur import database
-from lemur.extensions import metrics
 from lemur.users import service as user_service
 from lemur.roles import service as role_service
-from lemur.certificates import service as cert_service
-from lemur.authorities import service as authority_service
 from lemur.notifications import service as notification_service
 
-from lemur.certificates.service import get_name_from_arn
-from lemur.certificates.verify import verify_string
 
-from lemur.plugins.lemur_aws import elb
-
-from lemur.sources import service as source_service
+from lemur.common.utils import validate_conf
 
 from lemur import create_app
 
@@ -46,6 +39,7 @@ from lemur.destinations.models import Destination  # noqa
 from lemur.domains.models import Domain  # noqa
 from lemur.notifications.models import Notification  # noqa
 from lemur.sources.models import Source  # noqa
+from lemur.logs.models import Log  # noqa
 
 
 manager = Manager(create_app)
@@ -53,12 +47,21 @@ manager.add_option('-c', '--config', dest='config')
 
 migrate = Migrate(create_app)
 
+REQUIRED_VARIABLES = [
+    'LEMUR_SECURITY_TEAM_EMAIL',
+    'LEMUR_DEFAULT_ORGANIZATIONAL_UNIT',
+    'LEMUR_DEFAULT_ORGANIZATION',
+    'LEMUR_DEFAULT_LOCATION',
+    'LEMUR_DEFAULT_COUNTRY',
+    'LEMUR_DEFAULT_STATE',
+    'SQLALCHEMY_DATABASE_URI'
+]
+
 KEY_LENGTH = 40
 DEFAULT_CONFIG_PATH = '~/.lemur/lemur.conf.py'
 DEFAULT_SETTINGS = 'lemur.conf.server'
 SETTINGS_ENVVAR = 'LEMUR_CONF'
 
-
 CONFIG_TEMPLATE = """
 # This is just Python which means you can inherit and tweak settings
 
@@ -110,7 +113,6 @@ LOG_FILE = "lemur.log"
 # modify this if you are not using a local database
 SQLALCHEMY_DATABASE_URI = 'postgresql://lemur:lemur@localhost:5432/lemur'
 
-
 # AWS
 
 #LEMUR_INSTANCE_PROFILE = 'Lemur'
@@ -139,28 +141,6 @@ def drop_all():
     database.db.drop_all()
 
 
-@manager.command
-def check_revoked():
-    """
-    Function attempts to update Lemur's internal cache with revoked
-    certificates. This is called periodically by Lemur. It checks both
-    CRLs and OCSP to see if a certificate is revoked. If Lemur is unable
-    encounters an issue with verification it marks the certificate status
-    as `unknown`.
-    """
-    for cert in cert_service.get_all_certs():
-        try:
-            if cert.chain:
-                status = verify_string(cert.body, cert.chain)
-            else:
-                status = verify_string(cert.body, "")
-
-            cert.status = 'valid' if status else 'invalid'
-        except Exception as e:
-            cert.status = 'unknown'
-        database.update(cert)
-
-
 @manager.shell
 def make_shell_context():
     """
@@ -181,31 +161,14 @@ def generate_settings():
     output = CONFIG_TEMPLATE.format(
         # we use Fernet.generate_key to make sure that the key length is
         # compatible with Fernet
-        encryption_key=Fernet.generate_key(),
-        secret_token=base64.b64encode(os.urandom(KEY_LENGTH)),
-        flask_secret_key=base64.b64encode(os.urandom(KEY_LENGTH)),
+        encryption_key=Fernet.generate_key().decode('utf-8'),
+        secret_token=base64.b64encode(os.urandom(KEY_LENGTH)).decode('utf-8'),
+        flask_secret_key=base64.b64encode(os.urandom(KEY_LENGTH)).decode('utf-8'),
     )
 
     return output
 
 
-@manager.command
-def notify():
-    """
-    Runs Lemur's notification engine, that looks for expired certificates and sends
-    notifications out to those that bave subscribed to them.
-
-    :return:
-    """
-    sys.stdout.write("Starting to notify subscribers about expiring certificates!\n")
-    count = notification_service.send_expiration_notifications()
-    sys.stdout.write(
-        "Finished notifying subscribers about expiring certificates! Sent {count} notifications!\n".format(
-            count=count
-        )
-    )
-
-
 class InitializeApp(Command):
     """
     This command will bootstrap our database with any destinations as
@@ -222,6 +185,33 @@ class InitializeApp(Command):
         create()
         user = user_service.get_by_username("lemur")
 
+        admin_role = role_service.get_by_name('admin')
+
+        if admin_role:
+            sys.stdout.write("[-] Admin role already created, skipping...!\n")
+        else:
+            # we create an admin role
+            admin_role = role_service.create('admin', description='This is the Lemur administrator role.')
+            sys.stdout.write("[+] Created 'admin' role\n")
+
+        operator_role = role_service.get_by_name('operator')
+
+        if operator_role:
+            sys.stdout.write("[-] Operator role already created, skipping...!\n")
+        else:
+            # we create an admin role
+            operator_role = role_service.create('operator', description='This is the Lemur operator role.')
+            sys.stdout.write("[+] Created 'operator' role\n")
+
+        read_only_role = role_service.get_by_name('read-only')
+
+        if read_only_role:
+            sys.stdout.write("[-] Operator role already created, skipping...!\n")
+        else:
+            # we create an admin role
+            read_only_role = role_service.create('read-only', description='This is the Lemur read only role.')
+            sys.stdout.write("[+] Created 'read-only' role\n")
+
         if not user:
             if not password:
                 sys.stdout.write("We need to set Lemur's password to continue!\n")
@@ -232,17 +222,8 @@ class InitializeApp(Command):
                     sys.stderr.write("[!] Passwords do not match!\n")
                     sys.exit(1)
 
-            role = role_service.get_by_name('admin')
-
-            if role:
-                sys.stdout.write("[-] Admin role already created, skipping...!\n")
-            else:
-                # we create an admin role
-                role = role_service.create('admin', description='this is the lemur administrator role')
-                sys.stdout.write("[+] Created 'admin' role\n")
-
-            user_service.create("lemur", password, 'lemur@nobody', True, None, [role])
-            sys.stdout.write("[+] Added a 'lemur' user and added it to the 'admin' role!\n")
+            user_service.create("lemur", password, 'lemur@nobody', True, None, [admin_role])
+            sys.stdout.write("[+] Created the user 'lemur' and granted it the 'admin' role!\n")
 
         else:
             sys.stdout.write("[-] Default user has already been created, skipping...!\n")
@@ -366,10 +347,16 @@ class LemurServer(Command):
 
     def get_options(self):
         settings = make_settings()
-        options = (
-            Option(*klass.cli, action=klass.action)
-            for setting, klass in settings.items() if klass.cli
-        )
+        options = []
+        for setting, klass in settings.items():
+            if klass.cli:
+                if klass.action:
+                    if klass.action == 'store_const':
+                        options.append(Option(*klass.cli, const=klass.const, action=klass.action))
+                    else:
+                        options.append(Option(*klass.cli, action=klass.action))
+                else:
+                    options.append(Option(*klass.cli))
 
         return options
 
@@ -377,7 +364,11 @@ class LemurServer(Command):
         from gunicorn.app.wsgiapp import WSGIApplication
 
         app = WSGIApplication()
-        app.app_uri = 'lemur:create_app(config="{0}")'.format(kwargs.get('config'))
+
+        # run startup tasks on a app like object
+        validate_conf(current_app, REQUIRED_VARIABLES)
+
+        app.app_uri = 'lemur:create_app(config="{0}")'.format(current_app.config.get('CONFIG_PATH'))
 
         return app.run()
 
@@ -392,6 +383,7 @@ def create_config(config_path=None):
 
     config_path = os.path.expanduser(config_path)
     dir = os.path.dirname(config_path)
+
     if not os.path.exists(dir):
         os.makedirs(dir)
 
@@ -479,263 +471,6 @@ def unlock(path=None):
     sys.stdout.write("[+] Keys have been unencrypted!\n")
 
 
-def unicode_(data):
-    import sys
-
-    if sys.version_info.major < 3:
-        return data.decode('UTF-8')
-    return data
-
-
-class RotateELBs(Command):
-    """
-    Rotates existing certificates to a new one on an ELB
-    """
-    option_list = (
-        Option('-e', '--elb-list', dest='elb_list', required=True),
-        Option('-p', '--chain-path', dest='chain_path'),
-        Option('-c', '--cert-name', dest='cert_name'),
-        Option('-a', '--cert-prefix', dest='cert_prefix'),
-        Option('-d', '--description', dest='description')
-    )
-
-    def run(self, elb_list, chain_path, cert_name, cert_prefix, description):
-
-        for e in open(elb_list, 'r').readlines():
-            elb_name, account_id, region, from_port, to_port, protocol = e.strip().split(',')
-
-            if cert_name:
-                arn = "arn:aws:iam::{0}:server-certificate/{1}".format(account_id, cert_name)
-
-            else:
-                # if no cert name is provided we need to discover it
-                listeners = elb.get_listeners(account_id, region, elb_name)
-
-                # get the listener we care about
-                for listener in listeners:
-                    if listener[0] == int(from_port) and listener[1] == int(to_port):
-                        arn = listener[4]
-                        name = get_name_from_arn(arn)
-                        certificate = cert_service.get_by_name(name)
-                        break
-                else:
-                    sys.stdout.write("[-] Could not find ELB {0}".format(elb_name))
-                    continue
-
-                if not certificate:
-                    sys.stdout.write("[-] Could not find certificate {0} in Lemur".format(name))
-                    continue
-
-                dests = []
-                for d in certificate.destinations:
-                    dests.append({'id': d.id})
-
-                nots = []
-                for n in certificate.notifications:
-                    nots.append({'id': n.id})
-
-                new_certificate = database.clone(certificate)
-
-                if cert_prefix:
-                    new_certificate.name = "{0}-{1}".format(cert_prefix, new_certificate.name)
-
-                new_certificate.chain = open(chain_path, 'r').read()
-                new_certificate.description = "{0} - {1}".format(new_certificate.description, description)
-
-                new_certificate = database.create(new_certificate)
-                database.update_list(new_certificate, 'destinations', Destination, dests)
-                database.update_list(new_certificate, 'notifications', Notification, nots)
-                database.update(new_certificate)
-
-                arn = new_certificate.get_arn(account_id)
-
-            elb.update_listeners(account_id, region, elb_name, [(from_port, to_port, protocol, arn)], [from_port])
-
-            sys.stdout.write("[+] Updated {0} to use {1}\n".format(elb_name, new_certificate.name))
-
-
-class ProvisionELB(Command):
-    """
-    Creates and provisions a certificate on an ELB based on command line arguments
-    """
-    option_list = (
-        Option('-d', '--dns', dest='dns', action='append', required=True, type=unicode_),
-        Option('-e', '--elb', dest='elb_name', required=True, type=unicode_),
-        Option('-o', '--owner', dest='owner', type=unicode_),
-        Option('-a', '--authority', dest='authority', required=True, type=unicode_),
-        Option('-s', '--description', dest='description', default=u'Command line provisioned keypair', type=unicode_),
-        Option('-t', '--destination', dest='destinations', action='append', type=unicode_, required=True),
-        Option('-n', '--notification', dest='notifications', action='append', type=unicode_, default=[]),
-        Option('-r', '--region', dest='region', default=u'us-east-1', type=unicode_),
-        Option('-p', '--dport', '--port', dest='dport', default=7002),
-        Option('--src-port', '--source-port', '--sport', dest='sport', default=443),
-        Option('--dry-run', dest='dryrun', action='store_true')
-    )
-
-    def configure_user(self, owner):
-        from flask import g
-        import lemur.users.service
-
-        # grab the user
-        g.user = lemur.users.service.get_by_username(owner)
-        # get the first user by default
-        if not g.user:
-            g.user = lemur.users.service.get_all()[0]
-
-        return g.user.username
-
-    def build_cert_options(self, destinations, notifications, description, owner, dns, authority):
-        from sqlalchemy.orm.exc import NoResultFound
-        from lemur.certificates.views import valid_authority
-        import sys
-
-        # convert argument lists to arrays, or empty sets
-        destinations = self.get_destinations(destinations)
-        if not destinations:
-            sys.stderr.write("Valid destinations provided\n")
-            sys.exit(1)
-
-        # get the primary CN
-        common_name = dns[0]
-
-        # If there are more than one fqdn, add them as alternate names
-        extensions = {}
-        if len(dns) > 1:
-            extensions['subAltNames'] = {'names': map(lambda x: {'nameType': 'DNSName', 'value': x}, dns)}
-
-        try:
-            authority = valid_authority({"name": authority})
-        except NoResultFound:
-            sys.stderr.write("Invalid authority specified: '{}'\naborting\n".format(authority))
-            sys.exit(1)
-
-        options = {
-            # Convert from the Destination model to the JSON input expected further in the code
-            'destinations': map(lambda x: {'id': x.id, 'label': x.label}, destinations),
-            'description': description,
-            'notifications': notifications,
-            'commonName': common_name,
-            'extensions': extensions,
-            'authority': authority,
-            'owner': owner,
-            # defaults:
-            'organization': current_app.config.get('LEMUR_DEFAULT_ORGANIZATION'),
-            'organizationalUnit': current_app.config.get('LEMUR_DEFAULT_ORGANIZATIONAL_UNIT'),
-            'country': current_app.config.get('LEMUR_DEFAULT_COUNTRY'),
-            'state': current_app.config.get('LEMUR_DEFAULT_STATE'),
-            'location': current_app.config.get('LEMUR_DEFAULT_LOCATION')
-        }
-
-        return options
-
-    def get_destinations(self, destination_names):
-        from lemur.destinations import service
-
-        destinations = []
-
-        for destination_name in destination_names:
-            destination = service.get_by_label(destination_name)
-
-            if not destination:
-                sys.stderr.write("Invalid destination specified: '{}'\nAborting...\n".format(destination_name))
-                sys.exit(1)
-
-            destinations.append(service.get_by_label(destination_name))
-
-        return destinations
-
-    def check_duplicate_listener(self, elb_name, region, account, sport, dport):
-        from lemur.plugins.lemur_aws import elb
-
-        listeners = elb.get_listeners(account, region, elb_name)
-        for listener in listeners:
-            if listener[0] == sport and listener[1] == dport:
-                return True
-        return False
-
-    def get_destination_account(self, destinations):
-        for destination in self.get_destinations(destinations):
-            if destination.plugin_name == 'aws-destination':
-
-                account_number = destination.plugin.get_option('accountNumber', destination.options)
-                return account_number
-
-        sys.stderr.write("No destination AWS account provided, failing\n")
-        sys.exit(1)
-
-    def run(self, dns, elb_name, owner, authority, description, notifications, destinations, region, dport, sport,
-            dryrun):
-        from lemur.certificates import service
-        from lemur.plugins.lemur_aws import elb
-        from boto.exception import BotoServerError
-
-        # configure the owner if we can find it, or go for default, and put it in the global
-        owner = self.configure_user(owner)
-
-        # make a config blob from the command line arguments
-        cert_options = self.build_cert_options(
-            destinations=destinations,
-            notifications=notifications,
-            description=description,
-            owner=owner,
-            dns=dns,
-            authority=authority)
-
-        aws_account = self.get_destination_account(destinations)
-
-        if dryrun:
-            import json
-
-            cert_options['authority'] = cert_options['authority'].name
-            sys.stdout.write('Will create certificate using options: {}\n'
-                             .format(json.dumps(cert_options, sort_keys=True, indent=2)))
-            sys.stdout.write('Will create listener {}->{} HTTPS using the new certificate to elb {}\n'
-                             .format(sport, dport, elb_name))
-            sys.exit(0)
-
-        if self.check_duplicate_listener(elb_name, region, aws_account, sport, dport):
-            sys.stderr.write("ELB {} already has a listener {}->{}\nAborting...\n".format(elb_name, sport, dport))
-            sys.exit(1)
-
-        # create the certificate
-        try:
-            sys.stdout.write('Creating certificate for {}\n'.format(cert_options['commonName']))
-            cert = service.create(**cert_options)
-        except Exception as e:
-            if e.message == 'Duplicate certificate: a certificate with the same common name exists already':
-                sys.stderr.write("Certificate already exists named: {}\n".format(dns[0]))
-                sys.exit(1)
-            raise e
-
-        cert_arn = cert.get_arn(aws_account)
-        sys.stderr.write('cert arn: {}\n'.format(cert_arn))
-
-        sys.stderr.write('Configuring elb {} from port {} to port {} in region {} with cert {}\n'
-                         .format(elb_name, sport, dport, region, cert_arn))
-
-        delay = 1
-        done = False
-        retries = 5
-        while not done and retries > 0:
-            try:
-                elb.create_new_listeners(aws_account, region, elb_name, [(sport, dport, 'HTTPS', cert_arn)])
-            except BotoServerError as bse:
-                # if the server returns ad error, the certificate
-                if bse.error_code == 'CertificateNotFound':
-                    sys.stderr.write('Certificate not available yet in the AWS account, waiting {}, {} retries left\n'
-                                     .format(delay, retries))
-                    time.sleep(delay)
-                    delay *= 2
-                    retries -= 1
-                elif bse.error_code == 'DuplicateListener':
-                    sys.stderr.write('ELB {} already has a listener {}->{}'.format(elb_name, sport, dport))
-                    sys.exit(1)
-                else:
-                    raise bse
-            else:
-                done = True
-
-
 @manager.command
 def publish_verisign_units():
     """
@@ -781,153 +516,6 @@ def publish_unapproved_verisign_certificates():
     metrics.send('pending_certificates', 'gauge', certs)
 
 
-class Report(Command):
-    """
-    Defines a set of reports to be run periodically against Lemur.
-    """
-    option_list = (
-        Option('-n', '--name', dest='name', default=None, help='Name of the report to run.'),
-        Option('-d', '--duration', dest='duration', default=356, help='Number of days to run the report'),
-    )
-
-    def run(self, name, duration):
-
-        end = datetime.utcnow()
-        start = end - timedelta(days=duration)
-        self.certificates_issued(name, start, end)
-
-    @staticmethod
-    def certificates_issued(name=None, start=None, end=None):
-        """
-        Generates simple report of number of certificates issued by the authority, if no authority
-        is specified report on total number of certificates.
-
-        :param name:
-        :param start:
-        :param end:
-        :return:
-        """
-
-        def _calculate_row(authority):
-            day_cnt = Counter()
-            month_cnt = Counter()
-            year_cnt = Counter()
-
-            for cert in authority.certificates:
-                date = cert.date_created.date()
-                day_cnt[date.day] += 1
-                month_cnt[date.month] += 1
-                year_cnt[date.year] += 1
-
-            try:
-                day_avg = int(sum(day_cnt.values()) / len(day_cnt.keys()))
-            except ZeroDivisionError:
-                day_avg = 0
-
-            try:
-                month_avg = int(sum(month_cnt.values()) / len(month_cnt.keys()))
-            except ZeroDivisionError:
-                month_avg = 0
-
-            try:
-                year_avg = int(sum(year_cnt.values()) / len(year_cnt.keys()))
-            except ZeroDivisionError:
-                year_avg = 0
-
-            return [authority.name, authority.description, day_avg, month_avg, year_avg]
-
-        rows = []
-        if not name:
-            for authority in authority_service.get_all():
-                rows.append(_calculate_row(authority))
-
-        else:
-            authority = authority_service.get_by_name(name)
-
-            if not authority:
-                sys.stderr.write('[!] Authority {0} was not found.'.format(name))
-                sys.exit(1)
-
-            rows.append(_calculate_row(authority))
-
-        sys.stdout.write(tabulate(rows, headers=["Authority Name", "Description", "Daily Average", "Monthy Average", "Yearly Average"]) + "\n")
-
-
-class Sources(Command):
-    """
-    Defines a set of actions to take against Lemur's sources.
-    """
-    option_list = (
-        Option('-s', '--sources', dest='source_strings', action='append', help='Sources to operate on.'),
-        Option('-a', '--action', choices=['sync', 'clean'], dest='action', help='Action to take on source.')
-    )
-
-    def run(self, source_strings, action):
-        sources = []
-        if not source_strings:
-            table = []
-            for source in source_service.get_all():
-                table.append([source.label, source.active, source.description])
-
-            sys.stdout.write(tabulate(table, headers=['Label', 'Active', 'Description']))
-            sys.exit(1)
-
-        elif 'all' in source_strings:
-            sources = source_service.get_all()
-
-        else:
-            for source_str in source_strings:
-                source = source_service.get_by_label(source_str)
-
-                if not source:
-                    sys.stderr.write("Unable to find specified source with label: {0}".format(source_str))
-
-                sources.append(source)
-
-        for source in sources:
-            if action == 'sync':
-                self.sync(source)
-
-            if action == 'clean':
-                self.clean(source)
-
-    @staticmethod
-    def sync(source):
-        start_time = time.time()
-        sys.stdout.write("[+] Staring to sync source: {label}!\n".format(label=source.label))
-
-        user = user_service.get_by_username('lemur')
-
-        try:
-            source_service.sync(source, user)
-            sys.stdout.write(
-                "[+] Finished syncing source: {label}. Run Time: {time}\n".format(
-                    label=source.label,
-                    time=(time.time() - start_time)
-                )
-            )
-        except Exception as e:
-            current_app.logger.exception(e)
-
-            sys.stdout.write(
-                "[X] Failed syncing source {label}!\n".format(label=source.label)
-            )
-
-            metrics.send('sync_failed', 'counter', 1, metric_tags={'source': source.label})
-
-    @staticmethod
-    def clean(source):
-        start_time = time.time()
-        sys.stdout.write("[+] Staring to clean source: {label}!\n".format(label=source.label))
-        source_service.clean(source)
-        sys.stdout.write(
-            "[+] Finished cleaning source: {label}. Run Time: {time}\n".format(
-                label=source.label,
-                time=(time.time() - start_time)
-            )
-        )
-
-
 def main():
     manager.add_command("start", LemurServer())
     manager.add_command("runserver", Server(host='127.0.0.1', threaded=True))
@@ -938,11 +526,13 @@ def main():
     manager.add_command("create_user", CreateUser())
     manager.add_command("reset_password", ResetPassword())
     manager.add_command("create_role", CreateRole())
-    manager.add_command("provision_elb", ProvisionELB())
-    manager.add_command("rotate_elbs", RotateELBs())
-    manager.add_command("sources", Sources())
-    manager.add_command("report", Report())
+    manager.add_command("source", source_manager)
+    manager.add_command("certificate", certificate_manager)
+    manager.add_command("notify", notification_manager)
+    manager.add_command("endpoint", endpoint_manager)
+    manager.add_command("report", report_manager)
     manager.run()
 
+
 if __name__ == "__main__":
     main()

lemur/migrations/versions/131ec6accff5_.py

@@ -0,0 +1,28 @@
+"""Ensuring we have endpoint updated times and certificate rotation availability.
+
+Revision ID: 131ec6accff5
+Revises: e3691fc396e9
+Create Date: 2016-12-07 17:29:42.049986
+
+"""
+
+# revision identifiers, used by Alembic.
+revision = '131ec6accff5'
+down_revision = 'e3691fc396e9'
+
+from alembic import op
+import sqlalchemy as sa
+
+
+def upgrade():
+    # ### commands auto generated by Alembic - please adjust! ###
+    op.add_column('certificates', sa.Column('rotation', sa.Boolean(), nullable=False, server_default=False))
+    op.add_column('endpoints', sa.Column('last_updated', sa.DateTime(), server_default=sa.text('now()'), nullable=False))
+    # ### end Alembic commands ###
+
+
+def downgrade():
+    # ### commands auto generated by Alembic - please adjust! ###
+    op.drop_column('endpoints', 'last_updated')
+    op.drop_column('certificates', 'rotation')
+    # ### end Alembic commands ###

lemur/migrations/versions/5e680529b666_.py

@@ -0,0 +1,30 @@
+"""Sync up endpoints properties
+
+Revision ID: 5e680529b666
+Revises: 131ec6accff5
+Create Date: 2017-01-26 05:05:25.168125
+
+"""
+
+# revision identifiers, used by Alembic.
+revision = '5e680529b666'
+down_revision = '131ec6accff5'
+
+from alembic import op
+import sqlalchemy as sa
+
+
+def upgrade():
+    ### commands auto generated by Alembic - please adjust! ###
+    op.add_column('endpoints', sa.Column('sensitive', sa.Boolean(), nullable=True))
+    op.add_column('endpoints', sa.Column('source_id', sa.Integer(), nullable=True))
+    op.create_foreign_key(None, 'endpoints', 'sources', ['source_id'], ['id'])
+    ### end Alembic commands ###
+
+
+def downgrade():
+    ### commands auto generated by Alembic - please adjust! ###
+    op.drop_constraint(None, 'endpoints', type_='foreignkey')
+    op.drop_column('endpoints', 'source_id')
+    op.drop_column('endpoints', 'sensitive')
+    ### end Alembic commands ###

lemur/migrations/versions/7f71c0cea31a_.py

@@ -32,4 +32,4 @@ def upgrade():
 
 
 def downgrade():
-    op.drop_constraint(None, 'certificates', type_='unique')
+    op.drop_constraint(None, 'certificates', type_='unique')

lemur/migrations/versions/e3691fc396e9_.py

@@ -0,0 +1,35 @@
+"""Adding logging database tables.
+
+Revision ID: e3691fc396e9
+Revises: 932525b82f1a
+Create Date: 2016-11-28 13:15:46.995219
+
+"""
+
+# revision identifiers, used by Alembic.
+revision = 'e3691fc396e9'
+down_revision = '932525b82f1a'
+
+from alembic import op
+import sqlalchemy as sa
+import sqlalchemy_utils
+
+def upgrade():
+    ### commands auto generated by Alembic - please adjust! ###
+    op.create_table('logs',
+    sa.Column('id', sa.Integer(), nullable=False),
+    sa.Column('certificate_id', sa.Integer(), nullable=True),
+    sa.Column('log_type', sa.Enum('key_view', name='log_type'), nullable=False),
+    sa.Column('logged_at', sqlalchemy_utils.types.arrow.ArrowType(), server_default=sa.text('now()'), nullable=False),
+    sa.Column('user_id', sa.Integer(), nullable=False),
+    sa.ForeignKeyConstraint(['certificate_id'], ['certificates.id'], ),
+    sa.ForeignKeyConstraint(['user_id'], ['users.id'], ),
+    sa.PrimaryKeyConstraint('id')
+    )
+    ### end Alembic commands ###
+
+
+def downgrade():
+    ### commands auto generated by Alembic - please adjust! ###
+    op.drop_table('logs')
+    ### end Alembic commands ###

lemur/notifications/cli.py

@@ -0,0 +1,35 @@
+"""
+.. module: lemur.notifications.cli
+    :platform: Unix
+    :copyright: (c) 2015 by Netflix Inc., see AUTHORS for more
+    :license: Apache, see LICENSE for more details.
+.. moduleauthor:: Kevin Glisson <kglisson@netflix.com>
+"""
+from flask_script import Manager
+
+from lemur.notifications.messaging import send_expiration_notifications
+
+manager = Manager(usage="Handles notification related tasks.")
+
+
+@manager.option('-e', '--exclude', dest='exclude', nargs="*", help='Common name matching of certificates that should be excluded from notification')
+def expirations(exclude):
+    """
+    Runs Lemur's notification engine, that looks for expired certificates and sends
+    notifications out to those that have subscribed to them.
+
+    Every certificate receives notifications by default. When expiration notifications are handled outside of Lemur
+    we exclude their names (or matching) from expiration notifications.
+
+    It performs simple subset matching and is case insensitive.
+
+    :return:
+    """
+    print("Starting to notify subscribers about expiring certificates!")
+    success, failed = send_expiration_notifications(exclude)
+    print(
+        "Finished notifying subscribers about expiring certificates! Sent: {success} Failed: {failed}".format(
+            success=success,
+            failed=failed
+        )
+    )

lemur/notifications/messaging.py

@@ -0,0 +1,195 @@
+"""
+.. module: lemur.notifications.messaging
+    :platform: Unix
+
+    :copyright: (c) 2015 by Netflix Inc., see AUTHORS for more
+    :license: Apache, see LICENSE for more details.
+
+.. moduleauthor:: Kevin Glisson <kglisson@netflix.com>
+
+"""
+from itertools import groupby
+from collections import defaultdict
+
+import arrow
+from datetime import timedelta
+from flask import current_app
+
+from sqlalchemy import and_
+
+from lemur import database, metrics
+from lemur.common.utils import windowed_query
+
+from lemur.certificates.schemas import certificate_notification_output_schema
+from lemur.certificates.models import Certificate
+
+from lemur.plugins import plugins
+from lemur.plugins.utils import get_plugin_option
+
+
+def get_certificates(exclude=None):
+    """
+    Finds all certificates that are eligible for notifications.
+    :param exclude:
+    :return:
+    """
+    now = arrow.utcnow()
+    max = now + timedelta(days=90)
+
+    q = database.db.session.query(Certificate) \
+        .filter(Certificate.not_after <= max) \
+        .filter(Certificate.notify == True) \
+        .filter(Certificate.expired == False)  # noqa
+
+    exclude_conditions = []
+    if exclude:
+        for e in exclude:
+            exclude_conditions.append(~Certificate.name.ilike('%{}%'.format(e)))
+
+        q = q.filter(and_(*exclude_conditions))
+
+    certs = []
+
+    for c in windowed_query(q, Certificate.id, 100):
+        if needs_notification(c):
+            certs.append(c)
+
+    return certs
+
+
+def get_eligible_certificates(exclude=None):
+    """
+    Finds all certificates that are eligible for certificate expiration.
+    :param exclude:
+    :return:
+    """
+    certificates = defaultdict(dict)
+    certs = get_certificates(exclude=exclude)
+
+    # group by owner
+    for owner, items in groupby(certs, lambda x: x.owner):
+        notification_groups = []
+
+        for certificate in items:
+            notification = needs_notification(certificate)
+
+            if notification:
+                notification_groups.append((notification, certificate))
+
+        # group by notification
+        for notification, items in groupby(notification_groups, lambda x: x[0].label):
+            certificates[owner][notification] = list(items)
+
+    return certificates
+
+
+def send_notification(event_type, data, targets, notification):
+    """
+    Executes the plugin and handles failure.
+
+    :param event_type:
+    :param data:
+    :param targets:
+    :param notification:
+    :return:
+    """
+    try:
+        notification.plugin.send(event_type, data, targets, notification.options)
+        metrics.send('{0}_notification_sent'.format(event_type), 'counter', 1)
+        return True
+    except Exception as e:
+        metrics.send('{0}_notification_failure'.format(event_type), 'counter', 1)
+        current_app.logger.exception(e)
+
+
+def send_expiration_notifications(exclude):
+    """
+    This function will check for upcoming certificate expiration,
+    and send out notification emails at given intervals.
+    """
+    success = failure = 0
+
+    # security team gets all
+    security_email = current_app.config.get('LEMUR_SECURITY_TEAM_EMAIL')
+
+    security_data = []
+    for owner, notification_group in get_eligible_certificates(exclude=exclude).items():
+
+        for notification_label, certificates in notification_group.items():
+            notification_data = []
+
+            notification = certificates[0][0]
+
+            for data in certificates:
+                n, certificate = data
+                cert_data = certificate_notification_output_schema.dump(certificate).data
+                notification_data.append(cert_data)
+                security_data.append(cert_data)
+
+            if send_notification('expiration', notification_data, [owner], notification):
+                success += 1
+            else:
+                failure += 1
+
+    if send_notification('expiration', security_data, security_email, notification):
+        success += 1
+    else:
+        failure += 1
+
+    return success, failure
+
+
+def send_rotation_notification(certificate, notification_plugin=None):
+    """
+    Sends a report to certificate owners when their certificate as been
+    rotated.
+
+    :param certificate:
+    :return:
+    """
+    if not notification_plugin:
+        notification_plugin = plugins.get(current_app.config.get('LEMUR_DEFAULT_NOTIFICATION_PLUGIN'))
+
+    data = certificate_notification_output_schema.dump(certificate).data
+
+    try:
+        notification_plugin.send('rotation', data, [data['owner']])
+        metrics.send('rotation_notification_sent', 'counter', 1)
+        return True
+    except Exception as e:
+        metrics.send('rotation_notification_failure', 'counter', 1)
+        current_app.logger.exception(e)
+
+
+def needs_notification(certificate):
+    """
+    Determine if notifications for a given certificate should
+    currently be sent
+
+    :param certificate:
+    :return:
+    """
+    now = arrow.utcnow()
+    days = (certificate.not_after - now).days
+
+    for notification in certificate.notifications:
+        if not notification.options:
+            return
+
+        interval = get_plugin_option('interval', notification.options)
+        unit = get_plugin_option('unit', notification.options)
+
+        if unit == 'weeks':
+            interval *= 7
+
+        elif unit == 'months':
+            interval *= 30
+
+        elif unit == 'days':  # it's nice to be explicit about the base unit
+            pass
+
+        else:
+            raise Exception("Invalid base unit for expiration interval: {0}".format(unit))
+
+        if days == interval:
+            return notification

lemur/notifications/schemas.py

@@ -30,7 +30,8 @@ class NotificationOutputSchema(LemurOutputSchema):
 
     @post_dump
     def fill_object(self, data):
-        data['plugin']['pluginOptions'] = data['options']
+        if data:
+            data['plugin']['pluginOptions'] = data['options']
         return data
 
 

lemur/notifications/service.py

@@ -1,5 +1,5 @@
 """
-.. module: lemur.notifications
+.. module: lemur.notifications.service
     :platform: Unix
 
     :copyright: (c) 2015 by Netflix Inc., see AUTHORS for more
@@ -8,181 +8,11 @@
 .. moduleauthor:: Kevin Glisson <kglisson@netflix.com>
 
 """
-import ssl
-
-import arrow
-
 from flask import current_app
+
 from lemur import database
-from lemur.domains.models import Domain
-from lemur.notifications.models import Notification
 from lemur.certificates.models import Certificate
-
-from lemur.certificates import service as cert_service
-
-from lemur.plugins.base import plugins
-
-
-def get_options(name, options):
-    for o in options:
-        if o.get('name') == name:
-            return o
-
-
-def _get_message_data(cert):
-    """
-    Parse our the certification information needed for our notification
-
-    :param cert:
-    :return:
-    """
-    cert_dict = {}
-
-    if cert.user:
-        cert_dict['creator'] = cert.user.email
-
-    cert_dict['not_after'] = cert.not_after
-    cert_dict['owner'] = cert.owner
-    cert_dict['name'] = cert.name
-    cert_dict['body'] = cert.body
-    cert_dict['endpoints'] = [{'name': x.name, 'dnsname': x.dnsname} for x in cert.endpoints]
-
-    return cert_dict
-
-
-def _deduplicate(messages):
-    """
-    Take all of the messages that should be sent and provide
-    a roll up to the same set if the recipients are the same
-    """
-    roll_ups = []
-    for data, options in messages:
-        o = get_options('recipients', options)
-        targets = o['value'].split(',')
-
-        for m, r, o in roll_ups:
-            if r == targets:
-                for cert in m:
-                    if cert['body'] == data['body']:
-                        break
-                else:
-                    m.append(data)
-                    current_app.logger.info(
-                        "Sending expiration alert about {0} to {1}".format(
-                            data['name'], ",".join(targets)))
-                break
-        else:
-            roll_ups.append(([data], targets, options))
-
-    return roll_ups
-
-
-def send_expiration_notifications():
-    """
-    This function will check for upcoming certificate expiration,
-    and send out notification emails at given intervals.
-    """
-    sent = 0
-
-    for plugin in plugins.all(plugin_type='notification'):
-        notifications = database.db.session.query(Notification)\
-            .filter(Notification.plugin_name == plugin.slug)\
-            .filter(Notification.active == True).all()  # noqa
-
-        messages = []
-        for n in notifications:
-            for c in n.certificates:
-                if _is_eligible_for_notifications(c):
-                    messages.append((_get_message_data(c), n.options))
-
-        messages = _deduplicate(messages)
-
-        for data, targets, options in messages:
-            sent += 1
-            plugin.send('expiration', data, targets, options)
-
-        current_app.logger.info("Lemur has sent {0} certification notifications".format(sent))
-    return sent
-
-
-def _get_domain_certificate(name):
-    """
-    Fetch the SSL certificate currently hosted at a given domain (if any) and
-    compare it against our all of our know certificates to determine if a new
-    SSL certificate has already been deployed
-
-    :param name:
-    :return:
-    """
-    try:
-        pub_key = ssl.get_server_certificate((name, 443))
-        return cert_service.find_duplicates(pub_key.strip())
-    except Exception as e:
-        current_app.logger.info(str(e))
-        return []
-
-
-def _find_superseded(cert):
-    """
-    Here we try to fetch any domain in the certificate to see if we can resolve it
-    and to try and see if it is currently serving the certificate we are
-    alerting on.
-
-    :param domains:
-    :return:
-    """
-    query = database.session_query(Certificate)
-    ss_list = []
-
-    # determine what is current host at our domains
-    for domain in cert.domains:
-        dups = _get_domain_certificate(domain.name)
-        for c in dups:
-            if c.body != cert.body:
-                ss_list.append(dups)
-
-        current_app.logger.info("Trying to resolve {0}".format(domain.name))
-
-    # look for other certificates that may not be hosted but cover the same domains
-    query = query.filter(Certificate.domains.any(Domain.name.in_([x.name for x in cert.domains])))
-    query = query.filter(Certificate.active == True)  # noqa
-    query = query.filter(Certificate.not_after >= arrow.utcnow().format('YYYY-MM-DD'))
-    query = query.filter(Certificate.body != cert.body)
-    ss_list.extend(query.all())
-    return ss_list
-
-
-def _is_eligible_for_notifications(cert):
-    """
-    Determine if notifications for a given certificate should
-    currently be sent
-
-    :param cert:
-    :return:
-    """
-    if not cert.notify:
-        return
-
-    now = arrow.utcnow()
-    days = (cert.not_after - now.naive).days
-
-    for notification in cert.notifications:
-        interval = get_options('interval', notification.options)['value']
-        unit = get_options('unit', notification.options)['value']
-        if unit == 'weeks':
-            interval *= 7
-
-        elif unit == 'months':
-            interval *= 30
-
-        elif unit == 'days':  # it's nice to be explicit about the base unit
-            pass
-
-        else:
-            raise Exception("Invalid base unit for expiration interval: {0}".format(unit))
-
-        if days == interval:
-            return cert
+from lemur.notifications.models import Notification
 
 
 def create_default_expiration_notifications(name, recipients):
@@ -191,6 +21,7 @@ def create_default_expiration_notifications(name, recipients):
     already exist these will be returned instead of new notifications.
 
     :param name:
+    :param recipients:
     :return:
     """
     if not recipients:
@@ -297,7 +128,7 @@ def delete(notification_id):
 
 def get(notification_id):
     """
-    Retrieves an notification by it's lemur assigned ID.
+    Retrieves an notification by its lemur assigned ID.
 
     :param notification_id: Lemur assigned ID
     :rtype : Notification
@@ -308,7 +139,7 @@ def get(notification_id):
 
 def get_by_label(label):
     """
-    Retrieves a notification by it's label
+    Retrieves a notification by its label
 
     :param label:
     :return:

lemur/notifications/views.py

@@ -7,7 +7,7 @@
 .. moduleauthor:: Kevin Glisson <kglisson@netflix.com>
 """
 from flask import Blueprint
-from flask.ext.restful import Api, reqparse
+from flask_restful import Api, reqparse
 from lemur.notifications import service
 from lemur.notifications.schemas import notification_input_schema, notification_output_schema, notifications_output_schema
 
@@ -95,7 +95,7 @@ class NotificationsList(AuthenticatedResource):
                }
 
            :query sortBy: field to sort on
-           :query sortDir: acs or desc
+           :query sortDir: asc or desc
            :query page: int default is 1
            :query filter: key value pair format is k;v
            :query count: count number default is 10
@@ -419,7 +419,7 @@ class CertificateNotifications(AuthenticatedResource):
                }
 
            :query sortBy: field to sort on
-           :query sortDir: acs or desc
+           :query sortDir: asc or desc
            :query page: int default is 1
            :query filter: key value pair format is k;v
            :query count: count number default is 10

lemur/plugins/bases/destination.py

@@ -1,5 +1,5 @@
 """
-.. module: lemur.bases.destination
+.. module: lemur.plugins.bases.destination
     :platform: Unix
     :copyright: (c) 2015 by Netflix Inc., see AUTHORS for more
     :license: Apache, see LICENSE for more details.
@@ -13,5 +13,5 @@ class DestinationPlugin(Plugin):
     type = 'destination'
     requires_key = True
 
-    def upload(self):
-        raise NotImplemented
+    def upload(self, name, body, private_key, cert_chain, options, **kwargs):
+        raise NotImplementedError

lemur/plugins/bases/export.py

@@ -1,5 +1,5 @@
 """
-.. module: lemur.bases.export
+.. module: lemur.plugins.bases.export
     :platform: Unix
     :copyright: (c) 2015 by Netflix Inc., see AUTHORS for more
     :license: Apache, see LICENSE for more details.
@@ -17,5 +17,5 @@ class ExportPlugin(Plugin):
     type = 'export'
     requires_key = True
 
-    def export(self):
-        raise NotImplemented
+    def export(self, body, chain, key, options, **kwargs):
+        raise NotImplementedError

lemur/plugins/bases/issuer.py

@@ -1,5 +1,5 @@
 """
-.. module: lemur.bases.issuer
+.. module: lemur.plugins.bases.issuer
     :platform: Unix
     :copyright: (c) 2015 by Netflix Inc., see AUTHORS for more
     :license: Apache, see LICENSE for more details.
@@ -16,8 +16,8 @@ class IssuerPlugin(Plugin):
     """
     type = 'issuer'
 
-    def create_certificate(self):
+    def create_certificate(self, csr, issuer_options):
         raise NotImplementedError
 
-    def create_authority(self):
-        raise NotImplemented
+    def create_authority(self, options):
+        raise NotImplementedError

lemur/plugins/bases/metric.py

@@ -1,5 +1,5 @@
 """
-.. module: lemur.bases.metric
+.. module: lemur.plugins.bases.metric
     :platform: Unix
     :copyright: (c) 2015 by Netflix Inc., see AUTHORS for more
     :license: Apache, see LICENSE for more details.
@@ -12,5 +12,5 @@ from lemur.plugins.base import Plugin
 class MetricPlugin(Plugin):
     type = 'metric'
 
-    def submit(self, *args, **kwargs):
-        raise NotImplemented
+    def submit(self, metric_name, metric_type, metric_value, metric_tags=None, options=None):
+        raise NotImplementedError

lemur/plugins/bases/notification.py

@@ -1,5 +1,5 @@
 """
-.. module: lemur.bases.notification
+.. module: lemur.plugins.bases.notification
     :platform: Unix
     :copyright: (c) 2015 by Netflix Inc., see AUTHORS for more
     :license: Apache, see LICENSE for more details.
@@ -16,7 +16,7 @@ class NotificationPlugin(Plugin):
     """
     type = 'notification'
 
-    def send(self):
+    def send(self, notification_type, message, targets, options, **kwargs):
         raise NotImplementedError
 
 
@@ -48,5 +48,5 @@ class ExpirationNotificationPlugin(NotificationPlugin):
     def options(self):
         return list(self.default_options) + self.additional_options
 
-    def send(self):
+    def send(self, notification_type, message, targets, options, **kwargs):
         raise NotImplementedError

lemur/plugins/bases/source.py

@@ -1,5 +1,5 @@
 """
-.. module: lemur.bases.source
+.. module: lemur.plugins.bases.source
     :platform: Unix
     :copyright: (c) 2015 by Netflix Inc., see AUTHORS for more
     :license: Apache, see LICENSE for more details.
@@ -22,14 +22,14 @@ class SourcePlugin(Plugin):
         }
     ]
 
-    def get_certificates(self):
-        raise NotImplemented
+    def get_certificates(self, options, **kwargs):
+        raise NotImplementedError
 
-    def get_endpoints(self):
-        raise NotImplemented
+    def get_endpoints(self, options, **kwargs):
+        raise NotImplementedError
 
-    def clean(self):
-        raise NotImplemented
+    def clean(self, certificate, options, **kwargs):
+        raise NotImplementedError
 
     @property
     def options(self):

lemur/plugins/lemur_acme/plugin.py

@@ -1,7 +1,7 @@
 """
-.. module: lemur.plugins.lemur_acme.acme
+.. module: lemur.plugins.lemur_acme.plugin
     :platform: Unix
-    :synopsis: This module is responsible for communicating with a ACME CA.
+    :synopsis: This module is responsible for communicating with an ACME CA.
     :copyright: (c) 2015 by Netflix Inc., see AUTHORS for more
     :license: Apache, see LICENSE for more details.
 
@@ -15,24 +15,26 @@ from flask import current_app
 from acme.client import Client
 from acme import jose
 from acme import messages
+from acme import challenges
+
+from lemur.common.utils import generate_private_key
 
-from cryptography.hazmat.backends import default_backend
 from cryptography.hazmat.primitives import serialization
-from cryptography.hazmat.primitives.asymmetric import rsa
 
 import OpenSSL.crypto
 
+from lemur.common.utils import validate_conf
 from lemur.plugins.bases import IssuerPlugin
 from lemur.plugins import lemur_acme as acme
 
-from .route53 import delete_txt_record, create_txt_record, wait_for_change
+from .route53 import delete_txt_record, create_txt_record, wait_for_r53_change
 
 
 def find_dns_challenge(authz):
     for combo in authz.body.resolved_combinations:
         if (
             len(combo) == 1 and
-            isinstance(combo[0].chall, acme.challenges.DNS01)
+            isinstance(combo[0].chall, challenges.DNS01)
         ):
             yield combo[0]
 
@@ -45,18 +47,17 @@ class AuthorizationRecord(object):
         self.change_id = change_id
 
 
-def start_dns_challenge(acme_client, host):
-    authz = acme_client.request_domain_challenges(
-        host, acme_client.directory.new_authz
-    )
+def start_dns_challenge(acme_client, account_number, host):
+    authz = acme_client.request_domain_challenges(host)
 
     [dns_challenge] = find_dns_challenge(authz)
 
     change_id = create_txt_record(
         dns_challenge.validation_domain_name(host),
         dns_challenge.validation(acme_client.key),
-
+        account_number
     )
+
     return AuthorizationRecord(
         host,
         authz,
@@ -65,8 +66,8 @@ def start_dns_challenge(acme_client, host):
     )
 
 
-def complete_dns_challenge(acme_client, authz_record):
-    wait_for_change(authz_record.change_id)
+def complete_dns_challenge(acme_client, account_number, authz_record):
+    wait_for_r53_change(authz_record.change_id, account_number=account_number)
 
     response = authz_record.dns_challenge.response(acme_client.key)
 
@@ -75,6 +76,7 @@ def complete_dns_challenge(acme_client, authz_record):
         authz_record.host,
         acme_client.key.public_key()
     )
+
     if not verified:
         raise ValueError("Failed verification")
 
@@ -91,50 +93,35 @@ def request_certificate(acme_client, authorizations, csr):
         ),
         authzrs=[authz_record.authz for authz_record in authorizations],
     )
+
     pem_certificate = OpenSSL.crypto.dump_certificate(
         OpenSSL.crypto.FILETYPE_PEM, cert_response.body
     )
+
     pem_certificate_chain = "\n".join(
         OpenSSL.crypto.dump_certificate(OpenSSL.crypto.FILETYPE_PEM, cert)
         for cert in acme_client.fetch_chain(cert_response)
     )
+
     return pem_certificate, pem_certificate_chain
 
 
-def generate_rsa_private_key():
-    return rsa.generate_private_key(
-        public_exponent=65537, key_size=2048, backend=default_backend()
-    )
-
-
 def setup_acme_client():
-    key = current_app.config.get('ACME_PRIVATE_KEY').strip()
-    acme_email = current_app.config.get('ACME_EMAIL')
-    acme_tel = current_app.config.get('ACME_TEL')
-    acme_directory_url = current_app.config.get('ACME_DIRECTORY_URL'),
-    contact = ('mailto:{}'.format(acme_email), 'tel:{}'.format(acme_tel))
+    email = current_app.config.get('ACME_EMAIL')
+    tel = current_app.config.get('ACME_TEL')
+    directory_url = current_app.config.get('ACME_DIRECTORY_URL')
+    contact = ('mailto:{}'.format(email), 'tel:{}'.format(tel))
 
-    key = serialization.load_pem_private_key(
-        key, password=None, backend=default_backend()
-    )
-    return acme_client_for_private_key(acme_directory_url, key)
+    key = jose.JWKRSA(key=generate_private_key('RSA2048'))
 
+    client = Client(directory_url, key)
 
-def acme_client_for_private_key(acme_directory_url, private_key):
-    return Client(
-        acme_directory_url, key=jose.JWKRSA(key=private_key)
-    )
-
-
-def register(email):
-    private_key = generate_rsa_private_key()
-    acme_client = acme_client_for_private_key(current_app.config('ACME_DIRECTORY_URL'), private_key)
-
-    registration = acme_client.register(
+    registration = client.register(
         messages.NewRegistration.from_data(email=email)
     )
-    acme_client.agree_to_tos(registration)
-    return private_key
+
+    client.agree_to_tos(registration)
+    return client, registration
 
 
 def get_domains(options):
@@ -144,27 +131,29 @@ def get_domains(options):
     :return:
     """
     domains = [options['common_name']]
-    for name in options['extensions']['sub_alt_name']['names']:
-        domains.append(name)
+    if options.get('extensions'):
+        for name in options['extensions']['sub_alt_names']['names']:
+            domains.append(name)
     return domains
 
 
-def get_authorizations(acme_client, domains):
+def get_authorizations(acme_client, account_number, domains):
     authorizations = []
     try:
         for domain in domains:
-            authz_record = start_dns_challenge(acme_client, domain)
+            authz_record = start_dns_challenge(acme_client, account_number, domain)
             authorizations.append(authz_record)
 
         for authz_record in authorizations:
-            complete_dns_challenge(acme_client, authz_record)
+            complete_dns_challenge(acme_client, account_number, authz_record)
     finally:
         for authz_record in authorizations:
             dns_challenge = authz_record.dns_challenge
             delete_txt_record(
                 authz_record.change_id,
+                account_number,
                 dns_challenge.validation_domain_name(authz_record.host),
-                dns_challenge.validation(acme_client.key),
+                dns_challenge.validation(acme_client.key)
             )
 
     return authorizations
@@ -180,20 +169,30 @@ class ACMEIssuerPlugin(IssuerPlugin):
     author_url = 'https://github.com/netflix/lemur.git'
 
     def __init__(self, *args, **kwargs):
+        required_vars = [
+            'ACME_DIRECTORY_URL',
+            'ACME_TEL',
+            'ACME_EMAIL',
+            'ACME_AWS_ACCOUNT_NUMBER',
+            'ACME_ROOT'
+        ]
+
+        validate_conf(current_app, required_vars)
         super(ACMEIssuerPlugin, self).__init__(*args, **kwargs)
 
     def create_certificate(self, csr, issuer_options):
         """
-        Creates a ACME certificate.
+        Creates an ACME certificate.
 
         :param csr:
         :param issuer_options:
         :return: :raise Exception:
         """
         current_app.logger.debug("Requesting a new acme certificate: {0}".format(issuer_options))
-        acme_client = setup_acme_client()
+        acme_client, registration = setup_acme_client()
+        account_number = current_app.config.get('ACME_AWS_ACCOUNT_NUMBER')
         domains = get_domains(issuer_options)
-        authorizations = get_authorizations(acme_client, domains)
+        authorizations = get_authorizations(acme_client, account_number, domains)
         pem_certificate, pem_certificate_chain = request_certificate(acme_client, authorizations, csr)
         return pem_certificate, pem_certificate_chain
 

lemur/plugins/lemur_acme/route53.py

@@ -54,33 +54,24 @@ def change_txt_record(action, zone_id, domain, value, client=None):
     return response["ChangeInfo"]["Id"]
 
 
-def create_txt_record(host, value):
-    zone_id = find_zone_id(host)
+def create_txt_record(account_number, host, value):
+    zone_id = find_zone_id(host, account_number=account_number)
     change_id = change_txt_record(
         "CREATE",
         zone_id,
         host,
         value,
+        account_number=account_number
     )
     return zone_id, change_id
 
 
-def delete_txt_record(change_id, host, value):
+def delete_txt_record(change_id, account_number, host, value):
     zone_id, _ = change_id
     change_txt_record(
         "DELETE",
         zone_id,
         host,
-        value
+        value,
+        account_number=account_number
     )
-
-
-@sts_client('route53')
-def wait_for_change(change_id, client=None):
-    _, change_id = change_id
-
-    while True:
-        response = client.get_change(Id=change_id)
-        if response["ChangeInfo"]["Status"] == "INSYNC":
-            return
-        time.sleep(5)

lemur/plugins/lemur_aws/ec2.py

@@ -1,5 +1,5 @@
 """
-.. module: lemur.plugins.lemur_aws.elb
+.. module: lemur.plugins.lemur_aws.ec2
     :synopsis: Module contains some often used and helpful classes that
     are used to deal with ELBs
 

lemur/plugins/lemur_aws/elb.py

@@ -10,20 +10,26 @@ from flask import current_app
 
 from retrying import retry
 
+from lemur.extensions import metrics
 from lemur.exceptions import InvalidListener
-from lemur.plugins.lemur_aws.sts import sts_client, assume_service
+from lemur.plugins.lemur_aws.sts import sts_client
 
 
 def retry_throttled(exception):
     """
-    Determiens if this exception is due to throttling
+    Determines if this exception is due to throttling
     :param exception:
     :return:
     """
     if isinstance(exception, botocore.exceptions.ClientError):
-        if 'Throttling' in exception.message:
-            return True
-    return False
+        if exception.response['Error']['Code'] == 'LoadBalancerNotFound':
+            return False
+
+        if exception.response['Error']['Code'] == 'CertificateNotFound':
+            return False
+
+    metrics.send('elb_retry', 'counter', 1)
+    return True
 
 
 def is_valid(listener_tuple):
@@ -42,7 +48,6 @@ def is_valid(listener_tuple):
     :param listener_tuple:
     """
     lb_port, i_port, lb_protocol, arn = listener_tuple
-    current_app.logger.debug(lb_protocol)
     if lb_protocol.lower() in ['ssl', 'https']:
         if not arn:
             raise InvalidListener
@@ -50,16 +55,6 @@ def is_valid(listener_tuple):
     return listener_tuple
 
 
-@sts_client('elb')
-@retry(retry_on_exception=retry_throttled, stop_max_attempt_number=7, wait_exponential_multiplier=1000)
-def get_elbs(**kwargs):
-    """
-    Fetches one page elb objects for a given account and region.
-    """
-    client = kwargs.pop('client')
-    return client.describe_load_balancers(**kwargs)
-
-
 def get_all_elbs(**kwargs):
     """
     Fetches all elbs for a given account/region
@@ -74,14 +69,87 @@ def get_all_elbs(**kwargs):
 
         elbs += response['LoadBalancerDescriptions']
 
-        if not response.get('IsTruncated'):
+        if not response.get('NextMarker'):
             return elbs
+        else:
+            kwargs.update(dict(Marker=response['NextMarker']))
 
-        if response['NextMarker']:
-            kwargs.update(dict(marker=response['NextMarker']))
+
+def get_all_elbs_v2(**kwargs):
+    """
+    Fetches all elbs for a given account/region
+
+    :param kwargs:
+    :return:
+    """
+    elbs = []
+
+    while True:
+        response = get_elbs_v2(**kwargs)
+        elbs += response['LoadBalancers']
+
+        if not response.get('NextMarker'):
+            return elbs
+        else:
+            kwargs.update(dict(Marker=response['NextMarker']))
+
+
+@sts_client('elbv2')
+@retry(retry_on_exception=retry_throttled, stop_max_attempt_number=7, wait_exponential_multiplier=1000)
+def get_listener_arn_from_endpoint(endpoint_name, endpoint_port, **kwargs):
+    """
+    Get a listener ARN from a endpoint.
+    :param endpoint_name:
+    :param endpoint_port:
+    :return:
+    """
+    client = kwargs.pop('client')
+    elbs = client.describe_load_balancers(Names=[endpoint_name])
+    for elb in elbs['LoadBalancers']:
+        listeners = client.describe_listeners(LoadBalancerArn=elb['LoadBalancerArn'])
+        for listener in listeners['Listeners']:
+            if listener['Port'] == endpoint_port:
+                return listener['ListenerArn']
 
 
 @sts_client('elb')
+@retry(retry_on_exception=retry_throttled, stop_max_attempt_number=7, wait_exponential_multiplier=1000)
+def get_elbs(**kwargs):
+    """
+    Fetches one page elb objects for a given account and region.
+    """
+    client = kwargs.pop('client')
+    return client.describe_load_balancers(**kwargs)
+
+
+@sts_client('elbv2')
+@retry(retry_on_exception=retry_throttled, stop_max_attempt_number=7, wait_exponential_multiplier=1000)
+def get_elbs_v2(**kwargs):
+    """
+    Fetches one page of elb objects for a given account and region.
+
+    :param kwargs:
+    :return:
+    """
+    client = kwargs.pop('client')
+    return client.describe_load_balancers(**kwargs)
+
+
+@sts_client('elbv2')
+@retry(retry_on_exception=retry_throttled, stop_max_attempt_number=7, wait_exponential_multiplier=1000)
+def describe_listeners_v2(**kwargs):
+    """
+    Fetches one page of listener objects for a given elb arn.
+
+    :param kwargs:
+    :return:
+    """
+    client = kwargs.pop('client')
+    return client.describe_listeners(**kwargs)
+
+
+@sts_client('elb')
+@retry(retry_on_exception=retry_throttled, stop_max_attempt_number=7, wait_exponential_multiplier=1000)
 def describe_load_balancer_policies(load_balancer_name, policy_names, **kwargs):
     """
     Fetching all policies currently associated with an ELB.
@@ -92,7 +160,20 @@ def describe_load_balancer_policies(load_balancer_name, policy_names, **kwargs):
     return kwargs['client'].describe_load_balancer_policies(LoadBalancerName=load_balancer_name, PolicyNames=policy_names)
 
 
+@sts_client('elbv2')
+@retry(retry_on_exception=retry_throttled, stop_max_attempt_number=7, wait_exponential_multiplier=1000)
+def describe_ssl_policies_v2(policy_names, **kwargs):
+    """
+    Fetching all policies currently associated with an ELB.
+
+    :param policy_names:
+    :return:
+    """
+    return kwargs['client'].describe_ssl_policies(Names=policy_names)
+
+
 @sts_client('elb')
+@retry(retry_on_exception=retry_throttled, stop_max_attempt_number=7, wait_exponential_multiplier=1000)
 def describe_load_balancer_types(policies, **kwargs):
     """
     Describe the policies with policy details.
@@ -103,81 +184,41 @@ def describe_load_balancer_types(policies, **kwargs):
     return kwargs['client'].describe_load_balancer_policy_types(PolicyTypeNames=policies)
 
 
-def attach_certificate(account_number, region, name, port, certificate_id):
+@sts_client('elb')
+@retry(retry_on_exception=retry_throttled, stop_max_attempt_number=7, wait_exponential_multiplier=1000)
+def attach_certificate(name, port, certificate_id, **kwargs):
     """
     Attaches a certificate to a listener, throws exception
     if certificate specified does not exist in a particular account.
 
-    :param account_number:
-    :param region:
     :param name:
     :param port:
     :param certificate_id:
     """
-    return assume_service(account_number, 'elb', region).set_lb_listener_SSL_certificate(name, port, certificate_id)
+    try:
+        return kwargs['client'].set_load_balancer_listener_ssl_certificate(LoadBalancerName=name, LoadBalancerPort=port, SSLCertificateId=certificate_id)
+    except botocore.exceptions.ClientError as e:
+        if e.response['Error']['Code'] == 'LoadBalancerNotFound':
+            current_app.logger.warning("Loadbalancer does not exist.")
+        else:
+            raise e
 
 
-# def create_new_listeners(account_number, region, name, listeners=None):
-#     """
-#     Creates a new listener and attaches it to the ELB.
-#
-#     :param account_number:
-#     :param region:
-#     :param name:
-#     :param listeners:
-#     :return:
-#     """
-#     listeners = [is_valid(x) for x in listeners]
-#     return assume_service(account_number, 'elb', region).create_load_balancer_listeners(name, listeners=listeners)
-#
-#
-# def update_listeners(account_number, region, name, listeners, ports):
-#     """
-#     We assume that a listener with a specified port already exists. We can then
-#     delete the old listener on the port and create a new one in it's place.
-#
-#     If however we are replacing a listener e.g. changing a port from 80 to 443 we need
-#     to make sure we kept track of which ports we needed to delete so that we don't create
-#     two listeners (one 80 and one 443)
-#
-#     :param account_number:
-#     :param region:
-#     :param name:
-#     :param listeners:
-#     :param ports:
-#     """
-#     # you cannot update a listeners port/protocol instead we remove the only one and
-#     # create a new one in it's place
-#     listeners = [is_valid(x) for x in listeners]
-#
-#     assume_service(account_number, 'elb', region).delete_load_balancer_listeners(name, ports)
-#     return create_new_listeners(account_number, region, name, listeners=listeners)
-#
-#
-# def delete_listeners(account_number, region, name, ports):
-#     """
-#     Deletes a listener from an ELB.
-#
-#     :param account_number:
-#     :param region:
-#     :param name:
-#     :param ports:
-#     :return:
-#     """
-#     return assume_service(account_number, 'elb', region).delete_load_balancer_listeners(name, ports)
-#
-#
-# def get_listeners(account_number, region, name):
-#     """
-#     Gets the listeners configured on an elb and returns a array of tuples
-#
-#     :param account_number:
-#     :param region:
-#     :param name:
-#     :return: list of tuples
-#     """
-#
-#     conn = assume_service(account_number, 'elb', region)
-#     elbs = conn.get_all_load_balancers(load_balancer_names=[name])
-#     if elbs:
-#         return elbs[0].listeners
+@sts_client('elbv2')
+@retry(retry_on_exception=retry_throttled, stop_max_attempt_number=7, wait_exponential_multiplier=1000)
+def attach_certificate_v2(listener_arn, port, certificates, **kwargs):
+    """
+    Attaches a certificate to a listener, throws exception
+    if certificate specified does not exist in a particular account.
+
+    :param listener_arn:
+    :param port:
+    :param certificates:
+    """
+    try:
+        return kwargs['client'].modify_listener(ListenerArn=listener_arn, Port=port, Certificates=certificates)
+    except botocore.exceptions.ClientError as e:
+        if e.response['Error']['Code'] == 'LoadBalancerNotFound':
+            current_app.logger.warning("Loadbalancer does not exist.")
+        else:
+            raise e

lemur/plugins/lemur_aws/iam.py

@@ -6,7 +6,26 @@
     :license: Apache, see LICENSE for more details.
 .. moduleauthor:: Kevin Glisson <kglisson@netflix.com>
 """
-from lemur.plugins.lemur_aws.sts import assume_service
+import botocore
+
+from retrying import retry
+
+from lemur.extensions import metrics
+from lemur.plugins.lemur_aws.sts import sts_client
+
+
+def retry_throttled(exception):
+    """
+    Determines if this exception is due to throttling
+    :param exception:
+    :return:
+    """
+    if isinstance(exception, botocore.exceptions.ClientError):
+        if exception.response['Error']['Code'] == 'NoSuchEntity':
+            return False
+
+    metrics.send('iam_retry', 'counter', 1)
+    return True
 
 
 def get_name_from_arn(arn):
@@ -19,79 +38,109 @@ def get_name_from_arn(arn):
     return arn.split("/", 1)[1]
 
 
-def upload_cert(account_number, name, body, private_key, cert_chain=None):
+def create_arn_from_cert(account_number, region, certificate_name):
+    """
+    Create an ARN from a certificate.
+    :param account_number:
+    :param region:
+    :param certificate_name:
+    :return:
+    """
+    return "arn:aws:iam::{account_number}:server-certificate/{certificate_name}".format(
+        account_number=account_number,
+        certificate_name=certificate_name)
+
+
+@sts_client('iam')
+@retry(retry_on_exception=retry_throttled, stop_max_attempt_number=7, wait_exponential_multiplier=100)
+def upload_cert(name, body, private_key, cert_chain=None, **kwargs):
     """
     Upload a certificate to AWS
 
-    :param account_number:
     :param name:
+    :param body:
     :param private_key:
     :param cert_chain:
     :return:
     """
-    return assume_service(account_number, 'iam').upload_server_cert(name, str(body), str(private_key),
-                                                                    cert_chain=str(cert_chain))
+    client = kwargs.pop('client')
+    try:
+        if cert_chain:
+            return client.upload_server_certificate(
+                ServerCertificateName=name,
+                CertificateBody=str(body),
+                PrivateKey=str(private_key),
+                CertificateChain=str(cert_chain)
+            )
+        else:
+            return client.upload_server_certificate(
+                ServerCertificateName=name,
+                CertificateBody=str(body),
+                PrivateKey=str(private_key)
+            )
+    except botocore.exceptions.ClientError as e:
+        if e.response['Error']['Code'] != 'EntityAlreadyExists':
+            raise e
 
 
-def delete_cert(account_number, cert_name):
+@sts_client('iam')
+@retry(retry_on_exception=retry_throttled, stop_max_attempt_number=7, wait_exponential_multiplier=100)
+def delete_cert(cert_name, **kwargs):
     """
     Delete a certificate from AWS
 
-    :param account_number:
     :param cert_name:
     :return:
     """
-    return assume_service(account_number, 'iam').delete_server_cert(cert_name)
+    client = kwargs.pop('client')
+    try:
+        client.delete_server_certificate(ServerCertificateName=cert_name)
+    except botocore.exceptions.ClientError as e:
+        if e.response['Error']['Code'] != 'NoSuchEntity':
+            raise e
 
 
-def get_all_server_certs(account_number):
+@sts_client('iam')
+@retry(retry_on_exception=retry_throttled, stop_max_attempt_number=7, wait_exponential_multiplier=100)
+def get_certificate(name, **kwargs):
+    """
+    Retrieves an SSL certificate.
+
+    :return:
+    """
+    client = kwargs.pop('client')
+    return client.get_server_certificate(
+        ServerCertificateName=name
+    )['ServerCertificate']
+
+
+@sts_client('iam')
+@retry(retry_on_exception=retry_throttled, stop_max_attempt_number=7, wait_exponential_multiplier=100)
+def get_certificates(**kwargs):
+    """
+    Fetches one page of certificate objects for a given account.
+    :param kwargs:
+    :return:
+    """
+    client = kwargs.pop('client')
+    return client.list_server_certificates(**kwargs)
+
+
+def get_all_certificates(**kwargs):
     """
     Use STS to fetch all of the SSL certificates from a given account
-
-    :param account_number:
     """
-    marker = None
-    certs = []
+    certificates = []
+    account_number = kwargs.get('account_number')
+
     while True:
-        response = assume_service(account_number, 'iam').get_all_server_certs(marker=marker)
-        result = response['list_server_certificates_response']['list_server_certificates_result']
+        response = get_certificates(**kwargs)
+        metadata = response['ServerCertificateMetadataList']
 
-        for cert in result['server_certificate_metadata_list']:
-            certs.append(cert['arn'])
+        for m in metadata:
+            certificates.append(get_certificate(m['ServerCertificateName'], account_number=account_number))
 
-        if result['is_truncated'] == 'true':
-            marker = result['marker']
+        if not response.get('Marker'):
+            return certificates
         else:
-            return certs
-
-
-def get_cert_from_arn(arn):
-    """
-    Retrieves an SSL certificate from a given ARN.
-
-    :param arn:
-    :return:
-    """
-    name = get_name_from_arn(arn)
-    account_number = arn.split(":")[4]
-    name = name.split("/")[-1]
-
-    response = assume_service(account_number, 'iam').get_server_certificate(name.strip())
-    return digest_aws_cert_response(response)
-
-
-def digest_aws_cert_response(response):
-    """
-    Processes an AWS certifcate response and retrieves the certificate body and chain.
-
-    :param response:
-    :return:
-    """
-    chain = None
-    cert = response['get_server_certificate_response']['get_server_certificate_result']['server_certificate']
-    body = cert['certificate_body']
-
-    if 'certificate_chain' in cert:
-        chain = cert['certificate_chain']
-
-    return str(body), str(chain),
+            kwargs.update(dict(Marker=response['Marker']))

lemur/plugins/lemur_aws/plugin.py

@@ -1,5 +1,5 @@
 """
-.. module: lemur.plugins.lemur_aws.aws
+.. module: lemur.plugins.lemur_aws.plugin
     :platform: Unix
     :copyright: (c) 2015 by Netflix Inc., see AUTHORS for more
     :license: Apache, see LICENSE for more details.
@@ -33,15 +33,118 @@
 .. moduleauthor:: Harm Weites <harm@weites.com>
 """
 from flask import current_app
-from boto.exception import BotoServerError
 
 from lemur.plugins.bases import DestinationPlugin, SourcePlugin
-from lemur.plugins.lemur_aws.ec2 import get_regions
-from lemur.plugins.lemur_aws.elb import get_all_elbs, describe_load_balancer_policies, attach_certificate
-from lemur.plugins.lemur_aws import iam, s3
+from lemur.plugins.lemur_aws import iam, s3, elb, ec2
 from lemur.plugins import lemur_aws as aws
 
 
+def get_region_from_dns(dns):
+    return dns.split('.')[-4]
+
+
+def format_elb_cipher_policy_v2(policy):
+    """
+    Attempts to format cipher policy information for elbv2 into a common format.
+    :param policy:
+    :return:
+    """
+    ciphers = []
+    name = None
+
+    for descr in policy['SslPolicies']:
+        name = descr['Name']
+        for cipher in descr['Ciphers']:
+            ciphers.append(cipher['Name'])
+
+    return dict(name=name, ciphers=ciphers)
+
+
+def format_elb_cipher_policy(policy):
+    """
+    Attempts to format cipher policy information into a common format.
+    :param policy:
+    :return:
+    """
+    ciphers = []
+    name = None
+    for descr in policy['PolicyDescriptions']:
+        for attr in descr['PolicyAttributeDescriptions']:
+            if attr['AttributeName'] == 'Reference-Security-Policy':
+                name = attr['AttributeValue']
+                continue
+
+            if attr['AttributeValue'] == 'true':
+                ciphers.append(attr['AttributeName'])
+
+    return dict(name=name, ciphers=ciphers)
+
+
+def get_elb_endpoints(account_number, region, elb_dict):
+    """
+    Retrieves endpoint information from elb response data.
+    :param account_number:
+    :param region:
+    :param elb_dict:
+    :return:
+    """
+    endpoints = []
+    for listener in elb_dict['ListenerDescriptions']:
+        if not listener['Listener'].get('SSLCertificateId'):
+            continue
+
+        if listener['Listener']['SSLCertificateId'] == 'Invalid-Certificate':
+            continue
+
+        endpoint = dict(
+            name=elb_dict['LoadBalancerName'],
+            dnsname=elb_dict['DNSName'],
+            type='elb',
+            port=listener['Listener']['LoadBalancerPort'],
+            certificate_name=iam.get_name_from_arn(listener['Listener']['SSLCertificateId'])
+        )
+
+        if listener['PolicyNames']:
+            policy = elb.describe_load_balancer_policies(elb_dict['LoadBalancerName'], listener['PolicyNames'], account_number=account_number, region=region)
+            endpoint['policy'] = format_elb_cipher_policy(policy)
+
+        endpoints.append(endpoint)
+
+    return endpoints
+
+
+def get_elb_endpoints_v2(account_number, region, elb_dict):
+    """
+    Retrieves endpoint information from elbv2 response data.
+    :param account_number:
+    :param region:
+    :param elb_dict:
+    :return:
+    """
+    endpoints = []
+    listeners = elb.describe_listeners_v2(account_number=account_number, region=region, LoadBalancerArn=elb_dict['LoadBalancerArn'])
+    for listener in listeners['Listeners']:
+        if not listener.get('Certificates'):
+            continue
+
+        for certificate in listener['Certificates']:
+            endpoint = dict(
+                name=elb_dict['LoadBalancerName'],
+                dnsname=elb_dict['DNSName'],
+                type='elbv2',
+                port=listener['Port'],
+                certificate_name=iam.get_name_from_arn(certificate['CertificateArn'])
+            )
+
+        if listener['SslPolicy']:
+            policy = elb.describe_ssl_policies_v2([listener['SslPolicy']], account_number=account_number, region=region)
+            endpoint['policy'] = format_elb_cipher_policy_v2(policy)
+
+        endpoints.append(endpoint)
+
+    return endpoints
+
+
 class AWSDestinationPlugin(DestinationPlugin):
     title = 'AWS'
     slug = 'aws-destination'
@@ -68,16 +171,12 @@ class AWSDestinationPlugin(DestinationPlugin):
     # }
 
     def upload(self, name, body, private_key, cert_chain, options, **kwargs):
-        try:
-            iam.upload_cert(self.get_option('accountNumber', options), name, body, private_key,
-                            cert_chain=cert_chain)
-        except BotoServerError as e:
-            if e.error_code != 'EntityAlreadyExists':
-                raise Exception(e)
+        iam.upload_cert(name, body, private_key,
+                        cert_chain=cert_chain,
+                        account_number=self.get_option('accountNumber', options))
 
-        e = self.get_option('elb', options)
-        if e:
-            attach_certificate(kwargs['accountNumber'], ['region'], e['name'], e['port'], e['certificateId'])
+    def deploy(self, elb_name, account, region, certificate):
+        pass
 
 
 class AWSSourcePlugin(SourcePlugin):
@@ -105,18 +204,8 @@ class AWSSourcePlugin(SourcePlugin):
     ]
 
     def get_certificates(self, options, **kwargs):
-        certs = []
-        arns = iam.get_all_server_certs(self.get_option('accountNumber', options))
-        for arn in arns:
-            cert_body, cert_chain = iam.get_cert_from_arn(arn)
-            cert_name = iam.get_name_from_arn(arn)
-            cert = dict(
-                body=cert_body,
-                chain=cert_chain,
-                name=cert_name
-            )
-            certs.append(cert)
-        return certs
+        cert_data = iam.get_all_certificates(account_number=self.get_option('accountNumber', options))
+        return [dict(body=c['CertificateBody'], chain=c.get('CertificateChain'), name=c['ServerCertificateMetadata']['ServerCertificateName']) for c in cert_data]
 
     def get_endpoints(self, options, **kwargs):
         endpoints = []
@@ -124,72 +213,43 @@ class AWSSourcePlugin(SourcePlugin):
         regions = self.get_option('regions', options)
 
         if not regions:
-            regions = get_regions(account_number=account_number)
+            regions = ec2.get_regions(account_number=account_number)
         else:
             regions = regions.split(',')
 
         for region in regions:
-            elbs = get_all_elbs(account_number=account_number, region=region)
-            current_app.logger.info("Describing load balancers in {0}-{1}".format(account_number, region))
-            for elb in elbs:
-                for listener in elb['ListenerDescriptions']:
-                    if not listener['Listener'].get('SSLCertificateId'):
-                        continue
+            elbs = elb.get_all_elbs(account_number=account_number, region=region)
+            current_app.logger.info("Describing classic load balancers in {0}-{1}".format(account_number, region))
 
-                    if listener['Listener']['SSLCertificateId'] == 'Invalid-Certificate':
-                        continue
+            for e in elbs:
+                endpoints.extend(get_elb_endpoints(account_number, region, e))
 
-                    endpoint = dict(
-                        name=elb['LoadBalancerName'],
-                        dnsname=elb['DNSName'],
-                        type='elb',
-                        port=listener['Listener']['LoadBalancerPort'],
-                        certificate_name=iam.get_name_from_arn(listener['Listener']['SSLCertificateId'])
-                    )
+            # fetch advanced ELBs
+            elbs_v2 = elb.get_all_elbs_v2(account_number=account_number, region=region)
+            current_app.logger.info("Describing advanced load balancers in {0}-{1}".format(account_number, region))
 
-                    if listener['PolicyNames']:
-                        policy = describe_load_balancer_policies(elb['LoadBalancerName'], listener['PolicyNames'], account_number=account_number, region=region)
-                        endpoint['policy'] = format_elb_cipher_policy(policy)
-
-                    endpoints.append(endpoint)
+            for e in elbs_v2:
+                endpoints.extend(get_elb_endpoints_v2(account_number, region, e))
 
         return endpoints
 
-    def clean(self, options, **kwargs):
+    def update_endpoint(self, endpoint, certificate):
+        options = endpoint.source.options
         account_number = self.get_option('accountNumber', options)
-        certificates = self.get_certificates(options)
-        endpoints = self.get_endpoints(options)
 
-        orphaned = []
-        for certificate in certificates:
-            for endpoint in endpoints:
-                if certificate['name'] == endpoint['certificate_name']:
-                    break
-            else:
-                orphaned.append(certificate['name'])
-                iam.delete_cert(account_number, certificate)
+        # relies on the fact that region is included in DNS name
+        region = get_region_from_dns(endpoint.dnsname)
+        arn = iam.create_arn_from_cert(account_number, region, certificate.name)
 
-        return orphaned
+        if endpoint.type == 'elbv2':
+            listener_arn = elb.get_listener_arn_from_endpoint(endpoint.name, endpoint.port, account_number=account_number, region=region)
+            elb.attach_certificate_v2(listener_arn, endpoint.port, [{'CertificateArn': arn}], account_number=account_number, region=region)
+        else:
+            elb.attach_certificate(endpoint.name, endpoint.port, arn, account_number=account_number, region=region)
 
-
-def format_elb_cipher_policy(policy):
-    """
-    Attempts to format cipher policy information into a common format.
-    :param policy:
-    :return:
-    """
-    ciphers = []
-    name = None
-    for descr in policy['PolicyDescriptions']:
-        for attr in descr['PolicyAttributeDescriptions']:
-            if attr['AttributeName'] == 'Reference-Security-Policy':
-                name = attr['AttributeValue']
-                continue
-
-            if attr['AttributeValue'] == 'true':
-                ciphers.append(attr['AttributeName'])
-
-    return dict(name=name, ciphers=ciphers)
+    def clean(self, certificate, options, **kwargs):
+        account_number = self.get_option('accountNumber', options)
+        iam.delete_cert(certificate.name, account_number=account_number)
 
 
 class S3DestinationPlugin(DestinationPlugin):

lemur/plugins/lemur_aws/sts.py

@@ -1,5 +1,5 @@
 """
-.. module: lemur.common.services.aws.sts
+.. module: lemur.plugins.lemur_aws.sts
     :platform: Unix
     :copyright: (c) 2015 by Netflix Inc., see AUTHORS for more
     :license: Apache, see LICENSE for more details.
@@ -56,6 +56,7 @@ def sts_client(service, service_type='client'):
                 kwargs.pop('account_number'),
                 current_app.config.get('LEMUR_INSTANCE_PROFILE', 'Lemur')
             )
+
             # TODO add user specific information to RoleSessionName
             role = sts.assume_role(RoleArn=arn, RoleSessionName='lemur')
 

lemur/plugins/lemur_aws/tests/test_iam.py

@@ -14,16 +14,7 @@ def test_get_name_from_arn():
 @mock_sts()
 @mock_iam()
 def test_get_all_server_certs(app):
-    from lemur.plugins.lemur_aws.iam import upload_cert, get_all_server_certs
-    upload_cert('123456789012', 'testCert', EXTERNAL_VALID_STR.decode('utf-8'), PRIVATE_KEY_STR.decode('utf-8'))
-    certs = get_all_server_certs('123456789012')
+    from lemur.plugins.lemur_aws.iam import upload_cert, get_all_certificates
+    upload_cert('123456789012', 'testCert', EXTERNAL_VALID_STR, PRIVATE_KEY_STR)
+    certs = get_all_certificates('123456789012')
     assert len(certs) == 1
-
-
-@mock_sts()
-@mock_iam()
-def test_get_cert_from_arn(app):
-    from lemur.plugins.lemur_aws.iam import upload_cert, get_cert_from_arn
-    upload_cert('123456789012', 'testCert', EXTERNAL_VALID_STR.decode('utf-8'), PRIVATE_KEY_STR.decode('utf-8'))
-    body, chain = get_cert_from_arn('arn:aws:iam::123456789012:server-certificate/testCert')
-    assert body.replace('\n', '') == EXTERNAL_VALID_STR.decode('utf-8').replace('\n', '')

lemur/plugins/lemur_cfssl/plugin.py

@@ -42,7 +42,7 @@ class CfsslIssuerPlugin(IssuerPlugin):
 
         url = "{0}{1}".format(current_app.config.get('CFSSL_URL'), '/api/v1/cfssl/sign')
 
-        data = {'certificate_request': csr.decode('utf_8')}
+        data = {'certificate_request': csr}
         data = json.dumps(data)
 
         response = self.session.post(url, data=data.encode(encoding='utf_8', errors='strict'))

lemur/plugins/lemur_cryptography/plugin.py

@@ -13,86 +13,160 @@ from flask import current_app
 from cryptography import x509
 from cryptography.hazmat.backends import default_backend
 from cryptography.hazmat.primitives import hashes, serialization
-from cryptography.hazmat.primitives.asymmetric import rsa
-
 
 from lemur.plugins.bases import IssuerPlugin
 from lemur.plugins import lemur_cryptography as cryptography_issuer
 
+from lemur.certificates.service import create_csr
 
-def build_root_certificate(options):
-    private_key = rsa.generate_private_key(
-        public_exponent=65537,
-        key_size=2048,
-        backend=default_backend()
-    )
 
-    subject = issuer = x509.Name([
-        x509.NameAttribute(x509.OID_COUNTRY_NAME, options['country']),
-        x509.NameAttribute(x509.OID_STATE_OR_PROVINCE_NAME, options['state']),
-        x509.NameAttribute(x509.OID_LOCALITY_NAME, options['location']),
-        x509.NameAttribute(x509.OID_ORGANIZATION_NAME, options['organization']),
-        x509.NameAttribute(x509.OID_ORGANIZATIONAL_UNIT_NAME, options['organizational_unit']),
-        x509.NameAttribute(x509.OID_COMMON_NAME, options['common_name'])
-    ])
+def build_certificate_authority(options):
+    options['certificate_authority'] = True
+    csr, private_key = create_csr(**options)
+    cert_pem, chain_cert_pem = issue_certificate(csr, options, private_key)
+
+    return cert_pem, private_key, chain_cert_pem
+
+
+def issue_certificate(csr, options, private_key=None):
+    csr = x509.load_pem_x509_csr(csr.encode('utf-8'), default_backend())
+
+    if options.get("parent"):
+        # creating intermediate authorities will have options['parent'] to specify the issuer
+        # creating certificates will have options['authority'] to specify the issuer
+        # This works around that by making sure options['authority'] can be referenced for either
+        options['authority'] = options['parent']
+
+    if options.get("authority"):
+        # Issue certificate signed by an existing lemur_certificates authority
+        issuer_subject = options['authority'].authority_certificate.subject
+        issuer_private_key = options['authority'].authority_certificate.private_key
+        chain_cert_pem = options['authority'].authority_certificate.body
+        authority_key_identifier_public = options['authority'].authority_certificate.public_key
+        authority_key_identifier_subject = x509.SubjectKeyIdentifier.from_public_key(authority_key_identifier_public)
+        authority_key_identifier_issuer = issuer_subject
+        authority_key_identifier_serial = int(options['authority'].authority_certificate.serial)
+        # TODO figure out a better way to increment serial
+        # New authorities have a value at options['serial_number'] that is being ignored here.
+        serial = int(uuid.uuid4())
+    else:
+        # Issue certificate that is self-signed (new lemur_certificates root authority)
+        issuer_subject = csr.subject
+        issuer_private_key = private_key
+        chain_cert_pem = ""
+        authority_key_identifier_public = csr.public_key()
+        authority_key_identifier_subject = None
+        authority_key_identifier_issuer = csr.subject
+        authority_key_identifier_serial = options['serial_number']
+        # TODO figure out a better way to increment serial
+        serial = int(uuid.uuid4())
+
+    extensions = normalize_extensions(csr)
 
     builder = x509.CertificateBuilder(
-        subject_name=subject,
-        issuer_name=issuer,
-        public_key=private_key.public_key(),
-        not_valid_after=options['validity_end'],
-        not_valid_before=options['validity_start'],
-        serial_number=options['first_serial']
-    )
-
-    builder.add_extension(x509.SubjectAlternativeName([x509.DNSName(options['common_name'])]), critical=False)
-
-    cert = builder.sign(private_key, hashes.SHA256(), default_backend())
-
-    cert_pem = cert.public_bytes(
-        encoding=serialization.Encoding.PEM
-    ).decode('utf-8')
-
-    private_key_pem = private_key.private_bytes(
-        encoding=serialization.Encoding.PEM,
-        format=serialization.PrivateFormat.TraditionalOpenSSL,  # would like to use PKCS8 but AWS ELBs don't like it
-        encryption_algorithm=serialization.NoEncryption()
-    )
-
-    return cert_pem, private_key_pem
-
-
-def issue_certificate(csr, options):
-    csr = x509.load_pem_x509_csr(csr, default_backend())
-
-    builder = x509.CertificateBuilder(
-        issuer_name=x509.Name([
-            x509.NameAttribute(
-                x509.OID_ORGANIZATION_NAME,
-                options['authority'].authority_certificate.issuer
-            )]
-        ),
+        issuer_name=issuer_subject,
         subject_name=csr.subject,
         public_key=csr.public_key(),
         not_valid_before=options['validity_start'],
         not_valid_after=options['validity_end'],
-        extensions=csr.extensions)
+        serial_number=serial,
+        extensions=extensions)
 
-    # TODO figure out a better way to increment serial
-    builder = builder.serial_number(int(uuid.uuid4()))
+    for k, v in options.get('extensions', {}).items():
+        if k == 'authority_key_identifier':
+            # One or both of these options may be present inside the aki extension
+            (authority_key_identifier, authority_identifier) = (False, False)
+            for k2, v2 in v.items():
+                if k2 == 'use_key_identifier' and v2:
+                    authority_key_identifier = True
+                if k2 == 'use_authority_cert' and v2:
+                    authority_identifier = True
+            if authority_key_identifier:
+                if authority_key_identifier_subject:
+                    # FIXME in python-cryptography.
+                    # from_issuer_subject_key_identifier(cls, ski) is looking for ski.value.digest
+                    # but the digest of the ski is at just ski.digest. Until that library is fixed,
+                    # this function won't work. The second line has the same result.
+                    # aki = x509.AuthorityKeyIdentifier.from_issuer_subject_key_identifier(authority_key_identifier_subject)
+                    aki = x509.AuthorityKeyIdentifier(authority_key_identifier_subject.digest, None, None)
+                else:
+                    aki = x509.AuthorityKeyIdentifier.from_issuer_public_key(authority_key_identifier_public)
+            elif authority_identifier:
+                aki = x509.AuthorityKeyIdentifier(None, [x509.DirectoryName(authority_key_identifier_issuer)], authority_key_identifier_serial)
+            builder = builder.add_extension(aki, critical=False)
+        if k == 'certificate_info_access':
+            # FIXME: Implement the AuthorityInformationAccess extension
+            # descriptions = [
+            #     x509.AccessDescription(x509.oid.AuthorityInformationAccessOID.OCSP, x509.UniformResourceIdentifier(u"http://FIXME")),
+            #     x509.AccessDescription(x509.oid.AuthorityInformationAccessOID.CA_ISSUERS, x509.UniformResourceIdentifier(u"http://FIXME"))
+            # ]
+            # for k2, v2 in v.items():
+            #     if k2 == 'include_aia' and v2 == True:
+            #         builder = builder.add_extension(
+            #             x509.AuthorityInformationAccess(descriptions),
+            #             critical=False
+            #         )
+            pass
+        if k == 'crl_distribution_points':
+            # FIXME: Implement the CRLDistributionPoints extension
+            # FIXME: Not implemented in lemur/schemas.py yet https://github.com/Netflix/lemur/issues/662
+            pass
 
     private_key = serialization.load_pem_private_key(
-        bytes(str(options['authority'].authority_certificate.private_key).encode('utf-8')),
+        bytes(str(issuer_private_key).encode('utf-8')),
         password=None,
         backend=default_backend()
     )
 
     cert = builder.sign(private_key, hashes.SHA256(), default_backend())
-
-    return cert.public_bytes(
+    cert_pem = cert.public_bytes(
         encoding=serialization.Encoding.PEM
     ).decode('utf-8')
 
+    return cert_pem, chain_cert_pem
+
+
+def normalize_extensions(csr):
+    try:
+        san_extension = csr.extensions.get_extension_for_oid(x509.oid.ExtensionOID.SUBJECT_ALTERNATIVE_NAME)
+        san_dnsnames = san_extension.value.get_values_for_type(x509.DNSName)
+    except x509.extensions.ExtensionNotFound:
+        san_dnsnames = []
+        san_extension = x509.Extension(x509.oid.ExtensionOID.SUBJECT_ALTERNATIVE_NAME, True, x509.SubjectAlternativeName(san_dnsnames))
+
+    common_name = csr.subject.get_attributes_for_oid(x509.oid.NameOID.COMMON_NAME)
+    common_name = common_name[0].value
+
+    if common_name not in san_dnsnames and " " not in common_name:
+        # CommonName isn't in SAN and CommonName has no spaces that will cause idna errors
+        # Create new list of GeneralNames for including in the SAN extension
+        general_names = []
+        try:
+            # Try adding Subject CN as first SAN general_name
+            general_names.append(x509.DNSName(common_name))
+        except TypeError:
+            # CommonName probably not a valid string for DNSName
+            pass
+
+        # Add all submitted SAN names to general_names
+        for san in san_extension.value:
+            general_names.append(san)
+
+        san_extension = x509.Extension(x509.oid.ExtensionOID.SUBJECT_ALTERNATIVE_NAME, True, x509.SubjectAlternativeName(general_names))
+
+    # Remove original san extension from CSR and add new SAN extension
+    extensions = list(filter(filter_san_extensions, csr.extensions._extensions))
+    if san_extension is not None and len(san_extension.value._general_names) > 0:
+        extensions.append(san_extension)
+
+    return extensions
+
+
+def filter_san_extensions(ext):
+    if ext.oid == x509.oid.ExtensionOID.SUBJECT_ALTERNATIVE_NAME:
+        return False
+    return True
+
 
 class CryptographyIssuerPlugin(IssuerPlugin):
     title = 'Cryptography'
@@ -112,8 +186,8 @@ class CryptographyIssuerPlugin(IssuerPlugin):
         :return: :raise Exception:
         """
         current_app.logger.debug("Issuing new cryptography certificate with options: {0}".format(options))
-        cert = issue_certificate(csr, options)
-        return cert, ""
+        cert_pem, chain_cert_pem = issue_certificate(csr, options)
+        return cert_pem, chain_cert_pem
 
     @staticmethod
     def create_authority(options):
@@ -125,9 +199,9 @@ class CryptographyIssuerPlugin(IssuerPlugin):
         :return:
         """
         current_app.logger.debug("Issuing new cryptography authority with options: {0}".format(options))
-        cert, private_key = build_root_certificate(options)
+        cert_pem, private_key, chain_cert_pem = build_certificate_authority(options)
         roles = [
             {'username': '', 'password': '', 'name': options['name'] + '_admin'},
             {'username': '', 'password': '', 'name': options['name'] + '_operator'}
         ]
-        return cert, private_key, "", roles
+        return cert_pem, private_key, chain_cert_pem, roles

lemur/plugins/lemur_cryptography/tests/test_cryptography.py

@@ -0,0 +1,40 @@
+import arrow
+
+
+def test_build_certificate_authority():
+    from lemur.plugins.lemur_cryptography.plugin import build_certificate_authority
+
+    options = {
+        'key_type': 'RSA2048',
+        'country': 'US',
+        'state': 'CA',
+        'location': 'Example place',
+        'organization': 'Example, Inc.',
+        'organizational_unit': 'Example Unit',
+        'common_name': 'Example ROOT',
+        'validity_start': arrow.get('2016-12-01').datetime,
+        'validity_end': arrow.get('2016-12-02').datetime,
+        'first_serial': 1,
+        'serial_number': 1,
+        'owner': 'owner@example.com'
+    }
+    cert_pem, private_key_pem, chain_cert_pem = build_certificate_authority(options)
+
+    assert cert_pem
+    assert private_key_pem
+    assert chain_cert_pem == ''
+
+
+def test_issue_certificate(authority):
+    from lemur.tests.vectors import CSR_STR
+    from lemur.plugins.lemur_cryptography.plugin import issue_certificate
+
+    options = {
+        'common_name': 'Example.com',
+        'authority': authority,
+        'validity_start': arrow.get('2016-12-01').datetime,
+        'validity_end': arrow.get('2016-12-02').datetime
+    }
+    cert_pem, chain_cert_pem = issue_certificate(CSR_STR, options)
+    assert cert_pem
+    assert chain_cert_pem

lemur/plugins/lemur_digicert/plugin.py

@@ -1,5 +1,5 @@
 """
-.. module: lemur.plugins.lemur_digicert.digicert
+.. module: lemur.plugins.lemur_digicert.plugin
     :platform: Unix
     :synopsis: This module is responsible for communicating with the DigiCert '
     Advanced API.
@@ -22,12 +22,27 @@ from retrying import retry
 
 from flask import current_app
 
+from cryptography import x509
+
 from lemur.extensions import metrics
+from lemur.common.utils import validate_conf
 from lemur.plugins.bases import IssuerPlugin, SourcePlugin
 
 from lemur.plugins import lemur_digicert as digicert
 
 
+def log_status_code(r, *args, **kwargs):
+    """
+    Is a request hook that logs all status codes to the digicert api.
+
+    :param r:
+    :param args:
+    :param kwargs:
+    :return:
+    """
+    metrics.send('digicert_status_code_{}'.format(r.status_code), 'counter', 1)
+
+
 def signature_hash(signing_algorithm):
     """Converts Lemur's signing algorithm into a format DigiCert understands.
 
@@ -74,46 +89,88 @@ def get_issuance(options):
     :param options:
     :return:
     """
-    if not options.get('validity_end'):
-        options['validity_end'] = arrow.utcnow().replace(years=current_app.config.get('DIGICERT_DEFAULT_VALIDITY', 1))
 
-    validity_years = determine_validity_years(options['validity_end'])
-    return validity_years
+    validity_years = options.get('validity_years')
+
+    if validity_years:
+        options['validity_end'] = None
+        return options
+    else:
+        if not options.get('validity_end'):
+            options['validity_end'] = arrow.utcnow().replace(years=current_app.config.get('DIGICERT_DEFAULT_VALIDITY', 1))
+
+        options['validity_years'] = determine_validity_years(options['validity_end'])
+        return options
 
 
-def process_options(options, csr):
+def get_additional_names(options):
+    """
+    Return a list of strings to be added to a SAN certificates.
+
+    :param options:
+    :return:
+    """
+    names = []
+    # add SANs if present
+    if options.get('extensions'):
+        for san in options['extensions']['sub_alt_names']['names']:
+            if isinstance(san, x509.DNSName):
+                names.append(san.value)
+    return names
+
+
+def map_fields(options, csr):
     """Set the incoming issuer options to DigiCert fields/options.
 
     :param options:
     :param csr:
     :return: dict or valid DigiCert options
     """
+    options = get_issuance(options)
+
+    data = dict(certificate={
+        "common_name": options['common_name'],
+        "csr": csr,
+        "signature_hash":
+            signature_hash(options.get('signing_algorithm')),
+    }, organization={
+        "id": current_app.config.get("DIGICERT_ORG_ID")
+    })
+
+    data['certificate']['dns_names'] = get_additional_names(options)
+
+    if options.get('validity_end'):
+        data['custom_expiration_date'] = options['validity_end'].format('YYYY-MM-DD')
+
+    data['validity_years'] = options.get('validity_years')
+
+    return data
+
+
+def map_cis_fields(options, csr):
+    """
+    MAP issuer options to DigiCert CIS fields/options.
+
+    :param options:
+    :param csr:
+    :return:
+    """
+    options = get_issuance(options)
     data = {
-        "certificate":
-            {
-                "common_name": options['common_name'],
-                "csr": csr.decode('utf-8'),
-                "signature_hash":
-                    signature_hash(options.get('signing_algorithm')),
-            },
-        "organization":
-            {
-                "id": current_app.config.get("DIGICERT_ORG_ID")
-            },
+        "profile_name": current_app.config.get('DIGICERT_CIS_PROFILE_NAME'),
+        "common_name": options['common_name'],
+        "additional_dns_names": get_additional_names(options),
+        "csr": csr,
+        "signature_hash": signature_hash(options.get('signing_algorithm')),
+        "validity": {
+            "valid_to": options['validity_end'].format('YYYY-MM-DD')
+        },
+        "organization": {
+            "name": options['organization'],
+            "units": [options['organizational_unit']]
+        }
     }
 
-    # add SANs if present
-    if options.get('extensions', 'sub_alt_names'):
-        dns_names = []
-        for san in options['extensions']['sub_alt_names']['names']:
-            dns_names.append(san['value'])
-
-        data['certificate']['dns_names'] = dns_names
-
-    validity_years = get_issuance(options)
-    data['custom_expiration_date'] = options['validity_end'].format('YYYY-MM-DD')
-    data['validity_years'] = validity_years
-
     return data
 
 
@@ -123,30 +180,22 @@ def handle_response(response):
     :param response:
     :return:
     """
-    metrics.send('digicert_status_code_{0}'.format(response.status_code), 'counter', 1)
-
-    if response.status_code not in [200, 201, 302, 301]:
+    if response.status_code > 399:
         raise Exception(response.json()['message'])
 
     return response.json()
 
 
-def verify_configuration():
-    """Verify that needed configuration variables are set before plugin startup."""
-    if not current_app.config.get('DIGICERT_API_KEY'):
-        raise Exception("No Digicert API key found. Ensure that 'DIGICERT_API_KEY' is set in the Lemur conf.")
+def handle_cis_response(response):
+    """
+    Handle the DigiCert CIS API response and any errors it might have experienced.
+    :param response:
+    :return:
+    """
+    if response.status_code > 399:
+        raise Exception(response.json()['errors'][0]['message'])
 
-    if not current_app.config.get('DIGICERT_URL'):
-        raise Exception("No Digicert URL found. Ensure that 'DIGICERT_URL' is set in the Lemur conf.")
-
-    if not current_app.config.get('DIGICERT_ORG_ID'):
-        raise Exception("No Digicert organization ID found. Ensure that 'DIGICERT_ORG_ID' is set in Lemur conf.")
-
-    if not current_app.config.get('DIGICERT_ROOT'):
-        raise Exception("No Digicert root found. Ensure that 'DIGICERT_ROOT' is set in the Lemur conf.")
-
-    if not current_app.config.get('DIGICERT_INTERMEDIATE'):
-        raise Exception("No Digicert intermediate found. Ensure that 'DIGICERT_INTERMEDIATE is set in Lemur conf.")
+    return response.json()
 
 
 @retry(stop_max_attempt_number=10, wait_fixed=10000)
@@ -160,6 +209,21 @@ def get_certificate_id(session, base_url, order_id):
     return response_data['certificate']['id']
 
 
+@retry(stop_max_attempt_number=10, wait_fixed=10000)
+def get_cis_certificate(session, base_url, order_id):
+    """Retrieve certificate order id from Digicert API."""
+    certificate_url = '{0}/platform/cis/certificate/{1}'.format(base_url, order_id)
+    session.headers.update(
+        {'Accept': 'application/x-pem-file'}
+    )
+    response = session.get(certificate_url)
+
+    if response.status_code == 404:
+        raise Exception("Order not in issued state.")
+
+    return response.content
+
+
 class DigiCertSourcePlugin(SourcePlugin):
     """Wrap the Digicert Certifcate API."""
     title = 'DigiCert'
@@ -172,16 +236,25 @@ class DigiCertSourcePlugin(SourcePlugin):
 
     def __init__(self, *args, **kwargs):
         """Initialize source with appropriate details."""
-        verify_configuration()
+        required_vars = [
+            'DIGICERT_API_KEY',
+            'DIGICERT_URL',
+            'DIGICERT_ORG_ID',
+            'DIGICERT_ROOT',
+            'DIGICERT_INTERMEDIATE'
+        ]
+        validate_conf(current_app, required_vars)
 
         self.session = requests.Session()
         self.session.headers.update(
             {
-                'X-DC-DEVKEY': current_app.config.get('DIGICERT_API_KEY'),
+                'X-DC-DEVKEY': current_app.config['DIGICERT_API_KEY'],
                 'Content-Type': 'application/json'
             }
         )
 
+        self.session.hooks = dict(response=log_status_code)
+
         super(DigiCertSourcePlugin, self).__init__(*args, **kwargs)
 
     def get_certificates(self):
@@ -190,7 +263,6 @@ class DigiCertSourcePlugin(SourcePlugin):
 
 class DigiCertIssuerPlugin(IssuerPlugin):
     """Wrap the Digicert Issuer API."""
-
     title = 'DigiCert'
     slug = 'digicert-issuer'
     description = "Enables the creation of certificates by"
@@ -202,16 +274,26 @@ class DigiCertIssuerPlugin(IssuerPlugin):
 
     def __init__(self, *args, **kwargs):
         """Initialize the issuer with the appropriate details."""
-        verify_configuration()
+        required_vars = [
+            'DIGICERT_API_KEY',
+            'DIGICERT_URL',
+            'DIGICERT_ORG_ID',
+            'DIGICERT_ROOT',
+            'DIGICERT_INTERMEDIATE'
+        ]
+
+        validate_conf(current_app, required_vars)
 
         self.session = requests.Session()
         self.session.headers.update(
             {
-                'X-DC-DEVKEY': current_app.config.get('DIGICERT_API_KEY'),
+                'X-DC-DEVKEY': current_app.config['DIGICERT_API_KEY'],
                 'Content-Type': 'application/json'
             }
         )
 
+        self.session.hooks = dict(response=log_status_code)
+
         super(DigiCertIssuerPlugin, self).__init__(*args, **kwargs)
 
     def create_certificate(self, csr, issuer_options):
@@ -225,16 +307,20 @@ class DigiCertIssuerPlugin(IssuerPlugin):
 
         # make certificate request
         determinator_url = "{0}/services/v2/order/certificate/ssl".format(base_url)
-        data = process_options(issuer_options, csr)
+        data = map_fields(issuer_options, csr)
         response = self.session.post(determinator_url, data=json.dumps(data))
+
+        if response.status_code > 399:
+            raise Exception(response.json()['message'])
+
         order_id = response.json()['id']
 
         certificate_id = get_certificate_id(self.session, base_url, order_id)
 
-        # retrieve certificate
+        # retrieve ceqrtificate
         certificate_url = "{0}/services/v2/certificate/{1}/download/format/pem_all".format(base_url, certificate_id)
         end_entity, intermediate, root = pem.parse(self.session.get(certificate_url).content)
-        return str(end_entity), str(intermediate)
+        return "\n".join(str(end_entity).splitlines()), "\n".join(str(end_entity).splitlines())
 
     @staticmethod
     def create_authority(options):
@@ -249,3 +335,70 @@ class DigiCertIssuerPlugin(IssuerPlugin):
         """
         role = {'username': '', 'password': '', 'name': 'digicert'}
         return current_app.config.get('DIGICERT_ROOT'), "", [role]
+
+
+class DigiCertCISIssuerPlugin(IssuerPlugin):
+    """Wrap the Digicert Certificate Issuing API."""
+    title = 'DigiCert CIS'
+    slug = 'digicert-cis-issuer'
+    description = "Enables the creation of certificates by the DigiCert CIS REST API."
+    version = digicert.VERSION
+
+    author = 'Kevin Glisson'
+    author_url = 'https://github.com/netflix/lemur.git'
+
+    def __init__(self, *args, **kwargs):
+        """Initialize the issuer with the appropriate details."""
+        required_vars = [
+            'DIGICERT_CIS_API_KEY',
+            'DIGICERT_CIS_URL',
+            'DIGICERT_CIS_ROOT',
+            'DIGICERT_CIS_INTERMEDIATE',
+            'DIGICERT_CIS_PROFILE_NAME'
+        ]
+
+        validate_conf(current_app, required_vars)
+
+        self.session = requests.Session()
+        self.session.headers.update(
+            {
+                'X-DC-DEVKEY': current_app.config['DIGICERT_CIS_API_KEY'],
+                'Content-Type': 'application/json'
+            }
+        )
+
+        self.session.hooks = dict(response=log_status_code)
+
+        super(DigiCertCISIssuerPlugin, self).__init__(*args, **kwargs)
+
+    def create_certificate(self, csr, issuer_options):
+        """Create a DigiCert certificate."""
+        base_url = current_app.config.get('DIGICERT_CIS_URL')
+
+        # make certificate request
+        create_url = '{0}/platform/cis/certificate'.format(base_url)
+
+        data = map_cis_fields(issuer_options, csr)
+        response = self.session.post(create_url, data=json.dumps(data))
+        data = handle_cis_response(response)
+
+        # retrieve certificate
+        certificate_pem = get_cis_certificate(self.session, base_url, data['id'])
+
+        self.session.headers.pop('Accept')
+        end_entity = pem.parse(certificate_pem)[0]
+        return "\n".join(str(end_entity).splitlines()), current_app.config.get('DIGICERT_CIS_INTERMEDIATE')
+
+    @staticmethod
+    def create_authority(options):
+        """Create an authority.
+
+        Creates an authority, this authority is then used by Lemur to
+        allow a user to specify which Certificate Authority they want
+        to sign their certificate.
+
+        :param options:
+        :return:
+        """
+        role = {'username': '', 'password': '', 'name': 'digicert'}
+        return current_app.config.get('DIGICERT_CIS_ROOT'), "", [role]

lemur/plugins/lemur_digicert/tests/test_digicert.py

@@ -4,11 +4,13 @@ from freezegun import freeze_time
 
 from lemur.tests.vectors import CSR_STR
 
+from cryptography import x509
 
-def test_process_options(app):
-    from lemur.plugins.lemur_digicert.plugin import process_options
 
-    names = ['one.example.com', 'two.example.com', 'three.example.com']
+def test_map_fields_with_validity_end_and_start(app):
+    from lemur.plugins.lemur_digicert.plugin import map_fields
+
+    names = [u'one.example.com', u'two.example.com', u'three.example.com']
 
     options = {
         'common_name': 'example.com',
@@ -16,25 +18,92 @@ def test_process_options(app):
         'description': 'test certificate',
         'extensions': {
             'sub_alt_names': {
-                'names': [{'name_type': 'DNSName', 'value': x} for x in names]
+                'names': [x509.DNSName(x) for x in names]
             }
         },
         'validity_end': arrow.get(2017, 5, 7),
         'validity_start': arrow.get(2016, 10, 30)
     }
 
-    data = process_options(options, CSR_STR)
+    data = map_fields(options, CSR_STR)
 
     assert data == {
         'certificate': {
-            'csr': CSR_STR.decode('utf-8'),
+            'csr': CSR_STR,
             'common_name': 'example.com',
             'dns_names': names,
             'signature_hash': 'sha256'
         },
         'organization': {'id': 111111},
-        'validity_years': 1,
-        'custom_expiration_date': arrow.get(2017, 5, 7).format('YYYY-MM-DD')
+        'custom_expiration_date': arrow.get(2017, 5, 7).format('YYYY-MM-DD'),
+        'validity_years': 1
+    }
+
+
+def test_map_fields_with_validity_years(app):
+    from lemur.plugins.lemur_digicert.plugin import map_fields
+
+    names = [u'one.example.com', u'two.example.com', u'three.example.com']
+
+    options = {
+        'common_name': 'example.com',
+        'owner': 'bob@example.com',
+        'description': 'test certificate',
+        'extensions': {
+            'sub_alt_names': {
+                'names': [x509.DNSName(x) for x in names]
+            }
+        },
+        'validity_years': 2,
+        'validity_end': arrow.get(2017, 10, 30)
+    }
+
+    data = map_fields(options, CSR_STR)
+
+    assert data == {
+        'certificate': {
+            'csr': CSR_STR,
+            'common_name': 'example.com',
+            'dns_names': names,
+            'signature_hash': 'sha256'
+        },
+        'organization': {'id': 111111},
+        'validity_years': 2
+    }
+
+
+def test_map_cis_fields(app):
+    from lemur.plugins.lemur_digicert.plugin import map_cis_fields
+
+    names = [u'one.example.com', u'two.example.com', u'three.example.com']
+
+    options = {
+        'common_name': 'example.com',
+        'owner': 'bob@example.com',
+        'description': 'test certificate',
+        'extensions': {
+            'sub_alt_names': {
+                'names': [x509.DNSName(x) for x in names]
+            }
+        },
+        'organization': 'Example, Inc.',
+        'organizational_unit': 'Example Org',
+        'validity_end': arrow.get(2017, 5, 7),
+        'validity_start': arrow.get(2016, 10, 30)
+    }
+
+    data = map_cis_fields(options, CSR_STR)
+
+    assert data == {
+        'common_name': 'example.com',
+        'csr': CSR_STR,
+        'additional_dns_names': names,
+        'signature_hash': 'sha256',
+        'organization': {'name': 'Example, Inc.', 'units': ['Example Org']},
+        'validity': {
+            'valid_to': arrow.get(2017, 5, 7).format('YYYY-MM-DD')
+        },
+        'profile_name': None
     }
 
 
@@ -47,14 +116,16 @@ def test_issuance():
             'validity_start': arrow.get(2016, 10, 30)
         }
 
-        assert get_issuance(options) == 2
+        new_options = get_issuance(options)
+        assert new_options['validity_years'] == 2
 
         options = {
             'validity_end': arrow.get(2017, 5, 7),
             'validity_start': arrow.get(2016, 10, 30)
         }
 
-        assert get_issuance(options) == 1
+        new_options = get_issuance(options)
+        assert new_options['validity_years'] == 1
 
         options = {
             'validity_end': arrow.get(2020, 5, 7),

lemur/plugins/lemur_email/plugin.py

@@ -1,5 +1,5 @@
 """
-.. module: lemur.plugins.lemur_aws.aws
+.. module: lemur.plugins.lemur_email.plugin
     :platform: Unix
     :copyright: (c) 2015 by Netflix Inc., see AUTHORS for more
     :license: Apache, see LICENSE for more details.
@@ -11,14 +11,53 @@ from flask import current_app
 from flask_mail import Message
 
 from lemur.extensions import smtp_mail
+from lemur.exceptions import InvalidConfiguration
 
 from lemur.plugins.bases import ExpirationNotificationPlugin
 from lemur.plugins import lemur_email as email
 
-
 from lemur.plugins.lemur_email.templates.config import env
 
 
+def render_html(template_name, message):
+    """
+    Renders the html for our email notification.
+
+    :param template_name:
+    :param message:
+    :return:
+    """
+    template = env.get_template('{}.html'.format(template_name))
+    return template.render(dict(message=message, hostname=current_app.config.get('LEMUR_HOSTNAME')))
+
+
+def send_via_smtp(subject, body, targets):
+    """
+    Attempts to deliver email notification via SES service.
+
+    :param subject:
+    :param body:
+    :param targets:
+    :return:
+    """
+    msg = Message(subject, recipients=targets, sender=current_app.config.get("LEMUR_EMAIL"))
+    msg.body = ""  # kinda a weird api for sending html emails
+    msg.html = body
+    smtp_mail.send(msg)
+
+
+def send_via_ses(subject, body, targets):
+    """
+    Attempts to deliver email notification via SMTP.
+    :param subject:
+    :param body:
+    :param targets:
+    :return:
+    """
+    conn = boto.connect_ses()
+    conn.send_email(current_app.config.get("LEMUR_EMAIL"), subject, body, targets, format='html')
+
+
 class EmailNotificationPlugin(ExpirationNotificationPlugin):
     title = 'Email'
     slug = 'email-notification'
@@ -38,33 +77,25 @@ class EmailNotificationPlugin(ExpirationNotificationPlugin):
         },
     ]
 
+    def __init__(self, *args, **kwargs):
+        """Initialize the plugin with the appropriate details."""
+        sender = current_app.config.get('LEMUR_EMAIL_SENDER', 'ses').lower()
+
+        if sender not in ['ses', 'smtp']:
+            raise InvalidConfiguration('Email sender type {0} is not recognized.')
+
     @staticmethod
-    def send(event_type, message, targets, options, **kwargs):
-        """
-        Configures all Lemur email messaging
+    def send(notification_type, message, targets, options, **kwargs):
 
-        :param event_type:
-        :param options:
-        """
-        subject = 'Notification: Lemur'
+        subject = 'Lemur: {0} Notification'.format(notification_type.capitalize())
 
-        if event_type == 'expiration':
-            subject = 'Notification: SSL Certificate Expiration '
-
-        # jinja template depending on type
-        template = env.get_template('{}.html'.format(event_type))
-        body = template.render(dict(messages=message, hostname=current_app.config.get('LEMUR_HOSTNAME')))
+        data = {'options': options, 'certificates': message}
+        body = render_html(notification_type, data)
 
         s_type = current_app.config.get("LEMUR_EMAIL_SENDER", 'ses').lower()
+
         if s_type == 'ses':
-            conn = boto.connect_ses()
-            conn.send_email(current_app.config.get("LEMUR_EMAIL"), subject, body, targets, format='html')
+            send_via_ses(subject, body, targets)
 
         elif s_type == 'smtp':
-            msg = Message(subject, recipients=targets)
-            msg.body = ""  # kinda a weird api for sending html emails
-            msg.html = body
-            smtp_mail.send(msg)
-
-        else:
-            current_app.logger.error("No mail carrier specified, notification emails were not able to be sent!")
+            send_via_smtp(subject, body, targets)

lemur/plugins/lemur_email/templates/config.py

@@ -2,6 +2,8 @@ import os
 import arrow
 from jinja2 import Environment, FileSystemLoader
 
+from lemur.plugins.utils import get_plugin_option
+
 loader = FileSystemLoader(searchpath=os.path.dirname(os.path.realpath(__file__)))
 env = Environment(loader=loader)
 
@@ -9,4 +11,15 @@ env = Environment(loader=loader)
 def human_time(time):
     return arrow.get(time).format('dddd, MMMM D, YYYY')
 
+
+def interval(options):
+    return get_plugin_option('interval', options)
+
+
+def unit(options):
+    return get_plugin_option('unit', options)
+
+
 env.filters['time'] = human_time
+env.filters['interval'] = interval
+env.filters['unit'] = unit

lemur/plugins/lemur_email/templates/expiration.html

@@ -43,7 +43,7 @@
                                 <tr>
                                     <td width="32px"></td>
                                     <td style="font-family:Roboto-Regular,Helvetica,Arial,sans-serif;font-size:24px;color:#ffffff;line-height:1.25">
-                                       Your certificate(s) are expiring!
+                                       Your certificate(s) are expiring in {{ message.options | interval }} {{ message.options | unit }}!
                                     </td>
                                     <td width="32px"></td>
                                 </tr>
@@ -79,18 +79,18 @@
                                                     <table border="0" cellspacing="0" cellpadding="0"
                                                            style="margin-top:48px;margin-bottom:48px">
                                                         <tbody>
-                                                        {%  for message in messages %}
+                                                        {%  for certificate in message.certificates %}
                                                             <tr valign="middle">
                                                                 <td width="32px"></td>
                                                                 <td width="16px"></td>
                                                                 <td style="line-height:1.2">
-                                                                    <span style="font-family:Roboto-Regular,Helvetica,Arial,sans-serif;font-size:20px;color:#202020">{{ message.name }}</span>
+                                                                    <span style="font-family:Roboto-Regular,Helvetica,Arial,sans-serif;font-size:20px;color:#202020">{{ certificate.name }}</span>
                                                                     <br>
                                                                     <span style="font-family:Roboto-Regular,Helvetica,Arial,sans-serif;font-size:13px;color:#727272">
-                                                                        {{ message.endpoints | length }} Endpoints
-                                                                        <br>{{ message.owner }}
-                                                                        <br>{{ message.not_after | time }}
-                                                                        <a href="https://{{ hostname }}/#/certificates/{{ message.name }}" target="_blank">Details</a>
+                                                                        {{ certificate.endpoints | length }} Endpoints
+                                                                        <br>{{ certificate.owner }}
+                                                                        <br>{{ certificate.validityEnd | time }}
+                                                                        <a href="https://{{ hostname }}/#/certificates/{{ certificate.name }}" target="_blank">Details</a>
                                                                     </span>
                                                                 </td>
                                                             </tr>
@@ -146,13 +146,13 @@
                             <table style="font-family:Roboto-Regular,Helvetica,Arial,sans-serif;font-size:10px;color:#666666;line-height:18px;padding-bottom:10px">
                                 <tbody>
                                 <tr>
-                                    <td>You received this mandatory email service announcement to update you about
+                                    <td>You received this mandatory email announcement to update you about
                                         important changes to your <span class="il">TLS certificate</span>.
                                     </td>
                                 </tr>
                                 <tr>
                                     <td>
-                                        <div style="direction:ltr;text-align:left">© 2015 <span class="il">Lemur</span></div>
+                                        <div style="direction:ltr;text-align:left">© 2016 <span class="il">Lemur</span></div>
                                     </td>
                                 </tr>
                                 </tbody>

lemur/plugins/lemur_email/templates/issued.html

@@ -0,0 +1,168 @@
+<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"
+        "http://www.w3.org/TR/html4/loose.dtd">
+<html lang="en">
+<head>
+    <meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
+    <meta name="viewport" content="initial-scale=1.0">    <!-- So that mobile webkit will display zoomed in -->
+    <meta name="format-detection" content="telephone=no"> <!-- disable auto telephone linking in iOS -->
+
+    <title>Lemur</title>
+</head>
+
+<div style="margin:0;padding:0" bgcolor="#FFFFFF">
+    <table width="100%" height="100%" style="min-width:348px" border="0" cellspacing="0" cellpadding="0">
+        <tbody>
+        <tr height="32px"></tr>
+        <tr align="center">
+            <td width="32px"></td>
+            <td>
+                <table border="0" cellspacing="0" cellpadding="0" style="max-width:600px">
+                    <tbody>
+                    <tr>
+                        <td>
+                            <table width="100%" border="0" cellspacing="0" cellpadding="0">
+                                <tbody>
+                                <tr>
+                                    <td align="left" style="font-family:Roboto-Regular,Helvetica,Arial,sans-serif;font-size:35px;color:#727272; line-height:1.5">
+                                        Lemur
+                                    </td>
+                                </tr>
+                                </tbody>
+                            </table>
+                        </td>
+                    </tr>
+                    <tr height="16"></tr>
+                    <tr>
+                        <td>
+                            <table bgcolor="#F44336" width="100%" border="0" cellspacing="0" cellpadding="0"
+                                   style="min-width:332px;max-width:600px;border:1px solid #e0e0e0;border-bottom:0;border-top-left-radius:3px;border-top-right-radius:3px">
+                                <tbody>
+                                <tr>
+                                    <td height="72px" colspan="3"></td>
+                                </tr>
+                                <tr>
+                                    <td width="32px"></td>
+                                    <td style="font-family:Roboto-Regular,Helvetica,Arial,sans-serif;font-size:24px;color:#ffffff;line-height:1.25">
+                                       Your certificate(s) are being rotated!
+                                    </td>
+                                    <td width="32px"></td>
+                                </tr>
+                                <tr>
+                                    <td height="18px" colspan="3"></td>
+                                </tr>
+                                </tbody>
+                            </table>
+                        </td>
+                    </tr>
+                    <tr>
+                        <td>
+                            <table bgcolor="#3681E4" width="100%" border="0" cellspacing="0" cellpadding="0"
+                                   style="min-width:332px;max-width:600px;border:1px solid #f0f0f0;border-bottom:1px solid #c0c0c0;border-top:0;border-bottom-left-radius:3px;border-bottom-right-radius:3px">
+                                <tbody>
+                                <tr height="16px">
+                                    <td width="32px" rowspan="3"></td>
+                                    <td></td>
+                                    <td width="32px" rowspan="3"></td>
+                                </tr>
+                                <tr>
+                                    <td>
+                                        <table style="min-width:300px" border="0" cellspacing="0" cellpadding="0">
+                                            <tbody>
+                                            <tr>
+                                                <td style="font-family:Roboto-Regular,Helvetica,Arial,sans-serif;font-size:13px;color:#202020;line-height:1.5">
+                                                    Hi,
+                                                </td>
+                                            </tr>
+                                            <tr>
+                                                <td style="font-family:Roboto-Regular,Helvetica,Arial,sans-serif;font-size:13px;color:#202020;line-height:1.5">
+                                                    <br>Your certificate is ready.
+                                                    <table border="0" cellspacing="0" cellpadding="0"
+                                                           style="margin-top:48px;margin-bottom:48px">
+                                                        <tbody>
+                                                        {%  for message in messages %}
+                                                            <tr valign="middle">
+                                                                <td width="32px"></td>
+                                                                <td width="16px"></td>
+                                                                <td style="line-height:1.2">
+                                                                    <span style="font-family:Roboto-Regular,Helvetica,Arial,sans-serif;font-size:20px;color:#202020">{{ message.name }}</span>
+                                                                    <br>
+                                                                    <span style="font-family:Roboto-Regular,Helvetica,Arial,sans-serif;font-size:13px;color:#727272">
+                                                                        {{ message.body }}
+                                                                    </span>
+                                                                    <br>
+                                                                    <span style="font-family:Roboto-Regular,Helvetica,Arial,sans-serif;font-size:13px;color:#727272">
+                                                                        <br>{{ message.owner }}
+                                                                        <br>{{ message.not_after | time }}
+                                                                        <a href="https://{{ hostname }}/#/certificates/{{ message.name }}" target="_blank">Details</a>
+                                                                    </span>
+                                                                </td>
+                                                            </tr>
+                                                                {%  if not loop.last %}
+                                                                <tr valign="middle">
+                                                                    <td width="32px" height="24px"></td>
+                                                                </tr>
+                                                                {% endif %}
+                                                        {%  endfor %}
+                                                        </tbody>
+                                                    </table>
+                                                </td>
+                                            </tr>
+                                            <tr>
+                                            </tr>
+                                            <tr>
+                                                <td style="font-family:Roboto-Regular,Helvetica,Arial,sans-serif;font-size:13px;color:#202020;line-height:1.5">
+                                                    <br>Best,<br><span class="il">Lemur</span>
+                                                </td>
+                                            </tr>
+                                            <tr height="16px"></tr>
+                                            <tr>
+                                                <td>
+                                                    <table style="font-family:Roboto-Regular,Helvetica,Arial,sans-serif;font-size:12px;color:#b9b9b9;line-height:1.5">
+                                                        <tbody>
+                                                        <tr>
+                                                            <td>*All times are in UTC<br></td>
+                                                        </tr>
+                                                        </tbody>
+                                                    </table>
+                                                </td>
+                                            </tr>
+                                            </tbody>
+                                        </table>
+                                    </td>
+                                </tr>
+                                <tr height="32px"></tr>
+                                </tbody>
+                            </table>
+                        </td>
+                    </tr>
+                    <tr height="16"></tr>
+                    <tr>
+                        <td style="max-width:600px;font-family:Roboto-Regular,Helvetica,Arial,sans-serif;font-size:10px;color:#bcbcbc;line-height:1.5"></td>
+                    </tr>
+                    <tr>
+                        <td>
+                            <table style="font-family:Roboto-Regular,Helvetica,Arial,sans-serif;font-size:10px;color:#666666;line-height:18px;padding-bottom:10px">
+                                <tbody>
+                                <tr>
+                                    <td>You received this mandatory email announcement to update you about
+                                        important changes to your <span class="il">TLS certificate</span>.
+                                    </td>
+                                </tr>
+                                <tr>
+                                    <td>
+                                        <div style="direction:ltr;text-align:left">© 2016 <span class="il">Lemur</span></div>
+                                    </td>
+                                </tr>
+                                </tbody>
+                            </table>
+                        </td>
+                    </tr>
+                    </tbody>
+                </table>
+            </td>
+            <td width="32px"></td>
+        </tr>
+        <tr height="32px"></tr>
+        </tbody>
+    </table>
+</div>

lemur/plugins/lemur_email/templates/rotation.html

@@ -0,0 +1,222 @@
+<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"
+        "http://www.w3.org/TR/html4/loose.dtd">
+<html lang="en">
+<head>
+    <meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
+    <meta name="viewport" content="initial-scale=1.0">    <!-- So that mobile webkit will display zoomed in -->
+    <meta name="format-detection" content="telephone=no"> <!-- disable auto telephone linking in iOS -->
+
+    <title>Lemur</title>
+</head>
+
+<div style="margin:0;padding:0" bgcolor="#FFFFFF">
+    <table width="100%" height="100%" style="min-width:348px" border="0" cellspacing="0" cellpadding="0">
+        <tbody>
+        <tr height="32px"></tr>
+        <tr align="center">
+            <td width="32px"></td>
+            <td>
+                <table border="0" cellspacing="0" cellpadding="0" style="max-width:600px">
+                    <tbody>
+                    <tr>
+                        <td>
+                            <table width="100%" border="0" cellspacing="0" cellpadding="0">
+                                <tbody>
+                                <tr>
+                                    <td align="left" style="font-family:Roboto-Regular,Helvetica,Arial,sans-serif;font-size:35px;color:#727272; line-height:1.5">
+                                        Lemur
+                                    </td>
+                                </tr>
+                                </tbody>
+                            </table>
+                        </td>
+                    </tr>
+                    <tr height="16"></tr>
+                    <tr>
+                        <td>
+                            <table bgcolor="#3681E4" width="100%" border="0" cellspacing="0" cellpadding="0"
+                                   style="min-width:332px;max-width:600px;border:1px solid #e0e0e0;border-bottom:0;border-top-left-radius:3px;border-top-right-radius:3px">
+                                <tbody>
+                                <tr>
+                                    <td height="72px" colspan="3"></td>
+                                </tr>
+                                <tr>
+                                    <td width="32px"></td>
+                                    <td style="font-family:Roboto-Regular,Helvetica,Arial,sans-serif;font-size:24px;color:#ffffff;line-height:1.25">
+                                       Your certificate has been rotated!
+                                    </td>
+                                    <td width="32px"></td>
+                                </tr>
+                                <tr>
+                                    <td height="18px" colspan="3"></td>
+                                </tr>
+                                </tbody>
+                            </table>
+                        </td>
+                    </tr>
+                    <tr>
+                        <td>
+                            <table width="100%" border="0" cellspacing="0" cellpadding="0"
+                                   style="min-width:332px;max-width:600px;border:1px solid #f0f0f0;border-bottom:1px solid #c0c0c0;border-top:0;border-bottom-left-radius:3px;border-bottom-right-radius:3px">
+                                <tbody>
+                                <tr height="16px">
+                                    <td width="32px" rowspan="3"></td>
+                                    <td></td>
+                                    <td width="32px" rowspan="3"></td>
+                                </tr>
+                                <tr>
+                                    <td>
+                                        <table style="min-width:300px" border="0" cellspacing="0" cellpadding="0">
+                                            <tbody>
+                                            <tr>
+                                                <td style="font-family:Roboto-Regular,Helvetica,Arial,sans-serif;font-size:13px;color:#202020;line-height:1.5">
+                                                    Hi,
+                                                    <br>This is a Lemur certificate rotation notice. Your certificate has been re-issued and re-provisioned.
+                                                </td>
+                                            </tr>
+                                            <tr>
+                                                <td style="font-family:Roboto-Regular,Helvetica,Arial,sans-serif;font-size:13px;color:#202020;line-height:1.5">
+                                                    <table border="0" cellspacing="0" cellpadding="0"
+                                                           style="margin-top:48px;margin-bottom:48px">
+                                                        <tbody>
+                                                            <tr valign="middle">
+                                                                <td width="32px"></td>
+                                                                <td width="16px"></td>
+                                                                <td style="line-height:1.2">
+                                                                    <span style="font-family:Roboto-Regular,Helvetica,Arial,sans-serif;font-size:20px;color:#202020">{{ certificate.name }}</span>
+                                                                    <br>
+                                                                    <span style="font-family:Roboto-Regular,Helvetica,Arial,sans-serif;font-size:13px;color:#727272">
+                                                                        <br>{{ certificate.owner }}
+                                                                        <br>{{ certificate.validityEnd | time }}
+                                                                        <a href="https://{{ hostname }}/#/certificates/{{ certificate.name }}" target="_blank">Details</a>
+                                                                    </span>
+                                                                </td>
+                                                            </tr>
+                                                        </tbody>
+                                                    </table>
+                                                </td>
+                                            </tr>
+                                            <tr>
+                                                <td style="font-family:Roboto-Regular,Helvetica,Arial,sans-serif;font-size:15px;color:#202020;line-height:1.5">
+                                                    Replaced by:
+                                                </td>
+                                            </tr>
+                                            <tr>
+                                                <td style="font-family:Roboto-Regular,Helvetica,Arial,sans-serif;font-size:13px;color:#202020;line-height:1.5">
+                                                    <table border="0" cellspacing="0" cellpadding="0"
+                                                           style="margin-top:48px;margin-bottom:48px">
+                                                        <tbody>
+                                                        <tr valign="middle">
+                                                            <td width="32px"></td>
+                                                            <td width="16px"></td>
+                                                            <td style="line-height:1.2">
+                                                                <span style="font-family:Roboto-Regular,Helvetica,Arial,sans-serif;font-size:20px;color:#202020">{{ certificate.replacedBy[0].name }}</span>
+                                                                <br>
+                                                                <span style="font-family:Roboto-Regular,Helvetica,Arial,sans-serif;font-size:13px;color:#727272">
+                                                                        <br>{{ certificate.replacedBy[0].owner }}
+                                                                        <br>{{ certificate.replacedBy[0].validityEnd | time }}
+                                                                        <a href="https://{{ hostname }}/#/certificates/{{ certificate.replacedBy[0].name }}" target="_blank">Details</a>
+                                                                    </span>
+                                                            </td>
+                                                        </tr>
+                                                        </tbody>
+                                                    </table>
+                                                </td>
+                                            </tr>
+                                            <tr>
+                                                <td style="font-family:Roboto-Regular,Helvetica,Arial,sans-serif;font-size:15px;color:#202020;line-height:1.5">
+                                                    Endpoints Rotated:
+                                                </td>
+                                            </tr>
+                                            <tr>
+                                                <td style="font-family:Roboto-Regular,Helvetica,Arial,sans-serif;font-size:13px;color:#202020;line-height:1.5">
+                                                    <table border="0" cellspacing="0" cellpadding="0"
+                                                           style="margin-top:48px;margin-bottom:48px">
+                                                        <tbody>
+                                                        {%  for endpoint in certificate.endpoints %}
+                                                            <tr valign="middle">
+                                                                <td width="32px"></td>
+                                                                <td width="16px"></td>
+                                                                <td style="line-height:1.2">
+                                                                    <span style="font-family:Roboto-Regular,Helvetica,Arial,sans-serif;font-size:20px;color:#202020">{{ endpoint.name }}</span>
+                                                                    <br>
+                                                                    <span style="font-family:Roboto-Regular,Helvetica,Arial,sans-serif;font-size:13px;color:#727272">
+                                                                        <br>{{ endpoint.dnsname }} | {{ endpoint.port }}
+                                                                        <a href="https://{{ hostname }}/#/endpoints/{{ endpoint.name }}" target="_blank">Details</a>
+                                                                    </span>
+                                                                </td>
+                                                            </tr>
+                                                                {%  if not loop.last %}
+                                                                <tr valign="middle">
+                                                                    <td width="32px" height="24px"></td>
+                                                                </tr>
+                                                                {% endif %}
+                                                        {%  endfor %}
+                                                        </tbody>
+                                                    </table>
+                                                </td>
+                                            </tr>
+                                            <tr>
+                                                <td style="font-family:Roboto-Regular,Helvetica,Arial,sans-serif;font-size:13px;color:#202020;line-height:1.5">
+                                                    If any of the above information looks incorrect, or if the endpoints mentioned are no longer in use please reach out to {{ security_email }}.</span>
+                                                </td>
+                                            </tr>
+                                            <tr>
+                                            </tr>
+                                            <tr>
+                                                <td style="font-family:Roboto-Regular,Helvetica,Arial,sans-serif;font-size:13px;color:#202020;line-height:1.5">
+                                                    <br>Best,<br><span class="il">Lemur</span>
+                                                </td>
+                                            </tr>
+                                            <tr height="16px"></tr>
+                                            <tr>
+                                                <td>
+                                                    <table style="font-family:Roboto-Regular,Helvetica,Arial,sans-serif;font-size:12px;color:#b9b9b9;line-height:1.5">
+                                                        <tbody>
+                                                        <tr>
+                                                            <td>*All times are in UTC<br></td>
+                                                        </tr>
+                                                        </tbody>
+                                                    </table>
+                                                </td>
+                                            </tr>
+                                            </tbody>
+                                        </table>
+                                    </td>
+                                </tr>
+                                <tr height="32px"></tr>
+                                </tbody>
+                            </table>
+                        </td>
+                    </tr>
+                    <tr height="16"></tr>
+                    <tr>
+                        <td style="max-width:600px;font-family:Roboto-Regular,Helvetica,Arial,sans-serif;font-size:10px;color:#bcbcbc;line-height:1.5"></td>
+                    </tr>
+                    <tr>
+                        <td>
+                            <table style="font-family:Roboto-Regular,Helvetica,Arial,sans-serif;font-size:10px;color:#666666;line-height:18px;padding-bottom:10px">
+                                <tbody>
+                                <tr>
+                                    <td>You received this mandatory email announcement to update you about
+                                        important changes to your <span class="il">TLS certificate</span>.
+                                    </td>
+                                </tr>
+                                <tr>
+                                    <td>
+                                        <div style="direction:ltr;text-align:left">© 2016 <span class="il">Lemur</span></div>
+                                    </td>
+                                </tr>
+                                </tbody>
+                            </table>
+                        </td>
+                    </tr>
+                    </tbody>
+                </table>
+            </td>
+            <td width="32px"></td>
+        </tr>
+        <tr height="32px"></tr>
+        </tbody>
+    </table>
+</div>

lemur/plugins/lemur_email/tests/conftest.py

@@ -0,0 +1 @@
+from lemur.tests.conftest import *  # noqa

lemur/plugins/lemur_email/tests/test_email.py

@@ -1,12 +1,37 @@
+import os
 from lemur.plugins.lemur_email.templates.config import env
 
+from lemur.tests.factories import CertificateFactory
 
-def test_render():
-    messages = [{
-        'name': 'a-really-really-long-certificate-name',
-        'owner': 'bob@example.com',
-        'not_after': '2015-12-14 23:59:59'
-    }] * 10
+dir_path = os.path.dirname(os.path.realpath(__file__))
+
+
+def test_render(certificate, endpoint):
+    from lemur.certificates.schemas import certificate_notification_output_schema
+
+    new_cert = CertificateFactory()
+    new_cert.replaces.append(certificate)
+
+    data = {
+        'certificates': [certificate_notification_output_schema.dump(certificate).data],
+        'options': [{'name': 'interval', 'value': 10}, {'name': 'unit', 'value': 'days'}]
+    }
 
     template = env.get_template('{}.html'.format('expiration'))
-    body = template.render(dict(messages=messages, hostname='lemur.test.example.com'))
+
+    with open(os.path.join(dir_path, 'expiration-rendered.html'), 'w') as f:
+        body = template.render(dict(message=data, hostname='lemur.test.example.com'))
+        f.write(body)
+
+    template = env.get_template('{}.html'.format('rotation'))
+
+    certificate.endpoints.append(endpoint)
+
+    with open(os.path.join(dir_path, 'rotation-rendered.html'), 'w') as f:
+        body = template.render(
+            dict(
+                certificate=certificate_notification_output_schema.dump(certificate).data,
+                hostname='lemur.test.example.com'
+            )
+        )
+        f.write(body)

lemur/plugins/lemur_java/plugin.py

@@ -102,10 +102,14 @@ def create_keystore(cert, chain, jks_tmp, key, alias, passphrase):
 
     if isinstance(key, bytes):
         key = key.decode('utf-8')
+
     # Create PKCS12 keystore from private key and public certificate
     with mktempfile() as cert_tmp:
         with open(cert_tmp, 'w') as f:
-            f.writelines([key.strip() + "\n", cert.strip() + "\n", chain.strip() + "\n"])
+            if chain:
+                f.writelines([key.strip() + "\n", cert.strip() + "\n", chain.strip() + "\n"])
+            else:
+                f.writelines([key.strip() + "\n", cert.strip() + "\n"])
 
         with mktempfile() as p12_tmp:
             run_process([

lemur/plugins/lemur_java/tests/test_java.py

@@ -1,9 +1,9 @@
 import pytest
-import six
 
 from lemur.tests.vectors import INTERNAL_CERTIFICATE_A_STR, INTERNAL_PRIVATE_KEY_A_STR
 
 
+@pytest.mark.skip(reason="no way of currently testing this")
 def test_export_truststore(app):
     from lemur.plugins.base import plugins
 
@@ -16,6 +16,7 @@ def test_export_truststore(app):
     assert isinstance(actual[2], bytes)
 
 
+@pytest.mark.skip(reason="no way of currently testing this")
 def test_export_truststore_default_password(app):
     from lemur.plugins.base import plugins
 
@@ -24,10 +25,11 @@ def test_export_truststore_default_password(app):
     actual = p.export(INTERNAL_CERTIFICATE_A_STR, "", "", options)
 
     assert actual[0] == 'jks'
-    assert isinstance(actual[1], six.string_types)
+    assert isinstance(actual[1], str)
     assert isinstance(actual[2], bytes)
 
 
+@pytest.mark.skip(reason="no way of currently testing this")
 def test_export_keystore(app):
     from lemur.plugins.base import plugins
 
@@ -44,6 +46,7 @@ def test_export_keystore(app):
     assert isinstance(actual[2], bytes)
 
 
+@pytest.mark.skip(reason="no way of currently testing this")
 def test_export_keystore_default_password(app):
     from lemur.plugins.base import plugins
 
@@ -56,5 +59,5 @@ def test_export_keystore_default_password(app):
     actual = p.export(INTERNAL_CERTIFICATE_A_STR, "", INTERNAL_PRIVATE_KEY_A_STR, options)
 
     assert actual[0] == 'jks'
-    assert isinstance(actual[1], six.string_types)
+    assert isinstance(actual[1], str)
     assert isinstance(actual[2], bytes)