6.0 release. (#1033)

Per RFC 5280 section 6.3.3 (k):
https://tools.ietf.org/html/rfc5280#section-6.3.3

.coveragerc

@@ -1,4 +1,5 @@
 [report]
 include = lemur/*.py
 omit = lemur/migrations/*
+       lemur/tests/*
 

.gitignore

@@ -13,6 +13,7 @@
 MANIFEST
 test.conf
 pip-log.txt
+package-lock.json
 /htmlcov
 /cover
 /build
@@ -27,5 +28,7 @@ pip-log.txt
 docs/_build
 .editorconfig
 .idea
-test.conf
+lemur/tests/tmp
-lemur/tests/tmp
+
+/lemur/plugins/lemur_email/tests/expiration-rendered.html
+/lemur/plugins/lemur_email/tests/rotation-rendered.html

.pre-commit-config.yaml

@@ -1,10 +1,10 @@
 -   repo: git://github.com/pre-commit/pre-commit-hooks
-    sha: 18d7035de5388cc7775be57f529c154bf541aab9
+    sha: v0.9.1
     hooks:
     -   id: trailing-whitespace
     -   id: flake8
     -   id: check-merge-conflict
 -   repo: git://github.com/pre-commit/mirrors-jshint
-    sha: e72140112bdd29b18b0c8257956c896c4c3cebcb
+    sha: v2.9.5
     hooks:
     -   id: jshint

.travis.yml

@@ -34,9 +34,11 @@ before_script:
 
 install:
   - pip install coveralls
+  - pip install bandit
 
 script:
   - make test
+  - bandit -r . -ll -ii -x lemur/tests/,docs
 
 after_success:
   - coveralls

CHANGELOG.rst

@@ -1,6 +1,50 @@
 Changelog
 =========
 
+0.6 - `2018-01-02`
+~~~~~~~~~~~~~~~~~~
+
+Happy Holidays! This is a big release with lots of bug fixes and features. Below are the highlights and are not exhaustive.
+
+
+Features:
+
+* Per-certificate rotation policies, requires a database migration. The default rotation policy for all certificates.
+is 30 days. Every certificate will gain a policy regardless of if auto-rotation is used.
+* Adds per-user API Keys, allows users to issue multiple long-lived API tokens with the same permission as the user creating them.
+* Adds the ability to revoke certificates from the Lemur UI/API, this is currently only supported for the digicert CIS and cfssl plugins.
+* Allow destinations to support an export function. Useful for file system destinations e.g. S3 to specify the export plugin you wish to run before being sent to the destination.
+* Adds support for uploading certificates to Cloudfront.
+* Re-worked certificate metadata pane for improved readability.
+* Adds support for LDAP user authentication
+
+Bugs:
+
+* Closed `#767 <https://github.com/Netflix/lemur/issues/767>`_ - Fixed issue with login redirect loop.
+* Closed `#792 <https://github.com/Netflix/lemur/issues/792>`_ - Fixed an issue with a unique constraint was violated when replacing certificates.
+* Closed `#752 <https://github.com/Netflix/lemur/issues/752>`_ - Fixed an internal server error when validating notification units.
+* Closed `#684 <https://github.com/Netflix/lemur/issues/684>`_ - Fixed migration failure when null values encountered.
+* Closes `#661 <https://github.com/Netflix/lemur/issues/661>`_ - Fixed an issue where default values were missing during clone operations.
+
+
+Special thanks to all who helped with this release, notably:
+
+- intgr
+- SecurityInsanity
+- johanneslange
+- RickB17
+- pr8kerl
+- bunjiboys
+
+See the full list of issues closed in `0.6 <https://github.com/Netflix/lemur/milestone/5>`_.
+
+Upgrading
+---------
+
+.. note:: This release will need a migration change. Please follow the `documentation <https://lemur.readthedocs.io/en/latest/administration.html#upgrading-lemur>`_ to upgrade Lemur.
+
+
+
 0.5 - `2016-04-08`
 ~~~~~~~~~~~~~~~~~~
 
@@ -13,19 +57,15 @@ Other Highlights:
 * Closed `#501 <https://github.com/Netflix/lemur/issues/501>`_ - Endpoint resource as now kept in sync via an
 expiration mechanism. Such that non-existant endpoints gracefully fall out of Lemur. Certificates are never
 removed from Lemur.
-
 * Closed `#551 <https://github.com/Netflix/lemur/pull/551>`_ - Added the ability to create a 4096 bit key during certificate
 creation. Closed `#528 <https://github.com/Netflix/lemur/pull/528>`_ to ensure that issuer plugins supported the new 4096 bit keys.
-
 * Closed `#566 <https://github.com/Netflix/lemur/issues/566>`_ - Fixed an issue changing the notification status for  certificates
 without private keys.
-
 * Closed `#594 <https://github.com/Netflix/lemur/issues/594>`_ - Added `replaced` field indicating if a certificate has been superseded.
-
 * Closed `#602 <https://github.com/Netflix/lemur/issues/602>`_ - AWS plugin added support for ALBs for endpoint tracking.
 
 
-Special thanks to all who helped with with this release, notably:
+Special thanks to all who helped with this release, notably:
 
 - RcRonco
 - harmw
@@ -87,7 +127,7 @@ Issuer Plugin Owners
 --------------------
 
 This release may break your plugins, the keys in `issuer_options` have been changed from `camelCase` to `under_score`.
-This change was made to break a undue reliance on downstream options maintains a more pythonic naming convention. Renaming
+This change was made to break an undue reliance on downstream options maintains a more pythonic naming convention. Renaming
 these keys should be fairly trivial, additionally pull requests have been submitted to affected plugins to help ease the transition.
 
 .. note:: This change only affects issuer plugins and does not affect any other types of plugins.
@@ -97,10 +137,10 @@ these keys should be fairly trivial, additionally pull requests have been submit
     stricter input validation and better error messages when validation fails.
 * Closed `#146 <https://github.com/Netflix/lemur/issues/146>`_ - Moved authority type to first pane of authority creation wizard.
 * Closed `#147 <https://github.com/Netflix/lemur/issues/147>`_ - Added and refactored the relationship between authorities and their
-    root certificates. Displays the certificates (and chains) next the the authority in question.
+    root certificates. Displays the certificates (and chains) next to the authority in question.
 * Closed `#199 <https://github.com/Netflix/lemur/issues/199>`_ - Ensures that the dates submitted to Lemur during authority and
     certificate creation are actually dates.
-* Closed `#230 <https://github.com/Netflix/lemur/issues/230>`_ - Migrated authority dropdown to a ui-select based dropdown, this
+* Closed `#230 <https://github.com/Netflix/lemur/issues/230>`_ - Migrated authority dropdown to an ui-select based dropdown, this
     should be easier to determine what authorities are available and when an authority has actually been selected.
 * Closed `#254 <https://github.com/Netflix/lemur/issues/254>`_ - Forces certificate names to be generally unique. If a certificate name
     (generated or otherwise) is found to be a duplicate we increment by appending a counter.
@@ -112,7 +152,7 @@ these keys should be fairly trivial, additionally pull requests have been submit
     via the UI.
 * Closed `#289 <https://github.com/Netflix/lemur/issues/289>`_ - Fixed and issue where intermediates were not being properly exported.
 * Closed `#315 <https://github.com/Netflix/lemur/issues/315>`_ - Made how roles are associated with certificates and authorities much more
-    explict, including adding the ability to add roles directly to certificates and authorities on creation.
+    explicit, including adding the ability to add roles directly to certificates and authorities on creation.
 
 
 
@@ -162,6 +202,6 @@ these keys should be fairly trivial, additionally pull requests have been submit
 0.1.5 - 2015-10-26
 ~~~~~~~~~~~~~~~~~~
 
-* **SECURITY ISSUE**: Switched from use a AES static key to Fernet encryption.
+* **SECURITY ISSUE**: Switched from use an AES static key to Fernet encryption.
   Affects all versions prior to 0.1.5. If upgrading this will require a data migration.
-  see: `Upgrading Lemur <https://lemur.readthedocs.com/adminstration#UpgradingLemur>`_
+  see: `Upgrading Lemur <https://lemur.readthedocs.io/administration#UpgradingLemur>`_

Dockerfile

@@ -0,0 +1,13 @@
+FROM python:3.5
+RUN apt-get update
+RUN apt-get install -y make python-software-properties curl
+RUN curl -sL https://deb.nodesource.com/setup_7.x | bash -
+RUN apt-get update
+RUN apt-get install -y nodejs libldap2-dev libsasl2-dev libldap2-dev libssl-dev
+RUN pip install -U setuptools
+RUN pip install coveralls bandit
+WORKDIR /app
+COPY . /app/
+RUN pip install -e .
+RUN pip install "file://`pwd`#egg=lemur[dev]"
+RUN pip install "file://`pwd`#egg=lemur[tests]"

README.rst

@@ -5,8 +5,8 @@ Lemur
    :alt: Join the chat at https://gitter.im/Netflix/lemur
    :target: https://gitter.im/Netflix/lemur?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge
 
-.. image:: https://readthedocs.org/projects/lemur/badge/?version=latest
+.. image:: https://readthedocs.io/projects/lemur/badge/?version=latest
-    :target: https://lemur.readthedocs.org
+    :target: https://lemur.readthedocs.io
     :alt: Latest Docs
 
 .. image:: https://img.shields.io/badge/NetflixOSS-active-brightgreen.svg
@@ -25,7 +25,7 @@ Project resources
 =================
 
 - `Lemur Blog Post <http://techblog.netflix.com/2015/09/introducing-lemur.html>`_
-- `Documentation <http://lemur.readthedocs.org/>`_
+- `Documentation <http://lemur.readthedocs.io/>`_
 - `Source code <https://github.com/netflix/lemur>`_
 - `Issue tracker <https://github.com/netflix/lemur/issues>`_
 - `Docker <https://github.com/Netflix/lemur-docker>`_

bower.json

@@ -31,7 +31,7 @@
     "font-awesome": "~4.5.0",
     "lodash": "~4.0.1",
     "underscore": "~1.8.3",
-    "angular-smart-table": "~2.1.6",
+    "angular-smart-table": "2.1.8",
     "angular-strap": ">= 2.2.2",
     "angular-underscore": "^0.5.0",
     "angular-translate": "^2.9.0",

docker-compose.yml

@@ -0,0 +1,18 @@
+---
+version: '2.0'
+services:
+  test:
+    build: .
+    volumes:
+      - ".:/app"
+    links:
+      - postgres
+    command: make test
+    environment:
+      SQLALCHEMY_DATABASE_URI: postgresql://lemur:lemur@postgres:5432/lemur
+
+  postgres:
+    image: postgres:9.4
+    environment:
+      POSTGRES_USER: lemur
+      POSTGRES_PASSWORD: lemur

docs/administration.rst

@@ -7,6 +7,10 @@ Configuration
     that allow you to encrypt files at rest and decrypt them when it's time for deployment. See :ref:`Credential Management <CredentialManagement>`
     for more information.
 
+.. note::
+    All configuration values are python strings unless otherwise noted.
+
+
 Basic Configuration
 -------------------
 
@@ -24,14 +28,14 @@ Basic Configuration
 
         LOG_FILE = "/logs/lemur/lemur-test.log"
 
-.. data:: debug
+.. data:: DEBUG
     :noindex:
 
     Sets the flask debug flag to true (if supported by the webserver)
 
     ::
 
-        debug = False
+        DEBUG = False
 
     .. warning::
         This should never be used in a production environment as it exposes Lemur to
@@ -66,11 +70,24 @@ Basic Configuration
 
         Specifies whether to allow certificates created by Lemur to expire on weekends. Default is True.
 
-.. data:: LEMUR_RESTRICTED_DOMAINS
+.. data:: LEMUR_WHITELISTED_DOMAINS
     :noindex:
 
-        This allows the administrator to mark a subset of domains or domains matching a particular regex as
+        List of regular expressions for domain restrictions; if the list is not empty, normal users can only issue
-        *restricted*. This means that only an administrator is allows to issue the domains in question.
+        certificates for domain names matching at least one pattern on this list. Administrators are exempt from this
+        restriction.
+
+        Cerificate common name is matched against these rules *if* it does not contain a space. SubjectAltName DNS names
+        are always matched against these rules.
+
+        Take care to write patterns in such way to not allow the `*` wildcard character inadvertently. To match a `.`
+        character, it must be escaped (as `\.`).
+
+.. data:: LEMUR_OWNER_EMAIL_IN_SUBJECT
+    :noindex:
+
+        By default, Lemur will add the certificate owner's email address to certificate subject (for CAs that allow it).
+        Set this to `False` to disable this.
 
 .. data:: LEMUR_TOKEN_SECRET
     :noindex:
@@ -109,6 +126,12 @@ Basic Configuration
         LEMUR_ENCRYPTION_KEYS = ['1YeftooSbxCiX2zo8m1lXtpvQjy27smZcUUaGmffhMY=', 'LAfQt6yrkLqOK5lwpvQcT4jf2zdeTQJV1uYeh9coT5s=']
 
 
+.. data:: DEBUG_DUMP
+    :noindex:
+
+        Dump all imported or generated CSR and certificate details to stdout using OpenSSL. (default: `False`)
+
+
 Certificate Default Options
 ---------------------------
 
@@ -202,14 +225,14 @@ Lemur supports sending certification expiration notifications through SES and SM
         you can send any mail. See: `Verifying Email Address in Amazon SES <http://docs.aws.amazon.com/ses/latest/DeveloperGuide/verify-email-addresses.html>`_
 
 
-.. data:: LEMUR_MAIL
+.. data:: LEMUR_EMAIL
     :noindex:
 
         Lemur sender's email
 
         ::
 
-            LEMUR_MAIL = 'lemur.example.com'
+            LEMUR_EMAIL = 'lemur.example.com'
 
 
 .. data:: LEMUR_SECURITY_TEAM_EMAIL
@@ -234,7 +257,120 @@ Lemur supports sending certification expiration notifications through SES and SM
 
 Authentication Options
 ----------------------
-Lemur currently supports Basic Authentication, Ping OAuth2, and Google out of the box. Additional flows can be added relatively easily.
+Lemur currently supports Basic Authentication, LDAP Authentication, Ping OAuth2, and Google out of the box. Additional flows can be added relatively easily.
+
+LDAP Options
+~~~~~~~~~~~~
+
+Lemur supports the use of an LDAP server in conjunction with Basic Authentication. Lemur local users can still be defined and take precedence over LDAP users. If a local user does not exist, LDAP will be queried for authentication. Only simple ldap binding with or without TLS is supported.
+
+LDAP support requires the pyldap python library, which also depends on the following openldap packages.
+
+.. code-block:: bash
+
+      $ sudo apt-get update
+      $ sudo apt-get install libldap2-dev libsasl2-dev libldap2-dev libssl-dev
+
+
+To configure the use of an LDAP server, a number of settings need to be configured in `lemur.conf.py`.
+
+Here is an example LDAP configuration stanza you can add to your config. Adjust to suit your environment of course.
+ 
+.. code-block:: python
+
+        LDAP_AUTH = True
+        LDAP_BIND_URI='ldaps://secure.evilcorp.net'
+        LDAP_BASE_DN='DC=users,DC=evilcorp,DC=net'
+        LDAP_EMAIL_DOMAIN='evilcorp.net'
+        LDAP_USE_TLS = True
+        LDAP_CACERT_FILE = '/opt/lemur/trusted.pem'
+        LDAP_REQUIRED_GROUP = 'certificate-management-access'
+        LDAP_GROUPS_TO_ROLES = {'certificate-management-admin': 'admin', 'certificate-management-read-only': 'read-only'}
+
+
+The lemur ldap module uses the `user principal name` (upn) of the authenticating user to bind. This is done once for each user at login time. The UPN is effectively the email address in AD/LDAP of the user. If the user doesn't provide the email address, it constructs one based on the username supplied (which should normally match the samAccountName) and the value provided by the config LDAP_EMAIL_DOMAIN.
+The config LDAP_BASE_DN tells lemur where to search within the AD/LDAP tree for the given UPN (user). If the bind with those credentials is successful - there is a valid user in AD with correct password.
+
+Each of the LDAP options are described below.
+
+.. data:: LDAP_AUTH
+    :noindex:
+
+        This enables the use of LDAP
+
+        ::
+
+            LDAP_AUTH = True
+
+.. data:: LDAP_BIND_URI
+    :noindex:
+
+        Specifies the LDAP server connection string
+
+        ::
+
+            LDAP_BIND_URI = 'ldaps://hostname'
+
+.. data:: LDAP_BASE_DN
+    :noindex:
+
+        Specifies the LDAP distinguished name location to search for users
+
+        ::
+
+            LDAP_BASE_DN = 'DC=Users,DC=Evilcorp,DC=com'
+
+.. data:: LDAP_EMAIL_DOMAIN
+    :noindex:
+
+        The email domain used by users in your directory. This is used to build the userPrincipalName to search with.
+
+        ::
+
+            LDAP_EMAIL_DOMAIN = 'evilcorp.com'
+
+The following LDAP options are not required, however TLS is always recommended.
+
+.. data:: LDAP_USE_TLS
+    :noindex:
+
+        Enables the use of TLS when connecting to the LDAP server. Ensure the LDAP_BIND_URI is using ldaps scheme.
+
+        ::
+
+            LDAP_USE_TLS = True
+
+.. data:: LDAP_CACERT_FILE
+    :noindex:
+
+        Specify a Certificate Authority file containing PEM encoded trusted issuer certificates. This can be used if your LDAP server is using certificates issued by a private CA.
+
+        ::
+
+            LDAP_CACERT_FILE = '/path/to/cacert/file'
+
+.. data:: LDAP_REQUIRED_GROUP
+    :noindex:
+
+        Lemur has pretty open permissions. You can define an LDAP group to specify who can access Lemur. Only members of this group will be able to login.
+
+        ::
+
+            LDAP_REQUIRED_GROUP = 'Lemur LDAP Group Name'
+
+.. data:: LDAP_GROUPS_TO_ROLES
+    :noindex:
+
+        You can also define a dictionary of ldap groups mapped to lemur roles. This allows you to use ldap groups to manage access to owner/creator roles in Lemur
+
+        ::
+
+            LDAP_GROUPS_TO_ROLES = {'lemur_admins': 'admin', 'Lemur Team DL Group': 'team@example.com'}
+
+
+Authentication Providers
+~~~~~~~~~~~~~~~~~~~~~~~~
+
 If you are not using an authentication provider you do not need to configure any of these options.
 
 For more information about how to use social logins, see: `Satellizer <https://github.com/sahat/satellizer>`_
@@ -360,6 +496,13 @@ For more information about how to use social logins, see: `Satellizer <https://g
 
             OAUTH2_AUTH_ENDPOINT = "https://<youroauthserver>/oauth2/v1/authorize"
 
+.. data:: OAUTH2_VERIFY_CERT
+    :noindex:
+
+        ::
+
+            OAUTH2_VERIFY_CERT = True
+
 .. data:: GOOGLE_CLIENT_ID
     :noindex:
 
@@ -435,7 +578,7 @@ The following configuration properties are required to use the Digicert issuer p
 .. data:: DIGICERT_URL
     :noindex:
 
-            This is the url for the Digicert API
+            This is the url for the Digicert API (e.g. https://www.digicert.com)
 
 
 .. data:: DIGICERT_API_KEY
@@ -450,12 +593,6 @@ The following configuration properties are required to use the Digicert issuer p
             This is the Digicert organization ID tied to your API key
 
 
-.. data:: DIGICERT_INTERMEDIATE
-    :noindex:
-
-            This is the intermediate to be used for your CA chain
-
-
 .. data:: DIGICERT_ROOT
     :noindex:
 
@@ -468,6 +605,11 @@ The following configuration properties are required to use the Digicert issuer p
             This is the default validity (in years), if no end date is specified. (Default: 1)
 
 
+.. data:: DIGICERT_PRIVATE
+    :noindex:
+
+            This is whether or not to issue a private certificate. (Default: False)
+
 
 CFSSL Issuer Plugin
 ^^^^^^^^^^^^^^^^^^^
@@ -824,7 +966,7 @@ After you have the latest version of the Lemur code base you must run any needed
 This will ensure that any needed tables or columns are created or destroyed.
 
 .. note::
-    Internally, this uses `Alembic <https://alembic.readthedocs.org/en/latest/>`_ to manage database migrations.
+    Internally, this uses `Alembic <http://alembic.zzzcomputing.com/en/latest/>`_ to manage database migrations.
 
 .. note::
     By default Alembic looks for the `migrations` folder in the current working directory.The migrations folder is

docs/conf.py

@@ -13,12 +13,24 @@
 # serve to show the default.
 import sys
 import os
+from unittest.mock import MagicMock
 
 # If extensions (or modules to document with autodoc) are in another directory,
 # add these directories to sys.path here. If the directory is relative to the
 # documentation root, use os.path.abspath to make it absolute, like shown here.
 sys.path.insert(0, os.path.abspath('..'))
 
+# Mock packages that cannot be installed on rtd
+on_rtd = os.environ.get('READTHEDOCS') == 'True'
+if on_rtd:
+    class Mock(MagicMock):
+        @classmethod
+        def __getattr__(cls, name):
+            return MagicMock()
+
+    MOCK_MODULES = ['ldap']
+    sys.modules.update((mod_name, Mock()) for mod_name in MOCK_MODULES)
+
 # -- General configuration ------------------------------------------------
 
 # If your documentation needs a minimal Sphinx version, state it here.

docs/developer/index.rst

@@ -89,6 +89,12 @@ You'll likely want to make some changes to the default configuration (we recomme
 .. note:: The ``upgrade`` shortcut is simply a shortcut to Alembic's upgrade command.
 
 
+Running tests with Docker and docker-compose
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Alternatively you can use Docker and docker-compose for running the tests with ``docker-compose run test``.
+
+
 Coding Standards
 ----------------
 
@@ -277,6 +283,31 @@ Domains
     :undoc-members:
     :show-inheritance:
 
+Endpoints
+---------
+
+.. automodule:: lemur.endpoints.views
+    :members:
+    :undoc-members:
+    :show-inheritance:
+
+Logs
+----
+
+.. automodule:: lemur.logs.views
+    :members:
+    :undoc-members:
+    :show-inheritance:
+
+
+Sources
+-------
+
+.. automodule:: lemur.sources.views
+    :members:
+    :undoc-members:
+    :show-inheritance:
+
 
 Internals
 =========

docs/developer/internals/lemur.certificates.rst

@@ -1,15 +1,6 @@
 certificates Package
 ====================
 
-:mod:`exceptions` Module
-------------------------
-
-.. automodule:: lemur.certificates.exceptions
-    :noindex:
-    :members:
-    :undoc-members:
-    :show-inheritance:
-
 :mod:`models` Module
 --------------------
 

docs/developer/internals/lemur.plugins.lemur_verisign.rst

@@ -10,15 +10,6 @@ lemur_verisign Package
     :undoc-members:
     :show-inheritance:
 
-:mod:`constants` Module
------------------------
-
-.. automodule:: lemur.plugins.lemur_verisign.constants
-    :noindex:
-    :members:
-    :undoc-members:
-    :show-inheritance:
-
 :mod:`plugin` Module
 --------------------
 

docs/developer/internals/lemur.rst

@@ -97,3 +97,19 @@ Subpackages
     lemur.plugins
     lemur.roles
     lemur.users
+    lemur.sources
+    lemur.logs
+    lemur.reporting
+    lemur.tests
+    lemur.deployment
+    lemur.endpoints
+    lemur.defaults
+    lemur.plugins.lemur_acme
+    lemur.plugins.lemur_atlas
+    lemur.plugins.lemur_cryptography
+    lemur.plugins.lemur_digicert
+    lemur.plugins.lemur_java
+    lemur.plugins.lemur_kubernetes
+    lemur.plugins.lemur_linuxdst
+    lemur.plugins.lemur_openssl
+    lemur.plugins.lemur_slack

docs/faq.rst

@@ -6,7 +6,7 @@ Common Problems
 
 In my startup logs I see *'Aborting... Lemur cannot locate db encryption key, is LEMUR_ENCRYPTION_KEYS set?'*
   You likely have not correctly configured **LEMUR_ENCRYPTION_KEYS**. See
-  :doc:`administration/index` for more information.
+  :doc:`administration` for more information.
 
 
 I am seeing Lemur's javascript load in my browser but not the CSS.

docs/production/index.rst

@@ -217,6 +217,23 @@ An example apache config::
         # HSTS (mod_headers is required) (15768000 seconds = 6 months)
         Header always set Strict-Transport-Security "max-age=15768000"
         ...
+        
+     # Set the lemur DocumentRoot to static/dist
+     DocumentRoot /www/lemur/lemur/static/dist
+     
+     # Uncomment to force http 1.0 connections to proxy
+     # SetEnv force-proxy-request-1.0 1
+     
+     #Don't keep proxy connections alive
+     SetEnv proxy-nokeepalive 1
+     
+     # Only need to do reverse proxy
+     ProxyRequests Off
+     
+     # Proxy requests to the api to the lemur service (and sanitize redirects from it)
+     ProxyPass "/api" "http://127.0.0.1:8000/api"
+     ProxyPassReverse "/api" "http://127.0.0.1:8000/api"
+     
     </VirtualHost>
 
 Also included in the configurations above are several best practices when it comes to deploying TLS. Things like enabling
@@ -314,6 +331,6 @@ How often you run these commands is largely up to the user. `notify` and `check_
 
 Example cron entries::
 
-    0 22 * * * lemuruser export LEMUR_CONF=/Users/me/.lemur/lemur.conf.py; /www/lemur/bin/lemur notify
+    0 22 * * * lemuruser export LEMUR_CONF=/Users/me/.lemur/lemur.conf.py; /www/lemur/bin/lemur notify expirations
-    */15 * * * * lemuruser export LEMUR_CONF=/Users/me/.lemur/lemur.conf.py; /www/lemur/bin/lemur sync -s all
+    */15 * * * * lemuruser export LEMUR_CONF=/Users/me/.lemur/lemur.conf.py; /www/lemur/bin/lemur source sync -s all
-    0 22 * * * lemuruser export LEMUR_CONF=/Users/me/.lemur/lemur.conf.py; /www/lemur/bin/lemur check_revoked
+    0 22 * * * lemuruser export LEMUR_CONF=/Users/me/.lemur/lemur.conf.py; /www/lemur/bin/lemur certificate check_revoked

docs/quickstart/index.rst

@@ -27,10 +27,13 @@ If installing Lemur on a bare Ubuntu OS you will need to grab the following pack
 .. code-block:: bash
 
     $ sudo apt-get update
-    $ sudo apt-get install nodejs-legacy python-pip python-dev python3-dev libpq-dev build-essential libssl-dev libffi-dev nginx git supervisor npm postgresql
+    $ sudo apt-get install nodejs nodejs-legacy python-pip python-dev python3-dev libpq-dev build-essential libssl-dev libffi-dev libsasl2-dev libldap2-dev nginx git supervisor npm postgresql
 
 .. note:: PostgreSQL is only required if your database is going to be on the same host as the webserver.  npm is needed if you're installing Lemur from the source (e.g., from git).
 
+.. note:: Installing node from a package manager may creat the nodejs bin at  /usr/bin/nodejs instead of /usr/bin/node If that is the case run the following
+    $ sudo ln -s /user/bin/nodejs /usr/bin/node
+
 Now, install Python ``virtualenv`` package:
 
 .. code-block:: bash
@@ -89,7 +92,7 @@ And then run:
 .. note:: This command will install npm dependencies as well as compile static assets.
 
 
-You may also run with the urlContextPath variable set. If this is set it will add the desired context path for subsequent calls back to lemur.
+You may also run with the urlContextPath variable set. If this is set it will add the desired context path for subsequent calls back to lemur. This will only edit the front end code for calls back to the server, you will have to make sure the server knows about these routes.
 ::
 
   Example:
@@ -138,7 +141,7 @@ Before Lemur will run you need to fill in a few required variables in the config
     LEMUR_DEFAULT_COUNTRY
     LEMUR_DEFAULT_STATE
     LEMUR_DEFAULT_LOCATION
-    LEMUR_DEFAUTL_ORGANIZATION
+    LEMUR_DEFAULT_ORGANIZATION
     LEMUR_DEFAULT_ORGANIZATIONAL_UNIT
 
 Setup Postgres
@@ -151,9 +154,8 @@ First, set a password for the postgres user.  For this guide, we will use ``lemu
 .. code-block:: bash
 
     $ sudo -u postgres -i
-    # \password postgres
+    $ psql
-    Enter new password: lemur
+    postgres=# CREATE USER lemur WITH PASSWORD 'lemur';
-    Enter it again: lemur
 
 Once successful, type CTRL-D to exit the Postgres shell.
 
@@ -269,8 +271,8 @@ Configuring Supervisor couldn't be more simple. Just point it to the ``lemur`` e
   autostart=true
   autorestart=true
   redirect_stderr=true
-  stdout_logfile syslog
+  stdout_logfile=syslog
-  stderr_logfile syslog
+  stderr_logfile=syslog
 
 See :ref:`Using Supervisor <UsingSupervisor>` for more details on using Supervisor.
 

docs/requirements.txt

@@ -1,54 +1,35 @@
-alabaster==0.7.8
+Flask==0.12
-alembic==0.8.6
+Flask-RESTful==0.3.6
-aniso8601==1.1.0
-arrow==0.7.0
-Babel==2.3.4
-bcrypt==2.0.0
-beautifulsoup4==4.4.1
-blinker==1.4
-boto==2.38.0
-cffi==1.7.0
-cryptography==1.3.2
-docutils==0.12
-enum34==1.1.6
-Flask==0.10.1
-Flask-Bcrypt==0.7.1
-Flask-Mail==0.9.1
-Flask-Migrate==1.7.0
-Flask-Principal==0.4.0
-Flask-RESTful==0.3.3
-Flask-Script==2.0.5
 Flask-SQLAlchemy==2.1
-future==0.15.2
+Flask-Script==2.0.5
-gunicorn==19.4.1
+Flask-Migrate==2.1.1
-idna==2.1
+Flask-Bcrypt==0.7.1
-imagesize==0.7.1
+Flask-Principal==0.4.0
-inflection==0.3.1
+Flask-Mail==0.9.1
-ipaddress==1.0.16
+SQLAlchemy-Utils==0.32.14
-itsdangerous==0.24
+requests==2.11.1
-Jinja2==2.8
+ndg-httpsclient==0.4.2
-lockfile==0.12.2
+psycopg2==2.7.3
-Mako==1.0.4
+arrow==0.10.0
-MarkupSafe==0.23
-marshmallow==2.4.0
-marshmallow-sqlalchemy==0.8.0
-psycopg2==2.6.1
-pyasn1==0.1.9
-pycparser==2.14
-pycrypto==2.6.1
-Pygments==2.1.3
-PyJWT==1.4.0
-pyOpenSSL==0.15.1
-python-dateutil==2.5.3
-python-editor==1.0.1
-pytz==2016.4
-requests==2.9.1
 six==1.10.0
-snowballstemmer==1.2.1
+marshmallow-sqlalchemy==0.13.1
-Sphinx==1.4.4
+gunicorn==19.7.1
-sphinx-rtd-theme==0.1.9
+marshmallow==2.13.6
-sphinxcontrib-httpdomain==1.5.0
+cryptography==1.9
-SQLAlchemy==1.0.13
+xmltodict==0.11.0
-SQLAlchemy-Utils==0.31.4
+pyjwt==1.5.2
-Werkzeug==0.11.10
+lockfile==0.12.2
-xmltodict==0.9.2
+inflection==0.3.1
+future==0.16.0
+boto3==1.4.6
+acme==0.18.1
+retrying==1.3.3
+tabulate==0.7.7
+pem==17.1.0
+raven[flask]==6.1.0
+jinja2==2.9.6
+# pyldap==2.4.37  # cannot be installed on rtd - required by ldap auth provider
+paramiko==2.2.1  # required for lemur_linuxdst plugin
+sphinx
+sphinxcontrib-httpdomain
+sphinx-rtd-theme

gulp/build.js

@@ -232,10 +232,16 @@ gulp.task('package:strip', function () {
 
 gulp.task('addUrlContextPath',['addUrlContextPath:revreplace'], function(){
   var urlContextPathExists = argv.urlContextPath ? true : false;
-  return gulp.src('lemur/static/dist/scripts/main*.js')
+  ['lemur/static/dist/scripts/main*.js',
-    .pipe(gulpif(urlContextPathExists, replace('api/', argv.urlContextPath + '/api/')))
+  'lemur/static/dist/angular/**/*.html']
-    .pipe(gulpif(urlContextPathExists, replace('angular/', argv.urlContextPath + '/angular/')))
+  .forEach(function(file){
-    .pipe(gulp.dest('lemur/static/dist/scripts'))
+    return gulp.src(file)
+      .pipe(gulpif(urlContextPathExists, replace('api/', argv.urlContextPath + '/api/')))
+      .pipe(gulpif(urlContextPathExists, replace('angular/', argv.urlContextPath + '/angular/')))
+      .pipe(gulp.dest(function(file){
+        return file.base;
+      }))
+  })
 });
 
 gulp.task('addUrlContextPath:revision', function(){

lemur/__about__.py

@@ -9,10 +9,10 @@ __title__ = "lemur"
 __summary__ = ("Certificate management and orchestration service")
 __uri__ = "https://github.com/Netflix/lemur"
 
-__version__ = "0.5.0"
+__version__ = "0.6.0"
 
 __author__ = "The Lemur developers"
 __email__ = "security@netflix.com"
 
 __license__ = "Apache License, Version 2.0"
-__copyright__ = "Copyright 2016 {0}".format(__author__)
+__copyright__ = "Copyright 2017 {0}".format(__author__)

lemur/__init__.py

@@ -26,6 +26,7 @@ from lemur.notifications.views import mod as notifications_bp
 from lemur.sources.views import mod as sources_bp
 from lemur.endpoints.views import mod as endpoints_bp
 from lemur.logs.views import mod as logs_bp
+from lemur.api_keys.views import mod as api_key_bp
 
 from lemur.__about__ import (
     __author__, __copyright__, __email__, __license__, __summary__, __title__,
@@ -51,7 +52,8 @@ LEMUR_BLUEPRINTS = (
     notifications_bp,
     sources_bp,
     endpoints_bp,
-    logs_bp
+    logs_bp,
+    api_key_bp
 )
 
 

lemur/api_keys/cli.py

@@ -0,0 +1,41 @@
+"""
+.. module: lemur.api_keys.cli
+    :platform: Unix
+    :copyright: (c) 2017 by Netflix Inc., see AUTHORS for more
+    :license: Apache, see LICENSE for more details.
+.. moduleauthor:: Eric Coan <kungfury@instructure.com>
+"""
+from flask_script import Manager
+from lemur.api_keys import service as api_key_service
+from lemur.auth.service import create_token
+
+from datetime import datetime
+
+manager = Manager(usage="Handles all api key related tasks.")
+
+
+@manager.option('-u', '--user-id', dest='uid', help='The User ID this access key belongs too.')
+@manager.option('-n', '--name', dest='name', help='The name of this API Key.')
+@manager.option('-t', '--ttl', dest='ttl', help='The TTL of this API Key. -1 for forever.')
+def create(uid, name, ttl):
+    """
+    Create a new api key for a user.
+    :return:
+    """
+    print("[+] Creating a new api key.")
+    key = api_key_service.create(user_id=uid, name=name,
+        ttl=ttl, issued_at=int(datetime.utcnow().timestamp()), revoked=False)
+    print("[+] Successfully created a new api key. Generating a JWT...")
+    jwt = create_token(uid, key.id, key.ttl)
+    print("[+] Your JWT is: {jwt}".format(jwt=jwt))
+
+
+@manager.option('-a', '--api-key-id', dest='aid', help='The API Key ID to revoke.')
+def revoke(aid):
+    """
+    Revokes an api key for a user.
+    :return:
+    """
+    print("[-] Revoking the API Key api key.")
+    api_key_service.revoke(aid=aid)
+    print("[+] Successfully revoked the api key")

lemur/api_keys/models.py

@@ -0,0 +1,25 @@
+"""
+.. module: lemur.api_keys.models
+    :platform: Unix
+    :synopsis: This module contains all of the models need to create an api key within Lemur.
+    :copyright: (c) 2017 by Netflix Inc., see AUTHORS for more
+    :license: Apache, see LICENSE for more details.
+.. moduleauthor:: Eric Coan <kungfury@instructure.com>
+"""
+from sqlalchemy import BigInteger, Boolean, Column, ForeignKey, Integer, String
+
+from lemur.database import db
+
+
+class ApiKey(db.Model):
+    __tablename__ = 'api_keys'
+    id = Column(Integer, primary_key=True)
+    name = Column(String)
+    user_id = Column(Integer, ForeignKey('users.id'))
+    ttl = Column(BigInteger)
+    issued_at = Column(BigInteger)
+    revoked = Column(Boolean)
+
+    def __repr__(self):
+        return "ApiKey(name={name}, user_id={user_id}, ttl={ttl}, issued_at={iat}, revoked={revoked})".format(
+            user_id=self.user_id, name=self.name, ttl=self.ttl, iat=self.issued_at, revoked=self.revoked)

lemur/api_keys/schemas.py

@@ -0,0 +1,57 @@
+"""
+.. module: lemur.api_keys.schemas
+    :platform: Unix
+    :copyright: (c) 2017 by Netflix Inc., see AUTHORS for more
+    :license: Apache, see LICENSE for more details.
+.. moduleauthor:: Eric Coan <kungfury@instructure.com>
+"""
+from flask import g
+from marshmallow import fields
+
+from lemur.common.schema import LemurInputSchema, LemurOutputSchema
+from lemur.users.schemas import UserNestedOutputSchema, UserInputSchema
+
+
+def current_user_id():
+    return {'id': g.current_user.id, 'email': g.current_user.email, 'username': g.current_user.username}
+
+
+class ApiKeyInputSchema(LemurInputSchema):
+    name = fields.String(required=False)
+    user = fields.Nested(UserInputSchema, missing=current_user_id, default=current_user_id)
+    ttl = fields.Integer()
+
+
+class ApiKeyRevokeSchema(LemurInputSchema):
+    id = fields.Integer(required=True)
+    name = fields.String()
+    user = fields.Nested(UserInputSchema, required=True)
+    revoked = fields.Boolean()
+    ttl = fields.Integer()
+    issued_at = fields.Integer(required=False)
+
+
+class UserApiKeyInputSchema(LemurInputSchema):
+    name = fields.String(required=False)
+    ttl = fields.Integer()
+
+
+class ApiKeyOutputSchema(LemurOutputSchema):
+    jwt = fields.String()
+
+
+class ApiKeyDescribedOutputSchema(LemurOutputSchema):
+    id = fields.Integer()
+    name = fields.String()
+    user = fields.Nested(UserNestedOutputSchema)
+    ttl = fields.Integer()
+    issued_at = fields.Integer()
+    revoked = fields.Boolean()
+
+
+api_key_input_schema = ApiKeyInputSchema()
+api_key_revoke_schema = ApiKeyRevokeSchema()
+api_key_output_schema = ApiKeyOutputSchema()
+api_keys_output_schema = ApiKeyDescribedOutputSchema(many=True)
+api_key_described_output_schema = ApiKeyDescribedOutputSchema()
+user_api_key_input_schema = UserApiKeyInputSchema()

lemur/api_keys/service.py

@@ -0,0 +1,97 @@
+"""
+.. module: lemur.api_keys.service
+    :platform: Unix
+    :copyright: (c) 2017 by Netflix Inc., see AUTHORS for more
+    :license: Apache, see LICENSE for more details.
+.. moduleauthor:: Eric Coan <kungfury@instructure.com>
+"""
+from lemur import database
+from lemur.api_keys.models import ApiKey
+
+
+def get(aid):
+    """
+    Retrieves an api key by its ID.
+    :param aid: The access key id to get.
+    :return:
+    """
+    return database.get(ApiKey, aid)
+
+
+def delete(access_key):
+    """
+    Delete an access key. This is one way to remove a key, though you probably should just set revoked.
+    :param access_key:
+    :return:
+    """
+    database.delete(access_key)
+
+
+def revoke(aid):
+    """
+    Revokes an api key.
+    :param aid:
+    :return:
+    """
+    api_key = get(aid)
+    setattr(api_key, 'revoked', False)
+
+    return database.update(api_key)
+
+
+def get_all_api_keys():
+    """
+    Retrieves all Api Keys.
+    :return:
+    """
+    return ApiKey.query.all()
+
+
+def create(**kwargs):
+    """
+    Creates a new API Key.
+
+    :param kwargs:
+    :return:
+    """
+    api_key = ApiKey(**kwargs)
+    database.create(api_key)
+    return api_key
+
+
+def update(api_key, **kwargs):
+    """
+    Updates an api key.
+    :param api_key:
+    :param kwargs:
+    :return:
+    """
+    for key, value in kwargs.items():
+        setattr(api_key, key, value)
+
+    return database.update(api_key)
+
+
+def render(args):
+    """
+    Helper to parse REST Api requests
+
+    :param args:
+    :return:
+    """
+    query = database.session_query(ApiKey)
+    user_id = args.pop('user_id', None)
+    aid = args.pop('id', None)
+    has_permission = args.pop('has_permission', False)
+    requesting_user_id = args.pop('requesting_user_id')
+
+    if user_id:
+        query = query.filter(ApiKey.user_id == user_id)
+
+    if aid:
+        query = query.filter(ApiKey.id == aid)
+
+    if not has_permission:
+        query = query.filter(ApiKey.user_id == requesting_user_id)
+
+    return database.sort_and_page(query, ApiKey, args)

lemur/api_keys/views.py

@@ -0,0 +1,579 @@
+"""
+.. module: lemur.api_keys.views
+    :platform: Unix
+    :copyright: (c) 2017 by Netflix Inc., see AUTHORS for more
+    :license: Apache, see LICENSE for more details.
+
+.. moduleauthor:: Eric Coan <kungfury@instructure.com>
+
+"""
+from datetime import datetime
+
+from flask import Blueprint, g
+from flask_restful import reqparse, Api
+
+from lemur.api_keys import service
+from lemur.auth.service import AuthenticatedResource, create_token
+from lemur.auth.permissions import ApiKeyCreatorPermission
+
+from lemur.common.schema import validate_schema
+from lemur.common.utils import paginated_parser
+
+from lemur.api_keys.schemas import api_key_input_schema, api_key_revoke_schema, api_key_output_schema, \
+    api_keys_output_schema, api_key_described_output_schema, user_api_key_input_schema
+
+mod = Blueprint('api_keys', __name__)
+api = Api(mod)
+
+
+class ApiKeyList(AuthenticatedResource):
+    """ Defines the 'api_keys' endpoint """
+
+    def __init__(self):
+        super(ApiKeyList, self).__init__()
+
+    @validate_schema(None, api_keys_output_schema)
+    def get(self):
+        """
+        .. http:get:: /keys
+
+           The current list of api keys, that you can see.
+
+           **Example request**:
+
+           .. sourcecode:: http
+
+              GET /keys HTTP/1.1
+              Host: example.com
+              Accept: application/json, text/javascript
+
+           **Example response**:
+
+           .. sourcecode:: http
+
+              HTTP/1.1 200 OK
+              Vary: Accept
+              Content-Type: text/javascript
+
+              {
+                "items": [
+                    {
+                      "id": 1,
+                      "name": "custom name",
+                      "user_id": 1,
+                      "ttl": -1,
+                      "issued_at": 12,
+                      "revoked": false
+                    }
+                ],
+                "total": 1
+              }
+
+           :query sortBy: field to sort on
+           :query sortDir: asc or desc
+           :query page: int default is 1
+           :query count: count number. default is 10
+           :query user_id: a user to filter by.
+           :query id: an access key to filter by.
+           :reqheader Authorization: OAuth token to authenticate
+           :statuscode 200: no error
+           :statuscode 403: unauthenticated
+        """
+        parser = paginated_parser.copy()
+        args = parser.parse_args()
+        args['has_permission'] = ApiKeyCreatorPermission().can()
+        args['requesting_user_id'] = g.current_user.id
+        return service.render(args)
+
+    @validate_schema(api_key_input_schema, api_key_output_schema)
+    def post(self, data=None):
+        """
+        .. http:post:: /keys
+
+           Creates an API Key.
+
+           **Example request**:
+
+           .. sourcecode:: http
+
+              POST /keys HTTP/1.1
+              Host: example.com
+              Accept: application/json, text/javascript
+
+              {
+                "name": "my custom name",
+                "user_id": 1,
+                "ttl": -1
+              }
+
+           **Example response**:
+
+           .. sourcecode:: http
+
+              HTTP/1.1 200 OK
+              Vary: Accept
+              Content-Type: text/javascript
+
+              {
+
+                "jwt": ""
+              }
+
+           :reqheader Authorization: OAuth token to authenticate
+           :statuscode 200: no error
+           :statuscode 403: unauthenticated
+        """
+        if not ApiKeyCreatorPermission().can():
+            if data['user']['id'] != g.current_user.id:
+                return dict(message="You are not authorized to create tokens for: {0}".format(data['user']['username'])), 403
+
+        access_token = service.create(name=data['name'], user_id=data['user']['id'], ttl=data['ttl'],
+                                      revoked=False, issued_at=int(datetime.utcnow().timestamp()))
+        return dict(jwt=create_token(access_token.user_id, access_token.id, access_token.ttl))
+
+
+class ApiKeyUserList(AuthenticatedResource):
+    """ Defines the 'keys' endpoint on the 'users' endpoint. """
+
+    def __init__(self):
+        super(ApiKeyUserList, self).__init__()
+
+    @validate_schema(None, api_keys_output_schema)
+    def get(self, user_id):
+        """
+        .. http:get:: /users/:user_id/keys
+
+           The current list of api keys for a user, that you can see.
+
+           **Example request**:
+
+           .. sourcecode:: http
+
+              GET /users/1/keys HTTP/1.1
+              Host: example.com
+              Accept: application/json, text/javascript
+
+           **Example response**:
+
+           .. sourcecode:: http
+
+              HTTP/1.1 200 OK
+              Vary: Accept
+              Content-Type: text/javascript
+
+              {
+                "items": [
+                    {
+                      "id": 1,
+                      "name": "custom name",
+                      "user_id": 1,
+                      "ttl": -1,
+                      "issued_at": 12,
+                      "revoked": false
+                    }
+                ],
+                "total": 1
+              }
+
+           :query sortBy: field to sort on
+           :query sortDir: asc or desc
+           :query page: int default is 1
+           :query count: count number. default is 10
+           :query id: an access key to filter by.
+           :reqheader Authorization: OAuth token to authenticate
+           :statuscode 200: no error
+           :statuscode 403: unauthenticated
+        """
+        parser = paginated_parser.copy()
+        args = parser.parse_args()
+        args['has_permission'] = ApiKeyCreatorPermission().can()
+        args['requesting_user_id'] = g.current_user.id
+        args['user_id'] = user_id
+        return service.render(args)
+
+    @validate_schema(user_api_key_input_schema, api_key_output_schema)
+    def post(self, user_id, data=None):
+        """
+        .. http:post:: /users/:user_id/keys
+
+           Creates an API Key for a user.
+
+           **Example request**:
+
+           .. sourcecode:: http
+
+              POST /users/1/keys HTTP/1.1
+              Host: example.com
+              Accept: application/json, text/javascript
+
+              {
+                "name": "my custom name"
+                "ttl": -1
+              }
+
+           **Example response**:
+
+           .. sourcecode:: http
+
+              HTTP/1.1 200 OK
+              Vary: Accept
+              Content-Type: text/javascript
+
+              {
+
+                "jwt": ""
+              }
+
+           :reqheader Authorization: OAuth token to authenticate
+           :statuscode 200: no error
+           :statuscode 403: unauthenticated
+        """
+        if not ApiKeyCreatorPermission().can():
+            if user_id != g.current_user.id:
+                return dict(message="You are not authorized to create tokens for: {0}".format(user_id)), 403
+
+        access_token = service.create(name=data['name'], user_id=user_id, ttl=data['ttl'],
+                                      revoked=False, issued_at=int(datetime.utcnow().timestamp()))
+        return dict(jwt=create_token(access_token.user_id, access_token.id, access_token.ttl))
+
+
+class ApiKeys(AuthenticatedResource):
+    def __init__(self):
+        self.reqparse = reqparse.RequestParser()
+        super(ApiKeys, self).__init__()
+
+    @validate_schema(None, api_key_output_schema)
+    def get(self, aid):
+        """
+        .. http:get:: /keys/1
+
+           Fetch one api key
+
+           **Example request**:
+
+           .. sourcecode:: http
+
+              GET /keys/1 HTTP/1.1
+              Host: example.com
+              Accept: application/json, text/javascript
+
+           **Example response**:
+
+           .. sourcecode:: http
+
+              HTTP/1.1 200 OK
+              Vary: Accept
+              Content-Type: text/javascript
+
+              {
+                  "jwt": ""
+              }
+
+           :reqheader Authorization: OAuth token to authenticate
+           :statuscode 200: no error
+           :statuscode 403: unauthenticated
+        """
+        access_key = service.get(aid)
+
+        if access_key is None:
+            return dict(message="This token does not exist!"), 404
+
+        if access_key.user_id != g.current_user.id:
+            if not ApiKeyCreatorPermission().can():
+                return dict(message="You are not authorized to view this token!"), 403
+
+        return dict(jwt=create_token(access_key.user_id, access_key.id, access_key.ttl))
+
+    @validate_schema(api_key_revoke_schema, api_key_output_schema)
+    def put(self, aid, data=None):
+        """
+        .. http:put:: /keys/1
+
+           update one api key
+
+           **Example request**:
+
+           .. sourcecode:: http
+
+              PUT /keys/1 HTTP/1.1
+              Host: example.com
+              Accept: application/json, text/javascript
+
+              {
+                  "name": "new_name",
+                  "revoked": false,
+                  "ttl": -1
+              }
+
+           **Example response**:
+
+           .. sourcecode:: http
+
+              HTTP/1.1 200 OK
+              Vary: Accept
+              Content-Type: text/javascript
+
+              {
+                  "jwt": ""
+              }
+
+           :reqheader Authorization: OAuth token to authenticate
+           :statuscode 200: no error
+           :statuscode 403: unauthenticated
+        """
+        access_key = service.get(aid)
+        if access_key is None:
+            return dict(message="This token does not exist!"), 404
+
+        if access_key.user_id != g.current_user.id:
+            if not ApiKeyCreatorPermission().can():
+                return dict(message="You are not authorized to update this token!"), 403
+
+        service.update(access_key, name=data['name'], revoked=data['revoked'], ttl=data['ttl'])
+        return dict(jwt=create_token(access_key.user_id, access_key.id, access_key.ttl))
+
+    def delete(self, aid):
+        """
+        .. http:delete:: /keys/1
+
+           deletes one api key
+
+           **Example request**:
+
+           .. sourcecode:: http
+
+              DELETE /keys/1 HTTP/1.1
+              Host: example.com
+              Accept: application/json, text/javascript
+
+           **Example response**:
+
+           .. sourcecode:: http
+
+              HTTP/1.1 200 OK
+              Vary: Accept
+              Content-Type: text/javascript
+
+              {
+                  "result": true
+              }
+
+           :reqheader Authorization: OAuth token to authenticate
+           :statuscode 200: no error
+           :statuscode 403: unauthenticated
+        """
+        access_key = service.get(aid)
+        if access_key is None:
+            return dict(message="This token does not exist!"), 404
+
+        if access_key.user_id != g.current_user.id:
+            if not ApiKeyCreatorPermission().can():
+                return dict(message="You are not authorized to delete this token!"), 403
+
+        service.delete(access_key)
+        return {'result': True}
+
+
+class UserApiKeys(AuthenticatedResource):
+    def __init__(self):
+        self.reqparse = reqparse.RequestParser()
+        super(UserApiKeys, self).__init__()
+
+    @validate_schema(None, api_key_output_schema)
+    def get(self, uid, aid):
+        """
+        .. http:get:: /users/1/keys/1
+
+           Fetch one api key
+
+           **Example request**:
+
+           .. sourcecode:: http
+
+              GET /users/1/api_keys/1 HTTP/1.1
+              Host: example.com
+              Accept: application/json, text/javascript
+
+           **Example response**:
+
+           .. sourcecode:: http
+
+              HTTP/1.1 200 OK
+              Vary: Accept
+              Content-Type: text/javascript
+
+              {
+                  "jwt": ""
+              }
+
+           :reqheader Authorization: OAuth token to authenticate
+           :statuscode 200: no error
+           :statuscode 403: unauthenticated
+        """
+        if uid != g.current_user.id:
+            if not ApiKeyCreatorPermission().can():
+                return dict(message="You are not authorized to view this token!"), 403
+
+        access_key = service.get(aid)
+
+        if access_key is None:
+            return dict(message="This token does not exist!"), 404
+
+        if access_key.user_id != uid:
+            return dict(message="You are not authorized to view this token!"), 403
+
+        return dict(jwt=create_token(access_key.user_id, access_key.id, access_key.ttl))
+
+    @validate_schema(api_key_revoke_schema, api_key_output_schema)
+    def put(self, uid, aid, data=None):
+        """
+        .. http:put:: /users/1/keys/1
+
+           update one api key
+
+           **Example request**:
+
+           .. sourcecode:: http
+
+              PUT /users/1/keys/1 HTTP/1.1
+              Host: example.com
+              Accept: application/json, text/javascript
+
+              {
+                  "name": "new_name",
+                  "revoked": false,
+                  "ttl": -1
+              }
+
+           **Example response**:
+
+           .. sourcecode:: http
+
+              HTTP/1.1 200 OK
+              Vary: Accept
+              Content-Type: text/javascript
+
+              {
+                  "jwt": ""
+              }
+
+           :reqheader Authorization: OAuth token to authenticate
+           :statuscode 200: no error
+           :statuscode 403: unauthenticated
+        """
+        if uid != g.current_user.id:
+            if not ApiKeyCreatorPermission().can():
+                return dict(message="You are not authorized to view this token!"), 403
+
+        access_key = service.get(aid)
+        if access_key is None:
+            return dict(message="This token does not exist!"), 404
+
+        if access_key.user_id != uid:
+            return dict(message="You are not authorized to update this token!"), 403
+
+        service.update(access_key, name=data['name'], revoked=data['revoked'], ttl=data['ttl'])
+        return dict(jwt=create_token(access_key.user_id, access_key.id, access_key.ttl))
+
+    def delete(self, uid, aid):
+        """
+        .. http:delete:: /users/1/keys/1
+
+           deletes one api key
+
+           **Example request**:
+
+           .. sourcecode:: http
+
+              DELETE /users/1/keys/1 HTTP/1.1
+              Host: example.com
+              Accept: application/json, text/javascript
+
+           **Example response**:
+
+           .. sourcecode:: http
+
+              HTTP/1.1 200 OK
+              Vary: Accept
+              Content-Type: text/javascript
+
+              {
+                  "result": true
+              }
+
+           :reqheader Authorization: OAuth token to authenticate
+           :statuscode 200: no error
+           :statuscode 403: unauthenticated
+        """
+        if uid != g.current_user.id:
+            if not ApiKeyCreatorPermission().can():
+                return dict(message="You are not authorized to view this token!"), 403
+
+        access_key = service.get(aid)
+        if access_key is None:
+            return dict(message="This token does not exist!"), 404
+
+        if access_key.user_id != uid:
+            return dict(message="You are not authorized to delete this token!"), 403
+
+        service.delete(access_key)
+        return {'result': True}
+
+
+class ApiKeysDescribed(AuthenticatedResource):
+    def __init__(self):
+        self.reqparse = reqparse.RequestParser()
+        super(ApiKeysDescribed, self).__init__()
+
+    @validate_schema(None, api_key_described_output_schema)
+    def get(self, aid):
+        """
+        .. http:get:: /keys/1/described
+
+           Fetch one api key
+
+           **Example request**:
+
+           .. sourcecode:: http
+
+              GET /keys/1 HTTP/1.1
+              Host: example.com
+              Accept: application/json, text/javascript
+
+           **Example response**:
+
+           .. sourcecode:: http
+
+              HTTP/1.1 200 OK
+              Vary: Accept
+              Content-Type: text/javascript
+
+              {
+                  "id": 2,
+                  "name": "hoi",
+                  "user_id": 2,
+                  "ttl": -1,
+                  "issued_at": 1222222,
+                  "revoked": false
+              }
+
+           :reqheader Authorization: OAuth token to authenticate
+           :statuscode 200: no error
+           :statuscode 403: unauthenticated
+        """
+        access_key = service.get(aid)
+        if access_key is None:
+            return dict(message="This token does not exist!"), 404
+
+        if access_key.user_id != g.current_user.id:
+            if not ApiKeyCreatorPermission().can():
+                return dict(message="You are not authorized to view this token!"), 403
+
+        return access_key
+
+
+api.add_resource(ApiKeyList, '/keys', endpoint='api_keys')
+api.add_resource(ApiKeys, '/keys/<int:aid>', endpoint='api_key')
+api.add_resource(ApiKeysDescribed, '/keys/<int:aid>/described', endpoint='api_key_described')
+api.add_resource(ApiKeyUserList, '/users/<int:user_id>/keys', endpoint='user_api_keys')
+api.add_resource(UserApiKeys, '/users/<int:uid>/keys/<int:aid>', endpoint='user_api_key')

lemur/auth/ldap.py

@@ -0,0 +1,187 @@
+"""
+.. module: lemur.auth.ldap
+    :platform: Unix
+    :copyright: (c) 2015 by Netflix Inc., see AUTHORS for more
+    :license: Apache, see LICENSE for more details.
+.. moduleauthor:: Ian Stahnke <ian.stahnke@myob.com>
+"""
+import ldap
+
+from flask import current_app
+
+from lemur.users import service as user_service
+from lemur.roles import service as role_service
+from lemur.common.utils import validate_conf, get_psuedo_random_string
+
+
+class LdapPrincipal():
+    """
+    Provides methods for authenticating against an LDAP server.
+    """
+    def __init__(self, args):
+        self._ldap_validate_conf()
+        # setup ldap config
+        if not args['username']:
+            raise Exception("missing ldap username")
+        if not args['password']:
+            self.error_message = "missing ldap password"
+            raise Exception("missing ldap password")
+        self.ldap_principal = args['username']
+        self.ldap_email_domain = current_app.config.get("LDAP_EMAIL_DOMAIN", None)
+        if '@' not in self.ldap_principal:
+            self.ldap_principal = '%s@%s' % (self.ldap_principal, self.ldap_email_domain)
+        self.ldap_username = args['username']
+        if '@' in self.ldap_username:
+            self.ldap_username = args['username'].split("@")[0]
+        self.ldap_password = args['password']
+        self.ldap_server = current_app.config.get('LDAP_BIND_URI', None)
+        self.ldap_base_dn = current_app.config.get("LDAP_BASE_DN", None)
+        self.ldap_use_tls = current_app.config.get("LDAP_USE_TLS", False)
+        self.ldap_cacert_file = current_app.config.get("LDAP_CACERT_FILE", None)
+        self.ldap_default_role = current_app.config.get("LEMUR_DEFAULT_ROLE", None)
+        self.ldap_required_group = current_app.config.get("LDAP_REQUIRED_GROUP", None)
+        self.ldap_groups_to_roles = current_app.config.get("LDAP_GROUPS_TO_ROLES", None)
+        self.ldap_attrs = ['memberOf']
+        self.ldap_client = None
+        self.ldap_groups = None
+
+    def _update_user(self, roles):
+        """
+        create or update a local user instance.
+        """
+        # try to get user from local database
+        user = user_service.get_by_email(self.ldap_principal)
+
+        # create them a local account
+        if not user:
+            user = user_service.create(
+                self.ldap_username,
+                get_psuedo_random_string(),
+                self.ldap_principal,
+                True,
+                '',     # thumbnailPhotoUrl
+                list(roles)
+            )
+        else:
+            # we add 'lemur' specific roles, so they do not get marked as removed
+            for ur in user.roles:
+                if not ur.third_party:
+                    roles.add(ur)
+
+            # update any changes to the user
+            user_service.update(
+                user.id,
+                self.ldap_username,
+                self.ldap_principal,
+                user.active,
+                user.profile_picture,
+                list(roles)
+            )
+        return user
+
+    def _authorize(self):
+        """
+        check groups and roles to confirm access.
+        return a list of roles if ok.
+        raise an exception on error.
+        """
+        if not self.ldap_principal:
+            return None
+
+        if self.ldap_required_group:
+            # ensure the user has the required group in their group list
+            if self.ldap_required_group not in self.ldap_groups:
+                return None
+
+        roles = set()
+        if self.ldap_default_role:
+            role = role_service.get_by_name(self.ldap_default_role)
+            if role:
+                if not role.third_party:
+                    role = role.set_third_party(role.id, third_party_status=True)
+                roles.add(role)
+
+        # update their 'roles'
+        role = role_service.get_by_name(self.ldap_principal)
+        if not role:
+            description = "auto generated role based on owner: {0}".format(self.ldap_principal)
+            role = role_service.create(self.ldap_principal, description=description,
+                third_party=True)
+        if not role.third_party:
+            role = role_service.set_third_party(role.id, third_party_status=True)
+        roles.add(role)
+        if not self.ldap_groups_to_roles:
+            return roles
+
+        for ldap_group_name, role_name in self.ldap_groups_to_roles.items():
+            role = role_service.get_by_name(role_name)
+            if role:
+                if ldap_group_name in self.ldap_groups:
+                    current_app.logger.debug("assigning role {0} to ldap user {1}".format(self.ldap_principal, role))
+                    if not role.third_party:
+                        role = role_service.set_third_party(role.id, third_party_status=True)
+                    roles.add(role)
+        return roles
+
+    def authenticate(self):
+        """
+        orchestrate the ldap login.
+        raise an exception on error.
+        """
+        self._bind()
+        roles = self._authorize()
+        if not roles:
+            raise Exception('ldap authorization failed')
+        return self._update_user(roles)
+
+    def _bind(self):
+        """
+        authenticate an ldap user.
+        list groups for a user.
+        raise an exception on error.
+        """
+        if '@' not in self.ldap_principal:
+            self.ldap_principal = '%s@%s' % (self.ldap_principal, self.ldap_email_domain)
+        ldap_filter = 'userPrincipalName=%s' % self.ldap_principal
+
+        # query ldap for auth
+        try:
+            # build a client
+            if not self.ldap_client:
+                self.ldap_client = ldap.initialize(self.ldap_server)
+            # perform a synchronous bind
+            self.ldap_client.set_option(ldap.OPT_REFERRALS, 0)
+            if self.ldap_use_tls:
+                ldap.set_option(ldap.OPT_X_TLS_REQUIRE_CERT, ldap.OPT_X_TLS_NEVER)
+                self.ldap_client.set_option(ldap.OPT_PROTOCOL_VERSION, 3)
+                self.ldap_client.set_option(ldap.OPT_X_TLS, ldap.OPT_X_TLS_DEMAND)
+                self.ldap_client.set_option(ldap.OPT_X_TLS_DEMAND, True)
+                self.ldap_client.set_option(ldap.OPT_DEBUG_LEVEL, 255)
+            if self.ldap_cacert_file:
+                self.ldap_client.set_option(ldap.OPT_X_TLS_CACERTFILE, self.ldap_cacert_file)
+            self.ldap_client.simple_bind_s(self.ldap_principal, self.ldap_password)
+        except ldap.INVALID_CREDENTIALS:
+            self.ldap_client.unbind()
+            raise Exception('The supplied ldap credentials are invalid')
+        except ldap.SERVER_DOWN:
+            raise Exception('ldap server unavailable')
+        except ldap.LDAPError as e:
+            raise Exception("ldap error: {0}".format(e))
+
+        lgroups = self.ldap_client.search_s(self.ldap_base_dn,
+                        ldap.SCOPE_SUBTREE, ldap_filter, self.ldap_attrs)[0][1]['memberOf']
+        # lgroups is a list of utf-8 encoded strings
+        # convert to a single string of groups to allow matching
+        self.ldap_groups = b''.join(lgroups).decode('ascii')
+        self.ldap_client.unbind()
+
+    def _ldap_validate_conf(self):
+        """
+        Confirms required ldap config settings exist.
+        """
+        required_vars = [
+            'LDAP_BIND_URI',
+            'LDAP_BASE_DN',
+            'LDAP_EMAIL_DOMAIN',
+        ]
+        validate_conf(current_app, required_vars)

lemur/auth/permissions.py

@@ -33,6 +33,11 @@ class CertificatePermission(Permission):
         super(CertificatePermission, self).__init__(*needs)
 
 
+class ApiKeyCreatorPermission(Permission):
+    def __init__(self):
+        super(ApiKeyCreatorPermission, self).__init__(RoleNeed('admin'))
+
+
 RoleMember = namedtuple('role', ['method', 'value'])
 RoleMemberNeed = partial(RoleMember, 'member')
 

lemur/auth/service.py

@@ -27,6 +27,7 @@ from cryptography.hazmat.primitives import serialization
 from cryptography.hazmat.primitives.asymmetric.rsa import RSAPublicNumbers
 
 from lemur.users import service as user_service
+from lemur.api_keys import service as api_key_service
 from lemur.auth.permissions import AuthorityCreatorNeed, RoleMemberNeed
 
 
@@ -48,9 +49,9 @@ def get_rsa_public_key(n, e):
     )
 
 
-def create_token(user):
+def create_token(user, aid=None, ttl=None):
     """
-    Create a valid JWT for a given user, this token is then used to authenticate
+    Create a valid JWT for a given user/api key, this token is then used to authenticate
     sessions until the token expires.
 
     :param user:
@@ -58,10 +59,24 @@ def create_token(user):
     """
     expiration_delta = timedelta(days=int(current_app.config.get('LEMUR_TOKEN_EXPIRATION', 1)))
     payload = {
-        'sub': user.id,
         'iat': datetime.utcnow(),
         'exp': datetime.utcnow() + expiration_delta
     }
+
+    # Handle Just a User ID & User Object.
+    if isinstance(user, int):
+        payload['sub'] = user
+    else:
+        payload['sub'] = user.id
+    if aid is not None:
+        payload['aid'] = aid
+    # Custom TTLs are only supported on Access Keys.
+    if ttl is not None and aid is not None:
+        # Tokens that are forever until revoked.
+        if ttl == -1:
+            del payload['exp']
+        else:
+            payload['exp'] = ttl
     token = jwt.encode(payload, current_app.config['LEMUR_TOKEN_SECRET'])
     return token.decode('unicode_escape')
 
@@ -94,6 +109,16 @@ def login_required(f):
         except jwt.InvalidTokenError:
             return dict(message='Token is invalid'), 403
 
+        if 'aid' in payload:
+            access_key = api_key_service.get(payload['aid'])
+            if access_key.revoked:
+                return dict(message='Token has been revoked'), 403
+            if access_key.ttl != -1:
+                current_time = datetime.utcnow()
+                expired_time = datetime.fromtimestamp(access_key.issued_at + access_key.ttl)
+                if current_time >= expired_time:
+                    return dict(message='Token has expired'), 403
+
         user = user_service.get(payload['sub'])
 
         if not user.active:

lemur/auth/views.py

@@ -7,7 +7,6 @@
 """
 import jwt
 import base64
-import sys
 import requests
 
 from flask import Blueprint, current_app
@@ -21,12 +20,180 @@ from lemur.common.utils import get_psuedo_random_string
 from lemur.users import service as user_service
 from lemur.roles import service as role_service
 from lemur.auth.service import create_token, fetch_token_header, get_rsa_public_key
+import lemur.auth.ldap as ldap
 
 
 mod = Blueprint('auth', __name__)
 api = Api(mod)
 
 
+def exchange_for_access_token(code, redirect_uri, client_id, secret, access_token_url=None, verify_cert=True):
+    """
+    Exchanges authorization code for access token.
+
+    :param code:
+    :param redirect_uri:
+    :param client_id:
+    :param secret:
+    :param access_token_url:
+    :param verify_cert:
+    :return:
+    :return:
+    """
+    # take the information we have received from the provider to create a new request
+    params = {
+        'grant_type': 'authorization_code',
+        'scope': 'openid email profile address',
+        'code': code,
+        'redirect_uri': redirect_uri,
+        'client_id': client_id
+    }
+
+    # the secret and cliendId will be given to you when you signup for the provider
+    token = '{0}:{1}'.format(client_id, secret)
+
+    basic = base64.b64encode(bytes(token, 'utf-8'))
+    headers = {
+        'Content-Type': 'application/x-www-form-urlencoded',
+        'authorization': 'basic {0}'.format(basic.decode('utf-8'))
+    }
+
+    # exchange authorization code for access token.
+    r = requests.post(access_token_url, headers=headers, params=params, verify=verify_cert)
+    if r.status_code == 400:
+        r = requests.post(access_token_url, headers=headers, data=params, verify=verify_cert)
+    id_token = r.json()['id_token']
+    access_token = r.json()['access_token']
+
+    return id_token, access_token
+
+
+def validate_id_token(id_token, client_id, jwks_url):
+    """
+    Ensures that the token we receive is valid.
+
+    :param id_token:
+    :param client_id:
+    :param jwks_url:
+    :return:
+    """
+    # fetch token public key
+    header_data = fetch_token_header(id_token)
+
+    # retrieve the key material as specified by the token header
+    r = requests.get(jwks_url)
+    for key in r.json()['keys']:
+        if key['kid'] == header_data['kid']:
+            secret = get_rsa_public_key(key['n'], key['e'])
+            algo = header_data['alg']
+            break
+    else:
+        return dict(message='Key not found'), 401
+
+    # validate your token based on the key it was signed with
+    try:
+        jwt.decode(id_token, secret.decode('utf-8'), algorithms=[algo], audience=client_id)
+    except jwt.DecodeError:
+        return dict(message='Token is invalid'), 401
+    except jwt.ExpiredSignatureError:
+        return dict(message='Token has expired'), 401
+    except jwt.InvalidTokenError:
+        return dict(message='Token is invalid'), 401
+
+
+def retrieve_user(user_api_url, access_token):
+    """
+    Fetch user information from provided user api_url.
+
+    :param user_api_url:
+    :param access_token:
+    :return:
+    """
+    user_params = dict(access_token=access_token, schema='profile')
+
+    # retrieve information about the current user.
+    r = requests.get(user_api_url, params=user_params)
+    profile = r.json()
+
+    user = user_service.get_by_email(profile['email'])
+    metrics.send('successful_login', 'counter', 1)
+    return user, profile
+
+
+def create_user_roles(profile):
+    """Creates new roles based on profile information.
+
+    :param profile:
+    :return:
+    """
+    roles = []
+
+    # update their google 'roles'
+    for group in profile['googleGroups']:
+        role = role_service.get_by_name(group)
+        if not role:
+            role = role_service.create(group, description='This is a google group based role created by Lemur', third_party=True)
+        if not role.third_party:
+            role = role_service.set_third_party(role.id, third_party_status=True)
+        roles.append(role)
+
+    role = role_service.get_by_name(profile['email'])
+
+    if not role:
+        role = role_service.create(profile['email'], description='This is a user specific role', third_party=True)
+    if not role.third_party:
+        role = role_service.set_third_party(role.id, third_party_status=True)
+
+    roles.append(role)
+
+    # every user is an operator (tied to a default role)
+    if current_app.config.get('LEMUR_DEFAULT_ROLE'):
+        default = role_service.get_by_name(current_app.config['LEMUR_DEFAULT_ROLE'])
+        if not default:
+            default = role_service.create(current_app.config['LEMUR_DEFAULT_ROLE'], description='This is the default Lemur role.')
+        if not default.third_party:
+            role_service.set_third_party(default.id, third_party_status=True)
+        roles.append(default)
+
+    return roles
+
+
+def update_user(user, profile, roles):
+    """Updates user with current profile information and associated roles.
+
+    :param user:
+    :param profile:
+    :param roles:
+    """
+
+    # if we get an sso user create them an account
+    if not user:
+        user = user_service.create(
+            profile['email'],
+            get_psuedo_random_string(),
+            profile['email'],
+            True,
+            profile.get('thumbnailPhotoUrl'),
+            roles
+        )
+
+    else:
+        # we add 'lemur' specific roles, so they do not get marked as removed
+        for ur in user.roles:
+            if not ur.third_party:
+                roles.append(ur)
+
+        # update any changes to the user
+        user_service.update(
+            user.id,
+            profile['email'],
+            profile['email'],
+            True,
+            profile.get('thumbnailPhotoUrl'),  # profile isn't google+ enabled
+            roles
+        )
+
+
 class Login(Resource):
     """
     Provides an endpoint for Lemur's basic authentication. It takes a username and password
@@ -94,6 +261,7 @@ class Login(Resource):
         else:
             user = user_service.get_by_username(args['username'])
 
+        # default to local authentication
         if user and user.check_password(args['password']) and user.active:
             # Tell Flask-Principal the identity changed
             identity_changed.send(current_app._get_current_object(),
@@ -102,8 +270,26 @@ class Login(Resource):
             metrics.send('successful_login', 'counter', 1)
             return dict(token=create_token(user))
 
+        # try ldap login
+        if current_app.config.get("LDAP_AUTH"):
+            try:
+                ldap_principal = ldap.LdapPrincipal(args)
+                user = ldap_principal.authenticate()
+                if user and user.active:
+                    # Tell Flask-Principal the identity changed
+                    identity_changed.send(current_app._get_current_object(),
+                                  identity=Identity(user.id))
+                    metrics.send('successful_login', 'counter', 1)
+                    return dict(token=create_token(user))
+            except Exception as e:
+                    current_app.logger.error("ldap error: {0}".format(e))
+                    ldap_message = 'ldap error: %s' % e
+                    metrics.send('invalid_login', 'counter', 1)
+                    return dict(message=ldap_message), 403
+
+        # if not valid user - no certificates for you
         metrics.send('invalid_login', 'counter', 1)
-        return dict(message='The supplied credentials are invalid'), 401
+        return dict(message='The supplied credentials are invalid'), 403
 
 
 class Ping(Resource):
@@ -112,14 +298,17 @@ class Ping(Resource):
     this example we use an OpenIDConnect authentication flow, that is essentially OAuth2 underneath. If you have an
     OAuth2 provider you want to use Lemur there would be two steps:
 
-    1. Define your own class that inherits from :class:`flask.ext.restful.Resource` and create the HTTP methods the \
+    1. Define your own class that inherits from :class:`flask_restful.Resource` and create the HTTP methods the \
-    provider uses for it's callbacks.
+    provider uses for its callbacks.
     2. Add or change the Lemur AngularJS Configuration to point to your new provider
     """
     def __init__(self):
         self.reqparse = reqparse.RequestParser()
         super(Ping, self).__init__()
 
+    def get(self):
+        return 'Redirecting...'
+
     def post(self):
         self.reqparse.add_argument('clientId', type=str, required=True, location='json')
         self.reqparse.add_argument('redirectUri', type=str, required=True, location='json')
@@ -127,111 +316,26 @@ class Ping(Resource):
 
         args = self.reqparse.parse_args()
 
-        # take the information we have received from the provider to create a new request
-        params = {
-            'client_id': args['clientId'],
-            'grant_type': 'authorization_code',
-            'scope': 'openid email profile address',
-            'redirect_uri': args['redirectUri'],
-            'code': args['code']
-        }
-
         # you can either discover these dynamically or simply configure them
         access_token_url = current_app.config.get('PING_ACCESS_TOKEN_URL')
         user_api_url = current_app.config.get('PING_USER_API_URL')
 
-        # the secret and cliendId will be given to you when you signup for the provider
+        secret = current_app.config.get('PING_SECRET')
-        token = '{0}:{1}'.format(args['clientId'], current_app.config.get("PING_SECRET"))
 
-        basic = base64.b64encode(bytes(token, 'utf-8'))
+        id_token, access_token = exchange_for_access_token(
-        headers = {'authorization': 'basic {0}'.format(basic.decode('utf-8'))}
+            args['code'],
+            args['redirectUri'],
+            args['clientId'],
+            secret,
+            access_token_url=access_token_url
+        )
 
-        # exchange authorization code for access token.
-
-        r = requests.post(access_token_url, headers=headers, params=params)
-        id_token = r.json()['id_token']
-        access_token = r.json()['access_token']
-
-        # fetch token public key
-        header_data = fetch_token_header(id_token)
         jwks_url = current_app.config.get('PING_JWKS_URL')
+        validate_id_token(id_token, args['clientId'], jwks_url)
 
-        # retrieve the key material as specified by the token header
+        user, profile = retrieve_user(user_api_url, access_token)
-        r = requests.get(jwks_url)
+        roles = create_user_roles(profile)
-        for key in r.json()['keys']:
+        update_user(user, profile, roles)
-            if key['kid'] == header_data['kid']:
-                secret = get_rsa_public_key(key['n'], key['e'])
-                algo = header_data['alg']
-                break
-        else:
-            return dict(message='Key not found'), 403
-
-        # validate your token based on the key it was signed with
-        try:
-            jwt.decode(id_token, secret.decode('utf-8'), algorithms=[algo], audience=args['clientId'])
-        except jwt.DecodeError:
-            return dict(message='Token is invalid'), 403
-        except jwt.ExpiredSignatureError:
-            return dict(message='Token has expired'), 403
-        except jwt.InvalidTokenError:
-            return dict(message='Token is invalid'), 403
-
-        user_params = dict(access_token=access_token, schema='profile')
-
-        # retrieve information about the current user.
-        r = requests.get(user_api_url, params=user_params)
-        profile = r.json()
-
-        user = user_service.get_by_email(profile['email'])
-        metrics.send('successful_login', 'counter', 1)
-
-        # update their google 'roles'
-        roles = []
-
-        for group in profile['googleGroups']:
-            role = role_service.get_by_name(group)
-            if not role:
-                role = role_service.create(group, description='This is a google group based role created by Lemur')
-            roles.append(role)
-
-        role = role_service.get_by_name(profile['email'])
-
-        if not role:
-            role = role_service.create(profile['email'], description='This is a user specific role')
-        roles.append(role)
-
-        # if we get an sso user create them an account
-        if not user:
-            # every user is an operator (tied to a default role)
-            if current_app.config.get('LEMUR_DEFAULT_ROLE'):
-                v = role_service.get_by_name(current_app.config.get('LEMUR_DEFAULT_ROLE'))
-                if v:
-                    roles.append(v)
-
-            user = user_service.create(
-                profile['email'],
-                get_psuedo_random_string(),
-                profile['email'],
-                True,
-                profile.get('thumbnailPhotoUrl'),
-                roles
-            )
-
-        else:
-            # we add 'lemur' specific roles, so they do not get marked as removed
-            for ur in user.roles:
-                if ur.authority_id:
-                    roles.append(ur)
-
-            # update any changes to the user
-            user_service.update(
-                user.id,
-                profile['email'],
-                profile['email'],
-                True,
-                profile.get('thumbnailPhotoUrl'),  # incase profile isn't google+ enabled
-                roles
-            )
 
         if not user.active:
             metrics.send('invalid_login', 'counter', 1)
@@ -249,6 +353,9 @@ class OAuth2(Resource):
         self.reqparse = reqparse.RequestParser()
         super(OAuth2, self).__init__()
 
+    def get(self):
+        return 'Redirecting...'
+
     def post(self):
         self.reqparse.add_argument('clientId', type=str, required=True, location='json')
         self.reqparse.add_argument('redirectUri', type=str, required=True, location='json')
@@ -256,109 +363,32 @@ class OAuth2(Resource):
 
         args = self.reqparse.parse_args()
 
-        # take the information we have received from the provider to create a new request
-        params = {
-            'grant_type': 'authorization_code',
-            'scope': 'openid email profile groups',
-            'redirect_uri': args['redirectUri'],
-            'code': args['code'],
-        }
-
         # you can either discover these dynamically or simply configure them
         access_token_url = current_app.config.get('OAUTH2_ACCESS_TOKEN_URL')
         user_api_url = current_app.config.get('OAUTH2_USER_API_URL')
+        verify_cert = current_app.config.get('OAUTH2_VERIFY_CERT')
 
-        # the secret and cliendId will be given to you when you signup for the provider
+        secret = current_app.config.get('OAUTH2_SECRET')
-        token = '{0}:{1}'.format(args['clientId'], current_app.config.get("OAUTH2_SECRET"))
 
-        basic = base64.b64encode(bytes(token, 'utf-8'))
+        id_token, access_token = exchange_for_access_token(
+            args['code'],
+            args['redirectUri'],
+            args['clientId'],
+            secret,
+            access_token_url=access_token_url,
+            verify_cert=verify_cert
+        )
 
-        headers = {
+        jwks_url = current_app.config.get('PING_JWKS_URL')
-            'Content-Type': 'application/x-www-form-urlencoded',
+        validate_id_token(id_token, args['clientId'], jwks_url)
-            'authorization': 'basic {0}'.format(basic.decode('utf-8'))
-        }
 
-        # exchange authorization code for access token.
+        user, profile = retrieve_user(user_api_url, access_token)
-        r = requests.post(access_token_url, headers=headers, params=params)
+        roles = create_user_roles(profile)
-        id_token = r.json()['id_token']
+        update_user(user, profile, roles)
-        access_token = r.json()['access_token']
 
-        # fetch token public key
+        if not user.active:
-        header_data = fetch_token_header(id_token)
+            metrics.send('invalid_login', 'counter', 1)
-        jwks_url = current_app.config.get('OAUTH2_JWKS_URL')
+            return dict(message='The supplied credentials are invalid'), 403
-
-        # retrieve the key material as specified by the token header
-        r = requests.get(jwks_url)
-        for key in r.json()['keys']:
-            if key['kid'] == header_data['kid']:
-                secret = get_rsa_public_key(key['n'], key['e'])
-                algo = header_data['alg']
-                break
-        else:
-            return dict(message='Key not found'), 403
-
-        # validate your token based on the key it was signed with
-        try:
-            if sys.version_info >= (3, 0):
-                jwt.decode(id_token, secret.decode('utf-8'), algorithms=[algo], audience=args['clientId'])
-            else:
-                jwt.decode(id_token, secret, algorithms=[algo], audience=args['clientId'])
-        except jwt.DecodeError:
-            return dict(message='Token is invalid'), 403
-        except jwt.ExpiredSignatureError:
-            return dict(message='Token has expired'), 403
-        except jwt.InvalidTokenError:
-            return dict(message='Token is invalid'), 403
-
-        headers = {'authorization': 'Bearer {0}'.format(access_token)}
-
-        # retrieve information about the current user.
-        r = requests.get(user_api_url, headers=headers)
-        profile = r.json()
-
-        user = user_service.get_by_email(profile['email'])
-        metrics.send('successful_login', 'counter', 1)
-
-        # update their google 'roles'
-        roles = []
-
-        role = role_service.get_by_name(profile['email'])
-        if not role:
-            role = role_service.create(profile['email'], description='This is a user specific role')
-        roles.append(role)
-
-        # if we get an sso user create them an account
-        if not user:
-            # every user is an operator (tied to a default role)
-            if current_app.config.get('LEMUR_DEFAULT_ROLE'):
-                v = role_service.get_by_name(current_app.config.get('LEMUR_DEFAULT_ROLE'))
-                if v:
-                    roles.append(v)
-
-            user = user_service.create(
-                profile['name'],
-                get_psuedo_random_string(),
-                profile['email'],
-                True,
-                profile.get('thumbnailPhotoUrl'),
-                roles
-            )
-
-        else:
-            # we add 'lemur' specific roles, so they do not get marked as removed
-            for ur in user.roles:
-                if ur.authority_id:
-                    roles.append(ur)
-
-            # update any changes to the user
-            user_service.update(
-                user.id,
-                profile['name'],
-                profile['email'],
-                True,
-                profile.get('thumbnailPhotoUrl'),  # incase profile isn't google+ enabled
-                roles
-            )
 
         # Tell Flask-Principal the identity changed
         identity_changed.send(current_app._get_current_object(), identity=Identity(user.id))
@@ -403,7 +433,7 @@ class Google(Resource):
 
         if not user.active:
             metrics.send('invalid_login', 'counter', 1)
-            return dict(message='The supplied credentials are invalid.'), 401
+            return dict(message='The supplied credentials are invalid.'), 403
 
         if user:
             metrics.send('successful_login', 'counter', 1)

lemur/authorities/schemas.py

@@ -23,7 +23,7 @@ class AuthorityInputSchema(LemurInputSchema):
     name = fields.String(required=True)
     owner = fields.Email(required=True)
     description = fields.String()
-    common_name = fields.String(required=True, validate=validators.sensitive_domain)
+    common_name = fields.String(required=True, validate=validators.common_name)
 
     validity_start = ArrowDateTime()
     validity_end = ArrowDateTime()

lemur/authorities/service.py

@@ -16,7 +16,7 @@ from lemur.roles import service as role_service
 from lemur.certificates.service import upload
 
 
-def update(authority_id, description=None, owner=None, active=None, roles=None):
+def update(authority_id, description, owner, active, roles):
     """
     Update an authority with new values.
 
@@ -26,12 +26,11 @@ def update(authority_id, description=None, owner=None, active=None, roles=None):
     """
     authority = get(authority_id)
 
-    if roles:
+    authority.roles = roles
-        authority.roles = roles
-
     authority.active = active
     authority.description = description
     authority.owner = owner
+
     return database.update(authority)
 
 

lemur/certificates/cli.py

@@ -6,17 +6,35 @@
 .. moduleauthor:: Kevin Glisson <kglisson@netflix.com>
 """
 import sys
+import multiprocessing
+from tabulate import tabulate
+from sqlalchemy import or_
 
 from flask import current_app
 
 from flask_script import Manager
+from flask_principal import Identity, identity_changed
+
 
 from lemur import database
+from lemur.extensions import sentry
 from lemur.extensions import metrics
+from lemur.plugins.base import plugins
 from lemur.deployment import service as deployment_service
 from lemur.endpoints import service as endpoint_service
 from lemur.notifications.messaging import send_rotation_notification
-from lemur.certificates.service import reissue_certificate, get_certificate_primitives, get_all_pending_reissue, get_by_name, get_all_certs
+from lemur.domains.models import Domain
+from lemur.authorities.models import Authority
+from lemur.certificates.schemas import CertificateOutputSchema
+from lemur.certificates.models import Certificate
+from lemur.certificates.service import (
+    reissue_certificate,
+    get_certificate_primitives,
+    get_all_pending_reissue,
+    get_by_name,
+    get_all_certs,
+    get
+)
 
 from lemur.certificates.verify import verify_string
 
@@ -29,28 +47,19 @@ def print_certificate_details(details):
     :param details:
     :return:
     """
+    details, errors = CertificateOutputSchema().dump(details)
     print("[+] Re-issuing certificate with the following details: ")
     print(
         "\t[+] Common Name: {common_name}\n"
         "\t[+] Subject Alternate Names: {sans}\n"
         "\t[+] Authority: {authority_name}\n"
         "\t[+] Validity Start: {validity_start}\n"
-        "\t[+] Validity End: {validity_end}\n"
+        "\t[+] Validity End: {validity_end}\n".format(
-        "\t[+] Organization: {organization}\n"
+            common_name=details['commonName'],
-        "\t[+] Organizational Unit: {organizational_unit}\n"
+            sans=",".join(x['value'] for x in details['extensions']['subAltNames']['names']) or None,
-        "\t[+] Country: {country}\n"
+            authority_name=details['authority']['name'],
-        "\t[+] State: {state}\n"
+            validity_start=details['validityStart'],
-        "\t[+] Location: {location}".format(
+            validity_end=details['validityEnd']
-            common_name=details['common_name'],
-            sans=",".join(x['value'] for x in details['extensions']['sub_alt_names']['names']) or None,
-            authority_name=details['authority'].name,
-            validity_start=details['validity_start'].isoformat(),
-            validity_end=details['validity_end'].isoformat(),
-            organization=details['organization'],
-            organizational_unit=details['organizational_unit'],
-            country=details['country'],
-            state=details['state'],
-            location=details['location']
         )
     )
 
@@ -123,22 +132,16 @@ def request_reissue(certificate, commit):
     :param commit:
     :return:
     """
-    details = get_certificate_primitives(certificate)
+    # set the lemur identity for all cli commands
+    identity_changed.send(current_app._get_current_object(), identity=Identity(1))
 
+    details = get_certificate_primitives(certificate)
     print_certificate_details(details)
+
     if commit:
-        try:
+        new_cert = reissue_certificate(certificate, replace=True)
-            new_cert = reissue_certificate(certificate, replace=True)
+        metrics.send('certificate_reissue_success', 'counter', 1)
-            metrics.send('certificate_reissue_success', 'counter', 1)
+        print("[+] New certificate named: {0}".format(new_cert.name))
-            print("[+] New certificate named: {0}".format(new_cert.name))
-        except Exception as e:
-            metrics.send('certificate_reissue_failure', 'counter', 1)
-            print(
-                "[!] Failed to reissue certificate {1} reason: {2}".format(
-                    certificate.name,
-                    e
-                )
-            )
 
 
 @manager.option('-e', '--endpoint', dest='endpoint_name', help='Name of the endpoint you wish to rotate.')
@@ -156,34 +159,36 @@ def rotate(endpoint_name, new_certificate_name, old_certificate_name, message, c
 
     print("[+] Starting endpoint rotation.")
 
-    old_cert = validate_certificate(old_certificate_name)
+    try:
-    new_cert = validate_certificate(new_certificate_name)
+        old_cert = validate_certificate(old_certificate_name)
-    endpoint = validate_endpoint(endpoint_name)
+        new_cert = validate_certificate(new_certificate_name)
+        endpoint = validate_endpoint(endpoint_name)
 
-    if endpoint and new_cert:
+        if endpoint and new_cert:
-        print("[+] Rotating endpoint: {0} to certificate {1}".format(endpoint.name, new_cert.name))
+            print("[+] Rotating endpoint: {0} to certificate {1}".format(endpoint.name, new_cert.name))
-        request_rotation(endpoint, new_cert, message, commit)
-
-    elif old_cert and new_cert:
-        print("[+] Rotating all endpoints from {0} to {1}".format(old_cert.name, new_cert.name))
-
-        for endpoint in old_cert.endpoints:
-            print("[+] Rotating {0}".format(endpoint.name))
             request_rotation(endpoint, new_cert, message, commit)
 
-    else:
+        elif old_cert and new_cert:
-        print("[+] Rotating all endpoints that have new certificates available")
+            print("[+] Rotating all endpoints from {0} to {1}".format(old_cert.name, new_cert.name))
-        for endpoint in endpoint_service.get_all_pending_rotation():
-            if len(endpoint.certificate.replaced) == 1:
-                print("[+] Rotating {0} to {1}".format(endpoint.name, endpoint.certificate.replaced[0].name))
-                request_rotation(endpoint, endpoint.certificate.replaced[0], message, commit)
-            else:
-                metrics.send('endpoint_rotation_failure', 'counter', 1)
-                print("[!] Failed to rotate endpoint {0} reason: Multiple replacement certificates found.".format(
-                    endpoint.name
-                ))
 
-    print("[+] Done!")
+            for endpoint in old_cert.endpoints:
+                print("[+] Rotating {0}".format(endpoint.name))
+                request_rotation(endpoint, new_cert, message, commit)
+
+        else:
+            print("[+] Rotating all endpoints that have new certificates available")
+            for endpoint in endpoint_service.get_all_pending_rotation():
+                if len(endpoint.certificate.replaced) == 1:
+                    print("[+] Rotating {0} to {1}".format(endpoint.name, endpoint.certificate.replaced[0].name))
+                    request_rotation(endpoint, endpoint.certificate.replaced[0], message, commit)
+                else:
+                    metrics.send('endpoint_rotation_failure', 'counter', 1)
+                    print("[!] Failed to rotate endpoint {0} reason: Multiple replacement certificates found.".format(
+                        endpoint.name
+                    ))
+        print("[+] Done!")
+    except Exception as e:
+        sentry.captureException()
 
 
 @manager.option('-o', '--old-certificate', dest='old_certificate_name', help='Name of the certificate you wish to reissue.')
@@ -199,16 +204,114 @@ def reissue(old_certificate_name, commit):
 
     print("[+] Starting certificate re-issuance.")
 
-    old_cert = validate_certificate(old_certificate_name)
+    try:
+        old_cert = validate_certificate(old_certificate_name)
 
-    if not old_cert:
+        if not old_cert:
-        for certificate in get_all_pending_reissue():
+            for certificate in get_all_pending_reissue():
-            print("[+] {0} is eligible for re-issuance".format(certificate.name))
+                print("[+] {0} is eligible for re-issuance".format(certificate.name))
-            request_reissue(certificate, commit)
+                request_reissue(certificate, commit)
-    else:
+        else:
-        request_reissue(old_cert, commit)
+            request_reissue(old_cert, commit)
 
-    print("[+] Done!")
+        print("[+] Done!")
+    except Exception as e:
+        sentry.captureException()
+        metrics.send('certificate_reissue_failure', 'counter', 1)
+        print(
+            "[!] Failed to reissue certificates. Reason: {}".format(
+                e
+            )
+        )
+
+
+@manager.option('-f', '--fqdns', dest='fqdns', help='FQDNs to query. Multiple fqdns specified via comma.')
+@manager.option('-i', '--issuer', dest='issuer', help='Issuer to query for.')
+@manager.option('-o', '--owner', dest='owner', help='Owner to query for.')
+@manager.option('-e', '--expired', dest='expired', type=bool, default=False, help='Include expired certificates.')
+def query(fqdns, issuer, owner, expired):
+    """Prints certificates that match the query params."""
+    table = []
+
+    q = database.session_query(Certificate)
+
+    sub_query = database.session_query(Authority.id) \
+        .filter(Authority.name.ilike('%{0}%'.format(issuer))) \
+        .subquery()
+
+    q = q.filter(
+        or_(
+            Certificate.issuer.ilike('%{0}%'.format(issuer)),
+            Certificate.authority_id.in_(sub_query)
+        )
+    )
+
+    q = q.filter(Certificate.owner.ilike('%{0}%'.format(owner)))
+
+    if not expired:
+        q = q.filter(Certificate.expired == False)  # noqa
+
+    for f in fqdns.split(','):
+        q = q.filter(
+            or_(
+                Certificate.cn.ilike('%{0}%'.format(f)),
+                Certificate.domains.any(Domain.name.ilike('%{0}%'.format(f)))
+            )
+        )
+
+    for c in q.all():
+        table.append([c.id, c.name, c.owner, c.issuer])
+
+    print(tabulate(table, headers=['Id', 'Name', 'Owner', 'Issuer'], tablefmt='csv'))
+
+
+def worker(data, commit, reason):
+    parts = [x for x in data.split(' ') if x]
+    try:
+        cert = get(int(parts[0].strip()))
+        plugin = plugins.get(cert.authority.plugin_name)
+
+        print('[+] Revoking certificate. Id: {0} Name: {1}'.format(cert.id, cert.name))
+        if commit:
+            plugin.revoke_certificate(cert, reason)
+
+    except Exception as e:
+        sentry.captureException()
+        metrics.send('certificate_revoke_failure', 'counter', 1)
+        print(
+            "[!] Failed to revoke certificates. Reason: {}".format(
+                e
+            )
+        )
+
+
+@manager.command
+def clear_pending():
+    """
+    Function clears all pending certificates.
+    :return:
+    """
+    v = plugins.get('verisign-issuer')
+    v.clear_pending_certificates()
+
+
+@manager.option('-p', '--path', dest='path', help='Absolute file path to a Lemur query csv.')
+@manager.option('-r', '--reason', dest='reason', help='Reason to revoke certificate.')
+@manager.option('-c', '--commit', dest='commit', action='store_true', default=False, help='Persist changes.')
+def revoke(path, reason, commit):
+    """
+    Revokes given certificate.
+    """
+    if commit:
+        print("[!] Running in COMMIT mode.")
+
+    print("[+] Starting certificate revocation.")
+
+    with open(path, 'r') as f:
+        args = [[x, commit, reason] for x in f.readlines()[2:]]
+
+    with multiprocessing.Pool(processes=3) as pool:
+        pool.starmap(worker, args)
 
 
 @manager.command
@@ -227,9 +330,10 @@ def check_revoked():
             else:
                 status = verify_string(cert.body, "")
 
-            cert.status = 'valid' if status else 'invalid'
+            cert.status = 'valid' if status else 'revoked'
 
         except Exception as e:
+            sentry.captureException()
             current_app.logger.exception(e)
             cert.status = 'unknown'
 

lemur/certificates/hooks.py

@@ -0,0 +1,38 @@
+"""
+Debugging hooks for dumping imported or generated CSR and certificate details to stdout via OpenSSL.
+
+.. module: lemur.certificates.hooks
+    :platform: Unix
+    :copyright: (c) 2016-2017 by Marti Raudsepp, see AUTHORS for more
+    :license: Apache, see LICENSE for more details.
+
+.. moduleauthor:: Marti Raudsepp <marti@juffo.org>
+"""
+import subprocess
+
+from flask import current_app
+
+from lemur.certificates.service import csr_created, csr_imported, certificate_issued, certificate_imported
+
+
+def csr_dump_handler(sender, csr, **kwargs):
+    try:
+        subprocess.run(['openssl', 'req', '-text', '-noout', '-reqopt', 'no_sigdump,no_pubkey'],
+                       input=csr.encode('utf8'))
+    except Exception as err:
+        current_app.logger.warning("Error inspecting CSR: %s", err)
+
+
+def cert_dump_handler(sender, certificate, **kwargs):
+    try:
+        subprocess.run(['openssl', 'x509', '-text', '-noout', '-certopt', 'no_sigdump,no_pubkey'],
+                       input=certificate.body.encode('utf8'))
+    except Exception as err:
+        current_app.logger.warning("Error inspecting certificate: %s", err)
+
+
+def activate_debug_dump():
+    csr_created.connect(csr_dump_handler)
+    csr_imported.connect(csr_dump_handler)
+    certificate_issued.connect(cert_dump_handler)
+    certificate_imported.connect(cert_dump_handler)

lemur/certificates/models.py

@@ -6,6 +6,7 @@
 .. moduleauthor:: Kevin Glisson <kglisson@netflix.com>
 """
 import arrow
+from datetime import timedelta
 
 from flask import current_app
 
@@ -15,7 +16,7 @@ from cryptography.hazmat.primitives.asymmetric import rsa
 from idna.core import InvalidCodepoint
 
 from sqlalchemy.orm import relationship
-from sqlalchemy.sql.expression import case
+from sqlalchemy.sql.expression import case, extract
 from sqlalchemy.ext.hybrid import hybrid_property
 from sqlalchemy import event, Integer, ForeignKey, String, PassiveDefault, func, Column, Text, Boolean
 
@@ -24,6 +25,7 @@ from sqlalchemy_utils.types.arrow import ArrowType
 import lemur.common.utils
 
 from lemur.database import db
+from lemur.extensions import sentry
 
 from lemur.utils import Vault
 from lemur.common import defaults
@@ -37,6 +39,7 @@ from lemur.models import certificate_associations, certificate_source_associatio
     certificate_replacement_associations, roles_certificates
 
 from lemur.domains.models import Domain
+from lemur.policies.models import RotationPolicy
 
 
 def get_sequence(name):
@@ -44,29 +47,35 @@ def get_sequence(name):
         return name, None
 
     parts = name.split('-')
-    end = parts.pop(-1)
-    root = '-'.join(parts)
-
-    if len(end) == 8:
-        return root + '-' + end, None
 
+    # see if we have an int at the end of our name
     try:
-        end = int(end)
+        seq = int(parts[-1])
     except ValueError:
-        end = None
+        return name, None
 
-    return root, end
+    # we might have a date at the end of our name
+    if len(parts[-1]) == 8:
+        return name, None
+
+    root = '-'.join(parts[:-1])
+    return root, seq
 
 
-def get_or_increase_name(name):
+def get_or_increase_name(name, serial):
-    name = '-'.join(name.strip().split(' '))
     certificates = Certificate.query.filter(Certificate.name.ilike('{0}%'.format(name))).all()
 
     if not certificates:
         return name
 
+    serial_name = '{0}-{1}'.format(name, hex(int(serial))[2:].upper())
+    certificates = Certificate.query.filter(Certificate.name.ilike('{0}%'.format(serial_name))).all()
+
+    if not certificates:
+        return serial_name
+
     ends = [0]
-    root, end = get_sequence(name)
+    root, end = get_sequence(serial_name)
     for cert in certificates:
         root, end = get_sequence(cert.name)
         if end:
@@ -78,8 +87,9 @@ def get_or_increase_name(name):
 class Certificate(db.Model):
     __tablename__ = 'certificates'
     id = Column(Integer, primary_key=True)
+    external_id = Column(String(128))
     owner = Column(String(128), nullable=False)
-    name = Column(String(128), unique=True)
+    name = Column(String(256), unique=True)
     description = Column(String(1024))
     notify = Column(Boolean, default=True)
 
@@ -102,10 +112,10 @@ class Certificate(db.Model):
     san = Column(String(1024))  # TODO this should be migrated to boolean
 
     rotation = Column(Boolean, default=False)
-
     user_id = Column(Integer, ForeignKey('users.id'))
     authority_id = Column(Integer, ForeignKey('authorities.id', ondelete="CASCADE"))
     root_authority_id = Column(Integer, ForeignKey('authorities.id', ondelete="CASCADE"))
+    rotation_policy_id = Column(Integer, ForeignKey('rotation_policies.id'))
 
     notifications = relationship('Notification', secondary=certificate_notification_associations, backref='certificate')
     destinations = relationship('Destination', secondary=certificate_destination_associations, backref='certificate')
@@ -120,6 +130,9 @@ class Certificate(db.Model):
 
     logs = relationship('Log', backref='certificate')
     endpoints = relationship('Endpoint', backref='certificate')
+    rotation_policy = relationship("RotationPolicy")
+
+    sensitive_fields = ('private_key',)
 
     def __init__(self, **kwargs):
         cert = lemur.common.utils.parse_certificate(kwargs['body'])
@@ -129,12 +142,14 @@ class Certificate(db.Model):
         self.san = defaults.san(cert)
         self.not_before = defaults.not_before(cert)
         self.not_after = defaults.not_after(cert)
+        self.serial = defaults.serial(cert)
 
         # when destinations are appended they require a valid name.
         if kwargs.get('name'):
-            self.name = get_or_increase_name(kwargs['name'])
+            self.name = get_or_increase_name(defaults.text_to_slug(kwargs['name']), self.serial)
         else:
-            self.name = get_or_increase_name(defaults.certificate_name(self.cn, self.issuer, self.not_before, self.not_after, self.san))
+            self.name = get_or_increase_name(
+                defaults.certificate_name(self.cn, self.issuer, self.not_before, self.not_after, self.san), self.serial)
 
         self.owner = kwargs['owner']
         self.body = kwargs['body'].strip()
@@ -152,9 +167,10 @@ class Certificate(db.Model):
         self.roles = list(set(kwargs.get('roles', [])))
         self.replaces = kwargs.get('replaces', [])
         self.rotation = kwargs.get('rotation')
+        self.rotation_policy = kwargs.get('rotation_policy')
         self.signing_algorithm = defaults.signing_algorithm(cert)
         self.bits = defaults.bitstrength(cert)
-        self.serial = defaults.serial(cert)
+        self.external_id = kwargs.get('external_id')
 
         for domain in defaults.domains(cert):
             self.domains.append(Domain(name=domain))
@@ -240,6 +256,33 @@ class Certificate(db.Model):
             else_=False
         )
 
+    @hybrid_property
+    def in_rotation_window(self):
+        """
+        Determines if a certificate is available for rotation based
+        on the rotation policy associated.
+        :return:
+        """
+        now = arrow.utcnow()
+        end = now + timedelta(days=self.rotation_policy.days)
+
+        if self.not_after <= end:
+            return True
+
+    @in_rotation_window.expression
+    def in_rotation_window(cls):
+        """
+        Determines if a certificate is available for rotation based
+        on the rotation policy associated.
+        :return:
+        """
+        return case(
+            [
+                (extract('day', cls.not_after - func.now()) <= RotationPolicy.days, True)
+            ],
+            else_=False
+        )
+
     @property
     def extensions(self):
         # setup default values
@@ -291,20 +334,15 @@ class Certificate(db.Model):
                 else:
                     current_app.logger.warning('Custom OIDs not yet supported for clone operation.')
         except InvalidCodepoint as e:
+            sentry.captureException()
             current_app.logger.warning('Unable to parse extensions due to underscore in dns name')
+        except ValueError as e:
+            sentry.captureException()
+            current_app.logger.warning('Unable to parse')
+            current_app.logger.exception(e)
 
         return return_extensions
 
-    def get_arn(self, account_number):
-        """
-        Generate a valid AWS IAM arn
-
-        :rtype : str
-        :param account_number:
-        :return:
-        """
-        return "arn:aws:iam::{}:server-certificate/{}".format(account_number, self.name)
-
     def __repr__(self):
         return "Certificate(name={name})".format(name=self.name)
 
@@ -325,8 +363,10 @@ def update_destinations(target, value, initiator):
         if target.private_key:
             destination_plugin.upload(target.name, target.body, target.private_key, target.chain, value.options)
     except Exception as e:
+        sentry.captureException()
         current_app.logger.exception(e)
-        metrics.send('destination_upload_failure', 'counter', 1, metric_tags={'certificate': target.name, 'destination': value.label})
+        metrics.send('destination_upload_failure', 'counter', 1,
+                     metric_tags={'certificate': target.name, 'destination': value.label})
 
 
 @event.listens_for(Certificate.replaces, 'append')

lemur/certificates/schemas.py

@@ -9,8 +9,17 @@ from flask import current_app
 from marshmallow import fields, validate, validates_schema, post_load, pre_load
 from marshmallow.exceptions import ValidationError
 
-from lemur.schemas import AssociatedAuthoritySchema, AssociatedDestinationSchema, AssociatedCertificateSchema, \
+from lemur.schemas import (
-    AssociatedNotificationSchema, PluginInputSchema, ExtensionSchema, AssociatedRoleSchema, EndpointNestedOutputSchema
+    AssociatedAuthoritySchema,
+    AssociatedDestinationSchema,
+    AssociatedCertificateSchema,
+    AssociatedNotificationSchema,
+    PluginInputSchema,
+    ExtensionSchema,
+    AssociatedRoleSchema,
+    EndpointNestedOutputSchema,
+    AssociatedRotationPolicySchema
+)
 
 from lemur.authorities.schemas import AuthorityNestedOutputSchema
 from lemur.destinations.schemas import DestinationNestedOutputSchema
@@ -18,17 +27,18 @@ from lemur.notifications.schemas import NotificationNestedOutputSchema
 from lemur.roles.schemas import RoleNestedOutputSchema
 from lemur.domains.schemas import DomainNestedOutputSchema
 from lemur.users.schemas import UserNestedOutputSchema
+from lemur.policies.schemas import RotationPolicyNestedOutputSchema
 
 from lemur.common.schema import LemurInputSchema, LemurOutputSchema
 from lemur.common import validators, missing
 from lemur.notifications import service as notification_service
 
-from lemur.common.fields import ArrowDateTime
+from lemur.common.fields import ArrowDateTime, Hex
 
 
 class CertificateSchema(LemurInputSchema):
     owner = fields.Email(required=True)
-    description = fields.String()
+    description = fields.String(missing='', allow_none=True)
 
 
 class CertificateCreationSchema(CertificateSchema):
@@ -45,7 +55,7 @@ class CertificateCreationSchema(CertificateSchema):
 
 class CertificateInputSchema(CertificateCreationSchema):
     name = fields.String()
-    common_name = fields.String(required=True, validate=validators.sensitive_domain)
+    common_name = fields.String(required=True, validate=validators.common_name)
     authority = fields.Nested(AssociatedAuthoritySchema, required=True)
 
     validity_start = ArrowDateTime()
@@ -63,6 +73,7 @@ class CertificateInputSchema(CertificateCreationSchema):
 
     notify = fields.Boolean(default=True)
     rotation = fields.Boolean()
+    rotation_policy = fields.Nested(AssociatedRotationPolicySchema, missing={'name': 'default'}, default={'name': 'default'})
 
     # certificate body fields
     organizational_unit = fields.String(missing=lambda: current_app.config.get('LEMUR_DEFAULT_ORGANIZATIONAL_UNIT'))
@@ -73,6 +84,11 @@ class CertificateInputSchema(CertificateCreationSchema):
 
     extensions = fields.Nested(ExtensionSchema)
 
+    @validates_schema
+    def validate_authority(self, data):
+        if not data['authority'].active:
+            raise ValidationError("The authority is inactive.", ['authority'])
+
     @validates_schema
     def validate_dates(self, data):
         validators.dates(data)
@@ -124,7 +140,7 @@ class CertificateNestedOutputSchema(LemurOutputSchema):
     creator = fields.Nested(UserNestedOutputSchema)
     description = fields.String()
 
-    status = fields.Boolean()
+    status = fields.String()
 
     bits = fields.Integer()
     body = fields.String()
@@ -133,8 +149,9 @@ class CertificateNestedOutputSchema(LemurOutputSchema):
 
     rotation = fields.Boolean()
     notify = fields.Boolean()
+    rotation_policy = fields.Nested(RotationPolicyNestedOutputSchema)
 
-    # Note aliasing  is the first step in deprecating these fields.
+    # Note aliasing is the first step in deprecating these fields.
     cn = fields.String()  # deprecated
     common_name = fields.String(attribute='cn')
 
@@ -155,6 +172,7 @@ class CertificateCloneSchema(LemurOutputSchema):
 
 class CertificateOutputSchema(LemurOutputSchema):
     id = fields.Integer()
+    external_id = fields.String()
     bits = fields.Integer()
     body = fields.String()
     chain = fields.String()
@@ -165,7 +183,7 @@ class CertificateOutputSchema(LemurOutputSchema):
 
     rotation = fields.Boolean()
 
-    # Note aliasing  is the first step in deprecating these fields.
+    # Note aliasing is the first step in deprecating these fields.
     notify = fields.Boolean()
     active = fields.Boolean(attribute='notify')
 
@@ -181,9 +199,10 @@ class CertificateOutputSchema(LemurOutputSchema):
     owner = fields.Email()
     san = fields.Boolean()
     serial = fields.String()
+    serial_hex = Hex(attribute='serial')
     signing_algorithm = fields.String()
 
-    status = fields.Boolean()
+    status = fields.String()
     user = fields.Nested(UserNestedOutputSchema)
 
     extensions = fields.Nested(ExtensionSchema)
@@ -197,6 +216,7 @@ class CertificateOutputSchema(LemurOutputSchema):
     roles = fields.Nested(RoleNestedOutputSchema, many=True)
     endpoints = fields.Nested(EndpointNestedOutputSchema, many=True, missing=[])
     replaced_by = fields.Nested(CertificateNestedOutputSchema, many=True, attribute='replaced')
+    rotation_policy = fields.Nested(RotationPolicyNestedOutputSchema)
 
 
 class CertificateUploadInputSchema(CertificateCreationSchema):
@@ -234,6 +254,10 @@ class CertificateNotificationOutputSchema(LemurOutputSchema):
     endpoints = fields.Nested(EndpointNestedOutputSchema, many=True, missing=[])
 
 
+class CertificateRevokeSchema(LemurInputSchema):
+    comments = fields.String()
+
+
 certificate_input_schema = CertificateInputSchema()
 certificate_output_schema = CertificateOutputSchema()
 certificates_output_schema = CertificateOutputSchema(many=True)
@@ -241,3 +265,4 @@ certificate_upload_input_schema = CertificateUploadInputSchema()
 certificate_export_input_schema = CertificateExportInputSchema()
 certificate_edit_input_schema = CertificateEditInputSchema()
 certificate_notification_output_schema = CertificateNotificationOutputSchema()
+certificate_revoke_schema = CertificateRevokeSchema()

lemur/certificates/service.py

@@ -6,7 +6,6 @@
 .. moduleauthor:: Kevin Glisson <kglisson@netflix.com>
 """
 import arrow
-from datetime import timedelta
 
 from flask import current_app
 from sqlalchemy import func, or_, not_, cast, Boolean, Integer
@@ -16,7 +15,7 @@ from cryptography.hazmat.backends import default_backend
 from cryptography.hazmat.primitives import hashes, serialization
 
 from lemur import database
-from lemur.extensions import metrics
+from lemur.extensions import metrics, signals
 from lemur.plugins.base import plugins
 from lemur.common.utils import generate_private_key
 
@@ -32,6 +31,12 @@ from lemur.certificates.schemas import CertificateOutputSchema, CertificateInput
 from lemur.roles import service as role_service
 
 
+csr_created = signals.signal('csr_created', "CSR generated")
+csr_imported = signals.signal('csr_imported', "CSR imported from external source")
+certificate_issued = signals.signal('certificate_issued', "Authority issued a certificate")
+certificate_imported = signals.signal('certificate_imported', "Certificate imported from external source")
+
+
 def get(cert_id):
     """
     Retrieves certificate by its ID.
@@ -52,6 +57,15 @@ def get_by_name(name):
     return database.get(Certificate, name, field='name')
 
 
+def get_by_serial(serial):
+    """
+    Retrieves certificate by it's Serial.
+    :param serial:
+    :return:
+    """
+    return Certificate.query.filter(Certificate.serial == serial).all()
+
+
 def delete(cert_id):
     """
     Delete's a certificate.
@@ -85,20 +99,15 @@ def get_all_pending_reissue():
     """
     Retrieves all certificates that need to be rotated.
 
-    Must be X days from expiration, uses `LEMUR_DEFAULT_ROTATION_INTERVAL`
+    Must be X days from expiration, uses the certificates rotation
-    to determine how many days from expiration the certificate must be
+    policy to determine how many days from expiration the certificate must be
     for rotation to be pending.
 
     :return:
     """
-    now = arrow.utcnow()
-    interval = current_app.config.get('LEMUR_DEFAULT_ROTATION_INTERVAL', 30)
-    end = now + timedelta(days=interval)
-
     return Certificate.query.filter(Certificate.rotation == True)\
-        .filter(Certificate.endpoints.any())\
         .filter(not_(Certificate.replaced.any()))\
-        .filter(Certificate.not_after <= end.format('YYYY-MM-DD')).all()  # noqa
+        .filter(Certificate.in_rotation_window == True).all()  # noqa
 
 
 def find_duplicates(cert):
@@ -174,12 +183,14 @@ def mint(**kwargs):
     # allow the CSR to be specified by the user
     if not kwargs.get('csr'):
         csr, private_key = create_csr(**kwargs)
+        csr_created.send(authority=authority, csr=csr)
     else:
         csr = str(kwargs.get('csr'))
         private_key = None
+        csr_imported.send(authority=authority, csr=csr)
 
-    cert_body, cert_chain = issuer.create_certificate(csr, kwargs)
+    cert_body, cert_chain, external_id = issuer.create_certificate(csr, kwargs)
-    return cert_body, private_key, cert_chain,
+    return cert_body, private_key, cert_chain, external_id
 
 
 def import_certificate(**kwargs):
@@ -222,17 +233,21 @@ def upload(**kwargs):
     cert = database.create(cert)
 
     kwargs['creator'].certificates.append(cert)
-    return database.update(cert)
+
+    cert = database.update(cert)
+    certificate_imported.send(certificate=cert, authority=cert.authority)
+    return cert
 
 
 def create(**kwargs):
     """
     Creates a new certificate.
     """
-    cert_body, private_key, cert_chain = mint(**kwargs)
+    cert_body, private_key, cert_chain, external_id = mint(**kwargs)
     kwargs['body'] = cert_body
     kwargs['private_key'] = private_key
     kwargs['chain'] = cert_chain
+    kwargs['external_id'] = external_id
 
     roles = create_certificate_roles(**kwargs)
 
@@ -245,6 +260,8 @@ def create(**kwargs):
 
     kwargs['creator'].certificates.append(cert)
     cert.authority = kwargs['authority']
+    certificate_issued.send(certificate=cert, authority=cert.authority)
+
     database.commit()
 
     metrics.send('certificate_issued', 'counter', 1, metric_tags=dict(owner=cert.owner, issuer=cert.issuer))
@@ -284,7 +301,6 @@ def render(args):
                     Certificate.authority_id.in_(sub_query)
                 )
             )
-            return database.sort_and_page(query, Certificate, args)
 
         elif 'destination' in terms:
             query = query.filter(Certificate.destinations.any(Destination.id == terms[1]))
@@ -337,8 +353,9 @@ def create_csr(**csr_config):
     private_key = generate_private_key(csr_config.get('key_type'))
 
     builder = x509.CertificateSigningRequestBuilder()
-    name_list = [x509.NameAttribute(x509.OID_COMMON_NAME, csr_config['common_name']),
+    name_list = [x509.NameAttribute(x509.OID_COMMON_NAME, csr_config['common_name'])]
-                 x509.NameAttribute(x509.OID_EMAIL_ADDRESS, csr_config['owner'])]
+    if current_app.config.get('LEMUR_OWNER_EMAIL_IN_SUBJECT', True):
+        name_list.append(x509.NameAttribute(x509.OID_EMAIL_ADDRESS, csr_config['owner']))
     if 'organization' in csr_config and csr_config['organization'].strip():
         name_list.append(x509.NameAttribute(x509.OID_ORGANIZATION_NAME, csr_config['organization']))
     if 'organizational_unit' in csr_config and csr_config['organizational_unit'].strip():
@@ -359,7 +376,8 @@ def create_csr(**csr_config):
             if k in critical_extensions:
                 current_app.logger.debug('Adding Critical Extension: {0} {1}'.format(k, v))
                 if k == 'sub_alt_names':
-                    builder = builder.add_extension(v['names'], critical=True)
+                    if v['names']:
+                        builder = builder.add_extension(v['names'], critical=True)
                 else:
                     builder = builder.add_extension(v, critical=True)
 
@@ -475,6 +493,12 @@ def get_certificate_primitives(certificate):
     # we will rely on the Lemur generated name
     data.pop('name', None)
 
+    # TODO this can be removed once we migrate away from cn
+    data['cn'] = data['common_name']
+
+    # needed until we move off not_*
+    data['not_before'] = start
+    data['not_after'] = end
     data['validity_start'] = start
     data['validity_end'] = end
     return data

lemur/certificates/verify.py

@@ -7,7 +7,7 @@
 """
 import requests
 import subprocess
-from requests.exceptions import ConnectionError
+from requests.exceptions import ConnectionError, InvalidSchema
 from cryptography import x509
 from cryptography.hazmat.backends import default_backend
 
@@ -19,7 +19,7 @@ def ocsp_verify(cert_path, issuer_chain_path):
     """
     Attempts to verify a certificate via OCSP. OCSP is a more modern version
     of CRL in that it will query the OCSP URI in order to determine if the
-    certificate as been revoked
+    certificate has been revoked
 
     :param cert_path:
     :param issuer_chain_path:
@@ -69,6 +69,9 @@ def crl_verify(cert_path):
 
             if response.status_code != 200:
                 raise Exception("Unable to retrieve CRL: {0}".format(point))
+        except InvalidSchema:
+            # Unhandled URI scheme (like ldap://); skip this distribution point.
+            continue
         except ConnectionError:
             raise Exception("Unable to retrieve CRL: {0}".format(point))
 
@@ -76,6 +79,15 @@ def crl_verify(cert_path):
 
         for r in crl:
             if cert.serial == r.serial_number:
+                try:
+                    reason = r.extensions.get_extension_for_class(x509.CRLReason).value
+                    # Handle "removeFromCRL" revoke reason as unrevoked; continue with the next distribution point.
+                    # Per RFC 5280 section 6.3.3 (k): https://tools.ietf.org/html/rfc5280#section-6.3.3
+                    if reason == x509.ReasonFlags.remove_from_crl:
+                        break
+                except x509.ExtensionNotFound:
+                    pass
+
                 return
 
     return True

lemur/certificates/views.py

@@ -18,8 +18,15 @@ from lemur.auth.service import AuthenticatedResource
 from lemur.auth.permissions import AuthorityPermission, CertificatePermission
 
 from lemur.certificates import service
-from lemur.certificates.schemas import certificate_input_schema, certificate_output_schema, \
+from lemur.plugins.base import plugins
-    certificate_upload_input_schema, certificates_output_schema, certificate_export_input_schema, certificate_edit_input_schema
+from lemur.certificates.schemas import (
+    certificate_input_schema,
+    certificate_output_schema,
+    certificate_upload_input_schema,
+    certificates_output_schema,
+    certificate_export_input_schema,
+    certificate_edit_input_schema
+)
 
 from lemur.roles import service as role_service
 from lemur.logs import service as log_service
@@ -84,7 +91,7 @@ class CertificatesList(AuthenticatedResource):
                     "deleted": null,
                     "notifications": [{
                         "id": 1
-                    }]
+                    }],
                     "signingAlgorithm": "sha256",
                     "user": {
                         "username": "jane",
@@ -169,7 +176,7 @@ class CertificatesList(AuthenticatedResource):
                   },
                   "replacements": [{
                     "id": 1
-                  },
+                  }],
                   "notify": true,
                   "validityEnd": "2026-01-01T08:00:00.000Z",
                   "authority": {
@@ -215,7 +222,7 @@ class CertificatesList(AuthenticatedResource):
                 "deleted": null,
                 "notifications": [{
                     "id": 1
-                }]
+                }],
                 "signingAlgorithm": "sha256",
                 "user": {
                     "username": "jane",
@@ -232,6 +239,8 @@ class CertificatesList(AuthenticatedResource):
                 "replaces": [{
                     "id": 1
                 }],
+                "rotation": true,
+                "rotationPolicy": {"name": "default"},
                 "name": "WILDCARD.test.example.net-SymantecCorporation-20160603-20180112",
                 "roles": [{
                     "id": 464,
@@ -241,18 +250,6 @@ class CertificatesList(AuthenticatedResource):
                 "san": null
               }
 
-
-           :arg extensions: extensions to be used in the certificate
-           :arg description: description for new certificate
-           :arg owner: owner email
-           :arg validityStart: when the certificate should start being valid
-           :arg validityEnd: when the certificate should expire
-           :arg authority: authority that should issue the certificate
-           :arg country: country for the CSR
-           :arg state: state for the CSR
-           :arg location: location for the CSR
-           :arg organization: organization for CSR
-           :arg commonName: certificate common name
            :reqheader Authorization: OAuth token to authenticate
            :statuscode 200: no error
            :statuscode 403: unauthenticated
@@ -269,9 +266,11 @@ class CertificatesList(AuthenticatedResource):
 
         if authority_permission.can():
             data['creator'] = g.user
-            return service.create(**data)
+            cert = service.create(**data)
+            log_service.create(g.user, 'create_cert', certificate=cert)
+            return cert
 
-        return dict(message="You are not authorized to use {0}".format(data['authority'].name)), 403
+        return dict(message="You are not authorized to use the authority: {0}".format(data['authority'].name)), 403
 
 
 class CertificatesUpload(AuthenticatedResource):
@@ -339,7 +338,7 @@ class CertificatesUpload(AuthenticatedResource):
                 "deleted": null,
                 "notifications": [{
                     "id": 1
-                }]
+                }],
                 "signingAlgorithm": "sha256",
                 "user": {
                     "username": "jane",
@@ -354,6 +353,8 @@ class CertificatesUpload(AuthenticatedResource):
                     "name": "*.test.example.net"
                 }],
                 "replaces": [],
+                "rotation": true,
+                "rotationPolicy": {"name": "default"},
                 "name": "WILDCARD.test.example.net-SymantecCorporation-20160603-20180112",
                 "roles": [{
                     "id": 464,
@@ -363,11 +364,6 @@ class CertificatesUpload(AuthenticatedResource):
                 "san": null
               }
 
-           :arg owner: owner email for certificate
-           :arg publicCert: valid PEM public key for certificate
-           :arg intermediateCert valid PEM intermediate key for certificate
-           :arg privateKey: valid PEM private key for certificate
-           :arg destinations: list of aws destinations to upload the certificate to
            :reqheader Authorization: OAuth token to authenticate
            :statuscode 403: unauthenticated
            :statuscode 200: no error
@@ -428,7 +424,7 @@ class CertificatePrivateKey(AuthenticatedResource):
               Content-Type: text/javascript
 
               {
-                 "key": "-----BEGIN ...",
+                 "key": "-----BEGIN ..."
               }
 
            :reqheader Authorization: OAuth token to authenticate
@@ -506,7 +502,7 @@ class Certificates(AuthenticatedResource):
                 "deleted": null,
                 "notifications": [{
                     "id": 1
-                }]
+                }],
                 "signingAlgorithm": "sha256",
                 "user": {
                     "username": "jane",
@@ -520,6 +516,8 @@ class Certificates(AuthenticatedResource):
                     "id": 1090,
                     "name": "*.test.example.net"
                 }],
+                "rotation": true,
+                "rotationPolicy": {"name": "default"},
                 "replaces": [],
                 "replaced": [],
                 "name": "WILDCARD.test.example.net-SymantecCorporation-20160603-20180112",
@@ -614,6 +612,8 @@ class Certificates(AuthenticatedResource):
                     "description": "This is a google group based role created by Lemur",
                     "name": "joe@example.com"
                 }],
+                "rotation": true,
+                "rotationPolicy": {"name": "default"},
                 "san": null
               }
 
@@ -644,7 +644,9 @@ class Certificates(AuthenticatedResource):
                         )
                     ), 400
 
-        return service.update(certificate_id, **data)
+        cert = service.update(certificate_id, **data)
+        log_service.create(g.current_user, 'update_cert', certificate=cert)
+        return cert
 
 
 class NotificationCertificatesList(AuthenticatedResource):
@@ -702,7 +704,7 @@ class NotificationCertificatesList(AuthenticatedResource):
                     "deleted": null,
                     "notifications": [{
                         "id": 1
-                    }]
+                    }],
                     "signingAlgorithm": "sha256",
                     "user": {
                         "username": "jane",
@@ -718,6 +720,8 @@ class NotificationCertificatesList(AuthenticatedResource):
                     }],
                     "replaces": [],
                     "replaced": [],
+                    "rotation": true,
+                    "rotationPolicy": {"name": "default"},
                     "name": "WILDCARD.test.example.net-SymantecCorporation-20160603-20180112",
                     "roles": [{
                         "id": 464,
@@ -823,6 +827,8 @@ class CertificatesReplacementsList(AuthenticatedResource):
                     }],
                     "replaces": [],
                     "replaced": [],
+                    "rotation": true,
+                    "rotationPolicy": {"name": "default"},
                     "name": "WILDCARD.test.example.net-SymantecCorporation-20160603-20180112",
                     "roles": [{
                         "id": 464,
@@ -945,6 +951,69 @@ class CertificateExport(AuthenticatedResource):
         return dict(extension=extension, passphrase=passphrase, data=base64.b64encode(data).decode('utf-8'))
 
 
+class CertificateRevoke(AuthenticatedResource):
+    def __init__(self):
+        self.reqparse = reqparse.RequestParser()
+        super(CertificateRevoke, self).__init__()
+
+    @validate_schema(None, None)
+    def put(self, certificate_id, data=None):
+        """
+        .. http:put:: /certificates/1/revoke
+
+           Revoke a certificate
+
+           **Example request**:
+
+           .. sourcecode:: http
+
+              POST /certificates/1/revoke HTTP/1.1
+              Host: example.com
+              Accept: application/json, text/javascript
+
+           **Example response**:
+
+           .. sourcecode:: http
+
+              HTTP/1.1 200 OK
+              Vary: Accept
+              Content-Type: text/javascript
+
+              {
+                'id': 1
+              }
+
+           :reqheader Authorization: OAuth token to authenticate
+           :statuscode 200: no error
+           :statuscode 403: unauthenticated
+
+        """
+        cert = service.get(certificate_id)
+
+        if not cert:
+            return dict(message="Cannot find specified certificate"), 404
+
+        # allow creators
+        if g.current_user != cert.user:
+            owner_role = role_service.get_by_name(cert.owner)
+            permission = CertificatePermission(owner_role, [x.name for x in cert.roles])
+
+            if not permission.can():
+                return dict(message='You are not authorized to revoke this certificate.'), 403
+
+        if not cert.external_id:
+            return dict(message='Cannot revoke certificate. No external id found.'), 400
+
+        if cert.endpoints:
+            return dict(message='Cannot revoke certificate. Endpoints are deployed with the given certificate.'), 403
+
+        plugin = plugins.get(cert.authority.plugin_name)
+        plugin.revoke_certificate(cert, data)
+        log_service.create(g.current_user, 'revoke_cert', certificate=cert)
+        return dict(id=cert.id)
+
+
+api.add_resource(CertificateRevoke, '/certificates/<int:certificate_id>/revoke', endpoint='revokeCertificate')
 api.add_resource(CertificatesList, '/certificates', endpoint='certificates')
 api.add_resource(Certificates, '/certificates/<int:certificate_id>', endpoint='certificate')
 api.add_resource(CertificatesStats, '/certificates/stats', endpoint='certificateStats')

lemur/common/defaults.py

@@ -1,8 +1,26 @@
+import re
+import unicodedata
+
 from cryptography import x509
 from flask import current_app
+from lemur.extensions import sentry
 from lemur.constants import SAN_NAMING_TEMPLATE, DEFAULT_NAMING_TEMPLATE
 
 
+def text_to_slug(value):
+    """Normalize a string to a "slug" value, stripping character accents and removing non-alphanum characters."""
+
+    # Strip all character accents (ä => a): decompose Unicode characters and then drop combining chars.
+    value = ''.join(c for c in unicodedata.normalize('NFKD', value) if not unicodedata.combining(c))
+
+    # Replace all remaining non-alphanumeric characters with '-'. Multiple characters get collapsed into a single dash.
+    # Except, keep 'xn--' used in IDNA domain names as is.
+    value = re.sub(r'[^A-Za-z0-9.]+(?<!xn--)', '-', value)
+
+    # '-' in the beginning or end of string looks ugly.
+    return value.strip('-')
+
+
 def certificate_name(common_name, issuer, not_before, not_after, san):
     """
     Create a name for our certificate. A naming standard
@@ -25,21 +43,13 @@ def certificate_name(common_name, issuer, not_before, not_after, san):
 
     temp = t.format(
         subject=common_name,
-        issuer=issuer,
+        issuer=issuer.replace(' ', ''),
         not_before=not_before.strftime('%Y%m%d'),
         not_after=not_after.strftime('%Y%m%d')
     )
 
-    disallowed_chars = ''.join(c for c in map(chr, range(256)) if not c.isalnum())
-    disallowed_chars = disallowed_chars.replace("-", "")
-    disallowed_chars = disallowed_chars.replace(".", "")
     temp = temp.replace('*', "WILDCARD")
-
+    return text_to_slug(temp)
-    for c in disallowed_chars:
-        temp = temp.replace(c, "")
-
-    # white space is silly too
-    return temp.replace(" ", "-")
 
 
 def signing_algorithm(cert):
@@ -58,6 +68,7 @@ def common_name(cert):
             x509.OID_COMMON_NAME
         )[0].value.strip()
     except Exception as e:
+        sentry.captureException()
         current_app.logger.error("Unable to get common name! {0}".format(e))
 
 
@@ -72,6 +83,7 @@ def organization(cert):
             x509.OID_ORGANIZATION_NAME
         )[0].value.strip()
     except Exception as e:
+        sentry.captureException()
         current_app.logger.error("Unable to get organization! {0}".format(e))
 
 
@@ -86,6 +98,7 @@ def organizational_unit(cert):
             x509.OID_ORGANIZATIONAL_UNIT_NAME
         )[0].value.strip()
     except Exception as e:
+        sentry.captureException()
         current_app.logger.error("Unable to get organizational unit! {0}".format(e))
 
 
@@ -100,6 +113,7 @@ def country(cert):
             x509.OID_COUNTRY_NAME
         )[0].value.strip()
     except Exception as e:
+        sentry.captureException()
         current_app.logger.error("Unable to get country! {0}".format(e))
 
 
@@ -114,6 +128,7 @@ def state(cert):
             x509.OID_STATE_OR_PROVINCE_NAME
         )[0].value.strip()
     except Exception as e:
+        sentry.captureException()
         current_app.logger.error("Unable to get state! {0}".format(e))
 
 
@@ -128,6 +143,7 @@ def location(cert):
             x509.OID_LOCALITY_NAME
         )[0].value.strip()
     except Exception as e:
+        sentry.captureException()
         current_app.logger.error("Unable to get location! {0}".format(e))
 
 
@@ -147,7 +163,7 @@ def domains(cert):
         for entry in entries:
             domains.append(entry)
     except Exception as e:
-        pass
+        sentry.captureException()
 
     return domains
 
@@ -199,6 +215,7 @@ def bitstrength(cert):
     try:
         return cert.public_key().key_size
     except AttributeError:
+        sentry.captureException()
         current_app.logger.debug('Unable to get bitstrength.')
 
 
@@ -219,6 +236,7 @@ def issuer(cert):
             issuer = issuer.replace(c, "")
         return issuer
     except Exception as e:
+        sentry.captureException()
         current_app.logger.error("Unable to get issuer! {0}".format(e))
         return "Unknown"
 

lemur/common/fields.py

@@ -18,6 +18,18 @@ from marshmallow import utils
 from marshmallow.fields import Field
 from marshmallow.exceptions import ValidationError
 
+from lemur.common import validators
+
+
+class Hex(Field):
+    """
+    A hex formatted string.
+    """
+    def _serialize(self, value, attr, obj):
+        if value:
+            value = hex(int(value))[2:].upper()
+        return value
+
 
 class ArrowDateTime(Field):
     """A formatted datetime string in UTC.
@@ -317,7 +329,12 @@ class SubjectAlternativeNameExtension(Field):
                     name_type = 'DNSName'
 
                 elif isinstance(name, x509.IPAddress):
-                    name_type = 'IPAddress'
+                    if isinstance(value, ipaddress.IPv4Network):
+                        name_type = 'IPNetwork'
+                    else:
+                        name_type = 'IPAddress'
+
+                    value = str(value)
 
                 elif isinstance(name, x509.UniformResourceIdentifier):
                     name_type = 'uniformResourceIdentifier'
@@ -342,6 +359,7 @@ class SubjectAlternativeNameExtension(Field):
         general_names = []
         for name in value:
             if name['nameType'] == 'DNSName':
+                validators.sensitive_domain(name['value'])
                 general_names.append(x509.DNSName(name['value']))
 
             elif name['nameType'] == 'IPAddress':

lemur/common/health.py

@@ -7,10 +7,23 @@
 .. moduleauthor:: Kevin Glisson <kglisson@netflix.com>
 """
 from flask import Blueprint
+from lemur.database import db
+from lemur.extensions import sentry
 
 mod = Blueprint('healthCheck', __name__)
 
 
 @mod.route('/healthcheck')
 def health():
-    return 'ok'
+    try:
+        if healthcheck(db):
+            return 'ok'
+    except Exception:
+        sentry.captureException()
+        return 'db check failed'
+
+
+def healthcheck(db):
+    with db.engine.connect() as connection:
+        connection.execute('SELECT 1;')
+    return True

lemur/common/schema.py

@@ -15,6 +15,8 @@ from sqlalchemy.orm.collections import InstrumentedList
 from inflection import camelize, underscore
 from marshmallow import Schema, post_dump, pre_load
 
+from lemur.extensions import sentry
+
 
 class LemurSchema(Schema):
     """
@@ -157,6 +159,7 @@ def validate_schema(input_schema, output_schema):
             try:
                 resp = f(*args, **kwargs)
             except Exception as e:
+                sentry.captureException()
                 current_app.logger.exception(e)
                 return dict(message=str(e)), 500
 

lemur/common/utils.py

@@ -53,6 +53,14 @@ def parse_certificate(body):
     return x509.load_pem_x509_certificate(body, default_backend())
 
 
+def get_authority_key(body):
+    """Returns the authority key for a given certificate in hex format"""
+    parsed_cert = parse_certificate(body)
+    authority_key = parsed_cert.extensions.get_extension_for_class(
+        x509.AuthorityKeyIdentifier).value.key_identifier
+    return authority_key.hex()
+
+
 def generate_private_key(key_type):
     """
     Generates a new private key based on key_type.

lemur/common/validators.py

@@ -1,9 +1,10 @@
 import re
 
-from flask import current_app
 from cryptography import x509
 from cryptography.hazmat.backends import default_backend
 from cryptography.hazmat.primitives import serialization
+from cryptography.x509 import NameOID
+from flask import current_app
 from marshmallow.exceptions import ValidationError
 
 from lemur.auth.permissions import SensitiveDomainPermission
@@ -41,22 +42,33 @@ def private_key(key):
         raise ValidationError('Private key presented is not valid.')
 
 
+def common_name(value):
+    """If the common name could be a domain name, apply domain validation rules."""
+    # Common name could be a domain name, or a human-readable name of the subject (often used in CA names or client
+    # certificates). As a simple heuristic, we assume that human-readable names always include a space.
+    # However, to avoid confusion for humans, we also don't count spaces at the beginning or end of the string.
+    if ' ' not in value.strip():
+        return sensitive_domain(value)
+
+
 def sensitive_domain(domain):
     """
-    Determines if domain has been marked as sensitive.
+    Checks if user has the admin role, the domain does not match sensitive domains and whitelisted domain patterns.
-    :param domain:
+    :param domain: domain name (str)
     :return:
     """
-    restricted_domains = current_app.config.get('LEMUR_RESTRICTED_DOMAINS', [])
+    if SensitiveDomainPermission().can():
-    if restricted_domains:
+        # User has permission, no need to check anything
-        domains = domain_service.get_by_name(domain)
+        return
-        for domain in domains:
+
-            # we only care about non-admins
+    whitelist = current_app.config.get('LEMUR_WHITELISTED_DOMAINS', [])
-            if not SensitiveDomainPermission().can():
+    if whitelist and not any(re.match(pattern, domain) for pattern in whitelist):
-                if domain.sensitive or any([re.match(pattern, domain.name) for pattern in restricted_domains]):
+        raise ValidationError('Domain {0} does not match whitelisted domain patterns. '
-                    raise ValidationError(
+                              'Contact an administrator to issue the certificate.'.format(domain))
-                        'Domain {0} has been marked as sensitive, contact and administrator \
+
-                        to issue the certificate.'.format(domain))
+    if any(d.sensitive for d in domain_service.get_by_name(domain)):
+        raise ValidationError('Domain {0} has been marked as sensitive. '
+                              'Contact an administrator to issue the certificate.'.format(domain))
 
 
 def encoding(oid_encoding):
@@ -84,15 +96,27 @@ def sub_alt_type(alt_type):
 
 def csr(data):
     """
-    Determines if the CSR is valid.
+    Determines if the CSR is valid and allowed.
     :param data:
     :return:
     """
     try:
-        x509.load_pem_x509_csr(data.encode('utf-8'), default_backend())
+        request = x509.load_pem_x509_csr(data.encode('utf-8'), default_backend())
     except Exception:
         raise ValidationError('CSR presented is not valid.')
 
+    # Validate common name and SubjectAltNames
+    for name in request.subject.get_attributes_for_oid(NameOID.COMMON_NAME):
+        common_name(name.value)
+
+    try:
+        alt_names = request.extensions.get_extension_for_class(x509.SubjectAlternativeName)
+
+        for name in alt_names.value.get_values_for_type(x509.DNSName):
+            sensitive_domain(name)
+    except x509.ExtensionNotFound:
+        pass
+
 
 def dates(data):
     if not data.get('validity_start') and data.get('validity_end'):

lemur/database.py

@@ -9,6 +9,7 @@
 
 .. moduleauthor:: Kevin Glisson <kglisson@netflix.com>
 """
+from inflection import underscore
 from sqlalchemy import exc
 from sqlalchemy.sql import and_, or_
 from sqlalchemy.orm import make_transient
@@ -75,6 +76,16 @@ def add(model):
     db.session.add(model)
 
 
+def get_model_column(model, field):
+    if field in getattr(model, 'sensitive_fields', ()):
+        raise AttrNotFound(field)
+    column = model.__table__.columns._data.get(field, None)
+    if column is None:
+        raise AttrNotFound(field)
+
+    return column
+
+
 def find_all(query, model, kwargs):
     """
     Returns a query object that ensures that all kwargs
@@ -91,7 +102,7 @@ def find_all(query, model, kwargs):
         if not isinstance(value, list):
             value = value.split(',')
 
-        conditions.append(getattr(model, attr).in_(value))
+        conditions.append(get_model_column(model, attr).in_(value))
 
     return query.filter(and_(*conditions))
 
@@ -108,7 +119,7 @@ def find_any(query, model, kwargs):
     """
     or_args = []
     for attr, value in kwargs.items():
-        or_args.append(or_(getattr(model, attr) == value))
+        or_args.append(or_(get_model_column(model, attr) == value))
     exprs = or_(*or_args)
     return query.filter(exprs)
 
@@ -123,7 +134,7 @@ def get(model, value, field="id"):
     :return:
     """
     query = session_query(model)
-    return query.filter(getattr(model, field) == value).scalar()
+    return query.filter(get_model_column(model, field) == value).scalar()
 
 
 def get_all(model, value, field="id"):
@@ -136,7 +147,7 @@ def get_all(model, value, field="id"):
     :return:
     """
     query = session_query(model)
-    return query.filter(getattr(model, field) == value)
+    return query.filter(get_model_column(model, field) == value)
 
 
 def create(model):
@@ -188,7 +199,8 @@ def filter(query, model, terms):
     :param terms:
     :return:
     """
-    return query.filter(getattr(model, terms[0]).ilike('%{}%'.format(terms[1])))
+    column = get_model_column(model, underscore(terms[0]))
+    return query.filter(column.ilike('%{}%'.format(terms[1])))
 
 
 def sort(query, model, field, direction):
@@ -201,13 +213,8 @@ def sort(query, model, field, direction):
     :param field:
     :param direction:
     """
-    try:
+    column = get_model_column(model, underscore(field))
-        field = getattr(model, field)
+    return query.order_by(column.desc() if direction == 'desc' else column.asc())
-        direction = getattr(field, direction)
-        query = query.order_by(direction())
-        return query
-    except AttributeError:
-        raise AttrNotFound(field)
 
 
 def paginate(query, page, count):

lemur/default.conf.py

@@ -9,7 +9,7 @@ THREADS_PER_PAGE = 8
 
 # These will need to be set to `True` if you are developing locally
 CORS = False
-debug = False
+DEBUG = False
 
 # Logging
 

lemur/deployment/service.py

@@ -10,7 +10,6 @@ def rotate_certificate(endpoint, new_cert):
     :return:
     """
     # ensure that certificate is available for rotation
-
     endpoint.source.plugin.update_endpoint(endpoint, new_cert)
     endpoint.certificate = new_cert
     database.update(endpoint)

lemur/destinations/service.py

@@ -22,6 +22,11 @@ def create(label, plugin_name, options, description=None):
     :rtype : Destination
     :return: New destination
     """
+    # remove any sub-plugin objects before try to save the json options
+    for option in options:
+        if 'plugin' in option['type']:
+            del option['value']['plugin_object']
+
     destination = Destination(label=label, options=options, plugin_name=plugin_name, description=description)
     return database.create(destination)
 

lemur/domains/schemas.py

@@ -15,7 +15,7 @@ from lemur.schemas import AssociatedCertificateSchema
 class DomainInputSchema(LemurInputSchema):
     id = fields.Integer()
     name = fields.String(required=True)
-    sensitive = fields.Boolean()
+    sensitive = fields.Boolean(missing=False)
     certificates = fields.Nested(AssociatedCertificateSchema, many=True, missing=[])
 
 

lemur/domains/service.py

@@ -76,7 +76,7 @@ def render(args):
     :param args:
     :return:
     """
-    query = database.session_query(Domain).join(Certificate, Domain.certificate)
+    query = database.session_query(Domain)
     filt = args.pop('filter')
     certificate_id = args.pop('certificate_id', None)
 
@@ -85,6 +85,7 @@ def render(args):
         query = database.filter(query, Domain, terms)
 
     if certificate_id:
+        query = query.join(Certificate, Domain.certificates)
         query = query.filter(Certificate.id == certificate_id)
 
     return database.sort_and_page(query, Domain, args)

lemur/endpoints/cli.py

@@ -14,7 +14,7 @@ from sqlalchemy import cast
 from sqlalchemy_utils import ArrowType
 
 from lemur import database
-from lemur.extensions import metrics
+from lemur.extensions import metrics, sentry
 from lemur.endpoints.models import Endpoint
 
 
@@ -27,13 +27,17 @@ def expire(ttl):
     Removed all endpoints that have not been recently updated.
     """
     print("[+] Staring expiration of old endpoints.")
-    now = arrow.utcnow()
-    expiration = now - timedelta(hours=ttl)
-    endpoints = database.session_query(Endpoint).filter(cast(Endpoint.last_updated, ArrowType) <= expiration)
 
-    for endpoint in endpoints:
+    try:
-        print("[!] Expiring endpoint: {name} Last Updated: {last_updated}".format(name=endpoint.name, last_updated=endpoint.last_updated))
+        now = arrow.utcnow()
-        database.delete(endpoint)
+        expiration = now - timedelta(hours=ttl)
-        metrics.send('endpoint_expired', 'counter', 1)
+        endpoints = database.session_query(Endpoint).filter(cast(Endpoint.last_updated, ArrowType) <= expiration)
 
-    print("[+] Finished expiration.")
+        for endpoint in endpoints:
+            print("[!] Expiring endpoint: {name} Last Updated: {last_updated}".format(name=endpoint.name, last_updated=endpoint.last_updated))
+            database.delete(endpoint)
+            metrics.send('endpoint_expired', 'counter', 1)
+
+        print("[+] Finished expiration.")
+    except Exception as e:
+        sentry.captureException()

lemur/endpoints/service.py

@@ -60,6 +60,16 @@ def get_by_dnsname(dnsname):
     return database.get(Endpoint, dnsname, field='dnsname')
 
 
+def get_by_dnsname_and_port(dnsname, port):
+    """
+    Retrieves and endpoint by it's dnsname and port.
+    :param dnsname:
+    :param port:
+    :return:
+    """
+    return Endpoint.query.filter(Endpoint.dnsname == dnsname).filter(Endpoint.port == port).scalar()
+
+
 def get_by_source(source_label):
     """
     Retrieves all endpoints for a given source.

lemur/exceptions.py

@@ -29,7 +29,7 @@ class AttrNotFound(LemurException):
         self.field = field
 
     def __str__(self):
-        return repr("The field '{0}' is not sortable".format(self.field))
+        return repr("The field '{0}' is not sortable or filterable".format(self.field))
 
 
 class InvalidConfiguration(Exception):

lemur/extensions.py

@@ -13,10 +13,16 @@ from flask_bcrypt import Bcrypt
 bcrypt = Bcrypt()
 
 from flask_principal import Principal
-principal = Principal()
+principal = Principal(use_sessions=False)
 
 from flask_mail import Mail
 smtp_mail = Mail()
 
 from lemur.metrics import Metrics
 metrics = Metrics()
+
+from raven.contrib.flask import Sentry
+sentry = Sentry()
+
+from blinker import Namespace
+signals = Namespace()

lemur/factory.py

@@ -18,8 +18,10 @@ from logging import Formatter, StreamHandler
 from logging.handlers import RotatingFileHandler
 
 from flask import Flask
+
+from lemur.certificates.hooks import activate_debug_dump
 from lemur.common.health import mod as health
-from lemur.extensions import db, migrate, principal, smtp_mail, metrics
+from lemur.extensions import db, migrate, principal, smtp_mail, metrics, sentry
 
 
 DEFAULT_BLUEPRINTS = (
@@ -73,7 +75,8 @@ def from_file(file_path, silent=False):
     d.__file__ = file_path
     try:
         with open(file_path) as config_file:
-            exec(compile(config_file.read(), file_path, 'exec'), d.__dict__)
+            exec(compile(config_file.read(),  # nosec: config file safe
+                 file_path, 'exec'), d.__dict__)
     except IOError as e:
         if silent and e.errno in (errno.ENOENT, errno.EISDIR):
             return False
@@ -120,6 +123,7 @@ def configure_extensions(app):
     principal.init_app(app)
     smtp_mail.init_app(app)
     metrics.init_app(app)
+    sentry.init_app(app)
 
 
 def configure_blueprints(app, blueprints):
@@ -152,9 +156,12 @@ def configure_logging(app):
     app.logger.addHandler(handler)
 
     stream_handler = StreamHandler()
-    stream_handler.setLevel(app.config.get('LOG_LEVEL'))
+    stream_handler.setLevel(app.config.get('LOG_LEVEL', 'DEBUG'))
     app.logger.addHandler(stream_handler)
 
+    if app.config.get('DEBUG_DUMP', False):
+        activate_debug_dump()
+
 
 def install_plugins(app):
     """
@@ -181,8 +188,10 @@ def install_plugins(app):
 
     # ensure that we have some way to notify
     with app.app_context():
+        slug = app.config.get("LEMUR_DEFAULT_NOTIFICATION_PLUGIN", "email-notification")
         try:
-            slug = app.config.get("LEMUR_DEFAULT_NOTIFICATION_PLUGIN", "email-notification")
             plugins.get(slug)
         except KeyError:
-            raise Exception("Unable to location notification plugin: {slug}. Ensure that LEMUR_DEFAULT_NOTIFICATION_PLUGIN is set to a valid and installed notification plugin.".format(slug=slug))
+            raise Exception("Unable to location notification plugin: {slug}. Ensure that "
+                            "LEMUR_DEFAULT_NOTIFICATION_PLUGIN is set to a valid and installed notification plugin."
+                            .format(slug=slug))

lemur/logs/models.py

@@ -18,6 +18,6 @@ class Log(db.Model):
     __tablename__ = 'logs'
     id = Column(Integer, primary_key=True)
     certificate_id = Column(Integer, ForeignKey('certificates.id'))
-    log_type = Column(Enum('key_view', name='log_type'), nullable=False)
+    log_type = Column(Enum('key_view', 'create_cert', 'update_cert', 'revoke_cert', name='log_type'), nullable=False)
     logged_at = Column(ArrowType(), PassiveDefault(func.now()), nullable=False)
     user_id = Column(Integer, ForeignKey('users.id'), nullable=False)

lemur/logs/service.py

@@ -7,8 +7,12 @@
     :license: Apache, see LICENSE for more details.
 .. moduleauthor:: Kevin Glisson <kglisson@netflix.com>
 """
+from flask import current_app
+
 from lemur import database
 from lemur.logs.models import Log
+from lemur.users.models import User
+from lemur.certificates.models import Certificate
 
 
 def create(user, type, certificate=None):
@@ -20,6 +24,7 @@ def create(user, type, certificate=None):
     :param certificate:
     :return:
     """
+    current_app.logger.info("[lemur-audit] action: {0}, user: {1}, certificate: {2}.".format(type, user.email, certificate.name))
     view = Log(user_id=user.id, log_type=type, certificate_id=certificate.id)
     database.add(view)
     database.commit()
@@ -49,6 +54,20 @@ def render(args):
 
     if filt:
         terms = filt.split(';')
-        query = database.filter(query, Log, terms)
+
+        if 'certificate.name' in terms:
+            sub_query = database.session_query(Certificate.id)\
+                .filter(Certificate.name.ilike('%{0}%'.format(terms[1])))
+
+            query = query.filter(Log.certificate_id.in_(sub_query))
+
+        elif 'user.email' in terms:
+            sub_query = database.session_query(User.id)\
+                .filter(User.email.ilike('%{0}%'.format(terms[1])))
+
+            query = query.filter(Log.user_id.in_(sub_query))
+
+        else:
+            query = database.filter(query, Log, terms)
 
     return database.sort_and_page(query, Log, args)

lemur/manage.py

@@ -16,16 +16,18 @@ from flask_migrate import Migrate, MigrateCommand, stamp
 from flask_script.commands import ShowUrls, Clean, Server
 
 from lemur.sources.cli import manager as source_manager
+from lemur.policies.cli import manager as policy_manager
+from lemur.reporting.cli import manager as report_manager
+from lemur.endpoints.cli import manager as endpoint_manager
 from lemur.certificates.cli import manager as certificate_manager
 from lemur.notifications.cli import manager as notification_manager
-from lemur.endpoints.cli import manager as endpoint_manager
+
-from lemur.reporting.cli import manager as report_manager
 from lemur import database
 from lemur.users import service as user_service
 from lemur.roles import service as role_service
+from lemur.policies import service as policy_service
 from lemur.notifications import service as notification_service
 
-
 from lemur.common.utils import validate_conf
 
 from lemur import create_app
@@ -40,6 +42,8 @@ from lemur.domains.models import Domain  # noqa
 from lemur.notifications.models import Notification  # noqa
 from lemur.sources.models import Source  # noqa
 from lemur.logs.models import Log  # noqa
+from lemur.endpoints.models import Endpoint  # noqa
+from lemur.policies.models import RotationPolicy  # noqa
 
 
 manager = Manager(create_app)
@@ -83,8 +87,8 @@ SECRET_KEY = '{flask_secret_key}'
 LEMUR_TOKEN_SECRET = '{secret_token}'
 LEMUR_ENCRYPTION_KEYS = '{encryption_key}'
 
-# this is a list of domains as regexes that only admins can issue
+# List of domain regular expressions that non-admin users can issue
-LEMUR_RESTRICTED_DOMAINS = []
+LEMUR_WHITELISTED_DOMAINS = []
 
 # Mail Server
 
@@ -222,7 +226,7 @@ class InitializeApp(Command):
                     sys.stderr.write("[!] Passwords do not match!\n")
                     sys.exit(1)
 
-            user_service.create("lemur", password, 'lemur@nobody', True, None, [admin_role])
+            user_service.create("lemur", password, 'lemur@nobody.com', True, None, [admin_role])
             sys.stdout.write("[+] Created the user 'lemur' and granted it the 'admin' role!\n")
 
         else:
@@ -242,6 +246,12 @@ class InitializeApp(Command):
         recipients = current_app.config.get('LEMUR_SECURITY_TEAM_EMAIL')
         notification_service.create_default_expiration_notifications("DEFAULT_SECURITY", recipients=recipients)
 
+        days = current_app.config.get("LEMUR_DEFAULT_ROTATION_INTERVAL", 30)
+        sys.stdout.write("[+] Creating default certificate rotation policy of {days} days before issuance.\n".format(
+            days=days
+        ))
+
+        policy_service.create(days=days, name='default')
         sys.stdout.write("[/] Done!\n")
 
 
@@ -365,7 +375,7 @@ class LemurServer(Command):
 
         app = WSGIApplication()
 
-        # run startup tasks on a app like object
+        # run startup tasks on an app like object
         validate_conf(current_app, REQUIRED_VARIABLES)
 
         app.app_uri = 'lemur:create_app(config="{0}")'.format(current_app.config.get('CONFIG_PATH'))
@@ -531,6 +541,7 @@ def main():
     manager.add_command("notify", notification_manager)
     manager.add_command("endpoint", endpoint_manager)
     manager.add_command("report", report_manager)
+    manager.add_command("policy", policy_manager)
     manager.run()
 
 

lemur/migrations/env.py

@@ -3,6 +3,9 @@ from alembic import context
 from sqlalchemy import engine_from_config, pool
 from logging.config import fileConfig
 
+import alembic_autogenerate_enums
+
+
 # this is the Alembic Config object, which provides
 # access to the values within the .ini file in use.
 config = context.config

lemur/migrations/versions/131ec6accff5_.py

@@ -16,7 +16,7 @@ import sqlalchemy as sa
 
 def upgrade():
     # ### commands auto generated by Alembic - please adjust! ###
-    op.add_column('certificates', sa.Column('rotation', sa.Boolean(), nullable=False, server_default=False))
+    op.add_column('certificates', sa.Column('rotation', sa.Boolean(), nullable=False, server_default=sa.false()))
     op.add_column('endpoints', sa.Column('last_updated', sa.DateTime(), server_default=sa.text('now()'), nullable=False))
     # ### end Alembic commands ###
 

lemur/migrations/versions/1ae8e3104db8_.py

@@ -0,0 +1,21 @@
+"""Adds additional ENUM for creating and updating certificates.
+
+Revision ID: 1ae8e3104db8
+Revises: a02a678ddc25
+Create Date: 2017-07-13 12:32:09.162800
+
+"""
+
+# revision identifiers, used by Alembic.
+revision = '1ae8e3104db8'
+down_revision = 'a02a678ddc25'
+
+from alembic import op
+
+
+def upgrade():
+    op.sync_enum_values('public', 'log_type', ['key_view'], ['create_cert', 'key_view', 'update_cert'])
+
+
+def downgrade():
+    op.sync_enum_values('public', 'log_type', ['create_cert', 'key_view', 'update_cert'], ['key_view'])

lemur/migrations/versions/5bc47fa7cac4_.py

@@ -0,0 +1,22 @@
+"""add third party roles to lemur
+
+Revision ID: 5bc47fa7cac4
+Revises: c05a8998b371
+Create Date: 2017-12-08 14:19:11.903864
+
+"""
+
+# revision identifiers, used by Alembic.
+revision = '5bc47fa7cac4'
+down_revision = 'c05a8998b371'
+
+from alembic import op
+import sqlalchemy as sa
+
+
+def upgrade():
+    op.add_column('roles', sa.Column('third_party', sa.Boolean(), nullable=True, default=False))
+
+
+def downgrade():
+    op.drop_column('roles', 'third_party')

lemur/migrations/versions/5e680529b666_.py

@@ -15,16 +15,12 @@ import sqlalchemy as sa
 
 
 def upgrade():
-    ### commands auto generated by Alembic - please adjust! ###
     op.add_column('endpoints', sa.Column('sensitive', sa.Boolean(), nullable=True))
     op.add_column('endpoints', sa.Column('source_id', sa.Integer(), nullable=True))
     op.create_foreign_key(None, 'endpoints', 'sources', ['source_id'], ['id'])
-    ### end Alembic commands ###
 
 
 def downgrade():
-    ### commands auto generated by Alembic - please adjust! ###
     op.drop_constraint(None, 'endpoints', type_='foreignkey')
     op.drop_column('endpoints', 'source_id')
     op.drop_column('endpoints', 'sensitive')
-    ### end Alembic commands ###

lemur/migrations/versions/8ae67285ff14_.py

@@ -0,0 +1,24 @@
+"""Modifies the unique index for the certificate replacements
+
+Revision ID: 8ae67285ff14
+Revises: 5e680529b666
+Create Date: 2017-05-10 11:56:13.999332
+
+"""
+
+# revision identifiers, used by Alembic.
+revision = '8ae67285ff14'
+down_revision = '5e680529b666'
+
+from alembic import op
+import sqlalchemy as sa
+
+
+def upgrade():
+    op.drop_index('certificate_replacement_associations_ix')
+    op.create_index('certificate_replacement_associations_ix', 'certificate_replacement_associations', ['replaced_certificate_id', 'certificate_id'], unique=True)
+
+
+def downgrade():
+    op.drop_index('certificate_replacement_associations_ix')
+    op.create_index('certificate_replacement_associations_ix', 'certificate_replacement_associations', ['certificate_id', 'certificate_id'], unique=True)

lemur/migrations/versions/a02a678ddc25_.py

@@ -0,0 +1,54 @@
+"""Adds support for rotation policies.
+
+Creates a default rotation policy (30 days) with the name
+'default' ensures that all existing certificates use the default
+policy.
+
+Revision ID: a02a678ddc25
+Revises: 8ae67285ff14
+Create Date: 2017-07-12 11:45:49.257927
+"""
+
+# revision identifiers, used by Alembic.
+revision = 'a02a678ddc25'
+down_revision = '8ae67285ff14'
+
+from alembic import op
+import sqlalchemy as sa
+from sqlalchemy.sql import text
+
+
+def upgrade():
+    # ### commands auto generated by Alembic - please adjust! ###
+    op.create_table('rotation_policies',
+        sa.Column('id', sa.Integer(), nullable=False),
+        sa.Column('name', sa.String(), nullable=True),
+        sa.Column('days', sa.Integer(), nullable=True),
+        sa.PrimaryKeyConstraint('id')
+    )
+    op.add_column('certificates', sa.Column('rotation_policy_id', sa.Integer(), nullable=True))
+    op.create_foreign_key(None, 'certificates', 'rotation_policies', ['rotation_policy_id'], ['id'])
+
+    conn = op.get_bind()
+    stmt = text('insert into rotation_policies (days, name) values (:days, :name)')
+    stmt = stmt.bindparams(days=30, name='default')
+    conn.execute(stmt)
+
+    stmt = text('select id from rotation_policies where name=:name')
+    stmt = stmt.bindparams(name='default')
+    rotation_policy_id = conn.execute(stmt).fetchone()[0]
+
+    stmt = text('update certificates set rotation_policy_id=:rotation_policy_id')
+    stmt = stmt.bindparams(rotation_policy_id=rotation_policy_id)
+    conn.execute(stmt)
+    # ### end Alembic commands ###
+
+
+def downgrade():
+    # ### commands auto generated by Alembic - please adjust! ###
+    op.drop_constraint(None, 'certificates', type_='foreignkey')
+    op.drop_column('certificates', 'rotation_policy_id')
+    op.drop_index('certificate_replacement_associations_ix', table_name='certificate_replacement_associations')
+    op.create_index('certificate_replacement_associations_ix', 'certificate_replacement_associations', ['replaced_certificate_id', 'certificate_id'], unique=True)
+    op.drop_table('rotation_policies')
+    # ### end Alembic commands ###

lemur/migrations/versions/ac483cfeb230_.py

@@ -0,0 +1,27 @@
+"""empty message
+
+Revision ID: ac483cfeb230
+Revises: b29e2c4bf8c9
+Create Date: 2017-10-11 10:16:39.682591
+
+"""
+
+# revision identifiers, used by Alembic.
+revision = 'ac483cfeb230'
+down_revision = 'b29e2c4bf8c9'
+
+from alembic import op
+import sqlalchemy as sa
+from sqlalchemy.dialects import postgresql
+
+
+def upgrade():
+    op.alter_column('certificates', 'name',
+                    existing_type=sa.VARCHAR(length=128),
+                    type_=sa.String(length=256))
+
+
+def downgrade():
+    op.alter_column('certificates', 'name',
+                    existing_type=sa.VARCHAR(length=256),
+                    type_=sa.String(length=128))

lemur/migrations/versions/b29e2c4bf8c9_.py

@@ -0,0 +1,28 @@
+"""Adds external ID checking and modifying enum
+
+Revision ID: b29e2c4bf8c9
+Revises: 1ae8e3104db8
+Create Date: 2017-09-26 10:50:35.740367
+
+"""
+
+# revision identifiers, used by Alembic.
+revision = 'b29e2c4bf8c9'
+down_revision = '1ae8e3104db8'
+
+from alembic import op
+import sqlalchemy as sa
+
+
+def upgrade():
+    # ### commands auto generated by Alembic - please adjust! ###
+    op.add_column('certificates', sa.Column('external_id', sa.String(128), nullable=True))
+    op.sync_enum_values('public', 'log_type', ['create_cert', 'key_view', 'update_cert'], ['create_cert', 'key_view', 'revoke_cert', 'update_cert'])
+    # ### end Alembic commands ###
+
+
+def downgrade():
+    # ### commands auto generated by Alembic - please adjust! ###
+    op.sync_enum_values('public', 'log_type', ['create_cert', 'key_view', 'revoke_cert', 'update_cert'], ['create_cert', 'key_view', 'update_cert'])
+    op.drop_column('certificates', 'external_id')
+    # ### end Alembic commands ###

lemur/migrations/versions/c05a8998b371_.py

@@ -0,0 +1,31 @@
+"""Adds JWT Tokens to Users
+
+Revision ID: c05a8998b371
+Revises: ac483cfeb230
+Create Date: 2017-11-10 14:51:28.975927
+
+"""
+
+# revision identifiers, used by Alembic.
+revision = 'c05a8998b371'
+down_revision = 'ac483cfeb230'
+
+from alembic import op
+import sqlalchemy as sa
+import sqlalchemy_utils
+
+def upgrade():
+    op.create_table('api_keys',
+    sa.Column('id', sa.Integer(), nullable=False),
+    sa.Column('name', sa.String(length=128), nullable=True),
+    sa.Column('user_id', sa.Integer(), nullable=False),
+    sa.Column('ttl', sa.BigInteger(), nullable=False),
+    sa.Column('issued_at', sa.BigInteger(), nullable=False),
+    sa.Column('revoked', sa.Boolean(), nullable=False),
+    sa.ForeignKeyConstraint(['user_id'], ['users.id'], ),
+    sa.PrimaryKeyConstraint('id')
+    )
+
+
+def downgrade():
+    op.drop_table('api_keys')

lemur/models.py

@@ -53,7 +53,7 @@ certificate_replacement_associations = db.Table('certificate_replacement_associa
                                                        ForeignKey('certificates.id', ondelete='cascade'))
                                                 )
 
-Index('certificate_replacement_associations_ix', certificate_replacement_associations.c.certificate_id, certificate_replacement_associations.c.certificate_id)
+Index('certificate_replacement_associations_ix', certificate_replacement_associations.c.replaced_certificate_id, certificate_replacement_associations.c.certificate_id)
 
 roles_authorities = db.Table('roles_authorities',
                              Column('authority_id', Integer, ForeignKey('authorities.id')),

lemur/notifications/cli.py

@@ -12,7 +12,7 @@ from lemur.notifications.messaging import send_expiration_notifications
 manager = Manager(usage="Handles notification related tasks.")
 
 
-@manager.option('-e', '--exclude', dest='exclude', nargs="*", help='Common name matching of certificates that should be excluded from notification')
+@manager.option('-e', '--exclude', dest='exclude', action='append', default=[], help='Common name matching of certificates that should be excluded from notification')
 def expirations(exclude):
     """
     Runs Lemur's notification engine, that looks for expired certificates and sends

lemur/notifications/messaging.py

@@ -18,6 +18,7 @@ from flask import current_app
 from sqlalchemy import and_
 
 from lemur import database, metrics
+from lemur.extensions import sentry
 from lemur.common.utils import windowed_query
 
 from lemur.certificates.schemas import certificate_notification_output_schema
@@ -98,6 +99,7 @@ def send_notification(event_type, data, targets, notification):
         metrics.send('{0}_notification_sent'.format(event_type), 'counter', 1)
         return True
     except Exception as e:
+        sentry.captureException()
         metrics.send('{0}_notification_failure'.format(event_type), 'counter', 1)
         current_app.logger.exception(e)
 
@@ -131,17 +133,17 @@ def send_expiration_notifications(exclude):
             else:
                 failure += 1
 
-    if send_notification('expiration', security_data, security_email, notification):
+            if send_notification('expiration', security_data, security_email, notification):
-        success += 1
+                success += 1
-    else:
+            else:
-        failure += 1
+                failure += 1
 
     return success, failure
 
 
 def send_rotation_notification(certificate, notification_plugin=None):
     """
-    Sends a report to certificate owners when their certificate as been
+    Sends a report to certificate owners when their certificate has been
     rotated.
 
     :param certificate:
@@ -157,6 +159,7 @@ def send_rotation_notification(certificate, notification_plugin=None):
         metrics.send('rotation_notification_sent', 'counter', 1)
         return True
     except Exception as e:
+        sentry.captureException()
         metrics.send('rotation_notification_failure', 'counter', 1)
         current_app.logger.exception(e)
 
@@ -173,7 +176,7 @@ def needs_notification(certificate):
     days = (certificate.not_after - now).days
 
     for notification in certificate.notifications:
-        if not notification.options:
+        if not notification.active or not notification.options:
             return
 
         interval = get_plugin_option('interval', notification.options)

lemur/notifications/schemas.py

@@ -26,7 +26,6 @@ class NotificationOutputSchema(LemurOutputSchema):
     active = fields.Boolean()
     options = fields.List(fields.Dict())
     plugin = fields.Nested(PluginOutputSchema)
-    certificates = fields.Nested(AssociatedCertificateSchema, many=True, missing=[])
 
     @post_dump
     def fill_object(self, data):

lemur/plugins/base/v1.py

@@ -101,7 +101,7 @@ class IPlugin(local):
         Returns a list of tuples pointing to various resources for this plugin.
         >>> def get_resource_links(self):
         >>>     return [
-        >>>         ('Documentation', 'http://lemury.readthedocs.org'),
+        >>>         ('Documentation', 'https://lemur.readthedocs.io'),
         >>>         ('Bug Tracker', 'https://github.com/Netflix/lemur/issues'),
         >>>         ('Source', 'https://github.com/Netflix/lemur'),
         >>>     ]

lemur/plugins/bases/__init__.py

@@ -1,4 +1,4 @@
-from .destination import DestinationPlugin  # noqa
+from .destination import DestinationPlugin, ExportDestinationPlugin  # noqa
 from .issuer import IssuerPlugin  # noqa
 from .source import SourcePlugin  # noqa
 from .notification import NotificationPlugin, ExpirationNotificationPlugin  # noqa

lemur/plugins/bases/destination.py

@@ -6,7 +6,7 @@
 
 .. moduleauthor:: Kevin Glisson <kglisson@netflix.com>
 """
-from lemur.plugins.base import Plugin
+from lemur.plugins.base import Plugin, plugins
 
 
 class DestinationPlugin(Plugin):
@@ -15,3 +15,32 @@ class DestinationPlugin(Plugin):
 
     def upload(self, name, body, private_key, cert_chain, options, **kwargs):
         raise NotImplementedError
+
+
+class ExportDestinationPlugin(DestinationPlugin):
+    default_options = [
+        {
+            'name': 'exportPlugin',
+            'type': 'export-plugin',
+            'required': True,
+            'helpMessage': 'Export plugin to use before sending data to destination.'
+        }
+    ]
+
+    @property
+    def options(self):
+        return self.default_options + self.additional_options
+
+    def export(self, body, private_key, cert_chain, options):
+        export_plugin = self.get_option('exportPlugin', options)
+
+        if export_plugin:
+            plugin = plugins.get(export_plugin['slug'])
+            extension, passphrase, data = plugin.export(body, cert_chain, private_key, export_plugin['plugin_options'])
+            return [(extension, passphrase, data)]
+
+        data = body + '\n' + cert_chain + '\n' + private_key
+        return [('.pem', '', data)]
+
+    def upload(self, name, body, private_key, cert_chain, options, **kwargs):
+        raise NotImplementedError

lemur/plugins/bases/issuer.py

@@ -21,3 +21,6 @@ class IssuerPlugin(Plugin):
 
     def create_authority(self, options):
         raise NotImplementedError
+
+    def revoke_certificate(self, certificate, comments):
+        raise NotImplementedError

lemur/plugins/bases/notification.py

@@ -46,7 +46,7 @@ class ExpirationNotificationPlugin(NotificationPlugin):
 
     @property
     def options(self):
-        return list(self.default_options) + self.additional_options
+        return self.default_options + self.additional_options
 
     def send(self, notification_type, message, targets, options, **kwargs):
         raise NotImplementedError

lemur/plugins/bases/source.py

@@ -33,4 +33,4 @@ class SourcePlugin(Plugin):
 
     @property
     def options(self):
-        return list(self.default_options) + self.additional_options
+        return self.default_options + self.additional_options

lemur/plugins/lemur_acme/plugin.py

@@ -19,8 +19,6 @@ from acme import challenges
 
 from lemur.common.utils import generate_private_key
 
-from cryptography.hazmat.primitives import serialization
-
 import OpenSSL.crypto
 
 from lemur.common.utils import validate_conf
@@ -87,8 +85,8 @@ def request_certificate(acme_client, authorizations, csr):
     cert_response, _ = acme_client.poll_and_request_issuance(
         jose.util.ComparableX509(
             OpenSSL.crypto.load_certificate_request(
-                OpenSSL.crypto.FILETYPE_ASN1,
+                OpenSSL.crypto.FILETYPE_PEM,
-                csr.public_bytes(serialization.Encoding.DER),
+                csr
             )
         ),
         authzrs=[authz_record.authz for authz_record in authorizations],
@@ -96,12 +94,13 @@ def request_certificate(acme_client, authorizations, csr):
 
     pem_certificate = OpenSSL.crypto.dump_certificate(
         OpenSSL.crypto.FILETYPE_PEM, cert_response.body
-    )
+    ).decode('utf-8')
 
-    pem_certificate_chain = "\n".join(
+    # https://github.com/alex/letsencrypt-aws/commit/853ea7f93f141fe18d9ef12aee6b3388f98b4830
+    pem_certificate_chain = b"\n".join(
         OpenSSL.crypto.dump_certificate(OpenSSL.crypto.FILETYPE_PEM, cert)
         for cert in acme_client.fetch_chain(cert_response)
-    )
+    ).decode('utf-8')
 
     return pem_certificate, pem_certificate_chain
 
@@ -133,7 +132,7 @@ def get_domains(options):
     domains = [options['common_name']]
     if options.get('extensions'):
         for name in options['extensions']['sub_alt_names']['names']:
-            domains.append(name)
+            domains.append(name.value)
     return domains
 
 
@@ -194,7 +193,8 @@ class ACMEIssuerPlugin(IssuerPlugin):
         domains = get_domains(issuer_options)
         authorizations = get_authorizations(acme_client, account_number, domains)
         pem_certificate, pem_certificate_chain = request_certificate(acme_client, authorizations, csr)
-        return pem_certificate, pem_certificate_chain
+        # TODO add external ID (if possible)
+        return pem_certificate, pem_certificate_chain, None
 
     @staticmethod
     def create_authority(options):

lemur/plugins/lemur_acme/route53.py

@@ -27,6 +27,7 @@ def find_zone_id(domain, client=None):
         raise ValueError(
             "Unable to find a Route53 hosted zone for {}".format(domain)
         )
+    return zones[0][1]
 
 
 @sts_client('route53')
@@ -54,7 +55,7 @@ def change_txt_record(action, zone_id, domain, value, client=None):
     return response["ChangeInfo"]["Id"]
 
 
-def create_txt_record(account_number, host, value):
+def create_txt_record(host, value, account_number):
     zone_id = find_zone_id(host, account_number=account_number)
     change_id = change_txt_record(
         "CREATE",

lemur/plugins/lemur_aws/elb.py

@@ -98,7 +98,7 @@ def get_all_elbs_v2(**kwargs):
 @retry(retry_on_exception=retry_throttled, stop_max_attempt_number=7, wait_exponential_multiplier=1000)
 def get_listener_arn_from_endpoint(endpoint_name, endpoint_port, **kwargs):
     """
-    Get a listener ARN from a endpoint.
+    Get a listener ARN from an endpoint.
     :param endpoint_name:
     :param endpoint_port:
     :return:

lemur/plugins/lemur_aws/iam.py

@@ -53,7 +53,7 @@ def create_arn_from_cert(account_number, region, certificate_name):
 
 @sts_client('iam')
 @retry(retry_on_exception=retry_throttled, stop_max_attempt_number=7, wait_exponential_multiplier=100)
-def upload_cert(name, body, private_key, cert_chain=None, **kwargs):
+def upload_cert(name, body, private_key, path, cert_chain=None, **kwargs):
     """
     Upload a certificate to AWS
 
@@ -61,12 +61,20 @@ def upload_cert(name, body, private_key, cert_chain=None, **kwargs):
     :param body:
     :param private_key:
     :param cert_chain:
+    :param path:
     :return:
     """
     client = kwargs.pop('client')
+
+    if not path:
+        path = '/'
+    else:
+        name = name + '-' + path.strip('/')
+
     try:
         if cert_chain:
             return client.upload_server_certificate(
+                Path=path,
                 ServerCertificateName=name,
                 CertificateBody=str(body),
                 PrivateKey=str(private_key),
@@ -74,6 +82,7 @@ def upload_cert(name, body, private_key, cert_chain=None, **kwargs):
             )
         else:
             return client.upload_server_certificate(
+                Path=path,
                 ServerCertificateName=name,
                 CertificateBody=str(body),
                 PrivateKey=str(private_key)

lemur/plugins/lemur_aws/plugin.py

@@ -34,9 +34,9 @@
 """
 from flask import current_app
 
-from lemur.plugins.bases import DestinationPlugin, SourcePlugin
-from lemur.plugins.lemur_aws import iam, s3, elb, ec2
 from lemur.plugins import lemur_aws as aws
+from lemur.plugins.lemur_aws import iam, s3, elb, ec2
+from lemur.plugins.bases import DestinationPlugin, ExportDestinationPlugin, SourcePlugin
 
 
 def get_region_from_dns(dns):
@@ -105,9 +105,12 @@ def get_elb_endpoints(account_number, region, elb_dict):
         )
 
         if listener['PolicyNames']:
-            policy = elb.describe_load_balancer_policies(elb_dict['LoadBalancerName'], listener['PolicyNames'], account_number=account_number, region=region)
+            policy = elb.describe_load_balancer_policies(elb_dict['LoadBalancerName'], listener['PolicyNames'],
+                                                         account_number=account_number, region=region)
             endpoint['policy'] = format_elb_cipher_policy(policy)
 
+        current_app.logger.debug("Found new endpoint. Endpoint: {}".format(endpoint))
+
         endpoints.append(endpoint)
 
     return endpoints
@@ -122,7 +125,8 @@ def get_elb_endpoints_v2(account_number, region, elb_dict):
     :return:
     """
     endpoints = []
-    listeners = elb.describe_listeners_v2(account_number=account_number, region=region, LoadBalancerArn=elb_dict['LoadBalancerArn'])
+    listeners = elb.describe_listeners_v2(account_number=account_number, region=region,
+                                          LoadBalancerArn=elb_dict['LoadBalancerArn'])
     for listener in listeners['Listeners']:
         if not listener.get('Certificates'):
             continue
@@ -161,6 +165,12 @@ class AWSDestinationPlugin(DestinationPlugin):
             'required': True,
             'validation': '/^[0-9]{12,12}$/',
             'helpMessage': 'Must be a valid AWS account number!',
+        },
+        {
+            'name': 'path',
+            'type': 'str',
+            'default': '/',
+            'helpMessage': 'Path to upload certificate.'
         }
     ]
 
@@ -172,6 +182,7 @@ class AWSDestinationPlugin(DestinationPlugin):
 
     def upload(self, name, body, private_key, cert_chain, options, **kwargs):
         iam.upload_cert(name, body, private_key,
+                        self.get_option('path', options),
                         cert_chain=cert_chain,
                         account_number=self.get_option('accountNumber', options))
 
@@ -205,7 +216,8 @@ class AWSSourcePlugin(SourcePlugin):
 
     def get_certificates(self, options, **kwargs):
         cert_data = iam.get_all_certificates(account_number=self.get_option('accountNumber', options))
-        return [dict(body=c['CertificateBody'], chain=c.get('CertificateChain'), name=c['ServerCertificateMetadata']['ServerCertificateName']) for c in cert_data]
+        return [dict(body=c['CertificateBody'], chain=c.get('CertificateChain'),
+                     name=c['ServerCertificateMetadata']['ServerCertificateName']) for c in cert_data]
 
     def get_endpoints(self, options, **kwargs):
         endpoints = []
@@ -242,8 +254,10 @@ class AWSSourcePlugin(SourcePlugin):
         arn = iam.create_arn_from_cert(account_number, region, certificate.name)
 
         if endpoint.type == 'elbv2':
-            listener_arn = elb.get_listener_arn_from_endpoint(endpoint.name, endpoint.port, account_number=account_number, region=region)
+            listener_arn = elb.get_listener_arn_from_endpoint(endpoint.name, endpoint.port,
-            elb.attach_certificate_v2(listener_arn, endpoint.port, [{'CertificateArn': arn}], account_number=account_number, region=region)
+                                                              account_number=account_number, region=region)
+            elb.attach_certificate_v2(listener_arn, endpoint.port, [{'CertificateArn': arn}],
+                                      account_number=account_number, region=region)
         else:
             elb.attach_certificate(endpoint.name, endpoint.port, arn, account_number=account_number, region=region)
 
@@ -252,7 +266,7 @@ class AWSSourcePlugin(SourcePlugin):
         iam.delete_cert(certificate.name, account_number=account_number)
 
 
-class S3DestinationPlugin(DestinationPlugin):
+class S3DestinationPlugin(ExportDestinationPlugin):
     title = 'AWS-S3'
     slug = 'aws-s3'
     description = 'Allow the uploading of certificates to Amazon S3'
@@ -260,7 +274,7 @@ class S3DestinationPlugin(DestinationPlugin):
     author = 'Mikhail Khodorovskiy, Harm Weites <harm@weites.com>'
     author_url = 'https://github.com/Netflix/lemur'
 
-    options = [
+    additional_options = [
         {
             'name': 'bucket',
             'type': 'str',
@@ -278,56 +292,42 @@ class S3DestinationPlugin(DestinationPlugin):
         {
             'name': 'region',
             'type': 'str',
-            'default': 'eu-west-1',
+            'default': 'us-east-1',
             'required': False,
-            'validation': '/^\w+-\w+-\d+$/',
+            'helpMessage': 'Region bucket exists',
-            'helpMessage': 'Availability zone to use',
+            'available': ['us-east-1', 'us-west-2', 'eu-west-1']
         },
         {
             'name': 'encrypt',
             'type': 'bool',
             'required': False,
-            'helpMessage': 'Availability zone to use',
+            'helpMessage': 'Enable server side encryption',
             'default': True
         },
         {
-            'name': 'key',
+            'name': 'prefix',
             'type': 'str',
             'required': False,
             'validation': '/^$|\s+/',
-            'helpMessage': 'Must be a valid S3 object key!',
+            'helpMessage': 'Must be a valid S3 object prefix!',
-        },
-        {
-            'name': 'caKey',
-            'type': 'str',
-            'required': False,
-            'validation': '/^$|\s+/',
-            'helpMessage': 'Must be a valid S3 object key!',
-        },
-        {
-            'name': 'certKey',
-            'type': 'str',
-            'required': False,
-            'validation': '/^$|\s+/',
-            'helpMessage': 'Must be a valid S3 object key!',
         }
     ]
 
     def __init__(self, *args, **kwargs):
         super(S3DestinationPlugin, self).__init__(*args, **kwargs)
 
-    def upload(self, name, body, private_key, cert_chain, options, **kwargs):
+    def upload(self, name, body, private_key, chain, options, **kwargs):
-        account_number = self.get_option('accountNumber', options)
+        files = self.export(body, private_key, chain, options)
-        encrypt = self.get_option('encrypt', options)
-        bucket = self.get_option('bucket', options)
-        key = self.get_option('key', options)
-        ca_key = self.get_option('caKey', options)
-        cert_key = self.get_option('certKey', options)
 
-        if key and ca_key and cert_key:
+        for ext, passphrase, data in files:
-            s3.write_to_s3(account_number, bucket, key, private_key, encrypt=encrypt)
+            s3.put(
-            s3.write_to_s3(account_number, bucket, ca_key, cert_chain, encrypt=encrypt)
+                self.get_option('bucket', options),
-            s3.write_to_s3(account_number, bucket, cert_key, body, encrypt=encrypt)
+                self.get_option('region', options),
-        else:
+                '{prefix}/{name}.{extension}'.format(
-            pem_body = key + '\n' + body + '\n' + cert_chain + '\n'
+                    prefix=self.get_option('prefix', options),
-            s3.write_to_s3(account_number, bucket, name, pem_body, encrypt=encrypt)
+                    name=name,
+                    extension=ext),
+                data,
+                self.get_option('encrypt', options),
+                account_number=self.get_option('accountNumber', options)
+            )

lemur/plugins/lemur_aws/s3.py

@@ -6,21 +6,32 @@
     :license: Apache, see LICENSE for more details.
 .. moduleauthor:: Kevin Glisson <kglisson@netflix.com>
 """
-from boto.s3.key import Key
+from flask import current_app
-from lemur.plugins.lemur_aws.sts import assume_service
+from .sts import sts_client
 
 
-def write_to_s3(account_number, bucket_name, key, data, encrypt=True):
+@sts_client('s3', service_type='resource')
+def put(bucket_name, region, prefix, data, encrypt, **kwargs):
     """
     Use STS to write to an S3 bucket
-
-    :param account_number:
-    :param bucket_name:
-    :param data:
     """
-    conn = assume_service(account_number, 's3')
+    bucket = kwargs['resource'].Bucket(bucket_name)
-    b = conn.get_bucket(bucket_name, validate=False)  # validate=False removes need for ListObjects permission
+    current_app.logger.debug('Persisting data to S3. Bucket: {0} Prefix: {1}'.format(bucket_name, prefix))
 
-    k = Key(bucket=b, name=key)
+    # get data ready for writing
-    k.set_contents_from_string(data, encrypt_key=encrypt)
+    if isinstance(data, str):
-    k.set_canned_acl("bucket-owner-read")
+        data = data.encode('utf-8')
+
+    if encrypt:
+        bucket.put_object(
+            Key=prefix,
+            Body=data,
+            ACL='bucket-owner-full-control',
+            ServerSideEncryption='AES256'
+        )
+    else:
+        bucket.put_object(
+            Key=prefix,
+            Body=data,
+            ACL='bucket-owner-full-control'
+        )

lemur/plugins/lemur_aws/sts.py

@@ -7,46 +7,11 @@
 """
 from functools import wraps
 
-import boto
-import boto.ec2.elb
 import boto3
 
 from flask import current_app
 
 
-def assume_service(account_number, service, region='us-east-1'):
-    conn = boto.connect_sts()
-
-    role = conn.assume_role('arn:aws:iam::{0}:role/{1}'.format(
-        account_number, current_app.config.get('LEMUR_INSTANCE_PROFILE', 'Lemur')), 'blah')
-
-    if service in 'iam':
-        return boto.connect_iam(
-            aws_access_key_id=role.credentials.access_key,
-            aws_secret_access_key=role.credentials.secret_key,
-            security_token=role.credentials.session_token)
-
-    elif service in 'elb':
-        return boto.ec2.elb.connect_to_region(
-            region,
-            aws_access_key_id=role.credentials.access_key,
-            aws_secret_access_key=role.credentials.secret_key,
-            security_token=role.credentials.session_token)
-
-    elif service in 'vpc':
-        return boto.connect_vpc(
-            aws_access_key_id=role.credentials.access_key,
-            aws_secret_access_key=role.credentials.secret_key,
-            security_token=role.credentials.session_token)
-
-    elif service in 's3':
-        return boto.s3.connect_to_region(
-            region,
-            aws_access_key_id=role.credentials.access_key,
-            aws_secret_access_key=role.credentials.secret_key,
-            security_token=role.credentials.session_token)
-
-
 def sts_client(service, service_type='client'):
     def decorator(f):
         @wraps(f)

lemur/plugins/lemur_aws/tests/test_elb.py

@@ -1,4 +1,4 @@
-import boto
+import boto3
 from moto import mock_sts, mock_elb
 
 
@@ -6,9 +6,23 @@ from moto import mock_sts, mock_elb
 @mock_elb()
 def test_get_all_elbs(app):
     from lemur.plugins.lemur_aws.elb import get_all_elbs
-    conn = boto.ec2.elb.connect_to_region('us-east-1')
+    client = boto3.client('elb', region_name='us-east-1')
+
     elbs = get_all_elbs(account_number='123456789012', region='us-east-1')
     assert not elbs
-    conn.create_load_balancer('example-lb', ['us-east-1a', 'us-east-1b'], [(443, 5443, 'tcp')])
+
+    client.create_load_balancer(
+        LoadBalancerName='example-lb',
+        Listeners=[
+            {
+                'Protocol': 'string',
+                'LoadBalancerPort': 443,
+                'InstanceProtocol': 'tcp',
+                'InstancePort': 5443,
+                'SSLCertificateId': 'tcp'
+            }
+        ]
+    )
+
     elbs = get_all_elbs(account_number='123456789012', region='us-east-1')
     assert elbs

lemur/plugins/lemur_cfssl/plugin.py

@@ -13,8 +13,11 @@ import requests
 
 from flask import current_app
 
+from lemur.common.utils import parse_certificate
+from lemur.common.utils import get_authority_key
 from lemur.plugins.bases import IssuerPlugin
 from lemur.plugins import lemur_cfssl as cfssl
+from lemur.extensions import metrics
 
 
 class CfsslIssuerPlugin(IssuerPlugin):
@@ -46,10 +49,15 @@ class CfsslIssuerPlugin(IssuerPlugin):
         data = json.dumps(data)
 
         response = self.session.post(url, data=data.encode(encoding='utf_8', errors='strict'))
+        if response.status_code > 399:
+            metrics.send('cfssl_create_certificate_failure', 'counter', 1)
+            raise Exception(
+                "Error revoking cert. Please check your CFSSL API server")
         response_json = json.loads(response.content.decode('utf_8'))
         cert = response_json['result']['certificate']
-
+        parsed_cert = parse_certificate(cert)
-        return cert, current_app.config.get('CFSSL_INTERMEDIATE'),
+        metrics.send('cfssl_create_certificate_success', 'counter', 1)
+        return cert, current_app.config.get('CFSSL_INTERMEDIATE'), parsed_cert.serial_number
 
     @staticmethod
     def create_authority(options):
@@ -62,3 +70,20 @@ class CfsslIssuerPlugin(IssuerPlugin):
         """
         role = {'username': '', 'password': '', 'name': 'cfssl'}
         return current_app.config.get('CFSSL_ROOT'), "", [role]
+
+    def revoke_certificate(self, certificate, comments):
+        """Revoke a CFSSL certificate."""
+        base_url = current_app.config.get('CFSSL_URL')
+        create_url = '{0}/api/v1/cfssl/revoke'.format(base_url)
+        data = '{"serial": "' + certificate.external_id + '","authority_key_id": "' + \
+            get_authority_key(certificate.body) + \
+            '", "reason": "superseded"}'
+        current_app.logger.debug("Revoking cert: {0}".format(data))
+        response = self.session.post(
+            create_url, data=data.encode(encoding='utf_8', errors='strict'))
+        if response.status_code > 399:
+            metrics.send('cfssl_revoke_certificate_failure', 'counter', 1)
+            raise Exception(
+                "Error revoking cert. Please check your CFSSL API server")
+        metrics.send('cfssl_revoke_certificate_success', 'counter', 1)
+        return response.json()

lemur/plugins/lemur_cryptography/plugin.py

@@ -187,7 +187,7 @@ class CryptographyIssuerPlugin(IssuerPlugin):
         """
         current_app.logger.debug("Issuing new cryptography certificate with options: {0}".format(options))
         cert_pem, chain_cert_pem = issue_certificate(csr, options)
-        return cert_pem, chain_cert_pem
+        return cert_pem, chain_cert_pem, None
 
     @staticmethod
     def create_authority(options):

lemur/plugins/lemur_csr/__init__.py

@@ -0,0 +1,5 @@
+try:
+    VERSION = __import__('pkg_resources') \
+        .get_distribution(__name__).version
+except Exception as e:
+    VERSION = 'unknown'

lemur/plugins/lemur_csr/plugin.py

@@ -0,0 +1,104 @@
+"""
+.. module: lemur.plugins.lemur_csr.plugin
+
+An export plugin that exports CSR from a private key and certificate.
+"""
+from io import open
+import subprocess
+
+from flask import current_app
+
+from lemur.utils import mktempfile, mktemppath
+from lemur.plugins.bases import ExportPlugin
+from lemur.plugins import lemur_csr as csr
+
+
+def run_process(command):
+    """
+    Runs a given command with pOpen and wraps some
+    error handling around it.
+    :param command:
+    :return:
+    """
+    p = subprocess.Popen(command, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+    current_app.logger.debug(command)
+    stdout, stderr = p.communicate()
+
+    if p.returncode != 0:
+        current_app.logger.debug(" ".join(command))
+        current_app.logger.error(stderr)
+        raise Exception(stderr)
+
+
+def create_csr(cert, chain, csr_tmp, key):
+    """
+    Creates a csr from key and cert file.
+    :param cert:
+    :param chain:
+    :param csr_tmp:
+    :param key:
+    """
+    if isinstance(cert, bytes):
+        cert = cert.decode('utf-8')
+
+    if isinstance(chain, bytes):
+        chain = chain.decode('utf-8')
+
+    if isinstance(key, bytes):
+        key = key.decode('utf-8')
+
+    with mktempfile() as key_tmp:
+        with open(key_tmp, 'w') as f:
+            f.write(key)
+
+        with mktempfile() as cert_tmp:
+            with open(cert_tmp, 'w') as f:
+                if chain:
+                    f.writelines([cert.strip() + "\n", chain.strip() + "\n"])
+                else:
+                    f.writelines([cert.strip() + "\n"])
+
+            output = subprocess.check_output([
+                "openssl",
+                "x509",
+                "-x509toreq",
+                "-in", cert_tmp,
+                "-signkey", key_tmp,
+            ])
+            subprocess.run([
+                "openssl",
+                "req",
+                "-out", csr_tmp
+            ], input=output)
+
+
+class CSRExportPlugin(ExportPlugin):
+    title = 'CSR'
+    slug = 'openssl-csr'
+    description = 'Exports a CSR'
+    version = csr.VERSION
+
+    author = 'jchuong'
+    author_url = 'https://github.com/jchuong'
+
+    def export(self, body, chain, key, options, **kwargs):
+        """
+        Creates CSR from certificate
+
+        :param key:
+        :param chain:
+        :param body:
+        :param options:
+        :param kwargs:
+        """
+        with mktemppath() as output_tmp:
+            if not key:
+                raise Exception("Private Key required by CSR")
+
+            create_csr(body, chain, output_tmp, key)
+            extension = "csr"
+
+            with open(output_tmp, 'rb') as f:
+                raw = f.read()
+        # passphrase is None
+        return extension, None, raw

lemur/plugins/lemur_csr/tests/conftest.py

@@ -0,0 +1 @@
+from lemur.tests.conftest import *  # noqa

lemur/plugins/lemur_csr/tests/test_csr_export.py

@@ -0,0 +1,13 @@
+import pytest
+from lemur.tests.vectors import INTERNAL_PRIVATE_KEY_A_STR, INTERNAL_CERTIFICATE_A_STR
+
+
+def test_export_certificate_to_csr(app):
+    from lemur.plugins.base import plugins
+    p = plugins.get('openssl-csr')
+    options = []
+    with pytest.raises(Exception):
+        p.export(INTERNAL_CERTIFICATE_A_STR, "", "", options)
+
+    raw = p.export(INTERNAL_CERTIFICATE_A_STR, "", INTERNAL_PRIVATE_KEY_A_STR, options)
+    assert raw != b""

lemur/plugins/lemur_digicert/plugin.py

@@ -83,26 +83,6 @@ def determine_validity_years(end_date):
                     " years in validity")
 
 
-def get_issuance(options):
-    """Get the time range for certificates.
-
-    :param options:
-    :return:
-    """
-
-    validity_years = options.get('validity_years')
-
-    if validity_years:
-        options['validity_end'] = None
-        return options
-    else:
-        if not options.get('validity_end'):
-            options['validity_end'] = arrow.utcnow().replace(years=current_app.config.get('DIGICERT_DEFAULT_VALIDITY', 1))
-
-        options['validity_years'] = determine_validity_years(options['validity_end'])
-        return options
-
-
 def get_additional_names(options):
     """
     Return a list of strings to be added to a SAN certificates.
@@ -126,7 +106,9 @@ def map_fields(options, csr):
     :param csr:
     :return: dict or valid DigiCert options
     """
-    options = get_issuance(options)
+    if not options.get('validity_years'):
+        if not options.get('validity_end'):
+            options['validity_years'] = current_app.config.get('DIGICERT_DEFAULT_VALIDITY', 1)
 
     data = dict(certificate={
         "common_name": options['common_name'],
@@ -139,10 +121,16 @@ def map_fields(options, csr):
 
     data['certificate']['dns_names'] = get_additional_names(options)
 
-    if options.get('validity_end'):
+    if options.get('validity_years'):
+        data['validity_years'] = options['validity_years']
+    else:
         data['custom_expiration_date'] = options['validity_end'].format('YYYY-MM-DD')
 
-    data['validity_years'] = options.get('validity_years')
+    if current_app.config.get('DIGICERT_PRIVATE', False):
+        if 'product' in data:
+            data['product']['type_hint'] = 'private'
+        else:
+            data['product'] = dict(type_hint='private')
 
     return data
 
@@ -155,7 +143,13 @@ def map_cis_fields(options, csr):
     :param csr:
     :return:
     """
-    options = get_issuance(options)
+    if not options.get('validity_years'):
+        if not options.get('validity_end'):
+            options['validity_end'] = arrow.utcnow().replace(years=current_app.config.get('DIGICERT_DEFAULT_VALIDITY', 1))
+        options['validity_years'] = determine_validity_years(options['validity_end'])
+    else:
+        options['validity_end'] = arrow.utcnow().replace(years=options['validity_years'])
+
     data = {
         "profile_name": current_app.config.get('DIGICERT_CIS_PROFILE_NAME'),
         "common_name": options['common_name'],
@@ -181,7 +175,7 @@ def handle_response(response):
     :return:
     """
     if response.status_code > 399:
-        raise Exception(response.json()['message'])
+        raise Exception(response.json()['errors'][0]['message'])
 
     return response.json()
 
@@ -241,7 +235,6 @@ class DigiCertSourcePlugin(SourcePlugin):
             'DIGICERT_URL',
             'DIGICERT_ORG_ID',
             'DIGICERT_ROOT',
-            'DIGICERT_INTERMEDIATE'
         ]
         validate_conf(current_app, required_vars)
 
@@ -279,7 +272,6 @@ class DigiCertIssuerPlugin(IssuerPlugin):
             'DIGICERT_URL',
             'DIGICERT_ORG_ID',
             'DIGICERT_ROOT',
-            'DIGICERT_INTERMEDIATE'
         ]
 
         validate_conf(current_app, required_vars)
@@ -311,16 +303,26 @@ class DigiCertIssuerPlugin(IssuerPlugin):
         response = self.session.post(determinator_url, data=json.dumps(data))
 
         if response.status_code > 399:
-            raise Exception(response.json()['message'])
+            raise Exception(response.json()['errors'][0]['message'])
 
         order_id = response.json()['id']
 
         certificate_id = get_certificate_id(self.session, base_url, order_id)
 
-        # retrieve ceqrtificate
+        # retrieve certificate
         certificate_url = "{0}/services/v2/certificate/{1}/download/format/pem_all".format(base_url, certificate_id)
         end_entity, intermediate, root = pem.parse(self.session.get(certificate_url).content)
-        return "\n".join(str(end_entity).splitlines()), "\n".join(str(end_entity).splitlines())
+        return "\n".join(str(end_entity).splitlines()), "\n".join(str(intermediate).splitlines()), certificate_id
+
+    def revoke_certificate(self, certificate, comments):
+        """Revoke a Digicert certificate."""
+        base_url = current_app.config.get('DIGICERT_URL')
+
+        # make certificate revoke request
+        create_url = '{0}/certificate/{1}/revoke'.format(base_url, certificate.external_id)
+        metrics.send('digicert_revoke_certificate', 'counter', 1)
+        response = self.session.put(create_url, data=json.dumps({'comments': comments}))
+        return handle_response(response)
 
     @staticmethod
     def create_authority(options):
@@ -337,6 +339,74 @@ class DigiCertIssuerPlugin(IssuerPlugin):
         return current_app.config.get('DIGICERT_ROOT'), "", [role]
 
 
+class DigiCertCISSourcePlugin(SourcePlugin):
+    """Wrap the Digicert CIS Certifcate API."""
+    title = 'DigiCert'
+    slug = 'digicert-cis-source'
+    description = "Enables the use of Digicert as a source of existing certificates."
+    version = digicert.VERSION
+
+    author = 'Kevin Glisson'
+    author_url = 'https://github.com/netflix/lemur.git'
+
+    additional_options = []
+
+    def __init__(self, *args, **kwargs):
+        """Initialize source with appropriate details."""
+        required_vars = [
+            'DIGICERT_CIS_API_KEY',
+            'DIGICERT_CIS_URL',
+            'DIGICERT_CIS_ROOT',
+            'DIGICERT_CIS_INTERMEDIATE',
+            'DIGICERT_CIS_PROFILE_NAME'
+        ]
+        validate_conf(current_app, required_vars)
+
+        self.session = requests.Session()
+        self.session.headers.update(
+            {
+                'X-DC-DEVKEY': current_app.config['DIGICERT_CIS_API_KEY'],
+                'Content-Type': 'application/json'
+            }
+        )
+
+        self.session.hooks = dict(response=log_status_code)
+
+        a = requests.adapters.HTTPAdapter(max_retries=3)
+        self.session.mount('https://', a)
+
+        super(DigiCertCISSourcePlugin, self).__init__(*args, **kwargs)
+
+    def get_certificates(self, options, **kwargs):
+        """Fetch all Digicert certificates."""
+        base_url = current_app.config.get('DIGICERT_CIS_URL')
+
+        # make request
+        search_url = '{0}/platform/cis/certificate/search'.format(base_url)
+
+        certs = []
+        page = 1
+
+        while True:
+            response = self.session.get(search_url, params={'status': ['issued'], 'page': page})
+            data = handle_cis_response(response)
+
+            for c in data['certificates']:
+                download_url = '{0}/platform/cis/certificate/{1}'.format(base_url, c['id'])
+                certificate = self.session.get(download_url)
+
+                # normalize serial
+                serial = str(int(c['serial_number'], 16))
+                cert = {'body': certificate.content, 'serial': serial, 'external_id': c['id']}
+                certs.append(cert)
+
+            if page == data['total_pages']:
+                break
+
+            page += 1
+        return certs
+
+
 class DigiCertCISIssuerPlugin(IssuerPlugin):
     """Wrap the Digicert Certificate Issuing API."""
     title = 'DigiCert CIS'
@@ -387,7 +457,22 @@ class DigiCertCISIssuerPlugin(IssuerPlugin):
 
         self.session.headers.pop('Accept')
         end_entity = pem.parse(certificate_pem)[0]
-        return "\n".join(str(end_entity).splitlines()), current_app.config.get('DIGICERT_CIS_INTERMEDIATE')
+        return "\n".join(str(end_entity).splitlines()), current_app.config.get('DIGICERT_CIS_INTERMEDIATE'), data['id']
+
+    def revoke_certificate(self, certificate, comments):
+        """Revoke a Digicert certificate."""
+        base_url = current_app.config.get('DIGICERT_CIS_URL')
+
+        # make certificate revoke request
+        revoke_url = '{0}/platform/cis/certificate/{1}/revoke'.format(base_url, certificate.external_id)
+        metrics.send('digicert_revoke_certificate_success', 'counter', 1)
+        response = self.session.put(revoke_url, data=json.dumps({'comments': comments}))
+
+        if response.status_code != 204:
+            metrics.send('digicert_revoke_certificate_failure', 'counter', 1)
+            raise Exception('Failed to revoke certificate.')
+
+        metrics.send('digicert_revoke_certificate_success', 'counter', 1)
 
     @staticmethod
     def create_authority(options):

lemur/plugins/lemur_digicert/tests/test_digicert.py

@@ -1,5 +1,6 @@
 import pytest
 import arrow
+import json
 from freezegun import freeze_time
 
 from lemur.tests.vectors import CSR_STR
@@ -35,8 +36,7 @@ def test_map_fields_with_validity_end_and_start(app):
             'signature_hash': 'sha256'
         },
         'organization': {'id': 111111},
-        'custom_expiration_date': arrow.get(2017, 5, 7).format('YYYY-MM-DD'),
+        'custom_expiration_date': arrow.get(2017, 5, 7).format('YYYY-MM-DD')
-        'validity_years': 1
     }
 
 
@@ -106,35 +106,35 @@ def test_map_cis_fields(app):
         'profile_name': None
     }
 
-
+    options = {
-def test_issuance():
+        'common_name': 'example.com',
-    from lemur.plugins.lemur_digicert.plugin import get_issuance
+        'owner': 'bob@example.com',
+        'description': 'test certificate',
+        'extensions': {
+            'sub_alt_names': {
+                'names': [x509.DNSName(x) for x in names]
+            }
+        },
+        'organization': 'Example, Inc.',
+        'organizational_unit': 'Example Org',
+        'validity_years': 2
+    }
 
     with freeze_time(time_to_freeze=arrow.get(2016, 11, 3).datetime):
-        options = {
+        data = map_cis_fields(options, CSR_STR)
-            'validity_end': arrow.get(2018, 5, 7),
+
-            'validity_start': arrow.get(2016, 10, 30)
+        assert data == {
+            'common_name': 'example.com',
+            'csr': CSR_STR,
+            'additional_dns_names': names,
+            'signature_hash': 'sha256',
+            'organization': {'name': 'Example, Inc.', 'units': ['Example Org']},
+            'validity': {
+                'valid_to': arrow.get(2018, 11, 3).format('YYYY-MM-DD')
+            },
+            'profile_name': None
         }
 
-        new_options = get_issuance(options)
-        assert new_options['validity_years'] == 2
-
-        options = {
-            'validity_end': arrow.get(2017, 5, 7),
-            'validity_start': arrow.get(2016, 10, 30)
-        }
-
-        new_options = get_issuance(options)
-        assert new_options['validity_years'] == 1
-
-        options = {
-            'validity_end': arrow.get(2020, 5, 7),
-            'validity_start': arrow.get(2016, 10, 30)
-        }
-
-        with pytest.raises(Exception):
-            period = get_issuance(options)
-
 
 def test_signature_hash(app):
     from lemur.plugins.lemur_digicert.plugin import signature_hash
@@ -146,3 +146,32 @@ def test_signature_hash(app):
 
     with pytest.raises(Exception):
         signature_hash('sdfdsf')
+
+
+def test_issuer_plugin_create_certificate():
+    import requests_mock
+    from lemur.plugins.lemur_digicert.plugin import DigiCertIssuerPlugin
+
+    pem_fixture = """\
+-----BEGIN CERTIFICATE-----
+abc
+-----END CERTIFICATE-----
+-----BEGIN CERTIFICATE-----
+def
+-----END CERTIFICATE-----
+-----BEGIN CERTIFICATE-----
+ghi
+-----END CERTIFICATE-----
+"""
+
+    subject = DigiCertIssuerPlugin()
+    adapter = requests_mock.Adapter()
+    adapter.register_uri('POST', 'mock://www.digicert.com/services/v2/order/certificate/ssl', text=json.dumps({'id': 'id123'}))
+    adapter.register_uri('GET', 'mock://www.digicert.com/services/v2/order/certificate/id123', text=json.dumps({'status': 'issued', 'certificate': {'id': 'cert123'}}))
+    adapter.register_uri('GET', 'mock://www.digicert.com/services/v2/certificate/cert123/download/format/pem_all', text=pem_fixture)
+    subject.session.mount('mock', adapter)
+
+    cert, intermediate, external_id = subject.create_certificate("", {'common_name': 'test.com'})
+
+    assert cert == "-----BEGIN CERTIFICATE-----\nabc\n-----END CERTIFICATE-----"
+    assert intermediate == "-----BEGIN CERTIFICATE-----\ndef\n-----END CERTIFICATE-----"

lemur/plugins/lemur_email/plugin.py

@@ -6,7 +6,7 @@
 
 .. moduleauthor:: Kevin Glisson <kglisson@netflix.com>
 """
-import boto.ses
+import boto3
 from flask import current_app
 from flask_mail import Message
 
@@ -54,8 +54,25 @@ def send_via_ses(subject, body, targets):
     :param targets:
     :return:
     """
-    conn = boto.connect_ses()
+    client = boto3.client('ses', region_name='us-east-1')
-    conn.send_email(current_app.config.get("LEMUR_EMAIL"), subject, body, targets, format='html')
+    client.send_email(
+        Source=current_app.config.get('LEMUR_EMAIL'),
+        Destination={
+            'ToAddresses': targets
+        },
+        Message={
+            'Subject': {
+                'Data': subject,
+                'Charset': 'UTF-8'
+            },
+            'Body': {
+                'Html': {
+                    'Data': body,
+                    'Charset': 'UTF-8'
+                }
+            }
+        }
+    )
 
 
 class EmailNotificationPlugin(ExpirationNotificationPlugin):

lemur/plugins/lemur_email/templates/config.py

@@ -1,11 +1,12 @@
 import os
 import arrow
-from jinja2 import Environment, FileSystemLoader
+from jinja2 import Environment, FileSystemLoader, select_autoescape
 
 from lemur.plugins.utils import get_plugin_option
 
 loader = FileSystemLoader(searchpath=os.path.dirname(os.path.realpath(__file__)))
-env = Environment(loader=loader)
+env = Environment(loader=loader,  # nosec: potentially dangerous types esc.
+                  autoescape=select_autoescape(['html', 'xml']))
 
 
 def human_time(time):

lemur/plugins/lemur_email/tests/test_email.py

@@ -19,19 +19,15 @@ def test_render(certificate, endpoint):
 
     template = env.get_template('{}.html'.format('expiration'))
 
-    with open(os.path.join(dir_path, 'expiration-rendered.html'), 'w') as f:
+    body = template.render(dict(message=data, hostname='lemur.test.example.com'))
-        body = template.render(dict(message=data, hostname='lemur.test.example.com'))
-        f.write(body)
 
     template = env.get_template('{}.html'.format('rotation'))
 
     certificate.endpoints.append(endpoint)
 
-    with open(os.path.join(dir_path, 'rotation-rendered.html'), 'w') as f:
+    body = template.render(
-        body = template.render(
+        dict(
-            dict(
+            certificate=certificate_notification_output_schema.dump(certificate).data,
-                certificate=certificate_notification_output_schema.dump(certificate).data,
+            hostname='lemur.test.example.com'
-                hostname='lemur.test.example.com'
-            )
         )
-        f.write(body)
+    )