Merge pull request #1372 from castrapel/acme_validation_dns_provider_option

R53: Extend only TXT records

.coveragerc

@@ -1,4 +1,5 @@
 [report]
 include = lemur/*.py
 omit = lemur/migrations/*
+       lemur/tests/*
 

.gitignore

@@ -13,6 +13,7 @@
 MANIFEST
 test.conf
 pip-log.txt
+package-lock.json
 /htmlcov
 /cover
 /build
@@ -27,5 +28,7 @@ pip-log.txt
 docs/_build
 .editorconfig
 .idea
-test.conf
-lemur/tests/tmp
+lemur/tests/tmp
+
+/lemur/plugins/lemur_email/tests/expiration-rendered.html
+/lemur/plugins/lemur_email/tests/rotation-rendered.html

.pre-commit-config.yaml

@@ -1,10 +1,10 @@
 -   repo: git://github.com/pre-commit/pre-commit-hooks
-    sha: 18d7035de5388cc7775be57f529c154bf541aab9
+    sha: v0.9.1
     hooks:
     -   id: trailing-whitespace
     -   id: flake8
     -   id: check-merge-conflict
 -   repo: git://github.com/pre-commit/mirrors-jshint
-    sha: e72140112bdd29b18b0c8257956c896c4c3cebcb
+    sha: v2.9.5
     hooks:
     -   id: jshint

.travis.yml

@@ -34,9 +34,11 @@ before_script:
 
 install:
   - pip install coveralls
+  - pip install bandit
 
 script:
   - make test
+  - bandit -r . -ll -ii -x lemur/tests/,docs
 
 after_success:
   - coveralls

CHANGELOG.rst

@@ -1,6 +1,81 @@
 Changelog
 =========
 
+
+0.7 - `2018-05-07`
+~~~~~~~~~~~~~~
+
+This release adds LetsEncrypt support with DNS providers Dyn, Route53, and Cloudflare, and expands on the pending certificate functionality.
+The linux_dst plugin will also be deprecated and removed.
+
+The pending_dns_authorizations and dns_providers tables were created. New columns
+were added to the certificates and pending_certificates tables, (For the DNS provider ID), and authorities (For options).
+Please run a database migration when upgrading.
+
+The Let's Encrypt flow will run asynchronously. When a certificate is requested through the acme-issuer, a pending certificate
+will be created. A cron needs to be defined to run `lemur pending_certs fetch_all_acme`. This command will iterate through all of the pending
+certificates, request a DNS challenge token from Let's Encrypt, and set the appropriate _acme-challenge TXT entry. It will
+then iterate through and resolve the challenges before requesting a certificate for each pending certificate. If a certificate
+is successfully obtained, the pending_certificate will be moved to the certificates table with the appropriate properties.
+
+Special thanks to all who helped with this release, notably:
+
+- The folks at Cloudflare
+- dmitryzykov
+- jchuong
+- seils
+- titouanc
+
+
+Upgrading
+---------
+
+.. note:: This release will need a migration change. Please follow the `documentation <https://lemur.readthedocs.io/en/latest/administration.html#upgrading-lemur>`_ to upgrade Lemur.
+
+0.6 - `2018-01-02`
+~~~~~~~~~~~~~~~~~~
+
+Happy Holidays! This is a big release with lots of bug fixes and features. Below are the highlights and are not exhaustive.
+
+
+Features:
+
+* Per-certificate rotation policies, requires a database migration. The default rotation policy for all certificates.
+is 30 days. Every certificate will gain a policy regardless of if auto-rotation is used.
+* Adds per-user API Keys, allows users to issue multiple long-lived API tokens with the same permission as the user creating them.
+* Adds the ability to revoke certificates from the Lemur UI/API, this is currently only supported for the digicert CIS and cfssl plugins.
+* Allow destinations to support an export function. Useful for file system destinations e.g. S3 to specify the export plugin you wish to run before being sent to the destination.
+* Adds support for uploading certificates to Cloudfront.
+* Re-worked certificate metadata pane for improved readability.
+* Adds support for LDAP user authentication
+
+Bugs:
+
+* Closed `#767 <https://github.com/Netflix/lemur/issues/767>`_ - Fixed issue with login redirect loop.
+* Closed `#792 <https://github.com/Netflix/lemur/issues/792>`_ - Fixed an issue with a unique constraint was violated when replacing certificates.
+* Closed `#752 <https://github.com/Netflix/lemur/issues/752>`_ - Fixed an internal server error when validating notification units.
+* Closed `#684 <https://github.com/Netflix/lemur/issues/684>`_ - Fixed migration failure when null values encountered.
+* Closes `#661 <https://github.com/Netflix/lemur/issues/661>`_ - Fixed an issue where default values were missing during clone operations.
+
+
+Special thanks to all who helped with this release, notably:
+
+- intgr
+- SecurityInsanity
+- johanneslange
+- RickB17
+- pr8kerl
+- bunjiboys
+
+See the full list of issues closed in `0.6 <https://github.com/Netflix/lemur/milestone/5>`_.
+
+Upgrading
+---------
+
+.. note:: This release will need a migration change. Please follow the `documentation <https://lemur.readthedocs.io/en/latest/administration.html#upgrading-lemur>`_ to upgrade Lemur.
+
+
+
 0.5 - `2016-04-08`
 ~~~~~~~~~~~~~~~~~~
 
@@ -13,19 +88,15 @@ Other Highlights:
 * Closed `#501 <https://github.com/Netflix/lemur/issues/501>`_ - Endpoint resource as now kept in sync via an
 expiration mechanism. Such that non-existant endpoints gracefully fall out of Lemur. Certificates are never
 removed from Lemur.
-
 * Closed `#551 <https://github.com/Netflix/lemur/pull/551>`_ - Added the ability to create a 4096 bit key during certificate
 creation. Closed `#528 <https://github.com/Netflix/lemur/pull/528>`_ to ensure that issuer plugins supported the new 4096 bit keys.
-
 * Closed `#566 <https://github.com/Netflix/lemur/issues/566>`_ - Fixed an issue changing the notification status for  certificates
 without private keys.
-
 * Closed `#594 <https://github.com/Netflix/lemur/issues/594>`_ - Added `replaced` field indicating if a certificate has been superseded.
-
 * Closed `#602 <https://github.com/Netflix/lemur/issues/602>`_ - AWS plugin added support for ALBs for endpoint tracking.
 
 
-Special thanks to all who helped with with this release, notably:
+Special thanks to all who helped with this release, notably:
 
 - RcRonco
 - harmw
@@ -87,7 +158,7 @@ Issuer Plugin Owners
 --------------------
 
 This release may break your plugins, the keys in `issuer_options` have been changed from `camelCase` to `under_score`.
-This change was made to break a undue reliance on downstream options maintains a more pythonic naming convention. Renaming
+This change was made to break an undue reliance on downstream options maintains a more pythonic naming convention. Renaming
 these keys should be fairly trivial, additionally pull requests have been submitted to affected plugins to help ease the transition.
 
 .. note:: This change only affects issuer plugins and does not affect any other types of plugins.
@@ -97,10 +168,10 @@ these keys should be fairly trivial, additionally pull requests have been submit
     stricter input validation and better error messages when validation fails.
 * Closed `#146 <https://github.com/Netflix/lemur/issues/146>`_ - Moved authority type to first pane of authority creation wizard.
 * Closed `#147 <https://github.com/Netflix/lemur/issues/147>`_ - Added and refactored the relationship between authorities and their
-    root certificates. Displays the certificates (and chains) next the the authority in question.
+    root certificates. Displays the certificates (and chains) next to the authority in question.
 * Closed `#199 <https://github.com/Netflix/lemur/issues/199>`_ - Ensures that the dates submitted to Lemur during authority and
     certificate creation are actually dates.
-* Closed `#230 <https://github.com/Netflix/lemur/issues/230>`_ - Migrated authority dropdown to a ui-select based dropdown, this
+* Closed `#230 <https://github.com/Netflix/lemur/issues/230>`_ - Migrated authority dropdown to an ui-select based dropdown, this
     should be easier to determine what authorities are available and when an authority has actually been selected.
 * Closed `#254 <https://github.com/Netflix/lemur/issues/254>`_ - Forces certificate names to be generally unique. If a certificate name
     (generated or otherwise) is found to be a duplicate we increment by appending a counter.
@@ -112,7 +183,7 @@ these keys should be fairly trivial, additionally pull requests have been submit
     via the UI.
 * Closed `#289 <https://github.com/Netflix/lemur/issues/289>`_ - Fixed and issue where intermediates were not being properly exported.
 * Closed `#315 <https://github.com/Netflix/lemur/issues/315>`_ - Made how roles are associated with certificates and authorities much more
-    explict, including adding the ability to add roles directly to certificates and authorities on creation.
+    explicit, including adding the ability to add roles directly to certificates and authorities on creation.
 
 
 
@@ -162,6 +233,6 @@ these keys should be fairly trivial, additionally pull requests have been submit
 0.1.5 - 2015-10-26
 ~~~~~~~~~~~~~~~~~~
 
-* **SECURITY ISSUE**: Switched from use a AES static key to Fernet encryption.
+* **SECURITY ISSUE**: Switched from use an AES static key to Fernet encryption.
   Affects all versions prior to 0.1.5. If upgrading this will require a data migration.
-  see: `Upgrading Lemur <https://lemur.readthedocs.com/adminstration#UpgradingLemur>`_
+  see: `Upgrading Lemur <https://lemur.readthedocs.io/administration#UpgradingLemur>`_

Dockerfile

@@ -0,0 +1,13 @@
+FROM python:3.5
+RUN apt-get update
+RUN apt-get install -y make python-software-properties curl
+RUN curl -sL https://deb.nodesource.com/setup_7.x | bash -
+RUN apt-get update
+RUN apt-get install -y nodejs libldap2-dev libsasl2-dev libldap2-dev libssl-dev
+RUN pip install -U setuptools
+RUN pip install coveralls bandit
+WORKDIR /app
+COPY . /app/
+RUN pip install -e .
+RUN pip install "file://`pwd`#egg=lemur[dev]"
+RUN pip install "file://`pwd`#egg=lemur[tests]"

LICENSE

@@ -186,7 +186,7 @@
       same "printed page" as the copyright notice for easier
       identification within third-party archives.
 
-   Copyright 2014 Netflix, Inc.
+   Copyright 2018 Netflix, Inc.
 
    Licensed under the Apache License, Version 2.0 (the "License");
    you may not use this file except in compliance with the License.

Makefile

@@ -1,6 +1,6 @@
 NPM_ROOT = ./node_modules
 STATIC_DIR = src/lemur/static/app
-
+SHELL=/bin/bash
 USER := $(shell whoami)
 
 develop: update-submodules setup-git
@@ -104,4 +104,24 @@ coverage: develop
 publish:
 	python setup.py sdist bdist_wheel upload
 
+up-reqs:
+ifndef VIRTUAL_ENV
+    $(error Please activate virtualenv first)
+endif
+	@echo "--> Updating Python requirements"
+	pip install --upgrade pip
+	pip install --upgrade pip-tools
+	pip-compile --output-file requirements-docs.txt requirements-docs.in -U --no-index
+	pip-compile --output-file requirements-dev.txt requirements-dev.in -U --no-index
+	pip-compile --output-file requirements-tests.txt requirements-tests.in -U --no-index
+	pip-compile --output-file requirements.txt requirements.in -U --no-index
+	@echo "--> Done updating Python requirements"
+	@echo "--> Removing python-ldap from requirements-docs.txt"
+	grep -v "python-ldap" requirements-docs.txt > tempreqs && mv tempreqs requirements-docs.txt
+	@echo "--> Installing new dependencies"
+	pip install -e .
+	@echo "--> Done installing new dependencies"
+	@echo ""
+
+
 .PHONY: develop dev-postgres dev-docs setup-git build clean update-submodules test testloop test-cli test-js test-python lint lint-python lint-js coverage publish release

README.rst

@@ -6,7 +6,7 @@ Lemur
    :target: https://gitter.im/Netflix/lemur?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge
 
 .. image:: https://readthedocs.org/projects/lemur/badge/?version=latest
-    :target: https://lemur.readthedocs.org
+    :target: https://lemur.readthedocs.io
     :alt: Latest Docs
 
 .. image:: https://img.shields.io/badge/NetflixOSS-active-brightgreen.svg
@@ -14,6 +14,10 @@ Lemur
 .. image:: https://travis-ci.org/Netflix/lemur.svg
     :target: https://travis-ci.org/Netflix/lemur
 
+.. image:: https://coveralls.io/repos/github/Netflix/lemur/badge.svg?branch=master
+    :target: https://coveralls.io/github/Netflix/lemur?branch=master
+
+
 
 Lemur manages TLS certificate creation. While not able to issue certificates itself, Lemur acts as a broker between CAs
 and environments providing a central portal for developers to issue TLS certificates with 'sane' defaults.
@@ -25,7 +29,7 @@ Project resources
 =================
 
 - `Lemur Blog Post <http://techblog.netflix.com/2015/09/introducing-lemur.html>`_
-- `Documentation <http://lemur.readthedocs.org/>`_
+- `Documentation <http://lemur.readthedocs.io/>`_
 - `Source code <https://github.com/netflix/lemur>`_
 - `Issue tracker <https://github.com/netflix/lemur/issues>`_
 - `Docker <https://github.com/Netflix/lemur-docker>`_

bower.json

@@ -31,7 +31,7 @@
     "font-awesome": "~4.5.0",
     "lodash": "~4.0.1",
     "underscore": "~1.8.3",
-    "angular-smart-table": "~2.1.6",
+    "angular-smart-table": "2.1.8",
     "angular-strap": ">= 2.2.2",
     "angular-underscore": "^0.5.0",
     "angular-translate": "^2.9.0",

docker-compose.yml

@@ -0,0 +1,19 @@
+---
+version: '2.0'
+services:
+  test:
+    build: .
+    volumes:
+      - ".:/app"
+    links:
+      - postgres
+    command: make test
+    environment:
+      SQLALCHEMY_DATABASE_URI: postgresql://lemur:lemur@postgres:5432/lemur
+      VIRTUAL_ENV: 'true'
+
+  postgres:
+    image: postgres:9.4
+    environment:
+      POSTGRES_USER: lemur
+      POSTGRES_PASSWORD: lemur

docs/administration.rst

@@ -7,6 +7,10 @@ Configuration
     that allow you to encrypt files at rest and decrypt them when it's time for deployment. See :ref:`Credential Management <CredentialManagement>`
     for more information.
 
+.. note::
+    All configuration values are python strings unless otherwise noted.
+
+
 Basic Configuration
 -------------------
 
@@ -24,14 +28,14 @@ Basic Configuration
 
         LOG_FILE = "/logs/lemur/lemur-test.log"
 
-.. data:: debug
+.. data:: DEBUG
     :noindex:
 
     Sets the flask debug flag to true (if supported by the webserver)
 
     ::
 
-        debug = False
+        DEBUG = False
 
     .. warning::
         This should never be used in a production environment as it exposes Lemur to
@@ -61,16 +65,59 @@ Basic Configuration
         SQLALCHEMY_DATABASE_URI = 'postgresql://<user>:<password>@<hostname>:5432/lemur'
 
 
+.. data:: SQLALCHEMY_POOL_SIZE
+:noindex:
+
+            The default connection pool size is 5 for sqlalchemy managed connections.   Depending on the number of Lemur instances,
+            please specify per instance connection pool size.  Below is an example to set connection pool size to 10.
+
+        ::
+
+        SQLALCHEMY_POOL_SIZE = 10
+
+
+    .. warning::
+This is an optional setting but important to review and set for optimal database connection usage and for overall database performance.
+
+.. data:: SQLALCHEMY_MAX_OVERFLOW
+:noindex:
+
+        This setting allows to create connections in addition to specified number of connections in pool size.   By default, sqlalchemy
+        allows 10 connections to create in addition to the pool size.  This is also an optional setting.  If `SQLALCHEMY_POOL_SIZE` and
+        `SQLALCHEMY_MAX_OVERFLOW` are not speficied then each Lemur instance may create maximum of 15 connections.
+
+    ::
+
+        SQLALCHECK_MAX_OVERFLOW = 0
+
+
+    .. note::
+Specifying the `SQLALCHEMY_MAX_OVERFLOW` to 0 will enforce limit to not create connections above specified pool size.
+
+
 .. data:: LEMUR_ALLOW_WEEKEND_EXPIRATION
     :noindex:
 
         Specifies whether to allow certificates created by Lemur to expire on weekends. Default is True.
 
-.. data:: LEMUR_RESTRICTED_DOMAINS
+.. data:: LEMUR_WHITELISTED_DOMAINS
     :noindex:
 
-        This allows the administrator to mark a subset of domains or domains matching a particular regex as
-        *restricted*. This means that only an administrator is allows to issue the domains in question.
+        List of regular expressions for domain restrictions; if the list is not empty, normal users can only issue
+        certificates for domain names matching at least one pattern on this list. Administrators are exempt from this
+        restriction.
+
+        Cerificate common name is matched against these rules *if* it does not contain a space. SubjectAltName DNS names
+        are always matched against these rules.
+
+        Take care to write patterns in such way to not allow the `*` wildcard character inadvertently. To match a `.`
+        character, it must be escaped (as `\.`).
+
+.. data:: LEMUR_OWNER_EMAIL_IN_SUBJECT
+    :noindex:
+
+        By default, Lemur will add the certificate owner's email address to certificate subject (for CAs that allow it).
+        Set this to `False` to disable this.
 
 .. data:: LEMUR_TOKEN_SECRET
     :noindex:
@@ -109,6 +156,12 @@ Basic Configuration
         LEMUR_ENCRYPTION_KEYS = ['1YeftooSbxCiX2zo8m1lXtpvQjy27smZcUUaGmffhMY=', 'LAfQt6yrkLqOK5lwpvQcT4jf2zdeTQJV1uYeh9coT5s=']
 
 
+.. data:: DEBUG_DUMP
+    :noindex:
+
+        Dump all imported or generated CSR and certificate details to stdout using OpenSSL. (default: `False`)
+
+
 Certificate Default Options
 ---------------------------
 
@@ -202,14 +255,14 @@ Lemur supports sending certification expiration notifications through SES and SM
         you can send any mail. See: `Verifying Email Address in Amazon SES <http://docs.aws.amazon.com/ses/latest/DeveloperGuide/verify-email-addresses.html>`_
 
 
-.. data:: LEMUR_MAIL
+.. data:: LEMUR_EMAIL
     :noindex:
 
         Lemur sender's email
 
         ::
 
-            LEMUR_MAIL = 'lemur.example.com'
+            LEMUR_EMAIL = 'lemur.example.com'
 
 
 .. data:: LEMUR_SECURITY_TEAM_EMAIL
@@ -234,7 +287,120 @@ Lemur supports sending certification expiration notifications through SES and SM
 
 Authentication Options
 ----------------------
-Lemur currently supports Basic Authentication, Ping OAuth2, and Google out of the box. Additional flows can be added relatively easily.
+Lemur currently supports Basic Authentication, LDAP Authentication, Ping OAuth2, and Google out of the box. Additional flows can be added relatively easily.
+
+LDAP Options
+~~~~~~~~~~~~
+
+Lemur supports the use of an LDAP server in conjunction with Basic Authentication. Lemur local users can still be defined and take precedence over LDAP users. If a local user does not exist, LDAP will be queried for authentication. Only simple ldap binding with or without TLS is supported.
+
+LDAP support requires the pyldap python library, which also depends on the following openldap packages.
+
+.. code-block:: bash
+
+      $ sudo apt-get update
+      $ sudo apt-get install libldap2-dev libsasl2-dev libldap2-dev libssl-dev
+
+
+To configure the use of an LDAP server, a number of settings need to be configured in `lemur.conf.py`.
+
+Here is an example LDAP configuration stanza you can add to your config. Adjust to suit your environment of course.
+ 
+.. code-block:: python
+
+        LDAP_AUTH = True
+        LDAP_BIND_URI='ldaps://secure.evilcorp.net'
+        LDAP_BASE_DN='DC=users,DC=evilcorp,DC=net'
+        LDAP_EMAIL_DOMAIN='evilcorp.net'
+        LDAP_USE_TLS = True
+        LDAP_CACERT_FILE = '/opt/lemur/trusted.pem'
+        LDAP_REQUIRED_GROUP = 'certificate-management-access'
+        LDAP_GROUPS_TO_ROLES = {'certificate-management-admin': 'admin', 'certificate-management-read-only': 'read-only'}
+
+
+The lemur ldap module uses the `user principal name` (upn) of the authenticating user to bind. This is done once for each user at login time. The UPN is effectively the email address in AD/LDAP of the user. If the user doesn't provide the email address, it constructs one based on the username supplied (which should normally match the samAccountName) and the value provided by the config LDAP_EMAIL_DOMAIN.
+The config LDAP_BASE_DN tells lemur where to search within the AD/LDAP tree for the given UPN (user). If the bind with those credentials is successful - there is a valid user in AD with correct password.
+
+Each of the LDAP options are described below.
+
+.. data:: LDAP_AUTH
+    :noindex:
+
+        This enables the use of LDAP
+
+        ::
+
+            LDAP_AUTH = True
+
+.. data:: LDAP_BIND_URI
+    :noindex:
+
+        Specifies the LDAP server connection string
+
+        ::
+
+            LDAP_BIND_URI = 'ldaps://hostname'
+
+.. data:: LDAP_BASE_DN
+    :noindex:
+
+        Specifies the LDAP distinguished name location to search for users
+
+        ::
+
+            LDAP_BASE_DN = 'DC=Users,DC=Evilcorp,DC=com'
+
+.. data:: LDAP_EMAIL_DOMAIN
+    :noindex:
+
+        The email domain used by users in your directory. This is used to build the userPrincipalName to search with.
+
+        ::
+
+            LDAP_EMAIL_DOMAIN = 'evilcorp.com'
+
+The following LDAP options are not required, however TLS is always recommended.
+
+.. data:: LDAP_USE_TLS
+    :noindex:
+
+        Enables the use of TLS when connecting to the LDAP server. Ensure the LDAP_BIND_URI is using ldaps scheme.
+
+        ::
+
+            LDAP_USE_TLS = True
+
+.. data:: LDAP_CACERT_FILE
+    :noindex:
+
+        Specify a Certificate Authority file containing PEM encoded trusted issuer certificates. This can be used if your LDAP server is using certificates issued by a private CA.
+
+        ::
+
+            LDAP_CACERT_FILE = '/path/to/cacert/file'
+
+.. data:: LDAP_REQUIRED_GROUP
+    :noindex:
+
+        Lemur has pretty open permissions. You can define an LDAP group to specify who can access Lemur. Only members of this group will be able to login.
+
+        ::
+
+            LDAP_REQUIRED_GROUP = 'Lemur LDAP Group Name'
+
+.. data:: LDAP_GROUPS_TO_ROLES
+    :noindex:
+
+        You can also define a dictionary of ldap groups mapped to lemur roles. This allows you to use ldap groups to manage access to owner/creator roles in Lemur
+
+        ::
+
+            LDAP_GROUPS_TO_ROLES = {'lemur_admins': 'admin', 'Lemur Team DL Group': 'team@example.com'}
+
+
+Authentication Providers
+~~~~~~~~~~~~~~~~~~~~~~~~
+
 If you are not using an authentication provider you do not need to configure any of these options.
 
 For more information about how to use social logins, see: `Satellizer <https://github.com/sahat/satellizer>`_
@@ -360,6 +526,13 @@ For more information about how to use social logins, see: `Satellizer <https://g
 
             OAUTH2_AUTH_ENDPOINT = "https://<youroauthserver>/oauth2/v1/authorize"
 
+.. data:: OAUTH2_VERIFY_CERT
+    :noindex:
+
+        ::
+
+            OAUTH2_VERIFY_CERT = True
+
 .. data:: GOOGLE_CLIENT_ID
     :noindex:
 
@@ -375,6 +548,21 @@ For more information about how to use social logins, see: `Satellizer <https://g
             GOOGLE_SECRET = "somethingsecret"
 
 
+Metric Providers
+~~~~~~~~~~~~~~~~
+
+If you are not using a metric provider you do not need to configure any of these options.
+
+.. data:: ACTIVE_PROVIDERS
+    :noindex:
+
+        A list of metric plugins slugs to be ativated.
+
+        ::
+
+            METRIC_PROVIDERS = ['atlas-metric']
+
+
 Plugin Specific Options
 -----------------------
 
@@ -435,7 +623,13 @@ The following configuration properties are required to use the Digicert issuer p
 .. data:: DIGICERT_URL
     :noindex:
 
-            This is the url for the Digicert API
+            This is the url for the Digicert API (e.g. https://www.digicert.com)
+
+
+.. data:: DIGICERT_ORDER_TYPE
+    :noindex:
+
+            This is the type of certificate to order. (e.g. ssl_plus, ssl_ev_plus see: https://www.digicert.com/services/v2/documentation/order/overview-submit)
 
 
 .. data:: DIGICERT_API_KEY
@@ -450,12 +644,6 @@ The following configuration properties are required to use the Digicert issuer p
             This is the Digicert organization ID tied to your API key
 
 
-.. data:: DIGICERT_INTERMEDIATE
-    :noindex:
-
-            This is the intermediate to be used for your CA chain
-
-
 .. data:: DIGICERT_ROOT
     :noindex:
 
@@ -468,6 +656,11 @@ The following configuration properties are required to use the Digicert issuer p
             This is the default validity (in years), if no end date is specified. (Default: 1)
 
 
+.. data:: DIGICERT_PRIVATE
+    :noindex:
+
+            This is whether or not to issue a private certificate. (Default: False)
+
 
 CFSSL Issuer Plugin
 ^^^^^^^^^^^^^^^^^^^
@@ -824,7 +1017,7 @@ After you have the latest version of the Lemur code base you must run any needed
 This will ensure that any needed tables or columns are created or destroyed.
 
 .. note::
-    Internally, this uses `Alembic <https://alembic.readthedocs.org/en/latest/>`_ to manage database migrations.
+    Internally, this uses `Alembic <http://alembic.zzzcomputing.com/en/latest/>`_ to manage database migrations.
 
 .. note::
     By default Alembic looks for the `migrations` folder in the current working directory.The migrations folder is
@@ -993,6 +1186,31 @@ Digicert
     https://github.com/opendns/lemur-digicert
 
 
+InfluxDB
+--------
+
+:Authors:
+    Titouan Christophe
+:Type:
+    Metric
+:Description:
+    Sends key metrics to InfluxDB
+:Links:
+    https://github.com/titouanc/lemur-influxdb
+
+Hashicorp Vault
+---------------
+
+:Authors:
+    Ron Cohen
+:Type:
+    Issuer
+:Description:
+    Adds support for basic Vault PKI secret backend.
+:Links:
+    https://github.com/RcRonco/lemur_vault
+
+
 Have an extension that should be listed here? Submit a `pull request <https://github.com/netflix/lemur>`_ and we'll
 get it added.
 

docs/conf.py

@@ -13,12 +13,24 @@
 # serve to show the default.
 import sys
 import os
+from unittest.mock import MagicMock
 
 # If extensions (or modules to document with autodoc) are in another directory,
 # add these directories to sys.path here. If the directory is relative to the
 # documentation root, use os.path.abspath to make it absolute, like shown here.
 sys.path.insert(0, os.path.abspath('..'))
 
+# Mock packages that cannot be installed on rtd
+on_rtd = os.environ.get('READTHEDOCS') == 'True'
+if on_rtd:
+    class Mock(MagicMock):
+        @classmethod
+        def __getattr__(cls, name):
+            return MagicMock()
+
+    MOCK_MODULES = ['ldap']
+    sys.modules.update((mod_name, Mock()) for mod_name in MOCK_MODULES)
+
 # -- General configuration ------------------------------------------------
 
 # If your documentation needs a minimal Sphinx version, state it here.
@@ -47,7 +59,7 @@ master_doc = 'index'
 
 # General information about the project.
 project = u'lemur'
-copyright = u'2015, Netflix Inc.'
+copyright = u'2018, Netflix Inc.'
 
 # The version info for the project you're documenting, acts as replacement for
 # |version| and |release|, also used in various other places throughout the

docs/developer/index.rst

@@ -48,7 +48,7 @@ of Lemur. You'll want to make sure you have a few things on your local system fi
 * pip
 * virtualenv (ideally virtualenvwrapper)
 * node.js (for npm and building css/javascript)
-* (Optional) PostgreSQL
++* `PostgreSQL <https://lemur.readthedocs.io/en/latest/quickstart/index.html#setup-postgres>`_
 
 Once you've got all that, the rest is simple:
 
@@ -77,6 +77,7 @@ Create a default Lemur configuration just as if this were a production instance:
 
 ::
 
+    lemur create_config
     lemur init
 
 You'll likely want to make some changes to the default configuration (we recommend developing against Postgres, for example). Once done, migrate your database using the following command:
@@ -89,6 +90,12 @@ You'll likely want to make some changes to the default configuration (we recomme
 .. note:: The ``upgrade`` shortcut is simply a shortcut to Alembic's upgrade command.
 
 
+Running tests with Docker and docker-compose
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Alternatively you can use Docker and docker-compose for running the tests with ``docker-compose run test``.
+
+
 Coding Standards
 ----------------
 
@@ -277,6 +284,31 @@ Domains
     :undoc-members:
     :show-inheritance:
 
+Endpoints
+---------
+
+.. automodule:: lemur.endpoints.views
+    :members:
+    :undoc-members:
+    :show-inheritance:
+
+Logs
+----
+
+.. automodule:: lemur.logs.views
+    :members:
+    :undoc-members:
+    :show-inheritance:
+
+
+Sources
+-------
+
+.. automodule:: lemur.sources.views
+    :members:
+    :undoc-members:
+    :show-inheritance:
+
 
 Internals
 =========

docs/developer/internals/lemur.certificates.rst

@@ -1,15 +1,6 @@
 certificates Package
 ====================
 
-:mod:`exceptions` Module
-------------------------
-
-.. automodule:: lemur.certificates.exceptions
-    :noindex:
-    :members:
-    :undoc-members:
-    :show-inheritance:
-
 :mod:`models` Module
 --------------------
 

docs/developer/internals/lemur.plugins.lemur_verisign.rst

@@ -10,15 +10,6 @@ lemur_verisign Package
     :undoc-members:
     :show-inheritance:
 
-:mod:`constants` Module
------------------------
-
-.. automodule:: lemur.plugins.lemur_verisign.constants
-    :noindex:
-    :members:
-    :undoc-members:
-    :show-inheritance:
-
 :mod:`plugin` Module
 --------------------
 

docs/developer/internals/lemur.rst

@@ -97,3 +97,18 @@ Subpackages
     lemur.plugins
     lemur.roles
     lemur.users
+    lemur.sources
+    lemur.logs
+    lemur.reporting
+    lemur.tests
+    lemur.deployment
+    lemur.endpoints
+    lemur.defaults
+    lemur.plugins.lemur_acme
+    lemur.plugins.lemur_atlas
+    lemur.plugins.lemur_cryptography
+    lemur.plugins.lemur_digicert
+    lemur.plugins.lemur_java
+    lemur.plugins.lemur_kubernetes
+    lemur.plugins.lemur_openssl
+    lemur.plugins.lemur_slack

docs/developer/plugins/index.rst

@@ -100,10 +100,16 @@ If you have a third party or internal service that creates authorities (EJBCA, e
 it can treat any issuer plugin as both a source of creating new certificates as well as new authorities.
 
 
-The `IssuerPlugin` exposes two functions::
+The `IssuerPlugin` exposes four functions functions::
 
     def create_certificate(self, csr, issuer_options):
         # requests.get('a third party')
+    def revoke_certificate(self, certificate, comments):
+        # requests.put('a third party')
+    def get_ordered_certificate(self, order_id):
+        # requests.get('already existing certificate')
+    def canceled_ordered_certificate(self, pending_cert, **kwargs):
+        # requests.put('cancel an order that has yet to be issued')
 
 Lemur will pass a dictionary of all possible options for certificate creation. Including a valid CSR, and the raw options associated with the request.
 
@@ -139,6 +145,19 @@ The `IssuerPlugin` doesn't have any options like Destination, Source, and Notifi
 any fields you might need to submit a request to a third party. If there are additional options you need
 in your plugin feel free to open an issue, or look into adding additional options to issuers yourself.
 
+Asynchronous Certificates
+^^^^^^^^^^^^^^^^^^^^^^^^^
+An issuer may take some time to actually issue a certificate for an order.  In this case, a `PendingCertificate` is returned, which holds information to recreate a `Certificate` object at a later time.  Then, `get_ordered_certificate()` should be run periodically via `python manage.py pending_certs fetch -i all` to attempt to retrieve an ordered certificate::
+
+    def get_ordered_ceriticate(self, order_id):
+        # order_id is the external id of the order, not the external_id of the certificate
+        # retrieve an order, and check if there is an issued certificate attached to it
+
+`cancel_ordered_certificate()` should be implemented to allow an ordered certificate to be canceled before it is issued::
+    def cancel_ordered_certificate(self, pending_cert, **kwargs):
+        # pending_cert should contain the necessary information to match an order
+        # kwargs can be given to provide information to the issuer for canceling
+
 Destination
 -----------
 

docs/faq.rst

@@ -6,7 +6,7 @@ Common Problems
 
 In my startup logs I see *'Aborting... Lemur cannot locate db encryption key, is LEMUR_ENCRYPTION_KEYS set?'*
   You likely have not correctly configured **LEMUR_ENCRYPTION_KEYS**. See
-  :doc:`administration/index` for more information.
+  :doc:`administration` for more information.
 
 
 I am seeing Lemur's javascript load in my browser but not the CSS.

docs/production/index.rst

@@ -217,6 +217,23 @@ An example apache config::
         # HSTS (mod_headers is required) (15768000 seconds = 6 months)
         Header always set Strict-Transport-Security "max-age=15768000"
         ...
+        
+     # Set the lemur DocumentRoot to static/dist
+     DocumentRoot /www/lemur/lemur/static/dist
+     
+     # Uncomment to force http 1.0 connections to proxy
+     # SetEnv force-proxy-request-1.0 1
+     
+     #Don't keep proxy connections alive
+     SetEnv proxy-nokeepalive 1
+     
+     # Only need to do reverse proxy
+     ProxyRequests Off
+     
+     # Proxy requests to the api to the lemur service (and sanitize redirects from it)
+     ProxyPass "/api" "http://127.0.0.1:8000/api"
+     ProxyPassReverse "/api" "http://127.0.0.1:8000/api"
+     
     </VirtualHost>
 
 Also included in the configurations above are several best practices when it comes to deploying TLS. Things like enabling
@@ -314,6 +331,6 @@ How often you run these commands is largely up to the user. `notify` and `check_
 
 Example cron entries::
 
-    0 22 * * * lemuruser export LEMUR_CONF=/Users/me/.lemur/lemur.conf.py; /www/lemur/bin/lemur notify
-    */15 * * * * lemuruser export LEMUR_CONF=/Users/me/.lemur/lemur.conf.py; /www/lemur/bin/lemur sync -s all
-    0 22 * * * lemuruser export LEMUR_CONF=/Users/me/.lemur/lemur.conf.py; /www/lemur/bin/lemur check_revoked
+    0 22 * * * lemuruser export LEMUR_CONF=/Users/me/.lemur/lemur.conf.py; /www/lemur/bin/lemur notify expirations
+    */15 * * * * lemuruser export LEMUR_CONF=/Users/me/.lemur/lemur.conf.py; /www/lemur/bin/lemur source sync -s all
+    0 22 * * * lemuruser export LEMUR_CONF=/Users/me/.lemur/lemur.conf.py; /www/lemur/bin/lemur certificate check_revoked

docs/quickstart/index.rst

@@ -27,10 +27,13 @@ If installing Lemur on a bare Ubuntu OS you will need to grab the following pack
 .. code-block:: bash
 
     $ sudo apt-get update
-    $ sudo apt-get install nodejs-legacy python-pip python-dev python3-dev libpq-dev build-essential libssl-dev libffi-dev nginx git supervisor npm postgresql
+    $ sudo apt-get install nodejs nodejs-legacy python-pip python-dev python3-dev libpq-dev build-essential libssl-dev libffi-dev libsasl2-dev libldap2-dev nginx git supervisor npm postgresql
 
 .. note:: PostgreSQL is only required if your database is going to be on the same host as the webserver.  npm is needed if you're installing Lemur from the source (e.g., from git).
 
+.. note:: Installing node from a package manager may creat the nodejs bin at  /usr/bin/nodejs instead of /usr/bin/node If that is the case run the following
+    $ sudo ln -s /user/bin/nodejs /usr/bin/node
+
 Now, install Python ``virtualenv`` package:
 
 .. code-block:: bash
@@ -89,7 +92,7 @@ And then run:
 .. note:: This command will install npm dependencies as well as compile static assets.
 
 
-You may also run with the urlContextPath variable set. If this is set it will add the desired context path for subsequent calls back to lemur.
+You may also run with the urlContextPath variable set. If this is set it will add the desired context path for subsequent calls back to lemur. This will only edit the front end code for calls back to the server, you will have to make sure the server knows about these routes.
 ::
 
   Example:
@@ -138,7 +141,7 @@ Before Lemur will run you need to fill in a few required variables in the config
     LEMUR_DEFAULT_COUNTRY
     LEMUR_DEFAULT_STATE
     LEMUR_DEFAULT_LOCATION
-    LEMUR_DEFAUTL_ORGANIZATION
+    LEMUR_DEFAULT_ORGANIZATION
     LEMUR_DEFAULT_ORGANIZATIONAL_UNIT
 
 Setup Postgres
@@ -151,9 +154,8 @@ First, set a password for the postgres user.  For this guide, we will use ``lemu
 .. code-block:: bash
 
     $ sudo -u postgres -i
-    # \password postgres
-    Enter new password: lemur
-    Enter it again: lemur
+    $ psql
+    postgres=# CREATE USER lemur WITH PASSWORD 'lemur';
 
 Once successful, type CTRL-D to exit the Postgres shell.
 
@@ -269,8 +271,8 @@ Configuring Supervisor couldn't be more simple. Just point it to the ``lemur`` e
   autostart=true
   autorestart=true
   redirect_stderr=true
-  stdout_logfile syslog
-  stderr_logfile syslog
+  stdout_logfile=syslog
+  stderr_logfile=syslog
 
 See :ref:`Using Supervisor <UsingSupervisor>` for more details on using Supervisor.
 

docs/requirements.txt

@@ -1,54 +0,0 @@
-alabaster==0.7.8
-alembic==0.8.6
-aniso8601==1.1.0
-arrow==0.7.0
-Babel==2.3.4
-bcrypt==2.0.0
-beautifulsoup4==4.4.1
-blinker==1.4
-boto==2.38.0
-cffi==1.7.0
-cryptography==1.3.2
-docutils==0.12
-enum34==1.1.6
-Flask==0.10.1
-Flask-Bcrypt==0.7.1
-Flask-Mail==0.9.1
-Flask-Migrate==1.7.0
-Flask-Principal==0.4.0
-Flask-RESTful==0.3.3
-Flask-Script==2.0.5
-Flask-SQLAlchemy==2.1
-future==0.15.2
-gunicorn==19.4.1
-idna==2.1
-imagesize==0.7.1
-inflection==0.3.1
-ipaddress==1.0.16
-itsdangerous==0.24
-Jinja2==2.8
-lockfile==0.12.2
-Mako==1.0.4
-MarkupSafe==0.23
-marshmallow==2.4.0
-marshmallow-sqlalchemy==0.8.0
-psycopg2==2.6.1
-pyasn1==0.1.9
-pycparser==2.14
-pycrypto==2.6.1
-Pygments==2.1.3
-PyJWT==1.4.0
-pyOpenSSL==0.15.1
-python-dateutil==2.5.3
-python-editor==1.0.1
-pytz==2016.4
-requests==2.9.1
-six==1.10.0
-snowballstemmer==1.2.1
-Sphinx==1.4.4
-sphinx-rtd-theme==0.1.9
-sphinxcontrib-httpdomain==1.5.0
-SQLAlchemy==1.0.13
-SQLAlchemy-Utils==0.31.4
-Werkzeug==0.11.10
-xmltodict==0.9.2

gulp/build.js

@@ -232,10 +232,16 @@ gulp.task('package:strip', function () {
 
 gulp.task('addUrlContextPath',['addUrlContextPath:revreplace'], function(){
   var urlContextPathExists = argv.urlContextPath ? true : false;
-  return gulp.src('lemur/static/dist/scripts/main*.js')
-    .pipe(gulpif(urlContextPathExists, replace('api/', argv.urlContextPath + '/api/')))
-    .pipe(gulpif(urlContextPathExists, replace('angular/', argv.urlContextPath + '/angular/')))
-    .pipe(gulp.dest('lemur/static/dist/scripts'))
+  ['lemur/static/dist/scripts/main*.js',
+  'lemur/static/dist/angular/**/*.html']
+  .forEach(function(file){
+    return gulp.src(file)
+      .pipe(gulpif(urlContextPathExists, replace('api/', argv.urlContextPath + '/api/')))
+      .pipe(gulpif(urlContextPathExists, replace('angular/', argv.urlContextPath + '/angular/')))
+      .pipe(gulp.dest(function(file){
+        return file.base;
+      }))
+  })
 });
 
 gulp.task('addUrlContextPath:revision', function(){

lemur/__about__.py

@@ -9,10 +9,10 @@ __title__ = "lemur"
 __summary__ = ("Certificate management and orchestration service")
 __uri__ = "https://github.com/Netflix/lemur"
 
-__version__ = "0.5.0"
+__version__ = "0.7.0"
 
 __author__ = "The Lemur developers"
 __email__ = "security@netflix.com"
 
 __license__ = "Apache License, Version 2.0"
-__copyright__ = "Copyright 2016 {0}".format(__author__)
+__copyright__ = "Copyright 2018 {0}".format(__author__)

lemur/__init__.py

@@ -1,14 +1,15 @@
 """
 .. module: lemur
     :platform: Unix
-    :copyright: (c) 2015 by Netflix Inc., see AUTHORS for more
+    :copyright: (c) 2018 by Netflix Inc., see AUTHORS for more
     :license: Apache, see LICENSE for more details.
 
 .. moduleauthor:: Kevin Glisson <kglisson@netflix.com>
 
 
 """
-from __future__ import absolute_import, division, print_function
+import time
+from flask import g, request
 
 from lemur import factory
 from lemur.extensions import metrics
@@ -26,6 +27,9 @@ from lemur.notifications.views import mod as notifications_bp
 from lemur.sources.views import mod as sources_bp
 from lemur.endpoints.views import mod as endpoints_bp
 from lemur.logs.views import mod as logs_bp
+from lemur.api_keys.views import mod as api_key_bp
+from lemur.pending_certificates.views import mod as pending_certificates_bp
+from lemur.dns_providers.views import mod as dns_providers_bp
 
 from lemur.__about__ import (
     __author__, __copyright__, __email__, __license__, __summary__, __title__,
@@ -51,7 +55,10 @@ LEMUR_BLUEPRINTS = (
     notifications_bp,
     sources_bp,
     endpoints_bp,
-    logs_bp
+    logs_bp,
+    api_key_bp,
+    pending_certificates_bp,
+    dns_providers_bp,
 )
 
 
@@ -69,17 +76,6 @@ def configure_hook(app):
     """
     from flask import jsonify
     from werkzeug.exceptions import HTTPException
-    from lemur.decorators import crossdomain
-    if app.config.get('CORS'):
-        @app.after_request
-        @crossdomain(origin=u"http://localhost:3000", methods=['PUT', 'HEAD', 'GET', 'POST', 'OPTIONS', 'DELETE'])
-        def after(response):
-            return response
-
-    @app.after_request
-    def log_status(response):
-        metrics.send('status_code_{}'.format(response.status_code), 'counter', 1)
-        return response
 
     @app.errorhandler(Exception)
     def handle_error(e):
@@ -89,3 +85,29 @@ def configure_hook(app):
 
         app.logger.exception(e)
         return jsonify(error=str(e)), code
+
+    @app.before_request
+    def before_request():
+        g.request_start_time = time.time()
+
+    @app.after_request
+    def after_request(response):
+        # Return early if we don't have the start time
+        if not hasattr(g, 'request_start_time'):
+            return response
+
+        # Get elapsed time in milliseconds
+        elapsed = time.time() - g.request_start_time
+        elapsed = int(round(1000 * elapsed))
+
+        # Collect request/response tags
+        tags = {
+            'endpoint': request.endpoint,
+            'request_method': request.method.lower(),
+            'status_code': response.status_code
+        }
+
+        # Record our response time metric
+        metrics.send('response_time', 'TIMER', elapsed, metric_tags=tags)
+        metrics.send('status_code_{}'.format(response.status_code), 'counter', 1)
+        return response

lemur/api_keys/cli.py

@@ -0,0 +1,41 @@
+"""
+.. module: lemur.api_keys.cli
+    :platform: Unix
+    :copyright: (c) 2018 by Netflix Inc., see AUTHORS for more
+    :license: Apache, see LICENSE for more details.
+.. moduleauthor:: Eric Coan <kungfury@instructure.com>
+"""
+from flask_script import Manager
+from lemur.api_keys import service as api_key_service
+from lemur.auth.service import create_token
+
+from datetime import datetime
+
+manager = Manager(usage="Handles all api key related tasks.")
+
+
+@manager.option('-u', '--user-id', dest='uid', help='The User ID this access key belongs too.')
+@manager.option('-n', '--name', dest='name', help='The name of this API Key.')
+@manager.option('-t', '--ttl', dest='ttl', help='The TTL of this API Key. -1 for forever.')
+def create(uid, name, ttl):
+    """
+    Create a new api key for a user.
+    :return:
+    """
+    print("[+] Creating a new api key.")
+    key = api_key_service.create(user_id=uid, name=name,
+        ttl=ttl, issued_at=int(datetime.utcnow().timestamp()), revoked=False)
+    print("[+] Successfully created a new api key. Generating a JWT...")
+    jwt = create_token(uid, key.id, key.ttl)
+    print("[+] Your JWT is: {jwt}".format(jwt=jwt))
+
+
+@manager.option('-a', '--api-key-id', dest='aid', help='The API Key ID to revoke.')
+def revoke(aid):
+    """
+    Revokes an api key for a user.
+    :return:
+    """
+    print("[-] Revoking the API Key api key.")
+    api_key_service.revoke(aid=aid)
+    print("[+] Successfully revoked the api key")

lemur/api_keys/models.py

@@ -0,0 +1,25 @@
+"""
+.. module: lemur.api_keys.models
+    :platform: Unix
+    :synopsis: This module contains all of the models need to create an api key within Lemur.
+    :copyright: (c) 2018 by Netflix Inc., see AUTHORS for more
+    :license: Apache, see LICENSE for more details.
+.. moduleauthor:: Eric Coan <kungfury@instructure.com>
+"""
+from sqlalchemy import BigInteger, Boolean, Column, ForeignKey, Integer, String
+
+from lemur.database import db
+
+
+class ApiKey(db.Model):
+    __tablename__ = 'api_keys'
+    id = Column(Integer, primary_key=True)
+    name = Column(String)
+    user_id = Column(Integer, ForeignKey('users.id'))
+    ttl = Column(BigInteger)
+    issued_at = Column(BigInteger)
+    revoked = Column(Boolean)
+
+    def __repr__(self):
+        return "ApiKey(name={name}, user_id={user_id}, ttl={ttl}, issued_at={iat}, revoked={revoked})".format(
+            user_id=self.user_id, name=self.name, ttl=self.ttl, iat=self.issued_at, revoked=self.revoked)

lemur/api_keys/schemas.py

@@ -0,0 +1,57 @@
+"""
+.. module: lemur.api_keys.schemas
+    :platform: Unix
+    :copyright: (c) 2018 by Netflix Inc., see AUTHORS for more
+    :license: Apache, see LICENSE for more details.
+.. moduleauthor:: Eric Coan <kungfury@instructure.com>
+"""
+from flask import g
+from marshmallow import fields
+
+from lemur.common.schema import LemurInputSchema, LemurOutputSchema
+from lemur.users.schemas import UserNestedOutputSchema, UserInputSchema
+
+
+def current_user_id():
+    return {'id': g.current_user.id, 'email': g.current_user.email, 'username': g.current_user.username}
+
+
+class ApiKeyInputSchema(LemurInputSchema):
+    name = fields.String(required=False)
+    user = fields.Nested(UserInputSchema, missing=current_user_id, default=current_user_id)
+    ttl = fields.Integer()
+
+
+class ApiKeyRevokeSchema(LemurInputSchema):
+    id = fields.Integer(required=True)
+    name = fields.String()
+    user = fields.Nested(UserInputSchema, required=True)
+    revoked = fields.Boolean()
+    ttl = fields.Integer()
+    issued_at = fields.Integer(required=False)
+
+
+class UserApiKeyInputSchema(LemurInputSchema):
+    name = fields.String(required=False)
+    ttl = fields.Integer()
+
+
+class ApiKeyOutputSchema(LemurOutputSchema):
+    jwt = fields.String()
+
+
+class ApiKeyDescribedOutputSchema(LemurOutputSchema):
+    id = fields.Integer()
+    name = fields.String()
+    user = fields.Nested(UserNestedOutputSchema)
+    ttl = fields.Integer()
+    issued_at = fields.Integer()
+    revoked = fields.Boolean()
+
+
+api_key_input_schema = ApiKeyInputSchema()
+api_key_revoke_schema = ApiKeyRevokeSchema()
+api_key_output_schema = ApiKeyOutputSchema()
+api_keys_output_schema = ApiKeyDescribedOutputSchema(many=True)
+api_key_described_output_schema = ApiKeyDescribedOutputSchema()
+user_api_key_input_schema = UserApiKeyInputSchema()

lemur/api_keys/service.py

@@ -0,0 +1,97 @@
+"""
+.. module: lemur.api_keys.service
+    :platform: Unix
+    :copyright: (c) 2018 by Netflix Inc., see AUTHORS for more
+    :license: Apache, see LICENSE for more details.
+.. moduleauthor:: Eric Coan <kungfury@instructure.com>
+"""
+from lemur import database
+from lemur.api_keys.models import ApiKey
+
+
+def get(aid):
+    """
+    Retrieves an api key by its ID.
+    :param aid: The access key id to get.
+    :return:
+    """
+    return database.get(ApiKey, aid)
+
+
+def delete(access_key):
+    """
+    Delete an access key. This is one way to remove a key, though you probably should just set revoked.
+    :param access_key:
+    :return:
+    """
+    database.delete(access_key)
+
+
+def revoke(aid):
+    """
+    Revokes an api key.
+    :param aid:
+    :return:
+    """
+    api_key = get(aid)
+    setattr(api_key, 'revoked', False)
+
+    return database.update(api_key)
+
+
+def get_all_api_keys():
+    """
+    Retrieves all Api Keys.
+    :return:
+    """
+    return ApiKey.query.all()
+
+
+def create(**kwargs):
+    """
+    Creates a new API Key.
+
+    :param kwargs:
+    :return:
+    """
+    api_key = ApiKey(**kwargs)
+    database.create(api_key)
+    return api_key
+
+
+def update(api_key, **kwargs):
+    """
+    Updates an api key.
+    :param api_key:
+    :param kwargs:
+    :return:
+    """
+    for key, value in kwargs.items():
+        setattr(api_key, key, value)
+
+    return database.update(api_key)
+
+
+def render(args):
+    """
+    Helper to parse REST Api requests
+
+    :param args:
+    :return:
+    """
+    query = database.session_query(ApiKey)
+    user_id = args.pop('user_id', None)
+    aid = args.pop('id', None)
+    has_permission = args.pop('has_permission', False)
+    requesting_user_id = args.pop('requesting_user_id')
+
+    if user_id:
+        query = query.filter(ApiKey.user_id == user_id)
+
+    if aid:
+        query = query.filter(ApiKey.id == aid)
+
+    if not has_permission:
+        query = query.filter(ApiKey.user_id == requesting_user_id)
+
+    return database.sort_and_page(query, ApiKey, args)

lemur/api_keys/views.py

@@ -0,0 +1,579 @@
+"""
+.. module: lemur.api_keys.views
+    :platform: Unix
+    :copyright: (c) 2018 by Netflix Inc., see AUTHORS for more
+    :license: Apache, see LICENSE for more details.
+
+.. moduleauthor:: Eric Coan <kungfury@instructure.com>
+
+"""
+from datetime import datetime
+
+from flask import Blueprint, g
+from flask_restful import reqparse, Api
+
+from lemur.api_keys import service
+from lemur.auth.service import AuthenticatedResource, create_token
+from lemur.auth.permissions import ApiKeyCreatorPermission
+
+from lemur.common.schema import validate_schema
+from lemur.common.utils import paginated_parser
+
+from lemur.api_keys.schemas import api_key_input_schema, api_key_revoke_schema, api_key_output_schema, \
+    api_keys_output_schema, api_key_described_output_schema, user_api_key_input_schema
+
+mod = Blueprint('api_keys', __name__)
+api = Api(mod)
+
+
+class ApiKeyList(AuthenticatedResource):
+    """ Defines the 'api_keys' endpoint """
+
+    def __init__(self):
+        super(ApiKeyList, self).__init__()
+
+    @validate_schema(None, api_keys_output_schema)
+    def get(self):
+        """
+        .. http:get:: /keys
+
+           The current list of api keys, that you can see.
+
+           **Example request**:
+
+           .. sourcecode:: http
+
+              GET /keys HTTP/1.1
+              Host: example.com
+              Accept: application/json, text/javascript
+
+           **Example response**:
+
+           .. sourcecode:: http
+
+              HTTP/1.1 200 OK
+              Vary: Accept
+              Content-Type: text/javascript
+
+              {
+                "items": [
+                    {
+                      "id": 1,
+                      "name": "custom name",
+                      "user_id": 1,
+                      "ttl": -1,
+                      "issued_at": 12,
+                      "revoked": false
+                    }
+                ],
+                "total": 1
+              }
+
+           :query sortBy: field to sort on
+           :query sortDir: asc or desc
+           :query page: int default is 1
+           :query count: count number. default is 10
+           :query user_id: a user to filter by.
+           :query id: an access key to filter by.
+           :reqheader Authorization: OAuth token to authenticate
+           :statuscode 200: no error
+           :statuscode 403: unauthenticated
+        """
+        parser = paginated_parser.copy()
+        args = parser.parse_args()
+        args['has_permission'] = ApiKeyCreatorPermission().can()
+        args['requesting_user_id'] = g.current_user.id
+        return service.render(args)
+
+    @validate_schema(api_key_input_schema, api_key_output_schema)
+    def post(self, data=None):
+        """
+        .. http:post:: /keys
+
+           Creates an API Key.
+
+           **Example request**:
+
+           .. sourcecode:: http
+
+              POST /keys HTTP/1.1
+              Host: example.com
+              Accept: application/json, text/javascript
+
+              {
+                "name": "my custom name",
+                "user_id": 1,
+                "ttl": -1
+              }
+
+           **Example response**:
+
+           .. sourcecode:: http
+
+              HTTP/1.1 200 OK
+              Vary: Accept
+              Content-Type: text/javascript
+
+              {
+
+                "jwt": ""
+              }
+
+           :reqheader Authorization: OAuth token to authenticate
+           :statuscode 200: no error
+           :statuscode 403: unauthenticated
+        """
+        if not ApiKeyCreatorPermission().can():
+            if data['user']['id'] != g.current_user.id:
+                return dict(message="You are not authorized to create tokens for: {0}".format(data['user']['username'])), 403
+
+        access_token = service.create(name=data['name'], user_id=data['user']['id'], ttl=data['ttl'],
+                                      revoked=False, issued_at=int(datetime.utcnow().timestamp()))
+        return dict(jwt=create_token(access_token.user_id, access_token.id, access_token.ttl))
+
+
+class ApiKeyUserList(AuthenticatedResource):
+    """ Defines the 'keys' endpoint on the 'users' endpoint. """
+
+    def __init__(self):
+        super(ApiKeyUserList, self).__init__()
+
+    @validate_schema(None, api_keys_output_schema)
+    def get(self, user_id):
+        """
+        .. http:get:: /users/:user_id/keys
+
+           The current list of api keys for a user, that you can see.
+
+           **Example request**:
+
+           .. sourcecode:: http
+
+              GET /users/1/keys HTTP/1.1
+              Host: example.com
+              Accept: application/json, text/javascript
+
+           **Example response**:
+
+           .. sourcecode:: http
+
+              HTTP/1.1 200 OK
+              Vary: Accept
+              Content-Type: text/javascript
+
+              {
+                "items": [
+                    {
+                      "id": 1,
+                      "name": "custom name",
+                      "user_id": 1,
+                      "ttl": -1,
+                      "issued_at": 12,
+                      "revoked": false
+                    }
+                ],
+                "total": 1
+              }
+
+           :query sortBy: field to sort on
+           :query sortDir: asc or desc
+           :query page: int default is 1
+           :query count: count number. default is 10
+           :query id: an access key to filter by.
+           :reqheader Authorization: OAuth token to authenticate
+           :statuscode 200: no error
+           :statuscode 403: unauthenticated
+        """
+        parser = paginated_parser.copy()
+        args = parser.parse_args()
+        args['has_permission'] = ApiKeyCreatorPermission().can()
+        args['requesting_user_id'] = g.current_user.id
+        args['user_id'] = user_id
+        return service.render(args)
+
+    @validate_schema(user_api_key_input_schema, api_key_output_schema)
+    def post(self, user_id, data=None):
+        """
+        .. http:post:: /users/:user_id/keys
+
+           Creates an API Key for a user.
+
+           **Example request**:
+
+           .. sourcecode:: http
+
+              POST /users/1/keys HTTP/1.1
+              Host: example.com
+              Accept: application/json, text/javascript
+
+              {
+                "name": "my custom name"
+                "ttl": -1
+              }
+
+           **Example response**:
+
+           .. sourcecode:: http
+
+              HTTP/1.1 200 OK
+              Vary: Accept
+              Content-Type: text/javascript
+
+              {
+
+                "jwt": ""
+              }
+
+           :reqheader Authorization: OAuth token to authenticate
+           :statuscode 200: no error
+           :statuscode 403: unauthenticated
+        """
+        if not ApiKeyCreatorPermission().can():
+            if user_id != g.current_user.id:
+                return dict(message="You are not authorized to create tokens for: {0}".format(user_id)), 403
+
+        access_token = service.create(name=data['name'], user_id=user_id, ttl=data['ttl'],
+                                      revoked=False, issued_at=int(datetime.utcnow().timestamp()))
+        return dict(jwt=create_token(access_token.user_id, access_token.id, access_token.ttl))
+
+
+class ApiKeys(AuthenticatedResource):
+    def __init__(self):
+        self.reqparse = reqparse.RequestParser()
+        super(ApiKeys, self).__init__()
+
+    @validate_schema(None, api_key_output_schema)
+    def get(self, aid):
+        """
+        .. http:get:: /keys/1
+
+           Fetch one api key
+
+           **Example request**:
+
+           .. sourcecode:: http
+
+              GET /keys/1 HTTP/1.1
+              Host: example.com
+              Accept: application/json, text/javascript
+
+           **Example response**:
+
+           .. sourcecode:: http
+
+              HTTP/1.1 200 OK
+              Vary: Accept
+              Content-Type: text/javascript
+
+              {
+                  "jwt": ""
+              }
+
+           :reqheader Authorization: OAuth token to authenticate
+           :statuscode 200: no error
+           :statuscode 403: unauthenticated
+        """
+        access_key = service.get(aid)
+
+        if access_key is None:
+            return dict(message="This token does not exist!"), 404
+
+        if access_key.user_id != g.current_user.id:
+            if not ApiKeyCreatorPermission().can():
+                return dict(message="You are not authorized to view this token!"), 403
+
+        return dict(jwt=create_token(access_key.user_id, access_key.id, access_key.ttl))
+
+    @validate_schema(api_key_revoke_schema, api_key_output_schema)
+    def put(self, aid, data=None):
+        """
+        .. http:put:: /keys/1
+
+           update one api key
+
+           **Example request**:
+
+           .. sourcecode:: http
+
+              PUT /keys/1 HTTP/1.1
+              Host: example.com
+              Accept: application/json, text/javascript
+
+              {
+                  "name": "new_name",
+                  "revoked": false,
+                  "ttl": -1
+              }
+
+           **Example response**:
+
+           .. sourcecode:: http
+
+              HTTP/1.1 200 OK
+              Vary: Accept
+              Content-Type: text/javascript
+
+              {
+                  "jwt": ""
+              }
+
+           :reqheader Authorization: OAuth token to authenticate
+           :statuscode 200: no error
+           :statuscode 403: unauthenticated
+        """
+        access_key = service.get(aid)
+        if access_key is None:
+            return dict(message="This token does not exist!"), 404
+
+        if access_key.user_id != g.current_user.id:
+            if not ApiKeyCreatorPermission().can():
+                return dict(message="You are not authorized to update this token!"), 403
+
+        service.update(access_key, name=data['name'], revoked=data['revoked'], ttl=data['ttl'])
+        return dict(jwt=create_token(access_key.user_id, access_key.id, access_key.ttl))
+
+    def delete(self, aid):
+        """
+        .. http:delete:: /keys/1
+
+           deletes one api key
+
+           **Example request**:
+
+           .. sourcecode:: http
+
+              DELETE /keys/1 HTTP/1.1
+              Host: example.com
+              Accept: application/json, text/javascript
+
+           **Example response**:
+
+           .. sourcecode:: http
+
+              HTTP/1.1 200 OK
+              Vary: Accept
+              Content-Type: text/javascript
+
+              {
+                  "result": true
+              }
+
+           :reqheader Authorization: OAuth token to authenticate
+           :statuscode 200: no error
+           :statuscode 403: unauthenticated
+        """
+        access_key = service.get(aid)
+        if access_key is None:
+            return dict(message="This token does not exist!"), 404
+
+        if access_key.user_id != g.current_user.id:
+            if not ApiKeyCreatorPermission().can():
+                return dict(message="You are not authorized to delete this token!"), 403
+
+        service.delete(access_key)
+        return {'result': True}
+
+
+class UserApiKeys(AuthenticatedResource):
+    def __init__(self):
+        self.reqparse = reqparse.RequestParser()
+        super(UserApiKeys, self).__init__()
+
+    @validate_schema(None, api_key_output_schema)
+    def get(self, uid, aid):
+        """
+        .. http:get:: /users/1/keys/1
+
+           Fetch one api key
+
+           **Example request**:
+
+           .. sourcecode:: http
+
+              GET /users/1/api_keys/1 HTTP/1.1
+              Host: example.com
+              Accept: application/json, text/javascript
+
+           **Example response**:
+
+           .. sourcecode:: http
+
+              HTTP/1.1 200 OK
+              Vary: Accept
+              Content-Type: text/javascript
+
+              {
+                  "jwt": ""
+              }
+
+           :reqheader Authorization: OAuth token to authenticate
+           :statuscode 200: no error
+           :statuscode 403: unauthenticated
+        """
+        if uid != g.current_user.id:
+            if not ApiKeyCreatorPermission().can():
+                return dict(message="You are not authorized to view this token!"), 403
+
+        access_key = service.get(aid)
+
+        if access_key is None:
+            return dict(message="This token does not exist!"), 404
+
+        if access_key.user_id != uid:
+            return dict(message="You are not authorized to view this token!"), 403
+
+        return dict(jwt=create_token(access_key.user_id, access_key.id, access_key.ttl))
+
+    @validate_schema(api_key_revoke_schema, api_key_output_schema)
+    def put(self, uid, aid, data=None):
+        """
+        .. http:put:: /users/1/keys/1
+
+           update one api key
+
+           **Example request**:
+
+           .. sourcecode:: http
+
+              PUT /users/1/keys/1 HTTP/1.1
+              Host: example.com
+              Accept: application/json, text/javascript
+
+              {
+                  "name": "new_name",
+                  "revoked": false,
+                  "ttl": -1
+              }
+
+           **Example response**:
+
+           .. sourcecode:: http
+
+              HTTP/1.1 200 OK
+              Vary: Accept
+              Content-Type: text/javascript
+
+              {
+                  "jwt": ""
+              }
+
+           :reqheader Authorization: OAuth token to authenticate
+           :statuscode 200: no error
+           :statuscode 403: unauthenticated
+        """
+        if uid != g.current_user.id:
+            if not ApiKeyCreatorPermission().can():
+                return dict(message="You are not authorized to view this token!"), 403
+
+        access_key = service.get(aid)
+        if access_key is None:
+            return dict(message="This token does not exist!"), 404
+
+        if access_key.user_id != uid:
+            return dict(message="You are not authorized to update this token!"), 403
+
+        service.update(access_key, name=data['name'], revoked=data['revoked'], ttl=data['ttl'])
+        return dict(jwt=create_token(access_key.user_id, access_key.id, access_key.ttl))
+
+    def delete(self, uid, aid):
+        """
+        .. http:delete:: /users/1/keys/1
+
+           deletes one api key
+
+           **Example request**:
+
+           .. sourcecode:: http
+
+              DELETE /users/1/keys/1 HTTP/1.1
+              Host: example.com
+              Accept: application/json, text/javascript
+
+           **Example response**:
+
+           .. sourcecode:: http
+
+              HTTP/1.1 200 OK
+              Vary: Accept
+              Content-Type: text/javascript
+
+              {
+                  "result": true
+              }
+
+           :reqheader Authorization: OAuth token to authenticate
+           :statuscode 200: no error
+           :statuscode 403: unauthenticated
+        """
+        if uid != g.current_user.id:
+            if not ApiKeyCreatorPermission().can():
+                return dict(message="You are not authorized to view this token!"), 403
+
+        access_key = service.get(aid)
+        if access_key is None:
+            return dict(message="This token does not exist!"), 404
+
+        if access_key.user_id != uid:
+            return dict(message="You are not authorized to delete this token!"), 403
+
+        service.delete(access_key)
+        return {'result': True}
+
+
+class ApiKeysDescribed(AuthenticatedResource):
+    def __init__(self):
+        self.reqparse = reqparse.RequestParser()
+        super(ApiKeysDescribed, self).__init__()
+
+    @validate_schema(None, api_key_described_output_schema)
+    def get(self, aid):
+        """
+        .. http:get:: /keys/1/described
+
+           Fetch one api key
+
+           **Example request**:
+
+           .. sourcecode:: http
+
+              GET /keys/1 HTTP/1.1
+              Host: example.com
+              Accept: application/json, text/javascript
+
+           **Example response**:
+
+           .. sourcecode:: http
+
+              HTTP/1.1 200 OK
+              Vary: Accept
+              Content-Type: text/javascript
+
+              {
+                  "id": 2,
+                  "name": "hoi",
+                  "user_id": 2,
+                  "ttl": -1,
+                  "issued_at": 1222222,
+                  "revoked": false
+              }
+
+           :reqheader Authorization: OAuth token to authenticate
+           :statuscode 200: no error
+           :statuscode 403: unauthenticated
+        """
+        access_key = service.get(aid)
+        if access_key is None:
+            return dict(message="This token does not exist!"), 404
+
+        if access_key.user_id != g.current_user.id:
+            if not ApiKeyCreatorPermission().can():
+                return dict(message="You are not authorized to view this token!"), 403
+
+        return access_key
+
+
+api.add_resource(ApiKeyList, '/keys', endpoint='api_keys')
+api.add_resource(ApiKeys, '/keys/<int:aid>', endpoint='api_key')
+api.add_resource(ApiKeysDescribed, '/keys/<int:aid>/described', endpoint='api_key_described')
+api.add_resource(ApiKeyUserList, '/users/<int:user_id>/keys', endpoint='user_api_keys')
+api.add_resource(UserApiKeys, '/users/<int:uid>/keys/<int:aid>', endpoint='user_api_key')

lemur/auth/ldap.py

@@ -0,0 +1,187 @@
+"""
+.. module: lemur.auth.ldap
+    :platform: Unix
+    :copyright: (c) 2018 by Netflix Inc., see AUTHORS for more
+    :license: Apache, see LICENSE for more details.
+.. moduleauthor:: Ian Stahnke <ian.stahnke@myob.com>
+"""
+import ldap
+
+from flask import current_app
+
+from lemur.users import service as user_service
+from lemur.roles import service as role_service
+from lemur.common.utils import validate_conf, get_psuedo_random_string
+
+
+class LdapPrincipal():
+    """
+    Provides methods for authenticating against an LDAP server.
+    """
+    def __init__(self, args):
+        self._ldap_validate_conf()
+        # setup ldap config
+        if not args['username']:
+            raise Exception("missing ldap username")
+        if not args['password']:
+            self.error_message = "missing ldap password"
+            raise Exception("missing ldap password")
+        self.ldap_principal = args['username']
+        self.ldap_email_domain = current_app.config.get("LDAP_EMAIL_DOMAIN", None)
+        if '@' not in self.ldap_principal:
+            self.ldap_principal = '%s@%s' % (self.ldap_principal, self.ldap_email_domain)
+        self.ldap_username = args['username']
+        if '@' in self.ldap_username:
+            self.ldap_username = args['username'].split("@")[0]
+        self.ldap_password = args['password']
+        self.ldap_server = current_app.config.get('LDAP_BIND_URI', None)
+        self.ldap_base_dn = current_app.config.get("LDAP_BASE_DN", None)
+        self.ldap_use_tls = current_app.config.get("LDAP_USE_TLS", False)
+        self.ldap_cacert_file = current_app.config.get("LDAP_CACERT_FILE", None)
+        self.ldap_default_role = current_app.config.get("LEMUR_DEFAULT_ROLE", None)
+        self.ldap_required_group = current_app.config.get("LDAP_REQUIRED_GROUP", None)
+        self.ldap_groups_to_roles = current_app.config.get("LDAP_GROUPS_TO_ROLES", None)
+        self.ldap_attrs = ['memberOf']
+        self.ldap_client = None
+        self.ldap_groups = None
+
+    def _update_user(self, roles):
+        """
+        create or update a local user instance.
+        """
+        # try to get user from local database
+        user = user_service.get_by_email(self.ldap_principal)
+
+        # create them a local account
+        if not user:
+            user = user_service.create(
+                self.ldap_username,
+                get_psuedo_random_string(),
+                self.ldap_principal,
+                True,
+                '',     # thumbnailPhotoUrl
+                list(roles)
+            )
+        else:
+            # we add 'lemur' specific roles, so they do not get marked as removed
+            for ur in user.roles:
+                if not ur.third_party:
+                    roles.add(ur)
+
+            # update any changes to the user
+            user_service.update(
+                user.id,
+                self.ldap_username,
+                self.ldap_principal,
+                user.active,
+                user.profile_picture,
+                list(roles)
+            )
+        return user
+
+    def _authorize(self):
+        """
+        check groups and roles to confirm access.
+        return a list of roles if ok.
+        raise an exception on error.
+        """
+        if not self.ldap_principal:
+            return None
+
+        if self.ldap_required_group:
+            # ensure the user has the required group in their group list
+            if self.ldap_required_group not in self.ldap_groups:
+                return None
+
+        roles = set()
+        if self.ldap_default_role:
+            role = role_service.get_by_name(self.ldap_default_role)
+            if role:
+                if not role.third_party:
+                    role = role.set_third_party(role.id, third_party_status=True)
+                roles.add(role)
+
+        # update their 'roles'
+        role = role_service.get_by_name(self.ldap_principal)
+        if not role:
+            description = "auto generated role based on owner: {0}".format(self.ldap_principal)
+            role = role_service.create(self.ldap_principal, description=description,
+                third_party=True)
+        if not role.third_party:
+            role = role_service.set_third_party(role.id, third_party_status=True)
+        roles.add(role)
+        if not self.ldap_groups_to_roles:
+            return roles
+
+        for ldap_group_name, role_name in self.ldap_groups_to_roles.items():
+            role = role_service.get_by_name(role_name)
+            if role:
+                if ldap_group_name in self.ldap_groups:
+                    current_app.logger.debug("assigning role {0} to ldap user {1}".format(self.ldap_principal, role))
+                    if not role.third_party:
+                        role = role_service.set_third_party(role.id, third_party_status=True)
+                    roles.add(role)
+        return roles
+
+    def authenticate(self):
+        """
+        orchestrate the ldap login.
+        raise an exception on error.
+        """
+        self._bind()
+        roles = self._authorize()
+        if not roles:
+            raise Exception('ldap authorization failed')
+        return self._update_user(roles)
+
+    def _bind(self):
+        """
+        authenticate an ldap user.
+        list groups for a user.
+        raise an exception on error.
+        """
+        if '@' not in self.ldap_principal:
+            self.ldap_principal = '%s@%s' % (self.ldap_principal, self.ldap_email_domain)
+        ldap_filter = 'userPrincipalName=%s' % self.ldap_principal
+
+        # query ldap for auth
+        try:
+            # build a client
+            if not self.ldap_client:
+                self.ldap_client = ldap.initialize(self.ldap_server)
+            # perform a synchronous bind
+            self.ldap_client.set_option(ldap.OPT_REFERRALS, 0)
+            if self.ldap_use_tls:
+                ldap.set_option(ldap.OPT_X_TLS_REQUIRE_CERT, ldap.OPT_X_TLS_NEVER)
+                self.ldap_client.set_option(ldap.OPT_PROTOCOL_VERSION, 3)
+                self.ldap_client.set_option(ldap.OPT_X_TLS, ldap.OPT_X_TLS_DEMAND)
+                self.ldap_client.set_option(ldap.OPT_X_TLS_DEMAND, True)
+                self.ldap_client.set_option(ldap.OPT_DEBUG_LEVEL, 255)
+            if self.ldap_cacert_file:
+                self.ldap_client.set_option(ldap.OPT_X_TLS_CACERTFILE, self.ldap_cacert_file)
+            self.ldap_client.simple_bind_s(self.ldap_principal, self.ldap_password)
+        except ldap.INVALID_CREDENTIALS:
+            self.ldap_client.unbind()
+            raise Exception('The supplied ldap credentials are invalid')
+        except ldap.SERVER_DOWN:
+            raise Exception('ldap server unavailable')
+        except ldap.LDAPError as e:
+            raise Exception("ldap error: {0}".format(e))
+
+        lgroups = self.ldap_client.search_s(self.ldap_base_dn,
+                        ldap.SCOPE_SUBTREE, ldap_filter, self.ldap_attrs)[0][1]['memberOf']
+        # lgroups is a list of utf-8 encoded strings
+        # convert to a single string of groups to allow matching
+        self.ldap_groups = b''.join(lgroups).decode('ascii')
+        self.ldap_client.unbind()
+
+    def _ldap_validate_conf(self):
+        """
+        Confirms required ldap config settings exist.
+        """
+        required_vars = [
+            'LDAP_BIND_URI',
+            'LDAP_BASE_DN',
+            'LDAP_EMAIL_DOMAIN',
+        ]
+        validate_conf(current_app, required_vars)

lemur/auth/permissions.py

@@ -2,7 +2,7 @@
 .. module: lemur.auth.permissions
     :platform: Unix
     :synopsis: This module defines all the permission used within Lemur
-    :copyright: (c) 2015 by Netflix Inc., see AUTHORS for more
+    :copyright: (c) 2018 by Netflix Inc., see AUTHORS for more
     :license: Apache, see LICENSE for more details.
 .. moduleauthor:: Kevin Glisson <kglisson@netflix.com>
 """
@@ -33,6 +33,11 @@ class CertificatePermission(Permission):
         super(CertificatePermission, self).__init__(*needs)
 
 
+class ApiKeyCreatorPermission(Permission):
+    def __init__(self):
+        super(ApiKeyCreatorPermission, self).__init__(RoleNeed('admin'))
+
+
 RoleMember = namedtuple('role', ['method', 'value'])
 RoleMemberNeed = partial(RoleMember, 'member')
 

lemur/auth/service.py

@@ -3,7 +3,7 @@
     :platform: Unix
     :synopsis: This module contains all of the authentication duties for
     lemur
-    :copyright: (c) 2015 by Netflix Inc., see AUTHORS for more
+    :copyright: (c) 2018 by Netflix Inc., see AUTHORS for more
     :license: Apache, see LICENSE for more details.
 .. moduleauthor:: Kevin Glisson <kglisson@netflix.com>
 
@@ -27,6 +27,7 @@ from cryptography.hazmat.primitives import serialization
 from cryptography.hazmat.primitives.asymmetric.rsa import RSAPublicNumbers
 
 from lemur.users import service as user_service
+from lemur.api_keys import service as api_key_service
 from lemur.auth.permissions import AuthorityCreatorNeed, RoleMemberNeed
 
 
@@ -48,9 +49,9 @@ def get_rsa_public_key(n, e):
     )
 
 
-def create_token(user):
+def create_token(user, aid=None, ttl=None):
     """
-    Create a valid JWT for a given user, this token is then used to authenticate
+    Create a valid JWT for a given user/api key, this token is then used to authenticate
     sessions until the token expires.
 
     :param user:
@@ -58,10 +59,24 @@ def create_token(user):
     """
     expiration_delta = timedelta(days=int(current_app.config.get('LEMUR_TOKEN_EXPIRATION', 1)))
     payload = {
-        'sub': user.id,
         'iat': datetime.utcnow(),
         'exp': datetime.utcnow() + expiration_delta
     }
+
+    # Handle Just a User ID & User Object.
+    if isinstance(user, int):
+        payload['sub'] = user
+    else:
+        payload['sub'] = user.id
+    if aid is not None:
+        payload['aid'] = aid
+    # Custom TTLs are only supported on Access Keys.
+    if ttl is not None and aid is not None:
+        # Tokens that are forever until revoked.
+        if ttl == -1:
+            del payload['exp']
+        else:
+            payload['exp'] = ttl
     token = jwt.encode(payload, current_app.config['LEMUR_TOKEN_SECRET'])
     return token.decode('unicode_escape')
 
@@ -94,6 +109,16 @@ def login_required(f):
         except jwt.InvalidTokenError:
             return dict(message='Token is invalid'), 403
 
+        if 'aid' in payload:
+            access_key = api_key_service.get(payload['aid'])
+            if access_key.revoked:
+                return dict(message='Token has been revoked'), 403
+            if access_key.ttl != -1:
+                current_time = datetime.utcnow()
+                expired_time = datetime.fromtimestamp(access_key.issued_at + access_key.ttl)
+                if current_time >= expired_time:
+                    return dict(message='Token has expired'), 403
+
         user = user_service.get(payload['sub'])
 
         if not user.active:

lemur/auth/views.py

@@ -1,13 +1,12 @@
 """
 .. module: lemur.auth.views
     :platform: Unix
-    :copyright: (c) 2015 by Netflix Inc., see AUTHORS for more
+    :copyright: (c) 2018 by Netflix Inc., see AUTHORS for more
     :license: Apache, see LICENSE for more details.
 .. moduleauthor:: Kevin Glisson <kglisson@netflix.com>
 """
 import jwt
 import base64
-import sys
 import requests
 
 from flask import Blueprint, current_app
@@ -15,18 +14,186 @@ from flask import Blueprint, current_app
 from flask_restful import reqparse, Resource, Api
 from flask_principal import Identity, identity_changed
 
+from lemur.constants import SUCCESS_METRIC_STATUS, FAILURE_METRIC_STATUS
 from lemur.extensions import metrics
 from lemur.common.utils import get_psuedo_random_string
 
 from lemur.users import service as user_service
 from lemur.roles import service as role_service
 from lemur.auth.service import create_token, fetch_token_header, get_rsa_public_key
+import lemur.auth.ldap as ldap
 
 
 mod = Blueprint('auth', __name__)
 api = Api(mod)
 
 
+def exchange_for_access_token(code, redirect_uri, client_id, secret, access_token_url=None, verify_cert=True):
+    """
+    Exchanges authorization code for access token.
+
+    :param code:
+    :param redirect_uri:
+    :param client_id:
+    :param secret:
+    :param access_token_url:
+    :param verify_cert:
+    :return:
+    :return:
+    """
+    # take the information we have received from the provider to create a new request
+    params = {
+        'grant_type': 'authorization_code',
+        'scope': 'openid email profile address',
+        'code': code,
+        'redirect_uri': redirect_uri,
+        'client_id': client_id
+    }
+
+    # the secret and cliendId will be given to you when you signup for the provider
+    token = '{0}:{1}'.format(client_id, secret)
+
+    basic = base64.b64encode(bytes(token, 'utf-8'))
+    headers = {
+        'Content-Type': 'application/x-www-form-urlencoded',
+        'authorization': 'basic {0}'.format(basic.decode('utf-8'))
+    }
+
+    # exchange authorization code for access token.
+    r = requests.post(access_token_url, headers=headers, params=params, verify=verify_cert)
+    if r.status_code == 400:
+        r = requests.post(access_token_url, headers=headers, data=params, verify=verify_cert)
+    id_token = r.json()['id_token']
+    access_token = r.json()['access_token']
+
+    return id_token, access_token
+
+
+def validate_id_token(id_token, client_id, jwks_url):
+    """
+    Ensures that the token we receive is valid.
+
+    :param id_token:
+    :param client_id:
+    :param jwks_url:
+    :return:
+    """
+    # fetch token public key
+    header_data = fetch_token_header(id_token)
+
+    # retrieve the key material as specified by the token header
+    r = requests.get(jwks_url)
+    for key in r.json()['keys']:
+        if key['kid'] == header_data['kid']:
+            secret = get_rsa_public_key(key['n'], key['e'])
+            algo = header_data['alg']
+            break
+    else:
+        return dict(message='Key not found'), 401
+
+    # validate your token based on the key it was signed with
+    try:
+        jwt.decode(id_token, secret.decode('utf-8'), algorithms=[algo], audience=client_id)
+    except jwt.DecodeError:
+        return dict(message='Token is invalid'), 401
+    except jwt.ExpiredSignatureError:
+        return dict(message='Token has expired'), 401
+    except jwt.InvalidTokenError:
+        return dict(message='Token is invalid'), 401
+
+
+def retrieve_user(user_api_url, access_token):
+    """
+    Fetch user information from provided user api_url.
+
+    :param user_api_url:
+    :param access_token:
+    :return:
+    """
+    user_params = dict(access_token=access_token, schema='profile')
+
+    # retrieve information about the current user.
+    r = requests.get(user_api_url, params=user_params)
+    profile = r.json()
+
+    user = user_service.get_by_email(profile['email'])
+    return user, profile
+
+
+def create_user_roles(profile):
+    """Creates new roles based on profile information.
+
+    :param profile:
+    :return:
+    """
+    roles = []
+
+    # update their google 'roles'
+    for group in profile['googleGroups']:
+        role = role_service.get_by_name(group)
+        if not role:
+            role = role_service.create(group, description='This is a google group based role created by Lemur', third_party=True)
+        if not role.third_party:
+            role = role_service.set_third_party(role.id, third_party_status=True)
+        roles.append(role)
+
+    role = role_service.get_by_name(profile['email'])
+
+    if not role:
+        role = role_service.create(profile['email'], description='This is a user specific role', third_party=True)
+    if not role.third_party:
+        role = role_service.set_third_party(role.id, third_party_status=True)
+
+    roles.append(role)
+
+    # every user is an operator (tied to a default role)
+    if current_app.config.get('LEMUR_DEFAULT_ROLE'):
+        default = role_service.get_by_name(current_app.config['LEMUR_DEFAULT_ROLE'])
+        if not default:
+            default = role_service.create(current_app.config['LEMUR_DEFAULT_ROLE'], description='This is the default Lemur role.')
+        if not default.third_party:
+            role_service.set_third_party(default.id, third_party_status=True)
+        roles.append(default)
+
+    return roles
+
+
+def update_user(user, profile, roles):
+    """Updates user with current profile information and associated roles.
+
+    :param user:
+    :param profile:
+    :param roles:
+    """
+
+    # if we get an sso user create them an account
+    if not user:
+        user = user_service.create(
+            profile['email'],
+            get_psuedo_random_string(),
+            profile['email'],
+            True,
+            profile.get('thumbnailPhotoUrl'),
+            roles
+        )
+
+    else:
+        # we add 'lemur' specific roles, so they do not get marked as removed
+        for ur in user.roles:
+            if not ur.third_party:
+                roles.append(ur)
+
+        # update any changes to the user
+        user_service.update(
+            user.id,
+            profile['email'],
+            profile['email'],
+            True,
+            profile.get('thumbnailPhotoUrl'),  # profile isn't google+ enabled
+            roles
+        )
+
+
 class Login(Resource):
     """
     Provides an endpoint for Lemur's basic authentication. It takes a username and password
@@ -94,16 +261,35 @@ class Login(Resource):
         else:
             user = user_service.get_by_username(args['username'])
 
+        # default to local authentication
         if user and user.check_password(args['password']) and user.active:
             # Tell Flask-Principal the identity changed
             identity_changed.send(current_app._get_current_object(),
                                   identity=Identity(user.id))
 
-            metrics.send('successful_login', 'counter', 1)
+            metrics.send('login', 'counter', 1, metric_tags={'status': SUCCESS_METRIC_STATUS})
             return dict(token=create_token(user))
 
-        metrics.send('invalid_login', 'counter', 1)
-        return dict(message='The supplied credentials are invalid'), 401
+        # try ldap login
+        if current_app.config.get("LDAP_AUTH"):
+            try:
+                ldap_principal = ldap.LdapPrincipal(args)
+                user = ldap_principal.authenticate()
+                if user and user.active:
+                    # Tell Flask-Principal the identity changed
+                    identity_changed.send(current_app._get_current_object(),
+                                  identity=Identity(user.id))
+                    metrics.send('login', 'counter', 1, metric_tags={'status': SUCCESS_METRIC_STATUS})
+                    return dict(token=create_token(user))
+            except Exception as e:
+                    current_app.logger.error("ldap error: {0}".format(e))
+                    ldap_message = 'ldap error: %s' % e
+                    metrics.send('login', 'counter', 1, metric_tags={'status': FAILURE_METRIC_STATUS})
+                    return dict(message=ldap_message), 403
+
+        # if not valid user - no certificates for you
+        metrics.send('login', 'counter', 1, metric_tags={'status': FAILURE_METRIC_STATUS})
+        return dict(message='The supplied credentials are invalid'), 403
 
 
 class Ping(Resource):
@@ -112,14 +298,17 @@ class Ping(Resource):
     this example we use an OpenIDConnect authentication flow, that is essentially OAuth2 underneath. If you have an
     OAuth2 provider you want to use Lemur there would be two steps:
 
-    1. Define your own class that inherits from :class:`flask.ext.restful.Resource` and create the HTTP methods the \
-    provider uses for it's callbacks.
+    1. Define your own class that inherits from :class:`flask_restful.Resource` and create the HTTP methods the \
+    provider uses for its callbacks.
     2. Add or change the Lemur AngularJS Configuration to point to your new provider
     """
     def __init__(self):
         self.reqparse = reqparse.RequestParser()
         super(Ping, self).__init__()
 
+    def get(self):
+        return 'Redirecting...'
+
     def post(self):
         self.reqparse.add_argument('clientId', type=str, required=True, location='json')
         self.reqparse.add_argument('redirectUri', type=str, required=True, location='json')
@@ -127,120 +316,35 @@ class Ping(Resource):
 
         args = self.reqparse.parse_args()
 
-        # take the information we have received from the provider to create a new request
-        params = {
-            'client_id': args['clientId'],
-            'grant_type': 'authorization_code',
-            'scope': 'openid email profile address',
-            'redirect_uri': args['redirectUri'],
-            'code': args['code']
-        }
-
         # you can either discover these dynamically or simply configure them
         access_token_url = current_app.config.get('PING_ACCESS_TOKEN_URL')
         user_api_url = current_app.config.get('PING_USER_API_URL')
 
-        # the secret and cliendId will be given to you when you signup for the provider
-        token = '{0}:{1}'.format(args['clientId'], current_app.config.get("PING_SECRET"))
+        secret = current_app.config.get('PING_SECRET')
 
-        basic = base64.b64encode(bytes(token, 'utf-8'))
-        headers = {'authorization': 'basic {0}'.format(basic.decode('utf-8'))}
+        id_token, access_token = exchange_for_access_token(
+            args['code'],
+            args['redirectUri'],
+            args['clientId'],
+            secret,
+            access_token_url=access_token_url
+        )
 
-        # exchange authorization code for access token.
-
-        r = requests.post(access_token_url, headers=headers, params=params)
-        id_token = r.json()['id_token']
-        access_token = r.json()['access_token']
-
-        # fetch token public key
-        header_data = fetch_token_header(id_token)
         jwks_url = current_app.config.get('PING_JWKS_URL')
+        validate_id_token(id_token, args['clientId'], jwks_url)
 
-        # retrieve the key material as specified by the token header
-        r = requests.get(jwks_url)
-        for key in r.json()['keys']:
-            if key['kid'] == header_data['kid']:
-                secret = get_rsa_public_key(key['n'], key['e'])
-                algo = header_data['alg']
-                break
-        else:
-            return dict(message='Key not found'), 403
+        user, profile = retrieve_user(user_api_url, access_token)
+        roles = create_user_roles(profile)
+        update_user(user, profile, roles)
 
-        # validate your token based on the key it was signed with
-        try:
-            jwt.decode(id_token, secret.decode('utf-8'), algorithms=[algo], audience=args['clientId'])
-        except jwt.DecodeError:
-            return dict(message='Token is invalid'), 403
-        except jwt.ExpiredSignatureError:
-            return dict(message='Token has expired'), 403
-        except jwt.InvalidTokenError:
-            return dict(message='Token is invalid'), 403
-
-        user_params = dict(access_token=access_token, schema='profile')
-
-        # retrieve information about the current user.
-        r = requests.get(user_api_url, params=user_params)
-        profile = r.json()
-
-        user = user_service.get_by_email(profile['email'])
-        metrics.send('successful_login', 'counter', 1)
-
-        # update their google 'roles'
-        roles = []
-
-        for group in profile['googleGroups']:
-            role = role_service.get_by_name(group)
-            if not role:
-                role = role_service.create(group, description='This is a google group based role created by Lemur')
-            roles.append(role)
-
-        role = role_service.get_by_name(profile['email'])
-
-        if not role:
-            role = role_service.create(profile['email'], description='This is a user specific role')
-        roles.append(role)
-
-        # if we get an sso user create them an account
-        if not user:
-            # every user is an operator (tied to a default role)
-            if current_app.config.get('LEMUR_DEFAULT_ROLE'):
-                v = role_service.get_by_name(current_app.config.get('LEMUR_DEFAULT_ROLE'))
-                if v:
-                    roles.append(v)
-
-            user = user_service.create(
-                profile['email'],
-                get_psuedo_random_string(),
-                profile['email'],
-                True,
-                profile.get('thumbnailPhotoUrl'),
-                roles
-            )
-
-        else:
-            # we add 'lemur' specific roles, so they do not get marked as removed
-            for ur in user.roles:
-                if ur.authority_id:
-                    roles.append(ur)
-
-            # update any changes to the user
-            user_service.update(
-                user.id,
-                profile['email'],
-                profile['email'],
-                True,
-                profile.get('thumbnailPhotoUrl'),  # incase profile isn't google+ enabled
-                roles
-            )
-
-        if not user.active:
-            metrics.send('invalid_login', 'counter', 1)
+        if not user or not user.active:
+            metrics.send('login', 'counter', 1, metric_tags={'status': FAILURE_METRIC_STATUS})
             return dict(message='The supplied credentials are invalid'), 403
 
         # Tell Flask-Principal the identity changed
         identity_changed.send(current_app._get_current_object(), identity=Identity(user.id))
 
-        metrics.send('successful_login', 'counter', 1)
+        metrics.send('login', 'counter', 1, metric_tags={'status': SUCCESS_METRIC_STATUS})
         return dict(token=create_token(user))
 
 
@@ -249,6 +353,9 @@ class OAuth2(Resource):
         self.reqparse = reqparse.RequestParser()
         super(OAuth2, self).__init__()
 
+    def get(self):
+        return 'Redirecting...'
+
     def post(self):
         self.reqparse.add_argument('clientId', type=str, required=True, location='json')
         self.reqparse.add_argument('redirectUri', type=str, required=True, location='json')
@@ -256,113 +363,38 @@ class OAuth2(Resource):
 
         args = self.reqparse.parse_args()
 
-        # take the information we have received from the provider to create a new request
-        params = {
-            'grant_type': 'authorization_code',
-            'scope': 'openid email profile groups',
-            'redirect_uri': args['redirectUri'],
-            'code': args['code'],
-        }
-
         # you can either discover these dynamically or simply configure them
         access_token_url = current_app.config.get('OAUTH2_ACCESS_TOKEN_URL')
         user_api_url = current_app.config.get('OAUTH2_USER_API_URL')
+        verify_cert = current_app.config.get('OAUTH2_VERIFY_CERT')
 
-        # the secret and cliendId will be given to you when you signup for the provider
-        token = '{0}:{1}'.format(args['clientId'], current_app.config.get("OAUTH2_SECRET"))
+        secret = current_app.config.get('OAUTH2_SECRET')
 
-        basic = base64.b64encode(bytes(token, 'utf-8'))
+        id_token, access_token = exchange_for_access_token(
+            args['code'],
+            args['redirectUri'],
+            args['clientId'],
+            secret,
+            access_token_url=access_token_url,
+            verify_cert=verify_cert
+        )
 
-        headers = {
-            'Content-Type': 'application/x-www-form-urlencoded',
-            'authorization': 'basic {0}'.format(basic.decode('utf-8'))
-        }
+        jwks_url = current_app.config.get('PING_JWKS_URL')
+        validate_id_token(id_token, args['clientId'], jwks_url)
 
-        # exchange authorization code for access token.
-        r = requests.post(access_token_url, headers=headers, params=params)
-        id_token = r.json()['id_token']
-        access_token = r.json()['access_token']
+        user, profile = retrieve_user(user_api_url, access_token)
+        roles = create_user_roles(profile)
+        update_user(user, profile, roles)
 
-        # fetch token public key
-        header_data = fetch_token_header(id_token)
-        jwks_url = current_app.config.get('OAUTH2_JWKS_URL')
-
-        # retrieve the key material as specified by the token header
-        r = requests.get(jwks_url)
-        for key in r.json()['keys']:
-            if key['kid'] == header_data['kid']:
-                secret = get_rsa_public_key(key['n'], key['e'])
-                algo = header_data['alg']
-                break
-        else:
-            return dict(message='Key not found'), 403
-
-        # validate your token based on the key it was signed with
-        try:
-            if sys.version_info >= (3, 0):
-                jwt.decode(id_token, secret.decode('utf-8'), algorithms=[algo], audience=args['clientId'])
-            else:
-                jwt.decode(id_token, secret, algorithms=[algo], audience=args['clientId'])
-        except jwt.DecodeError:
-            return dict(message='Token is invalid'), 403
-        except jwt.ExpiredSignatureError:
-            return dict(message='Token has expired'), 403
-        except jwt.InvalidTokenError:
-            return dict(message='Token is invalid'), 403
-
-        headers = {'authorization': 'Bearer {0}'.format(access_token)}
-
-        # retrieve information about the current user.
-        r = requests.get(user_api_url, headers=headers)
-        profile = r.json()
-
-        user = user_service.get_by_email(profile['email'])
-        metrics.send('successful_login', 'counter', 1)
-
-        # update their google 'roles'
-        roles = []
-
-        role = role_service.get_by_name(profile['email'])
-        if not role:
-            role = role_service.create(profile['email'], description='This is a user specific role')
-        roles.append(role)
-
-        # if we get an sso user create them an account
-        if not user:
-            # every user is an operator (tied to a default role)
-            if current_app.config.get('LEMUR_DEFAULT_ROLE'):
-                v = role_service.get_by_name(current_app.config.get('LEMUR_DEFAULT_ROLE'))
-                if v:
-                    roles.append(v)
-
-            user = user_service.create(
-                profile['name'],
-                get_psuedo_random_string(),
-                profile['email'],
-                True,
-                profile.get('thumbnailPhotoUrl'),
-                roles
-            )
-
-        else:
-            # we add 'lemur' specific roles, so they do not get marked as removed
-            for ur in user.roles:
-                if ur.authority_id:
-                    roles.append(ur)
-
-            # update any changes to the user
-            user_service.update(
-                user.id,
-                profile['name'],
-                profile['email'],
-                True,
-                profile.get('thumbnailPhotoUrl'),  # incase profile isn't google+ enabled
-                roles
-            )
+        if not user.active:
+            metrics.send('login', 'counter', 1, metric_tags={'status': FAILURE_METRIC_STATUS})
+            return dict(message='The supplied credentials are invalid'), 403
 
         # Tell Flask-Principal the identity changed
         identity_changed.send(current_app._get_current_object(), identity=Identity(user.id))
 
+        metrics.send('login', 'counter', 1, metric_tags={'status': SUCCESS_METRIC_STATUS})
+
         return dict(token=create_token(user))
 
 
@@ -401,15 +433,15 @@ class Google(Resource):
 
         user = user_service.get_by_email(profile['email'])
 
-        if not user.active:
-            metrics.send('invalid_login', 'counter', 1)
-            return dict(message='The supplied credentials are invalid.'), 401
+        if not (user and user.active):
+            metrics.send('login', 'counter', 1, metric_tags={'status': FAILURE_METRIC_STATUS})
+            return dict(message='The supplied credentials are invalid.'), 403
 
         if user:
-            metrics.send('successful_login', 'counter', 1)
+            metrics.send('login', 'counter', 1, metric_tags={'status': SUCCESS_METRIC_STATUS})
             return dict(token=create_token(user))
 
-        metrics.send('invalid_login', 'counter', 1)
+        metrics.send('login', 'counter', 1, metric_tags={'status': FAILURE_METRIC_STATUS})
 
 
 class Providers(Resource):

lemur/authorities/models.py

@@ -2,7 +2,7 @@
 .. module: lemur.authorities.models
     :platform: unix
     :synopsis: This module contains all of the models need to create an authority within Lemur.
-    :copyright: (c) 2015 by Netflix Inc., see AUTHORS for more
+    :copyright: (c) 2018 by Netflix Inc., see AUTHORS for more
     :license: Apache, see LICENSE for more details.
 .. moduleauthor:: Kevin Glisson <kglisson@netflix.com>
 """
@@ -32,6 +32,9 @@ class Authority(db.Model):
     authority_certificate = relationship("Certificate", backref='root_authority', uselist=False, foreign_keys='Certificate.root_authority_id')
     certificates = relationship("Certificate", backref='authority', foreign_keys='Certificate.authority_id')
 
+    authority_pending_certificate = relationship("PendingCertificate", backref='root_authority', uselist=False, foreign_keys='PendingCertificate.root_authority_id')
+    pending_certificates = relationship('PendingCertificate', backref='authority', foreign_keys='PendingCertificate.authority_id')
+
     def __init__(self, **kwargs):
         self.owner = kwargs['owner']
         self.roles = kwargs.get('roles', [])
@@ -39,6 +42,7 @@ class Authority(db.Model):
         self.description = kwargs.get('description')
         self.authority_certificate = kwargs['authority_certificate']
         self.plugin_name = kwargs['plugin']['slug']
+        self.options = kwargs.get('options')
 
     @property
     def plugin(self):

lemur/authorities/schemas.py

@@ -1,7 +1,7 @@
 """
 .. module: lemur.authorities.schemas
     :platform: unix
-    :copyright: (c) 2015 by Netflix Inc., see AUTHORS for more
+    :copyright: (c) 2018 by Netflix Inc., see AUTHORS for more
     :license: Apache, see LICENSE for more details.
 .. moduleauthor:: Kevin Glisson <kglisson@netflix.com>
 """
@@ -23,7 +23,7 @@ class AuthorityInputSchema(LemurInputSchema):
     name = fields.String(required=True)
     owner = fields.Email(required=True)
     description = fields.String()
-    common_name = fields.String(required=True, validate=validators.sensitive_domain)
+    common_name = fields.String(required=True, validate=validators.common_name)
 
     validity_start = ArrowDateTime()
     validity_end = ArrowDateTime()

lemur/authorities/service.py

@@ -3,12 +3,16 @@
     :platform: Unix
     :synopsis: This module contains all of the services level functions used to
     administer authorities in Lemur
-    :copyright: (c) 2015 by Netflix Inc., see AUTHORS for more
+    :copyright: (c) 2018 by Netflix Inc., see AUTHORS for more
     :license: Apache, see LICENSE for more details.
 .. moduleauthor:: Kevin Glisson <kglisson@netflix.com>
 
 """
+
+import json
+
 from lemur import database
+from lemur.common.utils import truthiness
 from lemur.extensions import metrics
 from lemur.authorities.models import Authority
 from lemur.roles import service as role_service
@@ -16,7 +20,7 @@ from lemur.roles import service as role_service
 from lemur.certificates.service import upload
 
 
-def update(authority_id, description=None, owner=None, active=None, roles=None):
+def update(authority_id, description, owner, active, roles):
     """
     Update an authority with new values.
 
@@ -26,12 +30,11 @@ def update(authority_id, description=None, owner=None, active=None, roles=None):
     """
     authority = get(authority_id)
 
-    if roles:
-        authority.roles = roles
-
+    authority.roles = roles
     authority.active = active
     authority.description = description
     authority.owner = owner
+
     return database.update(authority)
 
 
@@ -107,6 +110,8 @@ def create(**kwargs):
 
     cert = upload(**kwargs)
     kwargs['authority_certificate'] = cert
+    if kwargs.get('plugin', {}).get('plugin_options', []):
+        kwargs['options'] = json.dumps(kwargs['plugin']['plugin_options'])
 
     authority = Authority(**kwargs)
     authority = database.create(authority)
@@ -171,8 +176,8 @@ def render(args):
 
     if filt:
         terms = filt.split(';')
-        if 'active' in filt:  # this is really weird but strcmp seems to not work here??
-            query = query.filter(Authority.active == terms[1])
+        if 'active' in filt:
+            query = query.filter(Authority.active == truthiness(terms[1]))
         else:
             query = database.filter(query, Authority, terms)
 

lemur/authorities/views.py

@@ -1,7 +1,7 @@
 """
 .. module: lemur.authorities.views
     :platform: Unix
-    :copyright: (c) 2015 by Netflix Inc., see AUTHORS for more
+    :copyright: (c) 2018 by Netflix Inc., see AUTHORS for more
     :license: Apache, see LICENSE for more details.
 .. moduleauthor:: Kevin Glisson <kglisson@netflix.com>
 """

lemur/authorizations/models.py

@@ -0,0 +1,34 @@
+"""
+.. module: lemur.authorizations.models
+    :platform: unix
+    :copyright: (c) 2018 by Netflix Inc., see AUTHORS for more
+    :license: Apache, see LICENSE for more details.
+.. moduleauthor:: Netflix Secops <secops@netflix.com>
+"""
+from sqlalchemy import Column, Integer, String
+from sqlalchemy_utils import JSONType
+from lemur.database import db
+
+from lemur.plugins.base import plugins
+
+
+class Authorization(db.Model):
+    __tablename__ = 'pending_dns_authorizations'
+    id = Column(Integer, primary_key=True, autoincrement=True)
+    account_number = Column(String(128))
+    domains = Column(JSONType)
+    dns_provider_type = Column(String(128))
+    options = Column(JSONType)
+
+    @property
+    def plugin(self):
+        return plugins.get(self.plugin_name)
+
+    def __repr__(self):
+        return "Authorization(id={id})".format(label=self.id)
+
+    def __init__(self, account_number, domains, dns_provider_type, options=None):
+        self.account_number = account_number
+        self.domains = domains
+        self.dns_provider_type = dns_provider_type
+        self.options = options

lemur/authorizations/service.py

@@ -0,0 +1,24 @@
+"""
+.. module: lemur.pending_certificates.service
+    Copyright (c) 2018 and onwards Netflix, Inc.  All rights reserved.
+.. moduleauthor:: Secops <secops@netflix.com>
+"""
+from lemur import database
+
+from lemur.authorizations.models import Authorization
+
+
+def get(authorization_id):
+    """
+    Retrieve dns authorization by ID
+    """
+    return database.get(Authorization, authorization_id)
+
+
+def create(account_number, domains, dns_provider_type, options=None):
+    """
+    Creates a new dns authorization.
+    """
+
+    authorization = Authorization(account_number, domains, dns_provider_type, options)
+    return database.create(authorization)

lemur/certificates/cli.py

@@ -1,22 +1,41 @@
 """
 .. module: lemur.certificate.cli
     :platform: Unix
-    :copyright: (c) 2015 by Netflix Inc., see AUTHORS for more
+    :copyright: (c) 2018 by Netflix Inc., see AUTHORS for more
     :license: Apache, see LICENSE for more details.
 .. moduleauthor:: Kevin Glisson <kglisson@netflix.com>
 """
 import sys
+import multiprocessing
+from tabulate import tabulate
+from sqlalchemy import or_
 
 from flask import current_app
 
 from flask_script import Manager
+from flask_principal import Identity, identity_changed
+
 
 from lemur import database
+from lemur.extensions import sentry
 from lemur.extensions import metrics
+from lemur.plugins.base import plugins
+from lemur.constants import SUCCESS_METRIC_STATUS, FAILURE_METRIC_STATUS
 from lemur.deployment import service as deployment_service
 from lemur.endpoints import service as endpoint_service
 from lemur.notifications.messaging import send_rotation_notification
-from lemur.certificates.service import reissue_certificate, get_certificate_primitives, get_all_pending_reissue, get_by_name, get_all_certs
+from lemur.domains.models import Domain
+from lemur.authorities.models import Authority
+from lemur.certificates.schemas import CertificateOutputSchema
+from lemur.certificates.models import Certificate
+from lemur.certificates.service import (
+    reissue_certificate,
+    get_certificate_primitives,
+    get_all_pending_reissue,
+    get_by_name,
+    get_all_certs,
+    get
+)
 
 from lemur.certificates.verify import verify_string
 
@@ -29,28 +48,19 @@ def print_certificate_details(details):
     :param details:
     :return:
     """
+    details, errors = CertificateOutputSchema().dump(details)
     print("[+] Re-issuing certificate with the following details: ")
     print(
         "\t[+] Common Name: {common_name}\n"
         "\t[+] Subject Alternate Names: {sans}\n"
         "\t[+] Authority: {authority_name}\n"
         "\t[+] Validity Start: {validity_start}\n"
-        "\t[+] Validity End: {validity_end}\n"
-        "\t[+] Organization: {organization}\n"
-        "\t[+] Organizational Unit: {organizational_unit}\n"
-        "\t[+] Country: {country}\n"
-        "\t[+] State: {state}\n"
-        "\t[+] Location: {location}".format(
-            common_name=details['common_name'],
-            sans=",".join(x['value'] for x in details['extensions']['sub_alt_names']['names']) or None,
-            authority_name=details['authority'].name,
-            validity_start=details['validity_start'].isoformat(),
-            validity_end=details['validity_end'].isoformat(),
-            organization=details['organization'],
-            organizational_unit=details['organizational_unit'],
-            country=details['country'],
-            state=details['state'],
-            location=details['location']
+        "\t[+] Validity End: {validity_end}\n".format(
+            common_name=details['commonName'],
+            sans=",".join(x['value'] for x in details['extensions']['subAltNames']['names']) or None,
+            authority_name=details['authority']['name'],
+            validity_start=details['validityStart'],
+            validity_end=details['validityEnd']
         )
     )
 
@@ -97,16 +107,17 @@ def request_rotation(endpoint, certificate, message, commit):
     :param commit:
     :return:
     """
+    status = FAILURE_METRIC_STATUS
     if commit:
         try:
             deployment_service.rotate_certificate(endpoint, certificate)
-            metrics.send('endpoint_rotation_success', 'counter', 1)
 
             if message:
                 send_rotation_notification(certificate)
 
+            status = SUCCESS_METRIC_STATUS
+
         except Exception as e:
-            metrics.send('endpoint_rotation_failure', 'counter', 1)
             print(
                 "[!] Failed to rotate endpoint {0} to certificate {1} reason: {2}".format(
                     endpoint.name,
@@ -115,6 +126,8 @@ def request_rotation(endpoint, certificate, message, commit):
                 )
             )
 
+    metrics.send('endpoint_rotation', 'counter', 1, metric_tags={'status': status})
+
 
 def request_reissue(certificate, commit):
     """
@@ -123,22 +136,32 @@ def request_reissue(certificate, commit):
     :param commit:
     :return:
     """
-    details = get_certificate_primitives(certificate)
+    status = FAILURE_METRIC_STATUS
+    try:
+        print("[+] {0} is eligible for re-issuance".format(certificate.name))
 
-    print_certificate_details(details)
-    if commit:
-        try:
+        # set the lemur identity for all cli commands
+        identity_changed.send(current_app._get_current_object(), identity=Identity(1))
+
+        details = get_certificate_primitives(certificate)
+        print_certificate_details(details)
+
+        if commit:
             new_cert = reissue_certificate(certificate, replace=True)
-            metrics.send('certificate_reissue_success', 'counter', 1)
             print("[+] New certificate named: {0}".format(new_cert.name))
-        except Exception as e:
-            metrics.send('certificate_reissue_failure', 'counter', 1)
-            print(
-                "[!] Failed to reissue certificate {1} reason: {2}".format(
-                    certificate.name,
-                    e
-                )
+
+        status = SUCCESS_METRIC_STATUS
+
+    except Exception as e:
+        sentry.captureException()
+        current_app.logger.exception("Error reissuing certificate.", exc_info=True)
+        print(
+            "[!] Failed to reissue certificates. Reason: {}".format(
+                e
             )
+        )
+
+    metrics.send('certificate_reissue', 'counter', 1, metric_tags={'status': status})
 
 
 @manager.option('-e', '--endpoint', dest='endpoint_name', help='Name of the endpoint you wish to rotate.')
@@ -156,34 +179,43 @@ def rotate(endpoint_name, new_certificate_name, old_certificate_name, message, c
 
     print("[+] Starting endpoint rotation.")
 
-    old_cert = validate_certificate(old_certificate_name)
-    new_cert = validate_certificate(new_certificate_name)
-    endpoint = validate_endpoint(endpoint_name)
+    status = FAILURE_METRIC_STATUS
 
-    if endpoint and new_cert:
-        print("[+] Rotating endpoint: {0} to certificate {1}".format(endpoint.name, new_cert.name))
-        request_rotation(endpoint, new_cert, message, commit)
+    try:
+        old_cert = validate_certificate(old_certificate_name)
+        new_cert = validate_certificate(new_certificate_name)
+        endpoint = validate_endpoint(endpoint_name)
 
-    elif old_cert and new_cert:
-        print("[+] Rotating all endpoints from {0} to {1}".format(old_cert.name, new_cert.name))
-
-        for endpoint in old_cert.endpoints:
-            print("[+] Rotating {0}".format(endpoint.name))
+        if endpoint and new_cert:
+            print("[+] Rotating endpoint: {0} to certificate {1}".format(endpoint.name, new_cert.name))
             request_rotation(endpoint, new_cert, message, commit)
 
-    else:
-        print("[+] Rotating all endpoints that have new certificates available")
-        for endpoint in endpoint_service.get_all_pending_rotation():
-            if len(endpoint.certificate.replaced) == 1:
-                print("[+] Rotating {0} to {1}".format(endpoint.name, endpoint.certificate.replaced[0].name))
-                request_rotation(endpoint, endpoint.certificate.replaced[0], message, commit)
-            else:
-                metrics.send('endpoint_rotation_failure', 'counter', 1)
-                print("[!] Failed to rotate endpoint {0} reason: Multiple replacement certificates found.".format(
-                    endpoint.name
-                ))
+        elif old_cert and new_cert:
+            print("[+] Rotating all endpoints from {0} to {1}".format(old_cert.name, new_cert.name))
 
-    print("[+] Done!")
+            for endpoint in old_cert.endpoints:
+                print("[+] Rotating {0}".format(endpoint.name))
+                request_rotation(endpoint, new_cert, message, commit)
+
+        else:
+            print("[+] Rotating all endpoints that have new certificates available")
+            for endpoint in endpoint_service.get_all_pending_rotation():
+                if len(endpoint.certificate.replaced) == 1:
+                    print("[+] Rotating {0} to {1}".format(endpoint.name, endpoint.certificate.replaced[0].name))
+                    request_rotation(endpoint, endpoint.certificate.replaced[0], message, commit)
+                else:
+                    metrics.send('endpoint_rotation', 'counter', 1, metric_tags={'status': FAILURE_METRIC_STATUS})
+                    print("[!] Failed to rotate endpoint {0} reason: Multiple replacement certificates found.".format(
+                        endpoint.name
+                    ))
+
+        status = SUCCESS_METRIC_STATUS
+        print("[+] Done!")
+
+    except Exception as e:
+        sentry.captureException()
+
+    metrics.send('endpoint_rotation_job', 'counter', 1, metric_tags={'status': status})
 
 
 @manager.option('-o', '--old-certificate', dest='old_certificate_name', help='Name of the certificate you wish to reissue.')
@@ -199,16 +231,120 @@ def reissue(old_certificate_name, commit):
 
     print("[+] Starting certificate re-issuance.")
 
-    old_cert = validate_certificate(old_certificate_name)
+    status = FAILURE_METRIC_STATUS
 
-    if not old_cert:
-        for certificate in get_all_pending_reissue():
-            print("[+] {0} is eligible for re-issuance".format(certificate.name))
-            request_reissue(certificate, commit)
-    else:
-        request_reissue(old_cert, commit)
+    try:
+        old_cert = validate_certificate(old_certificate_name)
 
-    print("[+] Done!")
+        if not old_cert:
+            for certificate in get_all_pending_reissue():
+                request_reissue(certificate, commit)
+        else:
+            request_reissue(old_cert, commit)
+
+        status = SUCCESS_METRIC_STATUS
+        print("[+] Done!")
+    except Exception as e:
+        sentry.captureException()
+        current_app.logger.exception("Error reissuing certificate.", exc_info=True)
+        print(
+            "[!] Failed to reissue certificates. Reason: {}".format(
+                e
+            )
+        )
+
+    metrics.send('certificate_reissue_job', 'counter', 1, metric_tags={'status': status})
+
+
+@manager.option('-f', '--fqdns', dest='fqdns', help='FQDNs to query. Multiple fqdns specified via comma.')
+@manager.option('-i', '--issuer', dest='issuer', help='Issuer to query for.')
+@manager.option('-o', '--owner', dest='owner', help='Owner to query for.')
+@manager.option('-e', '--expired', dest='expired', type=bool, default=False, help='Include expired certificates.')
+def query(fqdns, issuer, owner, expired):
+    """Prints certificates that match the query params."""
+    table = []
+
+    q = database.session_query(Certificate)
+
+    sub_query = database.session_query(Authority.id) \
+        .filter(Authority.name.ilike('%{0}%'.format(issuer))) \
+        .subquery()
+
+    q = q.filter(
+        or_(
+            Certificate.issuer.ilike('%{0}%'.format(issuer)),
+            Certificate.authority_id.in_(sub_query)
+        )
+    )
+
+    q = q.filter(Certificate.owner.ilike('%{0}%'.format(owner)))
+
+    if not expired:
+        q = q.filter(Certificate.expired == False)  # noqa
+
+    for f in fqdns.split(','):
+        q = q.filter(
+            or_(
+                Certificate.cn.ilike('%{0}%'.format(f)),
+                Certificate.domains.any(Domain.name.ilike('%{0}%'.format(f)))
+            )
+        )
+
+    for c in q.all():
+        table.append([c.id, c.name, c.owner, c.issuer])
+
+    print(tabulate(table, headers=['Id', 'Name', 'Owner', 'Issuer'], tablefmt='csv'))
+
+
+def worker(data, commit, reason):
+    parts = [x for x in data.split(' ') if x]
+    try:
+        cert = get(int(parts[0].strip()))
+        plugin = plugins.get(cert.authority.plugin_name)
+
+        print('[+] Revoking certificate. Id: {0} Name: {1}'.format(cert.id, cert.name))
+        if commit:
+            plugin.revoke_certificate(cert, reason)
+
+        metrics.send('certificate_revoke', 'counter', 1, metric_tags={'status': SUCCESS_METRIC_STATUS})
+
+    except Exception as e:
+        sentry.captureException()
+        metrics.send('certificate_revoke', 'counter', 1, metric_tags={'status': FAILURE_METRIC_STATUS})
+        print(
+            "[!] Failed to revoke certificates. Reason: {}".format(
+                e
+            )
+        )
+
+
+@manager.command
+def clear_pending():
+    """
+    Function clears all pending certificates.
+    :return:
+    """
+    v = plugins.get('verisign-issuer')
+    v.clear_pending_certificates()
+
+
+@manager.option('-p', '--path', dest='path', help='Absolute file path to a Lemur query csv.')
+@manager.option('-r', '--reason', dest='reason', help='Reason to revoke certificate.')
+@manager.option('-c', '--commit', dest='commit', action='store_true', default=False, help='Persist changes.')
+def revoke(path, reason, commit):
+    """
+    Revokes given certificate.
+    """
+    if commit:
+        print("[!] Running in COMMIT mode.")
+
+    print("[+] Starting certificate revocation.")
+
+    with open(path, 'r') as f:
+        args = [[x, commit, reason] for x in f.readlines()[2:]]
+
+    with multiprocessing.Pool(processes=3) as pool:
+        pool.starmap(worker, args)
 
 
 @manager.command
@@ -227,9 +363,10 @@ def check_revoked():
             else:
                 status = verify_string(cert.body, "")
 
-            cert.status = 'valid' if status else 'invalid'
+            cert.status = 'valid' if status else 'revoked'
 
         except Exception as e:
+            sentry.captureException()
             current_app.logger.exception(e)
             cert.status = 'unknown'
 

lemur/certificates/hooks.py

@@ -0,0 +1,38 @@
+"""
+Debugging hooks for dumping imported or generated CSR and certificate details to stdout via OpenSSL.
+
+.. module: lemur.certificates.hooks
+    :platform: Unix
+    :copyright: (c) 2018 by Marti Raudsepp, see AUTHORS for more
+    :license: Apache, see LICENSE for more details.
+
+.. moduleauthor:: Marti Raudsepp <marti@juffo.org>
+"""
+import subprocess
+
+from flask import current_app
+
+from lemur.certificates.service import csr_created, csr_imported, certificate_issued, certificate_imported
+
+
+def csr_dump_handler(sender, csr, **kwargs):
+    try:
+        subprocess.run(['openssl', 'req', '-text', '-noout', '-reqopt', 'no_sigdump,no_pubkey'],
+                       input=csr.encode('utf8'))
+    except Exception as err:
+        current_app.logger.warning("Error inspecting CSR: %s", err)
+
+
+def cert_dump_handler(sender, certificate, **kwargs):
+    try:
+        subprocess.run(['openssl', 'x509', '-text', '-noout', '-certopt', 'no_sigdump,no_pubkey'],
+                       input=certificate.body.encode('utf8'))
+    except Exception as err:
+        current_app.logger.warning("Error inspecting certificate: %s", err)
+
+
+def activate_debug_dump():
+    csr_created.connect(csr_dump_handler)
+    csr_imported.connect(csr_dump_handler)
+    certificate_issued.connect(cert_dump_handler)
+    certificate_imported.connect(cert_dump_handler)

lemur/certificates/models.py

@@ -1,11 +1,12 @@
 """
 .. module: lemur.certificates.models
     :platform: Unix
-    :copyright: (c) 2015 by Netflix Inc., see AUTHORS for more
+    :copyright: (c) 2018 by Netflix Inc., see AUTHORS for more
     :license: Apache, see LICENSE for more details.
 .. moduleauthor:: Kevin Glisson <kglisson@netflix.com>
 """
 import arrow
+from datetime import timedelta
 
 from flask import current_app
 
@@ -15,7 +16,7 @@ from cryptography.hazmat.primitives.asymmetric import rsa
 from idna.core import InvalidCodepoint
 
 from sqlalchemy.orm import relationship
-from sqlalchemy.sql.expression import case
+from sqlalchemy.sql.expression import case, extract
 from sqlalchemy.ext.hybrid import hybrid_property
 from sqlalchemy import event, Integer, ForeignKey, String, PassiveDefault, func, Column, Text, Boolean
 
@@ -24,6 +25,7 @@ from sqlalchemy_utils.types.arrow import ArrowType
 import lemur.common.utils
 
 from lemur.database import db
+from lemur.extensions import sentry
 
 from lemur.utils import Vault
 from lemur.common import defaults
@@ -31,12 +33,14 @@ from lemur.common import defaults
 from lemur.plugins.base import plugins
 
 from lemur.extensions import metrics
+from lemur.constants import SUCCESS_METRIC_STATUS, FAILURE_METRIC_STATUS
 
 from lemur.models import certificate_associations, certificate_source_associations, \
     certificate_destination_associations, certificate_notification_associations, \
-    certificate_replacement_associations, roles_certificates
+    certificate_replacement_associations, roles_certificates, pending_cert_replacement_associations
 
 from lemur.domains.models import Domain
+from lemur.policies.models import RotationPolicy
 
 
 def get_sequence(name):
@@ -44,29 +48,35 @@ def get_sequence(name):
         return name, None
 
     parts = name.split('-')
-    end = parts.pop(-1)
-    root = '-'.join(parts)
-
-    if len(end) == 8:
-        return root + '-' + end, None
 
+    # see if we have an int at the end of our name
     try:
-        end = int(end)
+        seq = int(parts[-1])
     except ValueError:
-        end = None
+        return name, None
 
-    return root, end
+    # we might have a date at the end of our name
+    if len(parts[-1]) == 8:
+        return name, None
+
+    root = '-'.join(parts[:-1])
+    return root, seq
 
 
-def get_or_increase_name(name):
-    name = '-'.join(name.strip().split(' '))
+def get_or_increase_name(name, serial):
     certificates = Certificate.query.filter(Certificate.name.ilike('{0}%'.format(name))).all()
 
     if not certificates:
         return name
 
+    serial_name = '{0}-{1}'.format(name, hex(int(serial))[2:].upper())
+    certificates = Certificate.query.filter(Certificate.name.ilike('{0}%'.format(serial_name))).all()
+
+    if not certificates:
+        return serial_name
+
     ends = [0]
-    root, end = get_sequence(name)
+    root, end = get_sequence(serial_name)
     for cert in certificates:
         root, end = get_sequence(cert.name)
         if end:
@@ -78,8 +88,9 @@ def get_or_increase_name(name):
 class Certificate(db.Model):
     __tablename__ = 'certificates'
     id = Column(Integer, primary_key=True)
+    external_id = Column(String(128))
     owner = Column(String(128), nullable=False)
-    name = Column(String(128), unique=True)
+    name = Column(String(256), unique=True)
     description = Column(String(1024))
     notify = Column(Boolean, default=True)
 
@@ -91,6 +102,7 @@ class Certificate(db.Model):
     serial = Column(String(128))
     cn = Column(String(128))
     deleted = Column(Boolean, index=True)
+    dns_provider_id = Column(Integer(), ForeignKey('dns_providers.id', ondelete='cascade'), nullable=True)
 
     not_before = Column(ArrowType)
     not_after = Column(ArrowType)
@@ -102,10 +114,10 @@ class Certificate(db.Model):
     san = Column(String(1024))  # TODO this should be migrated to boolean
 
     rotation = Column(Boolean, default=False)
-
     user_id = Column(Integer, ForeignKey('users.id'))
     authority_id = Column(Integer, ForeignKey('authorities.id', ondelete="CASCADE"))
     root_authority_id = Column(Integer, ForeignKey('authorities.id', ondelete="CASCADE"))
+    rotation_policy_id = Column(Integer, ForeignKey('rotation_policies.id'))
 
     notifications = relationship('Notification', secondary=certificate_notification_associations, backref='certificate')
     destinations = relationship('Destination', secondary=certificate_destination_associations, backref='certificate')
@@ -118,8 +130,16 @@ class Certificate(db.Model):
                             secondaryjoin=id == certificate_replacement_associations.c.replaced_certificate_id,  # noqa
                             backref='replaced')
 
+    replaced_by_pending = relationship('PendingCertificate',
+                                       secondary=pending_cert_replacement_associations,
+                                       backref='pending_replace',
+                                       viewonly=True)
+
     logs = relationship('Log', backref='certificate')
     endpoints = relationship('Endpoint', backref='certificate')
+    rotation_policy = relationship("RotationPolicy")
+
+    sensitive_fields = ('private_key',)
 
     def __init__(self, **kwargs):
         cert = lemur.common.utils.parse_certificate(kwargs['body'])
@@ -129,12 +149,14 @@ class Certificate(db.Model):
         self.san = defaults.san(cert)
         self.not_before = defaults.not_before(cert)
         self.not_after = defaults.not_after(cert)
+        self.serial = defaults.serial(cert)
 
         # when destinations are appended they require a valid name.
         if kwargs.get('name'):
-            self.name = get_or_increase_name(kwargs['name'])
+            self.name = get_or_increase_name(defaults.text_to_slug(kwargs['name']), self.serial)
         else:
-            self.name = get_or_increase_name(defaults.certificate_name(self.cn, self.issuer, self.not_before, self.not_after, self.san))
+            self.name = get_or_increase_name(
+                defaults.certificate_name(self.cn, self.issuer, self.not_before, self.not_after, self.san), self.serial)
 
         self.owner = kwargs['owner']
         self.body = kwargs['body'].strip()
@@ -152,9 +174,12 @@ class Certificate(db.Model):
         self.roles = list(set(kwargs.get('roles', [])))
         self.replaces = kwargs.get('replaces', [])
         self.rotation = kwargs.get('rotation')
+        self.rotation_policy = kwargs.get('rotation_policy')
         self.signing_algorithm = defaults.signing_algorithm(cert)
         self.bits = defaults.bitstrength(cert)
-        self.serial = defaults.serial(cert)
+        self.external_id = kwargs.get('external_id')
+        self.authority_id = kwargs.get('authority_id')
+        self.dns_provider_id = kwargs.get('dns_provider_id')
 
         for domain in defaults.domains(cert):
             self.domains.append(Domain(name=domain))
@@ -240,6 +265,33 @@ class Certificate(db.Model):
             else_=False
         )
 
+    @hybrid_property
+    def in_rotation_window(self):
+        """
+        Determines if a certificate is available for rotation based
+        on the rotation policy associated.
+        :return:
+        """
+        now = arrow.utcnow()
+        end = now + timedelta(days=self.rotation_policy.days)
+
+        if self.not_after <= end:
+            return True
+
+    @in_rotation_window.expression
+    def in_rotation_window(cls):
+        """
+        Determines if a certificate is available for rotation based
+        on the rotation policy associated.
+        :return:
+        """
+        return case(
+            [
+                (extract('day', cls.not_after - func.now()) <= RotationPolicy.days, True)
+            ],
+            else_=False
+        )
+
     @property
     def extensions(self):
         # setup default values
@@ -283,28 +335,22 @@ class Certificate(db.Model):
 
                     return_extensions['authority_key_identifier'] = aki
 
-                # TODO: Don't support CRLDistributionPoints yet https://github.com/Netflix/lemur/issues/662
                 elif isinstance(value, x509.CRLDistributionPoints):
-                    current_app.logger.warning('CRLDistributionPoints not yet supported for clone operation.')
+                    return_extensions['crl_distribution_points'] = {'include_crl_dp': value}
 
                 # TODO: Not supporting custom OIDs yet. https://github.com/Netflix/lemur/issues/665
                 else:
                     current_app.logger.warning('Custom OIDs not yet supported for clone operation.')
         except InvalidCodepoint as e:
+            sentry.captureException()
             current_app.logger.warning('Unable to parse extensions due to underscore in dns name')
+        except ValueError as e:
+            sentry.captureException()
+            current_app.logger.warning('Unable to parse')
+            current_app.logger.exception(e)
 
         return return_extensions
 
-    def get_arn(self, account_number):
-        """
-        Generate a valid AWS IAM arn
-
-        :rtype : str
-        :param account_number:
-        :return:
-        """
-        return "arn:aws:iam::{}:server-certificate/{}".format(account_number, self.name)
-
     def __repr__(self):
         return "Certificate(name={name})".format(name=self.name)
 
@@ -320,13 +366,16 @@ def update_destinations(target, value, initiator):
     :return:
     """
     destination_plugin = plugins.get(value.plugin_name)
-
+    status = FAILURE_METRIC_STATUS
     try:
         if target.private_key:
             destination_plugin.upload(target.name, target.body, target.private_key, target.chain, value.options)
+            status = SUCCESS_METRIC_STATUS
     except Exception as e:
-        current_app.logger.exception(e)
-        metrics.send('destination_upload_failure', 'counter', 1, metric_tags={'certificate': target.name, 'destination': value.label})
+        sentry.captureException()
+
+    metrics.send('destination_upload', 'counter', 1,
+                 metric_tags={'status': status, 'certificate': target.name, 'destination': value.label})
 
 
 @event.listens_for(Certificate.replaces, 'append')

lemur/certificates/schemas.py

@@ -1,7 +1,7 @@
 """
 .. module: lemur.certificates.schemas
     :platform: unix
-    :copyright: (c) 2015 by Netflix Inc., see AUTHORS for more
+    :copyright: (c) 2018 by Netflix Inc., see AUTHORS for more
     :license: Apache, see LICENSE for more details.
 .. moduleauthor:: Kevin Glisson <kglisson@netflix.com>
 """
@@ -9,43 +9,55 @@ from flask import current_app
 from marshmallow import fields, validate, validates_schema, post_load, pre_load
 from marshmallow.exceptions import ValidationError
 
-from lemur.schemas import AssociatedAuthoritySchema, AssociatedDestinationSchema, AssociatedCertificateSchema, \
-    AssociatedNotificationSchema, PluginInputSchema, ExtensionSchema, AssociatedRoleSchema, EndpointNestedOutputSchema
-
 from lemur.authorities.schemas import AuthorityNestedOutputSchema
-from lemur.destinations.schemas import DestinationNestedOutputSchema
-from lemur.notifications.schemas import NotificationNestedOutputSchema
-from lemur.roles.schemas import RoleNestedOutputSchema
-from lemur.domains.schemas import DomainNestedOutputSchema
-from lemur.users.schemas import UserNestedOutputSchema
-
-from lemur.common.schema import LemurInputSchema, LemurOutputSchema
 from lemur.common import validators, missing
+from lemur.common.fields import ArrowDateTime, Hex
+from lemur.common.schema import LemurInputSchema, LemurOutputSchema
+from lemur.constants import CERTIFICATE_KEY_TYPES
+from lemur.destinations.schemas import DestinationNestedOutputSchema
+from lemur.domains.schemas import DomainNestedOutputSchema
 from lemur.notifications import service as notification_service
-
-from lemur.common.fields import ArrowDateTime
+from lemur.notifications.schemas import NotificationNestedOutputSchema
+from lemur.policies.schemas import RotationPolicyNestedOutputSchema
+from lemur.roles.schemas import RoleNestedOutputSchema
+from lemur.schemas import (
+    AssociatedAuthoritySchema,
+    AssociatedDestinationSchema,
+    AssociatedCertificateSchema,
+    AssociatedNotificationSchema,
+    AssociatedDnsProviderSchema,
+    PluginInputSchema,
+    ExtensionSchema,
+    AssociatedRoleSchema,
+    EndpointNestedOutputSchema,
+    AssociatedRotationPolicySchema,
+)
+from lemur.users.schemas import UserNestedOutputSchema
 
 
 class CertificateSchema(LemurInputSchema):
     owner = fields.Email(required=True)
-    description = fields.String()
+    description = fields.String(missing='', allow_none=True)
 
 
 class CertificateCreationSchema(CertificateSchema):
     @post_load
     def default_notification(self, data):
         if not data['notifications']:
-            notification_name = "DEFAULT_{0}".format(data['owner'].split('@')[0].upper())
-            data['notifications'] += notification_service.create_default_expiration_notifications(notification_name, [data['owner']])
-
-        notification_name = 'DEFAULT_SECURITY'
-        data['notifications'] += notification_service.create_default_expiration_notifications(notification_name, current_app.config.get('LEMUR_SECURITY_TEAM_EMAIL'))
+            data['notifications'] += notification_service.create_default_expiration_notifications(
+                "DEFAULT_{0}".format(data['owner'].split('@')[0].upper()),
+                [data['owner']],
+            )
+            data['notifications'] += notification_service.create_default_expiration_notifications(
+                'DEFAULT_SECURITY',
+                current_app.config.get('LEMUR_SECURITY_TEAM_EMAIL')
+            )
         return data
 
 
 class CertificateInputSchema(CertificateCreationSchema):
     name = fields.String()
-    common_name = fields.String(required=True, validate=validators.sensitive_domain)
+    common_name = fields.String(required=True, validate=validators.common_name)
     authority = fields.Nested(AssociatedAuthoritySchema, required=True)
 
     validity_start = ArrowDateTime()
@@ -57,12 +69,17 @@ class CertificateInputSchema(CertificateCreationSchema):
     replaces = fields.Nested(AssociatedCertificateSchema, missing=[], many=True)
     replacements = fields.Nested(AssociatedCertificateSchema, missing=[], many=True)  # deprecated
     roles = fields.Nested(AssociatedRoleSchema, missing=[], many=True)
+    dns_provider = fields.Nested(AssociatedDnsProviderSchema, missing=None, allow_none=True, required=False)
 
     csr = fields.String(validate=validators.csr)
-    key_type = fields.String(validate=validate.OneOf(['RSA2048', 'RSA4096']), missing='RSA2048')
+
+    key_type = fields.String(
+        validate=validate.OneOf(CERTIFICATE_KEY_TYPES),
+        missing='RSA2048')
 
     notify = fields.Boolean(default=True)
     rotation = fields.Boolean()
+    rotation_policy = fields.Nested(AssociatedRotationPolicySchema, missing={'name': 'default'}, default={'name': 'default'})
 
     # certificate body fields
     organizational_unit = fields.String(missing=lambda: current_app.config.get('LEMUR_DEFAULT_ORGANIZATIONAL_UNIT'))
@@ -73,6 +90,11 @@ class CertificateInputSchema(CertificateCreationSchema):
 
     extensions = fields.Nested(ExtensionSchema)
 
+    @validates_schema
+    def validate_authority(self, data):
+        if not data['authority'].active:
+            raise ValidationError("The authority is inactive.", ['authority'])
+
     @validates_schema
     def validate_dates(self, data):
         validators.dates(data)
@@ -124,7 +146,7 @@ class CertificateNestedOutputSchema(LemurOutputSchema):
     creator = fields.Nested(UserNestedOutputSchema)
     description = fields.String()
 
-    status = fields.Boolean()
+    status = fields.String()
 
     bits = fields.Integer()
     body = fields.String()
@@ -133,8 +155,9 @@ class CertificateNestedOutputSchema(LemurOutputSchema):
 
     rotation = fields.Boolean()
     notify = fields.Boolean()
+    rotation_policy = fields.Nested(RotationPolicyNestedOutputSchema)
 
-    # Note aliasing  is the first step in deprecating these fields.
+    # Note aliasing is the first step in deprecating these fields.
     cn = fields.String()  # deprecated
     common_name = fields.String(attribute='cn')
 
@@ -155,6 +178,7 @@ class CertificateCloneSchema(LemurOutputSchema):
 
 class CertificateOutputSchema(LemurOutputSchema):
     id = fields.Integer()
+    external_id = fields.String()
     bits = fields.Integer()
     body = fields.String()
     chain = fields.String()
@@ -162,10 +186,11 @@ class CertificateOutputSchema(LemurOutputSchema):
     description = fields.String()
     issuer = fields.String()
     name = fields.String()
+    dns_provider_id = fields.Integer(required=False, allow_none=True)
 
     rotation = fields.Boolean()
 
-    # Note aliasing  is the first step in deprecating these fields.
+    # Note aliasing is the first step in deprecating these fields.
     notify = fields.Boolean()
     active = fields.Boolean(attribute='notify')
 
@@ -181,9 +206,10 @@ class CertificateOutputSchema(LemurOutputSchema):
     owner = fields.Email()
     san = fields.Boolean()
     serial = fields.String()
+    serial_hex = Hex(attribute='serial')
     signing_algorithm = fields.String()
 
-    status = fields.Boolean()
+    status = fields.String()
     user = fields.Nested(UserNestedOutputSchema)
 
     extensions = fields.Nested(ExtensionSchema)
@@ -197,6 +223,7 @@ class CertificateOutputSchema(LemurOutputSchema):
     roles = fields.Nested(RoleNestedOutputSchema, many=True)
     endpoints = fields.Nested(EndpointNestedOutputSchema, many=True, missing=[])
     replaced_by = fields.Nested(CertificateNestedOutputSchema, many=True, attribute='replaced')
+    rotation_policy = fields.Nested(RotationPolicyNestedOutputSchema)
 
 
 class CertificateUploadInputSchema(CertificateCreationSchema):
@@ -234,6 +261,10 @@ class CertificateNotificationOutputSchema(LemurOutputSchema):
     endpoints = fields.Nested(EndpointNestedOutputSchema, many=True, missing=[])
 
 
+class CertificateRevokeSchema(LemurInputSchema):
+    comments = fields.String()
+
+
 certificate_input_schema = CertificateInputSchema()
 certificate_output_schema = CertificateOutputSchema()
 certificates_output_schema = CertificateOutputSchema(many=True)
@@ -241,3 +272,4 @@ certificate_upload_input_schema = CertificateUploadInputSchema()
 certificate_export_input_schema = CertificateExportInputSchema()
 certificate_edit_input_schema = CertificateEditInputSchema()
 certificate_notification_output_schema = CertificateNotificationOutputSchema()
+certificate_revoke_schema = CertificateRevokeSchema()

lemur/certificates/service.py

@@ -1,24 +1,23 @@
 """
 .. module: lemur.certificate.service
     :platform: Unix
-    :copyright: (c) 2015 by Netflix Inc., see AUTHORS for more
+    :copyright: (c) 2018 by Netflix Inc., see AUTHORS for more
     :license: Apache, see LICENSE for more details.
 .. moduleauthor:: Kevin Glisson <kglisson@netflix.com>
 """
 import arrow
-from datetime import timedelta
 
 from flask import current_app
-from sqlalchemy import func, or_, not_, cast, Boolean, Integer
+from sqlalchemy import func, or_, not_, cast, Integer
 
 from cryptography import x509
 from cryptography.hazmat.backends import default_backend
 from cryptography.hazmat.primitives import hashes, serialization
 
 from lemur import database
-from lemur.extensions import metrics
+from lemur.extensions import metrics, signals
 from lemur.plugins.base import plugins
-from lemur.common.utils import generate_private_key
+from lemur.common.utils import generate_private_key, truthiness
 
 from lemur.roles.models import Role
 from lemur.domains.models import Domain
@@ -26,12 +25,19 @@ from lemur.authorities.models import Authority
 from lemur.destinations.models import Destination
 from lemur.certificates.models import Certificate
 from lemur.notifications.models import Notification
+from lemur.pending_certificates.models import PendingCertificate
 
 from lemur.certificates.schemas import CertificateOutputSchema, CertificateInputSchema
 
 from lemur.roles import service as role_service
 
 
+csr_created = signals.signal('csr_created', "CSR generated")
+csr_imported = signals.signal('csr_imported', "CSR imported from external source")
+certificate_issued = signals.signal('certificate_issued', "Authority issued a certificate")
+certificate_imported = signals.signal('certificate_imported', "Certificate imported from external source")
+
+
 def get(cert_id):
     """
     Retrieves certificate by its ID.
@@ -52,6 +58,18 @@ def get_by_name(name):
     return database.get(Certificate, name, field='name')
 
 
+def get_by_serial(serial):
+    """
+    Retrieves certificate by it's Serial.
+    :param serial:
+    :return:
+    """
+    if isinstance(serial, int):
+        # although serial is a number, the DB column is String(128)
+        serial = str(serial)
+    return Certificate.query.filter(Certificate.serial == serial).all()
+
+
 def delete(cert_id):
     """
     Delete's a certificate.
@@ -85,20 +103,15 @@ def get_all_pending_reissue():
     """
     Retrieves all certificates that need to be rotated.
 
-    Must be X days from expiration, uses `LEMUR_DEFAULT_ROTATION_INTERVAL`
-    to determine how many days from expiration the certificate must be
+    Must be X days from expiration, uses the certificates rotation
+    policy to determine how many days from expiration the certificate must be
     for rotation to be pending.
 
     :return:
     """
-    now = arrow.utcnow()
-    interval = current_app.config.get('LEMUR_DEFAULT_ROTATION_INTERVAL', 30)
-    end = now + timedelta(days=interval)
-
     return Certificate.query.filter(Certificate.rotation == True)\
-        .filter(Certificate.endpoints.any())\
         .filter(not_(Certificate.replaced.any()))\
-        .filter(Certificate.not_after <= end.format('YYYY-MM-DD')).all()  # noqa
+        .filter(Certificate.in_rotation_window == True).all()  # noqa
 
 
 def find_duplicates(cert):
@@ -174,12 +187,14 @@ def mint(**kwargs):
     # allow the CSR to be specified by the user
     if not kwargs.get('csr'):
         csr, private_key = create_csr(**kwargs)
+        csr_created.send(authority=authority, csr=csr)
     else:
         csr = str(kwargs.get('csr'))
         private_key = None
+        csr_imported.send(authority=authority, csr=csr)
 
-    cert_body, cert_chain = issuer.create_certificate(csr, kwargs)
-    return cert_body, private_key, cert_chain,
+    cert_body, cert_chain, external_id = issuer.create_certificate(csr, kwargs)
+    return cert_body, private_key, cert_chain, external_id, csr
 
 
 def import_certificate(**kwargs):
@@ -222,17 +237,22 @@ def upload(**kwargs):
     cert = database.create(cert)
 
     kwargs['creator'].certificates.append(cert)
-    return database.update(cert)
+
+    cert = database.update(cert)
+    certificate_imported.send(certificate=cert, authority=cert.authority)
+    return cert
 
 
 def create(**kwargs):
     """
     Creates a new certificate.
     """
-    cert_body, private_key, cert_chain = mint(**kwargs)
+    cert_body, private_key, cert_chain, external_id, csr = mint(**kwargs)
     kwargs['body'] = cert_body
     kwargs['private_key'] = private_key
     kwargs['chain'] = cert_chain
+    kwargs['external_id'] = external_id
+    kwargs['csr'] = csr
 
     roles = create_certificate_roles(**kwargs)
 
@@ -241,13 +261,20 @@ def create(**kwargs):
     else:
         kwargs['roles'] = roles
 
-    cert = Certificate(**kwargs)
+    if cert_body:
+        cert = Certificate(**kwargs)
+        kwargs['creator'].certificates.append(cert)
+    else:
+        cert = PendingCertificate(**kwargs)
+        kwargs['creator'].pending_certificates.append(cert)
 
-    kwargs['creator'].certificates.append(cert)
     cert.authority = kwargs['authority']
+
     database.commit()
 
-    metrics.send('certificate_issued', 'counter', 1, metric_tags=dict(owner=cert.owner, issuer=cert.issuer))
+    if isinstance(cert, Certificate):
+        certificate_issued.send(certificate=cert, authority=cert.authority)
+        metrics.send('certificate_issued', 'counter', 1, metric_tags=dict(owner=cert.owner, issuer=cert.issuer))
     return cert
 
 
@@ -271,36 +298,47 @@ def render(args):
 
     if filt:
         terms = filt.split(';')
+        term = '%{0}%'.format(terms[1])
+        # Exact matches for quotes. Only applies to name, issuer, and cn
+        if terms[1].startswith('"') and terms[1].endswith('"'):
+            term = terms[1][1:-1]
 
         if 'issuer' in terms:
             # we can't rely on issuer being correct in the cert directly so we combine queries
             sub_query = database.session_query(Authority.id)\
-                .filter(Authority.name.ilike('%{0}%'.format(terms[1])))\
+                .filter(Authority.name.ilike(term))\
                 .subquery()
 
             query = query.filter(
                 or_(
-                    Certificate.issuer.ilike('%{0}%'.format(terms[1])),
+                    Certificate.issuer.ilike(term),
                     Certificate.authority_id.in_(sub_query)
                 )
             )
-            return database.sort_and_page(query, Certificate, args)
 
         elif 'destination' in terms:
             query = query.filter(Certificate.destinations.any(Destination.id == terms[1]))
         elif 'notify' in filt:
-            query = query.filter(Certificate.notify == cast(terms[1], Boolean))
+            query = query.filter(Certificate.notify == truthiness(terms[1]))
         elif 'active' in filt:
-            query = query.filter(Certificate.active == terms[1])
+            query = query.filter(Certificate.active == truthiness(terms[1]))
         elif 'cn' in terms:
             query = query.filter(
                 or_(
-                    Certificate.cn.ilike('%{0}%'.format(terms[1])),
-                    Certificate.domains.any(Domain.name.ilike('%{0}%'.format(terms[1])))
+                    Certificate.cn.ilike(term),
+                    Certificate.domains.any(Domain.name.ilike(term))
                 )
             )
         elif 'id' in terms:
             query = query.filter(Certificate.id == cast(terms[1], Integer))
+        elif 'name' in terms:
+            query = query.filter(
+                or_(
+                    Certificate.name.ilike(term),
+                    Certificate.domains.any(Domain.name.ilike(term)),
+                    Certificate.cn.ilike(term),
+                )
+            )
         else:
             query = database.filter(query, Certificate, terms)
 
@@ -337,8 +375,9 @@ def create_csr(**csr_config):
     private_key = generate_private_key(csr_config.get('key_type'))
 
     builder = x509.CertificateSigningRequestBuilder()
-    name_list = [x509.NameAttribute(x509.OID_COMMON_NAME, csr_config['common_name']),
-                 x509.NameAttribute(x509.OID_EMAIL_ADDRESS, csr_config['owner'])]
+    name_list = [x509.NameAttribute(x509.OID_COMMON_NAME, csr_config['common_name'])]
+    if current_app.config.get('LEMUR_OWNER_EMAIL_IN_SUBJECT', True):
+        name_list.append(x509.NameAttribute(x509.OID_EMAIL_ADDRESS, csr_config['owner']))
     if 'organization' in csr_config and csr_config['organization'].strip():
         name_list.append(x509.NameAttribute(x509.OID_ORGANIZATION_NAME, csr_config['organization']))
     if 'organizational_unit' in csr_config and csr_config['organizational_unit'].strip():
@@ -359,7 +398,8 @@ def create_csr(**csr_config):
             if k in critical_extensions:
                 current_app.logger.debug('Adding Critical Extension: {0} {1}'.format(k, v))
                 if k == 'sub_alt_names':
-                    builder = builder.add_extension(v['names'], critical=True)
+                    if v['names']:
+                        builder = builder.add_extension(v['names'], critical=True)
                 else:
                     builder = builder.add_extension(v, critical=True)
 
@@ -475,6 +515,12 @@ def get_certificate_primitives(certificate):
     # we will rely on the Lemur generated name
     data.pop('name', None)
 
+    # TODO this can be removed once we migrate away from cn
+    data['cn'] = data['common_name']
+
+    # needed until we move off not_*
+    data['not_before'] = start
+    data['not_after'] = end
     data['validity_start'] = start
     data['validity_end'] = end
     return data

lemur/certificates/verify.py

@@ -1,13 +1,13 @@
 """
 .. module: lemur.certificates.verify
     :platform: Unix
-    :copyright: (c) 2015 by Netflix Inc., see AUTHORS for more
+    :copyright: (c) 2018 by Netflix Inc., see AUTHORS for more
     :license: Apache, see LICENSE for more details.
 .. moduleauthor:: Kevin Glisson <kglisson@netflix.com>
 """
 import requests
 import subprocess
-from requests.exceptions import ConnectionError
+from requests.exceptions import ConnectionError, InvalidSchema
 from cryptography import x509
 from cryptography.hazmat.backends import default_backend
 
@@ -19,7 +19,7 @@ def ocsp_verify(cert_path, issuer_chain_path):
     """
     Attempts to verify a certificate via OCSP. OCSP is a more modern version
     of CRL in that it will query the OCSP URI in order to determine if the
-    certificate as been revoked
+    certificate has been revoked
 
     :param cert_path:
     :param issuer_chain_path:
@@ -69,6 +69,9 @@ def crl_verify(cert_path):
 
             if response.status_code != 200:
                 raise Exception("Unable to retrieve CRL: {0}".format(point))
+        except InvalidSchema:
+            # Unhandled URI scheme (like ldap://); skip this distribution point.
+            continue
         except ConnectionError:
             raise Exception("Unable to retrieve CRL: {0}".format(point))
 
@@ -76,6 +79,15 @@ def crl_verify(cert_path):
 
         for r in crl:
             if cert.serial == r.serial_number:
+                try:
+                    reason = r.extensions.get_extension_for_class(x509.CRLReason).value
+                    # Handle "removeFromCRL" revoke reason as unrevoked; continue with the next distribution point.
+                    # Per RFC 5280 section 6.3.3 (k): https://tools.ietf.org/html/rfc5280#section-6.3.3
+                    if reason == x509.ReasonFlags.remove_from_crl:
+                        break
+                except x509.ExtensionNotFound:
+                    pass
+
                 return
 
     return True

lemur/certificates/views.py

@@ -1,7 +1,7 @@
 """
 .. module: lemur.certificates.views
     :platform: Unix
-    :copyright: (c) 2015 by Netflix Inc., see AUTHORS for more
+    :copyright: (c) 2018 by Netflix Inc., see AUTHORS for more
     :license: Apache, see LICENSE for more details.
 .. moduleauthor:: Kevin Glisson <kglisson@netflix.com>
 """
@@ -18,8 +18,16 @@ from lemur.auth.service import AuthenticatedResource
 from lemur.auth.permissions import AuthorityPermission, CertificatePermission
 
 from lemur.certificates import service
-from lemur.certificates.schemas import certificate_input_schema, certificate_output_schema, \
-    certificate_upload_input_schema, certificates_output_schema, certificate_export_input_schema, certificate_edit_input_schema
+from lemur.certificates.models import Certificate
+from lemur.plugins.base import plugins
+from lemur.certificates.schemas import (
+    certificate_input_schema,
+    certificate_output_schema,
+    certificate_upload_input_schema,
+    certificates_output_schema,
+    certificate_export_input_schema,
+    certificate_edit_input_schema
+)
 
 from lemur.roles import service as role_service
 from lemur.logs import service as log_service
@@ -84,7 +92,7 @@ class CertificatesList(AuthenticatedResource):
                     "deleted": null,
                     "notifications": [{
                         "id": 1
-                    }]
+                    }],
                     "signingAlgorithm": "sha256",
                     "user": {
                         "username": "jane",
@@ -169,7 +177,7 @@ class CertificatesList(AuthenticatedResource):
                   },
                   "replacements": [{
                     "id": 1
-                  },
+                  }],
                   "notify": true,
                   "validityEnd": "2026-01-01T08:00:00.000Z",
                   "authority": {
@@ -215,7 +223,7 @@ class CertificatesList(AuthenticatedResource):
                 "deleted": null,
                 "notifications": [{
                     "id": 1
-                }]
+                }],
                 "signingAlgorithm": "sha256",
                 "user": {
                     "username": "jane",
@@ -232,6 +240,8 @@ class CertificatesList(AuthenticatedResource):
                 "replaces": [{
                     "id": 1
                 }],
+                "rotation": true,
+                "rotationPolicy": {"name": "default"},
                 "name": "WILDCARD.test.example.net-SymantecCorporation-20160603-20180112",
                 "roles": [{
                     "id": 464,
@@ -241,18 +251,6 @@ class CertificatesList(AuthenticatedResource):
                 "san": null
               }
 
-
-           :arg extensions: extensions to be used in the certificate
-           :arg description: description for new certificate
-           :arg owner: owner email
-           :arg validityStart: when the certificate should start being valid
-           :arg validityEnd: when the certificate should expire
-           :arg authority: authority that should issue the certificate
-           :arg country: country for the CSR
-           :arg state: state for the CSR
-           :arg location: location for the CSR
-           :arg organization: organization for CSR
-           :arg commonName: certificate common name
            :reqheader Authorization: OAuth token to authenticate
            :statuscode 200: no error
            :statuscode 403: unauthenticated
@@ -269,9 +267,13 @@ class CertificatesList(AuthenticatedResource):
 
         if authority_permission.can():
             data['creator'] = g.user
-            return service.create(**data)
+            cert = service.create(**data)
+            if isinstance(cert, Certificate):
+                # only log if created, not pending
+                log_service.create(g.user, 'create_cert', certificate=cert)
+            return cert
 
-        return dict(message="You are not authorized to use {0}".format(data['authority'].name)), 403
+        return dict(message="You are not authorized to use the authority: {0}".format(data['authority'].name)), 403
 
 
 class CertificatesUpload(AuthenticatedResource):
@@ -298,12 +300,14 @@ class CertificatesUpload(AuthenticatedResource):
 
               {
                  "owner": "joe@example.com",
-                 "publicCert": "-----BEGIN CERTIFICATE-----...",
-                 "intermediateCert": "-----BEGIN CERTIFICATE-----...",
+                 "body": "-----BEGIN CERTIFICATE-----...",
+                 "chain": "-----BEGIN CERTIFICATE-----...",
                  "privateKey": "-----BEGIN RSA PRIVATE KEY-----..."
                  "destinations": [],
                  "notifications": [],
                  "replacements": [],
+                 "roles": [],
+                 "notify": true,
                  "name": "cert1"
               }
 
@@ -339,7 +343,7 @@ class CertificatesUpload(AuthenticatedResource):
                 "deleted": null,
                 "notifications": [{
                     "id": 1
-                }]
+                }],
                 "signingAlgorithm": "sha256",
                 "user": {
                     "username": "jane",
@@ -354,6 +358,8 @@ class CertificatesUpload(AuthenticatedResource):
                     "name": "*.test.example.net"
                 }],
                 "replaces": [],
+                "rotation": true,
+                "rotationPolicy": {"name": "default"},
                 "name": "WILDCARD.test.example.net-SymantecCorporation-20160603-20180112",
                 "roles": [{
                     "id": 464,
@@ -363,11 +369,6 @@ class CertificatesUpload(AuthenticatedResource):
                 "san": null
               }
 
-           :arg owner: owner email for certificate
-           :arg publicCert: valid PEM public key for certificate
-           :arg intermediateCert valid PEM intermediate key for certificate
-           :arg privateKey: valid PEM private key for certificate
-           :arg destinations: list of aws destinations to upload the certificate to
            :reqheader Authorization: OAuth token to authenticate
            :statuscode 403: unauthenticated
            :statuscode 200: no error
@@ -428,7 +429,7 @@ class CertificatePrivateKey(AuthenticatedResource):
               Content-Type: text/javascript
 
               {
-                 "key": "-----BEGIN ...",
+                 "key": "-----BEGIN ..."
               }
 
            :reqheader Authorization: OAuth token to authenticate
@@ -506,7 +507,7 @@ class Certificates(AuthenticatedResource):
                 "deleted": null,
                 "notifications": [{
                     "id": 1
-                }]
+                }],
                 "signingAlgorithm": "sha256",
                 "user": {
                     "username": "jane",
@@ -520,6 +521,8 @@ class Certificates(AuthenticatedResource):
                     "id": 1090,
                     "name": "*.test.example.net"
                 }],
+                "rotation": true,
+                "rotationPolicy": {"name": "default"},
                 "replaces": [],
                 "replaced": [],
                 "name": "WILDCARD.test.example.net-SymantecCorporation-20160603-20180112",
@@ -614,6 +617,8 @@ class Certificates(AuthenticatedResource):
                     "description": "This is a google group based role created by Lemur",
                     "name": "joe@example.com"
                 }],
+                "rotation": true,
+                "rotationPolicy": {"name": "default"},
                 "san": null
               }
 
@@ -644,7 +649,9 @@ class Certificates(AuthenticatedResource):
                         )
                     ), 400
 
-        return service.update(certificate_id, **data)
+        cert = service.update(certificate_id, **data)
+        log_service.create(g.current_user, 'update_cert', certificate=cert)
+        return cert
 
 
 class NotificationCertificatesList(AuthenticatedResource):
@@ -702,7 +709,7 @@ class NotificationCertificatesList(AuthenticatedResource):
                     "deleted": null,
                     "notifications": [{
                         "id": 1
-                    }]
+                    }],
                     "signingAlgorithm": "sha256",
                     "user": {
                         "username": "jane",
@@ -718,6 +725,8 @@ class NotificationCertificatesList(AuthenticatedResource):
                     }],
                     "replaces": [],
                     "replaced": [],
+                    "rotation": true,
+                    "rotationPolicy": {"name": "default"},
                     "name": "WILDCARD.test.example.net-SymantecCorporation-20160603-20180112",
                     "roles": [{
                         "id": 464,
@@ -823,6 +832,8 @@ class CertificatesReplacementsList(AuthenticatedResource):
                     }],
                     "replaces": [],
                     "replaced": [],
+                    "rotation": true,
+                    "rotationPolicy": {"name": "default"},
                     "name": "WILDCARD.test.example.net-SymantecCorporation-20160603-20180112",
                     "roles": [{
                         "id": 464,
@@ -945,6 +956,69 @@ class CertificateExport(AuthenticatedResource):
         return dict(extension=extension, passphrase=passphrase, data=base64.b64encode(data).decode('utf-8'))
 
 
+class CertificateRevoke(AuthenticatedResource):
+    def __init__(self):
+        self.reqparse = reqparse.RequestParser()
+        super(CertificateRevoke, self).__init__()
+
+    @validate_schema(None, None)
+    def put(self, certificate_id, data=None):
+        """
+        .. http:put:: /certificates/1/revoke
+
+           Revoke a certificate
+
+           **Example request**:
+
+           .. sourcecode:: http
+
+              POST /certificates/1/revoke HTTP/1.1
+              Host: example.com
+              Accept: application/json, text/javascript
+
+           **Example response**:
+
+           .. sourcecode:: http
+
+              HTTP/1.1 200 OK
+              Vary: Accept
+              Content-Type: text/javascript
+
+              {
+                'id': 1
+              }
+
+           :reqheader Authorization: OAuth token to authenticate
+           :statuscode 200: no error
+           :statuscode 403: unauthenticated
+
+        """
+        cert = service.get(certificate_id)
+
+        if not cert:
+            return dict(message="Cannot find specified certificate"), 404
+
+        # allow creators
+        if g.current_user != cert.user:
+            owner_role = role_service.get_by_name(cert.owner)
+            permission = CertificatePermission(owner_role, [x.name for x in cert.roles])
+
+            if not permission.can():
+                return dict(message='You are not authorized to revoke this certificate.'), 403
+
+        if not cert.external_id:
+            return dict(message='Cannot revoke certificate. No external id found.'), 400
+
+        if cert.endpoints:
+            return dict(message='Cannot revoke certificate. Endpoints are deployed with the given certificate.'), 403
+
+        plugin = plugins.get(cert.authority.plugin_name)
+        plugin.revoke_certificate(cert, data)
+        log_service.create(g.current_user, 'revoke_cert', certificate=cert)
+        return dict(id=cert.id)
+
+
+api.add_resource(CertificateRevoke, '/certificates/<int:certificate_id>/revoke', endpoint='revokeCertificate')
 api.add_resource(CertificatesList, '/certificates', endpoint='certificates')
 api.add_resource(Certificates, '/certificates/<int:certificate_id>', endpoint='certificate')
 api.add_resource(CertificatesStats, '/certificates/stats', endpoint='certificateStats')

lemur/common/defaults.py

@@ -1,8 +1,26 @@
+import re
+import unicodedata
+
 from cryptography import x509
 from flask import current_app
+from lemur.extensions import sentry
 from lemur.constants import SAN_NAMING_TEMPLATE, DEFAULT_NAMING_TEMPLATE
 
 
+def text_to_slug(value):
+    """Normalize a string to a "slug" value, stripping character accents and removing non-alphanum characters."""
+
+    # Strip all character accents: decompose Unicode characters and then drop combining chars.
+    value = ''.join(c for c in unicodedata.normalize('NFKD', value) if not unicodedata.combining(c))
+
+    # Replace all remaining non-alphanumeric characters with '-'. Multiple characters get collapsed into a single dash.
+    # Except, keep 'xn--' used in IDNA domain names as is.
+    value = re.sub(r'[^A-Za-z0-9.]+(?<!xn--)', '-', value)
+
+    # '-' in the beginning or end of string looks ugly.
+    return value.strip('-')
+
+
 def certificate_name(common_name, issuer, not_before, not_after, san):
     """
     Create a name for our certificate. A naming standard
@@ -25,21 +43,13 @@ def certificate_name(common_name, issuer, not_before, not_after, san):
 
     temp = t.format(
         subject=common_name,
-        issuer=issuer,
+        issuer=issuer.replace(' ', ''),
         not_before=not_before.strftime('%Y%m%d'),
         not_after=not_after.strftime('%Y%m%d')
     )
 
-    disallowed_chars = ''.join(c for c in map(chr, range(256)) if not c.isalnum())
-    disallowed_chars = disallowed_chars.replace("-", "")
-    disallowed_chars = disallowed_chars.replace(".", "")
     temp = temp.replace('*', "WILDCARD")
-
-    for c in disallowed_chars:
-        temp = temp.replace(c, "")
-
-    # white space is silly too
-    return temp.replace(" ", "-")
+    return text_to_slug(temp)
 
 
 def signing_algorithm(cert):
@@ -58,6 +68,7 @@ def common_name(cert):
             x509.OID_COMMON_NAME
         )[0].value.strip()
     except Exception as e:
+        sentry.captureException()
         current_app.logger.error("Unable to get common name! {0}".format(e))
 
 
@@ -72,6 +83,7 @@ def organization(cert):
             x509.OID_ORGANIZATION_NAME
         )[0].value.strip()
     except Exception as e:
+        sentry.captureException()
         current_app.logger.error("Unable to get organization! {0}".format(e))
 
 
@@ -86,6 +98,7 @@ def organizational_unit(cert):
             x509.OID_ORGANIZATIONAL_UNIT_NAME
         )[0].value.strip()
     except Exception as e:
+        sentry.captureException()
         current_app.logger.error("Unable to get organizational unit! {0}".format(e))
 
 
@@ -100,6 +113,7 @@ def country(cert):
             x509.OID_COUNTRY_NAME
         )[0].value.strip()
     except Exception as e:
+        sentry.captureException()
         current_app.logger.error("Unable to get country! {0}".format(e))
 
 
@@ -114,6 +128,7 @@ def state(cert):
             x509.OID_STATE_OR_PROVINCE_NAME
         )[0].value.strip()
     except Exception as e:
+        sentry.captureException()
         current_app.logger.error("Unable to get state! {0}".format(e))
 
 
@@ -128,6 +143,7 @@ def location(cert):
             x509.OID_LOCALITY_NAME
         )[0].value.strip()
     except Exception as e:
+        sentry.captureException()
         current_app.logger.error("Unable to get location! {0}".format(e))
 
 
@@ -146,8 +162,11 @@ def domains(cert):
         entries = ext.value.get_values_for_type(x509.DNSName)
         for entry in entries:
             domains.append(entry)
+    except x509.ExtensionNotFound:
+        if current_app.config.get("LOG_SSL_SUBJ_ALT_NAME_ERRORS", True):
+            sentry.captureException()
     except Exception as e:
-        pass
+        sentry.captureException()
 
     return domains
 
@@ -159,7 +178,7 @@ def serial(cert):
     :param cert:
     :return: serial number
     """
-    return cert.serial
+    return cert.serial_number
 
 
 def san(cert):
@@ -199,6 +218,7 @@ def bitstrength(cert):
     try:
         return cert.public_key().key_size
     except AttributeError:
+        sentry.captureException()
         current_app.logger.debug('Unable to get bitstrength.')
 
 
@@ -219,6 +239,7 @@ def issuer(cert):
             issuer = issuer.replace(c, "")
         return issuer
     except Exception as e:
+        sentry.captureException()
         current_app.logger.error("Unable to get issuer! {0}".format(e))
         return "Unknown"
 

lemur/common/fields.py

@@ -1,7 +1,7 @@
 """
 .. module: lemur.common.fields
     :platform: Unix
-    :copyright: (c) 2015 by Netflix Inc., see AUTHORS for more
+    :copyright: (c) 2018 by Netflix Inc., see AUTHORS for more
     :license: Apache, see LICENSE for more details.
 .. moduleauthor:: Kevin Glisson <kglisson@netflix.com>
 """
@@ -18,6 +18,18 @@ from marshmallow import utils
 from marshmallow.fields import Field
 from marshmallow.exceptions import ValidationError
 
+from lemur.common import validators
+
+
+class Hex(Field):
+    """
+    A hex formatted string.
+    """
+    def _serialize(self, value, attr, obj):
+        if value:
+            value = hex(int(value))[2:].upper()
+        return value
+
 
 class ArrowDateTime(Field):
     """A formatted datetime string in UTC.
@@ -317,7 +329,12 @@ class SubjectAlternativeNameExtension(Field):
                     name_type = 'DNSName'
 
                 elif isinstance(name, x509.IPAddress):
-                    name_type = 'IPAddress'
+                    if isinstance(value, ipaddress.IPv4Network):
+                        name_type = 'IPNetwork'
+                    else:
+                        name_type = 'IPAddress'
+
+                    value = str(value)
 
                 elif isinstance(name, x509.UniformResourceIdentifier):
                     name_type = 'uniformResourceIdentifier'
@@ -342,6 +359,7 @@ class SubjectAlternativeNameExtension(Field):
         general_names = []
         for name in value:
             if name['nameType'] == 'DNSName':
+                validators.sensitive_domain(name['value'])
                 general_names.append(x509.DNSName(name['value']))
 
             elif name['nameType'] == 'IPAddress':

lemur/common/health.py

@@ -1,16 +1,29 @@
 """
 .. module: lemur.common.health
     :platform: Unix
-    :copyright: (c) 2015 by Netflix Inc., see AUTHORS for more
+    :copyright: (c) 2018 by Netflix Inc., see AUTHORS for more
     :license: Apache, see LICENSE for more details.
 
 .. moduleauthor:: Kevin Glisson <kglisson@netflix.com>
 """
 from flask import Blueprint
+from lemur.database import db
+from lemur.extensions import sentry
 
 mod = Blueprint('healthCheck', __name__)
 
 
 @mod.route('/healthcheck')
 def health():
-    return 'ok'
+    try:
+        if healthcheck(db):
+            return 'ok'
+    except Exception:
+        sentry.captureException()
+        return 'db check failed'
+
+
+def healthcheck(db):
+    with db.engine.connect() as connection:
+        connection.execute('SELECT 1;')
+    return True

lemur/common/managers.py

@@ -1,7 +1,7 @@
 """
 .. module: lemur.common.managers
     :platform: Unix
-    :copyright: (c) 2015 by Netflix Inc., see AUTHORS for more
+    :copyright: (c) 2018 by Netflix Inc., see AUTHORS for more
     :license: Apache, see LICENSE for more details.
 
 .. moduleauthor:: Kevin Glisson <kglisson@netflix.com>

lemur/common/schema.py

@@ -1,7 +1,7 @@
 """
 .. module: lemur.common.schema
     :platform: unix
-    :copyright: (c) 2015 by Netflix Inc., see AUTHORS for more
+    :copyright: (c) 2018 by Netflix Inc., see AUTHORS for more
     :license: Apache, see LICENSE for more details.
 
 .. moduleauthor:: Kevin Glisson <kglisson@netflix.com>
@@ -15,6 +15,8 @@ from sqlalchemy.orm.collections import InstrumentedList
 from inflection import camelize, underscore
 from marshmallow import Schema, post_dump, pre_load
 
+from lemur.extensions import sentry
+
 
 class LemurSchema(Schema):
     """
@@ -133,7 +135,6 @@ def unwrap_pagination(data, output_schema):
         marshaled_data = {'total': len(data)}
         marshaled_data['items'] = output_schema.dump(data, many=True).data
         return marshaled_data
-
     return output_schema.dump(data).data
 
 
@@ -157,6 +158,7 @@ def validate_schema(input_schema, output_schema):
             try:
                 resp = f(*args, **kwargs)
             except Exception as e:
+                sentry.captureException()
                 current_app.logger.exception(e)
                 return dict(message=str(e)), 500
 

lemur/common/utils.py

@@ -1,23 +1,22 @@
 """
 .. module: lemur.common.utils
     :platform: Unix
-    :copyright: (c) 2015 by Netflix Inc., see AUTHORS for more
+    :copyright: (c) 2018 by Netflix Inc., see AUTHORS for more
     :license: Apache, see LICENSE for more details.
 
 .. moduleauthor:: Kevin Glisson <kglisson@netflix.com>
 """
-import string
 import random
+import string
 
 import sqlalchemy
-from sqlalchemy import and_, func
-
 from cryptography import x509
 from cryptography.hazmat.backends import default_backend
-from cryptography.hazmat.primitives.asymmetric import rsa
-
+from cryptography.hazmat.primitives.asymmetric import rsa, ec
 from flask_restful.reqparse import RequestParser
+from sqlalchemy import and_, func
 
+from lemur.constants import CERTIFICATE_KEY_TYPES
 from lemur.exceptions import InvalidConfiguration
 
 paginated_parser = RequestParser()
@@ -53,21 +52,68 @@ def parse_certificate(body):
     return x509.load_pem_x509_certificate(body, default_backend())
 
 
+def parse_csr(csr):
+    """
+    Helper function that parses a CSR.
+
+    :param csr:
+    :return:
+    """
+    if isinstance(csr, str):
+        csr = csr.encode('utf-8')
+
+    return x509.load_pem_x509_csr(csr, default_backend())
+
+
+def get_authority_key(body):
+    """Returns the authority key for a given certificate in hex format"""
+    parsed_cert = parse_certificate(body)
+    authority_key = parsed_cert.extensions.get_extension_for_class(
+        x509.AuthorityKeyIdentifier).value.key_identifier
+    return authority_key.hex()
+
+
 def generate_private_key(key_type):
     """
     Generates a new private key based on key_type.
 
-    Valid key types: RSA2048, RSA4096
+    Valid key types: RSA2048, RSA4096', 'ECCPRIME192V1', 'ECCPRIME256V1', 'ECCSECP192R1',
+        'ECCSECP224R1', 'ECCSECP256R1', 'ECCSECP384R1', 'ECCSECP521R1', 'ECCSECP256K1',
+        'ECCSECT163K1', 'ECCSECT233K1', 'ECCSECT283K1', 'ECCSECT409K1', 'ECCSECT571K1',
+        'ECCSECT163R2', 'ECCSECT233R1', 'ECCSECT283R1', 'ECCSECT409R1', 'ECCSECT571R2'
 
     :param key_type:
     :return:
     """
-    valid_key_types = ['RSA2048', 'RSA4096']
 
-    if key_type not in valid_key_types:
+    _CURVE_TYPES = {
+        "ECCPRIME192V1": ec.SECP192R1(),
+        "ECCPRIME256V1": ec.SECP256R1(),
+
+        "ECCSECP192R1": ec.SECP192R1(),
+        "ECCSECP224R1": ec.SECP224R1(),
+        "ECCSECP256R1": ec.SECP256R1(),
+        "ECCSECP384R1": ec.SECP384R1(),
+        "ECCSECP521R1": ec.SECP521R1(),
+        "ECCSECP256K1": ec.SECP256K1(),
+
+        "ECCSECT163K1": ec.SECT163K1(),
+        "ECCSECT233K1": ec.SECT233K1(),
+        "ECCSECT283K1": ec.SECT283K1(),
+        "ECCSECT409K1": ec.SECT409K1(),
+        "ECCSECT571K1": ec.SECT571K1(),
+
+        "ECCSECT163R2": ec.SECT163R2(),
+        "ECCSECT233R1": ec.SECT233R1(),
+        "ECCSECT283R1": ec.SECT283R1(),
+        "ECCSECT409R1": ec.SECT409R1(),
+        "ECCSECT571R2": ec.SECT571R1(),
+    }
+
+    if key_type not in CERTIFICATE_KEY_TYPES:
         raise Exception("Invalid key type: {key_type}. Supported key types: {choices}".format(
             key_type=key_type,
-            choices=",".join(valid_key_types)
+            choices=",".join(CERTIFICATE_KEY_TYPES)
         ))
 
     if 'RSA' in key_type:
@@ -77,6 +123,11 @@ def generate_private_key(key_type):
             key_size=key_size,
             backend=default_backend()
         )
+    elif 'ECC' in key_type:
+        return ec.generate_private_key(
+            curve=_CURVE_TYPES[key_type],
+            backend=default_backend()
+        )
 
 
 def is_weekend(date):
@@ -154,3 +205,9 @@ def windowed_query(q, column, windowsize):
             column, windowsize):
         for row in q.filter(whereclause).order_by(column):
             yield row
+
+
+def truthiness(s):
+    """If input string resembles something truthy then return True, else False."""
+
+    return s.lower() in ('true', 'yes', 'on', 't', '1')

lemur/common/validators.py

@@ -1,9 +1,10 @@
 import re
 
-from flask import current_app
 from cryptography import x509
 from cryptography.hazmat.backends import default_backend
 from cryptography.hazmat.primitives import serialization
+from cryptography.x509 import NameOID
+from flask import current_app
 from marshmallow.exceptions import ValidationError
 
 from lemur.auth.permissions import SensitiveDomainPermission
@@ -41,22 +42,33 @@ def private_key(key):
         raise ValidationError('Private key presented is not valid.')
 
 
+def common_name(value):
+    """If the common name could be a domain name, apply domain validation rules."""
+    # Common name could be a domain name, or a human-readable name of the subject (often used in CA names or client
+    # certificates). As a simple heuristic, we assume that human-readable names always include a space.
+    # However, to avoid confusion for humans, we also don't count spaces at the beginning or end of the string.
+    if ' ' not in value.strip():
+        return sensitive_domain(value)
+
+
 def sensitive_domain(domain):
     """
-    Determines if domain has been marked as sensitive.
-    :param domain:
+    Checks if user has the admin role, the domain does not match sensitive domains and whitelisted domain patterns.
+    :param domain: domain name (str)
     :return:
     """
-    restricted_domains = current_app.config.get('LEMUR_RESTRICTED_DOMAINS', [])
-    if restricted_domains:
-        domains = domain_service.get_by_name(domain)
-        for domain in domains:
-            # we only care about non-admins
-            if not SensitiveDomainPermission().can():
-                if domain.sensitive or any([re.match(pattern, domain.name) for pattern in restricted_domains]):
-                    raise ValidationError(
-                        'Domain {0} has been marked as sensitive, contact and administrator \
-                        to issue the certificate.'.format(domain))
+    if SensitiveDomainPermission().can():
+        # User has permission, no need to check anything
+        return
+
+    whitelist = current_app.config.get('LEMUR_WHITELISTED_DOMAINS', [])
+    if whitelist and not any(re.match(pattern, domain) for pattern in whitelist):
+        raise ValidationError('Domain {0} does not match whitelisted domain patterns. '
+                              'Contact an administrator to issue the certificate.'.format(domain))
+
+    if any(d.sensitive for d in domain_service.get_by_name(domain)):
+        raise ValidationError('Domain {0} has been marked as sensitive. '
+                              'Contact an administrator to issue the certificate.'.format(domain))
 
 
 def encoding(oid_encoding):
@@ -84,15 +96,27 @@ def sub_alt_type(alt_type):
 
 def csr(data):
     """
-    Determines if the CSR is valid.
+    Determines if the CSR is valid and allowed.
     :param data:
     :return:
     """
     try:
-        x509.load_pem_x509_csr(data.encode('utf-8'), default_backend())
+        request = x509.load_pem_x509_csr(data.encode('utf-8'), default_backend())
     except Exception:
         raise ValidationError('CSR presented is not valid.')
 
+    # Validate common name and SubjectAltNames
+    for name in request.subject.get_attributes_for_oid(NameOID.COMMON_NAME):
+        common_name(name.value)
+
+    try:
+        alt_names = request.extensions.get_extension_for_class(x509.SubjectAlternativeName)
+
+        for name in alt_names.value.get_values_for_type(x509.DNSName):
+            sensitive_domain(name)
+    except x509.ExtensionNotFound:
+        pass
+
 
 def dates(data):
     if not data.get('validity_start') and data.get('validity_end'):

lemur/constants.py

@@ -1,8 +1,34 @@
 """
 .. module: lemur.constants
-    :copyright: (c) 2015 by Netflix Inc.
+    :copyright: (c) 2018 by Netflix Inc.
     :license: Apache, see LICENSE for more details.
 """
 SAN_NAMING_TEMPLATE = "SAN-{subject}-{issuer}-{not_before}-{not_after}"
 DEFAULT_NAMING_TEMPLATE = "{subject}-{issuer}-{not_before}-{not_after}"
 NONSTANDARD_NAMING_TEMPLATE = "{issuer}-{not_before}-{not_after}"
+
+SUCCESS_METRIC_STATUS = 'success'
+FAILURE_METRIC_STATUS = 'failure'
+
+CERTIFICATE_KEY_TYPES = [
+    'RSA2048',
+    'RSA4096',
+    'ECCPRIME192V1',
+    'ECCPRIME256V1',
+    'ECCSECP192R1',
+    'ECCSECP224R1',
+    'ECCSECP256R1',
+    'ECCSECP384R1',
+    'ECCSECP521R1',
+    'ECCSECP256K1',
+    'ECCSECT163K1',
+    'ECCSECT233K1',
+    'ECCSECT283K1',
+    'ECCSECT409K1',
+    'ECCSECT571K1',
+    'ECCSECT163R2',
+    'ECCSECT233R1',
+    'ECCSECT283R1',
+    'ECCSECT409R1',
+    'ECCSECT571R2'
+]

lemur/database.py

@@ -4,12 +4,13 @@
     :synopsis: This module contains all of the database related methods
     needed for lemur to interact with a datastore
 
-    :copyright: (c) 2015 by Netflix Inc., see AUTHORS for more
+    :copyright: (c) 2018 by Netflix Inc., see AUTHORS for more
     :license: Apache, see LICENSE for more details.
 
 .. moduleauthor:: Kevin Glisson <kglisson@netflix.com>
 """
-from sqlalchemy import exc
+from inflection import underscore
+from sqlalchemy import exc, func
 from sqlalchemy.sql import and_, or_
 from sqlalchemy.orm import make_transient
 
@@ -75,6 +76,16 @@ def add(model):
     db.session.add(model)
 
 
+def get_model_column(model, field):
+    if field in getattr(model, 'sensitive_fields', ()):
+        raise AttrNotFound(field)
+    column = model.__table__.columns._data.get(field, None)
+    if column is None:
+        raise AttrNotFound(field)
+
+    return column
+
+
 def find_all(query, model, kwargs):
     """
     Returns a query object that ensures that all kwargs
@@ -91,7 +102,7 @@ def find_all(query, model, kwargs):
         if not isinstance(value, list):
             value = value.split(',')
 
-        conditions.append(getattr(model, attr).in_(value))
+        conditions.append(get_model_column(model, attr).in_(value))
 
     return query.filter(and_(*conditions))
 
@@ -108,7 +119,7 @@ def find_any(query, model, kwargs):
     """
     or_args = []
     for attr, value in kwargs.items():
-        or_args.append(or_(getattr(model, attr) == value))
+        or_args.append(or_(get_model_column(model, attr) == value))
     exprs = or_(*or_args)
     return query.filter(exprs)
 
@@ -123,7 +134,7 @@ def get(model, value, field="id"):
     :return:
     """
     query = session_query(model)
-    return query.filter(getattr(model, field) == value).scalar()
+    return query.filter(get_model_column(model, field) == value).scalar()
 
 
 def get_all(model, value, field="id"):
@@ -136,7 +147,7 @@ def get_all(model, value, field="id"):
     :return:
     """
     query = session_query(model)
-    return query.filter(getattr(model, field) == value)
+    return query.filter(get_model_column(model, field) == value)
 
 
 def create(model):
@@ -188,7 +199,8 @@ def filter(query, model, terms):
     :param terms:
     :return:
     """
-    return query.filter(getattr(model, terms[0]).ilike('%{}%'.format(terms[1])))
+    column = get_model_column(model, underscore(terms[0]))
+    return query.filter(column.ilike('%{}%'.format(terms[1])))
 
 
 def sort(query, model, field, direction):
@@ -201,13 +213,8 @@ def sort(query, model, field, direction):
     :param field:
     :param direction:
     """
-    try:
-        field = getattr(model, field)
-        direction = getattr(field, direction)
-        query = query.order_by(direction())
-        return query
-    except AttributeError:
-        raise AttrNotFound(field)
+    column = get_model_column(model, underscore(field))
+    return query.order_by(column.desc() if direction == 'desc' else column.asc())
 
 
 def paginate(query, page, count):
@@ -260,6 +267,17 @@ def clone(model):
     return model
 
 
+def get_count(q):
+    """
+    Count the number of rows in a table. More efficient than count(*)
+    :param q:
+    :return:
+    """
+    count_q = q.statement.with_only_columns([func.count()]).order_by(None)
+    count = q.session.execute(count_q).scalar()
+    return count
+
+
 def sort_and_page(query, model, args):
     """
     Helper that allows us to combine sorting and paging
@@ -282,7 +300,7 @@ def sort_and_page(query, model, args):
     if sort_by and sort_dir:
         query = sort(query, model, sort_by, sort_dir)
 
-    total = query.count()
+    total = get_count(query)
 
     # offset calculated at zero
     page -= 1

lemur/decorators.py

@@ -1,56 +0,0 @@
-"""
-.. module: lemur.decorators
-    :copyright: (c) 2015 by Netflix Inc., see AUTHORS for more
-    :license: Apache, see LICENSE for more details.
-"""
-from builtins import str
-
-from datetime import timedelta
-from flask import make_response, request, current_app
-
-from functools import update_wrapper
-
-
-# this is only used for dev
-def crossdomain(origin=None, methods=None, headers=None,
-                max_age=21600, attach_to_all=True,
-                automatic_options=True):  # pragma: no cover
-    if methods is not None:
-        methods = ', '.join(sorted(x.upper() for x in methods))
-
-    if headers is not None and not isinstance(headers, str):
-        headers = ', '.join(x.upper() for x in headers)
-
-    if not isinstance(origin, str):
-        origin = ', '.join(origin)
-
-    if isinstance(max_age, timedelta):
-        max_age = max_age.total_seconds()
-
-    def get_methods():
-        if methods is not None:
-            return methods
-
-        options_resp = current_app.make_default_options_response()
-        return options_resp.headers['allow']
-
-    def decorator(f):
-        def wrapped_function(*args, **kwargs):
-            if automatic_options and request.method == 'OPTIONS':
-                resp = current_app.make_default_options_response()
-            else:
-                resp = make_response(f(*args, **kwargs))
-            if not attach_to_all and request.method != 'OPTIONS':
-                return resp
-
-            h = resp.headers
-            h['Access-Control-Allow-Origin'] = origin
-            h['Access-Control-Allow-Methods'] = get_methods()
-            h['Access-Control-Max-Age'] = str(max_age)
-            h['Access-Control-Allow-Headers'] = "Origin, X-Requested-With, Content-Type, Accept, Authorization "
-            h['Access-Control-Allow-Credentials'] = 'true'
-            return resp
-
-        f.provide_automatic_options = False
-        return update_wrapper(wrapped_function, f)
-    return decorator

lemur/default.conf.py

@@ -9,7 +9,7 @@ THREADS_PER_PAGE = 8
 
 # These will need to be set to `True` if you are developing locally
 CORS = False
-debug = False
+DEBUG = False
 
 # Logging
 

lemur/defaults/schemas.py

@@ -1,7 +1,7 @@
 """
 .. module: lemur.defaults.schemas
     :platform: unix
-    :copyright: (c) 2015 by Netflix Inc., see AUTHORS for more
+    :copyright: (c) 2018 by Netflix Inc., see AUTHORS for more
     :license: Apache, see LICENSE for more details.
 .. moduleauthor:: Kevin Glisson <kglisson@netflix.com>
 """

lemur/defaults/views.py

@@ -1,6 +1,6 @@
 """
 .. module: lemur.defaults.views
-    :copyright: (c) 2015 by Netflix Inc., see AUTHORS for more
+    :copyright: (c) 2018 by Netflix Inc., see AUTHORS for more
     :license: Apache, see LICENSE for more details.
 """
 from flask import current_app, Blueprint
@@ -50,7 +50,8 @@ class LemurDefaults(AuthenticatedResource):
                  "state": "CA",
                  "location": "Los Gatos",
                  "organization": "Netflix",
-                 "organizationalUnit": "Operations"
+                 "organizationalUnit": "Operations",
+                 "dnsProviders": [{"name": "test", ...}, {...}],
               }
 
            :reqheader Authorization: OAuth token to authenticate
@@ -67,7 +68,7 @@ class LemurDefaults(AuthenticatedResource):
             organization=current_app.config.get('LEMUR_DEFAULT_ORGANIZATION'),
             organizational_unit=current_app.config.get('LEMUR_DEFAULT_ORGANIZATIONAL_UNIT'),
             issuer_plugin=current_app.config.get('LEMUR_DEFAULT_ISSUER_PLUGIN'),
-            authority=default_authority
+            authority=default_authority,
         )
 
 

lemur/deployment/service.py

@@ -10,7 +10,6 @@ def rotate_certificate(endpoint, new_cert):
     :return:
     """
     # ensure that certificate is available for rotation
-
     endpoint.source.plugin.update_endpoint(endpoint, new_cert)
     endpoint.certificate = new_cert
     database.update(endpoint)

lemur/destinations/models.py

@@ -1,7 +1,7 @@
 """
 .. module: lemur.destinations.models
     :platform: unix
-    :copyright: (c) 2015 by Netflix Inc., see AUTHORS for more
+    :copyright: (c) 2018 by Netflix Inc., see AUTHORS for more
     :license: Apache, see LICENSE for more details.
 .. moduleauthor:: Kevin Glisson <kglisson@netflix.com>
 """

lemur/destinations/schemas.py

@@ -1,7 +1,7 @@
 """
 .. module: lemur.destinations.schemas
     :platform: unix
-    :copyright: (c) 2015 by Netflix Inc., see AUTHORS for more
+    :copyright: (c) 2018 by Netflix Inc., see AUTHORS for more
     :license: Apache, see LICENSE for more details.
 .. moduleauthor:: Kevin Glisson <kglisson@netflix.com>
 """

lemur/destinations/service.py

@@ -1,7 +1,7 @@
 """
 .. module: lemur.destinations.service
     :platform: Unix
-    :copyright: (c) 2015 by Netflix Inc., see AUTHORS for more
+    :copyright: (c) 2018 by Netflix Inc., see AUTHORS for more
     :license: Apache, see LICENSE for more details.
 .. moduleauthor:: Kevin Glisson <kglisson@netflix.com>
 """
@@ -22,6 +22,11 @@ def create(label, plugin_name, options, description=None):
     :rtype : Destination
     :return: New destination
     """
+    # remove any sub-plugin objects before try to save the json options
+    for option in options:
+        if 'plugin' in option['type']:
+            del option['value']['plugin_object']
+
     destination = Destination(label=label, options=options, plugin_name=plugin_name, description=description)
     return database.create(destination)
 

lemur/destinations/views.py

@@ -2,7 +2,7 @@
 .. module: lemur.destinations.views
     :platform: Unix
     :synopsis: This module contains all of the accounts view code.
-    :copyright: (c) 2015 by Netflix Inc., see AUTHORS for more
+    :copyright: (c) 2018 by Netflix Inc., see AUTHORS for more
     :license: Apache, see LICENSE for more details.
 .. moduleauthor:: Kevin Glisson <kglisson@netflix.com>
 """

lemur/dns_providers/models.py

@@ -0,0 +1,37 @@
+from sqlalchemy import Column, Integer, String, text, Text
+from sqlalchemy.dialects.postgresql import JSON
+from sqlalchemy_utils import ArrowType
+
+from lemur.database import db
+from lemur.plugins.base import plugins
+from lemur.utils import Vault
+
+
+class DnsProvider(db.Model):
+    __tablename__ = 'dns_providers'
+    id = Column(
+        Integer(),
+        primary_key=True,
+    )
+    name = Column(String(length=256), unique=True, nullable=True)
+    description = Column(Text(), nullable=True)
+    provider_type = Column(String(length=256), nullable=True)
+    credentials = Column(Vault, nullable=True)
+    api_endpoint = Column(String(length=256), nullable=True)
+    date_created = Column(ArrowType(), server_default=text('now()'), nullable=False)
+    status = Column(String(length=128), nullable=True)
+    options = Column(JSON, nullable=True)
+    domains = Column(JSON, nullable=True)
+
+    def __init__(self, name, description, provider_type, credentials):
+        self.name = name
+        self.description = description
+        self.provider_type = provider_type
+        self.credentials = credentials
+
+    @property
+    def plugin(self):
+        return plugins.get(self.plugin_name)
+
+    def __repr__(self):
+        return "DnsProvider(name={name})".format(name=self.name)

lemur/dns_providers/schemas.py

@@ -0,0 +1,27 @@
+from lemur.common.fields import ArrowDateTime
+from lemur.common.schema import LemurInputSchema, LemurOutputSchema
+
+from marshmallow import fields
+
+
+class DnsProvidersNestedOutputSchema(LemurOutputSchema):
+    __envelope__ = False
+    id = fields.Integer()
+    name = fields.String()
+    providerType = fields.String()
+    description = fields.String()
+    credentials = fields.String()
+    api_endpoint = fields.String()
+    date_created = ArrowDateTime()
+
+
+class DnsProvidersNestedInputSchema(LemurInputSchema):
+    __envelope__ = False
+    name = fields.String()
+    description = fields.String()
+    provider_type = fields.Dict()
+
+
+dns_provider_output_schema = DnsProvidersNestedOutputSchema()
+
+dns_provider_input_schema = DnsProvidersNestedInputSchema()

lemur/dns_providers/service.py

@@ -0,0 +1,111 @@
+import json
+
+from flask import current_app
+from lemur import database
+from lemur.dns_providers.models import DnsProvider
+
+
+def render(args):
+    """
+    Helper that helps us render the REST Api responses.
+    :param args:
+    :return:
+    """
+    query = database.session_query(DnsProvider)
+
+    return database.sort_and_page(query, DnsProvider, args)
+
+
+def get(dns_provider_id):
+    provider = database.get(DnsProvider, dns_provider_id)
+    return provider
+
+
+def get_friendly(dns_provider_id):
+    """
+    Retrieves a dns provider by its lemur assigned ID.
+
+    :param dns_provider_id: Lemur assigned ID
+    :rtype : DnsProvider
+    :return:
+    """
+    dns_provider = get(dns_provider_id)
+    dns_provider_friendly = {
+        "name": dns_provider.name,
+        "description": dns_provider.description,
+        "providerType": dns_provider.provider_type,
+        "options": dns_provider.options,
+        "credentials": dns_provider.credentials,
+    }
+
+    if dns_provider.provider_type == "route53":
+        dns_provider_friendly["account_id"] = json.loads(dns_provider.credentials).get("account_id")
+    return dns_provider_friendly
+
+
+def delete(dns_provider_id):
+    """
+    Deletes a DNS provider.
+
+    :param dns_provider_id: Lemur assigned ID
+    """
+    database.delete(get(dns_provider_id))
+
+
+def get_types():
+    provider_config = current_app.config.get(
+        'ACME_DNS_PROVIDER_TYPES',
+        {"items": [
+            {
+                'name': 'route53',
+                'requirements': [
+                    {
+                        'name': 'account_id',
+                        'type': 'int',
+                        'required': True,
+                        'helpMessage': 'AWS Account number'
+                    },
+                ]
+            },
+            {
+                'name': 'cloudflare',
+                'requirements': [
+                    {
+                        'name': 'email',
+                        'type': 'str',
+                        'required': True,
+                        'helpMessage': 'Cloudflare Email'
+                    },
+                    {
+                        'name': 'key',
+                        'type': 'str',
+                        'required': True,
+                        'helpMessage': 'Cloudflare Key'
+                    },
+                ]
+            },
+            {
+                'name': 'dyn',
+            },
+        ]}
+    )
+    if not provider_config:
+        raise Exception("No DNS Provider configuration specified.")
+    provider_config["total"] = len(provider_config.get("items"))
+    return provider_config
+
+
+def create(data):
+    provider_name = data.get("name")
+
+    credentials = {}
+    for item in data.get("provider_type", {}).get("requirements", []):
+        credentials[item["name"]] = item["value"]
+    dns_provider = DnsProvider(
+        name=provider_name,
+        description=data.get("description"),
+        provider_type=data.get("provider_type").get("name"),
+        credentials=json.dumps(credentials),
+    )
+    created = database.create(dns_provider)
+    return created.id

lemur/dns_providers/views.py

@@ -0,0 +1,170 @@
+"""
+.. module: lemur.dns)providers.views
+    :platform: Unix
+    :copyright: (c) 2018 by Netflix Inc., see AUTHORS for more
+    :license: Apache, see LICENSE for more details.
+.. moduleauthor:: Curtis Castrapel <ccastrapel@netflix.com>
+"""
+from flask import Blueprint, g
+from flask_restful import reqparse, Api
+
+from lemur.auth.permissions import admin_permission
+from lemur.auth.service import AuthenticatedResource
+from lemur.common.schema import validate_schema
+from lemur.common.utils import paginated_parser
+from lemur.dns_providers import service
+from lemur.dns_providers.schemas import dns_provider_output_schema, dns_provider_input_schema
+
+mod = Blueprint('dns_providers', __name__)
+api = Api(mod)
+
+
+class DnsProvidersList(AuthenticatedResource):
+    """ Defines the 'dns_providers' endpoint """
+    def __init__(self):
+        self.reqparse = reqparse.RequestParser()
+        super(DnsProvidersList, self).__init__()
+
+    @validate_schema(None, dns_provider_output_schema)
+    def get(self):
+        """
+        .. http:get:: /dns_providers
+
+           The current list of DNS Providers
+
+           **Example request**:
+
+           .. sourcecode:: http
+
+              GET /dns_providers HTTP/1.1
+              Host: example.com
+              Accept: application/json, text/javascript
+
+           **Example response**:
+
+           .. sourcecode:: http
+
+              HTTP/1.1 200 OK
+              Vary: Accept
+              Content-Type: text/javascript
+
+              {
+                "items": [{
+                    "id": 1,
+                    "name": "test",
+                    "description": "test",
+                    "provider_type": "dyn",
+                    "status": "active",
+                }],
+                "total": 1
+              }
+
+           :query sortBy: field to sort on
+           :query sortDir: asc or desc
+           :query page: int. default is 1
+           :query filter: key value pair format is k;v
+           :query count: count number. default is 10
+           :reqheader Authorization: OAuth token to authenticate
+           :statuscode 200: no error
+           :statuscode 403: unauthenticated
+
+        """
+        parser = paginated_parser.copy()
+        parser.add_argument('dns_provider_id', type=int, location='args')
+        parser.add_argument('name', type=str, location='args')
+        parser.add_argument('type', type=str, location='args')
+
+        args = parser.parse_args()
+        args['user'] = g.user
+        return service.render(args)
+
+    @validate_schema(dns_provider_input_schema, None)
+    @admin_permission.require(http_exception=403)
+    def post(self, data=None):
+        """
+        Creates a DNS Provider
+
+        **Example request**:
+        {
+          "providerType": {
+            "name": "route53",
+            "requirements": [
+              {
+                "name": "account_id",
+                "type": "int",
+                "required": true,
+                "helpMessage": "AWS Account number",
+                "value": 12345
+              }
+            ],
+            "route": "dns_provider_options",
+            "reqParams": null,
+            "restangularized": true,
+            "fromServer": true,
+            "parentResource": null,
+            "restangularCollection": false
+          },
+          "name": "provider_name",
+          "description": "provider_description"
+        }
+
+        **Example request 2**
+        {
+          "providerType": {
+            "name": "cloudflare",
+            "requirements": [
+              {
+                "name": "email",
+                "type": "str",
+                "required": true,
+                "helpMessage": "Cloudflare Email",
+                "value": "test@example.com"
+              },
+              {
+                "name": "key",
+                "type": "str",
+                "required": true,
+                "helpMessage": "Cloudflare Key",
+                "value": "secretkey"
+              }
+            ],
+            "route": "dns_provider_options",
+            "reqParams": null,
+            "restangularized": true,
+            "fromServer": true,
+            "parentResource": null,
+            "restangularCollection": false
+          },
+          "name": "provider_name",
+          "description": "provider_description"
+        }
+        :return:
+        """
+        return service.create(data)
+
+
+class DnsProviders(AuthenticatedResource):
+    @validate_schema(None, dns_provider_output_schema)
+    def get(self, dns_provider_id):
+        return service.get_friendly(dns_provider_id)
+
+    @admin_permission.require(http_exception=403)
+    def delete(self, dns_provider_id):
+        service.delete(dns_provider_id)
+        return {'result': True}
+
+
+class DnsProviderOptions(AuthenticatedResource):
+    """ Defines the 'dns_provider_types' endpoint """
+
+    def __init__(self):
+        self.reqparse = reqparse.RequestParser()
+        super(DnsProviderOptions, self).__init__()
+
+    def get(self):
+        return service.get_types()
+
+
+api.add_resource(DnsProvidersList, '/dns_providers', endpoint='dns_providers')
+api.add_resource(DnsProviders, '/dns_providers/<int:dns_provider_id>', endpoint='dns_provider')
+api.add_resource(DnsProviderOptions, '/dns_provider_options', endpoint='dns_provider_options')

lemur/domains/models.py

@@ -1,7 +1,7 @@
 """
 .. module: lemur.domains.models
     :platform: Unix
-    :copyright: (c) 2015 by Netflix Inc., see AUTHORS for more
+    :copyright: (c) 2018 by Netflix Inc., see AUTHORS for more
     :license: Apache, see LICENSE for more details.
 
 .. moduleauthor:: Kevin Glisson <kglisson@netflix.com>

lemur/domains/schemas.py

@@ -1,7 +1,7 @@
 """
 .. module: lemur.domains.schemas
     :platform: unix
-    :copyright: (c) 2015 by Netflix Inc., see AUTHORS for more
+    :copyright: (c) 2018 by Netflix Inc., see AUTHORS for more
     :license: Apache, see LICENSE for more details.
 .. moduleauthor:: Kevin Glisson <kglisson@netflix.com>
 """
@@ -15,7 +15,7 @@ from lemur.schemas import AssociatedCertificateSchema
 class DomainInputSchema(LemurInputSchema):
     id = fields.Integer()
     name = fields.String(required=True)
-    sensitive = fields.Boolean()
+    sensitive = fields.Boolean(missing=False)
     certificates = fields.Nested(AssociatedCertificateSchema, many=True, missing=[])
 
 

lemur/domains/service.py

@@ -1,7 +1,7 @@
 """
 .. module: lemur.domains.service
     :platform: Unix
-    :copyright: (c) 2015 by Netflix Inc., see AUTHORS for more
+    :copyright: (c) 2018 by Netflix Inc., see AUTHORS for more
     :license: Apache, see LICENSE for more details.
 
 .. moduleauthor:: Kevin Glisson <kglisson@netflix.com>
@@ -76,7 +76,7 @@ def render(args):
     :param args:
     :return:
     """
-    query = database.session_query(Domain).join(Certificate, Domain.certificate)
+    query = database.session_query(Domain)
     filt = args.pop('filter')
     certificate_id = args.pop('certificate_id', None)
 
@@ -85,6 +85,7 @@ def render(args):
         query = database.filter(query, Domain, terms)
 
     if certificate_id:
+        query = query.join(Certificate, Domain.certificates)
         query = query.filter(Certificate.id == certificate_id)
 
     return database.sort_and_page(query, Domain, args)

lemur/domains/views.py

@@ -1,7 +1,7 @@
 """
 .. module: lemur.domains.views
     :platform: Unix
-    :copyright: (c) 2015 by Netflix Inc., see AUTHORS for more
+    :copyright: (c) 2018 by Netflix Inc., see AUTHORS for more
     :license: Apache, see LICENSE for more details.
 
 .. moduleauthor:: Kevin Glisson <kglisson@netflix.com>

lemur/endpoints/cli.py

@@ -1,7 +1,7 @@
 """
 .. module: lemur.endpoints.cli
     :platform: Unix
-    :copyright: (c) 2015 by Netflix Inc., see AUTHORS for more
+    :copyright: (c) 2018 by Netflix Inc., see AUTHORS for more
     :license: Apache, see LICENSE for more details.
 .. moduleauthor:: Kevin Glisson <kglisson@netflix.com>
 """
@@ -14,7 +14,7 @@ from sqlalchemy import cast
 from sqlalchemy_utils import ArrowType
 
 from lemur import database
-from lemur.extensions import metrics
+from lemur.extensions import metrics, sentry
 from lemur.endpoints.models import Endpoint
 
 
@@ -27,13 +27,17 @@ def expire(ttl):
     Removed all endpoints that have not been recently updated.
     """
     print("[+] Staring expiration of old endpoints.")
-    now = arrow.utcnow()
-    expiration = now - timedelta(hours=ttl)
-    endpoints = database.session_query(Endpoint).filter(cast(Endpoint.last_updated, ArrowType) <= expiration)
 
-    for endpoint in endpoints:
-        print("[!] Expiring endpoint: {name} Last Updated: {last_updated}".format(name=endpoint.name, last_updated=endpoint.last_updated))
-        database.delete(endpoint)
-        metrics.send('endpoint_expired', 'counter', 1)
+    try:
+        now = arrow.utcnow()
+        expiration = now - timedelta(hours=ttl)
+        endpoints = database.session_query(Endpoint).filter(cast(Endpoint.last_updated, ArrowType) <= expiration)
 
-    print("[+] Finished expiration.")
+        for endpoint in endpoints:
+            print("[!] Expiring endpoint: {name} Last Updated: {last_updated}".format(name=endpoint.name, last_updated=endpoint.last_updated))
+            database.delete(endpoint)
+            metrics.send('endpoint_expired', 'counter', 1)
+
+        print("[+] Finished expiration.")
+    except Exception as e:
+        sentry.captureException()

lemur/endpoints/models.py

@@ -2,7 +2,7 @@
 .. module: lemur.endpoints.models
     :platform: unix
     :synopsis: This module contains all of the models need to create an authority within Lemur.
-    :copyright: (c) 2015 by Netflix Inc., see AUTHORS for more
+    :copyright: (c) 2018 by Netflix Inc., see AUTHORS for more
     :license: Apache, see LICENSE for more details.
 .. moduleauthor:: Kevin Glisson <kglisson@netflix.com>
 """

lemur/endpoints/schemas.py

@@ -1,7 +1,7 @@
 """
 .. module: lemur.endpoints.schemas
     :platform: unix
-    :copyright: (c) 2015 by Netflix Inc., see AUTHORS for more
+    :copyright: (c) 2018 by Netflix Inc., see AUTHORS for more
     :license: Apache, see LICENSE for more details.
 .. moduleauthor:: Kevin Glisson <kglisson@netflix.com>
 """

lemur/endpoints/service.py

@@ -3,18 +3,17 @@
     :platform: Unix
     :synopsis: This module contains all of the services level functions used to
     administer endpoints in Lemur
-    :copyright: (c) 2015 by Netflix Inc., see AUTHORS for more
+    :copyright: (c) 2018 by Netflix Inc., see AUTHORS for more
     :license: Apache, see LICENSE for more details.
 .. moduleauthor:: Kevin Glisson <kglisson@netflix.com>
 
 """
 import arrow
 
-from flask import current_app
-
 from sqlalchemy import func
 
 from lemur import database
+from lemur.common.utils import truthiness
 from lemur.endpoints.models import Endpoint, Policy, Cipher
 from lemur.extensions import metrics
 
@@ -60,6 +59,16 @@ def get_by_dnsname(dnsname):
     return database.get(Endpoint, dnsname, field='dnsname')
 
 
+def get_by_dnsname_and_port(dnsname, port):
+    """
+    Retrieves and endpoint by it's dnsname and port.
+    :param dnsname:
+    :param port:
+    :return:
+    """
+    return Endpoint.query.filter(Endpoint.dnsname == dnsname).filter(Endpoint.port == port).scalar()
+
+
 def get_by_source(source_label):
     """
     Retrieves all endpoints for a given source.
@@ -122,19 +131,6 @@ def update(endpoint_id, **kwargs):
     return endpoint
 
 
-def rotate_certificate(endpoint, new_cert):
-    """Rotates a certificate on a given endpoint."""
-    try:
-        endpoint.source.plugin.update_endpoint(endpoint, new_cert)
-        endpoint.certificate = new_cert
-        database.update(endpoint)
-        metrics.send('certificate_rotate_success', 'counter', 1, metric_tags={'endpoint': endpoint.name, 'source': endpoint.source.label})
-    except Exception as e:
-        metrics.send('certificate_rotate_failure', 'counter', 1, metric_tags={'endpoint': endpoint.name})
-        current_app.logger.exception(e)
-        raise e
-
-
 def render(args):
     """
     Helper that helps us render the REST Api responses.
@@ -147,7 +143,7 @@ def render(args):
     if filt:
         terms = filt.split(';')
         if 'active' in filt:  # this is really weird but strcmp seems to not work here??
-            query = query.filter(Endpoint.active == terms[1])
+            query = query.filter(Endpoint.active == truthiness(terms[1]))
         elif 'port' in filt:
             if terms[1] != 'null':  # ng-table adds 'null' if a number is removed
                 query = query.filter(Endpoint.port == terms[1])

lemur/endpoints/views.py

@@ -1,7 +1,7 @@
 """
 .. module: lemur.endpoints.views
     :platform: Unix
-    :copyright: (c) 2015 by Netflix Inc., see AUTHORS for more
+    :copyright: (c) 2018 by Netflix Inc., see AUTHORS for more
     :license: Apache, see LICENSE for more details.
 .. moduleauthor:: Kevin Glisson <kglisson@netflix.com>
 """

lemur/exceptions.py

@@ -1,6 +1,6 @@
 """
 .. module: lemur.exceptions
-    :copyright: (c) 2015 by Netflix Inc., see AUTHORS for more
+    :copyright: (c) 2018 by Netflix Inc., see AUTHORS for more
     :license: Apache, see LICENSE for more details.
 """
 from flask import current_app
@@ -29,8 +29,16 @@ class AttrNotFound(LemurException):
         self.field = field
 
     def __str__(self):
-        return repr("The field '{0}' is not sortable".format(self.field))
+        return repr("The field '{0}' is not sortable or filterable".format(self.field))
 
 
 class InvalidConfiguration(Exception):
     pass
+
+
+class InvalidAuthority(Exception):
+    pass
+
+
+class UnknownProvider(Exception):
+    pass

lemur/extensions.py

@@ -1,6 +1,6 @@
 """
 .. module: lemur.extensions
-    :copyright: (c) 2015 by Netflix Inc., see AUTHORS for more
+    :copyright: (c) 2018 by Netflix Inc., see AUTHORS for more
     :license: Apache, see LICENSE for more details.
 """
 from flask_sqlalchemy import SQLAlchemy
@@ -13,10 +13,19 @@ from flask_bcrypt import Bcrypt
 bcrypt = Bcrypt()
 
 from flask_principal import Principal
-principal = Principal()
+principal = Principal(use_sessions=False)
 
 from flask_mail import Mail
 smtp_mail = Mail()
 
 from lemur.metrics import Metrics
 metrics = Metrics()
+
+from raven.contrib.flask import Sentry
+sentry = Sentry()
+
+from blinker import Namespace
+signals = Namespace()
+
+from flask_cors import CORS
+cors = CORS()

lemur/factory.py

@@ -4,7 +4,7 @@
     :synopsis: This module contains all the needed functions to allow
     the factory app creation.
 
-    :copyright: (c) 2015 by Netflix Inc., see AUTHORS for more
+    :copyright: (c) 2018 by Netflix Inc., see AUTHORS for more
     :license: Apache, see LICENSE for more details.
 .. moduleauthor:: Kevin Glisson <kglisson@netflix.com>
 
@@ -18,8 +18,10 @@ from logging import Formatter, StreamHandler
 from logging.handlers import RotatingFileHandler
 
 from flask import Flask
+
+from lemur.certificates.hooks import activate_debug_dump
 from lemur.common.health import mod as health
-from lemur.extensions import db, migrate, principal, smtp_mail, metrics
+from lemur.extensions import db, migrate, principal, smtp_mail, metrics, sentry, cors
 
 
 DEFAULT_BLUEPRINTS = (
@@ -73,7 +75,8 @@ def from_file(file_path, silent=False):
     d.__file__ = file_path
     try:
         with open(file_path) as config_file:
-            exec(compile(config_file.read(), file_path, 'exec'), d.__dict__)
+            exec(compile(config_file.read(),  # nosec: config file safe
+                 file_path, 'exec'), d.__dict__)
     except IOError as e:
         if silent and e.errno in (errno.ENOENT, errno.EISDIR):
             return False
@@ -120,6 +123,11 @@ def configure_extensions(app):
     principal.init_app(app)
     smtp_mail.init_app(app)
     metrics.init_app(app)
+    sentry.init_app(app)
+
+    if app.config['CORS']:
+        app.config['CORS_HEADERS'] = 'Content-Type'
+        cors.init_app(app, resources=r'/api/*', headers='Content-Type', origin='*', supports_credentials=True)
 
 
 def configure_blueprints(app, blueprints):
@@ -152,9 +160,12 @@ def configure_logging(app):
     app.logger.addHandler(handler)
 
     stream_handler = StreamHandler()
-    stream_handler.setLevel(app.config.get('LOG_LEVEL'))
+    stream_handler.setLevel(app.config.get('LOG_LEVEL', 'DEBUG'))
     app.logger.addHandler(stream_handler)
 
+    if app.config.get('DEBUG_DUMP', False):
+        activate_debug_dump()
+
 
 def install_plugins(app):
     """
@@ -181,8 +192,10 @@ def install_plugins(app):
 
     # ensure that we have some way to notify
     with app.app_context():
+        slug = app.config.get("LEMUR_DEFAULT_NOTIFICATION_PLUGIN", "email-notification")
         try:
-            slug = app.config.get("LEMUR_DEFAULT_NOTIFICATION_PLUGIN", "email-notification")
             plugins.get(slug)
         except KeyError:
-            raise Exception("Unable to location notification plugin: {slug}. Ensure that LEMUR_DEFAULT_NOTIFICATION_PLUGIN is set to a valid and installed notification plugin.".format(slug=slug))
+            raise Exception("Unable to location notification plugin: {slug}. Ensure that "
+                            "LEMUR_DEFAULT_NOTIFICATION_PLUGIN is set to a valid and installed notification plugin."
+                            .format(slug=slug))

lemur/logs/models.py

@@ -2,7 +2,7 @@
 .. module: lemur.logs.models
     :platform: unix
     :synopsis: This module contains all of the models related private key audit log.
-    :copyright: (c) 2015 by Netflix Inc., see AUTHORS for more
+    :copyright: (c) 2018 by Netflix Inc., see AUTHORS for more
     :license: Apache, see LICENSE for more details.
 
 .. moduleauthor:: Kevin Glisson <kglisson@netflix.com>
@@ -18,6 +18,6 @@ class Log(db.Model):
     __tablename__ = 'logs'
     id = Column(Integer, primary_key=True)
     certificate_id = Column(Integer, ForeignKey('certificates.id'))
-    log_type = Column(Enum('key_view', name='log_type'), nullable=False)
+    log_type = Column(Enum('key_view', 'create_cert', 'update_cert', 'revoke_cert', name='log_type'), nullable=False)
     logged_at = Column(ArrowType(), PassiveDefault(func.now()), nullable=False)
     user_id = Column(Integer, ForeignKey('users.id'), nullable=False)

lemur/logs/schemas.py

@@ -1,7 +1,7 @@
 """
 .. module: lemur.logs.schemas
     :platform: unix
-    :copyright: (c) 2015 by Netflix Inc., see AUTHORS for more
+    :copyright: (c) 2018 by Netflix Inc., see AUTHORS for more
     :license: Apache, see LICENSE for more details.
 .. moduleauthor:: Kevin Glisson <kglisson@netflix.com>
 """

lemur/logs/service.py

@@ -3,12 +3,16 @@
     :platform: Unix
     :synopsis: This module contains all of the services level functions used to
     administer logs in Lemur
-    :copyright: (c) 2015 by Netflix Inc., see AUTHORS for more
+    :copyright: (c) 2018 by Netflix Inc., see AUTHORS for more
     :license: Apache, see LICENSE for more details.
 .. moduleauthor:: Kevin Glisson <kglisson@netflix.com>
 """
+from flask import current_app
+
 from lemur import database
 from lemur.logs.models import Log
+from lemur.users.models import User
+from lemur.certificates.models import Certificate
 
 
 def create(user, type, certificate=None):
@@ -20,6 +24,7 @@ def create(user, type, certificate=None):
     :param certificate:
     :return:
     """
+    current_app.logger.info("[lemur-audit] action: {0}, user: {1}, certificate: {2}.".format(type, user.email, certificate.name))
     view = Log(user_id=user.id, log_type=type, certificate_id=certificate.id)
     database.add(view)
     database.commit()
@@ -49,6 +54,20 @@ def render(args):
 
     if filt:
         terms = filt.split(';')
-        query = database.filter(query, Log, terms)
+
+        if 'certificate.name' in terms:
+            sub_query = database.session_query(Certificate.id)\
+                .filter(Certificate.name.ilike('%{0}%'.format(terms[1])))
+
+            query = query.filter(Log.certificate_id.in_(sub_query))
+
+        elif 'user.email' in terms:
+            sub_query = database.session_query(User.id)\
+                .filter(User.email.ilike('%{0}%'.format(terms[1])))
+
+            query = query.filter(Log.user_id.in_(sub_query))
+
+        else:
+            query = database.filter(query, Log, terms)
 
     return database.sort_and_page(query, Log, args)

lemur/logs/views.py

@@ -1,7 +1,7 @@
 """
 .. module: lemur.logs.views
     :platform: Unix
-    :copyright: (c) 2015 by Netflix Inc., see AUTHORS for more
+    :copyright: (c) 2018 by Netflix Inc., see AUTHORS for more
     :license: Apache, see LICENSE for more details.
 .. moduleauthor:: Kevin Glisson <kglisson@netflix.com>
 """

lemur/manage.py

@@ -16,16 +16,19 @@ from flask_migrate import Migrate, MigrateCommand, stamp
 from flask_script.commands import ShowUrls, Clean, Server
 
 from lemur.sources.cli import manager as source_manager
+from lemur.policies.cli import manager as policy_manager
+from lemur.reporting.cli import manager as report_manager
+from lemur.endpoints.cli import manager as endpoint_manager
 from lemur.certificates.cli import manager as certificate_manager
 from lemur.notifications.cli import manager as notification_manager
-from lemur.endpoints.cli import manager as endpoint_manager
-from lemur.reporting.cli import manager as report_manager
+from lemur.pending_certificates.cli import manager as pending_certificate_manager
+
 from lemur import database
 from lemur.users import service as user_service
 from lemur.roles import service as role_service
+from lemur.policies import service as policy_service
 from lemur.notifications import service as notification_service
 
-
 from lemur.common.utils import validate_conf
 
 from lemur import create_app
@@ -40,6 +43,9 @@ from lemur.domains.models import Domain  # noqa
 from lemur.notifications.models import Notification  # noqa
 from lemur.sources.models import Source  # noqa
 from lemur.logs.models import Log  # noqa
+from lemur.endpoints.models import Endpoint  # noqa
+from lemur.policies.models import RotationPolicy  # noqa
+from lemur.pending_certificates.models import PendingCertificate  # noqa
 
 
 manager = Manager(create_app)
@@ -83,8 +89,8 @@ SECRET_KEY = '{flask_secret_key}'
 LEMUR_TOKEN_SECRET = '{secret_token}'
 LEMUR_ENCRYPTION_KEYS = '{encryption_key}'
 
-# this is a list of domains as regexes that only admins can issue
-LEMUR_RESTRICTED_DOMAINS = []
+# List of domain regular expressions that non-admin users can issue
+LEMUR_WHITELISTED_DOMAINS = []
 
 # Mail Server
 
@@ -102,6 +108,9 @@ LEMUR_DEFAULT_ORGANIZATIONAL_UNIT = ''
 # Authentication Providers
 ACTIVE_PROVIDERS = []
 
+# Metrics Providers
+METRIC_PROVIDERS = []
+
 # Logging
 
 LOG_LEVEL = "DEBUG"
@@ -199,16 +208,16 @@ class InitializeApp(Command):
         if operator_role:
             sys.stdout.write("[-] Operator role already created, skipping...!\n")
         else:
-            # we create an admin role
+            # we create an operator role
             operator_role = role_service.create('operator', description='This is the Lemur operator role.')
             sys.stdout.write("[+] Created 'operator' role\n")
 
         read_only_role = role_service.get_by_name('read-only')
 
         if read_only_role:
-            sys.stdout.write("[-] Operator role already created, skipping...!\n")
+            sys.stdout.write("[-] Read only role already created, skipping...!\n")
         else:
-            # we create an admin role
+            # we create an read only role
             read_only_role = role_service.create('read-only', description='This is the Lemur read only role.')
             sys.stdout.write("[+] Created 'read-only' role\n")
 
@@ -222,15 +231,12 @@ class InitializeApp(Command):
                     sys.stderr.write("[!] Passwords do not match!\n")
                     sys.exit(1)
 
-            user_service.create("lemur", password, 'lemur@nobody', True, None, [admin_role])
+            user_service.create("lemur", password, 'lemur@nobody.com', True, None, [admin_role])
             sys.stdout.write("[+] Created the user 'lemur' and granted it the 'admin' role!\n")
 
         else:
             sys.stdout.write("[-] Default user has already been created, skipping...!\n")
 
-        sys.stdout.write("[+] Creating expiration email notifications!\n")
-        sys.stdout.write("[!] Using {0} as specified by LEMUR_SECURITY_TEAM_EMAIL for notifications\n".format("LEMUR_SECURITY_TEAM_EMAIL"))
-
         intervals = current_app.config.get("LEMUR_DEFAULT_EXPIRATION_NOTIFICATION_INTERVALS", [])
         sys.stdout.write(
             "[!] Creating {num} notifications for {intervals} days as specified by LEMUR_DEFAULT_EXPIRATION_NOTIFICATION_INTERVALS\n".format(
@@ -240,8 +246,21 @@ class InitializeApp(Command):
         )
 
         recipients = current_app.config.get('LEMUR_SECURITY_TEAM_EMAIL')
+        sys.stdout.write("[+] Creating expiration email notifications!\n")
+        sys.stdout.write("[!] Using {0} as specified by LEMUR_SECURITY_TEAM_EMAIL for notifications\n".format(recipients))
         notification_service.create_default_expiration_notifications("DEFAULT_SECURITY", recipients=recipients)
 
+        _DEFAULT_ROTATION_INTERVAL = 'default'
+        default_rotation_interval = policy_service.get_by_name(_DEFAULT_ROTATION_INTERVAL)
+
+        if default_rotation_interval:
+            sys.stdout.write("[-] Default rotation interval policy already created, skipping...!\n")
+        else:
+            days = current_app.config.get("LEMUR_DEFAULT_ROTATION_INTERVAL", 30)
+            sys.stdout.write("[+] Creating default certificate rotation policy of {days} days before issuance.\n".format(
+                days=days))
+            policy_service.create(days=days, name=_DEFAULT_ROTATION_INTERVAL)
+
         sys.stdout.write("[/] Done!\n")
 
 
@@ -365,7 +384,7 @@ class LemurServer(Command):
 
         app = WSGIApplication()
 
-        # run startup tasks on a app like object
+        # run startup tasks on an app like object
         validate_conf(current_app, REQUIRED_VARIABLES)
 
         app.app_uri = 'lemur:create_app(config="{0}")'.format(current_app.config.get('CONFIG_PATH'))
@@ -503,19 +522,6 @@ def publish_verisign_units():
         requests.post('http://localhost:8078/metrics', data=json.dumps(metric))
 
 
-@manager.command
-def publish_unapproved_verisign_certificates():
-    """
-    Query the Verisign for any certificates that need to be approved.
-    :return:
-    """
-    from lemur.plugins import plugins
-    from lemur.extensions import metrics
-    v = plugins.get('verisign-issuer')
-    certs = v.get_pending_certificates()
-    metrics.send('pending_certificates', 'gauge', certs)
-
-
 def main():
     manager.add_command("start", LemurServer())
     manager.add_command("runserver", Server(host='127.0.0.1', threaded=True))
@@ -531,6 +537,8 @@ def main():
     manager.add_command("notify", notification_manager)
     manager.add_command("endpoint", endpoint_manager)
     manager.add_command("report", report_manager)
+    manager.add_command("policy", policy_manager)
+    manager.add_command("pending_certs", pending_certificate_manager)
     manager.run()
 
 

lemur/metrics.py

@@ -1,6 +1,6 @@
 """
 .. module: lemur.metrics
-    :copyright: (c) 2015 by Netflix Inc., see AUTHORS for more
+    :copyright: (c) 2018 by Netflix Inc., see AUTHORS for more
     :license: Apache, see LICENSE for more details.
 """
 from flask import current_app

lemur/migrations/env.py

@@ -3,6 +3,9 @@ from alembic import context
 from sqlalchemy import engine_from_config, pool
 from logging.config import fileConfig
 
+import alembic_autogenerate_enums
+
+
 # this is the Alembic Config object, which provides
 # access to the values within the .ini file in use.
 config = context.config

lemur/migrations/versions/131ec6accff5_.py

@@ -16,7 +16,7 @@ import sqlalchemy as sa
 
 def upgrade():
     # ### commands auto generated by Alembic - please adjust! ###
-    op.add_column('certificates', sa.Column('rotation', sa.Boolean(), nullable=False, server_default=False))
+    op.add_column('certificates', sa.Column('rotation', sa.Boolean(), nullable=False, server_default=sa.false()))
     op.add_column('endpoints', sa.Column('last_updated', sa.DateTime(), server_default=sa.text('now()'), nullable=False))
     # ### end Alembic commands ###
 

lemur/migrations/versions/1ae8e3104db8_.py

@@ -0,0 +1,21 @@
+"""Adds additional ENUM for creating and updating certificates.
+
+Revision ID: 1ae8e3104db8
+Revises: a02a678ddc25
+Create Date: 2017-07-13 12:32:09.162800
+
+"""
+
+# revision identifiers, used by Alembic.
+revision = '1ae8e3104db8'
+down_revision = 'a02a678ddc25'
+
+from alembic import op
+
+
+def upgrade():
+    op.sync_enum_values('public', 'log_type', ['key_view'], ['create_cert', 'key_view', 'update_cert'])
+
+
+def downgrade():
+    op.sync_enum_values('public', 'log_type', ['create_cert', 'key_view', 'update_cert'], ['key_view'])

lemur/migrations/versions/3adfdd6598df_.py

@@ -0,0 +1,105 @@
+"""Create tables and columns for the acme issuer.
+
+Revision ID: 3adfdd6598df
+Revises: 556ceb3e3c3e
+Create Date: 2018-04-10 13:25:47.007556
+
+"""
+
+# revision identifiers, used by Alembic.
+revision = '3adfdd6598df'
+down_revision = '556ceb3e3c3e'
+
+import sqlalchemy as sa
+from alembic import op
+from sqlalchemy.dialects.postgresql import JSON
+from sqlalchemy_utils import ArrowType
+
+from lemur.utils import Vault
+
+
+def upgrade():
+    # create provider table
+    print("Creating dns_providers table")
+    op.create_table(
+        'dns_providers',
+        sa.Column('id', sa.Integer(), nullable=False),
+        sa.Column('name', sa.String(length=256), nullable=True),
+        sa.Column('description', sa.String(length=1024), nullable=True),
+        sa.Column('provider_type', sa.String(length=256), nullable=True),
+        sa.Column('credentials', Vault(), nullable=True),
+        sa.Column('api_endpoint', sa.String(length=256), nullable=True),
+        sa.Column('date_created', ArrowType(), server_default=sa.text('now()'), nullable=False),
+        sa.Column('status', sa.String(length=128), nullable=True),
+        sa.Column('options', JSON),
+        sa.Column('domains', sa.JSON(), nullable=True),
+        sa.PrimaryKeyConstraint('id'),
+        sa.UniqueConstraint('name')
+    )
+    print("Adding dns_provider_id column to certificates")
+    op.add_column('certificates', sa.Column('dns_provider_id', sa.Integer(), nullable=True))
+    print("Adding dns_provider_id column to pending_certs")
+    op.add_column('pending_certs', sa.Column('dns_provider_id', sa.Integer(), nullable=True))
+    print("Adding options column to pending_certs")
+    op.add_column('pending_certs', sa.Column('options', JSON))
+
+    print("Creating pending_dns_authorizations table")
+    op.create_table(
+        'pending_dns_authorizations',
+        sa.Column('id', sa.Integer(), primary_key=True, autoincrement=True),
+        sa.Column('account_number', sa.String(length=128), nullable=True),
+        sa.Column('domains', JSON, nullable=True),
+        sa.Column('dns_provider_type', sa.String(length=128), nullable=True),
+        sa.Column('options', JSON, nullable=True),
+    )
+
+    print("Creating certificates_dns_providers_fk foreign key")
+    op.create_foreign_key('certificates_dns_providers_fk', 'certificates', 'dns_providers', ['dns_provider_id'], ['id'],
+                          ondelete='cascade')
+
+    print("Altering column types in the api_keys table")
+    op.alter_column('api_keys', 'issued_at',
+                    existing_type=sa.BIGINT(),
+                    nullable=True)
+    op.alter_column('api_keys', 'revoked',
+                    existing_type=sa.BOOLEAN(),
+                    nullable=True)
+    op.alter_column('api_keys', 'ttl',
+                    existing_type=sa.BIGINT(),
+                    nullable=True)
+    op.alter_column('api_keys', 'user_id',
+                    existing_type=sa.INTEGER(),
+                    nullable=True)
+
+    print("Creating dns_providers_id foreign key on pending_certs table")
+    op.create_foreign_key(None, 'pending_certs', 'dns_providers', ['dns_provider_id'], ['id'], ondelete='CASCADE')
+
+def downgrade():
+    print("Removing dns_providers_id foreign key on pending_certs table")
+    op.drop_constraint(None, 'pending_certs', type_='foreignkey')
+    print("Reverting column types in the api_keys table")
+    op.alter_column('api_keys', 'user_id',
+                    existing_type=sa.INTEGER(),
+                    nullable=False)
+    op.alter_column('api_keys', 'ttl',
+                    existing_type=sa.BIGINT(),
+                    nullable=False)
+    op.alter_column('api_keys', 'revoked',
+                    existing_type=sa.BOOLEAN(),
+                    nullable=False)
+    op.alter_column('api_keys', 'issued_at',
+                    existing_type=sa.BIGINT(),
+                    nullable=False)
+    print("Reverting certificates_dns_providers_fk foreign key")
+    op.drop_constraint('certificates_dns_providers_fk', 'certificates', type_='foreignkey')
+
+    print("Dropping pending_dns_authorizations table")
+    op.drop_table('pending_dns_authorizations')
+    print("Undoing modifications to pending_certs table")
+    op.drop_column('pending_certs', 'options')
+    op.drop_column('pending_certs', 'dns_provider_id')
+    print("Undoing modifications to certificates table")
+    op.drop_column('certificates', 'dns_provider_id')
+
+    print("Deleting dns_providers table")
+    op.drop_table('dns_providers')

lemur/migrations/versions/449c3d5c7299_.py

@@ -0,0 +1,28 @@
+"""Add unique constraint to certificate_notification_associations table.
+
+Revision ID: 449c3d5c7299
+Revises: 5770674184de
+Create Date: 2018-02-24 22:51:35.369229
+
+"""
+
+# revision identifiers, used by Alembic.
+revision = '449c3d5c7299'
+down_revision = '5770674184de'
+
+from alembic import op
+from flask_sqlalchemy import SQLAlchemy
+
+db = SQLAlchemy()
+
+CONSTRAINT_NAME = "uq_dest_not_ids"
+TABLE = "certificate_notification_associations"
+COLUMNS = ["notification_id", "certificate_id"]
+
+
+def upgrade():
+    op.create_unique_constraint(CONSTRAINT_NAME, TABLE, COLUMNS)
+
+
+def downgrade():
+    op.drop_constraint(CONSTRAINT_NAME, TABLE)

lemur/migrations/versions/556ceb3e3c3e_.py

@@ -0,0 +1,99 @@
+"""Add Pending Certificates models and relations
+
+Revision ID: 556ceb3e3c3e
+Revises: 47baffaae1a7
+Create Date: 2018-01-05 01:18:45.571595
+
+"""
+
+# revision identifiers, used by Alembic.
+revision = '556ceb3e3c3e'
+down_revision = '449c3d5c7299'
+
+from alembic import op
+import sqlalchemy as sa
+from lemur.utils import Vault
+from sqlalchemy.dialects import postgresql
+from sqlalchemy_utils import ArrowType
+
+def upgrade():
+    # ### commands auto generated by Alembic - please adjust! ###
+    op.create_table('pending_certs',
+    sa.Column('id', sa.Integer(), nullable=False),
+    sa.Column('external_id', sa.String(length=128), nullable=True),
+    sa.Column('owner', sa.String(length=128), nullable=False),
+    sa.Column('name', sa.String(length=256), nullable=True),
+    sa.Column('description', sa.String(length=1024), nullable=True),
+    sa.Column('notify', sa.Boolean(), nullable=True),
+    sa.Column('number_attempts', sa.Integer(), nullable=True),
+    sa.Column('rename', sa.Boolean(), nullable=True),
+    sa.Column('cn', sa.String(length=128), nullable=True),
+    sa.Column('csr', sa.Text(), nullable=False),
+    sa.Column('chain', sa.Text(), nullable=True),
+    sa.Column('private_key', Vault(), nullable=True),
+    sa.Column('date_created', ArrowType(), server_default=sa.text('now()'), nullable=False),
+    sa.Column('status', sa.String(length=128), nullable=True),
+    sa.Column('rotation', sa.Boolean(), nullable=True),
+    sa.Column('user_id', sa.Integer(), nullable=True),
+    sa.Column('authority_id', sa.Integer(), nullable=True),
+    sa.Column('root_authority_id', sa.Integer(), nullable=True),
+    sa.Column('rotation_policy_id', sa.Integer(), nullable=True),
+    sa.ForeignKeyConstraint(['authority_id'], ['authorities.id'], ondelete='CASCADE'),
+    sa.ForeignKeyConstraint(['root_authority_id'], ['authorities.id'], ondelete='CASCADE'),
+    sa.ForeignKeyConstraint(['rotation_policy_id'], ['rotation_policies.id'], ),
+    sa.ForeignKeyConstraint(['user_id'], ['users.id'], ),
+    sa.PrimaryKeyConstraint('id'),
+    sa.UniqueConstraint('name')
+    )
+    op.create_table('pending_cert_destination_associations',
+    sa.Column('destination_id', sa.Integer(), nullable=True),
+    sa.Column('pending_cert_id', sa.Integer(), nullable=True),
+    sa.ForeignKeyConstraint(['destination_id'], ['destinations.id'], ondelete='cascade'),
+    sa.ForeignKeyConstraint(['pending_cert_id'], ['pending_certs.id'], ondelete='cascade')
+    )
+    op.create_index('pending_cert_destination_associations_ix', 'pending_cert_destination_associations', ['destination_id', 'pending_cert_id'], unique=False)
+    op.create_table('pending_cert_notification_associations',
+    sa.Column('notification_id', sa.Integer(), nullable=True),
+    sa.Column('pending_cert_id', sa.Integer(), nullable=True),
+    sa.ForeignKeyConstraint(['notification_id'], ['notifications.id'], ondelete='cascade'),
+    sa.ForeignKeyConstraint(['pending_cert_id'], ['pending_certs.id'], ondelete='cascade')
+    )
+    op.create_index('pending_cert_notification_associations_ix', 'pending_cert_notification_associations', ['notification_id', 'pending_cert_id'], unique=False)
+    op.create_table('pending_cert_replacement_associations',
+    sa.Column('replaced_certificate_id', sa.Integer(), nullable=True),
+    sa.Column('pending_cert_id', sa.Integer(), nullable=True),
+    sa.ForeignKeyConstraint(['pending_cert_id'], ['pending_certs.id'], ondelete='cascade'),
+    sa.ForeignKeyConstraint(['replaced_certificate_id'], ['certificates.id'], ondelete='cascade')
+    )
+    op.create_index('pending_cert_replacement_associations_ix', 'pending_cert_replacement_associations', ['replaced_certificate_id', 'pending_cert_id'], unique=False)
+    op.create_table('pending_cert_role_associations',
+    sa.Column('pending_cert_id', sa.Integer(), nullable=True),
+    sa.Column('role_id', sa.Integer(), nullable=True),
+    sa.ForeignKeyConstraint(['pending_cert_id'], ['pending_certs.id'], ),
+    sa.ForeignKeyConstraint(['role_id'], ['roles.id'], )
+    )
+    op.create_index('pending_cert_role_associations_ix', 'pending_cert_role_associations', ['pending_cert_id', 'role_id'], unique=False)
+    op.create_table('pending_cert_source_associations',
+    sa.Column('source_id', sa.Integer(), nullable=True),
+    sa.Column('pending_cert_id', sa.Integer(), nullable=True),
+    sa.ForeignKeyConstraint(['pending_cert_id'], ['pending_certs.id'], ondelete='cascade'),
+    sa.ForeignKeyConstraint(['source_id'], ['sources.id'], ondelete='cascade')
+    )
+    op.create_index('pending_cert_source_associations_ix', 'pending_cert_source_associations', ['source_id', 'pending_cert_id'], unique=False)
+    # ### end Alembic commands ###
+
+
+def downgrade():
+    # ### commands auto generated by Alembic - please adjust! ###
+    op.drop_index('pending_cert_source_associations_ix', table_name='pending_cert_source_associations')
+    op.drop_table('pending_cert_source_associations')
+    op.drop_index('pending_cert_role_associations_ix', table_name='pending_cert_role_associations')
+    op.drop_table('pending_cert_role_associations')
+    op.drop_index('pending_cert_replacement_associations_ix', table_name='pending_cert_replacement_associations')
+    op.drop_table('pending_cert_replacement_associations')
+    op.drop_index('pending_cert_notification_associations_ix', table_name='pending_cert_notification_associations')
+    op.drop_table('pending_cert_notification_associations')
+    op.drop_index('pending_cert_destination_associations_ix', table_name='pending_cert_destination_associations')
+    op.drop_table('pending_cert_destination_associations')
+    op.drop_table('pending_certs')
+    # ### end Alembic commands ###

lemur/migrations/versions/5770674184de_.py

@@ -0,0 +1,44 @@
+"""Remove duplicates from certificate_notification_associations.
+
+Revision ID: 5770674184de
+Revises: ce547319f7be
+Create Date: 2018-02-23 15:27:30.335435
+
+"""
+
+# revision identifiers, used by Alembic.
+revision = '5770674184de'
+down_revision = 'ce547319f7be'
+
+from flask_sqlalchemy import SQLAlchemy
+from lemur.models import certificate_notification_associations
+
+db = SQLAlchemy()
+session = db.session()
+
+
+def upgrade():
+    print("Querying for all entries in certificate_notification_associations.")
+    # Query for all entries in table
+    results = session.query(certificate_notification_associations).with_entities(
+        certificate_notification_associations.c.certificate_id,
+        certificate_notification_associations.c.notification_id,
+        certificate_notification_associations.c.id,
+    )
+
+    seen = {}
+    # Iterate through all entries and mark as seen for each certificate_id and notification_id pair
+    for x in results:
+        # If we've seen a pair already, delete the duplicates
+        if seen.get("{}-{}".format(x.certificate_id, x.notification_id)):
+            print("Deleting duplicate: {}".format(x))
+            d = session.query(certificate_notification_associations).filter(certificate_notification_associations.c.id==x.id)
+            d.delete(synchronize_session=False)
+        seen["{}-{}".format(x.certificate_id, x.notification_id)] = True
+    db.session.commit()
+    db.session.flush()
+
+
+def downgrade():
+    # No way to downgrade this
+    pass

lemur/migrations/versions/5bc47fa7cac4_.py

@@ -0,0 +1,22 @@
+"""add third party roles to lemur
+
+Revision ID: 5bc47fa7cac4
+Revises: c05a8998b371
+Create Date: 2017-12-08 14:19:11.903864
+
+"""
+
+# revision identifiers, used by Alembic.
+revision = '5bc47fa7cac4'
+down_revision = 'c05a8998b371'
+
+from alembic import op
+import sqlalchemy as sa
+
+
+def upgrade():
+    op.add_column('roles', sa.Column('third_party', sa.Boolean(), nullable=True, default=False))
+
+
+def downgrade():
+    op.drop_column('roles', 'third_party')

lemur/migrations/versions/5e680529b666_.py

@@ -15,16 +15,12 @@ import sqlalchemy as sa
 
 
 def upgrade():
-    ### commands auto generated by Alembic - please adjust! ###
     op.add_column('endpoints', sa.Column('sensitive', sa.Boolean(), nullable=True))
     op.add_column('endpoints', sa.Column('source_id', sa.Integer(), nullable=True))
     op.create_foreign_key(None, 'endpoints', 'sources', ['source_id'], ['id'])
-    ### end Alembic commands ###
 
 
 def downgrade():
-    ### commands auto generated by Alembic - please adjust! ###
     op.drop_constraint(None, 'endpoints', type_='foreignkey')
     op.drop_column('endpoints', 'source_id')
     op.drop_column('endpoints', 'sensitive')
-    ### end Alembic commands ###

lemur/migrations/versions/8ae67285ff14_.py

@@ -0,0 +1,24 @@
+"""Modifies the unique index for the certificate replacements
+
+Revision ID: 8ae67285ff14
+Revises: 5e680529b666
+Create Date: 2017-05-10 11:56:13.999332
+
+"""
+
+# revision identifiers, used by Alembic.
+revision = '8ae67285ff14'
+down_revision = '5e680529b666'
+
+from alembic import op
+import sqlalchemy as sa
+
+
+def upgrade():
+    op.drop_index('certificate_replacement_associations_ix')
+    op.create_index('certificate_replacement_associations_ix', 'certificate_replacement_associations', ['replaced_certificate_id', 'certificate_id'], unique=True)
+
+
+def downgrade():
+    op.drop_index('certificate_replacement_associations_ix')
+    op.create_index('certificate_replacement_associations_ix', 'certificate_replacement_associations', ['certificate_id', 'certificate_id'], unique=True)

lemur/migrations/versions/a02a678ddc25_.py

@@ -0,0 +1,54 @@
+"""Adds support for rotation policies.
+
+Creates a default rotation policy (30 days) with the name
+'default' ensures that all existing certificates use the default
+policy.
+
+Revision ID: a02a678ddc25
+Revises: 8ae67285ff14
+Create Date: 2017-07-12 11:45:49.257927
+"""
+
+# revision identifiers, used by Alembic.
+revision = 'a02a678ddc25'
+down_revision = '8ae67285ff14'
+
+from alembic import op
+import sqlalchemy as sa
+from sqlalchemy.sql import text
+
+
+def upgrade():
+    # ### commands auto generated by Alembic - please adjust! ###
+    op.create_table('rotation_policies',
+        sa.Column('id', sa.Integer(), nullable=False),
+        sa.Column('name', sa.String(), nullable=True),
+        sa.Column('days', sa.Integer(), nullable=True),
+        sa.PrimaryKeyConstraint('id')
+    )
+    op.add_column('certificates', sa.Column('rotation_policy_id', sa.Integer(), nullable=True))
+    op.create_foreign_key(None, 'certificates', 'rotation_policies', ['rotation_policy_id'], ['id'])
+
+    conn = op.get_bind()
+    stmt = text('insert into rotation_policies (days, name) values (:days, :name)')
+    stmt = stmt.bindparams(days=30, name='default')
+    conn.execute(stmt)
+
+    stmt = text('select id from rotation_policies where name=:name')
+    stmt = stmt.bindparams(name='default')
+    rotation_policy_id = conn.execute(stmt).fetchone()[0]
+
+    stmt = text('update certificates set rotation_policy_id=:rotation_policy_id')
+    stmt = stmt.bindparams(rotation_policy_id=rotation_policy_id)
+    conn.execute(stmt)
+    # ### end Alembic commands ###
+
+
+def downgrade():
+    # ### commands auto generated by Alembic - please adjust! ###
+    op.drop_constraint(None, 'certificates', type_='foreignkey')
+    op.drop_column('certificates', 'rotation_policy_id')
+    op.drop_index('certificate_replacement_associations_ix', table_name='certificate_replacement_associations')
+    op.create_index('certificate_replacement_associations_ix', 'certificate_replacement_associations', ['replaced_certificate_id', 'certificate_id'], unique=True)
+    op.drop_table('rotation_policies')
+    # ### end Alembic commands ###