Merge branch 'entrust-plugin' of github.com:sirferl/lemur into azure-plugin

docs/doing-a-release.rst

@@ -11,22 +11,47 @@ software.
 
 * Update the version number in ``lemur/__about__.py``.
 * Set the release date in the :doc:`/changelog`.
-* Do a commit indicating this.
+* Do a commit indicating this, and raise a pull request with this.
-* Send a pull request with this.
 * Wait for it to be merged.
 
 Performing the release
 ----------------------
 
 The commit that merged the version number bump is now the official release
-commit for this release. You will need to have ``gpg`` installed and a ``gpg``
+commit for this release. You need an `API key <https://pypi.org/manage/account/#api-tokens>`_,
-key in order to do a release. Once this has happened:
+which requires permissions to maintain the Lemur `project  <https://pypi.org/project/lemur/>`_.
 
-* Run ``invoke release {version}``.
+For creating the release, follow these steps (more details `here <https://packaging.python.org/tutorials/packaging-projects/#generating-distribution-archives>`_)
 
-The release should now be available on PyPI and a tag should be available in
+* Make sure you have the latest versions of setuptools and wheel installed:
+
+    ``python3 -m pip install --user --upgrade setuptools wheel``
+
+* Now run this command from the same directory where setup.py is located:
+
+    ``python3 setup.py sdist bdist_wheel``
+
+* Once completed it should generate two files in the dist directory:
+
+.. code-block:: pycon
+
+    $ ls dist/
+    lemur-0.8.0-py2.py3-none-any.whl	lemur-0.8.0.tar.gz
+
+
+* In this step, the distribution will be uploaded. You’ll need to install Twine:
+
+    ``python3 -m pip install --user --upgrade twine``
+
+* Once installed, run Twine to upload all of the archives under dist. Once installed, run Twine to upload all of the archives under dist:
+
+    ``python3 -m twine upload --repository pypi dist/*``
+
+The release should now be available on `PyPI Lemur <https://pypi.org/project/lemur/>`_ and a tag should be available in
 the repository.
 
+Make sure to also make a github `release <https://github.com/Netflix/lemur/releases>`_ which will pick up the latest version.
+
 Verifying the release
 ---------------------
 

docs/production/index.rst

@@ -415,8 +415,8 @@ And the worker can be started with desired options such as the following::
 
 supervisor or systemd configurations should be created for these in production environments as appropriate.
 
-Add support for LetsEncrypt
+Add support for LetsEncrypt/ACME
-===========================
+================================
 
 LetsEncrypt is a free, limited-feature certificate authority that offers publicly trusted certificates that are valid
 for 90 days. LetsEncrypt does not use organizational validation (OV), and instead relies on domain validation (DV).
@@ -424,7 +424,10 @@ LetsEncrypt requires that we prove ownership of a domain before we're able to is
 time we want a certificate.
 
 The most common methods to prove ownership are HTTP validation and DNS validation. Lemur supports DNS validation
-through the creation of DNS TXT records.
+through the creation of DNS TXT records as well as HTTP validation, reusing the destination concept.
+
+ACME DNS Challenge
+------------------
 
 In a nutshell, when we send a certificate request to LetsEncrypt, they generate a random token and ask us to put that
 token in a DNS text record to prove ownership of a domain. If a certificate request has multiple domains, we must
@@ -462,6 +465,24 @@ possible. To enable this functionality, periodically (or through Cron/Celery) ru
 This command will traverse all DNS providers, determine which zones they control, and upload this list of zones to
 Lemur's database (in the dns_providers table). Alternatively, you can manually input this data.
 
+ACME HTTP Challenge
+-------------------
+
+The flow for requesting a certificate using the HTTP challenge is not that different from the one described for the DNS
+challenge. The only difference is, that instead of creating a DNS TXT record, a file is uploaded to a Webserver which
+serves the file at `http://<domain>/.well-known/acme-challenge/<token>`
+
+Currently the HTTP challenge also works without Celery, since it's done while creating the certificate, and doesn't
+rely on celery to create the DNS record. This will change when we implement mix & match of acme challenge types.
+
+To create a HTTP compatible Authority, you first need to create a new destination that will be used to deploy the
+challenge token. Visit `Admin` -> `Destination` and click `Create`. The path you provide for the destination needs to
+be the exact path that is called when the ACME providers calls ``http://<domain>/.well-known/acme-challenge/`. The
+token part will be added dynamically by the acme_upload.
+Currently only the SFTP and S3 Bucket destination support the ACME HTTP challenge.
+
+Afterwards you can create a new certificate authority as described in the DNS challenge, but need to choose
+`Acme HTTP-01` as the plugin type, and then the destination you created beforehand.
 
 LetsEncrypt: pinning to cross-signed ICA
 ----------------------------------------

lemur/notifications/messaging.py

@@ -103,8 +103,9 @@ def send_plugin_notification(event_type, data, recipients, notification):
     function = f"{__name__}.{sys._getframe().f_code.co_name}"
     log_data = {
         "function": function,
-        "message": f"Sending expiration notification for to recipients {recipients}",
+        "message": f"Sending {event_type} notification for to recipients {recipients}",
-        "notification_type": "expiration",
+        "notification_type": event_type,
+        "notification_plugin": notification.plugin.slug,
         "certificate_targets": recipients,
     }
     status = FAILURE_METRIC_STATUS
@@ -121,7 +122,7 @@ def send_plugin_notification(event_type, data, recipients, notification):
         "notification",
         "counter",
         1,
-        metric_tags={"status": status, "event_type": event_type},
+        metric_tags={"status": status, "event_type": event_type, "plugin": notification.plugin.slug},
     )
 
     if status == SUCCESS_METRIC_STATUS:
@@ -142,7 +143,6 @@ def send_expiration_notifications(exclude):
 
         for notification_label, certificates in notification_group.items():
             notification_data = []
-            security_data = []
 
             notification = certificates[0][0]
 
@@ -152,33 +152,26 @@ def send_expiration_notifications(exclude):
                     certificate
                 ).data
                 notification_data.append(cert_data)
-                security_data.append(cert_data)
-
-            if send_default_notification(
-                    "expiration", notification_data, [owner], notification.options
-            ):
-                success += 1
-            else:
-                failure += 1
-
-            recipients = notification.plugin.filter_recipients(notification.options, security_email + [owner])
 
+            email_recipients = notification.plugin.get_recipients(notification.options, security_email + [owner])
+            # Plugin will ONLY use the provided recipients if it's email; any other notification plugin ignores them
             if send_plugin_notification(
-                "expiration",
+                    "expiration", notification_data, email_recipients, notification
-                notification_data,
-                recipients,
-                notification,
             ):
-                success += 1
+                success += len(email_recipients)
             else:
-                failure += 1
+                failure += len(email_recipients)
-
+            # If we're using an email plugin, we're done,
+            #   since "security_email + [owner]" were added as email_recipients.
+            # If we're not using an email plugin, we also need to send an email to the security team and owner,
+            #   since the plugin notification didn't send anything to them.
+            if notification.plugin.slug != "email-notification":
                 if send_default_notification(
-                "expiration", security_data, security_email, notification.options
+                        "expiration", notification_data, email_recipients, notification.options
                 ):
-                success += 1
+                    success = 1 + len(email_recipients)
                 else:
-                failure += 1
+                    failure = 1 + len(email_recipients)
 
     return success, failure
 
@@ -195,15 +188,16 @@ def send_default_notification(notification_type, data, targets, notification_opt
     :return:
     """
     function = f"{__name__}.{sys._getframe().f_code.co_name}"
-    log_data = {
-        "function": function,
-        "message": f"Sending notification for certificate data {data}",
-        "notification_type": notification_type,
-    }
     status = FAILURE_METRIC_STATUS
     notification_plugin = plugins.get(
         current_app.config.get("LEMUR_DEFAULT_NOTIFICATION_PLUGIN", "email-notification")
     )
+    log_data = {
+        "function": function,
+        "message": f"Sending {notification_type} notification for certificate data {data} to targets {targets}",
+        "notification_type": notification_type,
+        "notification_plugin": notification_plugin.slug,
+    }
 
     try:
         current_app.logger.debug(log_data)
@@ -212,7 +206,7 @@ def send_default_notification(notification_type, data, targets, notification_opt
         status = SUCCESS_METRIC_STATUS
     except Exception as e:
         log_data["message"] = f"Unable to send {notification_type} notification for certificate data {data} " \
-                              f"to target {targets}"
+                              f"to targets {targets}"
         current_app.logger.error(log_data, exc_info=True)
         sentry.captureException()
 
@@ -220,7 +214,7 @@ def send_default_notification(notification_type, data, targets, notification_opt
         "notification",
         "counter",
         1,
-        metric_tags={"status": status, "event_type": notification_type},
+        metric_tags={"status": status, "event_type": notification_type, "plugin": notification_plugin.slug},
     )
 
     if status == SUCCESS_METRIC_STATUS:
@@ -247,15 +241,14 @@ def send_pending_failure_notification(
     data = pending_certificate_output_schema.dump(pending_cert).data
     data["security_email"] = current_app.config.get("LEMUR_SECURITY_TEAM_EMAIL")
 
-    notify_owner_success = False
+    email_recipients = []
     if notify_owner:
-        notify_owner_success = send_default_notification("failed", data, [data["owner"]], pending_cert)
+        email_recipients = email_recipients + [data["owner"]]
 
-    notify_security_success = False
     if notify_security:
-        notify_security_success = send_default_notification("failed", data, data["security_email"], pending_cert)
+        email_recipients = email_recipients + data["security_email"]
 
-    return notify_owner_success or notify_security_success
+    return send_default_notification("failed", data, email_recipients, pending_cert)
 
 
 def needs_notification(certificate):

lemur/plugins/bases/notification.py

@@ -20,14 +20,14 @@ class NotificationPlugin(Plugin):
     def send(self, notification_type, message, targets, options, **kwargs):
         raise NotImplementedError
 
-    def filter_recipients(self, options, excluded_recipients):
+    def get_recipients(self, options, additional_recipients):
         """
-        Given a set of options (which should include configured recipient info), filters out recipients that
+        Given a set of options (which should include configured recipient info), returns the parsed list of recipients
-        we do NOT want to notify.
+        from those options plus the additional recipients specified. The returned value has no duplicates.
 
-        For any notification types where recipients can't be dynamically modified, this returns an empty list.
+        For any notification types where recipients can't be dynamically modified, this returns only the additional recipients.
         """
-        return []
+        return additional_recipients
 
 
 class ExpirationNotificationPlugin(NotificationPlugin):

lemur/plugins/lemur_acme/acme_handlers.py

@@ -224,7 +224,7 @@ class AcmeHandler(object):
     def revoke_certificate(self, certificate):
         if not self.reuse_account(certificate.authority):
             raise InvalidConfiguration("There is no ACME account saved, unable to revoke the certificate.")
-        acme_client, _ = self.acme.setup_acme_client(certificate.authority)
+        acme_client, _ = self.setup_acme_client(certificate.authority)
 
         fullchain_com = jose.ComparableX509(
             OpenSSL.crypto.load_certificate(

lemur/plugins/lemur_email/plugin.py

@@ -105,6 +105,8 @@ class EmailNotificationPlugin(ExpirationNotificationPlugin):
 
     @staticmethod
     def send(notification_type, message, targets, options, **kwargs):
+        if not targets:
+            return
 
         subject = "Lemur: {0} Notification".format(notification_type.capitalize())
 
@@ -119,11 +121,9 @@ class EmailNotificationPlugin(ExpirationNotificationPlugin):
             send_via_smtp(subject, body, targets)
 
     @staticmethod
-    def filter_recipients(options, excluded_recipients, **kwargs):
+    def get_recipients(options, additional_recipients, **kwargs):
         notification_recipients = get_plugin_option("recipients", options)
         if notification_recipients:
             notification_recipients = notification_recipients.split(",")
-            # removing owner and security_email from notification_recipient
-            notification_recipients = [i for i in notification_recipients if i not in excluded_recipients]
 
-        return notification_recipients
+        return list(set(notification_recipients + additional_recipients))

lemur/plugins/lemur_email/tests/test_email.py

@@ -21,7 +21,6 @@ def get_options():
 
 
 def test_render_expiration(certificate, endpoint):
-
     new_cert = CertificateFactory()
     new_cert.replaces.append(certificate)
 
@@ -54,7 +53,7 @@ def test_send_expiration_notification():
     certificate.notifications[0].options = get_options()
 
     verify_sender_email()
-    assert send_expiration_notifications([]) == (3, 0)  # owner, recipients (only counted as 1), and security
+    assert send_expiration_notifications([]) == (4, 0)  # owner (1), recipients (2), and security (1)
 
 
 @mock_ses
@@ -76,15 +75,20 @@ def test_send_pending_failure_notification(user, pending_certificate, async_issu
 
     verify_sender_email()
     assert send_pending_failure_notification(pending_certificate)
+    assert send_pending_failure_notification(pending_certificate, True, True)
+    assert send_pending_failure_notification(pending_certificate, True, False)
+    assert send_pending_failure_notification(pending_certificate, False, True)
+    assert send_pending_failure_notification(pending_certificate, False, False)
 
 
-def test_filter_recipients(certificate, endpoint):
+def test_get_recipients(certificate, endpoint):
     from lemur.plugins.lemur_email.plugin import EmailNotificationPlugin
 
-    options = [{"name": "recipients", "value": "security@example.com,bob@example.com,joe@example.com"}]
+    options = [{"name": "recipients", "value": "security@example.com,joe@example.com"}]
-    assert EmailNotificationPlugin.filter_recipients(options, []) == ["security@example.com", "bob@example.com",
+    two_emails = sorted(["security@example.com", "joe@example.com"])
-                                                                      "joe@example.com"]
+    assert sorted(EmailNotificationPlugin.get_recipients(options, [])) == two_emails
-    assert EmailNotificationPlugin.filter_recipients(options, ["security@example.com"]) == ["bob@example.com",
+    assert sorted(EmailNotificationPlugin.get_recipients(options, ["security@example.com"])) == two_emails
-                                                                                            "joe@example.com"]
+    three_emails = sorted(["security@example.com", "bob@example.com", "joe@example.com"])
-    assert EmailNotificationPlugin.filter_recipients(options, ["security@example.com", "bob@example.com",
+    assert sorted(EmailNotificationPlugin.get_recipients(options, ["bob@example.com"])) == three_emails
-                                                               "joe@example.com"]) == []
+    assert sorted(EmailNotificationPlugin.get_recipients(options, ["security@example.com", "bob@example.com",
+                                                                   "joe@example.com"])) == three_emails

requirements-dev.txt

@@ -24,7 +24,7 @@ keyring==21.2.0           # via twine
 mccabe==0.6.1             # via flake8
 nodeenv==1.5.0            # via -r requirements-dev.in, pre-commit
 pkginfo==1.5.0.1          # via twine
-pre-commit==2.8.2         # via -r requirements-dev.in
+pre-commit==2.9.0         # via -r requirements-dev.in
 pycodestyle==2.6.0        # via flake8
 pycparser==2.20           # via cffi
 pyflakes==2.2.0           # via flake8
@@ -32,7 +32,7 @@ pygments==2.6.1           # via readme-renderer
 pyyaml==5.3.1             # via -r requirements-dev.in, pre-commit
 readme-renderer==25.0     # via twine
 requests-toolbelt==0.9.1  # via twine
-requests==2.24.0          # via requests-toolbelt, twine
+requests==2.25.0          # via requests-toolbelt, twine
 rfc3986==1.4.0            # via twine
 secretstorage==3.1.2      # via keyring
 six==1.15.0               # via bleach, cryptography, readme-renderer, virtualenv

requirements-docs.txt

@@ -17,8 +17,8 @@ bcrypt==3.1.7             # via -r requirements.txt, flask-bcrypt, paramiko
 beautifulsoup4==4.9.1     # via -r requirements.txt, cloudflare
 billiard==3.6.3.0         # via -r requirements.txt, celery
 blinker==1.4              # via -r requirements.txt, flask-mail, flask-principal, raven
-boto3==1.16.14            # via -r requirements.txt
+boto3==1.16.24            # via -r requirements.txt
-botocore==1.19.14         # via -r requirements.txt, boto3, s3transfer
+botocore==1.19.24         # via -r requirements.txt, boto3, s3transfer
 celery[redis]==4.4.2      # via -r requirements.txt
 certifi==2020.11.8        # via -r requirements.txt, requests
 certsrv==2.1.1            # via -r requirements.txt
@@ -79,19 +79,20 @@ pyrfc3339==1.1            # via -r requirements.txt, acme
 python-dateutil==2.8.1    # via -r requirements.txt, alembic, arrow, botocore
 python-editor==1.0.4      # via -r requirements.txt, alembic
 python-json-logger==0.1.11  # via -r requirements.txt, logmatic-python
+python-ldap==3.3.1        # via -r requirements.txt
 pytz==2019.3              # via -r requirements.txt, acme, babel, celery, flask-restful, pyrfc3339
 pyyaml==5.3.1             # via -r requirements.txt, cloudflare
 raven[flask]==6.10.0      # via -r requirements.txt
 redis==3.5.3              # via -r requirements.txt, celery
 requests-toolbelt==0.9.1  # via -r requirements.txt, acme
-requests[security]==2.24.0  # via -r requirements.txt, acme, certsrv, cloudflare, hvac, requests-toolbelt, sphinx
+requests[security]==2.25.0  # via -r requirements.txt, acme, certsrv, cloudflare, hvac, requests-toolbelt, sphinx
 retrying==1.3.3           # via -r requirements.txt
 s3transfer==0.3.3         # via -r requirements.txt, boto3
 six==1.15.0               # via -r requirements.txt, acme, bcrypt, cryptography, flask-cors, flask-restful, hvac, josepy, jsonlines, packaging, pynacl, pyopenssl, python-dateutil, retrying, sphinxcontrib-httpdomain, sqlalchemy-utils
 snowballstemmer==2.0.0    # via sphinx
 soupsieve==2.0.1          # via -r requirements.txt, beautifulsoup4
 sphinx-rtd-theme==0.5.0   # via -r requirements-docs.in
-sphinx==3.3.0             # via -r requirements-docs.in, sphinx-rtd-theme, sphinxcontrib-httpdomain
+sphinx==3.3.1             # via -r requirements-docs.in, sphinx-rtd-theme, sphinxcontrib-httpdomain
 sphinxcontrib-applehelp==1.0.2  # via sphinx
 sphinxcontrib-devhelp==1.0.2  # via sphinx
 sphinxcontrib-htmlhelp==1.0.3  # via sphinx

requirements-tests.txt

@@ -10,9 +10,9 @@ aws-sam-translator==1.22.0  # via cfn-lint
 aws-xray-sdk==2.5.0       # via moto
 bandit==1.6.2             # via -r requirements-tests.in
 black==20.8b1             # via -r requirements-tests.in
-boto3==1.16.14            # via aws-sam-translator, moto
+boto3==1.16.24            # via aws-sam-translator, moto
 boto==2.49.0              # via moto
-botocore==1.19.14         # via aws-xray-sdk, boto3, moto, s3transfer
+botocore==1.19.24         # via aws-xray-sdk, boto3, moto, s3transfer
 certifi==2020.11.8        # via requests
 cffi==1.14.0              # via cryptography
 cfn-lint==0.29.5          # via moto
@@ -24,7 +24,7 @@ decorator==4.4.2          # via networkx
 docker==4.2.0             # via moto
 ecdsa==0.14.1             # via moto, python-jose, sshpubkeys
 factory-boy==3.1.0        # via -r requirements-tests.in
-faker==4.14.2             # via -r requirements-tests.in, factory-boy
+faker==4.17.1             # via -r requirements-tests.in, factory-boy
 fakeredis==1.4.4          # via -r requirements-tests.in
 flask==1.1.2              # via pytest-flask
 freezegun==1.0.0          # via -r requirements-tests.in
@@ -69,7 +69,7 @@ pyyaml==5.3.1             # via -r requirements-tests.in, bandit, cfn-lint, moto
 redis==3.5.3              # via fakeredis
 regex==2020.4.4           # via black
 requests-mock==1.8.0      # via -r requirements-tests.in
-requests==2.24.0          # via docker, moto, requests-mock, responses
+requests==2.25.0          # via docker, moto, requests-mock, responses
 responses==0.10.12        # via moto
 rsa==4.0                  # via python-jose
 s3transfer==0.3.3         # via boto3

requirements.txt

@@ -15,8 +15,8 @@ bcrypt==3.1.7             # via flask-bcrypt, paramiko
 beautifulsoup4==4.9.1     # via cloudflare
 billiard==3.6.3.0         # via celery
 blinker==1.4              # via flask-mail, flask-principal, raven
-boto3==1.16.14            # via -r requirements.in
+boto3==1.16.24            # via -r requirements.in
-botocore==1.19.14         # via -r requirements.in, boto3, s3transfer
+botocore==1.19.24         # via -r requirements.in, boto3, s3transfer
 celery[redis]==4.4.2      # via -r requirements.in
 certifi==2020.11.8        # via -r requirements.in, requests
 certsrv==2.1.1            # via -r requirements.in
@@ -78,7 +78,7 @@ pyyaml==5.3.1             # via -r requirements.in, cloudflare
 raven[flask]==6.10.0      # via -r requirements.in
 redis==3.5.3              # via -r requirements.in, celery
 requests-toolbelt==0.9.1  # via acme
-requests[security]==2.24.0  # via -r requirements.in, acme, certsrv, cloudflare, hvac, requests-toolbelt
+requests[security]==2.25.0  # via -r requirements.in, acme, certsrv, cloudflare, hvac, requests-toolbelt
 retrying==1.3.3           # via -r requirements.in
 s3transfer==0.3.3         # via boto3
 six==1.15.0               # via -r requirements.in, acme, bcrypt, cryptography, flask-cors, flask-restful, hvac, josepy, jsonlines, pynacl, pyopenssl, python-dateutil, retrying, sqlalchemy-utils