Merge branch 'entrust-plugin' of github.com:sirferl/lemur into azure-plugin

This commit is contained in:
sirferl 2020-11-24 12:17:14 +01:00
commit eedd2e91ee
11 changed files with 125 additions and 81 deletions

View File

@ -11,22 +11,47 @@ software.
* Update the version number in ``lemur/__about__.py``.
* Set the release date in the :doc:`/changelog`.
* Do a commit indicating this.
* Send a pull request with this.
* Do a commit indicating this, and raise a pull request with this.
* Wait for it to be merged.
Performing the release
----------------------
The commit that merged the version number bump is now the official release
commit for this release. You will need to have ``gpg`` installed and a ``gpg``
key in order to do a release. Once this has happened:
commit for this release. You need an `API key <https://pypi.org/manage/account/#api-tokens>`_,
which requires permissions to maintain the Lemur `project <https://pypi.org/project/lemur/>`_.
* Run ``invoke release {version}``.
For creating the release, follow these steps (more details `here <https://packaging.python.org/tutorials/packaging-projects/#generating-distribution-archives>`_)
The release should now be available on PyPI and a tag should be available in
* Make sure you have the latest versions of setuptools and wheel installed:
``python3 -m pip install --user --upgrade setuptools wheel``
* Now run this command from the same directory where setup.py is located:
``python3 setup.py sdist bdist_wheel``
* Once completed it should generate two files in the dist directory:
.. code-block:: pycon
$ ls dist/
lemur-0.8.0-py2.py3-none-any.whl lemur-0.8.0.tar.gz
* In this step, the distribution will be uploaded. Youll need to install Twine:
``python3 -m pip install --user --upgrade twine``
* Once installed, run Twine to upload all of the archives under dist. Once installed, run Twine to upload all of the archives under dist:
``python3 -m twine upload --repository pypi dist/*``
The release should now be available on `PyPI Lemur <https://pypi.org/project/lemur/>`_ and a tag should be available in
the repository.
Make sure to also make a github `release <https://github.com/Netflix/lemur/releases>`_ which will pick up the latest version.
Verifying the release
---------------------

View File

@ -415,8 +415,8 @@ And the worker can be started with desired options such as the following::
supervisor or systemd configurations should be created for these in production environments as appropriate.
Add support for LetsEncrypt
===========================
Add support for LetsEncrypt/ACME
================================
LetsEncrypt is a free, limited-feature certificate authority that offers publicly trusted certificates that are valid
for 90 days. LetsEncrypt does not use organizational validation (OV), and instead relies on domain validation (DV).
@ -424,7 +424,10 @@ LetsEncrypt requires that we prove ownership of a domain before we're able to is
time we want a certificate.
The most common methods to prove ownership are HTTP validation and DNS validation. Lemur supports DNS validation
through the creation of DNS TXT records.
through the creation of DNS TXT records as well as HTTP validation, reusing the destination concept.
ACME DNS Challenge
------------------
In a nutshell, when we send a certificate request to LetsEncrypt, they generate a random token and ask us to put that
token in a DNS text record to prove ownership of a domain. If a certificate request has multiple domains, we must
@ -462,6 +465,24 @@ possible. To enable this functionality, periodically (or through Cron/Celery) ru
This command will traverse all DNS providers, determine which zones they control, and upload this list of zones to
Lemur's database (in the dns_providers table). Alternatively, you can manually input this data.
ACME HTTP Challenge
-------------------
The flow for requesting a certificate using the HTTP challenge is not that different from the one described for the DNS
challenge. The only difference is, that instead of creating a DNS TXT record, a file is uploaded to a Webserver which
serves the file at `http://<domain>/.well-known/acme-challenge/<token>`
Currently the HTTP challenge also works without Celery, since it's done while creating the certificate, and doesn't
rely on celery to create the DNS record. This will change when we implement mix & match of acme challenge types.
To create a HTTP compatible Authority, you first need to create a new destination that will be used to deploy the
challenge token. Visit `Admin` -> `Destination` and click `Create`. The path you provide for the destination needs to
be the exact path that is called when the ACME providers calls ``http://<domain>/.well-known/acme-challenge/`. The
token part will be added dynamically by the acme_upload.
Currently only the SFTP and S3 Bucket destination support the ACME HTTP challenge.
Afterwards you can create a new certificate authority as described in the DNS challenge, but need to choose
`Acme HTTP-01` as the plugin type, and then the destination you created beforehand.
LetsEncrypt: pinning to cross-signed ICA
----------------------------------------

View File

@ -103,8 +103,9 @@ def send_plugin_notification(event_type, data, recipients, notification):
function = f"{__name__}.{sys._getframe().f_code.co_name}"
log_data = {
"function": function,
"message": f"Sending expiration notification for to recipients {recipients}",
"notification_type": "expiration",
"message": f"Sending {event_type} notification for to recipients {recipients}",
"notification_type": event_type,
"notification_plugin": notification.plugin.slug,
"certificate_targets": recipients,
}
status = FAILURE_METRIC_STATUS
@ -121,7 +122,7 @@ def send_plugin_notification(event_type, data, recipients, notification):
"notification",
"counter",
1,
metric_tags={"status": status, "event_type": event_type},
metric_tags={"status": status, "event_type": event_type, "plugin": notification.plugin.slug},
)
if status == SUCCESS_METRIC_STATUS:
@ -142,7 +143,6 @@ def send_expiration_notifications(exclude):
for notification_label, certificates in notification_group.items():
notification_data = []
security_data = []
notification = certificates[0][0]
@ -152,33 +152,26 @@ def send_expiration_notifications(exclude):
certificate
).data
notification_data.append(cert_data)
security_data.append(cert_data)
if send_default_notification(
"expiration", notification_data, [owner], notification.options
):
success += 1
else:
failure += 1
recipients = notification.plugin.filter_recipients(notification.options, security_email + [owner])
email_recipients = notification.plugin.get_recipients(notification.options, security_email + [owner])
# Plugin will ONLY use the provided recipients if it's email; any other notification plugin ignores them
if send_plugin_notification(
"expiration",
notification_data,
recipients,
notification,
"expiration", notification_data, email_recipients, notification
):
success += 1
success += len(email_recipients)
else:
failure += 1
if send_default_notification(
"expiration", security_data, security_email, notification.options
):
success += 1
else:
failure += 1
failure += len(email_recipients)
# If we're using an email plugin, we're done,
# since "security_email + [owner]" were added as email_recipients.
# If we're not using an email plugin, we also need to send an email to the security team and owner,
# since the plugin notification didn't send anything to them.
if notification.plugin.slug != "email-notification":
if send_default_notification(
"expiration", notification_data, email_recipients, notification.options
):
success = 1 + len(email_recipients)
else:
failure = 1 + len(email_recipients)
return success, failure
@ -195,15 +188,16 @@ def send_default_notification(notification_type, data, targets, notification_opt
:return:
"""
function = f"{__name__}.{sys._getframe().f_code.co_name}"
log_data = {
"function": function,
"message": f"Sending notification for certificate data {data}",
"notification_type": notification_type,
}
status = FAILURE_METRIC_STATUS
notification_plugin = plugins.get(
current_app.config.get("LEMUR_DEFAULT_NOTIFICATION_PLUGIN", "email-notification")
)
log_data = {
"function": function,
"message": f"Sending {notification_type} notification for certificate data {data} to targets {targets}",
"notification_type": notification_type,
"notification_plugin": notification_plugin.slug,
}
try:
current_app.logger.debug(log_data)
@ -212,7 +206,7 @@ def send_default_notification(notification_type, data, targets, notification_opt
status = SUCCESS_METRIC_STATUS
except Exception as e:
log_data["message"] = f"Unable to send {notification_type} notification for certificate data {data} " \
f"to target {targets}"
f"to targets {targets}"
current_app.logger.error(log_data, exc_info=True)
sentry.captureException()
@ -220,7 +214,7 @@ def send_default_notification(notification_type, data, targets, notification_opt
"notification",
"counter",
1,
metric_tags={"status": status, "event_type": notification_type},
metric_tags={"status": status, "event_type": notification_type, "plugin": notification_plugin.slug},
)
if status == SUCCESS_METRIC_STATUS:
@ -247,15 +241,14 @@ def send_pending_failure_notification(
data = pending_certificate_output_schema.dump(pending_cert).data
data["security_email"] = current_app.config.get("LEMUR_SECURITY_TEAM_EMAIL")
notify_owner_success = False
email_recipients = []
if notify_owner:
notify_owner_success = send_default_notification("failed", data, [data["owner"]], pending_cert)
email_recipients = email_recipients + [data["owner"]]
notify_security_success = False
if notify_security:
notify_security_success = send_default_notification("failed", data, data["security_email"], pending_cert)
email_recipients = email_recipients + data["security_email"]
return notify_owner_success or notify_security_success
return send_default_notification("failed", data, email_recipients, pending_cert)
def needs_notification(certificate):

View File

@ -20,14 +20,14 @@ class NotificationPlugin(Plugin):
def send(self, notification_type, message, targets, options, **kwargs):
raise NotImplementedError
def filter_recipients(self, options, excluded_recipients):
def get_recipients(self, options, additional_recipients):
"""
Given a set of options (which should include configured recipient info), filters out recipients that
we do NOT want to notify.
Given a set of options (which should include configured recipient info), returns the parsed list of recipients
from those options plus the additional recipients specified. The returned value has no duplicates.
For any notification types where recipients can't be dynamically modified, this returns an empty list.
For any notification types where recipients can't be dynamically modified, this returns only the additional recipients.
"""
return []
return additional_recipients
class ExpirationNotificationPlugin(NotificationPlugin):

View File

@ -224,7 +224,7 @@ class AcmeHandler(object):
def revoke_certificate(self, certificate):
if not self.reuse_account(certificate.authority):
raise InvalidConfiguration("There is no ACME account saved, unable to revoke the certificate.")
acme_client, _ = self.acme.setup_acme_client(certificate.authority)
acme_client, _ = self.setup_acme_client(certificate.authority)
fullchain_com = jose.ComparableX509(
OpenSSL.crypto.load_certificate(

View File

@ -105,6 +105,8 @@ class EmailNotificationPlugin(ExpirationNotificationPlugin):
@staticmethod
def send(notification_type, message, targets, options, **kwargs):
if not targets:
return
subject = "Lemur: {0} Notification".format(notification_type.capitalize())
@ -119,11 +121,9 @@ class EmailNotificationPlugin(ExpirationNotificationPlugin):
send_via_smtp(subject, body, targets)
@staticmethod
def filter_recipients(options, excluded_recipients, **kwargs):
def get_recipients(options, additional_recipients, **kwargs):
notification_recipients = get_plugin_option("recipients", options)
if notification_recipients:
notification_recipients = notification_recipients.split(",")
# removing owner and security_email from notification_recipient
notification_recipients = [i for i in notification_recipients if i not in excluded_recipients]
return notification_recipients
return list(set(notification_recipients + additional_recipients))

View File

@ -21,7 +21,6 @@ def get_options():
def test_render_expiration(certificate, endpoint):
new_cert = CertificateFactory()
new_cert.replaces.append(certificate)
@ -54,7 +53,7 @@ def test_send_expiration_notification():
certificate.notifications[0].options = get_options()
verify_sender_email()
assert send_expiration_notifications([]) == (3, 0) # owner, recipients (only counted as 1), and security
assert send_expiration_notifications([]) == (4, 0) # owner (1), recipients (2), and security (1)
@mock_ses
@ -76,15 +75,20 @@ def test_send_pending_failure_notification(user, pending_certificate, async_issu
verify_sender_email()
assert send_pending_failure_notification(pending_certificate)
assert send_pending_failure_notification(pending_certificate, True, True)
assert send_pending_failure_notification(pending_certificate, True, False)
assert send_pending_failure_notification(pending_certificate, False, True)
assert send_pending_failure_notification(pending_certificate, False, False)
def test_filter_recipients(certificate, endpoint):
def test_get_recipients(certificate, endpoint):
from lemur.plugins.lemur_email.plugin import EmailNotificationPlugin
options = [{"name": "recipients", "value": "security@example.com,bob@example.com,joe@example.com"}]
assert EmailNotificationPlugin.filter_recipients(options, []) == ["security@example.com", "bob@example.com",
"joe@example.com"]
assert EmailNotificationPlugin.filter_recipients(options, ["security@example.com"]) == ["bob@example.com",
"joe@example.com"]
assert EmailNotificationPlugin.filter_recipients(options, ["security@example.com", "bob@example.com",
"joe@example.com"]) == []
options = [{"name": "recipients", "value": "security@example.com,joe@example.com"}]
two_emails = sorted(["security@example.com", "joe@example.com"])
assert sorted(EmailNotificationPlugin.get_recipients(options, [])) == two_emails
assert sorted(EmailNotificationPlugin.get_recipients(options, ["security@example.com"])) == two_emails
three_emails = sorted(["security@example.com", "bob@example.com", "joe@example.com"])
assert sorted(EmailNotificationPlugin.get_recipients(options, ["bob@example.com"])) == three_emails
assert sorted(EmailNotificationPlugin.get_recipients(options, ["security@example.com", "bob@example.com",
"joe@example.com"])) == three_emails

View File

@ -24,7 +24,7 @@ keyring==21.2.0 # via twine
mccabe==0.6.1 # via flake8
nodeenv==1.5.0 # via -r requirements-dev.in, pre-commit
pkginfo==1.5.0.1 # via twine
pre-commit==2.8.2 # via -r requirements-dev.in
pre-commit==2.9.0 # via -r requirements-dev.in
pycodestyle==2.6.0 # via flake8
pycparser==2.20 # via cffi
pyflakes==2.2.0 # via flake8
@ -32,7 +32,7 @@ pygments==2.6.1 # via readme-renderer
pyyaml==5.3.1 # via -r requirements-dev.in, pre-commit
readme-renderer==25.0 # via twine
requests-toolbelt==0.9.1 # via twine
requests==2.24.0 # via requests-toolbelt, twine
requests==2.25.0 # via requests-toolbelt, twine
rfc3986==1.4.0 # via twine
secretstorage==3.1.2 # via keyring
six==1.15.0 # via bleach, cryptography, readme-renderer, virtualenv

View File

@ -17,8 +17,8 @@ bcrypt==3.1.7 # via -r requirements.txt, flask-bcrypt, paramiko
beautifulsoup4==4.9.1 # via -r requirements.txt, cloudflare
billiard==3.6.3.0 # via -r requirements.txt, celery
blinker==1.4 # via -r requirements.txt, flask-mail, flask-principal, raven
boto3==1.16.14 # via -r requirements.txt
botocore==1.19.14 # via -r requirements.txt, boto3, s3transfer
boto3==1.16.24 # via -r requirements.txt
botocore==1.19.24 # via -r requirements.txt, boto3, s3transfer
celery[redis]==4.4.2 # via -r requirements.txt
certifi==2020.11.8 # via -r requirements.txt, requests
certsrv==2.1.1 # via -r requirements.txt
@ -79,19 +79,20 @@ pyrfc3339==1.1 # via -r requirements.txt, acme
python-dateutil==2.8.1 # via -r requirements.txt, alembic, arrow, botocore
python-editor==1.0.4 # via -r requirements.txt, alembic
python-json-logger==0.1.11 # via -r requirements.txt, logmatic-python
python-ldap==3.3.1 # via -r requirements.txt
pytz==2019.3 # via -r requirements.txt, acme, babel, celery, flask-restful, pyrfc3339
pyyaml==5.3.1 # via -r requirements.txt, cloudflare
raven[flask]==6.10.0 # via -r requirements.txt
redis==3.5.3 # via -r requirements.txt, celery
requests-toolbelt==0.9.1 # via -r requirements.txt, acme
requests[security]==2.24.0 # via -r requirements.txt, acme, certsrv, cloudflare, hvac, requests-toolbelt, sphinx
requests[security]==2.25.0 # via -r requirements.txt, acme, certsrv, cloudflare, hvac, requests-toolbelt, sphinx
retrying==1.3.3 # via -r requirements.txt
s3transfer==0.3.3 # via -r requirements.txt, boto3
six==1.15.0 # via -r requirements.txt, acme, bcrypt, cryptography, flask-cors, flask-restful, hvac, josepy, jsonlines, packaging, pynacl, pyopenssl, python-dateutil, retrying, sphinxcontrib-httpdomain, sqlalchemy-utils
snowballstemmer==2.0.0 # via sphinx
soupsieve==2.0.1 # via -r requirements.txt, beautifulsoup4
sphinx-rtd-theme==0.5.0 # via -r requirements-docs.in
sphinx==3.3.0 # via -r requirements-docs.in, sphinx-rtd-theme, sphinxcontrib-httpdomain
sphinx==3.3.1 # via -r requirements-docs.in, sphinx-rtd-theme, sphinxcontrib-httpdomain
sphinxcontrib-applehelp==1.0.2 # via sphinx
sphinxcontrib-devhelp==1.0.2 # via sphinx
sphinxcontrib-htmlhelp==1.0.3 # via sphinx

View File

@ -10,9 +10,9 @@ aws-sam-translator==1.22.0 # via cfn-lint
aws-xray-sdk==2.5.0 # via moto
bandit==1.6.2 # via -r requirements-tests.in
black==20.8b1 # via -r requirements-tests.in
boto3==1.16.14 # via aws-sam-translator, moto
boto3==1.16.24 # via aws-sam-translator, moto
boto==2.49.0 # via moto
botocore==1.19.14 # via aws-xray-sdk, boto3, moto, s3transfer
botocore==1.19.24 # via aws-xray-sdk, boto3, moto, s3transfer
certifi==2020.11.8 # via requests
cffi==1.14.0 # via cryptography
cfn-lint==0.29.5 # via moto
@ -24,7 +24,7 @@ decorator==4.4.2 # via networkx
docker==4.2.0 # via moto
ecdsa==0.14.1 # via moto, python-jose, sshpubkeys
factory-boy==3.1.0 # via -r requirements-tests.in
faker==4.14.2 # via -r requirements-tests.in, factory-boy
faker==4.17.1 # via -r requirements-tests.in, factory-boy
fakeredis==1.4.4 # via -r requirements-tests.in
flask==1.1.2 # via pytest-flask
freezegun==1.0.0 # via -r requirements-tests.in
@ -69,7 +69,7 @@ pyyaml==5.3.1 # via -r requirements-tests.in, bandit, cfn-lint, moto
redis==3.5.3 # via fakeredis
regex==2020.4.4 # via black
requests-mock==1.8.0 # via -r requirements-tests.in
requests==2.24.0 # via docker, moto, requests-mock, responses
requests==2.25.0 # via docker, moto, requests-mock, responses
responses==0.10.12 # via moto
rsa==4.0 # via python-jose
s3transfer==0.3.3 # via boto3

View File

@ -15,8 +15,8 @@ bcrypt==3.1.7 # via flask-bcrypt, paramiko
beautifulsoup4==4.9.1 # via cloudflare
billiard==3.6.3.0 # via celery
blinker==1.4 # via flask-mail, flask-principal, raven
boto3==1.16.14 # via -r requirements.in
botocore==1.19.14 # via -r requirements.in, boto3, s3transfer
boto3==1.16.24 # via -r requirements.in
botocore==1.19.24 # via -r requirements.in, boto3, s3transfer
celery[redis]==4.4.2 # via -r requirements.in
certifi==2020.11.8 # via -r requirements.in, requests
certsrv==2.1.1 # via -r requirements.in
@ -78,7 +78,7 @@ pyyaml==5.3.1 # via -r requirements.in, cloudflare
raven[flask]==6.10.0 # via -r requirements.in
redis==3.5.3 # via -r requirements.in, celery
requests-toolbelt==0.9.1 # via acme
requests[security]==2.24.0 # via -r requirements.in, acme, certsrv, cloudflare, hvac, requests-toolbelt
requests[security]==2.25.0 # via -r requirements.in, acme, certsrv, cloudflare, hvac, requests-toolbelt
retrying==1.3.3 # via -r requirements.in
s3transfer==0.3.3 # via boto3
six==1.15.0 # via -r requirements.in, acme, bcrypt, cryptography, flask-cors, flask-restful, hvac, josepy, jsonlines, pynacl, pyopenssl, python-dateutil, retrying, sqlalchemy-utils