Merge branch 'source_options' of github.com:sirferl/lemur into source_options

docs/administration.rst

@@ -262,22 +262,107 @@ and are used when Lemur creates the CSR for your certificates.
         LEMUR_DEFAULT_AUTHORITY = "verisign"
 
 
+.. _NotificationOptions:
+
 Notification Options
 --------------------
 
-Lemur currently has very basic support for notifications. Currently only expiration notifications are supported. Actual notification
+Lemur supports a small variety of notification types through a set of notification plugins.
-is handled by the notification plugins that you have configured. Lemur ships with the 'Email' notification that allows expiration emails
+By default, Lemur configures a standard set of email notifications for all certificates.
-to be sent to subscribers.
 
-Templates for expiration emails are located under `lemur/plugins/lemur_email/templates` and can be modified for your needs.
+**Plugin-capable notifications**
-Notifications are sent to the certificate creator, owner and security team as specified by the `LEMUR_SECURITY_TEAM_EMAIL` configuration parameter.
 
-Certificates marked as inactive will **not** be notified of upcoming expiration. This enables a user to essentially
+These notifications can be configured to use all available notification plugins.
-silence the expiration. If a certificate is active and is expiring the above will be notified according to the `LEMUR_DEFAULT_EXPIRATION_NOTIFICATION_INTERVALS` or
-30, 15, 2 days before expiration if no intervals are set.
 
-Lemur supports sending certificate expiration notifications through SES and SMTP.
+Supported types:
 
+* Certificate expiration
+
+**Email-only notifications**
+
+These notifications can only be sent via email and cannot use other notification plugins.
+
+Supported types:
+
+* CA certificate expiration
+* Pending ACME certificate failure
+* Certificate rotation
+
+**Default notifications**
+
+When a certificate is created, the following email notifications are created for it if they do not exist.
+If these notifications already exist, they will be associated with the new certificate.
+
+* ``DEFAULT_<OWNER>_X_DAY``, where X is the set of values specified in ``LEMUR_DEFAULT_EXPIRATION_NOTIFICATION_INTERVALS`` and defaults to 30, 15, and 2 if not specified. The owner's username will replace ``<OWNER>``.
+* ``DEFAULT_SECURITY_X_DAY``, where X is the set of values specified in ``LEMUR_SECURITY_TEAM_EMAIL_INTERVALS`` and defaults to ``LEMUR_DEFAULT_EXPIRATION_NOTIFICATION_INTERVALS`` if not specified (which also defaults to 30, 15, and 2 if not specified).
+
+These notifications can be disabled if desired. They can also be unassociated with a specific certificate.
+
+**Disabling notifications**
+
+Notifications can be disabled either for an individual certificate (which disables all notifications for that certificate)
+or for an individual notification object (which disables that notification for all associated certificates).
+At present, disabling a notification object will only disable certificate expiration notifications, and not other types,
+since other notification types don't use notification objects.
+
+**Certificate expiration**
+
+Certificate expiration notifications are sent when the scheduled task to send certificate expiration notifications runs
+(see :ref:`PeriodicTasks`). Specific patterns of certificate names may be excluded using ``--exclude`` (when using
+cron; you may specify this multiple times for multiple patterns) or via the config option ``EXCLUDE_CN_FROM_NOTIFICATION``
+(when using celery; this is a list configuration option, meaning you specify multiple values, such as
+``['exclude', 'also exclude']``). The specified exclude pattern will match if found anywhere in the certificate name.
+
+When the periodic task runs, Lemur checks for certificates meeting the following conditions:
+
+* Certificate has notifications enabled
+* Certificate is not expired
+* Certificate is not revoked
+* Certificate name does not match the `exclude` parameter
+* Certificate has at least one associated notification object
+* That notification is active
+* That notification's configured interval and unit match the certificate's remaining lifespan
+
+All eligible certificates are then grouped by owner and applicable notification. For each notification and certificate group,
+Lemur will send the expiration notification using whichever plugin was configured for that notification object.
+In addition, Lemur will send an email to the certificate owner and security team (as specified by the
+``LEMUR_SECURITY_TEAM_EMAIL`` configuration parameter).
+
+**CA certificate expiration**
+
+Certificate authority certificate expiration notifications are sent when the scheduled task to send authority certificate
+expiration notifications runs (see :ref:`PeriodicTasks`). Notifications are sent via the intervals configured in the
+configuration parameter ``LEMUR_AUTHORITY_CERT_EXPIRATION_EMAIL_INTERVALS``, with a default of 365 and 180 days.
+
+When the periodic task runs, Lemur checks for certificates meeting the following conditions:
+
+* Certificate has notifications enabled
+* Certificate is not expired
+* Certificate is not revoked
+* Certificate is associated with a CA
+* Certificate's remaining lifespan matches one of the configured intervals
+
+All eligible certificates are then grouped by owner and expiration interval. For each interval and certificate group,
+Lemur will send the CA certificate expiration notification via email to the certificate owner and security team
+(as specified by the ``LEMUR_SECURITY_TEAM_EMAIL`` configuration parameter).
+
+**Pending ACME certificate failure**
+
+Whenever a pending ACME certificate fails to be issued, Lemur will send a notification via email to the certificate owner
+and security team (as specified by the ``LEMUR_SECURITY_TEAM_EMAIL`` configuration parameter). This email is not sent if
+the pending certificate had notifications disabled.
+
+**Certificate rotation**
+
+Whenever a cert is rotated, Lemur will send a notification via email to the certificate owner. This notification is
+disabled by default; to enable it, you must set the option ``--notify`` (when using cron) or the configuration parameter
+``ENABLE_ROTATION_NOTIFICATION`` (when using celery).
+
+**Email notifications**
+
+Templates for emails are located under `lemur/plugins/lemur_email/templates` and can be modified for your needs.
+
+The following configuration options are supported:
 
 .. data:: LEMUR_EMAIL_SENDER
     :noindex:
@@ -318,7 +403,7 @@ Lemur supports sending certificate expiration notifications through SES and SMTP
 
         ::
 
-            LEMUR_EMAIL = 'lemur.example.com'
+            LEMUR_EMAIL = 'lemur@example.com'
 
 
 .. data:: LEMUR_SECURITY_TEAM_EMAIL
@@ -333,7 +418,7 @@ Lemur supports sending certificate expiration notifications through SES and SMTP
 .. data:: LEMUR_DEFAULT_EXPIRATION_NOTIFICATION_INTERVALS
     :noindex:
 
-        Lemur notification intervals
+        Lemur notification intervals. If unspecified, the value [30, 15, 2] is used.
 
         ::
 
@@ -348,6 +433,15 @@ Lemur supports sending certificate expiration notifications through SES and SMTP
 
           LEMUR_SECURITY_TEAM_EMAIL_INTERVALS = [15, 2]
 
+.. data:: LEMUR_AUTHORITY_CERT_EXPIRATION_EMAIL_INTERVALS
+    :noindex:
+
+       Notification interval set for CA certificate expiration notifications. If unspecified, the value [365, 180] is used (roughly one year and 6 months).
+
+       ::
+
+          LEMUR_AUTHORITY_CERT_EXPIRATION_EMAIL_INTERVALS = [365, 180]
+
 
 Celery Options
 ---------------

docs/developer/plugins/index.rst

@@ -215,12 +215,13 @@ Notification
 ------------
 
 Lemur includes the ability to create Email notifications by **default**. These notifications
-currently come in the form of expiration and rotation notices. Lemur periodically checks certificate expiration dates and
+currently come in the form of expiration and rotation notices for all certificates, expiration notices for CA certificates,
+and ACME certificate creation failure notices. Lemur periodically checks certificate expiration dates and
 determines if a given certificate is eligible for notification. There are currently only two parameters used to
 determine if a certificate is eligible; validity expiration (date the certificate is no longer valid) and the number
 of days the current date (UTC) is from that expiration date.
 
-Expiration notifications can also be configured for Slack or AWS SNS. Rotation notifications are not configurable.
+Certificate expiration notifications can also be configured for Slack or AWS SNS. Other notifications are not configurable.
 Notifications sent to a certificate owner and security team (`LEMUR_SECURITY_TEAM_EMAIL`) can currently only be sent via email.
 
 There are currently two objects that are available for notification plugins. The first is `NotificationPlugin`, which is the base object for

docs/production/index.rst

@@ -325,7 +325,7 @@ celery tasks or cron jobs that run these commands.
 
 There are currently three commands that could/should be run on a periodic basis:
 
-- `notify`
+- `notify expirations` and `notify authority_expirations` (see :ref:`NotificationOptions` for configuration info)
 - `check_revoked`
 - `sync`
 
@@ -334,13 +334,15 @@ If you are using LetsEncrypt, you must also run the following:
 - `fetch_all_pending_acme_certs`
 - `remove_old_acme_certs`
 
-How often you run these commands is largely up to the user. `notify` and `check_revoked` are typically run at least once a day.
+How often you run these commands is largely up to the user. `notify` should be run once a day (more often will result in
+duplicate notifications). `check_revoked` is typically run at least once a day.
 `sync` is typically run every 15 minutes. `fetch_all_pending_acme_certs` should be ran frequently (Every minute is fine).
 `remove_old_acme_certs` can be ran more rarely, such as once every week.
 
 Example cron entries::
 
     0 22 * * * lemuruser export LEMUR_CONF=/Users/me/.lemur/lemur.conf.py; /www/lemur/bin/lemur notify expirations
+    0 22 * * * lemuruser export LEMUR_CONF=/Users/me/.lemur/lemur.conf.py; /www/lemur/bin/lemur notify authority_expirations
     */15 * * * * lemuruser export LEMUR_CONF=/Users/me/.lemur/lemur.conf.py; /www/lemur/bin/lemur source sync -s all
     0 22 * * * lemuruser export LEMUR_CONF=/Users/me/.lemur/lemur.conf.py; /www/lemur/bin/lemur certificate check_revoked
 
@@ -382,6 +384,20 @@ Example Celery configuration (To be placed in your configuration file)::
                 'expires': 180
             },
             'schedule': crontab(hour="*"),
+        },
+        'notify_expirations': {
+            'task': 'lemur.common.celery.notify_expirations',
+            'options': {
+                'expires': 180
+            },
+            'schedule': crontab(hour=22, minute=0),
+        },
+        'notify_authority_expirations': {
+            'task': 'lemur.common.celery.notify_authority_expirations',
+            'options': {
+                'expires': 180
+            },
+            'schedule': crontab(hour=22, minute=0),
         }
     }
 

lemur/certificates/service.py

@@ -864,3 +864,12 @@ def cleanup_after_revoke(certificate):
 
     database.update(certificate)
     return error_message
+
+
+def get_issued_cert_count_for_authority(authority):
+    """
+    Returns the count of certs issued by the specified authority.
+
+    :return:
+    """
+    return database.db.session.query(Certificate).filter(Certificate.authority_id == authority.id).count()

lemur/common/celery.py

@@ -656,11 +656,12 @@ def certificate_rotate(**kwargs):
 
     current_app.logger.debug(log_data)
     try:
+        notify = current_app.config.get("ENABLE_ROTATION_NOTIFICATION", None)
         if region:
             log_data["region"] = region
-            cli_certificate.rotate_region(None, None, None, None, True, region)
+            cli_certificate.rotate_region(None, None, None, notify, True, region)
         else:
-            cli_certificate.rotate(None, None, None, None, True)
+            cli_certificate.rotate(None, None, None, notify, True)
     except SoftTimeLimitExceeded:
         log_data["message"] = "Certificate rotate: Time limit exceeded."
         current_app.logger.error(log_data)
@@ -820,6 +821,42 @@ def notify_expirations():
     return log_data
 
 
+@celery.task(soft_time_limit=3600)
+def notify_authority_expirations():
+    """
+    This celery task notifies about expiring certificate authority certs
+    :return:
+    """
+    function = f"{__name__}.{sys._getframe().f_code.co_name}"
+    task_id = None
+    if celery.current_task:
+        task_id = celery.current_task.request.id
+
+    log_data = {
+        "function": function,
+        "message": "notify for certificate authority cert expiration",
+        "task_id": task_id,
+    }
+
+    if task_id and is_task_active(function, task_id, None):
+        log_data["message"] = "Skipping task: Task is already active"
+        current_app.logger.debug(log_data)
+        return
+
+    current_app.logger.debug(log_data)
+    try:
+        cli_notification.authority_expirations()
+    except SoftTimeLimitExceeded:
+        log_data["message"] = "Notify expiring CA Time limit exceeded."
+        current_app.logger.error(log_data)
+        sentry.captureException()
+        metrics.send("celery.timeout", "counter", 1, metric_tags={"function": function})
+        return
+
+    metrics.send(f"{function}.success", "counter", 1)
+    return log_data
+
+
 @celery.task(soft_time_limit=3600)
 def enable_autorotate_for_certs_attached_to_endpoint():
     """

lemur/notifications/cli.py

@@ -10,6 +10,7 @@ from flask_script import Manager
 from lemur.constants import SUCCESS_METRIC_STATUS, FAILURE_METRIC_STATUS
 from lemur.extensions import sentry, metrics
 from lemur.notifications.messaging import send_expiration_notifications
+from lemur.notifications.messaging import send_authority_expiration_notifications
 
 manager = Manager(usage="Handles notification related tasks.")
 
@@ -24,7 +25,7 @@ manager = Manager(usage="Handles notification related tasks.")
 )
 def expirations(exclude):
     """
-    Runs Lemur's notification engine, that looks for expired certificates and sends
+    Runs Lemur's notification engine, that looks for expiring certificates and sends
     notifications out to those that have subscribed to them.
 
     Every certificate receives notifications by default. When expiration notifications are handled outside of Lemur
@@ -39,9 +40,7 @@ def expirations(exclude):
         print("Starting to notify subscribers about expiring certificates!")
         success, failed = send_expiration_notifications(exclude)
         print(
-            "Finished notifying subscribers about expiring certificates! Sent: {success} Failed: {failed}".format(
+            f"Finished notifying subscribers about expiring certificates! Sent: {success} Failed: {failed}"
-                success=success, failed=failed
-            )
         )
         status = SUCCESS_METRIC_STATUS
     except Exception as e:
@@ -50,3 +49,27 @@ def expirations(exclude):
     metrics.send(
         "expiration_notification_job", "counter", 1, metric_tags={"status": status}
     )
+
+
+def authority_expirations():
+    """
+    Runs Lemur's notification engine, that looks for expiring certificate authority certificates and sends
+    notifications out to the security team and owner.
+
+    :return:
+    """
+    status = FAILURE_METRIC_STATUS
+    try:
+        print("Starting to notify subscribers about expiring certificate authority certificates!")
+        success, failed = send_authority_expiration_notifications()
+        print(
+            "Finished notifying subscribers about expiring certificate authority certificates! "
+            f"Sent: {success} Failed: {failed}"
+        )
+        status = SUCCESS_METRIC_STATUS
+    except Exception as e:
+        sentry.captureException()
+
+    metrics.send(
+        "authority_expiration_notification_job", "counter", 1, metric_tags={"status": status}
+    )

lemur/notifications/messaging.py

@@ -19,9 +19,10 @@ from sqlalchemy import and_
 from sqlalchemy.sql.expression import false, true
 
 from lemur import database
+from lemur.certificates import service as certificates_service
 from lemur.certificates.models import Certificate
 from lemur.certificates.schemas import certificate_notification_output_schema
-from lemur.common.utils import windowed_query
+from lemur.common.utils import windowed_query, is_selfsigned
 from lemur.constants import FAILURE_METRIC_STATUS, SUCCESS_METRIC_STATUS
 from lemur.extensions import metrics, sentry
 from lemur.pending_certificates.schemas import pending_certificate_output_schema
@@ -62,6 +63,34 @@ def get_certificates(exclude=None):
     return certs
 
 
+def get_expiring_authority_certificates():
+    """
+    Finds all certificate authority certificates that are eligible for expiration notifications.
+    :return:
+    """
+    now = arrow.utcnow()
+    authority_expiration_intervals = current_app.config.get("LEMUR_AUTHORITY_CERT_EXPIRATION_EMAIL_INTERVALS",
+                                                            [365, 180])
+    max_not_after = now + timedelta(days=max(authority_expiration_intervals) + 1)
+
+    q = (
+        database.db.session.query(Certificate)
+        .filter(Certificate.not_after < max_not_after)
+        .filter(Certificate.notify == true())
+        .filter(Certificate.expired == false())
+        .filter(Certificate.revoked == false())
+        .filter(Certificate.root_authority_id.isnot(None))
+        .filter(Certificate.authority_id.is_(None))
+    )
+
+    certs = []
+    for c in windowed_query(q, Certificate.id, 10000):
+        days_remaining = (c.not_after - now).days
+        if days_remaining in authority_expiration_intervals:
+            certs.append(c)
+    return certs
+
+
 def get_eligible_certificates(exclude=None):
     """
     Finds all certificates that are eligible for certificate expiration notification.
@@ -90,6 +119,25 @@ def get_eligible_certificates(exclude=None):
     return certificates
 
 
+def get_eligible_authority_certificates():
+    """
+    Finds all certificate authority certificates that are eligible for certificate expiration notification.
+    Returns the set of all eligible CA certificates, grouped by owner and interval, with a list of applicable certs.
+    :return:
+    """
+    certificates = defaultdict(dict)
+    all_certs = get_expiring_authority_certificates()
+    now = arrow.utcnow()
+
+    # group by owner
+    for owner, owner_certs in groupby(all_certs, lambda x: x.owner):
+        # group by expiration interval
+        for interval, interval_certs in groupby(owner_certs, lambda x: (x.not_after - now).days):
+            certificates[owner][interval] = list(interval_certs)
+
+    return certificates
+
+
 def send_plugin_notification(event_type, data, recipients, notification):
     """
     Executes the plugin and handles failure.
@@ -176,6 +224,40 @@ def send_expiration_notifications(exclude):
     return success, failure
 
 
+def send_authority_expiration_notifications():
+    """
+    This function will check for upcoming certificate authority certificate expiration,
+    and send out notification emails at configured intervals.
+    """
+    success = failure = 0
+
+    # security team gets all
+    security_email = current_app.config.get("LEMUR_SECURITY_TEAM_EMAIL")
+
+    for owner, owner_cert_groups in get_eligible_authority_certificates().items():
+        for interval, certificates in owner_cert_groups.items():
+            notification_data = []
+
+            for certificate in certificates:
+                cert_data = certificate_notification_output_schema.dump(
+                    certificate
+                ).data
+                cert_data['self_signed'] = is_selfsigned(certificate.parsed_cert)
+                cert_data['issued_cert_count'] = certificates_service.get_issued_cert_count_for_authority(certificate.root_authority)
+                notification_data.append(cert_data)
+
+            email_recipients = security_email + [owner]
+            if send_default_notification(
+                    "authority_expiration", notification_data, email_recipients,
+                    notification_options=[{'name': 'interval', 'value': interval}]
+            ):
+                success = len(email_recipients)
+            else:
+                failure = len(email_recipients)
+
+    return success, failure
+
+
 def send_default_notification(notification_type, data, targets, notification_options=None):
     """
     Sends a report to the specified target via the default notification plugin. Applicable for any notification_type.

lemur/plugins/lemur_email/plugin.py

@@ -108,7 +108,8 @@ class EmailNotificationPlugin(ExpirationNotificationPlugin):
         if not targets:
             return
 
-        subject = "Lemur: {0} Notification".format(notification_type.capitalize())
+        readable_notification_type = ' '.join(map(lambda x: x.capitalize(), notification_type.split('_')))
+        subject = f"Lemur: {readable_notification_type} Notification"
 
         body = render_html(notification_type, options, message)
 

lemur/plugins/lemur_email/templates/authority_expiration.html

@@ -0,0 +1,179 @@
+<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"
+        "http://www.w3.org/TR/html4/loose.dtd">
+<html lang="en">
+<head>
+    <meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
+    <meta name="viewport" content="initial-scale=1.0">    <!-- So that mobile webkit will display zoomed in -->
+    <meta name="format-detection" content="telephone=no"> <!-- disable auto telephone linking in iOS -->
+
+    <title>Lemur</title>
+</head>
+
+<div style="margin:0;padding:0" bgcolor="#FFFFFF">
+    <table width="100%" height="100%" style="min-width:348px" border="0" cellspacing="0" cellpadding="0">
+        <tbody>
+        <tr height="32px"></tr>
+        <tr align="center">
+            <td width="32px"></td>
+            <td>
+                <table border="0" cellspacing="0" cellpadding="0" style="max-width:600px">
+                    <tbody>
+                    <tr>
+                        <td>
+                            <table width="100%" border="0" cellspacing="0" cellpadding="0">
+                                <tbody>
+                                <tr>
+                                    <td align="left" style="font-family:Roboto-Regular,Helvetica,Arial,sans-serif;font-size:35px;color:#727272; line-height:1.5">
+                                        Lemur
+                                    </td>
+                                </tr>
+                                </tbody>
+                            </table>
+                        </td>
+                    </tr>
+                    <tr height="16"></tr>
+                    <tr>
+                        <td>
+                            <table bgcolor="#F44336" width="100%" border="0" cellspacing="0" cellpadding="0"
+                                   style="min-width:332px;max-width:600px;border:1px solid #e0e0e0;border-bottom:0;border-top-left-radius:3px;border-top-right-radius:3px">
+                                <tbody>
+                                <tr>
+                                    <td height="72px" colspan="3"></td>
+                                </tr>
+                                <tr>
+                                    <td width="32px"></td>
+                                    <td style="font-family:Roboto-Regular,Helvetica,Arial,sans-serif;font-size:24px;color:#ffffff;line-height:1.25">
+                                       Your CA certificate(s) are expiring in {{ message.options | interval }} days!
+                                    </td>
+                                    <td width="32px"></td>
+                                </tr>
+                                <tr>
+                                    <td height="18px" colspan="3"></td>
+                                </tr>
+                                </tbody>
+                            </table>
+                        </td>
+                    </tr>
+                    <tr>
+                        <td>
+                            <table bgcolor="#FAFAFA" width="100%" border="0" cellspacing="0" cellpadding="0"
+                                   style="min-width:332px;max-width:600px;border:1px solid #f0f0f0;border-bottom:1px solid #c0c0c0;border-top:0;border-bottom-left-radius:3px;border-bottom-right-radius:3px">
+                                <tbody>
+                                <tr height="16px">
+                                    <td width="32px" rowspan="3"></td>
+                                    <td></td>
+                                    <td width="32px" rowspan="3"></td>
+                                </tr>
+                                <tr>
+                                    <td>
+                                        <table style="min-width:300px" border="0" cellspacing="0" cellpadding="0">
+                                            <tbody>
+                                            <tr>
+                                                <td style="font-family:Roboto-Regular,Helvetica,Arial,sans-serif;font-size:13px;color:#202020;line-height:1.5">
+                                                    Hi,
+                                                </td>
+                                            </tr>
+                                            <tr>
+                                                <td style="font-family:Roboto-Regular,Helvetica,Arial,sans-serif;font-size:13px;color:#202020;line-height:1.5">
+                                                    <br>This is a Lemur CA certificate expiration notice. The following CA certificates are expiring soon;
+                                                    please take manual action to renew them if necessary. Note that rotating a root CA requires
+                                                    advanced planing and the respective trustStores need to be updated. A sub-CA, on the other hand,
+                                                    does not require any changes to the trustStore. You may also disable notifications via the
+                                                    Notify toggle in Lemur if they are no longer in use.
+                                                    <table border="0" cellspacing="0" cellpadding="0"
+                                                           style="margin-top:48px;margin-bottom:48px">
+                                                        <tbody>
+                                                        {%  for certificate in message.certificates %}
+                                                            <tr valign="middle">
+                                                                <td width="32px"></td>
+                                                                <td width="16px"></td>
+                                                                <td style="line-height:1.2">
+                                                                    <span style="font-family:Roboto-Regular,Helvetica,Arial,sans-serif;font-size:20px;color:#202020">{{ certificate.name }}</span>
+                                                                    <br>
+                                                                    <span style="font-family:Roboto-Regular,Helvetica,Arial,sans-serif;font-size:13px;color:#727272">
+                                                                       {% if certificate.self_signed %}
+                                                                           <b>Root</b>
+                                                                       {% else %}
+                                                                           Intermediate
+                                                                       {% endif %} CA
+                                                                        <br>{{ certificate.issued_cert_count }} issued certificates
+                                                                        <br>{{ certificate.owner }}
+                                                                        <br>{{ certificate.validityEnd | time }}
+                                                                        <a href="https://{{ hostname }}/#/certificates/{{ certificate.name }}" target="_blank">Details</a>
+                                                                    </span>
+                                                                </td>
+                                                            </tr>
+                                                                {%  if not loop.last %}
+                                                                <tr valign="middle">
+                                                                    <td width="32px" height="24px"></td>
+                                                                </tr>
+                                                                {% endif %}
+                                                        {%  endfor %}
+                                                        </tbody>
+                                                    </table>
+                                                </td>
+                                            </tr>
+                                            <tr>
+                                                <td style="font-family:Roboto-Regular,Helvetica,Arial,sans-serif;font-size:13px;color:#202020;line-height:1.5">
+                                                    Your action is required if the above CA certificates are still needed.
+                                                </td>
+                                            </tr>
+                                            <tr>
+                                            </tr>
+                                            <tr>
+                                                <td style="font-family:Roboto-Regular,Helvetica,Arial,sans-serif;font-size:13px;color:#202020;line-height:1.5">
+                                                    <br>Best,<br><span class="il">Lemur</span>
+                                                </td>
+                                            </tr>
+                                            <tr height="16px"></tr>
+                                            <tr>
+                                                <td>
+                                                    <table style="font-family:Roboto-Regular,Helvetica,Arial,sans-serif;font-size:12px;color:#b9b9b9;line-height:1.5">
+                                                        <tbody>
+                                                        <tr>
+                                                            <td>*All expiration times are in UTC<br></td>
+                                                        </tr>
+                                                        </tbody>
+                                                    </table>
+                                                </td>
+                                            </tr>
+                                            </tbody>
+                                        </table>
+                                    </td>
+                                </tr>
+                                <tr height="32px"></tr>
+                                </tbody>
+                            </table>
+                        </td>
+                    </tr>
+                    <tr height="16"></tr>
+                    <tr>
+                        <td style="max-width:600px;font-family:Roboto-Regular,Helvetica,Arial,sans-serif;font-size:10px;color:#bcbcbc;line-height:1.5"></td>
+                    </tr>
+                    <tr>
+                        <td>
+                            <table style="font-family:Roboto-Regular,Helvetica,Arial,sans-serif;font-size:10px;color:#666666;line-height:18px;padding-bottom:10px">
+                                <tbody>
+                                <tr>
+                                    <td>You received this mandatory email announcement to update you about
+                                        important changes to your <span class="il">TLS certificate</span>.
+                                    </td>
+                                </tr>
+                                <tr>
+                                    <td>
+                                        <div style="direction:ltr;text-align:left">© 2020 <span class="il">Lemur</span></div>
+                                    </td>
+                                </tr>
+                                </tbody>
+                            </table>
+                        </td>
+                    </tr>
+                    </tbody>
+                </table>
+            </td>
+            <td width="32px"></td>
+        </tr>
+        <tr height="32px"></tr>
+        </tbody>
+    </table>
+</div>

lemur/plugins/lemur_entrust/plugin.py

@@ -385,7 +385,7 @@ class EntrustSourcePlugin(SourcePlugin):
                     "external_id": str(certificate["trackingId"]),
                     "csr": certificate["csr"],
                     "owner": certificate["tracking"]["requesterEmail"],
-                    "description": f"Type: Entrust {certificate['certType']}\nExtended Key Usage: {certificate['eku']}"
+                    "description": f"Imported by Lemur; Type: Entrust {certificate['certType']}\nExtended Key Usage: {certificate['eku']}"
                 }
                 certs.append(cert)
                 processed_certs += 1

lemur/tests/test_certificates.py

@@ -1377,3 +1377,17 @@ def test_boolean_filter(client):
         headers=VALID_ADMIN_HEADER_TOKEN,
     )
     assert resp.status_code == 200
+
+
+def test_issued_cert_count_for_authority(authority):
+    from lemur.tests.factories import CertificateFactory
+    from lemur.certificates.service import get_issued_cert_count_for_authority
+
+    assert get_issued_cert_count_for_authority(authority) == 0
+
+    # create a few certs issued by the authority
+    CertificateFactory(authority=authority, name="test_issued_cert_count_for_authority1")
+    CertificateFactory(authority=authority, name="test_issued_cert_count_for_authority2")
+    CertificateFactory(authority=authority, name="test_issued_cert_count_for_authority3")
+
+    assert get_issued_cert_count_for_authority(authority) == 3

lemur/tests/test_messaging.py

@@ -5,6 +5,7 @@ import boto3
 import pytest
 from freezegun import freeze_time
 from moto import mock_ses
+from lemur.tests.factories import AuthorityFactory, CertificateFactory
 
 
 @mock_ses
@@ -125,3 +126,47 @@ def test_send_pending_failure_notification(notification_plugin, async_issuer_plu
     verify_sender_email()
 
     assert send_pending_failure_notification(pending_certificate)
+
+
+def test_get_authority_certificates():
+    from lemur.notifications.messaging import get_expiring_authority_certificates
+
+    certificate_1 = create_ca_cert_that_expires_in_days(180)
+    certificate_2 = create_ca_cert_that_expires_in_days(365)
+    create_ca_cert_that_expires_in_days(364)
+    create_ca_cert_that_expires_in_days(366)
+    create_ca_cert_that_expires_in_days(179)
+    create_ca_cert_that_expires_in_days(181)
+    create_ca_cert_that_expires_in_days(1)
+
+    assert set(get_expiring_authority_certificates()) == {certificate_1, certificate_2}
+
+
+@mock_ses
+def test_send_authority_expiration_notifications():
+    from lemur.notifications.messaging import send_authority_expiration_notifications
+    verify_sender_email()
+
+    create_ca_cert_that_expires_in_days(180)
+    create_ca_cert_that_expires_in_days(180)  # two on the same day results in a single email
+    create_ca_cert_that_expires_in_days(365)
+    create_ca_cert_that_expires_in_days(364)
+    create_ca_cert_that_expires_in_days(366)
+    create_ca_cert_that_expires_in_days(179)
+    create_ca_cert_that_expires_in_days(181)
+    create_ca_cert_that_expires_in_days(1)
+
+    assert send_authority_expiration_notifications() == (2, 0)
+
+
+def create_ca_cert_that_expires_in_days(days):
+    now = arrow.utcnow()
+    not_after = now + timedelta(days=days, hours=1)  # a bit more than specified since we'll check in the future
+
+    authority = AuthorityFactory()
+    certificate = CertificateFactory()
+    certificate.not_after = not_after
+    certificate.notify = True
+    certificate.root_authority_id = authority.id
+    certificate.authority_id = None
+    return certificate

requirements-dev.txt

@@ -6,7 +6,7 @@
 #
 appdirs==1.4.3            # via virtualenv
 bleach==3.1.4             # via readme-renderer
-certifi==2020.11.8        # via requests
+certifi==2020.12.5        # via requests
 cffi==1.14.0              # via cryptography
 cfgv==3.1.0               # via pre-commit
 chardet==3.0.4            # via requests

requirements-docs.txt

@@ -4,92 +4,22 @@
 #
 #    pip-compile --no-index --output-file=requirements-docs.txt requirements-docs.in
 #
-acme==1.9.0               # via -r requirements.txt
 alabaster==0.7.12         # via sphinx
-alembic-autogenerate-enums==0.0.2  # via -r requirements.txt
-alembic==1.4.2            # via -r requirements.txt, flask-migrate
-amqp==2.5.2               # via -r requirements.txt, kombu
-aniso8601==8.0.0          # via -r requirements.txt, flask-restful
-arrow==0.17.0             # via -r requirements.txt
-asyncpool==1.0            # via -r requirements.txt
 babel==2.8.0              # via sphinx
-bcrypt==3.1.7             # via -r requirements.txt, flask-bcrypt, paramiko
+certifi==2020.12.5        # via requests
-beautifulsoup4==4.9.1     # via -r requirements.txt, cloudflare
+chardet==3.0.4            # via requests
-billiard==3.6.3.0         # via -r requirements.txt, celery
-blinker==1.4              # via -r requirements.txt, flask-mail, flask-principal, raven
-boto3==1.16.25            # via -r requirements.txt
-botocore==1.19.25         # via -r requirements.txt, boto3, s3transfer
-celery[redis]==4.4.2      # via -r requirements.txt
-certifi==2020.11.8        # via -r requirements.txt, requests
-certsrv==2.1.1            # via -r requirements.txt
-cffi==1.14.0              # via -r requirements.txt, bcrypt, cryptography, pynacl
-chardet==3.0.4            # via -r requirements.txt, requests
-click==7.1.2              # via -r requirements.txt, flask
-cloudflare==2.8.13        # via -r requirements.txt
-cryptography==3.2.1       # via -r requirements.txt, acme, josepy, paramiko, pyopenssl, requests
-dnspython3==1.15.0        # via -r requirements.txt
-dnspython==1.15.0         # via -r requirements.txt, dnspython3
 docutils==0.15.2          # via sphinx
-dyn==1.8.1                # via -r requirements.txt
+idna==2.9                 # via requests
-flask-bcrypt==0.7.1       # via -r requirements.txt
-flask-cors==3.0.9         # via -r requirements.txt
-flask-mail==0.9.1         # via -r requirements.txt
-flask-migrate==2.5.3      # via -r requirements.txt
-flask-principal==0.4.0    # via -r requirements.txt
-flask-replicated==1.4     # via -r requirements.txt
-flask-restful==0.3.8      # via -r requirements.txt
-flask-script==2.0.6       # via -r requirements.txt
-flask-sqlalchemy==2.4.4   # via -r requirements.txt, flask-migrate
-flask==1.1.2              # via -r requirements.txt, flask-bcrypt, flask-cors, flask-mail, flask-migrate, flask-principal, flask-restful, flask-script, flask-sqlalchemy, raven
-future==0.18.2            # via -r requirements.txt
-gunicorn==20.0.4          # via -r requirements.txt
-hvac==0.10.5              # via -r requirements.txt
-idna==2.9                 # via -r requirements.txt, requests
 imagesize==1.2.0          # via sphinx
-inflection==0.5.1         # via -r requirements.txt
+jinja2==2.11.2            # via sphinx
-itsdangerous==1.1.0       # via -r requirements.txt, flask
+markupsafe==1.1.1         # via jinja2
-javaobj-py3==0.4.0.1      # via -r requirements.txt, pyjks
-jinja2==2.11.2            # via -r requirements.txt, flask, sphinx
-jmespath==0.9.5           # via -r requirements.txt, boto3, botocore
-josepy==1.3.0             # via -r requirements.txt, acme
-jsonlines==1.2.0          # via -r requirements.txt, cloudflare
-kombu==4.6.8              # via -r requirements.txt, celery
-lockfile==0.12.2          # via -r requirements.txt
-logmatic-python==0.1.7    # via -r requirements.txt
-mako==1.1.2               # via -r requirements.txt, alembic
-markupsafe==1.1.1         # via -r requirements.txt, jinja2, mako
-marshmallow-sqlalchemy==0.23.1  # via -r requirements.txt
-marshmallow==2.20.4       # via -r requirements.txt, marshmallow-sqlalchemy
-ndg-httpsclient==0.5.1    # via -r requirements.txt
 packaging==20.3           # via sphinx
-paramiko==2.7.2           # via -r requirements.txt
-pem==20.1.0               # via -r requirements.txt
-psycopg2==2.8.6           # via -r requirements.txt
-pyasn1-modules==0.2.8     # via -r requirements.txt, pyjks, python-ldap
-pyasn1==0.4.8             # via -r requirements.txt, ndg-httpsclient, pyasn1-modules, pyjks, python-ldap
-pycparser==2.20           # via -r requirements.txt, cffi
-pycryptodomex==3.9.7      # via -r requirements.txt, pyjks
 pygments==2.6.1           # via sphinx
-pyjks==20.0.0             # via -r requirements.txt
-pyjwt==1.7.1              # via -r requirements.txt
-pynacl==1.3.0             # via -r requirements.txt, paramiko
-pyopenssl==20.0.0         # via -r requirements.txt, acme, josepy, ndg-httpsclient, requests
 pyparsing==2.4.7          # via packaging
-pyrfc3339==1.1            # via -r requirements.txt, acme
+pytz==2019.3              # via babel
-python-dateutil==2.8.1    # via -r requirements.txt, alembic, arrow, botocore
+requests==2.25.0          # via sphinx
-python-editor==1.0.4      # via -r requirements.txt, alembic
+six==1.15.0               # via packaging, sphinxcontrib-httpdomain
-python-json-logger==0.1.11  # via -r requirements.txt, logmatic-python
-pytz==2019.3              # via -r requirements.txt, acme, babel, celery, flask-restful, pyrfc3339
-pyyaml==5.3.1             # via -r requirements.txt, cloudflare
-raven[flask]==6.10.0      # via -r requirements.txt
-redis==3.5.3              # via -r requirements.txt, celery
-requests-toolbelt==0.9.1  # via -r requirements.txt, acme
-requests[security]==2.25.0  # via -r requirements.txt, acme, certsrv, cloudflare, hvac, requests-toolbelt, sphinx
-retrying==1.3.3           # via -r requirements.txt
-s3transfer==0.3.3         # via -r requirements.txt, boto3
-six==1.15.0               # via -r requirements.txt, acme, bcrypt, cryptography, flask-cors, flask-restful, hvac, josepy, jsonlines, packaging, pynacl, pyopenssl, python-dateutil, retrying, sphinxcontrib-httpdomain, sqlalchemy-utils
 snowballstemmer==2.0.0    # via sphinx
-soupsieve==2.0.1          # via -r requirements.txt, beautifulsoup4
 sphinx-rtd-theme==0.5.0   # via -r requirements-docs.in
 sphinx==3.3.1             # via -r requirements-docs.in, sphinx-rtd-theme, sphinxcontrib-httpdomain
 sphinxcontrib-applehelp==1.0.2  # via sphinx
@@ -99,14 +29,7 @@ sphinxcontrib-httpdomain==1.7.0  # via -r requirements-docs.in
 sphinxcontrib-jsmath==1.0.1  # via sphinx
 sphinxcontrib-qthelp==1.0.3  # via sphinx
 sphinxcontrib-serializinghtml==1.1.4  # via sphinx
-sqlalchemy-utils==0.36.8  # via -r requirements.txt
+urllib3==1.25.8           # via requests
-sqlalchemy==1.3.16        # via -r requirements.txt, alembic, flask-sqlalchemy, marshmallow-sqlalchemy, sqlalchemy-utils
-tabulate==0.8.7           # via -r requirements.txt
-twofish==0.3.0            # via -r requirements.txt, pyjks
-urllib3==1.25.8           # via -r requirements.txt, botocore, requests
-vine==1.3.0               # via -r requirements.txt, amqp, celery
-werkzeug==1.0.1           # via -r requirements.txt, flask
-xmltodict==0.12.0         # via -r requirements.txt
 
 # The following packages are considered to be unsafe in a requirements file:
 # setuptools

requirements-tests.txt

@@ -8,12 +8,12 @@ appdirs==1.4.3            # via black
 attrs==19.3.0             # via jsonschema, pytest
 aws-sam-translator==1.22.0  # via cfn-lint
 aws-xray-sdk==2.5.0       # via moto
-bandit==1.6.2             # via -r requirements-tests.in
+bandit==1.6.3             # via -r requirements-tests.in
 black==20.8b1             # via -r requirements-tests.in
-boto3==1.16.25            # via aws-sam-translator, moto
+boto3==1.16.30            # via aws-sam-translator, moto
 boto==2.49.0              # via moto
-botocore==1.19.25         # via aws-xray-sdk, boto3, moto, s3transfer
+botocore==1.19.30         # via aws-xray-sdk, boto3, moto, s3transfer
-certifi==2020.11.8        # via requests
+certifi==2020.12.5        # via requests
 cffi==1.14.0              # via cryptography
 cfn-lint==0.29.5          # via moto
 chardet==3.0.4            # via requests
@@ -24,7 +24,7 @@ decorator==4.4.2          # via networkx
 docker==4.2.0             # via moto
 ecdsa==0.14.1             # via moto, python-jose, sshpubkeys
 factory-boy==3.1.0        # via -r requirements-tests.in
-faker==4.17.1             # via -r requirements-tests.in, factory-boy
+faker==5.0.0              # via -r requirements-tests.in, factory-boy
 fakeredis==1.4.5          # via -r requirements-tests.in
 flask==1.1.2              # via pytest-flask
 freezegun==1.0.0          # via -r requirements-tests.in

requirements.txt

@@ -4,7 +4,7 @@
 #
 #    pip-compile --no-index --output-file=requirements.txt requirements.in
 #
-acme==1.9.0               # via -r requirements.in
+acme==1.10.1              # via -r requirements.in
 alembic-autogenerate-enums==0.0.2  # via -r requirements.in
 alembic==1.4.2            # via flask-migrate
 amqp==2.5.2               # via kombu
@@ -15,15 +15,15 @@ bcrypt==3.1.7             # via flask-bcrypt, paramiko
 beautifulsoup4==4.9.1     # via cloudflare
 billiard==3.6.3.0         # via celery
 blinker==1.4              # via flask-mail, flask-principal, raven
-boto3==1.16.25            # via -r requirements.in
+boto3==1.16.30            # via -r requirements.in
-botocore==1.19.25         # via -r requirements.in, boto3, s3transfer
+botocore==1.19.30         # via -r requirements.in, boto3, s3transfer
 celery[redis]==4.4.2      # via -r requirements.in
-certifi==2020.11.8        # via -r requirements.in, requests
+certifi==2020.12.5        # via -r requirements.in, requests
 certsrv==2.1.1            # via -r requirements.in
 cffi==1.14.0              # via bcrypt, cryptography, pynacl
 chardet==3.0.4            # via requests
 click==7.1.2              # via flask
-cloudflare==2.8.13        # via -r requirements.in
+cloudflare==2.8.14        # via -r requirements.in
 cryptography==3.2.1       # via -r requirements.in, acme, josepy, paramiko, pyopenssl, requests
 dnspython3==1.15.0        # via -r requirements.in
 dnspython==1.15.0         # via dnspython3