Merge pull request #3425 from jtschladen/fix-docs

Debug docs

CHANGELOG.rst

@@ -2,7 +2,7 @@ Changelog
 =========
 
 0.8.0 - `2020-11-13`
-~~~~~~~~~~~~~~
+~~~~~~~~~~~~~~~~~~~~
 
 This release comes after more than two years and contains many interesting new features and improvements.
 In addition to multiple new plugins, such as ACME-http01, ADCS, PowerDNS, UltraDNS, Entrust, SNS, many of Lemur's existing
@@ -84,7 +84,7 @@ Upgrading
 
 
 0.7 - `2018-05-07`
-~~~~~~~~~~~~~~
+~~~~~~~~~~~~~~~~~~
 
 This release adds LetsEncrypt support with DNS providers Dyn, Route53, and Cloudflare, and expands on the pending certificate functionality.
 The linux_dst plugin will also be deprecated and removed.
@@ -121,8 +121,7 @@ Happy Holidays! This is a big release with lots of bug fixes and features. Below
 
 Features:
 
-* Per-certificate rotation policies, requires a database migration. The default rotation policy for all certificates.
+* Per-certificate rotation policies, requires a database migration. The default rotation policy for all certificates is 30 days. Every certificate will gain a policy regardless of if auto-rotation is used.
-is 30 days. Every certificate will gain a policy regardless of if auto-rotation is used.
 * Adds per-user API Keys, allows users to issue multiple long-lived API tokens with the same permission as the user creating them.
 * Adds the ability to revoke certificates from the Lemur UI/API, this is currently only supported for the digicert CIS and cfssl plugins.
 * Allow destinations to support an export function. Useful for file system destinations e.g. S3 to specify the export plugin you wish to run before being sent to the destination.
@@ -166,13 +165,9 @@ Big thanks to neilschelly for quite a lot of improvements to the `lemur-cryptogr
 
 Other Highlights:
 
-* Closed `#501 <https://github.com/Netflix/lemur/issues/501>`_ - Endpoint resource as now kept in sync via an
+* Closed `#501 <https://github.com/Netflix/lemur/issues/501>`_ - Endpoint resource as now kept in sync via an expiration mechanism. Such that non-existant endpoints gracefully fall out of Lemur. Certificates are never removed from Lemur.
-expiration mechanism. Such that non-existant endpoints gracefully fall out of Lemur. Certificates are never
+* Closed `#551 <https://github.com/Netflix/lemur/pull/551>`_ - Added the ability to create a 4096 bit key during certificate creation. Closed `#528 <https://github.com/Netflix/lemur/pull/528>`_ to ensure that issuer plugins supported the new 4096 bit keys.
-removed from Lemur.
+* Closed `#566 <https://github.com/Netflix/lemur/issues/566>`_ - Fixed an issue changing the notification status for  certificates without private keys.
-* Closed `#551 <https://github.com/Netflix/lemur/pull/551>`_ - Added the ability to create a 4096 bit key during certificate
-creation. Closed `#528 <https://github.com/Netflix/lemur/pull/528>`_ to ensure that issuer plugins supported the new 4096 bit keys.
-* Closed `#566 <https://github.com/Netflix/lemur/issues/566>`_ - Fixed an issue changing the notification status for  certificates
-without private keys.
 * Closed `#594 <https://github.com/Netflix/lemur/issues/594>`_ - Added `replaced` field indicating if a certificate has been superseded.
 * Closed `#602 <https://github.com/Netflix/lemur/issues/602>`_ - AWS plugin added support for ALBs for endpoint tracking.
 
@@ -196,12 +191,8 @@ Upgrading
 
 There have been quite a few issues closed in this release. Some notables:
 
-* Closed `#284 <https://github.com/Netflix/lemur/issues/284>`_ - Created new models for `Endpoints` created associated
+* Closed `#284 <https://github.com/Netflix/lemur/issues/284>`_ - Created new models for `Endpoints` created associated AWS ELB endpoint tracking code. This was the major stated goal of this milestone and should serve as the basis for future enhancements of Lemur's certificate 'deployment' capabilities.
-AWS ELB endpoint tracking code. This was the major stated goal of this milestone and should serve as the basis for
+* Closed `#334 <https://github.com/Netflix/lemur/issues/334>`_ - Lemur not has the ability to restrict certificate expiration dates to weekdays.
-future enhancements of Lemur's certificate 'deployment' capabilities.
-
-* Closed `#334 <https://github.com/Netflix/lemur/issues/334>`_ - Lemur not has the ability
-to restrict certificate expiration dates to weekdays.
 
 Several fixes/tweaks to Lemurs python3 support (thanks chadhendrie!)
 

docs/administration.rst

@@ -78,13 +78,13 @@ Basic Configuration
             The default connection pool size is 5 for sqlalchemy managed connections.   Depending on the number of Lemur instances,
             please specify per instance connection pool size.  Below is an example to set connection pool size to 10.
 
-        ::
+    ::
 
         SQLALCHEMY_POOL_SIZE = 10
 
 
     .. warning::
-This is an optional setting but important to review and set for optimal database connection usage and for overall database performance.
+        This is an optional setting but important to review and set for optimal database connection usage and for overall database performance.
 
 .. data:: SQLALCHEMY_MAX_OVERFLOW
     :noindex:
@@ -99,7 +99,7 @@ This is an optional setting but important to review and set for optimal database
 
 
     .. note::
-Specifying the `SQLALCHEMY_MAX_OVERFLOW` to 0 will enforce limit to not create connections above specified pool size.
+        Specifying the `SQLALCHEMY_MAX_OVERFLOW` to 0 will enforce limit to not create connections above specified pool size.
 
 
 .. data:: LEMUR_ALLOW_WEEKEND_EXPIRATION
@@ -174,6 +174,7 @@ Specifying the `SQLALCHEMY_MAX_OVERFLOW` to 0 will enforce limit to not create c
 
 .. data:: PUBLIC_CA_MAX_VALIDITY_DAYS
     :noindex:
+
         Use this config to override the limit of 397 days of validity for certificates issued by CA/Browser compliant authorities.
         The authorities with cab_compliant option set to true will use this config. The example below overrides the default validity
         of 397 days and sets it to 365 days.
@@ -185,6 +186,7 @@ Specifying the `SQLALCHEMY_MAX_OVERFLOW` to 0 will enforce limit to not create c
 
 .. data:: DEFAULT_VALIDITY_DAYS
     :noindex:
+
         Use this config to override the default validity of 365 days for certificates offered through Lemur UI. Any CA which
         is not CA/Browser Forum compliant will be using this value as default validity to be displayed on UI. Please
         note that this config is used for cert issuance only through Lemur UI. The example below overrides the default validity
@@ -904,10 +906,12 @@ Active Directory Certificate Services Plugin
 
 .. data:: ADCS_START
     :noindex:
+
         Used in ADCS-Sourceplugin. Minimum id of the first certificate to be returned. ID is increased by one until ADCS_STOP. Missing cert-IDs are ignored
 
 .. data:: ADCS_STOP
     :noindex:
+
         Used for ADCS-Sourceplugin. Maximum id of the certificates returned. 
         
 
@@ -1640,7 +1644,7 @@ Slack
 
 
 AWS (Source)
-----
+------------
 
 :Authors:
     Kevin Glisson <kglisson@netflix.com>,
@@ -1653,7 +1657,7 @@ AWS (Source)
 
 
 AWS (Destination)
-----
+-----------------
 
 :Authors:
     Kevin Glisson <kglisson@netflix.com>,
@@ -1666,7 +1670,7 @@ AWS (Destination)
 
 
 AWS (SNS Notification)
------
+----------------------
 
 :Authors:
     Jasmine Schladen <jschladen@netflix.com>

docs/developer/plugins/index.rst

@@ -145,8 +145,7 @@ The `IssuerPlugin` doesn't have any options like Destination, Source, and Notifi
 any fields you might need to submit a request to a third party. If there are additional options you need
 in your plugin feel free to open an issue, or look into adding additional options to issuers yourself.
 
-Asynchronous Certificates
+**Asynchronous Certificates**
-^^^^^^^^^^^^^^^^^^^^^^^^^
 An issuer may take some time to actually issue a certificate for an order.  In this case, a `PendingCertificate` is returned, which holds information to recreate a `Certificate` object at a later time.  Then, `get_ordered_certificate()` should be run periodically via `python manage.py pending_certs fetch -i all` to attempt to retrieve an ordered certificate::
 
     def get_ordered_ceriticate(self, order_id):
@@ -154,9 +153,10 @@ An issuer may take some time to actually issue a certificate for an order.  In t
         # retrieve an order, and check if there is an issued certificate attached to it
 
 `cancel_ordered_certificate()` should be implemented to allow an ordered certificate to be canceled before it is issued::
-    def cancel_ordered_certificate(self, pending_cert, **kwargs):
+
-        # pending_cert should contain the necessary information to match an order
+        def cancel_ordered_certificate(self, pending_cert, **kwargs):
-        # kwargs can be given to provide information to the issuer for canceling
+            # pending_cert should contain the necessary information to match an order
+            # kwargs can be given to provide information to the issuer for canceling
 
 Destination
 -----------
@@ -286,7 +286,7 @@ The `ExportPlugin` object requires the implementation of one function::
 
 
 Custom TLS Provider
-------
+-------------------
 
 Managing TLS at the enterprise scale could be hard and often organizations offer custom wrapper implementations. It could
 be ideal to use those while making calls to internal services. The `TLSPlugin` would help to achieve this. It requires the

docs/guide/index.rst

@@ -65,6 +65,7 @@ Import an Existing Certificate
     You can add notification options and upload the created certificate to a destination, both
     of these are editable features and can be changed after the certificate has been created.
 
+.. _CreateANewUser:
 
 Create a New User
 ~~~~~~~~~~~~~~~~~

docs/production/index.rst

@@ -501,7 +501,7 @@ rely on celery to create the DNS record. This will change when we implement mix
 
 To create a HTTP compatible Authority, you first need to create a new destination that will be used to deploy the
 challenge token. Visit `Admin` -> `Destination` and click `Create`. The path you provide for the destination needs to
-be the exact path that is called when the ACME providers calls ``http://<domain>/.well-known/acme-challenge/`. The
+be the exact path that is called when the ACME providers calls `http://<domain>/.well-known/acme-challenge/`. The
 token part will be added dynamically by the acme_upload.
 Currently only the SFTP and S3 Bucket destination support the ACME HTTP challenge.
 

docs/quickstart/index.rst

@@ -148,7 +148,7 @@ Before Lemur will run you need to fill in a few required variables in the config
     LEMUR_DEFAULT_ORGANIZATIONAL_UNIT
 
 Set Up Postgres
---------------
+---------------
 
 For production, a dedicated database is recommended, for this guide we will assume postgres has been installed and is on the same machine that Lemur is installed on.
 
@@ -186,11 +186,12 @@ In addition to creating a new user, Lemur also creates a few default email notif
 Your database installation requires the pg_trgm extension. If you do not have this installed already, you can allow the script to install this for you by adding the SUPERUSER permission to the lemur database user.
 
 .. code-block:: bash
+
     sudo -u postgres -i
     psql
     postgres=# ALTER USER lemur WITH SUPERUSER
 
-Additional notifications can be created through the UI or API.  See :ref:`Creating Notifications <CreatingNotifications>` and :ref:`Command Line Interface <CommandLineInterface>` for details.
+Additional notifications can be created through the UI or API.  See :ref:`Notification Options <NotificationOptions>` and :ref:`Command Line Interface <CommandLineInterface>` for details.
 
 **Make note of the password used as this will be used during first login to the Lemur UI.**
 
@@ -202,15 +203,16 @@ Additional notifications can be created through the UI or API.  See :ref:`Creati
 .. note:: If you added the SUPERUSER permission to the lemur database user above, it is recommended you revoke that permission now.
 
 .. code-block:: bash
+
     sudo -u postgres -i
     psql
     postgres=# ALTER USER lemur WITH NOSUPERUSER
 
 
-.. note:: It is recommended that once the ``lemur`` user is created that you create individual users for every day access.  There is currently no way for a user to self enroll for Lemur access, they must have an administrator create an account for them or be enrolled automatically through SSO.  This can be done through the CLI or UI.  See :ref:`Creating Users <CreatingUsers>` and :ref:`Command Line Interface <CommandLineInterface>` for details.
+.. note:: It is recommended that once the ``lemur`` user is created that you create individual users for every day access.  There is currently no way for a user to self enroll for Lemur access, they must have an administrator create an account for them or be enrolled automatically through SSO.  This can be done through the CLI or UI.  See :ref:`Creating a New User <CreateANewUser>` and :ref:`Command Line Interface <CommandLineInterface>` for details.
 
 Set Up a Reverse Proxy
----------------------
+----------------------
 
 By default, Lemur runs on port 8000.  Even if you change this, under normal conditions you won't be able to bind to port 80. To get around this (and to avoid running Lemur as a privileged user, which you shouldn't), we need to set up a simple web proxy. There are many different web servers you can use for this, we like and recommend Nginx.
 

lemur/authorities/views.py

@@ -132,31 +132,31 @@ class AuthoritiesList(AuthenticatedResource):
               Accept: application/json, text/javascript
               Content-Type: application/json;charset=UTF-8
 
-             {
+              {
-                "country": "US",
+                 "country": "US",
-                "state": "California",
+                 "state": "California",
-                "location": "Los Gatos",
+                 "location": "Los Gatos",
-                "organization": "Netflix",
+                 "organization": "Netflix",
-                "organizationalUnit": "Operations",
+                 "organizationalUnit": "Operations",
-                "type": "root",
+                 "type": "root",
-                "signingAlgorithm": "sha256WithRSA",
+                 "signingAlgorithm": "sha256WithRSA",
-                "sensitivity": "medium",
+                 "sensitivity": "medium",
-                "keyType": "RSA2048",
+                 "keyType": "RSA2048",
-                "plugin": {
+                 "plugin": {
-                    "slug": "cloudca-issuer"
+                     "slug": "cloudca-issuer"
-                },
+                 },
-                "name": "TimeTestAuthority5",
+                 "name": "TimeTestAuthority5",
-                "owner": "secure@example.com",
+                 "owner": "secure@example.com",
-                "description": "test",
+                 "description": "test",
-                "commonName": "AcommonName",
+                 "commonName": "AcommonName",
-                "validityYears": "20",
+                 "validityYears": "20",
-                "extensions": {
+                 "extensions": {
-                    "subAltNames": {
+                     "subAltNames": {
-                        "names": []
+                         "names": []
-                    },
+                     },
-                    "custom": []
+                     "custom": []
-                }
+                 }
-             }
+              }
 
            **Example response**:
 
@@ -218,8 +218,7 @@ class AuthoritiesList(AuthenticatedResource):
            :arg parent: the parent authority if this is to be a subca
            :arg signingAlgorithm: algorithm used to sign the authority
            :arg keyType: key type
-           :arg sensitivity: the sensitivity of the root key, for CloudCA this determines if the root keys are stored
+           :arg sensitivity: the sensitivity of the root key, for CloudCA this determines if the root keys are stored in an HSM
-           in an HSM
            :arg keyName: name of the key to store in the HSM (CloudCA)
            :arg serialNumber: serial number of the authority
            :arg firstSerial: specifies the starting serial number for certificates issued off of this authority
@@ -494,23 +493,48 @@ class CertificateAuthority(AuthenticatedResource):
 class AuthorityVisualizations(AuthenticatedResource):
     def get(self, authority_id):
         """
-        {"name": "flare",
+        .. http:get:: /authorities/1/visualize
-        "children": [
+
-            {
+           Authority visualization
-                "name": "analytics",
+
-                "children": [
+           **Example request**:
-                    {
+
-                        "name": "cluster",
+           .. sourcecode:: http
-                        "children": [
+
-                            {"name": "AgglomerativeCluster", "size": 3938},
+              GET /certificates/1/visualize HTTP/1.1
-                            {"name": "CommunityStructure", "size": 3812},
+              Host: example.com
-                            {"name": "HierarchicalCluster", "size": 6714},
+              Accept: application/json, text/javascript
-                            {"name": "MergeEdge", "size": 743}
+
-                        ]
+           **Example response**:
-                    }
+
-                ]
+           .. sourcecode:: http
-            }
+
-        ]}
+              HTTP/1.1 200 OK
+              Vary: Accept
+              Content-Type: text/javascript
+
+                {"name": "flare",
+                    "children": [
+                        {
+                            "name": "analytics",
+                            "children": [
+                                {
+                                    "name": "cluster",
+                                    "children": [
+                                        {"name": "AgglomerativeCluster", "size": 3938},
+                                        {"name": "CommunityStructure", "size": 3812},
+                                        {"name": "HierarchicalCluster", "size": 6714},
+                                        {"name": "MergeEdge", "size": 743}
+                                    ]
+                                }
+                            ]
+                        }
+                    ]
+                }
+
+           :reqheader Authorization: OAuth token to authenticate
+           :statuscode 200: no error
+           :statuscode 403: unauthenticated
         """
         authority = service.get(authority_id)
         return dict(

lemur/certificates/views.py

@@ -59,6 +59,7 @@ class CertificatesListValid(AuthenticatedResource):
            **Example request**:
 
            .. sourcecode:: http
+
               GET /certificates/valid?filter=cn;*.test.example.net&owner=joe@example.com&page=1&count=20
               HTTP/1.1
               Host: example.com

lemur/destinations/service.py

@@ -21,7 +21,7 @@ def create(label, plugin_name, options, description=None):
 
     :param label: Destination common name
     :param description:
-    :rtype : Destination
+    :rtype: Destination
     :return: New destination
     """
     # remove any sub-plugin objects before try to save the json options
@@ -50,7 +50,7 @@ def update(destination_id, label, plugin_name, options, description):
     :param plugin_name:
     :param options:
     :param description:
-    :rtype : Destination
+    :rtype: Destination
     :return:
     """
     destination = get(destination_id)
@@ -81,7 +81,7 @@ def get(destination_id):
     Retrieves an destination by its lemur assigned ID.
 
     :param destination_id: Lemur assigned ID
-    :rtype : Destination
+    :rtype: Destination
     :return:
     """
     return database.get(Destination, destination_id)

lemur/notifications/service.py

@@ -94,7 +94,7 @@ def create(label, plugin_name, options, description, certificates):
     :param options:
     :param description:
     :param certificates:
-    :rtype : Notification
+    :rtype: Notification
     :return:
     """
     notification = Notification(
@@ -115,7 +115,7 @@ def update(notification_id, label, plugin_name, options, description, active, ce
     :param description:
     :param active:
     :param certificates:
-    :rtype : Notification
+    :rtype: Notification
     :return:
     """
     notification = get(notification_id)
@@ -144,7 +144,7 @@ def get(notification_id):
     Retrieves an notification by its lemur assigned ID.
 
     :param notification_id: Lemur assigned ID
-    :rtype : Notification
+    :rtype: Notification
     :return:
     """
     return database.get(Notification, notification_id)

lemur/plugins/lemur_aws/plugin.py

@@ -450,7 +450,8 @@ class S3DestinationPlugin(ExportDestinationPlugin):
 
     def upload_acme_token(self, token_path, token, options, **kwargs):
         """
-         This is called from the acme http challenge
+        This is called from the acme http challenge
+
         :param self:
         :param token_path:
         :param token:

requirements-docs.in

@@ -3,8 +3,38 @@
 # from requirements-docs.txt
 # However, dependabot doesn't use `make up-reqs`, so we have to replicate the necessary dependencies here
 # Without including these dependencies, the docs are unable to include generated autodocs
+acme
+arrow
+boto3
+botocore
+CloudFlare
+cryptography
+dnspython3
+dyn
 Flask
+Flask-Bcrypt
+Flask-Cors
+Flask-Mail
+Flask-Migrate
+Flask-Principal
+Flask-RESTful
+Flask-Script
+Flask-SQLAlchemy
 flask_replicated
+gunicorn
+inflection
+josepy
+logmatic-python
+marshmallow-sqlalchemy
+marshmallow<2.20.5 #schema duplicate issues https://github.com/marshmallow-code/marshmallow-sqlalchemy/issues/121
+pem
+pyjwt
+pyOpenSSL
+raven[flask]
+retrying
+SQLAlchemy-Utils
+tabulate
+xmltodict
 
 # docs specific
 sphinx

requirements-docs.txt

@@ -4,36 +4,92 @@
 #
 #    pip-compile --no-index --output-file=requirements-docs.txt requirements-docs.in
 #
+acme==1.12.0
+    # manual debug
 alabaster==0.7.12
     # via sphinx
+arrow==0.17.0
+    # manual debug
 babel==2.8.0
     # via sphinx
+boto3==1.17.7
+    # manual debug
+botocore==1.20.7
+    # manual debug
 certifi==2020.12.5
     # via requests
 chardet==3.0.4
     # via requests
+cloudflare==2.8.15
+    # manual debug
+cryptography==3.4.5
+    # manual debug
+dnspython3==1.15.0
+    # manual debug
+dnspython==1.15.0
+    # manual debug
 docutils==0.15.2
     # via sphinx
+dyn==1.8.1
+    # manual debug
 idna==2.9
     # via requests
 imagesize==1.2.0
     # via sphinx
 flask==1.1.2
     # manual debug
+flask-bcrypt==0.7.1
+    # manual debug
+flask-cors==3.0.10
+    # manual debug
+flask-mail==0.9.1
+    # manual debug
+flask-migrate==2.6.0
+    # manual debug
+flask-principal==0.4.0
+    # manual debug
 flask-replicated==1.4
     # manual debug
+flask-restful==0.3.8
+    # manual debug
+flask-script==2.0.6
+    # manual debug
+flask-sqlalchemy==2.4.4
+    # manual debug
+gunicorn==20.0.4
+    # manual debug
+inflection==0.5.1
+    # manual debug
 jinja2==2.11.3
     # via sphinx
+josepy==1.3.0
+    # manual debug
+logmatic-python==0.1.7
+    # manual debug
 markupsafe==1.1.1
     # via jinja2
+marshmallow-sqlalchemy==0.23.1
+    # manual debug
+marshmallow==2.20.4
+    # manual debug
 packaging==20.3
     # via sphinx
+pem==21.1.0
+    # manual debug
 pygments==2.6.1
     # via sphinx
+pyjwt==2.0.1
+    # manual debug
+pyopenssl==20.0.1
+    # manual debug
 pyparsing==2.4.7
     # via packaging
 pytz==2019.3
     # via babel
+raven[flask]==6.10.0
+    # manual debug
+retrying==1.3.3
+    # manual debug
 requests==2.25.1
     # via sphinx
 six==1.15.0
@@ -63,8 +119,14 @@ sphinxcontrib-qthelp==1.0.3
     # via sphinx
 sphinxcontrib-serializinghtml==1.1.4
     # via sphinx
+sqlalchemy-utils==0.36.8
+    # manual debug
+tabulate==0.8.7
+    # manual debug
 urllib3==1.25.8
     # via requests
+xmltodict==0.12.0
+    # manual debug
 
 # The following packages are considered to be unsafe in a requirements file:
 # setuptools