diff --git a/CHANGELOG.rst b/CHANGELOG.rst index 67b792f8..22a9341f 100644 --- a/CHANGELOG.rst +++ b/CHANGELOG.rst @@ -2,7 +2,7 @@ Changelog ========= 0.8.0 - `2020-11-13` -~~~~~~~~~~~~~~ +~~~~~~~~~~~~~~~~~~~~ This release comes after more than two years and contains many interesting new features and improvements. In addition to multiple new plugins, such as ACME-http01, ADCS, PowerDNS, UltraDNS, Entrust, SNS, many of Lemur's existing @@ -84,7 +84,7 @@ Upgrading 0.7 - `2018-05-07` -~~~~~~~~~~~~~~ +~~~~~~~~~~~~~~~~~~ This release adds LetsEncrypt support with DNS providers Dyn, Route53, and Cloudflare, and expands on the pending certificate functionality. The linux_dst plugin will also be deprecated and removed. @@ -121,8 +121,7 @@ Happy Holidays! This is a big release with lots of bug fixes and features. Below Features: -* Per-certificate rotation policies, requires a database migration. The default rotation policy for all certificates. -is 30 days. Every certificate will gain a policy regardless of if auto-rotation is used. +* Per-certificate rotation policies, requires a database migration. The default rotation policy for all certificates is 30 days. Every certificate will gain a policy regardless of if auto-rotation is used. * Adds per-user API Keys, allows users to issue multiple long-lived API tokens with the same permission as the user creating them. * Adds the ability to revoke certificates from the Lemur UI/API, this is currently only supported for the digicert CIS and cfssl plugins. * Allow destinations to support an export function. Useful for file system destinations e.g. S3 to specify the export plugin you wish to run before being sent to the destination. @@ -166,13 +165,9 @@ Big thanks to neilschelly for quite a lot of improvements to the `lemur-cryptogr Other Highlights: -* Closed `#501 `_ - Endpoint resource as now kept in sync via an -expiration mechanism. Such that non-existant endpoints gracefully fall out of Lemur. Certificates are never -removed from Lemur. -* Closed `#551 `_ - Added the ability to create a 4096 bit key during certificate -creation. Closed `#528 `_ to ensure that issuer plugins supported the new 4096 bit keys. -* Closed `#566 `_ - Fixed an issue changing the notification status for certificates -without private keys. +* Closed `#501 `_ - Endpoint resource as now kept in sync via an expiration mechanism. Such that non-existant endpoints gracefully fall out of Lemur. Certificates are never removed from Lemur. +* Closed `#551 `_ - Added the ability to create a 4096 bit key during certificate creation. Closed `#528 `_ to ensure that issuer plugins supported the new 4096 bit keys. +* Closed `#566 `_ - Fixed an issue changing the notification status for certificates without private keys. * Closed `#594 `_ - Added `replaced` field indicating if a certificate has been superseded. * Closed `#602 `_ - AWS plugin added support for ALBs for endpoint tracking. @@ -196,12 +191,8 @@ Upgrading There have been quite a few issues closed in this release. Some notables: -* Closed `#284 `_ - Created new models for `Endpoints` created associated -AWS ELB endpoint tracking code. This was the major stated goal of this milestone and should serve as the basis for -future enhancements of Lemur's certificate 'deployment' capabilities. - -* Closed `#334 `_ - Lemur not has the ability -to restrict certificate expiration dates to weekdays. +* Closed `#284 `_ - Created new models for `Endpoints` created associated AWS ELB endpoint tracking code. This was the major stated goal of this milestone and should serve as the basis for future enhancements of Lemur's certificate 'deployment' capabilities. +* Closed `#334 `_ - Lemur not has the ability to restrict certificate expiration dates to weekdays. Several fixes/tweaks to Lemurs python3 support (thanks chadhendrie!) diff --git a/docs/administration.rst b/docs/administration.rst index 3623f311..4cf8e769 100644 --- a/docs/administration.rst +++ b/docs/administration.rst @@ -78,13 +78,13 @@ Basic Configuration The default connection pool size is 5 for sqlalchemy managed connections. Depending on the number of Lemur instances, please specify per instance connection pool size. Below is an example to set connection pool size to 10. - :: + :: SQLALCHEMY_POOL_SIZE = 10 .. warning:: -This is an optional setting but important to review and set for optimal database connection usage and for overall database performance. + This is an optional setting but important to review and set for optimal database connection usage and for overall database performance. .. data:: SQLALCHEMY_MAX_OVERFLOW :noindex: @@ -99,7 +99,7 @@ This is an optional setting but important to review and set for optimal database .. note:: -Specifying the `SQLALCHEMY_MAX_OVERFLOW` to 0 will enforce limit to not create connections above specified pool size. + Specifying the `SQLALCHEMY_MAX_OVERFLOW` to 0 will enforce limit to not create connections above specified pool size. .. data:: LEMUR_ALLOW_WEEKEND_EXPIRATION @@ -174,6 +174,7 @@ Specifying the `SQLALCHEMY_MAX_OVERFLOW` to 0 will enforce limit to not create c .. data:: PUBLIC_CA_MAX_VALIDITY_DAYS :noindex: + Use this config to override the limit of 397 days of validity for certificates issued by CA/Browser compliant authorities. The authorities with cab_compliant option set to true will use this config. The example below overrides the default validity of 397 days and sets it to 365 days. @@ -185,6 +186,7 @@ Specifying the `SQLALCHEMY_MAX_OVERFLOW` to 0 will enforce limit to not create c .. data:: DEFAULT_VALIDITY_DAYS :noindex: + Use this config to override the default validity of 365 days for certificates offered through Lemur UI. Any CA which is not CA/Browser Forum compliant will be using this value as default validity to be displayed on UI. Please note that this config is used for cert issuance only through Lemur UI. The example below overrides the default validity @@ -904,10 +906,12 @@ Active Directory Certificate Services Plugin .. data:: ADCS_START :noindex: + Used in ADCS-Sourceplugin. Minimum id of the first certificate to be returned. ID is increased by one until ADCS_STOP. Missing cert-IDs are ignored .. data:: ADCS_STOP :noindex: + Used for ADCS-Sourceplugin. Maximum id of the certificates returned. @@ -1640,7 +1644,7 @@ Slack AWS (Source) ----- +------------ :Authors: Kevin Glisson , @@ -1653,7 +1657,7 @@ AWS (Source) AWS (Destination) ----- +----------------- :Authors: Kevin Glisson , @@ -1666,7 +1670,7 @@ AWS (Destination) AWS (SNS Notification) ------ +---------------------- :Authors: Jasmine Schladen diff --git a/docs/developer/plugins/index.rst b/docs/developer/plugins/index.rst index 3834b0b9..517b5a0d 100644 --- a/docs/developer/plugins/index.rst +++ b/docs/developer/plugins/index.rst @@ -145,8 +145,7 @@ The `IssuerPlugin` doesn't have any options like Destination, Source, and Notifi any fields you might need to submit a request to a third party. If there are additional options you need in your plugin feel free to open an issue, or look into adding additional options to issuers yourself. -Asynchronous Certificates -^^^^^^^^^^^^^^^^^^^^^^^^^ +**Asynchronous Certificates** An issuer may take some time to actually issue a certificate for an order. In this case, a `PendingCertificate` is returned, which holds information to recreate a `Certificate` object at a later time. Then, `get_ordered_certificate()` should be run periodically via `python manage.py pending_certs fetch -i all` to attempt to retrieve an ordered certificate:: def get_ordered_ceriticate(self, order_id): @@ -154,9 +153,10 @@ An issuer may take some time to actually issue a certificate for an order. In t # retrieve an order, and check if there is an issued certificate attached to it `cancel_ordered_certificate()` should be implemented to allow an ordered certificate to be canceled before it is issued:: - def cancel_ordered_certificate(self, pending_cert, **kwargs): - # pending_cert should contain the necessary information to match an order - # kwargs can be given to provide information to the issuer for canceling + + def cancel_ordered_certificate(self, pending_cert, **kwargs): + # pending_cert should contain the necessary information to match an order + # kwargs can be given to provide information to the issuer for canceling Destination ----------- @@ -286,7 +286,7 @@ The `ExportPlugin` object requires the implementation of one function:: Custom TLS Provider ------- +------------------- Managing TLS at the enterprise scale could be hard and often organizations offer custom wrapper implementations. It could be ideal to use those while making calls to internal services. The `TLSPlugin` would help to achieve this. It requires the diff --git a/docs/guide/index.rst b/docs/guide/index.rst index b06a95e0..f3efcb14 100644 --- a/docs/guide/index.rst +++ b/docs/guide/index.rst @@ -65,6 +65,7 @@ Import an Existing Certificate You can add notification options and upload the created certificate to a destination, both of these are editable features and can be changed after the certificate has been created. +.. _CreateANewUser: Create a New User ~~~~~~~~~~~~~~~~~ diff --git a/docs/production/index.rst b/docs/production/index.rst index fa0a7dec..3082ee4a 100644 --- a/docs/production/index.rst +++ b/docs/production/index.rst @@ -501,7 +501,7 @@ rely on celery to create the DNS record. This will change when we implement mix To create a HTTP compatible Authority, you first need to create a new destination that will be used to deploy the challenge token. Visit `Admin` -> `Destination` and click `Create`. The path you provide for the destination needs to -be the exact path that is called when the ACME providers calls ``http:///.well-known/acme-challenge/`. The +be the exact path that is called when the ACME providers calls `http:///.well-known/acme-challenge/`. The token part will be added dynamically by the acme_upload. Currently only the SFTP and S3 Bucket destination support the ACME HTTP challenge. diff --git a/docs/quickstart/index.rst b/docs/quickstart/index.rst index 3056029d..f972c2ef 100644 --- a/docs/quickstart/index.rst +++ b/docs/quickstart/index.rst @@ -148,7 +148,7 @@ Before Lemur will run you need to fill in a few required variables in the config LEMUR_DEFAULT_ORGANIZATIONAL_UNIT Set Up Postgres --------------- +--------------- For production, a dedicated database is recommended, for this guide we will assume postgres has been installed and is on the same machine that Lemur is installed on. @@ -186,11 +186,12 @@ In addition to creating a new user, Lemur also creates a few default email notif Your database installation requires the pg_trgm extension. If you do not have this installed already, you can allow the script to install this for you by adding the SUPERUSER permission to the lemur database user. .. code-block:: bash + sudo -u postgres -i psql postgres=# ALTER USER lemur WITH SUPERUSER -Additional notifications can be created through the UI or API. See :ref:`Creating Notifications ` and :ref:`Command Line Interface ` for details. +Additional notifications can be created through the UI or API. See :ref:`Notification Options ` and :ref:`Command Line Interface ` for details. **Make note of the password used as this will be used during first login to the Lemur UI.** @@ -202,15 +203,16 @@ Additional notifications can be created through the UI or API. See :ref:`Creati .. note:: If you added the SUPERUSER permission to the lemur database user above, it is recommended you revoke that permission now. .. code-block:: bash + sudo -u postgres -i psql postgres=# ALTER USER lemur WITH NOSUPERUSER -.. note:: It is recommended that once the ``lemur`` user is created that you create individual users for every day access. There is currently no way for a user to self enroll for Lemur access, they must have an administrator create an account for them or be enrolled automatically through SSO. This can be done through the CLI or UI. See :ref:`Creating Users ` and :ref:`Command Line Interface ` for details. +.. note:: It is recommended that once the ``lemur`` user is created that you create individual users for every day access. There is currently no way for a user to self enroll for Lemur access, they must have an administrator create an account for them or be enrolled automatically through SSO. This can be done through the CLI or UI. See :ref:`Creating a New User ` and :ref:`Command Line Interface ` for details. Set Up a Reverse Proxy ---------------------- +---------------------- By default, Lemur runs on port 8000. Even if you change this, under normal conditions you won't be able to bind to port 80. To get around this (and to avoid running Lemur as a privileged user, which you shouldn't), we need to set up a simple web proxy. There are many different web servers you can use for this, we like and recommend Nginx. diff --git a/lemur/authorities/views.py b/lemur/authorities/views.py index 094a5a74..16441719 100644 --- a/lemur/authorities/views.py +++ b/lemur/authorities/views.py @@ -132,31 +132,31 @@ class AuthoritiesList(AuthenticatedResource): Accept: application/json, text/javascript Content-Type: application/json;charset=UTF-8 - { - "country": "US", - "state": "California", - "location": "Los Gatos", - "organization": "Netflix", - "organizationalUnit": "Operations", - "type": "root", - "signingAlgorithm": "sha256WithRSA", - "sensitivity": "medium", - "keyType": "RSA2048", - "plugin": { - "slug": "cloudca-issuer" - }, - "name": "TimeTestAuthority5", - "owner": "secure@example.com", - "description": "test", - "commonName": "AcommonName", - "validityYears": "20", - "extensions": { - "subAltNames": { - "names": [] - }, - "custom": [] - } - } + { + "country": "US", + "state": "California", + "location": "Los Gatos", + "organization": "Netflix", + "organizationalUnit": "Operations", + "type": "root", + "signingAlgorithm": "sha256WithRSA", + "sensitivity": "medium", + "keyType": "RSA2048", + "plugin": { + "slug": "cloudca-issuer" + }, + "name": "TimeTestAuthority5", + "owner": "secure@example.com", + "description": "test", + "commonName": "AcommonName", + "validityYears": "20", + "extensions": { + "subAltNames": { + "names": [] + }, + "custom": [] + } + } **Example response**: @@ -218,8 +218,7 @@ class AuthoritiesList(AuthenticatedResource): :arg parent: the parent authority if this is to be a subca :arg signingAlgorithm: algorithm used to sign the authority :arg keyType: key type - :arg sensitivity: the sensitivity of the root key, for CloudCA this determines if the root keys are stored - in an HSM + :arg sensitivity: the sensitivity of the root key, for CloudCA this determines if the root keys are stored in an HSM :arg keyName: name of the key to store in the HSM (CloudCA) :arg serialNumber: serial number of the authority :arg firstSerial: specifies the starting serial number for certificates issued off of this authority @@ -494,23 +493,48 @@ class CertificateAuthority(AuthenticatedResource): class AuthorityVisualizations(AuthenticatedResource): def get(self, authority_id): """ - {"name": "flare", - "children": [ - { - "name": "analytics", - "children": [ - { - "name": "cluster", - "children": [ - {"name": "AgglomerativeCluster", "size": 3938}, - {"name": "CommunityStructure", "size": 3812}, - {"name": "HierarchicalCluster", "size": 6714}, - {"name": "MergeEdge", "size": 743} - ] - } - ] - } - ]} + .. http:get:: /authorities/1/visualize + + Authority visualization + + **Example request**: + + .. sourcecode:: http + + GET /certificates/1/visualize HTTP/1.1 + Host: example.com + Accept: application/json, text/javascript + + **Example response**: + + .. sourcecode:: http + + HTTP/1.1 200 OK + Vary: Accept + Content-Type: text/javascript + + {"name": "flare", + "children": [ + { + "name": "analytics", + "children": [ + { + "name": "cluster", + "children": [ + {"name": "AgglomerativeCluster", "size": 3938}, + {"name": "CommunityStructure", "size": 3812}, + {"name": "HierarchicalCluster", "size": 6714}, + {"name": "MergeEdge", "size": 743} + ] + } + ] + } + ] + } + + :reqheader Authorization: OAuth token to authenticate + :statuscode 200: no error + :statuscode 403: unauthenticated """ authority = service.get(authority_id) return dict( diff --git a/lemur/certificates/views.py b/lemur/certificates/views.py index 8d4e6954..f453ac4f 100644 --- a/lemur/certificates/views.py +++ b/lemur/certificates/views.py @@ -59,6 +59,7 @@ class CertificatesListValid(AuthenticatedResource): **Example request**: .. sourcecode:: http + GET /certificates/valid?filter=cn;*.test.example.net&owner=joe@example.com&page=1&count=20 HTTP/1.1 Host: example.com diff --git a/lemur/destinations/service.py b/lemur/destinations/service.py index 7bae57f0..5e302c6d 100644 --- a/lemur/destinations/service.py +++ b/lemur/destinations/service.py @@ -21,7 +21,7 @@ def create(label, plugin_name, options, description=None): :param label: Destination common name :param description: - :rtype : Destination + :rtype: Destination :return: New destination """ # remove any sub-plugin objects before try to save the json options @@ -50,7 +50,7 @@ def update(destination_id, label, plugin_name, options, description): :param plugin_name: :param options: :param description: - :rtype : Destination + :rtype: Destination :return: """ destination = get(destination_id) @@ -81,7 +81,7 @@ def get(destination_id): Retrieves an destination by its lemur assigned ID. :param destination_id: Lemur assigned ID - :rtype : Destination + :rtype: Destination :return: """ return database.get(Destination, destination_id) diff --git a/lemur/notifications/service.py b/lemur/notifications/service.py index 5bc5f3e1..fd8ba20f 100644 --- a/lemur/notifications/service.py +++ b/lemur/notifications/service.py @@ -94,7 +94,7 @@ def create(label, plugin_name, options, description, certificates): :param options: :param description: :param certificates: - :rtype : Notification + :rtype: Notification :return: """ notification = Notification( @@ -115,7 +115,7 @@ def update(notification_id, label, plugin_name, options, description, active, ce :param description: :param active: :param certificates: - :rtype : Notification + :rtype: Notification :return: """ notification = get(notification_id) @@ -144,7 +144,7 @@ def get(notification_id): Retrieves an notification by its lemur assigned ID. :param notification_id: Lemur assigned ID - :rtype : Notification + :rtype: Notification :return: """ return database.get(Notification, notification_id) diff --git a/lemur/plugins/lemur_aws/plugin.py b/lemur/plugins/lemur_aws/plugin.py index efcce4d0..61c64dab 100644 --- a/lemur/plugins/lemur_aws/plugin.py +++ b/lemur/plugins/lemur_aws/plugin.py @@ -450,7 +450,8 @@ class S3DestinationPlugin(ExportDestinationPlugin): def upload_acme_token(self, token_path, token, options, **kwargs): """ - This is called from the acme http challenge + This is called from the acme http challenge + :param self: :param token_path: :param token: diff --git a/requirements-docs.in b/requirements-docs.in index f2551059..07b3e987 100644 --- a/requirements-docs.in +++ b/requirements-docs.in @@ -3,8 +3,38 @@ # from requirements-docs.txt # However, dependabot doesn't use `make up-reqs`, so we have to replicate the necessary dependencies here # Without including these dependencies, the docs are unable to include generated autodocs +acme +arrow +boto3 +botocore +CloudFlare +cryptography +dnspython3 +dyn Flask +Flask-Bcrypt +Flask-Cors +Flask-Mail +Flask-Migrate +Flask-Principal +Flask-RESTful +Flask-Script +Flask-SQLAlchemy flask_replicated +gunicorn +inflection +josepy +logmatic-python +marshmallow-sqlalchemy +marshmallow<2.20.5 #schema duplicate issues https://github.com/marshmallow-code/marshmallow-sqlalchemy/issues/121 +pem +pyjwt +pyOpenSSL +raven[flask] +retrying +SQLAlchemy-Utils +tabulate +xmltodict # docs specific sphinx diff --git a/requirements-docs.txt b/requirements-docs.txt index 7d094b77..41d5133b 100644 --- a/requirements-docs.txt +++ b/requirements-docs.txt @@ -4,36 +4,92 @@ # # pip-compile --no-index --output-file=requirements-docs.txt requirements-docs.in # +acme==1.12.0 + # manual debug alabaster==0.7.12 # via sphinx +arrow==0.17.0 + # manual debug babel==2.8.0 # via sphinx +boto3==1.17.7 + # manual debug +botocore==1.20.7 + # manual debug certifi==2020.12.5 # via requests chardet==3.0.4 # via requests +cloudflare==2.8.15 + # manual debug +cryptography==3.4.5 + # manual debug +dnspython3==1.15.0 + # manual debug +dnspython==1.15.0 + # manual debug docutils==0.15.2 # via sphinx +dyn==1.8.1 + # manual debug idna==2.9 # via requests imagesize==1.2.0 # via sphinx flask==1.1.2 # manual debug +flask-bcrypt==0.7.1 + # manual debug +flask-cors==3.0.10 + # manual debug +flask-mail==0.9.1 + # manual debug +flask-migrate==2.6.0 + # manual debug +flask-principal==0.4.0 + # manual debug flask-replicated==1.4 # manual debug +flask-restful==0.3.8 + # manual debug +flask-script==2.0.6 + # manual debug +flask-sqlalchemy==2.4.4 + # manual debug +gunicorn==20.0.4 + # manual debug +inflection==0.5.1 + # manual debug jinja2==2.11.3 # via sphinx +josepy==1.3.0 + # manual debug +logmatic-python==0.1.7 + # manual debug markupsafe==1.1.1 # via jinja2 +marshmallow-sqlalchemy==0.23.1 + # manual debug +marshmallow==2.20.4 + # manual debug packaging==20.3 # via sphinx +pem==21.1.0 + # manual debug pygments==2.6.1 # via sphinx +pyjwt==2.0.1 + # manual debug +pyopenssl==20.0.1 + # manual debug pyparsing==2.4.7 # via packaging pytz==2019.3 # via babel +raven[flask]==6.10.0 + # manual debug +retrying==1.3.3 + # manual debug requests==2.25.1 # via sphinx six==1.15.0 @@ -63,8 +119,14 @@ sphinxcontrib-qthelp==1.0.3 # via sphinx sphinxcontrib-serializinghtml==1.1.4 # via sphinx +sqlalchemy-utils==0.36.8 + # manual debug +tabulate==0.8.7 + # manual debug urllib3==1.25.8 # via requests +xmltodict==0.12.0 + # manual debug # The following packages are considered to be unsafe in a requirements file: # setuptools