lemur/docs/developer/plugins/index.rst
James Chuong 2578970f7d Async Certificate Issuing using Pending Certificates ()
* Add PendingCertificate model

This change creates a DB table called pending_certificates and
associated mapping relationship tables from pending certificate to
roles, rotation policy, destination, sources, etc.

The table is generated on initialization of Lemur. A pending
certificate holds most of the information of a Certificate, while it has
not be issued so that it can later backfill the information when the CA
has issued the certificate.

Change-Id: I277c16b776a71fe5edaf0fa0e76bbedc88924db0
Tickets: PBL-36499

* Create a PendingCertificate if cert is empty

IssuePlugins should return empty cert bodies if the request failed to
complete immediately (such as Digicert).  This way, we can immediately
return the certificate, or if not just place into PendingCertificates
for later processing.

+ Fix relation from Certificate to Pending Certificate, as view only.
There is no real need for anything more than that since Pending cert
only needs to know the cert to replace when it is issued later.

+ Made PendingCertificate private key be empty: UI does not allow
private key on 'Create' but only on 'Import'.  For Instart, we require
the private key but upstream does not necessarily need it.  Thus, if
someone at Instart wants to create a CSR / key combo, they should
manually issue the cert themselves and import later.  Otherwise you
should let Lemur generate that.  This keeps the workflow transparent for
upstream Lemur users.

Change-Id: Ib74722a5ed5792d4b10ca702659422739c95ae26
Tickets: PBL-36343

* Fix empty private_key when create Pending Cert

On creation of a certificate with a CSR, there is no option for private
key.  In this case, we actually have a dictionary with private_key as
key, but the value is None.  This fixes the strip() called on NoneType.

Change-Id: I7b265564d8095bfc83d9d4cd14ae13fea3c03199
Tickets: PBL-36499

* Source sync finds and uses pending certificate

When a source syncs certificates, it will check for a pending
certificate.  If that is found via external_id (given by digicert as
order_id) then it will use the found Pending Certificate's fields to
create a new certificate.  Then the pending certificate is deleted.

Tickets: PBL-36343
Change-Id: I4f7959da29275ebc47a3996741f7e98d3e2d29d9

* Add Lemur static files and views for pending certs

This adds the basic static files to view pending certificates in a
table.

Tickets: PBL-36343
Change-Id: Ia4362e6664ec730d05d280c5ef5c815a6feda0d9

* Add CLI and plugin based pending fetch

This change uses the adds a new function to issuer plugins to fetch
certificates like source, but for one order.  This way, we can control
which pending certificates to try and populate instead of getting all
certificates from source.

Tickets: PBL-36343
Change-Id: Ifc1747ccdc2cba09a81f298b31ddddebfee1b1d6

* Revert source using Pending Certificate

Tickets: PBL-36343
Change-Id: I05121bc951e0530d804070afdb9c9e09baa0bc51

* Fix PendingCertificate init getting authority id

Should get authority id from authority.id instead of the authority_id
key in kwargs.

Change-Id: Ie56df1a5fb0ab2729e91050f3ad1a831853e0623
Tickets: n/a

* Add fixtures and basic test for PendingCertificate

Change-Id: I4cca34105544d40dac1cc50a87bba93d8af9ab34
Tickets: PBL-36343

* Add User to create_certificate parameters

create_certificate now takes a User, which will be used to populate the
'creator' field in certificates.service.upload().  This allows the UI
populate with the current user if the owner does not exist in Lemur.

+ Fix chain being replaced with version from pending certificate, which
may be empty (depends on plugin implementation).

Change-Id: I516027b36bc643c4978b9c4890060569e03f3049
Tickets: n/a

* Fix permalink and filters to pending certs

Fixes the permalink button to get a single pending certificate
Add argument filter parsing for the pending certificate API
Fix comment on API usage
Added get_by_name for pending_certificate (currently unused, but useful
for CLI, instead of using IDs)

Change-Id: Iaa48909c45606bec65dfb193c13d6bd0e816f6db
Tickets: PBL-36910

* Update displayed fields for Pending Certificates

There are a number of unused / unpopulated fields from Certificate UI
that does apply to Pending Certificates.  Those ones were removed, and
added other useful fields:
Owner, number of attempts to fetch and date created

Change-Id: I3010a715f0357ba149cf539a19fdb5974c5ce08b
Tickets: PBL-36910

* Add common name (cn) to Pending Certificate model

Fixes the UI missing the CN for Pending Certificate, as it was
originally being parsed from the generated certificate.  In the case of
pending certificate, the CN from the user generates the request, which
means a pending cert can trust the original user putting in the CN
instead of having to parse the not-yet-generated certificate.  There is
no real possibility to return a certificate from a pending certificate
where the CN has changed since it was initially ordered.

Change-Id: I88a4fa28116d5d8d293e58970d9777ce73fbb2ab
Tickets: PBL-36910

* Fix missing imports for service filter

+ Removed duplicate get_by_name function from old merge

Change-Id: I04ae6852533aa42988433338de74390e2868d69b
Tickets: PBL-36910

* Add private key viewing to Pending Certificates

Add private key API for Pending Certificates, with the same
authorization as Certificates (only owner, creator or owner-roles can
view private key).

Change-Id: Ie5175154a10fe0007cc0e9f35b80c0a01ed48d5b
Tickets: PBL-36910

* Add edit capability to pending certificates

Like editing certificates, we should be able to modify some parts of a
pending certificate so the resulting certificate has the right
references, owner, etc.

+ Added API to update pending certificate
+ Fix UI to use pending certificate scope instead of reusing Certificate
+ Change pending_certificate.replaces to non-passive association, so
that updates do affect it (similar to roles/notifications/etc)

Tickets: PBL-36910
Change-Id: Ibbcb166a33f0337e1b14f426472261222f790ce6

* Add common_name parsing instead using kwargs

To fix tests where common name may not be passed in, use the CSR
generated to find the official common name.

Change-Id: I09f9258fa92c2762d095798676ce210c5d7a3da4
Tickets: PBL-36343

* Add Cancel to pending certificates and plugins

This allows pending certificates to be cancelled, which will be handled
by the issuer plugin.

Change-Id: Ibd6b5627c3977e33aca7860690cfb7f677236ca9
Tickets: PBL-36910

* Add API for Cancelling Pending Certificate

Added the DELETE handler for pending_certificates, which will cancel and
delete the pending certificate from the pending certs table on
successful cancellation via Issuer Plugin.

+ Add UT for testing cancel API

Change-Id: I11b1d87872e4284f6e4f9c366a15da4ddba38bc4
Tickets: PBL-36910

* Remove Export from Pending Certificates

Pending Certificates doesn't need an export since it should just be
fetched by Lemur via plugins, and the CSR is viewable via the UI.

Change-Id: I9a3e65ea11ac5a85316f6428e7f526c3c09178ae
Tickets: PBL-36910

* Add cancel button functionality to UI

This adds the Cancel option to the dropdown of pending certificates.

+ Adds modal window for Note (may not be required for all issuers, just
Digicert)
+ Add schema for cancel input
+ Fix Digitcert plugin for non-existant orders

When an order is actually issued, then attempting to cancel will return
a 403 from Digicert.  This is a case where it should only be done once
we know the pending cert has been sitting for too long.

Change-Id: I256c81ecd142dd51dcf8e38802d2c202829887b0
Tickets: PBL-36910

* Fix test_pending_cancel UT

This change creates and injects a pending cert, which will then be used
for the ID so it can be canceled by the unit test.

Change-Id: I686e7e0fafd68cdaeb26438fb8504d79de77c346
Tickets: PBL-36343

* Fix test_digicert on non-existent order

cancelling a non-existent order is fine since we're cancelling it

Change-Id: I70c0e82ba2f4b8723a7f65b113c19e6eeff7e68c
Tickets: PBL-36343

* Add migrations for PendingCertificates

Added revision for Pending Certificates table and foreign key mapping
tables.

Change-Id: Ife8202cef1e6b99db377851264639ba540b749db
Tickets: n/a

* Fix relationship copy from Pending to Certificate

When a Pending Certificate is changed to a full Certificate, the
relationship fields are not copied via vars() function, as it's not a
column but mapped via association table.  This adds an explicit copy for
these relations.  Which will properly copy them to the new Certificate,
and thus also update destinations.

Change-Id: I322032ce4a9e3e67773f7cf39ee4971054c92685
Tickets: PBL-36343

* Fix renaming of certificates and unit tests

The rename flag was not used to rename certificates on creation as
expected.

Fixed unit test, instead of expunging the session, just copy the
pending_certificate so we don't have a weird reference to the object
that can't be copied via vars() function.

Change-Id: I962943272ed92386ab6eab2af4ed6d074d4cffa0
Tickets: PBL-36343

* Updated developer docs for async certs

Added blurb for implementing new issuer functions.

Change-Id: I1caed6e914bcd73214eae2d241e4784e1b8a0c4c
Tickets: n/a
2018-02-22 08:13:16 -08:00

357 lines
15 KiB
ReStructuredText

Several interfaces exist for extending Lemur:
* Issuer (lemur.plugins.base.issuer)
* Destination (lemur.plugins.base.destination)
* Source (lemur.plugins.base.source)
* Notification (lemur.plugins.base.notification)
Each interface has its own functions that will need to be defined in order for
your plugin to work correctly. See :ref:`Plugin Interfaces <PluginInterfaces>` for details.
Structure
---------
A plugins layout generally looks like the following::
setup.py
lemur_pluginname/
lemur_pluginname/__init__.py
lemur_pluginname/plugin.py
The ``__init__.py`` file should contain no plugin logic, and at most, a VERSION = 'x.x.x' line. For example,
if you want to pull the version using pkg_resources (which is what we recommend), your file might contain::
try:
VERSION = __import__('pkg_resources') \
.get_distribution(__name__).version
except Exception as e:
VERSION = 'unknown'
Inside of ``plugin.py``, you'll declare your Plugin class::
import lemur_pluginname
from lemur.plugins.base.issuer import IssuerPlugin
class PluginName(IssuerPlugin):
title = 'Plugin Name'
slug = 'pluginname'
description = 'My awesome plugin!'
version = lemur_pluginname.VERSION
author = 'Your Name'
author_url = 'https://github.com/yourname/lemur_pluginname'
def widget(self, request, group, **kwargs):
return "<p>Absolutely useless widget</p>"
And you'll register it via ``entry_points`` in your ``setup.py``::
setup(
# ...
entry_points={
'lemur.plugins': [
'pluginname = lemur_pluginname.issuers:PluginName'
],
},
)
You can potentially package multiple plugin types in one package, say you want to create a source and
destination plugins for the same third-party. To accomplish this simply alias the plugin in entry points to point
at multiple plugins within your package::
setup(
# ...
entry_points={
'lemur.plugins': [
'pluginnamesource = lemur_pluginname.plugin:PluginNameSource',
'pluginnamedestination = lemur_pluginname.plugin:PluginNameDestination'
],
},
)
Once your plugin files are in place and the ``/www/lemur/setup.py`` file has been modified, you can load your plugin into your instance by reinstalling lemur:
::
(lemur)$cd /www/lemur
(lemur)$pip install -e .
That's it! Users will be able to install your plugin via ``pip install <package name>``.
.. SeeAlso:: For more information about python packages see `Python Packaging <https://packaging.python.org/en/latest/distributing.html>`_
.. SeeAlso:: For an example of a plugin operation outside of Lemur's core, see `lemur-digicert <https://github.com/opendns/lemur-digicert>`_
.. _PluginInterfaces:
Plugin Interfaces
=================
In order to use the interfaces all plugins are required to inherit and override unimplemented functions
of the parent object.
Issuer
------
Issuer plugins are used when you have an external service that creates certificates or authorities.
In the simple case the third party only issues certificates (Verisign, DigiCert, etc.).
If you have a third party or internal service that creates authorities (EJBCA, etc.), Lemur has you covered,
it can treat any issuer plugin as both a source of creating new certificates as well as new authorities.
The `IssuerPlugin` exposes four functions functions::
def create_certificate(self, csr, issuer_options):
# requests.get('a third party')
def revoke_certificate(self, certificate, comments):
# requests.put('a third party')
def get_ordered_certificate(self, order_id):
# requests.get('already existing certificate')
def canceled_ordered_certificate(self, pending_cert, **kwargs):
# requests.put('cancel an order that has yet to be issued')
Lemur will pass a dictionary of all possible options for certificate creation. Including a valid CSR, and the raw options associated with the request.
If you wish to be able to create new authorities implement the following function and ensure that the ROOT_CERTIFICATE and the INTERMEDIATE_CERTIFICATE (if any) for the new authority is returned::
def create_authority(self, options):
root_cert, intermediate_cert, username, password = request.get('a third party')
# if your provider creates specific credentials for each authority you can associated them with the role associated with the authority
# these credentials will be provided along with any other options when a certificate is created
role = dict(username=username, password=password, name='generatedAuthority')
return root_cert, intermediate_cert, [role]
.. Note::
Lemur uses PEM formatted certificates as it's internal standard, if you receive certificates in other formats convert them to PEM before returning.
If instead you do not need need to generate authorities but instead use a static authority (Verisign, DigiCert), you can use publicly available constants::
def create_authority(self, options):
# optionally associate a role with authority to control who can use it
role = dict(username='', password='', name='exampleAuthority')
# username and password don't really matter here because we do no need to authenticate our authority against a third party
return EXAMPLE_ROOT_CERTIFICATE, EXAMPLE_INTERMEDIATE_CERTIFICATE, [role]
.. Note:: You do not need to associate roles to the authority at creation time as they can always be associated after the fact.
The `IssuerPlugin` doesn't have any options like Destination, Source, and Notification plugins. Essentially Lemur **should** already have
any fields you might need to submit a request to a third party. If there are additional options you need
in your plugin feel free to open an issue, or look into adding additional options to issuers yourself.
Asynchronous Certificates
^^^^^^^^^^^^^^^^^^^^^^^^^
An issuer may take some time to actually issue a certificate for an order. In this case, a `PendingCertificate` is returned, which holds information to recreate a `Certificate` object at a later time. Then, `get_ordered_certificate()` should be run periodically via `python manage.py pending_certs fetch -i all` to attempt to retrieve an ordered certificate::
def get_ordered_ceriticate(self, order_id):
# order_id is the external id of the order, not the external_id of the certificate
# retrieve an order, and check if there is an issued certificate attached to it
`cancel_ordered_certificate()` should be implemented to allow an ordered certificate to be canceled before it is issued::
def cancel_ordered_certificate(self, pending_cert, **kwargs):
# pending_cert should contain the necessary information to match an order
# kwargs can be given to provide information to the issuer for canceling
Destination
-----------
Destination plugins allow you to propagate certificates managed by Lemur to additional third parties. This provides flexibility when
different orchestration systems have their own way of manage certificates or there is an existing system you wish to integrate with Lemur.
By default destination plugins have a private key requirement. If your plugin does not require a certificates private key mark `requires_key = False`
in the plugins base class like so::
class MyDestinationPlugin(DestinationPlugin):
requires_key = False
The DestinationPlugin requires only one function to be implemented::
def upload(self, name, body, private_key, cert_chain, options, **kwargs):
# request.post('a third party')
Additionally the DestinationPlugin allows the plugin author to add additional options
that can be used to help define sub-destinations.
For example, if we look at the aws-destination plugin we can see that it defines an `accountNumber` option::
options = [
{
'name': 'accountNumber',
'type': 'int',
'required': True,
'validation': '/^[0-9]{12,12}$/',
'helpMessage': 'Must be a valid AWS account number!',
}
]
By defining an `accountNumber` we can make this plugin handle many N number of AWS accounts instead of just one.
The schema for defining plugin options are pretty straightforward:
- **Name**: name of the variable you wish to present the user, snake case (snakeCase) is preferred as Lemur
will parse these and create pretty variable titles
- **Type** there are currently four supported variable types
- **Int** creates an html integer box for the user to enter integers into
- **Str** creates a html text input box
- **Boolean** creates a checkbox for the user to signify truthiness
- **Select** creates a select box that gives the user a list of options
- When used a `available` key must be provided with a list of selectable options
- **Required** determines if this option is required, this **must be a boolean value**
- **Validation** simple JavaScript regular expression used to give the user an indication if the input value is valid
- **HelpMessage** simple string that provides more detail about the option
.. Note::
DestinationPlugin, NotificationPlugin and SourcePlugin all support the option
schema outlined above.
Notification
------------
Lemur includes the ability to create Email notifications by **default**. These notifications
currently come in the form of expiration notices. Lemur periodically checks certifications expiration dates and
determines if a given certificate is eligible for notification. There are currently only two parameters used to
determine if a certificate is eligible; validity expiration (date the certificate is no longer valid) and the number
of days the current date (UTC) is from that expiration date.
There are currently two objects that available for notification plugins the first is `NotficationPlugin`. This is the base object for
any notification within Lemur. Currently the only support notification type is an certificate expiration notification. If you
are trying to create a new notification type (audit, failed logins, etc.) this would be the object to base your plugin on.
You would also then need to build additional code to trigger the new notification type.
The second is `ExpirationNotificationPlugin`, this object inherits from `NotificationPlugin` object.
You will most likely want to base your plugin on, if you want to add new channels for expiration notices (Slack, HipChat, Jira, etc.). It adds default options that are required by
all expiration notifications (interval, unit). This interface expects for the child to define the following function::
def send(self, notification_type, message, targets, options, **kwargs):
# request.post("some alerting infrastructure")
Source
------
When building Lemur we realized that although it would be nice if every certificate went through Lemur to get issued, but this is not
always be the case. Oftentimes there are third parties that will issue certificates on your behalf and these can get deployed
to infrastructure without any interaction with Lemur. In an attempt to combat this and try to track every certificate, Lemur has a notion of
certificate **Sources**. Lemur will contact the source at periodic intervals and attempt to **sync** against the source. This means downloading or discovering any
certificate Lemur does not know about and adding the certificate to its inventory to be tracked and alerted on.
The `SourcePlugin` object has one default option of `pollRate`. This controls the number of seconds which to get new certificates.
.. warning::
Lemur currently has a very basic polling system of running a cron job every 15min to see which source plugins need to be run. A lock file is generated to guarantee that
only one sync is running at a time. It also means that the minimum resolution of a source plugin poll rate is effectively 15min. You can always specify a faster cron
job if you need a higher resolution sync job.
The `SourcePlugin` object requires implementation of one function::
def get_certificates(self, options, **kwargs):
# request.get("some source of certificates")
.. note::
Oftentimes to facilitate code re-use it makes sense put source and destination plugins into one package.
Export
------
Formats, formats and more formats. That's the current PKI landscape. See the always relevant `xkcd <https://xkcd.com/927/>`_.
Thankfully Lemur supports the ability to output your certificates into whatever format you want. This integration comes by the way
of Export plugins. Support is still new and evolving, the goal of these plugins is to return raw data in a new format that
can then be used by any number of applications. Included in Lemur is the `JavaExportPlugin` which currently supports generating
a Java Key Store (JKS) file for use in Java based applications.
The `ExportPlugin` object requires the implementation of one function::
def export(self, body, chain, key, options, **kwargs):
# sys.call('openssl hokuspocus')
# return "extension", passphrase, raw
.. note::
Support of various formats sometimes relies on external tools system calls. Always be mindful of sanitizing any input to these calls.
Testing
=======
Lemur provides a basic py.test-based testing framework for extensions.
In a simple project, you'll need to do a few things to get it working:
setup.py
--------
Augment your setup.py to ensure at least the following:
.. code-block:: python
setup(
# ...
install_requires=[
'lemur',
]
)
conftest.py
-----------
The ``conftest.py`` file is our main entry-point for py.test. We need to configure it to load the Lemur pytest configuration:
.. code-block:: python
from lemur.tests.conftest import * # noqa
Test Cases
----------
You can now inherit from Lemur's core test classes. These are Django-based and ensure the database and other basic utilities are in a clean state:
.. code-block:: python
import pytest
from lemur.tests.vectors import INTERNAL_CERTIFICATE_A_STR, INTERNAL_PRIVATE_KEY_A_STR
def test_export_keystore(app):
from lemur.plugins.base import plugins
p = plugins.get('java-keystore-jks')
options = [{'name': 'passphrase', 'value': 'test1234'}]
with pytest.raises(Exception):
p.export(INTERNAL_CERTIFICATE_A_STR, "", "", options)
raw = p.export(INTERNAL_CERTIFICATE_A_STR, "", INTERNAL_PRIVATE_KEY_A_STR, options)
assert raw != b""
Running Tests
-------------
Running tests follows the py.test standard. As long as your test files and methods are named appropriately (``test_filename.py`` and ``test_function()``) you can simply call out to py.test:
::
$ py.test -v
============================== test session starts ==============================
platform darwin -- Python 2.7.10, pytest-2.8.5, py-1.4.30, pluggy-0.3.1
cachedir: .cache
plugins: flask-0.10.0
collected 346 items
lemur/plugins/lemur_acme/tests/test_acme.py::test_get_certificates PASSED
=========================== 1 passed in 0.35 seconds ============================
.. SeeAlso:: Lemur bundles several plugins that use the same interfaces mentioned above.