HSM Operations

HSM Operations

Back to Documentation Map

This page gives an overview of deploying HSM Solutions operationally. "Operator" is the
organisation responsible for operating the HSMs and "Application Owner" is an entity responsible
for application development and deployment that uses HSMs. This document considers both deployments with or without custom firmware.

Responsible Operations

The security benefits of using HSMs can only be realised if the HSMs are deployed responsibly.
This involves at least:

1. An auditable and maintained Plan describing how the HSMs are to be operated.
2. Auditable proof of origin and initial integrity of the HSM Devices.
3. Auditable trail of all actions carried out on the HSMs since initial taking into service.
4. Defined, Strong and Auditable governance of HSM access roles and actors. Credentials must be
   defined and securely stored in a manner supporting the access model.
5. Defined Interfaces and Change Control processes for interaction with Application Owners.
6. Assigned and Appropriately trained Staff with defined skill levels to operate the system.
7. Atomic, Ceremonial approach to HSM Actions with defined prerequisites, reviewed content,
   defined outputs and output destinations including an Authenticated Execution Report for
   the audit trail.
8. Auditors that will actually look at the audit material with informed scrutiny and a governance
   path to deal with their reports.

Owning HSMs

This picture shows an overview of aspects involved in responsibly operating HSMs:

This diagram is intended as a warning not to underestimate the consequences of deciding to
use and operate HSMs.

It also scares off many potential HSM users with the perceived complexity. It is the task of
a sensible operations plan to hide most of this complexity from the application owner level.
This can be achieved by a strong categorisation of HSM Interactions and careful
choice of where the hand-off of the HSM between the operator and application owner occurs.

There are many potential synergies when operating HSMs for multiple end-use application owners which
makes "HSM as a Service" offerings attractive. These are discussed below

HSM Interactions

This Diagram shows the different areas of interactions with the HSMs and the likely actors (Use
Cases). It makes an assumption that the HSM Hardware devices (global operations) support a number of
virtualised HSM Modules (Slots, cHSMs/ContainerHSMs). It also allows that there may be an element of
installing custom firmware on the HSMs. Note that there can be HSM Manufacturer differences in this
area, e.g. On Thales HSMs a custom firmware is global to the hardware device, whereas on Utimaco HSMs it
is local to a single cHSM.

Boundaries

The set of actors and actions will be split according to several types of boundaries which may have complex overlapping and interlocking relationships:

• Organisational Boundaries separate the actors into groups that share a single management, staffing
  and resource allocation.
• Governance Boundaries separate the actors into groups with different approval roles, implicit when
  everything is in a single hand with one-person governance, but which must be defined clearly, especially
  when multi-person approvals are intended and where these may cross organisational or institutional boundaries.
• Institutional Boundaries. These separate the actors into institutionally distinct groups, e.g. Companies, Government agencies, independent watchdogs, etc.

High Availability

Real World applications have high availability requirements for scalability and resilience that can
only be met by a clustered HSM solution. Designing a solution that will achieve this requires coordination between Operator and Application Owner since it involves architectural choices and implementation.

• Load Management (Scalability) can be achieved by changing the number of available HSMs processing requests for an application. This can be done on a variety of different timescales, though true dynamic scaling as known from containerised (Kubernetes) clusters is rarely practicable.
• Resilience can be provided by having enough HSM instances available to meet expected loads, even allowing for one or more HSMs to fail.

For this to work

1. All HSMs must be capable of processing HSM individual transaction requests. This means that:
   • Either all keys are loaded into all HSMs in the cluster and do not change at transaction runtime, or
   • HSM Storage is externalised, effectively making the HSMs stateless and converting the HSM interface
     into a REST API, or
   • There is a mechanism in place to synchronise the state of HSMs at runtime. (Horrible, strongly
     recommended against!), or
   • There is a sticky-session management put in place which ensures that any HSM Request that creates
     state has all follow-up requests using that state directed to the same HSM. (Strongly unrecommended.)
2. HSM Request routing must be dynamic and adaptive. This usually means through a load balancing proxy,
   such as e.g. HA Proxy.
3. The HA Routing must be able to track and update HSM state of health at runtime and reroute requests as required.

HSM Propagation

Different HSM Manufacturers support different levels and modes of propagation of HSM State. This
becomes important as soon as we start to talk about a cluster of HSMs rather than a single device.
Here we discuss the options on Utimaco HSMs.

Virtual HSM propagation

A virtual (cHSM) HSM can be copied (Snapshotting) and restored completely intact (Cloning). This
allows simple and fast proliferation of HSM instances, most useful when addressing loading, HSM
Recovery and Redundancy issues.

The cloned HSMs are exact copies of the original and (on Utimaco) labelled with a tag assigned when
creating the initial virtual HSM. This can be annoying to manage if we wish to clone cHSMs
successively with aging state.

The virtual HSM state is neither divisible nor discrete, which means that selective reversion of part
of the state is not possible. This makes it unsuitable for general backup and restore purposes.
For that we must use the virtual HSM backup and recovery below.

Operator Secret

To control where a snap-shotted cHSM may be cloned, Utimaco offers an operator secret which is baked
into the snapshotted archive. The secret is defined by the operator when taking ownership of the
HSM and the same value should be installed into all physical HSMs which are intended to be interoperable
at the cloned cHSM level.

A snapshotted cHSM may only be cloned onto a physical HSM with the same operator secret as was present
on the physical HSM from where the snapshot was made.

Virtual HSM Backup/Restore

A finer-grained level of backup and restore can be achieved using the cHSM Master Backup Key. This is
a key which, when shared by different cHSMs allows data from one cHSM to be backed up off-device and
then restored onto the other device.

This data can be single users, a complete set of users, the database for a particular HSM module (e.g.
CXI, CCHSM) or any combination of these.

The Master Backup Key is created off-device by the Utimaco tools and can be stored in multiple parts.
It is then installed in the cHSM during the process of taking a cHSM into service. Because restore
can be a significant attack vector, Access to the MBK Parts should be strictly defined and controlled.

It is possible to define a user authorised to do JUST Backup and Restore operations allowing a split in responsibilities at this level between backup and other administration operations.

Catastrophe Recovery

The minimum scenarios for a complete environment recovery are also the attack vectors for cloning an
environment with compromised governance. Compromised Governance can lead to a complete exposure
and manipulation of the application owner's sensitive data. Avoid it. At the same time, it
should never be possible to lose both of the reconstruction paths - if you do, then the system is
dead and protected data CANNOT be recovered.

One working cHSM with working ADMIN User

If there is one working cHSM with a working ADMIN user then this can be used to install an new MBK,
create new backup files and then propagate these to newly created cHSM instances.

Working Set of MBK Parts plus Backup Files

With a working set of MBK parts (e.g. 2 out of 3 in an M of N setup) the MBK can be installed on a
newly configured empty cHSM and then the backup files used to restore the HSM State. Note that
there is no guarantee that the new cHSM has been properly sealed (e.g. no default ADMIN User,
correct firmware etc.) and so this must be done carefully, exactly as for an initial take into service.

Special MBK Case: HSM Attest Storage Keys

The HSM Attest firmware allows export and import of single HSM Attest storage keys. This is the
only way to rebuild a desired state of the CCHSM module database if conflicting configuration errors
have been made, for example wrongful deletion of a key followed creation of a new one in the Database.
This import and export is secured by the same master backup key as other cHSM backups.

Limitations of Backup by Snapshot

With a cHSM snapshot, new cHSM Instances can be started, but the following limitations should prevent
its abuse, but also limit its usefulness:

1. The new cHSM can only be on a device with the correct operator secret.
2. The internal configuration of the cHSM cannot be changed without the cHSM Governance (ADMIN Users,
   MBK, etc...)

So, for example, this cannot be used to recover from a lost MBK or ADMIN Logon Credentials, or a detailed
error in one of the HSM Module databases.

Operational Models

These are the main fault lines on which HSM operations can be split between operator and application owner and between Organisational, Governance and Institutional Boundaries.

On Premise, Self-managed

Do it all yourself...

Operator and application owner are one institutional entity, the HSMs are the property of the owner,
are physically installed on owner Premises and fully managed through all the phases of their lifecycle
by the owner. The owner may create internal organisational boundaries to clarify role and expertise
requirements.

Governance may be completely internal or may involve external actors invited to act as multi-person governance parties.

Off-premise, Self-Managed

Same as above, but the physical location of the HSMs is in a data centre run by a third party which
takes over the physical operations role, but hands-off at the point of a network connected, factory
clean HSM to the Owner who then manages all further operations.

Hosted, Owned Hardware

Here the HSMs still belong or are uniquely assigned to the Application Owner, but they are located
in a data centre run by an institutionally distinct operator who also manages at least the global HSM Administration layer.

The Hand off between Operator and Application Owner can be at

• the boundary between global and cryptoserver operations. In this case the Application Owner receives a clean empty cHSM from the operator and takes over responsibility for it from that point.
• CryptoServer Operations can also be outsourced to the operator organisation in which case, the configured cHSM is made available to the Customer fully setup, with only a change in user credentials required to transfer ownership to the Application Owner.

In either case, the Application Owner will issue directives to the Operator for desired HSM Propagation and backup operations.

Hosted, Shared Hardware

As above, but a given single physical HSM may be used to supply cHSMs to a number of different Application Owners. There are questions of load quotas and scalability reserves to be addressed contractually here, but this model allows much more flexibility in capacity planning by the operator.

Hosted, HSM Attest (or similar) Managed

Same as either the shared or owned hardware model above but, in this case, the cHSM would be preconfigured with custom firmware, a base set of storage keys and a set of appropriate users to quickly set up an application using HSM Attest.

This is likely to be offered together with similar managed service offerings for Confidential Computing and HSM Attest Client pre-configurations designed to interoperate smoothly.

This operating model could also be applied to other specific HSM configurations, e.g. PKCS11 cHSM, Payments HSM, Certificate Authority HSM...