LDM-148.tex

\documentclass[DM,toc,lsstdraft]{lsstdoc}
\usepackage{enumitem}

\title{Data Management System Design}
\setDocRef{LDM-148}
\author{
  K.-T.~Lim,
  J.~Bosch,
  G.~Dubois-Felsmann,
  T.~Jenness,
  J.~Kantor,
  W.~O'Mullane,
  D.~Petravick,
  G.~Comoretto,
  and
  the DM Leadership Team}
\setDocCurator{Kian-Tat Lim}
\setDocUpstreamLocation{\url{https://github.com/lsst/LDM-148}}

\date{\today}
\setDocAbstract{%
  The LSST Data Management System (DMS) is a set of services
  employing a variety of software components running on
  computational and networking infrastructure that combine to
  deliver science data products to the observatory's users and
  support observatory operations.  This document describes the
  components, their service instances, and their deployment
  environments as well as the interfaces among them, the rest
  of the LSST system, and the outside world.
}
\setDocChangeRecord{%
  \addtohist{2}{2011-08-09}{Copied from MREFC Proposal into LDM-148 handle, reformatted}{Robert McKercher}
  \addtohist{3}{2011-08-15}{Updated for Preliminary Design Review}{Tim Axelrod, K-T Lim, Mike Freemon, Jeffrey Kantor}
  \addtohist{4}{2013-10-09}{Updated for Final Design Review}{Mario Juric, K-T Lim, Jeffrey Kantor}
  \addtohist{5.0}{2017-07-04}{Rewritten for Construction and Operations. Approved in \jira{RFC-358}.}{K-T Lim}
  \addtohist{6.0}{2018-07-12}{Add Observatory Operations Data and Planned Observation Publishing Services; synchronize terminology with new product tree, LDM-129, and LDM-230. Approved in \jira{RFC-456}.}{K-T~Lim}
  \addtohist{6.1}{2019-01-23}{Add historical note on DMCS.  Approved in \jira{RFC-530}.}{K-T~Lim}
  \addtohist{6.2}{2019-01-30}{Add text about outbound bandwidths and modify near-realtime dataflow figure.  Approved in \jira{RFC-538}.}{K-T~Lim}
  \addtohist{6.3}{2019-05-29}{Add service levels for enclaves.  Approved in \jira{RFC-581}.}{K-T~Lim}
  \addtohist{7.0}{2019-07-17}{Synchronize with product tree and add LSE-400.  Approved in \jira{RFC-614}.}{K-T~Lim}
}

\begin{document}
\maketitle

\section{Introduction}\label{introduction}

The purpose of the LSST Data Management System (DMS) is to deliver science data
products to the observatory's users and to support observatory operations.  The
DMS is a set of services employing a variety of software components running on
computational and networking infrastructure.  The DMS is constructed by the DM
subsystem in the NSF MREFC project; in the Operations era, it is operated by a
combination of the LSST Data Facility, Science Operations, and Observatory
Operations departments.

The data products to be delivered are defined and described in the \textit{Data
Products Definition Document} \citedsp{LSE-163}. These are divided into three
major categories.

One category of data products is generated on a nightly or daily cadence
and comprises raw, processed/calibrated, and difference images as well as alerts
of transient, moving, and variable objects detected from the images,
published within 60 seconds, and recorded in searchable catalogs. These
data products can be considered ``online'', as they are driven primarily
by the observing cadence of the observatory. This category of Prompt data products has
historically been referred to as ``Level 1''.  These products are intended to
enable detection and follow-up of time-sensitive time-domain events.

Make some change.

A second category of data products is generated on an annual cadence and
represents a complete reprocessing of the set of images taken to date to
generate astronomical catalogs containing measurements and
characterization of tens of billions of stars and galaxies with high and
uniform astrometric and photometric accuracy. As part of this
reprocessing, all of the first category of data products is regenerated,
often using more accurate algorithms. This category also includes other
data products such as calibration products and templates that are
generated in an ``offline'' mode, not directly tied to the observing
cadence. This category of Data Release data products has historically been referred to as ``Level 2'',
including the regenerated data products from the first category.

The third category of data products is not generated by the LSST DMS but is
instead generated, created, or imported by science users for their own science
goals. These products derive value from their close association with or
derivation from other LSST data products. The DMS is responsible for providing
facilities, services, and software for their generation and storage.  This
category of User Generated data products has historically been referred to as ``Level 3''.

Data products are delivered to science users through Data Access
Centers (DACs). In addition, streams of near-realtime alerts and planned
observations are provided.  Each LSST data product has associated
metadata providing provenance and quality metrics and tracing it to relevant
calibration information in the archive. The DACs are composed of modest but
significant computational, storage, networking, and other resources intended
for use as a flexible, multi-tenant environment for professional astronomers
with LSST data rights to retrieve, manipulate, and annotate LSST data products
in order to perform scientific discovery and inquiry.

The first section of this document describes how the DMS components work
together to generate and distribute the data products.  The next section
describes how the size of the DMS computing environments was estimated.
Subsequent sections describe the individual components of the DMS in more
detail, including their interfaces with each other, with other LSST subsystems,
and with the outside world.

\section{Summary Concept of Operations}\label{summary-concept-of-operations}

The principal functions of the DMS are to:
\begin{itemize}
	\item Process the incoming stream of images generated by the camera system during observing by archiving raw images, generating transient alerts, and updating difference source and object catalogs.
	\item Periodically (at least annually) process the accumulated survey data to provide a uniform photometric and astrometric calibration, measure the properties of fainter objects, and characterize the time-dependent behavior of objects. The results of such a processing run form a data release (DR), which is a static, self-consistent data set for use in performing scientific analysis of LSST data and publication of the results. All data releases are archived for the entire operational life of the LSST archive.
	\item Periodically create new calibration data products, such as bias frames and flat fields, to be used by the other processing functions.
	\item Make all LSST data available through an interface that utilizes, to the maximum practical extent, community-based standards such as those being developed by the Virtual Observatory (VO) in collaboration with the International Virtual Observatory Alliance (IVOA).  Provide enough processing, storage, and network bandwidth to enable user analysis of the data without petabyte-scale data transfers.
\end{itemize}

The latency requirements for alerts determine several aspects of the DMS design
and overall cost.  An alert is triggered by an unexpected excursion in
brightness of a known object or the appearance of a previously undetected
object such as a supernova or a GRB. The astrophysical time scale of some of
these events may warrant follow-up by other telescopes on short time scales.
These excursions in brightness must be recognized by the pipeline, and the
resulting alert data product sent on its way, within 60 seconds. This drives
the DMS design in the decision to acquire high-bandwidth/high-reliability
long-haul networking from the Summit at Cerro Pachon to the Base in La Serena and from Chile to the U.S. These networks allow the significant computational
resources necessary for promptly processing incoming images to be located in
cost-effective locations: the Base has far fewer limitations on power, cooling,
and rack space capacity than the Summit, and placing the scientific
processing at NCSA allows for far greater flexibility in the allocation of
resources to ensure that deadlines are met. Performing cross-talk correction
on the data in the data acquisition system and parallelizing the alert
processing at the amplifier and CCD levels, where possible, also help to
minimize the latency to alert delivery.

The Data Release processing requires extensive computation, combining
information from all images of an object in order to measure it as
accurately as possible.  A sophisticated workload and workflow management
system and Task Framework are used to divide the processing into
manageable units of work that can be assigned to available resources,
including the two dedicated processing clusters at NCSA and CC-IN2P3.

Calibration data products must be created and updated at cadences in between
the Alert and Data Release periods.  The stability of the system is expected to
require daily, monthly, and annual calibration productions.  The daily
production must be synchronized with the observatory schedule, occurring after
raw calibration frames have been taken but well before science observing is
planned.  This requirement necessitates the inclusion of a service that allows
the Observatory Control System to trigger remote calibration processing at
NCSA.

The DACs are a key component of the DMS, giving the community resources and an
interface to interact with and utilize the LSST data products to perform
science.  An instance of the LSST Science Platform (LSP) is deployed in each
DAC to support science users with its Portal, JupyterLab (notebook), and Web
API aspects.  Substantial compute, storage, and storage bandwidth is devoted to
ensuring that the LSP is responsive and allows for exploration of the vast
LSST data products.

Underlying all of the above is a Data Backbone that provides storage, tracking,
and replication for all LSST data products.  The Data Backbone links the
computing environments including the Data Access Centers, acting as the spine that
supports them all.

\input{sizing}

\section{Component Overview}\label{component-overview}

The services that make up the DMS are in turn made up of software and
underlying service components, instantiated in a particular
configuration in a particular computing environment to perform a
particular function. Some software components are specific to a service;
others are general-purpose and reused across multiple services. Many
services have only one instance in the production system; others have
several, and all have additional instances in the development and
integration environments for testing purposes.

The DMS services can be considered to consist of four tiers of software
components. The top tier is the LSST Science Platform, which is deployed
to provide a user
interface and analysis environment for science users and LSST staff. The
detailed design of this tier is given in \textit{Science Platform Design} \citedsp{LDM-542}. The next
tier is composed of science ``applications'' software that generates
data products. This software is used to build ``payloads'', sequences of
pipelines, that perform particular data analysis and product generation
tasks. It is also used by science users and staff to analyze the data
products. The detailed design of the components in this tier is given in
\textit{Data Management Science Pipelines Design} \citedsp{LDM-151}. A lower tier is
``middleware'' software components and services that execute the science
application payloads and isolate them from their environment, including
changes to underlying technologies. These components also provide data
access for science users and staff. The detailed design of the
components in this tier is given in \textit{Data Management Middleware Design} \citedsp{LDM-152}.
The bottom tier is ``infrastructure'': hardware, networking,
and low-level software and services that provide a computing
environment. The detailed design of components in this tier is given in
\textit{LSST Data Facility Logical Information Technology and Communications Design} \citedsp{LDM-129} and \textit{LSST Observatory Network Design} \citedsp{LSE-78}.

The DMS computing environments reside in four main physical locations:
the Summit Site including the main Observatory and Auxiliary Telescope
buildings on Cerro Pachon, Chile; the Base Facility data center located
at the Base Site in La Serena, Chile; the NCSA Facility data center
at the National Center for Supercomputing Applications (NCSA) in Urbana,
Illinois, USA; and the Satellite Facility at CC-IN2P3 in Lyon,
France. These are linked by high-speed networks to allow rapid data
movement.
These computing environments are separated into enclaves by their requirements for availability, access, and change management.

The Base Facility includes four enclaves: the Base portion of the Prompt Enclave directly supporting Observatory operations, the Commissioning Cluster, an Archive Enclave holding data products, and the Chilean Data Access Center.

The NCSA Facility also includes four enclaves: the NCSA portion of the Prompt Enclave, the Offline Production Enclave hosting all offline "data release" and calibration activities, another Archive Enclave, and the US Data Access Center.

Additionally, a separate Development and Integration Enclave at NCSA hosts many of the services and tools necessary to build and test the DMS.

These enclaves are distinguished by having different users, operations timescales, interfaces, and often components.
The service levels required of each enclave have relatively loose requirements set by the LSST System Requirements \citedsp{LSE-29} and the Observatory System Specification \citedsp{LSE-30}, which are then flowed down to the Data Management System Requirements \citedsp{LSE-61}.
Some enclaves have goals for availability, processing latency, or other parameters that may be significantly more strict, however.
The trade-offs between cost and availability and between availability and latency need to be evaluated carefully to ensure that the optimum balance is being struck.
Requirement DMS-REQ-0161 already specifies that reliability (i.e. correctness and consistency of service) should be prioritized over availability to end users, given equal cost.

DMS services are assigned to each of these enclaves.  Some enclave hardware may be dedicated; the remainder is allocated from Master Provisioning hardware pools at each Facility.
Each service is generally implemented by a corresponding software product.
In some cases, such as for the Archiving and Telemetry Gateway, multiple services are supported by a single software product.
In other cases, such as for the Image Ingest and EFD Transformation portions of the Archiving service, multiple software products support a single overall service.

The service instances that make up the DMS include (with the
enclave they are in noted):
\begin{itemize}
\item
  Archiving (Prompt Base)
\item
  Planned Observation Publication (Prompt Base)
\item
  Prompt Processing Ingest (Prompt Base)
\item
  Observatory Operations Data (Prompt Base)
\item
  Observatory Control System (OCS) Driven Batch (Prompt Base)
\item
  Telemetry Gateway (Prompt Base)
\item
  Prompt Processing (Prompt NCSA)
\item
  Alert Distribution (Prompt NCSA)
\item
  Prompt Quality Control (QC) (Prompt NCSA)
\item
  Batch Production (Offline Production, Satellite Facility)
\item
  Offline QC (Offline Production)
\item
  Bulk Distribution (Offline Production)
\item
  Data Backbone (Archive Base and NCSA)
\item
  LSST Science Platform Portal (Commissioning Cluster and DACs)
\item
  LSST Science Platform JupyterLab (Commissioning Cluster and DACs)
\item
  LSST Science Platform Web API (Commissioning Cluster and DACs)
\end{itemize}

The relationships between these services, their deployment enclaves
physical facilities, and science application ``payloads'' can be
visualized in Figure~\ref{fig:deployment}.

\begin{figure}
\centering
\includegraphics[width=\textwidth]{images/DMSDeployment.pdf}
\caption{Data Management System Deployment}
\label{fig:deployment}
\end{figure}

Other services necessary to build, test, and operate the DMS but that are not directly responsible for generating data products include:
\begin{itemize}
\item
  LSST Science Platform Science Validation instance (Development and Integration)
\item
  Developer Services (Development and Integration)
\item
  Managed Database (Infrastructure)
\item
  Batch Computing (Infrastructure)
\item
  Containerized Application Management (Infrastructure)
\item
  IT Security (Infrastructure)
\item
  Identity Management (Infrastructure)
\item
  ITC Provisioning and Management (Infrastructure)
\item
  Service Management/Monitoring (Infrastructure)
\end{itemize}

The common infrastructure services are illustrated in Figure~\ref{fig:commonservices}.

\begin{figure}
\centering
\includegraphics[height=0.9\textheight]{images/DMSCommonServices.pdf}
\caption{Data Management System Common Infrastructure Services}
\label{fig:commonservices}
\end{figure}

The science application software for the Alert Production, daytime
processing, Data Release Production, and calibration processing is built
out of a set of libraries and frameworks that accept plugins. In turn, those
frameworks build on middleware that provides portability and
scalability.  The relationships between the packages implementing
these frameworks and plugins and the underlying middleware packages
are shown in Figure~\ref{fig:scipi}.

The Science Pipelines Software product category contains the production pipelines that generate data products.
Those rely on the Science Pipelines Libraries supporting software product, which includes these key components:
\begin{itemize}
\item
  Low-level astronomical software primitives and data structures
  (\texttt{afw})
\item
  Image processing and measurement framework with core algorithms
  (\texttt{ip\_*}, \texttt{meas\_*})
\item
  Additional image processing and measurement algorithms
  (\texttt{meas\_extensions\_*})
\item
  High-level algorithms and driver scripts that define pipelines
  (\texttt{pipe\_tasks}, \texttt{pipe\_drivers})
\item
  Camera-specific customizations (\texttt{obs\_*})
\end{itemize}
Documentation for the Science Pipelines is available at \href{https://pipelines.lsst.io/}{pipelines.lsst.io}.

\begin{figure}
\centering
\includegraphics[width=\textwidth]{images/DM_Application_Software_Arch.png}
\caption{Data Management Science Pipelines Software ``Stack''}
\label{fig:scipi}
\end{figure}

The middleware layer includes software products that allow the building of algorithmic pipelines and abstract their data access.
\begin{itemize}
\item
  Data access client (Data Butler) (\texttt{daf\_butler}, \texttt{daf\_persistence})
\item
  Task framework (\texttt{pex\_*}, \texttt{log}, \texttt{pipe\_base}, \texttt{ctrl\_pool})
\end{itemize}

These pipelines are in turn executed and controlled by Batch Production software products:
\begin{itemize}
\item
  Campaign management (\texttt{ctrl\_bps})
\item
  Workload/workflow management
\end{itemize}

Infrastructure components include:
\begin{itemize}
\item
  Parallel distributed database (\texttt{qserv})
\item
  Other databases (typically relational)
\item
  Common off-the-shelf software and other third-party products
\item
  Low-level software such as operating systems
\item
  Filesystems
\item
  Authentication and authorization (identity management)
\item
  Provisioning and resource management
\item
  Monitoring
\item
  Hardware (compute, storage, and network) configurations
\end{itemize}

The relationships between the middleware and infrastructure components
are illustrated in Figure~\ref{fig:mwandinfra}.

\begin{figure}
\centering
\includegraphics[width=\textwidth]{images/MiddlewareInfrastructure.pdf}
\caption{Data Management Middleware and Infrastructure}
\label{fig:mwandinfra}
\end{figure}


\section{Prompt Base Enclave}\label{prompt-base-enclave}

Services located in this enclave are located at the Base solely because
they must interact with the OCS or the Camera Data System (also known as
the Camera DAQ) or both. In several cases, services located here
interact closely with corresponding services in the Prompt NCSA Enclave,
to the point where the Base service cannot function if the
NCSA service is not operational. This reliance has been taken into
account in the fault tolerance strategies used.

The primary goals of the services in this enclave are to transfer data
to appropriate locations, either to NCSA, from NCSA, or to the Data
Backbone.

The services in this enclave and their partners in the Prompt NCSA Enclave
need to run rapidly and reliably. They run at times
(outside office hours) and with latencies that are not amenable to a
human-in-the-loop design. Instead, they are designed to execute
autonomously, often under the control of the OCS, with human oversight,
monitoring, and control only at the highest level.

Historical designs for the equivalent of this enclave envisioned an overarching DM Control System (DMCS) that would be the sole externally-visible service controlling all others.
The current design exposes each service as a separate entity with potentially different service levels.
A DMCS component still exists, but it is an implementation feature of the services' design rather than an architectural component.

The overall service level for this enclave should be high, with the goal being to have the enclave functional whenever the Observatory is taking data.
Since these services are part of the normal functioning of the Observatory, they are also referred to as Observatory Operations Services.

Of the services within the enclave, the Header Generator component of the Archiving service is critical to observing and should have a target uptime equal to the Engineering and Facility Database.

The Archiving, Observatory Operations Data, OCS Driven Batch, Prompt Processing Ingest, and Telemetry Gateway services are important for maintaining high-quality observing; lack of these services constitutes a degraded mode of operation without useful feedback loops.
Downtime, defined as when observing is occurring but one or more of these services is unavailable, must be limited in both length and frequency.
There are daily and annual observatory scheduled maintenance periods during which no observing occurs; accordingly, these services do not need to remain available during these periods.
During normal observing, the stability of the telescope systems is expected to allow at most 24 consecutive hours of downtime during any single incident for the Archiving, Observatory Operations Data and OCS Driven Batch services.
(This might require reconfiguring OCS Driven Batch to execute on the Commissioning Cluster rather than in the Offline Production enclave.)
Similarly, DMS-REQ-0008 requires that the pipelines fed by Prompt Processing Ingest and producing output via the Telemetry Gateway be down for no more than 24 hours at a time.
These services must be available for at least 24 continuous hours out of any 48 hour period in order to maintain optimal Observatory scientific performance.

The overall Archiving service can prevent further observing if the capacity of the buffer in the Camera Data System (DAQ) at the Summit is exceeded, so it must be up sufficiently, particularly after a Summit-to-Base network outage, to drain that buffer.

All of these services need to function sufficiently to meet requirement DMS-REQ-0318, which says that DM as a whole must not prevent observing for more than 1 day total per year.
As a result, the above limits on downtime can be exceeded, but only as long as total observing outages do not exceed 1 day when integrated over a year.

The Planned Observation Publication service provides information for external clients.
Because it publishes advance predictions that are not guaranteed, it is acceptable for this service to be down during observing for an amount of time less than the prediction window.
Having this service down does not prevent further observing.

\subsection{Service Descriptions}\label{base-service-descriptions}

Detailed concepts of operations for each service can be found in
\textit{Concept of Operations for the LSST Production Services} \citedsp{LDM-230}.


\subsubsection{Archiving}\label{archiving}

This component is composed of several Image Archiver service and
Catch-Up Image Archiver instances: one pair each for the LSSTCam, the
ComCam, and the Auxiliary Telescope Spectrograph, all of which may be
operated simultaneously. These capture raw images taken by each camera,
including the wavefront sensors and the guide sensors of the LSSTCam or
ComCam when so configured, retrieving them from their respective Camera
Data System instances.

A Header Generator written by Data Management but operated by the Observatory captures specific sets of metadata
associated with the images, including telemetry values and event
timings, from the OCS publish/subscribe middleware and/or from the EFD.
It formats these into a metadata package that is recorded in the EFD Large File Annex.
The Archiver and Catch-Up Archiver instances retrieve this metadata package and attach it to the captured image pixels.

The image pixels and metadata are passed to the Observatory Operations Data Service (OODS), which serves as a buffer from which observing-critical data can be retrieved.
They are also passed to a staging area for ingestion into the permanent archive in the Data Backbone.
The catch-up versions archive into the OODS and Data Backbone any raw
images and metadata that were missed by the primary archiving services
due to network or other outage, retrieving them from the flash storage
in the Camera Data System instances and the EFD.

This component also includes an EFD Transformation service that extracts
all information (including telemetry, events, configurations, and
commands) from the EFD and its large file annex, transforms it into a
form more suitable for querying by image timestamp, and loads it into
the permanently archived ``Transformed EFD'' database in the Data
Backbone.

\subsubsection{Planned Observation Publication}\label{planned-observation-publication}

This service receives telemetry from the OCS describing the next visit location and the telescope scheduler's predictions of its future observations.
It publishes these as an unauthenticated, globally-accessible web service comprising both a web page for human inspection and a web API for usage by automated tools.

\subsubsection{Prompt Processing Ingest}\label{prompt-processing-ingest}

This component is composed of two instances that capture
crosstalk-corrected images from the LSSTCam and ComCam Camera Data
Systems along with selected metadata from the OCS and/or EFD and
transfer them to the Prompt Processing service in the Prompt NCSA Enclave.

There is no Prompt Processing Ingest instance for the auxiliary
telescope spectrograph.

\subsubsection{Observatory Operations Data}\label{obs-ops-data}

This service provides low-latency access to images, other files, and metadata for use by Observatory systems and the Commissioning Cluster LSP instance.
It maintains a higher level of service availability than the Data Backbone.
After images, files, and metadata are ingested, they remain available through the OODS for a policy-configured amount of time.
Files stored in the OODS include raw images from the Camera as well as master calibration images from the Calibration Products Production payloads, intended for use by the Active Optics and Guider components of the Telescope and Site subsystem.

\subsubsection{OCS Driven Batch}\label{ocs-driven-batch}

This service receives commands from the OCS and invokes a Batch Computing service to execute configured science payloads.
The service can be configured to execute on the Commissioning Cluster at the Base or in the Offline Production Enclave at NCSA.
It is used for modest-latency analysis of images during Commissioning and for processing daily calibration images in normal observing operations.
Images and metadata are taken from the Data Backbone, and results are provided back to the Data Backbone; there is no direct connection from this service to the Camera Data System.
This obviously bounds the minimum latency from image acquisition to processing start by the latency of the Archiving service and Data Backbone transfer.
A summary status for the processing performed is sent to the OCS Driven Batch Control service to be returned to the OCS.

\subsubsection{Telemetry Gateway}\label{telemetry-gateway}

This service obtains information from the Prompt NCSA Enclave,
particularly status and quality metrics from Prompt Processing of images
and the Prompt Quality Control service, and transmits it to the OCS as
specified in the \textit{Data Management-OCS Software Communication Interface}
\citedsp{LSE-72}. Note that more detailed information on the status and
performance of DMS services will also be available to Observatory
operators through remote displays originated from the
Service Management/Monitoring infrastructure services in all DMS enclaves.

\subsection{Interfaces}\label{base-interfaces}

OCS to all Prompt Base Enclave services: these interface through the SAL
library provided by the OCS subsystem.

Archiver and Catch-Up Archiver to Observatory Operations Data Service:
image files with associated metadata are written to OODS storage.

Archiver and Catch-Up Archiver to Data Backbone: files are copied to
Data Backbone storage via a file transfer mechanism, and their
information and metadata are registered with Data Backbone management
databases. The Data Butler is not used for this low-level,
non-science-payload interface.

Observatory Operations Data Service to Data Backbone: files are copied from the Data Backbone to the OODS after completion and validation of calibration production results.

Observatory Operations Data Service to Commissioning Cluster LSP and other users: the OODS provides a mountable POSIX filesystem interface.
A web interface (e.g. WebDAV) may also be provided, but this will be as simple as possible to enable maintaining a very high level of service availability and reliability.

EFD to EFD Transformer: this interface is via connection to the
databases that make up the EFD as well as file transfer from the EFD's
Large File Annex.

EFD Transformer to Data Backbone: Transformed EFD entries are inserted
into the ``Transformed EFD'' database resident within the Data Backbone.

Camera Data System to Archiver, Catch-Up Archiver, Prompt Processing
Ingest: these interface through the custom library provided by the
Camera Data System.

Prompt Processing Ingest to Prompt Processing: BBFTP is used to transfer
files over the international network from the ingest service to the
processing service.

OCS Driven Batch to Batch Computing: HTCondor is
used to transfer execution instructions
from the control service to the batch service, whether to the Commissioning Cluster or over the international network to the Offline Production Enclave, and return status and
result information.

Telemetry Gateway from Prompt NCSA Enclave services: RabbitMQ is
used to transfer status and quality metrics to the gateway over the
international network.

\section{Prompt NCSA Enclave}\label{prompt-ncsa-enclave}

This enclave is responsible for the compute-intensive processing for all
near-realtime operations and other operations closely tied with the
Observatory. Its primary goals are to process images and metadata from
the Observatory into ``online'' science data products and publish them
to the DACs, alert subscribers, and back to the OCS.

The Prompt Processing service executes science payloads that are tightly integrated with the observing cadence and is intended to function in near-realtime with strict result deadlines for both science and raw calibration images.

Note that offline (typically daytime) processing to generate Prompt data products occurs under the control of the Batch Production service in the Offline Production Enclave using its Batch Computing resources.

The Alert Distribution service receives batches of alerts resulting from Prompt Processing of each science visit; it then provides bulk alert streams to community alert brokers and filtered alert streams to LSST data rights holders.

The Prompt Quality Control service monitors the ``online'' science data
products, including alerts, notifying operators if any anomalies are
found.

Like the services in the Base Center, these services need to run
rapidly and reliably and so are designed to execute autonomously.

The services in this enclave are important parts of the observing feedback system and the prompt DM pipelines and accordingly should have a target of being up whenever the Observatory is taking data.
They should be down (while observing is taking place) for no more than 24 hours at a time (DMS-REQ-0008), although additional downtime adding up to no more than 1 day per year (DMS-REQ-0318) can be tolerated.
Latency for services in this enclave must be low in order to meet the 60 second alert delivery requirement.

\subsection{Service Descriptions}\label{prompt-ncsa-service-descriptions}

Detailed concepts of operations for each service can be found in
\textit{Concept of Operations for the LSST Production Services} \citedsp{LDM-230}.

\subsubsection{Prompt Processing}\label{prompt-processing}

This service receives crosstalk-corrected images and metadata from the
Prompt Processing Ingest service at the Base and executes the Alert
Production science payload on them, generating ``online'' data products
that are stored in the Data Backbone. The Alert Production payload then
sends alerts to the Alert Distribution service.

The Prompt Processing service has calibration (including Collimated Beam
Projector images), science, and deep drilling modes. In calibration
mode, it executes a Raw Calibration Validation payload that provides
rapid feedback of raw calibration image quality. In normal science mode,
two consecutive exposures are grouped and processed as a single visit.
Definitions of exposure groupings to be processed as visits in deep
drilling and other modes are TBD. The service is required to deliver
Alerts within 60 seconds of the final camera readout of a standard
science visit with 98\% availability.

There is no Prompt Processing service instance for the Auxiliary
Telescope Spectrograph.

\subsubsection{Alert Distribution}\label{alert-distribution}

This service obtains alerts generated by the Alert Production science payload and distributes them to community alert brokers.
A full Alert stream is also fed to a filtering component (also known as the "mini-broker"); that component allows individual LSST data rights holders to execute limited filters against the stream, producing filtered feeds that are then distributed to the individuals.

\subsubsection{Prompt Quality Control}\label{prompt-quality-control}

This service collects information on Prompt science and calibration
payload execution, post-processes the science data products from the
Data Backbone to generate additional measurements, and monitors the
measurement values against defined thresholds, providing an automated
quality control capability for potentially detecting issues with the
environment, telescope, camera, data acquisition, or data processing.
Alarms stemming from threshold crossings are delivered to Observatory
operators and to LSST Data Facility Production Scientists for
verification, analysis, and resolution.

\subsection{Interfaces}\label{prompt-ncsa-interfaces}

Prompt Processing to Alert Distribution: these
interface through a reliable transport system.

Prompt Processing to Batch Production: in the event that Prompt
Processing runs over its allotted time window, processing can be
cancelled and the failure recorded, after which Offline Processing within
the Batch Production service will
redo the processing at a later time. Note that it may be possible, if
sufficient computational resources have been provisioned, for the Prompt
Processing to be allowed to continue to run, with spare capacity used to
maintain latency for future visits. In that case, there would
effectively be an infinite time window.

Science Payloads to Data Backbone: payloads use the Data Butler as a
client to access files and catalog databases within the Data Backbone.

\section{Offline Production Enclave}\label{offline-production-enclave}

This enclave is responsible for all longer-period data processing
operations, including the largest and most complex payloads supported by
the DMS: the annual Data Release Production (DRP) and periodic
Calibration Products Productions (CPPs). Note that CPPs will execute
even while the annual DRP is executing.
The Offline Quality Control Service monitors the science data
products, notifying operators if any anomalies are found.
The enclave also includes a service for distributing bulk data on daily and annual (Data Release) timescales to partner institutions, collaborations, and LSST Education and Public Outreach (EPO).
This bulk data transfer may occur in large batches (e.g. once per day or when a Data Release is about to occur) or in narrower streams (e.g. as candidate data products become available from the Data Release Production).

The services in this enclave need to run efficiently and reliably over
long periods of time, spanning weeks or months. They need to execute
millions or billions of tasks when their input data becomes available
while tracking the status of each and preserving its output. They are
designed to execute autonomously with human oversight, monitoring, and
control primarily at the highest level, although provisions are made for
manual intervention if absolutely necessary.

This enclave does not have direct users (besides the operators of its
services); the services within it obtain inputs from the Data Backbone
and place their outputs (if any) into the Data Backbone.

Requirement DMS-REQ-0008 places a limit of 24 hours on each unscheduled outage of this enclave.
Otherwise, downtime is limited by the need to generate timely calibration products and annual Data Release data products.
This is expected to allow no more than 1 week of integrated scheduled or unscheduled downtime (e.g. half of the enclave down for two weeks) per year for this enclave.
Because the total capacity of this enclave is needed in order to provide the throughput necessary to complete the productions in the allotted time, any outage that decreases capacity is relevant, even if it occurs as part of a rolling maintenance in which services remain running.
There are no particular latency requirements for services in this enclave, other than what is necessary in order to meet the periodic Calibration Products and annual Data Release cycles.

\subsection{Service Descriptions}\label{ncsa-gen-prod-service-descriptions}

\subsubsection{Batch Production}\label{batch-production}

This service executes science payloads as "campaigns" consisting of a defined pipeline, a defined configuration, and defined inputs and outputs.
Many different payloads may be executed on many different campaign cadences.
These include:
\begin{itemize}
	\item Offline processing for Prompt data products
	\item Calibration Products Production
	\item Template Production
	\item Special Programs Productions
	\item Data Release Production
\end{itemize}

The service is able to handle massively distributed computing, executing jobs when their inputs become available and tracking their status and outputs.
It ensures that the data needed for a job is accessible to it and that outputs (including log files, if any) are preserved.
It can allocate work across multiple enclaves, in particular between NCSA and the Satellite Facility at CC-IN2P3, which will have capacity for half of the DRP processing.
It utilizes the Campaign Management and Workload/Workflow Management software products to accomplish these goals.

Offline processing ensures that Prompt data products are generated within the nominal 24 hours.
It includes catch-up of missed nightly processing as well as daytime processing such as the Moving Object Processing System.

Calibration Products Production campaigns execute various CPP science payloads at intervals to
generate Master Calibration Images and populate the Calibration Database
with information derived from analysis of raw calibration images from
the Data Backbone and information in the Transformed EFD. This includes
the computation of crosstalk correction matrices.
Additional information such as external catalogs are also
taken from the Data Backbone. The intervals at which this service
executes will depend on the stability of Observatory systems but are
expected to include at least monthly and annual executions. The annual
execution is a prerequisite for the subsequent execution of the Data
Release Production. The service involves human scientist/operator input
to determine initial configurations of the payload, to monitor and
analyze the results, and possibly to provide additional configuration
information during execution.

Template Production campaigns, typically run annually, generate the static sky templates used by Alert Production, based on raw science images from the Data Backbone.

Special Programs Productions campaigns perform custom analyses on raw science images taken for these programs.
The cadence for these campaigns may vary based on the particular Special Program.

Data Release Production campaigns execute the DRP science payload annually to generate all
Data Release data products after the annual CPP is executed. A small-scale
(about 10\% of the sky) mini-production is executed first to ensure
readiness, followed by the full production. Raw science images are taken
from the Data Backbone along with Master Calibration Images and
information from the Transformed EFD. Additional information such as
external catalogs may also be taken from the Data Backbone.

Output data products from both the mini-production and the main
production are loaded into the Data Backbone, including both images and
catalogs. From there, they are analyzed by LSST staff scientists and
selected external scientists using the Science Validation instance of
the LSST Science Platform to ensure quality and readiness for release.
The to-be-released data products are loaded into the Data Access Center
services, and access is then enabled on the release date. The service
involves human scientist/operator/programmer input to determine initial
configurations of the payload, to monitor and analyze results, and, when
absolutely necessary, to make ``hot fixes'' during execution that
maintain adequate consistency of the resulting data products.

\subsubsection{Offline Quality Control}\label{offline-quality-control}

This collects information on Calibration, Template Generation, and Data Release science payload execution,
post-processes the science data products from the Data Backbone to
generate additional measurements, and monitors the measurement values
against defined thresholds, providing an automated quality control
capability for potentially detecting issues with the data processing but
also the environment, telescope, camera, or data acquisition. Alarms
stemming from threshold crossings are delivered to LSST Data Facility
Production Scientists for verification, analysis, and resolution.

\subsubsection{Bulk Distribution}\label{bulk-distribution}

This service is used to transmit Prompt and Data Release data products to partners such as LSST Education and Public Outreach, the UK LSST project, and the Dark Energy Science Collaboration.
It extracts data products from the Data Backbone and transmits them over high bandwidth connections to designated, pre-subscribed partners.


\subsection{Interfaces}\label{ncsa-general-production-interfaces}

Batch Production to Data Backbone: for large-scale productions, a workflow
system is expected to stage files and selected database entries from the
Data Backbone to local storage for access by the science payloads via
the Data Butler. Similarly, the staging system will ingest output images
and catalogs into the Data Backbone.

Batch Production to Satellite Facility: the Data Backbone will transfer raw data, including images, metadata, and the Transformed EFD, to the Satellite Facility.
Intermediate data products will be transferred back via the Data Backbone for further computations in the Offline Production Enclave.

Bulk Distribution to Data Backbone: all data products sent via Bulk Distribution are retrieved from the Data Backbone.

Bulk Distribution to partners: the exact delivery mechanism for
large-scale data distribution is TBD.


\section{Data Access Centers}\label{data-access-centers}

There are two Data Access Centers, one in the US at NCSA and one in
Chile at the Base. These DACs are responsible for all
science-user-facing services, primarily instances of the LSST Science
Platform (LSP). The LSP is the preferred analytic interface to LSST data
products in the DAC. It provides computation and data access on both
interactive and asynchronous timescales.

The services in each enclave must support multiple users simultaneously
and securely. The LSP must be responsive to science user needs; updates
are likely to occur at a different cadence from the other enclaves as a
result. The LSP must operate reliably enough that scientific work is not
impeded.

OSS-REQ-0180 states that data products should be available from the DACs 98\% of the time, including scheduled and unscheduled downtime, and that a single outage can last at most 3 working days.
The Bulk Distribution service is expected to be used heavily around the time of a Data Release (possibly both before and after); it should have minimal downtime during those periods but can have lower availability otherwise.

\subsection{Service Descriptions}\label{dac-service-descriptions}

\subsubsection{LSST Science Platform DAC
instances}\label{lsst-science-platform-dac-instances}

This service provides an exploratory analysis environment for science
users. It can be further broken down into three ``Aspects'' that it
presents to end users, along with underlying ``backend services'' that
users can take advantage of, as illustrated in Figure~\ref{fig:lsp}.

\begin{figure}
\centering
\includegraphics[width=0.5\textwidth]{images/SciencePlatform.pdf}
\caption{LSST Science Platform}
\label{fig:lsp}
\end{figure}

The ``Portal'' Aspect provides a pre-specified yet flexible discovery,
query, and viewing tool. The ``JupyterLab'' Aspect provides a fully
flexible (``notebook'') environment incorporating rendering of images,
catalogs, and plots and providing for execution of LSST-provided and
custom algorithms. The ``Web API'' Aspect provides a
language-independent, VO-compliant Web Services data access API with
extensions for LSST capabilities and volumes. Access is provided via all
three Aspects to all data products, including images, catalogs, and
metadata. The Web API Aspect regenerates ``virtual'' data products on
demand when required.

The backend services provide general-purpose user computation, including
batch job submission from the Containerized Application Management Service and Batch Computing Service pools; file storage for User Generated data products which is accessible to all three Aspects and in particular exposed through one or more Web API Aspect services; and database storage for User Generated relational tables which is also accessible to all three Aspects.
User Generated data may be shared with
individual users, with groups, or with all DAC users (data rights
holders). Resource management of the backend services is based on a
small ``birthright'' quota with additional resources allocated by a
committee.

LSST Science Platform instances access read-only data products, including files and databases, from the Data Backbone.
In addition, files may be cached within the LSP instance for speed, and large-scale catalogs are typically loaded into an instance of the Qserv database management system for efficient query and analysis.

All usage of any LSST Science Platform instance requires authentication
to ensure availability only to LSST data rights holders or LSST
operations staff.

\subsection{Interfaces}\label{dac-interfaces}

LSST Science Platform to Data Backbone: the LSP services retrieve their data, including raw images, nightly and
annual image and catalog data products, metadata, and provenance, from the Data
Backbone.  The LSP Portal Aspect uses the LSP Web APIs to retrieve data.  The
LSP JupyterLab Aspect can use the LSP Web APIs and also can use the Data Butler
client library to access the Data Backbone.


\section{Commissioning Cluster}\label{commissioning-cluster}

The Commissioning Cluster enclave provides important services to the Observatory for rapid-turnaround human-driven ad hoc analysis of data during the Commissioning period and any re-commissioning of the system (e.g. after extended maintenance).
These analyses will be used for debugging the Observatory system; inability to perform such analyses can delay verification and validation in Commissioning.
In addition, this enclave supports human-driven quality control at the Base.
As a result, the goal should be to have this cluster functioning whenever the Observatory is taking data as well as during the day, except for scheduled daytime outages coordinated with Commissioning activities.
While it is operating, the latency of data delivery to the enclave by the OODS needs to be as low as feasible to support tight feedback loops during Commissioning.
It is desirable but not required to have low Data Backbone latency for delivery of pipeline outputs.
This enclave is separate as its human-centered services are distinct from the automated ones in the Prompt Base enclave.

\subsection{Service Description}\label{commcluster-service}

\subsubsection{LSST Science Platform Commissioning
instance}\label{lsst-science-platform-commissioning-instance}

This instance of the LSST Science Platform for Science Validation runs
on the Commissioning Cluster at the Base Facility (but also has access
to computational resources at the Archive) and accesses a Base endpoint
for the Data Backbone. This location at the Base lowers the latency of
both access to Data Backbone-resident data (which does not have to wait
for transfer over the international network) and, perhaps more
importantly, for user interface operations for staff in Chile, which are
served locally. Note that the Commissioning Cluster does not have direct
access to the Camera Data System; it relies on the Archiver service to
obtain data. The Commissioning Cluster will have direct access to the
OCS's Base replica of the EFD (before transformation).

\subsection{Interfaces}\label{commcluster-interfaces}

Commissioning Cluster to Data Backbone: The Commissioning Cluster relies on the
Data Backbone for its historical data, like the other instances of the LSST Science
Platform.

Commissioning Cluster to OODS: The Commissioning Cluster obtains its freshest, most recent data from the OODS.

Commissioning Cluster to EFD: The Commissioning Cluster has direct read-only
client access to the Base replica of the EFD (before transformation).

\section{Backbone Services}\label{backbone-services}

Detailed concepts of operations for each service can be found in \textit{Concept of Operations for the LSST Production Services} \citedsp{LDM-230}.

The Backbone enclaves supply data to other services.
As a result, the availability and latency requirements of the enclaves are constrained by those of the services it feeds.
In general, the system has been designed so that the Data Backbone is not essential for normal Observatory data-taking.
Latency requirements are thus loose, with permissible worst-case delivery times of up to 24 hours, although shorter normal-case latencies on the order of seconds are desirable for staff and science users alike.
Availability is primarily set by the need to support the Data Access Centers' 98\% availability and maximum three-working-day downtime.

\subsection{Service Descriptions}\label{backbone-service-descriptions}

The Data Backbone (DBB) is a key component that provides for data storage, transport, and replication, allowing data products to move between enclaves.
This service provides policy-based replication of files (including images and flat files to be loaded into databases as well as other raw and intermediate files) and databases (including metadata about files as well as other miscellaneous databases but not including the large Data Release catalogs) across multiple physical locations, including the Base, Commissioning Cluster, NCSA, and DACs.
It manages caches of files at each endpoint as well as persistence to long-term archival storage (e.g. tape).
It provides a registration mechanism for new datasets and database entries and a retrieval mechanism compatible with the Data Butler.

The relationships between the Data Backbone components are illustrated
in Figure~\ref{fig:dbb}.

\begin{figure}
\centering
\includegraphics[width=0.7\textwidth]{images/DataBackbone.pdf}
\caption{Data Backbone}
\label{fig:dbb}
\end{figure}

\subsubsection{DBB Ingest/Metadata Management}\label{dbb-ingest-metadata}

This service within the Data Backbone is responsible for maintaining and providing access to the metadata describing the location, characteristics, and provenance of the data products it manages.
Part of this service involves creating the appropriate metadata during ingest when data from external sources is incorporated into the DBB.
The Batch Production services will generally create the necessary DBB metadata as part of their operation, so only a minimal ingest process is needed for internally-generated data products.
Metadata is kept in a database that is a superset of the registry required by the Data Butler, allowing the Butler to directly access data within the DBB.

\subsubsection{DBB Lifetime Management}\label{dbb-lifetime-metadata}

This service is responsible for managing the lifetimes of data products within the DBB based on a set of policies.
Products may move from high-speed storage to near-line or offline storage or may be deleted completely.
Some products are kept permanently.
Some are kept for defined time periods as specified in requirements.
Intermediates may be kept until all downstream products have been generated.

\subsubsection{DBB Transport/Replication/Backup}\label{dbb-transport-repl}

This service is responsible for moving data products from one Facility to another and to backup and disaster recovery storage.
It handles recovery if a data product is found to be missing or corrupt.

\subsubsection{DBB Storage}\label{dbb-storage}

This service is responsible for storage of data products in the DBB.
The storage service provides an interface usable by the Data Butler as a datastore.

\subsection{Interfaces}\label{backbone-interfaces}

The Data Backbone services interact with most other deployed services.


\section{Software Components}\label{software-components}

DM is responsible for delivering a set of operational services, as described above, and most of those are implemented by custom or customized software components.
The implementing software products are distinct from the service products as they do not include configurations needed for deploying the service in different environments.
This section describes some of the software products that are not directly tied to corresponding services.

\subsection{Science Payloads}\label{science-payloads}

These payloads are described in more detail in the DM Applications Design Document \citedsp{LDM-151}.
Payloads are built from application software components from the Science Pipelines Libraries or third-party products.
They specify inputs, outputs, and algorithmic configurations as well as the order of execution of science algorithms from those libraries.
Most payloads execute under control of the Batch Production service.
Exceptions include the Alert Production Payload and some of the Calibration Software.

\subsubsection{Alert Production Payload}\label{alert-production-payload}

Executes under control of the Prompt Processing service. Generates all
Prompt science data products including Alerts (with the exception of
Solar System object orbits) and loads them into the Data Backbone and
Prompt Products Database. Transmits Alerts to Alert Distribution service.
Generates image quality feedback to the OCS and observers via the
Telemetry Gateway. Uses crosstalk-corrected science images and
associated metadata delivered by the Prompt Processing service; uses
Master Calibration Images, Template Images, Prompt Products Database, and
Calibration Database information from the Data Backbone.

\subsubsection{Calibration Software}\label{calibration-software-payload}

Executes under control of the Prompt Processing service, OCS-controlled batch processing, and offline batch production at intervals ranging from per-exposure to daily to annual, depending on the stability of Observatory systems and their calibrations.
Generates all calibration products.
Provides raw calibration image quality feedback to the OCS and observers via the
Telemetry Gateway. Uses crosstalk-corrected science images and
associated metadata delivered by the Prompt Processing service in addition to raw calibration images, Auxiliary Telescope images, and Master
Calibration Images from the Data Backbone and EFD information from EFD Transformation and the Data Backbone.

\subsubsection{Data Release Production
Payload}\label{data-release-production-payload}

Executes at annual intervals,
first running a ``mini-DRP'' over a small portion of the sky, followed
by the full DRP over the entire sky. Produces science data products in
the Data Backbone.

\subsubsection{MOPS and Forced Photometry Payload}\label{mops-payload}

Executes after a night's
observations are complete. Generates entries in the MOPS Database and
the Prompt Products Database, including Solar System Object records,
measurements, and orbits. Performs precovery forced photometry of
transients. Uses Prompt Products Database entries and images from the Data
Backbone.

\subsubsection{Special Programs Productions
Payload}\label{special-programs-productions-payload}

Executes at program-defined intervals.
Uses raw science images to generate special programs science data products, placing them in the Data Backbone.

\subsubsection{Template Generation
Payload}\label{template-generation-payload}

Executes if necessary to generate templates for Alert Production in between annual
Data Release Productions. Uses raw science images to generate the
templates, placing them in the Data Backbone.

\subsection{SUIT}\label{suit}

The Science User Interface and Tools, primarily composed of the Firefly product, provide visualization, plotting,
catalog rendering, browsing, and searching elements that can be
assembled into predetermined ``portals'' but can also be used flexibly
within dynamic ``notebook'' environments.

\subsection{Supporting Software}\label{supporting-software}

These software products, plus the middleware in the next subsection, are key underlying components to services and other software products.

\subsubsection{Science Pipelines Libraries}\label{science-pipelines-libraries}

These libraries provide objects, data structures, algorithms, and frameworks for writing Science Payloads.
These libraries contain the key scientific analysis routines used to perform processing and measurement to generate the LSST data products.

\subsubsection{Distributed Database
(Qserv)}\label{distributed-database-qserv}

The petabyte-scale, tens-of-trillions-of-rows catalogs produced by LSST are served to users by the catalog query portion of the LSST Science Platform Web APIs aspect in the Data Access Centers.
A custom parallel distributed database provides the required performance for these large and heavily-used catalogs that are resident in the DACs.

\subsubsection{ADQL Translator}\label{adql-translator}

The ADQL (Astronomical Data Query Language) Translator serves to connect the catalog query portion of the LSST Science Platform Web APIs aspect, including its VO-standard TAP interface, with underlying databases, including both the Parallel Distributed Database (Qserv) and conventional relational databases.

\subsubsection{Image/Cutout Server}\label{image-server}

The Image Server connects the image query portion of the LSST Science Platform Web APIs aspect with underlying image data products in the Data Backbone.
It also serves to regenerate "virtual" data products that have been removed for space reasons.

\subsection{Middleware}\label{middleware}

The detailed design of the Middleware components is in \textit{Data Management Middleware Design} \citedsp{LDM-152}.

\subsubsection{Data Butler Access
Client}\label{data-butler-access-client}

The Data Butler provides an access abstraction for all science payloads
that enables their underlying data sources and destinations to be
configured at runtime with a variety of back-ends ranging from local
disk to network locations and a variety of serializations ranging from
YAML and FITS files (extensible to HDF5 or ASDF) to database tables. The
Butler client is also available within the LSST Science Platform
JupyterLab environment.

\subsubsection{Task Framework}\label{task-framework}

The Task Framework is a Python class library that provides a structure
(standardized class entry points and conventions) to organize low-level
algorithms into potentially-reusable algorithmic components (Tasks; e.g.
dark frame subtraction, object detection, object measurement), and to
organize tasks into basic pipelines (SuperTasks; e.g., process a single
visit, build a coadd, difference a visit). The algorithmic code is
written into (Super)Tasks by overriding classes and providing
implementation for standard entry points. The Task Framework allows the
pipelines to be constructed and run at the level of a single node or a
group of tightly-synchronized nodes. It allows for sub-node
parallelization: trivial parallelization of Task execution, as well as
providing (in the future) parallelization primitives for development of
multi-core Tasks and synchronized multi-node Tasks.

The Task Framework serves as an interface layer between orchestration
and the algorithmic code. It exposes a standard interface to
``activators'' (command-line runners as well as the orchestration layer
and QA systems), which use it to execute the code wrapped in tasks. The
Task Framework does not concern itself with fault-tolerant massively
parallel execution of the pipelines over multiple (thousands) of nodes
nor any staging of data that might be required; this is the concern of
the orchestration middleware.

The Task Framework exposes to the orchestration system needs and
capabilities of the underlying algorithmic code (i.e., the number of
cores needed, expected memory-per-core, expected need for data). It may
also receive from the orchestration layer the information on how to
optimally run the particular task (i.e., which level of intra-node
parallelization is be desired).

It also includes a configuration API and a logging API.


\section{Infrastructure Services}\label{infrastructure-services}

The information technology and communications infrastructure is composed of services and systems that form the computing environments on top of which the services in the enclaves described above are deployed and operate.

\subsection{Service Descriptions}\label{infrastructure-service-descriptions}

Detailed concepts of operations for each service can be found in \textit{Concept of Operations for the LSST Production Services} \citedsp{LDM-230}.
The detailed design of the infrastructure is in \textit{LSST Data Facility Logical Information Technology and Communications Design} \citedsp{LDM-129} and \textit{Network Design} \citedsp{LSE-78}.

\subsubsection{Managed Database}\label{managed-database}

This service provides general-purpose relational database management that supports other services.
It includes metadata and provenance, but it does not include the large catalog science data products that are generated as files and loaded into the Qserv parallel distributed database.
For efficiency of resource usage and management, most databases are consolidated into a single RDBMS instance.

\subsubsection{Batch Processing}\label{batch-processing}

This service provides execution of batch jobs with a variety of priorities from a variety of users in a variety of environments (e.g. OS and software configurations) on the underlying provisioned compute resources.
It will use containerization to handle heterogeneity of environments.
HTCondor is the baseline technology choice for this service.

Some compute resources are reserved for particular uses, but others can
be flexibly provisioned, up to a certain maximum quota, if needed to
deal with surges in processing.

The priority order for processing is:
\begin{itemize}
\item
  Prompt processing
\item
  Offline processing for Prompt data products
\item
  OCS-controlled batch processing
\item
  LSP Commissioning Cluster processing
\item
  LSP Science Validation processing
\item
  LSP Data Access Center processing
\item
  Template and Calibration Products Production
\item
  Special Programs Productions
\item
  Data Release Production
\end{itemize}

\subsubsection{Containerized Application Management}\label{containerized-application-management}

This service provides compute, local-to-node storage, and local-to-LAN storage resources for deploying containerized service-oriented systems, especially the Science Platforms but also including parts of the Quality Control systems and Developer Services.
It allows allocation of compute and storage resources as well as reproducible, controlled deployment of services onto those resources.
Kubernetes is the baseline technology choice for this service.

\subsubsection{IT Security}\label{it-security}

This service provides security, including monitoring, vulnerability management, incident detection and response, access controls, intrusion detection, and configuration management.
Information security is provided in accordance with the \textit{LSST Master Infromation Security Policy} \citedsp{LPM-121}.

\subsubsection{Identity Management}\label{identity-management}

This service provides authentication and authorization for all users of
any DMS component, especially the LSST Science Platform instances.

\subsubsection{ITC Provisioning and Management}\label{itc-provisioning-management}

This service provides for acquisition, change and configuration management, and provisioning of information technology components, whether purchased ``bare metal'', provided by the NCSA Commons, or provided by agreements with outside providers.

This service includes deployment of non-containerized services.
For example, many of the services at the Base Facility are not highly dynamic or flexible, as they primarily provide interfacing to the OCS and Camera Data System.
The baseline provisioning mechanism for them is vSphere; they will be deployed using Puppet.

\subsubsection{Service Management/Monitoring}\label{service-management-monitoring}

These services provide management and monitoring at the service level for each computing environment.
They include standard IT processes such as service design, service transition (including change, release, and configuration management), and service delivery (including incident and request response plus problem management).

Service monitoring reports in dashboard and document form on the health and status of all services.

\subsection{Interfaces}\label{infrastructure-interfaces}

The infrastructure services generally interact with all other deployed
services.

Identity management instances are present in the Base Center and at NCSA.
(Another replica will be maintained at the Summit.)  These are used to support
authentication and authorization for the other physically co-located environments:
the Commissioning Cluster and the two Data Access Centers.


\section{NCSA Development and Integration Enclave}\label{ncsa-development-integration-enclave}

This enclave encompasses environments for analysts, developers, and
integration and test. Its users are the Observatory staff as they
analyze raw data and processed data products to characterize them,
develop new algorithms and systems, and test new versions of components
and services before deployment.

Integration environments in this enclave represent various deployment environments, deployment services, test datasets, test execution services, metric measurement and tracking services.
This environment includes the ``Level 1'' Test Stand, which includes a DM instance of the Camera DAQ used in an integration environment simulating the Base Center.
It also includes the Prototype Data Access Center (PDAC), which is an integration environment simulating a Data Access Center.

\subsection{Service Descriptions}\label{ncsa-dev-int-service-descriptions}

\subsubsection{LSST Science Platform Science Validation
instance}\label{lsst-science-platform-science-validation-instance}

This instance of the LSST Science Platform is customized to allow access
to unreleased and intermediate data products from the Alert, Calibration
Products, and Data Release Productions. It is optimized for usage by
scientists within the LSST Operations team, although selected external
scientists can be granted access to assist with Science Validation. Part
of the optimization is to size and configure the three Aspects of the
LSP appropriately; in particular, more JupyterLab usage and less portal
usage is expected.

\subsubsection{Developer Services}\label{developer-services}

These services include a software version control service, a build, unit test, and continuous integration service, a documentation publication service, developer communications services, an issue/ticket tracking service, etc., necessary to support science and system developers as they create and debug new versions of the Data Management System.

\subsection{Interfaces}\label{ncsa-development-integration-interfaces}

LSP Science Validation instance and Developer Services to Data Backbone:
All services in this enclave interface with the Data Backbone.  The LSP
Science Validation instance is used to inspect, analyze, and validate the data
products of the Data Release Production prior to their release and so has
access to those products in the Data Backbone; since it may be used to annotate
the data products, it can also write to the Data Backbone.  The Developer
Services may use raw data, intermediate data products, and final
data products to do development, perform tests, and debug problems.

Developer Services do not have direct interfaces with the rest of the
operational system; they communicate via the distributed source version control
system, the package management system, and the configuration system.



\input{standards}
\input{trace}

\input{refs}
\end{document}