From ea00d8280dc70c50f3b00b767e297a2d68e7dd05 Mon Sep 17 00:00:00 2001 From: Dan Gottlieb Date: Thu, 21 Jan 2021 11:35:19 -0500 Subject: [PATCH 1/7] rough publish operation --- FHIR-bulkdata.xml | 1 + input/bulk-data.xml | 5 + input/includes/menu.xml | 1 + input/pagecontent/bulk-publish.md | 238 ++++++++++++++++++++++++++++++ 4 files changed, 245 insertions(+) create mode 100644 input/pagecontent/bulk-publish.md diff --git a/FHIR-bulkdata.xml b/FHIR-bulkdata.xml index f67db0d..291822a 100644 --- a/FHIR-bulkdata.xml +++ b/FHIR-bulkdata.xml @@ -21,6 +21,7 @@ + diff --git a/input/bulk-data.xml b/input/bulk-data.xml index adca380..8d5535e 100644 --- a/input/bulk-data.xml +++ b/input/bulk-data.xml @@ -90,6 +90,11 @@ <generation value="html"/> </page> + <page> + <nameUrl value="bulk-publish.html"/> + <title value="Bulk Data Publish"/> + <generation value="markdown"/> + </page> <!-- <page> <nameUrl value="operations.html"/> <title value="Bulk Data Export Operations"/> diff --git a/input/includes/menu.xml b/input/includes/menu.xml index 6a22504..ee8abe2 100644 --- a/input/includes/menu.xml +++ b/input/includes/menu.xml @@ -2,6 +2,7 @@ <li><a href="index.html">Home</a></li> <li><a href="export.html">Export</a></li> <li><a href="authorization.html">Authorization</a></li> + <li><a href="bulk-publish.html">Publish</a></li> <li><a href="artifacts.html">FHIR Artifacts</a></li> <li><a href="downloads.html">Downloads</a></li> <li><a href="abbreviations.html">Abbreviations</a></li> diff --git a/input/pagecontent/bulk-publish.md b/input/pagecontent/bulk-publish.md new file mode 100644 index 0000000..ec5184b --- /dev/null +++ b/input/pagecontent/bulk-publish.md @@ -0,0 +1,238 @@ +### Audience and Scope + +This implementation guide is intended to be used by developers of backend services (clients) and data providers (servers) that aim to interoperate by sharing large FHIR datasets. The guide defines the application programming interfaces (APIs) through which a client may retrieve pre-generated FHIR bulk-data files from a server. These files may be provided at an open endpoint, or may require the client to authenticate and authorize access to retrieve the data. + +In contrast to the FHIR bulk data export operation ($export), the publish operation ($bulk-publish) returns a static set of pre-generated bulk data files and does not provide a mechanism for a client to retrieve a filtered subset of the available data. Expected use cases include publication of provider directories by payor organizations and healthcare organizations, and publication of formulary information by payors. + +### Underlying Standards + +* [HL7 FHIR](https://www.hl7.org/fhir/) +* [Newline-delimited JSON](http://ndjson.org) +* [RFC5246, Transport Layer Security (TLS) Protocol Version 1.2](https://tools.ietf.org/html/rfc5246) +* [RFC6749, The OAuth 2.0 Authorization Framework](https://tools.ietf.org/html/rfc6749) +* [RFC6750, The OAuth 2.0 Authorization Framework: Bearer Token Usage](https://tools.ietf.org/html/rfc6750) +* [RFC7159, The JavaScript Object Notation (JSON) Data Interchange Format](https://tools.ietf.org/html/rfc7159) + +### Terminology + +This profile inherits terminology from the standards referenced above. +The key words "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this specification are to be interpreted as described in RFC2119. + +### Security Considerations + +All exchanges described herein between a client and a server SHALL be secured using [Transport Layer Security (TLS) Protocol Version 1.2 (RFC5246)](https://tools.ietf.org/html/rfc5246) or a more recent version of TLS. Use of mutual TLS is OPTIONAL. With each of the requests described herein, implementers MAY implement OAuth 2.0 access management in accordance with the [SMART Backend Services: Authorization Guide](authorization.html). + +### Request Flow + +#### Manifest Request + +Binary Resources MAY be serialized as DocumentReference Resources with the content.attachment element populated as described in the [Attachments](#attachments) section below. + +References in the resources returned MAY be relative URLs with the format <code><resource type>/<id></code>, or absolute URLs with the same structure rooted in the base URL for the server from which the export was performed. References will be resolved by looking for a resource with the specified type and id within the file set. + +##### Endpoint - Data relating to a group of patients + +`[fhir base]/Group/[id]/$bulk-publish` + +FHIR Operation to obtain a set of FHIR resources of diverse resource types pertaining to to the set of patients in the specified [Group](https://www.hl7.org/fhir/group.html). + +If a FHIR server supports Group-level data export, it SHOULD support reading and searching for `Group` resource. This enables clients to discover available groups based on stable characteristics such as `Group.identifier`. + +Note: How these Groups are defined is specific to each FHIR system's implementation. Group membership could be based upon explicit attributes of the patient, such as age, sex or a particular condition such as PTSD or Chronic Opioid use, or on more complex attributes, such as a recent inpatient discharge or membership in the population used to calculate a quality measure. FHIR-based group management is out of scope for the current version of this implementation guide. + +##### Endpoint - Data relating to all patients + +`[fhir base]/$bulk-publish` + +Publish fully static or periodically updated data in FHIR format. + +##### Response - Error + +- HTTP status code of ```4XX``` or ```5XX``` +- `Content-Type` header of `application/fhir+json` +- The body of the response SHOULD be a FHIR `OperationOutcome` resource in JSON format. If this is not possible (for example, the infrastructure layer returning the error is not FHIR aware), the server MAY return an error message in another format and include a corresponding value for the `Content-Type` header. + + +##### Response - Manifest + +- HTTP status of ```200 OK``` +- ```Content-Type``` header of ```application/json``` +- The server MAY return an ```Expires``` header indicating when the files listed will no longer be available for access. +- A body containing a JSON object providing metadata, and links to the generated bulk data files. The files SHALL be accessible to the client at the URLs advertised. These URLs MAY be served by file servers other than a FHIR-specific server. + +Required Fields: + +<table class="table"> + <thead> + <th>Field</th> + <th>Optionality</th> + <th>Type</th> + <th>Description</th> + </thead> + <tbody> + <tr> + <td><code>transactionTime</code></td> + <td><span class="label label-success">required</span></td> + <td>FHIR instant</td> + <td>Indicates the server's time when the query is run. The response SHOULD NOT include any resources modified after this instant, and SHALL include any matching resources modified up to and including this instant. + <br/> + <br/> + Note: To properly meet these constraints, a FHIR Server might need to wait for any pending transactions to resolve in its database before generating the export files. + </td> + </tr> + <tr> + <td><code>request</code></td> + <td><span class="label label-success">required</span></td> + <td>String</td> + <td>The full URL of the original manifest request.</td> + </tr> + <tr> + <td><code>requiresAccessToken</code></td> + <td><span class="label label-success">required</span></td> + <td>Boolean</td> + <td>Indicates whether downloading the generated files requires a bearer access token + <br/> + <br/> + Value SHALL be <code>true</code> if both the file server and the FHIR API server control access using OAuth 2.0 bearer tokens. Value MAY be <code>false</code> for file servers that use access-control schemes other than OAuth 2.0, such as downloads from Amazon S3 bucket URLs or verifiable file servers within an organization's firewall. + </td> + </tr> + <tr> + <td><code>output</code></td> + <td><span class="label label-success">required</span></td> + <td>Array</td> + <td>An array of file items with one entry for each generated file. If no resources are returned from the kick-off request, the server SHOULD return an empty array. + <br/> + <br/> + Each file item SHALL contain the following fields: + <br/> + <br/> + - <code>type</code> - the FHIR resource type that is contained in the file. + <br/> + <br/> + Each file SHALL contain resources of only one type, but a server MAY create more than one file for each resource type returned. The number of resources contained in a file MAY vary between servers. If no data are found for a resource, the server SHOULD NOT return an output item for that resource in the response. These rules apply only to top-level resources within the response; as always in FHIR, any resource MAY have a "contained" array that includes referenced resources of other types. + <br/> + <br/> + - <code>url</code> - the path to the file. + <br/> + <br/> + - <code>extension.format</code> - MIME type of the output file. All data must be made available as `application/fhir+ndjson`, but servers MAY also provide the data in other formats. Servers SHOULD include this extension for files in ndjson format, but SHALL include it for files in other formats. If more than one format is supported, clients should filter the file items in the output array and only retrieve those in the desired format. The number of files may differ by format (eg. each ndjson format file may contain fewer resources than a binary file format of the same size, resulting in fewer entries in the output array). + <br/> + <br/> + Each file item MAY optionally contain the following field: + <br/> + <br/> + - <code>count</code> - the number of resources in the file, represented as a JSON number. + </td> + </tr> + + <tr> + <td><code>error</code></td> + <td><span class="label label-success">required</span></td> + <td>Array</td> + <td>Array of message file items following the same structure as the <code>output</code> array. + <br/> + <br/> + Error, warning, and information messages related to the export should be included here (not in output). If there are no relevant messages, the server SHOULD return an empty array. Only the <code>OperationOutcome</code> resource type is currently supported, so a server SHALL generate files in the same format as bulk data output files that contain <code>OperationOutcome</code> resources. + <br/><br/>Note: this field may be renamed in a future version of this IG to reflect the inclusion of <code>OperationOutcome</code> resources with severity levels other than error. + </td> + </tr> + <tr> + <td><code>extension</code></td> + <td><span class="label label-info">optional</span></td> + <td>JSON Object</td> + <td>To support extensions, this implementation guide reserves the name <code>extension</code> and will never define a field with that name, allowing server implementations to use it to provide custom behavior and information. For example, a server may choose to provide a custom extension that contains a decryption key for encrypted ndjson files. The value of an extension element SHALL be a pre-coordinated JSON object. + <br/> + <br/> + Note: In addition to extensions being supported on the root object level, extensions may also be included within the fields above (e.g., in the 'output' object). + </td> + </tr> + </tbody> +</table> + +Example response body: + +```json + { + "transactionTime": "[instant]", + "request" : "[base]/Patient/$export?_type=Patient,Observation", + "requiresAccessToken" : true, + "output" : [{ + "type" : "Patient", + "url" : "http://serverpath2/patient_file_1.ndjson", + "extension" : { + "format" : "application/fhir+ndjson" + } + },{ + "type" : "Patient", + "url" : "http://serverpath2/patient_file_2.ndjson", + "extension" : { + "format" : "application/fhir+ndjson" + } + },{ + "type" : "Observation", + "url" : "http://serverpath2/observation_file_1.ndjson", + "extension" : { + "format" : "application/fhir+ndjson" + } + }], + "error" : [{ + "type" : "OperationOutcome", + "url" : "http://serverpath2/err_file_1.ndjson", + "extension" : { + "format" : "application/fhir+ndjson" + } + }] + } +``` + +--- +#### Output File Request + +Using the URLs supplied by the FHIR server in the Complete Status response body, a client MAY download the generated bulk data files (one or more per resource type) within the time period specified in the ```Expires``` header (if present). If the ```requiresAccessToken``` field in the Complete Status body is set to ```true```, the request SHALL include a valid access token. See the Security Considerations section above. + +The exported data SHALL include only the most recent version of any exported resources unless the client explicitly requests different behavior in a fashion supported by the server (e.g. via a new query parameter yet to be defined). Inclusion of the .meta information is at the discretion of the server (as it is for all FHIR interactions). + +Example NDJSON output file: +``` +{"id":"5c41cecf-cf81-434f-9da7-e24e5a99dbc2","name":[{"given":["Brenda"],"family":["Jackson"]}],"gender":"female","birthDate":"1956-10-14T00:00:00.000Z","resourceType":"Patient"} +{"id":"3fabcb98-0995-447d-a03f-314d202b32f4","name":[{"given":["Bram"],"family":["Sandeep"]}],"gender":"male","birthDate":"1994-11-01T00:00:00.000Z","resourceType":"Patient"} +{"id":"945e5c7f-504b-43bd-9562-a2ef82c244b2","name":[{"given":["Sandy"],"family":["Hamlin"]}],"gender":"female","birthDate":"1988-01-24T00:00:00.000Z","resourceType":"Patient"} + +``` + +##### Endpoint + +`GET [url from status request output field]` + +##### Headers + +- ```Accept``` (optional, defaults to ```application/fhir+ndjson```) + +Specifies the format of the file being requested. + +##### Response - Success + +- HTTP status of ```200 OK``` +- ```Content-Type``` header that matches the file format being delivered. For files in ndjson format, SHALL be ```application/fhir+ndjson``` +- Body of FHIR resources in newline delimited json - [ndjson](http://ndjson.org/) or other requested format + +##### Response - Error + +- HTTP Status Code of ```4XX``` or ```5XX``` + +##### Attachments + +If resources in an output file contain elements of the type ```Attachment```, servers SHALL populate the ```Attachment.contentType``` code as well as either the ```data``` element or the ```url``` element. The ```url``` element SHALL be an absolute url that can be de-referenced to the attachment's content. + +When the ```url``` element is populated with an absolute URL and the ```requiresAccessToken``` field in the Complete Status body is set to ```true```, the url location must be accessible by a client with a valid access token, and SHALL NOT require the use of additional authentication credentials. When the ```url``` element is populated and the ```requiresAccessToken``` field in the Complete Status body is set to ```false```, the url location must be accessible by a client without an access token. + +Note that if a server copies files to the bulk data output endpoint or proxies requests to facilitate access from this endpoint, it may need to modify the ```Attachment.url``` element when generating the FHIR bulk data output files. + +### Server Capability Documentation + +This implementation guide is structured to support a wide variety of bulk data export use cases and server architectures. To provide clarity to developers on which capabilities are implemented in a particular server, server providers should ensure their documentation addresses the topics below. Future versions of this IG may define a computable format for this information as well. + +- Does the server restrict responses to a specific "profile" like US Core, USCDI, or Blue Button? +- What approach does the server take to divide datasets into multiple files (eg. single file per the resource type, limit file size to 100MB, limit number of resources per file to 100,000)? +- Does the server support system-wide or Group-level export? +- What file formats does this server return? \ No newline at end of file From 2c6620a6c0bfdb5cded28da0e7ca7f2f78ad2ddc Mon Sep 17 00:00:00 2001 From: Dan Gottlieb <dan@gotdan.org> Date: Fri, 22 Jan 2021 10:03:49 -0500 Subject: [PATCH 2/7] update example and intro --- input/pagecontent/bulk-publish.md | 37 +++++++++++++++++-------------- 1 file changed, 20 insertions(+), 17 deletions(-) diff --git a/input/pagecontent/bulk-publish.md b/input/pagecontent/bulk-publish.md index ec5184b..d403ec0 100644 --- a/input/pagecontent/bulk-publish.md +++ b/input/pagecontent/bulk-publish.md @@ -2,7 +2,10 @@ This implementation guide is intended to be used by developers of backend services (clients) and data providers (servers) that aim to interoperate by sharing large FHIR datasets. The guide defines the application programming interfaces (APIs) through which a client may retrieve pre-generated FHIR bulk-data files from a server. These files may be provided at an open endpoint, or may require the client to authenticate and authorize access to retrieve the data. -In contrast to the FHIR bulk data export operation ($export), the publish operation ($bulk-publish) returns a static set of pre-generated bulk data files and does not provide a mechanism for a client to retrieve a filtered subset of the available data. Expected use cases include publication of provider directories by payor organizations and healthcare organizations, and publication of formulary information by payors. +In contrast to the FHIR bulk data [export operation ($export)](./export.html), the publish operation ($bulk-publish) returns a static manifest and set of pre-generated bulk data files and does not provide a mechanism for a client to retrieve a filtered subset of the available data. Systems that dynamically manage information about individual patients should use the export operation and not the bulk-publish operation. Systems that return infrequently updated reference information may wish to use the bulk-publish operation instead of the export operation to reduce the complexity and cost involved in hosting and providing this information. + +Expected use cases include publication of provider directories by payor organizations and healthcare organizations, and publication of formulary information by payors. + ### Underlying Standards @@ -38,7 +41,7 @@ FHIR Operation to obtain a set of FHIR resources of diverse resource types perta If a FHIR server supports Group-level data export, it SHOULD support reading and searching for `Group` resource. This enables clients to discover available groups based on stable characteristics such as `Group.identifier`. -Note: How these Groups are defined is specific to each FHIR system's implementation. Group membership could be based upon explicit attributes of the patient, such as age, sex or a particular condition such as PTSD or Chronic Opioid use, or on more complex attributes, such as a recent inpatient discharge or membership in the population used to calculate a quality measure. FHIR-based group management is out of scope for the current version of this implementation guide. +Note: How these Groups are defined is specific to each FHIR system's implementation. For example, Group membership could be based upon explicit attributes of the dataset, such as provider participation in a provider network. FHIR-based group management is out of scope for the current version of this implementation guide. ##### Endpoint - Data relating to all patients @@ -153,35 +156,35 @@ Example response body: ```json { - "transactionTime": "[instant]", - "request" : "[base]/Patient/$export?_type=Patient,Observation", + "transactionTime": "2021-01-01T00:00:00Z", + "request" : "http://example.com/pd/Group/network-1/$bulk-publish", "requiresAccessToken" : true, "output" : [{ - "type" : "Patient", - "url" : "http://serverpath2/patient_file_1.ndjson", + "type" : "Practitioner", + "url" : "http://example.com/pd/Group/network-1/practitioner_file_1.ndjson", "extension" : { "format" : "application/fhir+ndjson" } },{ - "type" : "Patient", - "url" : "http://serverpath2/patient_file_2.ndjson", + "type" : "Practitioner", + "url" : "http://example.com/pd/Group/network-1/practitioner_file_2.ndjson", "extension" : { "format" : "application/fhir+ndjson" } },{ - "type" : "Observation", - "url" : "http://serverpath2/observation_file_1.ndjson", + "type" : "Organization", + "url" : "http://example.com/pd/Group/network-1/organization_file_1.ndjson", + "extension" : { + "format" : "application/fhir+ndjson" + } + },{ + "type" : "Location", + "url" : "http://example.com/pd/Group/network-1/location_file_1.ndjson", "extension" : { "format" : "application/fhir+ndjson" } }], - "error" : [{ - "type" : "OperationOutcome", - "url" : "http://serverpath2/err_file_1.ndjson", - "extension" : { - "format" : "application/fhir+ndjson" - } - }] + "error" : [] } ``` From d19e60ede11a774eb460b1ff4b32a0d2fc1f0ed3 Mon Sep 17 00:00:00 2001 From: Dan Gottlieb <dan@gotdan.org> Date: Fri, 22 Jan 2021 10:49:06 -0500 Subject: [PATCH 3/7] tweak group language --- input/pagecontent/bulk-publish.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/input/pagecontent/bulk-publish.md b/input/pagecontent/bulk-publish.md index d403ec0..65997af 100644 --- a/input/pagecontent/bulk-publish.md +++ b/input/pagecontent/bulk-publish.md @@ -39,7 +39,7 @@ References in the resources returned MAY be relative URLs with the format <code> FHIR Operation to obtain a set of FHIR resources of diverse resource types pertaining to to the set of patients in the specified [Group](https://www.hl7.org/fhir/group.html). -If a FHIR server supports Group-level data export, it SHOULD support reading and searching for `Group` resource. This enables clients to discover available groups based on stable characteristics such as `Group.identifier`. +If a FHIR server supports Group-level data export, it SHOULD support listing the available `Group` resources. This enables clients to discover groups based on stable characteristics such as `Group.identifier`. Note: How these Groups are defined is specific to each FHIR system's implementation. For example, Group membership could be based upon explicit attributes of the dataset, such as provider participation in a provider network. FHIR-based group management is out of scope for the current version of this implementation guide. From 41074ffb8548512ba6ab3c8f2b6962e337a2c674 Mon Sep 17 00:00:00 2001 From: Dan Gottlieb <dan@gotdan.org> Date: Tue, 26 Jan 2021 17:22:37 -0500 Subject: [PATCH 4/7] remove groups --- input/pagecontent/bulk-publish.md | 34 ++++++++++--------------------- 1 file changed, 11 insertions(+), 23 deletions(-) diff --git a/input/pagecontent/bulk-publish.md b/input/pagecontent/bulk-publish.md index 65997af..5d51674 100644 --- a/input/pagecontent/bulk-publish.md +++ b/input/pagecontent/bulk-publish.md @@ -29,25 +29,10 @@ All exchanges described herein between a client and a server SHALL be secured us #### Manifest Request -Binary Resources MAY be serialized as DocumentReference Resources with the content.attachment element populated as described in the [Attachments](#attachments) section below. - -References in the resources returned MAY be relative URLs with the format <code><resource type>/<id></code>, or absolute URLs with the same structure rooted in the base URL for the server from which the export was performed. References will be resolved by looking for a resource with the specified type and id within the file set. - -##### Endpoint - Data relating to a group of patients - -`[fhir base]/Group/[id]/$bulk-publish` - -FHIR Operation to obtain a set of FHIR resources of diverse resource types pertaining to to the set of patients in the specified [Group](https://www.hl7.org/fhir/group.html). - -If a FHIR server supports Group-level data export, it SHOULD support listing the available `Group` resources. This enables clients to discover groups based on stable characteristics such as `Group.identifier`. - -Note: How these Groups are defined is specific to each FHIR system's implementation. For example, Group membership could be based upon explicit attributes of the dataset, such as provider participation in a provider network. FHIR-based group management is out of scope for the current version of this implementation guide. - -##### Endpoint - Data relating to all patients +Request for fully static or periodically updated dataset in FHIR format. `[fhir base]/$bulk-publish` -Publish fully static or periodically updated data in FHIR format. ##### Response - Error @@ -96,7 +81,7 @@ Required Fields: <td>Indicates whether downloading the generated files requires a bearer access token <br/> <br/> - Value SHALL be <code>true</code> if both the file server and the FHIR API server control access using OAuth 2.0 bearer tokens. Value MAY be <code>false</code> for file servers that use access-control schemes other than OAuth 2.0, such as downloads from Amazon S3 bucket URLs or verifiable file servers within an organization's firewall. + Value SHALL be <code>true</code> if both the file server and the FHIR API server control access using OAuth 2.0 bearer tokens. </td> </tr> <tr> @@ -127,7 +112,6 @@ Required Fields: - <code>count</code> - the number of resources in the file, represented as a JSON number. </td> </tr> - <tr> <td><code>error</code></td> <td><span class="label label-success">required</span></td> @@ -157,29 +141,29 @@ Example response body: ```json { "transactionTime": "2021-01-01T00:00:00Z", - "request" : "http://example.com/pd/Group/network-1/$bulk-publish", + "request" : "http://example.com/pd/$bulk-publish", "requiresAccessToken" : true, "output" : [{ "type" : "Practitioner", - "url" : "http://example.com/pd/Group/network-1/practitioner_file_1.ndjson", + "url" : "http://example.com/pd/practitioner_file_1.ndjson", "extension" : { "format" : "application/fhir+ndjson" } },{ "type" : "Practitioner", - "url" : "http://example.com/pd/Group/network-1/practitioner_file_2.ndjson", + "url" : "http://example.com/pd/practitioner_file_2.ndjson", "extension" : { "format" : "application/fhir+ndjson" } },{ "type" : "Organization", - "url" : "http://example.com/pd/Group/network-1/organization_file_1.ndjson", + "url" : "http://example.com/pd/organization_file_1.ndjson", "extension" : { "format" : "application/fhir+ndjson" } },{ "type" : "Location", - "url" : "http://example.com/pd/Group/network-1/location_file_1.ndjson", + "url" : "http://example.com/pd/location_file_1.ndjson", "extension" : { "format" : "application/fhir+ndjson" } @@ -195,6 +179,10 @@ Using the URLs supplied by the FHIR server in the Complete Status response body, The exported data SHALL include only the most recent version of any exported resources unless the client explicitly requests different behavior in a fashion supported by the server (e.g. via a new query parameter yet to be defined). Inclusion of the .meta information is at the discretion of the server (as it is for all FHIR interactions). +Binary Resources MAY be serialized as DocumentReference Resources with the content.attachment element populated as described in the [Attachments](#attachments) section below. + +References in the resources returned MAY be relative URLs with the format <code><resource type>/<id></code>, or absolute URLs with the same structure rooted in the base URL for the server from which the export was performed. References will be resolved by looking for a resource with the specified type and id within the file set. + Example NDJSON output file: ``` {"id":"5c41cecf-cf81-434f-9da7-e24e5a99dbc2","name":[{"given":["Brenda"],"family":["Jackson"]}],"gender":"female","birthDate":"1956-10-14T00:00:00.000Z","resourceType":"Patient"} From 05409235a3ec5fbaaf5f26548b93d4dcce83b82d Mon Sep 17 00:00:00 2001 From: Dan Gottlieb <dan@gotdan.org> Date: Tue, 23 Feb 2021 14:43:57 -0500 Subject: [PATCH 5/7] _since param, clarify transactionTime, ssl in example --- input/pagecontent/bulk-publish.md | 16 +++++++++------- 1 file changed, 9 insertions(+), 7 deletions(-) diff --git a/input/pagecontent/bulk-publish.md b/input/pagecontent/bulk-publish.md index 5d51674..e71721a 100644 --- a/input/pagecontent/bulk-publish.md +++ b/input/pagecontent/bulk-publish.md @@ -31,7 +31,9 @@ All exchanges described herein between a client and a server SHALL be secured us Request for fully static or periodically updated dataset in FHIR format. -`[fhir base]/$bulk-publish` +GET `[fhir base]/$bulk-publish` + +- A client MAY include an `_since` parameter with a value that is a [FHIR instant](https://www.hl7.org/fhir/datatypes.html#instant). If this parameter is provided, the server MAY restrict the FHIR resources included in the response to only those that have been created or modified after the supplied time (e.g., `Resource.meta.lastUpdated` is later). ##### Response - Error @@ -62,7 +64,7 @@ Required Fields: <td><code>transactionTime</code></td> <td><span class="label label-success">required</span></td> <td>FHIR instant</td> - <td>Indicates the server's time when the query is run. The response SHOULD NOT include any resources modified after this instant, and SHALL include any matching resources modified up to and including this instant. + <td>Indicates the server's time when the bulk data files included in the manifest were generated. The response SHOULD NOT include any resources modified after this instant, and SHALL include any matching resources modified up to and including this instant. <br/> <br/> Note: To properly meet these constraints, a FHIR Server might need to wait for any pending transactions to resolve in its database before generating the export files. @@ -141,29 +143,29 @@ Example response body: ```json { "transactionTime": "2021-01-01T00:00:00Z", - "request" : "http://example.com/pd/$bulk-publish", + "request" : "https://example.com/pd/$bulk-publish", "requiresAccessToken" : true, "output" : [{ "type" : "Practitioner", - "url" : "http://example.com/pd/practitioner_file_1.ndjson", + "url" : "https://example.com/pd/practitioner_file_1.ndjson", "extension" : { "format" : "application/fhir+ndjson" } },{ "type" : "Practitioner", - "url" : "http://example.com/pd/practitioner_file_2.ndjson", + "url" : "https://example.com/pd/practitioner_file_2.ndjson", "extension" : { "format" : "application/fhir+ndjson" } },{ "type" : "Organization", - "url" : "http://example.com/pd/organization_file_1.ndjson", + "url" : "https://example.com/pd/organization_file_1.ndjson", "extension" : { "format" : "application/fhir+ndjson" } },{ "type" : "Location", - "url" : "http://example.com/pd/location_file_1.ndjson", + "url" : "https://example.com/pd/location_file_1.ndjson", "extension" : { "format" : "application/fhir+ndjson" } From 094e080178e7513f537fa6dbc555e4e437010f53 Mon Sep 17 00:00:00 2001 From: Dan Gottlieb <gotdan@users.noreply.github.com> Date: Tue, 28 Nov 2023 09:31:02 -0500 Subject: [PATCH 6/7] Clean up bulk-publish and align language --- input/pagecontent/bulk-publish.md | 91 +++++++++++++++---------------- 1 file changed, 45 insertions(+), 46 deletions(-) diff --git a/input/pagecontent/bulk-publish.md b/input/pagecontent/bulk-publish.md index e71721a..dbdfbb6 100644 --- a/input/pagecontent/bulk-publish.md +++ b/input/pagecontent/bulk-publish.md @@ -19,11 +19,11 @@ Expected use cases include publication of provider directories by payor organiza ### Terminology This profile inherits terminology from the standards referenced above. -The key words "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this specification are to be interpreted as described in RFC2119. +The key words "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this specification are to be interpreted as described in [RFC2119](https://tools.ietf.org/html/rfc2119). ### Security Considerations -All exchanges described herein between a client and a server SHALL be secured using [Transport Layer Security (TLS) Protocol Version 1.2 (RFC5246)](https://tools.ietf.org/html/rfc5246) or a more recent version of TLS. Use of mutual TLS is OPTIONAL. With each of the requests described herein, implementers MAY implement OAuth 2.0 access management in accordance with the [SMART Backend Services: Authorization Guide](authorization.html). +All exchanges described herein between a client and a server SHALL be secured using [Transport Layer Security (TLS) Protocol Version 1.2 (RFC5246)](https://tools.ietf.org/html/rfc5246) or a more recent version of TLS. Use of mutual TLS is OPTIONAL. With each of the requests described herein, implementers MAY implement OAuth 2.0 access management in accordance with the [SMART Backend Services Authorization Profile](authorization.html). ### Request Flow @@ -38,16 +38,16 @@ GET `[fhir base]/$bulk-publish` ##### Response - Error -- HTTP status code of ```4XX``` or ```5XX``` +- HTTP status code of `4XX` or `5XX` - `Content-Type` header of `application/fhir+json` - The body of the response SHOULD be a FHIR `OperationOutcome` resource in JSON format. If this is not possible (for example, the infrastructure layer returning the error is not FHIR aware), the server MAY return an error message in another format and include a corresponding value for the `Content-Type` header. ##### Response - Manifest -- HTTP status of ```200 OK``` -- ```Content-Type``` header of ```application/json``` -- The server MAY return an ```Expires``` header indicating when the files listed will no longer be available for access. +- HTTP status of `200 OK` +- `Content-Type` header of `application/json` +- The server MAY return an `Expires` header indicating when the files listed will no longer be available for access. - A body containing a JSON object providing metadata, and links to the generated bulk data files. The files SHALL be accessible to the client at the URLs advertised. These URLs MAY be served by file servers other than a FHIR-specific server. Required Fields: @@ -83,14 +83,14 @@ Required Fields: <td>Indicates whether downloading the generated files requires a bearer access token <br/> <br/> - Value SHALL be <code>true</code> if both the file server and the FHIR API server control access using OAuth 2.0 bearer tokens. + Value SHALL be <code>true</code> if both the file server and the FHIR API server control access using OAuth 2.0 bearer tokens. Value MAY be <code>false</code> for file servers that use access-control schemes other than OAuth 2.0, such as downloads from Amazon S3 bucket URLs or verifiable file servers within an organization's firewall. </td> </tr> <tr> <td><code>output</code></td> <td><span class="label label-success">required</span></td> - <td>Array</td> - <td>An array of file items with one entry for each generated file. If no resources are returned from the kick-off request, the server SHOULD return an empty array. + <td>JSON array</td> + <td>An array of file items with one entry for each generated file. If no resources are returned from the manifest request, the server SHOULD return an empty array. <br/> <br/> Each file item SHALL contain the following fields: @@ -121,18 +121,18 @@ Required Fields: <td>Array of message file items following the same structure as the <code>output</code> array. <br/> <br/> - Error, warning, and information messages related to the export should be included here (not in output). If there are no relevant messages, the server SHOULD return an empty array. Only the <code>OperationOutcome</code> resource type is currently supported, so a server SHALL generate files in the same format as bulk data output files that contain <code>OperationOutcome</code> resources. - <br/><br/>Note: this field may be renamed in a future version of this IG to reflect the inclusion of <code>OperationOutcome</code> resources with severity levels other than error. + Error, warning, and information messages related to the export SHOULD be included here (not in output). If there are no relevant messages, the server SHOULD return an empty array. Only the FHIR <code>OperationOutcome</code> resource type is currently supported, so the server SHALL generate files in the same format(s) as bulk data output files that contain FHIR <code>OperationOutcome</code> resources. + <br/><br/>Note: this field may be renamed in a future version of this IG to reflect the inclusion of <code>OperationOutcome</code> resources with severity levels other than <code>error</code>. </td> </tr> <tr> <td><code>extension</code></td> <td><span class="label label-info">optional</span></td> - <td>JSON Object</td> + <td>JSON object</td> <td>To support extensions, this implementation guide reserves the name <code>extension</code> and will never define a field with that name, allowing server implementations to use it to provide custom behavior and information. For example, a server may choose to provide a custom extension that contains a decryption key for encrypted ndjson files. The value of an extension element SHALL be a pre-coordinated JSON object. <br/> <br/> - Note: In addition to extensions being supported on the root object level, extensions may also be included within the fields above (e.g., in the 'output' object). + Note: In addition to extensions being supported on the root object level, extensions may also be included within the fields above. For example, a server could include an extension for each object in the 'output' array to provide details that help clients fetch only the files containing data associated with a specific geographic location. </td> </tr> </tbody> @@ -146,29 +146,29 @@ Example response body: "request" : "https://example.com/pd/$bulk-publish", "requiresAccessToken" : true, "output" : [{ - "type" : "Practitioner", - "url" : "https://example.com/pd/practitioner_file_1.ndjson", - "extension" : { - "format" : "application/fhir+ndjson" - } + "type" : "Practitioner", + "url" : "https://example.com/pd/practitioner_file_1.ndjson", + "extension" : { + "format" : "application/fhir+ndjson" + } },{ "type" : "Practitioner", - "url" : "https://example.com/pd/practitioner_file_2.ndjson", - "extension" : { + "url" : "https://example.com/pd/practitioner_file_2.ndjson", + "extension" : { "format" : "application/fhir+ndjson" - } + } },{ "type" : "Organization", - "url" : "https://example.com/pd/organization_file_1.ndjson", - "extension" : { - "format" : "application/fhir+ndjson" - } - },{ + "url" : "https://example.com/pd/organization_file_1.ndjson", + "extension" : { + "format" : "application/fhir+ndjson" + } + },{ "type" : "Location", - "url" : "https://example.com/pd/location_file_1.ndjson", - "extension" : { - "format" : "application/fhir+ndjson" - } + "url" : "https://example.com/pd/location_file_1.ndjson", + "extension" : { + "format" : "application/fhir+ndjson" + } }], "error" : [] } @@ -177,11 +177,11 @@ Example response body: --- #### Output File Request -Using the URLs supplied by the FHIR server in the Complete Status response body, a client MAY download the generated bulk data files (one or more per resource type) within the time period specified in the ```Expires``` header (if present). If the ```requiresAccessToken``` field in the Complete Status body is set to ```true```, the request SHALL include a valid access token. See the Security Considerations section above. +Using the URLs supplied by the FHIR server in the manifest, a client MAY download the generated bulk data files (one or more per resource type) within the time period specified in the `Expires` header (if present). If the `requiresAccessToken` field in the manifest is set to `true`, the request SHALL include a valid access token. See the [Security Considerations](#security-considerations) section above. -The exported data SHALL include only the most recent version of any exported resources unless the client explicitly requests different behavior in a fashion supported by the server (e.g. via a new query parameter yet to be defined). Inclusion of the .meta information is at the discretion of the server (as it is for all FHIR interactions). +The exported data SHALL include only the most recent version of any exported resources unless the client explicitly requests different behavior in a fashion supported by the server (e.g. via a new query parameter yet to be defined). Inclusion of the `.meta` information is at the discretion of the server (as it is for all FHIR interactions). -Binary Resources MAY be serialized as DocumentReference Resources with the content.attachment element populated as described in the [Attachments](#attachments) section below. +Binary Resources MAY be serialized as `DocumentReference` resources with the `content.attachment` element populated as described in the [Attachments](#attachments) section below. References in the resources returned MAY be relative URLs with the format <code><resource type>/<id></code>, or absolute URLs with the same structure rooted in the base URL for the server from which the export was performed. References will be resolved by looking for a resource with the specified type and id within the file set. @@ -190,42 +190,41 @@ Example NDJSON output file: {"id":"5c41cecf-cf81-434f-9da7-e24e5a99dbc2","name":[{"given":["Brenda"],"family":["Jackson"]}],"gender":"female","birthDate":"1956-10-14T00:00:00.000Z","resourceType":"Patient"} {"id":"3fabcb98-0995-447d-a03f-314d202b32f4","name":[{"given":["Bram"],"family":["Sandeep"]}],"gender":"male","birthDate":"1994-11-01T00:00:00.000Z","resourceType":"Patient"} {"id":"945e5c7f-504b-43bd-9562-a2ef82c244b2","name":[{"given":["Sandy"],"family":["Hamlin"]}],"gender":"female","birthDate":"1988-01-24T00:00:00.000Z","resourceType":"Patient"} - ``` ##### Endpoint -`GET [url from status request output field]` +`GET [url from manifest output object]` ##### Headers -- ```Accept``` (optional, defaults to ```application/fhir+ndjson```) +- `Accept` (optional, defaults to `application/fhir+ndjson`) Specifies the format of the file being requested. ##### Response - Success -- HTTP status of ```200 OK``` -- ```Content-Type``` header that matches the file format being delivered. For files in ndjson format, SHALL be ```application/fhir+ndjson``` +- HTTP status of `200 OK` +- `Content-Type` header that matches the file format being delivered. For files in ndjson format, SHALL be `application/fhir+ndjson` - Body of FHIR resources in newline delimited json - [ndjson](http://ndjson.org/) or other requested format ##### Response - Error -- HTTP Status Code of ```4XX``` or ```5XX``` +- HTTP Status Code of `4XX` or `5XX` ##### Attachments -If resources in an output file contain elements of the type ```Attachment```, servers SHALL populate the ```Attachment.contentType``` code as well as either the ```data``` element or the ```url``` element. The ```url``` element SHALL be an absolute url that can be de-referenced to the attachment's content. +If resources in an output file contain elements of the type `Attachment`, the server SHOULD populate the `Attachment.contentType` code as well as either the `data` element or the `url` element. When populated, the `url` element SHALL be an absolute url that can be de-referenced to the attachment's content. -When the ```url``` element is populated with an absolute URL and the ```requiresAccessToken``` field in the Complete Status body is set to ```true```, the url location must be accessible by a client with a valid access token, and SHALL NOT require the use of additional authentication credentials. When the ```url``` element is populated and the ```requiresAccessToken``` field in the Complete Status body is set to ```false```, the url location must be accessible by a client without an access token. +When the `url` element is populated with an absolute URL and the `requiresAccessToken` field in the manifest is set to `true`, the url location SHALL be accessible by a client with a valid access token, and SHALL NOT require the use of additional authentication credentials. When the `url` element is populated and the `requiresAccessToken` field in the manifest is set to `false`, and no additional authorization-related extensions are present on the manifest's output entry, the url location must be accessible by a client without an access token. -Note that if a server copies files to the bulk data output endpoint or proxies requests to facilitate access from this endpoint, it may need to modify the ```Attachment.url``` element when generating the FHIR bulk data output files. +Note that if a server copies files to the bulk data output endpoint or proxies requests to facilitate access from this endpoint, it may need to modify the `Attachment.url` element when generating the bulk data output files. ### Server Capability Documentation -This implementation guide is structured to support a wide variety of bulk data export use cases and server architectures. To provide clarity to developers on which capabilities are implemented in a particular server, server providers should ensure their documentation addresses the topics below. Future versions of this IG may define a computable format for this information as well. +This implementation guide is structured to support a wide variety of use cases and server architectures. To provide clarity to developers on which capabilities are implemented in a particular server, data providers should ensure their documentation addresses the topics below. Future versions of this IG may define a computable format for this information as well. -- Does the server restrict responses to a specific "profile" like US Core, USCDI, or Blue Button? +- Does the server restrict responses to a specific profile like the [US Core Implementation Guide](http://www.hl7.org/fhir/us/core/) or the [Blue Button Implementation Guide](http://hl7.org/fhir/us/carin-bb/)? - What approach does the server take to divide datasets into multiple files (eg. single file per the resource type, limit file size to 100MB, limit number of resources per file to 100,000)? -- Does the server support system-wide or Group-level export? -- What file formats does this server return? \ No newline at end of file +- What file formats does this server return? +- Are there extension fields populated in the manifest and how should a client use them? From 1b7ec68cebd828ccbc8729f3f84449eee039fc2a Mon Sep 17 00:00:00 2001 From: Dan Gottlieb <dan@gotdan.org> Date: Wed, 29 Nov 2023 16:42:12 -0500 Subject: [PATCH 7/7] Add performance related header guidance --- input/pagecontent/bulk-publish.md | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/input/pagecontent/bulk-publish.md b/input/pagecontent/bulk-publish.md index dbdfbb6..8e1e79b 100644 --- a/input/pagecontent/bulk-publish.md +++ b/input/pagecontent/bulk-publish.md @@ -34,7 +34,7 @@ Request for fully static or periodically updated dataset in FHIR format. GET `[fhir base]/$bulk-publish` - A client MAY include an `_since` parameter with a value that is a [FHIR instant](https://www.hl7.org/fhir/datatypes.html#instant). If this parameter is provided, the server MAY restrict the FHIR resources included in the response to only those that have been created or modified after the supplied time (e.g., `Resource.meta.lastUpdated` is later). - +- A client MAY include the conditional request HTTP header `If-None-Match` or `If-Modified-Since` with each request to avoid retrieving data when nothing has changed since the last request. Servers SHOULD support the use of these headers. ##### Response - Error @@ -48,6 +48,7 @@ GET `[fhir base]/$bulk-publish` - HTTP status of `200 OK` - `Content-Type` header of `application/json` - The server MAY return an `Expires` header indicating when the files listed will no longer be available for access. +- The server SHOULD return an `ETag` header - A body containing a JSON object providing metadata, and links to the generated bulk data files. The files SHALL be accessible to the client at the URLs advertised. These URLs MAY be served by file servers other than a FHIR-specific server. Required Fields: @@ -200,12 +201,15 @@ Example NDJSON output file: - `Accept` (optional, defaults to `application/fhir+ndjson`) -Specifies the format of the file being requested. + Specifies the format of the file being requested. + +- A client MAY include the conditional request HTTP header `If-None-Match` or `If-Modified-Since` with each request to avoid retrieving data when nothing has changed since the last request. Servers SHOULD support the use of these headers. ##### Response - Success - HTTP status of `200 OK` - `Content-Type` header that matches the file format being delivered. For files in ndjson format, SHALL be `application/fhir+ndjson` +- The server SHOULD return an `ETag` header - Body of FHIR resources in newline delimited json - [ndjson](http://ndjson.org/) or other requested format ##### Response - Error