All notable changes to this project will be documented in this file.
The format is based on Keep a Changelog and this project adheres to Semantic Versioning.
- #500: fix error with remote datasource
- #499: add CORS headers to response from cache
- #494: fix issue with template loading process
- #493: Allow duplicate nodes in public path
Web 7.0 modifies the way domain redirects are configured. The existing functionality is to supply only a hostname to redirect to, as a String property. This is a breaking change in Web 7.0, and requires configuring forceDomain
to be an object containing hostname, port and redirect type (now also a number
rather than a string
).
The default is an empty hostname, effectively disabling the functionality.
"default": {
"hostname": "",
"port": 80,
"type": 301
}
To redirect to example.com
from www.example.com
temporarily you could configure it thus:
"rewrites": {
"forceDomain": {
"hostname": "example.com",
"type": 307
}
}
Web 7.0 allows flattened datasource specification files, to match the page specification files. Notice in the example below there is no top-level datasource
property as specified here. No changes to existing projects are required, this change is backwards compatible with previous versions.
{
"key": "cars",
"name": "Cars datasource",
"source": {},
"count": 20,
"search": {},
"filter": {}
}
Web 7.0 allows page specification files without the "page" block, using the filename as the page name internally.
Both the following examples will work for index.json
{
"page": {
"name": "index"
},
"routes": [
{
"path": "/"
}
]
}
{
"routes": [
{
"path": "/"
}
]
}
The value of the composed
property will be added to the API endpoint querystring.
{
"datasource": {
"key": "cars",
"name": "Cars datasource",
"source": {
"endpoint": "/1.0/db/cars"
},
"compose": "all",
"count": 20
}
}
The generated endpoint will become /1.0/db/cars?count=20&compose=all
Start the server with support for HTTP2 by changing the protocol used in the server
configuration:
"server": {
"host": "example.com",
"port": 443,
"protocol": "http2",
"sslPrivateKeyPath": "keys/server.key",
"sslCertificatePath": "keys/server.crt",
}
This basic level of support gives Web most of the HTTP2 benefits such as multiplexing, binary transfers and header compression, and is backwards compatible with clients that do not support it 🎉
- #263: allow composed option in datasource specification files
- #438: simplify page specification files
- #469: allow adding/modifying/deleting page.json files while the application is running
- #479: allow flattened datasource specification files
- #489: fix an issue whereby the querystring parameters of a URL were not taken into account when constructing a cache key
-
Add property to markdown datasource to specify whether the HTML should be rendered and added to the output. To disable HTML, simply extend the
source
block with arenderHtml
property:"source": { "type": "markdown", "path": "./docs", "renderHtml": false }
- #405: Multi-language support
- #402: Add caching to markdown datasource status: needs tests
- #416: Markdown provider: error when document body contains --- type: bug
- #428: 404 returned for files containing spaces type: bug
- #399: improve the process of selecting a loaded endpoint based on the request URL within the cache layer
A new debug view has been added to allow developers to understand how a particular page has been generated. In previous versions of Web we've used a configuration setting allowJsonView
which enabled the option of seeing the data context used to build a rendered page - this could be used on any page by appending json=true
to the URL.
The new debug view in this version is far more powerful and provides greater insight into how your data is being used within Web. Documentation is available on the DADI Documentation website.
Version 6.0 removes Web's 1:1 relationship with DADI API, allowing multiple DADI API configurations which can be referenced from a datasource by API name:
Main configuration
"api": {
"main": {
"host": "api-one.somedomain.com",
"port": 80,
"auth": {
"tokenUrl": "/token",
"clientId": "your-client-id",
"secret": "your-secret"
}
},
"secondary": {
"host": "api-two.somedomain.com",
"port": 80,
"auth": {
"tokenUrl": "/token",
"clientId": "your-client-id",
"secret": "your-secret"
}
}
}
Datasource configuration
{
"datasource": {
"key": "articles",
"source": {
"api": "main",
"endpoint": "1.0/library/articles"
},
"count": 12,
"paginate": false
}
}
Version 6.0 removes the wordpress
and twitter
data providers, replacing them with a restapi
provider. Details are available here: https://docs.dadi.tech/web/latest#rest-api. The main difference between the existing remote
provider and the new restapi
provider is that restapi
provider can be supplied with authentication configuration.
- #258: give
globalEvents
access to full page data, and run per request instead of only at startup - #262: default workspace folders no longer created unnecessarily, even if the config
paths
specified different locations than the default - #267: fix Markdown provider crash if data was malformed
- #336: middleware can now intercept public folder requests
- #350: add support for range header requests on static assets so a browser can specify which part of a file it wants to receieve and when.
- #370: add configuration options for @dadi/status
- Deprecated configuration setting for compression removed.
headers.useGzipCompression
was deprecated in Web 4.0 in favour of the more genericuseCompression
.
Note: this is a breaking change
This configuration option has been moved within the page specification file to a block dedicated to the chosen template engine. Modify existing page.json
files as follows:
Existing
"settings": {
"keepWhitespace": true
}
New
"settings": {
"engine": {
"keepWhitespace": true
}
}
Template engine developers can also use engine
to pass any page specific setting to their engine.
- upgrade to Brotli 1.0 compression engine
- upgrade router to use version 2.0 of path-to-regexp
- ensure page parameter passed to datasources is numeric
- include uuid in hashed filename for uniqueness
- #170: feature: file upload support
- #253: 📬 feature: post processors
- #297: feature: datasource parameter extension
- #193: fix expired token datasource failure
- #216: construct endpoints with chained datasources
- #288: fix required data check fails when given no data
- #310: simplify start page: removes Dust.js and it's dependencies from the Web core. Instead there is a simple start page powered by the ES6 template engine (rather than the previous blog configuration)
- #312: allow Mongo aggregation queries
- fix bug where datasource parameters would be added to a DADI API endpoint when they already exist in the endpoint - leading to incorrect results loaded from the API
See the full release notes.
- #238: update metadata for Markdown posts after filtering
- fix: for every datasource request, the DatasourceCache file instantiated a new DadiCache - combining this with Redis caching results in ever-increasing (and unreleased) Redis connections. This change makes use of the primary Cache layer, instantiated once at application startup.
See the full release notes.
- #158: compress response before caching
- #174: introduce Brotli compression
- Static assets now obey configured compression settings; previously public folder assets were not subject to compression. Files will only be compressed if doing so will save space.
To support the introduction of the new compression engine, the configuration setting for compression has changed. To enable compression in Version 4.0, use config.headers.useCompression
rather than config.headers.useGzipCompression
. The config.headers.useGzipCompression
property is deprecated and will be removed in a future release.
DADI Web 4.0 adds CSRF security, giving developers the ability to add a per-request CSRF token into the view context, and ensures that all POST requests supply a correct CSRF token. Without a correct token, and with CSRF enabled, users will be greeted with an HTTP 403 response.
To enable CSRF, set the security.csrf
configuration option:
"security": {
"csrf": true
}
Once enabled, the property csrfToken
will be added to the view context. You will need to add this to any forms which perform a POST using the field name _csrf, like so:
<form action="/" method="post">
<input type="text" name="test_input_safe">
<input type="hidden" name="_csrf" value="{csrfToken}">
<input type="submit" value="Submit form">
</form>
Launching the application now returns a Promise which, when resolved, returns an object containing the application instance and the loaded route/page components.
// start the application
require('@dadi/web')({
"engines":[
require("@dadi/web-dustjs")
]
}).then(loaded => {
console.log(loaded.App)
console.log(loaded.Components)
})
This change replaces the exported modules in previous versions. To obtain a reference to these modules when the application has already started (for example when loading template helpers), require @dadi/web without passing an engine argument:
require('@dadi/web')().then(loaded => {
console.log(loaded.App)
console.log(loaded.Components)
})
Page caching is now on by default if caching
is specified in the configuration. Page specification files no longer require cache: true
for caching to be enabled.
Version 4.0 performs route determination faster. In previous versions a request was tested against all loaded page components at the beginning of the request, and an array of matching routes was added to the middleware stack. In this version matching app-specific routes are loaded only if processing the middleware stack yields no matching handlers.
Requests for static files are now passed through the request logger, giving more detailed access logs for the full request cycle.
- Removed support for event-logging system "Sentry". This feature was untested and unused
- Added new middleware to serve content from the public folder, removing dependency on Express.js modules serve-static and serve-favicon.
- Moved helper methods
sendBackJSON
,sendBackHTML
intoview/send.js
- Removed unused helper
sendBackJSONP
- Removed outdated/unused
media
path. - Refactor of cache flush under
api/flush
. Added corresponding error page when method is notPOST
. - Added
npm run format
to run for standard & prettier - Hide the err.stack from default error pages when the
NODE_ENV
environment variable isproduction
(NODE_ENV=production
) - An improved developer experience: changes to event files & template partials/includes reinitialises the application without requiring a restart.
- #51: cache flush command fails when no matching page is found
- #59: add CSRF token
- #168: process routes after middleware
- #173: listener should trigger a 302 redirect
- #175: remove 'server' response header
- #193: reload templates and event files when changed on disk (without restarting app)
- #212: fix default workspace config error
- #204: remove Sentry support in favour of a future error handing implementation
- #209: wait for components and routes to be loaed before exporting
- ensure static datasources are passed request parameters for filtering data
- throw error at startup when the pages directory contains templates without an engine that can handle them
- fix an issue where information about the loaded engines on the app startup screen was sometimes incorrect
- add https://github.com/dadi/web-dustjs as dependency
- modify post install script to include the above plugin when creating the server.js file
- #103: multiple template engine support
- #184: unrestricted environment configurations allowed
- #175: response header "SERVER (DADI)" removed
- #165: only check matching routes after all middleware functions have completed
- #98: log more informative error when an event fails
- ensure request parameters passed to Markdown provider
- #120: add post install scripts to copy configuration and workspace files
- #135: allow requestParams to replace placeholders in datasource endpoints by specifying an additional property:
"target": "endpoint"
- #137: allow configuration of virtual hosts which can specify overriding
workspace
andglobal
properties. configuration details at #138 - add new remote data provider, using the original as a dedicated DADI API data provider
- #128: attach compression middleware prior to static middleware
- #130: leave url params unmodified when lowercasing urls
- #139 remove datasource path from attributes
- #144: check host header against specified hosts before serving static files
- don’t modify original schema endpoint (4901ecc)
- rename sample-workspace to workspace (c307b8e)
- #117: allow an array of partial paths
- add metadata for pagination (e34be99)
- add pagination to Markdown provider (09618e7)
- add transportSecurity configuration option (03fd4e8)
- API connection disabled by default (1199527)
- start in cluster mode by default (5987dd2)
- #123: xxx
- #110: filter hidden files for xx
- use debug statements instead of log statements (e425040)
- implement transportSecurity properly (f18764c)
- pass authstrategy to Bearer module (3a6bacc)
- rebuild chained datasource endpoints after applying results from chainee
- add example https config file (d069b2b)
- add http/http+https/https support (05cee15)
- add markdown provider (71070a0)
- add search/filter/sort/count/fields methods (a8e88ce)
- allow a 404 route to skip validation (9ba36b6)
- extend ds cache options using main config options (c0e9a12)
- make hard-coded protocol redirect dynamic (52f7fe8)
- make tests run cross platform (2f3f708)
- redirect http to https with 301 instead of 302 (ae6c13a)
- remove call to hasOwnProperty for node >= 6 (cfa48ad)
- Now runs on latest Node versions 4.7.0, 5.12.0, 6.9.2
- Improved build and test for cross platform usage, now works under Windows 10
- Fix: configured 404 pages now render the associated template and don't simply return the content of the page specification file (#101)
A cache-control
header can be added to a 301/302 redirect by adding to the headers
configuration block:
"headers": {
"cacheControl": {
"301": "no-cache"
}
}
The Web configuration file now has provision for specifying datasources that should be loaded at startup. Add preload datasources by adding a block to the configuration file:
data: {
preload: [
"categories"
]
}
Accessing preloaded data in Web events is as simple as passing the datasource key to the Preload
module:
var Preload = require('@dadi/web').Preload
var categories = Preload().get('categories')
The preloader stores the contents of the inner results property of the data that is returned, so there is no metadata
section as you would normally find in data returned from DADI API.
Note: There is no refresh of this preloaded data, yet. Refresh support will be added in a future release.
DADI Web 1.7.0 introduces several new ways to ensure the correct route is loaded for a particular request, hopefully reducing the hacky workarounds we've implemented for some projects.
See the documentation at http://docs.dadi.tech/web/concepts/routing/#route-validation
For assistance with routing changes, please add an issue to your project and assign it to
jimlambie
.
Version 1.7.0 sees the completion of the extraction of the built-in Dust helpers. These are no longer available in the core Web product and must be loaded separately, see https://www.npmjs.com/package/@dadi/dustjs-helpers for usage information.
DADI Web 1.7.0 introduces a more explicit way of specifying multiple routes per page . The route
property has been replaced with routes
which should be an Array of route objects.
Each route object must contain, at the very least, a path
property. At startup, Web adds the value of each path
property to an internal collection of routes for matching incoming requests.
In this example, the same page (and therefore it's template) will be loaded for requests matching any of the formats specified by the path
properties:
"routes": [
{
"path": "/movies/:title"
},
{
"path": "/movies/news/:title?/"
},
{
"path": "/movies/news/:page?/"
}
]
The DADI Cache module has replaced the caching code in Web 1.7.0. DADI Cache includes better failover support for when Redis connections fail. Datasources now require a caching configuration similar the main config. See the next section, Migration to Version 1.7.0.
To help you migrate your Web installation to v1.7.0, the router will inform you of any changes required to page specifications if the existing route
property has not yet been modified.
Look out for console messages similar to the following:
The `route` property for pages has been extended to provide better routing functionality.
Please modify the route property for page 'movies_news'. The schema should change to the below:
{
"page": {
"name": "movies_news",
"description": "movies news",
"language": "en"
},
"settings": {
"cache": true
},
"datasources": [],
"events": [],
"routes": [
{
"path": "/movies/news/:furl?/"
}
]
}
Note that it isn't a requirement to specify caching for datasources, as they will use the main configuration settings if none are specified.
Prior to 1.7.0, caching configuration took the following format:
"caching": {
"enabled": true,
"ttl": 300,
"directory": "./cache/web/",
"extension": "json"
}
In 1.7.0 this has changed. If you want your datasources to be cached differently to pages, the caching configuration should now appear similar to the main configuration file:
"caching": {
"ttl": 300,
"directory": {
"enabled": true,
"path": "./cache/web/",
"extension": "json"
},
"redis": {
"enabled": false
}
}
This version update adds the ability to use a datasource as the main source of the htaccess style rewrites
Add the following block to configuration, specifying an existing datasource name:
rewrites: {
datasource: "rewrites"
path: ""
}
- Cluster support This version adds support for running in cluster mode to take advantage of multiple core CPUs.
The default configuration does not use cluster mode. To enable, add the following to configuration:
"cluster": true
Web will report it is running in cluster mode at startup.
-
fix #63: don't use main config options for ds extension
if no cache config options are provided for a datasource the main config settings are used. for a datasource, we need to use .json by default, rather than the main config setting
-
add 503 server too busy response
-
allow port configuration to be set from environment variable PORT
-
add support for HTTPS
-
add page template metadata to returned JSON:
"page": { "name": "movies_news", "description": "movies news", "language": "en" }
-
add config option to force a domain redirect
this allows a setting in config as below which will force redirecting to the specified domain, e.g. useful to redirect a root domain to www.
"rewrites": { "forceDomain": "www.example.com" }
- allow Redis use for session store configuration
- add token wallet path to
paths
configuration blocks - integration of @dadi/passport for token generation/validation
- add protocol option to
api
andauth
configuration blocks
-
fix: bug where an ID parameter from the request URL was added to all subsequent datasource filters
-
fix: bug where caching was performed routing, therefore sometimes ignoring routes
-
fix: allow datasource configuration to override protocol:
- Previous versions assumed if the API settings used HTTPS then the datasource calls should too. Fix to allow a datasource to specify it's protocol and have that override the API setting.
-
fix #48: test current URL against returned redirect results:
- when using a datasource for redirects, ensure that any results returned are matched against the URL before redirecting the request
-
fix: allow any format for config setting sessions.cookie.maxAge
the default for express-session is to use
cookie.maxAge = null
which means no "expires" parameter is set so the cookie becomes a browser-session cookie.When the user closes the browser the cookie (and session) will be removed.
Adds an endpoint at /api/status
which returns server/application data in JSON format.
The request must be a POST and the body must have a clientId
and secret
that match those stored in the application's config file.
POST /api/status HTTP/1.1
Host: www.example.com
Content-Type: application/json
{"clientId": "testClient","secret": "superSecret"}
Now allows the ability to specify a type definition of 'Number' on requestParams in a datasource schema to override default of String. Thanks @mingard!
"requestParams": [
{ "param": "author", "field": "authorId", "type": "Number" }
]
Storing partials in subdirectories of the main partials folder previously caused the application crash. Now it doesn't. Thanks @eduardoboucas!
- Add: additional routing/rewriting config properties
- Add #37: Global events loader
- Fix #38: replace datasource loader in router
- Fix #36: load events in the order they were specified
- Fix #32: load template from filesystem if Dust cache is_disabled
- Fix #31: define the zlib variable
- Fix #29: refresh endpoint filter on subsequent page loads
Version 1.1.0 introduces a cache invalidation endpoint which allows an authorised user to flush the cache for either a specific path or the entire website. This process clears both page and datasource cache files.
The user must send a POST request to /api/flush
with a request body containing the path to flush and
a set of credentials that match those held in the configuration file:
Flush all cache files
POST /api/flush HTTP/1.1
Host: www.example.com
{ "path": "*", "clientId": "testClient", "secret": "superSecret" }
Flush cache files for a specific path
POST /api/flush HTTP/1.1
Host: www.example.com
{ "path": "/books/crime", "clientId": "testClient", "secret": "superSecret" }
A datasource can now specify a filterEvent
property. Before the datasource attempts to load data
it will load and run an event file matching the filterEvent
property. Filter events are identical to normal
event files, but they should return a filter object that the datasource will use when querying the API for data.
Example filter event: /app/events/datasourceFilterTest.js
// the `data` parameter contains the data already loaded by
// the page's datasources
var Event = function (req, res, data, callback) {
var filter = { "x": "1" }
callback(null, filter)
}
module.exports = function (req, res, data, callback) {
return new Event(req, res, data, callback)
}
module.exports.Event = Event;
The sort property in a datasource schema has been extended to allow a variety of styles. The sort property can now take one of the following forms:
To sort by the field "name" ascending, as an array of field names:
"sort": [{ "field": "name", "order": "asc" }]
To sort by the field "name" ascending, as a single field name:
"sort": { "field": "name", "order": "asc" }
To sort by the field "name" ascending, as a MongoDB-style object:
"sort": { "name": 1 }
To sort by multiple fields, as a MongoDB-style object:
"sort": { "name": 1, "age": -1 }
- Cache:
- Ensure a more unique datasource cache key by including the datasource name as well as the endpoint
- Ensure a more unique page cache key by including the query as well as the pathname
- Improve search for loaded component based on request URL
- Ensure contentType is passed from loaded component (page) settings when returning cached data
- Config:
- Remove
sentry.enabled
and rely solely on the existence of thesentry.dsn
property - Rationalise included config properties in sample files, most can be handled by the sensible defaults
- Datasource:
- Add
skip
property to give the option of specifying an offset when querying for API data - Use main config api settings when endpoint host or port are not specified by the datasource schema
- Event:
- Pass 404 errors from event files to the NotFound handler
- Views:
- Added new
replace
helper, usage: {@replace str="hello.world" search="." replace="-" /}
- Add config option for socket timeout, defaults to 30 seconds
- Keepalive header added to API data & auth requests
- Server:
- Error if the configured API can't be reached
- Simplify middleware loading
- Logging:
- Simplify error logging
- Provide configuration for logging to a Sentry server
- Remove Slack logging option, as this can be done from Sentry
- Config:
- Provide configuration for logging to a Sentry server
- Cache:
- Don't cache requests that use ?json=true in the querystring
- Provides better integration with Express patterns
- Debug:
- When debug: true is set in config, a debug panel is added to the right side of the page with a view of the loaded data and various execution stats. Note this won't work if your data object contains Javascript ads with no CDATA declaration.
- Logging:
- Locate client IP in request headers when behind a load balancer
- Fix logging in default error handler
- Addition of AWS Kinesis and Slack logging options (see example files or config.js)
- Don't log to slack in test mode
- Config:
- Add support for serving content from virtual directories (see example files or config.js)
- Add support for Gzip compression (see example files or config.js)
- Add support for configurable cache control headers (see example files or config.js)
- Add application name (see example files or config.js)
- Pages:
- Extend
toPath
method to cover all routes in a page - Addition of new getComponent method on the Server to return a page by key
- Extend
- Datasources:
- Remove json param from query before building filter
- Tests
- improve test suite coverage
- ensure test environment is set before tests run
- Misc:
- rename public to sample-public to avoid overwrite when updating
- check page file extension when reloading to avoid processing swap files