Added option to store digital assets into static folder instead of using Gatsby cache

Author: kmacdonaO

README.md

@@ -36,6 +36,11 @@ module.exports = {
           query: '',
         },
         renditions: 'none',
+        preview: false,
+        auth: '',
+        staticAssetDownload: 'false'.
+        staticAssetRootDir: 'asset'
+
       },
     },
   ],
@@ -73,22 +78,55 @@ This setting controls which renditions of digital assets will be downloaded. Thi
 
 - `custom` (default) means that only custom renditions will be downloaded.
 - `none` means that only the original data will be downloaded.  
-- `all` means that all of the OCE-generated renditions (*thumbnail*, *small*, etc.) will be downloaded as well as any custom renditions.
+- `all` means that all of the system generated renditions (*thumbnail*, *small*, etc.) will be downloaded as well as any custom renditions.
+
+> NOTE: For each image downloaded, Gatsby will generate multiple renditions of its own to use for responsive display. This means that if you use the `all` value for renditions, Gatsby will generate resized copies of both the original image and for each of the system renditions (*thumbnail*, *small*, etc.). This can add a lot of processing time and network load to the build process for support that you may not need. It is recommended that you use the `none` or `custom` option unless you wish to explicitly use the system renditions.  
+
+**`preview`** [optional]  
+This setting controls whether the client will be retrieving preview (unpublished) data or not. If true, it must be used with the auth parameter.  It defaults to false.
+
+**`auth`** [optional]  
+This setting is used to set authentication tokens required when downloading from a secure publishing channel (preview=false) or preview items. It may use basic auth or
+an OAuth bearer token. It defaults to an empty string.
+
+**`staticAssetDownload`** [optional]  
+This setting is used to make the plugin download binary assets into the public directory of the site as opposed to storing them in the internal Gatsby cache. This allows items such as image files to be referenced as direct URLs rather than using GraphQL to access them. It is used in combination with the staticAssetRootDir (described below). The default value is false which means all data will be in the cache instead. 
+
+**`staticAssetRootDir`** [optional]  
+This setting is ignored unless staticAssetDownload is set to true. This allows a developer to define a directory in the website that will contain all the downloaded assets. This can help to segregate the items coming from the server from other static data.  The default value is "asset", but it can be set to any string that is a valid directory name. 
+
+**`How to use static download:`**  
+  Assume there is a digital asset on the server called Logo.jpg that contains an image to be used in a site.  
+If staticAssetDownload is false then the binary image will be stored in the cache as well as any selected renditions. (See 'renditions' flag above)   These binary files can then be queried and traversed via GraphQL  
+  
+If staticAssetDownload is true and staticAssetRootDir is set to 'server' then the file will be available in the website as: /server/Logo.jpg and could be displayed
+in an \<img\> tag as \<img src="/server/Logo.jpg"\/>  . If the user has chosen to download all the renditions of the digital assets in their site then the following urls
+will be available:   
+/server/Logo.jpg     (the original)  
+/server/Large/Logo.jpg     (the large rendition)  
+/server/Medium/Logo.jpg     (the medium rendition)  
+/server/Small/Logo.jpg     (the small rendition)  
+/server/Thumbnail/Logo.jpg     (the thumbnail rendition)  
+
+If there are any custom renditions, then they will also appear as
+/server/*custom rendition name*/Logo.jpg
+ 
+
+
 
-> NOTE: For each image downloaded, Gatsby will generate multiple renditions of its own to use for responsive display. This means that if you use the `all` value for renditions, Gatsby will generate resized copies of both the original image and for each of the OCE renditions (*thumbnail*, *small*, etc.). This can add a lot of processing time and network load to the build process for support that you may not need. It is recommended that you use the `none` or `custom` option unless you wish to explicitly use the OCE renditions.  
 
 ### Content Model
 
-The data model used for OCE data in Gatsby closely models the JSON data provided by the OCE REST interface. There are a few small exceptions though that are needed to avoid conflicts between Gatsby node reserved fields and OCE data. These include:
+The data model used for content data in Gatsby closely models the JSON data provided by the Oracle Content Management REST interface. There are a few small exceptions though that are needed to avoid conflicts between Gatsby node reserved fields and Oracle Content Management data. These include:
 
 - All assets are typed as `oceAsset` so that they can be distinguished from other assets in GraphQL queries.
-- Some OCE asset fields are renamed to avoid conflicts with reserved Gatsby node field names. `id` becomes `oceId`,  `fields` becomes `oceFields`, and `type` becomes `oceType`.
-- In traditional OCE usage, a digital asset provides a URL that can be used to retrieve the binary form(s) of the asset from the server. As Gatsby builds static sites, these binary values must be stored locally and are placed in file nodes in the GraphQL store.  To allow these binary forms to be retrieved easily in a site, a link is established between the file nodes and the digital asset nodes. What this means is that it is possible to traverse an `oceAsset` and find the internal Gatsby representations of its binary data without having to load the file nodes as well. All of an `oceAsset`'s  file data can be found under the field `childrenFile`.
+- Some Oracle Content Management asset fields are renamed to avoid conflicts with reserved Gatsby node field names. `id` becomes `oceId`,  `fields` becomes `oceFields`, and `type` becomes `oceType`.
+- In traditional Oracle Content Management usage, a digital asset provides a URL that can be used to retrieve the binary form(s) of the asset from the server. As Gatsby builds static sites, these binary values must be stored locally and are placed in file nodes in the GraphQL store.  To allow these binary forms to be retrieved easily in a site, a link is established between the file nodes and the digital asset nodes. What this means is that it is possible to traverse an `oceAsset` and find the internal Gatsby representations of its binary data without having to load the file nodes as well. All of an `oceAsset`'s  file data can be found under the field `childrenFile`.
 
 ## Examples
 
 You can query `OceAsset` nodes in the following manner:  
-(This returns the names and types of all assets. Note that the `type` field is internal to Gatsby, so we use the `oceType` field to get the OCE name for each definition )  
+(This returns the names and types of all assets. Note that the `type` field is internal to Gatsby, so we use the `oceType` field to get the Oracle Content Management name for each definition )  
 
 ```graphql
 {
@@ -124,10 +162,7 @@ Load the information needed to display the image `Banner1.jpg` as a fluid (respo
         rendition
       }
       childImageSharp {
-        fluid(quality: 100) {
-            
-          ...GatsbyImageSharpFluid_withWebp
-        }                
+        gatsbyImageData(quality: 100, layout: FULL_WIDTH)
       }
     }
   }

gatsby-node.js

@@ -28,14 +28,16 @@ const fetch = require('./src/fetch');
  *         query: '',
  *       },
  *       renditions: 'all',
+ *       staticAssetDownload: false
+ *       staticAssetRootDir: 'asset'
  *     },
  * },
  * ]
  *</code></pre></div>
  * <b>contentServer</b> should be set to the base url for your server. For example: https://oceserver.domain.com<br>
  * <b>channelToken</b> should be set to the publishing channel id on the given server<br>
- * <b>proxyUrl</b> this is only needed if there is a proxy between the local computer and
- * the OCE server. It should be of the form: "http://proxyserver.company.com:port"
+ * <b>proxyUrl</b> (optional) this is only needed if there is a proxy between the local computer and
+ * the OCE server. It should be of the form: "http://proxyserver.company.com:port"<br>
  * <b>limit</b> (optional) How many assets  should be queried from the server in one call.
  * This can generally be left alone <br>
  * <b>query</b> (optional) It is used to limit what class of assets should be downloaded
@@ -57,6 +59,15 @@ const fetch = require('./src/fetch');
  *  If your project uses 100 image assets,the plugin will then download 500+ files from the server
  * (1 original + 4 generated renditions, plus all custom renditions)
  * This is a waste of bandwidth unless you plan to use the oce renditions directly in your site.
+ * <br>
+ * <b>staticAssetDownload</b> (optional) should be set to true if you want to download assets as
+ * files to be included via urls in your site. If it is set to the default value of false the
+ * binary files will be placed in the Gatsby cache and will be accessible via GraphQL<br>
+ * <b>staticAssetRootDir</b> this setting is only used if staticAssetDownload is true. It is used
+ * to set a a prefix that will be prepended to all of the urls for downloaded files. For example:
+ *  If it is set to "content" then the url of a downloaded image called Logo.jpg
+ * will be /content/Logo.jpg.
+ *
  *
  */
 exports.sourceNodes = async (
@@ -72,11 +83,25 @@ exports.sourceNodes = async (
   delete configOptions.plugins;
 
   // plugin code goes here...
-  console.log('Using OCE plugin with the options: ', configOptions);
 
   const {
-    contentServer, channelToken, proxyUrl = '', items: { limit } = 100, items: { query } = '', renditions = 'custom', debug = false,
+    contentServer, channelToken, proxyUrl = '', items: { limit } = 100, items: { query } = '', auth = '', preview = false, renditions = 'custom',
+    staticAssetDownload = false, staticAssetRootDir = 'assets', debug = false,
   } = configOptions;
+
+  console.log('Using OCE plugin with the options: ', {
+    contentServer,
+    channelToken,
+    proxyUrl,
+    limit,
+    query,
+    auth: `${auth.length} characters`,
+    preview,
+    renditions,
+    staticAssetDownload,
+    staticAssetRootDir,
+    debug,
+  });
   try {
     if (proxyUrl !== '') {
       try {
@@ -87,7 +112,8 @@ exports.sourceNodes = async (
         console.log(`ERROR ${e}`);
       }
     }
-    let entities = await fetch.all(contentServer, channelToken, limit, query, debug);
+    let entities = await fetch.all(contentServer, channelToken, limit, query,
+      auth, preview, debug);
     // tidy up the data for further processing
     entities = entities.filter((e) => e != null && (typeof e === 'object'));
     entities = process.cleanUp(entities);
@@ -100,16 +126,30 @@ exports.sourceNodes = async (
     // Generate unique Gatsby ids for indexing
     entities = process.createGatsbyIds(createNodeId, entities, channelToken);
 
+    if (staticAssetDownload === true || staticAssetDownload === 'true') {
+      console.log('Starting static download');
+      await process.prepareForStaticDownload({ staticAssetRootDir });
+      await process.downloadMediaFilesToStaticDir({
+        entities,
+        staticAssetRootDir,
+        renditions,
+        auth,
+      });
+      console.log('Ending static download');
+    } else {
     // Get a copy of all digital asset files (.jpg, .png, etc...) and make Gatsby nodes for them
-    entities = await process.downloadMediaFiles({
-      entities,
-      store,
-      cache,
-      createNode,
-      createNodeId,
-      touchNode,
-      renditions,
-    });
+      entities = await process.downloadMediaFiles({
+        entities,
+        store,
+        cache,
+        createNode,
+        createNodeId,
+        touchNode,
+        renditions,
+        auth,
+      });
+    }
+
     console.log('Completed download of assets');
     // Create nodes for all the downloaded assets
     await process.createNodesFromEntities({ entities, createNode, createContentDigest });

package.json

@@ -2,7 +2,7 @@
   "name": "@oracle/gatsby-source-oce",
   "description": "Gatsby source plugin for using Oracle Content Management as a data source",
   "author": "Oracle America, Inc.",
-  "version": "1.0.1",
+  "version": "1.1.0",
   "homepage": "https://github.com/oracle/gatsby-source-oce",
   "repository": {
     "type": "git",
@@ -35,5 +35,14 @@
     "eslint": "~7.11.0",
     "eslint-config-airbnb-base": "~14.2.1",
     "eslint-plugin-import": "~2.22.1"
-  }
+  },
+  "files": [
+    "CONTRIBUTING.md",
+    "LICENSE.txt",
+    "README.md",
+    "SECURITY.md",
+    "THIRD_PARTY*",
+    "*.js",
+    "src"
+  ]
 }

src/fetch.js

@@ -6,6 +6,7 @@
 /* eslint-disable no-console */
 /* eslint-disable no-await-in-loop */
 const fetch = require('node-fetch');
+const { Headers } = require('node-fetch');
 const fs = require('fs').promises;
 
 /**
@@ -23,7 +24,7 @@ const fs = require('fs').promises;
  * data retrieved from the OCE server.
  */
 
-exports.all = async (contentServer, channelToken, limit, query, debug) => {
+exports.all = async (contentServer, channelToken, limit, query, auth, preview, debug) => {
   try {
     await fs.rmdir('.data', { recursive: true });
   } catch (err) {
@@ -32,33 +33,62 @@ exports.all = async (contentServer, channelToken, limit, query, debug) => {
   if (debug) {
     await fs.mkdir('.data');
   }
-  const fetchLimit = ((limit != null && limit > 0) ? limit : 10);
 
-  const fetchQuery = query ? `&q=${query}` : '';
-  const allItemsUrl = `${contentServer}/content/published/api/v1.1/items?limit=${fetchLimit}&offset=0&totalResults=true&offset=0&channelToken=${channelToken}${fetchQuery}`;
+  // const pretty = (o) => JSON.stringify(o, null, 2);
 
   const fetchItem = async (id) => {
-    const response = await fetch(
-      `${contentServer}/content/published/api/v1.1/items/${id}?channelToken=${channelToken}&expand=all`,
-    );
-    const item = await response.json();
-
-    // Remove hyphens which can cause issues
-    item.type = item.type.replace('-', '');
-    if (debug) {
-      await fs.writeFile(
-        `.data/${id}.json`,
-        JSON.stringify(item, null, 2),
-        'utf-8',
-      );
+    let item = null;
+    const itemUrl = preview === 'true' ? `${contentServer}/content/management/api/v1.1/items/${id}` : `${contentServer}/content/published/api/v1.1/items/${id}?channelToken=${channelToken}&expand=all`;
+
+    try {
+      const headers = new Headers({
+        Authorization: `${auth}`,
+        Accept: '*/*',
+      });
+
+      const response = await fetch(itemUrl, { headers });
+      item = await response.json();
+
+      // Remove hyphens which can cause issues
+      item.type = item.type.replace('-', '');
+      if (debug) {
+        await fs.writeFile(
+          `.data/${id}.json`,
+          JSON.stringify(item, null, 2),
+          'utf-8',
+        );
+      }
+    } catch (e) {
+      console.log(`ERROR Download of item ${id} with URL ${itemUrl} ${e} `);
+      throw (e);
     }
     return item;
   };
 
   const fetchAll = async () => {
+    const fetchLimit = ((limit != null && limit > 0) ? limit : 10);
+
+    const fetchQuery = query ? `&q=${query}` : '';
+    const allPublishedItemsUrl = `${contentServer}/content/published/api/v1.1/items?limit=${fetchLimit}&offset=0&totalResults=true&offset=0&channelToken=${channelToken}${fetchQuery}`;
+    const allManagementItemsUrl = `${contentServer}/content/management/api/v1.1/items?limit=${fetchLimit}&offset=0&totalResults=true&offset=0&channelToken=${channelToken}${fetchQuery}`;
+
+    const allItemsUrl = preview === 'true' ? allManagementItemsUrl : allPublishedItemsUrl;
+
     // Fetch a response from the apiUrl
 
-    let response = await fetch(allItemsUrl);
+    let response = null;
+    let headers = '';
+    try {
+      if (auth !== '') {
+        headers = new Headers({
+          Authorization: `${auth}`,
+          Accept: '*/*',
+        });
+      }
+      response = await fetch(allItemsUrl, { headers });
+    } catch (e) {
+      console.log(`ERROR Failed downloading item list using  ${allItemsUrl}`);
+    }
     // Parse the response as JSON
     try {
       const data = await response.json();
@@ -70,8 +100,7 @@ exports.all = async (contentServer, channelToken, limit, query, debug) => {
 
         for (let offset = fetchLimit; offset < totalResults; offset += fetchLimit) {
           const partialQuery = allItemsUrl.replace('&offset=0', `&offset=${offset}`);
-          // console.log(`partialQuery: ${partialQuery}`);
-          response = await fetch(partialQuery);
+          response = await fetch(partialQuery, { headers });
           const partialData = await response.json();
           data.items = data.items.concat(partialData.items);
         }
@@ -93,6 +122,7 @@ exports.all = async (contentServer, channelToken, limit, query, debug) => {
       throw err;
     }
   };
+
   const entities = await fetchAll();
   return entities;
 };