feat(plugin-import-export)!: refactor plugin and add import functionality

[paulpopus]

This PR works to refactor the entire plugin so it's easier for us to work with it going forward and also implements import functionality.

It expands the int tests and adds some e2e tests as well.

Previous config

Previously the config for the plugin looked like this. If you wanted to disable jobs queue you'd have to do it for all collections, the collections config was optional and the plugin applied both import and export functionality to all collections which isn't normally how our plugins work.

If you want to customise certain behaviour you need to instantiate the plugin multiple times, which causes additional problems by doubling up providers, UI components and endpoints.

As such we're committing some breaking changes while the plugin is in beta to standardise the config and allow more customisation.

importExportPlugin({
  overrideExportCollection: (collection) => {
    collection.admin.group = 'System'
    collection.upload.staticDir = path.resolve(dirname, 'uploads')
    return collection
  },
  overrideImportCollection: (collection) => {
    collection.admin.group = 'System'
    collection.upload.staticDir = path.resolve(dirname, 'uploads')
    return collection
  },
  disableJobsQueue: true,
  debug: true,
  // collections: ['posts'] // optionally provided - defaults to all collections
}),

New config

The new config pushes most customisation to a per collection level so that you only ever need to instantiate the plugin once.

You can now provide export/import collections per collection config if needed, do not instantiate the plugin multiple times.

collections is now required.

importExportPlugin({
   overrideExportCollection: (collection) => {
    collection.admin.group = 'System'
    collection.upload.staticDir = path.resolve(dirname, 'uploads')
    return collection
  },
  overrideImportCollection: (collection) => {
    collection.admin.group = 'System'
    collection.upload.staticDir = path.resolve(dirname, 'uploads')
    return collection
  },
  collections: [
    {
      slug: 'posts',
      import: false, // disables import functionality, export enabled by default
    },
    {
      slug: 'pages',
      // customise the export collection per collection if needed
      export: ({ collection }) => {
        collection.admin.group = 'System'
        collection.upload.staticDir = path.resolve(dirname, 'uploads')
        return collection
      },
      disableJobsQueue: true, // disable jobs queue for this collection only
    },
  ],
  debug: true, // debug remains top level
})

This PR also fixes the following:

• Drafts being exported even if 'no' is selected to include them
• Fetch endpoints from preview not respecting the API routes
• Downloads not respecting drafts
• Dependencies on csv-parse and csv-stringify are now pinned

Todo:

• Regenerate all translations
• Add docs for import
• Review code structure/patterns
• Polish the preview UI to allow for pagination, show loading states and make it clear what data is being previewed

[github-actions]

📦 esbuild Bundle Analysis for payload

This analysis was generated by esbuild-bundle-analyzer. 🤖

Meta File	Out File	Size (raw)	Note
packages/next/meta_index.json	esbuild/index.js	764.57 KB	🆕 Added
packages/payload/meta_index.json	esbuild/index.js	1.23 MB	🆕 Added
packages/payload/meta_shared.json	esbuild/exports/shared.js	163.07 KB	🆕 Added
packages/richtext-lexical/meta_client.json	esbuild/exports/client_optimized/index.js	280.71 KB	🆕 Added
packages/ui/meta_client.json	esbuild/exports/client_optimized/index.js	1.16 MB	🆕 Added
packages/ui/meta_shared.json	esbuild/exports/shared_optimized/index.js	14.39 KB	🆕 Added

Largest paths

These visualization shows top 20 largest paths in the bundle.

Meta file: packages/next/meta_index.json, Out file: esbuild/index.js

Path	Size
../../node_modules	${{\color{Goldenrod}{ ████████████████████ }}}$ 80.2%, 609.13 KB
dist/views/Version	${{\color{Goldenrod}{ █▋ }}}$ 6.6%, 50.19 KB
dist/views/Document	${{\color{Goldenrod}{ ▌ }}}$ 2.0%, 15.51 KB
dist/views/List	${{\color{Goldenrod}{ ▍ }}}$ 1.5%, 11.27 KB
dist/views/Root	${{\color{Goldenrod}{ ▎ }}}$ 1.2%, 8.97 KB
dist/views/API	${{\color{Goldenrod}{ ▏ }}}$ 0.8%, 5.98 KB
dist/views/Versions	${{\color{Goldenrod}{ ▏ }}}$ 0.8%, 5.96 KB
dist/elements/Nav	${{\color{Goldenrod}{ ▏ }}}$ 0.7%, 5.53 KB
dist/views/Account	${{\color{Goldenrod}{ ▏ }}}$ 0.7%, 5.32 KB
dist/elements/DocumentHeader	${{\color{Goldenrod}{ ▏ }}}$ 0.6%, 4.81 KB
dist/views/Login	${{\color{Goldenrod}{ ▏ }}}$ 0.6%, 4.39 KB
dist/views/Dashboard	${{\color{Goldenrod}{ ▏ }}}$ 0.5%, 3.69 KB
dist/views/ForgotPassword	${{\color{Goldenrod}{ }}}$ 0.4%, 3.09 KB
dist/layouts/Root	${{\color{Goldenrod}{ }}}$ 0.4%, 2.91 KB
dist/templates/Default	${{\color{Goldenrod}{ }}}$ 0.4%, 2.83 KB
dist/views/CreateFirstUser	${{\color{Goldenrod}{ }}}$ 0.4%, 2.76 KB
dist/views/BrowseByFolder	${{\color{Goldenrod}{ }}}$ 0.3%, 2.60 KB
dist/views/CollectionFolders	${{\color{Goldenrod}{ }}}$ 0.3%, 2.46 KB
dist/views/ResetPassword	${{\color{Goldenrod}{ }}}$ 0.3%, 2.41 KB
dist/views/Logout	${{\color{Goldenrod}{ }}}$ 0.3%, 1.92 KB
(other)	${{\color{Goldenrod}{ ████▉ }}}$ 19.8%, 150.76 KB

Meta file: packages/payload/meta_index.json, Out file: esbuild/index.js

Path	Size
../../node_modules	${{\color{Goldenrod}{ █████████████████▏ }}}$ 68.7%, 841.09 KB
dist/fields/hooks	${{\color{Goldenrod}{ ▉ }}}$ 3.5%, 43.31 KB
dist/collections/operations	${{\color{Goldenrod}{ ▊ }}}$ 3.0%, 37.29 KB
dist/auth/operations	${{\color{Goldenrod}{ ▎ }}}$ 1.3%, 15.65 KB
dist/queues/operations	${{\color{Goldenrod}{ ▎ }}}$ 1.0%, 12.11 KB
dist/globals/operations	${{\color{Goldenrod}{ ▎ }}}$ 1.0%, 12.04 KB
dist/fields/config	${{\color{Goldenrod}{ ▎ }}}$ 1.0%, 11.95 KB
dist/utilities/configToJSONSchema.js	${{\color{Goldenrod}{ ▎ }}}$ 1.0%, 11.83 KB
dist/fields/validations.js	${{\color{Goldenrod}{ ▏ }}}$ 0.8%, 10.21 KB
dist/bin/generateImportMap	${{\color{Goldenrod}{ ▏ }}}$ 0.7%, 8.38 KB
dist/database/migrations	${{\color{Goldenrod}{ ▏ }}}$ 0.7%, 8.07 KB
dist/collections/config	${{\color{Goldenrod}{ ▏ }}}$ 0.6%, 7.88 KB
dist/uploads/fetchAPI-multipart	${{\color{Goldenrod}{ ▏ }}}$ 0.6%, 7.74 KB
dist/index.js	${{\color{Goldenrod}{ ▏ }}}$ 0.6%, 7.59 KB
dist/config/orderable	${{\color{Goldenrod}{ ▏ }}}$ 0.5%, 6.27 KB
dist/collections/endpoints	${{\color{Goldenrod}{ ▏ }}}$ 0.5%, 6.00 KB
dist/config/sanitize.js	${{\color{Goldenrod}{ ▏ }}}$ 0.5%, 5.54 KB
dist/auth/strategies	${{\color{Goldenrod}{ }}}$ 0.4%, 5.50 KB
dist/auth/endpoints	${{\color{Goldenrod}{ }}}$ 0.4%, 5.42 KB
dist/utilities/telemetry	${{\color{Goldenrod}{ }}}$ 0.4%, 5.31 KB
(other)	${{\color{Goldenrod}{ ███████▊ }}}$ 31.3%, 383.88 KB

Meta file: packages/payload/meta_shared.json, Out file: esbuild/exports/shared.js

Path	Size
../../node_modules	${{\color{Goldenrod}{ ███████████████████▉ }}}$ 79.5%, 126.93 KB
dist/fields/validations.js	${{\color{Goldenrod}{ █▌ }}}$ 6.4%, 10.21 KB
dist/fields/baseFields	${{\color{Goldenrod}{ ▍ }}}$ 1.7%, 2.79 KB
dist/utilities/deepCopyObject.js	${{\color{Goldenrod}{ ▍ }}}$ 1.6%, 2.48 KB
dist/auth/cookies.js	${{\color{Goldenrod}{ ▎ }}}$ 1.0%, 1.55 KB
dist/utilities/flattenTopLevelFields.js	${{\color{Goldenrod}{ ▏ }}}$ 0.9%, 1.42 KB
dist/fields/config	${{\color{Goldenrod}{ ▏ }}}$ 0.8%, 1.28 KB
dist/utilities/flattenAllFields.js	${{\color{Goldenrod}{ ▏ }}}$ 0.6%, 943 B
dist/folders/utils	${{\color{Goldenrod}{ ▏ }}}$ 0.6%, 916 B
dist/utilities/unflatten.js	${{\color{Goldenrod}{ ▏ }}}$ 0.5%, 779 B
dist/utilities/sanitizeUserDataForEmail.js	${{\color{Goldenrod}{ }}}$ 0.4%, 713 B
dist/utilities/getFieldPermissions.js	${{\color{Goldenrod}{ }}}$ 0.4%, 651 B
dist/collections/config	${{\color{Goldenrod}{ }}}$ 0.4%, 570 B
dist/bin/generateImportMap	${{\color{Goldenrod}{ }}}$ 0.4%, 561 B
dist/auth/sessions.js	${{\color{Goldenrod}{ }}}$ 0.3%, 525 B
dist/utilities/getSafeRedirect.js	${{\color{Goldenrod}{ }}}$ 0.3%, 423 B
dist/utilities/deepMerge.js	${{\color{Goldenrod}{ }}}$ 0.3%, 413 B
dist/utilities/formatLabels.js	${{\color{Goldenrod}{ }}}$ 0.2%, 380 B
dist/utilities/appendUploadSelectFields.js	${{\color{Goldenrod}{ }}}$ 0.2%, 360 B
dist/utilities/transformColumnPreferences.js	${{\color{Goldenrod}{ }}}$ 0.2%, 348 B
(other)	${{\color{Goldenrod}{ █████▏ }}}$ 20.5%, 32.80 KB

Meta file: packages/richtext-lexical/meta_client.json, Out file: esbuild/exports/client_optimized/index.js

Path	Size
dist/features/blocks	${{\color{Goldenrod}{ ███▏ }}}$ 12.6%, 35.02 KB
dist/lexical/plugins	${{\color{Goldenrod}{ ██▊ }}}$ 11.4%, 31.68 KB
dist/lexical/ui	${{\color{Goldenrod}{ ██▏ }}}$ 8.8%, 24.36 KB
dist/features/experimental_table	${{\color{Goldenrod}{ ██▏ }}}$ 8.5%, 23.70 KB
dist/packages/@lexical	${{\color{Goldenrod}{ █▋ }}}$ 6.8%, 18.99 KB
dist/features/link	${{\color{Goldenrod}{ █▋ }}}$ 6.5%, 18.04 KB
dist/features/toolbars	${{\color{Goldenrod}{ █▌ }}}$ 6.4%, 17.75 KB
dist/features/upload	${{\color{Goldenrod}{ █▏ }}}$ 4.9%, 13.73 KB
dist/features/textState	${{\color{Goldenrod}{ █ }}}$ 4.0%, 11.08 KB
dist/features/relationship	${{\color{Goldenrod}{ ▊ }}}$ 3.2%, 8.96 KB
dist/lexical/utils	${{\color{Goldenrod}{ ▊ }}}$ 3.0%, 8.22 KB
dist/features/debug	${{\color{Goldenrod}{ ▋ }}}$ 2.7%, 7.39 KB
dist/utilities/fieldsDrawer	${{\color{Goldenrod}{ ▋ }}}$ 2.6%, 7.12 KB
dist/features/converters	${{\color{Goldenrod}{ ▋ }}}$ 2.5%, 7.04 KB
dist/lexical/config	${{\color{Goldenrod}{ ▍ }}}$ 1.8%, 5.08 KB
dist/features/lists	${{\color{Goldenrod}{ ▍ }}}$ 1.8%, 5.00 KB
dist/features/format	${{\color{Goldenrod}{ ▎ }}}$ 1.2%, 3.46 KB
dist/lexical/LexicalEditor.js	${{\color{Goldenrod}{ ▎ }}}$ 1.1%, 3.17 KB
dist/lexical/theme	${{\color{Goldenrod}{ ▏ }}}$ 0.9%, 2.62 KB
dist/features/indent	${{\color{Goldenrod}{ ▏ }}}$ 0.9%, 2.50 KB
(other)	${{\color{Goldenrod}{ █████████████████████▊ }}}$ 87.4%, 242.43 KB

Meta file: packages/ui/meta_client.json, Out file: esbuild/exports/client_optimized/index.js

Path	Size
../../node_modules	${{\color{Goldenrod}{ ████████████▍ }}}$ 49.9%, 572.85 KB
dist/elements/FolderView	${{\color{Goldenrod}{ ▋ }}}$ 2.5%, 29.18 KB
dist/elements/BulkUpload	${{\color{Goldenrod}{ ▌ }}}$ 2.4%, 27.14 KB
dist/elements/WhereBuilder	${{\color{Goldenrod}{ ▍ }}}$ 1.5%, 16.85 KB
dist/views/Edit	${{\color{Goldenrod}{ ▎ }}}$ 1.4%, 16.08 KB
dist/fields/Relationship	${{\color{Goldenrod}{ ▎ }}}$ 1.4%, 15.74 KB
dist/elements/Table	${{\color{Goldenrod}{ ▎ }}}$ 1.3%, 15.46 KB
dist/forms/Form	${{\color{Goldenrod}{ ▎ }}}$ 1.3%, 15.13 KB
dist/fields/Upload	${{\color{Goldenrod}{ ▎ }}}$ 1.2%, 14.06 KB
dist/fields/Blocks	${{\color{Goldenrod}{ ▎ }}}$ 1.2%, 13.70 KB
dist/elements/PublishButton	${{\color{Goldenrod}{ ▏ }}}$ 0.8%, 8.73 KB
dist/elements/QueryPresets	${{\color{Goldenrod}{ ▏ }}}$ 0.7%, 8.58 KB
dist/providers/Folders	${{\color{Goldenrod}{ ▏ }}}$ 0.7%, 8.49 KB
dist/elements/LivePreview	${{\color{Goldenrod}{ ▏ }}}$ 0.7%, 8.38 KB
dist/elements/ListHeader	${{\color{Goldenrod}{ ▏ }}}$ 0.7%, 7.83 KB
dist/elements/HTMLDiff	${{\color{Goldenrod}{ ▏ }}}$ 0.7%, 7.81 KB
dist/fields/Array	${{\color{Goldenrod}{ ▏ }}}$ 0.7%, 7.55 KB
dist/views/CollectionFolder	${{\color{Goldenrod}{ ▏ }}}$ 0.6%, 7.37 KB
dist/elements/ReactSelect	${{\color{Goldenrod}{ ▏ }}}$ 0.6%, 7.31 KB
dist/views/List	${{\color{Goldenrod}{ ▏ }}}$ 0.6%, 6.96 KB
(other)	${{\color{Goldenrod}{ ████████████▌ }}}$ 50.1%, 574.33 KB

Meta file: packages/ui/meta_shared.json, Out file: esbuild/exports/shared_optimized/index.js

Path	Size
dist/graphics/Logo	${{\color{Goldenrod}{ █████▋ }}}$ 22.6%, 3.12 KB
../../node_modules	${{\color{Goldenrod}{ ████▊ }}}$ 19.2%, 2.65 KB
dist/graphics/Icon	${{\color{Goldenrod}{ ██▊ }}}$ 11.0%, 1.52 KB
dist/utilities/formatDocTitle	${{\color{Goldenrod}{ ██▍ }}}$ 9.6%, 1.32 KB
dist/providers/TableColumns	${{\color{Goldenrod}{ █▌ }}}$ 6.2%, 862 B
dist/utilities/groupNavItems.js	${{\color{Goldenrod}{ █▍ }}}$ 5.9%, 814 B
dist/utilities/api.js	${{\color{Goldenrod}{ █▍ }}}$ 5.5%, 756 B
dist/elements/Translation	${{\color{Goldenrod}{ ▉ }}}$ 3.6%, 493 B
dist/utilities/handleTakeOver.js	${{\color{Goldenrod}{ ▊ }}}$ 3.2%, 440 B
dist/elements/withMergedProps	${{\color{Goldenrod}{ ▋ }}}$ 2.5%, 339 B
dist/elements/WithServerSideProps	${{\color{Goldenrod}{ ▍ }}}$ 1.7%, 232 B
dist/utilities/handleGoBack.js	${{\color{Goldenrod}{ ▎ }}}$ 1.2%, 168 B
dist/fields/mergeFieldStyles.js	${{\color{Goldenrod}{ ▎ }}}$ 1.2%, 159 B
dist/forms/Form	${{\color{Goldenrod}{ ▎ }}}$ 1.1%, 147 B
dist/utilities/abortAndIgnore.js	${{\color{Goldenrod}{ ▎ }}}$ 1.1%, 146 B
dist/utilities/hasSavePermission.js	${{\color{Goldenrod}{ ▎ }}}$ 1.0%, 136 B
dist/utilities/handleBackToDashboard.js	${{\color{Goldenrod}{ ▏ }}}$ 0.9%, 129 B
dist/utilities/findLocaleFromCode.js	${{\color{Goldenrod}{ ▏ }}}$ 0.6%, 84 B
dist/utilities/sanitizeID.js	${{\color{Goldenrod}{ ▏ }}}$ 0.6%, 77 B
dist/utilities/isEditing.js	${{\color{Goldenrod}{ }}}$ 0.4%, 59 B
(other)	${{\color{Goldenrod}{ ███████████████████▎ }}}$ 77.4%, 10.68 KB

Details

Next to the size is how much the size has increased or decreased compared with the base branch of this PR.

• ‼️: Size increased by 20% or more. Special attention should be given to this.
• ⚠️: Size increased in acceptable range (lower than 20%).
• ✅: No change or even downsized.
• 🗑️: The out file is deleted: not found in base branch.
• 🆕: The out file is newly found: will be added to base branch.