Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 11 additions & 0 deletions src/commands/add.ts
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,8 @@ import { Command } from 'commander'
import { runAdd } from '../add/add.js'
import type { AddOptions } from '../add/types.js'
import { MIN_RUNWAY_DAYS } from '../common/constants.js'
import { isCarFile } from '../utils/car-detection.js'
import { log } from '../utils/cli-logger.js'
import { addAuthOptions, addProviderOptions } from '../utils/cli-options.js'
import { addMetadataOptions, resolveMetadataOptions } from '../utils/cli-options-metadata.js'

Expand Down Expand Up @@ -29,6 +31,15 @@ export const addCommand = new Command('add')
...(dataSetMetadata && { dataSetMetadata }),
}

// Check if the file is a CAR file and warn the user
if (await isCarFile(path)) {
log.warn(
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you run this branch and do an add for a valid CAR and show us what the output looks like? then also do it and | less the output and show us that output too? I'm not convinced about the multi-line here, but also want to know that a log.warn actually does something useful.

One for @SgtPooki to consider too, is this the right way we want to output a warning in here?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For this scenario, I think we should display a spinnerSection telling the users to "use filecoin-pin import or provide paths to a non-CAR file or a directory" instead of using log.warn. log.* with the pino logger will only be visible if someone explicitly enables pino log levels.

In general, inside the CLI "UI" code, i.e. /src/commands/* we should display warnings explicitly via clack, similar to payments setup (non-auto) and potentially prompt them to confirm understanding.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should do this check from inside runAdd itself. There is already an await validatePath inside there that determines if it's a directory or a file, we could type that as "contentType" instead of a simple isDirectory boolean, and pivot display messaging there.

then we can move on to the future i'm imagining if Rod is in agreement.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fine by me

`Warning: You are adding a CAR file. Did you mean to 'import' it?
'add' wraps the file in a new UnixFS DAG.
To import existing CAR data, use 'filecoin-pin import'.`
)
}

await runAdd(addOptions)
} catch (error) {
console.error('Add failed:', error instanceof Error ? error.message : error)
Expand Down
58 changes: 58 additions & 0 deletions src/test/unit/car-detection.test.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,58 @@
import { createWriteStream } from 'node:fs'
import { unlink } from 'node:fs/promises'
import { join } from 'node:path'
import { CarWriter } from '@ipld/car'
import { describe, expect, it } from 'vitest'
import { isCarFile } from '../../utils/car-detection.js'
import { CID } from 'multiformats/cid'
import * as raw from 'multiformats/codecs/raw'
import { sha256 } from 'multiformats/hashes/sha2'

describe('isCarFile', () => {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this filename should be updated as well, likely is-car-file.test.ts

const tempDir = process.cwd()
const validCarPath = join(tempDir, 'valid.car')
const invalidCarPath = join(tempDir, 'invalid.car')
const textFilePath = join(tempDir, 'text.txt')
Comment on lines +11 to +15
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The tests here should be pretty simple and the same for each scenario. This file could be simplified a lot by doing some table-driven/data-driven test scenarios in an array of tuples, with a single it block:

const tests = [
 ['valid.car', 'returns true for valid car files', true],
 ['invalid.car', 'returns false for invalid car files', false],
 ['text.txt', 'returns false for regular files', false]
]
describe('isCarFile', () => {
  beforeAll(() => { /* set up all files */ })
  afterAll(() => { /* cleanup */ })
  for([fileName, msg, result] of tests) {
    it(msg, () => {
      // read file, call isCar, assert result
    })
  }
})


it('should return true for a valid CAR file', async () => {
// Create a valid CAR file
const { out, writer } = await CarWriter.create([
CID.create(1, raw.code, await sha256.digest(new Uint8Array([1, 2, 3])))
])
const { Readable } = await import('node:stream')
const stream = createWriteStream(validCarPath)
await new Promise<void>((resolve, reject) => {
Readable.from(out).pipe(stream)
stream.on('error', reject)
stream.on('finish', () => resolve())
writer.close()
})

expect(await isCarFile(validCarPath)).toBe(true)
})

it('should return false for a text file', async () => {
const stream = createWriteStream(textFilePath)
stream.write('This is just a text file')
stream.end()
await new Promise<void>((resolve) => stream.on('finish', () => resolve()))
Comment on lines +35 to +38
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the two instances of non-CAR could just be a plain await writeFile('...'), no need to use streams here


expect(await isCarFile(textFilePath)).toBe(false)
})

it('should return false for a random binary file', async () => {
const stream = createWriteStream(invalidCarPath)
stream.write(Buffer.from([0, 1, 2, 3, 4, 5]))
stream.end()
await new Promise<void>((resolve) => stream.on('finish', () => resolve()))

expect(await isCarFile(invalidCarPath)).toBe(false)
})

// Cleanup
it('cleanup', async () => {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This isn't a test so don't make it one, you have afterEach or afterAll to help with this; perhaps make an array within this test group, then within each test append the file, then use an afterAll to clean up.

@SgtPooki might have more creative ideas of how to structure this though.

try { await unlink(validCarPath) } catch { }
try { await unlink(textFilePath) } catch { }
try { await unlink(invalidCarPath) } catch { }
})
})
20 changes: 20 additions & 0 deletions src/utils/car-detection.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
import { createReadStream } from 'node:fs'
import { asyncIterableReader, createDecoder } from '@ipld/car/decoder'

export async function isCarFile(filePath: string): Promise<boolean> {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

file name should match exported function.. is-car-file.ts

let stream: ReturnType<typeof createReadStream> | undefined

try {
stream = createReadStream(filePath)
const reader = asyncIterableReader(stream)
const decoder = createDecoder(reader)
const header = await decoder.header()
return !!(header && header.version === 1 && header.roots)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

version can also be 2; but also it's pretty strict in there so you shouldn't need to check this output so you should be able to just await decoder.header() without bothering to look at the return value

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
return !!(header && header.version === 1 && header.roots)
return header != null

} catch (error) {
return false
} finally {
if (stream) {
stream.destroy()
}
}
}