can we reuse CAS directory in filesystem_store across working directory

[zpzjzj]

Thanks for the opt on evicting_map .
We used v0.7.5 for remote-execution, found that fetch time is improved but still cost a lot comparing to execution time even when filesystem store is mostly hit, especially in cases which needs many inputs like linking and testing (We do Python remote test on pods with GPU).

We used Buildfarm years ago, it will create a directory with same prefix of its digest, and will be hardlinked into working directory if needed. To make sure the directory is not changed, it will set directory as readonly, which can make some tests trying to write on local directory fail but may reduce the cost of creating directory in each job.

I'll try to collect metrics in fetch stage later, produce data of (time of reconstructing directory) / ( time of fetch across threads)

[MarcusSorealheis]

Thanks for the opt on evicting_map .

Yw. In general, my hope is to move expensive outside of locks but there are a few other areas where I need to dig in. We can be better. For example, we could parallelize directory construction, but that could also cause other issues.

As for Buildfarm, I am familiar with that approach from a while and we should probably have done something similar.

In five or six hours, I should have a branch up that you could try as well.

[MarcusSorealheis]

ok @zpzjzj You can try this branch out. I've tested it with a few projects. My expectation is that it will almost certainly be faster. I don't have your source code or all the answers about your infrastructure, but I have some limited familiarity with this problem.

[MarcusSorealheis]

@zpzjzj could you confirm that my PR solved your problem? I Didn't mean to close this issue without your confirmation.

[MarcusSorealheis]

to use the directory cache, you can add it as directory_cache to the config like so:

{
  workers: [
    {
      local_worker: {
        worker_api_endpoint: {
          uri: "grpc://127.0.0.1:50061",
        },
        cas_fast_slow_store: "CAS_MAIN_STORE",
        upload_action_result: {
          ac_store: "AC_MAIN_STORE",
        },
        work_directory: "/tmp/nativelink/work",
        
        // Add this section to enable directory caching
        directory_cache: {
          // Maximum number of cached directories (default: 1000)
          max_entries: 1000,
          
          // Maximum total cache size (default: 10GB)
          // Supports human-readable sizes: "10GB", "1TB", etc.
          max_size_bytes: "10GB",
          
          // Cache storage location (optional)
          // Default: {work_directory}/../directory_cache
          cache_root: "/tmp/nativelink/directory_cache",
        },
      },
    },
  ],
  // ... rest of config
}

[MarcusSorealheis]

@zpzjzj I think this branch should work Ping. Let me know if not.

[zpzjzj]

Many thanks for your swift and effective work on this issue — much appreciated!
I’m tied up with a few tasks right now, but I’ll make sure to try out this branch soon.