Added a simple btd example

[guj]

This is a simple version of BTD. While writing this, it looks like something unexpected is observed.

to reproduce:

1. compile and run 16_btd_write_parallel
2. use more than one rank (2 is ok)
3. use -s (span) with -b (mpi barrier before flush)
4. run "mpirun -n 2 16_btd_write_parallel -s " works, however "mpirun -n 2 16_btd_write_parallel -s -b " hangs. It is not able to finish through the storeChunk call.

@franzpoeschel if you have time, please verify.

[franzpoeschel]

Yep, I see the hangup, too. Will have a look.

[franzpoeschel]

Hey Junmin, the issue in here is that the Span-based storeChunk API is collective, which was not properly documented so far. Fixing that with #1794.

For your code, this means that all ranks must call storeChunk() when using the Span API. The following patch should do that:

diff --git a/examples/16_btd_write_parallel.cpp b/examples/16_btd_write_parallel.cpp
index a7b420ba2..65a602d94 100644
--- a/examples/16_btd_write_parallel.cpp
+++ b/examples/16_btd_write_parallel.cpp
@@ -204,6 +204,10 @@ void doWork(
                 input.get(), input.get() + numElements, spanBuffer.data());
         }
     }
+    if (m_span)
+    {
+        mymesh.storeChunk<double>({0, 0, 0}, {0, 0, 0}).currentBuffer();
+    }
 }
 
 int main(int argc, char *argv[])

[guj]

Hey Junmin, the issue in here is that the Span-based storeChunk API is collective, which was not properly documented so far. Fixing that with #1794.

For your code, this means that all ranks must call storeChunk() when using the Span API. The following patch should do that:

diff --git a/examples/16_btd_write_parallel.cpp b/examples/16_btd_write_parallel.cpp
index a7b420ba2..65a602d94 100644
--- a/examples/16_btd_write_parallel.cpp
+++ b/examples/16_btd_write_parallel.cpp
@@ -204,6 +204,10 @@ void doWork(
                 input.get(), input.get() + numElements, spanBuffer.data());
         }
     }
+    if (m_span)
+    {
+        mymesh.storeChunk<double>({0, 0, 0}, {0, 0, 0}).currentBuffer();
+    }
 }
 
 int main(int argc, char *argv[])

Oh, I did not know that.
Then why is the barrier call cause the hanging? while without the barrier call, store chunk with span succeeded?

[franzpoeschel]

Hey Junmin, the issue in here is that the Span-based storeChunk API is collective, which was not properly documented so far. Fixing that with #1794.
For your code, this means that all ranks must call storeChunk() when using the Span API. The following patch should do that:

diff --git a/examples/16_btd_write_parallel.cpp b/examples/16_btd_write_parallel.cpp
index a7b420ba2..65a602d94 100644
--- a/examples/16_btd_write_parallel.cpp
+++ b/examples/16_btd_write_parallel.cpp
@@ -204,6 +204,10 @@ void doWork(
                 input.get(), input.get() + numElements, spanBuffer.data());
         }
     }
+    if (m_span)
+    {
+        mymesh.storeChunk<double>({0, 0, 0}, {0, 0, 0}).currentBuffer();
+    }
 }
 
 int main(int argc, char *argv[])

Oh, I did not know that. Then why is the barrier call cause the hanging? while without the barrier call, store chunk with span succeeded?

Okay, I take it back, the Span API can be used non-collectively, but there is currently a bug in Iteration::open(), fixed by #1794.
So you can roll back the last commit and use the fix on #1794 instead.

[guj]

Hey Junmin, the issue in here is that the Span-based storeChunk API is collective, which was not properly documented so far. Fixing that with #1794.
For your code, this means that all ranks must call storeChunk() when using the Span API. The following patch should do that:

diff --git a/examples/16_btd_write_parallel.cpp b/examples/16_btd_write_parallel.cpp
index a7b420ba2..65a602d94 100644
--- a/examples/16_btd_write_parallel.cpp
+++ b/examples/16_btd_write_parallel.cpp
@@ -204,6 +204,10 @@ void doWork(
                 input.get(), input.get() + numElements, spanBuffer.data());
         }
     }
+    if (m_span)
+    {
+        mymesh.storeChunk<double>({0, 0, 0}, {0, 0, 0}).currentBuffer();
+    }
 }
 
 int main(int argc, char *argv[])

Oh, I did not know that. Then why is the barrier call cause the hanging? while without the barrier call, store chunk with span succeeded?

Okay, I take it back, the Span API can be used non-collectively, but there is currently a bug in Iteration::open(), fixed by #1794. So you can roll back the last commit and use the fix on #1794 instead.

I just checked, and it works. will you merge your fix?

[franzpoeschel]

I have merged it now, you can rebase this branch

[guj]

@ax3l @franzpoeschel can we merge this one?

[guj]

@franzpoeschel I adjusted the examples, and looks like it is worth to tryout the following commands:
mpirun -n 4 ./bin/16_btd_write_parallel -s -v -d 4

It will finish successfully (h5 or bp) however, if you try "h5ls -r" for each of the h5 files, not all of them show datasets. And those files are not readable by h5dump.

mpirun -n 4 ./bin/16_btd_write_parallel -s -v -d 0
This gives good h5 files.
The difference is with -d 0, basically a flush is called after each store chunk. Otherwise it will call flush after a few store chunk calls.
I believe this has something to do with the umbrella span storechunk() support. I followed the pointers they are valid up to H5Dwrite. So not sure what the root cause is.

To reduce run time, you can change line 65 to use one field "B" instead of three fields "B" "j" "E".

[ax3l]

You could store the series just as a variable and pass it around by reference, but ok so as well if you try to emulate some pattern we use in codes?

[guj]

As it gets created and destroyed in main(), this way is cleaner to me

[ax3l]

@franzpoeschel should we use

Suggested change

	series->iterations[w.whichSnapshot].open();
	series->snapshots()[w.whichSnapshot].open();

now?

[franzpoeschel]

Ideally, yes

[guj]

for writing, more straightforward to use iterations[]?

[ax3l]

@franzpoeschel can you please take a look at the flush call issues described by @guj?

[ax3l]

@franzpoeschel should we add a helper function to calculate he number of elements from chunk_offset and chunk_extent (as used in storeChunk below)?

Something like

std::size_t numElements = openPMD::count_chunk_elements(chunk_offset, chunk_extent);

[franzpoeschel]

This probably depends on chunk_extent only? But yeah, I have manually written that often enough, a helper would be nice.

[ax3l]

Hm, not sure what I meant here, this is independent of the offset...

[guj]

This is a simple version of BTD. While writing this, it looks like something unexpected is observed.

to reproduce:

1. compile and run 16_btd_write_parallel
2. use more than one rank (2 is ok)
3. use -s (span) with -b (mpi barrier before flush)
4. run "mpirun -n 2 16_btd_write_parallel -s " works, however "mpirun -n 2 16_btd_write_parallel -s -b " hangs. It is not able to finish through the storeChunk call.

@franzpoeschel if you have time, please verify.

just verified that this bug is gone

[guj]

@franzpoeschel I adjusted the examples, and looks like it is worth to tryout the following commands: mpirun -n 4 ./bin/16_btd_write_parallel -s -v -d 4

It will finish successfully (h5 or bp) however, if you try "h5ls -r" for each of the h5 files, not all of them show datasets. And those files are not readable by h5dump.

mpirun -n 4 ./bin/16_btd_write_parallel -s -v -d 0 This gives good h5 files. The difference is with -d 0, basically a flush is called after each store chunk. Otherwise it will call flush after a few store chunk calls. I believe this has something to do with the umbrella span storechunk() support. I followed the pointers they are valid up to H5Dwrite. So not sure what the root cause is.

To reduce run time, you can change line 65 to use one field "B" instead of three fields "B" "j" "E".

Also I retried this one.
---- file 1 & 3 has metadata error: (only has info of B fields, not E/j)

h5dump -H ../samples/16_btd_0000003.h5
h5dump error: internal error (file h5dump.c:line 1440)
h5ls -r ../samples/16_btd_0000003.h5
/ Group
/data Group
/data/3 Group
/data/3/fields Group
/data/3/fields/B Group
/data/3/fields/B/x Dataset {192, 64, 64}
/data/3/fields/B/y Dataset {192, 64, 64}
/data/3/fields/B/z Dataset {192, 64, 64}

[guj]

@franzpoeschel can you please take a look at the flush call issues described by @guj?

The branch is up to date, and you can reproduce h5 issue with: " mpirun -n 4 ./bin/16_btd_write_parallel -v -s" which invokes span, and files 0 2 are good but files 1 & 3 are not. " mpirun -n 3 ./bin/16_btd_write_parallel -v -s" will be ok, as well as " mpirun -n 4 ./bin/16_btd_write_parallel -v -s -t 3". In the later two cases, rank 0 processed the first buffer of all files. Not sure why that makes difference in span call of h5 files. If needed, I am happy to enforce this test to use span only with bp. what do you think?