Sidre Interaction with Conduit

Internally, Sidre uses the in-memory data description capabilities of the Conduit library. Sidre also leverages Conduit to facilitate data exchange, shown here applied to visualization. The following discussion gives a basic overview of Sidre capabilities when combined with Conduit.

Mesh Blueprint

The Mesh Blueprint is a data exchange protocol supported by Conduit, consisting of a properly-structured Sidre datastore hierarchy saved as an HDF5 or JSON file and a Conduit index file. The Blueprint can accomodate structured or unstructured meshes, with node- or element-centered fields. The following example shows how to create a Blueprint-conforming datastore hierarchy containing two unstructured adjacent hexahedrons with one node-centered field and one element-centered field. In the diagram, nodes are labeled in black, the node-centered field values are in blue, and the element-centered field values are in green.

../../../../_images/tiny_mesh.png

A simulation organizes Sidre data as the code design dictates. Here is a simple example data hierarchy.

../../../../_images/ds.png

Here is the code to create that example data hierarchy.

  auto ds = std::unique_ptr<sidre::DataStore> {new sidre::DataStore()};

  int nodecount = 12;
  int elementcount = 2;

  // Create views and buffers to hold node positions and field values
  sidre::Group* nodes = ds->getRoot()->createGroup("nodes");
  sidre::View* xs =
    nodes->createViewAndAllocate("xs", sidre::DOUBLE_ID, nodecount);
  sidre::View* ys =
    nodes->createViewAndAllocate("ys", sidre::DOUBLE_ID, nodecount);
  sidre::View* zs =
    nodes->createViewAndAllocate("zs", sidre::DOUBLE_ID, nodecount);

  sidre::Group* fields = ds->getRoot()->createGroup("fields");
  sidre::View* nodefield =
    fields->createViewAndAllocate("nodefield", sidre::INT_ID, nodecount);
  sidre::View* eltfield =
    fields->createViewAndAllocate("eltfield", sidre::DOUBLE_ID, elementcount);

  // Set node position for two adjacent hexahedrons
  double* xptr = xs->getArray();
  double* yptr = ys->getArray();
  double* zptr = zs->getArray();
  for(int pos = 0; pos < nodecount; ++pos)
  {
    xptr[pos] = ((pos + 1) / 2) % 2;
    yptr[pos] = (pos / 2) % 2;
    zptr[pos] = pos / 4;
  }

  // Assign a value to the node field
  int* nf = nodefield->getArray();
  for(int pos = 0; pos < nodecount; ++pos)
  {
    nf[pos] = static_cast<int>(xptr[pos] + yptr[pos] + zptr[pos]);
  }
  // and to the element field.
  double* ef = eltfield->getArray();
  // There are only two elements.
  ef[0] = 2.65;
  ef[1] = 1.96;

  return ds;

To use the Conduit Mesh Blueprint, make a group hierarchy tinymesh conforming to the Mesh Blueprint protocol. The structure of the group hierarchy is shown below (summarizing the Mesh Blueprint documentation).

First build top-level groups required by the Mesh Blueprint.

../../../../_images/cds.png
  // Conduit needs a specific hierarchy.
  // We'll make a new DataStore with that hierarchy, pointing at the
  // application's data.
  std::string mesh_name = "tinymesh";

  // The Conduit specifies top-level groups:
  sidre::Group* mroot = ds->getRoot()->createGroup(mesh_name);
  sidre::Group* coords = mroot->createGroup("coordsets/coords");
  sidre::Group* topos = mroot->createGroup("topologies");
  // no material sets in this example
  sidre::Group* fields = mroot->createGroup("fields");
  // no adjacency sets in this (single-domain) example

Add the node coordinates. The views under tinymesh will point to the same buffers that were created for the views under nodes so that tinymesh can use the data without new data allocations or data copying.

../../../../_images/cdscoords.png
  // Set up the coordinates as Mesh Blueprint requires
  coords->createViewString("type", "explicit");
  // We use prior knowledge of the layout of the original datastore
  sidre::View* origv = ds->getRoot()->getView("nodes/xs");
  sidre::Group* conduitval = coords->createGroup("values");
  conduitval->createView("x",
                         sidre::DOUBLE_ID,
                         origv->getNumElements(),
                         origv->getBuffer());
  origv = ds->getRoot()->getView("nodes/ys");
  conduitval->createView("y",
                         sidre::DOUBLE_ID,
                         origv->getNumElements(),
                         origv->getBuffer());
  origv = ds->getRoot()->getView("nodes/zs");
  conduitval->createView("z",
                         sidre::DOUBLE_ID,
                         origv->getNumElements(),
                         origv->getBuffer());

Arrange the nodes into elements. Each simulation has its own knowledge of its mesh topology. This tiny example didn’t previously encode the topology, so we must explicitly specify it.

../../../../_images/cdstopo.png
  // Sew the nodes together into the two hexahedra, using prior knowledge.
  sidre::Group* connmesh = topos->createGroup("mesh");
  connmesh->createViewString("type", "unstructured");
  connmesh->createViewString("coordset", "coords");
  sidre::Group* elts = connmesh->createGroup("elements");
  elts->createViewString("shape", "hex");

  // We have two eight-node hex elements, so we need 2 * 8 = 16 ints.
  sidre::View* connectivity =
    elts->createViewAndAllocate("connectivity", sidre::INT_ID, 16);

  // The Mesh Blueprint connectivity array for a hexahedron lists four nodes on
  // one face arranged by right-hand rule to indicate a normal pointing into
  // the element, then the four nodes of the opposite face arranged to point
  // the normal the same way (out of the element).  This is the same as for
  // a VTK_HEXAHEDRON.  See
  // https://www.vtk.org/wp-content/uploads/2015/04/file-formats.pdf.

  int* c = connectivity->getArray();

  // First hex.  In this example, the Blueprint node ordering matches the
  // dataset layout.  This is fortuitous but not required.
  c[0] = 0;
  c[1] = 1;
  c[2] = 2;
  c[3] = 3;
  c[4] = 4;
  c[5] = 5;
  c[6] = 6;
  c[7] = 7;

  // Second and last hex
  c[8] = 4;
  c[9] = 5;
  c[10] = 6;
  c[11] = 7;
  c[12] = 8;
  c[13] = 9;
  c[14] = 10;
  c[15] = 11;

Link the fields into tinymesh. As with the node positions, the views point to the existing buffers containing the field data.

../../../../_images/cdsfields.png
  // Set up the node-centered field
  // Get the original data
  sidre::View* origv = ds->getRoot()->getView("fields/nodefield");
  sidre::Group* nodefield = fields->createGroup("nodefield");
  nodefield->createViewString("association", "vertex");
  nodefield->createViewString("type", "scalar");
  nodefield->createViewString("topology", "mesh");
  nodefield->createView("values",
                        sidre::INT_ID,
                        origv->getNumElements(),
                        origv->getBuffer());

  // Set up the element-centered field
  // Get the original data
  origv = ds->getRoot()->getView("fields/eltfield");
  sidre::Group* eltfield = fields->createGroup("eltfield");
  eltfield->createViewString("association", "element");
  eltfield->createViewString("type", "scalar");
  eltfield->createViewString("topology", "mesh");
  eltfield->createView("values",
                       sidre::DOUBLE_ID,
                       origv->getNumElements(),
                       origv->getBuffer());

Conduit includes a verify method to test if the structure of the tinymesh conforms to the Mesh Blueprint. This is valuable for writing and debugging data adapters. Once the datastore hierarchy is properly structured, save it, then use Conduit to save the index file (ending with .root). This toy data set is small enough that we can choose to save it in a JSON format.

  conduit::Node info, mesh_node, root_node;
  ds->getRoot()->createNativeLayout(mesh_node);
  std::string bp_protocol = "mesh";
  if(conduit::blueprint::verify(bp_protocol, mesh_node[mesh_name], info))
  {
    // Generate the Conduit index
    conduit::Node& index = root_node["blueprint_index"];
    conduit::blueprint::mesh::generate_index(mesh_node[mesh_name],
                                             mesh_name,
                                             1,
                                             index[mesh_name]);

    std::string root_output_path = mesh_name + ".root";
    std::string output_path = mesh_name + ".json";

    root_node["protocol/name"] = "json";
    root_node["protocol/version"] = "0.1";
    root_node["number_of_files"] = 1;
    root_node["number_of_trees"] = 1;
    root_node["file_pattern"] = output_path;
    root_node["tree_pattern"] = "/";

    // Now save both the index and the data set
    conduit::relay::io::save(root_node, root_output_path, "json");
    conduit::relay::io::save(mesh_node, output_path, "json");
  }
  else
  {
    std::cout << "does not conform to Mesh Blueprint: ";
    info.print();
    std::cout << std::endl;
  }

The code listed above produces the files tinymesh.json and tinymesh.root. Any code that uses Mesh Blueprint can open and use this pair of files.

The DataStore also contains a method that can automatically generate the Blueprint index within a Sidre group rather than calling directly into Conduit. Set up a mesh similarly to the example above.

  // Conduit needs a specific hierarchy.
  // We'll make a new Group with that hierarchy, pointing at the
  // application's data.
  std::string domain_name = "domain";
  std::string domain_location = "domain_data/" + domain_name;
  std::string mesh_name = "mesh";
  std::string domain_mesh = domain_location + "/" + mesh_name;

  sidre::Group* mroot = ds->getRoot()->createGroup(domain_location);
  sidre::Group* coords = mroot->createGroup(mesh_name + "/coordsets/coords");
  sidre::Group* topos = mroot->createGroup(mesh_name + "/topologies");
  // no material sets in this example
  sidre::Group* fields = mroot->createGroup(mesh_name + "/fields");
  // no adjacency sets in this (single-domain) example

Then call the DataStore::generateBlueprintIndex() method to generate the index within a group in the datastore. Additional data needed in the root file can be added and saved using Sidre I/O calls.

  conduit::Node info, mesh_node, root_node;
  ds->getRoot()->createNativeLayout(mesh_node);
  std::string bp_protocol = "mesh";
  if(conduit::blueprint::verify(bp_protocol, mesh_node[domain_mesh], info))
  {
    std::string bp("rootfile_data/blueprint_index/automesh");

    ds->generateBlueprintIndex(domain_mesh, mesh_name, bp, 1);

    sidre::Group* rootfile_grp = ds->getRoot()->getGroup("rootfile_data");
    rootfile_grp->createViewString("protocol/name", "json");
    rootfile_grp->createViewString("protocol/version", "0.1");
    rootfile_grp->createViewScalar("number_of_files", 1);
    rootfile_grp->createViewScalar("number_of_trees", 1);
    rootfile_grp->createViewScalar("file_pattern", "bpgen.json");
    rootfile_grp->createViewScalar("tree_pattern", "/domain");
    rootfile_grp->save("bpgen.root", "json");

    ds->getRoot()->getGroup("domain_data")->save("bpgen.json", "json");
  }
  else
  {
    std::cout << "does not conform to Mesh Blueprint: ";
    info.print();
    std::cout << std::endl;
  }

Additionally, the Sidre Parallel I/O (SPIO) class IOManager provides a method that both generates a Blueprint index and adds it to a root file. Using the same mesh data from the last example, first write out all of the parallel data using the IOManager::write() method. This will output to files all of the data for all domains, and will also create a basic root file. Then the IOManager::writeBlueprintIndexToRootFile() methods can be called to generate the Blueprint index and add it to the root file. This is currently only implemented to work with the sidre_hdf5 I/O protocol.

  sidre::IOManager writer(MPI_COMM_WORLD);

  conduit::Node info, mesh_node, root_node;
  ds->getRoot()->createNativeLayout(mesh_node);
  std::string bp_protocol = "mesh";
  if(conduit::blueprint::mpi::verify(bp_protocol,
                                     mesh_node[domain_mesh],
                                     info,
                                     MPI_COMM_WORLD))
  {
  #if defined(AXOM_USE_HDF5)
    std::string protocol = "sidre_hdf5";
  #else
    std::string protocol = "sidre_json";
  #endif
    std::string output_name = "bpspio";
    if(comm_size > 1)
    {
      output_name = output_name + "_par";
    }

    std::string bp_rootfile = output_name + ".root";

    writer.write(ds->getRoot()->getGroup(domain_location), 1, output_name, protocol);

    writer.writeBlueprintIndexToRootFile(ds, domain_mesh, bp_rootfile, mesh_name);
  }

Data Visualization

The VisIt tool can read in a Conduit Mesh Blueprint conforming file, interpret the index file, and sensibly display the data contained in the data file. Starting from version 2.13.1, VisIt can open a .root file just like any other data file. VisIt produced the following image from the Mesh Blueprint file saved above.

../../../../_images/tiny_mesh_rendered.png

Conduit is also a foundational building block for the Ascent project, which provides a powerful in situ data analytics and visualization facility (without copying memory) to distributed-memory simulation codes.