“XANSONS for COD” (http://xansons4cod.com) is a new BOINC project aimed at creating the open-access database of simulated x-ray and neutron powder diffraction patterns for nanocrystalline phase of materials from the collection of the Crystallography Open Database (COD). The project uses original open-source software XaNSoNS to simulate diffraction patterns on CPU and GPU. This paper describes the scientific problem this project solves, the project’s internal structure, its operation principles and organization of the final database.
Crystallography Open Database (COD)  is an open-access database which contains the information about the crystal structure of inorganic and organic materials excluding biopolymers. These data were obtained using various experimental methods but in the first place the x-ray, neutron and electron diffraction on single crystals or powder samples. Usually, a crystallite with a size of at least few micrometers is required to accurately derive the unit cell properties from its diffraction pattern. When determining the structural content of a powder sample, the presence of already known phases should be checked first. There is a web application called “Full Profile Search Match”  that performs fitting of the experimental x-ray powder diffraction pattern with the simulated ones using the data from COD. However, these simulated powder diffraction patterns are obtained for the crystallites of infinite size. This approximation can be applied to the microcrystalline samples with high accuracy, but for the nanocrystalline samples, one need to adjust for the finite size of the crystallites. The Gaussians or other broadening functions are used for that purpose to broaden the delta-functional Bragg peaks. The same broadening functions are used to adjust for the finite resolution of the measurement device. The artificial broadening of the simulated diffraction peaks is not suitable for determining the sizes of the crystallites in the nanocrystalline sample, which is usually required in addition to determining the structural content. This is so because the function that describes the dependence of the width of the diffraction peaks on the linear size of the crystallite is not universal and depends on the structure of the material. The alternative approach is to use powder diffraction patterns simulated for the nanocrystallites of various sizes. This approach, however, requires calculating the diffraction patterns using the general equation (namely, the well-known Debye equation) without infinite crystal approximation. This is time consuming but this is something that distributed computing can help with. Using the Debye equation have additional benefits such as accounting for lattice defects (e.g., site vacancies, atom replacements and displacements) in a natural way. Many structures from the COD have such defects.
The open-access database of simulated x-ray and neutron powder diffraction patterns of the nanocrystalline materials will be useful for the diagnostics of these materials. It may complement the Full Profile Search Match method in the analysis of the sizes of the crystallites in the sample. The “XANSONS for COD” (http://xansons4cod.com) project is aimed at creating such database for most of entries of the COD using volunteer computing. The project is powered by BOINC . To simulate powder diffraction patterns, it uses the original open source software XaNSoNS (X-ray and Neutron Scattering on Nanoscale Structures) . XaNSoNS supports parallel CPU computing with OpenMP and GPU-computing with Nvidia CUDA and OpenCL making it possible to utilize the power of Nvidia, AMD and Intel GPUs to simulate the diffraction patters. XaNSoNS is a cross-platform application, so the BOINC project has the client applications for Linux, MacOS and Windows.
The paper is organized as follows. Section 2 provides information on what exactly is calculated in this project and how. Section 3 gives a look at the internal structure of the project. The results of alpha-testing and the current state of the project are given in Section 4. Section 5 describes the structure of the database with diffraction patterns. Conclusions are given in Section 6.
2 Calculating powder diffraction patterns
In the “XANSONS for COD” project, x-ray and neutron powder diffraction patterns are calculated for two different types of model samples: (a) isolated spherical crystalline particle of a given size (diameter), D, (b) solid crystalline material with long-range order broken on the distances greater than a given value, D. These two samples are referred as isolated particle and solid material below. For both cases, the value of D varies in the range from 6 nm to 30 nm with the step of 3 nm resulting in 18 powder diffraction patterns in total for a given structure from the COD and for a given type of incident radiation source (x-ray or neutron). Notwithstanding that the shape of the nanocrystals affects the diffraction pattern (especially at small scattering angles), the variation of crystallite shape in addition to the variation of its size is out of the scope of this project. Note that during the alpha-testing of the project, which took place in November 2016, only isolated particles were considered.
For both types of model sample, the calculation algorithm is the same and consists of 3 steps. Each step, however, performs slightly different for an isolated particle and a solid material. These steps are:
calculating the atomic ensemble using the lattice parameters and unit cell configuration extracted from the Crystallography Information File (CIF),
calculating the histogram of interatomic distances for this atomic ensemble,
calculating the scattering intensity using the histogram of interatomic distances.
The step 1 performs on a CPU and is not parallelized. It has computational complexity O(N), where N is the total number of atoms in atomic ensemble. For both types of model samples, the atomic ensemble is a spherical nanoparticle of radius R. While for an isolated particle, R equals to D/2, for a solid material, R equals to RBS + D, where RBS is the radius of “bounding sphere” (by analogy with bounding box). The radius of bounding sphere is set to 6 nm for all values of D. Creating such a big atomic ensemble for a solid material sample guarantees that any atom from the bounding sphere is surrounded by other atoms on any distance less than D. This emulates the periodic boundary conditions which are not supported in XaNSoNS.
The step 2 has complexity O(N2) for an isolated particle and O(N·NBS) for a solid material, where NBS is the number of atoms in the bounding sphere. The width of histogram bin, Δbin, is set to 0.001Å (see Eq. (2) in ). This step is parallelized with OpenMP in the CPU version of the application. In the CUDA and OpenCL versions it performs on a GPU. Since the total number of atoms in the isolated particle or in the correlation sphere could be up to 107, the histogram of interatomic distances must be a 64-bit integer array. Not being a problem when computing on a CPU, this becomes the main factor affecting the performance when computing on a GPU because each read-modify-write (RMW) operation applied to this array must be synchronized between all parallel threads. For AMD and Nvidia GPUs, atomic operations on 64-bit integers are used for synchronized writing in GPU memory. These operations are not supported by Intel GPUs and therefore 32-bit atomic operations, performed separately for two halves of a 64-bit integer, are used for Intel. The GPU-specific implementation of the atomic operations and the GPU memory throughput strongly influence the performance. Despite all the limitations, calculation of the histogram of interatomic distances performs 20-40 times faster on high-end GPU than on high-end CPU (if all CPU cores are loaded).
The computational complexity of step 3 is O(Nbin · NDP), where Nbin is the number of histogram bins and NDP is the resolution of diffraction pattern. This step is also parallelized in the CPU version of the application and performs on a GPU in the CUDA and OpenCL versions. For an isolated particle, Eq. (3) from  is used to calculate the scattering intensity. For a solid material, this equation is modified as follows:
where S is the scattering intensity, q is the scattering vector magnitude, ρ is the average atomic density of the model sample, Rcut = D is the cutoff radius (the longest distance at which the atoms are still correlated), fav is the average atomic form-factor of the model sample, eli and elj are the indexes of chemical elements, Nel is the total number of different chemical elements in the model sample, feli is the atomic from-factor of the chemical element with the index eli,
is the total number of atoms of the element eliin the bounding sphere, Heli, elj is the histogram of interatomic distances, k is the index of histogram bin, and rk = (k + 0.5) Δbin.
is the weight coefficient that smoothly suppresses the contribution of long-range interatomic correlations. This weight coefficient is also used in . The effect of this coefficient on the scattering intensity is shown in Figure 1 in . The first term in (1) describes the contribution of a continuous sphere with atomic density
and radius Rcut.
The behavior of simulated powder diffraction patterns for these two types of the sample is different at small values of q. Notice that the large sinusoid seen on the plot “a” is gone on the plot “b”. The diffraction peaks are always broader for the isolated nanoparticles of the diameter D than for the material with interatomic distances cutoff at the Rcut = D. Artificially cutting off the interatomic distances results in negative values of scattering intensity at some values of q. However, this should not significantly reduce the accuracy of interpretation of experimental data where the simulated diffraction patterns are used.
3 Internal structure of “XANSONS for COD”
The modular structure of the original application simplified developing of the BOINC application. The original application consists of the wrapper written in Python and the main module written in C/C++. The wrapper implements the API which allows to use XaNSoNS in Python applications. Figure 2 shows side-to-side the structures of original XaNSoNS application and the BOINC application based on it.
The wrapper was transformed into the work generator. It works as a periodic task and has the following algorithm:
getting the numbers of unsent and inactive results for CPU and GPU applications,
from the results database,
calculating (based on the values of
the number of COD structures, NS, for which the new workunits (WU) will be created,
downloading the CIF files for selected NS structures from the COD,
importing the required data from CIF files to XML files readable by client application,
creating atomic form-factor tables for selected NS structures,
specifying the sizes of nanocrystallites and additional parameters of the structures,
staging input files (XML files and form-factor tables) with the stage_file command,
creating WUs for CPU and GPU applications with the create_work command.
Steps 3 and 4 are the most time-consuming among all the steps because some CIF files can be up to several MB in size. Despite this, during the alpha-testing it was found that this work generator can maintain the required number of jobs in the queue. The work generator does not calculate the atomic form-factor in step 5, but uses the available data from the Periodictable Python library (http://www.reflectometry.org/danse/elements.html) instead. This step is carried out fast, since the number of different chemical elements (or ions) in the structure rarely exceeds ten. For a given COD structure for a given type of the radiation source, 18 WUs are created in total. For an isolated particle, the time required to complete the WU, tWU ≈ a · D6 + bNbin · NDP, while for a solid material, tWU ≈ c · (RBS (D+RBS))3 + bNbin · NDP .Here, a, b and c are constant parameters. If D > 12 nm for an isolated particle and D > 6 nm for a solid material, the value of tWU is determined by the first term of equation. Since D varies by a factor of 5 (from 6 nm to 30 nm), tWU varies by hundreds of times. The three smallest WUs are always sent to the queue for CPU application, while the seven largest are always sent to the queue for GPU application. The remaining 8 WUs are distributed between the CPU and GPU queues depending on the values of
All the diffraction patterns are calculated for the values of q in the range from 1 nm–1 to 81.6 nm–1 and resolution NDP = 4096 points (~50.8 points per nm–1). The value q = 81.6 nm–1 correspond to the scattering angle (2θ) of π for CuKα x-ray radiation source with 0.154 nm wavelength.
Built-in daemon script_validator is used to validate the results. It calls init_results.py and compare_results.py Python scripts. Init_results.py mark the result as valid if stdout does not contain error messages, and all the values (for an isolated particle) or at least 90% of the values (for a solid material) of the scattering intensity array are positive. If two successful results from two different hosts are obtained for the same WU, these results are compared with Compare_results.py scripts. The results are marked as valid if the following condition is satisfied:
where S1 and S2 are the scattering intensities calculated by different hosts for the same WU. The tolerance value T takes into account possible randomization of the calculated scattering intensity caused by the Monte-Carlo method used in XaNSoNS to simulate defects in the crystal structure such as replacements and displacement of atoms. These defects are described in the input data with site occupancy parameters and atomic displacement parameters respectively. Different runs of XaNSoNS generate slightly different atomic ensembles for the same input data if these defects are present. This in turn leads to slightly different diffraction patterns obtained from the two different runs (see Section 3 and Figure 3 in  for the details). The maximum allowed value of T =0.05 was obtained experimentally by processing different samples of data from the COD before the launch of the BOINC project. Fortunately, various failures during the computation usually result in much larger values of T and are separated by the condition (2).
The assimilator consists of two parts. The first part is the built-in sample_assimilator daemon. The second part is the periodic task (Python script) that compresses all diffraction patterns calculated for the same structure and the same radiation source into a single archive file which is stored in the final database then.
4 Results of alpha-testing and current state of the project
The alpha-testing of the project took place in November 2016. 126 volunteers participated in the test donating the power of 307 PCs (only the volunteers granted with credits are counted). The BOINC wrapper was used to run the client application during the alpha-testing. About 5.6% of COD entries (i.e. 20500 structures) were processed. The diffraction patterns were calculated only for the isolated nanoparticles. Below are the main results.
A few critical bugs in the client application were fixed thanks to volunteers who participated in debugging both on the project website and on http://boinc.ru. At the end of the test, all versions of the client application were sufficiently stable: only about 1% of the jobs returned errors.
Intel GPUs below 8-th generation and all integrated AMD GPUs tested in the project experienced significant performance issues. These issues were probably caused by specific implementation of atomic operations on these GPUs and could not be fixed by updating the software.
Due to the large number of small jobs, the server load was very high. The test allowed to determine the optimal parameters of the virtual server: 2 cores of Intel Xeon E5-2666, 4 GB RAM, 500 GB SSD storage with read/wright throughput of 1500 IOPS, network load: 60 MB/s.
The processing power donated by volunteers was much higher than expected. This allowed to increase the upper limit of crystallite sizes from 21 nm to 30 nm.
During the long-term maintenance, the new native versions of the client application were developed. The simulation of the powder diffraction patterns of the solid crystalline material with broken long-range order was implemented. The calculations for the project were relaunched on June 18th, 2017. Since then, additional 848 volunteers joined the project (only the volunteers granted with credits are counted). All COD entries available at the moment were processed by October 10, 2017. Since the new structures are added to COD every day, it is important to keep the database of diffraction patterns up to date. The rate with which COD is updated (about 1500 new structures per month) is not enough to keep constantly busy even 10 hosts, therefore the new jobs will be created in batches about twice a month starting from late November 2017.
5 Database of powder diffraction patterns
The open access database of calculated powder diffraction patterns is not finished yet. This database is hosted on a separate server and accessible by the url: http://database. xansons4cod.com. Since the COD contains all necessary information about the structures, there is no need to duplicate it in this database. The database schema is shown in Figure 3. The field “source_id” there refers to the type of incident radiation source (x-ray or neutron), the field “sample_id” refers to the type of the model sample (isolated particle or solid material), the field “size_min” contains the minimal size of the crystalline (it equals to 6 nm for the most structures, but may be larger for the structures with large unit cells).
The web interface to the database is very simple. The structure can be searched by its COD ID number. The structure’s webpage contains interactive charts showing simulated powder diffraction patterns, links to the data files (txt) with these diffraction patterns and the list of volunteers who contributed to this page by donating their computing power. The screenshot of the webpage for the structure with COD ID 4111341  is shown in Figure 4. The user can zoom in/out the diffraction patterns and show/hide the curves by clicking on the respective entries in the legend. By default, only the diffraction patterns for solid materials for 6, 12, 18, 24 and 30 nm samples are shown. Note that the limits of y-axis are set in such a way as to show the main features of diffraction pattern, therefore for the single particles the large sinusoid at small values of q is cropped the same way as in Figure 1.
In its present form the database does not allow for automated processing of the experimental data similar to Full Profile Search Match  but for nanocrystalline samples. In the future we will try to implement the data processing service which uses the results obtained with  as the initial guess to refine the structural composition of the sample and estimate the sizes of crystallites in it.
The successful operation of the “XANSONS for COD” BOINC project shows that the creation of the open-access database of simulated x-ray and neutron powder diffraction patterns of nanocrystalline materials is possible using resources of volunteer computing. Thanks to the power of GPU, all entries of the Crystallography Open Database were processed in less than 4 months. The final database is under development and will be completed by the end of 2017. We hope that this database will be useful for the diagnostics of nanocrystalline materials.
The authors would like to thank Alexander Kukushkin and Mikhail Posypkin for helpful discussions, all the volunteers participating in the “XANSONS for COD” BOINC project and the creators and maintainers of the Crystallography Open Database for making this project possible. This work is supported by the Russian Foundation for Basic Research (project RFBR #15-07-07901-a).
 Gražulis S., Chateigner D., Downs R.T., Yokochi A.F.T., Quirós M., Lutterotti L., Manakova E., Butkus J., Moeck P., Le Bail A., Crystallography Open Database - An open-access collection of crystal structures, J. Appl. Cryst. 2009, 42, 726–729, 10.1107/S0021889809016690, http://crystallography.net/cod/. Search in Google Scholar
 Anderson D.P., BOINC: A System for Public-Resource Computing and Storage, Proceedings of the 5th IEEE/ACM International Workshop on Grid Computing, IEEE Computer Society, Washington, DC, USA, 2004, 4–11, 10.1109/GRID.2004.14. Search in Google Scholar
 Neverov V.S., XaNSoNS: GPU-accelerated simulator of diffraction patterns of nanoparticles, SoftwareX, 2017, 6, 63–68, 10.1016/j.softx.2017.01.004, https://gitlab.com/vsnever/XaNSoNS/. Search in Google Scholar
 Lin Z., Zhigilei L.V., Time-resolved diffraction profiles and atomic dynamics in short-pulse laser-induced structural transformations: Molecular dynamics study, Phys. Rev. B, 2006, 73, 184113, 10.1103/PhysRevB.73.184113. Search in Google Scholar
 Bentrup U., Le Bail A., Duroy H., Fourquet J.L., ChemInform Abstract: Polymorphism of CsAlF4. Synthesis and Structure of Two New Crystalline Forms, Physical Inorganic Chemistry, 1992, 23(29), 10.1002/chin.199229001 Search in Google Scholar
 Tong R., Valentine J.C., McDonald F.E., Cao R., Fang X., Hardcastle K.I., Total Syntheses of Durgamone, Nakorone, and Abudinol B via Biomimetic Oxa- and Carbacyclizations, J. Am. Chem. Soc., 2007, 129 (5), 1050–1051, 10.1021/ja068826+. Search in Google Scholar
© 2018 V. S. Neverov and N. P. Khrapov
This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 License.