pgII data post processing for use in Window 6 and other software

Once data is measured and stored in the PabOpto MySQL database it has to be converted to a format that is readable for Window 6.
Code will be shared when it is more stable, general ideas and pitfalls are described for now.
All our processing is carried out using MATLAB programs which is readable for people with C/FORTRAN/Java or similar language skills.

Process overview

The current process has quite a few steps, and a brief overview is in order before a step-by-step description follows. Each line leads to the next and the last column describes how many unique calls to the "function" (or process) must be done.
Data In
Function
Data out
Number of instances
MySQL database
Webinterface
filename_thetain_phiin.txt
4 channels*9 angles of incidence = 36
textfile, each row theta phi BSDF
pubload.m
4 MATLAB variables, thetai, phii, BRDF and BTDF
36*2 (BTDF and BRDF split) = 72
One x*3 matrix containing either a BTDF or BRDF for a single angle of incidence. x typically >100000
points2klems.m
1x145 vector outgoing values according to Klems coordinate system
72
9 of the 1x145 vectors to generate a BTDF for all incident theta
mergeAoi.m
9x145 matrix. Actually 145x145 matrix with 9 of the 145 incident angles correct. 136 incident directions empty
8 (4 channels* BRDF/BTDF)
9x145 matrix
fillisometric.m
145x145 matrix with each theta isotropic
8
8 of the 145x145 matrices
saveW6fourband.m
1 XML file with 4 bands of both BRDF and BTDF. But equal for both facings.
1
The final file acutally contains both internal and external BRDF for each band. For samples with different front and back the number of instances is doubled.

0. Data formats

A single BSDF value is commonly given using four angle indices obtained for each combination of incident and outgoing angle pairs (theta and phi).
Window 6 XML datafiles typically uses a coordinate system designed by Joe Klems, having 9 theta bands with a varying number of phi values for each band.
The total number of directions in the Klems coordinate system is 145, resulting in a total of 145x145 unique BSDF values for each wavelength band.

1. Downloading files from the SQL database

This is very straight-forward just click download.txt for all the files you need.

2. points2klems.m

Let us begin here even though this function is not the first in the chain, it is the core of the process and everything else is more or less book-keeping, extrapolation, and formatting.
  This function starts with all the points for a single BTDF or BRDF (N.B. the filename_thetain_phiin.txt contains both).
  A Voronoi diagram is generated in a Carthesian coordinate system (the theta and phi variables are projected down on the sample p lane) associating an area with each measured data point (basica lly a nearest neighbor approximation but moving from points to areas).  The image to the right (full sized images available at the bottom of the page) shows a random 10000 points (less than 10% of a typical data file) and the patches of the Klems coordinate system outlined in yellow. The dark points outside the unit circle are mirrored points that are inserted to guarantee that no polygons inside the unit circle are open ended. This image is from an old prototype and mirroring is done in a smarter way in the current version of the function.
 
The BSDF for each patch in the Klems coordinate system is calculated by adding up the areas of the measured points. The image to the left is a zoom on a patch in the whole showing how polygons around  data points are in several of the Klems patches. Contributions are obviously weighted by the measured data value, but also by the size of the area and the cos(theta) term for that area due to the nature of the BSDF. Points outside of the patch can still contribute since the the area contribution is based on the intersection of the data point area and the patch in the Klems.
  The core function in this part is obviously the creation of the Voronoi diagram. MATLAB has built in functions, for other platforms I recommend starting at http://www.qhull.org.

3. pabload.m

The text files from the mySQL database contain some text information. This function basically grabs the BSDF data and sorts it based on theta so that BRDF and BTDF values can be separated. This moves from the pgII coordinate system to the Klems coordinate system.

A future project is to replace this with direct calls to the database instead of accessing text files.

4. mergeAoI.m

points2klems returns 145 values for one given angle of incidence. This function merges 9 such sets as a step towards filling the whole matrix.

5. fillIsometric.m

This function assumes that the data is isometric and extrapolates the 9 measured sets accordingly.

6. saveW6fourbands.m

This only combines the filled matrices and writes them to a text file with the correct Window 6 XML tags.

Conclusion

These functions are all called from a single function that only asks for the sample name part of the filename and a directory where 36 such files are stored. It then uses the ordering of the theta values and ids to figure out which belongs to which channel.

Status update

Sep 25, 2009: The chain is running, but a lot is not properly validated. A lot of the code contains more DEBUG code than actual code. Source will be shared as confidence in functionality grows.
ą
Unknown user,
Sep 25, 2009, 11:33 AM
ą
Unknown user,
Sep 25, 2009, 11:33 AM
Comments