For 5 years (1998-2003), I worked in the Advanced Computations Department (ACD) at the Stanford Linear Accelerator Center (SLAC). ACD is a group of physicists, mathematicians, and computer scientists that develops and runs high performance scientific applications. I was a software developer and spent most of my time developing high performance electromagnetic applications that are used in advanced particle accelerator design.
Tau3P
My primary responsibility was to develop high performance electromagnetic software. The code that I spent the most time working on was Tau3P, a parallel 3-D distributed-memory time domain electromagnetic solver. Tau3P uses Madsen’s Discrete Surface Integral (DSI) Method [1] on unstructured grid to solve Maxwell’s equations. Tau3P is a distributed-memory code that using MPI for process communication. We ran Tau3P primarily on our 32 processor Linux cluster and the IBM SP at NERSC.
Load Balancing
I also spent a good bit of time working on improving the performance of Tau3P by finding a more optimal mesh partitioning scheme. I collaborated with Karen Devine at SNL to integrate Zoltan into our software framework so that we could use the mesh partitioning schemes that they provided. Some of my work is summarized here. I presented this research at SIAM PP04 (pdf).
Linux Cluster
I also researched Linux clusters, focusing on efficient simulations of accelerator structures on these clusters. Here’s a picture of our SLAC (SCS, ACD, BABAR) and LBL (NERSC) collaboration in front of our first Linux cluster (back row, left to right: Bob Cook, Bill Saphir, Randy Melen, myself; front row, left to right: Paul Raines, Brian McCandless).
The second generation SLAC Linux cluster was a 32 processor VA Linux cluster connected by Myrinet. As I was leaving SLAC in 2003, we were working on the purchase of our third generation cluster. Our Linux clusters have been useful to running the small to medium side parallel problems that don’t need a supercomputer to run.
Papers
Michael Wolf, Adam Guetz, Cho-Kuen Ng, "Modeling Large Accelerator Structures with the Parallel Field Solver Tau3P," 18th Annual Review of Progress in Applied Computational Electromagnetics ACES 2002 (Monterey, CA).
V. Ivanov, C Adolphsen, N. Folwell, L. Ge, A. Guetz, Z. Li, C.-K. Ng, J.W. Wang, M. Wolf, K. Ko, G. Schussman, M. Weiner. "Simulating Accelerator Structure Operation at High Power," Proceedings of the 2003 Particle Accelerator Conference, Pages 2664-2666.
Nathaniel Folwell, Lixin Ge, Valentin Ivanov, Zenghai Li, Cho-Kuen Ng, Greg Schussman, Martin Weiner, Michael Wolf, and Kwok Ko. "Numerical Studies of Field Gradients and Dark Currents in SLAC Structures," Paper presented at International Computational Accelerator Physics Conference 2002 (East Lansing, MI).
Lie-Quan Lee, Lixin Ge, Marc Kowalski, Zenghai Li, Cho-Kuen Ng, Greg Schussman, Michael Wolf, Kwok Ko, "Solving Large Sparse Linear Systems in End-to-end Accelerator Structure Simulations," Proceedings of the 18th International Parallel and Distributed Processing Symposium (IPDPS’04) 2004.
Z. Li, N. Folwell, L. Ge, A. Guetz, V. Ivanov, M. Kowalski, L. Lee, C. Ng, G. Schussman, R. Uplenchwar, M. Wolf, and K. Ko. "X-band Linear Collider R&D in Accelerating Structures through Advanced Computing," SLAC-PUB-10563, in Proceedings of the 9th European Particle Accelerator Conference, July 2004.
References:
[1] N.K. Madsen. Divergence Preseving Discrete Surface Integral Methods for Maxwell’s Curl Equations Using Non-orthogonal Unstructured Grids, Journal of Computation Physics, 119, pp. 34-45, 1995.