There's no lack of mailing list threads out there on the subject. This time it was on the mpi4py mailing list, where the feeling was that nobody has really catered to the "HPC" or "large cluster" segment. Rather than plunging ahead and develop yet another scientific Python distribution or set of standards for another special case, let's take a breath and make sure we understand the full problem first. Why aren't the current solutions working, and what do we really want?
Goals:
- Avoid redoing the mistakes of the past
- Save time by pooling our efforts
- Take a step towards more reproducible research
- Make scientific Python more attractive
Round 1: Getting to know each other + surveying problems and solutions
Everybody can participate (HPC or desktop, Python or Fortran shouldn't matter at this point). Send me an email to d.s.seljebotn@astro.uio.no and I'll give you posting rights. Then write a post where you:
- Introduce yourself and the problems you work with
- What have you tried? What are you currently using? Why does this cause problems?
- What are the key insights you can draw from your experience?
- What way do you see forward? Are you already working on something? How much time and other resources (money for workshops etc.) do you have to work towards a solution?
We hope many will participate so that we all get a broad collection of the problems faced and the solutions tried.
Sorry it took me so long to post. I am quite busy at work and I am posting this during a lull.
ReplyDeleteIntroducing myself:
I am currently employed at the university of Canterbury in New Zealand with the title of "supercomputing support consultant". I got my PhD in physics (particle physics to be precise) in 2002 and I landed this job because of my experience in Lattice QCD amongst other things.
My perspective is quite different from Dag or Chris. While I was studying for my PhD the software stack they are talking about was still in its infancy. Most of the computing I have done for my PhD was using Mathematica some fortran code and some C code and a copy of numerical recipes.
Before landing my current job at the end of last year I was mainly teaching with doing development for Gentoo linux and sage in my free time. Because of my background in Gentoo I have a vision including package management.
There are several things I should note:
* I am currently a big bad admin.
* Physics/maths users are more sophisticated on average.
* Package management sucks - rpm sucks even more.
* No package management at all is worse.
Now, I am a big bad admin but the cluster I help run here is not managed by IT. They provide facilities but not the support, the cluster is part of a separate unit administered by a scientist (applied maths). My job is to bend over backwards as far as I can to give you the package you want and I can help users setting up their sandpit if they want too.
I have users from different fields like biology chemistry and so on. Their need can usually be easily accommodated.
I have been developing for sage for a number of years now. I do sage-on-gentoo and more recently with Burcin Erochal sage-on-prefix which is a light version of Gentoo prefix primarily aimed at an alternate build system for sage. Of course it can be expanded to other stuff. Thye main point of sage-on-prefix is too try to rely more on the package provided by the host. There is a pull on one side we can simplify the packages a bit but if we do that we move away from Gentoo and increase the maintenance burden.
I like to have some package management. It manages my upgrade nicely and ideally keep track of installed files. Which is one of the failing of sage's spkg in my opinion. If you update something did you leave undesirable cruft behind?
I'd really to hear from Ondřej Čertík from sympy and his femhub project which can use spkg but can do more.
Francois Bissey