From: "Jim Linnemann" <linnemann@pa.msu.edu>
Date: July 17, 2006 10:39:14 AM EDT (CA)
To: <06w5054-par@lists.pims.math.ca>
Subject: [BIRS] Single top multivariate problems of interest

This message from Reinhard Schweinhorst, introducing a data set of interest,
didn't get distributed to a  broader audience due to a communication glitch
between Byron and me, back in May.

	Jim Linnemann

I would like to provide an additional problem set for multivariate analyses.
We have performed a search for single top quark production using neural
networks on D0 RunII data.

We have unfortunately not been allowed to make the actual observed data
available, but we can provide the event sets used to model the signal and
backgrounds. 

The analysis is described in http://arxiv.org/abs/hep-ex/0505063
and in more detail in http://arxiv.org/abs/hep-ex/0604020.
The challenge in the single top analysis is that no single variable shows
good signal-background separation, and this set is thus ideally suited for a
multivariate analysis.
Besides the published neural network search and a simple cut-based analysis,
we have also done a decision tree analysis on this dataset (no
boosting) and a likelihood analysis on a slightly larger dataset.

I have already provided some of the files to Raoul Lepage, in a format
suitable for Mathematica. I can easily make them available in other formats
for other analyses. 
These are text files that contain one row for each event, and each row
contains an event weight and the 25 variables used in the neural network
search. The set of files for the full set will be large, since we divide the
dataset into electron and muon channels as well as single and double
b-tagged events. There are two different signals (s-channel and
t-channel) and 5 main backgrounds. Thus, the complete set would consist of
28 files.

Best regards,
Reinhard

On Mon, 2006-05-08 at 17:31 -0400, Byron Roe wrote:
This note is intended as a (very) brief introduction, especially for 
statisticians, into some of the multivariate problems in which 
physicists are interested.  The problems are, broadly speaking, 
classification (1.), variable selection(2.), and goodness of fit(3.).

1. In many experiments, physicists are interested in separating 
candidate signal events from background.  To do this, they have a 
number of "particle identification (PID)" variables which have 
different distributions for signal and various classes of background.
Some problems have a large number of PID variables, up to several 
hundred, others only a few.

One data set already used for the comparisons sited below is available 
at the end of my homepage:
http://gallatin.physics.lsa.umich.edu/~roe/
<http://gallatin.physics.lsa.umich.edu/%7Eroe/>
Ilya Narsky (narsky@hep.caltech.edu) has agreed to provide one or two 
other sets.  It would be very useful if participants would try 
favorite classification methods on these sets.

A number of statistical classification methods have been employed and 
the use and comparisons of some of them have been described at 
previous meetings:

(Several talks [especially those by Prosper, Bock and Vaiciulis] in 
the 'Multivariate Methods' section of the Durham IPPP meeting in 2002) 
http://www.ippp.dur.ac.uk/Workshops/02/statistics/proceedings.shtml

(Talk in Session 10 of PHYSTAT2003 by Jerry Friedman)
   http://www.slac.stanford.edu/econf/C030908/proceedings.html

(The invited talk by Jerry Friedman in PHYSTAT05and the response by 
Harrison Prosper; also the contributed talks by Bhat, Narsky, and Roe) 
http://www.physics.ox.ac.uk/phystat05/proceedings/default.htm

Comparisons have been made between the use of various types of boosted 
decision trees, random forests, bagging, and neural nets among others.
Although each problem is different, it would be useful to extend the 
comparison to techniques such as SVM and Bayesian neural nets.

Physicists have found that the use of some of these techniques for 
their problems requires different choices of parameters than the ones 
commonly used in statistics publications.  For example, for the use of 
boosted decision trees in my miniBooNE experiment, we have found the 
use of a large number of nodes(around 50) to be more efficient than 
the small number quoted in most statistical articles.

Although the comparisons are interesting, perhaps even more useful 
would be contrasting methods statisticians use in approaching data, in 
the use of graphical methods, and the use of R.  Physicists have 
tended to use C++, C, or even FORTRAN because of unfamiliarity with R 
and, for some groups, worries about its speed.

2.  It is also of considerable interest to examine ways of selecting 
the best PID (classification) variables to use.  In my experiment, 
MiniBooNE, we have over 300 potential PID variables and have worked 
out various ad hoc methods of winnowing these down to of the order of 
100.

3. Methods of dealing with systematics for chi- square and for log 
likelihood goodness of fit statistics, in several dimensions, is 
another current problem.  In many physics measurements there are some 
nuisance parameters which are not well estimated and whose uncertainty 
contributes significantly to the errors in the results for parameters 
of interest.  One method for chi-squared fits is described in:
D. Stump et al.,Phys. Rev. D65, 014012.  This method has several good 
features, but also some significant problems in practice.  Systematics 
also arises in multivariate event classification; systematic errors in 
the training sample (model or scale) errors are now being examined.
Graphical methods to examine these systematics, such as 1-D 
projections on random axes are being pursued.  The idea is that if the 
model fits badly, it may be because it needs more (nuisance) 
parameters to adequately describe the physical phenomenon.

As a general comment, applying to all three topics, there some 
reluctance among a number of physicists to use modern classification 
techniques as they seem "non-intuitive", and because the physicists 
believe it to be very difficult to  accurately model data in many 
dimensions.  Suggestions from statisticians concerning these issues 
would be welcome.

--
.....................................................................
. Reinhard Schwienhorst                       schwier@pa.msu.edu    .
. Research Associate                          Phone: (630) 840-6811 .
. Michigan State University                   Fax:   (630) 840-8886 .
.....................................................................