Day 2: Group C

A. Introduction

1.  go over classification methods: what can be said about applicability in 
different situations

2.  Models not perfect; what can we say about robustness or other flaws in the model.

3.  How do we get an intuitive feeling for the classification methods?  Graphical methods?  

4.  Methods for variable selection?  How to find the 'best' set of variables?  Why do we need to reduce the number of variables?

5. How many data-sets have been looked at by different people.

6. Unisims, multisims: estimating systematic uncertainties.

7.  Uniform framework?  (Should we all be re-inventing the wheel?)

8. Question: is subjectivity adding to the quality of the analysis or not?

9. Can we learn about statisticians' methodology:  Raoul, Nicolai

B.  Support vector machines:  Stephen Bailey

Palomar telescope

Searching for super-novae:  30k images, 2-5 galaxies/image,  = 100,000 per night

there are about 2-5 good SNe/night

around 300 objects on each image of which 2-5 are galaxies

new image subtracted from  a reference image; if a new spot has appeared it will be brighter on the difference image; there is a lot of background noise too;  in a typical night there will be 3 new images in the same region of the sky

one approach is to first identify the known galaxies; since these get subtracted out
our approach is to look for supernovae in galaxies that may not be identified

each object in the residual image are described by about 20 parameters (features),
one of which might be the statistical significance of the observed difference

the base code (old method) does the following: (on the difference images)

10^7 objects; 10^3 pass the cuts based on a computer model;  these are whittled down to 10 by visual inspection and analysed in detail; this leads to about 1 SN found

collaboration with LBL to find the boundary using support vector machines; rank order the 10^3 objects by the distance from the boundary and scan them in order.
This works, sort of, because the first 10% finds about 1/2 the objects, but the remaining objects usually need to be handled manually anyway

Dataset is not such high quality because it's piggy-backed on another experiment.  As a result the boundary is very diffuse.  Goal to increase the signal efficiency and reduce the work.  The features of the background vary from one night to the next.  There is some attempt to handle weather effects in the process of subtractions; but the types of artifacts that are left are still rather variable from night to night.

We have real data available to train for the future.

The variability of the background and the noise in the data are the main problems.

There are fake duplicates introduced into each search to check the accuracy of the observers.

The goal is to apply it to the full set of 10^7 objects when the problems in selecting the variables, and in correcting the background.

Question:  is there one classifier that should be optimized to the variation in background, or should the background effects be adjusted as part of the classification.  Suggestion:  include additional inputs that predict the general background for the image.

GLAST in decision trees:  Toby Burnett

Gamma Large Area Space Telescope:
attached to spacecraft, surround by shielding, tracking and conversion section, calibration section.  Gamma ray comes into tracking section, gamma ray showers captured in the calibration section.  Designed with Monte Carlo.

The data is not an image, but rather a HEP type event, reconstructed by hits in the tracking and energy in the calibration to determined where it came from.  

Decision trees:  optimize the resolution on the energy and direction reconstruction; the trees are used to decide if the reconstructed event is close to MC values

See: http://d0.phys.washington.edu/~burnett/d0/ctree/ and
         http://d0.phys.washington.edu/~burnett/BIRS/decision_trees.mht

Bayesian neural networks: Radford Neal

Bryon's data:  50 PID variables; classification into two types is the output

Radford's NN has two hidden layers; connections from all inputs to all hidden units; each connection has a weight attached to it; a 'bias' constant; tanh activation function:   output for each hidden unit is 

tanh(b + \Sum_i w_i V_i)

this is converted at the end to a binary classifier by the logistic function

This can be parametrized, by treating the weights and biases as parameters.  It can be shown that even with just one hidden layer any function can be approximated with enough units.  

In principle the parameters could be estimated by maximum likelihood, but this is not usually a good idea, since the data can be fitted exactly with enough units.  NonBayesian methods incorporate either regularization or a naive method called 'early stopping', or an ensemble version of that analogous to cross-validation.  This version of early stopping actually does quite well.

It turns out that he number of hidden units is not so crucial; early stopping basically corrects for this.

A Bayesian version works better, by integrating over the space of weights in the network, with a combination of prior and likelihood.  This integration is carried out by MCMC.  The Metropolis algorithm is very slow, but a hybrid MC version works well.  The result is a probability for classification, but this is done on a large number of networks, and averaged (at the last step).  The Bayesian version can incorporate a hyper-prior for the parameters on some of the parameters in the network.  This allows some improved classification if the data turns out to be very predictable.

You can show that there is no statistical necessity to constrain the number of hidden units.  However the computations do get slow as the number gets too large.

Now adding some boosting to this, to concentrate on the items that are hard to classify, but of course apply larger weights to the easy-to-classify items so that the output remains unbiased.  This hopefully will reduce the computation time.

Graphical Display Suggestion

plot $\Delta \ log(p/(1-p))$ which is the change in the classifier output when variable 27 is changed by $\epsilon$ in the training case, vs variable 27.  

If the plot is straight, variable 27 affects the logit linearly; if the plot is curved then the variable isn't linear, but it is additive (no interaction).   If the plot is scattered, then there are interactions going on with the other variables.

(The computation of $\Delta "prediction"$ adjusts for the presence of the other 49 variables.)

Note that one plot needed for each PID, and that each plot has as many points as there are events in the training sample.

There could be an 'ensemble' of such plots, one for each of the networks that go into the bagging, OR, it could be based on the averaged or bagged prediction.  

Some thoughts on robustifying the classifier

All in pictures, your scribe couldn't keep up

Some software for multivariate classification:  Ilya Narsky

Slides are posted at http://www.cithep.caltech.edu/~narsky/Software_Banff.pdf