Lecture 1 - Jet tagging with neural networks#

A first look at training deep neural networks to classify jets in proton-proton collisions.

Learning objectives#

Understand what jet tagging is and how to frame it as a machine learning task
Understand the main steps needed to train and evaluate a jet tagger
Learn how to download and process data with the 🤗 Datasets library
Gain an introduction to the fastai library and how to push models to the Hugging Face Hub

References#

Chapter 1 of Deep Learning for Coders with fastai & PyTorch by Jeremy Howard and Sylvain Gugger.
The Machine Learning Landscape of Top Taggers by G. Kasieczka et al.
How Much Information is in a Jet? by K. Datta and A. Larkowski.

The task and data#

For the first few lectures, we’ll be analysing the Top Quark Tagging dataset, which is a famous benchmark that’s used to compare the performance of jet classification algorithms. The dataset consists of around 2 million Monte Carlo simulated events in proton-proton collisions that have been clustered into jets.

Framed as a supervised machine learning task, the goal is to train a model that can classify each jet as either a top-quark signal or quark-gluon background.

_images/jet-tagging.png — Fig. 2 Figure reference: Particle Transformer for Jet Tagging#

Setup#

# Uncomment and run this cell if using Colab, Kaggle etc
# %pip install fastai==2.6.0 datasets git+https://github.com/huggingface/huggingface_hub

# Check we have the correct fastai version
import fastai

assert fastai.__version__ == "2.6.0"

Import libraries#

from datasets import load_dataset
from fastai.tabular.all import *
from huggingface_hub import from_pretrained_fastai, notebook_login, push_to_hub_fastai
from scipy.interpolate import interp1d
from sklearn.metrics import accuracy_score, auc, roc_auc_score, roc_curve
from sklearn.model_selection import train_test_split

import datasets

# Suppress logs
datasets.logging.set_verbosity_error()

Getting the data#

We will use the 🤗 Datasets library to download and process the datasets that we’ll encounter in this course. 🤗 Datasets provides smart caching and allows you to process larger-than-RAM datasets by exploiting a technique called memory-mapping that provides a mapping between RAM and filesystem storage.

To download the Top Quark Tagging dataset from the Hugging Face Hub, we can use the load_dataset() function:

top_tagging_ds = load_dataset("dl4phys/top_tagging")

If we look inside our top_tagging_ds object

top_tagging_ds

DatasetDict({
    train: Dataset({
        features: ['E_0', 'PX_0', 'PY_0', 'PZ_0', 'E_1', 'PX_1', 'PY_1', 'PZ_1', 'E_2', 'PX_2', 'PY_2', 'PZ_2', 'E_3', 'PX_3', 'PY_3', 'PZ_3', 'E_4', 'PX_4', 'PY_4', 'PZ_4', 'E_5', 'PX_5', 'PY_5', 'PZ_5', 'E_6', 'PX_6', 'PY_6', 'PZ_6', 'E_7', 'PX_7', 'PY_7', 'PZ_7', 'E_8', 'PX_8', 'PY_8', 'PZ_8', 'E_9', 'PX_9', 'PY_9', 'PZ_9', 'E_10', 'PX_10', 'PY_10', 'PZ_10', 'E_11', 'PX_11', 'PY_11', 'PZ_11', 'E_12', 'PX_12', 'PY_12', 'PZ_12', 'E_13', 'PX_13', 'PY_13', 'PZ_13', 'E_14', 'PX_14', 'PY_14', 'PZ_14', 'E_15', 'PX_15', 'PY_15', 'PZ_15', 'E_16', 'PX_16', 'PY_16', 'PZ_16', 'E_17', 'PX_17', 'PY_17', 'PZ_17', 'E_18', 'PX_18', 'PY_18', 'PZ_18', 'E_19', 'PX_19', 'PY_19', 'PZ_19', 'E_20', 'PX_20', 'PY_20', 'PZ_20', 'E_21', 'PX_21', 'PY_21', 'PZ_21', 'E_22', 'PX_22', 'PY_22', 'PZ_22', 'E_23', 'PX_23', 'PY_23', 'PZ_23', 'E_24', 'PX_24', 'PY_24', 'PZ_24', 'E_25', 'PX_25', 'PY_25', 'PZ_25', 'E_26', 'PX_26', 'PY_26', 'PZ_26', 'E_27', 'PX_27', 'PY_27', 'PZ_27', 'E_28', 'PX_28', 'PY_28', 'PZ_28', 'E_29', 'PX_29', 'PY_29', 'PZ_29', 'E_30', 'PX_30', 'PY_30', 'PZ_30', 'E_31', 'PX_31', 'PY_31', 'PZ_31', 'E_32', 'PX_32', 'PY_32', 'PZ_32', 'E_33', 'PX_33', 'PY_33', 'PZ_33', 'E_34', 'PX_34', 'PY_34', 'PZ_34', 'E_35', 'PX_35', 'PY_35', 'PZ_35', 'E_36', 'PX_36', 'PY_36', 'PZ_36', 'E_37', 'PX_37', 'PY_37', 'PZ_37', 'E_38', 'PX_38', 'PY_38', 'PZ_38', 'E_39', 'PX_39', 'PY_39', 'PZ_39', 'E_40', 'PX_40', 'PY_40', 'PZ_40', 'E_41', 'PX_41', 'PY_41', 'PZ_41', 'E_42', 'PX_42', 'PY_42', 'PZ_42', 'E_43', 'PX_43', 'PY_43', 'PZ_43', 'E_44', 'PX_44', 'PY_44', 'PZ_44', 'E_45', 'PX_45', 'PY_45', 'PZ_45', 'E_46', 'PX_46', 'PY_46', 'PZ_46', 'E_47', 'PX_47', 'PY_47', 'PZ_47', 'E_48', 'PX_48', 'PY_48', 'PZ_48', 'E_49', 'PX_49', 'PY_49', 'PZ_49', 'E_50', 'PX_50', 'PY_50', 'PZ_50', 'E_51', 'PX_51', 'PY_51', 'PZ_51', 'E_52', 'PX_52', 'PY_52', 'PZ_52', 'E_53', 'PX_53', 'PY_53', 'PZ_53', 'E_54', 'PX_54', 'PY_54', 'PZ_54', 'E_55', 'PX_55', 'PY_55', 'PZ_55', 'E_56', 'PX_56', 'PY_56', 'PZ_56', 'E_57', 'PX_57', 'PY_57', 'PZ_57', 'E_58', 'PX_58', 'PY_58', 'PZ_58', 'E_59', 'PX_59', 'PY_59', 'PZ_59', 'E_60', 'PX_60', 'PY_60', 'PZ_60', 'E_61', 'PX_61', 'PY_61', 'PZ_61', 'E_62', 'PX_62', 'PY_62', 'PZ_62', 'E_63', 'PX_63', 'PY_63', 'PZ_63', 'E_64', 'PX_64', 'PY_64', 'PZ_64', 'E_65', 'PX_65', 'PY_65', 'PZ_65', 'E_66', 'PX_66', 'PY_66', 'PZ_66', 'E_67', 'PX_67', 'PY_67', 'PZ_67', 'E_68', 'PX_68', 'PY_68', 'PZ_68', 'E_69', 'PX_69', 'PY_69', 'PZ_69', 'E_70', 'PX_70', 'PY_70', 'PZ_70', 'E_71', 'PX_71', 'PY_71', 'PZ_71', 'E_72', 'PX_72', 'PY_72', 'PZ_72', 'E_73', 'PX_73', 'PY_73', 'PZ_73', 'E_74', 'PX_74', 'PY_74', 'PZ_74', 'E_75', 'PX_75', 'PY_75', 'PZ_75', 'E_76', 'PX_76', 'PY_76', 'PZ_76', 'E_77', 'PX_77', 'PY_77', 'PZ_77', 'E_78', 'PX_78', 'PY_78', 'PZ_78', 'E_79', 'PX_79', 'PY_79', 'PZ_79', 'E_80', 'PX_80', 'PY_80', 'PZ_80', 'E_81', 'PX_81', 'PY_81', 'PZ_81', 'E_82', 'PX_82', 'PY_82', 'PZ_82', 'E_83', 'PX_83', 'PY_83', 'PZ_83', 'E_84', 'PX_84', 'PY_84', 'PZ_84', 'E_85', 'PX_85', 'PY_85', 'PZ_85', 'E_86', 'PX_86', 'PY_86', 'PZ_86', 'E_87', 'PX_87', 'PY_87', 'PZ_87', 'E_88', 'PX_88', 'PY_88', 'PZ_88', 'E_89', 'PX_89', 'PY_89', 'PZ_89', 'E_90', 'PX_90', 'PY_90', 'PZ_90', 'E_91', 'PX_91', 'PY_91', 'PZ_91', 'E_92', 'PX_92', 'PY_92', 'PZ_92', 'E_93', 'PX_93', 'PY_93', 'PZ_93', 'E_94', 'PX_94', 'PY_94', 'PZ_94', 'E_95', 'PX_95', 'PY_95', 'PZ_95', 'E_96', 'PX_96', 'PY_96', 'PZ_96', 'E_97', 'PX_97', 'PY_97', 'PZ_97', 'E_98', 'PX_98', 'PY_98', 'PZ_98', 'E_99', 'PX_99', 'PY_99', 'PZ_99', 'E_100', 'PX_100', 'PY_100', 'PZ_100', 'E_101', 'PX_101', 'PY_101', 'PZ_101', 'E_102', 'PX_102', 'PY_102', 'PZ_102', 'E_103', 'PX_103', 'PY_103', 'PZ_103', 'E_104', 'PX_104', 'PY_104', 'PZ_104', 'E_105', 'PX_105', 'PY_105', 'PZ_105', 'E_106', 'PX_106', 'PY_106', 'PZ_106', 'E_107', 'PX_107', 'PY_107', 'PZ_107', 'E_108', 'PX_108', 'PY_108', 'PZ_108', 'E_109', 'PX_109', 'PY_109', 'PZ_109', 'E_110', 'PX_110', 'PY_110', 'PZ_110', 'E_111', 'PX_111', 'PY_111', 'PZ_111', 'E_112', 'PX_112', 'PY_112', 'PZ_112', 'E_113', 'PX_113', 'PY_113', 'PZ_113', 'E_114', 'PX_114', 'PY_114', 'PZ_114', 'E_115', 'PX_115', 'PY_115', 'PZ_115', 'E_116', 'PX_116', 'PY_116', 'PZ_116', 'E_117', 'PX_117', 'PY_117', 'PZ_117', 'E_118', 'PX_118', 'PY_118', 'PZ_118', 'E_119', 'PX_119', 'PY_119', 'PZ_119', 'E_120', 'PX_120', 'PY_120', 'PZ_120', 'E_121', 'PX_121', 'PY_121', 'PZ_121', 'E_122', 'PX_122', 'PY_122', 'PZ_122', 'E_123', 'PX_123', 'PY_123', 'PZ_123', 'E_124', 'PX_124', 'PY_124', 'PZ_124', 'E_125', 'PX_125', 'PY_125', 'PZ_125', 'E_126', 'PX_126', 'PY_126', 'PZ_126', 'E_127', 'PX_127', 'PY_127', 'PZ_127', 'E_128', 'PX_128', 'PY_128', 'PZ_128', 'E_129', 'PX_129', 'PY_129', 'PZ_129', 'E_130', 'PX_130', 'PY_130', 'PZ_130', 'E_131', 'PX_131', 'PY_131', 'PZ_131', 'E_132', 'PX_132', 'PY_132', 'PZ_132', 'E_133', 'PX_133', 'PY_133', 'PZ_133', 'E_134', 'PX_134', 'PY_134', 'PZ_134', 'E_135', 'PX_135', 'PY_135', 'PZ_135', 'E_136', 'PX_136', 'PY_136', 'PZ_136', 'E_137', 'PX_137', 'PY_137', 'PZ_137', 'E_138', 'PX_138', 'PY_138', 'PZ_138', 'E_139', 'PX_139', 'PY_139', 'PZ_139', 'E_140', 'PX_140', 'PY_140', 'PZ_140', 'E_141', 'PX_141', 'PY_141', 'PZ_141', 'E_142', 'PX_142', 'PY_142', 'PZ_142', 'E_143', 'PX_143', 'PY_143', 'PZ_143', 'E_144', 'PX_144', 'PY_144', 'PZ_144', 'E_145', 'PX_145', 'PY_145', 'PZ_145', 'E_146', 'PX_146', 'PY_146', 'PZ_146', 'E_147', 'PX_147', 'PY_147', 'PZ_147', 'E_148', 'PX_148', 'PY_148', 'PZ_148', 'E_149', 'PX_149', 'PY_149', 'PZ_149', 'E_150', 'PX_150', 'PY_150', 'PZ_150', 'E_151', 'PX_151', 'PY_151', 'PZ_151', 'E_152', 'PX_152', 'PY_152', 'PZ_152', 'E_153', 'PX_153', 'PY_153', 'PZ_153', 'E_154', 'PX_154', 'PY_154', 'PZ_154', 'E_155', 'PX_155', 'PY_155', 'PZ_155', 'E_156', 'PX_156', 'PY_156', 'PZ_156', 'E_157', 'PX_157', 'PY_157', 'PZ_157', 'E_158', 'PX_158', 'PY_158', 'PZ_158', 'E_159', 'PX_159', 'PY_159', 'PZ_159', 'E_160', 'PX_160', 'PY_160', 'PZ_160', 'E_161', 'PX_161', 'PY_161', 'PZ_161', 'E_162', 'PX_162', 'PY_162', 'PZ_162', 'E_163', 'PX_163', 'PY_163', 'PZ_163', 'E_164', 'PX_164', 'PY_164', 'PZ_164', 'E_165', 'PX_165', 'PY_165', 'PZ_165', 'E_166', 'PX_166', 'PY_166', 'PZ_166', 'E_167', 'PX_167', 'PY_167', 'PZ_167', 'E_168', 'PX_168', 'PY_168', 'PZ_168', 'E_169', 'PX_169', 'PY_169', 'PZ_169', 'E_170', 'PX_170', 'PY_170', 'PZ_170', 'E_171', 'PX_171', 'PY_171', 'PZ_171', 'E_172', 'PX_172', 'PY_172', 'PZ_172', 'E_173', 'PX_173', 'PY_173', 'PZ_173', 'E_174', 'PX_174', 'PY_174', 'PZ_174', 'E_175', 'PX_175', 'PY_175', 'PZ_175', 'E_176', 'PX_176', 'PY_176', 'PZ_176', 'E_177', 'PX_177', 'PY_177', 'PZ_177', 'E_178', 'PX_178', 'PY_178', 'PZ_178', 'E_179', 'PX_179', 'PY_179', 'PZ_179', 'E_180', 'PX_180', 'PY_180', 'PZ_180', 'E_181', 'PX_181', 'PY_181', 'PZ_181', 'E_182', 'PX_182', 'PY_182', 'PZ_182', 'E_183', 'PX_183', 'PY_183', 'PZ_183', 'E_184', 'PX_184', 'PY_184', 'PZ_184', 'E_185', 'PX_185', 'PY_185', 'PZ_185', 'E_186', 'PX_186', 'PY_186', 'PZ_186', 'E_187', 'PX_187', 'PY_187', 'PZ_187', 'E_188', 'PX_188', 'PY_188', 'PZ_188', 'E_189', 'PX_189', 'PY_189', 'PZ_189', 'E_190', 'PX_190', 'PY_190', 'PZ_190', 'E_191', 'PX_191', 'PY_191', 'PZ_191', 'E_192', 'PX_192', 'PY_192', 'PZ_192', 'E_193', 'PX_193', 'PY_193', 'PZ_193', 'E_194', 'PX_194', 'PY_194', 'PZ_194', 'E_195', 'PX_195', 'PY_195', 'PZ_195', 'E_196', 'PX_196', 'PY_196', 'PZ_196', 'E_197', 'PX_197', 'PY_197', 'PZ_197', 'E_198', 'PX_198', 'PY_198', 'PZ_198', 'E_199', 'PX_199', 'PY_199', 'PZ_199', 'truthE', 'truthPX', 'truthPY', 'truthPZ', 'ttv', 'is_signal_new'],
        num_rows: 1211000
    })
    test: Dataset({
        features: ['E_0', 'PX_0', 'PY_0', 'PZ_0', 'E_1', 'PX_1', 'PY_1', 'PZ_1', 'E_2', 'PX_2', 'PY_2', 'PZ_2', 'E_3', 'PX_3', 'PY_3', 'PZ_3', 'E_4', 'PX_4', 'PY_4', 'PZ_4', 'E_5', 'PX_5', 'PY_5', 'PZ_5', 'E_6', 'PX_6', 'PY_6', 'PZ_6', 'E_7', 'PX_7', 'PY_7', 'PZ_7', 'E_8', 'PX_8', 'PY_8', 'PZ_8', 'E_9', 'PX_9', 'PY_9', 'PZ_9', 'E_10', 'PX_10', 'PY_10', 'PZ_10', 'E_11', 'PX_11', 'PY_11', 'PZ_11', 'E_12', 'PX_12', 'PY_12', 'PZ_12', 'E_13', 'PX_13', 'PY_13', 'PZ_13', 'E_14', 'PX_14', 'PY_14', 'PZ_14', 'E_15', 'PX_15', 'PY_15', 'PZ_15', 'E_16', 'PX_16', 'PY_16', 'PZ_16', 'E_17', 'PX_17', 'PY_17', 'PZ_17', 'E_18', 'PX_18', 'PY_18', 'PZ_18', 'E_19', 'PX_19', 'PY_19', 'PZ_19', 'E_20', 'PX_20', 'PY_20', 'PZ_20', 'E_21', 'PX_21', 'PY_21', 'PZ_21', 'E_22', 'PX_22', 'PY_22', 'PZ_22', 'E_23', 'PX_23', 'PY_23', 'PZ_23', 'E_24', 'PX_24', 'PY_24', 'PZ_24', 'E_25', 'PX_25', 'PY_25', 'PZ_25', 'E_26', 'PX_26', 'PY_26', 'PZ_26', 'E_27', 'PX_27', 'PY_27', 'PZ_27', 'E_28', 'PX_28', 'PY_28', 'PZ_28', 'E_29', 'PX_29', 'PY_29', 'PZ_29', 'E_30', 'PX_30', 'PY_30', 'PZ_30', 'E_31', 'PX_31', 'PY_31', 'PZ_31', 'E_32', 'PX_32', 'PY_32', 'PZ_32', 'E_33', 'PX_33', 'PY_33', 'PZ_33', 'E_34', 'PX_34', 'PY_34', 'PZ_34', 'E_35', 'PX_35', 'PY_35', 'PZ_35', 'E_36', 'PX_36', 'PY_36', 'PZ_36', 'E_37', 'PX_37', 'PY_37', 'PZ_37', 'E_38', 'PX_38', 'PY_38', 'PZ_38', 'E_39', 'PX_39', 'PY_39', 'PZ_39', 'E_40', 'PX_40', 'PY_40', 'PZ_40', 'E_41', 'PX_41', 'PY_41', 'PZ_41', 'E_42', 'PX_42', 'PY_42', 'PZ_42', 'E_43', 'PX_43', 'PY_43', 'PZ_43', 'E_44', 'PX_44', 'PY_44', 'PZ_44', 'E_45', 'PX_45', 'PY_45', 'PZ_45', 'E_46', 'PX_46', 'PY_46', 'PZ_46', 'E_47', 'PX_47', 'PY_47', 'PZ_47', 'E_48', 'PX_48', 'PY_48', 'PZ_48', 'E_49', 'PX_49', 'PY_49', 'PZ_49', 'E_50', 'PX_50', 'PY_50', 'PZ_50', 'E_51', 'PX_51', 'PY_51', 'PZ_51', 'E_52', 'PX_52', 'PY_52', 'PZ_52', 'E_53', 'PX_53', 'PY_53', 'PZ_53', 'E_54', 'PX_54', 'PY_54', 'PZ_54', 'E_55', 'PX_55', 'PY_55', 'PZ_55', 'E_56', 'PX_56', 'PY_56', 'PZ_56', 'E_57', 'PX_57', 'PY_57', 'PZ_57', 'E_58', 'PX_58', 'PY_58', 'PZ_58', 'E_59', 'PX_59', 'PY_59', 'PZ_59', 'E_60', 'PX_60', 'PY_60', 'PZ_60', 'E_61', 'PX_61', 'PY_61', 'PZ_61', 'E_62', 'PX_62', 'PY_62', 'PZ_62', 'E_63', 'PX_63', 'PY_63', 'PZ_63', 'E_64', 'PX_64', 'PY_64', 'PZ_64', 'E_65', 'PX_65', 'PY_65', 'PZ_65', 'E_66', 'PX_66', 'PY_66', 'PZ_66', 'E_67', 'PX_67', 'PY_67', 'PZ_67', 'E_68', 'PX_68', 'PY_68', 'PZ_68', 'E_69', 'PX_69', 'PY_69', 'PZ_69', 'E_70', 'PX_70', 'PY_70', 'PZ_70', 'E_71', 'PX_71', 'PY_71', 'PZ_71', 'E_72', 'PX_72', 'PY_72', 'PZ_72', 'E_73', 'PX_73', 'PY_73', 'PZ_73', 'E_74', 'PX_74', 'PY_74', 'PZ_74', 'E_75', 'PX_75', 'PY_75', 'PZ_75', 'E_76', 'PX_76', 'PY_76', 'PZ_76', 'E_77', 'PX_77', 'PY_77', 'PZ_77', 'E_78', 'PX_78', 'PY_78', 'PZ_78', 'E_79', 'PX_79', 'PY_79', 'PZ_79', 'E_80', 'PX_80', 'PY_80', 'PZ_80', 'E_81', 'PX_81', 'PY_81', 'PZ_81', 'E_82', 'PX_82', 'PY_82', 'PZ_82', 'E_83', 'PX_83', 'PY_83', 'PZ_83', 'E_84', 'PX_84', 'PY_84', 'PZ_84', 'E_85', 'PX_85', 'PY_85', 'PZ_85', 'E_86', 'PX_86', 'PY_86', 'PZ_86', 'E_87', 'PX_87', 'PY_87', 'PZ_87', 'E_88', 'PX_88', 'PY_88', 'PZ_88', 'E_89', 'PX_89', 'PY_89', 'PZ_89', 'E_90', 'PX_90', 'PY_90', 'PZ_90', 'E_91', 'PX_91', 'PY_91', 'PZ_91', 'E_92', 'PX_92', 'PY_92', 'PZ_92', 'E_93', 'PX_93', 'PY_93', 'PZ_93', 'E_94', 'PX_94', 'PY_94', 'PZ_94', 'E_95', 'PX_95', 'PY_95', 'PZ_95', 'E_96', 'PX_96', 'PY_96', 'PZ_96', 'E_97', 'PX_97', 'PY_97', 'PZ_97', 'E_98', 'PX_98', 'PY_98', 'PZ_98', 'E_99', 'PX_99', 'PY_99', 'PZ_99', 'E_100', 'PX_100', 'PY_100', 'PZ_100', 'E_101', 'PX_101', 'PY_101', 'PZ_101', 'E_102', 'PX_102', 'PY_102', 'PZ_102', 'E_103', 'PX_103', 'PY_103', 'PZ_103', 'E_104', 'PX_104', 'PY_104', 'PZ_104', 'E_105', 'PX_105', 'PY_105', 'PZ_105', 'E_106', 'PX_106', 'PY_106', 'PZ_106', 'E_107', 'PX_107', 'PY_107', 'PZ_107', 'E_108', 'PX_108', 'PY_108', 'PZ_108', 'E_109', 'PX_109', 'PY_109', 'PZ_109', 'E_110', 'PX_110', 'PY_110', 'PZ_110', 'E_111', 'PX_111', 'PY_111', 'PZ_111', 'E_112', 'PX_112', 'PY_112', 'PZ_112', 'E_113', 'PX_113', 'PY_113', 'PZ_113', 'E_114', 'PX_114', 'PY_114', 'PZ_114', 'E_115', 'PX_115', 'PY_115', 'PZ_115', 'E_116', 'PX_116', 'PY_116', 'PZ_116', 'E_117', 'PX_117', 'PY_117', 'PZ_117', 'E_118', 'PX_118', 'PY_118', 'PZ_118', 'E_119', 'PX_119', 'PY_119', 'PZ_119', 'E_120', 'PX_120', 'PY_120', 'PZ_120', 'E_121', 'PX_121', 'PY_121', 'PZ_121', 'E_122', 'PX_122', 'PY_122', 'PZ_122', 'E_123', 'PX_123', 'PY_123', 'PZ_123', 'E_124', 'PX_124', 'PY_124', 'PZ_124', 'E_125', 'PX_125', 'PY_125', 'PZ_125', 'E_126', 'PX_126', 'PY_126', 'PZ_126', 'E_127', 'PX_127', 'PY_127', 'PZ_127', 'E_128', 'PX_128', 'PY_128', 'PZ_128', 'E_129', 'PX_129', 'PY_129', 'PZ_129', 'E_130', 'PX_130', 'PY_130', 'PZ_130', 'E_131', 'PX_131', 'PY_131', 'PZ_131', 'E_132', 'PX_132', 'PY_132', 'PZ_132', 'E_133', 'PX_133', 'PY_133', 'PZ_133', 'E_134', 'PX_134', 'PY_134', 'PZ_134', 'E_135', 'PX_135', 'PY_135', 'PZ_135', 'E_136', 'PX_136', 'PY_136', 'PZ_136', 'E_137', 'PX_137', 'PY_137', 'PZ_137', 'E_138', 'PX_138', 'PY_138', 'PZ_138', 'E_139', 'PX_139', 'PY_139', 'PZ_139', 'E_140', 'PX_140', 'PY_140', 'PZ_140', 'E_141', 'PX_141', 'PY_141', 'PZ_141', 'E_142', 'PX_142', 'PY_142', 'PZ_142', 'E_143', 'PX_143', 'PY_143', 'PZ_143', 'E_144', 'PX_144', 'PY_144', 'PZ_144', 'E_145', 'PX_145', 'PY_145', 'PZ_145', 'E_146', 'PX_146', 'PY_146', 'PZ_146', 'E_147', 'PX_147', 'PY_147', 'PZ_147', 'E_148', 'PX_148', 'PY_148', 'PZ_148', 'E_149', 'PX_149', 'PY_149', 'PZ_149', 'E_150', 'PX_150', 'PY_150', 'PZ_150', 'E_151', 'PX_151', 'PY_151', 'PZ_151', 'E_152', 'PX_152', 'PY_152', 'PZ_152', 'E_153', 'PX_153', 'PY_153', 'PZ_153', 'E_154', 'PX_154', 'PY_154', 'PZ_154', 'E_155', 'PX_155', 'PY_155', 'PZ_155', 'E_156', 'PX_156', 'PY_156', 'PZ_156', 'E_157', 'PX_157', 'PY_157', 'PZ_157', 'E_158', 'PX_158', 'PY_158', 'PZ_158', 'E_159', 'PX_159', 'PY_159', 'PZ_159', 'E_160', 'PX_160', 'PY_160', 'PZ_160', 'E_161', 'PX_161', 'PY_161', 'PZ_161', 'E_162', 'PX_162', 'PY_162', 'PZ_162', 'E_163', 'PX_163', 'PY_163', 'PZ_163', 'E_164', 'PX_164', 'PY_164', 'PZ_164', 'E_165', 'PX_165', 'PY_165', 'PZ_165', 'E_166', 'PX_166', 'PY_166', 'PZ_166', 'E_167', 'PX_167', 'PY_167', 'PZ_167', 'E_168', 'PX_168', 'PY_168', 'PZ_168', 'E_169', 'PX_169', 'PY_169', 'PZ_169', 'E_170', 'PX_170', 'PY_170', 'PZ_170', 'E_171', 'PX_171', 'PY_171', 'PZ_171', 'E_172', 'PX_172', 'PY_172', 'PZ_172', 'E_173', 'PX_173', 'PY_173', 'PZ_173', 'E_174', 'PX_174', 'PY_174', 'PZ_174', 'E_175', 'PX_175', 'PY_175', 'PZ_175', 'E_176', 'PX_176', 'PY_176', 'PZ_176', 'E_177', 'PX_177', 'PY_177', 'PZ_177', 'E_178', 'PX_178', 'PY_178', 'PZ_178', 'E_179', 'PX_179', 'PY_179', 'PZ_179', 'E_180', 'PX_180', 'PY_180', 'PZ_180', 'E_181', 'PX_181', 'PY_181', 'PZ_181', 'E_182', 'PX_182', 'PY_182', 'PZ_182', 'E_183', 'PX_183', 'PY_183', 'PZ_183', 'E_184', 'PX_184', 'PY_184', 'PZ_184', 'E_185', 'PX_185', 'PY_185', 'PZ_185', 'E_186', 'PX_186', 'PY_186', 'PZ_186', 'E_187', 'PX_187', 'PY_187', 'PZ_187', 'E_188', 'PX_188', 'PY_188', 'PZ_188', 'E_189', 'PX_189', 'PY_189', 'PZ_189', 'E_190', 'PX_190', 'PY_190', 'PZ_190', 'E_191', 'PX_191', 'PY_191', 'PZ_191', 'E_192', 'PX_192', 'PY_192', 'PZ_192', 'E_193', 'PX_193', 'PY_193', 'PZ_193', 'E_194', 'PX_194', 'PY_194', 'PZ_194', 'E_195', 'PX_195', 'PY_195', 'PZ_195', 'E_196', 'PX_196', 'PY_196', 'PZ_196', 'E_197', 'PX_197', 'PY_197', 'PZ_197', 'E_198', 'PX_198', 'PY_198', 'PZ_198', 'E_199', 'PX_199', 'PY_199', 'PZ_199', 'truthE', 'truthPX', 'truthPY', 'truthPZ', 'ttv', 'is_signal_new'],
        num_rows: 404000
    })
    validation: Dataset({
        features: ['E_0', 'PX_0', 'PY_0', 'PZ_0', 'E_1', 'PX_1', 'PY_1', 'PZ_1', 'E_2', 'PX_2', 'PY_2', 'PZ_2', 'E_3', 'PX_3', 'PY_3', 'PZ_3', 'E_4', 'PX_4', 'PY_4', 'PZ_4', 'E_5', 'PX_5', 'PY_5', 'PZ_5', 'E_6', 'PX_6', 'PY_6', 'PZ_6', 'E_7', 'PX_7', 'PY_7', 'PZ_7', 'E_8', 'PX_8', 'PY_8', 'PZ_8', 'E_9', 'PX_9', 'PY_9', 'PZ_9', 'E_10', 'PX_10', 'PY_10', 'PZ_10', 'E_11', 'PX_11', 'PY_11', 'PZ_11', 'E_12', 'PX_12', 'PY_12', 'PZ_12', 'E_13', 'PX_13', 'PY_13', 'PZ_13', 'E_14', 'PX_14', 'PY_14', 'PZ_14', 'E_15', 'PX_15', 'PY_15', 'PZ_15', 'E_16', 'PX_16', 'PY_16', 'PZ_16', 'E_17', 'PX_17', 'PY_17', 'PZ_17', 'E_18', 'PX_18', 'PY_18', 'PZ_18', 'E_19', 'PX_19', 'PY_19', 'PZ_19', 'E_20', 'PX_20', 'PY_20', 'PZ_20', 'E_21', 'PX_21', 'PY_21', 'PZ_21', 'E_22', 'PX_22', 'PY_22', 'PZ_22', 'E_23', 'PX_23', 'PY_23', 'PZ_23', 'E_24', 'PX_24', 'PY_24', 'PZ_24', 'E_25', 'PX_25', 'PY_25', 'PZ_25', 'E_26', 'PX_26', 'PY_26', 'PZ_26', 'E_27', 'PX_27', 'PY_27', 'PZ_27', 'E_28', 'PX_28', 'PY_28', 'PZ_28', 'E_29', 'PX_29', 'PY_29', 'PZ_29', 'E_30', 'PX_30', 'PY_30', 'PZ_30', 'E_31', 'PX_31', 'PY_31', 'PZ_31', 'E_32', 'PX_32', 'PY_32', 'PZ_32', 'E_33', 'PX_33', 'PY_33', 'PZ_33', 'E_34', 'PX_34', 'PY_34', 'PZ_34', 'E_35', 'PX_35', 'PY_35', 'PZ_35', 'E_36', 'PX_36', 'PY_36', 'PZ_36', 'E_37', 'PX_37', 'PY_37', 'PZ_37', 'E_38', 'PX_38', 'PY_38', 'PZ_38', 'E_39', 'PX_39', 'PY_39', 'PZ_39', 'E_40', 'PX_40', 'PY_40', 'PZ_40', 'E_41', 'PX_41', 'PY_41', 'PZ_41', 'E_42', 'PX_42', 'PY_42', 'PZ_42', 'E_43', 'PX_43', 'PY_43', 'PZ_43', 'E_44', 'PX_44', 'PY_44', 'PZ_44', 'E_45', 'PX_45', 'PY_45', 'PZ_45', 'E_46', 'PX_46', 'PY_46', 'PZ_46', 'E_47', 'PX_47', 'PY_47', 'PZ_47', 'E_48', 'PX_48', 'PY_48', 'PZ_48', 'E_49', 'PX_49', 'PY_49', 'PZ_49', 'E_50', 'PX_50', 'PY_50', 'PZ_50', 'E_51', 'PX_51', 'PY_51', 'PZ_51', 'E_52', 'PX_52', 'PY_52', 'PZ_52', 'E_53', 'PX_53', 'PY_53', 'PZ_53', 'E_54', 'PX_54', 'PY_54', 'PZ_54', 'E_55', 'PX_55', 'PY_55', 'PZ_55', 'E_56', 'PX_56', 'PY_56', 'PZ_56', 'E_57', 'PX_57', 'PY_57', 'PZ_57', 'E_58', 'PX_58', 'PY_58', 'PZ_58', 'E_59', 'PX_59', 'PY_59', 'PZ_59', 'E_60', 'PX_60', 'PY_60', 'PZ_60', 'E_61', 'PX_61', 'PY_61', 'PZ_61', 'E_62', 'PX_62', 'PY_62', 'PZ_62', 'E_63', 'PX_63', 'PY_63', 'PZ_63', 'E_64', 'PX_64', 'PY_64', 'PZ_64', 'E_65', 'PX_65', 'PY_65', 'PZ_65', 'E_66', 'PX_66', 'PY_66', 'PZ_66', 'E_67', 'PX_67', 'PY_67', 'PZ_67', 'E_68', 'PX_68', 'PY_68', 'PZ_68', 'E_69', 'PX_69', 'PY_69', 'PZ_69', 'E_70', 'PX_70', 'PY_70', 'PZ_70', 'E_71', 'PX_71', 'PY_71', 'PZ_71', 'E_72', 'PX_72', 'PY_72', 'PZ_72', 'E_73', 'PX_73', 'PY_73', 'PZ_73', 'E_74', 'PX_74', 'PY_74', 'PZ_74', 'E_75', 'PX_75', 'PY_75', 'PZ_75', 'E_76', 'PX_76', 'PY_76', 'PZ_76', 'E_77', 'PX_77', 'PY_77', 'PZ_77', 'E_78', 'PX_78', 'PY_78', 'PZ_78', 'E_79', 'PX_79', 'PY_79', 'PZ_79', 'E_80', 'PX_80', 'PY_80', 'PZ_80', 'E_81', 'PX_81', 'PY_81', 'PZ_81', 'E_82', 'PX_82', 'PY_82', 'PZ_82', 'E_83', 'PX_83', 'PY_83', 'PZ_83', 'E_84', 'PX_84', 'PY_84', 'PZ_84', 'E_85', 'PX_85', 'PY_85', 'PZ_85', 'E_86', 'PX_86', 'PY_86', 'PZ_86', 'E_87', 'PX_87', 'PY_87', 'PZ_87', 'E_88', 'PX_88', 'PY_88', 'PZ_88', 'E_89', 'PX_89', 'PY_89', 'PZ_89', 'E_90', 'PX_90', 'PY_90', 'PZ_90', 'E_91', 'PX_91', 'PY_91', 'PZ_91', 'E_92', 'PX_92', 'PY_92', 'PZ_92', 'E_93', 'PX_93', 'PY_93', 'PZ_93', 'E_94', 'PX_94', 'PY_94', 'PZ_94', 'E_95', 'PX_95', 'PY_95', 'PZ_95', 'E_96', 'PX_96', 'PY_96', 'PZ_96', 'E_97', 'PX_97', 'PY_97', 'PZ_97', 'E_98', 'PX_98', 'PY_98', 'PZ_98', 'E_99', 'PX_99', 'PY_99', 'PZ_99', 'E_100', 'PX_100', 'PY_100', 'PZ_100', 'E_101', 'PX_101', 'PY_101', 'PZ_101', 'E_102', 'PX_102', 'PY_102', 'PZ_102', 'E_103', 'PX_103', 'PY_103', 'PZ_103', 'E_104', 'PX_104', 'PY_104', 'PZ_104', 'E_105', 'PX_105', 'PY_105', 'PZ_105', 'E_106', 'PX_106', 'PY_106', 'PZ_106', 'E_107', 'PX_107', 'PY_107', 'PZ_107', 'E_108', 'PX_108', 'PY_108', 'PZ_108', 'E_109', 'PX_109', 'PY_109', 'PZ_109', 'E_110', 'PX_110', 'PY_110', 'PZ_110', 'E_111', 'PX_111', 'PY_111', 'PZ_111', 'E_112', 'PX_112', 'PY_112', 'PZ_112', 'E_113', 'PX_113', 'PY_113', 'PZ_113', 'E_114', 'PX_114', 'PY_114', 'PZ_114', 'E_115', 'PX_115', 'PY_115', 'PZ_115', 'E_116', 'PX_116', 'PY_116', 'PZ_116', 'E_117', 'PX_117', 'PY_117', 'PZ_117', 'E_118', 'PX_118', 'PY_118', 'PZ_118', 'E_119', 'PX_119', 'PY_119', 'PZ_119', 'E_120', 'PX_120', 'PY_120', 'PZ_120', 'E_121', 'PX_121', 'PY_121', 'PZ_121', 'E_122', 'PX_122', 'PY_122', 'PZ_122', 'E_123', 'PX_123', 'PY_123', 'PZ_123', 'E_124', 'PX_124', 'PY_124', 'PZ_124', 'E_125', 'PX_125', 'PY_125', 'PZ_125', 'E_126', 'PX_126', 'PY_126', 'PZ_126', 'E_127', 'PX_127', 'PY_127', 'PZ_127', 'E_128', 'PX_128', 'PY_128', 'PZ_128', 'E_129', 'PX_129', 'PY_129', 'PZ_129', 'E_130', 'PX_130', 'PY_130', 'PZ_130', 'E_131', 'PX_131', 'PY_131', 'PZ_131', 'E_132', 'PX_132', 'PY_132', 'PZ_132', 'E_133', 'PX_133', 'PY_133', 'PZ_133', 'E_134', 'PX_134', 'PY_134', 'PZ_134', 'E_135', 'PX_135', 'PY_135', 'PZ_135', 'E_136', 'PX_136', 'PY_136', 'PZ_136', 'E_137', 'PX_137', 'PY_137', 'PZ_137', 'E_138', 'PX_138', 'PY_138', 'PZ_138', 'E_139', 'PX_139', 'PY_139', 'PZ_139', 'E_140', 'PX_140', 'PY_140', 'PZ_140', 'E_141', 'PX_141', 'PY_141', 'PZ_141', 'E_142', 'PX_142', 'PY_142', 'PZ_142', 'E_143', 'PX_143', 'PY_143', 'PZ_143', 'E_144', 'PX_144', 'PY_144', 'PZ_144', 'E_145', 'PX_145', 'PY_145', 'PZ_145', 'E_146', 'PX_146', 'PY_146', 'PZ_146', 'E_147', 'PX_147', 'PY_147', 'PZ_147', 'E_148', 'PX_148', 'PY_148', 'PZ_148', 'E_149', 'PX_149', 'PY_149', 'PZ_149', 'E_150', 'PX_150', 'PY_150', 'PZ_150', 'E_151', 'PX_151', 'PY_151', 'PZ_151', 'E_152', 'PX_152', 'PY_152', 'PZ_152', 'E_153', 'PX_153', 'PY_153', 'PZ_153', 'E_154', 'PX_154', 'PY_154', 'PZ_154', 'E_155', 'PX_155', 'PY_155', 'PZ_155', 'E_156', 'PX_156', 'PY_156', 'PZ_156', 'E_157', 'PX_157', 'PY_157', 'PZ_157', 'E_158', 'PX_158', 'PY_158', 'PZ_158', 'E_159', 'PX_159', 'PY_159', 'PZ_159', 'E_160', 'PX_160', 'PY_160', 'PZ_160', 'E_161', 'PX_161', 'PY_161', 'PZ_161', 'E_162', 'PX_162', 'PY_162', 'PZ_162', 'E_163', 'PX_163', 'PY_163', 'PZ_163', 'E_164', 'PX_164', 'PY_164', 'PZ_164', 'E_165', 'PX_165', 'PY_165', 'PZ_165', 'E_166', 'PX_166', 'PY_166', 'PZ_166', 'E_167', 'PX_167', 'PY_167', 'PZ_167', 'E_168', 'PX_168', 'PY_168', 'PZ_168', 'E_169', 'PX_169', 'PY_169', 'PZ_169', 'E_170', 'PX_170', 'PY_170', 'PZ_170', 'E_171', 'PX_171', 'PY_171', 'PZ_171', 'E_172', 'PX_172', 'PY_172', 'PZ_172', 'E_173', 'PX_173', 'PY_173', 'PZ_173', 'E_174', 'PX_174', 'PY_174', 'PZ_174', 'E_175', 'PX_175', 'PY_175', 'PZ_175', 'E_176', 'PX_176', 'PY_176', 'PZ_176', 'E_177', 'PX_177', 'PY_177', 'PZ_177', 'E_178', 'PX_178', 'PY_178', 'PZ_178', 'E_179', 'PX_179', 'PY_179', 'PZ_179', 'E_180', 'PX_180', 'PY_180', 'PZ_180', 'E_181', 'PX_181', 'PY_181', 'PZ_181', 'E_182', 'PX_182', 'PY_182', 'PZ_182', 'E_183', 'PX_183', 'PY_183', 'PZ_183', 'E_184', 'PX_184', 'PY_184', 'PZ_184', 'E_185', 'PX_185', 'PY_185', 'PZ_185', 'E_186', 'PX_186', 'PY_186', 'PZ_186', 'E_187', 'PX_187', 'PY_187', 'PZ_187', 'E_188', 'PX_188', 'PY_188', 'PZ_188', 'E_189', 'PX_189', 'PY_189', 'PZ_189', 'E_190', 'PX_190', 'PY_190', 'PZ_190', 'E_191', 'PX_191', 'PY_191', 'PZ_191', 'E_192', 'PX_192', 'PY_192', 'PZ_192', 'E_193', 'PX_193', 'PY_193', 'PZ_193', 'E_194', 'PX_194', 'PY_194', 'PZ_194', 'E_195', 'PX_195', 'PY_195', 'PZ_195', 'E_196', 'PX_196', 'PY_196', 'PZ_196', 'E_197', 'PX_197', 'PY_197', 'PZ_197', 'E_198', 'PX_198', 'PY_198', 'PZ_198', 'E_199', 'PX_199', 'PY_199', 'PZ_199', 'truthE', 'truthPX', 'truthPY', 'truthPZ', 'ttv', 'is_signal_new'],
        num_rows: 403000
    })
})

we see it is similar to a Python dictionary, with each key corresponding to a different split. And we can use the usual dictionary syntax to access an individual split:

top_tagging_ds["train"]

Dataset({
    features: ['E_0', 'PX_0', 'PY_0', 'PZ_0', 'E_1', 'PX_1', 'PY_1', 'PZ_1', 'E_2', 'PX_2', 'PY_2', 'PZ_2', 'E_3', 'PX_3', 'PY_3', 'PZ_3', 'E_4', 'PX_4', 'PY_4', 'PZ_4', 'E_5', 'PX_5', 'PY_5', 'PZ_5', 'E_6', 'PX_6', 'PY_6', 'PZ_6', 'E_7', 'PX_7', 'PY_7', 'PZ_7', 'E_8', 'PX_8', 'PY_8', 'PZ_8', 'E_9', 'PX_9', 'PY_9', 'PZ_9', 'E_10', 'PX_10', 'PY_10', 'PZ_10', 'E_11', 'PX_11', 'PY_11', 'PZ_11', 'E_12', 'PX_12', 'PY_12', 'PZ_12', 'E_13', 'PX_13', 'PY_13', 'PZ_13', 'E_14', 'PX_14', 'PY_14', 'PZ_14', 'E_15', 'PX_15', 'PY_15', 'PZ_15', 'E_16', 'PX_16', 'PY_16', 'PZ_16', 'E_17', 'PX_17', 'PY_17', 'PZ_17', 'E_18', 'PX_18', 'PY_18', 'PZ_18', 'E_19', 'PX_19', 'PY_19', 'PZ_19', 'E_20', 'PX_20', 'PY_20', 'PZ_20', 'E_21', 'PX_21', 'PY_21', 'PZ_21', 'E_22', 'PX_22', 'PY_22', 'PZ_22', 'E_23', 'PX_23', 'PY_23', 'PZ_23', 'E_24', 'PX_24', 'PY_24', 'PZ_24', 'E_25', 'PX_25', 'PY_25', 'PZ_25', 'E_26', 'PX_26', 'PY_26', 'PZ_26', 'E_27', 'PX_27', 'PY_27', 'PZ_27', 'E_28', 'PX_28', 'PY_28', 'PZ_28', 'E_29', 'PX_29', 'PY_29', 'PZ_29', 'E_30', 'PX_30', 'PY_30', 'PZ_30', 'E_31', 'PX_31', 'PY_31', 'PZ_31', 'E_32', 'PX_32', 'PY_32', 'PZ_32', 'E_33', 'PX_33', 'PY_33', 'PZ_33', 'E_34', 'PX_34', 'PY_34', 'PZ_34', 'E_35', 'PX_35', 'PY_35', 'PZ_35', 'E_36', 'PX_36', 'PY_36', 'PZ_36', 'E_37', 'PX_37', 'PY_37', 'PZ_37', 'E_38', 'PX_38', 'PY_38', 'PZ_38', 'E_39', 'PX_39', 'PY_39', 'PZ_39', 'E_40', 'PX_40', 'PY_40', 'PZ_40', 'E_41', 'PX_41', 'PY_41', 'PZ_41', 'E_42', 'PX_42', 'PY_42', 'PZ_42', 'E_43', 'PX_43', 'PY_43', 'PZ_43', 'E_44', 'PX_44', 'PY_44', 'PZ_44', 'E_45', 'PX_45', 'PY_45', 'PZ_45', 'E_46', 'PX_46', 'PY_46', 'PZ_46', 'E_47', 'PX_47', 'PY_47', 'PZ_47', 'E_48', 'PX_48', 'PY_48', 'PZ_48', 'E_49', 'PX_49', 'PY_49', 'PZ_49', 'E_50', 'PX_50', 'PY_50', 'PZ_50', 'E_51', 'PX_51', 'PY_51', 'PZ_51', 'E_52', 'PX_52', 'PY_52', 'PZ_52', 'E_53', 'PX_53', 'PY_53', 'PZ_53', 'E_54', 'PX_54', 'PY_54', 'PZ_54', 'E_55', 'PX_55', 'PY_55', 'PZ_55', 'E_56', 'PX_56', 'PY_56', 'PZ_56', 'E_57', 'PX_57', 'PY_57', 'PZ_57', 'E_58', 'PX_58', 'PY_58', 'PZ_58', 'E_59', 'PX_59', 'PY_59', 'PZ_59', 'E_60', 'PX_60', 'PY_60', 'PZ_60', 'E_61', 'PX_61', 'PY_61', 'PZ_61', 'E_62', 'PX_62', 'PY_62', 'PZ_62', 'E_63', 'PX_63', 'PY_63', 'PZ_63', 'E_64', 'PX_64', 'PY_64', 'PZ_64', 'E_65', 'PX_65', 'PY_65', 'PZ_65', 'E_66', 'PX_66', 'PY_66', 'PZ_66', 'E_67', 'PX_67', 'PY_67', 'PZ_67', 'E_68', 'PX_68', 'PY_68', 'PZ_68', 'E_69', 'PX_69', 'PY_69', 'PZ_69', 'E_70', 'PX_70', 'PY_70', 'PZ_70', 'E_71', 'PX_71', 'PY_71', 'PZ_71', 'E_72', 'PX_72', 'PY_72', 'PZ_72', 'E_73', 'PX_73', 'PY_73', 'PZ_73', 'E_74', 'PX_74', 'PY_74', 'PZ_74', 'E_75', 'PX_75', 'PY_75', 'PZ_75', 'E_76', 'PX_76', 'PY_76', 'PZ_76', 'E_77', 'PX_77', 'PY_77', 'PZ_77', 'E_78', 'PX_78', 'PY_78', 'PZ_78', 'E_79', 'PX_79', 'PY_79', 'PZ_79', 'E_80', 'PX_80', 'PY_80', 'PZ_80', 'E_81', 'PX_81', 'PY_81', 'PZ_81', 'E_82', 'PX_82', 'PY_82', 'PZ_82', 'E_83', 'PX_83', 'PY_83', 'PZ_83', 'E_84', 'PX_84', 'PY_84', 'PZ_84', 'E_85', 'PX_85', 'PY_85', 'PZ_85', 'E_86', 'PX_86', 'PY_86', 'PZ_86', 'E_87', 'PX_87', 'PY_87', 'PZ_87', 'E_88', 'PX_88', 'PY_88', 'PZ_88', 'E_89', 'PX_89', 'PY_89', 'PZ_89', 'E_90', 'PX_90', 'PY_90', 'PZ_90', 'E_91', 'PX_91', 'PY_91', 'PZ_91', 'E_92', 'PX_92', 'PY_92', 'PZ_92', 'E_93', 'PX_93', 'PY_93', 'PZ_93', 'E_94', 'PX_94', 'PY_94', 'PZ_94', 'E_95', 'PX_95', 'PY_95', 'PZ_95', 'E_96', 'PX_96', 'PY_96', 'PZ_96', 'E_97', 'PX_97', 'PY_97', 'PZ_97', 'E_98', 'PX_98', 'PY_98', 'PZ_98', 'E_99', 'PX_99', 'PY_99', 'PZ_99', 'E_100', 'PX_100', 'PY_100', 'PZ_100', 'E_101', 'PX_101', 'PY_101', 'PZ_101', 'E_102', 'PX_102', 'PY_102', 'PZ_102', 'E_103', 'PX_103', 'PY_103', 'PZ_103', 'E_104', 'PX_104', 'PY_104', 'PZ_104', 'E_105', 'PX_105', 'PY_105', 'PZ_105', 'E_106', 'PX_106', 'PY_106', 'PZ_106', 'E_107', 'PX_107', 'PY_107', 'PZ_107', 'E_108', 'PX_108', 'PY_108', 'PZ_108', 'E_109', 'PX_109', 'PY_109', 'PZ_109', 'E_110', 'PX_110', 'PY_110', 'PZ_110', 'E_111', 'PX_111', 'PY_111', 'PZ_111', 'E_112', 'PX_112', 'PY_112', 'PZ_112', 'E_113', 'PX_113', 'PY_113', 'PZ_113', 'E_114', 'PX_114', 'PY_114', 'PZ_114', 'E_115', 'PX_115', 'PY_115', 'PZ_115', 'E_116', 'PX_116', 'PY_116', 'PZ_116', 'E_117', 'PX_117', 'PY_117', 'PZ_117', 'E_118', 'PX_118', 'PY_118', 'PZ_118', 'E_119', 'PX_119', 'PY_119', 'PZ_119', 'E_120', 'PX_120', 'PY_120', 'PZ_120', 'E_121', 'PX_121', 'PY_121', 'PZ_121', 'E_122', 'PX_122', 'PY_122', 'PZ_122', 'E_123', 'PX_123', 'PY_123', 'PZ_123', 'E_124', 'PX_124', 'PY_124', 'PZ_124', 'E_125', 'PX_125', 'PY_125', 'PZ_125', 'E_126', 'PX_126', 'PY_126', 'PZ_126', 'E_127', 'PX_127', 'PY_127', 'PZ_127', 'E_128', 'PX_128', 'PY_128', 'PZ_128', 'E_129', 'PX_129', 'PY_129', 'PZ_129', 'E_130', 'PX_130', 'PY_130', 'PZ_130', 'E_131', 'PX_131', 'PY_131', 'PZ_131', 'E_132', 'PX_132', 'PY_132', 'PZ_132', 'E_133', 'PX_133', 'PY_133', 'PZ_133', 'E_134', 'PX_134', 'PY_134', 'PZ_134', 'E_135', 'PX_135', 'PY_135', 'PZ_135', 'E_136', 'PX_136', 'PY_136', 'PZ_136', 'E_137', 'PX_137', 'PY_137', 'PZ_137', 'E_138', 'PX_138', 'PY_138', 'PZ_138', 'E_139', 'PX_139', 'PY_139', 'PZ_139', 'E_140', 'PX_140', 'PY_140', 'PZ_140', 'E_141', 'PX_141', 'PY_141', 'PZ_141', 'E_142', 'PX_142', 'PY_142', 'PZ_142', 'E_143', 'PX_143', 'PY_143', 'PZ_143', 'E_144', 'PX_144', 'PY_144', 'PZ_144', 'E_145', 'PX_145', 'PY_145', 'PZ_145', 'E_146', 'PX_146', 'PY_146', 'PZ_146', 'E_147', 'PX_147', 'PY_147', 'PZ_147', 'E_148', 'PX_148', 'PY_148', 'PZ_148', 'E_149', 'PX_149', 'PY_149', 'PZ_149', 'E_150', 'PX_150', 'PY_150', 'PZ_150', 'E_151', 'PX_151', 'PY_151', 'PZ_151', 'E_152', 'PX_152', 'PY_152', 'PZ_152', 'E_153', 'PX_153', 'PY_153', 'PZ_153', 'E_154', 'PX_154', 'PY_154', 'PZ_154', 'E_155', 'PX_155', 'PY_155', 'PZ_155', 'E_156', 'PX_156', 'PY_156', 'PZ_156', 'E_157', 'PX_157', 'PY_157', 'PZ_157', 'E_158', 'PX_158', 'PY_158', 'PZ_158', 'E_159', 'PX_159', 'PY_159', 'PZ_159', 'E_160', 'PX_160', 'PY_160', 'PZ_160', 'E_161', 'PX_161', 'PY_161', 'PZ_161', 'E_162', 'PX_162', 'PY_162', 'PZ_162', 'E_163', 'PX_163', 'PY_163', 'PZ_163', 'E_164', 'PX_164', 'PY_164', 'PZ_164', 'E_165', 'PX_165', 'PY_165', 'PZ_165', 'E_166', 'PX_166', 'PY_166', 'PZ_166', 'E_167', 'PX_167', 'PY_167', 'PZ_167', 'E_168', 'PX_168', 'PY_168', 'PZ_168', 'E_169', 'PX_169', 'PY_169', 'PZ_169', 'E_170', 'PX_170', 'PY_170', 'PZ_170', 'E_171', 'PX_171', 'PY_171', 'PZ_171', 'E_172', 'PX_172', 'PY_172', 'PZ_172', 'E_173', 'PX_173', 'PY_173', 'PZ_173', 'E_174', 'PX_174', 'PY_174', 'PZ_174', 'E_175', 'PX_175', 'PY_175', 'PZ_175', 'E_176', 'PX_176', 'PY_176', 'PZ_176', 'E_177', 'PX_177', 'PY_177', 'PZ_177', 'E_178', 'PX_178', 'PY_178', 'PZ_178', 'E_179', 'PX_179', 'PY_179', 'PZ_179', 'E_180', 'PX_180', 'PY_180', 'PZ_180', 'E_181', 'PX_181', 'PY_181', 'PZ_181', 'E_182', 'PX_182', 'PY_182', 'PZ_182', 'E_183', 'PX_183', 'PY_183', 'PZ_183', 'E_184', 'PX_184', 'PY_184', 'PZ_184', 'E_185', 'PX_185', 'PY_185', 'PZ_185', 'E_186', 'PX_186', 'PY_186', 'PZ_186', 'E_187', 'PX_187', 'PY_187', 'PZ_187', 'E_188', 'PX_188', 'PY_188', 'PZ_188', 'E_189', 'PX_189', 'PY_189', 'PZ_189', 'E_190', 'PX_190', 'PY_190', 'PZ_190', 'E_191', 'PX_191', 'PY_191', 'PZ_191', 'E_192', 'PX_192', 'PY_192', 'PZ_192', 'E_193', 'PX_193', 'PY_193', 'PZ_193', 'E_194', 'PX_194', 'PY_194', 'PZ_194', 'E_195', 'PX_195', 'PY_195', 'PZ_195', 'E_196', 'PX_196', 'PY_196', 'PZ_196', 'E_197', 'PX_197', 'PY_197', 'PZ_197', 'E_198', 'PX_198', 'PY_198', 'PZ_198', 'E_199', 'PX_199', 'PY_199', 'PZ_199', 'truthE', 'truthPX', 'truthPY', 'truthPZ', 'ttv', 'is_signal_new'],
    num_rows: 1211000
})

The Dataset object is one of the core data structures in 🤗 Datasets and behaves like an ordinary Python list, so we can query its length:

len(top_tagging_ds["train"])

or access a single element by its index:

top_tagging_ds["train"][0]

{'E_0': 474.0711364746094,
 'PX_0': -250.34703063964844,
 'PY_0': -223.65196228027344,
 'PZ_0': -334.73809814453125,
 'E_1': 103.23623657226562,
 'PX_1': -48.8662223815918,
 'PY_1': -56.790775299072266,
 'PZ_1': -71.0254898071289,
 'E_2': 105.25556945800781,
 'PX_2': -55.415000915527344,
 'PY_2': -49.96888732910156,
 'PZ_2': -74.23626708984375,
 'E_3': 40.17677688598633,
 'PX_3': -21.760696411132812,
 'PY_3': -18.71761131286621,
 'PZ_3': -28.112215042114258,
 'E_4': 22.4285831451416,
 'PX_4': -11.835756301879883,
 'PY_4': -10.374107360839844,
 'PZ_4': -15.979177474975586,
 'E_5': 20.334388732910156,
 'PX_5': -10.950518608093262,
 'PY_5': -9.545439720153809,
 'PZ_5': -14.228776931762695,
 'E_6': 19.030899047851562,
 'PX_6': -10.243264198303223,
 'PY_6': -9.004837036132812,
 'PZ_6': -13.272662162780762,
 'E_7': 13.460596084594727,
 'PX_7': -7.3433637619018555,
 'PY_7': -6.359743595123291,
 'PZ_7': -9.317526817321777,
 'E_8': 11.226107597351074,
 'PX_8': -5.981515884399414,
 'PY_8': -5.456268787384033,
 'PZ_8': -7.776637554168701,
 'E_9': 10.445060729980469,
 'PX_9': -5.460624694824219,
 'PY_9': -4.854524612426758,
 'PZ_9': -7.464211463928223,
 'E_10': 9.077269554138184,
 'PX_10': -5.811364650726318,
 'PY_10': -3.4854695796966553,
 'PZ_10': -6.039566993713379,
 'E_11': 9.056221008300781,
 'PX_11': -4.758406162261963,
 'PY_11': -4.0972113609313965,
 'PZ_11': -6.525762557983398,
 'E_12': 6.96318244934082,
 'PX_12': -3.490816593170166,
 'PY_12': -3.0960206985473633,
 'PZ_12': -5.168632984161377,
 'E_13': 5.772968769073486,
 'PX_13': -2.934152364730835,
 'PY_13': -2.7418923377990723,
 'PZ_13': -4.147281646728516,
 'E_14': 3.760998249053955,
 'PX_14': -2.1263434886932373,
 'PY_14': -1.7262251377105713,
 'PZ_14': -2.5775797367095947,
 'E_15': 2.9336676597595215,
 'PX_15': -1.5918165445327759,
 'PY_15': -1.277614951133728,
 'PZ_15': -2.107184410095215,
 'E_16': 2.729625940322876,
 'PX_16': -1.4158698320388794,
 'PY_16': -1.217659831047058,
 'PZ_16': -1.9908478260040283,
 'E_17': 2.7179172039031982,
 'PX_17': -1.3310680389404297,
 'PY_17': -1.0936245918273926,
 'PZ_17': -2.102217197418213,
 'E_18': 2.301811933517456,
 'PX_18': -1.1343910694122314,
 'PY_18': -1.1675245761871338,
 'PZ_18': -1.6273847818374634,
 'E_19': 1.5100955963134766,
 'PX_19': -1.0334879159927368,
 'PY_19': -0.7483676671981812,
 'PZ_19': -0.8076121807098389,
 'E_20': 1.340166687965393,
 'PX_20': -0.8890589475631714,
 'PY_20': -0.7178719639778137,
 'PZ_20': -0.700200617313385,
 'E_21': 1.0327688455581665,
 'PX_21': -0.11786821484565735,
 'PY_21': -0.49550384283065796,
 'PZ_21': -0.8984400629997253,
 'E_22': 0.4266541004180908,
 'PX_22': -0.2976595461368561,
 'PY_22': -0.10618320107460022,
 'PZ_22': -0.2866315543651581,
 'E_23': 0.0,
 'PX_23': 0.0,
 'PY_23': 0.0,
 'PZ_23': 0.0,
 'E_24': 0.0,
 'PX_24': 0.0,
 'PY_24': 0.0,
 'PZ_24': 0.0,
 'E_25': 0.0,
 'PX_25': 0.0,
 'PY_25': 0.0,
 'PZ_25': 0.0,
 'E_26': 0.0,
 'PX_26': 0.0,
 'PY_26': 0.0,
 'PZ_26': 0.0,
 'E_27': 0.0,
 'PX_27': 0.0,
 'PY_27': 0.0,
 'PZ_27': 0.0,
 'E_28': 0.0,
 'PX_28': 0.0,
 'PY_28': 0.0,
 'PZ_28': 0.0,
 'E_29': 0.0,
 'PX_29': 0.0,
 'PY_29': 0.0,
 'PZ_29': 0.0,
 'E_30': 0.0,
 'PX_30': 0.0,
 'PY_30': 0.0,
 'PZ_30': 0.0,
 'E_31': 0.0,
 'PX_31': 0.0,
 'PY_31': 0.0,
 'PZ_31': 0.0,
 'E_32': 0.0,
 'PX_32': 0.0,
 'PY_32': 0.0,
 'PZ_32': 0.0,
 'E_33': 0.0,
 'PX_33': 0.0,
 'PY_33': 0.0,
 'PZ_33': 0.0,
 'E_34': 0.0,
 'PX_34': 0.0,
 'PY_34': 0.0,
 'PZ_34': 0.0,
 'E_35': 0.0,
 'PX_35': 0.0,
 'PY_35': 0.0,
 'PZ_35': 0.0,
 'E_36': 0.0,
 'PX_36': 0.0,
 'PY_36': 0.0,
 'PZ_36': 0.0,
 'E_37': 0.0,
 'PX_37': 0.0,
 'PY_37': 0.0,
 'PZ_37': 0.0,
 'E_38': 0.0,
 'PX_38': 0.0,
 'PY_38': 0.0,
 'PZ_38': 0.0,
 'E_39': 0.0,
 'PX_39': 0.0,
 'PY_39': 0.0,
 'PZ_39': 0.0,
 'E_40': 0.0,
 'PX_40': 0.0,
 'PY_40': 0.0,
 'PZ_40': 0.0,
 'E_41': 0.0,
 'PX_41': 0.0,
 'PY_41': 0.0,
 'PZ_41': 0.0,
 'E_42': 0.0,
 'PX_42': 0.0,
 'PY_42': 0.0,
 'PZ_42': 0.0,
 'E_43': 0.0,
 'PX_43': 0.0,
 'PY_43': 0.0,
 'PZ_43': 0.0,
 'E_44': 0.0,
 'PX_44': 0.0,
 'PY_44': 0.0,
 'PZ_44': 0.0,
 'E_45': 0.0,
 'PX_45': 0.0,
 'PY_45': 0.0,
 'PZ_45': 0.0,
 'E_46': 0.0,
 'PX_46': 0.0,
 'PY_46': 0.0,
 'PZ_46': 0.0,
 'E_47': 0.0,
 'PX_47': 0.0,
 'PY_47': 0.0,
 'PZ_47': 0.0,
 'E_48': 0.0,
 'PX_48': 0.0,
 'PY_48': 0.0,
 'PZ_48': 0.0,
 'E_49': 0.0,
 'PX_49': 0.0,
 'PY_49': 0.0,
 'PZ_49': 0.0,
 'E_50': 0.0,
 'PX_50': 0.0,
 'PY_50': 0.0,
 'PZ_50': 0.0,
 'E_51': 0.0,
 'PX_51': 0.0,
 'PY_51': 0.0,
 'PZ_51': 0.0,
 'E_52': 0.0,
 'PX_52': 0.0,
 'PY_52': 0.0,
 'PZ_52': 0.0,
 'E_53': 0.0,
 'PX_53': 0.0,
 'PY_53': 0.0,
 'PZ_53': 0.0,
 'E_54': 0.0,
 'PX_54': 0.0,
 'PY_54': 0.0,
 'PZ_54': 0.0,
 'E_55': 0.0,
 'PX_55': 0.0,
 'PY_55': 0.0,
 'PZ_55': 0.0,
 'E_56': 0.0,
 'PX_56': 0.0,
 'PY_56': 0.0,
 'PZ_56': 0.0,
 'E_57': 0.0,
 'PX_57': 0.0,
 'PY_57': 0.0,
 'PZ_57': 0.0,
 'E_58': 0.0,
 'PX_58': 0.0,
 'PY_58': 0.0,
 'PZ_58': 0.0,
 'E_59': 0.0,
 'PX_59': 0.0,
 'PY_59': 0.0,
 'PZ_59': 0.0,
 'E_60': 0.0,
 'PX_60': 0.0,
 'PY_60': 0.0,
 'PZ_60': 0.0,
 'E_61': 0.0,
 'PX_61': 0.0,
 'PY_61': 0.0,
 'PZ_61': 0.0,
 'E_62': 0.0,
 'PX_62': 0.0,
 'PY_62': 0.0,
 'PZ_62': 0.0,
 'E_63': 0.0,
 'PX_63': 0.0,
 'PY_63': 0.0,
 'PZ_63': 0.0,
 'E_64': 0.0,
 'PX_64': 0.0,
 'PY_64': 0.0,
 'PZ_64': 0.0,
 'E_65': 0.0,
 'PX_65': 0.0,
 'PY_65': 0.0,
 'PZ_65': 0.0,
 'E_66': 0.0,
 'PX_66': 0.0,
 'PY_66': 0.0,
 'PZ_66': 0.0,
 'E_67': 0.0,
 'PX_67': 0.0,
 'PY_67': 0.0,
 'PZ_67': 0.0,
 'E_68': 0.0,
 'PX_68': 0.0,
 'PY_68': 0.0,
 'PZ_68': 0.0,
 'E_69': 0.0,
 'PX_69': 0.0,
 'PY_69': 0.0,
 'PZ_69': 0.0,
 'E_70': 0.0,
 'PX_70': 0.0,
 'PY_70': 0.0,
 'PZ_70': 0.0,
 'E_71': 0.0,
 'PX_71': 0.0,
 'PY_71': 0.0,
 'PZ_71': 0.0,
 'E_72': 0.0,
 'PX_72': 0.0,
 'PY_72': 0.0,
 'PZ_72': 0.0,
 'E_73': 0.0,
 'PX_73': 0.0,
 'PY_73': 0.0,
 'PZ_73': 0.0,
 'E_74': 0.0,
 'PX_74': 0.0,
 'PY_74': 0.0,
 'PZ_74': 0.0,
 'E_75': 0.0,
 'PX_75': 0.0,
 'PY_75': 0.0,
 'PZ_75': 0.0,
 'E_76': 0.0,
 'PX_76': 0.0,
 'PY_76': 0.0,
 'PZ_76': 0.0,
 'E_77': 0.0,
 'PX_77': 0.0,
 'PY_77': 0.0,
 'PZ_77': 0.0,
 'E_78': 0.0,
 'PX_78': 0.0,
 'PY_78': 0.0,
 'PZ_78': 0.0,
 'E_79': 0.0,
 'PX_79': 0.0,
 'PY_79': 0.0,
 'PZ_79': 0.0,
 'E_80': 0.0,
 'PX_80': 0.0,
 'PY_80': 0.0,
 'PZ_80': 0.0,
 'E_81': 0.0,
 'PX_81': 0.0,
 'PY_81': 0.0,
 'PZ_81': 0.0,
 'E_82': 0.0,
 'PX_82': 0.0,
 'PY_82': 0.0,
 'PZ_82': 0.0,
 'E_83': 0.0,
 'PX_83': 0.0,
 'PY_83': 0.0,
 'PZ_83': 0.0,
 'E_84': 0.0,
 'PX_84': 0.0,
 'PY_84': 0.0,
 'PZ_84': 0.0,
 'E_85': 0.0,
 'PX_85': 0.0,
 'PY_85': 0.0,
 'PZ_85': 0.0,
 'E_86': 0.0,
 'PX_86': 0.0,
 'PY_86': 0.0,
 'PZ_86': 0.0,
 'E_87': 0.0,
 'PX_87': 0.0,
 'PY_87': 0.0,
 'PZ_87': 0.0,
 'E_88': 0.0,
 'PX_88': 0.0,
 'PY_88': 0.0,
 'PZ_88': 0.0,
 'E_89': 0.0,
 'PX_89': 0.0,
 'PY_89': 0.0,
 'PZ_89': 0.0,
 'E_90': 0.0,
 'PX_90': 0.0,
 'PY_90': 0.0,
 'PZ_90': 0.0,
 'E_91': 0.0,
 'PX_91': 0.0,
 'PY_91': 0.0,
 'PZ_91': 0.0,
 'E_92': 0.0,
 'PX_92': 0.0,
 'PY_92': 0.0,
 'PZ_92': 0.0,
 'E_93': 0.0,
 'PX_93': 0.0,
 'PY_93': 0.0,
 'PZ_93': 0.0,
 'E_94': 0.0,
 'PX_94': 0.0,
 'PY_94': 0.0,
 'PZ_94': 0.0,
 'E_95': 0.0,
 'PX_95': 0.0,
 'PY_95': 0.0,
 'PZ_95': 0.0,
 'E_96': 0.0,
 'PX_96': 0.0,
 'PY_96': 0.0,
 'PZ_96': 0.0,
 'E_97': 0.0,
 'PX_97': 0.0,
 'PY_97': 0.0,
 'PZ_97': 0.0,
 'E_98': 0.0,
 'PX_98': 0.0,
 'PY_98': 0.0,
 'PZ_98': 0.0,
 'E_99': 0.0,
 'PX_99': 0.0,
 'PY_99': 0.0,
 'PZ_99': 0.0,
 'E_100': 0.0,
 'PX_100': 0.0,
 'PY_100': 0.0,
 'PZ_100': 0.0,
 'E_101': 0.0,
 'PX_101': 0.0,
 'PY_101': 0.0,
 'PZ_101': 0.0,
 'E_102': 0.0,
 'PX_102': 0.0,
 'PY_102': 0.0,
 'PZ_102': 0.0,
 'E_103': 0.0,
 'PX_103': 0.0,
 'PY_103': 0.0,
 'PZ_103': 0.0,
 'E_104': 0.0,
 'PX_104': 0.0,
 'PY_104': 0.0,
 'PZ_104': 0.0,
 'E_105': 0.0,
 'PX_105': 0.0,
 'PY_105': 0.0,
 'PZ_105': 0.0,
 'E_106': 0.0,
 'PX_106': 0.0,
 'PY_106': 0.0,
 'PZ_106': 0.0,
 'E_107': 0.0,
 'PX_107': 0.0,
 'PY_107': 0.0,
 'PZ_107': 0.0,
 'E_108': 0.0,
 'PX_108': 0.0,
 'PY_108': 0.0,
 'PZ_108': 0.0,
 'E_109': 0.0,
 'PX_109': 0.0,
 'PY_109': 0.0,
 'PZ_109': 0.0,
 'E_110': 0.0,
 'PX_110': 0.0,
 'PY_110': 0.0,
 'PZ_110': 0.0,
 'E_111': 0.0,
 'PX_111': 0.0,
 'PY_111': 0.0,
 'PZ_111': 0.0,
 'E_112': 0.0,
 'PX_112': 0.0,
 'PY_112': 0.0,
 'PZ_112': 0.0,
 'E_113': 0.0,
 'PX_113': 0.0,
 'PY_113': 0.0,
 'PZ_113': 0.0,
 'E_114': 0.0,
 'PX_114': 0.0,
 'PY_114': 0.0,
 'PZ_114': 0.0,
 'E_115': 0.0,
 'PX_115': 0.0,
 'PY_115': 0.0,
 'PZ_115': 0.0,
 'E_116': 0.0,
 'PX_116': 0.0,
 'PY_116': 0.0,
 'PZ_116': 0.0,
 'E_117': 0.0,
 'PX_117': 0.0,
 'PY_117': 0.0,
 'PZ_117': 0.0,
 'E_118': 0.0,
 'PX_118': 0.0,
 'PY_118': 0.0,
 'PZ_118': 0.0,
 'E_119': 0.0,
 'PX_119': 0.0,
 'PY_119': 0.0,
 'PZ_119': 0.0,
 'E_120': 0.0,
 'PX_120': 0.0,
 'PY_120': 0.0,
 'PZ_120': 0.0,
 'E_121': 0.0,
 'PX_121': 0.0,
 'PY_121': 0.0,
 'PZ_121': 0.0,
 'E_122': 0.0,
 'PX_122': 0.0,
 'PY_122': 0.0,
 'PZ_122': 0.0,
 'E_123': 0.0,
 'PX_123': 0.0,
 'PY_123': 0.0,
 'PZ_123': 0.0,
 'E_124': 0.0,
 'PX_124': 0.0,
 'PY_124': 0.0,
 'PZ_124': 0.0,
 'E_125': 0.0,
 'PX_125': 0.0,
 'PY_125': 0.0,
 'PZ_125': 0.0,
 'E_126': 0.0,
 'PX_126': 0.0,
 'PY_126': 0.0,
 'PZ_126': 0.0,
 'E_127': 0.0,
 'PX_127': 0.0,
 'PY_127': 0.0,
 'PZ_127': 0.0,
 'E_128': 0.0,
 'PX_128': 0.0,
 'PY_128': 0.0,
 'PZ_128': 0.0,
 'E_129': 0.0,
 'PX_129': 0.0,
 'PY_129': 0.0,
 'PZ_129': 0.0,
 'E_130': 0.0,
 'PX_130': 0.0,
 'PY_130': 0.0,
 'PZ_130': 0.0,
 'E_131': 0.0,
 'PX_131': 0.0,
 'PY_131': 0.0,
 'PZ_131': 0.0,
 'E_132': 0.0,
 'PX_132': 0.0,
 'PY_132': 0.0,
 'PZ_132': 0.0,
 'E_133': 0.0,
 'PX_133': 0.0,
 'PY_133': 0.0,
 'PZ_133': 0.0,
 'E_134': 0.0,
 'PX_134': 0.0,
 'PY_134': 0.0,
 'PZ_134': 0.0,
 'E_135': 0.0,
 'PX_135': 0.0,
 'PY_135': 0.0,
 'PZ_135': 0.0,
 'E_136': 0.0,
 'PX_136': 0.0,
 'PY_136': 0.0,
 'PZ_136': 0.0,
 'E_137': 0.0,
 'PX_137': 0.0,
 'PY_137': 0.0,
 'PZ_137': 0.0,
 'E_138': 0.0,
 'PX_138': 0.0,
 'PY_138': 0.0,
 'PZ_138': 0.0,
 'E_139': 0.0,
 'PX_139': 0.0,
 'PY_139': 0.0,
 'PZ_139': 0.0,
 'E_140': 0.0,
 'PX_140': 0.0,
 'PY_140': 0.0,
 'PZ_140': 0.0,
 'E_141': 0.0,
 'PX_141': 0.0,
 'PY_141': 0.0,
 'PZ_141': 0.0,
 'E_142': 0.0,
 'PX_142': 0.0,
 'PY_142': 0.0,
 'PZ_142': 0.0,
 'E_143': 0.0,
 'PX_143': 0.0,
 'PY_143': 0.0,
 'PZ_143': 0.0,
 'E_144': 0.0,
 'PX_144': 0.0,
 'PY_144': 0.0,
 'PZ_144': 0.0,
 'E_145': 0.0,
 'PX_145': 0.0,
 'PY_145': 0.0,
 'PZ_145': 0.0,
 'E_146': 0.0,
 'PX_146': 0.0,
 'PY_146': 0.0,
 'PZ_146': 0.0,
 'E_147': 0.0,
 'PX_147': 0.0,
 'PY_147': 0.0,
 'PZ_147': 0.0,
 'E_148': 0.0,
 'PX_148': 0.0,
 'PY_148': 0.0,
 'PZ_148': 0.0,
 'E_149': 0.0,
 'PX_149': 0.0,
 'PY_149': 0.0,
 'PZ_149': 0.0,
 'E_150': 0.0,
 'PX_150': 0.0,
 'PY_150': 0.0,
 'PZ_150': 0.0,
 'E_151': 0.0,
 'PX_151': 0.0,
 'PY_151': 0.0,
 'PZ_151': 0.0,
 'E_152': 0.0,
 'PX_152': 0.0,
 'PY_152': 0.0,
 'PZ_152': 0.0,
 'E_153': 0.0,
 'PX_153': 0.0,
 'PY_153': 0.0,
 'PZ_153': 0.0,
 'E_154': 0.0,
 'PX_154': 0.0,
 'PY_154': 0.0,
 'PZ_154': 0.0,
 'E_155': 0.0,
 'PX_155': 0.0,
 'PY_155': 0.0,
 'PZ_155': 0.0,
 'E_156': 0.0,
 'PX_156': 0.0,
 'PY_156': 0.0,
 'PZ_156': 0.0,
 'E_157': 0.0,
 'PX_157': 0.0,
 'PY_157': 0.0,
 'PZ_157': 0.0,
 'E_158': 0.0,
 'PX_158': 0.0,
 'PY_158': 0.0,
 'PZ_158': 0.0,
 'E_159': 0.0,
 'PX_159': 0.0,
 'PY_159': 0.0,
 'PZ_159': 0.0,
 'E_160': 0.0,
 'PX_160': 0.0,
 'PY_160': 0.0,
 'PZ_160': 0.0,
 'E_161': 0.0,
 'PX_161': 0.0,
 'PY_161': 0.0,
 'PZ_161': 0.0,
 'E_162': 0.0,
 'PX_162': 0.0,
 'PY_162': 0.0,
 'PZ_162': 0.0,
 'E_163': 0.0,
 'PX_163': 0.0,
 'PY_163': 0.0,
 'PZ_163': 0.0,
 'E_164': 0.0,
 'PX_164': 0.0,
 'PY_164': 0.0,
 'PZ_164': 0.0,
 'E_165': 0.0,
 'PX_165': 0.0,
 'PY_165': 0.0,
 'PZ_165': 0.0,
 'E_166': 0.0,
 'PX_166': 0.0,
 'PY_166': 0.0,
 'PZ_166': 0.0,
 'E_167': 0.0,
 'PX_167': 0.0,
 'PY_167': 0.0,
 'PZ_167': 0.0,
 'E_168': 0.0,
 'PX_168': 0.0,
 'PY_168': 0.0,
 'PZ_168': 0.0,
 'E_169': 0.0,
 'PX_169': 0.0,
 'PY_169': 0.0,
 'PZ_169': 0.0,
 'E_170': 0.0,
 'PX_170': 0.0,
 'PY_170': 0.0,
 'PZ_170': 0.0,
 'E_171': 0.0,
 'PX_171': 0.0,
 'PY_171': 0.0,
 'PZ_171': 0.0,
 'E_172': 0.0,
 'PX_172': 0.0,
 'PY_172': 0.0,
 'PZ_172': 0.0,
 'E_173': 0.0,
 'PX_173': 0.0,
 'PY_173': 0.0,
 'PZ_173': 0.0,
 'E_174': 0.0,
 'PX_174': 0.0,
 'PY_174': 0.0,
 'PZ_174': 0.0,
 'E_175': 0.0,
 'PX_175': 0.0,
 'PY_175': 0.0,
 'PZ_175': 0.0,
 'E_176': 0.0,
 'PX_176': 0.0,
 'PY_176': 0.0,
 'PZ_176': 0.0,
 'E_177': 0.0,
 'PX_177': 0.0,
 'PY_177': 0.0,
 'PZ_177': 0.0,
 'E_178': 0.0,
 'PX_178': 0.0,
 'PY_178': 0.0,
 'PZ_178': 0.0,
 'E_179': 0.0,
 'PX_179': 0.0,
 'PY_179': 0.0,
 'PZ_179': 0.0,
 'E_180': 0.0,
 'PX_180': 0.0,
 'PY_180': 0.0,
 'PZ_180': 0.0,
 'E_181': 0.0,
 'PX_181': 0.0,
 'PY_181': 0.0,
 'PZ_181': 0.0,
 'E_182': 0.0,
 'PX_182': 0.0,
 'PY_182': 0.0,
 'PZ_182': 0.0,
 'E_183': 0.0,
 'PX_183': 0.0,
 'PY_183': 0.0,
 'PZ_183': 0.0,
 'E_184': 0.0,
 'PX_184': 0.0,
 'PY_184': 0.0,
 'PZ_184': 0.0,
 'E_185': 0.0,
 'PX_185': 0.0,
 'PY_185': 0.0,
 'PZ_185': 0.0,
 'E_186': 0.0,
 'PX_186': 0.0,
 'PY_186': 0.0,
 'PZ_186': 0.0,
 'E_187': 0.0,
 'PX_187': 0.0,
 'PY_187': 0.0,
 'PZ_187': 0.0,
 'E_188': 0.0,
 'PX_188': 0.0,
 'PY_188': 0.0,
 'PZ_188': 0.0,
 'E_189': 0.0,
 'PX_189': 0.0,
 'PY_189': 0.0,
 'PZ_189': 0.0,
 'E_190': 0.0,
 'PX_190': 0.0,
 'PY_190': 0.0,
 'PZ_190': 0.0,
 'E_191': 0.0,
 'PX_191': 0.0,
 'PY_191': 0.0,
 'PZ_191': 0.0,
 'E_192': 0.0,
 'PX_192': 0.0,
 'PY_192': 0.0,
 'PZ_192': 0.0,
 'E_193': 0.0,
 'PX_193': 0.0,
 'PY_193': 0.0,
 'PZ_193': 0.0,
 'E_194': 0.0,
 'PX_194': 0.0,
 'PY_194': 0.0,
 'PZ_194': 0.0,
 'E_195': 0.0,
 'PX_195': 0.0,
 'PY_195': 0.0,
 'PZ_195': 0.0,
 'E_196': 0.0,
 'PX_196': 0.0,
 'PY_196': 0.0,
 'PZ_196': 0.0,
 'E_197': 0.0,
 'PX_197': 0.0,
 'PY_197': 0.0,
 'PZ_197': 0.0,
 'E_198': 0.0,
 'PX_198': 0.0,
 'PY_198': 0.0,
 'PZ_198': 0.0,
 'E_199': 0.0,
 'PX_199': 0.0,
 'PY_199': 0.0,
 'PZ_199': 0.0,
 'truthE': 0.0,
 'truthPX': 0.0,
 'truthPY': 0.0,
 'truthPZ': 0.0,
 'ttv': 0,
 'is_signal_new': 0}

Here we see that a single row is repesented as a dictionary where the keys correspond to the column names. Since we won’t need the top-quark 4-vector columns, let’s remove them along with the ttv one:

top_tagging_ds = top_tagging_ds.remove_columns(
    ["truthE", "truthPX", "truthPY", "truthPZ", "ttv"]
)

Although 🤗 Datasets provides a lot of low-level functionality for preprocessing datasets, it is often conventient to convert a Dataset object to a Pandas DataFrame. To enable the conversion, 🤗 Datasets provides a set_format() method that allows us to change the output format of the dataset:

# Convert output format to DataFrames
top_tagging_ds.set_format("pandas")
# Create DataFrames for the training and test splits
train_df, test_df = top_tagging_ds["train"][:], top_tagging_ds["test"][:]
# Peek at first few rows
train_df.head()

	E_0	PX_0	PY_0	PZ_0	E_1	PX_1	PY_1	PZ_1	E_2	PX_2	...
0	474.071136	-250.347031	-223.651962	-334.738098	103.236237	-48.866222	-56.790775	-71.025490	105.255569	-55.415001	...
1	150.504532	120.062393	76.852005	-48.274265	82.257057	63.801739	42.754807	-29.454842	48.573559	36.763199	...
2	251.645386	10.427651	-147.573746	203.564880	104.147797	10.718256	-54.497948	88.101395	78.043213	5.724113	...
3	451.566132	129.885437	-99.066292	-420.984100	208.410919	59.033958	-46.177090	-194.467941	190.183304	54.069675	...
4	399.093903	-168.432083	-47.205597	-358.717438	273.691956	-121.926941	-30.803854	-243.088928	152.837219	-44.400204	...

5 rows × 801 columns

As we can see, each row consists of 4-vectors \((E_i, p_{x_i}, p_{y_i}, p_{z_i})\) that correspond to the maximum 200 particles that make up each jet. We can also see that each jet has been padded with zeros, since most won’t have 200 particles. We also have an is_signal_new column that indicates whether the jet is a top quark signal (1) or QCD background (0).

Now that we’ve had a look at a sample of the raw data, let’s take a look at how we can convert it to a format that is suitable for neural networks!

Introducing fastai#

To train our model, we’ll use the fastai library. fastai is the most popular framework for training deep neural networks with PyTorch and provides various application-specific classes for different types of deep learning data structures and architectures. It is also designed with a layered API, which means:

We can use high-level components to quickly and easily get state-of-the-art results in standard deep learning domains
Low-level components can be mixed and matched to build new approaches

In particular, this approach will allow us in later lessons to use pure PyTorch code to define our models, and then let fastai take care of the training loop (which is often an error-prone process).

Basics of the API

The most common steps one takes when training a model in fastai are:

Create DataLoaders to feed batches of data to the model
Create a Learner which wraps the architecture, optimizer, and data, and automatically chooses an appropriate loss function where possible
Find a good learning rate
Train your model
Evaluate your model

Let’s go through each of these steps to build a neural network that can classify top quark jets from the QCD background!

From data to DataLoaders#

To wrangle our data in a format that’s suitable for training neural nets, we need to create an object called DataLoaders. To turn our dataset into a DataLoaders object we need to specify:

What type of data we are dealing with (tabular, images, etc)
How to get the examples
How to label each example
How to create the validation set

fastai provides a number of classes for different kinds of deep learning datasets and problems. In our case, the data is in a tabular format (i.e. a table of rows and columns), so we can use the TabularDataLoaders class:

# Downsample to ~0.5 if you're running on Colab / Kaggle which have limited RAM
frac_of_samples = 1.0
train_df = train_df.sample(int(frac_of_samples * len(train_df)), random_state=42)

features = list(train_df.drop(columns=["is_signal_new"]).columns)
splits = RandomSplitter(valid_pct=0.20, seed=42)(range_of(train_df))

dls = TabularDataLoaders.from_df(
    df=train_df,
    cont_names=features,
    y_names="is_signal_new",
    y_block=CategoryBlock,
    splits=splits,
    bs=1024,
)

Let’s unpack this code a bit. The first thing we’ve specified is which columns of our dataset correspond to continuous features via the cont_names argument. To do this, we’ve simply grabbed all column names from our DataFrame, except for the label column is_signal_new. Next, we’ve specified which column is the target in y_names and that this is a categorical feature with CategoryBlock. Finally we’ve specified the training and validation splits with RandomSplitter and picked a batch size of 1,024 examples.

After we’ve defined a DataLoaders object, we can take a look at the data by using the show_batch() method:

dls.show_batch()

	E_0	PX_0	PY_0	PZ_0	E_1	PX_1	PY_1	PZ_1	E_2	PX_2	PY_2	PZ_2	E_3	PX_3	PY_3	PZ_3	E_4	PX_4	PY_4	PZ_4	E_5	PX_5	PY_5	PZ_5	E_6	PX_6	PY_6	PZ_6	E_7	PX_7	PY_7	PZ_7	E_8	PX_8	PY_8	PZ_8	E_9	PX_9	PY_9	PZ_9	E_10	PX_10	PY_10	PZ_10	E_11	PX_11	PY_11	PZ_11	E_12	PX_12	PY_12	PZ_12	E_13	PX_13	PY_13	PZ_13	E_14	PX_14	PY_14	PZ_14	E_15	PX_15	PY_15	PZ_15	E_16	PX_16	PY_16	PZ_16	E_17	PX_17	PY_17	PZ_17	E_18	PX_18	PY_18	PZ_18	E_19	PX_19	PY_19	PZ_19	E_20	PX_20	PY_20	PZ_20	E_21	PX_21	PY_21	PZ_21	E_22	PX_22	PY_22	PZ_22	E_23	PX_23	PY_23	PZ_23	E_24	PX_24	PY_24	PZ_24	E_25	PX_25	PY_25	PZ_25	E_26	PX_26	PY_26	PZ_26	E_27	PX_27	PY_27	PZ_27	E_28	PX_28	PY_28	PZ_28	E_29	PX_29	PY_29	PZ_29	E_30	PX_30	PY_30	PZ_30	E_31	PX_31	PY_31	PZ_31	E_32	PX_32	PY_32	PZ_32	E_33	PX_33	PY_33	PZ_33	E_34	PX_34	PY_34	PZ_34	E_35	PX_35	PY_35	PZ_35	E_36	PX_36	PY_36	PZ_36	E_37	PX_37	PY_37	PZ_37	E_38	PX_38	PY_38	PZ_38	E_39	PX_39	PY_39	PZ_39	E_40	PX_40	PY_40	PZ_40	E_41	PX_41	PY_41	PZ_41	E_42	PX_42	PY_42	PZ_42	E_43	PX_43	PY_43	PZ_43	E_44	PX_44	PY_44	PZ_44	E_45	PX_45	PY_45	PZ_45	E_46	PX_46	PY_46	PZ_46	E_47	PX_47	PY_47	PZ_47	E_48	PX_48	PY_48	PZ_48	E_49	PX_49	PY_49	PZ_49	E_50	PX_50	PY_50	PZ_50	E_51	PX_51	PY_51	PZ_51	E_52	PX_52	PY_52	PZ_52	E_53	PX_53	PY_53	PZ_53	E_54	PX_54	PY_54	PZ_54	E_55	PX_55	PY_55	PZ_55	E_56	PX_56	PY_56	PZ_56	E_57	PX_57	PY_57	PZ_57	E_58	PX_58	PY_58	PZ_58	E_59	PX_59	PY_59	PZ_59	E_60	PX_60	PY_60	PZ_60	E_61	PX_61	PY_61	PZ_61	E_62	PX_62	PY_62	PZ_62	E_63	PX_63	PY_63	PZ_63	E_64	PX_64	PY_64	PZ_64	E_65	PX_65	PY_65	PZ_65	E_66	PX_66	PY_66	PZ_66	E_67	PX_67	PY_67	PZ_67	E_68	PX_68	PY_68	PZ_68	E_69	PX_69	PY_69	PZ_69	E_70	PX_70	PY_70	PZ_70	E_71	PX_71	PY_71	PZ_71	E_72	PX_72	PY_72	PZ_72	E_73	PX_73	PY_73	PZ_73	E_74	PX_74	PY_74	PZ_74	E_75	PX_75	PY_75	PZ_75	E_76	PX_76	PY_76	PZ_76	E_77	PX_77	PY_77	PZ_77	E_78	PX_78	PY_78	PZ_78	E_79	PX_79	PY_79	PZ_79	E_80	PX_80	PY_80	PZ_80	E_81	PX_81	PY_81	PZ_81	E_82	PX_82	PY_82	PZ_82	E_83	PX_83	PY_83	PZ_83	E_84	PX_84	PY_84	PZ_84	E_85	PX_85	PY_85	PZ_85	E_86	PX_86	PY_86	PZ_86	E_87	PX_87	PY_87	PZ_87	is_signal_new
0	156.651123	147.405258	26.006823	-46.205074	66.166000	61.714252	13.618260	-19.591679	59.187538	43.524006	18.544239	-35.565948	52.280418	39.901722	14.766282	-30.381767	46.479649	37.455952	-4.363357	-27.172979	37.165035	35.369465	1.354363	-11.331660	36.232121	27.362452	10.842762	-21.130484	35.059006	26.817917	10.583280	-19.948118	27.499258	21.504807	-7.419861	-15.449861	24.723345	18.445026	7.439287	-14.686109	21.618416	17.497467	5.555872	-11.416079	20.788595	15.854955	6.082806	-11.991064	18.213100	13.441398	6.698515	-10.304162	12.677663	12.036289	2.944411	-2.679800	12.660934	11.887339	1.900199	-3.921694	9.484063	7.706819	2.073808	-5.123643	8.043362	6.180357	2.354782	-4.577538	7.829610	5.816335	2.422960	-4.647829	7.204622	5.786285	-0.912594	-4.194359	6.569118	4.983078	-0.247118	-4.273309	5.204410	4.723341	0.999273	-1.943549	5.212553	4.281167	1.108515	-2.759259	5.343522	4.059355	1.723231	-3.017504	3.995430	2.993573	1.290984	-2.309836	3.862169	3.138426	0.844511	-2.086487	3.488208	3.158298	0.699826	-1.304989	2.978119	2.682876	-0.341816	-1.246809	3.168274	2.370644	-0.752072	-1.962752	2.761582	2.272017	0.939171	-1.257868	3.092704	2.281398	-0.749264	-1.949011	3.142989	2.093421	1.106632	-2.066720	2.061056	1.755722	0.585744	-0.906805	2.281940	1.579042	-0.750299	-1.466605	1.614905	1.292173	-0.374225	-0.893399	1.415592	1.206074	0.533570	-0.514383	1.675033	1.266876	0.103127	-1.090928	1.120713	1.010384	0.072985	-0.479369	0.886037	0.871607	0.097665	-0.125795	0.932767	0.800974	0.097008	-0.468065	1.065342	0.602590	0.389746	-0.787360	0.728177	0.659168	0.136783	-0.277542	0.746282	0.631983	0.138891	-0.371811	0.668374	0.626190	-0.143476	-0.184459	0.616981	0.575796	0.220805	-0.019220	0.595697	0.543712	-0.019745	-0.242575	0.647482	0.387422	0.197245	-0.479825	0.660883	0.346850	0.072733	-0.557828	0.000000	0.000000	0.000000	0.00000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.00000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.00000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.00000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.00000	0.00000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.00000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.0000	0.000000	0.000000	0.000000	0.00000	0.00000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	1
1	217.770920	112.669815	-55.818962	-177.803070	169.487793	57.047081	-86.189850	-134.324432	79.572060	52.154568	-31.429808	-51.222851	56.272415	34.468712	-25.402637	-36.512993	47.898685	33.165764	-16.905451	-30.141701	38.172230	26.431019	-13.472577	-24.021032	43.246582	14.914506	-23.153084	-33.343048	42.543930	22.392347	-10.330865	-34.667595	32.264137	20.407352	-13.483160	-21.040884	28.707653	18.947687	-11.550921	-18.212378	17.897223	10.066465	-7.947463	-12.482575	15.136470	9.821433	-5.684728	-10.016788	12.714173	7.650305	-4.211721	-9.240370	10.947527	6.395720	-3.739368	-8.059792	9.969221	6.233003	-3.332090	-7.030806	9.037607	5.644869	-2.770133	-6.491545	9.774871	3.194082	-5.116557	-7.691995	7.610077	4.280361	-3.379340	-5.307716	7.548386	4.600593	-2.243847	-5.547777	7.722602	4.280883	-2.738911	-5.814722	7.851254	2.219324	-3.738460	-6.537639	7.487554	3.762560	-2.020533	-6.150126	3.724781	2.421021	-1.775541	-2.204564	3.201555	2.680049	-1.256285	-1.220262	3.649110	2.674477	-1.156612	-2.196685	3.758146	2.474789	-1.060927	-2.621739	3.019944	1.376533	-1.026755	-2.484149	2.923555	1.332597	-0.993983	-2.404861	2.215418	1.350253	-0.658559	-1.628248	1.959735	0.949403	-0.752075	-1.540642	0.888710	0.648152	-0.440543	-0.419079	0.848160	0.685332	-0.233436	-0.441818	0.675687	0.371343	-0.327507	-0.459779	0.911277	0.391090	-0.264506	-0.779430	0.496820	0.104868	-0.304911	-0.377972	0.500294	0.158566	-0.132538	-0.455614	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.00000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.00000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.00000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.00000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.00000	0.00000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.00000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.0000	0.000000	0.000000	0.000000	0.00000	0.00000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	1
2	197.560150	-56.235020	167.053986	89.222221	165.007935	-46.307407	140.065323	73.925278	58.558525	7.904080	38.231747	43.645844	24.165867	-7.660753	20.127789	10.962393	31.479214	6.973133	20.029806	23.262056	25.901262	3.065581	17.752800	18.609558	19.532864	2.014424	13.778230	13.698002	15.889723	1.921973	10.279412	11.963403	11.679363	-2.999971	9.845611	5.520113	15.129631	2.799529	9.602891	11.351337	10.367282	-4.385475	7.676659	5.414523	10.548188	1.402773	7.611049	7.167179	10.266623	1.767461	7.162259	7.140145	10.980591	1.992374	7.085339	8.148730	8.134279	-1.721148	6.898781	3.951072	10.490783	1.322561	6.552466	8.085329	10.063974	1.033956	6.523703	7.593143	6.653511	-2.662420	5.401315	2.829580	7.802452	0.912603	5.774929	5.166779	6.776989	-4.736222	3.121546	3.708334	7.355442	0.376328	5.583453	4.773464	7.668360	1.090786	5.371960	5.362460	5.996251	1.288733	4.357318	3.912539	5.400266	1.283929	4.101466	3.269920	4.625764	0.696765	3.567205	2.861340	4.286886	-0.753782	3.452291	2.427115	4.278675	0.500963	3.018395	2.990884	3.064641	-1.061758	2.513021	1.396214	2.755710	-0.583086	2.337152	1.338534	2.760834	0.142814	2.016001	1.880838	2.745739	-0.898090	1.681090	1.976475	2.824965	-0.544976	1.690960	2.196380	2.142634	-1.143083	1.296435	1.266293	2.576994	0.463196	1.605608	1.961727	2.119251	0.168004	1.398939	1.583024	1.918472	0.472835	1.186520	1.431480	1.664753	0.230758	1.224832	1.103603	1.378997	-0.583331	1.021106	0.720209	1.180363	-0.468122	0.946170	0.528092	1.263083	0.311305	0.781181	0.942457	1.097036	0.266244	0.731841	0.772664	1.048604	-0.213505	0.648294	0.796054	1.238005	-0.316156	0.560488	1.057617	0.870312	-0.057704	0.548431	0.673304	0.437953	0.123507	0.333553	0.255522	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.00000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.00000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.00000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.00000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.00000	0.00000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.00000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.0000	0.000000	0.000000	0.000000	0.00000	0.00000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0
3	610.647827	226.345108	380.794891	420.302155	63.992065	20.026573	40.651062	45.181984	34.088505	12.317686	21.222988	23.661900	19.412596	7.212401	11.676986	13.728735	12.422605	4.078647	7.691472	8.861547	8.593320	3.107085	5.503000	5.823071	8.785389	1.611511	5.721509	6.469190	6.476299	2.561637	3.974964	4.424943	6.261754	2.436332	4.014475	4.142202	4.367147	-0.180545	3.841729	2.068937	5.324480	1.876532	2.614121	4.242061	3.464139	1.269829	2.293913	2.264013	3.984544	1.646987	2.019110	3.014501	3.175199	1.474765	1.881939	2.089322	2.759936	-0.058105	2.299892	1.524586	3.104477	0.312899	1.925831	2.414755	1.867687	0.454739	1.536920	0.958824	2.131482	0.814061	1.346203	1.438144	2.045079	0.624774	1.419305	1.333259	1.957313	0.909101	1.160098	1.287937	1.731558	0.245909	1.255657	1.166682	1.484838	0.382009	1.185589	0.808202	2.347440	0.466752	1.148688	1.993272	1.491139	0.204910	1.188494	0.876921	1.666883	0.304074	0.909618	1.363317	1.286895	0.051610	0.841874	0.971948	0.931831	-0.012308	0.794448	0.486837	0.872798	0.092387	0.769380	0.401616	1.031228	0.033445	0.730115	0.727493	0.881844	0.453599	0.541407	0.527991	0.791480	0.007617	0.681956	0.401644	0.905531	0.234966	0.577660	0.656571	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.00000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.00000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.00000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.00000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.00000	0.00000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.00000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.0000	0.000000	0.000000	0.000000	0.00000	0.00000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0
4	267.671173	-102.091461	-158.199081	-190.258362	186.682709	-106.923073	-120.735558	-94.025612	43.500698	-25.432400	-32.653172	-13.389325	33.176712	-16.007950	-26.435410	-12.066850	25.473227	-16.826694	-14.994489	-11.870672	22.224855	-12.763152	-14.124808	-11.468911	22.096191	-11.916757	-13.761514	-12.524111	20.754120	-11.957877	-13.286635	-10.545519	8.926038	-4.627290	-6.742624	-3.577618	9.328678	-5.104222	-6.056514	-4.928466	8.100491	-4.943729	-4.181729	-4.867303	6.637937	-4.049828	-3.903955	-3.524235	5.622627	-2.480986	-4.644761	-1.971001	7.360441	-2.632272	-4.206151	-5.436500	6.706080	-2.552033	-3.882224	-4.836009	6.284374	-2.633040	-3.749263	-4.301568	4.440441	-2.336462	-3.143116	-2.092673	3.807630	-2.125382	-2.990122	-1.019787	4.108629	-2.040272	-2.983819	-1.953190	4.327999	-2.688244	-2.379228	-2.417477	4.888242	-2.052459	-2.825049	-3.420734	2.397446	-1.613496	-1.423905	-1.056823	1.777853	-0.819084	-1.452035	-0.617621	2.864586	-0.385838	-1.549062	-2.378527	2.145798	-0.754262	-0.644166	-1.902784	1.232828	-0.363937	-0.790984	-0.872789	0.577701	-0.361531	-0.389291	-0.226905	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.00000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.00000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.00000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.00000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.00000	0.00000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.00000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.0000	0.000000	0.000000	0.000000	0.00000	0.00000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	1
5	164.499893	99.829689	114.322830	-63.439232	123.595650	73.272484	87.150879	-48.080688	46.899860	29.951080	30.025049	-20.025631	44.196377	26.861120	30.690706	-17.025875	44.664692	16.855186	30.713501	-27.704121	31.599348	18.721317	22.216320	-12.428443	31.945690	25.068062	13.442268	-14.540455	29.875772	26.329243	6.924061	-12.304070	32.041317	9.733595	22.010416	-21.152889	26.031197	15.625027	18.247799	-10.024946	22.373161	7.985788	16.161669	-13.250886	17.998421	15.480885	5.098110	-7.635093	12.085132	10.545874	3.623783	-4.658664	11.461868	7.127863	7.795332	-4.449806	10.355254	9.125990	2.399951	-4.264719	9.254187	3.680014	6.454139	-5.517386	7.729150	4.620795	5.246796	-3.295321	8.619956	3.220232	6.199579	-5.049650	6.765135	5.536580	2.547601	-2.936506	6.015748	1.827709	4.381334	-3.694945	4.865225	3.817785	2.047214	-2.214464	4.902287	1.720700	3.702501	-2.713503	4.228647	2.289000	2.801015	-2.190035	2.536126	1.562588	1.672837	-1.091728	2.611094	1.076857	1.843695	-1.502991	2.746674	1.359373	1.173785	-2.078112	2.139244	1.714339	0.432192	-1.204416	1.951081	0.655775	1.540169	-1.002275	2.171378	0.748067	1.082292	-1.727404	1.497544	1.267451	0.323287	-0.729171	1.528786	1.284921	0.187024	-0.806961	1.535819	1.136983	0.444848	-0.931729	1.685910	0.422827	1.132676	-1.174970	1.542091	0.713069	0.951747	-0.981710	1.360734	0.912283	0.647170	-0.774924	1.329683	0.369132	0.933771	-0.871704	0.872655	0.632663	0.316496	-0.510975	0.673151	0.473445	0.421085	-0.227308	0.701269	0.168290	0.481972	-0.480791	0.320729	0.179812	0.263012	-0.036872	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.00000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.00000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.00000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.00000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.00000	0.00000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.00000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.0000	0.000000	0.000000	0.000000	0.00000	0.00000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	1
6	275.454407	-122.148170	245.166122	29.129313	118.976463	-54.020538	103.525024	22.798040	56.767715	-25.233208	50.541958	5.600825	39.765671	-17.675812	35.404541	3.923367	28.905994	-12.309326	25.961252	3.170231	17.560022	-7.289474	15.827994	2.166228	15.634098	-6.248994	14.285532	1.139592	13.518690	-4.890047	12.514712	1.491443	8.883497	-3.600181	8.090115	0.710819	6.004735	0.449896	5.872437	1.170002	5.401881	-2.026861	4.984735	0.473888	5.193821	-2.154214	4.684274	0.626673	4.936502	-2.095330	4.465212	0.201305	4.855865	-1.840441	4.480408	0.343733	4.459344	-2.125681	3.889884	0.485834	4.096872	-1.933278	3.561832	0.600119	3.042507	-1.206348	2.791826	-0.085335	2.960997	-0.065342	2.920553	0.483327	2.535212	-0.774514	2.407406	-0.178395	2.354776	-0.906100	2.123487	0.463419	2.187518	-0.917977	1.983081	-0.099711	2.039711	-0.906492	1.764101	0.476066	1.375132	-0.661072	1.196265	0.151399	1.180829	0.083918	1.159590	-0.206555	1.024851	-0.611182	0.819252	0.074846	1.115528	-0.765417	0.610163	0.535015	0.744857	-0.101565	0.703446	-0.222846	0.665840	-0.228176	0.603181	-0.165684	0.556547	-0.100163	0.538030	0.101175	0.579779	-0.117168	0.395411	0.407512	0.316470	-0.109241	0.294108	0.041476	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.00000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.00000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.00000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.00000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.00000	0.00000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.00000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.0000	0.000000	0.000000	0.000000	0.00000	0.00000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0
7	195.776382	71.457642	-43.164070	-177.084900	89.740707	31.712599	-42.891129	-72.166870	97.034958	28.485353	-19.710289	-90.641449	105.457581	29.350021	-17.268545	-99.808189	91.525826	26.652744	-18.645784	-85.550819	45.013363	17.892401	-20.705256	-35.740135	67.904793	21.080061	-14.514310	-62.896954	65.139336	19.158308	-13.387456	-60.801876	28.043575	10.094787	-11.583507	-23.459747	33.811947	12.049286	-9.135211	-30.242525	39.968479	11.967908	-7.647948	-37.359837	19.605148	6.826134	-10.283878	-15.231794	19.917049	7.889309	-9.450686	-15.656697	26.502373	8.925963	-8.377806	-23.505644	28.139923	10.174318	-6.100360	-25.517134	21.598833	8.947879	-7.567963	-18.143070	32.135963	9.605417	-6.528434	-29.963903	21.106709	7.316864	-8.853605	-17.707916	17.236910	6.328868	-7.247945	-14.301180	22.669622	6.760088	-6.758747	-20.555592	17.584393	6.982410	-5.573163	-15.145845	13.972600	4.864989	-7.329326	-10.855709	14.518909	5.529046	-6.366781	-11.819158	19.141750	6.333881	-5.494456	-17.207541	14.319798	6.128025	-4.859442	-11.995406	18.058144	5.734624	-5.234973	-16.303547	15.012374	5.689663	-3.815546	-13.358171	16.057026	5.201401	-3.606006	-14.757041	14.916764	5.126918	-3.422858	-13.583395	10.874345	4.409315	-3.699378	-9.226262	9.346753	3.026163	-4.479315	-7.624949	11.745160	4.989849	-0.893933	-10.594861	11.756345	4.166164	-1.612040	-10.874559	7.894176	3.271113	-3.024977	-6.516698	9.953744	2.944449	-3.067981	-8.999707	11.413699	3.805331	-1.026229	-10.711622	8.001363	2.741758	-2.334764	-7.145170	8.111948	3.066202	-1.367840	-7.384519	8.008175	2.509926	-2.168585	-7.288922	9.635747	3.264674	-0.398368	-9.057087	6.581693	2.098787	-2.274143	-5.808791	6.415633	2.281583	-1.853963	-5.702415	5.414214	2.242979	-1.897073	-4.547952	6.342005	2.173160	-1.850571	-5.663373	6.010907	1.867151	-1.812217	-5.418544	5.255612	2.101340	-1.434451	-4.598715	6.058692	2.138798	-0.903120	-5.596219	4.413128	1.552851	-1.639159	-3.79177	5.400943	1.975864	-1.027184	-4.920471	4.535516	2.191744	-0.04438	-3.970541	3.779752	1.223756	-1.811399	-3.083469	6.278596	2.127186	-0.423808	-5.892048	5.818707	1.999318	-0.456253	-5.445357	3.557932	1.409252	-1.339549	-2.979681	5.123084	1.30832	-1.308706	-4.777193	4.661511	1.370572	-1.163642	-4.300832	2.679124	1.142659	-1.387828	-1.986446	3.639842	0.980154	-1.325995	-3.244917	3.713544	1.413734	-0.797830	-3.339945	2.017078	1.468336	-0.412986	-1.319862	3.527530	1.128044	-0.920737	-3.212978	3.425990	1.389430	-0.381687	-3.108249	1.761514	0.426779	-1.088822	-1.317293	2.786923	0.932916	-0.644128	-2.545920	2.483886	1.029132	-0.321483	-2.237684	2.400906	0.922831	0.072338	-2.215288	1.222659	0.834952	-0.312236	-0.836814	2.629696	0.828367	-0.234595	-2.484768	2.126755	0.712590	-0.255136	-1.987513	1.422644	0.582351	-0.281356	-1.267131	2.066752	0.381331	-0.502643	-1.968096	1.200118	0.479841	-0.327557	-1.050116	1.053963	0.365367	-0.442104	-0.884244	1.075273	0.508198	-0.000647	-0.947601	1.185643	0.295148	-0.397664	-1.077266	1.479972	0.452914	-0.110404	-1.404634	1.581326	0.422413	-0.124003	-1.51881	1.291269	0.402151	0.002812	-1.227047	0.656545	0.290875	-0.275079	-0.52036	0.59395	0.143902	-0.367131	-0.444167	1.080945	0.380208	-0.100681	-1.00685	0.658771	0.354148	-0.156366	-0.533018	1.399878	0.162971	-0.321637	-1.352644	1.104601	0.268391	-0.233508	-1.045745	1.105273	0.325975	-0.1373	-1.047147	1.187471	0.241487	-0.15016	-1.15292	0.885596	0.166324	-0.165458	-0.853955	0.724353	0.151978	-0.074951	-0.704253	1
8	59.403610	41.055508	-33.719429	-26.575066	48.023834	40.540417	-25.325750	-4.622713	47.623581	40.172688	-25.056154	-5.133194	35.076084	34.462933	-4.700911	-4.532028	35.169415	24.306574	-19.963308	-15.733547	22.860498	18.416842	-13.533061	-0.527802	22.762016	21.673370	-6.532049	-2.388879	21.087252	17.847937	-11.186087	-0.997427	18.204227	17.811428	-2.865510	-2.436343	17.363245	16.520151	-4.841825	-2.263539	15.720620	12.664824	-9.306363	-0.362957	14.507853	10.184656	-8.142515	-6.360033	13.173553	11.744920	-5.335850	-2.669840	12.037468	11.905970	-1.162715	-1.340377	11.400192	9.603645	-6.033511	-1.153745	10.687634	9.820236	-4.214749	-0.156130	10.765874	8.830809	-5.947473	-1.596380	9.998008	9.060877	-3.770721	-1.908486	10.647401	7.320719	-6.443269	-4.272997	9.769657	8.575953	-4.528944	-1.178091	9.726794	9.184681	-2.947534	-1.250682	8.962761	7.351788	-4.951365	-1.329011	9.045572	5.860114	-5.534487	-4.104985	7.799853	6.898966	-3.384279	-1.337398	7.589919	7.217942	-2.322251	-0.339617	6.618104	6.475303	-1.041749	-0.885727	6.445922	5.841820	-2.608860	-0.785435	6.303173	5.880459	-2.012818	-1.048216	6.115631	5.616018	-2.389849	-0.387172	6.016768	5.741394	-1.726236	-0.507951	6.582354	3.213797	-4.717912	-3.277223	5.111702	5.041956	-0.749910	-0.381860	4.543345	4.468598	-0.692640	-0.440309	4.549988	4.033710	-1.856231	-0.992964	4.491254	3.125853	-2.687415	-1.782752	4.198073	2.618282	-2.850502	-1.625747	3.552185	2.808232	-2.172750	0.104925	3.797282	2.665923	-2.336805	-1.360715	3.882375	2.666025	-2.036456	-1.953969	3.085772	2.756250	-1.275881	-0.545163	3.055829	1.976014	-2.128738	-0.949700	3.033531	2.217443	-1.842004	-0.944604	3.127451	2.226434	-1.816478	-1.234647	2.779321	2.721802	-0.370564	-0.423200	3.759017	2.410880	-1.294332	-2.577319	2.813013	1.806992	-2.005503	-0.791062	2.800653	2.178456	-1.355162	-1.123176	2.481514	2.173129	-1.088288	-0.50105	2.400734	2.313423	-0.617865	-0.172744	2.395828	2.278410	-0.73304	-0.107203	2.409425	2.258392	-0.777159	-0.317829	2.358977	2.272147	-0.592347	0.226370	2.375695	2.239820	-0.533427	-0.585310	2.162638	2.148124	-0.213261	-0.130732	2.099411	1.46773	-1.411818	-0.509966	1.727220	1.635701	-0.544216	-0.107705	1.541837	1.498864	-0.149107	-0.329293	1.610131	1.107682	-0.602702	-1.001155	1.256446	1.116295	-0.575927	-0.029164	1.286798	0.692060	-0.986721	-0.450870	1.073181	0.527123	-0.928199	-0.110933	1.070529	0.900912	-0.571988	-0.084969	0.997909	0.895759	-0.437355	-0.046468	0.903666	0.737234	-0.346682	-0.391039	0.739323	0.653897	-0.341555	0.048551	0.710355	0.478017	-0.473676	-0.227451	0.642361	0.465885	-0.436092	-0.073507	0.706375	0.309000	-0.541972	-0.331286	0.573909	0.555006	0.001055	-0.146080	0.537981	0.472248	-0.249393	-0.064873	0.484754	0.408577	-0.260669	-0.010134	0.350200	0.182170	-0.299083	-0.001904	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.00000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.00000	0.00000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.00000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.0000	0.000000	0.000000	0.000000	0.00000	0.00000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	1
9	151.784470	0.949546	131.392471	75.984489	106.406952	0.097951	92.315750	52.917225	73.209137	1.746257	63.050304	37.164330	73.091263	1.297367	61.928951	38.800182	53.414776	0.531225	45.820644	27.446762	35.558651	0.306147	30.661337	18.005730	27.954166	-0.050373	23.910858	14.481151	14.535252	-0.025319	12.635176	7.185069	14.017663	0.100577	11.907260	7.396075	16.027166	0.939310	11.593787	11.025963	12.076696	0.419338	10.127374	6.565596	14.812226	0.261285	9.948937	10.970526	9.939997	0.096975	8.514777	5.127642	9.104223	-0.080639	7.039525	5.772820	7.128007	0.364237	6.171385	3.548216	8.472843	0.282425	5.559115	6.387922	6.245513	0.174957	5.436501	3.069250	5.853168	0.487733	4.812369	3.295876	6.324660	-0.116454	4.811591	4.103212	4.871553	0.481108	4.298071	2.242130	5.296984	0.249718	4.235509	3.171141	5.364757	-0.288473	3.970162	3.596556	5.233563	-0.014002	3.862339	3.531618	4.273739	-0.099422	3.496274	2.455816	3.384312	1.009371	2.485349	2.063439	4.170475	-0.155990	2.602640	3.254965	3.561823	0.072464	2.507604	2.528488	3.723295	0.337383	2.445301	2.787400	2.744171	0.090100	2.276320	1.529942	3.109154	0.366565	2.134917	2.230381	1.902180	-0.438700	1.833718	-0.251609	2.648119	-0.254911	1.761903	1.960421	1.751961	-0.780244	1.544798	0.272369	2.358303	0.538362	1.561525	1.683270	1.904660	1.039524	1.256800	0.983652	1.424168	0.096391	1.329254	0.502043	1.408695	0.064821	0.955400	1.033165	0.923758	0.337149	0.717163	0.474696	0.904179	0.380832	0.520913	0.633369	0.574015	0.301523	0.423995	0.242499	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.00000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.00000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.00000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.00000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.00000	0.00000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.00000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.0000	0.000000	0.000000	0.000000	0.00000	0.00000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0

This looks like the format we want: we have a matrix of numerical features encoded in the 4-vectors, along with a target denoted by the is_signal_new column.

Create a Learner#

We can now create the Learner for this data. fastai provides various application-specific learner classes, each of which come with a set of good defaults for training. In our case, we’ll use the tabular_learner class:

learn = tabular_learner(
    dls, layers=[200, 200, 50, 50], metrics=[accuracy, RocAucBinary()]
)

By default, tabular_learner creates a neural network with two hidden layers and 200 and 100 activations each. This works great for small datasets, but since our dataset is quite large, we’ve increased the depth of the network by adding two more layers. This also matches the architecture chosen in Section 3.2.2 of The Machine Learning Landscape of Top Taggers review that we’ll compare to later.

We’ve also provided two common classification metrics to track during training: accuracy and the Area Under the ROC Curve (ROC AUC). We’ll look at ROC AUC in more detail later, so for now let’s take a look at our network with the summary() method:

learn.summary()

TabularModel (Input shape: 1024 x 0)
============================================================================
Layer (type)         Output Shape         Param #    Trainable 
============================================================================
                     1024 x 800          
BatchNorm1d                               1600       True      
____________________________________________________________________________
                     1024 x 200          
Linear                                    160000     True      
ReLU                                                           
BatchNorm1d                               400        True      
Linear                                    40000      True      
ReLU                                                           
BatchNorm1d                               400        True      
____________________________________________________________________________
                     1024 x 50           
Linear                                    10000      True      
ReLU                                                           
BatchNorm1d                               100        True      
Linear                                    2500       True      
ReLU                                                           
BatchNorm1d                               100        True      
____________________________________________________________________________
                     1024 x 2            
Linear                                    102        True      
____________________________________________________________________________

Total params: 215,202
Total trainable params: 215,202
Total non-trainable params: 0

Optimizer used: <function Adam at 0x7faa1ad5e790>
Loss function: FlattenedLoss of CrossEntropyLoss()

Callbacks:
  - TrainEvalCallback
  - Recorder
  - ProgressCallback

Here we can see that this particular network has around 215,000 parameters - although this sounds like a lot, it’s actually a very small model by modern standards (e.g. in natural language processing, some models have hundreds of billions of parameters!).

Find a good learning rate#

The learning rate is one of the most important hyperparameters involved in training neural networks, so it’s important to make sure you’ve picked a good one. We’ll see in the next lesson exactly how this parameter impacts training, but for now it is enough to know that:

If our learning rate is too low, it will take a long time to train the model and there is a good chance of overfitting.
If our learning rate is too high, the training process can diverge.

To handle these two extremes, fastai provides a learning rate finder that tracks the loss as we increase the learning rate. You can see this in action by using the lr_find() method of any Learner:

learn.lr_find()

SuggestedLRs(valley=0.0006918309954926372)

From this curve we can see that the loss hits a minimum around a learning rate of \(3 \times 10^{-1}\), so we should select a learning rate lower than this point. The lr_find() method provides a handy heuristic to pick the learning rate 1-2 orders of magnitude less than the minimum, as indicated by the orange dot.

Train your model#

In the above learning rate plot, it appears a learning rate of around \(10^{-3}\) would be good, so let’s choose that and train our models for 3 epochs:

learn.fit_one_cycle(n_epoch=3, lr_max=1e-3)

epoch	train_loss	valid_loss	accuracy	roc_auc_score	time
0	0.512565	0.511615	0.732775	0.809289	00:20
1	0.419058	0.404574	0.810632	0.891700	00:20
2	0.381482	0.372323	0.830776	0.907655	00:20

Once the model is trained, we can view the results in various ways. A simple approach is to use the show_results() method to compare the model errors:

learn.show_results()

	E_0	PX_0	PY_0	PZ_0	E_1	PX_1	PY_1	PZ_1	E_2	PX_2	PY_2	PZ_2	E_3	PX_3	PY_3	PZ_3	E_4	PX_4	PY_4	PZ_4	E_5	PX_5	PY_5	PZ_5	E_6	PX_6	PY_6	PZ_6	E_7	PX_7	PY_7	PZ_7	E_8	PX_8	PY_8	PZ_8	E_9	PX_9	PY_9	PZ_9	E_10	PX_10	PY_10	PZ_10	E_11	PX_11	PY_11	PZ_11	E_12	PX_12	PY_12	PZ_12	E_13	PX_13	PY_13	PZ_13	E_14	PX_14	PY_14	PZ_14	E_15	PX_15	PY_15	PZ_15	E_16	PX_16	PY_16	PZ_16	E_17	PX_17	PY_17	PZ_17	E_18	PX_18	PY_18	PZ_18	E_19	PX_19	PY_19	PZ_19	E_20	PX_20	PY_20	PZ_20	E_21	PX_21	PY_21	PZ_21	E_22	PX_22	PY_22	PZ_22	E_23	PX_23	PY_23	PZ_23	E_24	PX_24	PY_24	PZ_24	E_25	PX_25	PY_25	PZ_25	E_26	PX_26	PY_26	PZ_26	E_27	PX_27	PY_27	PZ_27	E_28	PX_28	PY_28	PZ_28	E_29	PX_29	PY_29	PZ_29	E_30	PX_30	PY_30	PZ_30	E_31	PX_31	PY_31	PZ_31	E_32	PX_32	PY_32	PZ_32	E_33	PX_33	PY_33	PZ_33	E_34	PX_34	PY_34	PZ_34	E_35	PX_35	PY_35	PZ_35	E_36	PX_36	PY_36	PZ_36	E_37	PX_37	PY_37	PZ_37	E_38	PX_38	PY_38	PZ_38	E_39	PX_39	PY_39	PZ_39	E_40	PX_40	PY_40	PZ_40	E_41	PX_41	PY_41	PZ_41	E_42	PX_42	PY_42	PZ_42	E_43	PX_43	PY_43	PZ_43	E_44	PX_44	PY_44	PZ_44	E_45	PX_45	PY_45	PZ_45	E_46	PX_46	PY_46	PZ_46	E_47	PX_47	PY_47	PZ_47	E_48	PX_48	PY_48	PZ_48	E_49	PX_49	PY_49	PZ_49	E_50	PX_50	PY_50	PZ_50	E_51	PX_51	PY_51	PZ_51	E_52	PX_52	PY_52	PZ_52	E_53	PX_53	PY_53	PZ_53	E_54	PX_54	PY_54	PZ_54	E_55	PX_55	PY_55	PZ_55	E_56	PX_56	PY_56	PZ_56	E_57	PX_57	PY_57	PZ_57	E_58	PX_58	PY_58	PZ_58	E_59	PX_59	PY_59	PZ_59	E_60	PX_60	PY_60	PZ_60	E_61	PX_61	PY_61	PZ_61	E_62	PX_62	PY_62	PZ_62	E_63	PX_63	PY_63	PZ_63	E_64	PX_64	PY_64	PZ_64	E_65	PX_65	PY_65	PZ_65	E_66	PX_66	PY_66	PZ_66	E_67	PX_67	PY_67	PZ_67	E_68	PX_68	PY_68	PZ_68	E_69	PX_69	PY_69	PZ_69	E_70	PX_70	PY_70	PZ_70	E_71	PX_71	PY_71	PZ_71	E_72	PX_72	PY_72	PZ_72	E_73	PX_73	PY_73	PZ_73	E_74	PX_74	PY_74	PZ_74	E_75	PX_75	PY_75	PZ_75	E_76	PX_76	PY_76	PZ_76	E_77	PX_77	PY_77	PZ_77	E_78	PX_78	PY_78	PZ_78	E_79	PX_79	PY_79	PZ_79	E_80	PX_80	PY_80	PZ_80	E_81	PX_81	PY_81	PZ_81	E_82	PX_82	PY_82	PZ_82	E_83	PX_83	PY_83	PZ_83	E_84	PX_84	PY_84	PZ_84	E_85	PX_85	PY_85	PZ_85	E_86	PX_86	PY_86	PZ_86	E_87	PX_87	PY_87	PZ_87	E_88	PX_88	PY_88	PZ_88	E_89	PX_89	PY_89	PZ_89	E_90	PX_90	PY_90	PZ_90	E_91	PX_91	PY_91	PZ_91	E_92	PX_92	PY_92	PZ_92	E_93	PX_93	PY_93	PZ_93	E_94	PX_94	PY_94	PZ_94	E_95	PX_95	PY_95	PZ_95	E_96	PX_96	PY_96	PZ_96	E_97	PX_97	PY_97	PZ_97	E_98	PX_98	PY_98	PZ_98	E_99	PX_99	PY_99	PZ_99	E_100	PX_100	PY_100	PZ_100	E_101	PX_101	PY_101	PZ_101	E_102	PX_102	PY_102	PZ_102	E_103	PX_103	PY_103	PZ_103	E_104	PX_104	PY_104	PZ_104	E_105	PX_105	PY_105	PZ_105	is_signal_new	is_signal_new_pred
0	55.373371	15.447829	-53.108967	2.648058	39.483627	9.879107	-38.208023	1.227653	39.344952	28.682394	-26.352501	5.557986	34.630009	25.825630	-22.191050	6.311225	31.741671	22.453880	-22.047646	4.154309	28.607065	20.475050	-19.632759	3.700181	31.330889	12.412865	-21.484344	19.130299	22.874287	16.306553	-15.693566	3.322846	21.684135	11.369366	-17.951180	4.323710	20.443676	5.001358	-19.806828	0.787326	18.971832	14.148410	-12.157228	3.457565	22.302622	7.657867	-16.329651	13.118936	22.506207	8.257535	-15.369576	14.216842	17.488396	12.562215	-11.999881	2.009378	17.286680	12.460367	-11.671321	2.710874	15.806683	9.301882	-12.139855	3.993762	16.524759	6.189622	-11.225109	10.428478	10.510235	7.613899	-7.113340	1.376215	10.505521	6.357656	-7.655622	3.367139	8.879970	2.345350	-8.521793	0.855708	7.796289	2.116827	-7.502737	0.100512	7.792163	5.807485	-5.111705	0.928112	8.909110	3.027786	-6.752189	4.961118	7.138719	1.478972	-6.924917	-0.905249	7.019392	1.748098	-6.797865	0.071055	6.291604	4.547145	-3.957447	-1.801769	5.986259	4.363632	-4.026358	0.763186	7.380098	2.547143	-5.351652	4.397468	5.743817	0.075045	-5.278041	2.264527	6.931075	2.999568	-4.256332	4.574497	7.232418	1.939481	-4.349987	5.442784	5.510406	2.592241	-3.626578	3.239258	4.339673	2.618227	-3.349735	-0.870016	4.239345	3.116243	-2.873728	-0.052541	5.080171	1.750966	-3.859296	2.801444	5.380709	2.247479	-3.541767	3.369979	5.361376	1.817439	-3.761746	3.360140	4.103991	2.657733	-2.794918	1.402720	3.818110	2.906654	-2.466961	-0.208396	4.390255	1.492041	-3.327362	2.444753	3.485384	2.404282	-2.507123	-0.285769	3.187455	1.371662	-2.873592	-0.144521	3.490748	1.988637	-2.472357	1.455369	4.097678	2.209598	-2.229547	2.633963	3.930950	2.172712	-2.055131	2.551103	3.562002	1.982498	-2.168933	2.013279	3.120360	1.434136	-2.554029	-1.075563	2.888219	1.857622	-2.140393	-0.556570	2.678831	2.034264	-1.711286	0.330761	2.630311	1.963093	-1.706802	0.389398	3.244133	1.065371	-2.351692	1.964417	2.638533	1.724679	-1.895686	-0.627464	2.543895	2.038069	-1.515647	0.143134	3.280566	1.412071	-1.966430	2.213893	2.566380	1.855283	-1.529425	-0.897268	2.289888	1.568115	-1.578431	0.541439	2.143133	0.785485	-1.959918	0.367092	2.265625	0.370917	-2.031101	0.932795	2.155324	1.487690	-1.398882	-0.689440	2.088751	0.885557	-1.750641	0.716886	2.567099	0.351094	-1.901725	1.688246	2.626564	1.599039	-0.941216	1.859038	2.267673	0.917841	-1.602557	1.315948	1.809489	0.863159	-1.339359	0.857510	1.555773	0.946619	-1.225913	0.146564	1.840218	0.865688	-1.211108	1.081761	1.502715	0.671494	-1.305816	0.319518	2.117384	0.982598	-1.066445	1.542891	2.147986	0.576015	-1.291921	1.616475	1.620426	0.445077	-1.329756	0.812058	1.287042	1.104052	-0.659054	0.056528	1.346352	1.007555	-0.781745	0.431708	1.204614	0.038337	-1.201698	0.074481	1.388158	0.423047	-1.119640	0.703149	1.290550	0.594335	-0.993044	0.571094	1.259902	0.842324	-0.746376	0.566362	1.240312	0.823487	-0.695624	0.613474	1.114801	0.616407	-0.836268	0.404327	1.050791	0.398882	-0.922332	0.307179	1.127198	0.645933	-0.747799	-0.542350	1.396213	0.701842	-0.656147	1.013065	1.034949	0.493690	-0.766056	0.490459	0.911330	0.421269	-0.712346	-0.381599	0.814269	0.523715	-0.603436	-0.156913	0.954917	0.557657	-0.571621	0.523579	0.760974	0.312979	-0.693473	0.014826	0.846925	0.272219	-0.697219	0.396314	0.644899	0.546843	-0.334867	-0.068710	0.649957	0.325505	-0.550379	-0.116504	0.676266	0.024148	-0.598012	0.314857	0.570777	0.153775	-0.548854	-0.029968	0.518796	0.181761	-0.406142	0.266760	0.000000	0.000000	0.000000	0.00000	0.000000	0.000000	0.000000	0.000000	0.00000	0.00000	0.000000	0.000000	0.000000	0.000000	0.000000	0.00000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.00000	0.000000	0.000000	0.000000	0.00000	0.000000	0.000000	0.000000	0.00000	0.000000	0.00000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.00000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	1.0	1.0
1	85.271927	18.985689	80.161728	-22.021387	35.494274	14.089288	27.902519	16.816210	35.372967	5.981579	24.812439	24.491024	25.733261	9.897063	22.975033	6.032969	28.182434	10.784863	21.934576	14.028923	18.519686	4.554176	17.492123	-4.032852	17.722200	8.807404	15.307701	1.476581	16.917294	6.332823	15.572810	1.891498	24.142254	4.088970	16.139261	17.482935	18.184538	7.623066	14.188622	8.440929	14.736913	7.184424	12.294654	3.795013	14.942549	4.855879	12.715952	6.164801	13.729703	6.076223	11.742292	3.701733	12.704572	6.494713	10.371592	3.413933	15.580447	2.845028	9.984985	11.617065	9.553916	2.127168	8.981367	-2.467289	9.804935	3.186309	8.343896	4.045191	8.694020	3.851479	7.770153	0.613865	8.243057	5.284256	5.934836	2.191428	8.779967	3.985839	6.611471	4.182028	7.933362	4.407343	5.780066	3.178741	7.105115	3.368708	5.879133	2.137819	5.970509	2.737379	5.242725	0.817049	6.051137	2.366136	5.302313	1.703857	5.901711	3.073598	4.855549	1.344184	6.272639	4.037410	3.992417	2.665695	6.232912	5.105202	2.319947	2.721020	7.341536	1.614292	5.363025	4.746595	6.145093	4.572462	2.929341	2.876407	5.856678	2.805473	4.596078	2.303491	5.343754	2.587870	4.635347	0.610072	5.468257	2.527618	4.632632	1.432377	6.032536	4.560538	2.457248	3.091102	5.207212	1.688362	4.861693	-0.792737	5.531978	2.484682	4.359250	2.329393	5.121120	1.059114	4.798028	-1.443289	4.850181	2.497190	4.136650	0.420032	6.786995	1.429670	4.304793	5.048573	5.556354	1.293891	4.277099	3.302323	4.906525	3.403742	2.785759	2.174414	4.229742	3.379346	2.434534	-0.737415	4.091139	1.765958	3.664821	0.433472	4.395818	2.131417	3.452122	1.692080	3.884586	1.596526	3.251563	1.403015	4.026021	3.060858	1.747410	1.945906	3.470460	0.785172	3.339807	-0.522771	3.551084	2.793807	1.822081	1.218547	3.460228	2.306340	2.393177	0.962641	3.458021	1.377673	2.960962	1.136938	3.495999	2.029963	2.312546	1.659335	3.475843	2.760226	1.258267	1.696881	3.245294	1.344248	2.667978	1.267608	2.962028	1.531076	2.519693	-0.283835	3.051412	1.687248	2.174260	1.317916	2.512844	1.686045	1.779321	0.552860	2.689137	1.730874	1.711585	1.142807	2.823017	0.768076	2.290710	1.460183	2.411201	1.448410	1.926455	0.069062	3.871706	0.444843	2.284927	3.093757	2.409937	1.518276	1.763376	0.627008	2.762771	0.450254	2.190216	1.622692	2.339598	1.109260	1.935902	0.703949	2.320295	1.870794	1.161824	0.730795	2.191975	1.253329	1.720387	0.523628	2.147218	1.241978	1.723932	0.309991	2.067152	0.668318	1.797967	0.770573	2.486211	0.976978	1.561115	1.670233	1.853210	0.693731	1.705928	0.207205	1.954812	1.041473	1.383678	0.906673	1.887233	0.418697	1.644984	-0.824846	1.756695	0.891352	1.399602	0.576703	2.784866	0.735156	1.483188	2.239459	2.572018	0.816908	1.401982	1.995592	1.613443	0.533463	1.495458	0.286742	1.677924	0.851383	1.336842	0.550843	1.746369	0.290088	1.536734	0.777240	2.353512	0.480571	1.473505	1.771117	2.081337	0.122754	1.513445	1.423510	1.590024	0.158424	1.482136	0.553488	1.504820	0.578757	1.343526	0.352794	1.503309	0.523185	1.351987	0.397927	1.831516	0.319156	1.346600	1.199692	1.556591	0.420901	1.305272	0.736263	1.166046	-0.175975	1.152045	0.038587	1.352714	1.022639	0.551004	0.693138	1.228763	0.139722	1.110641	-0.506768	1.227743	0.433013	0.974023	0.609207	1.074988	0.600248	0.867498	0.206761	1.761592	0.317861	0.975899	1.431710	1.121050	0.744424	0.605544	0.579571	0.931303	0.605460	0.700810	0.098024	0.870360	0.526842	0.679842	0.133338	1.094996	0.182765	0.839526	0.67883	0.869425	0.694774	0.499537	0.153792	0.80607	0.43132	0.671326	-0.114161	0.757981	0.123427	0.695199	-0.27568	0.774677	0.211287	0.639215	0.383257	0.735012	0.439176	0.507381	0.299884	0.919752	0.23057	0.625046	0.634112	1.091753	0.43259	0.506562	0.864977	0.822304	0.41847	0.422452	0.56798	0.548454	0.200325	0.501501	-0.095753	0.662222	0.247508	0.440059	0.428516	0.722328	0.219603	0.416881	0.547487	0.45947	0.192403	0.385645	0.159286	0.381374	0.307673	0.205482	0.092526	1.0	0.0
2	516.210022	65.679703	-247.258301	-448.355103	239.182953	31.903074	-113.941299	-207.865479	179.469437	23.711529	-86.030853	-155.710419	31.219574	4.289379	-15.344496	-26.847895	37.876450	-1.978928	-11.897355	-35.904907	20.399145	-2.294662	-6.365724	-19.244148	12.406221	1.862238	-6.106205	-10.637700	9.219143	3.883240	-4.890146	-6.782294	19.560913	-0.260901	-5.983164	-18.621572	11.251606	2.552148	-5.037094	-9.732054	18.354359	-0.930482	-5.475431	-17.493895	16.252151	-2.212914	-5.068512	-15.282200	6.916037	2.132614	-4.821438	-4.476299	9.780379	0.747449	-5.211164	-8.242627	9.660797	1.918161	-4.794651	-8.164741	9.848786	1.837722	-4.671026	-8.473658	16.540770	-0.054699	-4.790403	-15.831807	14.726268	-0.474498	-4.746349	-13.932336	7.632535	1.056448	-4.623791	-5.979972	16.599749	-1.347675	-4.512690	-15.917634	12.049094	-0.836061	-4.101630	-11.298598	7.719193	1.126171	-3.958953	-6.530267	10.318196	-0.681848	-3.737458	-9.593314	6.383173	0.875908	-3.390751	-5.336711	5.925990	0.664168	-3.116205	-4.996550	9.041269	-0.083944	-3.101111	-8.492386	8.563301	-0.485526	-3.019691	-7.998491	4.828800	0.646107	-2.672212	-3.969778	3.304960	1.389704	-2.079199	-2.160651	4.622041	0.501523	-2.295092	-3.980489	5.623734	0.022133	-2.192067	-5.178874	4.362431	1.123515	-1.728674	-3.844503	7.586563	-0.431333	-1.987274	-7.308942	3.077971	0.304442	-1.948014	-2.363570	3.152456	0.365215	-1.881816	-2.502672	2.837118	0.741525	-1.701359	-2.145869	5.856131	0.142115	-1.631506	-5.622478	2.703051	0.351616	-1.503992	-2.218301	4.142230	0.702753	-1.246833	-3.887109	2.733151	0.492921	-1.205528	-2.402883	1.912597	0.737363	-1.061550	-1.409764	3.952762	-0.013051	-1.147835	-3.782410	3.149440	-0.272640	-1.041096	-2.959858	1.572127	0.540256	-0.918551	-1.155842	1.289259	0.397553	-0.898793	-0.834453	1.004981	0.386808	-0.679457	-0.631431	2.614485	0.011527	-0.770739	-2.498271	2.090822	-0.224414	-0.682523	-1.963501	1.025700	0.101452	-0.649154	-0.787633	0.876280	0.130027	-0.486805	-0.716924	0.851218	0.168147	-0.416703	-0.722950	1.414109	-0.082354	-0.420374	-1.347668	0.551654	0.105977	-0.407282	-0.356668	1.229190	0.026430	-0.400792	-1.161712	0.933148	0.246668	-0.309570	-0.845036	0.507530	0.109217	-0.332749	-0.367337	0.749032	0.160857	-0.235663	-0.692558	0.603806	0.142805	-0.159186	-0.564666	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.00000	0.000000	0.000000	0.000000	0.000000	0.00000	0.00000	0.000000	0.000000	0.000000	0.000000	0.000000	0.00000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.00000	0.000000	0.000000	0.000000	0.00000	0.000000	0.000000	0.000000	0.00000	0.000000	0.00000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.00000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.0	1.0
3	179.563461	-78.816872	-144.127274	72.513924	57.747158	-21.012482	-47.169498	25.850504	57.541145	-21.817780	-46.437847	26.047922	31.984177	-10.522673	-23.974773	18.370390	28.503929	-19.214449	-16.230444	13.410877	23.305548	-8.522502	-19.122160	10.240045	20.502831	-13.593023	-12.078649	9.471115	19.289295	-10.553767	-13.188070	9.315028	17.299089	-7.465109	-14.635800	5.415163	18.352409	-12.133499	-11.023665	8.250330	15.409149	-5.652246	-13.209612	5.567778	14.739727	-9.908389	-8.232552	7.162993	13.606773	-6.493807	-10.264329	6.133376	15.646438	-4.733969	-11.113035	9.944899	13.769272	-4.598180	-10.381672	7.789125	11.078214	-4.109962	-8.905683	5.150132	10.438019	-6.905755	-6.393700	4.514797	10.234579	-4.340665	-7.756525	5.073615	9.574831	-4.906097	-6.998511	4.316070	8.875442	-5.893004	-5.258232	4.049315	8.183035	-3.070297	-6.941494	3.057940	8.138783	-5.618798	-4.768335	3.454253	8.431035	-4.407260	-5.768741	4.287196	7.438922	-2.728679	-6.377074	2.687901	7.635469	-2.623368	-6.148635	3.689528	9.050841	-4.024401	-5.112267	6.291792	6.475564	-2.130436	-4.853968	3.719296	6.063044	-3.251853	-4.128994	3.022806	5.670615	-3.004871	-4.072957	2.556881	6.102602	-2.382156	-4.397341	3.497209	5.423337	-1.995948	-4.519047	2.237631	5.209931	-2.486430	-3.930134	2.348423	5.429140	-1.997184	-4.083933	2.967878	5.431569	-2.630232	-3.229809	3.485993	4.644033	-1.483754	-3.725301	2.342575	4.351503	-3.004162	-2.549451	1.846860	4.217671	-3.204675	-2.168622	1.678058	4.105029	-2.240312	-2.686709	2.147989	4.409565	-1.903473	-2.643459	2.972067	3.114978	-2.337030	-1.513010	1.397204	3.706148	-2.089397	-1.802283	2.474213	2.785452	-0.510653	-2.401492	1.315603	2.668029	-1.756768	-1.705230	1.060346	4.436053	-1.182912	-2.099723	3.724305	2.730992	-1.515908	-1.776266	1.416057	2.537111	-1.800472	-1.239602	1.287873	2.233770	-0.767471	-1.798794	1.079378	2.209532	-0.555434	-1.733758	1.252041	1.863158	-1.251323	-0.999087	0.952561	1.604812	-0.760796	-1.217320	0.717455	1.521268	-0.278892	-1.311569	0.718513	1.160954	-0.706225	-0.797743	0.461159	1.215907	-0.677310	-0.713421	0.714639	0.975926	-0.487160	-0.761194	0.368361	1.007666	-0.732924	-0.358831	0.591146	0.891673	-0.404192	-0.692181	0.390633	0.998600	-0.336022	-0.683391	0.645963	0.968041	-0.356868	-0.591964	0.677737	0.501999	-0.081097	-0.487034	-0.090685	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.00000	0.000000	0.000000	0.000000	0.000000	0.00000	0.00000	0.000000	0.000000	0.000000	0.000000	0.000000	0.00000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.00000	0.000000	0.000000	0.000000	0.00000	0.000000	0.000000	0.000000	0.00000	0.000000	0.00000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.00000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.0	1.0
4	329.562469	95.270767	-126.800941	288.888245	319.419220	100.912178	-120.495155	278.076050	77.337204	22.194319	-29.571247	67.926407	61.838978	18.147085	-23.431585	54.274334	49.351467	13.173972	-22.642370	41.825073	47.120506	12.578436	-21.618811	39.934349	46.389317	14.740288	-16.279430	40.861629	43.341885	12.493152	-16.581955	38.045746	31.633617	10.231354	-12.024124	27.412142	30.532423	8.062800	-11.625650	27.056688	26.103477	8.533171	-10.292779	22.419527	25.247374	7.628972	-9.912628	21.930992	17.856659	4.219609	-6.807502	15.959734	13.716174	4.209394	-5.481440	11.847711	11.635141	3.570374	-4.609353	10.068903	9.181920	4.273754	-2.714597	7.659873	8.440472	2.724650	-3.251005	7.297178	6.138549	2.320958	-2.505951	5.100505	6.732611	2.486831	-1.331028	6.113271	3.722566	2.305616	-0.828641	2.802675	4.201150	1.409204	-1.568503	3.633677	2.803007	1.142459	-1.012736	2.350745	4.100997	0.869796	-1.231139	3.813912	2.151438	0.791965	-0.791666	1.837048	1.975119	0.610799	-0.920475	1.637298	1.900072	0.872769	-0.254539	1.668460	1.301327	0.735834	-0.496105	0.951777	1.059441	0.427947	-0.446137	0.860372	1.012077	0.260065	-0.557012	0.803993	1.630392	0.172932	-0.532861	1.531121	0.824098	0.486938	-0.220189	0.627333	0.712525	0.349196	-0.376336	0.494091	0.904629	0.301461	-0.269676	0.809166	0.644888	0.158176	-0.218105	0.585911	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.00000	0.000000	0.000000	0.000000	0.000000	0.00000	0.00000	0.000000	0.000000	0.000000	0.000000	0.000000	0.00000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.00000	0.000000	0.000000	0.000000	0.00000	0.000000	0.000000	0.000000	0.00000	0.000000	0.00000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.00000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.0	0.0
5	611.084106	-305.877930	129.185684	513.004456	284.789490	-140.946121	61.773891	239.631439	76.014404	-37.440628	16.435490	64.080139	22.579498	-11.267144	5.153885	18.876511	18.358086	-9.069336	3.679590	15.531487	11.038916	-5.771944	2.414210	9.094719	5.687854	-2.857658	0.943238	4.826570	4.331696	-2.132928	0.182841	3.765737	5.723002	-1.590581	0.922244	5.419619	3.527958	-1.545158	0.410306	3.144937	2.790856	-1.413644	0.731161	2.292573	2.203614	-1.009327	0.244803	1.943514	1.776436	-0.946788	0.292658	1.474336	1.433138	-0.883230	0.212052	1.108524	2.893339	-0.881270	0.169735	2.750629	1.782020	-0.816223	0.197968	1.571682	1.645752	-0.726600	-0.050745	1.475797	1.530543	-0.678641	0.133411	1.365361	0.820616	-0.492704	0.282692	0.592233	1.151004	-0.297730	0.427764	1.026248	1.248217	-0.497593	-0.094049	1.140878	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.00000	0.000000	0.000000	0.000000	0.000000	0.00000	0.00000	0.000000	0.000000	0.000000	0.000000	0.000000	0.00000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.00000	0.000000	0.000000	0.000000	0.00000	0.000000	0.000000	0.000000	0.00000	0.000000	0.00000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.00000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.0	0.0
6	102.676163	-97.163437	-14.328404	29.939236	64.570015	-61.093929	-9.346408	18.693928	65.386238	-60.351425	-8.609890	23.641819	56.462463	-54.785923	-5.263214	12.602011	47.432133	-45.001816	-7.120251	13.188850	40.683399	-38.570587	-5.872537	11.530916	28.003189	-26.391380	-4.377386	8.277205	26.152256	-24.843985	-3.721982	7.270751	23.080246	-21.833197	-3.181887	6.773834	22.892691	-21.534258	-3.309304	7.028485	21.047714	-19.981674	-2.704503	6.035285	13.292037	-12.593484	-1.933046	3.787579	12.791776	-12.218965	-1.924688	3.259145	7.018115	-6.534798	-2.008228	1.586625	6.743606	-6.232697	-1.848569	1.792343	6.210994	-5.740381	-1.045328	2.128794	6.112596	-5.376745	-2.126432	1.983113	4.986853	-3.374813	-3.165563	1.859717	4.302229	-4.220151	-0.616500	0.565178	3.806959	-3.438175	-0.578166	1.528924	3.572317	-3.229210	-0.452246	1.459153	3.641506	-2.671299	1.595279	1.892039	2.801416	-2.653493	-0.783084	0.440098	2.662709	-2.382332	-0.915795	0.758835	2.815930	-2.255222	-1.117244	1.263014	2.460618	-2.231441	-0.676693	0.785749	2.745603	-2.279052	-0.092057	1.528328	2.000568	-1.662871	-0.776164	0.796682	1.834038	-1.740878	-0.446244	0.365928	1.715279	-1.317732	-1.063943	0.271645	1.531346	-1.449036	-0.133841	0.476866	1.474029	-1.246142	-0.586768	0.524973	1.532491	-1.294766	-0.469805	0.671858	1.328893	-1.285204	-0.337899	0.005790	1.271554	-1.238967	-0.213757	-0.190049	1.742561	-0.964922	-0.765100	1.232910	1.388186	-1.225337	0.026604	0.651846	1.618137	-1.154190	-0.382735	1.067580	1.032082	-0.862739	-0.565555	-0.031960	1.025964	-0.928927	-0.315983	-0.299751	1.010365	-0.591851	-0.631354	0.521482	0.823785	-0.701112	-0.424199	-0.084371	0.544246	-0.425272	-0.337233	0.040271	0.427590	-0.415362	-0.080952	0.061274	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.00000	0.000000	0.000000	0.000000	0.000000	0.00000	0.00000	0.000000	0.000000	0.000000	0.000000	0.000000	0.00000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.00000	0.000000	0.000000	0.000000	0.00000	0.000000	0.000000	0.000000	0.00000	0.000000	0.00000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.00000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.0	0.0
7	257.630188	106.853798	-69.073929	224.018692	83.139183	54.075981	-27.262121	56.962166	111.417793	46.211254	-29.872526	96.881767	41.762379	26.673393	-13.681358	29.076571	34.663544	21.791115	-12.034702	24.122076	25.538147	16.182859	-8.959289	17.608044	30.153559	7.585900	-16.008104	24.401472	24.515146	11.261580	-12.848575	17.580767	22.468275	14.698792	-7.485686	15.255603	19.120033	11.736723	-6.992858	13.376284	24.139442	9.408215	-6.032071	21.396549	18.365290	4.613016	-9.457962	15.051607	14.258600	8.633021	-5.624858	9.855942	14.425040	8.865430	-5.083375	10.180630	20.804403	8.410170	-5.317484	18.270649	13.994586	7.593967	-6.303789	9.921811	21.007475	8.456771	-4.788936	18.624262	13.029757	5.821420	-6.856107	9.427588	10.091116	5.895158	-4.701034	6.706565	9.571890	5.884108	-4.098946	6.340110	9.571373	6.144860	-3.171987	6.617430	8.580808	6.470941	-1.388490	5.461618	10.252076	2.282099	-5.927240	8.047665	8.047889	4.701519	-3.749179	5.348634	8.326077	4.827573	-3.402928	5.868406	8.132969	4.453572	-3.825441	5.628222	8.798279	3.072728	-4.789514	6.710336	12.012880	4.217518	-3.521853	10.682620	7.761537	4.680856	-2.098069	5.824873	9.441773	4.063155	-2.331676	8.197631	7.229572	2.543451	-3.838978	5.573134	5.715028	4.136883	-1.597178	3.605103	6.003904	3.465700	-2.197486	4.382560	7.316044	3.797707	-0.928238	6.183874	5.193681	2.735659	-2.342006	3.742393	5.353796	1.858556	-3.058202	3.981997	4.304455	3.083871	-1.632927	2.520243	6.899963	2.732437	-1.535975	6.146874	3.176030	2.508638	-1.098781	1.608285	3.084017	2.016692	-1.410590	1.858588	3.045140	2.359873	-0.078474	1.922945	3.249019	1.177708	-1.724877	2.488761	3.297309	1.671539	-1.190321	2.580956	2.947780	1.652419	-1.175254	2.139555	3.344485	1.645264	-1.121537	2.687162	2.092230	1.878650	-0.561380	0.730036	3.207179	0.990327	-1.383898	2.718470	3.152884	1.315663	-0.959724	2.699748	2.404118	0.960115	-1.258060	1.809764	2.967598	1.304544	-0.536729	2.610885	1.521293	1.291479	-0.466002	0.655176	1.381625	1.102953	-0.681875	0.476894	2.242080	0.981484	-0.568098	1.934135	1.653730	0.920067	-0.644304	1.213743	1.700750	1.043000	-0.157132	1.334170	2.208643	0.970911	-0.399462	1.943159	2.415330	0.449469	-0.907174	2.192905	1.785409	0.838798	-0.551518	1.476460	1.050630	0.569733	-0.767990	0.435223	1.919633	0.552084	-0.779635	1.665041	1.684097	0.741616	-0.561193	1.404012	1.794403	0.864213	-0.256112	1.551587	1.520116	0.504923	-0.647644	1.279204	0.914422	0.630762	-0.426319	0.506516	0.949885	0.526892	-0.370316	0.698234	1.031610	0.440354	-0.415594	0.835217	1.149833	0.153560	-0.499332	1.024306	1.124914	0.351487	-0.290792	1.028265	0.820899	0.451697	-0.020556	0.685145	0.526363	0.289947	-0.326733	0.293657	1.276224	0.369967	-0.214223	1.202489	0.516433	0.283546	-0.275611	0.332181	0.377315	0.064522	-0.181876	0.324229	0.167024	0.057483	-0.122019	0.098510	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.00000	0.000000	0.000000	0.000000	0.000000	0.00000	0.00000	0.000000	0.000000	0.000000	0.000000	0.000000	0.00000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.00000	0.000000	0.000000	0.000000	0.00000	0.000000	0.000000	0.000000	0.00000	0.000000	0.00000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.00000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	1.0	1.0
8	142.923752	78.678352	99.938553	65.185890	72.991386	68.257805	24.202347	9.102827	38.014297	35.750080	12.087891	4.571796	35.611473	33.123123	11.181815	6.782530	29.437706	25.464855	7.476874	12.736403	26.011148	24.477785	8.453134	2.441799	25.178915	23.511496	8.387461	3.292090	26.343893	22.708706	7.349658	11.148897	14.659115	13.821280	4.732279	1.211350	12.022038	11.286951	3.723090	1.809071	11.740626	11.090772	3.692733	1.095822	11.710329	10.720157	4.432101	1.602043	11.045879	9.710128	2.566499	4.597603	10.035184	9.327897	3.396616	1.469095	9.884888	8.961467	2.712603	3.169369	8.882207	7.292553	1.885134	4.707286	6.543412	6.489433	-0.837826	-0.039319	6.999748	6.033854	1.952853	2.962336	6.257664	5.496066	2.890849	0.771110	7.793723	5.734364	1.738196	4.983760	4.510467	4.178370	1.598105	0.575843	4.782681	4.108604	1.113110	2.180458	4.182361	3.912849	1.477021	-0.012807	4.109838	4.077982	0.468581	-0.203135	3.711725	3.349799	1.546700	0.404313	4.168599	3.340050	1.566669	1.940832	4.268133	3.465314	1.210372	2.177971	4.612126	2.487145	2.594614	2.890294	3.607978	3.168864	1.666775	0.444598	3.494550	1.425179	3.127082	0.634120	3.491037	3.115516	0.857281	1.321350	3.215504	1.748195	2.689125	-0.227790	3.137908	2.968556	1.016842	0.013167	3.484559	2.057550	2.275903	1.651939	2.793626	2.571566	1.040875	0.328598	2.581538	1.263143	2.250417	0.066577	3.051403	1.363279	2.085111	1.762057	2.360827	1.616907	1.685841	0.342134	2.474930	1.347824	1.682887	1.215131	1.996616	1.705726	1.037443	0.026195	1.940771	1.925673	-0.095188	-0.222073	1.953383	1.835132	0.535860	-0.401062	2.335166	1.851506	0.128525	1.417182	1.739371	1.590491	0.703631	0.025517	1.747788	1.568626	0.640641	0.428663	1.702729	1.054443	1.319898	0.212849	1.675363	1.218332	1.108690	0.305474	1.422847	1.390380	-0.218373	-0.208926	1.494798	0.871027	0.889037	0.827856	1.229201	1.005900	0.556967	0.434615	1.132050	0.610936	0.893196	0.332407	1.326933	0.782299	0.727508	0.787079	1.065655	0.966104	0.292436	0.341678	1.136415	0.824682	0.569743	0.535474	0.948853	0.304282	0.863336	0.249770	1.092636	0.401995	0.816073	0.605208	0.766739	0.709495	-0.179992	0.228271	0.945731	0.669617	0.267365	0.611993	0.752638	0.693571	0.178965	-0.231072	0.839839	0.686661	-0.165872	0.454216	0.596012	0.589230	-0.042138	-0.079139	0.426369	0.416082	0.085506	-0.036812	0.523865	0.241910	0.313563	0.342919	0.521278	0.356159	0.144610	0.352093	0.358706	0.269504	0.236387	0.012588	0.299515	0.137145	0.265864	0.014722	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.00000	0.000000	0.000000	0.000000	0.000000	0.00000	0.00000	0.000000	0.000000	0.000000	0.000000	0.000000	0.00000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.00000	0.000000	0.000000	0.000000	0.00000	0.000000	0.000000	0.000000	0.00000	0.000000	0.00000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.00000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	1.0	0.0

Here we can see that model made a handful of errors, which is expected since our accuracy is only around 82%. However, evaluating our model’s predictions on the same data it was trained on is almost always a recipe for disaster! Why? The problem is that the model may memorise the structure of the data it sees and fail to provide good predictions when shown new data. Let’s see how we can evaluate our model on examples from the test set that it has never seen.

Evaluate your model#

The learners in fastai are equipped with predict() and get_preds() methods that allow one to evaluae the model on new data. To use them, we’ll need a new DataLoader which we can create by simply passing in a DataFrame of the test events:

test_dl = learn.dls.test_dl(test_items=test_df)

Now that we have a DataLoader, it’s a simple matter to compute the predictions with the get_preds() method:

preds, targs = learn.get_preds(dl=test_dl)

Let’s take a look at the first few values of preds and targs:

preds[:5], targs[:5]

(tensor([[0.9980, 0.0020],
         [0.6680, 0.3320],
         [0.8597, 0.1403],
         [0.4682, 0.5318],
         [0.9532, 0.0468]]),
 tensor([[0],
         [0],
         [0],
         [0],
         [0]], dtype=torch.int8))

Here we can see that they are tensors. In PyTorch, a tensor is similar to the arrays that you may be familiar with in Numpy. Tensors have a rank that can be inspected by using the size() method:

preds.size(), targs.size()

(torch.Size([404000, 2]), torch.Size([404000, 1]))

In this case, we see that preds is a rank-2 tensor (i.e. a matrix), while targs is rank-1 (a vector). Note that each dimension in preds corresponds to the probabilities of the model for the two classes (signal vs background). We can visualise the distribution of the probabilities for each class by filtering the preds tensor according to the ground truch labels and then plotting the result as a histogram:

signal_test = preds[:, 1][targs.flatten() == 1].numpy()
background_test = preds[:, 1][targs.flatten() == 0].numpy()

plt.hist(signal_test, histtype="step", bins=20, range=(0, 1), label="Signal")
plt.hist(background_test, histtype="step", bins=20, range=(0, 1), label="Background")
plt.xlabel("Probability")
plt.ylabel("Events/bin")
plt.yscale("log")
plt.xlim(0, 1)
plt.legend(loc="lower right", frameon=False)
plt.show()

We see that although the model assigns high (low) probabilities to the signal (background) events, a fair amount of the signal events overlap with background ones. To handle this, one usually defines a “cut” or threshold that only includes events above that value. For example, if define the cut at 0, then all the events are counted and the signal efficiency \(\epsilon_S\) and background efficiency \(\epsilon_B\) are both 1. As we increase the cut, we reject more and more background events and the result is a curve with \(\epsilon_{B,S}\) ranging from 0 to 1.

This curve is equivalent to the Reciever Operating Characteristic (ROC) curve which plots the true positive rate

\[ \mathrm{TPR} = \frac{\mathrm{TP}}{\mathrm{TP} + \mathrm{FP}} \,, \qquad \mathrm{TP\, (FP)} = \mathrm{number\, of\, true\, (false) \,positives}\,, \]

against the false positive rate FPR, where the FPR is the ratio of negative instances that are incorrectly classified as positive. In general there is a tradeoff between these two quantities: the higher the TPR, the more false positives (FPR) the classifier produces.

To visualise the ROC curve for our model’s predictions, we can use the handy roc_curve() function from scikit-learn:

# fpr = epsilon_B, tpr = epsilon_S
fpr, tpr, thresholds = roc_curve(y_true=targs, y_score=preds[:, 1])

plt.plot(fpr, tpr)
plt.plot([0, 1], [0, 1], ls="--", color="k")
plt.xlabel(r"$\epsilon_B$")
plt.ylabel(r"$\epsilon_S$")
plt.tight_layout()

A perfect classifier would have a ROC curve with all signal and background events correctly identified, i.e. an Area Under the Curve (AUC) of 1. Let’s compute this area along with the accuracy on the test set:

acc_test = accuracy_score(targs, preds.argmax(dim=-1))
auc_test = auc(fpr, tpr)
print(f"Accuracy: {acc_test:.4f}")
print(f"AUC: {auc_test:.4f}")

Accuracy: 0.8314
AUC: 0.9077

Since the AUC is dominated by values at large \(\epsilon_B\), it is common to also report the background rejection at a fixed signal efficiency (often 30%). We can do that by defining an interpolating function across the tpr and fpr values as follows:

background_eff = interp1d(tpr, fpr)
background_eff_at_30 = background_eff(0.3)
print(f"Backround rejection at signal efficiency 0.3: {1/background_eff_at_30:0.3f}")

Backround rejection at signal efficiency 0.3: 42.233

Comparing these results again the The Machine Learning Landscape of Top Taggers review, shows that our baseline model falls short of the models in the review, which get a typical accuracy of 93% and an AUC of 98%.

_images/top-tagging-scores.png — Fig. 3 Figure reference: The Machine Learning Landscape of Top Taggers#

Let’s see if we can train a better model by choosing a clever representation of the input data! Before doing that, let’s collect this evaluation logic in a function that we can reuse later:

def compute_metrics(learn, test_df):
    test_dl = learn.dls.test_dl(test_items=test_df)
    preds, targs = learn.get_preds(dl=test_dl)
    fpr, tpr, _ = roc_curve(y_true=targs, y_score=preds[:, 1])
    acc_test = accuracy_score(targs, preds.argmax(dim=-1))
    auc_test = auc(fpr, tpr)
    background_eff = interp1d(tpr, fpr)
    background_eff_at_30 = background_eff(0.3)

    print(f"Accuracy: {acc_test:.4f}")
    print(f"AUC: {auc_test:.4f}")
    print(
        f"Backround rejection at signal efficiency 0.3: {1/background_eff_at_30:0.3f}"
    )
    return fpr, tpr

fpr_baseline, tpr_baseline = compute_metrics(learn, test_df)

Accuracy: 0.8314
AUC: 0.9077
Backround rejection at signal efficiency 0.3: 42.233

Jet representations#

In any machine learning problem, how we represent the data often has a large impact on the performance of the models we train. For jet tagging, the most common approaches one finds in the literature include:

Jets as images. A jet image is a pixelated grayscale image, where the pixel intensity represents the energy (or transverse momentum) of all particles that deposited energy in a particular location in the \(\eta-\phi\) plane. Typically, convolutional neural networks (CNNs) are used to process the images and we’ll ecplore these architectures in a future lesson.
Jets as sequences. Here the idea is to order the particles in a jet (usually by \(p_T\)) and use sequence-based architectures like recurrent neural networks (RNNs).
Jets as graphs. This approach treats each jet as a generic graph of nodes and edges. Graph neural networks (which we’ll also encounter later in the course) excel on this tpe of data.
Jets as sets. A generalisation of the previous representations, this approach simply treats each jets as an unordered collection or point cloud of 4-vectors.
Theory-inspired representations. Instead of representing the jets in formats to match specific neural network architectures, these approaches use results on IR safety from QCD to represent the jets as a simplified set of features. Fully-connected neural networks are then trained on these features.

You can find more details about each representation in a nice review article from 2017.

_images/jets.png — Fig. 4 Figure reference: Jet Substructure at the Large Hadron Collider#

In this lesson and the next, we’ll use one of the theory-inspired representations called \(N\)-subjettiness. Let’s take a look.

Representing jets with \(N\)-subjettiness observables#

\(N\)-subjettiness observables quantify how much of the radiation in a jet is aligned along different subjet axes. Although originally used for analytic approaches to distinguish different decays and event topologies, these observable can also be used as inputs for machine learning models and provide strong discriminating power.

To be precise, an \(N\)-subjettiness observable \(\tau_N^{(\beta)}\) measures the radiation about \(N\) axes in the jet according to an angular exponent \(\beta>0\):

\[ \tau_N^{(\beta)} = \frac{1}{p_{T,J}} \sum_{i\in J} p_{T,i} \min \left\{ R_{1,i}^\beta, R_{1,i}^\beta, \ldots , R_{1,i}^\beta \right\} \]

Here \(p_{T,J}\) is the transverse momentum of the jet, \(p_{T,i}\) is the transverse momentum of particle \(i\) in the jet, and \(R_{K,i}\) is the distance in the \(\eta-\phi\) plane of particle \(i\) to axis \(K\).

To measure substructure in a jet, one thus needs to measure a suitable number of \(N\)-subjettiness observables. In practice this is done by specifying the corrdinates of \(M\)-body phase space in terms of \(3M - 4\) \(N\)-subjettiness observables:

\[ \left\{ \tau_1^{(0.5)}, \tau_1^{(1)}, \tau_1^{(2)}, \tau_2^{(0.5)}, \tau_2^{(1)}, \tau_2^{(2)}, \ldots , \tau_{m-1}^{(0.5)}, \tau_{m-1}^{(1)}, \tau_{m-1}^{(2)} \right\} \]

To see how we can use this basis as features for a neural network, we have computed \(N\)-subjettiness observables up through 6-body phase space using the pyjet library. You can download these features via the load_dataset() function as follows:

nsubjet_ds = load_dataset("dl4phys/top_tagging_nsubjettiness")

Downloading and preparing dataset parquet/dl4phys--top_tagging_nsubjettiness to /home/lewis/.cache/huggingface/datasets/parquet/dl4phys--top_tagging_nsubjettiness-d7eca4f13187c4c4/0.0.0/0b6d5799bb726b24ad7fc7be720c170d8e497f575d02d47537de9a5bac074901...

Dataset parquet downloaded and prepared to /home/lewis/.cache/huggingface/datasets/parquet/dl4phys--top_tagging_nsubjettiness-d7eca4f13187c4c4/0.0.0/0b6d5799bb726b24ad7fc7be720c170d8e497f575d02d47537de9a5bac074901. Subsequent calls will reuse this data.

As before, we’ll convert our Dataset object to a pandas DataFrame:

nsubjet_ds.set_format("pandas")
train_df, test_df = nsubjet_ds["train"][:], nsubjet_ds["test"][:]
train_df.head()

	pT	mass	tau_1_0.5	tau_1_1	tau_1_2	tau_2_0.5	tau_2_1	tau_2_2	tau_3_0.5	tau_3_1	...	tau_4_0.5	tau_4_1	tau_4_2	tau_5_0.5	tau_5_1	tau_5_2	tau_6_0.5	tau_6_1	tau_6_2
0	543.633944	25.846792	0.165122	0.032661	0.002262	0.048830	0.003711	0.000044	0.030994	0.001630	...	0.024336	0.001115	0.000008	0.004252	0.000234	7.706005e-07	0.000000	0.000000	0.000000e+00
1	452.411860	13.388679	0.162938	0.027598	0.000876	0.095902	0.015461	0.000506	0.079750	0.009733	...	0.056854	0.005454	0.000072	0.044211	0.004430	6.175314e-05	0.037458	0.003396	3.670517e-05
2	429.495258	32.021091	0.244436	0.065901	0.005557	0.155202	0.038807	0.002762	0.123285	0.025339	...	0.078205	0.012678	0.000567	0.052374	0.005935	9.395772e-05	0.037572	0.002932	2.237277e-05
3	512.675443	6.684734	0.102580	0.011369	0.000170	0.086306	0.007760	0.000071	0.068169	0.005386	...	0.044705	0.002376	0.000008	0.027895	0.001364	4.400042e-06	0.009012	0.000379	6.731099e-07
4	527.956859	133.985415	0.407009	0.191839	0.065169	0.291460	0.105479	0.029753	0.209341	0.049187	...	0.143768	0.033249	0.003689	0.135407	0.029054	2.593460e-03	0.110805	0.023179	2.202088e-03

5 rows × 21 columns

Following Section 3.2.2 of The Machine Learning Landscape of Top Taggers revie, we’ve also included the jet mass and jet \(p_T\) as input variables to allow the network to learn physical scales.

Let’s now train a model using these features. As before, we need to first define our DataLoaders object:

features = list(train_df.drop(columns=["label"]).columns)
splits = RandomSplitter(valid_pct=0.20, seed=42)(range_of(train_df))

dls = TabularDataLoaders.from_df(
    df=train_df,
    cont_names=features,
    y_names="label",
    y_block=CategoryBlock,
    splits=splits,
    bs=1024,
)

And just like before it’s a good idea to sanity check your data is formatted correctly with the show_batch() method:

dls.show_batch()

	pT	mass	tau_1_0.5	tau_1_1	tau_1_2	tau_2_0.5	tau_2_1	tau_2_2	tau_3_0.5	tau_3_1	tau_3_2	tau_4_0.5	tau_4_1	tau_4_2	tau_5_0.5	tau_5_1	tau_5_2	tau_6_0.5	tau_6_1	tau_6_2	label
0	461.643372	58.081078	0.267102	0.088350	0.015706	0.182025	0.042736	0.005997	0.141589	0.029279	0.003416	0.120514	0.020182	0.001241	0.087524	0.013450	8.641806e-04	0.071813	0.010891	7.935878e-04	0
1	513.609070	129.866760	0.431575	0.202836	0.063762	0.327400	0.133254	0.040518	0.130512	0.023113	0.001422	0.114296	0.017393	0.000718	0.095355	0.013060	4.953301e-04	0.082081	0.009201	2.063908e-04	1
2	538.835754	21.680426	0.198053	0.039486	0.001619	0.038596	0.003683	0.000069	0.027622	0.002028	0.000024	0.018646	0.000869	0.000002	0.008453	0.000373	9.324909e-07	0.000000	0.000000	0.000000e+00	0
3	569.943970	14.177874	0.124071	0.018201	0.000619	0.091646	0.008896	0.000096	0.039875	0.002515	0.000014	0.008931	0.000650	0.000005	0.000000	0.000000	0.000000e+00	0.000000	0.000000	0.000000e+00	0
4	462.787750	167.859375	0.618895	0.366184	0.132532	0.274511	0.074367	0.006216	0.217250	0.049036	0.003042	0.185648	0.036743	0.001925	0.159761	0.027603	9.593506e-04	0.136480	0.022875	7.589663e-04	0
5	481.187836	21.275795	0.197569	0.041092	0.001955	0.117185	0.018937	0.000665	0.099367	0.012753	0.000281	0.085475	0.009725	0.000197	0.062330	0.005595	6.916394e-05	0.052907	0.004203	3.618403e-05	0
6	574.911987	42.508812	0.180569	0.037991	0.005424	0.108725	0.020727	0.004305	0.041011	0.003381	0.000071	0.028611	0.002046	0.000036	0.015887	0.001213	2.921311e-05	0.011662	0.000650	1.678485e-05	0
7	463.576508	53.591480	0.288984	0.092856	0.013347	0.224898	0.052918	0.003495	0.173352	0.032209	0.001432	0.133511	0.024035	0.001166	0.094518	0.015667	6.733207e-04	0.088208	0.012934	3.997425e-04	0
8	519.291260	17.618988	0.151518	0.026514	0.001151	0.085697	0.009568	0.000241	0.058985	0.004627	0.000063	0.039815	0.002514	0.000013	0.028490	0.001495	5.074164e-06	0.017248	0.000635	8.944363e-07	0
9	478.084625	124.346436	0.507225	0.254765	0.067330	0.182893	0.043455	0.006483	0.126383	0.019826	0.001121	0.089138	0.014123	0.000801	0.044779	0.005901	1.272047e-04	0.029706	0.003584	5.757449e-05	1

This looks good, so the last step is to create a Learner and find a good learning rate:

learn = tabular_learner(
    dls, layers=[200, 200, 50, 50], metrics=[accuracy, RocAucBinary()]
)

learn.lr_find()

SuggestedLRs(valley=0.002511886414140463)

This curve is similar to what we found before so let’s pick a learning rate of \(10^{-3}\) and train for 3 epochs:

learn.fit_one_cycle(n_epoch=3, lr_max=1e-3)

epoch	train_loss	valid_loss	accuracy	roc_auc_score	time
0	0.234615	0.228308	0.903832	0.966693	00:08
1	0.225498	0.227929	0.903348	0.967744	00:08
2	0.222219	0.224911	0.904133	0.968275	00:08

We can already see that training on the \(N\)-subjettiness features has produced a better model than our baseline, which achieved around 83% and \(91%\) accuracy and AUC score respectively. Let’s wrap up by computing these metrics on the test set with our compute_metrics() function:

test_df = nsubjet_ds["test"].to_pandas()
fpr_nsubjet, tpr_nsubjet = compute_metrics(learn, test_df)

Accuracy: 0.9037
AUC: 0.9677
Backround rejection at signal efficiency 0.3: 373.915

This is much better and now just a 1-2% the classifiers reported in the review paper! We can also compare both models by plotting the background rejection rate against the signal efficiency:

fig, ax = plt.subplots()

plt.plot(tpr_baseline, 1 / (fpr_baseline + 1e-6), label="Baseline")
plt.plot(tpr_nsubjet, 1 / (fpr_nsubjet + 1e-6), label="6-body N-subjettiness")
plt.xlabel("Signal befficiency $\epsilon_{S}$")
plt.ylabel("Background rejection $1/\epsilon_{B}$")
plt.xlim(0, 1)
plt.yscale("log")
plt.legend()
plt.show()

Exercises#

Try changing the network architecture by adjusting the layers argument in tabular_learner() on the \(N\)-subjettiness features. What happens if you keep the number of nodes fixed and increase the number of layers? Similarly, what happens if you have just a single layer, but increase the number of nodes on that layer?
Push one of your new models to the dl4phys organisation on the Hugging Face Hub.
Train a model on the \(N\)-subjettiness features without including the jet mass and \(p_T\). Does this have a positive or negative impact on performance?
Train a model using 2-body and 4-body \(N\)-subjettiness features to see if the performance is saturated with a smaller number of features.

Deep Learning for Particle Physicists

Lecture 1 - Jet tagging with neural networks

Contents

Lecture 1 - Jet tagging with neural networks#

Learning objectives#

References#

The task and data#

Setup#

Import libraries#

Getting the data#

Introducing fastai#

From data to DataLoaders#

Create a Learner#

Find a good learning rate#

Train your model#

Evaluate your model#

Jet representations#

Representing jets with \(N\)-subjettiness observables#

Exercises#

Deep Learning for Particle Physicists

Lecture 1 - Jet tagging with neural networks

Contents

Lecture 1 - Jet tagging with neural networks#

Learning objectives#

References#

The task and data#

Setup#

Import libraries#

Getting the data#

Introducing fastai#

From data to DataLoaders#

Create a Learner#

Find a good learning rate#

Train your model#

Evaluate your model#

Jet representations#

Representing jets with \(N\)-subjettiness observables#

Saving and sharing the model#

Exercises#