Skip to content

Fix sky scan orientation, p-value plots and wording in the tutorials #340

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
tomaskontrimas wants to merge 2 commits into
base: master
Choose a base branch
from
Open

Fix sky scan orientation, p-value plots and wording in the tutorials #340

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
6fb78dd
Fix sky scan orientation, p-value plots and wording in the tutorials
tomaskontrimas Oct 28, 2026
a4d3ff9
Fix the dataset concept notebook and DatasetOrigin string representation
tomaskontrimas Oct 28, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
113 changes: 94 additions & 19 deletions doc/sphinx/concepts/dataset.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,18 +19,18 @@
"collection of individual datasets, one for each partial IceCube detector\n",
"configuration.\n",
"\n",
"SkyLLh provides the :py:class:`skyllh.core.dataset.Dataset` class to create an\n",
"SkyLLH provides the :py:class:`skyllh.core.dataset.Dataset` class to create an\n",
"individual dataset definition. Such a definition defines the experimental and\n",
"monte-carlo data files and possibly additional information like data binning\n",
"definitions or auxilary data files.\n",
"definitions or auxiliary data files.\n",
"\n",
"Individual datasets can be combined into a dataset collection via the \n",
":py:class:`skyllh.core.dataset.DatasetCollection` class.\n",
"\n",
"A dataset collection is usually defined within one Python module providing the\n",
"function ``create_dataset_collection``. For instance the 10-year public \n",
"point-source data sample is defined in the \n",
":py:mod:`skyllh.datasets.i3.PublicData_10y_ps` module, and the its dataset \n",
":py:mod:`skyllh.datasets.i3.PublicData_10y_ps` module, and its dataset \n",
"collection can be created via the \n",
":py:func:`~skyllh.datasets.i3.PublicData_10y_ps.create_dataset_collection`\n",
"function. This function requires a configuration. If no data repository base \n",
Expand Down Expand Up @@ -71,7 +71,7 @@
{
"data": {
"text/plain": [
"<skyllh.core.dataset.DatasetCollection at 0x7fa0a37834f0>"
"<skyllh.core.dataset.DatasetCollection at 0x7fed6518b450>"
]
},
"execution_count": 3,
Expand Down Expand Up @@ -128,9 +128,9 @@
" Dataset \"My Dataset\": v001patch00\n",
" { livetime = 365.000 days }\n",
" Experimental data:\n",
" [\u001b[92mFOUND\u001b[0m] /data/ana/analyses/my_dataset_v001_p00/exp.npy\n",
" [\u001b[91mNOT FOUND\u001b[0m] /data/ana/analyses/my_dataset_v001_p00/exp.npy\n",
" MC data:\n",
" [\u001b[92mFOUND\u001b[0m] /data/ana/analyses/my_dataset_v001_p00/mc.npy\n",
" [\u001b[91mNOT FOUND\u001b[0m] /data/ana/analyses/my_dataset_v001_p00/mc.npy\n",
" \n"
]
}
Expand Down Expand Up @@ -161,9 +161,9 @@
"Dataset \"My Dataset\": v001patch00\n",
" { livetime = 365.000 days }\n",
" Experimental data:\n",
" [\u001b[92mFOUND\u001b[0m] /data/ana/analyses/my_dataset_v001_p00/exp.npy\n",
" [\u001b[91mNOT FOUND\u001b[0m] /data/ana/analyses/my_dataset_v001_p00/exp.npy\n",
" MC data:\n",
" [\u001b[92mFOUND\u001b[0m] /data/ana/analyses/my_dataset_v001_p00/mc.npy\n",
" [\u001b[91mNOT FOUND\u001b[0m] /data/ana/analyses/my_dataset_v001_p00/mc.npy\n",
" \n"
]
}
Expand Down Expand Up @@ -211,12 +211,12 @@
"Dataset \"My Dataset\": v001patch00\n",
" { livetime = 365.000 days }\n",
" Experimental data:\n",
" [\u001b[92mFOUND\u001b[0m] /data/ana/analyses/my_dataset_v001_p00/exp.npy\n",
" [\u001b[91mNOT FOUND\u001b[0m] /data/ana/analyses/my_dataset_v001_p00/exp.npy\n",
" MC data:\n",
" [\u001b[92mFOUND\u001b[0m] /data/ana/analyses/my_dataset_v001_p00/mc.npy\n",
" [\u001b[91mNOT FOUND\u001b[0m] /data/ana/analyses/my_dataset_v001_p00/mc.npy\n",
" Auxiliary data:\n",
" aux_file_key_1: \n",
" [\u001b[92mFOUND\u001b[0m] /data/ana/analyses/my_dataset_v001_p00/aux_data/aux_file1.dat\n"
" [\u001b[91mNOT FOUND\u001b[0m] /data/ana/analyses/my_dataset_v001_p00/aux_data/aux_file1.dat\n"
]
}
],
Expand Down Expand Up @@ -265,7 +265,11 @@
"at the origin, and a transfer function which will be used to perform the actual\n",
"data transfer.\n",
"\n",
"SkyLLH provides two dataset transfer methods, ``wget`` and ``rsync``. "
"SkyLLH provides three dataset transfer methods: ``rsync`` via the\n",
":py:class:`~skyllh.core.dataset.RSYNCDatasetTransfer` class, ``wget`` via the\n",
":py:class:`~skyllh.core.dataset.WGETDatasetTransfer` class, and Python's\n",
":py:func:`urllib.request.urlretrieve` function via the\n",
":py:class:`~skyllh.core.dataset.URLRetrieveDatasetTransfer` class."
]
},
{
Expand All @@ -276,7 +280,9 @@
"source": [
"from skyllh.core.dataset import (\n",
" DatasetOrigin,\n",
" URLRetrieveDatasetTransfer,\n",
" WGETDatasetTransfer,\n",
" post_transfer_unarchive,\n",
")"
]
},
Expand All @@ -302,7 +308,7 @@
},
"source": [
"In the example above we specified that the dataset is available at the URL\n",
"``data.mydomain.com/downloads/data/my_dataset``, which can be transfered\n",
"``data.mydomain.com/downloads/data/my_dataset``, which can be transferred\n",
"using ``wget`` via the https protocol.\n",
"\n",
"Hence, the experimental and monte-carlo files ``exp.npy`` and ``mc.npy`` \n",
Expand All @@ -328,23 +334,42 @@
"webserver. In such cases the ``filename`` argument of the \n",
":py:class:`~skyllh.core.dataset.DatasetOrigin` class constructor can be used in\n",
"combination with a post transfer function specified via the \n",
"``post_transfer_func`` argument of the constructor:"
"``post_transfer_func`` argument of the constructor. SkyLLH provides the\n",
":py:func:`~skyllh.core.dataset.post_transfer_unarchive` function, which\n",
"extracts ``.zip``, ``.tar.gz``, ``.tgz``, ``.tar.bz2``, and ``.tar.xz`` archive\n",
"files:"
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {},
"outputs": [],
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"DatasetOrigin {\n",
" base_path = \"/downloads/data\"\n",
" sub_path = \"\"\n",
" filename = my_dataset.zip\n",
" user@host:port = None@data.mydomain.com:None\n",
" password = not set\n",
" transfer class = WGETDatasetTransfer\n",
" }\n"
]
}
],
"source": [
"origin = DatasetOrigin(\n",
" host='data.mydomain.com',\n",
" base_path='/downloads/data',\n",
" sub_path='',\n",
" filename='my_dataset.zip',\n",
" transfer_func=WGETDatasetTransfer(protocol='https').transfer,\n",
" post_transfer_func=WGETDatasetTransfer.post_transfer_unzip,\n",
")"
" post_transfer_func=post_transfer_unarchive,\n",
")\n",
"print(origin)"
]
},
{
Expand All @@ -354,8 +379,58 @@
},
"source": [
"The example above will transfer the single archive file \n",
"``https://data.mydomain.com/downloads/data/my_dataset.zip`` and unzip the file\n",
"on the local host."
"``https://data.mydomain.com/downloads/data/my_dataset.zip`` and extract the\n",
"archive on the local host. The archive file is removed after the extraction."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Origin as download URL"
]
},
{
"cell_type": "raw",
"metadata": {
"raw_mimetype": "text/restructuredtext"
},
"source": [
"If the dataset is provided through a complete download URL, e.g. an API\n",
"endpoint with query parameters, the URL can be specified via the ``url``\n",
"argument of the :py:class:`~skyllh.core.dataset.DatasetOrigin` class\n",
"constructor. The ``filename`` argument then specifies the local filename of the\n",
"downloaded file. This is how the IceCube public data samples are downloaded\n",
"from the Harvard Dataverse, e.g. in the\n",
":py:mod:`skyllh.datasets.i3.PublicData_14y_ps_v2` module:"
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {},
"outputs": [],
"source": [
"origin = DatasetOrigin(\n",
" url='https://data.mydomain.com/api/access/dataset?id=my_dataset',\n",
" base_path='',\n",
" sub_path='my_dataset',\n",
" filename='my_dataset.zip',\n",
" transfer_func=URLRetrieveDatasetTransfer(protocol='https').transfer,\n",
" post_transfer_func=post_transfer_unarchive,\n",
")\n",
"my_dataset.origin = origin"
]
},
{
"cell_type": "raw",
"metadata": {
"raw_mimetype": "text/restructuredtext"
},
"source": [
"The example above will download the archive file from the given URL, save it as\n",
"``my_dataset.zip`` into the ``my_dataset`` sub directory of the dataset's base\n",
"path on the local host, and extract it there."
]
},
{
Expand Down
16 changes: 8 additions & 8 deletions doc/sphinx/tutorials/dataset_collections.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@
"id": "3cd8d21f",
"metadata": {},
"source": [
"The available data samples are accessible through the `skyllh.data_samples` dictionary"
"The available data samples are accessible through the `skyllh.datasets.data_samples` dictionary"
]
},
{
Expand All @@ -47,12 +47,12 @@
{
"data": {
"text/plain": [
"{'IceTracks-DR1': <module 'skyllh.datasets.i3.PublicData_10y_ps' from '/Users/tomas/projects/skyllh_development/skyllh/skyllh/datasets/i3/PublicData_10y_ps.py'>,\n",
" 'IceTracks-DR2': <module 'skyllh.datasets.i3.PublicData_14y_ps_v2' from '/Users/tomas/projects/skyllh_development/skyllh/skyllh/datasets/i3/PublicData_14y_ps_v2.py'>,\n",
" 'IceTracks-DR2-v1': <module 'skyllh.datasets.i3.PublicData_14y_ps' from '/Users/tomas/projects/skyllh_development/skyllh/skyllh/datasets/i3/PublicData_14y_ps.py'>,\n",
" 'IceTracks-DR2-v2': <module 'skyllh.datasets.i3.PublicData_14y_ps_v2' from '/Users/tomas/projects/skyllh_development/skyllh/skyllh/datasets/i3/PublicData_14y_ps_v2.py'>,\n",
" 'IceTracks-DR1_wMC': <module 'skyllh.datasets.i3.PublicData_10y_ps_wMC' from '/Users/tomas/projects/skyllh_development/skyllh/skyllh/datasets/i3/PublicData_10y_ps_wMC.py'>,\n",
" 'TestData': <module 'skyllh.datasets.i3.TestData' from '/Users/tomas/projects/skyllh_development/skyllh/skyllh/datasets/i3/TestData.py'>}"
"['IceTracks-DR1',\n",
" 'IceTracks-DR2',\n",
" 'IceTracks-DR2-v1',\n",
" 'IceTracks-DR2-v2',\n",
" 'IceTracks-DR1_wMC',\n",
" 'TestData']"
]
},
"execution_count": 2,
Expand All @@ -63,7 +63,7 @@
"source": [
"from skyllh.datasets import data_samples\n",
"\n",
"data_samples"
"list(data_samples.keys())"
]
},
{
Expand Down
Loading
Loading