Skip to content

SP-2832: update the commissioning tutorials #114

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
MelissaGraham merged 4 commits into from
Jan 29, 2026
Merged

SP-2832: update the commissioning tutorials #114

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
39f6337
add ddf-specific subsections
MelissaGraham Jan 28, 2026
1fbe853
add ddf and filter functionality
MelissaGraham Jan 29, 2026
0e1dfe3
fixes post review
MelissaGraham Jan 29, 2026
052da8f
fix wording
MelissaGraham Jan 29, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
86 changes: 79 additions & 7 deletions Commissioning/101_lsstcam_visits_database.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,12 +18,12 @@
"\n",
"</div>\n",
"\n",
"For the Rubin Science Platform at data.lsst.cloud. <br>\n",
"Container Size: Large <br>\n",
"LSST Science Pipelines version: v29.2.0 <br>\n",
"Last verified to run: 2025-11-12 <br>\n",
"Repository: <a href=\"https://github.com/lsst/tutorial-notebooks\">github.com/lsst/tutorial-notebooks</a> <br>\n",
"DOI: <a href=\"https://doi.org/10.11578/rubin/dc.20250909.20\">10.11578/rubin/dc.20250909.20</a> <br>"
"For the Rubin Science Platform at data.lsst.cloud.\\\n",
"Container Size: Large\\\n",
"LSST Science Pipelines version: v29.2.0\\\n",
"Last verified to run: 2026-01-28\\\n",
"Repository: <a href=\"https://github.com/lsst/tutorial-notebooks\">github.com/lsst/tutorial-notebooks</a>\\\n",
"DOI: <a href=\"https://doi.org/10.11578/rubin/dc.20250909.20\">10.11578/rubin/dc.20250909.20</a>"
]
},
{
Expand Down Expand Up @@ -1039,7 +1039,27 @@
"id": "99aa1d6b-4c85-4d41-aafb-0b1b5f7d8faa",
"metadata": {},
"source": [
"Recreate the table above, but for the SV DDF, and include all field (do not restrict to fields with >50 visits)."
"Recreate the two tables above, but for the SV DDF, and include all fields (do not restrict to fields with >50 visits)."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "b862d38b-706d-48a6-a243-4c239f96ec13",
"metadata": {},
"outputs": [],
"source": [
"query = \"observation_reason.str.contains('ddf')\"\n",
"df_ddf = df.query(query).groupby(['observation_reason', 'band']).agg({'seq_num': 'count'})\n",
"df_ddf.rename({'seq_num': 'count'}, axis=1, inplace=True)\n",
"df_ddf = df_ddf.reset_index('band').pivot(columns=[\"band\"]).droplevel(0, axis=1)\n",
"df_ddf = df_ddf[['u', 'g', 'r', 'i', 'z', 'y']]\n",
"df_ddf['all'] = df_ddf.sum(axis=1)\n",
"df_ddf = df_ddf.query(\"all > 0\").sort_values('all')\n",
"table = tabulate(pd.DataFrame(df_ddf.round(0)), headers='keys')\n",
"table = table.replace('nan', ' 0')\n",
"print(table)\n",
"del query, df_ddf, table"
]
},
{
Expand Down Expand Up @@ -1222,6 +1242,58 @@
"Read more about they sky coverage for the SV wide-area survey on the <a href='https://survey-strategy.lsst.io/progress/sv_status/sv_20250930.html#sv-wide'> Science Validation survey summary webpage </a>."
]
},
{
"cell_type": "markdown",
"id": "f40d0511-12bc-451f-832d-ff5eabffee03",
"metadata": {},
"source": [
"#### 5.1. Sky map for DDFs only\n",
"\n",
"Create a sky map for only the DDFs, which will be where alert production will start in early 2026."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "e95d55ce-5a17-403a-a7ea-2bf48caa653d",
"metadata": {},
"outputs": [],
"source": [
"constraint = \"observation_reason like \\'%ddf%\\'\"\n",
"bundle = maf.MetricBundle(metric, slicer, constraint, run_name=run_name)\n",
"group = maf.MetricBundleGroup({'nvisits': bundle},\n",
" opsim_fname, out_dir=output_path)\n",
"group.run_all()"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "113eccfe-9598-4ce9-8708-b50a1cbbc952",
"metadata": {},
"outputs": [],
"source": [
"ph = maf.PlotHandler(savefig=False, fig_format='png', thumbnail=False, dpi=270)\n",
"ph.set_metric_bundles([bundle])\n",
"ph.plot(plot_func=maf.plots.HealpixSkyMap(),\n",
" plot_dicts={'color_min': 20, 'color_max': 100, 'figsize': (6, 4),\n",
" 'labelsize': 'x-large', 'fontsize': 'x-large', 'extend': 'max',\n",
" 'title': 'Deep Drilling Field LSSTCam Visits as of 2025-09-30'})\n",
"plt.figtext(0.40, 0.62, 'XMM LSS', fontsize='large', fontweight='bold', color='white')\n",
"plt.figtext(0.36, 0.50, 'ECDFS', fontsize='large', fontweight='bold', color='white')\n",
"plt.figtext(0.36, 0.33, 'EDFS', fontsize='large', fontweight='bold', color='white')\n",
"plt.figtext(0.47, 0.43, 'ELAIS-S1', fontsize='large', fontweight='bold', color='white')\n",
"plt.show()"
]
},
{
"cell_type": "markdown",
"id": "2a9aaa25-6d90-41c7-acaf-485ea7dead82",
"metadata": {},
"source": [
"> **Figure 7:** The locations of the four Deep Drilling Fields (DDFs), with their names labeled. See Figure 1 for the field central coordinates."
]
},
{
"cell_type": "markdown",
"id": "135c0da0-4d13-4230-9abe-2add68bb1266",
Expand Down
156 changes: 131 additions & 25 deletions Commissioning/102_rubin_schedule_viewer.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,10 +18,10 @@
"\n",
"</div>\n",
"\n",
"For the Rubin Science Platform at data.lsst.cloud. <br>\n",
"Container Size: Large <br>\n",
"LSST Science Pipelines version: v29.2.0 <br>\n",
"Last verified to run: 2025-12-04 <br>\n",
"For the Rubin Science Platform at data.lsst.cloud.\\\n",
"Container Size: Large\\\n",
"LSST Science Pipelines version: v29.2.0\\\n",
"Last verified to run: 2025-01-28\\\n",
"Repository: [github.com/lsst/tutorial-notebooks](https://github.com/lsst/tutorial-notebooks)\\\n",
"DOI: [10.11578/rubin/dc.20250909.20](https://doi.org/10.11578/rubin/dc.20250909.20)"
]
Expand Down Expand Up @@ -88,28 +88,35 @@
"The ObsLocTAP data model (table schema) for the scheduled observations is defined in the `ivoa.obsplan` table -- <a href=\"https://www.ivoa.net/documents/ObsLocTAP/20210609/PR-ObsLocTAP-1.0-20210609.pdf\">Observation Locator \n",
"Table Access Protocol Version 1.0</a>\n",
"\n",
"The database contains 28 columns in total, but these are the key columns used in this tutorial.\n",
"The database contains 28 columns in total, and these are the key columns.\n",
"\n",
"* `t_planning` : Time (MJD) when this observation has been added or modified into the planning log.\n",
"* `target_name` : The name of the astronomical object observed, if any.\n",
"* `obs_id` : The observation ID from the Camera.\n",
"* `s_ra` : The Right Ascension of the visit or planned observation (degrees).\n",
"* `s_dec` : The Declination of the visit or planned observation (degrees).\n",
"* `s_fov` : The diameter (bounds) of the covered region.\n",
"* `s_region` : Sky region covered by the data product (expressed in ICRS frame).\n",
"* `s_resolution` : The spatial resolution of the data expressed as FWHM.\n",
"* `t_min` : The start time (MJD).\n",
"* `t_max` : The stop time (MJD).\n",
"* `t_exptime` : The total exposure time = `t_max - t_min`.\n",
"* `t_resolution` : Temporal resolution.\n",
"* `facility_name` : Name of the facility used for this observation e.g Rubin Simonyi, AuxTel.\n",
"* `instrument_name` : Name of the instrument used for this observation e.g LSSTCam, LSSTComCam.\n",
"* `t_plan_exptime` : The planned or scheduled exposure time.\n",
"* `s_ra` : The Right Ascension of the visit or planned observation (deg).\n",
"* `s_dec` : The Declination of the visit or planned observation (deg).\n",
"* `em_min` : The filter minimum wavelength (m).\n",
"* `em_max` : The filter maximum wavelength (m).\n",
"* `target_name` : The name of the astronomical object observed, if any.\n",
"* `tracking_type` : One of the following values: Sidereal, Solar-system-object-tracking, Fixed-az-el-transit.\n",
"* `priority` : The Priority level { 0, 1, 2}. 0 = in the target queue as the next visit\n",
"(meaning its execution is highly likely), 1 = in the 1 hour look-ahead summit queue (meaning likely),\n",
"2 = 24 hour pre-night simulation, meaning it is not in the summit queue and has a high degree of uncertainty. \n",
"* `t_planning` : Time (MJD) when this observation has been added or modified into the planning log.\n",
"* `execution_status` : Execution status of the prediction: One of the following values: Planned, Scheduled, Unscheduled, Performed, Aborted.\n",
"* `tracking_type` : One of the following values: Sidereal, Solar-system-object-tracking, Fixed-az-el-transit.\n",
"\n",
"Key columns that remained unpopulated as of the last update of this tutorial.\n",
"\n",
"* `s_region` : Sky region covered by the data product (expressed in ICRS frame).\n",
"* `obs_id` : The observation ID from the Camera (currently the same as `target_name`).\n",
"\n",
"These are also columns of interest, but are generally the same for most observations taken with LSSTCam.\n",
"\n",
"* `t_plan_exptime` : The planned or scheduled exposure time (s).\n",
"* `t_exptime` : The total exposure time = `t_max - t_min` (s).\n",
"* `s_fov` : The diameter (bounds) of the covered region (deg).\n",
"* `s_resolution` : The spatial resolution of the data expressed as FWHM (arcsec/pixel).\n",
"* `facility_name` : Name of the facility used for this observation e.g Rubin Simonyi, AuxTel.\n",
"* `instrument_name` : Name of the instrument used for this observation e.g LSSTCam, LSSTComCam.\n",
"\n",
"See <a href=\"https://dmtn-263.lsst.io/\">DMTN-263</a> for a description of all fields in the schema."
]
Expand Down Expand Up @@ -155,6 +162,53 @@
"warnings.filterwarnings(\"ignore\")"
]
},
{
"cell_type": "markdown",
"id": "cd2a8cab-f364-420d-80ec-3c9f99e47cf7",
"metadata": {},
"source": [
"### 1.4. Define functions\n",
"\n",
"Define a function to convert the `em_min` and `em_max` columns into the bandpass (filter) name."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "27684781-bdf6-4fb4-a57b-e8b510253375",
"metadata": {},
"outputs": [],
"source": [
"def em_min_max_to_band(em_min, em_max):\n",
" \"\"\"\n",
" Convert the wavelength minimum and maximum to band name.\n",
"\n",
" Parameters\n",
" ----------\n",
" em_min: float\n",
" Wavelength minimum, in m.\n",
" em_max: float\n",
" Wavelength maximum, in m.\n",
"\n",
" Returns\n",
" -------\n",
" band: string\n",
" Band (filter) name.\n",
" \"\"\"\n",
"\n",
" band = None\n",
" band_dict = {'u': (2.95e-7, 4.00e-7), 'g': (4.00e-7, 5.55e-7),\n",
" 'r': (5.50e-7, 6.92e-7), 'i': (6.90e-7, 8.20e-7),\n",
" 'z': (8.15e-7, 9.25e-7), 'y': (9.20e-7, 11.1e-7)}\n",
" for b in band_dict.keys():\n",
" if em_min > band_dict[b][0] and em_max < band_dict[b][1]:\n",
" band = b\n",
" if band is None:\n",
" print(em_min, em_max, ' no band found')\n",
"\n",
" return band"
]
},
{
"cell_type": "markdown",
"id": "d7cbe189-5a7c-4373-8fcb-076d661b0a6f",
Expand Down Expand Up @@ -196,7 +250,9 @@
"id": "9",
"metadata": {},
"source": [
"Programmatically check the health of the static UI by executing the following cell, and assert that the static user interface service must be alive."
"Programmatically check the health of the static UI by executing the following cell, and assert that the static user interface service must be alive.\n",
"\n",
"> **Warning:** If `RemoteDisconnect` errors are returned, it means the service is temporarily unstable or unavailable, and to try again later."
]
},
{
Expand All @@ -206,7 +262,7 @@
"metadata": {},
"outputs": [],
"source": [
"rubin_static_schedule_url = obsloctap_url + \"/static/viewer.html\"\n",
"rubin_static_schedule_url = obsloctap_url + \"/static/viewer.html\"\n",
"response = requests.get(rubin_static_schedule_url)\n",
"assert response.status_code == 200, f\"request failed with status {response.status_code}\"\n",
"print(f\"Rubin Schedule Viewer at {response.url} is alive.\")"
Expand Down Expand Up @@ -376,7 +432,57 @@
"id": "26",
"metadata": {},
"source": [
"Note that the values in the \"Execution Status\" field are all \"Scheduled\"."
"Note that the values in the \"Execution Status\" field are all \"Scheduled\".\n",
"\n",
"#### 3.1.1. Planned DDF observations\n",
"\n",
"When alerts begin streaming in early 2026, template images will only exist for the DDFs, and thus alerts will only stream in the DDFs.\n",
"\n",
"Print the number of planned observations per DDF in the next 24h, and in which filters the observations are planned.\n",
"\n",
"First, add a column for the band (filter), converted from the `em_min` and `em_max` columns."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "6769ff38-6b03-4649-9f39-01c475a9e478",
"metadata": {},
"outputs": [],
"source": [
"em_min = forward_24h_schedule['em_min'].values\n",
"em_max = forward_24h_schedule['em_max'].values\n",
"temp = []\n",
"for i in range(len(forward_24h_schedule)):\n",
" temp.append(em_min_max_to_band(em_min[i], em_max[i]))\n",
"forward_24h_schedule['band'] = temp\n",
"del temp"
]
},
{
"cell_type": "markdown",
"id": "4148b3ff-9173-484a-991b-495ee6d25b95",
"metadata": {},
"source": [
"Define the names of the DDF fields, in the format of the `target_name` column.\n",
"For each field, sum the number of planned observations and print the unique filters for those observations."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "fec6e949-a299-4eae-b4b4-f15da82c9e16",
"metadata": {},
"outputs": [],
"source": [
"ddf_names = ['ddf_cosmos', 'ddf_ecdfs', 'ddf_elaiss1',\n",
" 'ddf_edfs_a', 'ddf_edfs_b', 'ddf_xmm_lss']\n",
"\n",
"for ddf_name in ddf_names:\n",
" temp = forward_24h_schedule.query(\"target_name.str.contains(@ddf_name)\")\n",
" band_list = \"\".join(str(value) for value in np.unique(temp['band']))\n",
" print('%-12s %3i bands: %-15s' % (ddf_name, len(temp), band_list))\n",
" del temp, band_list"
]
},
{
Expand Down Expand Up @@ -650,9 +756,9 @@
"metadata": {},
"outputs": [],
"source": [
"print(f\"Min gap: {np.min(obs_visit_gap):.2f}\")\n",
"print(f\"Max gap: {np.max(obs_visit_gap):.2f}\")\n",
"print(f\"Mean gap: {np.mean(obs_visit_gap):.2f}\")"
"print(f\"Min gap: {np.min(obs_visit_gap): .2f}\")\n",
"print(f\"Max gap: {np.max(obs_visit_gap): .2f}\")\n",
"print(f\"Mean gap: {np.mean(obs_visit_gap): .2f}\")"
]
},
{
Expand Down