Introduction

The following is documentation for users of the SKA-Low system. This documentation is intended to help users understand the system and how to interact with it.

Acquire Access

There are two main clusters that you can access:

SKA-Low MCCS: This is the main cluster where you can use the actual stations (currently s8-6 and s8-1)
ITF: The integration test facility where you can use test software with generated signals

There are two ways to access the clusters:

JupyterHub: This is a web-based interface that allows you to run notebooks on the cluster. This is the easiest way to get started.
k9s/kubectl: This is a terminal-based interface that allows you to look at individual pod’s logs and kill them if necessary. This is more advanced and is useful for debugging.

ITF Cluster

To access the ITF cluster, submit a system team ticket to request VPN access. You will need to install the cisco VPN client on your machine and connect to the SKAO VPN before you can access the cluster.

You should then be able to access the ITF JupyterHub interface at https://k8s.lowitf.internal.skao.int/jupyterhub/hub. You will have to login with your SKA and GitLab credentials then click launch server. When you launch the server for the first time, several things will be installed which may cause the launch to time out. If this happens, try again a few times until you can see the JupyterLab interface.

If you plan to use the Tango devices, first make sure that no one else is using the ITF cluster, then reserve time using the calendar.

SKA-Low MCCS Cluster

To access the SKA-Low MCCS cluster, send a message (or email lucio.tirone@csiro.au) to Lucio Tirone to request VPN access (by adding you to the correct security group).

Make sure you are connected to the SKAO VPN and then you can access the SKA-Low JupyterHub interface at https://k8s.mccs.low.internal.skao.int/jupyterhub/hub. You will once again have to login with your credentials and have to try a few times until you can see the JupyterLab interface.

If you plan to use the Tango devices, first make sure that no one else is using the station you plan to use by checking the calendar. To reserve a station follow the instruction at the bottom of the calendar page.

k9s/kubectl Interface

k9s is a terminal-based interface that allows you to look at individual pod’s logs and kill them if necessary. Because this has the ability to kill pods, you will only be granted access if you are a more advanced user (such as part of the Vulcan team). To install k9s, follow the instructions on the k9s website.

To access the k9s interface, you will need to be connected to the SKAO VPN and be granted access through InfraHQ. After you have downloaded and installed infra, follow the SKA instructions to login. If you have the correct permissions, the command infra list will show you the available clusters, which should include au-aa-mccs-cloud01-k8s for SKA-Low MCCS cluster and au-itf-k8s-master01-k8s for the ITF cluster.

Useful Link Summary

Many of the following links are mentioned and explained in other sections, but for your convenience, here is a summary of useful links:

SKA-Low
- JupyterHub
- Calendar
- Grafana
- Taranta
- Kibana
ITF
Repositories
Documentation

High Level Overview

Telescope Components

An interferometer takes signals from multiple receivers and compares them over time. By applying a specific mathematical transformation to these comparisons, an image can be formed.

In SKA Low, the receivers are stations, which consist of a set of antennas whose signals are added together as a phased array to form station beams. The comparison between station beams is performed by the correlator and beamformer, or CBF, which is a part of the central signal processor, or CSP. The number crunching to take the outputs from CSP and produce images and other scientific products is performed by the science data processor, or SDP, and this whole process is orchestrated by telescope monitoring and control software, or TMC.

Note

Why is it called the correlator and beamformer if beams are already formed in MCCS?

Beamforming can happen at multiple levels - antenna signals are summed by TPMs to create station beams, and then those station beams may be summed by CBF to form extremely sensitive tied-array beams, which aren’t useful for imaging but are great for analysing time-domain phenomena like pulsars.

So, the software subsystems in play are:

MCCS, the Monitoring, Calibration and Control Subsystem - this is responsible for controlling the Low station hardware that captures, digitises and sums antenna signals. MCCS is also responsible for calibrating stations, and capturing engineering sample data from stations.
CSP, the Central Signal Processor - receives station beams from site, and does further cross-station processing, either correlating or further beamforming.
SDP, the Science Data Processor - takes products from CSP for further processing, e.g. creating images. Currently, SDP simply receives and stores visibilities from CSP.
TMC, Telescope Management and Control - the high-level control system which coordinates all the other subsystems in order to perform a scan.

MCCS, CSP, and SDP are functional subsystems which each implement a portion of the telescope signal chain. Each of these has a controller and its own representation of a subarray, implemented as Tango devices. The controller is responsible for turning things on and off, and for allocating resources within its subsystem to subarrays.

These functional subsystems are controlled by TMC, which also has its own representation of a subarray.

MCCS

The MCCS control system is loosely hierarchical. The primary interface to a Low station is an MccsStation Tango device. This device should represent the aggregate state of all station hardware, and provide commands to operate a station as an atomic unit. Unfortunately this abstraction is not watertight, so it’s often necessary to interact with devices beneath MccsStation in the hierarchy.

The next tier of devices are SpsStation, and FieldStation. These correspond to the two main functional areas of the hardware under MCCS control - PaSD and SPS.

(MCCS should also theoretically monitor and control the synchronisation and timing system (SAT) and infrastructure like PDUs and chillers, but this functionality is immature as of September 2024.)

MCCS Tango devices for high-level control
Tango device name	comment
MccsController	allocates resources to subarrays, turns the array on and off
MccsStation	represents a station, both the field node and signal processing system
MccsStationBeam	a station may eventually have up to 48 beams, each of which is configured for a specific subarray beam
MccsSubarray	MCCS’ internal representation of a subarray
MccsSubarrayBeam	a beam for a specific target and frequency that may be instantiated in multiple stations

There are three devices dedicated to calibration:

MCCS Tango devices for calibration
Tango device name	comment
StationCalibrationSolverDevice	a service device for performing calibration calculations
MccsStationCalibrator	orchestrates calibration: acquiring sample data, calculating and applying calibration solutions
MccsCalibrationStore	stores calibrations solutions?

PaSD

FieldStation represents the power and signal distribution (PaSD) system, which consists of the hardware that sits outside in the weather alongside the antennas - 24 smartboxes and one field node distribution hub (FNDH). The science inputs to this system are the sky, and the outputs are analogue optical signals on fibres, one per antenna.

PaSD hardware and Tango devices are initialised by the FieldNodeOn.ipynb operational notebook. (TODO: add a link to docs or source)

PaSD Tango devices are developed in the https://gitlab.com/ska-telescope/mccs/ska-low-mccs-pasd repository by the Wombat and MCCS teams.

MCCS Tango devices for hardware components
Tango device name	hardware component	comment
FieldStation		An abstract device representing a field node
MccsFNDH	field node distribution hub	sits on the edge of the mesh, and powers and controls smartboxes
MccsPasdBus	ethernet-modbus gateway	The central point of communication with all PaSD hardware. This is the only PaSD Tango device that directly communicates with hardware.
MccsSmartbox	smartbox	contains up to 12 FEMs (front-end modules) which convert an electrical signals from antennas into optical signals on fibre

SPS

SpsStation represents the signal processing system (SPS) for a station. This is high-performance dedicated computing hardware (tile processing modules or TPMs) that lives in climate controlled, EM-shielded structures: either remote processing facilities in the case of stations on the spiral arms, or the central processing facility in the case of stations in the core. The TPMs take analogue optical input from the PaSD, digitise them, add them together and send the resulting station beams to CSP.

The SPS hardware and Tango devices are initialised by the Initialise.ipynb operational notebook. (TODO: add a link to docs or source)

SPS Tango devices are developed in the https://gitlab.com/ska-telescope/mccs/ska-low-mccs-spshw repository by the MCCS team. Within MCCS is a specialist team who develop the signal processing firmware that runs on the TPMs.

MCCS Tango devices for hardware components
Tango device name	hardware component	comment
SpsStation		An aggregate device representing the set of 2 subracks and 8 TPMs in for a station.
MccsTile	TPM	Tile processing module - digitises and beamforms antenna signals
MccsSubrack	subrack	Each subrack distributes power and LMC network to 8 TPMs
MccsDaqReceiver		A software device which receives data samples from TPMs for engineering and calibration purposes

CSP

The central signal processor is responsible for correlating or beamforming the signals from stations and performing further processing for pulsars. Unlike MCCS, there are no Tango devices at the top level - high-level control is provided by the CSP-LMC component.

The CBF is implemented on special-purpose COTS hardware - FPGAs and programmable P4 switches, of which there are six and one respectively in AA0.5.

PSS and PST are implemented in software and run on COTS server hardware.

CSP-LMC

CSP-LMC Tango devices for high-level control
Tango device name	comment
CspController	allocates resources to subarrays, turns the array on and off
CspSubarray	CSP’s internal representation of a subarray

CBF

CBF Tango devices for control of the correlator and beamformer
Tango device name	hardware component	comment
CbfAllocator		allocates CBF hardware components (i.e. FPGAs) for particular observations
CbfConnector	P4 switch	controls the programmable P4 switch by dynamically configuring routing of station beams to FPGAs
CbfProcessor	Alveo FPGA	programs and commands the FPGAs on which the actual correlation and beamforming take place

PSS and PST

TODO - these are so far undeployed in the ITF or in AA0.5.

SDP

SDP has no specialised hardware, and uses less Tango than other areas of the signal chain. SDP relies Kubernetes as a part of its internal operation, using Helm to create on-demand deployments of Pods for receiving data and running scripts for observations. Much of this is unexposed to the Tango world.

SDP manages its state in an internal etcd database.

It still provides Tango devices for its high-level control interface:

SDP Tango devices for high-level control
Tango device name	comment
SdpController	allocates resources to subarrays, turns the array on and off
SdpSubarray	SDP’s internal representation of a subarray

TMC

TMC provides the highest-level Tango devices for controlling the telescope. The main public interfaces are the TmcCentralNode and TmcSubarrayNode. Internally, TMC also has Tango devices coupled to each of the subsystems’ subarray devices and to their controllers.

TMC Tango devices for top-level control
Tango device name	comment
CentralNode	allocates resources to subarrays, turns the array on and off
SubarrayNode	SDP’s internal representation of a subarray
CspMasterLeafNode	TMC’s proxy to CspController
CspSubarrayLeafNode	TMC’s proxy to a CspSubarray device
MccsMasterLeafNode	TMC’s proxy to MccsController
MccsSubarrayLeafNode	TMC’s proxy to an MccsSubarray device
SdpMasterLeafNode	TMC’s proxy to SdpController
SdpSubarrayLeafNode	TMC’s proxy to an SdpSubarray device

Kubernetes

Kubernetes is a container management system. Functionality is divided into groups of containers (“Pods”) which can be deployed amongst many servers (“Nodes”) to share the load. The total set of nodes and pods form a “cluster”. See https://kubernetes.io for more information.

The kubernetes system is used by SKAO so that we can automatically role out new software versions of the different components and roll back these versions if there are any issues with new releases. It also makes the cluster more fault tolerant as if one part of the telescope fails, then that pod can be killed and a new one will automatically be generated to take its place.

Clusters

Currently, the SKAO has three Clusters:

Cluster	Infra Context	Location
Low ITF	infra:au-itf-k8s-master01-k8s	Geraldton ITF
SKA-Low North (MCCS)	infra:au-aa-mccs-cloud01-k8s	AA0.5 Site tcpf
SKA-Low South (SUT)	infra:au-aa-k8s-master01-k8s	Pawsey Centre

See the Acquire Access section if you haven’t already been given access.

Pods

A pod is the smallest unit in the Kubernetes ecosystem. Kubernetes uses the description of the pod to assign it resources such as storage and ram, which container it shall use and other start up commands. These pods are ephemeral which allows automatic restarts in the case of failure.

Pods can be grouped into namespaces for easier management. The logs for a given pod can be also be viewed using Kibana, k9s or kubectl (see Kibana, k9s and kubectl sections).

For SKAO, you can think of each pod contains a docker container that contains a tango device which interacts with hardware or other tango devices.

Tango

Tango is a control system framework for physical hardware (as well as software abstractions). SKAO uses the python binding PyTango to create devices representing elements of the Telescope (such as TPMS and subarrays) as well as test equipment (e.g. an Arbritrary Waveform Generator (AWG) or a Noise Source). After definition and deployment to a cluster, these devices can be used from scripts or Jupyter Notebooks remotely.

Tango Device Servers and Proxies

The Tango devices are managed by a Tango Device Server which we can get DeviceProxy instances from. This device proxy is a class that can communicate with the tango device that is being run in a different kubernetes pod which in turn talks to hardware.

For example, on the ITF we can get a device proxy for an Arbitrary Waveform Generator (AWG) like this:

from tango import DeviceProxy
trl = "tango-databaseds.test-environment:10000/low-itf/awg/1"
awg = DeviceProxy(trl)

We used the Tango Resource Locator (trl) for the AWG to get a device proxy that we interact with in Jupyter Notebooks.

The trl shows the tango database name and port, along with the device name. In the case of SKAO, the tango database is in kubernetes which is partitioned into namespaces, so the namespace qualifier is used. To see a list of tango device servers (pods) residing in a given namespace one can use kubectl, k9s or Taranta (see Useful Link Summary). You can also directly get the list from the tango database using the following code:

from tango import Database
db = Database()
list(db.get_device_exported("*"))

The Vulcan group have made several helper functions to get some of the device proxies for you. For example, to get the Signal Processing System (SPS) device proxies you can use:

from aiv_utils.low_utils import get_sps_devices
STATION_NAME = "itf1"
station, subracks, tpms, (daq, _) = get_sps_devices(STATION_NAME)

This an other helper functions are documented in the API section.

Each tango device consists of sets of Attributes and Commands that can be read and executed respectively.

Attributes

Attributes are typed variables representing features of the device (e.g. frequency, temperature). To see a list of attributes for a given device type: device.get_attribute_list()

Attribute Operations
Operation	Example Code	Explanation
List available device attributes	`awg.get_attribute_list()`	This will work for all devices
Read Attribute	`awg.frequency`	Use dot notation: device.attribute
Write Attribute	`awg.frequency = 250e6`	Assignment of new value
Put in Admin Mode	`awg.adminMode = 1`	When a device is in adminMode it is not communicating with the hardware
Take Device out of admin mode	`awg.adminMode = 0`	Allows communication

Commands

Commands are operations a device can perform.

Commands
Operation	Example Code	Explanation
List available commands	`awg.get_command_list()`	This will work for all devices
Execute a command	`awg.Play()`
Execute a command with arguments	`car.Turn(90)`

Depending on the Tango Device, commands can be synchronous (blocking), where the user must wait for the command to finish or asynchronous (long running commands) which return immediately. The execution status of a long running command can be checked by issuing: device.longrunningcommandstatus

Tango Events

Tango devices emit events when attributes are changes or commands are run. Clients can subscribe to these events and add custom event handlers.

For example, a subscription looks like:

subscription_id = deviceproxy.subscribe_event(attribute, tango.EventType.CHANGE_EVENT, event_handler, [])

Where event_handler is any python function

To unsubscribe you can then run:

deviceproxy.unsubscribe_event(subscription_id)

Using and Monitoring the System

JupyterHub

JupyterHub is a web-based interface that allows you to run notebooks and terminal commands on the cluster. In the following sections we will describe the directories available to you and the basics of how to use JupyterHub.

Directories

When you first open JupyterHub, you will see a list of directories in your home directory on the cluster (/home/jovyan/). Some are mounted volumes that are shared with all users, such as daq-data, eep-data, cnic-data and shared.

The shared directory is where you can put data that you want to share with other users. It is recommended that you make a directory with your username in the shared directory and put your data there. The daq-data directory is where the data from the daq is stored. The eep-data directory is where the EEP simulated data files are stored. The cnic-data directory is where the Customisable Network Interface Card (CNIC) simulated data files are stored.

All other directories currently in /home/jovyan/ or that you create are only accessible by you. These means that any changes you make to the operational notebooks or the aiv_utils package will not affect other users. If you have made a change that will benefit others, see the Contributing to the SKA-Low Tests Repository section for an explanation of how to contribute.

The Launcher

If you press the blue + button in the top left of the JupyterHub interface, you will see a list of options for creating new files or terminal. If you open a terminal, you can run shell commands as normal which is useful for running git commands to ensure you are using the most up-to-date version of the operational notebooks and aiv_utils package. You can also open a new notebook or python console from the launcher. Make sure you select the Python 3 (ipykernal) option so that you have access to the aiv_utils package.

Jupyter Notebooks

Jupyter Notebooks are a way to run python code interactively. They are made up of cells which can be run individually. The default command to run a cell is Shift + Enter which you can press for each cell to run the whole notebook or click ⏩ in the toolbar to restart the kernel and run all cells.

If you want to keep the output of the cells, the easiest way is to copy the notebook and give it a new filename before rerunning it. If you want to clear the outputs (before git committing changes for example) you can click “Edit” then “Clear All Outputs”.

Grafana

Grafana is an open-source visualisation platform for the creation of near realtime charts and displays. Data appearing in SKAO Grafana is extracted from the Engineering Data Archive (EDA). Only device attributes archived in the EDA can be viewed in Grafana. To view the dashboards you must be connected to the VPN.

Low ITF

Low ITF Dashboards provides links to all dashboards.
System Under Test (SUT) Dashboard high level overview of the state of TMC, CSP and CBF devices.
CBF Connector Display Correlator Beamformer (CBF) connector status.
SPS Dashboard Signal Processing System (SPS) devices such as TPMs, subracks and the bandpass’s of the antennas.
PASD Devices the FNDH and Smartbox temperatures and currents.
Test Equipment Devices the noise source relay status.

SKA-Low

Make sure you use the filters so you view the station that you are interested in.

SPS Signal Processing System (SPS) devices such as TPMs, subracks and the bandpass’s of the antennas.
PASD the FNDH and Smartbox temperatures and currents.
Tango Devices Status of the MCCS devices.

Kibana

Kibana is a web interface primarily for searching and visualizing logs. It is useful for investigating the logs of the various Tango pods when you encounter an issue. Currently, only the ITF cluster has Kibana enabled.

To access Kibana, you must be connected to the VPN, then navigate to the following links: - ITF Kibana - SKA-Low Kibana

The stream will be overwhelming at first because it will display all logs from the cluster (not just pod logs). You can add several filters to the be more specific about the logs you want to see.

You often will want to see the logs for pod of the Tango DeviceProxy that gave you an error. You can use the device name as a filter to only see logs from that pod. You can get the device name with the .dev_name() command:

from aiv_utils.low_utils import get_sps_devices
STATION_NAME = "itf1"
%env TANGO_HOST=tango-databaseds.sut-mccs:10000
station, subracks, tpms, (daq, _) = get_sps_devices(STATION_NAME)

station.dev_name()

which outputs

'low-mccs/spsstation/itf1'

You can add a filter to Kibana to only see logs from this pod by adding the following filter in the filter bar: ska_tags_field.tango-device : low-mccs/spsstation/itf1

You may also need to change the time range to see logs from the time you encountered the error. You can do this by clicking the calendar icon in the top right and selecting the time range you are interested in.

You can use more than one filter at a time by separating them with an ” and ” in the filter bar. The following filters may be useful:

ska_severity : "ERROR" Only look for error messages (filter our warnings and info messages)
ska.application : kubernetes Only look for logs from the kubernetes pods (ignore cluster logs)
kubernetes.namespace : sut-mccs Only look for pods of a certain namespace if you’re only interested in MCCS devices for example
message : *Failed to connect* Find logs that contain the phrase “Failed to connect” or any other phrase you are interested in
kubernetes.pod.name : spsstations-spsstation-itf1-0 Find logs from a specific pod
ska_tags_field.tango-device : low-mccs/spsstation/itf1 As mentioned above, find logs for a Tango device’s pod

k9s

If you have the correct permissions (see k9s/kubectl Interface), you can use the graphical tool k9s to investigate and manage a cluster. Make sure you install k9s (and for full functionality also install kubectl) Then once you have logged in with infra login and connected to the VPN you should be able to use k9s to connect to the clusters.

Once in the k9s interface, make sure you are in the correct cluster (Clusters) which can be chosen using the :context command. You may then also have to press 0 to see all the namespaces in the cluster. You may then want to use / to filter the pod names to find what you are interested in. For example I could type /daq then click enter to see all the daq pods or /s8-6 to see all the pods for the s8-6 station.

Once you have found the pod you are interested in, you can click l to see the logs, d to describe the pod or Ctrl + k to kill the pod (don’t do this lightly!). Investigating the pods in this way can help you to understand what is happening in the cluster and debug any issues.

kubectl

If you want a more advanced command line tool, you can use kubectl to view and modify the cluster. Some common commands are shown below.

kubectl commands
Action	Example Command Line
List contexts (clusters)	`kubectl config get-contexts`
Use a context (cluster)	`kubectl config use-context infra:au-itf-k8s-master01-k8s`
List namespaces in a cluster	`kubectl get namespaces`
List all pods in namespace	`kubectl -n sut get pods`
Show logs of a given pod in a namespace	`kubectl -n sut logs centralnode-01-0`

SKA-Low Tests Repository

When the JupyterHub server is launched, the SKA-Low Tests repository is cloned and fetched in the directory /home/jovyan/ska-low-tests. The most important two directories in this repository are:

ska-low-tests/notebooks/operations: This directory contains the notebooks that are used for common operations such as initialisation of stations and acquiring data.
ska-low-tests/src/aiv_utils: This directory contains the helper functions that are used in a lot of the notebooks.

Operation Notebooks

The operation notebooks cover common station operations such as initialisation, data acquisition, and calibration. Each notebook is internally documented and should explain each step they take. We will now go through the purpose of each operation notebook in the ska-low-tests/notebooks/operations to make it clear how they should work together.

FieldNodeOn.ipynb: This notebook will attempt to get a field node into the state where the FNDH and all smartboxes are initialised and all FEM ports on. This notebook should be run after any maintenance that might required the field node to be turned off or if output data is not as expected (none or too small to look like a bandpass).

Initialise.ipynb: This notebook will bring the station up to the point where TPMs have started acquisition. It is often a good idea to run this notebook before an observing session to make sure that the station is in a known state. If the station is behaving unexpectedly (such as data files not being written) then reinitialising the station is a good first debug step.

Equalise.ipynb: This notebook will attempt to equalise the station by adjusting the preadu levels so that each antenna emits similar amounts of power. This is best done after initialisation and before the first observation.

CalibrateStation.ipynb: This notebook applies a previously calculated calibration solution. If this notebook is not run after initialisation, the station will not be accurately phase calibrated but this is sufficient for observations that do not require high sensitivity (such as observing the sun).

FrequencySweep.ipynb: This notebook acquires correlator data for each frequency channel. Such data are used to calculate calibration solutions.

AcquireBeamformed.ipynb: This notebook acquires beamformed data for a station.

AcquireRaw.ipynb: This notebook acquires a burst of raw voltage data for a station.

AcquireChannelised.ipynb:This notebook acquires a burst of channelised data for a station.

aiv_utils package

In ska-low-tests/src/aiv_utils is a python package full of helper functions to make running these notebooks easier. These functions are documented in the API section.

The most important functions are aiv_utils.low_utils.get_sps_devices() which returns the tango devices for a station and the functions in the metadata module which get metadata such as antenna positions for the station.

This package is interactively installed in your JupyterHub environment so you can make changes to the package and see the effects in the notebooks as soon as your restart the Python 3 (ipykernal) kernel.

Writing your own notebooks

Use Papermill to run operational notebooks

Papermill allows one Jupyter notebook to execute another notebook and supply it with parameters. This reduces replication of code. The Operation Notebooks have been written by Vulcan to be easily run using Papermill.

An example of calling a notebook via papermill is given here:

import papermill

parameters = {"STATION_NAME": "itf-1"}

papermill.execute_notebook("/home/jovyan/ska-low-tests/notebooks/operations/Initialise.ipynb",
                           "Initialise_local.ipynb",
                           cwd=".",
                           stdout_file=sys.stdout,
                           stderr_file=sys.stderr,
                           parameters=parameters)

When evaluated, this cell will create a copy of the notebook /home/jovyan/ska-low-tests/notebooks/operations/Initialise.ipynb and give it a new name Initialise_local.ipynb. It will be supplied a parameters dictionary which will overwrite any default parameters with the same name. The results will be printed to the output cell in the calling notebook. By creating local copies, test evidence can be accumulated as the local copy will contain the output cells when evaluated.

To know what parameters are exposed by a notebook click the “Property Inspector” icon in the right sidebar of an opened notebook and look in the “Common Tools” section. For a cell that contains parameters the “parameters” cell tag will be checked.

Use aiv_libraries from ska-low-tests

Vulcan has created python package called aiv_utils which includes helper functions which will make running notebooks easier.

Example Usage:

from aiv_utils.metadata import get_station_antenna_dataframe
STATION_NAME = "s8-6"
station_df = get_station_antenna_dataframe(STATION_NAME).sort_values(by="eep")

The aiv_utils package will already be in your python path in Jupyterhub. See the API documentation for further details.

Contributing to the SKA-Low Tests Repository

TODO write some stuff here

click Edit then Clear Outputs of All Cells

Common Issues

Bellow are some of the common issues that users may encounter when using the system. If you encounter an issue with one of the operational scripts that is not covered here, please contact the Vulcan team via their slack channel.

The Operation Notebooks Are Not Up to Date

If you encounter a bug in any of the operations notebooks, the first thing you should do is make sure they are up to date with the latest version. To do this, make sure you are on the main branch (git checkout main) and then run git pull. You may have to restore your notebooks to their initial state (git restore file_i_changed.ipynb) before pulling the latest changes. You can also git stash your changes before pulling and then git stash pop to reapply them after pulling.

Telescope is Not On

After maintenance that requires the field node to be turned off or if output data is not as expected (none or too small to look like a bandpass), the field node may need to be turned on. You can think of this as turning on the telescope. To do this, run the FieldNodeOn.ipynb and then the Intialise.ipynb operational notebooks, making sure that STATION_NAME is set to the correct station.

Station DevError CommunicationFailed

This error is usually happens during the initialisation of the station. It will output a message similar to the following:

PyTango.CommunicationFailed: DevFailed[
DevError[
    desc = TRANSIENT CORBA system exception: TRANSIENT_CallTimedout
  origin = DeviceProxy:read_attribute
  reason = API_CorbaException
severity = ERR]

DevError[
    desc = Timeout (3000 mS) exceeded on device low-mccs/tile/itf1-tpm01
  origin = DeviceProxy:read_attribute
  reason = API_DeviceTimedOut
severity = ERR]
]
Stack (most recent call last):
  File "/usr/lib/python3.10/threading.py", line 973, in _bootstrap
    self._bootstrap_inner()
  File "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.10/threading.py", line 953, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib/python3.10/concurrent/futures/thread.py", line 83, in _worker
    work_item.run()
  File "/usr/lib/python3.10/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/local/lib/python3.10/dist-packages/ska_tango_base/executor/executor.py", line 211, in _run
    self._call_task_callback(
  File "/usr/local/lib/python3.10/dist-packages/ska_tango_base/executor/executor.py", line 228, in _call_task_callback
    task_callback(**kwargs)
  File "/usr/local/lib/python3.10/dist-packages/ska_tango_base/base/command_tracker.py", line 142, in update_command_info
    self._exception_callback(command_id, exception)
  File "/usr/local/lib/python3.10/dist-packages/ska_tango_base/base/base_device.py", line 691, in _log_command_exception
    self.logger.error(message, exc_info=exc_info, stack_info=True)

This error is usually caused by the station not being able to communicate with the device. Sometimes the device is too busy and you just need to wait a few moments and try again. If the error persists, it may be due to the Taranta server making too many requests which can be fixed by turning the Taranta server off. If the Taranata servers is on you will be able to see the devices in the Taranata server (links for ITF and SKA-Low). You can then ask one of the Vulcan team members to turn the server off (scale the ska-tango-tangogql-skaffold replicas to 0) for you.

TPM unconnected

This error is often encountered when trying to initialise a station and it fails to do so after repeated attempts. If you check the logs for the TPMs you will see an error similar to the following:

1|2024-09-10T01:12:12.853Z|ERROR|Polling thread|check_communication|tile.py#2457|tango-device:low-mccs/tile/itf2-tpm02|Not able to communicate with CPLD: Cannot call function get_temperature on unconnected TPM
1|2024-09-10T01:12:13.257Z|WARNING|Polling thread|wrapper|tile.py#66|tango-device:low-mccs/tile/itf2-tpm02|Cannot call function get_temperature on unconnected TPM

This is due to the TPM not being able to connect to its hardware. The only currently know solution is to restart the TPM pod which you can ask one of the Vulcan team members to do for you.