============ Introduction ============ The following is documentation for users of the SKA-Low system. This documentation is intended to help users understand the system and how to interact with it. .. include:: access.inc .. _useful-links: =================== Useful Link Summary =================== Many of the following links are mentioned and explained in other sections, but for your convenience, here is a summary of useful links: - SKA-Low - `JupyterHub `__ - `Calendar `__ - `Grafana `__ - `Taranta `__ - `Kibana `__ - ITF - `JupyterHub `__ - `Calendar `__ - `Grafana `__ - `Taranta SUT `__ - `Taranta SUT MCCS `__ - `Kibana `__ - Repositories - `SKA-Low Tests `__ - `SKA-Low MCCS `__ - `SKA-Low MCCS SPSHW `__ - `SKA-Low MCCS Calibration `__ - Documentation - `SKA term glossary `__ - `SKA-Low MCCS `__ - `SKA-Low MCCS SPSHW `__ - `SKA-Low MCCS Calibration `__ =================== High Level Overview =================== .. include:: telescope-components.inc .. include:: kubernetes.inc .. include:: tango.inc =============================== Using and Monitoring the System =============================== .. include:: jupyterhub.inc .. include:: grafana.inc .. include:: kibana.inc .. _k9s: k9s --- If you have the correct permissions (see :ref:`k9s-access`), you can use the graphical tool `k9s `__ to investigate and manage a cluster. Make sure you `install k9s `_ (and for full functionality also `install kubectl `_) Then once you have logged in with ``infra login`` and connected to the VPN you should be able to use ``k9s`` to connect to the clusters. Once in the k9s interface, make sure you are in the correct cluster (:ref:`cluster-context`) which can be chosen using the ``:context`` command. You may then also have to press ``0`` to see all the namespaces in the cluster. You may then want to use ``/`` to filter the pod names to find what you are interested in. For example I could type ``/daq`` then click enter to see all the daq pods or ``/s8-6`` to see all the pods for the s8-6 station. Once you have found the pod you are interested in, you can click ``l`` to see the logs, ``d`` to describe the pod or ``Ctrl + k`` to kill the pod (don't do this lightly!). Investigating the pods in this way can help you to understand what is happening in the cluster and debug any issues. .. _kubectl: kubectl ------- If you want a more advanced command line tool, you can use `kubectl `__ to view and modify the cluster. Some common commands are shown below. .. csv-table:: kubectl commands :header: "Action", "Example Command Line" "List contexts (clusters)", "``kubectl config get-contexts``" "Use a context (cluster)", "``kubectl config use-context infra:au-itf-k8s-master01-k8s``" "List namespaces in a cluster", "``kubectl get namespaces``" "List all pods in namespace", "``kubectl -n sut get pods``" "Show logs of a given pod in a namespace", "``kubectl -n sut logs centralnode-01-0``" .. include:: ska_low_tests_repo.inc ========================== Writing your own notebooks ========================== .. include:: writing_notebooks.inc ============= Common Issues ============= Bellow are some of the common issues that users may encounter when using the system. If you encounter an issue with one of the operational scripts that is not covered here, please contact the Vulcan team via their slack channel. The Operation Notebooks Are Not Up to Date ------------------------------------------ If you encounter a bug in any of the operations notebooks, the first thing you should do is make sure they are up to date with the latest version. To do this, make sure you are on the main branch (``git checkout main``) and then run ``git pull``. You may have to restore your notebooks to their initial state (``git restore file_i_changed.ipynb``) before pulling the latest changes. You can also ``git stash`` your changes before pulling and then ``git stash pop`` to reapply them after pulling. Telescope is Not On ------------------- After maintenance that requires the field node to be turned off or if output data is not as expected (none or too small to look like a bandpass), the field node may need to be turned on. You can think of this as turning on the telescope. To do this, run the ``FieldNodeOn.ipynb`` and then the ``Intialise.ipynb`` operational notebooks, making sure that ``STATION_NAME`` is set to the correct station. Station DevError CommunicationFailed ------------------------------------- This error is usually happens during the initialisation of the station. It will output a message similar to the following: .. code-block:: python PyTango.CommunicationFailed: DevFailed[ DevError[ desc = TRANSIENT CORBA system exception: TRANSIENT_CallTimedout origin = DeviceProxy:read_attribute reason = API_CorbaException severity = ERR] DevError[ desc = Timeout (3000 mS) exceeded on device low-mccs/tile/itf1-tpm01 origin = DeviceProxy:read_attribute reason = API_DeviceTimedOut severity = ERR] ] Stack (most recent call last): File "/usr/lib/python3.10/threading.py", line 973, in _bootstrap self._bootstrap_inner() File "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner self.run() File "/usr/lib/python3.10/threading.py", line 953, in run self._target(*self._args, **self._kwargs) File "/usr/lib/python3.10/concurrent/futures/thread.py", line 83, in _worker work_item.run() File "/usr/lib/python3.10/concurrent/futures/thread.py", line 58, in run result = self.fn(*self.args, **self.kwargs) File "/usr/local/lib/python3.10/dist-packages/ska_tango_base/executor/executor.py", line 211, in _run self._call_task_callback( File "/usr/local/lib/python3.10/dist-packages/ska_tango_base/executor/executor.py", line 228, in _call_task_callback task_callback(**kwargs) File "/usr/local/lib/python3.10/dist-packages/ska_tango_base/base/command_tracker.py", line 142, in update_command_info self._exception_callback(command_id, exception) File "/usr/local/lib/python3.10/dist-packages/ska_tango_base/base/base_device.py", line 691, in _log_command_exception self.logger.error(message, exc_info=exc_info, stack_info=True) This error is usually caused by the station not being able to communicate with the device. Sometimes the device is too busy and you just need to wait a few moments and try again. If the error persists, it may be due to the Taranta server making too many requests which can be fixed by turning the Taranta server off. If the Taranata servers is on you will be able to see the devices in the Taranata server (links for `ITF `_ and `SKA-Low `_). You can then ask one of the Vulcan team members to turn the server off (scale the ``ska-tango-tangogql-skaffold`` replicas to 0) for you. TPM unconnected --------------- This error is often encountered when trying to initialise a station and it fails to do so after repeated attempts. If you check the logs for the TPMs you will see an error similar to the following: .. code-block:: output 1|2024-09-10T01:12:12.853Z|ERROR|Polling thread|check_communication|tile.py#2457|tango-device:low-mccs/tile/itf2-tpm02|Not able to communicate with CPLD: Cannot call function get_temperature on unconnected TPM 1|2024-09-10T01:12:13.257Z|WARNING|Polling thread|wrapper|tile.py#66|tango-device:low-mccs/tile/itf2-tpm02|Cannot call function get_temperature on unconnected TPM This is due to the TPM not being able to connect to its hardware. The only currently know solution is to restart the TPM pod which you can ask one of the Vulcan team members to do for you.