maravelias.info

I have followed the pip install approach to add everything that I wanted for Python. Even though I did the same for jupyter when I was starting it (“jupyter notebook”) it would only start the Python 2 kernel, although Python 3 “was there” but not active actually.

After some digging I found that a similar issue existed already (github/jupyter/issue#270 [1]). By doing


$jupyter kernelspec list
Available kernels:
python2 /home/grigoris/.local/share/jupyter/kernels/python2
python3 /home/grigoris/.local/share/jupyter/kernels/python3


we get the locations of the kernels that are loaded. Under each of the above directories there is a kernel.json file. In more detail:

{
"display_name": "Python 2",
"language": "python",
"argv": [
"python",
"-m",
"ipykernel_launcher",
"-f",
"{connection_file}"
]
}

{
"display_name": "Python 3",
"language": "python",
"argv": [
"python",
"-m",
"ipykernel_launcher",
"-f",
"{connection_file}"
]
}


corresponding to Python 2 and Python 3, respectively. The problem lies in the version of python used, as what we see after the ‘”argv”: [‘ line is “python” in both cases. This is ok for Python 2 since this is the default way to invoke it. But not for Python 3, which needs “python3” (by default). So, the first option may be left as it is, but the second needs to change to “python3” to properly point to the correct version. However, this was not working and I found no reason why (especially since “python” and “python3” invoke the proper versions). In any case a robust solution is to provide the full paths, i.e. “/usr/bin/python” and “/usr/bin/python3”.

As I wanted to become even more specific with the versions used I made the following changes:
{
"display_name": "Python 2.7",
"language": "python",
"argv": [
"/usr/bin/python2.7",
"-m",
"ipykernel_launcher",
"-f",
"{connection_file}"
]
}

{
"display_name": "Python 3.7",
"language": "python",
"argv": [
"/usr/bin/python3.7",
"-m",
"ipykernel_launcher",
"-f",
"{connection_file}"
]
}


so now I can use both versions at the same time in jupyter. This is a manual fix to what should have been an automatic process…

[1] For more details see also this link:
https://github.com/jupyter/jupyter/issues/270

It is exciting to have a new machine, such the Lenovo Thinkpad X-1, but the process for setting it up can be a bit tedious. So this is a small guide what worked or didn’t through my small experience in installing Debian linux. [Just for the fun of it I proceeded with the installation of Win which was really easy with all vocal commands! But they wouldn’t live long … ]

BIOS boot

I started by downloading the net/CD image for Debian 9 (‘stretch’) and putting it to a USB key. I plugged the key at the laptop and started the booting process. First problem: even though it could see the USB and it showed the Debian distro it wouldn’t start. I couldn’t figure out at all why up to the point that I noticed that the new hardware has new BIOS technology called UEFI (yes … I hadn’t installed anything for a looong time!). I went to BIOS > Startup > UEFI/Legacy Boot which was ‘UEFI only’ and I changed the CSM Support to ‘Yes’. But this alone was not enough. So, I changed the UEFI/Legacy Boot from ‘UEFI only’ to ‘Both’ [where I think it should be also to put ‘Legacy only’ if you plan to run on Linux only, but if Win are necessary then leave both].

Debian installation

Then the USB key would finally load and the installation could start. With the graphical installation of Debian the screen goes blank after some time of inactivity, which you can correct with a small movement of the mouse. But since I didn’t have any mouse connected I though that something was wrong. It took me a few iterations before I understand that, which I find ridiculous when you install an operating system. I think that the monitor should be on at all times! The installation went on without any other issues. After logging in the new system, I updated all packages.

Trackpoint/Trackpad issue

At the beginning the trackpad (touchpad) could work with some lag. The trackpoint didn’t work at all but I didn’t pay any serious attention to that. I installed the wifi driver (firmware-iwlwifi, for Intel chips) and after that the touchpad wouldn’t work at all. This is a known problem and many solutions are offered (updating to latest kernel 4.18 and to Debian testing didn’t work). However (and after some iterations of the installation process) I finally went to BIOS > Config > Keyboard/Mouse > Trackpoint and I ‘Disabled’ it. That way the touchpad works without any issue. [At this point it is not so urgent for me to fix this, but I will come back in the future].

Suspend mode

There is a know issue that certain BIOS version have serious problems to recover when suspending the laptop. I learned that the hard way: after suspending the laptop when I was trying to put my password over and over again, as every time it would accept only a few characters before it accepted that raised errors of incorrect password. I had to brutally shut it down (pressing the power button) but for whatever reason the x-server would start. So I had only text access and I couldn’t restarted it (of course I may have missed some more appropriate approaches). Anyway, I re-install Debian from scratch. In order to solve that I needed to update the BIOS.

BIOS update with fwupd

There is a new easy way to update firmware and BIOS through Linux by using the fwupd. After the fresh installation I set this as my first target before I install anything else. fwupd was at Debian repository so it was simply to install it. And the instruction are really easy to follow. Only do ‘fwupdmgr refresh’ to get the latest metadata for you firmware. But at this point it was failing to connect to the webpage (error message: “Failed to download https://s3.amazonaws.com/lvfsbucket/downloads/firmware.xml.gz.asc: Not Found”). The issue (Debian bug #912414) was simply a wrong site, which was easy to fix by going to the config file “/etc/fwupd.conf” and replace: “DownloadURI=https://s3.amazonaws.com/lvfsbucket/downloads/firmware.xml.gz” with the new site: “DownloadURI=https://cdn.fwupd.org/downloads/firmware.xml.gz”. That should do the work, right? NO! Because the version of fwupd installed in Debian 9 is 0.7 and according to Richard Hughsie: “LVFS will block old versions of fwupd for some firmware […] The ability to restrict firmware to specific versions of fwupd and the existing firmware version was added to fwupd in version 0.8.0. This functionality was added so that you could prevent the firmware being deployed if the upgrade was going to fail, either because: i. The old version of fwupd did not support the new hardware quirks, ii. If the upgraded-from firmware had broken upgrade functionality. Then, let’s upgrade fwupd by upgrading (at the same time it would be nice to have a fully fresh new system) and I switched the repositories to Debian 10 (testing, named ‘buster’), upgrading also the Linux kernel from 4.9 to 4.18. And indeed the fwupd now was working! Almost…

BIOS update with USB key

I run fwupd to get the latest BIOS version but it refused. The installed version was 1.25 while only those >1.27 are upgradable with fwupd. Now the only solution is the classic one, i.e. download the last Thinkpad BIOS update (bootable, for Windows) and install it directly from a USB. After having downloaded the .iso image I followed the instructions by Vivek Gite (with most important part the video showing what to do during the actual BIOS installation).

[In brief, using the El Torito boot image extractor (debian package name: genisoimage) do the following:
geteltorito -o bios.img n23ur13w.iso
sudo dd if=bios.img of=/dev/sdb1 bs=1M
Take care that you know exactly which device is the USB key you are going to write to, else you are going to delete data!
Then reboot and enter BIOS and boot from the USB. Select option 3 to verify that you have the right model (in case you press something and you cannot cancel just retype the model number) and then press 2 to start the actual BIOS update. This will run some things now and then when you reboot it will continue before it shows the booting screen.]

Crossing fingers and … booting again! phew… everything is working and indeed I have now the latest version (1.34).

As a (much) later version of the BIOS it should have fixed the issue with suspending mode. Did it work? So far I have experienced any critical issues. What I did noticed though is that that I could feel it a bit hot when suspending. I was going through the BIOS setting and I discovered that there is an option now with respect to the sleep state (I don’t know if it was present in the previous version): Config > Power > Sleep State, which has two options ‘Windows 10’ and ‘Linux’. I obviously picked the last one to optimize the performance and I think it is working now.

Python pip

Finally, the time that I would actually install what I need to start working has arrived! Since Python2.x and 3.x exist I was going to use pip only to install all necessary packages. I installed pip for both Python versions from the distro repositories. Checking the pip site I saw the following recommendation: “Ensure pip, setuptools, and wheel are up to date, by doing: python -m pip install –upgrade pip setuptools wheel”.

Well, first of all … DON’T DO IT! I missed the obvious warning sign above:

Warning
Be cautious if you’re using a Python install that’s managed by your operating system or another package manager. get-pip.py does not coordinate with those tools, and may leave your system in an inconsistent state. You can use python get-pip.py –prefix=/usr/local/ to install in /usr/local which is designed for locally-installed software.

By default the pip version provided by the Debian repositories is 9.x. When I did this upgrade I got the latest version of 18.x and I went on to install numpy, scipy, matplotlib. But when I started python to try out the installation nothing was working! Even worse:

pip install astropy
Traceback (most recent call last):
File "/usr/bin/pip3", line 9, in from pip import main
ImportError: cannot import name 'main'

What is going on now? Apparently is it not a good idea to upgrade with pip a system installation of pip, as then pip gets confused of which version to use exactly and, even worse, some scripts managed by apt may break (see e.g. the discussion in github issues #5447 and #5221). So the two important advice here are:
1. upgrade/install under user (with –user)
2. avoid running pip with sudo (which may affect root accessed files)

I found various ways to fix this over the internet, but I opted for a more … brutal way. I noticed that when installing Python packages from the repositories of the distribution everything is installed under the /usr/lib/python*.*/dist-packages/ while pip puts everything under /home/user/.local/lib/site-packages/. So, the idea was to remove (manually delete) everything under these directories. Then I could re-install pip and the Python packages and it should be clean from all errors. I did that but I continued to get errors, but probably that was because I didn’t re-source the terminal (so paths to the previous packages and scripts were still active). After a reboot I found everything working fine. [One note: after a fresh installation everything is new, such as the content of $PATH. When I installed jupyter and tried to run it I got an error “command not found”, obviously because I forgot to update the $PATH to include the ~/.local/bin/ directory, where binaries from pip are stored.]

Other resources

The above is a personal (certainly not the wisest) experience to install Debian 9/10 (stable/testing) to Thinkpad X-1. There are also more thorough and knowledgeable guides out there (which I have to check also for other issues), such as:
–> Installing Debian On Thinkpad X1 Carbon 5th Gen (previous generation)
–> Lenovo ThinkPad X1 Carbon (Gen 6) by Arch linux

Between 12 to 19 of September 2016, we organized the Ondřejov Summer School 2016, which was held at the Stellar Stellar Department of the Astronomical Institute (Czech Academy of Sciences), in Ondřejov. The main aim of the school was to provide students hands-on experience with real observational data, i.e. to understand observations, reduce them, and discuss their results. This project-oriented activity was supplemented with lectures on various techniques and one of them was a workshop on Python and IRAF/PyRAF.

As a workshop the main focus was given to show examples that the students could repeat at the own computers. Its structure was based on a very small introductory talk and then examples of how to use Python and IRAF in real problems. With this post I make public all of this material both as a reference and as a (hopeful) help to others.

Material:

Among the simplest installation methods to install IRAF are SciSoft (for RedHat/Fedora based distributions) and Fernando Avila Castro iso files (for Ubuntu/Debian based distributions). However, both require root privileges. What can you do when you do not have these permissions?

Ureka [1] comes in rescue! It is easy to install (download and run the installer![2]) and it doesn’t need any root privileges, so you can install it under your home directory.

Moreover, it offers, along with a simple installation of IRAF, a full installation of Python, with many pre-installed modules useful to perform almost any astronomical reduction (e.g. PyRAF, Astropy, matplotlib, pandas, [more]).

By default Ureka (1.4.1) installs with a Python version (2.7.3), and it modifies your shell starting scripts (bashrc,tcshrc) to create the aliases to use. E.g. the added lines in a .bashrc file are:

ur_setup() {
    eval `/var/home/maravelias/.ureka/ur_setup -sh $*`
}
ur_forget() {
    eval `/var/home/maravelias/.ureka/ur_forget -sh $*`
}

and you can invoke Ureka by simple typing: ur_setup. Now, this means that in every shell that you start you will have to type over and over again the same command, which becomes at least frustrating. In order to avoid this, you can simply comment the alias line:

#ur_setup() {
eval `/var/home/maravelias/.ureka/ur_setup -sh $*`
#}
ur_forget() {
    eval `/var/home/maravelias/.ureka/ur_forget -sh $*`
}

so Ureka starts with every shell now. This option allows Python to install any extra modules (through PyPI – pip install) to the Ureka path and not system’s Python path.

Ureka allows for multiple installations of difference versions or different variants so there is reasoning in using the alias – but still for many users it may not be necessary (in any case you just uncomment the lines and you get back to the normal use).

So, Ureka offers an easy and clean way to install IRAF and a powerfull Python installation at the same time, without any root permissions. The only drawback is that it needs a lot of space (~4-5 GB), but it does not seem dramatic nowadays. [As far as the sys-admins it is easier to provide you with more extra space rather than software installations/updates within their systems.]

References:
[1] Ureka: http://ssb.stsci.edu/ureka/
[2] Documentation 1.4: http://ssb.stsci.edu/ureka/1.4.1/docs/index.html

How to make a dictionary of dictionaries with lists inside?
For example we have a number of stellar spectral types (spec) and for each one of them we have a series of spectral lines (line) and their equivalent widths (ew). By creating a rather complex dictionary we can then call the list of values for the equivalent width for any spectral type and line we want.

Let’s look at this step by step.

1. A simple dictionary, with spetral types (spec) as keys and corresponding spectral lines as values:
dic = {}
...
dic[spec] = line

Output (which includes all spectral types but only the last spectral line parsed, because it just overrides the value for each key):
{'B1': 'HeI/4026', 'B0.5': 'HeI/4026'}

2. In order to get all spectral lines we have to make a list of values for each key. For this we have to import defaultdict first:

from collections import defaultdict
dic = defaultdict(list)
...
dic[spec].append(line)


Output:
defaultdict(, {'B1': ['CaIIK/3928', 'HeI/4009', 'HeI/4026'], 'B0.5': ['CaIIK/3928', 'HeI/4009', 'HeI/4026']})

3. Now, we want to add the value of ew for each line.

from collections import defaultdict
dic = defaultdict(defaultdict)
...
dic[spec][line] = ew


Output:
defaultdict(, {'B1': defaultdict(None, {'HeI/4009': 0.38, 'CaIIK/3928': 0.29, 'HeI/4026': 1.03}), 'B0.5': defaultdict(None, {'HeI/4009': 0.33, 'CaIIK/3928': 0.27, 'HeI/4026': 1.09})})

4. If we want to have not only a value but a list of values (per spectral line) we need to define a function within the defaultdict. If we just put defaultdict(defaultdict(list)) it will not work as the defaultdict(list) will return something that is not expected as input by the next level defaultdict. To overcome this we use a “lambda” (anonymous) function:

from collections import defaultdict
dic = defaultdict(lambda: defaultdict(list))
...
dic[spec][line].append(ew)


Output:
defaultdict( at 0x2508578>, {'B1': defaultdict(, {'HeI/4009': [0.38], 'CaIIK/3928': [0.29], 'HeI/4026': [1.03]}), 'B0.5': defaultdict(, {'HeI/4009': [0.33, 0.63], 'CaIIK/3928': [0.27, 0.13], 'HeI/4026': [1.09, 1.96]})})

We can easily call any spectral line per type by simply calling dic[specific_spec][specific_line], e.g. dic[‘B0.5’][‘CaIIK/3928’] will give us [0.27, 0.13].

Although the are many information on how to use the genfromtxt (from numpy) there is not a clear demonstration (that I found…) to show how to easily read a text file with columns of data (including strings – it is straightforward to read columns of numbers with loadtxt for example).

Suppose that we have the following text file (named ‘input.txt’):

#object	q1	q2
id-001	120.	2212.
id-002	145.	1222.
id-222	123.	1142.

Then we can just use:

g = genfromtxt('input.txt',dtype=None,names=True)

This will read all the file. dtype=None means that the command will decide what type of data is each element and name=True means that we can use the names from the first raw (header) to call each column, i.e.:

print g['object']
print g['q2']

will print:

['id-001' 'id-002' 'id-222']
[ 2212. 1222. 1142.]

That easy, that fast !

The easiest way to install any Python package is through PyPI. So, matplotlib is not an exception and we installed it on a CentOS (v6.4) machine without any errors (of course after updating numpy).

But when we tried to plot something we couldn’t see anything. This is actually a backend issue, and when we installed matplotlib there was no support for any backend (except the default agg which is supplied with matplotlib).

To solve this, we first installed the pygtk-dev version and then re-istalled matplotlib, through

pip install matplotlib

which was build now with the GTKAgg as default backend.

Happy plotting !

THELI is package for the automated reduction of astronomical imaging data. One of the prerequisite is Python 2.5 or higher (as THELI uses numerous shell and Python scripts). As I had already installed Scisoft 7.5 (with Python 2.5) I though that everything was ok.

I started using THELI and indeed no problem up to the point of Astrometry+Photometry tab. But there was an error kept coming up about a syntax error in the scampcat.py:


except IOError as (strerror):
  print "I/O error: {0}".format(strerror)
  result = False


While looking around why is this an error I found out that the syntax “except … as …” was a valid syntax from Python 2.6 and higher. So the first attempt was to correct for the syntax. By removing “as”:


except IOError, err:
  print "I/O error: {0}",format(err.errno, err.strerrorerr)
  result = False


the script was able to continue, although now the problem was that it couldn’t import the “multiprocessing” module! Indeed the multiprocessing module becomes standard after Python 2.6 and it was not included in Python 2.5 version of Scisoft.

So, what now? Let’s go on and install the module inside Scisoft’s Python (2.5). Probably by downloading the module and running the setup.py or by easy_install or pypi this could work easily, right? Somehow, none of them worked in an easy and clean way (either something missing or not working). Moreover, as I was looking around filesystem’s Python (2.6) I noticed that the multiprocessing module was not installed under the site-packages but under Python directory itself. This puzzled me a little bit more so I went on to see if there is any other approach.

The one I though was to check for the new version of Scisoft (7.7), which included Python 2.7. This version provided that the syntax will work and the module will be there. So I went on to upgrade to the new release and after the successful installation THELI did not complain about these problems any more!

But there was a something that it just came to my mind after finishing the THELI test run. Why hadn’t I change the $PATH (of bash) to include filesystem’s Python 2.6 (under /usr/bin/)? This would, probably, accept the syntax and included the multiprocessing module already! Then I would have avoided the whole fuzz about removing and installing programs and dependencies.

So, it’s better to think a little bit more and try different approaches (ask someone?)! At least the new installation is not in vain as there are some good upgrades (like Matplotlib 1.1) which will be helpful in the future.

While manipulating some data I came across to, what looked, a strange behavior with the errorbar of matplotlib, when plotting the results from linear to logarithmic scale.

Suppose these data:

import matplotlib.pyplot as plt
import numpy as np

s=[19.0, 20.0, 21.0, 22.0, 24.0]
v=[36.5, 66.814250000000001, 130.17750000000001, 498.57466666666664, 19.41]
verr=[0.28999999999999998, 80.075044597909169, 71.322124839818571, 650.11015891565125, 0.02]
plt.errorbar(s,v,yerr=verr)
plt.show()


and the result looked like this:

where the errorbars are obviously visible, but when I switched to the logarithmic scale, by using:

plt.yscale('log')
plt.show()

then the result was this:

where some errorbars are not visible (although their caps are still visible). So, what is going on?

I couldn’t figure out what was the problem, so I asked at StackOverflow. I got an answer by “Dan”, in which he pointed to the fact that some of the errors were (that large) that led to negative values (not valid for log). That’s why they were not displayed. The solution was to use asymmetric errors, by substituting basically the negative error values with a value close to 0 (actually the plotted value, but slightly over 0).
The piece of code that corrected the display is:

import matplotlib.pyplot as plt
import numpy as np

s=[19.0, 20.0, 21.0, 22.0, 24.0]
v=np.array([36.5, 66.814250000000001, 130.17750000000001, 498.57466666666664, 19.41])
verr=np.array([0.28999999999999998, 80.075044597909169, 71.322124839818571, 650.11015891565125, 0.02])
verr2 = np.array(verr)
verr2[np.where(v-verr<=.0)] = v[np.where(v-verr<=.0)]*.999999 plt.errorbar(s,v,yerr=[verr2,verr]) plt.ylim(1E1,1E4) plt.yscale('log') plt.show()
with the final output (by adjusting also the axes):

Perhaps looking for this (after importing os):

os.environ["VARIABLE"]="/path/to/program"


This is needed for example to set the jref path in order for multidrizzle package to run through a script.