maravelias.info

To properly process data (either imaging or multi-slit spectra) from the OSIRIS instrument at GTC you need first to convert the original data to single images. The raw fits data contain two extensions corresponding to the two CCD sensors. To combine these into a single 2110 X 2051 pixels image (including the gap between the sensors) they provide with a convenient IRAF script called Mosaic Tool. This takes into account more than simply adding the extensions. As written in the ReadmeFirst.txt file:

(i) corrects the input OSIRIS pre-image for overscan to provide a uniform background, (ii) executes the mosaic assembly according to the rotation/traslation parameters obtained from the MD Polynomial Manual (current version: V5; see contents of the attached MD_polynomials.pdf file) on a new created image, (iii) copies the zero extension of the input image in the new one, and (iv) updates the WCS keywords. Such updating consists in switch the input reference pixel (CRPIX1, CRPIX2) to the center of the mosaic (see MD_polynomials.pdf, p.12) and calculate the celestial coordinates (CRVAL1, CRVAL2) corresponding to the new reference, based on the original WCS parameters. Hence, the correction of the astrometric solution is the same of the input image.

So it does quite a work! To run it you have to start IRAF, define the task and apply it to an raw image:

-->task $mosaic_2x2=mosaic_2x2_v2.cl
-->mosaic_2x2 0002604740-20200628-OSIRIS-OsirisBroadBandImage.fits

(the name for the task, in this case mosaic_2x2 is not really important). When I tried that (using PyRAF v2.1.5 with IRAF 2.16.1 on a Debian 10 machine) I got an error:

Traceback (innermost last):
File "", line 1, in
File "", line 16, in mosaic
iraf.noao.PY()
AttributeError: Parameter PY not found

which I couldn’t understand. Even after contacting the support the mystery remained. Back in early August we had only a couple of images from our pre-imaging run (and not time to waste…) and the support team was kind enough to do it for us. But now that we have the actual spectroscopic data the images are way too more to ask again for this. Therefore, I tried to search a bit deeper than the first time.

I started by checking the mosaic_2x2_v2.cl file, and I removed all tasks to search where could be the problem. So I edited it to run this:

 
procedure mosaic (fobjeto)

file fobjeto {"",prompt="Object file:"}


begin

string objeto
file   text

objeto=mosaic.fobjeto

noao.
noao.artdata.
noao.imred
noao.imred.bias.
images.imgeom
ctio

#stsdas
#toolbox
#imgtools

end

In this form it only calls the basic packages of IRAF. Still though, when I attempted to run it again I took the same error! So the problem was already at the beginning. Then, I noticed these dots after the packages (e.g. noao, noao.artdata.) where obviously python was trying to get a non-existing attribute. After removing these dots it actually proceeded! Only after this revelation, it came to my mind to actually try the IRAF cl directly (when you have the python supported version there is no need to look back to cl!). However, it made a difference as (obviously) the cl doesn’t care about the cots… So the problem was python specific.

The script continued running only to … stuck again:


Traceback (innermost last):
File "", line 1, in
File "", line 44, in mosaic
iraf.imcreate(image = 'CCD_mosaic.fits', naxis = 2, naxis1 = 2110,naxis2 = 2051,pixtype = 'real')
AttributeError: Undefined IRAF task `imcreate'

However, this is easily interpreted. imcreate is a task in the ctio package, which is external (not part of the original core of IRAF). So you need to find the ctio package and put it under the iraf/extern/. This was actually not easy to spot since the support of IRAF by the STScI has dropped and most links are broken. The package is supported by the iraf-community, but I didn’t try to check exactly how to download (as I didn’t find it under my iraf/ directory). On the other hand, I found an alternative (in the iraf.net/forum) using the mkpattern task, under the artdata package which is already loaded in the file. Then, I just replaced the following line:

imcreate(image="CCD_mosaic.fits",naxis=2,naxis1=2110,naxis2=2051,pixtype="real")

with

mkpattern(input="CCD_mosaic.fits",pixtype="real", ndim=2, ncols=2110, nlines=2051)

and all problems are finally solved! The script runs to the end and provides the dinal mosaic image.

Bottomline, or what to do in three single steps

• If you are running cl/ecl directly AND have ctio package installed you don’t need to do anything
• If you are running cl/ecl directly and you don’t have ctio, just replace imcreate with mkpattern
• If you are running on python then remove the dots when the IRAF packages are called

I would therefore strongly suggest the support team to remove the dots and replace the imcreate task so that it can work in all cases.
(For any potential user here is my modified version: mosaic_2x2_v2_gm.cl.txt – remove .txt)

IRAF has become lately available through Debian‘s repositories. So I was curious to see how it works. But I was missing the right moment. About a month ago, I finally found the proper motivation so I went on to install it. As always … there were a few, minor, problems. Thanks to John K. we were able to resolve them! What follows is a small guide of what I faced during the installation (on a Debian 10/Buster).

Instead of previous (relatively) easy solutions to install IRAF now you can just do:

sudo apt-get install iraf

And the magic happens! It is typical to “mkiraf” then to build the necessary login.cl file. And that’s the very first problem as there is no such a thing (try mkiraf and you will get nothing).

Fortunately, mkiraf is not that necessary as the login.cl is just an ascii file with some default parameters set. We can easily copy such a file from another machine and we can put it here (under a proper directory, typically I create one under home, such as /home/user/iraf/). In all cases, what we need to change is the first few lines:
set home = "/home/user/iraf/"
set imdir = "home$images"
set cache = "home$cache"
set uparm = "home$uparm/"
set userid = "user"

where you replace the user with your username (or you put the appropriate path in the first line, and your username in the last one).

Another key feature is the shell used (typically xterm, xgterm, …). The original login.cl file will have already something (as the shell type is asked when IRAF is set with mkiraf), similar to:
# Set the terminal type. We assume the user has defined this correctly
# when issuing the MKIRAF and no longer key off the unix TERM to set a
# default.
if (access (".hushiraf") == no)
print "setting terminal type to 'xterm' ..."
stty xterm

Either you do not need to do anything (if you have xterm installed already) or you place the shell you are using.
Do not edit anything else from the rest of the file as these are the default IRAF parameters.

[NOTE: copy a fresh login.cl after running mkiraf, else you may end up with a version of a file with modified parameters set by the owner – although you should be always careful to ‘unlearn’ the commands and check the parameters used!)

Then you should be ready to use IRAF! Typing the usual “cl” or “ecl” unfortunately doesn’t work¹ (and it is a bit frightening at the beginning). But with this installation you just do “irafcl” and the normal ecl terminal starts immediately (nice!).

However, ecl is not that convenient, so we urgently needPyRAF. This can be easily installed through pip:

pip install pyraf
pip3 install pyraf

(for Python 2 and 3, respectively). Now even though pip instals all files under separate dirs it builds only one binary under ~/.local/bin/pyraf. So if you installed PyRAF for python2 first, you will see it running properly when using python2. If you add also pip3 version then you will notice that only PyRAF for python3 is available. Frustrating? You bet!

After some exploration we figured out that PyRAF creates only one binary so it overwrites the previous one. Although we tried to change the name and see if it can run individually this didn’t work (for whatever reason … probably bound to something else?). So there is no way (I wonder?) that you can have PyRAF for both Python versions. [Before asking why to keep both versions, keep in mind that somebody may want to use scripts from older versions! ]

The (not at all convenient) solution found so far is to:

1. install first PyRAF for Python v2
2. rename the pyraf binary (~/.local/bin/pyraf) to something else (e.g. pyraf-v2, this is only a back up as it doesn’t work even if you call it explicitly)
3. install PyRAF for Python v3
4. keep the pyraf binary as it is (for use with Python 3)
5. change the names accordingly if you want to use the older version

[There is another package python3-pyraf that I didn’t test², as I tend to keep as clean as possible Python’s installations by using pip only.]

However, even after that PyRAF may still not run properly. For example:
user@mymachine:~$ pyraf

Your “iraf” and “IRAFARCH” environment variables are not defined and could not
be determined from /usr/local/bin/cl. These are needed to find IRAF tasks.
Before starting pyraf, define them by doing (for example):

setenv iraf /iraf/iraf/
setenv IRAFARCH linux

at the Unix command line. Actual values will depend on your IRAF installation, and they are set during the IRAF user installation (see iraf.net), or via Ureka installation (see http://ssb.stsci.edu/ureka). Also be sure to run the “mkiraf” command to create a logion.cl (http://www.google.com/search?q=mkiraf).

What is missing now is to define these environment variables properly. You can either type the following lines in the terminal or add them to your .bashrc file, so that PyRAF is available at all times:

export iraf='/usr/lib/iraf/'
export IRAFARCH='linux'

Another working IRAF/PyRAF environment has been deployed!

17/7/2019 Update: Check olebole’s comment below regarding points 1 and 2.

During the last few days I have struggled a bit with the wavelength calibration of some long-slit spectra. When using the identify task of IRAF I found that I could select (“m”ark) only some of the lines, without any clear indication why the rest were not recognized (some were indeed stronger but others were equal or weaker). The actual error that kept popping out was “Center not found: check cursor position”. Although I went on to investigate all parameters (even the aidpars !) the one thing that solved the problem was finally to increase a bit the full-width of the features to be identified (fwidth), from the default value of 4.0 to 10 pixels. Apparently some of the lines seems too wide to be determined as real features by the task.

Between 12 to 19 of September 2016, we organized the Ondřejov Summer School 2016, which was held at the Stellar Stellar Department of the Astronomical Institute (Czech Academy of Sciences), in Ondřejov. The main aim of the school was to provide students hands-on experience with real observational data, i.e. to understand observations, reduce them, and discuss their results. This project-oriented activity was supplemented with lectures on various techniques and one of them was a workshop on Python and IRAF/PyRAF.

As a workshop the main focus was given to show examples that the students could repeat at the own computers. Its structure was based on a very small introductory talk and then examples of how to use Python and IRAF in real problems. With this post I make public all of this material both as a reference and as a (hopeful) help to others.

Material:

I have been working lately with IRAF/daophot with the PyRAF environment in a 32bit machine and everything is smooth except for the fact that it takes many hours to finish the run in each field I examine. As I would like to finish this analysis as soon as possible I opted to use a 64bit machine (with a larger number of more powerful processors). But still, the daophot task was failing, in particular during the psf task of daophot (when it needs to plot the stars from which you select the stars to create the psf model). The error message was the following:

ERROR (851, “Cannot restore graphics world coordinate systems”)

The same error was raised in both 64bit machines I used (a. PyRAF 2.1.3 and IRAF 2.16.1 in Debian 8, b. PyRAF 2.1.7 and IRAF 2.16.1 in CentOS 7.2), and actually regardless if I used the pyraf or the native cl environment. So the most probable cause should be somehow connected with the 64bit architecture (even though one possibility could be a typo in the iraf/unix/hlib/extpkg.cl file [1], but this is not the case for these PyRAF versions).

After asking this very question in iraf.net forum [2] I got an answer from fitz that

The problem happens because in the 64-bit v2.15 port the size of the WCS graphics structure was changed, and a corresponding change was required in pyraf (which is responsible for the graphics the same way the CL is). This bug was fixed in pyraf but apparently only for the 32-bit version

The proposed workaround is to download the 32bit version of daophot (at [3]) and put it under iraf$noao/bin.linux64 directory. If this is under /iraf (under root) then you need root permissions to copy there and make it executable, preferably as chmod +x x_daophot.e, to make it available for all users in the same machine.

So, we have to keep in mind that the 32bit versions could be the replacement for other non-working tasks as well. Let’s hope that there will be finally future releases of PyRAF [4].

[1] STScI / PyRAF FAQ #1.5
[2] iraf.net/forum: “PyRAF and DAOPHOT – ERROR 851”
[3] ftp://iraf.noao.edu/iraf/v216/support/linux/x_daophot.e
[4] 0. Support Status : “Due to reduced budgets for HST, we can only provide minimal pyraf support. If you have a problem that cannot be solved quickly by our first tier support, we may be unable to help. This is unfortunate, but it is the best we can do with the resources available. “,
STScI / PyRAF FAQ #0, accessed 15 March 2016

Does it seams that IRAF is getting too much time to start? And if it starts do you see cannot access host 'iraf.noao.edu:80' ?

If yes, then if you also try iraf.noao.edu then you will notice that it does take too much time to open (if…it opens!). So, in this case the server is down and IRAF cannot access it. But why IRAF tries to connect to the website? After all we just need it to run some tasks.

After v2.16 they have included in the login.cl a line which checks for updates. To avoid this the solution is just to comment the line which includes the ‘ckhupdate’ command in the login.cl, e.g.

#============================================================================
# Check for updates to the system
# chkupdate

# Notify the user if we're using the global login.
path (".") | scan (s1)
if ( osfn("home$") != substr (s1, strldx("!",s1)+1, strlen(s1)) ) {
printf (" *** Using global login file: %slogin.cl\n", osfn("home$"))
}
;


Definitely this is not new, but since I have seen this message a couple of times in the past (without looking into any further) and I had performed some installations yesterday, I got a bit worried if something broke. Fortunately not! So, a note of what I changed is a probable useful future tip!

Among the simplest installation methods to install IRAF are SciSoft (for RedHat/Fedora based distributions) and Fernando Avila Castro iso files (for Ubuntu/Debian based distributions). However, both require root privileges. What can you do when you do not have these permissions?

Ureka [1] comes in rescue! It is easy to install (download and run the installer![2]) and it doesn’t need any root privileges, so you can install it under your home directory.

Moreover, it offers, along with a simple installation of IRAF, a full installation of Python, with many pre-installed modules useful to perform almost any astronomical reduction (e.g. PyRAF, Astropy, matplotlib, pandas, [more]).

By default Ureka (1.4.1) installs with a Python version (2.7.3), and it modifies your shell starting scripts (bashrc,tcshrc) to create the aliases to use. E.g. the added lines in a .bashrc file are:

ur_setup() {
    eval `/var/home/maravelias/.ureka/ur_setup -sh $*`
}
ur_forget() {
    eval `/var/home/maravelias/.ureka/ur_forget -sh $*`
}

and you can invoke Ureka by simple typing: ur_setup. Now, this means that in every shell that you start you will have to type over and over again the same command, which becomes at least frustrating. In order to avoid this, you can simply comment the alias line:

#ur_setup() {
eval `/var/home/maravelias/.ureka/ur_setup -sh $*`
#}
ur_forget() {
    eval `/var/home/maravelias/.ureka/ur_forget -sh $*`
}

so Ureka starts with every shell now. This option allows Python to install any extra modules (through PyPI – pip install) to the Ureka path and not system’s Python path.

Ureka allows for multiple installations of difference versions or different variants so there is reasoning in using the alias – but still for many users it may not be necessary (in any case you just uncomment the lines and you get back to the normal use).

So, Ureka offers an easy and clean way to install IRAF and a powerfull Python installation at the same time, without any root permissions. The only drawback is that it needs a lot of space (~4-5 GB), but it does not seem dramatic nowadays. [As far as the sys-admins it is easier to provide you with more extra space rather than software installations/updates within their systems.]

References:
[1] Ureka: http://ssb.stsci.edu/ureka/
[2] Documentation 1.4: http://ssb.stsci.edu/ureka/1.4.1/docs/index.html

After a long time I finally managed to understand how to use IRAF’s sbands, along with a standard star and a filter.

1. If you want to use a spectrum from the standard star library of IRAF, you have to convert it from a .dat file to an image file .fits – Spectra files located in …/iraf/iraf/noao/lib/onedstds/ – by using the rspectext.
2. Having created an image file* (of the 1D spectrum), sbands uses this image and a bandpass file to provide the results (fluxes, or magnitudes). The contents of the bandpass file is the bands for which we want to measure flux/mag: name of the band, central wavelength, width, filter transmission file (if available).

*[2014/06/25 update]
Caution: the spectrophotometric files contain 2 columns, the wavelength and the magnitude, which means that they are reversed when plotted – so they have to be transformed into fluxes (in wavelength units) to be used properly.

[originated from: IRAF.net/ spectrophotometry with sbands or something else at ]

After having looked and searched into the whole IRAF/PyRAF directory to point out the directory that contains the template spectra for the stsdas/synphot package it was just a matter of a quick look at the front page of the package to resolve the issue!

As it is said: “Since the data files used by synphot in its calculations are rather large, they are distributed separately”
and they provide the links to the compressed files containing all the template data.

At the beginning we started looking into which path these files are stored since you can call each template by using the environmental variable crcalspec$template_spectrum. But this variable looks at the path where these templates are stored (outside IRAF), as it is defined (when installed) though the crref variable.

While doing photometry in some rather large images with DAOPHOT, the allstar I experienced a panic from IRAF:
PANIC in `/iraf/iraf/noao/bin.linux/x_daophot.e': Memory has been corrupted
By googling the exact error I hit on this post in iraf.net (#136679) where the problem presented was exactly the same. The solution is simple to change the cache parameter from yes to no (by epar allstar).

As Mike mentioned:

The ‘cache’ parameter tells the task to try to keep (up to) three images plus a starlist in memory to improve the processing time (as opposed to hitting the disk repeatedly). Practically speaking, what this does is consume a lot of the physical memory and if your images are very large you may be using a lot of swap space and losing any benefits, it may also be amplifying the effect of a memory leak and leading to the error (just a guess). In any case, if it works without the cache then keep going with that.