light curves with extractor

Alex requested a way of better controlling the start time of a light curve created using extractor. I've added a new parameter lcstart which sets the time of the first bin (in spacecraft time units). This is checked into the development version as extractor v5.12.

cleansis

We have seen a few cases of observations of bright sources using Suzaku where running cleansis creates a hole in the image. Part of the problem is running with too high a log probability parameter but there is also a subtle error when iterating. Pixels which are already identified as bad and have their counts set to zero are still included when calculating the local background. This artificially lowers the estimated background hence increases the probability that a pixel will be flagged as bad.

I changed the algorithm so that pixels flagged as bad in previous iterations are ignored when calculating the local background. This change has been checked into the development version.

New release

HEAsoft 6.7 is now available. This includes XSPEC v12.5.1. The biggest change is in gain fitting which has been reworked as the preliminary step towards allowing the creation of response models. Response models will be created analogously to spectral models and will be used to either replace or augment response matrices.

This release is also the first which does not include XSPEC v11. This version has in practice not been supported for some time and it is getting increasingly difficult to ensure that it builds and runs on all systems.

XSPEC bug fixes update

12.5.0af	The zredden multiplicative model component is calling the same function as redden, and therefore its redshift parameter has no effect. Our thanks to Brian Refsdal for pointing this out. Report added on May 05, 2009.
12.5.0ag	When in setplot wave mode and the user's Xspec.init file entry for WAVE_PLOT_UNITS is set to angstrom, plot efficiency and plot icounts don't display due to a missing plot label error. Our thanks to Maurice Leutenegger for pointing this out. Report added on May 08, 2009.
12.5.0ah	Fix to sqrt(2) error in the definition of the thermal broadening used in APEC models. Our thanks to Irina Zhuravleva for pointing this out. Report added on May 21, 2009.
12.5.0ai	This adds a new C-callable function to provide access to the version string for users linking the XSPEC models library into their own programs. The function is declared in src/XSUtil/FunctionUtils/xsFortran.h, and has the prototype: int xs_getVersion(char* buffer, int buffSize); Report added on May 22, 2009.
12.5.0aj	The model command fails to parse expressions that are explicitly of the form: M(M1(A1)+M2(A2)+M3(A3)). Our thanks to Jeremy Sanders for pointing this out. Report added on May 22, 2009.
12.5.0ak	When soft limits are in use during fitting, the pegged parameter test should be comparing the actual value against the hard limits, not the adjusted value. Otherwise the parameter may in some cases peg too soon. Report added on June 08, 2009.
12.5.0al	The same sqrt(2) fix mentioned in patch 12.5.0ah is needed when thermal broadening is turned on for v2.0 NEI models. Our thanks to Richard Sturm for pointing this out. Report added on June 08, 2009.

xselect cleansis

Fixed a bug spotted by Ed Cackett that the iterate_clean parameter is not recognized. Just needed to add it to the xselect.key file.

Note that there are issues turning up using the probabilistic sisclean tool for bright Suzaku sources. There is a tendency for the core of the PSF to be removed. Suggested fixes are either to change the critical probability or turn the iteration off.

self-irradiated funnel

Pavel Abolmasov sent in his self-irradiated funnel model (Abolmasov et al. 2009) and I added an entry to the new models web page.

addascaspec

The perl script addascapec, which is actually useful for other missions in addition to ASCA, had a couple of problems which I've fixed. The default errmeth should have been POISS-0 instead of POISS-1 and this parameter was not being used when summing background spectra.

Lexicon of handy statistical methods

Andrew Gelman's statistical lexicon.

bug in thermal broadening

Irina Zhuravleva at MPE pointed out that the thermal broadening option in the xspec apec model gives a sigma which is a factor of sqrt(2) too large. Fortunately, I don't think has any science implications since there have been no observations capable of measuring this broadening.

update on 6/8/09: Richard Sturm points out there is the same error in the NEI code.

panda/epanda/bpanda regions

I've added support to cfitsio region filtering for the panda/epanda/bpanda regions produced by ds9. Note that this assumes only one azimuthal region and one annulus although the region specification itself allows multiple for both. Made corresponding changes in extractor though note that the FITS region extension written in this case is not covered by the standard which does not include these shapes.

Spectral file format standard document

I updated the PHA file standard document to try to make it clearer. I fixed an error in the type II example which had HDUCLAS3 and HDUCLAS4 inverted.

XSPEC bug fixes update

Here are the bug fixes from the last month. They can be found in the usual place.

12.5.0w	A new tclout option has been added to make it easier to retrieve the fit parameters' sigma values. The syntax for this option is: tclout sigma [<>:]n where n is the parameter number. If it is not a variable parameter or if the fit was unable to calculate its sigma, a value of -1.0 is returned. Report added on Mar 06, 2009.
12.5.0x	The model.dat entry for the vequil model is missing the parameter for Ar abundance, causing the vequil parameters which follow it to be misinterpreted in the code. (This bug does not affect the equil model.) Report added on Mar 06, 2009.
12.5.0y	The kerrconv convolution model code is still using the xspec11-only LMODDIR environment variable for locating the kerrtable.dat model data file. This should be modified for usage in xspec12, where LMODDIR doesn't exist. Report added on Mar 11, 2009.
12.5.0z	This removes an ambiguous reference build error specific to gcc-4.1.1 on the Solaris-2.9 platform. Our thanks to Dacheng Lin for pointing this out. Report added on Mar 26, 2009.
12.5.0aa	The diskir model goes to infinity when its rirr parameter = 1.0, which is also the default value of rirr's lower limit. Our thanks to Brian Refsdal for pointing this out. Report added on Mar 26, 2009.
12.5.0ab	The nsmax model function is only able to find its auxiliary files when it is run directly from XSPEC's model data directory (heasoft-6.x/spectral/modelData). Our thanks to Stephen Doe for pointing this out. Report added on Mar 31, 2009.
12.5.0ac	Fixes to a couple of obscure cases of gain parameter usage: If a gain parameter belongs to a response temporarily replaced by a dummy response, or a gain parameter is indirectly removed through the data command erasing its associated spectrum and response, the gain shift may still be applied to a response occupying its former spectrum and source number. This patch also improves the show response output. Report added on Apr 06, 2009.
12.5.0ad	The nthcomp model has an uninitialized variable for the case of input energies less than kT_bb/10^4, which may cause a crash to occur. Our thanks to Gulab Dewangan for pointing this out. Report added on Apr 14, 2009.
12.5.0ae	For those linking the models library into their own programs, the swind1 model crashes when outputing a warning message as it indirectly accesses a variable intended for use only in XSPEC. Our thanks to Stephen Doe for pointing this out. Report added on Apr 14, 2009.

Notes on using SWIG to import C++ in Python

As a test case using my heasp library which is being converted from C to C++. Created heasp.i file to define interface. Basically a concatenation of the heasp.h, PHA.h and PHAII.h files with the following at the top. Adding additional classes will just require cat'ing their *.h file to the end and adding the appropriate #include at the top. Note the %include of "std_string.i" which is required to make the string arguments work. Similar %include specifications will be required if other STL classes are arguments for methods.

%module heasp

%include "std_string.i"

%{
#include
#include
#include
#include
#include
#include
#include
#include
#include
#include
#include
#include

#include

#include "heasp.h"
#include "PHA.h"
#include "PHAII.h"
%}

The interface file is then converted to C++ using

swig -python -c++ -classic heasp.i

which creates heasp_wrap.cxx and heasp.py. These must then be combined with the *.o files to build a shareable library called _heasp.so (note that the _ prefix and .so suffix are required). Compiling heasp_wrap.cxx will require include files from python. On my mac the relevant flag is
-I/Library/Frameworks/Python.framework/Versions/Current/include/python2.5
and on the lab Linux network
-I/usr1/local/include/python2.5

On the mac the command to make the shareable library is
ld -bundle -flat_namespace -undefined suppress -o _heasp.so ${OFILES} heasp_wrap.o -L${HEADAS}/lib -lCCfits_2.1 -lcfitsio_3.12
and on Linux
$(CXX) -shared -o _heasp.so ${OFILES} heasp_wrap.o -L${HEADAS}/lib -lCCfits_2.1 -lcfi
tsio_3.13

The module is loaded into python by
>>> import heasp
On the mac this generates an undefined symbol ___eprintf. This may require an update of Xcode to fix. On linux (running python2.5) there are no problems. Example use is

>>> import heasp
>>> spectrum = heasp.PHA()
>>> spectrum.read('file1.pha',1,1)
>>> spectrum.disp()
>>> spectrum.write('test.pha')

chkarf

Updated and much simplified chkarf so it actually lists the correct keywords as mandatory. This tool really should just be a perl script - the current program is overkill.

using openMP in gcc

I'm testing out parallelization using openMP on an 8-core Linux box. To build xspec to support openMP options requires the hmakerc to be edited to add -fopenmp to CFLAGS, FFLAGS, and CXXFLAGS and -lgomp to be added to F77LIBS4C (I added it immediately before -lgfortran).

First test is in sumape.f to parallelize over the individual elements when interpolating the continuum and pseudo-continuum...

C$OMP PARALLEL PRIVATE(ien,limdown,limup,ihigh,energy)

C$OMP DO SCHEDULE(DYNAMIC)

DO ielt = 1, nmelt

...


ENDDO

C$OMP END DO

C$OMP END PARALLEL

The variables defined as private are to avoid threads overwriting variables being set and used by other threads.

Timing tests show that parallelization doesn't win enough in this case. Including the parallel directives slows a newpar down from 0.02 seconds to 0.4 seconds due to the overhead.

Computing with GPUs

There is web site all about using GPUs for general purpose computing. One particularly useful looking article goes into some of the options and issues of coding for GPUs.

problem with extractor5.09

In the course of Suzaku pipeline testing Lorella spotted a problem I introduced during the attempted speed-up of the fixwmp routine. Pixels with non-zero counts show up even within the part of the region which is supposed to be set to -1.

The particular case of Suzaku demonstrated a conceptual issue here. Those pixels showing up within the supposed excluded region are real events. What is happening is that the region is defined in sky coordinates but the WMAP is written in detector coordinates. I calculate a conversion to determine which WMAP pixels are within the image region however it is a time-averaged solution. In the pre-5.09 version the satellite attitude wobble results in some events being incorrectly excluded from the WMAP. It was these events that the 5.09 version actually included however they are potentially confusing to users.

The fix is checked in as v5.10.

xselect and Chandra grating data

Jane Turner reported that the current version of xselect fails to read Chandra HETG evt2 data. This is due to a bug in xsel_mdb.f which I think only matters for Chandra grating files.

XSPEC units for setplot wave

In v12.5 we switched the y-axis units under setplot wave to per Hz instead of per Angstrom. Since this has not been universally popular we have added an initialization option to decide between the two cases. This is available as patch 12.5.0t. That and other recent patches can be obtained from the usual place and are summarized below

12.5.0r	Fix needed for all etable model calculations. Our thanks to Roderick Johnstone for pointing this out. Report added on Feb 12, 2009.
12.5.0s	When the both the absori and pexriv models are loaded during the same XSPEC session, whichever model is loaded second will not work properly. Note: this bug originated in version 12.5.0. Report added on Feb 13, 2009.
12.5.0t	This adds an option for choosing Y-axis units when plotting in setplot wave mode. The selection is made by the WAVE_PLOT_UNITS entry, which has been added to the new version of the Xspec.init file. The units may now be specified in Hz (the default for version 12.5.0), or angstroms as they were prior to 12.5.0. This applies to plots of data, counts, and all variants of model and ufspec plots. Report added on Feb 19, 2009.
12.5.0u	Minor code changes mostly to clean up the messages reported when running XSPEC under the valgrind software tool. Report added on Feb 19, 2009.
12.5.0v	Multiplicative model components which are created with the mdefine command should not be multiplied by the energy bin widths. Our thanks to Roderick Johnstone for pointing this out. Report added on Feb 19, 2009.

non-integer COUNTS column

Steve Snowden ran into a silent problem because he made spectrum files with non-integer COUNTS columns. This is in violation of the standard and many ftools will simply silently truncate the float value to the nearest integer. I've added checks in the low-level library routines to write warnings in this case.

Update on 5/7/09. It turns out that the RXTE s/w writes PHA files with a real COUNTS column although the values are integers. So, I've improved the checking so that no warnings are given if the values are integers even if the column is defined as real. If the values really are non-integer then a warning is written and the values truncated to integers.

Gelman on statistics

I'm going to start linking to interesting blog posts by Andrew Gelman because they often have important points to remember. Today he talks about a couple of reviews of "The Black Swan" and includes the following note on Bayesian inference :

From a philosophical point of view, I think the most important point of confusion about Bayesian inference is the idea that it's about computing the probability that a model is true. In all the areas I've ever worked on, the model is never true. But what you can do is find out that certain important aspects of the data are highly unlikely to be captured by the fitted model, which can facilitate a "model shift" moment. This sort of falsification is why I believe Popper's philosophy of science to be a good fit to Bayesian data analysis.

Recent XSPEC bug fixes

The following are all available from the usual place.

12.5.0l	This slightly modifies the state of newly untied parameters. When a parameter is untied, its frozen/unfrozen setting should retain the setting of the parameter(s) to which it was linked, rather than be restored to its original value. Our thanks to Phil Evans and Delphine Porquet for pointing this out. Report added on Jan 14, 2009.
12.5.0m	When steppar is executed, it should not automatically run a fit at the end to place the parameters in their best-fit state. Instead it should restore the parameters to their pre-steppar values, regardless of the fit state. Our thanks to Tim Kallman for pointing this out. Report added on Jan 14, 2009.
12.5.0n	Bug fix for a C-to-Fortran boolean conversion error, affecting the default setting for the thermal broadening flag of the bapec and bvapec models. Unless overridden by the APECTHERMAL setting, thermal broadening should be ON by default for bapec and bvapec. This bug has only been observed on Solaris platforms. Report added on Jan 15, 2009.
12.5.0o	When a delcomp operation is performed on the unnamed (default) model and no spectra are loaded, the model calculation is not updated. Our thanks to Laura Brenneman for pointing this out. Report added on Jan 21, 2009.
12.5.0p	This relaxes the OGIP requirement that spectral data files must contain a RESPFILE keyword, primarily to allow the loading of certain RXTE/PCA files. Our thanks to David Smith for pointing this out. Report added on Jan 30, 2009.
12.5.0q	A new option has been added to the tclout command: tclout gain. See the online tclout manual entry for proper syntax and usage. Prior to this, it was not possible to retrieve the values of gain parameters from tclout. Our thanks to Frank Haberl for pointing this out. Report added on Feb 03, 2009.

xselect and region filters

I've added clarification to the on-line help for xselect about region filters. It is dangerous to make region filters in Image coordinates - the correct options are WCS or Physical. If the xselect image binning is not 1 then regions in Image coordinates will not be handled correctly.

Chandra CY11 webspec updated

Updated webspec with the final ARFs for Chandra Cycle 11.

Word-cloud of the 171 Astro2010 NOI titles

recent xspec patches

12.5.0j	Parameter links both to and from a model which is not associated with any spectra (shown as "Active/Off" in the model display), should not be removed when the model's state gets updated (such as when using the dummyrsp command). Our thanks to Laura Brenneman for pointing this out. Report added on Dec 24, 2008.
12.5.0k	The dummyrsp command may cause a crash when entering a very large number of energy bins. Our thanks to Joel Frioriksson for pointing this out. Report added on Dec 31, 2008.

webspec for Chandra

I've updated webspec to use the Chandra CY11 rmfs and arfs released by the CUC. Note that in mid January they will be releasing new arfs with corrected contamination corrections and effective areas.

xspec patch for energies extend

12.5.0i The energies extend option does not work properly for the case of a data group containing multiple responses whose energy bins differ. Our thanks to Joel Frioriksson for pointing this out. Report added on Dec 15, 2008.

webspec updated to include pile-up

At the request of Andrea Prestwich at the CUC I've added the pile-up model to WebSpec. In the simple interface there is a check box to added it while in the advanced interface the user is expected to select pileup as the option for the first model component.

One tweak could be to set the default pile-up parameters appropriately for each instrument.

cfitsio region filtering bug

The recent cfitsio version region filtering fails when the first region is an exclude. This was an error I introduced when extending to multiple components as allowed in the FITS REGION extension. The fix should be available next week. Meanwhile a work-around is to include an initial region covering the entire field.

Update 12/15/08: There is also a memory-handling error in the code which reads a FITS REGION. This bug is only triggered if the extension contains a large number (>100) of exclude regions.

extractor bug for detector coordinate regions

Dave Henley reported problems with the new extractor when using detector coordinates for the region. This is because I wasn't correctly handling the case when the WCS type is blank. This is corrected in v5.09 which should be released as part of HEAsoft patch update next week.

xspec 12.5 patches

Another set of recent xspec patches. Two of these are minor mistakes in the 12.5 release, the others cover relatively obscure problems.

12.5.0e	In the FNINIT function (for users linking the XSPEC models library into their own programs), the default model data directory name needs to be updated from "modelIonData" to "modelData" to coincide with v12.5.0 restructuring. Our thanks to John Houck for pointing this out. Report added on Dec 05, 2008.
12.5.0f	Patch to improve the behavior of an exceptional case of Levenberg-Marquardt fitting, where a parameter becomes pegged due to a zero second-derivative matrix diagonal element AND the user chooses to exit before convergence. This prevents an attempt at a covariance calculation on the parameter, which otherwise might lead to a segmentation fault. Our thanks to Stefano Bianchi and Jeremy Sanders for pointing this out. Report added on Dec 05, 2008.
12.5.0g	Fix to the swind1 model function. Now that it is included as part of XSPEC's built-in models library, it should no longer look to the $LMODDIR symbol for finding the model data directory. Our thanks to Delphine Porquet for pointing this out. Report added on Dec 09, 2008.
12.5.0h	When doing a 2-panel plot of plot counts/lcounts chisq and the current fit statistic is cstat, the Y-axis scaling of the chisq panel is wrong. Our thanks to Fill Humphrey for pointing this out. Report added on Dec 09, 2008.

extractor bug for annular regions

Phil Evans spotted a problem in the new release when using an annulus region type. The region filtering is performed correctly however the WMAP created is too small in the case when the WMAP and image coordinates are the same. This is actually an error I introduced in the cfitsio code (which is however not used internally in cfitsio so doesn't change its standard use). A work-around is to use the elliptannulus region type instead of annulus. I also have a fix within the extractor code to get around this.

This bug is important for Swift analysis.

xselect and ASCA

The switch to wild cards in xselect.mdb was a little too enthusiastic in the case of the ASCA SIS - the catcol list still needs to be specified for individual detectors. The current version causes an error when running make obscat for ASCA SIS. I have checked in a fixed version.

script to split *.rsp files into *.rmf and *.arf

I've written a little perl script split_rsp_to_rmf_arf which takes an input response matrix file and splits it into a response matrix file with unit total response at each energy and an arf which gives the total response at each energy. This can be used to mock up in v12 xspec the behaviour of v11 xspec with /b models. An example is given on the wiki section on backgrounds.

three more patch updates for xspec 12.5

Three minor patches to the new release are available on the bugs page.

12.5.0b	When the editing commands addcomp, delcomp, or editmod have been used on a model that already contains linked parameters, the link expressions are not properly updated in the output to a save model file. Our thanks to Laura Brenneman for pointing this out. Report added on Dec 03, 2008.
12.5.0c	This removes the warnings issued if datasets contain CHANTYPE strings other than PI or PHA. It will still issue warnings if the CHANTYPE differs between the spectral data and RMF files. Report added on Dec 03, 2008.
12.5.0d	Additional C/Fortran function wrappers provided for the following recently added C++ models: cflux, partcov, simpl, and spexpcut. These are required for users who link the XSPEC models library into an external Fortran or C program. Our thanks to John Houck for pointing this out. Report added on Dec 03, 2008.

First xspec v12.5 patch released.

Li Xin-Li noted that his kerrbb model has an error in turning on/off self-irradiation and limb darkening. The model is fine if either both are on or both are off but if one is on and the other off then the wrong combination was calculated. Fixed as 12.5.0.

A couple of extractor fixes

A couple of bug fixes that were too late to get into the recent release.

1. When using a GTI file for time filtering the fall-back position of looking in the first table extension for the table data was not working. This should not be a problem if the gtinam parameter was set correctly.

2. The fixwmp routine was unnecessarily slow with the new method of region filtering. It is still slow but not as slow as it was.

These fixes are in v5.07 and will appear on the HEAsoft bugs page.

minor tweak to extractor region extension output

Switched the region extension written by extractor back to the XMM convention of REG001##. This works for Chandra because CIAO looks for the HDUNAME='REGION' keyword which is also included.

xspec 12.5

HEAsoft 6.6 which includes xspec v12.5 is now available.

updated xselect Chandra response matrix script

The xselect script xsl_chandra_acis_makeresp made some unwarranted assumptions about DSVAL1 holding the list of selected chips. Generalized this and fixed bug in one of the calls to mkacisrsp where I'd left off an wmap parameter specification. This change will make it into the release coming up this week.

writeFITS.tcl

Jörn Wilms pointed out it would be really useful in xspec to be able to write a bunch of information including best-fit parameters and errors to a single row in a FITS file. I've added a script writeFITS.tcl to do this. It either creates a new file or appends to an existing one. Obviously, this is not a general solution for all the information a user might want to save however it can easily be modified to save additional information.

Band model for GRB spectra

Dick Willingale pointed out that the grbm model has a hard-coded, artificial lower limit of 1 keV on the third (tem) parameter. There is no good reason for this so I have removed it for the upcoming release.

simftest.tcl

Moved the ftest.tcl in manager to simftest.tcl and added it to TclIndex so that it is automatically loaded in xspec and available as the command simftest.

Suzaku AO4 on WebSpec

I've updated WebSpec with Suzaku AO4 files. The HXD background files supplied had a total exposure of 3e6 seconds. The recommended procedure is to have a background file with the same exposure as the source file. This assumes that statistical errors on the background are Poisson and the systematic error can be represented by changing the total normalization (using corfile). I modified the standard background files to change them from COUNTS to RATE so an input background file could be created for a given exposure time just by changing the EXPOSURE keyword.

Note that fakeit with a background file creates a simulated source spectrum with the exposure requested and a simulated background spectrum with the same exposure as that of the input background file. A useful enhancement for fakeit would be to control the exposure of the simulated background file - one possibility would be to allow commands such as
XSPEC12> fakeit inbackground.pha 50000

UPDATE 11/04/08: Fixed an error on the energy ranges for the PIN. The input background spectrum is only valid up to 75 keV. Also, use an background exposure time of 10 x that of the source for the PIN as recommended by the HXD team.

latest xspec bug fixes

Three patches have been put out in the last couple of weeks. Two of these deal with the case when parameters differ by very large amounts. This is not recommended - xspec will work best if all the parameters are scaled to O(1).

12.4.0ap	In a Levenberg-Marquardt fit, the zero threshold for the diagonals of the second-derivatives matrix needs to be reduced. This is relevant for cases where fit parameters may differ by enormously large (~10^20) magnitudes. Our thanks to Roderick Johnstone for pointing this out. Report added on Oct 21, 2008.
12.4.0aq	The tclout model command no longer appends a newline character to the model name. This allows its output value to be used more easily as an argument to another XSPEC command. Our thanks to Jeremy Sanders for pointing this out. Report added on Oct 23, 2008.
12.4.0ar	This adds automatic normalization to the covariance calculation performed at the end of a Levenberg-Marquardt fit, which makes it more robust when dealing with large differences in magnitude among the fit parameters. Report added on Oct 29, 2008.

Redshifts in models

Here is a possible generic way to handle redshifts in model components. Allow any component to have a "z" prepended to its name eg pow-> zpow. The model parser strips off the z while setting a has-redshift flag. Any model component with the has-redshift flag set gets a redshift parameter added (after all the other parameters and before any normalization). When evaluating the model the function is passed a redshifted energy array if has-redshift is set. The output flux array could also be corrected for the (1+z) normalization factors although we would have to be careful to make this consistent with the flux and luminosity commands.

The only type of model where this would not work is one which uses the redshift for something in addition to the energy shift - I think the only such case is the cooling flow which needs the luminosity distance to convert the Mdot parameter to an amount of flux to add.

Partial covering convolution model

The new release will include a partial covering convolution model (partcov) which can be used to turn any absorption model into a partially covered version. This depends on the new model parsing that takes into account the non-associative nature of convolution models. So, if P is the partial covering model, M the multiplicative absorption model and A the additive model then (PM)A turns M into a partial covering absorption. However PMA or P(MA) applies the partial covering the product of M and A which does not give the desired result.

CHANTYPE problems

The Suzaku HXD PIN files use a CHANTYPE of PI_PIN. Strictly speaking this is a violation of the standard however it seems too limiting to just allow CHANTYPE the values of PHA or PI. xspec currently generates a spurious error message about mismatched CHANTYPE when reading the HXD spectrum and response. I've put a temporary workaround in RealResponse.cxx which suppresses the spurious warning however we need a more general solution.

update to Jeremy's spex python script

Jeremy Sanders pointed out that files created by his python script using SPEX have no pseudo-continuum causing a seg fault in sumape.f. I've checked in a fix to sumape and Jeremy has also supplied an updated version of the script which avoids the problem (by creating a zero pseudo-continuum).

recorn model

I've worked out how to implement the v11 recornrm model in v12 and allow the correction file norm to be a free parameter. This is done through a new model recorn which has parameters the spectrum number and the correction norm. It is implemented as a mixing model though it doesn't actually do anything to the model spectrum - just changes the correction norm stored in the dataset object. It has to be done as a mixing model to ensure that the XSFunctions library stays free of symbol references to xspec internals - mixing models are not included in XSFunctions but have their own library. This model will be included in the release planned for early November.

Fixed bug in new extractor

Pre-release testing on the Swift pipeline showed up an embarrassing bug in extractor v5 - the BACKSCAL keyword was not written correctly. This is now fixed in v5.02.

There was also a problem writing out the ROTBOX shape in the REGION extension. Further, Swift software does not recognize ROTBOX at present so I changed this to BOX even when there is a rotation angle. This may produce problems with other missions. These changes are v5.03.

And another bug this time when writing the REGION extension in the case when the input event file already has one and further region filtering is performed by extractor. v5.04.

More minor fixes. Removed the reference to number of events rejected because they fall outside the region (which extractor can no longer no). Eliminated duplication of identical regions in the same component being written to the REGION extension. v5.05.

new Comptonization model

Jack Steiner, Ramesh Narayan, Jeff McClintock (CfA) and Ken Ebisawa (ISAS) have supplied a new comptonization convolution model for xspec. This is available under the additional models website and will be included in the next xspec release.

xselect bug in save time

Roderick Johnstone reported that the command "save time cursor" ignores the final argument and always then prompts for it. Triggering this bug depended on which commands had been used earlier. Fixed xsel_save.f.

Fortran compiler notes

I've been working on building heasoft distributions using both gfortran and g95. This under Tiger with Xcode 2.4.1 (which is gcc 4.0.1).

g95 is the current snapshot binary from www.g95.org which is v0.92. It is installed under /usr/local. To build I had to edit the hmakerc files after configure to remove -fno-f2c from the FFLAGS macro.

gfortran is from "fink install gcc42" and is v4.2.2. To build requires editing hmakerc to change -O2 to -O in the FOPT macro and to add at the beginning of the F77LIBS4C the directories -L/usr/lib -L/usr/lib/i686-apple-darwin8/4.0.1 and -L/sw/lib/gcc4.2/lib/gcc/i686-apple-darwin8/4.2.2. Note that it is vital they be in this order and at the beginning of the macro or else the loader will link in gcc v4.2.2 libraries instead of gcc v4.0.1.

compbb documentation

Ken Ebisawa pointed out that the v12 description of the compbb model did not include a change made to the v11 help. I've corrected this and updated the on-line page.

gfortran on the mac

To get gfortran using fink get the package gcc42 or gcc43.

plot delchi

Jeremy Sanders points out that the label for delchi plots is wrong or at least misleading. Fixed this by changing the entry in plotLabels.dat. Note that the label is now hardwired for the chi-squared statistic however the delchi plot doesn't make any sense for other statistics.

Jeremy also noted that the y-axis label is a long way to the left and this looks strange. The reason for the position is that with LAB ROT we need to leave space to write numbers of the form
A x 10^(-B). Curiously, we are defaulting to rotated labels for 2-panel plots but unrotated labels for 1-panel plots. I've no idea why. Note that the easy way to move around the position of the y-axis label is
PLT> lab pos y 3.0

log files in xspec

Fixed bug that was adding a "#" to the next line instead of the current line of log file output. This is a partial fix to issue 1024. Input when setting parameter values (in model or newpar) is still not echoed to the log file - problem is in ModelContainer::setParameterFromPrompt.

Update: had to back this change out because it doesn't work with tcerr, which is not buffered and hence passes one character at a time to TclIO::write. Craig worked out how to fix xsLog.cxx to fix the problem.

Changing NOMAD password

Since I now have to change this every 60 days these notes are to help me remember what to change so that my Thunderbird e-mail keeps working.

1. Change NOMAD password on the webmail.nasa.gov site
2. Using Firefox Preferences remove saved password for webmail.nasa.gov
3. Using Thunderbird Preferences remove saved password for smtp://gsfc...
4. Click on NOMAD inbox and when prompted enter new password. Check the Password Manager box.
5. Restart Thunderbird.
6. Send an e-mail and when prompted enter new password. Check the Password Manager box.

Also need to fix CNE wireless connection. Open Internet Connect, go to 802.1X then "staff CNE wireless" in the configuration menu. Enter the new password in the appropriate box.

xspec priorities

I've created a Wiki page with a list of xspec priority enhancements. These are divided into large and small projects. At the moment they are not priority-ordered but I will add that along with additional information on the large projects.

xspec issue fixes

Updated documentation to match recent enhancements and additional models. Note that the description of the bvapec model is now included along with the bapec model, not in a separate file.

Fixed a plotting bug. The green line marking 0 or 1 in the res, ratio, chi etc. plots did not go all the way across the plot if a rescale was done so that only part of the x-range of the data was included. Note that a rescale to start at an x-value below the initial data will not draw the green line for the x values below the start of the data.

XMM ao8 webspec files

I've replaced the XMM rmfs and arfs in WebSpec with the appropriate ones for AO8.

super-exponential cutoff absorption

At the request of Jean Ballet added a super-exponential cutoff absorption model to the additional models page. This is useful for fitting gamma-ray observations of pulsars (eg Nel & de Jager 1995).

Also added model to the development version so it will be included in the next release.

energies command and table models

Roderick Johnstone pointed out that recent bug fixes have broken use of the energies command for table models. This is fixed in patch 12.4.0an.

added new models to development version

Added the following contributed additional models to the development version so they become standard in the next release : diskir, cflux, nsmax, zxipcf, swind1, kerrdisk, kerrconv, nthcomp.

latest xspec patches

12.4.0ak	The cutoffpl model can show a distortion for very low energies (E/B approx. less than 10^-3). The incomplete gamma function calculation which this is based on needs to be improved in this region. Our thanks to Joern Wilms, Moritz Boeck, and Manfred Hanke for pointing this out. Report added on Aug 28, 2008.
12.4.0al	This adds the Fortran wrapper function xgtcht(console,log) for those wishing to retrieve XSPEC's console and log chatter level settings from inside their own Fortran local models. The function is equivalent to the xgtcht contained in the xanlib library used by XSPEC11. Report added on Aug 28, 2008.
12.4.0am	Enhancement patch required for reading data sets and responses in the new format used by GLAST GBM files. The modifications include the removal of the SPEC_NUM column for type-II data files, and the storing of multiple response matrix extensions into one response file. XSPEC will now distinguish among multiple response extensions by appending a curly-bracket specifier to the response file name: myResponse.rsp{n} where n is the EXTVER keyword value of the extension. Report added on Aug 28, 2008.

parallelizing xspec

Craig has been experimenting with openMP which is included in gcc v4.2 and above. Since openMP uses a shared memory model the trick is to find places where multiple threads can be used which do not trash each others' memory. A valid case is the loop over responses in Model::fold. This speeds up the code if there are multiple datasets in the same datagroup. It involves changes to Model.cxx as well as MultiResponse.cxx and RealResponse.cxx.

Test was with a dual-processor machine running Solaris. Note that the -xopenmp compiler flag required either no optimization or -O3.

calculating fluxes and errors

Prompted by a question from Jeremy Sanders I have developed a new way of calculating fluxes and errors for models and parts of models. The trick is to provide a new convolution model called cflux whose first two parameters are the minimum and maximum energy over which the flux is calculated. The final parameter is the flux. The component is placed in front of the model component(s) whose flux is being calculated and one of the additive components has its normalization frozen. The standard fit and error command on the flux parameter will then provide the best fit flux and confidence region. This model is available through the extra models web page.

xselect and XMM-Newton spectra

The current version of xselect/extractor writes spectra with region extensions having the EXTNAME keyword set to REGION. This is required by CIAO. However, SAS requires EXTNAME to be REG00101 so added a command to xsl_xmm_epic_makeresp to change EXTNAME before calling the response and arf generation tools.

genrsp and LO_THRES

Allyn Tennant pointed out that genrsp writes a keyword LO_THRESH which should actually be LO_THRES. I fixed this as v2.04 by adding the new keyword while leaving the old just in case there is other s/w that expects it.

LISA xspec stuff current status

strain2xspectime.pl creates input file correctly for time series data and this is fit using the mbht model. However, residuals show high frequency correlations presumably because noise is being
applied in frequency space in synthlisa. The strain2xspec.pl and mbh model which work in frequency space are not working correctly - the model doesn't match the data when it should. All current changes are checked into the repository.

I'm not clear how and why the postBuffer is used - need to check this with Sean or Ira.

Larger question is how should the noise be generated ? Is it correct to do this in the frequency domain ? A better detector model may be required.

xspec patches

Some recent patches from the bugs page :

12.4.0ae	When using the gsmooth or swind1 models, certain combinations of parameters can cause an invalid array access. On 64-bit Linux systems this will cause a crash. Our thanks to Curtis McCully for pointing this out. Report added on Jul 08, 2008.
12.4.0af	When settings have changed through either the xsect or cosmo commands, the model calculations should be immediately updated. (This patch also improves the cosmo command-line parser.) Our thanks to Stefano Bianchi for pointing this out. Report added on Jul 08, 2008.
12.4.0ag	The kerrd and nsa additive models will cause XSPEC to crash in certain cases, the latter primarily on Intel/Mac platforms. Report added on Jul 14, 2008.

bug in genrsp

Allyn Tennant spotted a bug in genrsp when making a response for a grating. I'd missed a case when the channel boundaries are in decreasing energy order and the result was the center bin of the gaussian response was chopped out. This is fixed in v2.03.

extractor and filenames with "+"

Kim Page at Leicester reported that the HEAsoft 6.5 version of extractor dies when using a time filter if the filename contains a "+". It was interpreting this as an HDU number. Fixed this and checked in a change which will be included in extractor v5.0.

xspec model library

At the request mainly of the ISIS folks I've managed to make a stripped-down HEAsoft 6.5 distribution which only includes the Xspec model library files to build only libXSFunctions, libXSUtil and libXS. I assume that the user has a cfitsio library already. Some minor hacks of the top level and Xspec configure files were necessary to make sure that hmake didn't try to build any heacore libraries. Also, moved some include files into Xspec/src/include from directories that otherwise aren't needed.

documentation errors for table files

Mike Corcoran spotted a couple of minor problems. The ogip_92_009 memo refers to the incorrect location for the wftbmd.f and .c files. It should be $HEADAS/../ftools/spectral/tables/. Also the onepar and twopar Makefile don't work because they need to refer to the correct version numbers of xanlib and cfitsio - I fixed this by switching to standard HEADAS Makefiles so running hmake will automatically pick up current versions of libraries. These changes are checked in but won't make the imminent HEAsoft 6.5 release.

tclout goodness

goodness was listed as an option for the tclout command but we had forgotten to actually implement it. This is fixed as patch 12.4.0ab after Phil Evans pointed out the discrepancy.

Bug fix in xselect for suzaku

When I added the support for 5x5 to 3x3 conversion I missed the case of the user already having done an extract events. If this has happened then the event file to be used has a different name. Checked in a fix to this and was a bit cleverer about jumping out of the 5x5to3x3 routine if we don't need to run it.

modification to lrt.tcl

The likelihood-ratio test script doesn't work when there are multiple datagroups. The reason is that switching between models by making them active/inactive loses information on the parameters for groups > 1 when the model becomes inactive. We perhaps should modify the model inactive command to track multiple datagroups however as a workaround I modified lrt.tcl to use command files to store the model definitions. Each time the model is switched the appropriate command file is read.

started work on extractor v5

Started work on the new version of extractor which will use cfitsio internal filtering as much as possible.

In the course of switching the region filtering to cfitsio I found an obscure bug. If the image and wmap coordinates are the same but have different binning and wtmapfix is set then the WMAP bins with -1 are not quite correct. Several bins at the edge of the selected region can have counts in them when the value should be -1.

xselect and ROSAT

Dave Pooley reports that xselect has problems under Mac OS 10.4 but not 10.3 when using ROSAT data. I confirmed the error under 10.4. The error message is something like

Could not find keyword for Instrument for ROSAT
NONE
Got new mission: ?4????AF?'?AF?AF?AF??
'7????P???? R??????D???51

I don't know why this occurs but the fix is to edit xselect.mdb to use the new wildcard facility to deal with the PSPCB and PSPCC instrument variant names instead of using the instkey specification to only read the first four characters of the INSTRUME keyword.

xspec patches

A few new patches have been added recently. 12.4.0x fixes a number of problems that showed up when using the new g++-4.3.0 compiler and when using gcc on Solaris platforms. This patch level is released as part of HEAsoft 6.4.1. 12.4.0y improves the handling of cases when the L-M algorithm pegs a parameter due to a zero or negative pivot value in the Hessian. 12.4.0z keeps the fit valid when the fit method is changed. This allows the improve command to be run after an L-M fit. 12.4.0aa is a fix to allow filenames prompted for by the fakeit command to being with a / ie to be absolute paths. Patches available from the usual place.

xselect: Chandra response generation

Overhauled the response generation option on "save spec" for Chandra ACIS data. I added an extended? parameter to specify whether the source is a point source or not. For a point source the arf is created using mkarf, and for an extended source mkwarf is used. The rmf is generated using mkrmf or mkacisrmf depending on whether CTI correction has been performed. The script checks keywords in the spectrum to determine which is appropriate. Note that to find some of the auxiliary files required xselect should be being run in either main directory for the dataset or in a directory one level down (eg secondary).

Note that I also modified the FITS region extension written by extractor so that the EXTNAME is REGION and EXTVER is used to track multiple extensions. This is a change from an EXTNAME of REG001## and is made to conform with the CXC standard defined by McDowell & Rots. There may be an issue here is XMM-Newton SAS routines are expecting the REG001## names.

Science 2.0

SciAm has a brief survey of Web 2.0 applications in science. In particular, it argues that putting scientists working notes on-line will foster greater communication and collaboration.

reading old RMFs

Tahir points out that the BeppoSAX RMFs don't work in rbnrmf. The problem is that these files have only RMFVERSN='1992a' and do not have the HDUVERS keyword that the software now requires. The workaround is to add HDUVERS='1.0.0' to the MATRIX extension. I've updated the relevant subroutines to check for RMFVERSN if HDUVERS is not found.

setplot wave units in xspec

Changed the plotting units for the setplot wave option. Now showing f_nu and variants instead of f_lambda. The model and ufspec y-axis is photons/cm^2/s/Hz; emodel and eufspec are in Jansky (1e-23 erg/cm^2/s/Hz); eemodel and eeufspec in keV Jy. The latter is a bizarre mixed unit but it isn't clear to me what to use here. It might be an idea to have an option for the user to switch between f_lambda and f_nu.

Updated: decided that for the eemodel/eeufspec plots the units should be erg/cm^2/s corresponding to standard nu F_nu plots.

Independently, I fixed an error in setting the y-axis range for eufspec and eeufspec under setplot wave.

genrsp v2.02

Added an option for the resolution varying linearly with either the energy or the wavelength.

xspec updates

A number of updates and bug fixes, mainly minor.

12.4.0p This is the fix for suzpsf alluded to earlier.

12.4.0q CHANTYPE was not being set correctly after fakeit none leading to confusing warning messages.

12.4.0r A generalization of fakeit to work correctly with multiple models as requested by K.D. Kuntz.

12.4.0s A couple of patches for plot eff bizarreness that I ran across.

12.4.0t Fix for a seg fault in data when the user gives the filename as empty double quotes.

12.4.0u A fix for unusual compiler configurations.

12.4.0v Allow the unnamed model to be turned on and off.

12.4.0w Correction for a bug when using non-standard weighting and toggling between chi and cstat statistics.

Patches are available from the usual place.

likelihood-ratio test

Added a new Tcl script, lrt.tcl, to calibrate the likelihood ratio test. Currently only works with one data set. The user defines two models (by default model0 and model1) then the script generates a user-specified number of simulated data sets based on model0 and for each one evaluates the log(likelihood ratio) between model1 and model0. The script returns the fraction of simulations with a smaller log(LR) than the data.

I will then use lrt.tcl as the basis for a script to calculate F-test probabilities for
adding an additional model component.

fakeit with multiple sources

We now have the patch out for fakeit to simulate data from the case where there are multiple sources (with their own responses). This is xspec v12.4.0r and is available from the usual place.

suzaku xis response generation

Enabled the automated response generation in xselect for Suzaku XIS spectra. I'm still waiting for a decision from the XIS team on what default energy and channel binning to use in the response matrix. Ishisaki-san has suggested that the correct estepfile option for xissimarfgen is "medium" so the response generation uses this.

Currently using a "medium" binning scheme which produces 2048 channels. Spectra extracted from xselect are binned using this scheme whether or not the automated response generation is used.

Note that I also modified the MDB code to allow trailing wild cards eg XIS* will match to any of XIS0, XIS1, XIS2, or XIS3.

Also, updated addascaspec so it runs correctly if there is no ARF used.

suzaku xis 5x5 and 3x3 files

Checked in a change to xselect so that when there is a mix of 5x5 and 3x3 suzaku xis files read in then, when doing an extract, xis5x5to3x3 is used to create a temporary 3x3 file. This was requested by the xis team and ensures that doing an extract event creates a correct output 3x3 file. At the moment we aren't doing anything with the extra information in the 5x5 file.

local models in xspec v11 on the Mac OS X

Bryan points out that the documentation for local models in v11 doesn't include the special instructions required for the Mac OS X ie because we can't make a shared library the xspec11 executable must be rebuilt with the new local model library. I've added this to the relevant manual page although hopefully most people are using v12 now anyway.

grade strings in xselect/extractor

The specifications rules for grade strings in the xselect "filter grade" command do seem lead to a little confusion so made them a but more lenient. Now allow spaces in between individual elements. eg "0:0 2:4 6:6" is now allowed as well as "0:0,2:4,6:6". This change is in extractor v4.83.

twittering

There is now an xspector twitter. I'm not sure whether I will keep this up or where it will go - it's an experiment.

suzpsf model in xspec

As noted yesterday the current version of suzpsf assumes the WMAP is in sky coordinates, which is no longer true. I've now fixed this. There are two changes required. Firstly, extractor has been modified (v4.83) to write the REF* keywords which XMM files use to record the original sky WCS keywords. Secondly, the suzpsf model has been modified to use these keywords (as the xmmpsf model already did).

To fix spectra that were created using older versions of extractor add the following keywords to the primary header with the values of the corresponding keywords from the EVENTS extension of the input event file.
REFXCRVL <- TRCVL10 from events files
REFXCRPX <- TCRPX10 from events files
REFXCDLT <- TCDLT10 from events files
REFYCRVL <- TRCVL11 from events files
REFYCRPX <- TCRPX11 from events files
REFYCDLT <- TCDLT11 from events files

more minor xspec patches

12.4.0m The start-up statistic, method, and weight settings in the user's Xspec.init file are effectively ignored. Instead these always start with Xspec's hardcoded default settings (chisq, leven, and standard respectively). Our thanks to Maurice Leutenegger for pointing this out. Report added on Mar 05, 2008.

12.4.0n If the precise best-fit values are input as starting parameter values to a Levenberg-Marquardt fit, for certain data sets and models the fit may run through a large number of redundant loop iterations and produce "SVDCMP" warning messages. Our thanks to Jeremy Sanders for pointing this out. Report added on Mar 06, 2008.

12.4.0o For xmmpsf and suzpsf mixing components, Xspec is unable to parse RA and DEC strings with colon delimiters (entered with the xset command). Report added on Mar 06, 2008.

Note also the there is a problem with suzpsf for all options except that of assuming the surface brightness distribution is centered on the middle of the WMAP. This issue arises because the model was written when the Suzaku XIS spectra used sky coordinates in their WMAP. Since late 2006 we have used detector coordinates but due to an oversight the suzpsf model was not updated.

minor patch for error and named models.

Once the error command is run for a parameter belonging to a named model, it does not allow the user to specify unnamed-model parameters during future error runs. Our thanks to Kip Kuntz for pointing this out. Bug fix 12.4.0l.

background in xspec

I've started a wiki page on how to deal with background in xspec. At the moment it gives an example on how to deal with the case when the background requires a different response matrix to the source.