Skip to content

Coadd masking: Backmasking, Stellar Halo Suppression, and Star Asteroid Coadd level Masking

Jump to bottom
Adam Wright edited this page Sep 15, 2020 · 13 revisions

Once the coadds are made, there is a 2nd-round of masking which takes place on the lensing-band only (or at least, as long as the lensing band is the same as the detection band this is the case). This comes in three parts: Backmasking, Stellar Halo Suppression, and Asteroid Coadd-level Masking. The Backmasking uses coadd-level information to make chip-level masks/regions. The Stellar rings (only if bright stars are present) subtracts the halo signature around bright stars and where the brightest part of the rings are seen, a mask is placed. Lastly, after re-making the coadds with the backmasking/autosuppresion finished, the asteroids are masked from the final coadds.

In addition to that, there are two types of coadd-masks applied to all bands, not just the lensing band. The star masking and the edge masking.

In total we have backmasking autosuppression star asteroid edgemask

Then there is also the standard masking, which looks something like this: ./maskImages.pl -r /gpfs/slac/kipac/fs1/u/awright/SUBARU/Zw2089/W-J-V/SCIENCE/reg/ -l toMask_Zw2089.log -d /gpfs/slac/kipac/fs1/u/awright/SUBARU/Zw2089/W-J-V/SCIENCE_weighted/ SUP

Backmasking

Backmasking is when you make masks in coadds, but place regions in individual chips (i.e. SUPA*_[0-9].reg files) so regions are placed on individual chip exposures with backmask.py. Backmasking steps are as follows (see Evernote for more info, including examples of things to look for when masking):

  1. adam_make_backmask_ims.py: Make "backmaskables", i.e. smoothed exposure-coadd minus all-coadd images that draw out the features you'd like to mask. (e.g. ipython -i -- adam_make_backmask_ims.py /nfs/slac/g/ki/ki18/anja/SUBARU/MACS1226+21/W-C-RC/SCIENCE/). This also spits out all of the specific calls to baskmask.py that you'll need to put the regions in place. This script makes a BACKMASK/ directory with images used to mask reflections, ghosts, cross-talk, and maybe even asteroids. They have these names:
  • SUPAXXXXXXX-all.coadd.fits: difference images for each exposure ("coadd_cluster_SUPAXXXXXXX"-"coadd_cluster_all")
  • SUPAXXXXXXX-all.coadd.smoothed.fits: smoothed difference images

Set up options at the top of the script first:

#BEFORE RUNNING CODE (1): set make_smoothed if you want smoothed images for all types of coadd differences (all, gabodsid, gabrot)
#BEFORE RUNNING CODE (2): set make_gabodsid and make_gabrot to determine the types of coadd differences you want to make (makes "all" always), can make gabodsid & gabrot as well

make_smoothed=1                                                                                                                                                    
make_gabodsid=0
make_gabrot=0
  1. backmask.py: then use ds9 to make regions in the smoothed-difference coadds and copy them over to the chip-level using backmask.py (e.g. ipython -i -- backmask.py /nfs/slac/g/ki/ki18/anja/SUBARU/MACS0416-24/W-C-RC/SCIENCE/BACKMASK/SUPA0125907-all.coadd.smoothed.fits /nfs/slac/g/ki/ki18/anja/SUBARU/MACS0416-24/W-C-RC_*/SCIENCE/SUPA0125907_[0-9]*OCF.fits). More examples:
ipython -i -- backmask.py /nfs/slac/g/ki/ki18/anja/SUBARU/${cluster}/W-C-RC/SCIENCE/BACKMASK/${supa}-all.coadd.smoothed.fits /nfs/slac/g/ki/ki18/anja/SUBARU/${cluster}/W-C-RC_*/SCIENCE/${supa}_[0-9]*${ending}.fits

ipython -i -- backmask.py /nfs/slac/g/ki/ki18/anja/SUBARU/MACS0416-24/W-C-RC/SCIENCE/BACKMASK/SUPA0125912-all.coadd.smoothed.fits /nfs/slac/g/ki/ki18/anja/SUBARU/MACS0416-24/W-C-RC_*/SCIENCE/SUPA0125912_[0-9]*OCF.fits

Note: I don't fully trust these scripts, they need work. So make sure the regions you save ARE ACTUALLY BEING SAVED, because python may tell you it's saving them when it's really not.

Note from Doug: I think transferMasks.py is another tool that can be used to place stellar halo suppression rings (place rings in one image from each rotation, then it'll transfer those regions to the other images. Final adjustment with imageflipper). But, with two clusters, it's also pretty easy to copy and paste between images in ds9.

Stellar Rings (see Evernote)

  1. adam_make_autosuppression_ims.py #makes autosuppression directory with images that can be used to place stellar halos. This might help too: adam_use_fix_autosuppression.sh

  2. Make rings by fitting ~/wtgpipeline/rings.reg on top of individual stars at the chip-level in ds9. Here are some old notes on how I did this from Evernote, I'm not sure how accurate this is, but it's a decent guideline:

    1.) open /nfs/slac/g/ki/ki18/anja/SUBARU/MACS0416-24/W-C-RC/SCIENCE/coadd_MACS0416-24_all/coadd.fits in ds9 and for each star, load in ~/bonnpipeline/rings.reg and place the rings in the appropriate positions
    2.) when all of the rings are in place in the coadd_all, then save to ring_template_MACS0416-24.reg and make a backup somewhere just in case
        - template should be saved in wcs coordinates
        - should group each star and give different colors to make keeping track of things easier
    3.) make new directory ~/data/MACS0416-24/W-C-RC/SCIENCE/autosuppression and move ring_template_MACS0416-24.reg there
    4.) for each exposure, do this:
        (a) open coadd_exposure to see which stars go in which chip reg files:
            ds9 -geometry 2000x2000 -zscale -cmap bb /nfs/slac/g/ki/ki18/anja/SUBARU/MACS0416-24/W-C-RC/SCIENCE/coadd_MACS0416-24_SUPA0125911/coadd.fits -regions load /nfs/slac/g/ki/ki18/anja/SUBARU/MACS0416-24/W-C-RC/SCIENCE/autosuppression/ring_template_MACS0416-24.reg -zoom to fit &
        (b) Make chip region files (for each chip with a star on it) from header and whatever # star rings hit the chips (see ~/data/MACS0416-24/W-C-RC/SCIENCE/autosuppression/make_SUPA0125903.sh for example):
            head -n 3 ring_template_macs0416.reg > SUPA0125911_4.reg
            head -n 3 ring_template_macs0416.reg > SUPA0125911_5.reg
            head -n 3 ring_template_macs0416.reg > SUPA0125911_6.reg
            head -n 3 ring_template_macs0416.reg > SUPA0125911_7.reg
            head -n 3 ring_template_macs0416.reg > SUPA0125911_8.reg
            head -n 3 ring_template_macs0416.reg > SUPA0125911_9.reg
            head -n 3 ring_template_macs0416.reg > SUPA0125911_10.reg
            grep "Group 1" ring_template_macs0416.reg #>> SUPA0125911_4.reg
            grep "Group 4" ring_template_macs0416.reg >> SUPA0125911_4.reg
            grep "Group 4" ring_template_macs0416.reg >> SUPA0125911_5.reg
            grep "Group 1" ring_template_macs0416.reg >> SUPA0125911_6.reg
            grep "Group [1-2]" ring_template_macs0416.reg >> SUPA0125911_7.reg
            grep "Group [2-3]" ring_template_macs0416.reg >> SUPA0125911_8.reg
            grep "Group [3-5]" ring_template_macs0416.reg >> SUPA0125911_9.reg
            grep "Group [4-5]" ring_template_macs0416.reg >> SUPA0125911_10.reg
        (c) _for each chip_: Go through and place them in the exact location, then save the region files (in the autosuppression directory) using `physical` coordinates:
            ds9 -geometry 2000x2000 -zscale -cmap bb /nfs/slac/g/ki/ki18/anja/SUBARU/MACS0416-24/W-C-RC/SCIENCE/SUPA0125911_4OCF.fits -regions load SUPA0125911_4.reg -zoom to fit &
    5.) Once the rings are in place, do the actual suppression
        #before running batch_suppress.sh, make sure you have consistent endings matching for each science, weight, and flag image
        ./batch_suppress.sh MACS0416-24 W-C-RC /nfs/slac/g/ki/ki18/anja/SUBARU/MACS0416-24/W-C-RC/SCIENCE/autosuppression $ending $queue
        Currently $ending would be "OCF", once the IC works, it should be "OCFR".
        You should check the slac queues: probably $queue should be "long"
    6.) Lastly, move files into appropriate places (not sure why this isn't done as part of the previous script)
        ./move_suppression.sh MACS0416-24 W-C-RC  $ending
        Currently $ending would be "OCF", once the IC works, it should be "OCFR".

  1. batch_suppress.sh: This actually subtracts the gradual increase in counts approaching a ring. (e.g run it like ./batch_suppress.sh MACS0416-24 W-C-RC /nfs/slac/g/ki/ki18/anja/SUBARU/MACS0416-24/W-C-RC/SCIENCE/autosuppression $ending $queue and probably $queue should be "long" or "long -W 7000")

  2. This puts the regions in the right spot for them to be applied to the coadds. (e.g run it like ./move_suppression.sh MACS0416-24 W-C-RC $ending)

Note: Pat at some point wrote a utility that took in a text file of the ra & dec positions of all the bright stars and automatically output the individual image region files using the scamp headers...looks like that file is called create_rings.py. others:

Note: Remember to back up all region files! backup_rings.sh will do that ... and give you a clue about the directory structure.

Re-make masked coadds

Finish all of the backmasking/autosuppression, then apply all of the regions to the final coadds and remake the coadds using adam_reg2weights_filter.sh or adam_reg2weights-maybe_coadd_catchup.sh. First vim adam_reg2weights-maybe_coadd_catchup.sh and make sure you've commented out any of the stuff you don't want to do, then run it: ./adam_reg2weights-maybe_coadd_catchup.sh ${cluster} "W-C-RC_2010-11-04" > OUT-backmask_catchup.log 2>&1

star and asteroid coadd-level masking

Once all the backmasking/autosuppression is done and the coadds are re-made, then you can do the coadd-level masking like this:

  1. make coadd.stars.reg: ./mask_coadd.sh ${cluster} W-C-RC
  2. then fix it in ds9: ds9 -cmap bb -zscale ${SUBARUDIR}/${cluster}/${filter}/SCIENCE/coadd_${cluster}_all/coadd.fits -regions load ${SUBARUDIR}/${cluster}/masks/coadd.stars.reg &
  3. then apply the masks to coadd.flag.fits: ./mask_coadd.sh ${cluster} W-C-RC coadd.stars.reg 2>&1 | tee -a OUT-mask_coadd.stars.log
  4. make asteroid masks by opening ds9 in such a way that the asteroids jump out in red: ds9 -zscale -rgb -red ${SUBARUDIR}/${cluster}/${filter}/SCIENCE/coadd_${cluster}_good/coadd.fits -green ${SUBARUDIR}/${cluster}/${filter}/SCIENCE/coadd_${cluster}_all/coadd.fits -blue ${SUBARUDIR}/${cluster}/${filter}/SCIENCE/coadd_${cluster}_all/coadd.fits -regions load ${SUBARUDIR}/${cluster}/masks/coadd.asteroids.reg &
  5. then apply the masks to coadd.flag.fits: ./mask_coadd.sh ${cluster} W-C-RC coadd.asteroids.reg gabodsid1554 2>&1 | tee -a OUT-mask_coadd.asteroids.log

star masking for all bands

  1. make coadd.stars.reg: ./mask_coadd.sh ${cluster} W-C-RC
  2. then fix it in ds9: ds9 -cmap bb -zscale ${SUBARUDIR}/${cluster}/${filter}/SCIENCE/coadd_${cluster}_all/coadd.fits -regions load ${SUBARUDIR}/${cluster}/masks/coadd.stars.reg &
  3. then apply the masks to coadd.flag.fits: ./mask_coadd.sh ${cluster} W-C-RC coadd.stars.reg 2>&1 | tee -a OUT-mask_coadd.stars.log

edge masking for all bands

  1. copy over a template/previous mask and adjust it: cp ~/wtgpipeline/coadd_W-J-V_Zw2089_good.edgemask.reg ${SUBARUDIR}/${cluster}/masks/coadd_${filter}_${cluster}_all.edgemask.reg
  2. then fix it in ds9: ds9 -cmap bb -zscale ${SUBARUDIR}/${cluster}/${filter}/SCIENCE/coadd_${cluster}_all/coadd.fits -regions load ${SUBARUDIR}/${cluster}/masks/coadd_${filter}_${cluster}_all.edgemask.reg &

REMS masking of RXJ2129 (based on differences between coadds)

  1. Make the masks using coadd_CR_removal.py
  2. Use do_REMS_masks_RXJ2129.py to Apply the masks to resampled weights and flag files like those following the pattern /u/ki/awright/my_data/SUBARU/RXJ2129/W-C-RC/SCIENCE/coadd_RXJ2129_all/SUPA0*sub.flag.*resamp*.fits
  3. Use do_coadd_RXJ2129_after_REMS_mask.sh to re-make coadds WITHOUT overwriting the resampled weight/flag files
Clone this wiki locally