Low-latency, online search documentation

Table of Contents

• Introduction
  • Algorithm development and search pipeline
    • The Low Latency Online Inspiral Detection (LLOID) algorithm
  • Relationship to published work
• Preliminaries
  • Gotchas:
• Preparing the template banks
  • Resources used
• Setting up the analysis dag
• Running the analysis dag
  • Adjusting the gracedb FAR threshold
• Monitoring the output

Introduction

The prospect of detecting tens of compact binary mergers per year with Advanced LIGO and Virgo is ushering in a new era of gravitational-wave astronomy with potential for joint electromagnetic and astroparticle observations. The Compact Binary Coalescence group in the LIGO Scientific Collaboration (LSC) has identified the low-latency detection of gravitational waves, i.e., detection in less than 1 minute from the signal arriving at Earth, from coalescing neutron star and black hole binary systems as a science priority. Joint gravitational wave and electromagnetic observations will determine the progenitor models of some of violent transient astronomical events - such as gamma-ray bursts - and set the stage to discover completely unanticipated phenomena.

Algorithm development and search pipeline

Matched filtering is the baseline method to detect compact binaries. It is typically implemented in the frequency domain using fast Fourier transforms that are several times longer than the duration of the underlying signal (Phys. Rev. D 85, 122006 (2012)). Binary neutron stars may be observable for more than 30 minutes in the advanced LIGO band, which implies that the standard matched filtering paradigm incurs a O(1) hr latency. Naively implementing a time-domain matched filtering algorithm through brute force convolution would achieve the latency goals, but it would require O(10) GFLOPS per gravitational wave template (ApJ 748 136 (2012)), which is not feasible.

The Low Latency Online Inspiral Detection (LLOID) algorithm

Significant algorithmic development has led to computationally viable low-latency methods to implement matched filter searches for compact binaries (Astron Astrophys 541 A155 (2012), ApJ 748 136 (2012), Phys Rev D 86 024012 (2012)) by compressing the waveform parameter and/or sampling space. The results are algorithms with similar computational cost to the traditional FFT based search but with much lower latency.

The LLOID algorithm described in ApJ 748 136 (2012) applies two techniques to significantly reduce the computational cost of time-domain filtering.

1. Singular value decomposition (SVD) of template waveforms (Phys Rev D82 044025 (2010)) to exploit degeneracy in the compact binary parameter space and avoid performing unnecessary template waveform convolutions.

1. Exploiting the chirp (increasing frequency and amplitude) time-frequency evolution of compact binary systems to critically sample the templates according to the instantaneous Nyquist sampling limit by decomposing the strain data and templates into multiple power-of-two sample rates (multirate filtering).

The use of SVD and multirate filtering retains 99.7% of the signal-to-noise ratio (loses only 0.3%) obtained from conventional matched filtering. This can be compared to the acceptable signal-to-noise ratio loss from the discreteness of template banks, i.e., 3% (Phys. Rev. D 60, 022002 1999). The SVD and multirate filtering paradigm therefore leads to at most a 1% loss in event rate over a conventional matched filter search.

The LLOID algorithm applied to advanced LIGO BNS searches estimates the filtering cost for a real-time compact binary search to be O(1) MFLOPS per gravitational wave template for the advanced LIGO design sensitivity using the waveform decomposition described in ApJ 748 136 (2012). It is an open question, however, if different choices of template decomposition combined, perhaps, with special hardware features could further reduce the computation cost. Since that is not currently known the remaining estimates on this page use the decomposition described in ApJ 748 136 (2012).

The O(1) MFLOPs cost neglects all other costs of performing the analysis including additional signal consistency checks, coincidence analysis, background estimation, etc., but it roughly sets the order of magnitude required to do a low latency search with the LLOID algorithm.

Relationship to published work

• Phys Rev D82 044025 (2010): Provides basic expressions for computing the singular value decomposition of waveforms and expectations for signal-to-noise ratio loss, etc.
• Phys Rev D83 084053 (2011) describes the composite detection statistic that is formed from the singular value decomposition basis vectors. This statistic allows one to infer the presence of signals in the data without reconstructing the physical SNRs. This is used optionally in gstlal inspiral pipelines. Using it saves substantial CPU cost but at the cost of some detection efficiency. Unlike the published result, we do not threshold on the composite detection statistic value, rather we do "peak finding" with the composite detection statistic time series and reconstruct physical SNRs around the peaks. The peak time window is a free parameter.
• ApJ 748 136 (2012): The so called "LLOID" algorithm combines singular value decomposition with multirate sampling of the data and waveforms to produce a computationally efficient time-domain algorithm. The algorithm described in the paper is basically identical to what is done.
• Phys Rev D88 024025 (2013) The basic principle of significance estimation in the gstlal inspiral pipeline is described in this paper. Since this paper was published, the likelihood ratio ranking statistic has been extended to more accuratly account for correlated (between detector) signal probabilities, which was ignored in the published work. This has led to substantial sensitivity improvement, but also a more computationally complex background estimation procedure which is still being developed. The first 5 engineering runs used the simpler procedure as published. The 6th engineering run will be the first to use the new procedure.

Preliminaries

• Start by making a directory where you will run the analysis:

    $ mkdir /home/gstlalcbc/engineering/7
  

Gotchas:

• You will need a robot certificate in order to communicate with gracedb and other services requiring authentication. https://wiki.ligo.org/AuthProject/LIGOCARobotCertificate

• You will need your own lv alert account https://www.lsc-group.phys.uwm.edu/daswg/docs/howto/lvalert-howto.html

Preparing the template banks

Instructions

• First make a directory for the template banks, e.g.,

    $ mkdir /home/gstlalcbc/engineering/7/bns_bank
  

• Next obtain a Makefile to automate the bank generation, e.g. : this example

      $ wget http://www.lsc-group.phys.uwm.edu/cgit/gstlal/plain/gstlal-inspiral/share/Makefile.online_bank
  

• Then modify the make file to suit your needs, e.g. changing the masses or low frequency starting points

• Then run make

    $ make -f Makefile.online_bank
  

• After several minutes this will result in a dag file that can be submited by $ condor_submit_dag bank.dag

• You can track the progress of the dag by doing:

    $ tail -f bank.dag.dagman.out
  

Resources used

• gstlal_bank_splitter
• gstlal_psd_xml_from_asd_txt
• ligolw_add
• gstlal_inspiral_svd_bank_pipe

Setting up the analysis dag

A makefile automates the construction of a HTCondor DAG. The dag requires the template banks set up in the previous section.

• Begin by making a directory for the analysis dag to run, e.g.,

    $ mkdir /home/gstlalcbc/engineering/7/analysis
  

• Next obtain a makefile to automate the dag generation, e.g., this example

• Modify the makefile to your liking (make sure it knows where the files you made with the bank dag are) and then run make

    $ make Makefile.online_analysis
  

  Note that you will be prompted during the dag creation stage for your lvalert username and password. The password for lvalert is not secure. It will show up in plain text on the submit node. You should not use any password that is used elsewhere (like your ligo.org password) Since this dag is for running on LDG resources, the plain text should not be much of a problem. One should not check in any code or makefiles that contain this information (hence why you are asked for it). lvalert is a thin layer only sending announcments. gracedb, where the real data is stored, still requires proper ligo authentication.

• Take note of the last few lines of the dag generation step, it will provide a url where you can monitor the output (described below). It should look something like this:

      NOTE! You can monitor the analysis at this url: https://ldas-jobs.ligo.caltech.edu/~gstlalcbc/cgi-bin/gstlal_llcbcsummary?id=0001,0009&dir=/mnt/qfs3/gstlalcbc/engineering/5/bns_trigs_40Hz
  

• Pay attention to the gracedb far threshold. When set to -1

Running the analysis dag

• Once make is finished condor submit the dag

    $ condor_submit_dag trigger_pipe.dag
  

• Once the dag has run for a sufficient time to collect background statistics condor remove it and wait for the jobs to finish. NOTE: Since these online jobs run "forever man" they rely on condor sending them a soft kill (sig 15) rather than a hard kill (sig 9). The jobs intercept signal 15 and perform some cleanup steps to shutdown the analysis and write out the final output files. This is a necessary step, otherwise data will be lost.

• Next you can remake the dag with a positive gracedb far threshold, e.g., 0.0001.

    $ make -f Makefile.online_analysis
  

• This will overwrite your dag file, but not other files, like logs, so you will need to force resubmission

    $ condor_submit_dag -f trigger_pipe.dag
  

• Now gracedb event uploading will be enbabled and the analysis is in production mode

The running dag topology looks like this:

Adjusting the gracedb FAR threshold

Each gstlal_inspiral job that is running is also running its own webserver as a way to request information about the job or to post new configuration information to the job. One very useful result of this is a way to dynamically change the FAR threshold used to submit gracedb events. This can be done from the triggers directory with

    $ gstlal_ll_inspiral_gracedb_threshold --gracedb-far-threshold <FAR THRESH> *registry.txt

Resources used

• gstlal_inspiral
• gstlal_ll_inspiral_get_urls
• gstlal_ll_trigger_pipe
• gstlal_ll_inspiral_gracedb_threshold
• gstlal_inspiral_ll_create_prior_diststats
• gstlal_inspiral_marginalize_likelihood
• gstlal_inspiral_marginalize_likelihoods_online

Monitoring the output

As mentioned above you can monitor the output. Please see the gstlal_llcbcsummary for more information.

Events are uploaded to https://gracedb.ligo.org