MCEWiki - User contributions [en]

MCE script

2010-08-27T20:53:38Z

174.6.181.2:

* Auto-tuning system
** [[ Auto-setup (python) ]] - an integrated tool set for dealing with SQUID data.
** [[ Auto-setup (IDL) ]] - IDL-based auto-tuning code
** [[ mce_config_template system ]]
** [[ place_sq1.pro ]] for row selection for the purposes of sq1servo

Rectangle Mode Data

2010-08-02T17:28:59Z

174.6.181.2: /* Checking MCE configuration using rect_check.py */

"Rectangle Mode" refers to the sampling of a subset of the full array, typically at high rate. The data from the Readout Cards is packed into output frames efficiently to minimize overhead. Rectangle mode is available in MCE firmware starting with RC 5.0.1 and CC 5.0.0.

= Description of capabilities =

Rectangle mode eliminates strain on the acquisition system during fast acquisitions by eliminating read-out overhead. This means that the 'difficulty' of read-out is a fairly linear function of (sampling frequency) x (number of detectors read out) rather than depending on each.

The hardware limit on readout is roughly 4 MB/s. Since MCE data are 32-bit words, this permits 1e6 detector-samples to be read out per second.

So the maximum number of detectors that can be sampled at some frequency F is
D = (1 MHz) / F

You can't sample at 1 MHz. (Do your squids settle in 1us? Didn't think so.) But you could probably do a detector or two at 250 kHz.

= Firmware =

Rectangle mode requires "version 5" firmware on all cards, and at least RC 5.0.2. Here are links to the recommended SOFs at time of writing:
http://www.phas.ubc.ca/%7Emce/mcedocs/firmware/ac_v05000000_22dec2008.sof
http://www.phas.ubc.ca/%7Emce/mcedocs/firmware/bc_v05000000_22dec2008.sof
http://www.phas.ubc.ca/%7Emce/mcedocs/firmware/cc_v05000002_16jun2009.sof
http://www.phas.ubc.ca/%7Emce/mcedocs/firmware/rc_v05000002_16jun2009.sof

= Script Differences Between v4.x.x and v5.x.x =
The differences between basic frame setup in v4.x.x and v5.x.x are described here. The data frame structure has been made more flexible. A user can read out an arbitrarily rectangular patch of the array. The patch-size is specified using new commands.

*In v4.x.x: The array patch-size is flexible in terms of the number of rows, and the row start index.
wb cc num_rows_reported 41 //power-on default = 41
wb rcs readout_row_index 0 //power-on default = 0

*In v5.x.x: The array patch-size is now also flexible in terms of the number of columns, and the column start index. Note that matching parameters on the Readout Cards and the Clock Card must be consistent.
wb cc num_rows_reported 41 //power-on default as above
wb cc num_cols_reported 8 //power-on default = 8
wb rcs num_rows_reported 41 //power-on default = 41
wb rcs num_cols_reported 8 //power-on default = 8
wb rcs readout_row_index 0 //power-on default as above
wb rcs readout_col_index 0 //power-on default = 0

= Configuration =

The steps for configuring a rectangle mode acquisition are:
# Set the multiplexing rate using num_rows and row_len
# If necessary, adjust row_order to mux the channels of interest (you can only read out rows you are multiplexing...). If you do not intend to re-tune the array with the adjusted num_rows / row_order, you must also change the first entries of rc* adc_offset* to correspond to the desired rows and, for fast SQ2 switching, change row_order on the BAC/BC2.
# Identify a rectangle of detectors on the readout card to read out by setting the RC parameters
#* num_rows_reported, num_cols_reported, readout_row_index, readout_col_index
# Set the size of the readout frame by setting the CC parameters
#* num_rows_reported, num_cols_reported
# Set the CC decimation factor 'data_rate' so that the RC rectangles will stack perfectly into the CC readout frame.
#* e.g. suppose
#** RC num_rows_reported x num_cols_reported = 1x4
#** CC num_rows_reported x num_cols_reported = 33x8
#* then we set
#** CC data_rate = (33 x 8) / (1 x 4) = 66

= Readout limitations =

Rectangle mode configuration should respect the following hardware limits:
* Total data rate must not exceed 4 MB/s
* Read-out frequency must not exceed 20 kHz
* The multiplexing frame rate must not be set so fast that the MCE cannot communicate with the readout cards. This limit is probably in the 100s of kHz.

Other things to remember:
* num_rows >= 2 is encouraged.
* you can't sample a detector faster than it is being visited in the multiplexing cycle.

= Checking MCE configuration using rect_check.py =

Use the utility @rect_check.py@ to verify that data will be / were acquired reasonably.

e.g. To check the state of the MCE (requires python MCE bindings), run
rect_check.py --mce

To analyze the data framing in a runfile:
rect_check.py my_data.run

(Depending on stuff, you might need to invoke rect_check as:
$MAS_PYTHON/rect_check.py
or even
python $MAS_PYTHON/rect_check.py
Sorry.)

The output looks like this:

MCE configuration:
cc_rcs ( cc, rcs_to_report_data ): 60
cc_dec ( cc, data_rate ): 256
cc_nmux ( cc, num_rows ): 33
cc_cmux ( cc, row_len ): 100
cc_nr ( cc, num_rows_reported ): 32
cc_nc ( cc, num_cols_reported ): 8
rc_nmux ( rca, num_rows ): 33
rc_cmux ( rca, row_len ): 100
rc_nr ( rca, num_rows_reported ): 1
rc_nc ( rca, num_cols_reported ): 1
rc_r0 ( rca, readout_row_index ): 0
rc_c0 ( rca, readout_col_index ): 0
Framing:
RC storage per mux cycle: 1
CC read-out per frame: 256
CC decimation: 256
Contiguous? yes
Complete? no
Bizarro? no
Bounded? yes
Timing:
Mux freq: 15151.00
Mean sampling freq: 15104.00
Read-out freq: 59.00
Data volume:
Frame size (bytes/RC): 1200
Data rate (MB/s/RC): 0.07

The program has notions of what is "reasonable", and these are expressed in the lines "Contiguous?", "Complete?", "Bizarro?", and "Bounded?":
* 'Contiguous': all detectors that are read out will be read out at each multiplexing cycle
** When acquiring fast data from a subset of the detectors, at the multiplexing rate, you want "Contiguous = yes".
* 'Complete': all detectors that are multiplexed will be read out
** When acquiring normal data, you probably want "Complete = yes".
* 'Bizarro': the rectangle defined in the RC does not fit evenly into the readout frame.
** e.g. you have set up an 8x8 rectangle but you're reading CC frames that are 33x8. Don't do this. I'm not even sure what you'd get out.
* 'Bounded': the rectangle defined in the RC corresponds to detectors that are actually getting read/servoed. This basically just checks that rc? readout_row_index makes any sense at all given num_rows and num_rows_reported.
** This must be "yes" or your data could be weird.

Note that it is possible to be none of Complete, Contiguous, or Bizarro. For example, you could read out the first 33 of every 66 multiplexing cycles for 8 detectors, packing them efficiently into a 33 x 8 read-out frame. Maybe that should be called Bizarro. But right now, it isn't.

It is unlikely, but possible, to be both Contiguous and Complete.

If you are Bizarro, you are neither Contiguous nor Complete. See Venn diagram below.

[[Image:reasonable_rectangles.png]]

= Data acquisition =

Use "mce_run" in the usual way to acquire data. Note that the argument specifying the number of frames is always the number of 'read-out' frames, so for packed data the number of samples per detector will be higher.

e.g. for the above configuration
mce_run my_data 100 1

would acquire 100 read-out frames. Since the read-out frequency is 59 Hz, this would take about 2 seconds. Each readout frame contains 8 x 32 = 256 data from a single detector. So the resulting data stream contains 25600 samples.

= Data file loading =

Recent versions of mce_data.py can extract rectangle mode data from the packed frame structure. E.g. for the above case, extracting the data is as easy as
>>> d = MCEFile('my_data').Read()
>>> d.data.shape
(1, 25600)

= Rather long explanation and justification of structure, from an e-mail =

Yesterday Bryce and I were talking about paths towards a new rectangle readout mode. The following is a recommendation for the rough structure of the system that I think will be easy to manage and provide a maximum flexibility for backwards compatibility and future expansion.

The current rectangle mode is simpler than the planned one because the clock card and readout card need only agree on the number of data to readout, and no addditional waiting/synchronization is required between the clock card and readout card. The readout frame data sizes, internally and externally, are the same for everyone. The new mode is much more complex, because now the readout card has to accumulate several samples from each pixel into its internal buffer before sending the data to the clock card, and the clock card needs to ask for the right amount of data, which is probably some multiple of the rectangle size and is not necessarily a multiple of 8 or the number of multiplexing rows.

Anyway, the point is that here is a simple and flexible way to parametrize the new mode parameters and manage synchronization, and to provide flexibility and backwards compatibility.

1. The meaning of num_rows remains unchanged. It will always be a multiplexing parameter that all cards need to agree on.

2. The readout card has parameters (prepend "readout_" to all of these, if you want) "row_index", "num_rows", "col_index", "num_cols" that describe the rectangle of interest.

3. The clock card is programmed with a pair of numbers "readout_mult1" and "readout_mult2", that it uses to determine how many words of data to query the RCs for:
N_data = readout_mult1 * readout_mult2.
These numbers are analagous to "num_rows_reported" and "num_cols_reported", but the idea is to kill the idea of "rows" and "columns" in the readout frames because the payload size is not a simple product of n_rows * n_cols. The clock card continues to support "data rate", which indicates the period at which data queries are sent to the RCs. Software will handle the details of getting these factors right. Eventually we will only use one of mult?; the other is for backwards compat.

4. In the RC: at each mux frame, data from the desired rectangle is copied into a large buffer. The RC continues filling the buffer until the CC asks for data. When RC receives a request for N data from the CC, it returns the first N data in the buffer, and resets its write index to the start of the buffer. If the CC doesn't ask for data, and the RC buffer write index increments all the way past the end, the RC takes _no storage action_ in subsequent steps (i.e. it writes all new data to "/dev/null").

Setting up rectangle mode then proceeds as follows. Suppose the RC buffer size is 1600 and I want to readout a 4 x 33 patch of the array:

1. Software computes that 4x33 = 132 words fits into 1600 about 12 times. So it sets "cc data_rate" to 12, and the product of "mult1" and "mult2" to 12 * 132 = 1584. It also sets the readout card row and col rectangle parameters.

2. Software triggers frame acquisition. For the _first_ frame read out, the clock card and readout card may not be synchronized; i.e. the readout card's buffer almost certainly contains ancient, stale data. So the first frame is discarded. But since the first frame query sent the the buffer write index to 0, the data from subsequent frames is fresh and ordered. Because the clock card is asking for N data at the same rate that the RC can N-fill the buffer, this results in a contiguous data stream.

This scheme is simple because:
1. CC has only enough information to ask the readout cards for the right amount of data at the right times.
2. RC only needs to know the rectangle of interest, and it basically lets the CC manage the resetting of the buffer.
3. It is backwards-compatible with existing firmware; mult1 and mult2 can masquerade as num_rows_reported and "8" (num_cols_reported).

The scheme minimizes the amount of information that must be shared between the RC and CC, which provides flexiblity. With "mult" approach, the CC can remain truly ignorant as to what the RC is doing. In this scheme I have emphasized the rectangle mode features, but I also envision it being useful in raw mode firmware, since a lot of the features (such as a large, flexible RAM block in the RC) would seem useful in that context. (For raw mode you would need to add a lock to the buffer so that it could be read out with multiple queries.)

Rectangle Mode Data

2010-08-02T17:23:20Z

174.6.181.2: /* Configuration */

"Rectangle Mode" refers to the sampling of a subset of the full array, typically at high rate. The data from the Readout Cards is packed into output frames efficiently to minimize overhead. Rectangle mode is available in MCE firmware starting with RC 5.0.1 and CC 5.0.0.

= Description of capabilities =

Rectangle mode eliminates strain on the acquisition system during fast acquisitions by eliminating read-out overhead. This means that the 'difficulty' of read-out is a fairly linear function of (sampling frequency) x (number of detectors read out) rather than depending on each.

The hardware limit on readout is roughly 4 MB/s. Since MCE data are 32-bit words, this permits 1e6 detector-samples to be read out per second.

So the maximum number of detectors that can be sampled at some frequency F is
D = (1 MHz) / F

You can't sample at 1 MHz. (Do your squids settle in 1us? Didn't think so.) But you could probably do a detector or two at 250 kHz.

= Firmware =

Rectangle mode requires "version 5" firmware on all cards, and at least RC 5.0.2. Here are links to the recommended SOFs at time of writing:
http://www.phas.ubc.ca/%7Emce/mcedocs/firmware/ac_v05000000_22dec2008.sof
http://www.phas.ubc.ca/%7Emce/mcedocs/firmware/bc_v05000000_22dec2008.sof
http://www.phas.ubc.ca/%7Emce/mcedocs/firmware/cc_v05000002_16jun2009.sof
http://www.phas.ubc.ca/%7Emce/mcedocs/firmware/rc_v05000002_16jun2009.sof

= Script Differences Between v4.x.x and v5.x.x =
The differences between basic frame setup in v4.x.x and v5.x.x are described here. The data frame structure has been made more flexible. A user can read out an arbitrarily rectangular patch of the array. The patch-size is specified using new commands.

*In v4.x.x: The array patch-size is flexible in terms of the number of rows, and the row start index.
wb cc num_rows_reported 41 //power-on default = 41
wb rcs readout_row_index 0 //power-on default = 0

*In v5.x.x: The array patch-size is now also flexible in terms of the number of columns, and the column start index. Note that matching parameters on the Readout Cards and the Clock Card must be consistent.
wb cc num_rows_reported 41 //power-on default as above
wb cc num_cols_reported 8 //power-on default = 8
wb rcs num_rows_reported 41 //power-on default = 41
wb rcs num_cols_reported 8 //power-on default = 8
wb rcs readout_row_index 0 //power-on default as above
wb rcs readout_col_index 0 //power-on default = 0

= Configuration =

The steps for configuring a rectangle mode acquisition are:
# Set the multiplexing rate using num_rows and row_len
# If necessary, adjust row_order to mux the channels of interest (you can only read out rows you are multiplexing...). If you do not intend to re-tune the array with the adjusted num_rows / row_order, you must also change the first entries of rc* adc_offset* to correspond to the desired rows and, for fast SQ2 switching, change row_order on the BAC/BC2.
# Identify a rectangle of detectors on the readout card to read out by setting the RC parameters
#* num_rows_reported, num_cols_reported, readout_row_index, readout_col_index
# Set the size of the readout frame by setting the CC parameters
#* num_rows_reported, num_cols_reported
# Set the CC decimation factor 'data_rate' so that the RC rectangles will stack perfectly into the CC readout frame.
#* e.g. suppose
#** RC num_rows_reported x num_cols_reported = 1x4
#** CC num_rows_reported x num_cols_reported = 33x8
#* then we set
#** CC data_rate = (33 x 8) / (1 x 4) = 66

= Readout limitations =

Rectangle mode configuration should respect the following hardware limits:
* Total data rate must not exceed 4 MB/s
* Read-out frequency must not exceed 20 kHz
* The multiplexing frame rate must not be set so fast that the MCE cannot communicate with the readout cards. This limit is probably in the 100s of kHz.

Other things to remember:
* num_rows >= 2 is encouraged.
* you can't sample a detector faster than it is being visited in the multiplexing cycle.

= Checking MCE configuration using rect_check.py =

Use the utility @rect_check.py@ to verify that data will be / were acquired reasonably.

e.g. To check the state of the MCE (requires python MCE bindings), run
rect_check.py --mce

To analyze the data framing in a runfile:
rect_check.py my_data.run

(Depending on stuff, you might need to invoke rect_check as:
$MAS_PYTHON/rect_check.py
or even
python $MAS_PYTHON/rect_check.py
Sorry.)

The output looks like this:

MCE configuration:
cc_rcs ( cc, rcs_to_report_data ): 60
cc_dec ( cc, data_rate ): 256
cc_nmux ( cc, num_rows ): 33
cc_cmux ( cc, row_len ): 100
cc_nr ( cc, num_rows_reported ): 32
cc_nc ( cc, num_cols_reported ): 8
rc_nmux ( rca, num_rows ): 33
rc_cmux ( rca, row_len ): 100
rc_nr ( rca, num_rows_reported ): 1
rc_nc ( rca, num_cols_reported ): 1
rc_r0 ( rca, readout_row_index ): 0
rc_c0 ( rca, readout_col_index ): 0
Framing:
RC storage per mux cycle: 1
CC read-out per frame: 256
CC decimation: 256
Contiguous? yes
Complete? no
Bizarro? no
Timing:
Mux freq: 15151.00
Mean sampling freq: 15104.00
Read-out freq: 59.00
Data volume:
Frame size (bytes/RC): 1200
Data rate (MB/s/RC): 0.07

The program has notions of what is "reasonable", and these are expressed in the lines "Contiguous?", "Complete?", and "Bizarro?":
* 'Contiguous': all detectors that are read out will be read out at each multiplexing cycle
** When acquiring fast data from a subset of the detectors, at the multiplexing rate, you want "Contiguous = yes".
* 'Complete': all detectors that are multiplexed will be read out
** When acquiring normal data, you probably want "Complete = yes".
* 'Bizarro': the rectangle defined in the RC does not fit evenly into the readout frame.
** e.g. you have set up an 8x8 rectangle but you're reading CC frames that are 33x8. Don't do this. I'm not even sure what you'd get out.

Note that it is possible to be none of Complete, Contiguous, or Bizarro. For example, you could read out the first 33 of every 66 multiplexing cycles for 8 detectors, packing them efficiently into a 33 x 8 read-out frame. Maybe that should be called Bizarro. But right now, it isn't.

It is unlikely, but possible, to be both Contiguous and Complete.

If you are Bizarro, you are neither Contiguous nor Complete. See Venn diagram below.

[[Image:reasonable_rectangles.png]]

= Data acquisition =

Use "mce_run" in the usual way to acquire data. Note that the argument specifying the number of frames is always the number of 'read-out' frames, so for packed data the number of samples per detector will be higher.

e.g. for the above configuration
mce_run my_data 100 1

would acquire 100 read-out frames. Since the read-out frequency is 59 Hz, this would take about 2 seconds. Each readout frame contains 8 x 32 = 256 data from a single detector. So the resulting data stream contains 25600 samples.

= Data file loading =

Recent versions of mce_data.py can extract rectangle mode data from the packed frame structure. E.g. for the above case, extracting the data is as easy as
>>> d = MCEFile('my_data').Read()
>>> d.data.shape
(1, 25600)

= Rather long explanation and justification of structure, from an e-mail =

Yesterday Bryce and I were talking about paths towards a new rectangle readout mode. The following is a recommendation for the rough structure of the system that I think will be easy to manage and provide a maximum flexibility for backwards compatibility and future expansion.

The current rectangle mode is simpler than the planned one because the clock card and readout card need only agree on the number of data to readout, and no addditional waiting/synchronization is required between the clock card and readout card. The readout frame data sizes, internally and externally, are the same for everyone. The new mode is much more complex, because now the readout card has to accumulate several samples from each pixel into its internal buffer before sending the data to the clock card, and the clock card needs to ask for the right amount of data, which is probably some multiple of the rectangle size and is not necessarily a multiple of 8 or the number of multiplexing rows.

Anyway, the point is that here is a simple and flexible way to parametrize the new mode parameters and manage synchronization, and to provide flexibility and backwards compatibility.

1. The meaning of num_rows remains unchanged. It will always be a multiplexing parameter that all cards need to agree on.

2. The readout card has parameters (prepend "readout_" to all of these, if you want) "row_index", "num_rows", "col_index", "num_cols" that describe the rectangle of interest.

3. The clock card is programmed with a pair of numbers "readout_mult1" and "readout_mult2", that it uses to determine how many words of data to query the RCs for:
N_data = readout_mult1 * readout_mult2.
These numbers are analagous to "num_rows_reported" and "num_cols_reported", but the idea is to kill the idea of "rows" and "columns" in the readout frames because the payload size is not a simple product of n_rows * n_cols. The clock card continues to support "data rate", which indicates the period at which data queries are sent to the RCs. Software will handle the details of getting these factors right. Eventually we will only use one of mult?; the other is for backwards compat.

4. In the RC: at each mux frame, data from the desired rectangle is copied into a large buffer. The RC continues filling the buffer until the CC asks for data. When RC receives a request for N data from the CC, it returns the first N data in the buffer, and resets its write index to the start of the buffer. If the CC doesn't ask for data, and the RC buffer write index increments all the way past the end, the RC takes _no storage action_ in subsequent steps (i.e. it writes all new data to "/dev/null").

Setting up rectangle mode then proceeds as follows. Suppose the RC buffer size is 1600 and I want to readout a 4 x 33 patch of the array:

1. Software computes that 4x33 = 132 words fits into 1600 about 12 times. So it sets "cc data_rate" to 12, and the product of "mult1" and "mult2" to 12 * 132 = 1584. It also sets the readout card row and col rectangle parameters.

2. Software triggers frame acquisition. For the _first_ frame read out, the clock card and readout card may not be synchronized; i.e. the readout card's buffer almost certainly contains ancient, stale data. So the first frame is discarded. But since the first frame query sent the the buffer write index to 0, the data from subsequent frames is fresh and ordered. Because the clock card is asking for N data at the same rate that the RC can N-fill the buffer, this results in a contiguous data stream.

This scheme is simple because:
1. CC has only enough information to ask the readout cards for the right amount of data at the right times.
2. RC only needs to know the rectangle of interest, and it basically lets the CC manage the resetting of the buffer.
3. It is backwards-compatible with existing firmware; mult1 and mult2 can masquerade as num_rows_reported and "8" (num_cols_reported).

The scheme minimizes the amount of information that must be shared between the RC and CC, which provides flexiblity. With "mult" approach, the CC can remain truly ignorant as to what the RC is doing. In this scheme I have emphasized the rectangle mode features, but I also envision it being useful in raw mode firmware, since a lot of the features (such as a large, flexible RAM block in the RC) would seem useful in that context. (For raw mode you would need to add a lock to the buffer so that it could be read out with multiple queries.)

PCI card code assembly on Linux

2010-04-30T20:06:30Z

174.6.181.2:

= Assembler and linker =

Motorola/Frescale provide a Windows/DOS assembler/linker, called CLAS. The binaries run quite well under Linux via Wine. An exception is the final hex-file creation step that converts the .lod file to an Intel S-record file for use with an eprom burner; we have our own code for that.

== Install and configure Wine ==

To install wine (the Windows Emulator for Linux):

sudo apt-get install wine
wine

Running wine with no arguments should initialize it and create a place for the fake windows file system it uses.

== Install CLAS ==

The assembler binaries are available from Astrocam [ http://www.astro-cam.com/arcpage.php?txt=software.php#dsp ], or from mhasse [http://e-mode.phas.ubc.ca/~mhasse/mce/arc_pci/clas563.zip]. We need the versions for the DSP56300 family.

This will grab the binaries and put them in the right place:
wget http://e-mode.phas.ubc.ca/~mhasse/mce/arc_pci/clas563.zip
unzip clas563.zip
mv CLAS563 ~/.wine/drive_c/

You could now run the programs via, e.g.
wine ~/.wine/drive_c/CLAS563/BIN/ASM56300.EXE <args>

== Install wrapper scripts ==

The arc_pci repository includes a folder "clash" which contains the lod2s program and some wrapper scripts for launching the CLAS programs. The scripts, and the lod2s program, should probably be copied into the path somewhere. I use ~/bin.

MY_BIN=~/bin
cd src/arc_pci
cd clash
make -C srec lod2s
cp srec/lod2s script/* $MY_BIN

Now the makefiles in, e.g., arc_pci/trunk should work.

The wrapper scripts assume location ~/.wine/drive_c/CLAS563/BIN/ for the CLAS windows binaries. You can override this by putting
export CLAS_BIN=/somewhere/else
in your .bashrc .

PCI card firmware

2010-04-30T20:00:44Z

174.6.181.2:

= Download =

These S-record files should work with any reasonable EPROM programmer:
* U0105 (latest) - http://e-mode.phas.ubc.ca/~mhasse/mce/arc_pci/SDSU_RevU0105.s
* U0104 (stable) - http://e-mode.phas.ubc.ca/~mhasse/mce/arc_pci/SDSU_RevU0104.s

The full source code is available in these archives (it will not compile unless you get the correct assembler and stuff):
* U0105 - http://e-mode.phas.ubc.ca/~mhasse/mce/arc_pci/arc_pci-U0105.tar.gz
* U0104 - http://e-mode.phas.ubc.ca/~mhasse/mce/arc_pci/arc_pci-U0104.tar.gz

The subversion tree can be obtained like this (you may not have access to this repository; contact UBC);
svn checkout svn://e-mode.phas.ubc.ca/arc_pci

It is laid out roughly as follows:
/trunk main line of development
/releases tagged versions of code
/tools debugging tools
/clash stuff for making CLAS work with wine and linux
/docs useful (though maybe not to you) notes

= Other information =

* [[ PCI card bug list ]]
* [[ PCI card hacking ]]
* [[ PCI card code assembly on Linux ]]

= Firmware version notes =

== Latest release: U0105 (2009-06-08) ==

* Backward compatible with U0104.
* Support for MCE STOP commands and commands-on-the-fly.
* Accelerated MCE command code (along with Quiet-RP this increases commanding rate to ~6 kHz).
* Quiet-RP simplifies the protocol for MCE reply handling.
* Low-level improvements:
** CON is done as PCI burst
** Fibre-optic FIFO is emptied with timed read instead of polling.
** Hand-shaking for interrupts instead of host command to clear INTA and HC3.
** Non-interrupt context code disables interrupts when performing PCI transactions.
** Host vector interrupts are otherwise enabled, so PC doesn't have to force with HNMI bit.

== Last stable version: U0104 ==

* Implements quiet transfer mode! Remains backwards compatible with A1.4.
* Fixes the 64k boundary crossing issue
* Moves parameters that enter via interrupt out of registers and into variables
* Version reporting tag-along to RDM command (sending 'VER' to RDM's vector address returns the code version).
* Maximum burst length is reduced to 64 bytes, and is configurable.
* Reset (RST) clears the fibre fifo

== Oldest UBC release: U0103 ==

* Minor modifications of SCUBA2's A1.4 firmware, to improve PCI stability.
* Not compatible with non-realtime systems.

PCI card bug list

2010-03-12T23:39:08Z

174.6.181.2:

This is a list of things to fix in subsequent releases.

== U0105 bugs ==

* STATUS[DMA_RESUME] flag is not cleared after the error is handled. Innocuous.
* STATUS[HST_NFYD] flag is half-used, not cleared. Innocuous.
* Asynchronous MCE reply NFY does not seem to play well with QTS TAIL updates. I.e. some part of the system breaks when a QTS is issued between a CON and the associated NFY REP.
* Does CON_TRANSMIT need to disable PCI interrupts while it is sending the command to MCE? What is the MCE input time-out?

== U0103 bugs ==

* Data packets can fail to HST properly. This was observed in MAS driver, and only recently in DAS. In one instance, DAS failed with a timeout after receiving 1381/2000 frames and could not take data (though other mce commands worked) until the driver was reloaded. This looks to me like lost HST replies. The NFY comes through and the HST gets sent but then everything hangs.

* After some failures (card lookup?), one-behind state emerges. "Reset" doesn't help, nor does "reset_mce". A command must be issued (which returns as unmatched), and then reset dsp will fix the problem. I'm not even sure where the packet is trapped.

== A1.5 / U0103 design issues ==

* PCI reset doesn't empty the FIFO! FIFO is cleared on preamble error. But that means you have will definitely lose an MCE packet if buffer gets out of sync. Reset is 2 ms long; perhaps this is a way of waiting for an entire reply to have been transmitted? A PCI reset routine should clear the FIFO. Either that or a separate vector command, but that seems like overkill.

* Since packet choke is on after a reset, any words in fifo get discarded without a preamble check; this means that FIFO can't be reset before the first MCE command. This is silly; reset should clear the fifo!

Dirfile support

2009-11-20T20:00:36Z

174.6.181.2: /* System configuration */

= Support overview =

As of mas/trunk/:r384, MAS supports writing of dirfiles. For now we are using a very basic dirfile standard, compatible with older versions of KST as well more recent versions of libgetdata.

Consult the sourceforge page for dirfile libraries and documentation:
http://getdata.sourceforge.net/

Although MAS can write dirfiles, we do not currently provide a facility for reading dirfiles into python or IDL, as we do for MCE flatfiles. We recommend that flatfiles be used for tuning and diagnostic data, with dirfiles only used for long acquisitions that will be analyzed off-line or by non-MAS software.

dirfiles are very useful for large, multi-rate data sets. They are not ideal in all circumstances, however.

= Invocation =

To acquire data into a dirfile, pass the --dirfile option to mce_run:

mce_run 10000 dirfile_0526 s --dirfile

This will create a dirfile at $MAS_DATA/dirfile_0526/ , and acquire 10000 frames from all readout card channels.

Please note the following:
* currently the "--dirfile" option must be listed after the three mandatory arguments.
* the number of "samples" requested should be a multiple of 100, as this is the samples/frame ratio specified in the MCE dirfile format file.

= System configuration =

To prevent crazy programs from bringing down the system, Linux controls the maximum number of files that any one shell can have open. The dirfile writer needs to keep a file open for each data channel, and for MCE data this could be ~2000 separate files. If the OS limit is too low to accommodate this, we need to increase it.

To check current OS limit, run ulimit:
mhasse@gamow:~$ ulimit -n
1024

This is too small. To increase it, edit /etc/security/limits.conf . Add / edit lines for item "nofile":

* soft nofile 2048
* hard nofile 4096

After reboot, you should get
mhasse@gamow:~$ ulimit -n
2048

= Dirfile fields =

== Raw fields ==
The dirfile consists of raw fields and derived fields. For MCE data, the raw fields consist of fields from the frame header:
status
frame_ctr
row_len
num_rows_reported
data_rate
address0_ctr
header_version
ramp_value
ramp_addr
num_rows
sync_box_num
runfile_id
userfield

as well as the 32 bit words returned for each detector channel:
tesdatar00c00
tesdatar00c01
...

Note that MAS will determine exactly what channels are returning data and name the fields appropriately. So when reading RC2 only with readout_row_index=4 and num_rows_reported=1, the output fields will be
tesdatar04c08
tesdatar04c09
tesdatar04c10
tesdatar04c11
tesdatar04c12
tesdatar04c13
tesdatar04c14
tesdatar04c15

== Derived fields ==

Based on the [[ data_mode | Data Mode ]] of each readout card, derived signals are extracted from the raw data and made available to libgetdata users. The field extraction and conversion is accomplished according to the same rules used by [[ Python data and runfile modules | mce_data.py ]] and [[ mas_data.pro ]], i.e. the data are scaled to some reference level for that signal. See the discussion in the [[ Data Mode ]] page.

The field names for error, feedback, filtered feedback, and phi0 counter data are, respectively (for r00c00)
error_r00c00
fb_r00c00
filt_r00c00
fj_r00c00

For some conversions, it is necessary to first extract the bitfield from the raw MCE word and then scale it. In these cases an intermediate field will be created, e.g.:
INTER_filt_r00c00

Such fields can be ignored.

== Warning about signed fields in mixed-mode data ==

Currently the dirfile standard does not support sign extension of bit-field data. The result is that signed signals from mixed-mode MCE data will appear as positive numbers, with 'negative' values -D represented by the large positive number 2^n - D. For example, in data mode 4 the error signal is encoded in the lowest-order 14 bits of the MCE word. An error reading of 12 gets encoded as 00000000001100 in binary and would be extracted as 12, but an error reading of -12 would be encoded with the 14 binary digits 11111111110100, and upon extraction would end up as 2^14 - 12 = 16372.

Sign-related issues can be corrected in down-stream software; there is no information loss. But something like KST will not naturally 'do the right thing' here, so the time-stream and spectrum plots will be awful.

Auto-setup (IDL)

2009-10-19T19:44:43Z

174.6.181.2: /* IDL calling parameters */

This script tunes a squid array using the MCE.

== Overview ==

The steps in the tuning are roughly the following:
# "SA ramp": Ramp the SA feedback to select the SA output value of the lock-point (at this stage it is also possible to ramp the SA bias to find a V-phi curve of maximum amplitide).
# "SQ2 servo": Ramp the SQ2 feedback and servo the SA feedback to determine the SA feedback at lock-point.
# "SQ1 servo": For a single SQ1 in each column , ramp the SQ1 feedback and servo the SQ2 feedback to determine the SQ2 feedback at lock-point.
# "SQ1 ramp": For each SQ1, ramp the SQ1 feedback and determine the SA output value correction (ADC_offset) of the lock-point.

In addition to these, there is a final acquisition of a few hundred frames to evaluate the lock of each detector. This is called "frametest". Following this test, and independent of it, the servo parameters of channels known to be dead or problematic are turned off.

== Tuning parameters ==

Most tuning parameters (such as biases, servo gains, SQ1 servo row selections) are defined in the [[experiment.cfg]] file. Contact UBC for help with this.

== Auto script ==

Command-line access is via auto_setup_squids_sh, called like this:
auto_setup_squids_sh <level> <cards>

Where:
* level is one of 0,1,2 and selects what parts of the tuning to perform
** 0 (default): full tuning of all stages of the array.
** 1: perform only the "SQ1 servo" and the "SQ1 ramp"
** 2: perform only the "SQ1 ramp"
* cards is one of s,1,2,3,4, with s the default:
** Note for "s": this will tune all *4* readout cards. 'rcs' support for systems with 2 readout cards is still in development.

== IDL calling parameters ==

The IDL function declaration is (as of r397)
pro auto_setup_squids, COLUMN=column, ROW=row, $
RCs=rcs, $
interactive=interactive, text=text, $
numrows=numrows, note=note, $
ramp_sa_bias=ramp_sa_bias, check_bias=check_bias, $
short=short, quiet=quiet

Parameters that continue to be supported are:
* '''rcs''' (int array): an array of readout card numbers to tune. e.g. [1,2].
* '''note''' (string): a string to pass to the DB registration program (for logging)
* '''short''' (int): causes tuning of only particular stages of the array (useful for correcting minor drifts due to changing thermal or magnetic environments)
* '''ramp_sa_bias''' (switch): forces ramping of the SA bias during the SA ramp step. This can also be controlled through experiment.cfg
* '''check_bias''' (switch): causes the script to check for biasing of the SA and SQ2. If these are off, they are biased on and the script delays execution for a couple of minutes to let the system equilibrate.
* '''quiet''' (int): when set to 1 or 2, reduces text output to the terminal.

Parameters that no one really uses but might still work:
* '''column''' and '''row''' (int array): columns and rows to plot in the final frametest step.

Parameters that are probably broken or definitely meaningless:
* '''interactive'''
* '''text'''

Locking slopes

2009-10-19T19:42:53Z

174.6.181.2: /* Servos */

= MCE/MAS parameters that affect locking slopes =

== Slope controls ==

The following parameters in [[Mce config template system | experiment.cfg]] affect the locking slopes:

* '''sq2servo_gain''': sets the gain of the SQ2 servo and effectively determines the sign of the SA lock slope.
* '''sq1servo_gain''': sets the gain of the SQ1 servo and, with the sq2servo_gain, determines the SQ2 lock slope.
* '''default_servo_i''': determines the I gain of the MCE servo (rc# gaini#), and thus the sign of the locking slopeof the SQ1.

== Plotting controls ==

The following parameters affect plotting of locking slopes in IDL, as well as the choice of starting feedbacks for the next stage of servos:

* '''sq2servo_slope''' in experiment.cfg: (+1 or -1) the sign of the slope in the SQ2 servo plots to which we expect to lock.
* '''sq1servo_slope''' in experiment.cfg: (+1 or -1) the sign of the slope in the SQ1 servo plots to which we expect to lock.

In newer versions of auto-tuning, these parameters are ignored and the locking slope is determined from the signs of the servo gains.

== Consistency between locking and plotting ==

The above slope and plotting parameters should satisfy the following:
* sq2servo_slope = -sign( sq2servo_gain / sq1servo_gain)
* sq1servo_slope = -sign( sq1servo_gain / default_servo_i)

These relations follow from the discussion in the sections below.

Violation of the above will likely only result in misleading plots, and not bad locking. Misleading plots can indirectly lead to a user setting up bad locking, however; so keep it clean.

'''The sa_points.ps plot may be misleading''': older auto-tuning code (still in use all over the place) plots the positive-slope lock-points, regardless of any other parameters. Newer auto-tuning code resolves this.

= MCE and tuning servos =

== Servos ==

We will need to deal with three servos. One is the "MCE servo", which is the fast servoing of the TES signal via adjustment of the SQ1 feedback. The other two are the auto-tuning SQ2 and SQ1 servos, described [[auto_setup_squids.pro | here]].

In all cases, the servo is computed by measuring an error (<math>\Delta</math> y), multiplying by a gain (g), and using this to correct the output (x):
<math>\Delta x = g~\Delta y</math>

In the case of the MCE PID loop servo, we run with an I term only so we actually calculate
<math>x = P ~\sum \Delta y</math>

but differentially this is exactly <math>\Delta x = P * \Delta y</math>.

== MCE/MAS and sneaky signs ==

There are no sneaky, extra signs in any of the MCE/MAS servo calculations. This means that given an error (dy) and a gain g, we calculate the change in the feedback (dx) as:
<math>dx = g~dy</math>

and not
<math>dx = -g~dy</math>

There is a sneaky gain in the MCE servo, though; once the default_servo_i is set, the DAC value of the feedback is actually calculated as
<math>\Delta FB = (I~dy) / 2^{12}</math>

But there's no sneaky sign. Just a sneaky gain.

= The meaning of the plots =

== Definition of a lock point ==

Let the feedback signals on the three stages (in one channel) be <math>x_{SA}, x_{S2}, x_{S1}</math>, and let the error signal be y. Let s be the signal from the TES. The error signal is obviously a function of all 4 numbers:
<math>y = f(x_{SA}, x_{S2}, x_{S1}, t)</math>

But for small variations about one point in this space, we can expand in differentials
<math> dy = a_1 dx_{SA} + a_2 dx_{S2} + a_3 dx_{S1} + a_4 s</math>
but it is convenient to reparametrize this to resemble the chain of amplifiers:
<math> dy = m_{SA}(dx_{SA} - m_{S2}(dx_{S2} - m_{S1}(dx_{S1} - m_{TES} s)</math>

So the 'slopes' m account for the instantaneous gain between the stages at the lock point.

== Open loop ramps ==

Note that lock points are not ultimately chosen as a result of the open loop ramps. However, it is important to understand the relationship between what the open loop ramps and the settings for the auto-tuning and MCE servos.

For the "SA ramp", we hold <math>x_{S2}, x_{S1}</math> and s constant, and plot y vs. <math>x_{SA}</math>. Thus identifying a lock point permits the measurement of
<math>dy / dx_{SA} = m_{SA}</math>

For the "SQ1 ramp", we hold <math>x_{SA}, x_{S2}</math> and s constant, and plot y vs. <math>x_{S1}</math>. A slope measurement on this curve is a measurement of
<math>dy / dx_{S1} = m_{SA} m_{S2} m_{S1}</math>

== Servos ==

In servos, we ramp one stage's x while adjusting another x to keep the error at zero.

Thus in the "SQ2 servo" step (where SQ2 FB is ramped and SQA FB adjusted) we have:
<math>dx_{SA} = m_{S2} dx_{S2}</math>

However, this can only be accomplished via measurement of the error
<math>\Delta y = -m_{SA} m_{S2} \Delta x_{S2}</math>
so operationally we apply
<math>\Delta x_{SA} = - \Delta y / m_{SA}</math>

This tells us the ideal gain of the SQ2 servo:
<math>g_{S2} = -1/m_{SA}</math>

In the "SQ1 servo" step (where SQ1 FB is ramped and SQ2 FB adjusted) we have:
<math>dx_{S2} = m_{S1} dx_{S1}</math>
so we measure
<math>\Delta y = m_{SA} m_{S2} m_{S1} \Delta x_{S1}</math>
and apply
<math>\Delta x_{S2} = \Delta y / (m_{SA} m_{S2})</math>.

so the ideal gain of the SQ1 servo is
<math>g_{S1} = 1/(m_{SA} m_{S2})</math>.

== MCE servo I parameter ==

When the MCE servo is activated, it cancels changes in TES current by adjusting the SQ1 FB:
<math>dx_{S1} = m_{TES} ds</math>

i.e.
<math>\Delta x_{S1} = -\Delta y / (m_{S1} m_{S2} m_{S3})</math>
and ideal gain is thus
<math>I_{servo} = -1 / (m_{S1} m_{S2} m_{S3})</math>

== Consequences for tuning parameters ==

We have expressed, above, the ideal gains for the servos as a function of the slopes of the ramp and servo curves from auto-tuning. Solving these for the slopes as a function of the ideal gains yields the locking slopes' dependence on the signs of the gains:

=== SA locking point ===

The locking slope (on the SA ramp) is determined by the gain in the SQ2 servo:
<math>sgn(m_{SA}) = -sgn(1/g_{S2})</math>

=== SQ2 locking point ===

The locking slope (on the SQ2 servo curve) is determined by the ratio of the gains in the SQ1 and SQ2 servos:
<math>sgn(m_{S2}) = -sgn(g_{S2}/g_{S1})</math>

=== SQ1 locking point ===

The locking slope on the '''SQ1 servo curve''' is determined by the ratio of the gains in the SQ1 servo and the MCE servo I term:
<math>sgn(m_{S1}) = -sgn(g_{S1}/I)</math>

But the sign of the locking slope on the '''SQ1 ramp''' curves is simply:
<math>sgn(m_{SA} m_{S2} m_{S1}) = - sgn(1/I)</math>

Locking slopes

2009-10-19T19:41:50Z

174.6.181.2: /* Plotting controls */

= MCE/MAS parameters that affect locking slopes =

== Slope controls ==

The following parameters in [[Mce config template system | experiment.cfg]] affect the locking slopes:

* '''sq2servo_gain''': sets the gain of the SQ2 servo and effectively determines the sign of the SA lock slope.
* '''sq1servo_gain''': sets the gain of the SQ1 servo and, with the sq2servo_gain, determines the SQ2 lock slope.
* '''default_servo_i''': determines the I gain of the MCE servo (rc# gaini#), and thus the sign of the locking slopeof the SQ1.

== Plotting controls ==

The following parameters affect plotting of locking slopes in IDL, as well as the choice of starting feedbacks for the next stage of servos:

* '''sq2servo_slope''' in experiment.cfg: (+1 or -1) the sign of the slope in the SQ2 servo plots to which we expect to lock.
* '''sq1servo_slope''' in experiment.cfg: (+1 or -1) the sign of the slope in the SQ1 servo plots to which we expect to lock.

In newer versions of auto-tuning, these parameters are ignored and the locking slope is determined from the signs of the servo gains.

== Consistency between locking and plotting ==

The above slope and plotting parameters should satisfy the following:
* sq2servo_slope = -sign( sq2servo_gain / sq1servo_gain)
* sq1servo_slope = -sign( sq1servo_gain / default_servo_i)

These relations follow from the discussion in the sections below.

Violation of the above will likely only result in misleading plots, and not bad locking. Misleading plots can indirectly lead to a user setting up bad locking, however; so keep it clean.

'''The sa_points.ps plot may be misleading''': older auto-tuning code (still in use all over the place) plots the positive-slope lock-points, regardless of any other parameters. Newer auto-tuning code resolves this.

= MCE and tuning servos =

== Servos ==

We will need to deal with three servos. One is the "MCE servo", which is the fast servoing of the TES signal via adjustment of the SQ1 feedback. The other two are the auto-tuning SQ2 and SQ1 servos, described [here | auto_setup_squids.pro].

In all cases, the servo is computed by measuring an error (<math>\Delta</math> y), multiplying by a gain (g), and using this to correct the output (x):
<math>\Delta x = g~\Delta y</math>

In the case of the MCE PID loop servo, we run with an I term only so we actually calculate
<math>x = P ~\sum \Delta y</math>

but differentially this is exactly <math>\Delta x = P * \Delta y</math>.

== MCE/MAS and sneaky signs ==

There are no sneaky, extra signs in any of the MCE/MAS servo calculations. This means that given an error (dy) and a gain g, we calculate the change in the feedback (dx) as:
<math>dx = g~dy</math>

and not
<math>dx = -g~dy</math>

There is a sneaky gain in the MCE servo, though; once the default_servo_i is set, the DAC value of the feedback is actually calculated as
<math>\Delta FB = (I~dy) / 2^{12}</math>

But there's no sneaky sign. Just a sneaky gain.

= The meaning of the plots =

== Definition of a lock point ==

Let the feedback signals on the three stages (in one channel) be <math>x_{SA}, x_{S2}, x_{S1}</math>, and let the error signal be y. Let s be the signal from the TES. The error signal is obviously a function of all 4 numbers:
<math>y = f(x_{SA}, x_{S2}, x_{S1}, t)</math>

But for small variations about one point in this space, we can expand in differentials
<math> dy = a_1 dx_{SA} + a_2 dx_{S2} + a_3 dx_{S1} + a_4 s</math>
but it is convenient to reparametrize this to resemble the chain of amplifiers:
<math> dy = m_{SA}(dx_{SA} - m_{S2}(dx_{S2} - m_{S1}(dx_{S1} - m_{TES} s)</math>

So the 'slopes' m account for the instantaneous gain between the stages at the lock point.

== Open loop ramps ==

Note that lock points are not ultimately chosen as a result of the open loop ramps. However, it is important to understand the relationship between what the open loop ramps and the settings for the auto-tuning and MCE servos.

For the "SA ramp", we hold <math>x_{S2}, x_{S1}</math> and s constant, and plot y vs. <math>x_{SA}</math>. Thus identifying a lock point permits the measurement of
<math>dy / dx_{SA} = m_{SA}</math>

For the "SQ1 ramp", we hold <math>x_{SA}, x_{S2}</math> and s constant, and plot y vs. <math>x_{S1}</math>. A slope measurement on this curve is a measurement of
<math>dy / dx_{S1} = m_{SA} m_{S2} m_{S1}</math>

== Servos ==

In servos, we ramp one stage's x while adjusting another x to keep the error at zero.

Thus in the "SQ2 servo" step (where SQ2 FB is ramped and SQA FB adjusted) we have:
<math>dx_{SA} = m_{S2} dx_{S2}</math>

However, this can only be accomplished via measurement of the error
<math>\Delta y = -m_{SA} m_{S2} \Delta x_{S2}</math>
so operationally we apply
<math>\Delta x_{SA} = - \Delta y / m_{SA}</math>

This tells us the ideal gain of the SQ2 servo:
<math>g_{S2} = -1/m_{SA}</math>

In the "SQ1 servo" step (where SQ1 FB is ramped and SQ2 FB adjusted) we have:
<math>dx_{S2} = m_{S1} dx_{S1}</math>
so we measure
<math>\Delta y = m_{SA} m_{S2} m_{S1} \Delta x_{S1}</math>
and apply
<math>\Delta x_{S2} = \Delta y / (m_{SA} m_{S2})</math>.

so the ideal gain of the SQ1 servo is
<math>g_{S1} = 1/(m_{SA} m_{S2})</math>.

== MCE servo I parameter ==

When the MCE servo is activated, it cancels changes in TES current by adjusting the SQ1 FB:
<math>dx_{S1} = m_{TES} ds</math>

i.e.
<math>\Delta x_{S1} = -\Delta y / (m_{S1} m_{S2} m_{S3})</math>
and ideal gain is thus
<math>I_{servo} = -1 / (m_{S1} m_{S2} m_{S3})</math>

== Consequences for tuning parameters ==

We have expressed, above, the ideal gains for the servos as a function of the slopes of the ramp and servo curves from auto-tuning. Solving these for the slopes as a function of the ideal gains yields the locking slopes' dependence on the signs of the gains:

=== SA locking point ===

The locking slope (on the SA ramp) is determined by the gain in the SQ2 servo:
<math>sgn(m_{SA}) = -sgn(1/g_{S2})</math>

=== SQ2 locking point ===

The locking slope (on the SQ2 servo curve) is determined by the ratio of the gains in the SQ1 and SQ2 servos:
<math>sgn(m_{S2}) = -sgn(g_{S2}/g_{S1})</math>

=== SQ1 locking point ===

The locking slope on the '''SQ1 servo curve''' is determined by the ratio of the gains in the SQ1 servo and the MCE servo I term:
<math>sgn(m_{S1}) = -sgn(g_{S1}/I)</math>

But the sign of the locking slope on the '''SQ1 ramp''' curves is simply:
<math>sgn(m_{SA} m_{S2} m_{S1}) = - sgn(1/I)</math>

Locking slopes

2009-10-19T19:41:17Z

174.6.181.2: /* SQ1 locking point */

= MCE/MAS parameters that affect locking slopes =

== Slope controls ==

The following parameters in [[Mce config template system | experiment.cfg]] affect the locking slopes:

* '''sq2servo_gain''': sets the gain of the SQ2 servo and effectively determines the sign of the SA lock slope.
* '''sq1servo_gain''': sets the gain of the SQ1 servo and, with the sq2servo_gain, determines the SQ2 lock slope.
* '''default_servo_i''': determines the I gain of the MCE servo (rc# gaini#), and thus the sign of the locking slopeof the SQ1.

== Plotting controls ==

The following parameters affect plotting of locking slopes in IDL, as well as the choice of starting feedbacks for the next stage of servos:

* sq2servo_slope in experiment.cfg: (+1 or -1) the sign of the slope in the SQ2 servo plots to which we expect to lock.
* sq1servo_slope in experiment.cfg: (+1 or -1) the sign of the slope in the SQ1 servo plots to which we expect to lock.

In newer versions of auto-tuning, these parameters are ignored and the locking slope is determined from the signs of the servo gains.

== Consistency between locking and plotting ==

The above slope and plotting parameters should satisfy the following:
* sq2servo_slope = -sign( sq2servo_gain / sq1servo_gain)
* sq1servo_slope = -sign( sq1servo_gain / default_servo_i)

These relations follow from the discussion in the sections below.

Violation of the above will likely only result in misleading plots, and not bad locking. Misleading plots can indirectly lead to a user setting up bad locking, however; so keep it clean.

'''The sa_points.ps plot may be misleading''': older auto-tuning code (still in use all over the place) plots the positive-slope lock-points, regardless of any other parameters. Newer auto-tuning code resolves this.

= MCE and tuning servos =

== Servos ==

We will need to deal with three servos. One is the "MCE servo", which is the fast servoing of the TES signal via adjustment of the SQ1 feedback. The other two are the auto-tuning SQ2 and SQ1 servos, described [here | auto_setup_squids.pro].

In all cases, the servo is computed by measuring an error (<math>\Delta</math> y), multiplying by a gain (g), and using this to correct the output (x):
<math>\Delta x = g~\Delta y</math>

In the case of the MCE PID loop servo, we run with an I term only so we actually calculate
<math>x = P ~\sum \Delta y</math>

but differentially this is exactly <math>\Delta x = P * \Delta y</math>.

== MCE/MAS and sneaky signs ==

There are no sneaky, extra signs in any of the MCE/MAS servo calculations. This means that given an error (dy) and a gain g, we calculate the change in the feedback (dx) as:
<math>dx = g~dy</math>

and not
<math>dx = -g~dy</math>

There is a sneaky gain in the MCE servo, though; once the default_servo_i is set, the DAC value of the feedback is actually calculated as
<math>\Delta FB = (I~dy) / 2^{12}</math>

But there's no sneaky sign. Just a sneaky gain.

= The meaning of the plots =

== Definition of a lock point ==

Let the feedback signals on the three stages (in one channel) be <math>x_{SA}, x_{S2}, x_{S1}</math>, and let the error signal be y. Let s be the signal from the TES. The error signal is obviously a function of all 4 numbers:
<math>y = f(x_{SA}, x_{S2}, x_{S1}, t)</math>

But for small variations about one point in this space, we can expand in differentials
<math> dy = a_1 dx_{SA} + a_2 dx_{S2} + a_3 dx_{S1} + a_4 s</math>
but it is convenient to reparametrize this to resemble the chain of amplifiers:
<math> dy = m_{SA}(dx_{SA} - m_{S2}(dx_{S2} - m_{S1}(dx_{S1} - m_{TES} s)</math>

So the 'slopes' m account for the instantaneous gain between the stages at the lock point.

== Open loop ramps ==

Note that lock points are not ultimately chosen as a result of the open loop ramps. However, it is important to understand the relationship between what the open loop ramps and the settings for the auto-tuning and MCE servos.

For the "SA ramp", we hold <math>x_{S2}, x_{S1}</math> and s constant, and plot y vs. <math>x_{SA}</math>. Thus identifying a lock point permits the measurement of
<math>dy / dx_{SA} = m_{SA}</math>

For the "SQ1 ramp", we hold <math>x_{SA}, x_{S2}</math> and s constant, and plot y vs. <math>x_{S1}</math>. A slope measurement on this curve is a measurement of
<math>dy / dx_{S1} = m_{SA} m_{S2} m_{S1}</math>

== Servos ==

In servos, we ramp one stage's x while adjusting another x to keep the error at zero.

Thus in the "SQ2 servo" step (where SQ2 FB is ramped and SQA FB adjusted) we have:
<math>dx_{SA} = m_{S2} dx_{S2}</math>

However, this can only be accomplished via measurement of the error
<math>\Delta y = -m_{SA} m_{S2} \Delta x_{S2}</math>
so operationally we apply
<math>\Delta x_{SA} = - \Delta y / m_{SA}</math>

This tells us the ideal gain of the SQ2 servo:
<math>g_{S2} = -1/m_{SA}</math>

In the "SQ1 servo" step (where SQ1 FB is ramped and SQ2 FB adjusted) we have:
<math>dx_{S2} = m_{S1} dx_{S1}</math>
so we measure
<math>\Delta y = m_{SA} m_{S2} m_{S1} \Delta x_{S1}</math>
and apply
<math>\Delta x_{S2} = \Delta y / (m_{SA} m_{S2})</math>.

so the ideal gain of the SQ1 servo is
<math>g_{S1} = 1/(m_{SA} m_{S2})</math>.

== MCE servo I parameter ==

When the MCE servo is activated, it cancels changes in TES current by adjusting the SQ1 FB:
<math>dx_{S1} = m_{TES} ds</math>

i.e.
<math>\Delta x_{S1} = -\Delta y / (m_{S1} m_{S2} m_{S3})</math>
and ideal gain is thus
<math>I_{servo} = -1 / (m_{S1} m_{S2} m_{S3})</math>

== Consequences for tuning parameters ==

We have expressed, above, the ideal gains for the servos as a function of the slopes of the ramp and servo curves from auto-tuning. Solving these for the slopes as a function of the ideal gains yields the locking slopes' dependence on the signs of the gains:

=== SA locking point ===

The locking slope (on the SA ramp) is determined by the gain in the SQ2 servo:
<math>sgn(m_{SA}) = -sgn(1/g_{S2})</math>

=== SQ2 locking point ===

The locking slope (on the SQ2 servo curve) is determined by the ratio of the gains in the SQ1 and SQ2 servos:
<math>sgn(m_{S2}) = -sgn(g_{S2}/g_{S1})</math>

=== SQ1 locking point ===

The locking slope on the '''SQ1 servo curve''' is determined by the ratio of the gains in the SQ1 servo and the MCE servo I term:
<math>sgn(m_{S1}) = -sgn(g_{S1}/I)</math>

But the sign of the locking slope on the '''SQ1 ramp''' curves is simply:
<math>sgn(m_{SA} m_{S2} m_{S1}) = - sgn(1/I)</math>

Locking slopes

2009-10-19T19:37:27Z

174.6.181.2: /* Consistency between locking and plotting */

= MCE/MAS parameters that affect locking slopes =

== Slope controls ==

The following parameters in [[Mce config template system | experiment.cfg]] affect the locking slopes:

* '''sq2servo_gain''': sets the gain of the SQ2 servo and effectively determines the sign of the SA lock slope.
* '''sq1servo_gain''': sets the gain of the SQ1 servo and, with the sq2servo_gain, determines the SQ2 lock slope.
* '''default_servo_i''': determines the I gain of the MCE servo (rc# gaini#), and thus the sign of the locking slopeof the SQ1.

== Plotting controls ==

The following parameters affect plotting of locking slopes in IDL, as well as the choice of starting feedbacks for the next stage of servos:

* sq2servo_slope in experiment.cfg: (+1 or -1) the sign of the slope in the SQ2 servo plots to which we expect to lock.
* sq1servo_slope in experiment.cfg: (+1 or -1) the sign of the slope in the SQ1 servo plots to which we expect to lock.

In newer versions of auto-tuning, these parameters are ignored and the locking slope is determined from the signs of the servo gains.

== Consistency between locking and plotting ==

The above slope and plotting parameters should satisfy the following:
* sq2servo_slope = -sign( sq2servo_gain / sq1servo_gain)
* sq1servo_slope = -sign( sq1servo_gain / default_servo_i)

These relations follow from the discussion in the sections below.

Violation of the above will likely only result in misleading plots, and not bad locking. Misleading plots can indirectly lead to a user setting up bad locking, however; so keep it clean.

'''The sa_points.ps plot may be misleading''': older auto-tuning code (still in use all over the place) plots the positive-slope lock-points, regardless of any other parameters. Newer auto-tuning code resolves this.

= MCE and tuning servos =

== Servos ==

We will need to deal with three servos. One is the "MCE servo", which is the fast servoing of the TES signal via adjustment of the SQ1 feedback. The other two are the auto-tuning SQ2 and SQ1 servos, described [here | auto_setup_squids.pro].

In all cases, the servo is computed by measuring an error (<math>\Delta</math> y), multiplying by a gain (g), and using this to correct the output (x):
<math>\Delta x = g~\Delta y</math>

In the case of the MCE PID loop servo, we run with an I term only so we actually calculate
<math>x = P ~\sum \Delta y</math>

but differentially this is exactly <math>\Delta x = P * \Delta y</math>.

== MCE/MAS and sneaky signs ==

There are no sneaky, extra signs in any of the MCE/MAS servo calculations. This means that given an error (dy) and a gain g, we calculate the change in the feedback (dx) as:
<math>dx = g~dy</math>

and not
<math>dx = -g~dy</math>

There is a sneaky gain in the MCE servo, though; once the default_servo_i is set, the DAC value of the feedback is actually calculated as
<math>\Delta FB = (I~dy) / 2^{12}</math>

But there's no sneaky sign. Just a sneaky gain.

= The meaning of the plots =

== Definition of a lock point ==

Let the feedback signals on the three stages (in one channel) be <math>x_{SA}, x_{S2}, x_{S1}</math>, and let the error signal be y. Let s be the signal from the TES. The error signal is obviously a function of all 4 numbers:
<math>y = f(x_{SA}, x_{S2}, x_{S1}, t)</math>

But for small variations about one point in this space, we can expand in differentials
<math> dy = a_1 dx_{SA} + a_2 dx_{S2} + a_3 dx_{S1} + a_4 s</math>
but it is convenient to reparametrize this to resemble the chain of amplifiers:
<math> dy = m_{SA}(dx_{SA} - m_{S2}(dx_{S2} - m_{S1}(dx_{S1} - m_{TES} s)</math>

So the 'slopes' m account for the instantaneous gain between the stages at the lock point.

== Open loop ramps ==

Note that lock points are not ultimately chosen as a result of the open loop ramps. However, it is important to understand the relationship between what the open loop ramps and the settings for the auto-tuning and MCE servos.

For the "SA ramp", we hold <math>x_{S2}, x_{S1}</math> and s constant, and plot y vs. <math>x_{SA}</math>. Thus identifying a lock point permits the measurement of
<math>dy / dx_{SA} = m_{SA}</math>

For the "SQ1 ramp", we hold <math>x_{SA}, x_{S2}</math> and s constant, and plot y vs. <math>x_{S1}</math>. A slope measurement on this curve is a measurement of
<math>dy / dx_{S1} = m_{SA} m_{S2} m_{S1}</math>

== Servos ==

In servos, we ramp one stage's x while adjusting another x to keep the error at zero.

Thus in the "SQ2 servo" step (where SQ2 FB is ramped and SQA FB adjusted) we have:
<math>dx_{SA} = m_{S2} dx_{S2}</math>

However, this can only be accomplished via measurement of the error
<math>\Delta y = -m_{SA} m_{S2} \Delta x_{S2}</math>
so operationally we apply
<math>\Delta x_{SA} = - \Delta y / m_{SA}</math>

This tells us the ideal gain of the SQ2 servo:
<math>g_{S2} = -1/m_{SA}</math>

In the "SQ1 servo" step (where SQ1 FB is ramped and SQ2 FB adjusted) we have:
<math>dx_{S2} = m_{S1} dx_{S1}</math>
so we measure
<math>\Delta y = m_{SA} m_{S2} m_{S1} \Delta x_{S1}</math>
and apply
<math>\Delta x_{S2} = \Delta y / (m_{SA} m_{S2})</math>.

so the ideal gain of the SQ1 servo is
<math>g_{S1} = 1/(m_{SA} m_{S2})</math>.

== MCE servo I parameter ==

When the MCE servo is activated, it cancels changes in TES current by adjusting the SQ1 FB:
<math>dx_{S1} = m_{TES} ds</math>

i.e.
<math>\Delta x_{S1} = -\Delta y / (m_{S1} m_{S2} m_{S3})</math>
and ideal gain is thus
<math>I_{servo} = -1 / (m_{S1} m_{S2} m_{S3})</math>

== Consequences for tuning parameters ==

We have expressed, above, the ideal gains for the servos as a function of the slopes of the ramp and servo curves from auto-tuning. Solving these for the slopes as a function of the ideal gains yields the locking slopes' dependence on the signs of the gains:

=== SA locking point ===

The locking slope (on the SA ramp) is determined by the gain in the SQ2 servo:
<math>sgn(m_{SA}) = -sgn(1/g_{S2})</math>

=== SQ2 locking point ===

The locking slope (on the SQ2 servo curve) is determined by the ratio of the gains in the SQ1 and SQ2 servos:
<math>sgn(m_{S2}) = -sgn(g_{S2}/g_{S1})</math>

=== SQ1 locking point ===

The locking slope on the SQ1 servo curve is determined by the ratio of the gains in the SQ1 servo and the MCE servo I term:
<math>sgn(m_{S1}) = -sgn(g_{S1}/I)</math>

But the sign of the locking slope on the SQ1 ramp curve is simply:
<math>sgn(m_{SA} m_{S2} m_{S1}) = - sgn(1/I)</math>

Locking slopes

2009-10-19T19:36:34Z

174.6.181.2: /* Plotting controls */

= MCE/MAS parameters that affect locking slopes =

== Slope controls ==

The following parameters in [[Mce config template system | experiment.cfg]] affect the locking slopes:

* '''sq2servo_gain''': sets the gain of the SQ2 servo and effectively determines the sign of the SA lock slope.
* '''sq1servo_gain''': sets the gain of the SQ1 servo and, with the sq2servo_gain, determines the SQ2 lock slope.
* '''default_servo_i''': determines the I gain of the MCE servo (rc# gaini#), and thus the sign of the locking slopeof the SQ1.

== Plotting controls ==

The following parameters affect plotting of locking slopes in IDL, as well as the choice of starting feedbacks for the next stage of servos:

* sq2servo_slope in experiment.cfg: (+1 or -1) the sign of the slope in the SQ2 servo plots to which we expect to lock.
* sq1servo_slope in experiment.cfg: (+1 or -1) the sign of the slope in the SQ1 servo plots to which we expect to lock.

In newer versions of auto-tuning, these parameters are ignored and the locking slope is determined from the signs of the servo gains.

== Consistency between locking and plotting ==

The above slope and plotting parameters should satisfy the following:
* sq2servo_slope = -sign( sq2servo_gain / sq1servo_gain)
* sq1servo_slope = -sign( sq1servo_gain / default_servo_i)

These relations follow from the discussion in the sections below.

Violation of the above will likely only result in misleading plots, and not bad locking. Misleading plots can indirectly lead to a user setting up bad locking, however; so keep it clean.

'''The sa_points.ps plot may be misleading''': the code (at time of writing) plots the positive-slope lock-points, regardless of any other parameters.

Future code will eliminate sq2servo_slope and sq1servo_slope and determine all locking slopes from the servo gains...

= MCE and tuning servos =

== Servos ==

We will need to deal with three servos. One is the "MCE servo", which is the fast servoing of the TES signal via adjustment of the SQ1 feedback. The other two are the auto-tuning SQ2 and SQ1 servos, described [here | auto_setup_squids.pro].

In all cases, the servo is computed by measuring an error (<math>\Delta</math> y), multiplying by a gain (g), and using this to correct the output (x):
<math>\Delta x = g~\Delta y</math>

In the case of the MCE PID loop servo, we run with an I term only so we actually calculate
<math>x = P ~\sum \Delta y</math>

but differentially this is exactly <math>\Delta x = P * \Delta y</math>.

== MCE/MAS and sneaky signs ==

There are no sneaky, extra signs in any of the MCE/MAS servo calculations. This means that given an error (dy) and a gain g, we calculate the change in the feedback (dx) as:
<math>dx = g~dy</math>

and not
<math>dx = -g~dy</math>

There is a sneaky gain in the MCE servo, though; once the default_servo_i is set, the DAC value of the feedback is actually calculated as
<math>\Delta FB = (I~dy) / 2^{12}</math>

But there's no sneaky sign. Just a sneaky gain.

= The meaning of the plots =

== Definition of a lock point ==

Let the feedback signals on the three stages (in one channel) be <math>x_{SA}, x_{S2}, x_{S1}</math>, and let the error signal be y. Let s be the signal from the TES. The error signal is obviously a function of all 4 numbers:
<math>y = f(x_{SA}, x_{S2}, x_{S1}, t)</math>

But for small variations about one point in this space, we can expand in differentials
<math> dy = a_1 dx_{SA} + a_2 dx_{S2} + a_3 dx_{S1} + a_4 s</math>
but it is convenient to reparametrize this to resemble the chain of amplifiers:
<math> dy = m_{SA}(dx_{SA} - m_{S2}(dx_{S2} - m_{S1}(dx_{S1} - m_{TES} s)</math>

So the 'slopes' m account for the instantaneous gain between the stages at the lock point.

== Open loop ramps ==

Note that lock points are not ultimately chosen as a result of the open loop ramps. However, it is important to understand the relationship between what the open loop ramps and the settings for the auto-tuning and MCE servos.

For the "SA ramp", we hold <math>x_{S2}, x_{S1}</math> and s constant, and plot y vs. <math>x_{SA}</math>. Thus identifying a lock point permits the measurement of
<math>dy / dx_{SA} = m_{SA}</math>

For the "SQ1 ramp", we hold <math>x_{SA}, x_{S2}</math> and s constant, and plot y vs. <math>x_{S1}</math>. A slope measurement on this curve is a measurement of
<math>dy / dx_{S1} = m_{SA} m_{S2} m_{S1}</math>

== Servos ==

In servos, we ramp one stage's x while adjusting another x to keep the error at zero.

Thus in the "SQ2 servo" step (where SQ2 FB is ramped and SQA FB adjusted) we have:
<math>dx_{SA} = m_{S2} dx_{S2}</math>

However, this can only be accomplished via measurement of the error
<math>\Delta y = -m_{SA} m_{S2} \Delta x_{S2}</math>
so operationally we apply
<math>\Delta x_{SA} = - \Delta y / m_{SA}</math>

This tells us the ideal gain of the SQ2 servo:
<math>g_{S2} = -1/m_{SA}</math>

In the "SQ1 servo" step (where SQ1 FB is ramped and SQ2 FB adjusted) we have:
<math>dx_{S2} = m_{S1} dx_{S1}</math>
so we measure
<math>\Delta y = m_{SA} m_{S2} m_{S1} \Delta x_{S1}</math>
and apply
<math>\Delta x_{S2} = \Delta y / (m_{SA} m_{S2})</math>.

so the ideal gain of the SQ1 servo is
<math>g_{S1} = 1/(m_{SA} m_{S2})</math>.

== MCE servo I parameter ==

When the MCE servo is activated, it cancels changes in TES current by adjusting the SQ1 FB:
<math>dx_{S1} = m_{TES} ds</math>

i.e.
<math>\Delta x_{S1} = -\Delta y / (m_{S1} m_{S2} m_{S3})</math>
and ideal gain is thus
<math>I_{servo} = -1 / (m_{S1} m_{S2} m_{S3})</math>

== Consequences for tuning parameters ==

We have expressed, above, the ideal gains for the servos as a function of the slopes of the ramp and servo curves from auto-tuning. Solving these for the slopes as a function of the ideal gains yields the locking slopes' dependence on the signs of the gains:

=== SA locking point ===

The locking slope (on the SA ramp) is determined by the gain in the SQ2 servo:
<math>sgn(m_{SA}) = -sgn(1/g_{S2})</math>

=== SQ2 locking point ===

The locking slope (on the SQ2 servo curve) is determined by the ratio of the gains in the SQ1 and SQ2 servos:
<math>sgn(m_{S2}) = -sgn(g_{S2}/g_{S1})</math>

=== SQ1 locking point ===

The locking slope on the SQ1 servo curve is determined by the ratio of the gains in the SQ1 servo and the MCE servo I term:
<math>sgn(m_{S1}) = -sgn(g_{S1}/I)</math>

But the sign of the locking slope on the SQ1 ramp curve is simply:
<math>sgn(m_{SA} m_{S2} m_{S1}) = - sgn(1/I)</math>

PCI card firmware

2009-10-02T08:28:51Z

174.6.181.2:

= Download =

* U0105 (latest) - http://e-mode.phas.ubc.ca/~mhasse/mce/SDSU_RevU0105.s
* U0104 (stable) - http://e-mode.phas.ubc.ca/~mhasse/mce/SDSU_RevU0104.s

= Misc. =

* [[ PCI card bug list ]]
* [[ PCI card hacking ]]
* [[ PCI card code assembly on Linux ]]

= Firmware version notes =

== Latest release: U0105 (2009-06-08) ==

* Backward compatible with U0104.
* Support for MCE STOP commands and commands-on-the-fly.
* Accelerated MCE command code (along with Quiet-RP this increases commanding rate to ~6 kHz).
* Quiet-RP simplifies the protocol for MCE reply handling.
* Low-level improvements:
** CON is done as PCI burst
** Fibre-optic FIFO is emptied with timed read instead of polling.
** Hand-shaking for interrupts instead of host command to clear INTA and HC3.
** Non-interrupt context code disables interrupts when performing PCI transactions.
** Host vector interrupts are otherwise enabled, so PC doesn't have to force with HNMI bit.

== Last stable version: U0104 ==

* Implements quiet transfer mode! Remains backwards compatible with A1.4.
* Fixes the 64k boundary crossing issue
* Moves parameters that enter via interrupt out of registers and into variables
* Version reporting tag-along to RDM command (sending 'VER' to RDM's vector address returns the code version).
* Maximum burst length is reduced to 64 bytes, and is configurable.
* Reset (RST) clears the fibre fifo

== Oldest UBC release: U0103 ==

* Minor modifications of SCUBA2's A1.4 firmware, to improve PCI stability.
* Not compatible with non-realtime systems.

Rectangle Mode Data

2009-09-05T21:22:59Z

174.6.181.2:

"Rectangle Mode" refers to the sampling of a subset of the full array, typically at high rate. The data from the Readout Cards is packed into output frames efficiently to minimize overhead. Rectangle mode is available in MCE firmware starting with RC 5.0.1 and CC 5.0.0.

= Description of capabilities =

Rectangle mode eliminates strain on the acquisition system during fast acquisitions by eliminating read-out overhead. This means that the 'difficulty' of read-out is a fairly linear function of (sampling frequency) x (number of detectors read out) rather than depending on each.

The hardware limit on readout is roughly 4 MB/s. Since MCE data are 32-bit words, this permits 1e6 detector-samples to be read out per second.

So the maximum number of detectors that can be sampled at some frequency F is
D = (1 MHz) / F

You can't sample at 1 MHz. (Do your squids settle in 1us? Didn't think so.) But you could probably do a detector or two at 250 kHz.

= Firmware =

Rectangle mode requires "version 5" firmware on all cards, and at least RC 5.0.2. Here are links to the recommended SOFs at time of writing:
http://www.phas.ubc.ca/%7Emce/mcedocs/firmware/ac_v05000000_22dec2008.sof
http://www.phas.ubc.ca/%7Emce/mcedocs/firmware/bc_v05000000_22dec2008.sof
http://www.phas.ubc.ca/%7Emce/mcedocs/firmware/cc_v05000002_16jun2009.sof
http://www.phas.ubc.ca/%7Emce/mcedocs/firmware/rc_v05000002_16jun2009.sof

= Configuration =

The steps for configuring a rectangle mode acquisition are:
# Set the multiplexing rate using num_rows and row_len
# If necessary, adjust row_order to mux the channels of interest (you can only read out rows you are multiplexing...)
# Identify a rectangle of detectors on the readout card to read out by setting the RC parameters
#* num_rows_reported, num_cols_reported, readout_row_index, readout_col_index
# Set the size of the readout frame by setting the CC parameters
#* num_rows_reported, num_cols_reported
# Set the CC decimation factor 'data_rate' so that the RC rectangles will stack perfectly into the CC readout frame.
#* e.g. suppose
#** RC num_rows_reported x num_cols_reported = 1x4
#** CC num_rows_reported x num_cols_reported = 33x8
#* then we set
#** CC data_rate = (33 x 8) / (1 x 4) = 66

= Readout limitations =

Rectangle mode configuration should respect the following hardware limits:
* Total data rate must not exceed 4 MB/s
* Read-out frequency must not exceed 20 kHz
* The multiplexing frame rate must not be set so fast that the MCE cannot communicate with the readout cards. This limit is probably in the 100s of kHz.

Other things to remember:
* num_rows >= 2 is encouraged.
* you can't sample a detector faster than it is being visited in the multiplexing cycle.

= Checking MCE configuration using rect_check.py =

Use the utility @rect_check.py@ to verify that data will be / were acquired reasonably.

e.g. To check the state of the MCE (requires python MCE bindings), run
rect_check.py --mce

To analyze the data framing in a runfile:
rect_check.py my_data.run

(Depending on stuff, you might need to invoke rect_check as:
$MAS_PYTHON/rect_check.py
or even
python $MAS_PYTHON/rect_check.py
Sorry.)

The output looks like this:

MCE configuration:
cc_rcs ( cc, rcs_to_report_data ): 60
cc_dec ( cc, data_rate ): 256
cc_nmux ( cc, num_rows ): 33
cc_cmux ( cc, row_len ): 100
cc_nr ( cc, num_rows_reported ): 32
cc_nc ( cc, num_cols_reported ): 8
rc_nmux ( rca, num_rows ): 33
rc_cmux ( rca, row_len ): 100
rc_nr ( rca, num_rows_reported ): 1
rc_nc ( rca, num_cols_reported ): 1
rc_r0 ( rca, readout_row_index ): 0
rc_c0 ( rca, readout_col_index ): 0
Framing:
RC storage per mux cycle: 1
CC read-out per frame: 256
CC decimation: 256
Contiguous? yes
Complete? no
Bizarro? no
Timing:
Mux freq: 15151.00
Mean sampling freq: 15104.00
Read-out freq: 59.00
Data volume:
Frame size (bytes/RC): 1200
Data rate (MB/s/RC): 0.07

The program has notions of what is "reasonable", and these are expressed in the lines "Contiguous?", "Complete?", and "Bizarro?":
* 'Contiguous': all detectors that are read out will be read out at each multiplexing cycle
** When acquiring fast data from a subset of the detectors, at the multiplexing rate, you want "Contiguous = yes".
* 'Complete': all detectors that are multiplexed will be read out
** When acquiring normal data, you probably want "Complete = yes".
* 'Bizarro': the rectangle defined in the RC does not fit evenly into the readout frame.
** e.g. you have set up an 8x8 rectangle but you're reading CC frames that are 33x8. Don't do this. I'm not even sure what you'd get out.

Note that it is possible to be none of Complete, Contiguous, or Bizarro. For example, you could read out the first 33 of every 66 multiplexing cycles for 8 detectors, packing them efficiently into a 33 x 8 read-out frame. Maybe that should be called Bizarro. But right now, it isn't.

It is unlikely, but possible, to be both Contiguous and Complete.

If you are Bizarro, you are neither Contiguous nor Complete. See Venn diagram below.

[[Image:reasonable_rectangles.png]]

= Data acquisition =

Use "mce_run" in the usual way to acquire data. Note that the argument specifying the number of frames is always the number of 'read-out' frames, so for packed data the number of samples per detector will be higher.

e.g. for the above configuration
mce_run my_data 100 1

would acquire 100 read-out frames. Since the read-out frequency is 59 Hz, this would take about 2 seconds. Each readout frame contains 8 x 32 = 256 data from a single detector. So the resulting data stream contains 25600 samples.

= Data file loading =

Recent versions of mce_data.py can extract rectangle mode data from the packed frame structure. E.g. for the above case, extracting the data is as easy as
>>> d = MCEFile('my_data').Read()
>>> d.data.shape
(1, 25600)

= Rather long explanation and justification of structure, from an e-mail =

Yesterday Bryce and I were talking about paths towards a new rectangle readout mode. The following is a recommendation for the rough structure of the system that I think will be easy to manage and provide a maximum flexibility for backwards compatibility and future expansion.

The current rectangle mode is simpler than the planned one because the clock card and readout card need only agree on the number of data to readout, and no addditional waiting/synchronization is required between the clock card and readout card. The readout frame data sizes, internally and externally, are the same for everyone. The new mode is much more complex, because now the readout card has to accumulate several samples from each pixel into its internal buffer before sending the data to the clock card, and the clock card needs to ask for the right amount of data, which is probably some multiple of the rectangle size and is not necessarily a multiple of 8 or the number of multiplexing rows.

Anyway, the point is that here is a simple and flexible way to parametrize the new mode parameters and manage synchronization, and to provide flexibility and backwards compatibility.

1. The meaning of num_rows remains unchanged. It will always be a multiplexing parameter that all cards need to agree on.

2. The readout card has parameters (prepend "readout_" to all of these, if you want) "row_index", "num_rows", "col_index", "num_cols" that describe the rectangle of interest.

3. The clock card is programmed with a pair of numbers "readout_mult1" and "readout_mult2", that it uses to determine how many words of data to query the RCs for:
N_data = readout_mult1 * readout_mult2.
These numbers are analagous to "num_rows_reported" and "num_cols_reported", but the idea is to kill the idea of "rows" and "columns" in the readout frames because the payload size is not a simple product of n_rows * n_cols. The clock card continues to support "data rate", which indicates the period at which data queries are sent to the RCs. Software will handle the details of getting these factors right. Eventually we will only use one of mult?; the other is for backwards compat.

4. In the RC: at each mux frame, data from the desired rectangle is copied into a large buffer. The RC continues filling the buffer until the CC asks for data. When RC receives a request for N data from the CC, it returns the first N data in the buffer, and resets its write index to the start of the buffer. If the CC doesn't ask for data, and the RC buffer write index increments all the way past the end, the RC takes _no storage action_ in subsequent steps (i.e. it writes all new data to "/dev/null").

Setting up rectangle mode then proceeds as follows. Suppose the RC buffer size is 1600 and I want to readout a 4 x 33 patch of the array:

1. Software computes that 4x33 = 132 words fits into 1600 about 12 times. So it sets "cc data_rate" to 12, and the product of "mult1" and "mult2" to 12 * 132 = 1584. It also sets the readout card row and col rectangle parameters.

2. Software triggers frame acquisition. For the _first_ frame read out, the clock card and readout card may not be synchronized; i.e. the readout card's buffer almost certainly contains ancient, stale data. So the first frame is discarded. But since the first frame query sent the the buffer write index to 0, the data from subsequent frames is fresh and ordered. Because the clock card is asking for N data at the same rate that the RC can N-fill the buffer, this results in a contiguous data stream.

This scheme is simple because:
1. CC has only enough information to ask the readout cards for the right amount of data at the right times.
2. RC only needs to know the rectangle of interest, and it basically lets the CC manage the resetting of the buffer.
3. It is backwards-compatible with existing firmware; mult1 and mult2 can masquerade as num_rows_reported and "8" (num_cols_reported).

The scheme minimizes the amount of information that must be shared between the RC and CC, which provides flexiblity. With "mult" approach, the CC can remain truly ignorant as to what the RC is doing. In this scheme I have emphasized the rectangle mode features, but I also envision it being useful in raw mode firmware, since a lot of the features (such as a large, flexible RAM block in the RC) would seem useful in that context. (For raw mode you would need to add a lock to the buffer so that it could be read out with multiple queries.)

MAS OS setup

2009-06-11T16:23:41Z

174.6.181.2: /* Install python stuff */

'''Ubuntu 6.06 LTS server''': This is the most supported OS. The instructions below are mostly for 6.06.

'''Ubuntu 7.10 server''': We can support this now. Patched kernels are available. In the text below, major deviations from the 6.06 installation procedure are identified.

'''Ubuntu 8.04 LTS server''': This version is new enough that the kernel build system isn't producing portable kernel packages. MAS works with 8.04 but I don't want to install too many 8.04 systems until they clean up the build system.

= Install additional ubuntu packages =

== Disable CDROM seeking ==

The package manager knows that you have the Ubuntu disk and will say things like
Media change: please insert the disc labeled
'Ubuntu-Server 6.06.1 _Dapper Drake_ - Release i386 (20060807.1)'

To disable this (and download packages from the internet instead), open /etc/apt/sources.list
sudo pico /etc/apt/sources.list
and remove (comment) the line
deb cdrom:[Ubuntu-Server 6.06.1 _Dapper Drake_ - Release i386 (20060807.1)]/ dapper main restricted

== Install required packages ==

sudo apt-get update
sudo apt-get install build-essential subversion emacs21 libreadline5-dev

== Enable universe repository and install GGV ==

Uncomment the line in /etc/apt/sources.list so it says
deb http://us.archive.ubuntu.com/ubuntu/ dapper universe

Then run
sudo apt-get update
sudo apt-get install gnome-gv

gnome-gv doesn't exist on 7.10, you'll have to settle for
sudo apt-get install gv

== Install packages that you shouldn't even want to install (soon to be optional) ==
sudo apt-get install tcsh

== Install desktop manager (optional) ==
sudo apt-get install ubuntu-desktop gdm

== Install python stuff ==

The 'pyth' branch MAS has experimental support for python using wx and matplotlib:
sudo apt-get install python-dev python-wxglade python-matplotlib python-numarray-ext swig

For loading / plotting MCE data in python using [[Python data and runfile modules | mce_data.py]], you will want numpy / scipy:
sudo apt-get install python-numpy python-scipy

For Ubuntu 6.06, numpy and scipy can be obtained following instructions from this page:
http://debs.astraw.com/dapper/

On more recent, but not new, Ubuntu (e.g. 8.10), you may want to install a more current version of numpy. The easiest way to do this seems to get source code from
http://sourceforge.net/projects/numpy
unzip and run
./setup.py config
./setup.py build
sudo ./setup.py install
The config step may complain that it can't find BLAS and ATLAS, but that's ok because the numpy source has scaled-down substitutes.

== Install unpackaged libraries ==

MAS uses libconfig to manage its configuration files. The webpage is here: [ http://www.hyperrealm.com/libconfig/ ]. To install libconfig run the following:

wget http://www.hyperrealm.com/libconfig/libconfig-1.3.tar.gz
tar -xzf libconfig-1.3.tar.gz
cd libconfig-1.3
./configure
make
sudo make install

To make the system aware of this library, add "/usr/local/lib" to /etc/ld.so.conf and run "sudo ldconfig". i.e.
echo /usr/local/lib | sudo tee -a /etc/ld.so.conf
sudo ldconfig

On newer systems (Ubuntu 7.10) you can do this instead:
echo /usr/local/include | sudo tee /etc/ld.so.conf.d/libconfig.conf
sudo ldconfig

= Download and install MAS kernel patch =

== Download ==

If you're not compiling the kernel from scratch, download the binary packages from UBC:
wget http://e-mode.phas.ubc.ca/~mhasse/mce/kernel-headers-2.6.15.7-bigphys_10.00.Custom_i386.deb
wget http://e-mode.phas.ubc.ca/~mhasse/mce/kernel-image-2.6.15.7-bigphys_10.00.Custom_i386.deb
wget http://e-mode.phas.ubc.ca/~mhasse/mce/extras.patch

For '''Ubuntu 7.10''', get these instead:
wget http://e-mode.phas.ubc.ca/~mhasse/mce/linux-headers-2.6.22.9-bigphys_10.00.Custom_i386.deb
wget http://e-mode.phas.ubc.ca/~mhasse/mce/linux-image-2.6.22.9-bigphys_10.00.Custom_i386.deb

On '''Ubuntu 7.10''' machines '''without PAE''' hardware (i.e. a maximum of 4G of system memory) use:
wget http://e-mode.phas.ubc.ca/~mhasse/mce/linux-headers-2.6.22.14-bigphys_2.6.22.14-bigphys-10.00.Custom_i386.deb
wget http://e-mode.phas.ubc.ca/~mhasse/mce/linux-image-2.6.22.14-bigphys_2.6.22.14-bigphys-10.00.Custom_i386.deb

== Install ==

Then, install the packages using dpkg:
sudo dpkg -i kernel-headers-2.6.15.7-bigphys_10.00.Custom_i386.deb
sudo dpkg -i kernel-image-2.6.15.7-bigphys_10.00.Custom_i386.deb
The "image" file might complain about symbolic links, no big deal.

On '''Ubuntu 7.10''', instead install these:
sudo dpkg -i linux-headers-2.6.22.9-bigphys_10.00.Custom_i386.deb
sudo dpkg -i linux-image-2.6.22.9-bigphys_10.00.Custom_i386.deb

Why are these ones called "linux" instead of "kernel"? I think it's because I got them from Ubuntu repositories instead of kernel.org.

On '''Ubuntu 7.10''' machines '''without PAE''' install these:
sudo dpkg -i linux-headers-2.6.22.14-bigphys_2.6.22.14-bigphys-10.00.Custom_i386.deb
sudo dpkg -i linux-image-2.6.22.14-bigphys_2.6.22.14-bigphys-10.00.Custom_i386.deb

The '''Ubuntu 8.04''' kernel has a sound-card driver that lays claim to the Motorola DSP on the PCI card. We have to blacklist this module to prevent it from trying to configure the card as a sound card. Add the following to the bottom of /etc/modprobe.d/blacklist :
# Conflicts with Astro-cam PCI card!!
blacklist snd_asihpi

== Patch ==

This makes it possible to compile against the kernel package as though it had been locally compiled originally.
cd /usr/src/kernel-headers-2.6.15.7-bigphys/
sudo patch -p1 < ~/extras.patch

This step is not necessary for Ubuntu 7.10. It may be necessary to link the kernel headers into /lib/modules:
sudo ln -s /usr/src/linux-headers-2.6.22.9-bigphys/ /lib/modules/2.6.22.9-bigphys/build

== Boot menu ==

We need to add the kernel option that causes bigphys to allocate boot memory for our driver. In older versions of MAS (especially ACT's stable release) we also need to disable "acpi". It's a good idea to not run any DSP/MCE commands until all of your kernel options are in place.

Having installed the kernel "image" package, the kernel should show up in the boot loader (grub)'s kernel list.

=== Editing the kernel list ===
As root (or using sudo), edit the file /boot/grub/menu.lst .
sudo pico /boot/grub/menu.lst
Go to the list of kernels, below the line "## ## End Default Options ##", and find the new kernel entry. On Ubuntu 6.06, it should be the third block (index 2), and look like this:

title Ubuntu, kernel 2.6.15.7-bigphys
root (hd0,0)
kernel /vmlinuz-2.6.15.7-bigphys root=/dev/sda3 ro quiet splash
initrd /initrd.img-2.6.15.7-bigphys
savedefault
boot

Add the kernel options to the "kernel" line, producing either this (recent MAS):
kernel /vmlinuz-2.6.15.7-bigphys root=/dev/sda3 ro quiet splash bigphysarea=8192
or this (old MAS, esp. ACT):
kernel /vmlinuz-2.6.15.7-bigphys root=/dev/sda3 ro quiet splash bigphysarea=8192 acpi=off

Save and close the file, and reboot to test this kernel. If the kernel "works", you can edit menu.lst again and change the value of the "default" option to point to this kernel:
default 2

'''Ubuntu 7.10 and later''': the kernel packages install a bit differently so the block will likely be the first one in the list. The 'kernel' line must change from something like
kernel /boot/vmlinuz-2.6.22.9-bigphys root=UUID=1b6e7b54-894d-4571-9f0a-527fe0103975 ro quiet splash
to
kernel /boot/vmlinuz-2.6.22.9-bigphys root=UUID=1b6e7b54-894d-4571-9f0a-527fe0103975 ro quiet splash bigphysarea=8192
Note that the long hexadecimal serial numbers are system specifc and yours are probably different than these ones. Leave them as they are and just add the new kernel options. The "default" option will probably be
default 0
but it's a good idea to count these things out for yourself.

= Configure the system for mce users =

== Permissions and umask ==

Set the umask for all users to give write access for their group by default.

Edit /etc/profile and change the "umask 022" line to
umask 002

Edit /etc/login.defs and find the line that start "# UMASK" and change it to
UMASK 002

Any users wishing to use MCE under their own accounts should be added to the "mce" group and should have "mce" as their primary group. Their umask must be set properly, or other users will not be free to manipulate shared files.

To setup additional system users, see the page on [[MAS user setup]].

== Folders ==

This is the sort of thing that should be done by an install script...

Data directory:
sudo mkdir /data
sudo chown mce:mce /data
sudo chmod g+ws /data
mkdir /data/cryo/

Configuration directory:
sudo mkdir /etc/mce
sudo chgrp mce /etc/mce
sudo chmod g+ws /etc/mce

= If necessary, branch MAS and mce_script =

You can work with a complete checked-out tree, or not.

In the first case:

svn checkout svn://e-mode.phas.ubc.ca/mas
svn copy mas/trunk mas/branch/{your_project_name}
svn commit mas

Alternately, do the copy directly on the svn server (it is automatically committed):
svn copy svn://e-mode.phas.ubc.ca/mas/trunk svn://e-mode.phas.ubc.ca/mas/branch/{your_project_name}

For mce_script, replace "mas" with "mce_script".

Then check out the appropriate branch (or the whole tree) on the new machine.

= Download (checkout) MAS and mce_script =

See the page on [[ MAS svn repository ]]

= Compile and install MAS =

== Build the driver ==

By default, the driver will build with bigphysarea support enabled. To override build options, create a file Makefile.local in the "driver" folder, and add lines like:

# BIGPHYS=0 disables bigphysarea frame-buffering
BIGPHYS = 0

or

# FAKEMCE=1 provides simple emulation of a PCI card/MCE for testing
FAKEMCE = 1

or

# REALTIME=1 makes use of realtime interrupt routines
REALTIME = 1

In Makefiles, there should not be comments on the same line as the variable definitions. On a system running the bigphys kernel with the correct PCI card, Makefile.local will be created at compile time if it does not already exist. It can be left empty.

Anyway, once Makefile.local is setup, type
cd mas
make clean; make

From the driver folder we can test the driver:
cd driver
./reload
./mknodes

Note that since "reload" first unloads the driver if it is present, and then loads the driver from the current folder, it may report an "ERROR" message if the first step fails, even though the driver is successfully loaded. The definitive way to check that the driver is loaded is
cat /proc/mce_dsp

If this file does not exist, the driver isn't loaded. If the cat prints out a bunch of low-level driver information, you're in good shape.

If you're satisfied that the driver works, install the driver with the command

sudo make install

This should put mce_dsp.ko into /lib/modules/2.6.15.7-bigphys/kernel/drivers/misc/, and re-scan the module dependencies. To get the driver to load on boot you must add manually the following lines to /etc/rc.local:

modprobe mce_dsp
/home/mce/mas/driver/mknodes
sudo -u mce /usr/mce/bin/maslog_server

== Install MAS config files ==

Hardware and binary config files live in /etc/mce/ (created above).

Make sure you use the hardware config (mce_*.cfg) file appropriate for your system. Racks with 4 readout cards should use mce_v1.cfg; small sub-racks with readout card slots should use mce_v2.cfg.

cd ~/mas/config
cp mce_v1.cfg /etc/mce/mce.cfg
cp mas.cfg /etc/mce/mas.cfg

= Install mce_script =

The mce_script files are designed so that users can fairly easily run a different set of scripts. We will install a system default set in /usr/mce/mce_script

cd /usr/mce
svn checkout svn://e-mode.phas.ubc.ca/mce_script/branch/{your_experiment} mce_script

Current experiments include ACT and SPIDER.

== mas_env.bash ==

To set up the environment and path to use these scripts and idl codes, a user should do "source mas_env.bash". (This file used to live in mas/config, but has been moved into mce_script/template.) Non-developer users can source /usr/mce/mce_script/template/mas_env.bash directly; advanced users may want to keep their own copy (or several different copies) in their home folder.

Note that MAS/mce_script scripts should *not* source mas_env.bash! The whole point of it is that paths should be flexible.

== .bashrc ==

To have bash start up with the MAS variables defined, and the paths ready, add the following lines to .bashrc :

export MAS_ROOT=/usr/mce/mce_script/
source $MAS_ROOT/template/mas_env.bash
export IDL_PATH="<IDL_DEFAULT>:$MAS_IDL/mas"

While you're in there, you may want to define a few aliases. In particular, I always uncomment the alias ll='ls-l' line.

MAS PC specifications email template

2009-06-08T22:55:00Z

174.6.181.2:

MAS should run without difficulty on any modern PC that has a standard PCI slot and can run Ubuntu 8.04. In particular, MAS now supports 64-bit and multi-core machines. We have run MAS on fairly weak machines (1 GHz single core w/ 1GB RAM), but during detector characterization phases users will probably prefer a much more powerful desktop machine with lots of speed and graphics support.

Note also that MCE data can take up quite a bit of disk space; 1024 channels of 400 Hz will pile up at 1.6 MB/s. If your data rate is comparable to this then your system should have at least 500 GB of space for MCE data.

Again, each MCE requires a PCI slot for the fibre-optic interface board! Some new motherboards do not have standard PCI slots, as PCI-express is becoming more popular.

MCE flat-file format

2009-06-03T20:52:39Z

174.6.181.2: /* Frame status bits */

This document describes one of MAS/MCE's output file formats. The "DAS flat-file" format is closely related to the form that frame data have when they arrive at the PC from the MCE. This format has many weaknesses as far as end users are concerned. Only flat-files with "header version 6" are described here.

The data consist of any number of "frames". In a given flat-file, all frames should have the same size. The frame size can be determined from the first 28 bytes of the file. No MCE flat-file will be shorter than 28 bytes.

The natural data unit of MCE data is the 32-bit unsigned integer. We refer to 32-bit integers as "words". All offsets are expressed in terms of words, and should be multiplied by 4 to get the byte offset.

An MCE/DAS frame, up to and including header version 6, consists of the following:

* 43 words of header information
* 8*R*N words of data, where N is the number of readout cards reporting and R is the number of rows reporting.
* 1 word of checksum information

= Frame header =

The frame header is 43 words long. The table below describes the most useful fields in the header. Esoteric fields are omitted; absent offsets are not necessarily unused.

{| border="5"
|-
! Offset !! Name !! Category !! Detail
|-
| 0 || Frame status bits || Status/Structure || described below
|-
| 1 || CC frame counter || Status || internal counter, starts at the value of "cc ret_dat_s" and increments by 1 for each output frame.
|-
| 2 || row_len || Timing || Dwell time of the multiplexer on each row, in units of 1/50 MHz
|-
| 3 || num_rows_reported || Structure || Number of rows in each column that are reported in each output frame.
|-
| 4 || data_rate || Timing || Number of internal frames that MCE processes before outputting one to the user.
|-
| 5 || CC ARZ counter || Timing || internal counter, counts number of address-return-to-zero (ARZ) or number of internal frames.
|-
| 6 || header version || Structure || 6, the number "six". Subject to change in future header versions.
|-
| 7 || ramp value || Status || Current value of the internal ramping parameter
|-
| 8 || ramp address || Status || Card and parameter address of the internal ramping parameter (bits [31-16] are card_addr, bits [15-0] are param_id)
|-
| 9 || num_rows || Timing || Number of rows that the MCE is servoing
|-
| 10 || Sync box number || Status || Synchronization number, originating at the sync box, that is used to synchronize data from multiple MCEs and the housekeeping system
|-
| 11 || run_id || Status || Contents of "cc run_id", typically the ctime (unix time) associated with the acquisition.
|-
| 12 || user_word || Status || Contents of "cc user_word", which the user may use to store other significant information.
|}

A more complete description of the version 6 frame header can be found in the document "MCE File Formats", version 3.3:

http://www.phas.ubc.ca/%7Emce/mcedocs/index.html

This document is out of date in some respects, but the descriptions of the frame headers up to version 6 should be accurate.

== Frame status bits ==

{| border="5"
! Bit !! Category !! Detail
|-
| 0 || Status || Last frame bit, indicates that current acquisition cycle has completed. This does not necessarily indicate the last frame in a flat-file.
|-
| 1 || Status || Stop bit, indicates that MCE was commanded to stop acquisition and this is the last frame.
|-
| 2 || Status || Sync box free run mode bit, indicates that sync box is generating the data valid information (rather than accepting an external trigger)
|-
| 3 || Status || Sync box error bit.
|-
| 4 || Status || TES bias square wave level (not present in most firmware)
|-
| 10 || Structure || Readout card 1 data are present.
|-
| 11 || Structure || Readout card 2 data are present.
|-
| 12 || Structure || Readout card 3 data are present.
|-
| 13 || Structure || Readout card 4 data are present.
|-
| 19:16 (v7 only!) || Structure || Number of columns returning data, per readout card. Reads as 0 in v6.
|}

Bits 10-13 and 16-19 are especially relevant for determining the frame size and data structure.

== Timing parameters ==

The internal frame rate of the MCE (the rate at which a given row is servoed) and the output frame rate of the MCE (the rate at which frames are returned to the PC and recorded to disk) can be calculated as follows:

f_internal = 50 MHz / (num_rows * row_len)
f_output = f_internal / data_rate

== Structure determination ==

The size and structure of the readout data block can be determined from
* num_rows_reported
* bits 10-13 of the frame status bits, which we will call RC1_on, RC2_on, RC3_on, RC4_on (each taking a value of 1 or 0)

In particular we can determine the number of readout cards reporting, as

N_card = (RC1_on + RC2_on + RC3_on + RC4_on)

and this gives the size of the readout data block, in words, as

N_data = 8 * num_rows_reported * N_card

with 8 representing the number of columns managed by each readout card.

The organization of the frame data is described in the next section.

= Frame data =

== Normal acquisitions (row / column) ==

By "normal acquisitions", we mean everything except data mode 3.

Each word of MCE data is associated with a unique row and column in the multiplexing scheme. Each readout card is responsible for 8 columns and up to 41 rows. Columns 0-7 are associated with readout card 1, columns 8-15 are associated with readout card 2, and so on. MCE can support up to 32 columns.

Schematically, the data is organized like this (with rNNcMM indicating the data assoicated with row NN and column MM):

r00c00 r00c01 r00c02 ... r00c31
r01c00 r01c00 r01c02 ... r01c31
...
r40c00 r40c01 r40c02 ... r40c31

When num_rows_reported is less than 41, then the data for rows >= num_rows_reported is omitted. When a readout card is not reporting data, the data for its associated columns are omitted. For example, if num_rows_reported is 20 and only RC2 is reporting data then the frame structure is

r00c08 r00c09 r00c10 ... r00c15
r01c08 r01c09 r01c10 ... r01c15
r02c08 r02c09 r02c10 ... r02c15
...
r19c08 r19c09 r19c10 ... r19c15

Mathematically, the data for row R can be found starting at index
I_R = R * N_card * 8

The adjustment to this index to get the data associated with column C depends on which cards are reporting data.

Note that currently MCE only supports single-card and all-card modes; so the slightly more complicated cases of, for example, reporting from readout cards 2 and 4 are not available. The format described here does uniquely specify a way of handling those cases, however.

= Checksum =

The checksum word is the result of a cumulative XOR of all other words in the frame, including the header. Since the XOR of any number with itself is zero, the cumulative XOR of all words in the frame (including the header, data, and checksum) should be zero.

MCE flat-file format

2009-06-03T20:52:18Z

174.6.181.2: /* Frame status bits */

This document describes one of MAS/MCE's output file formats. The "DAS flat-file" format is closely related to the form that frame data have when they arrive at the PC from the MCE. This format has many weaknesses as far as end users are concerned. Only flat-files with "header version 6" are described here.

The data consist of any number of "frames". In a given flat-file, all frames should have the same size. The frame size can be determined from the first 28 bytes of the file. No MCE flat-file will be shorter than 28 bytes.

The natural data unit of MCE data is the 32-bit unsigned integer. We refer to 32-bit integers as "words". All offsets are expressed in terms of words, and should be multiplied by 4 to get the byte offset.

An MCE/DAS frame, up to and including header version 6, consists of the following:

* 43 words of header information
* 8*R*N words of data, where N is the number of readout cards reporting and R is the number of rows reporting.
* 1 word of checksum information

= Frame header =

The frame header is 43 words long. The table below describes the most useful fields in the header. Esoteric fields are omitted; absent offsets are not necessarily unused.

{| border="5"
|-
! Offset !! Name !! Category !! Detail
|-
| 0 || Frame status bits || Status/Structure || described below
|-
| 1 || CC frame counter || Status || internal counter, starts at the value of "cc ret_dat_s" and increments by 1 for each output frame.
|-
| 2 || row_len || Timing || Dwell time of the multiplexer on each row, in units of 1/50 MHz
|-
| 3 || num_rows_reported || Structure || Number of rows in each column that are reported in each output frame.
|-
| 4 || data_rate || Timing || Number of internal frames that MCE processes before outputting one to the user.
|-
| 5 || CC ARZ counter || Timing || internal counter, counts number of address-return-to-zero (ARZ) or number of internal frames.
|-
| 6 || header version || Structure || 6, the number "six". Subject to change in future header versions.
|-
| 7 || ramp value || Status || Current value of the internal ramping parameter
|-
| 8 || ramp address || Status || Card and parameter address of the internal ramping parameter (bits [31-16] are card_addr, bits [15-0] are param_id)
|-
| 9 || num_rows || Timing || Number of rows that the MCE is servoing
|-
| 10 || Sync box number || Status || Synchronization number, originating at the sync box, that is used to synchronize data from multiple MCEs and the housekeeping system
|-
| 11 || run_id || Status || Contents of "cc run_id", typically the ctime (unix time) associated with the acquisition.
|-
| 12 || user_word || Status || Contents of "cc user_word", which the user may use to store other significant information.
|}

A more complete description of the version 6 frame header can be found in the document "MCE File Formats", version 3.3:

http://www.phas.ubc.ca/%7Emce/mcedocs/index.html

This document is out of date in some respects, but the descriptions of the frame headers up to version 6 should be accurate.

== Frame status bits ==

{| border="5"
! Bit !! Category !! Detail
|-
| 0 || Status || Last frame bit, indicates that current acquisition cycle has completed. This does not necessarily indicate the last frame in a flat-file.
|-
| 1 || Status || Stop bit, indicates that MCE was commanded to stop acquisition and this is the last frame.
|-
| 2 || Status || Sync box free run mode bit, indicates that sync box is generating the data valid information (rather than accepting an external trigger)
|-
| 3 || Status || Sync box error bit.
|-
| 4 || Status || TES bias square wave level (not present in most firmware)
|-
| 10 || Structure || Readout card 1 data are present.
|-
| 11 || Structure || Readout card 2 data are present.
|-
| 12 || Structure || Readout card 3 data are present.
|-
| 13 || Structure || Readout card 4 data are present.
|-
| 19:16 (v7 only!) || Structure || Number of columns returning data. Reads as 0 in v6.
|}

Bits 10-13 and 16-19 are especially relevant for determining the frame size and data structure.

== Timing parameters ==

The internal frame rate of the MCE (the rate at which a given row is servoed) and the output frame rate of the MCE (the rate at which frames are returned to the PC and recorded to disk) can be calculated as follows:

f_internal = 50 MHz / (num_rows * row_len)
f_output = f_internal / data_rate

== Structure determination ==

The size and structure of the readout data block can be determined from
* num_rows_reported
* bits 10-13 of the frame status bits, which we will call RC1_on, RC2_on, RC3_on, RC4_on (each taking a value of 1 or 0)

In particular we can determine the number of readout cards reporting, as

N_card = (RC1_on + RC2_on + RC3_on + RC4_on)

and this gives the size of the readout data block, in words, as

N_data = 8 * num_rows_reported * N_card

with 8 representing the number of columns managed by each readout card.

The organization of the frame data is described in the next section.

= Frame data =

== Normal acquisitions (row / column) ==

By "normal acquisitions", we mean everything except data mode 3.

Each word of MCE data is associated with a unique row and column in the multiplexing scheme. Each readout card is responsible for 8 columns and up to 41 rows. Columns 0-7 are associated with readout card 1, columns 8-15 are associated with readout card 2, and so on. MCE can support up to 32 columns.

Schematically, the data is organized like this (with rNNcMM indicating the data assoicated with row NN and column MM):

r00c00 r00c01 r00c02 ... r00c31
r01c00 r01c00 r01c02 ... r01c31
...
r40c00 r40c01 r40c02 ... r40c31

When num_rows_reported is less than 41, then the data for rows >= num_rows_reported is omitted. When a readout card is not reporting data, the data for its associated columns are omitted. For example, if num_rows_reported is 20 and only RC2 is reporting data then the frame structure is

r00c08 r00c09 r00c10 ... r00c15
r01c08 r01c09 r01c10 ... r01c15
r02c08 r02c09 r02c10 ... r02c15
...
r19c08 r19c09 r19c10 ... r19c15

Mathematically, the data for row R can be found starting at index
I_R = R * N_card * 8

The adjustment to this index to get the data associated with column C depends on which cards are reporting data.

Note that currently MCE only supports single-card and all-card modes; so the slightly more complicated cases of, for example, reporting from readout cards 2 and 4 are not available. The format described here does uniquely specify a way of handling those cases, however.

= Checksum =

The checksum word is the result of a cumulative XOR of all other words in the frame, including the header. Since the XOR of any number with itself is zero, the cumulative XOR of all words in the frame (including the header, data, and checksum) should be zero.

MAS OS setup

2009-06-03T17:19:18Z

174.6.181.2: /* Install python stuff */

'''Ubuntu 6.06 LTS server''': This is the most supported OS. The instructions below are mostly for 6.06.

'''Ubuntu 7.10 server''': We can support this now. Patched kernels are available. In the text below, major deviations from the 6.06 installation procedure are identified.

'''Ubuntu 8.04 LTS server''': This version is new enough that the kernel build system isn't producing portable kernel packages. MAS works with 8.04 but I don't want to install too many 8.04 systems until they clean up the build system.

= Install additional ubuntu packages =

== Disable CDROM seeking ==

The package manager knows that you have the Ubuntu disk and will say things like
Media change: please insert the disc labeled
'Ubuntu-Server 6.06.1 _Dapper Drake_ - Release i386 (20060807.1)'

To disable this (and download packages from the internet instead), open /etc/apt/sources.list
sudo pico /etc/apt/sources.list
and remove (comment) the line
deb cdrom:[Ubuntu-Server 6.06.1 _Dapper Drake_ - Release i386 (20060807.1)]/ dapper main restricted

== Install required packages ==

sudo apt-get update
sudo apt-get install build-essential subversion emacs21 libreadline5-dev

== Enable universe repository and install GGV ==

Uncomment the line in /etc/apt/sources.list so it says
deb http://us.archive.ubuntu.com/ubuntu/ dapper universe

Then run
sudo apt-get update
sudo apt-get install gnome-gv

gnome-gv doesn't exist on 7.10, you'll have to settle for
sudo apt-get install gv

== Install packages that you shouldn't even want to install (soon to be optional) ==
sudo apt-get install tcsh

== Install desktop manager (optional) ==
sudo apt-get install ubuntu-desktop gdm

== Install python stuff ==

The 'pyth' branch MAS has experimental support for python using wx and matplotlib:
sudo apt-get install python-dev python-wxglade python-matplotlib python-numarray-ext swig

For loading / plotting MCE data in python using [[Python data and runfile modules | mce_data.py]], you will want numpy / scipy:
sudo apt-get install python-numpy python-scipy

For Ubuntu 6.06, numpy and scipy can be obtained following instructions from this page:
http://debs.astraw.com/dapper/

== Install unpackaged libraries ==

MAS uses libconfig to manage its configuration files. The webpage is here: [ http://www.hyperrealm.com/libconfig/ ]. To install libconfig run the following:

wget http://www.hyperrealm.com/libconfig/libconfig-1.3.tar.gz
tar -xzf libconfig-1.3.tar.gz
cd libconfig-1.3
./configure
make
sudo make install

To make the system aware of this library, add "/usr/local/lib" to /etc/ld.so.conf and run "sudo ldconfig". i.e.
echo /usr/local/lib | sudo tee -a /etc/ld.so.conf
sudo ldconfig

On newer systems (Ubuntu 7.10) you can do this instead:
echo /usr/local/include | sudo tee /etc/ld.so.conf.d/libconfig.conf
sudo ldconfig

= Download and install MAS kernel patch =

== Download ==

If you're not compiling the kernel from scratch, download the binary packages from UBC:
wget http://e-mode.phas.ubc.ca/~mhasse/mce/kernel-headers-2.6.15.7-bigphys_10.00.Custom_i386.deb
wget http://e-mode.phas.ubc.ca/~mhasse/mce/kernel-image-2.6.15.7-bigphys_10.00.Custom_i386.deb
wget http://e-mode.phas.ubc.ca/~mhasse/mce/extras.patch

For '''Ubuntu 7.10''', get these instead:
wget http://e-mode.phas.ubc.ca/~mhasse/mce/linux-headers-2.6.22.9-bigphys_10.00.Custom_i386.deb
wget http://e-mode.phas.ubc.ca/~mhasse/mce/linux-image-2.6.22.9-bigphys_10.00.Custom_i386.deb

On '''Ubuntu 7.10''' machines '''without PAE''' hardware (i.e. a maximum of 4G of system memory) use:
wget http://e-mode.phas.ubc.ca/~mhasse/mce/linux-headers-2.6.22.14-bigphys_2.6.22.14-bigphys-10.00.Custom_i386.deb
wget http://e-mode.phas.ubc.ca/~mhasse/mce/linux-image-2.6.22.14-bigphys_2.6.22.14-bigphys-10.00.Custom_i386.deb

== Install ==

Then, install the packages using dpkg:
sudo dpkg -i kernel-headers-2.6.15.7-bigphys_10.00.Custom_i386.deb
sudo dpkg -i kernel-image-2.6.15.7-bigphys_10.00.Custom_i386.deb
The "image" file might complain about symbolic links, no big deal.

On '''Ubuntu 7.10''', instead install these:
sudo dpkg -i linux-headers-2.6.22.9-bigphys_10.00.Custom_i386.deb
sudo dpkg -i linux-image-2.6.22.9-bigphys_10.00.Custom_i386.deb

Why are these ones called "linux" instead of "kernel"? I think it's because I got them from Ubuntu repositories instead of kernel.org.

On '''Ubuntu 7.10''' machines '''without PAE''' install these:
sudo dpkg -i linux-headers-2.6.22.14-bigphys_2.6.22.14-bigphys-10.00.Custom_i386.deb
sudo dpkg -i linux-image-2.6.22.14-bigphys_2.6.22.14-bigphys-10.00.Custom_i386.deb

The '''Ubuntu 8.04''' kernel has a sound-card driver that lays claim to the Motorola DSP on the PCI card. We have to blacklist this module to prevent it from trying to configure the card as a sound card. Add the following to the bottom of /etc/modprobe.d/blacklist :
# Conflicts with Astro-cam PCI card!!
blacklist snd_asihpi

== Patch ==

This makes it possible to compile against the kernel package as though it had been locally compiled originally.
cd /usr/src/kernel-headers-2.6.15.7-bigphys/
sudo patch -p1 < ~/extras.patch

This step is not necessary for Ubuntu 7.10. It may be necessary to link the kernel headers into /lib/modules:
sudo ln -s /usr/src/linux-headers-2.6.22.9-bigphys/ /lib/modules/2.6.22.9-bigphys/build

== Boot menu ==

We need to add the kernel option that causes bigphys to allocate boot memory for our driver. In older versions of MAS (especially ACT's stable release) we also need to disable "acpi". It's a good idea to not run any DSP/MCE commands until all of your kernel options are in place.

Having installed the kernel "image" package, the kernel should show up in the boot loader (grub)'s kernel list.

=== Editing the kernel list ===
As root (or using sudo), edit the file /boot/grub/menu.lst .
sudo pico /boot/grub/menu.lst
Go to the list of kernels, below the line "## ## End Default Options ##", and find the new kernel entry. On Ubuntu 6.06, it should be the third block (index 2), and look like this:

title Ubuntu, kernel 2.6.15.7-bigphys
root (hd0,0)
kernel /vmlinuz-2.6.15.7-bigphys root=/dev/sda3 ro quiet splash
initrd /initrd.img-2.6.15.7-bigphys
savedefault
boot

Add the kernel options to the "kernel" line, producing either this (recent MAS):
kernel /vmlinuz-2.6.15.7-bigphys root=/dev/sda3 ro quiet splash bigphysarea=8192
or this (old MAS, esp. ACT):
kernel /vmlinuz-2.6.15.7-bigphys root=/dev/sda3 ro quiet splash bigphysarea=8192 acpi=off

Save and close the file, and reboot to test this kernel. If the kernel "works", you can edit menu.lst again and change the value of the "default" option to point to this kernel:
default 2

'''Ubuntu 7.10 and later''': the kernel packages install a bit differently so the block will likely be the first one in the list. The 'kernel' line must change from something like
kernel /boot/vmlinuz-2.6.22.9-bigphys root=UUID=1b6e7b54-894d-4571-9f0a-527fe0103975 ro quiet splash
to
kernel /boot/vmlinuz-2.6.22.9-bigphys root=UUID=1b6e7b54-894d-4571-9f0a-527fe0103975 ro quiet splash bigphysarea=8192
Note that the long hexadecimal serial numbers are system specifc and yours are probably different than these ones. Leave them as they are and just add the new kernel options. The "default" option will probably be
default 0
but it's a good idea to count these things out for yourself.

= Configure the system for mce users =

== Permissions and umask ==

Set the umask for all users to give write access for their group by default.

Edit /etc/profile and change the "umask 022" line to
umask 002

Edit /etc/login.defs and find the line that start "# UMASK" and change it to
UMASK 002

Any users wishing to use MCE under their own accounts should be added to the "mce" group and should have "mce" as their primary group. Their umask must be set properly, or other users will not be free to manipulate shared files.

To setup additional system users, see the page on [[MAS user setup]].

== Folders ==

This is the sort of thing that should be done by an install script...

Data directory:
sudo mkdir /data
sudo chown mce:mce /data
sudo chmod g+ws /data
mkdir /data/cryo/

Configuration directory:
sudo mkdir /etc/mce
sudo chgrp mce /etc/mce
sudo chmod g+ws /etc/mce

= If necessary, branch MAS and mce_script =

You can work with a complete checked-out tree, or not.

In the first case:

svn checkout svn://e-mode.phas.ubc.ca/mas
svn copy mas/trunk mas/branch/{your_project_name}
svn commit mas

Alternately, do the copy directly on the svn server (it is automatically committed):
svn copy svn://e-mode.phas.ubc.ca/mas/trunk svn://e-mode.phas.ubc.ca/mas/branch/{your_project_name}

For mce_script, replace "mas" with "mce_script".

Then check out the appropriate branch (or the whole tree) on the new machine.

= Download (checkout) MAS and mce_script =

See the page on [[ MAS svn repository ]]

= Compile and install MAS =

== Build the driver ==

By default, the driver will build with bigphysarea support enabled. To override build options, create a file Makefile.local in the "driver" folder, and add lines like:

# BIGPHYS=0 disables bigphysarea frame-buffering
BIGPHYS = 0

or

# FAKEMCE=1 provides simple emulation of a PCI card/MCE for testing
FAKEMCE = 1

or

# REALTIME=1 makes use of realtime interrupt routines
REALTIME = 1

In Makefiles, there should not be comments on the same line as the variable definitions. On a system running the bigphys kernel with the correct PCI card, Makefile.local will be created at compile time if it does not already exist. It can be left empty.

Anyway, once Makefile.local is setup, type
cd mas
make clean; make

From the driver folder we can test the driver:
cd driver
./reload
./mknodes

Note that since "reload" first unloads the driver if it is present, and then loads the driver from the current folder, it may report an "ERROR" message if the first step fails, even though the driver is successfully loaded. The definitive way to check that the driver is loaded is
cat /proc/mce_dsp

If this file does not exist, the driver isn't loaded. If the cat prints out a bunch of low-level driver information, you're in good shape.

If you're satisfied that the driver works, install the driver with the command

sudo make install

This should put mce_dsp.ko into /lib/modules/2.6.15.7-bigphys/kernel/drivers/misc/, and re-scan the module dependencies. To get the driver to load on boot you must add manually the following lines to /etc/rc.local:

modprobe mce_dsp
/home/mce/mas/driver/mknodes
sudo -u mce /usr/mce/bin/maslog_server

== Install MAS config files ==

Hardware and binary config files live in /etc/mce/ (created above).

Make sure you use the hardware config (mce_*.cfg) file appropriate for your system. Racks with 4 readout cards should use mce_v1.cfg; small sub-racks with readout card slots should use mce_v2.cfg.

cd ~/mas/config
cp mce_v1.cfg /etc/mce/mce.cfg
cp mas.cfg /etc/mce/mas.cfg

= Install mce_script =

The mce_script files are designed so that users can fairly easily run a different set of scripts. We will install a system default set in /usr/mce/mce_script

cd /usr/mce
svn checkout svn://e-mode.phas.ubc.ca/mce_script/branch/{your_experiment} mce_script

Current experiments include ACT and SPIDER.

== mas_env.bash ==

To set up the environment and path to use these scripts and idl codes, a user should do "source mas_env.bash". (This file used to live in mas/config, but has been moved into mce_script/template.) Non-developer users can source /usr/mce/mce_script/template/mas_env.bash directly; advanced users may want to keep their own copy (or several different copies) in their home folder.

Note that MAS/mce_script scripts should *not* source mas_env.bash! The whole point of it is that paths should be flexible.

== .bashrc ==

To have bash start up with the MAS variables defined, and the paths ready, add the following lines to .bashrc :

export MAS_ROOT=/usr/mce/mce_script/
source $MAS_ROOT/template/mas_env.bash
export IDL_PATH="<IDL_DEFAULT>:$MAS_IDL/mas"

While you're in there, you may want to define a few aliases. In particular, I always uncomment the alias ll='ls-l' line.

Raw-mode readout

2009-04-27T20:22:37Z

174.6.181.2: /* General Information */

= General Information =

* "Raw data" and "Raw mode" refer to the acquisition of ADC samples at 50 MHz.
* Raw data acquisition occurs in the 'background', i.e. it can occur while multiplexing and servoing are going on.
* Special readout card firmware is required to support raw mode. This firmware supports most normal readout card features, but does not support the low-pass filter nor the reading back of some parameters. It should be loaded temporarily (i.e. as a .sof).
** For 8-column raw mode readout, maximum 8192 samples per column:
http://www.phas.ubc.ca/~mce/mcedocs/firmware/rc_v04030007_26may2008_raw.sof
** For 1-column raw mode readout, maximum 65536 samples:
http://www.phas.ubc.ca/~mce/mcedocs/firmware/rc_v0400000d_17apr2009.sof
* The ADC readings are 14 bits, signed, and are reported as 32-bit signed integers.
* Data read from the ADC is already 4 cycles old when it is written to the RC buffer.
** The row switches thus occur at indices (4 + n*row_len) in the output data.
** This applies to all raw mode firmware (as of 4.0.d).
** In normal operation, the delay is accounted for automatically.
* The ADC_offset '''is not''' included in raw mode readings. So when servoing, the settled values should be compared to ADC offset.

== Raw mode firmware ==

Raw data acquisition and 4-pole low-pass filter are both memory-intensive features of the MCE readout-card firmware. As a general rule, a given firmware has only one of these features available, e.g., the firmware that has raw mode enabled will not include the filter and vice versa. In some versions that support raw mode, read back of gaini0/gainp0/gaind0 is also not supported.

{| border=1
! Firmware Revision !! Reported data (signed)
|-
| 4.0.d || 14-bit ADC samples, single column, 65536 points.
|-
| 4.3.7 || 14-bit ADC samples, 8 columns, 8192 points.
|-
| 4.2.7 || upper 8-bit of the 14-bit ADC samples (deprecated)
|-
| 4.1.7 || upper 8-bit of the 14-bit ADC sample (deprecated)
|}

== High(er)-level support ==

The scripts "mce_raw_acq" and "mce_raw_acq_1col" handle the details of disabling the sync-box, computing the number of frames the user probably wants, issuing the capture command, and reading out the data. Use them like this:

For 8-column readout:
* Tune and configure your array.
* Run, for some RC (e.g 1):
mce_raw_acq 1

This will create files <ctime>_raw, <ctime>_raw.run.

For 1-column readout:
* Tune and configure your array.
* Run, for some RC (e.g. 1) and column (e.g. 3):
mce_raw_acq_1col 1 3

Due to differences in firmware implementation, mce_raw_acq and mce_raw_acq_1col have different default behaviours.
* mce_raw_acq:
** 8-column firmware supports the acquisition of either 8192 samples, or two full multiplexing frames, whichever is lower.
** By default, mce_raw_acq attempts to acquire this maximum number of samples.
* mce_raw_acq_1col:
** 1-column firmware supports the acquisition of 65536 samples, regardless of multiplexing settings.
** By default, mce_raw_acq will attempt to acquire 65536 samples. It will change num_rows_reported to 32 in order to make the readout frame size go evenly into 65536.

== Low-level support ==

Raw mode readout is accomplished by first triggering the MCE to store 50 MHz data, and then reading out that data using a special data mode.

* There are two RC parameters that have to be set for raw data acquisition:
** '''rc# captr_raw''' stores a snapshot of 50 MHz data, starting at the next row-return-to-zero.
*** In 8-column firmware, ''row_len'' samples of ''num_rows'' rows for '''2''' consecutive frames up to 8192 entries for each channel and stores them in MCE memory buffer for readout in raw mode. The user must delay here before proceeding to read out the data. The delay should be a few return-to-zeros. (Computers are fast, and RTZ can take a long time if row_len is increased for noise measurements... so be generous when running from within scripts.)
*** In 1-column firmware, 65536 samples are stored.
** '''rc# data_mode''' determines the pixel-data readout mode.
*** The data_mode must be 3 for 8-column raw mode readout.
*** The data_mode must be 12 for 1-column raw mode readout.
** '''rc# readout_col_index''', in 1-column raw mode, selects the column to store.

* A typical sequence of commands for data acquisition is (8-column mode):
wb cc ret_dat_s 1 1
wb rc1 data_mode 3
wb rc1 captr_raw 1
acq_config ''filename'' rc1
acq_go ''row_len*2''

== Data handling ==

Raw mode data files are MCE flatfiles and can be loaded using the usual mas_data.pro or mce_data.py programs. However, the data should be time-ordered for meaningfullness. This is done by re-ordering the dimensions and collapsing the row and time dimensions.

In newer MASes, the reader programs are aware of data modes 3 and 12 and automatically reform the output data into an n_cols x n_samples array:
d = SmallMCEFile('1234000000_raw_col3').Read()
print d.data.shape
(1, 65536)

If you wanted to do the unravelling yourself, though, you would do this (assuming 8-column raw mode):
d = SmallMCEFile('1234000000_raw').Read(row_col=True, data_mode=0)
_, nc, _ = d.data.shape
d.data = d.data.transpose((1,2,0)).reshape((nc,-1))

Or in IDL (it's practically a one-liner!):
data = mas_data('1234000000_raw', /no_runfile, data_mode=0)
ds = size(d)
nc = ds[1]
nt = ds[2]*ds[3]
data_raw = reform(data, nc, nt)

= Details =

== Data storage and packaging ==

'''What follows below is a description of 8-column raw mode data packaging. For 1-column raw data, the data are packed in time order, and read out in units of the frame size. Values past the end of the 1-column buffer will read as -2^31 = 0x80000000. '''

* Once the raw-buffer of 8192 samples is full, no more samples are stored till a new captr_raw is issued.
* The raw-buffer is shipped out of MCE in a regular frame format (header + num_rows_reported*num_columns + checksum). As compared with normal MCE flatfile data organization, in raw mode the time index takes the place of row. At each time, all columns are reported.
* For rc1 only, for example, time indices t = [0, num_rows_reported-1] are reported in the first frame, with the offset of column c and time t being
index = t*8 + c
* Subsequent frames contain subsequent data samples. Since the number of time samples, per column, in a full multiplexing cycle of the MCE is row_len*num_rows, the number of readout frames required to record an entire MCE internal frame's-worth of raw data is equal to row_len (assuming that num_rows = num_rows_reported).
* The readout frame and index of a given time t, column c, is thus
frame = floor[ (t*8 + c) / num_rows_reported ]
index = (t*8 + c) mod num_rows_reported
* Since the number of clock ticks spent on each row is row_len, the row associated with time t is
row = floor [ t / row_len ]
* It is possible to get raw mode data from multiple readout cards; in that case the data will be packaged in the usual way, with all columns reporting at each time.
* Schematically, the organization of raw data for num_rows = num_rows_reported = 33 and row_len = 100 can be represented as follows (here sNNNNNcXX represents a word containing the data for time sample NNNNN, column XX):
(header of frame 0, 43 words)
s0000c00 s0000c01 s0000c02 s0000c03 s0000c04 s0000c05 s0000c06 s0000c07
s0001c00 s0001c01 s0001c02 s0001c03 s0001c04 s0001c05 s0001c06 s0001c07
s0002c00 s0002c01 s0002c02 s0002c03 s0002c04 s0002c05 s0002c06 s0002c07
...
s0032c00 s0032c01 s0032c02 s0032c03 s0032c04 s0032c05 s0032c06 s0032c07
(checksum of frame 0, 1 word)

(header of frame 1, 43 words)
s0033c00 s0033c01 s0033c02 s0033c03 s0033c04 s0033c05 s0033c06 s0033c07
...
s0065c00 s0065c01 s0065c02 s0065c03 s0065c04 s0065c05 s0065c06 s0065c07
(checksum of frame 1, 1 word)

...

(header of frame 199, 43 words)
s6567c67 s6567c01 s6567c02 s6567c03 s6567c04 s6567c05 s6567c06 s6567c07
...
s6599c00 s6599c01 s6599c02 s6599c03 s6599c04 s6599c05 s6599c06 s6599c07
(checksum of frame 199, 1 word)

* Unfortunately, collecting data for up to '''2''' internal MCE frames is hard-coded in firmware. Even if you reduce number of rows, you will be restricted by 2 frames. If you really want to look at more samples, then you may want to increase row_len instead and using row_order parameter of the address card, move your particular row of interest to the beginning of the frame.

* readout_row_index parameter is ignored when data_mode is set to 3 or raw mode.

* if num_rows_reported < num_rows, it takes more frames (to be exact: n = (num_rows*row_len*2/num_rows_reported) frames) to read all the data out.

== Firmware ==
* To toggle between raw data firmware and filtering firmware, one must un-comment one or the other (not both) of the following data banks:
** i_raw_dat_bank in adc_sample_coadd.vhd, for enabling/disabling raw data.
** i_fsfb_wn11_Q, i_fsfb_wn12_Q, i_fsfb_wn21_Q, i_fsfb_wn22_Q in fsfb_fltr_regs.vhd, for enabling/disabling filtered data.
* For some versions of Quartus, a quartus.ini file must be present in the synthesis directory for the changes in RAM configurations to register. The file contains the following line:
rtl_no_super_dupe_reg = on

Raw-mode readout

2009-04-24T09:47:32Z

174.6.181.2: /* Data storage and packaging */

= General Information =

* "Raw data" and "Raw mode" refer to the acquisition of ADC samples at 50 MHz.
* Raw data acquisition occurs in the 'background', i.e. it can occur while multiplexing and servoing are going on.
* Special readout card firmware is required to support raw mode. This firmware supports most normal readout card features, but does not support the low-pass filter nor the reading back of some parameters. It should be loaded temporarily (i.e. as a .sof).
** For 8-column raw mode readout, maximum 8192 samples per column:
http://www.phas.ubc.ca/~mce/mcedocs/firmware/rc_v04030007_26may2008_raw.sof
** For 1-column raw mode readout, maximum 65536 samples:
http://www.phas.ubc.ca/~mce/mcedocs/firmware/rc_v0400000d_17apr2009.sof
* The ADC readings are 14 bits, signed, and are reported as 32-bit signed integers.
* The ADC_offset '''is not''' included in raw mode readings. So when servoing, the settled values should be compared to ADC offset.

== Raw mode firmware ==

Raw data acquisition and 4-pole low-pass filter are both memory-intensive features of the MCE readout-card firmware. As a general rule, a given firmware has only one of these features available, e.g., the firmware that has raw mode enabled will not include the filter and vice versa. In some versions that support raw mode, read back of gaini0/gainp0/gaind0 is also not supported.

{| border=1
! Firmware Revision !! Reported data (signed)
|-
| 4.0.d || 14-bit ADC samples, single column, 65536 points.
|-
| 4.3.7 || 14-bit ADC samples, 8 columns, 8192 points.
|-
| 4.2.7 || upper 8-bit of the 14-bit ADC samples (deprecated)
|-
| 4.1.7 || upper 8-bit of the 14-bit ADC sample (deprecated)
|}

== High(er)-level support ==

The scripts "mce_raw_acq" and "mce_raw_acq_1col" handle the details of disabling the sync-box, computing the number of frames the user probably wants, issuing the capture command, and reading out the data. Use them like this:

For 8-column readout:
* Tune and configure your array.
* Run, for some RC (e.g 1):
mce_raw_acq 1

This will create files <ctime>_raw, <ctime>_raw.run.

For 1-column readout:
* Tune and configure your array.
* Run, for some RC (e.g. 1) and column (e.g. 3):
mce_raw_acq_1col 1 3

Due to differences in firmware implementation, mce_raw_acq and mce_raw_acq_1col have different default behaviours.
* mce_raw_acq:
** 8-column firmware supports the acquisition of either 8192 samples, or two full multiplexing frames, whichever is lower.
** By default, mce_raw_acq attempts to acquire this maximum number of samples.
* mce_raw_acq_1col:
** 1-column firmware supports the acquisition of 65536 samples, regardless of multiplexing settings.
** By default, mce_raw_acq will attempt to acquire 65536 samples. It will change num_rows_reported to 32 in order to make the readout frame size go evenly into 65536.

== Low-level support ==

Raw mode readout is accomplished by first triggering the MCE to store 50 MHz data, and then reading out that data using a special data mode.

* There are two RC parameters that have to be set for raw data acquisition:
** '''rc# captr_raw''' stores a snapshot of 50 MHz data, starting at the next row-return-to-zero.
*** In 8-column firmware, ''row_len'' samples of ''num_rows'' rows for '''2''' consecutive frames up to 8192 entries for each channel and stores them in MCE memory buffer for readout in raw mode. The user must delay here before proceeding to read out the data. The delay should be a few return-to-zeros. (Computers are fast, and RTZ can take a long time if row_len is increased for noise measurements... so be generous when running from within scripts.)
*** In 1-column firmware, 65536 samples are stored.
** '''rc# data_mode''' determines the pixel-data readout mode.
*** The data_mode must be 3 for 8-column raw mode readout.
*** The data_mode must be 12 for 1-column raw mode readout.
** '''rc# readout_col_index''', in 1-column raw mode, selects the column to store.

* A typical sequence of commands for data acquisition is (8-column mode):
wb cc ret_dat_s 1 1
wb rc1 data_mode 3
wb rc1 captr_raw 1
acq_config ''filename'' rc1
acq_go ''row_len*2''

== Data handling ==

Raw mode data files are MCE flatfiles and can be loaded using the usual mas_data.pro or mce_data.py programs. However, the data should be time-ordered for meaningfullness. This is done by re-ordering the dimensions and collapsing the row and time dimensions.

In newer MASes, the reader programs are aware of data modes 3 and 12 and automatically reform the output data into an n_cols x n_samples array:
d = SmallMCEFile('1234000000_raw_col3').Read()
print d.data.shape
(1, 65536)

If you wanted to do the unravelling yourself, though, you would do this (assuming 8-column raw mode):
d = SmallMCEFile('1234000000_raw').Read(row_col=True, data_mode=0)
_, nc, _ = d.data.shape
d.data = d.data.transpose((1,2,0)).reshape((nc,-1))

Or in IDL (it's practically a one-liner!):
data = mas_data('1234000000_raw', /no_runfile, data_mode=0)
ds = size(d)
nc = ds[1]
nt = ds[2]*ds[3]
data_raw = reform(data, nc, nt)

= Details =

== Data storage and packaging ==

'''What follows below is a description of 8-column raw mode data packaging. For 1-column raw data, the data are packed in time order, and read out in units of the frame size. Values past the end of the 1-column buffer will read as -2^31 = 0x80000000. '''

* Once the raw-buffer of 8192 samples is full, no more samples are stored till a new captr_raw is issued.
* The raw-buffer is shipped out of MCE in a regular frame format (header + num_rows_reported*num_columns + checksum). As compared with normal MCE flatfile data organization, in raw mode the time index takes the place of row. At each time, all columns are reported.
* For rc1 only, for example, time indices t = [0, num_rows_reported-1] are reported in the first frame, with the offset of column c and time t being
index = t*8 + c
* Subsequent frames contain subsequent data samples. Since the number of time samples, per column, in a full multiplexing cycle of the MCE is row_len*num_rows, the number of readout frames required to record an entire MCE internal frame's-worth of raw data is equal to row_len (assuming that num_rows = num_rows_reported).
* The readout frame and index of a given time t, column c, is thus
frame = floor[ (t*8 + c) / num_rows_reported ]
index = (t*8 + c) mod num_rows_reported
* Since the number of clock ticks spent on each row is row_len, the row associated with time t is
row = floor [ t / row_len ]
* It is possible to get raw mode data from multiple readout cards; in that case the data will be packaged in the usual way, with all columns reporting at each time.
* Schematically, the organization of raw data for num_rows = num_rows_reported = 33 and row_len = 100 can be represented as follows (here sNNNNNcXX represents a word containing the data for time sample NNNNN, column XX):
(header of frame 0, 43 words)
s0000c00 s0000c01 s0000c02 s0000c03 s0000c04 s0000c05 s0000c06 s0000c07
s0001c00 s0001c01 s0001c02 s0001c03 s0001c04 s0001c05 s0001c06 s0001c07
s0002c00 s0002c01 s0002c02 s0002c03 s0002c04 s0002c05 s0002c06 s0002c07
...
s0032c00 s0032c01 s0032c02 s0032c03 s0032c04 s0032c05 s0032c06 s0032c07
(checksum of frame 0, 1 word)

(header of frame 1, 43 words)
s0033c00 s0033c01 s0033c02 s0033c03 s0033c04 s0033c05 s0033c06 s0033c07
...
s0065c00 s0065c01 s0065c02 s0065c03 s0065c04 s0065c05 s0065c06 s0065c07
(checksum of frame 1, 1 word)

...

(header of frame 199, 43 words)
s6567c67 s6567c01 s6567c02 s6567c03 s6567c04 s6567c05 s6567c06 s6567c07
...
s6599c00 s6599c01 s6599c02 s6599c03 s6599c04 s6599c05 s6599c06 s6599c07
(checksum of frame 199, 1 word)

* Unfortunately, collecting data for up to '''2''' internal MCE frames is hard-coded in firmware. Even if you reduce number of rows, you will be restricted by 2 frames. If you really want to look at more samples, then you may want to increase row_len instead and using row_order parameter of the address card, move your particular row of interest to the beginning of the frame.

* readout_row_index parameter is ignored when data_mode is set to 3 or raw mode.

* if num_rows_reported < num_rows, it takes more frames (to be exact: n = (num_rows*row_len*2/num_rows_reported) frames) to read all the data out.

== Firmware ==
* To toggle between raw data firmware and filtering firmware, one must un-comment one or the other (not both) of the following data banks:
** i_raw_dat_bank in adc_sample_coadd.vhd, for enabling/disabling raw data.
** i_fsfb_wn11_Q, i_fsfb_wn12_Q, i_fsfb_wn21_Q, i_fsfb_wn22_Q in fsfb_fltr_regs.vhd, for enabling/disabling filtered data.
* For some versions of Quartus, a quartus.ini file must be present in the synthesis directory for the changes in RAM configurations to register. The file contains the following line:
rtl_no_super_dupe_reg = on

Raw-mode readout

2009-04-24T09:42:38Z

174.6.181.2: /* Low-level support */

= General Information =

* "Raw data" and "Raw mode" refer to the acquisition of ADC samples at 50 MHz.
* Raw data acquisition occurs in the 'background', i.e. it can occur while multiplexing and servoing are going on.
* Special readout card firmware is required to support raw mode. This firmware supports most normal readout card features, but does not support the low-pass filter nor the reading back of some parameters. It should be loaded temporarily (i.e. as a .sof).
** For 8-column raw mode readout, maximum 8192 samples per column:
http://www.phas.ubc.ca/~mce/mcedocs/firmware/rc_v04030007_26may2008_raw.sof
** For 1-column raw mode readout, maximum 65536 samples:
http://www.phas.ubc.ca/~mce/mcedocs/firmware/rc_v0400000d_17apr2009.sof
* The ADC readings are 14 bits, signed, and are reported as 32-bit signed integers.
* The ADC_offset '''is not''' included in raw mode readings. So when servoing, the settled values should be compared to ADC offset.

== Raw mode firmware ==

Raw data acquisition and 4-pole low-pass filter are both memory-intensive features of the MCE readout-card firmware. As a general rule, a given firmware has only one of these features available, e.g., the firmware that has raw mode enabled will not include the filter and vice versa. In some versions that support raw mode, read back of gaini0/gainp0/gaind0 is also not supported.

{| border=1
! Firmware Revision !! Reported data (signed)
|-
| 4.0.d || 14-bit ADC samples, single column, 65536 points.
|-
| 4.3.7 || 14-bit ADC samples, 8 columns, 8192 points.
|-
| 4.2.7 || upper 8-bit of the 14-bit ADC samples (deprecated)
|-
| 4.1.7 || upper 8-bit of the 14-bit ADC sample (deprecated)
|}

== High(er)-level support ==

The scripts "mce_raw_acq" and "mce_raw_acq_1col" handle the details of disabling the sync-box, computing the number of frames the user probably wants, issuing the capture command, and reading out the data. Use them like this:

For 8-column readout:
* Tune and configure your array.
* Run, for some RC (e.g 1):
mce_raw_acq 1

This will create files <ctime>_raw, <ctime>_raw.run.

For 1-column readout:
* Tune and configure your array.
* Run, for some RC (e.g. 1) and column (e.g. 3):
mce_raw_acq_1col 1 3

Due to differences in firmware implementation, mce_raw_acq and mce_raw_acq_1col have different default behaviours.
* mce_raw_acq:
** 8-column firmware supports the acquisition of either 8192 samples, or two full multiplexing frames, whichever is lower.
** By default, mce_raw_acq attempts to acquire this maximum number of samples.
* mce_raw_acq_1col:
** 1-column firmware supports the acquisition of 65536 samples, regardless of multiplexing settings.
** By default, mce_raw_acq will attempt to acquire 65536 samples. It will change num_rows_reported to 32 in order to make the readout frame size go evenly into 65536.

== Low-level support ==

Raw mode readout is accomplished by first triggering the MCE to store 50 MHz data, and then reading out that data using a special data mode.

* There are two RC parameters that have to be set for raw data acquisition:
** '''rc# captr_raw''' stores a snapshot of 50 MHz data, starting at the next row-return-to-zero.
*** In 8-column firmware, ''row_len'' samples of ''num_rows'' rows for '''2''' consecutive frames up to 8192 entries for each channel and stores them in MCE memory buffer for readout in raw mode. The user must delay here before proceeding to read out the data. The delay should be a few return-to-zeros. (Computers are fast, and RTZ can take a long time if row_len is increased for noise measurements... so be generous when running from within scripts.)
*** In 1-column firmware, 65536 samples are stored.
** '''rc# data_mode''' determines the pixel-data readout mode.
*** The data_mode must be 3 for 8-column raw mode readout.
*** The data_mode must be 12 for 1-column raw mode readout.
** '''rc# readout_col_index''', in 1-column raw mode, selects the column to store.

* A typical sequence of commands for data acquisition is (8-column mode):
wb cc ret_dat_s 1 1
wb rc1 data_mode 3
wb rc1 captr_raw 1
acq_config ''filename'' rc1
acq_go ''row_len*2''

== Data handling ==

Raw mode data files are MCE flatfiles and can be loaded using the usual mas_data.pro or mce_data.py programs. However, the data should be time-ordered for meaningfullness. This is done by re-ordering the dimensions and collapsing the row and time dimensions.

In newer MASes, the reader programs are aware of data modes 3 and 12 and automatically reform the output data into an n_cols x n_samples array:
d = SmallMCEFile('1234000000_raw_col3').Read()
print d.data.shape
(1, 65536)

If you wanted to do the unravelling yourself, though, you would do this (assuming 8-column raw mode):
d = SmallMCEFile('1234000000_raw').Read(row_col=True, data_mode=0)
_, nc, _ = d.data.shape
d.data = d.data.transpose((1,2,0)).reshape((nc,-1))

Or in IDL (it's practically a one-liner!):
data = mas_data('1234000000_raw', /no_runfile, data_mode=0)
ds = size(d)
nc = ds[1]
nt = ds[2]*ds[3]
data_raw = reform(data, nc, nt)

= Details =

== Data storage and packaging ==

* Once the raw-buffer of 8192 samples is full, no more samples are stored till a new captr_raw is issued.
* The raw-buffer is shipped out of MCE in a regular frame format (header + num_rows_reported*num_columns + checksum). As compared with normal MCE flatfile data organization, in raw mode the time index takes the place of row. At each time, all columns are reported.
* For rc1 only, for example, time indices t = [0, num_rows_reported-1] are reported in the first frame, with the offset of column c and time t being
index = t*8 + c
* Subsequent frames contain subsequent data samples. Since the number of time samples, per column, in a full multiplexing cycle of the MCE is row_len*num_rows, the number of readout frames required to record an entire MCE internal frame's-worth of raw data is equal to row_len (assuming that num_rows = num_rows_reported).
* The readout frame and index of a given time t, column c, is thus
frame = floor[ (t*8 + c) / num_rows_reported ]
index = (t*8 + c) mod num_rows_reported
* Since the number of clock ticks spent on each row is row_len, the row associated with time t is
row = floor [ t / row_len ]
* It is possible to get raw mode data from multiple readout cards; in that case the data will be packaged in the usual way, with all columns reporting at each time.
* Schematically, the organization of raw data for num_rows = num_rows_reported = 33 and row_len = 100 can be represented as follows (here sNNNNNcXX represents a word containing the data for time sample NNNNN, column XX):
(header of frame 0, 43 words)
s0000c00 s0000c01 s0000c02 s0000c03 s0000c04 s0000c05 s0000c06 s0000c07
s0001c00 s0001c01 s0001c02 s0001c03 s0001c04 s0001c05 s0001c06 s0001c07
s0002c00 s0002c01 s0002c02 s0002c03 s0002c04 s0002c05 s0002c06 s0002c07
...
s0032c00 s0032c01 s0032c02 s0032c03 s0032c04 s0032c05 s0032c06 s0032c07
(checksum of frame 0, 1 word)

(header of frame 1, 43 words)
s0033c00 s0033c01 s0033c02 s0033c03 s0033c04 s0033c05 s0033c06 s0033c07
...
s0065c00 s0065c01 s0065c02 s0065c03 s0065c04 s0065c05 s0065c06 s0065c07
(checksum of frame 1, 1 word)

...

(header of frame 199, 43 words)
s6567c67 s6567c01 s6567c02 s6567c03 s6567c04 s6567c05 s6567c06 s6567c07
...
s6599c00 s6599c01 s6599c02 s6599c03 s6599c04 s6599c05 s6599c06 s6599c07
(checksum of frame 199, 1 word)

* Unfortunately, collecting data for up to '''2''' internal MCE frames is hard-coded in firmware. Even if you reduce number of rows, you will be restricted by 2 frames. If you really want to look at more samples, then you may want to increase row_len instead and using row_order parameter of the address card, move your particular row of interest to the beginning of the frame.

* readout_row_index parameter is ignored when data_mode is set to 3 or raw mode.

* if num_rows_reported < num_rows, it takes more frames (to be exact: n = (num_rows*row_len*2/num_rows_reported) frames) to read all the data out.

== Firmware ==
* To toggle between raw data firmware and filtering firmware, one must un-comment one or the other (not both) of the following data banks:
** i_raw_dat_bank in adc_sample_coadd.vhd, for enabling/disabling raw data.
** i_fsfb_wn11_Q, i_fsfb_wn12_Q, i_fsfb_wn21_Q, i_fsfb_wn22_Q in fsfb_fltr_regs.vhd, for enabling/disabling filtered data.
* For some versions of Quartus, a quartus.ini file must be present in the synthesis directory for the changes in RAM configurations to register. The file contains the following line:
rtl_no_super_dupe_reg = on

Raw-mode readout

2009-04-24T09:41:33Z

174.6.181.2: /* Low-level support */

= General Information =

* "Raw data" and "Raw mode" refer to the acquisition of ADC samples at 50 MHz.
* Raw data acquisition occurs in the 'background', i.e. it can occur while multiplexing and servoing are going on.
* Special readout card firmware is required to support raw mode. This firmware supports most normal readout card features, but does not support the low-pass filter nor the reading back of some parameters. It should be loaded temporarily (i.e. as a .sof).
** For 8-column raw mode readout, maximum 8192 samples per column:
http://www.phas.ubc.ca/~mce/mcedocs/firmware/rc_v04030007_26may2008_raw.sof
** For 1-column raw mode readout, maximum 65536 samples:
http://www.phas.ubc.ca/~mce/mcedocs/firmware/rc_v0400000d_17apr2009.sof
* The ADC readings are 14 bits, signed, and are reported as 32-bit signed integers.
* The ADC_offset '''is not''' included in raw mode readings. So when servoing, the settled values should be compared to ADC offset.

== Raw mode firmware ==

Raw data acquisition and 4-pole low-pass filter are both memory-intensive features of the MCE readout-card firmware. As a general rule, a given firmware has only one of these features available, e.g., the firmware that has raw mode enabled will not include the filter and vice versa. In some versions that support raw mode, read back of gaini0/gainp0/gaind0 is also not supported.

{| border=1
! Firmware Revision !! Reported data (signed)
|-
| 4.0.d || 14-bit ADC samples, single column, 65536 points.
|-
| 4.3.7 || 14-bit ADC samples, 8 columns, 8192 points.
|-
| 4.2.7 || upper 8-bit of the 14-bit ADC samples (deprecated)
|-
| 4.1.7 || upper 8-bit of the 14-bit ADC sample (deprecated)
|}

== High(er)-level support ==

The scripts "mce_raw_acq" and "mce_raw_acq_1col" handle the details of disabling the sync-box, computing the number of frames the user probably wants, issuing the capture command, and reading out the data. Use them like this:

For 8-column readout:
* Tune and configure your array.
* Run, for some RC (e.g 1):
mce_raw_acq 1

This will create files <ctime>_raw, <ctime>_raw.run.

For 1-column readout:
* Tune and configure your array.
* Run, for some RC (e.g. 1) and column (e.g. 3):
mce_raw_acq_1col 1 3

Due to differences in firmware implementation, mce_raw_acq and mce_raw_acq_1col have different default behaviours.
* mce_raw_acq:
** 8-column firmware supports the acquisition of either 8192 samples, or two full multiplexing frames, whichever is lower.
** By default, mce_raw_acq attempts to acquire this maximum number of samples.
* mce_raw_acq_1col:
** 1-column firmware supports the acquisition of 65536 samples, regardless of multiplexing settings.
** By default, mce_raw_acq will attempt to acquire 65536 samples. It will change num_rows_reported to 32 in order to make the readout frame size go evenly into 65536.

== Low-level support ==

Raw mode readout is accomplished by first triggering the MCE to store 50 MHz data, and then reading out that data using a special data mode.

* There are two RC parameters that have to be set for raw data acquisition:
** '''rc# captr_raw''' stores a snapshot of 50 MHz data, starting at the next row-return-to-zero.
*** In 8-column firmware, ''row_len'' samples of ''num_rows'' rows for '''2''' consecutive frames up to 8192 entries for each channel and stores them in MCE memory buffer for readout in raw mode. The user must delay here before proceeding to read out the data. The delay should be a few return-to-zeros. (Computers are fast, and RTZ can take a long time if row_len is increased for noise measurements... so be generous when running from within scripts.)
*** In 1-column firmware, 65536 samples are stored.
** '''rc# data_mode''' determines the pixel-data readout mode.
*** The data_mode must be 3 for 8-column raw mode readout.
*** The data_mode must be 12 for 1-column raw mode readout.
** In 1-column raw mode, the desired column is set through 'rc# readout_col_index'. This number will appear in the runfile.

* A typical sequence of commands for data acquisition is (8-column mode):
wb cc ret_dat_s 1 1
wb rc1 data_mode 3
wb rc1 captr_raw 1
acq_config ''filename'' rc1
acq_go ''row_len*2''

== Data handling ==

Raw mode data files are MCE flatfiles and can be loaded using the usual mas_data.pro or mce_data.py programs. However, the data should be time-ordered for meaningfullness. This is done by re-ordering the dimensions and collapsing the row and time dimensions.

In newer MASes, the reader programs are aware of data modes 3 and 12 and automatically reform the output data into an n_cols x n_samples array:
d = SmallMCEFile('1234000000_raw_col3').Read()
print d.data.shape
(1, 65536)

If you wanted to do the unravelling yourself, though, you would do this (assuming 8-column raw mode):
d = SmallMCEFile('1234000000_raw').Read(row_col=True, data_mode=0)
_, nc, _ = d.data.shape
d.data = d.data.transpose((1,2,0)).reshape((nc,-1))

Or in IDL (it's practically a one-liner!):
data = mas_data('1234000000_raw', /no_runfile, data_mode=0)
ds = size(d)
nc = ds[1]
nt = ds[2]*ds[3]
data_raw = reform(data, nc, nt)

= Details =

== Data storage and packaging ==

* Once the raw-buffer of 8192 samples is full, no more samples are stored till a new captr_raw is issued.
* The raw-buffer is shipped out of MCE in a regular frame format (header + num_rows_reported*num_columns + checksum). As compared with normal MCE flatfile data organization, in raw mode the time index takes the place of row. At each time, all columns are reported.
* For rc1 only, for example, time indices t = [0, num_rows_reported-1] are reported in the first frame, with the offset of column c and time t being
index = t*8 + c
* Subsequent frames contain subsequent data samples. Since the number of time samples, per column, in a full multiplexing cycle of the MCE is row_len*num_rows, the number of readout frames required to record an entire MCE internal frame's-worth of raw data is equal to row_len (assuming that num_rows = num_rows_reported).
* The readout frame and index of a given time t, column c, is thus
frame = floor[ (t*8 + c) / num_rows_reported ]
index = (t*8 + c) mod num_rows_reported
* Since the number of clock ticks spent on each row is row_len, the row associated with time t is
row = floor [ t / row_len ]
* It is possible to get raw mode data from multiple readout cards; in that case the data will be packaged in the usual way, with all columns reporting at each time.
* Schematically, the organization of raw data for num_rows = num_rows_reported = 33 and row_len = 100 can be represented as follows (here sNNNNNcXX represents a word containing the data for time sample NNNNN, column XX):
(header of frame 0, 43 words)
s0000c00 s0000c01 s0000c02 s0000c03 s0000c04 s0000c05 s0000c06 s0000c07
s0001c00 s0001c01 s0001c02 s0001c03 s0001c04 s0001c05 s0001c06 s0001c07
s0002c00 s0002c01 s0002c02 s0002c03 s0002c04 s0002c05 s0002c06 s0002c07
...
s0032c00 s0032c01 s0032c02 s0032c03 s0032c04 s0032c05 s0032c06 s0032c07
(checksum of frame 0, 1 word)

(header of frame 1, 43 words)
s0033c00 s0033c01 s0033c02 s0033c03 s0033c04 s0033c05 s0033c06 s0033c07
...
s0065c00 s0065c01 s0065c02 s0065c03 s0065c04 s0065c05 s0065c06 s0065c07
(checksum of frame 1, 1 word)

...

(header of frame 199, 43 words)
s6567c67 s6567c01 s6567c02 s6567c03 s6567c04 s6567c05 s6567c06 s6567c07
...
s6599c00 s6599c01 s6599c02 s6599c03 s6599c04 s6599c05 s6599c06 s6599c07
(checksum of frame 199, 1 word)

* Unfortunately, collecting data for up to '''2''' internal MCE frames is hard-coded in firmware. Even if you reduce number of rows, you will be restricted by 2 frames. If you really want to look at more samples, then you may want to increase row_len instead and using row_order parameter of the address card, move your particular row of interest to the beginning of the frame.

* readout_row_index parameter is ignored when data_mode is set to 3 or raw mode.

* if num_rows_reported < num_rows, it takes more frames (to be exact: n = (num_rows*row_len*2/num_rows_reported) frames) to read all the data out.

== Firmware ==
* To toggle between raw data firmware and filtering firmware, one must un-comment one or the other (not both) of the following data banks:
** i_raw_dat_bank in adc_sample_coadd.vhd, for enabling/disabling raw data.
** i_fsfb_wn11_Q, i_fsfb_wn12_Q, i_fsfb_wn21_Q, i_fsfb_wn22_Q in fsfb_fltr_regs.vhd, for enabling/disabling filtered data.
* For some versions of Quartus, a quartus.ini file must be present in the synthesis directory for the changes in RAM configurations to register. The file contains the following line:
rtl_no_super_dupe_reg = on

Raw-mode readout

2009-04-24T09:12:57Z

174.6.181.2: /* General Information */

= General Information =

* "Raw data" and "Raw mode" refer to the acquisition of ADC samples at 50 MHz.
* Raw data acquisition occurs in the 'background', i.e. it can occur while multiplexing and servoing are going on.
* Special readout card firmware is required to support raw mode. This firmware supports most normal readout card features, but does not support the low-pass filter nor the reading back of some parameters. It should be loaded temporarily (i.e. as a .sof).
** For 8-column raw mode readout, maximum 8192 samples per column:
http://www.phas.ubc.ca/~mce/mcedocs/firmware/rc_v04030007_26may2008_raw.sof
** For 1-column raw mode readout, maximum 65536 samples:
http://www.phas.ubc.ca/~mce/mcedocs/firmware/rc_v0400000d_17apr2009.sof
* The ADC readings are 14 bits, signed, and are reported as 32-bit signed integers.
* The ADC_offset '''is not''' included in raw mode readings. So when servoing, the settled values should be compared to ADC offset.

== Raw mode firmware ==

Raw data acquisition and 4-pole low-pass filter are both memory-intensive features of the MCE readout-card firmware. As a general rule, a given firmware has only one of these features available, e.g., the firmware that has raw mode enabled will not include the filter and vice versa. In some versions that support raw mode, read back of gaini0/gainp0/gaind0 is also not supported.

{| border=1
! Firmware Revision !! Reported data (signed)
|-
| 4.0.d || 14-bit ADC samples, single column, 65536 points.
|-
| 4.3.7 || 14-bit ADC samples, 8 columns, 8192 points.
|-
| 4.2.7 || upper 8-bit of the 14-bit ADC samples (deprecated)
|-
| 4.1.7 || upper 8-bit of the 14-bit ADC sample (deprecated)
|}

== High(er)-level support ==

The scripts "mce_raw_acq" and "mce_raw_acq_1col" handle the details of disabling the sync-box, computing the number of frames the user probably wants, issuing the capture command, and reading out the data. Use them like this:

For 8-column readout:
* Tune and configure your array.
* Run, for some RC (e.g 1):
mce_raw_acq 1

This will create files <ctime>_raw, <ctime>_raw.run.

For 1-column readout:
* Tune and configure your array.
* Run, for some RC (e.g. 1) and column (e.g. 3):
mce_raw_acq_1col 1 3

Due to differences in firmware implementation, mce_raw_acq and mce_raw_acq_1col have different default behaviours.
* mce_raw_acq:
** 8-column firmware supports the acquisition of either 8192 samples, or two full multiplexing frames, whichever is lower.
** By default, mce_raw_acq attempts to acquire this maximum number of samples.
* mce_raw_acq_1col:
** 1-column firmware supports the acquisition of 65536 samples, regardless of multiplexing settings.
** By default, mce_raw_acq will attempt to acquire 65536 samples. It will change num_rows_reported to 32 in order to make the readout frame size go evenly into 65536.

== Low-level support ==

Raw mode readout is accomplished by first triggering the MCE to store 50 MHz data, and then reading out that data using a special data mode.

* There are two RC parameters that have to be set for raw data acquisition:
** '''rc# captr_raw''' stores a snapshot of 50 MHz data, starting at the next row-return-to-zero.
*** In 8-column firmware, ''row_len'' samples of ''num_rows'' rows for '''2''' consecutive frames up to 8192 entries for each channel and stores them in MCE memory buffer for readout in raw mode. The user must delay here before proceeding to read out the data. The delay should be a few return-to-zeros. (Computers are fast, and RTZ can take a long time if row_len is increased for noise measurements... so be generous when running from within scripts.)
*** In 1-column firmware, 65536 samples are stored for the column indicated by 'rc# readout_col_index'
** '''rc# data_mode''' determines the pixel-data readout mode.
*** The data_mode must be 3 for 8-column raw mode readout.
*** The data_mode must be 12 for 8-column raw mode readout.
* A typical sequence of commands for data acquisition is:
wb cc ret_dat_s 1 1
wb rc1 data_mode 3
wb rc1 captr_raw 1
acq_config ''filename'' rc1
acq_go ''row_len*2''

== Data handling ==

Raw mode data files are MCE flatfiles and can be loaded using the usual mas_data.pro or mce_data.py programs. However, the data should be time-ordered for meaningfullness. This is done by re-ordering the dimensions and collapsing the row and time dimensions.

In newer MASes, the reader programs are aware of data modes 3 and 12 and automatically reform the output data into an n_cols x n_samples array:
d = SmallMCEFile('1234000000_raw_col3').Read()
print d.data.shape
(1, 65536)

If you wanted to do the unravelling yourself, though, you would do this (assuming 8-column raw mode):
d = SmallMCEFile('1234000000_raw').Read(row_col=True, data_mode=0)
_, nc, _ = d.data.shape
d.data = d.data.transpose((1,2,0)).reshape((nc,-1))

Or in IDL (it's practically a one-liner!):
data = mas_data('1234000000_raw', /no_runfile, data_mode=0)
ds = size(d)
nc = ds[1]
nt = ds[2]*ds[3]
data_raw = reform(data, nc, nt)

= Details =

== Data storage and packaging ==

* Once the raw-buffer of 8192 samples is full, no more samples are stored till a new captr_raw is issued.
* The raw-buffer is shipped out of MCE in a regular frame format (header + num_rows_reported*num_columns + checksum). As compared with normal MCE flatfile data organization, in raw mode the time index takes the place of row. At each time, all columns are reported.
* For rc1 only, for example, time indices t = [0, num_rows_reported-1] are reported in the first frame, with the offset of column c and time t being
index = t*8 + c
* Subsequent frames contain subsequent data samples. Since the number of time samples, per column, in a full multiplexing cycle of the MCE is row_len*num_rows, the number of readout frames required to record an entire MCE internal frame's-worth of raw data is equal to row_len (assuming that num_rows = num_rows_reported).
* The readout frame and index of a given time t, column c, is thus
frame = floor[ (t*8 + c) / num_rows_reported ]
index = (t*8 + c) mod num_rows_reported
* Since the number of clock ticks spent on each row is row_len, the row associated with time t is
row = floor [ t / row_len ]
* It is possible to get raw mode data from multiple readout cards; in that case the data will be packaged in the usual way, with all columns reporting at each time.
* Schematically, the organization of raw data for num_rows = num_rows_reported = 33 and row_len = 100 can be represented as follows (here sNNNNNcXX represents a word containing the data for time sample NNNNN, column XX):
(header of frame 0, 43 words)
s0000c00 s0000c01 s0000c02 s0000c03 s0000c04 s0000c05 s0000c06 s0000c07
s0001c00 s0001c01 s0001c02 s0001c03 s0001c04 s0001c05 s0001c06 s0001c07
s0002c00 s0002c01 s0002c02 s0002c03 s0002c04 s0002c05 s0002c06 s0002c07
...
s0032c00 s0032c01 s0032c02 s0032c03 s0032c04 s0032c05 s0032c06 s0032c07
(checksum of frame 0, 1 word)

(header of frame 1, 43 words)
s0033c00 s0033c01 s0033c02 s0033c03 s0033c04 s0033c05 s0033c06 s0033c07
...
s0065c00 s0065c01 s0065c02 s0065c03 s0065c04 s0065c05 s0065c06 s0065c07
(checksum of frame 1, 1 word)

...

(header of frame 199, 43 words)
s6567c67 s6567c01 s6567c02 s6567c03 s6567c04 s6567c05 s6567c06 s6567c07
...
s6599c00 s6599c01 s6599c02 s6599c03 s6599c04 s6599c05 s6599c06 s6599c07
(checksum of frame 199, 1 word)

* Unfortunately, collecting data for up to '''2''' internal MCE frames is hard-coded in firmware. Even if you reduce number of rows, you will be restricted by 2 frames. If you really want to look at more samples, then you may want to increase row_len instead and using row_order parameter of the address card, move your particular row of interest to the beginning of the frame.

* readout_row_index parameter is ignored when data_mode is set to 3 or raw mode.

* if num_rows_reported < num_rows, it takes more frames (to be exact: n = (num_rows*row_len*2/num_rows_reported) frames) to read all the data out.

== Firmware ==
* To toggle between raw data firmware and filtering firmware, one must un-comment one or the other (not both) of the following data banks:
** i_raw_dat_bank in adc_sample_coadd.vhd, for enabling/disabling raw data.
** i_fsfb_wn11_Q, i_fsfb_wn12_Q, i_fsfb_wn21_Q, i_fsfb_wn22_Q in fsfb_fltr_regs.vhd, for enabling/disabling filtered data.
* For some versions of Quartus, a quartus.ini file must be present in the synthesis directory for the changes in RAM configurations to register. The file contains the following line:
rtl_no_super_dupe_reg = on

Locking slopes

2009-04-17T09:16:25Z

174.6.181.2:

= MCE/MAS parameters that affect locking slopes =

== Slope controls ==

The following parameters in [[Mce config template system | experiment.cfg]] affect the locking slopes:

* '''sq2servo_gain''': sets the gain of the SQ2 servo and effectively determines the sign of the SA lock slope.
* '''sq1servo_gain''': sets the gain of the SQ1 servo and, with the sq2servo_gain, determines the SQ2 lock slope.
* '''default_servo_i''': determines the I gain of the MCE servo (rc# gaini#), and thus the sign of the locking slopeof the SQ1.

== Plotting controls ==

The following parameters affect plotting of locking slopes in IDL, as well as the choice of initial guesses for lock-points:

* sq2servo_slope in experiment.cfg: (+1 or -1) the sign of the slope in the SQ2 servo plots to which we expect to lock.
* sq1servo_slope in experiment.cfg: (+1 or -1) the sign of the slope in the SQ1 servo plots to which we expect to lock.

== Consistency between locking and plotting ==

The above slope and plotting parameters should satisfy the following:
* sq2servo_slope = -sign( sq2servo_gain / sq1servo_gain)
* sq1servo_slope = -sign( sq1servo_gain / default_servo_i)

These relations follow from the discussion in the sections below.

Violation of the above will likely only result in misleading plots, and not bad locking. Misleading plots can indirectly lead to a user setting up bad locking, however; so keep it clean.

'''The sa_points.ps plot may be misleading''': the code (at time of writing) plots the positive-slope lock-points, regardless of any other parameters.

Future code will eliminate sq2servo_slope and sq1servo_slope and determine all locking slopes from the servo gains...

= MCE and tuning servos =

== Servos ==

We will need to deal with three servos. One is the "MCE servo", which is the fast servoing of the TES signal via adjustment of the SQ1 feedback. The other two are the auto-tuning SQ2 and SQ1 servos, described [here | auto_setup_squids.pro].

In all cases, the servo is computed by measuring an error (<math>\Delta</math> y), multiplying by a gain (g), and using this to correct the output (x):
<math>\Delta x = g~\Delta y</math>

In the case of the MCE PID loop servo, we run with an I term only so we actually calculate
<math>x = P ~\sum \Delta y</math>

but differentially this is exactly <math>\Delta x = P * \Delta y</math>.

== MCE/MAS and sneaky signs ==

There are no sneaky, extra signs in any of the MCE/MAS servo calculations. This means that given an error (dy) and a gain g, we calculate the change in the feedback (dx) as:
<math>dx = g~dy</math>

and not
<math>dx = -g~dy</math>

There is a sneaky gain in the MCE servo, though; once the default_servo_i is set, the DAC value of the feedback is actually calculated as
<math>\Delta FB = (I~dy) / 2^{12}</math>

But there's no sneaky sign. Just a sneaky gain.

= The meaning of the plots =

== Definition of a lock point ==

Let the feedback signals on the three stages (in one channel) be <math>x_{SA}, x_{S2}, x_{S1}</math>, and let the error signal be y. Let s be the signal from the TES. The error signal is obviously a function of all 4 numbers:
<math>y = f(x_{SA}, x_{S2}, x_{S1}, t)</math>

But for small variations about one point in this space, we can expand in differentials
<math> dy = a_1 dx_{SA} + a_2 dx_{S2} + a_3 dx_{S1} + a_4 s</math>
but it is convenient to reparametrize this to resemble the chain of amplifiers:
<math> dy = m_{SA}(dx_{SA} - m_{S2}(dx_{S2} - m_{S1}(dx_{S1} - m_{TES} s)</math>

So the 'slopes' m account for the instantaneous gain between the stages at the lock point.

== Open loop ramps ==

Note that lock points are not ultimately chosen as a result of the open loop ramps. However, it is important to understand the relationship between what the open loop ramps and the settings for the auto-tuning and MCE servos.

For the "SA ramp", we hold <math>x_{S2}, x_{S1}</math> and s constant, and plot y vs. <math>x_{SA}</math>. Thus identifying a lock point permits the measurement of
<math>dy / dx_{SA} = m_{SA}</math>

For the "SQ1 ramp", we hold <math>x_{SA}, x_{S2}</math> and s constant, and plot y vs. <math>x_{S1}</math>. A slope measurement on this curve is a measurement of
<math>dy / dx_{S1} = m_{SA} m_{S2} m_{S1}</math>

== Servos ==

In servos, we ramp one stage's x while adjusting another x to keep the error at zero.

Thus in the "SQ2 servo" step (where SQ2 FB is ramped and SQA FB adjusted) we have:
<math>dx_{SA} = m_{S2} dx_{S2}</math>

However, this can only be accomplished via measurement of the error
<math>\Delta y = -m_{SA} m_{S2} \Delta x_{S2}</math>
so operationally we apply
<math>\Delta x_{SA} = - \Delta y / m_{SA}</math>

This tells us the ideal gain of the SQ2 servo:
<math>g_{S2} = -1/m_{SA}</math>

In the "SQ1 servo" step (where SQ1 FB is ramped and SQ2 FB adjusted) we have:
<math>dx_{S2} = m_{S1} dx_{S1}</math>
so we measure
<math>\Delta y = m_{SA} m_{S2} m_{S1} \Delta x_{S1}</math>
and apply
<math>\Delta x_{S2} = \Delta y / (m_{SA} m_{S2})</math>.

so the ideal gain of the SQ1 servo is
<math>g_{S1} = 1/(m_{SA} m_{S2})</math>.

== MCE servo I parameter ==

When the MCE servo is activated, it cancels changes in TES current by adjusting the SQ1 FB:
<math>dx_{S1} = m_{TES} ds</math>

i.e.
<math>\Delta x_{S1} = -\Delta y / (m_{S1} m_{S2} m_{S3})</math>
and ideal gain is thus
<math>I_{servo} = -1 / (m_{S1} m_{S2} m_{S3})</math>

== Consequences for tuning parameters ==

We have expressed, above, the ideal gains for the servos as a function of the slopes of the ramp and servo curves from auto-tuning. Solving these for the slopes as a function of the ideal gains yields the locking slopes' dependence on the signs of the gains:

=== SA locking point ===

The locking slope (on the SA ramp) is determined by the gain in the SQ2 servo:
<math>sgn(m_{SA}) = -sgn(1/g_{S2})</math>

=== SQ2 locking point ===

The locking slope (on the SQ2 servo curve) is determined by the ratio of the gains in the SQ1 and SQ2 servos:
<math>sgn(m_{S2}) = -sgn(g_{S2}/g_{S1})</math>

=== SQ1 locking point ===

The locking slope on the SQ1 servo curve is determined by the ratio of the gains in the SQ1 servo and the MCE servo I term:
<math>sgn(m_{S1}) = -sgn(g_{S1}/I)</math>

But the sign of the locking slope on the SQ1 ramp curve is simply:
<math>sgn(m_{SA} m_{S2} m_{S1}) = - sgn(1/I)</math>

Locking slopes

2009-04-17T08:45:56Z

174.6.181.2:

= MCE/MAS parameters that affect locking slopes =

== Slope controls ==

The following parameters in [[MCE config template system | experiment.cfg]] affect the locking slopes:

* '''sq2servo_gain''': sets the gain of the SQ2 servo and effectively determines the sign of the SA lock slope.
* '''sq1servo_gain''': sets the gain of the SQ1 servo and, with the sq2servo_gain, determines the SQ2 lock slope.
* '''default_servo_i''': determines the I gain of the MCE servo (rc# gaini#), and thus the sign of the locking slopeof the SQ1.

== Plotting controls ==

The following parameters affect plotting of locking slopes in IDL, as well as the choice of initial guesses for lock-points:

* sq2servo_slope in experiment.cfg: (+1 or -1) the sign of the slope in the SQ2 servo plots to which we expect to lock.
* sq1servo_slope in experiment.cfg: (+1 or -1) the sign of the slope in the SQ1 servo plots to which we expect to lock.

== Consistency between locking and plotting ==

The above slope and plotting parameters should satisfy the following:
* sq2servo_slope = -sign( sq2servo_gain / sq1servo_gain)
* sq1servo_slope = -sign( sq1servo_gain / default_servo_i)

These relations follow from the discussion in the sections below.

Violation of the above will likely only result in misleading plots, and not bad locking. Misleading plots can indirectly lead to a user setting up bad locking, however; so keep it clean.

'''The sa_points.ps plot may be misleading''': the code (at time of writing) plots the positive-slope lock-points, regardless of any other parameters.

Future code will eliminate sq2servo_slope and sq1servo_slope and determine all locking slopes from the servo gains...

= MCE and tuning servos =

== Servos ==

We will need to deal with three servos. One is the "MCE servo", which is the fast servoing of the TES signal via adjustment of the SQ1 feedback. The other two are the auto-tuning SQ2 and SQ1 servos, described [here | auto_setup_squids.pro].

In all cases, the servo is computed by measuring an error (\Delta y), multiplying by a gain (g), and using this to correct the output (x):
\Delta x = g \Delta y

In the case of the MCE PID loop servo, we run with an I term only so we actually calculate
x = P * \sum dy
but differentially this is exactly \Delta x = P * \Delta y.

== MCE/MAS and sneaky signs ==

There are no sneaky, extra signs in any of the MCE/MAS servo calculations. This means that given an error (dy) and a gain g, we calculate the change in the feedback (dx) as:
dx = g * dy
and not
dx = -g * dy (not used)

There is a sneaky gain in the MCE servo, though; once the default_servo_i is set, the DAC value of the feedback is actually calculated as
\Delta FB = (I dy) / 2^{12}

But there's no sneaky sign. Just a sneaky gain.

= The meaning of the plots =

== Definition of a lock point ==

Let the feedback signals on the three stages (in one channel) be x_{SA}, x_{S2}, x_{S1}, and let the error signal be y. Let s be the signal from the TES. The error signal is obviously a function of all 4 numbers:
y = f(x_{SA}, x_{S2}, x_{S1}, t)

But for small variations about one point in this space, we can expand in differentials
dy = a_1 dx_{SA} + a_2 dx_{S2} + a_3 dx_{S1} + a_4 s
but it is convenient to reparametrize this to resemble the chain of amplifiers:
dy = m_{SA}(dx_{SA} - m_{S2}(dx_{S2} - m_{S1}(dx_{S1} - m_{TES} s)

So the 'slopes' m account for the instantaneous gain between the stages at the lock point.

== Open loop ramps ==

Note that lock points are not ultimately chosen as a result of the open loop ramps. However, it is important to understand the relationship between what the open loop ramps and the settings for the auto-tuning and MCE servos.

For the "SA ramp", we hold x_{S2}, x_{S1} and s constant, and plot y vs. x_{SA}. Thus identifying a lock point permits the measurement of
dy / dx_{SA} = m_{SA}

For the "SQ1 ramp", we hold x_{SA}, x_{S2} and s constant, and plot y vs. x_{S1}. A slope measurement on this curve is a measurement of
dy / dx_{S1} = m_{SA} m_{S2} m_{S1}

== Servos ==

In servos, we ramp one stage's x while adjusting another x to keep the error at zero.

Thus in the "SQ2 servo" step (where SQ2 FB is ramped and SQA FB adjusted) we have:
dx_{SA} = m_{S2} dx_{S2}

However, this can only be accomplished via measurement of the error
\Delta y = -m_{SA} m_{S2} \Delta x_{S2}
so operationally we apply
\Delta x_{SA} = - \Delta y / m_{SA}

This tells us the ideal gain of the SQ2 servo:
g_{S2} = -1/m_{SA}

In the "SQ1 servo" step (where SQ1 FB is ramped and SQ2 FB adjusted) we have:
dx_{S2} = m_{S1} dx_{S1}
so we measure
\Delta y = m_{SA} m_{S2} m_{S1} \Delta x_{S1}
and apply
\Delta x_{S2} = \Delta y / (m_{SA} m_{S2}).

so the ideal gain of the sq1 servo is
g_{S1} = 1/(m_{SA} m_{S2})

== MCE servo I parameter ==

When the MCE servo is activated, it cancels changes in TES current by adjusting the SQ1 FB:
dx_{S1} = m_{TES} ds

i.e.
\Delta x_{S1} = -\Delta y / (m_{S1} m_{S2} m_{S3})
and ideal gain is thus
I_{servo} = -1 / (m_{S1} m_{S2} m_{S3})

== Consequences for tuning parameters ==

We have expressed, above, the ideal gains for the servos as a function of the slopes of the ramp and servo curves from auto-tuning. Solving these for the slopes as a function of the ideal gains yields the locking slopes' dependence on the signs of the gains:

=== SA locking point ===

The locking slope (on the SA ramp) is determined by the gain in the SQ2 servo:
sign(m_{SA}) = -sign(1/g_{S2})

=== SQ2 locking point ===

The locking slope (on the SQ2 servo curve) is determined by the ratio of the gains in the SQ1 and SQ2 servos:
sign(m_{S2}) = -sign(g_{S2}/g_{S1})

=== SQ1 locking point ===

The locking slope on the SQ1 servo curve is determined by the ratio of the gains in the SQ1 servo and the MCE servo I term:
sign(m_{S1}) = -sign(g_{S1}/I)

But the sign of the locking slope on the SQ1 ramp curve is simply:
sign(m_{SA} m_{S2} m_{S1}) = - sign(1/I)

Python data and runfile modules

2009-04-05T22:10:50Z

174.6.181.2:

Recent changes:
* mce_script/trunk r408: Merged mce_runfile.py into mce_data.py. Use mce_data.py for everything.
* mce_script/trunk r407: SmallMCEFile is smarter about frame counts, and less verbose unless challenged.

The interfaces described here are based on mce_script/trunk r408.

= mce_data.py and mce_runfile.py =

== Notes ==

The module mce_data.py provide roughly the same functionality that [[ mas_data.pro ]] and [[ mas_runfile.pro and mas_runparam.pro | mas_runfile.pro ]] provide for IDL.

In the latest versions of the scripts, mce_runfile.py is no longer needed as the MCERunfile class has been merged into mce_data.py

== System setup ==

Make sure python can find mce_data.py (and possibly mce_runfile.py). $MAS_PYTHON should be in your PYTHONPATH environment variable.
mce@mce-ubc-2:~$ export PYTHONPATH=$PYTHONPATH:$MAS_PYTHON

This should be done automatically if you "source $MAS_ROOT/template/mas_env.bash" in your .bashrc".

= MCEFile / SmallMCEFile classes =

The MCEFile class is an alias for the SmallMCEFile class. These can be used interchangeably. The "Small" is just meant to suggest that you shouldn't try to load more frame data than your machine's RAM can handle. The module will complain and prevent you from loading more than 100 MB of data in one go.

== Basic usage ==

In python, import the 'mce_data' module and create a SmallMCEFile object:
>>> from mce_data import *
>>> fn = '/data/cryo/current_data/data002'
>>> f = SmallMCEFile(fn)

To suppress loading of the runfile, or override the runfile name, use one of these:
>>> f = SmallMCEFile( fn, runfile=False)
>>> f = SmallMCEFIle( fn, runfile='template_runfile.run')

Then load the data like this:
>>> d = f.Read()

The data member of d is a 2d array, [n_detectors, n_frames]:
>>> d.data.shape
(328, 100)

The list of rows and columns associated with the 328 detectors is available through col_list and row_list:
>>> d.col_list
[8, 9, 10, 11, 12, 13, 14, 15, 8, 9, 10, 11, 12, 13, 14, 15,
8, 9, 10, 11, 12, 13, 14, 15, 8, 9, 10, 11, 12, 13, 14, 15,
...
8, 9, 10, 11, 12, 13, 14, 15]

To display a pixel's values in all the data frames of a file:
>>> d.data[col+row*8]

Or to see pixel index i:
>>> d.data[i,:]

To display all pixels in frame f:
>>> d.data[:,f]

The data from the first frame header is available in the header member:
>>> d.header
{'status': 2052, 'data_rate': 47, 'userfield': 0, 'num_rows_reported': 41,
'runfile_id': 1231969044, 'row_len': 64, 'header_version': 6, 'rc_present':
[False, True, False, False], 'address0_ctr': 23142826, 'num_rows': 41,
'ramp_value': 0, 'frame_counter': 0, 'sync_box_num': 0, 'ramp_addr': 0}

== Raw frame data ==

To get raw frames (including header and checksum), pass "raw_frames=True" to the Read call:

>>> raw = f.Read(raw_frames=True)
>>> raw.shape
(100, 1100)

Then you can look at, e.g., the checksum:
>>> raw[:,1099]
>>> dd[:,1099]
array([918158926, 918144283, 918156148, 918156615, 918153934, 918159205,
...
918146379, 918145339, 918158232, 918146376], dtype=int32)

== Force row/column ordering ==

To reformat the d.data array so that its indices are (row, column, frame), pass "row_col=True" to Read:
>>> d = f.Read(row_col=True)
>>> d.data.shape
(33, 32, 100)

Then to see frame k:
>>> d.data[:,:,k]

== Extracting fields from mixed mode data ==

The MCE signal names, for mce_data purposes, are:
error - co-added error (reference mode 0)
fb - feedback in sq1 DAC units (reference mode 1)
fb_filt - filtered feedback (reference mode 2)
fj - flux jump counter

For mixed modes, the default is to extract the feedback or filtered feedback signal by default. To extract an alternate signal, pass "field=..." to the Read call:
>>> d = f.Read(field='fj')
>>> print d.data
array([[0,0,0,0,1,0,0,0,0,
...

It is possible to extract a set of fields simultaneously, using the "fields=" option in the Read call. This will override the "field=" option. The extracted fields appear in a dictionary in d.data.

>>> d = f.Read(fields=['fj', 'fb_filt'])
>>> print d.data['fj']
...
>>> print d.data['fb_filt']
...

You can extract all fields by passing "fields='all'":

>>> d = f.Read(fields='all')
>>> print d.data['fj']
...
>>> print d.data['fb_filt']
...

To force the bitfield extraction code to assume a particular data_mode, pass "data_mode=..." to Read.

>>> d = f.Read(data_mode=10, field='fb_filt')

= MCERunfile class =

Note that in older versions of mce_script, the MCERunfile class is contained in the file mce_runfile.py. In that case, replace "from mce_data import ..." with "from mce_runfile import ..." in what follows.

== Load a runfile ==

>>> from mce_runfile import *
>>> runfile_name = '/data/cryo/current_data/1220531790_dat.run'
>>> rf = MCERunfile(runfile_name)

== Simple data ==

Recall that the structure of [[ Runfile format v2 | runfiles ]] is such that a line of data has an address defined by its 'block' and 'key' (where the key is the tag + specifiers...). The contents of rf include a dictionary of dictionaries of all the block / key pairs. For example:

>>> print rf.data['HEADER']['RB rc1 data_mode']
00000010
>>> print rf.data['SQUID']['SQ_tuning_dir']
1220510497

However, the member function "Item" allows us to repackage the runfile data by specifying a data type ('string', 'int', 'float') and whether or not we expect an array or a single value. For example:

>>> print rf.Item('HEADER', 'RB rc1 data_mode')
['00000010']
>>> print rf.Item('HEADER', 'RB rc1 data_mode', type='int')
[10]
>>> print rf.Item('HEADER', 'RB rc1 data_mode', type='int', array=False)
10
>>> print rf.Item('HEADER', 'RB rc1 data_mode', type='float')
[10.0]

== Two-dimensional data ==

Some runfile entries are really 2d arrays, entered row by row. For example, in the 'IV' block there are per-column entries for responsivity:
<IV>
...
<Responsivity(W/DACfb)_C0> 1.92290e-16 1.92119e-16 0.00000 1.93100e-16 1.92978e-16 1.89769e-16 1.91119e-16 ...
<Responsivity(W/DACfb)_C1> 0.00000 1.82838e-16 0.00000 1.84197e-16 1.84822e-16 1.84447e-16 1.83693e-16 ...
<Responsivity(W/DACfb)_C2> 1.89962e-16 1.88649e-16 0.00000 1.85339e-16 1.84462e-16 1.82965e-16 1.84045e-16 ...
...
</IV>

These can be extracted at once if you pass a printf-style format string to the member function Item2d:
>>> a = rf.Item2d('IV', 'Responsivity(W/DACfb)_C%i', type='float')
>>> print len(a)
32
>>> print len(a[0])
33
>>> print a[2][1]
1.88649e-16

(i.e. a[2][1] is the responsivity for column 2, row 1.)

Some MCE data are spread across readout cards... e.g. adc_offset is stored in rc# adc_offset#. To accumulate all of these into a single array, use Item2dRC, passing the format in a way such that it can be evaluated first with the RC number and then with the column index... :
>>> a = rf.Item2dRC('HEADER', 'RB rc%i adc_offset%%i', type='int')
>>> len(a)
32
>>> len(a[0])
41

Python data and runfile modules

2009-04-05T22:10:18Z

174.6.181.2:

Recent changes:
* mce_script/trunk r408: Merged mce_runfile.py into mce_data.py. Use mce_data.py for everything.
* mce_script/trunk r407: SmallMCEFile is smarter about frame counts, and less verbose unless challenged.

The interfaces described here are based on mce_script/trunk r408.

= mce_data.py and mce_runfile.py =

== Notes ==

The module mce_data.py provide roughly the same functionality that [[ mas_data.pro ]] and [[ mas_runfile.pro and mas_runparam.pro | mas_runfile.pro ]] provide for IDL.

In the latest versions of the scripts, mce_runfile.py is no longer needed as the MCERunfile class has been merged into mce_data.py

== System setup ==

Make sure python can find mce_data.py (and possibly mce_runfile.py). $MAS_PYTHON should be in your PYTHONPATH environment variable.
mce@mce-ubc-2:~$ export PYTHONPATH=$PYTHONPATH:$MAS_PYTHON

This should be done automatically if you "source $MAS_ROOT/template/mas_env.bash" in your .bashrc".

= MCEFile / SmallMCEFile classes =

The MCEFile class is an alias for the SmallMCEFile class. These can be used interchangeably. The "Small" is just meant to suggest that you shouldn't try to load more frame data than your machine's RAM can handle. The module will complain and prevent you from loading more than 100 MB of data in one go.

== Basic usage ==

In python, import the 'mce_data' module and create a SmallMCEFile object:
>>> from mce_data import *
>>> fn = '/data/cryo/current_data/data002'
>>> f = SmallMCEFile(fn)

To suppress loading of the runfile, or override the runfile name, use one of these:
>>> f = SmallMCEFile( fn, runfile=False)
>>> f = SmallMCEFIle( fn, runfile='template_runfile.run')

Then load the data like this:
>>> d = f.Read()

The data member of d is a 2d array, [n_detectors, n_frames]:
>>> d.data.shape
(328, 100)

The list of rows and columns associated with the 328 detectors is available through col_list and row_list:
>>> d.col_list
[8, 9, 10, 11, 12, 13, 14, 15, 8, 9, 10, 11, 12, 13, 14, 15,
8, 9, 10, 11, 12, 13, 14, 15, 8, 9, 10, 11, 12, 13, 14, 15,
...
8, 9, 10, 11, 12, 13, 14, 15]

To display a pixel's values in all the data frames of a file:
>>> d.data[col+row*8]

Or to see pixel index i:
>>> d.data[i,:]

To display all pixels in frame f:
>>> d.data[:,f]

The data from the first frame header is available in the header member:
>>> d.header
{'status': 2052, 'data_rate': 47, 'userfield': 0, 'num_rows_reported': 41,
'runfile_id': 1231969044, 'row_len': 64, 'header_version': 6, 'rc_present':
[False, True, False, False], 'address0_ctr': 23142826, 'num_rows': 41,
'ramp_value': 0, 'frame_counter': 0, 'sync_box_num': 0, 'ramp_addr': 0}

== Raw frame data ==

To get raw frames (including header and checksum), pass "raw_frames=True" to the Read call:

>>> raw = f.Read(raw_frames=True)
>>> raw.shape
(100, 1100)

Then you can look at, e.g., the checksum:
>>> raw[:,1099]
>>> dd[:,1099]
array([918158926, 918144283, 918156148, 918156615, 918153934, 918159205,
...
918146379, 918145339, 918158232, 918146376], dtype=int32)

== Force row/column ordering ==

To reformat the d.data array so that its indices are (row, column, frame), pass "row_col=True" to Read:
>>> d = f.Read(row_col=True)
>>> d.data.shape
(33, 32, 100)

Then to see frame k:
>>> d.data[:,:,k]

== Extracting fields from mixed mode data ==

The MCE signal names, for mce_data purposes, are:
error - co-added error (reference mode 0)
fb - feedback in sq1 DAC units (reference mode 1)
fb_filt - filtered feedback (reference mode 2)
fj - flux jump counter

For mixed modes, the default is to extract the feedback or filtered feedback signal by default. To extract an alternate signal, pass "field=..." to the Read call:
>>> d = f.Read(field='fj')
>>> print d.data
array([[0,0,0,0,1,0,0,0,0,
...

It is possible to extract a set of fields simultaneously, using the "fields=" option in the Read call. This will override the "field=" option. The extracted fields appear in a dictionary in d.data.

>>> d = f.Read(fields=['fj', 'fb_filt'])
>>> print d.data['fj']
...
>>> print d.data['fb_filt']
...

You can extract all fields by passing "fields='all'":

>>> d = f.Read(fields='all')
>>> print d.data['fj']
...
>>> print d.data['fb_filt']
...

To force the bitfield extraction code to assume a particular data_mode, pass "data_mode=..." to Read.

>>> d = f.Read(data_mode=10, field='fb_filt')

= mce_runfile.py =

Note that in older versions of mce_script, the MCERunfile class is contained in the file mce_runfile.py. In that case, replace "from mce_data import ..." with "from mce_runfile import ..." in what follows.

== Load a runfile ==

>>> from mce_runfile import *
>>> runfile_name = '/data/cryo/current_data/1220531790_dat.run'
>>> rf = MCERunfile(runfile_name)

== Simple data ==

Recall that the structure of [[ Runfile format v2 | runfiles ]] is such that a line of data has an address defined by its 'block' and 'key' (where the key is the tag + specifiers...). The contents of rf include a dictionary of dictionaries of all the block / key pairs. For example:

>>> print rf.data['HEADER']['RB rc1 data_mode']
00000010
>>> print rf.data['SQUID']['SQ_tuning_dir']
1220510497

However, the member function "Item" allows us to repackage the runfile data by specifying a data type ('string', 'int', 'float') and whether or not we expect an array or a single value. For example:

>>> print rf.Item('HEADER', 'RB rc1 data_mode')
['00000010']
>>> print rf.Item('HEADER', 'RB rc1 data_mode', type='int')
[10]
>>> print rf.Item('HEADER', 'RB rc1 data_mode', type='int', array=False)
10
>>> print rf.Item('HEADER', 'RB rc1 data_mode', type='float')
[10.0]

== Two-dimensional data ==

Some runfile entries are really 2d arrays, entered row by row. For example, in the 'IV' block there are per-column entries for responsivity:
<IV>
...
<Responsivity(W/DACfb)_C0> 1.92290e-16 1.92119e-16 0.00000 1.93100e-16 1.92978e-16 1.89769e-16 1.91119e-16 ...
<Responsivity(W/DACfb)_C1> 0.00000 1.82838e-16 0.00000 1.84197e-16 1.84822e-16 1.84447e-16 1.83693e-16 ...
<Responsivity(W/DACfb)_C2> 1.89962e-16 1.88649e-16 0.00000 1.85339e-16 1.84462e-16 1.82965e-16 1.84045e-16 ...
...
</IV>

These can be extracted at once if you pass a printf-style format string to the member function Item2d:
>>> a = rf.Item2d('IV', 'Responsivity(W/DACfb)_C%i', type='float')
>>> print len(a)
32
>>> print len(a[0])
33
>>> print a[2][1]
1.88649e-16

(i.e. a[2][1] is the responsivity for column 2, row 1.)

Some MCE data are spread across readout cards... e.g. adc_offset is stored in rc# adc_offset#. To accumulate all of these into a single array, use Item2dRC, passing the format in a way such that it can be evaluated first with the RC number and then with the column index... :
>>> a = rf.Item2dRC('HEADER', 'RB rc%i adc_offset%%i', type='int')
>>> len(a)
32
>>> len(a[0])
41