Digital Signal Processing

A. Introduction

1. Purpose and objectives. This is an experiment on the use of computerized digital signal processing techniques for enhancing and extracting information from instruments used in analytical chemistry, spectroscopy, chromatography, and in many other areas of physical measurement. This is a self-paced interactive exercise based on the software package S.P.E.C.T.R.U.M. (Signal Processing for Experimental Chemistry Teaching and Research/ University of Maryland) .

2. Requirements. In order to run this tutorial, you need a copy of the SPECTRUM program (SPECTRUM 1.1) and the set of associated tutorial data files. SPECTRUM was designed to be used on a Macintosh II (or IIx or IIcx), but it will run on any Macintosh from the Plus on up. It is assumes that you are familiar with basic Macintosh operation (inserting disks, clicking, etc.)

3. Basic Operation of SPECTRUM. The SPECTRUM software package is a straightforward program based on the standard Macintosh interface. To run the program, double-click on the SPECTRUM icon. The program displays four windows, numbered 1 through 4, for displaying plots of signals. The topmost window is the active or selected window. All of the operations in SPECTRUM apply to the active window, so you have to select the desired signal first, then select an operation.

a. Selecting a window. There are three ways to select a window: either

Click on the desired window,
or
Pull down the Window menu and choose Select Signal 1, etc.,
or
Press apple-1, or -2 or -3 or -4.

b. Loading signals from the disk. To load a signal from the disk into a window:

1. Select the desired window
2. Select Open XY file... from the File menu or press apple-O
3. If necessary, open the folder containing the file by double clicking the folder name.
4. Double click on the desired file.

For example, load the file "Absorption spectrum", in the SPECTRUM Tutorial Data Files folder, into Window 1. This is an experi mental u.v.-visible absorption spectrum. It has wavelength from 350 to 700 nm on the X-axis and absorbance on the Y-axis. (Note that the axis labels are always shown in exponential notation). At the top of the window, just above the plot, a line of text displays the number of points in that signal and the minimum and maximum X and Y values.

c. Transferring signals between windows. SPECTRUM uses the standard Cut and Paste metaphor to transfer signals between windows. To move a signal from one window to another:

1. Select the window of the signal to be moved,
2. Select Copy from the Edit menu (or press apple-C),
3. Select the window to receive the signal,
4. Select Paste from the Edit menu (or press apple-V).
(This operation replaces the signal in the receiving window.)

The Copy command puts a copy of the active signal into the clipboard (the active signal remains unchanged, but the former clipboard contents are replaced). The Paste command transfers the clipboard signal to the active window (the clipboard signal remains unchanged, but the former window contents are replaced).

For example, transfer the Absorption spectrum in Window 1 into Window 2.

d. Scale expansion. SPECTRUM automatically adjusts the X- and Y-axis scale expansion for each signal to fill the entire plotting area. However, you can override this with the manual scale expansion controls in the Window menu. Expand on Y-axis (apple-]) will expand the plotted signal in the Y direction by halving the range of the Y-axis. Contract on Y-axis (apple-[) will shorten the plot in the Y direction by doubling the range of the Y-axis. Expand on X-axis (apple->) will expand the plotted signal in the X direction by halving the range of the X-axis. Contract on X-axis (apple-<) will shorten the plot in the X direction by doubling the range of the X-axis. Autoscale (apple-') rescales both axes. These scale expansion operations do not really change the signal, only the way it is plotted.

For example, use the Expand on Y-axis (apple-]) command to expand the Absorption spectrum in Window 1 on the Y-axis so you can take a closer look at the region around 550 nm, where there are some weak bands. Each time you press apple-] the plot expands by a factor or two.

e. The cursor. If you press apple- ~ (or select Vertical line cursor from the Window menu), the normal arrow cursor will change into a vertical line cursor in the active window, and the top line of the window will show the current X-axis position of the cursor and the Y-value of the signal at that point. If the cursor is between points in the signal, the program calculates the Y-axis value by linear interpolation from the points before and after. Click the mouse or press any key to restore the normal arrow cursor.

For example, use the cursor to measure the wavelengths of the two weak bands around 500 - 550 nm in the Absorption spectrum in Window 1.

f. Window management. The four windows can be displayed in several ways, using commands in the Window menu. Stack windows (apple-6) displays signals in large, partly overlapping windows, but you can see only one signal at a time. Tile windows (apple-7) displays all four signals simultaneously, but in smaller windows. To expand any window to full-screen, click on the zoom box (upper right-hand corner) of the window or select Zoom windows.

You can also change the plotting style of the signal in the active window by selecting Plain line, Dots, or Connected dots from the Line menu. The chosen plotting style remains associated with that window until changed.

g. Operations involving the clipboard. Many of the signal processing operations in SPECTRUM operate only on the selected signal (e.g. adding a constant to a signal), but some operate between two signals (e.g. adding two signals together). These two-signal operations take place between the signal in the active window and the signal cur rently in the clipboard. (All such commands have "clipboard" in their names). To perform these operations, you select one signal, copy it (apple-C) to the clipboard, select the other signal, and then select the desired command. (To see what is currently in the clipboard, select Swap with clipboard from the Window menu, which exchanges the active signal with the clipboard signal, then select Swap with clipboard again to return it).

For example, select Window 2 (apple-2) , which should contain a copy of the Absorption spectrum loaded earlier, select Multiply by constant from the Math menu (or press apple-M), type in the number 10, and press return or click on the OK button. This multiplies the active signal by 10. Note that it does not look any different, because of the autoscaling, but notice that the Y-axis scale has changed. Now copy this signal to the clipboard (apple-C), select Window 1 (apple-1) , and select Superimpose clipboard from the Window menu (or press (apple-U). This temporarily superimposes the clipboard signal (which contains the absorption spectrum multiplied by 10) into the unmodified spectrum in Window 1. Note that the superimposed signal is plotted on the scale of Window 1 (and in green on a color monitor). This operation does not modify the signal in Window 1.

h. Undoing operations. Any time you perform an operation that changes the signal, you can undo that change and recover the previous signal by using the Undo command in the Edit menu. Only the last operation can be undone. You can also undo Undo.

i. Saving signals. Signals are stored on the disk as a list of X and Y values in plain ASCII text. To save a signal to the disk:

1. Select the desired signal.
2. Select Save as XY file... from the File menu, or press apple-S.
3. Type in the desired file name and select the target disk and folder in the usual way.
4. Press the return key or click the Save button.

Data are saved as two columns of decimal values, in the format X1 <TAB> Y1 <CR> X2 <TAB> Y2 <CR>, etc., where X1 is the first X value, <TAB> is a TAB character, Y1 is the first Y value, and <CR> is a carriage return. Data saved in this way can be directly read by text editors, word processors, spreadsheets, plotting programs, statistics packages, and other applications that read plain text. (You can also use the Save as Y file... command from the File menu to save the data in as a single column of Y values, without X values, in the format Y1 <CR> Y2 <CR>, etc. This takes up less disk space, but you loose the X values. Signals saved in this way should be opened with the Open Y-file... command. The X values are then simply the point numbers).

j. Printing. SPECTRUM can print graphic signal plots to the LaserWriter printer. To print a signal:

1. Select the desired signal.
2. Select Print... from the File menu, or press apple-P.
3. Press the return key or click the Print button.

K. Importing data from the outside world. SPECTRUM can read plain ASCII text files generated by many other applications. The Open XY file... command (apple-O) expects a text file in the format X1 <TAB or spaces> Y1 <CR> X2 <TAB or spaces>Y2 <CR> etc, where <CR> represents a carriage return. The columns can be separated by tabs or by any number of spaces. (SPECTRUM assumes that the X interval spacing is constant and sets the X Increment to X2-X1; however this can always be changed later with the Set X-axis function). Alphabetic labels and blank lines are automatically skipped. Data files may be prepared in a text editor, spreadsheet, telecommunications (terminal) programs, plotting or data acquisition program, etc, and saved in ASCII (text only) format.

l. Getting help. There is an on-line help file available in the [[apple]] menu that explains the operation of each menu selection in the program.

m. Quitting. To quit SPECTRUM and return to the Finder, select Quit from the File menu or press apple-Q.

B. Signal arithmetic.

The most elementary signal processing functions are those that involve point-by-point addition, subtraction, multiplication, or division of two signals. Although they may be mathematically trivial, these functions can nevertheless be very useful.

1. Subtraction of a known background. Load the file "Shale extract" into Window 1 and "Shale blank" into Window 2. "Shale extract" is the absorption spectrum of an extract of a sample of oil shale, a kind of rock that is is a source of petroleum. This spectrum exhibits two absorption bands, at about 515 nm and 550 nm, that are due to a class of petroleum-derived compounds called porphyrins. (Porphyrins are used as geomarkers in oil exploration). These bands are superimposed on a background absorption caused by the extracting solvents and by non-porphyrin compounds extracted from the shale. "Shale blank" is the spectrum of an extract of a non-petroleum-bearing shale, showing only the background absorption. To obtain the spectrum of the shale extract without the background, simply subtract Window 2 from Window 1: select Window 2 (click on its window or press apple-2), copy it to the clipboard (apple-C), select Window 1, and then select Subtract Clipboard Signal from the Math menu (or press apple-minus). The spectrum of the compound without the solvent background should be displayed in Window 1. It is now easier to measure precisely the absorbances and wavelengths of the two bands. Note the change in the Y-axis scale; the program always scale-expands signals along the Y-axis to fill the plotting area. Also, you will notice a sharp peak at about 655 nm in both spectra. This is almost certainly an instrumental artifact, caused by a "glitch" in the electronics or optics; it is very unlikely that any real absorbing compound would have an absorption band this narrow. Unfortunately the glitch is only partly removed by the subtraction.

For a more dramatic example, load the files "Sample 2" and "Blank 2". In this case the concentration of the analyte in Sample 2 is so low that there is no visible difference between the sample and blank signals. You can superimpose these two signals in one window by copying one to the clipboard, switching to the other signal, and selecting Superimpose clipboard from the Window menu (or press apple-U). The two signals are not quite identical, but is is hard to characterize the difference. However, subtracting Blank 2 from Sample 2 shows that there is a weak spectral band present in the sample that is not present in the blank and makes it possible to estimate the position, amplitude and width of that band. Clearly, there was more to these signals than was obvious on first inspection. Note also the evidence of background noise in the difference spectrum.

2. Comparison of similar signals. Often one needs to know whether two signals have the same shape, for example in comparing the spectrum of an unknown to a stored reference spectrum. Most likely the concentrations of the unknown and reference, and therefore the amplitudes of the spectra, will be different. Therefore a direct subtraction of the two spectra will not be useful. One possibility is to normalize the two spectra and then subtract them. Another possibility is simply to divide the two signals; if they have the same shape, the ratio will be a constant (within the noise, that is). Load the two files "Spectrum 1" and "Spectrum 2" into two windows. Spectrum 2 is much weaker (notice the Y-axis scale) and also slightly noisy (not surprising). But do they have the same shape? To divide the two signals, copy one to the clipboard (apple-C), select the other signal, and select Divide by clipboard from the Math menu (or press apple-/). The resulting ratio is relatively constant from 300 to 440 nm, with a value of 10 +/- 0.2. This means that the shape of these two signals is the same, within about +/- 2 %, over this wavelength range, and that Spectrum 1 is about 10 times more intense than spectrum 2. Above 440 nm the ratio is very noisy because Spectrum 2 has low intensity and a poor signal-to-noise ratio in this region.

3. Signal averaging. One of the classic ways of improving signal-to-noise ratio of experimental signals is to measure the signal many times in quick succession, add up all the measured signals point-by-point, and then divide by the number of repeat measurements. This is called co-adding or ensemble averaging. It is commonly applied in FT-NMR spectroscopy. It works because random noise is not repeatable from measurement to measurement and will gradually cancel out as more and more signals are added. Real signal, on the other hand, will continue to add up. This makes the assumption, of course, that the signal is repeatable and that the noise is random. (Ensemble averaging will not remove instrumental artifacts that are repeatable from measurement to measurement.) On your disk there is a subdirectory (folder) called Ensemble average, containing ten repeat measurements of a signal named "1" through "10". Load some of these signals for inspection. Obviously there is not much that is reproducible about these signals; you might be tempted to say that they are just noise. Now add them up in the following way: load signal "1" into any window and copy it to the clipboard (apple-C). Then load signal "2" into the same window, add the clipboard signal to it (select Add clipboard signal from the Math menu or press apple-plus), and copy the resulting sum to the clipboard (apple-C). Now repeat the last three steps for all the signals "3" through "10": that is, load signal "3", press apple-plus, press apple-C, load signal "4", etc. Gradually a pattern will emerge. After the last signal has been added, you should be able to see a definite signal: three peaks sticking out perceptibly above the noise. (Strictly speaking, you should divide by 10, the number of signals added, to compute the average, but this won't change the picture). So there were signals in those individual measurements after all, but they were below the statistical detection limit and therefore could not be seen. Theoretically, co-adding n signals improves the signal-to-noise ratio by the square root of n, or in this case by a little over 3. The larger n the greater the improvement. In practice, ensemble averaging is most efficiently accomplished by co-adding dozens or hundreds of repetitive signals in real time as they are measured and not by storing each measurement separately as has been done here. The entire process is readily automated.

C. Smoothing

1. Signals and noise. In many experiments in physical science, the true signal amplitudes (y-axis values) change rather smoothly as a function of the x-axis values, whereas many kinds of noise are seen as rapid, random changes in amplitude from point to point. In this situation it is common practice to attempt to reduce the noise by a process called smoothing. In smoothing, the data points of a signal are modified so that individual points that are higher than the immediately adjacent points (presumably because of noise) are reduced and points that are lower than the adjacent points are increased. This naturally leads to a smoother signal. As long as the true underlying signal is actually smooth, then the true signal will not be much effected by smoothing, but the noise will be reduced. As an example, load the file "Noisy signal" into Window 1 and transfer a copy to Window 2. This signal consists of a broad band with a substantial amount of random noise. Select Rectangular smooth... from the Transformation menu, or press apple-F. Then type the number 21 and press return or click OK. The signal is now much smoother and the noise has been visibly reduced. Select Undo from the Edit menu or press apple-Z to restore the original signal. The rectangular smooth, also called an unweighted sliding-average smooth, is the simplest smoothing algorithm; it simply replaces each point in the signal with the average of m adjacent points, where m is a positive odd integer called the smooth width. For a 3-point smooth (m=3), the jth point in the smoothed signal Sj is

S_j = (Y_j-1 + Y_j + Y_j+1)/3 for j=2 to n-1.

and similarly for other smooth widths. The (m-1)/2 points on either end (j1 and jn in this example) are replaced by the first and last values of the above series, respectively, because there are not enough data points to compute a full smooth for those points. As a result, important parts of the signal should not be positioned near the ends.

Try smoothing the signal with several other smooth widths, using Undo (apple-Z) to recover the signal each time. The larger the smooth width, the greater the noise reduction, but if the smooth width is too great, the signal will be distorted.

2. Optimization. Is there an optimum smooth width for any given type of signal? If signal-to-noise ratio is taken as the criterion for optimization, then yes, there is an optimum. The best way to demonstrate this quantitatively is to investigate the effect of smoothing on the signal and noise separately. (This is allowed because smoothing is a linear operator; therefore the smoothed sum of two signal components is equal to the sum of the two separately smoothed components.) This can be done with the two files "Gaussian 256", a computer-generated Gaussian band that represents the "signal", and "Noise 256", a sample of random noise that represents the "noise". Here is the procedure: Load "Gaussian 256" into Window 1, copy it to the clipboard (apple-C) and paste it (apple-V) into each of the other three windows. Note that the unsmoothed signal has a peak height of exactly 1.0. Smooth (apple-F) each window with a different smooth width according to this scheme:

Window 1: 9 points
Window 2: 13 points
Window 3: 25 points
Window 4: 51 points

Record the maximum Y value (peak height) reported at the top of each window. Compare the four smoothed peaks. Note that the peaks becomes shorter, broader, and somewhat distorted as the smooth width increases.

Now for the noise: Load "Noise 256" into Window 1, copy it to the clipboard (apple-C) and paste it (apple-V) into each of the other three windows as before. Measure the standard deviation of the unsmoothed noise by using the Standard deviation function in the Math menu (apple-; ). It should be about 1.0. Smooth (apple-F) each window with the same set of smooth widths as before. Record the standard deviation of each signal (select each window in turn and press apple-; ). You should end up with a table of peak heights and standard deviations for smooth widths of 1 (i.e. no smoothing), 9, 13, 25, and 51 points. Now compute the ratio of the peak heights to the standard deviations (i.e., the signal-to-noise ratios) for each smooth width. (You can use the Calculator available in the [[apple]] menu). As you can see, the signal-to-noise ratio starts out at 1,0 without smoothing, increases as smoothing is increased, reaches a maximum, then goes down. What is the optimum smooth width in this case?. In general, it is found that the optimum smooth width for a Gaussian band is about equal to the half-width of the band and that the relative signal-to-noise ratio improvement is about the square root of that width. Is that roughly what you observed?

In reality, of course, the signal and noise would appear added together in one signal, but they were treated separately above because if they they were combined, the presence of the noise would spoil the accuracy of peak height measurement and the presence of the signal would spoil the accuracy of the standard deviation measurement.

There are two particular things to note about these results: first, the standard deviation of the smoothed noise is seen to be inversely proportional to the square root of the smooth width: this is just what is expected from statistics. Second, the optimum signal-to-noise ratio occurs at a smooth width that causes rather significant decrease in the amplitude of the signal (generally about 10% to 20%). In other words, you have to be willing to tolerate a certain amount signal distortion in order to obtain optimum signal-to-noise ratio. On the other hand, if you were trying to measure the absolute peak height of the signal, then you probably wouldn't want to tolerate this much attenuation. In that case the optimum smooth width depends on the amount of noise: the nosier the signal, the more smoothing is desirable. In general, a smooth width of one-third to one-fourth of the peak width is optimum when absolute peak heights must be measured.

So how well does this optimum work? Load the two signals again. Copy the (unsmoothed) noise to the clipboard and add it to the Gaussian signal using the Add clipboard signal function in the Math menu (apple-plus). Since the signal-to-noise ratio without smoothing is is only 1, you should not be able to detect the signal in all that noise. (The minimum signal-to-noise ratio for reliable detection is about 3). Now smooth the combined signal+noise with a width of 25. Can you see the Gaussian "signal" now?

In general, the optimum choice of smooth width depends upon the width and shape of the signal, but the results just obtained are typical. Smoothing is most effective when the true signal is inherently very smooth and is digitized at closely-spaced intervals. In that case large smooth widths can be used without serious signal distortion.

3. Special cases. In most cases the benefits of smoothing are largely cosmetic, that is, it improves the appearance of signals but it does not yield a very dramatic increase in the ability of the observer to extract quantitative information from the signal, e.g. to estimate the amplitudes and peak positions and widths of signal peaks. This is because the "eye-brain" system is pretty good at estimating averages in noisy signals. In some situations, however, the benefits can be more substantial. One example, encountered in the previous section, is when the signal-to-noise is below the detection limit and the improvement by smoothing is just enough to make the signal detectable. For an even more dramatic example, load the file "Smooth Me". This signal contains a great deal of "high frequency" noise; that is, the noise is not totally random but has a strong tendency to oscillate between alternately positive and negative values. Noise of this type sometimes occurs in experimental signals as a result of capacitive coupling of radio-frequency (rf) signals from nearby equipment. Although the noise makes it impossible to extract any information about the underlying signal by visual inspection, it is very easily attenuated by smoothing. This is because the alternative positive and negative fl uctuations tend to cancel out nearly completely when adjacent points are averaged in the smoothing operation. Try smoothing this signal with an 11 point rectangular smooth. Note the dramatic improvement. Try another pass of the smooth, that is, smooth the smoothed signal again. Note the further improvement. The underlying signal is now fairly clean.

Another situation where smoothing is particularly effective is when the signal is limited by amplitude quantization noise. Load the file "QuantNoiseTest". This signal exhibits the stratification of Y-values that is commonly caused by digitization of a small analog signal near the lower limit of resolution of the analog-to-digital converter. The effect is more clearly seen by setting the line style to dots only (Select Dots from the Line menu). Now smooth the signal with a 7-point triangular smooth and note the improved appearance of the signal.

4. Other smoothing algorithms. The Triangular smooth... is like the rectangular smooth, above, except that it implements a weighted smoothing function. The smooth width m is the half-width of the triangle. For a 3-point smooth (m=3), the jth point in the smoothed signal Sj is

S_j = (Y_j-2 + 2Y_j-1 + 3Y_j + 2Y_j+1+ Y_j+2)/9 for j=3 to n-2.

and similarly for other smooth widths. This is equivalent to two passes of an m-point rectangular smooth.

You can always determine the underlying smoothing function of any smooth algorithm by applying the algorithm to a "delta function", that is, a signal that consists of only a single point above a zero baseline. For example, load the file "Delta 256" and smooth it with a 21-point rectangular smooth. You get a 21-point wide rectangle. Now Undo this and smooth it with a 21-point triangular smooth. You get a triangle with a 21-point half-width.

D. Differentiation

The differentiation of functions is a topic that is introduced in all elementary Calculus courses. The numerical differentiation of digitized signals is an application of this concept that has many uses in analytical signal processing. The first derivative of a signal is the rate of change of y with x, that is, dy/dx, which is interpreted as the slope of the signal. The second derivative d²y/dx² is a measure of the curvature of the signal, that is, the rate of change of the slope of the signal.

1. Slope measurement. As an example of a practical application of differentiation, load the file "Slope" into Window 1 and transfer a copy to Window 2. This signal is typical of the type of signal recorded in amperometric titrations and some kinds of thermal analysis and kinetic experiments: a series of straight line segments of different slope. The objective is to determine how many segments there are, where the breaks between then fall, and the slopes of each segment. This is difficult to do from the raw data, because the slope differences are small and the resolution of the screen display is limiting. The task is much simpler if the first derivative (slope) of the signal is calculated. Select First derivative from the Transformation menu (or press apple-D). Each segment is now clearly seen a separate step whose height (Y-axis value) is the slope. The Y-axis now takes on the units of dy/dx. Note that the steps are not completely flat, indicating that the line segments in the original signal were not perfectly straight. This is most likely due to random noise in the original signal. Although this noise was not particularly evident in the original signal, it has been accentuated in the derivative. This is normal behavior.

2. Application to titrimetry. A classic use of differentiation in chemical analysis is in the location of endpoints in potentiometric titration. Load the file "Titration Curve" into Window 1 and transfer to Window 2 This is a pH titration curve of a moderately weak acid with a strong base, with volume in mL on the X-axis and pH on the Y-axis. The endpoint is the point of greatest slope; this is also the inflection point, where the curvature of the signal is zero. Take the first derivative of this signal (apple-D). The maximum corresponds to the end point. (Note that there is also a smaller maximum very early in the titration, at about 1.0 mL, but this is clearly not the real endpoint.) Take the derivative again. The derivative of the derivative is the second derivative. The zero crossing of the second derivative corresponds to the endpoint. (It is generally easier to locate the zero crossing, particularly if the endpoint falls between two measured points, as it does here.) Note that there are actually several zero-crossings caused by noise, but it is clearly the main zero crossing at about 30 mL that we want. Use the cursor (apple-~) to measure the volume at this zero-crossing as precisely as possible. It should be about 31.77 mL. To make the measurement more precise, use the Extract subset of points... function in the Window menu (apple-K) to extract the points between 25 and 40 mL. (Fill in 25 for the first x-value and 40 for the last x-value in the dialog box). Now that portion of the curve will be expanded to fit the window and you will be able to make a very precise estimate of the end-point volume.
In automatic titrators, the second derivative is ordinarily measured in real time and the zero crossing is used to indicate the end point and trigger an electronic switch that turns off the titrant delivery pump.

3. Applications in spectroscopy. In spectroscopy, particularly in infra-red, u.v.-visible absorption, fluorescence, and reflectance spectrophotometry, differentiation of spectra is a widely used technique, referred to as derivative spectroscopy. Derivative methods have been used in analytical spectroscopy for three main purposes: (a) spectral discrimination, as a qualitative fingerprinting technique to accentuate small structural differences between nearly identical spectra; (b) spectral resolution enhancement, as a technique for increasing the apparent resolution of overlapping spectral bands in order to more easily determine the number of bands and their wavelengths; (c) quantitative analysis, as a technique for the correction for irrelevant background absorption and as a way to facilitate multicomponent analysis. Spectra commonly consist of one or more spectral bands, usually of approximately Gaussian or Lorentzian shape, often superimposed on a background of variable shape. Load the file "Two Bands" into Window 1 and transfer a copy to Window 2. This signal consists of two simulated (computer generated) bands: the one on the left is Gaussian and the one of the right is Lorentzian. They have the same peak height (maximum amplitude) and peak width. Gaussian bands are commonly encountered in u.v.-visible absorption and fluorescence spectrophotometry, and Lorentzian bands are characteristic of infra-red and nuclear magnetic resonance (NMR) spectroscopy. Now take the derivative of this signal (apple-D). The zero crossings correspond to the maxima of each band. Take the derivative again, yielding the second derivative. Both bands now have a negative central peak corresponding to the band maximum and two positive side lobes or satellite bands. Note the characteristic and distinct shapes of the two bands. In particular, note that the width of the negative central peak in the second derivative signals is narrower that the central peak of the original signal. Now take the derivative two more times, yielding the third and then the fourth derivative. (There are no simple physical interpretations of the third and higher-order derivatives; nevertheless, derivatives up to order four, and occasionally up to order six, have found use in analytical chemistry and spectroscopy.) Note that the central peaks are now positive and that there are now both negative and positive side lobes. The amplitude of the fourth derivative of the Lorentzian band is now substantially greater then that of the Gaussian band, and the magnitude of its side lobes is relatively less, due to their inherently different shapes. As was the case for the second derivative, the widths of the central peaks are narrower that those of the original signals, but even more so. This effect is much more pronounced for the Lorentzian band that for the Gaussian band. This behavior is the basis of the application of second and fourth derivatives as a simple resolution enhancement technique.

4. Background correction. Load the file "Two Gaussians" into Window 3 and transfer a copy to Window 4. This is computer-generated signal consisting of two Gaussian bands of equal height but different width: the one one the right has a width exactly twice the one on the left. Now take the first derivative of this signal. Note that the derivative of the narrower band now has an amplitude exactly twice that of the derivative of the wider band. In general, the amplitude of the first derivative of a band is inversely proportional to its width. Take the derivative again, yielding the second derivative. The narrower band now has an amplitude exactly four times that of the derivative of the wider band. Keep going to the third and fourth derivatives; the narrower band keeps getting stronger and stronger relative to the wider band. (In general, it can be shown that the amplitude of the n^th derivative of a band is inversely proportional to the n^th power of its width.) This is a very useful behavior, and it is the basis for the application of differentiation as a method of correction for background signals in quantitative spectrophotometric analysis. Very often in the practical applications of spectrophotometry to the analysis of complex samples, the spectral bands of the analyte (i.e. the compound to be measured) are superimposed on a broad, gradually curved background. For example, load the file "Sample 1". This is an absorption spectrum in which the analyte bands at 280 and 320 nm are superimposed on a broad background. If you take the first derivative of this spectrum, you can see that the background is significantly reduced relative to the analyte bands. Take the derivative again and you can see that the background is almost completely eliminated. Note that it is not necessary to have an analyte-free background spectrum (blank). (If you did, you could just subtract it out). It is only necessary that the background be broader (that is, have lower curvature) than the analyte spectral peaks.

In conventional condensed-phase absorption spectrophotometry, with quantitation based on the absorption at a single wavelength, measurement precision is often degraded by sample-to-sample baseline shifts due to non-specific broad-band interfering absorption, non-reproducible cuvette positioning, dirt or fingerprints on the cuvette walls, imperfect cuvette transmission matching, and solution turbidity. The use of a flow-cell or sipper cell can eliminate the variability associated with cuvette positioning and transmission, but the influence of non-specific interfering absorption and solution turbidity remains. Baseline shifts from these sources are usually either wavelength-independent (light blockage caused by bubbles or large suspended particles) or exhibit a weak wavelength dependence (small-particle turbidity). Therefore it can be expected that differentiation will in general help to discriminate relevant absorption from these sources of baseline shift.

An obvious benefit of the suppression of broad background by differentiation is that variations in the background amplitude from sample to sample are also reduced. This can result in improved precision or measurement in many instances, especially when the analyte signal is small compared to the background and if there is a lot of uncontrolled variability in the background. For example, load the files b1, b2, b3, and b4 into Windows 1,2,3, and 4, respectively. In each of these signals, the band at 320 actually has the same amplitude and position, but it is superimposed on a broad background that varies from signal to signal. The bands certainly do not look like they have the same intensity; even the peak positions seem to be different. Now take the Second derivative of each band. The background, having lower curvature than the peak, is reduced by second differentiation. The relative amplitudes and the peak positions (indicated by the minimum in the second derivative) of the peaks in these four signals are now much more nearly equal.

5. Quantitative measurement. Differentiation is a linear operation. This means that the amplitude of a derivative is directly proportional to the amplitude of the original signal. It also means that the derivative of the sum of two signals is the sum of the derivative of the two separate signals. This goes for all derivative orders. The significance of these facts is that it is possible to base a quantitative analysis on derivative signals just as surely as a conventional non-derivative measurement. Most quantitative analytical methods in spectroscopy or chromatography are relative methods based on calibration with standards. To perform an analysis based on derivative signals, one simply measures the derivatives of all the standards, samples, and blanks using the same calculational algorithm, that is the same derivative order and smoothing. The derivative signal amplitudes of the standards are plotted against their concentration to form the calibration curve, and then the derivative signal amplitudes of the samples are measured and "read off" the calibration curve to convert their readings into concentrations. It is not necessary to know how the process of differentiation and smoothing changes the signals amplitudes, because the samples and standards are all changed in the same way. As an example, load the file "Calibration series". This signal consists of a series of computer-generated peaks in a 1:2:3:4:5 ratio of peak heights, representing a series of standards of progressively higher concentration. Now take the first derivative of this signal, and you will see that the derivative signals also form a linear progression of signal amplitudes in a 1:2:3:4:5 ratio. You can try any derivative order, and any degree of smoothing (as long as the peaks are not broadened so much that they overlap), and you will observe the same linear progression of signal amplitudes.

6. Effect of Differentiation on noise and signal-to-noise ratio. When the signal is differentiated, the noise in it is also differentiated. Random noise is not "smooth" but rather changes rapidly from point to point. Therefore the noise has a relatively larger derivative (slope) than a smooth signal, with the result that the signal-to-noise ratio is degraded by differentiation. For example, load the file "Peak" into Window 1 and transfer it to the other three Windows. Take the first derivative of the signal (apple-D) in Window 2. Note that the signal-to-noise ratio has been decreased considerably. Press apple-D again to get the second derivative. The signal-to-noise ratio decreases even more; in fact, now it's hard to see the signal. This may look bad, but fortunately the situation can be improved greatly by smoothing. Smooth (apple-F) this signal with a 21-point rectangular smooth. The signal-to-noise ratio improves considerably. Smooth it again. Now the signal-to-noise ratio is almost as good as the original signal. This behavior is quite general; differentiation degrades signal-to-noise ratio dramatically but smoothing reverses the damage. In fact, smoothing is essential when using derivative signal processing. Whereas smoothing ordinary non-derivative signals yields only a modest signal-to-noise ratio improve ment, smoothing derivative signals yields a much more dramatic improvement, often a factor of 10 to 100 or even more, compared to the unsmoothed derivative signal. As was found to be the case for ordinary non-derivative peak-type signals, smoothing also causes distortion, i.e. reduction in the amplitude and an increase in width, but the effect is even more noticeable for derivative signals. The optimum smooth width for maximum signal-to-noise ratio of an isolated band is about the same as for non-derivative signals, that is, approximately equal to the half-width of the original peak. For a peak on a broad background, smaller smooth widths (about one-fourth to one-half the peak width) are usually used to avoid excessive peak attenuation relative to the background signal, which is not much attenuated by smoothing.

Some aspects of smoothing derivative signals are not necessarily obvious:

not

apple

-D

apple

-D

apple

-F

one

two

(b) In general, the optimum number of passes of a rectangular smooth is one more than the derivative order, i.e. 2 for the first derivative, 3 for the second, etc. (Recall that a triangular smooth is equivalent to two passes of a rectangular smooth.)

(c). The derivative and smooth operations can be done in any order; it makes no difference whether you smooth first and then differentiate or differentiate first and then smooth, or mix it up. The signal itself depends only on the total number of derivative and smooth steps and not on the route taken. The order of operations has an effect only on the edge effects, artifacts generated by the differentiation and smoothing algorithms at the very beginning and end of the signal record. For example, select the copy of the original peak signal in Window 4, smooth it (apple-F) with two passes of a 21-point rectangular smooth, and then differentiate it twice (apple-D, apple-D). Compare the result to the signal in Window 2, which was smoothed after differentiation. The central portion of the signal is identical, as can be proved by superimposing the two signals.

Because of the close association of differentiation and smoothing operation in dealing with noisy signals, SPECTRUM has a Smoothed second derivative... command in the Transformation menu that implements a second derivative followed by three passes of rectangular smooth. It is equivalent to the individual steps, but it is faster and can be undone.

7. Trace analysis. One of the widest uses of the derivative signal processing technique in practical analytical work is in the measurement of small amounts of substances in the presence of large amounts of potentially interfering materials. In such applications it is common that the analytical signals are weak, noisy, and superimposed on large background signals. The utility of derivative signal processing in such cases will depend upon the balance of two opposing effects: the within-spectrum signal-to-noise ratio, which is decreased by differentiation, and background variability, i.e., between-spectra shifts in the baseline and broadband interfering signals, which are reduced by differentiation. The overall effect of differentiation on analytical precision depends upon the balance of these two effects. If background variability is not significant, then the precision of a quantitative analysis based on derivative measurement will be the same as or poorer than the zeroth-order measurement. If background variability is a significant contribution to the overall analytical variability, then the derivative method will be more precise. Smoothing must also be optimized: too little and the results are imprecise due to within-spectrum random noise; too much, and excessive peak attenuation will increase the influence of background and its spectrum-to-spectrum variability.

As an example of the benefits of derivative signal processing, load the signals "t/1ppm", "t/2ppm", "t/4ppm", and "t/8ppm" into Windows 1, 2, 3, and 4, respectively. These signals are a calibration series for the quantitative determination of a component that gives a weak band near the center of these signals, superimposed on a very strong background. The concentration of the component in parts per million (ppm) is given in each file name. Use what you have learned to extract from these signals a quantitative measure that can be related to concentration and used as a basis for quantitative measurement. Is there really a detectable signal in "t/1ppm"? (Answer: Yes. Use 4th derivative with a total of 4 or 5 passes of an 11-point rectangular smooth).

E. Resolution enhancement

Load the file "Band head" into Window 1 and transfer a copy to Window 2 This spectrum consists of several poorly-resolved (that is, partly overlapping) bands. It actually has five bands, located at 100, 270, 340, 380, and 400, all with equal intensity, but it is obvious that the extensive overlap of the last three bands makes the accurate measurement of their intensities and positions impossible. Things would be easier if the bands were more completely resolved, that is, if the bands were narrower. Here use can be made of the peak-narrowing behavior of even-order derivatives. to artificially improve the apparent resolution. Take the second derivative of this signal by selecting Second derivative from the Transformation menu. The negative peaks now mark the positions of the band maxima. Multiply (apple-M) the signal by -1 so that the central peaks will be positive. Use the cursor (apple-`) to measure the positions and relative intensities of the last three peaks. As you can see, the second derivative has been effective in separating the last two peaks at 380 and 400), which were completely fused in the original spectrum. Moreover, the position and relative intensity of the third peak (at 340) is now nearly correct. However, the relative intensities of the last two peaks is still incorrect (they should all be equal). Would the fourth derivative be better? Take the fourth derivative by taking the second derivative of this second derivative signal, and then multiply it by -1 again. Now the noise has gotten out of hand, so smooth it with one or two passes of a small (3 or 5 point) triangular smooth. Clearly, the apparent resolution of the overlapping peaks has been further improved, but the generation of the side lobes complicates the resulting derivative spectrum and generates artifacts (small false peaks) that are not present in the original spectrum.

A slight modification of the above technique can give greatly improved results by reducing the troublesome side lobes. Try the following experiment. Transfer a copy of the original signal to Windows 3 and 4. Take the second derivative of Window 4 and multiply it by -700. Now copy it to the clipboard, switch to Window 3, and Superimpose the two signals (apple-U). Looking at the left-hand half of the first band, you can see that there is a region where the side lobe of the second derivative (the green line) is approximately equal in magnitude but opposite in sign to the original band (the red line). So, if these two signals were to be added together, a large portion of the side lobe would approximately cancel out, and yet the sum of the original band and the much narrower central peak of the second derivative would still be narrower that the original band. Select Add clipboard signal from the Math menu or press apple-plus. Compare the resulting signal to the original signal (which should still be in Window 1) and to the second derivative in Window 4. You can see that the side lobes have been greatly reduced across the entire spectrum, compared to the second derivative, and that there is still a substantial resolution enhancement compared to the original signal. More important, the relative intensities (i.e. peak heights) of the five bands are now much closer to their correct values, due to the reduced influence of the side lobes.
The trick in using this technique is to know how much to multiply the second derivative by before adding it to the original signal. This depends on the width and shape of the bands and has to be established by trial and error. As a rule, the number will be between -10 and -1000. If you choose too small a number, the resolution enhancement will be miminal. If you choose too large a number, the baseline will exhibit excessive negative "undershoot" caused by the derivative side lobes. Because this factor depends on peak width, it will not be possible to apply optimum enhancement simultaneously to several bands of unequal width. A further point is that it may be desirable to smooth the resolution-enhanced signal slightly to reduce the extra noise introduced by the derivative. Small smooth widths (no more that one-fourth the peak width) should be used to avoid significant peak broadening, which would counteract the resolution enhancement. You must always expect that resolution enhancement will be accompanied by at least some degradation in signal-to-noise ratio.

The resolution enhancement technique just described is incorporated into the Resolution enhancement function in the Transformation menu. This function combines the differentiation, multiplication, addition, and smoothing steps. You are prompted to enter the multiplication factor and the smooth width. As usual, you can Undo the result and try other numbers until the best results are obtained.

F. Integration and peak area measurement

The symbolic integration of functions and the calculation of definite integrals are topics that are introduced in elementary Calculus courses. The numerical integration of digitized signals finds application in analytical signal processing mainly as a method for measuring the areas under the curves of peak-type signals. As an example, load the file "NMR" into Window 1 and transfer a copy to Window 2. In nuclear magnetic resonance (NMR) spectroscopy, one often wants to know the relative area under a peak or group of peaks, because it is related to the number of nuclei responsible for a given resonance. In this spectrum, what is the relative area under the triplet at 200, relative to the doublet centered at 650? The standard way this is done is to compute the integral of the spectrum. Select Window 2, then Integrate from the Transformation menu (or press apple-I). The height of each step in the integral is proportional to the area under the corresponding peak. You can use the cursor (apple- ~) to measure the step heights. Here you can see that the area of the triplet is one-half that of the doublet.

Another way to measure peak areas is to use the Peak Area... function in the Math Menu on the original signal in Window 1.. A dialog box asks for the X-axis range over to which to perform the area measurement. You can fill in either the point number or (more likely) the X-axis values for the desired range, then press return or click on OK. The area (in XY units) is reported in a small box. The signal itself is not changed by the function.

The final way to measure peak areas involves the use of the Rubber band cursor (apple-5). This method allows you to use the mouse to mark the x-axis range over which the area is to be measured. Select Window 1, then press apple-5. A black vertical cursor line will be drawn on the window. Position the cursor somewhere to the left of the triplet and click and release the mouse button. The cursor line is erased. Now move the mouse to the right. You will now see two black vertical lines extending below the signal line, one fixed at the first place you clicked and the other tracking your mouse movements. They are connected by a "rubber band " stretching between their tops. The two vertical lines define the "integration limits", within which the area under the signal line is measured. Position the right-hand line somewhere to the right of the triplet. Meanwhile, in the first line at the top of the window, the display has changed. The first number reported there, labeled "Total area", is the area bounded by the signal curve, the X-axis, and two vertical sections of cursor that you just positioned. Note that the area display dynamically tracks changes in the mouse position, allowing you to observe the effect of changing the right-hand integration limit. When you have recorded the desired area, press the mouse button again (or press any key). The cursor is erased.

One area of analytical measurement where peak area measurements are important is in chromatography. Quantitation in analytical chromatography is customarily carried out on the basis of peak area rather than peak height measurement. The reason for this is that peak area is less sensitive to the influence of peak broadening (dispersion) mechanisms. These broadening effects, which arise from many sources, cause chromatographic peaks to become shorter, broader, and more unsymmetrical, but have little effect on the total area under the peak. The peak area remains proportional to the total quantity of substance passing into the detector. Therefore peak area measurements are often found to be more reliable than peak height measurement. As an example, open the file "Test area 1". This is a series of computer-generated "chromatographic" peaks; the one on the left is a pure, symmetrical Gaussian with a total peak area of about 1.0. The other peaks are derived from the first peak by a series of progressively more severe broadening functions that make the peaks shorter, broader, and slightly unsymmetrical. However, the area under each of these peaks is actually the same. Use the Rubber band cursor (apple-5) to measure the areas of each of these peaks in turn. The peak areas are seen to be much more nearly equal that the peak heights.

A common problem in chromatographic separations is that adjacent peaks are not always completely separated. Although the selection of columns and chromatographic conditions aims to optimize separation, it is often impossible or impractical to achieve a complete "baseline resolution" for all peaks. In such cases the measurement of peak area becomes more difficult. For example, load the file "test area 2". This series of four peaks all have the same height, width, and area, but the separation between the peaks on the right is insufficient to achieve complete resolution. The left-most peak is completely resolved; use the rubber band cursor to measure its area in order to establish the "correct" area. Now, if you try to measure the area of the second peak (centered at X=62.4), you will have to decide the best place to locate the right-hand integration limit. As you will discover, the best place is just about at the minimum of the valley between the second and third peaks. In fact, the area measured in this way is very close to the "correct" area. (This is often referred to as a "perpendicular drop" measurement). Next, measure the area of the third peak (centered at X=83.1). This is the most difficult case, because there are poorly-resolved peaks on both sides. Now you have to locate the left-hand and right-hand integration limits carefully. As before, locating these limits at the minima of the valleys on either side of the peak maximum gives good results. In general, this method of area measurement works well if the unresolved peaks are fairly symmetrical and are about the same height, and if the valleys between the peaks then are no higher than about 50% of the height of the measured peak. In practice, one or more of these conditions may not be met, in which case the area measurements will be only approximate.

Another common problem in methods is baseline drift. If the baseline (that is, the signal level when no peaks are occurring) is not zero, and particularly if it is drifting, then the area measurements have to be corrected for the baseline. In such cases another measure of peak area may be preferred. This is the "tangential skim" area. It is also measured using the rubber band cursor, and it is defined as the area between the signal curve and the rubber band line (as compared to the total area, which is the area between the signal curve and the X-axis). For an isolated band on a zero baseline, the total area and tangential skim areas are equal (because then the rubber band falls on the X-axis). The advantage of the tangential skim is that it is not effected by linear sloping background. For example, load the file "test area 3". This is yet another computer-generated synthetic signal. This one has four small peaks, all with equal area, but each sitting on a different kind of background. Measure the area of the left-most peak, which has a zero baseline. The total area and tangential skim areas are equal. Now measure the second peak, which is on a flat but non-zero baseline. Only the tangential skim area is correct. Next, measure the third peak, which is on a linear sloping baseline. Even in this case, the tangential skim area is correct, because the rubber band cursor is a good approximation to the real baseline. The fourth and last peak is on a curved baseline. As you will observe, the tangential skim method is only approximate in this case, because the rubber band cursor is only an approximation of the real baseline. However if the baseline is not too badly curved, the approximation is fairly reasonable. This last situation occurs very commonly in trace analysis, where the peak of the analyte falls on the sloping side of a very much larger peak due to a major component. The tangential skim method is useful, but not perfect, in such cases. More sophisticated methods, in which the baseline is approximated by higher-order polynomials or other function defined by more than two points, are beyond the scope of this program.

Lab Report
For each of the main parts A through F, your lab report should describe the results you obtained, giving numerical results if called for and describing whether that part of the experiment behaved at you thought it should. It is not necessary to print out the signal from every step of each part - that would generate far too many pages. In fact, since you now have to pay for print-outs, I don't require that you print out anything; ust describe in words what you observed.

This page is maintained by Tom O'Haver , Department of Chemistry and Biochemistry, The University of Maryland at College Park. Comments, suggestions and questions should be directed to Prof. O'Haver at toh@umd.edu.
Number of unique visits since May 17, 2008: