Communications Problem
Initially this was a problem in sorting out what had gone wrong with an undersea cable, about 50km long, which was failing to get control system messages through reliably. Sometimes it worked, sometimes it didn't. My task was to guide the work of finding the trouble and then find a fix.
There was a history of attempts to replace modems because of suspected design problems, but the whole record was confused by the reporting of the activities of the dive teams that had been sent out to make replacement changes and reconnections. It was made worse by having found, part way through the earlier work, that a software problem had been present, possibly affecting some of the test results. Bad weather had forced a temporary solution to be applied. A real solution was crucial.

Two Approaches
From the beginning I set about creating a mathematical model of the signal path from end to end so that we could use it to match and record, in a consistent framework, all we knew in real terms and hence be able to use it as a tool to verify that changes would have a good chance of working.
A test rig was set up on the shop floor with real equipment to model the whole system. The exception was the transmission line itself, modelled electrically using analog (operational amplifier circuits) components. It all worked perfectly of course!

A bit of Luck
We went very carefully through the history of the trials and tribulations, reconciling things like different ships recording dates using GMT, others using local time. We came up with hypotheses about partial component failures, failed connections underwater and the like. We set about testing them on the rig.
One of the tests for a simulated fault was done differently from McCann's request. However, it showed something like the behaviour of the actual equipment that was at sea -- some messages OK, others not.
With this clue we designed a "byte-grabber" that could monitor the RS422 line topsides and see exactly what came and went. Meanwhile we proceded with the Mathcad model of the signalling system, using the data originally used for the line properties that had guided the original design.
After a good deal of effort, involving sending people to sea to make measurements of the transmission line using network analysers, getting huge data-logs of bytes from the real system and from the test rig, we reworked our line model so that it was different from original design model and merged it into the evolving system model.

The mathematical model
Eventually we had a mathematical model (see diagram) that created a character string, such as the operating equipment would use, converted it to a bit stream (with all the start and stop bits and spacings) and made that into a time series representation of the FSK signal that was to be sent out. For computational feasibility, the time domain sampling, matched to the actual signals, was chosen to suit the Fast Fouier Transform (FFT) technique and to give step sizes that would work in the differential equation model of the demodulator (modem). By making the bit pattern long enough we could also see the spectral properties of typical signals. We pushed the FFT version of the signal through the frequency domain (transfer function) model of the pre-emphasis filters, the actual line and the filters at the receiving end. This was inverted (IFFT) to give a time series of the received signal and that then became input to a difference equation model of the dynamics of the demodulator, a highly non-linear device.

Conclusion
In the end we found that the original data for transmission line characteristics used in system design had been flawed -- actually a measurement technique problem. The resulting system was marginal and that showed up as the demodulation errors were repeatable, not random, and were message form dependent. The modem was repeatedly tripping up at the same point in some messages. I was able to redesign the filter networks so that they matched the spectral properties of the signals and the real line characteristics and could be modified without changing the PC board layouts (they were made as Sallen and Key filters). The best working criterion we found for deciding, in our model, if we had a good design was to match zero crossings of the recovered signal with the delayed original.
We had a recording made of the signals as they arrived at the undersea modem while a diver's visit was being made to recover equipment for modification.
We knew we were on the right track when we pushed that recording, unaltered, into our newly modified modem and it was decoded right first time! What's more, the modem worked fine when it was installed undersea.