ValMetTelem
Problem Statement
I found a steady stream of FSK activity on 452.625Mhz, 452.825Mhz, and 453.150Mhz. I looked this up and found it to be associated with the Phoenix area bus system. I wanted to know what exactly was being transmitted, maybe I can track the bus and never be late again?
Data Format
Using GNU-radio and baudline, I discovered that these FSK signals have a deviation of 4800Hz, and a bitrate of also 4800Hz. I constructed a demodulator and bitsync in GNUradio, and discovered the following packet structure. It is easy to see that this is some sort of manchester encoding as there is a transition during every 2 bis and there are no consecutive bits within the groups.
1 10 10 01 01 10 10 01 01 10 10 01 01 10 10 01 01 10 10 01 01 10 10 01 01 10 10 01 01 10 10 01 01 10 10 01 01 10 10 01 01 10 10 01 01 10 10 01 01 10 10 01 01 10 10 01 01 10 10 01 01 10 10 01 01 01 10 01 10 01 10 01 01 01 10 01 01 10 10 10 01 10 10 10 01 10 10 10 10 10 10 (the rest) 101010101010010101011001101001010101100110100101010110011010010101011001101001010101100110100101010110011010010101011001101001010101100110100101010110011010010101011001101001010101100110100101010110011010010101011001101001010101100110100101010110011010010101011001101001010101100110100101010110011010010101011001101001010101100110100101010110011010010101011001101001010101100110100101010110011010011001101010010101011001011010010110011001100101101001011010010
Decoding this using standard manchester (1 => 10 and 0 => 01) yielded some really funny patterns that did not make sense to me. Who uses a preamble like this?! The following is my best attempt to break the code into 8 bit chunks at boundaries that appeared to make sense... But the pattern does not make sense. This stumped me for a while...
00110011 00110011 00110011 00110011 00110011 00110011 00110011 00110011 10101011 11111111 11111111 10000000 01111111 11011000 11100000 11101111 1001 0011 00001011 00001011 00001011 00001011 00001011 00001011 00001011 00001011 00001011 00001011 00001011 00001011 00001011 00001011 00001011 00001011 00001011 00001011 00001011 10111010 11100111 10101011 00110011
It is clear there are far too many transitions still left in the data. You can just see it. Then suddenly it dawned on me. This is DIFFERENTIAL Manchester encoding! We can actually convert manchester decoded data to differential manchester data by looking at the bit vs the previous bit. A 1 is a change and a 0 is no change. Bit_previous == Bit_Current? Yes=0, No=1.
After we apply this differential transform, we arrive at data that makes much more sense.
10101010 10101010 10101010 10101010 10101010 10101010 10101010 10101010 01111110 01101001 10011000 00000000 10001110 10001110 10001110 10001110 10001110 10001110 10001110 10001110 10001110 10001110 10001110 10001110 10001110 10001110 10001110 10001110 10001110 10001110 10001110 10001110 10001110 10001110 10001110 10001110 11110010 00110101 01111110 1010101 [implied trailing 0] This is: AA AA 7E 69 98 00 8E8E8E8E8E8E8E8E8E8E8E8E8E8E8E8E8E8E8E8E8E8E8E8E F2 35 7E AA Confirmation of data structure made by grabbing a second packet: AA AA 7E 00 00 00 8E8E8E8E8E8E8E8E8E8E8E8E8E8E8E8E8E8E8E8E8E8E8E8E 97 80 7E AA
Matlab Script to Do All of This For Us
% Reads demodulated and bitsynced samples from the gnuradio file clear all; hfile = 'bits_all.raw'; fid = fopen(hfile,'rb'); data = fread(fid,'float32'); fclose(fid); Nfft = 1500000; Ts = 1/(170666); N = numel(data); RawTime=0:Ts:Ts*(N-1); %% *BITSTREAM - convert levels to bits clear bitstream; bitstream = uint8(0); idx2 = 1; for idx = 1:numel(data) if(data(idx) > 0) %reverse if sync words are inverted bitstream(idx2) = '1'; idx2 = idx2 + 1; elseif(data(idx) < -0) bitstream(idx2) = '0'; idx2 = idx2 + 1; end end fprintf(['Symbols found: ' num2str(numel(bitstream)) ' or ' num2str(numel(bitstream)/16) ' Bytes \n' ]); %% *BITSTREAM - Manchester Decode bitThreshold = 0.05; idx2=1; clockmod = 1; idxerr=1; %idxerr2=1; clear errx erry bitstream_manchester BitTime; bitstream_manchester(1) = uint8(0); for idx = 2:numel(bitstream)-1 %check if we need to re-sync the bit window if(mod(idx,2) ~= clockmod) %We should never see two 1's or two 0's in a row during a bit window if(bitstream(idx-1) == bitstream(idx)) %log sync errors errx(idxerr)=idx2; erry(idxerr)=data(idx); idxerr=idxerr+1; %But only resync if we have confidence in the bit decisions, if one of the values is weak, let it slide. if(abs(data(idx-1)) > bitThreshold && abs(data(idx)) > bitThreshold) clockmod = mod(idx,2); end end end %check for bit boundary, if true, then make decision the bit if(mod(idx,2) == clockmod) %use the strongest symbol of the pair to determine bit, to be more %better sure N stuff. if(abs(data(idx)) > abs(data(idx+1))) if(bitstream(idx) == '1') bitstream_manchester(idx2) = '1'; else bitstream_manchester(idx2) = '0'; end else if(bitstream(idx+1) == '1') bitstream_manchester(idx2) = '0'; else bitstream_manchester(idx2) = '1'; end end %Track new stream time BitTime(idx2)=RawTime(idx); idx2 = idx2+1; end end %% BITSTREAM - Test Differential Manchester Decode testStream = '11000101110'; clear bitstreamDManchester; for idx = 1:numel(testStream) if(idx == 1) %init the loop bitstreamDManchester(idx) = testStream(idx); elseif(testStream(idx) == testStream(idx-1)) bitstreamDManchester(idx) = '0'; else bitstreamDManchester(idx) = '1'; end end if(strcmp(bitstreamDManchester,'10100111001')) fprintf('success!\n'); else fprintf(bitstreamDManchester); fprintf('\n'); fprintf('failure!\n'); end %% *BITSTREAM - Differential Manchester Decode clear bitstreamDManchester; for idx = 1:numel(bitstream_manchester) if(idx == 1) %init the loop bitstreamDManchester(idx) = bitstream_manchester(idx); elseif(bitstream_manchester(idx) == bitstream_manchester(idx-1)) bitstreamDManchester(idx) = '0'; %0 indicates no change else bitstreamDManchester(idx) = '1'; %1 indicates a change end end %% *BITSTREAM - convert ascii binary stream to actual binary at matched syncword locations %Includes correct and inverted syncwords for periods of constellation reversal. SyncWord = '1010101001111110'; SyncWordInverse = '0101010110000001'; SyncWordIndex = strfind(bitstreamDManchester, SyncWord); SyncWordInvIndex = strfind(bitstreamDManchester, SyncWordInverse); fprintf([ '\n' num2str(numel(SyncWordInvIndex)+numel(SyncWordIndex)) ' detected\n' num2str(idxerr) ' errors\n']); SyncWordAllIndex = sort(cat(2,SyncWordIndex,SyncWordInvIndex)); clear minorFrames FrameTime; packetLength = 32; for frameIdx=1:numel(SyncWordAllIndex) %See if the frame is normal or inverted bits if isempty(find(SyncWordInvIndex == SyncWordAllIndex(frameIdx),1)) for frameByteIdx=0:packetLength byte=0; %Start of byte time FrameTime(frameIdx,frameByteIdx+1)=BitTime(SyncWordAllIndex(frameIdx)+frameByteIdx*8); %if this is a normal sync word, use normal bits for bit_idx=0:7 %bytes are 8 bits long ;) if(SyncWordAllIndex(frameIdx)+frameByteIdx*8+bit_idx > numel(bitstreamDManchester)) return; end if(bitstreamDManchester(SyncWordAllIndex(frameIdx)+frameByteIdx*8+bit_idx)=='0') byte = bitshift(byte,1); %This is a zero, just shift else byte = bitshift(byte,1); %This is a one, set the bit then shift byte = bitor(byte,1); end end minorFrames(frameIdx,frameByteIdx+1)=byte; end else %this minor frame is inverted for frameByteIdx=0:packetLength byte=0; %Start of byte time FrameTime(frameIdx,frameByteIdx+1)=BitTime(SyncWordAllIndex(frameIdx)+frameByteIdx*8); for bit_idx=0:7 %bytes are 8 bits long ;) if(SyncWordAllIndex(frameIdx)+frameByteIdx*8+bit_idx > numel(bitstreamDManchester)) return; end if(bitstreamDManchester(SyncWordAllIndex(frameIdx)+frameByteIdx*8+bit_idx)=='0') byte = bitshift(byte,1); %This is a zero, just shift byte = bitor(byte,1); else byte = bitshift(byte,1); %This is a one, set the bit then shift end end minorFrames(frameIdx,frameByteIdx+1)=byte; end end end %% Spit out frames as hex for idx=1:size(minorFrames,1) fprintf('%0.2X',minorFrames(idx,:)); fprintf('\n'); end %% Verify with CRC-16 and pull valid bytes into new array goodData = 0; idx2 = 1; minorFramesValid = zeros(1,29); for idx=1:size(minorFrames,1) if((minorFrames(idx,30)*2^8+minorFrames(idx,31)) == crc16(minorFrames(idx,3:29))) goodData = goodData+1; minorFramesValid(idx2,:) = minorFrames(idx,3:end-2); idx2 = idx2 + 1; end end; fprintf(['Valid packets: ' num2str(size(minorFramesValid,1)) ' (' num2str(goodData/size(minorFrames,1)*100,3) '%% valid)\n' ]);
Possible Packet Structure
- 8 bytes of transitions 0xAA for clock sync
- packet start flag of 0x7E
- Three bytes of, ID, status or mode
- 24 bytes of message structure which appears to be empty in most packets (very strange!)
- Two byte CRC maybe?
- An end of packet marker 0x7E (also weird!)
- another byte of transitions 0XAA, some sort of turn-off delay for the radio to avoid clipping off the end of a packet.
CRC Inspection
I had no idea if this was really CRC or not. It appeared to be, as identical packets had identical bytes at the end, without exception.
The CRC is not standard, but is close. I used the following website http://www.sunshine2k.de/coding/javascript/crc/crc_js.html and just tried the whole dropdown list until I landed on CRC16-genibus. If my bytes or polarity was off, this would not work, so this verifies the byte alignment and polarity of our bits. I actually didn't really think I would ever find the algorithm ;)
The CRC-16 settings are as follows: Polynomial: 0x1021 Initial Value: 0xFFFF Final XOR value: 0xFFFF
= CRC Matlab Code
The following code works, the CRC-16 matches!
function crc_val = crc16 (message) crc = uint16(hex2dec('FFFF')); for i = 1:length(message) crc = bitxor(crc,bitshift(message(i),8)); for j = 1:8 if (bitand(crc, hex2dec('8000')) > 0) crc = bitxor(bitshift(crc, 1), hex2dec('1021')); else crc = bitshift(crc, 1); end end end crc = bitxor(crc,65535); crc_val = dec2hex(crc,4); end Output: crc16(hex2dec(['00'; '00'; '00'; '8E'; '8E'; '8E'; '8E'; '8E'; '8E'; '8E'; '8E'; '8E'; '8E'; '8E'; '8E'; '8E'; '8E'; '8E'; '8E'; '8E'; '8E'; '8E'; '8E'; '8E'; '8E'; '8E'; '8E'])) ans = 0x9780 <- MATCH!!
Data Contents
Attempts to reverse engineer the packet structure are still in work. Multiple patterns have been already noted, but as most packets are actually empty, more data is required than would be assumed at first to decode what these messages might contain.