Difference between revisions of "ValMetTelem"

From NebarnixWiki
Jump to navigationJump to search
Line 3: Line 3:
 
I found a steady stream of FSK activity on 452.625Mhz, 452.825Mhz, and 453.150Mhz. I looked this up and found it to be associated with the Phoenix area bus system. I wanted to know what exactly was being transmitted, maybe I can track the bus and never be late again?
 
I found a steady stream of FSK activity on 452.625Mhz, 452.825Mhz, and 453.150Mhz. I looked this up and found it to be associated with the Phoenix area bus system. I wanted to know what exactly was being transmitted, maybe I can track the bus and never be late again?
  
==Data Format==
+
==Data Data what is the Data!==
  
 
Using GNU-radio and baudline, I discovered that these FSK signals have a deviation of 4800Hz, and a bitrate of also 4800Hz. I constructed a demodulator and bitsync in GNUradio, and discovered the following packet structure. It is  easy to see that this is some sort of manchester encoding as there is a transition during every 2 bis and there are no consecutive bits within the groups.  
 
Using GNU-radio and baudline, I discovered that these FSK signals have a deviation of 4800Hz, and a bitrate of also 4800Hz. I constructed a demodulator and bitsync in GNUradio, and discovered the following packet structure. It is  easy to see that this is some sort of manchester encoding as there is a transition during every 2 bis and there are no consecutive bits within the groups.  
Line 235: Line 235:
 
</pre>
 
</pre>
  
=== Possible Packet Structure===
+
== Possible Packet Structure Dissection==
 
* 8 bytes of transitions 0xAA for clock sync
 
* 8 bytes of transitions 0xAA for clock sync
 
* packet start flag of 0x7E
 
* packet start flag of 0x7E

Revision as of 08:00, 31 March 2016

Problem Statement

I found a steady stream of FSK activity on 452.625Mhz, 452.825Mhz, and 453.150Mhz. I looked this up and found it to be associated with the Phoenix area bus system. I wanted to know what exactly was being transmitted, maybe I can track the bus and never be late again?

Data Data what is the Data!

Using GNU-radio and baudline, I discovered that these FSK signals have a deviation of 4800Hz, and a bitrate of also 4800Hz. I constructed a demodulator and bitsync in GNUradio, and discovered the following packet structure. It is easy to see that this is some sort of manchester encoding as there is a transition during every 2 bis and there are no consecutive bits within the groups.

1 10 10 01 01 10 10 01 01 10 10 01 01 10 10 01 01 10 10 01 01 10 10 01 01 10 10 01 01 10 10 01 01 10 10 01 01 10 10 01 01 10 10 01 01 10 10 01 01 10 10 01 01 10 10 01 01 10 10 01 01 10 10 01 01 01 10 01 10 01 10 01 01 01 10 01 01 10 10 10 01 10 10 10 01 10 10 10 10 10 10 
(the rest) 101010101010010101011001101001010101100110100101010110011010010101011001101001010101100110100101010110011010010101011001101001010101100110100101010110011010010101011001101001010101100110100101010110011010010101011001101001010101100110100101010110011010010101011001101001010101100110100101010110011010010101011001101001010101100110100101010110011010010101011001101001010101100110100101010110011010011001101010010101011001011010010110011001100101101001011010010

Decoding this using standard manchester (1 => 10 and 0 => 01) yielded some really funny patterns that did not make sense to me. Who uses a preamble like this?! The following is my best attempt to break the code into 8 bit chunks at boundaries that appeared to make sense... But the pattern does not make sense. This stumped me for a while...

00110011 00110011 00110011 00110011 00110011 00110011 00110011 00110011  10101011  11111111 11111111  10000000 01111111 11011000 11100000 11101111 1001 0011 00001011 00001011 00001011 00001011 00001011 00001011 00001011 00001011 00001011 00001011 00001011 00001011 00001011 00001011 00001011 00001011 00001011 00001011 00001011 10111010 11100111 10101011 00110011

It is clear there are far too many transitions still left in the data. You can just see it. Then suddenly it dawned on me. This is DIFFERENTIAL Manchester encoding! We can actually convert manchester decoded data to differential manchester data by looking at the bit vs the previous bit. A 1 is a change and a 0 is no change. Bit_previous == Bit_Current? Yes=0, No=1.

After we apply this differential transform, we arrive at data that makes much more sense.

10101010 10101010 10101010 10101010 10101010 10101010 10101010 10101010 01111110 01101001 10011000 00000000 10001110 10001110 10001110 10001110 10001110 10001110 10001110 10001110 10001110 10001110 10001110 10001110 10001110 10001110 10001110 10001110 10001110 10001110 10001110 10001110 10001110 10001110 10001110 10001110 11110010 00110101 01111110 1010101 [implied trailing 0]
This is:
AA AA 7E 69 98 00 8E8E8E8E8E8E8E8E8E8E8E8E8E8E8E8E8E8E8E8E8E8E8E8E F2 35 7E AA

Confirmation of data structure made by grabbing a second packet:
AA AA 7E 00 00 00 8E8E8E8E8E8E8E8E8E8E8E8E8E8E8E8E8E8E8E8E8E8E8E8E 97 80 7E AA

Matlab Script to Do All of This For Us


% Reads demodulated and bitsynced samples from the gnuradio file
% Nebarnix 2016

clear all;

hfile = 'bits_all.raw';
fid = fopen(hfile,'rb');
data = fread(fid,'float32');
fclose(fid);

Nfft = 1500000;
Ts = 1/(170666);
N = numel(data);

RawTime=0:Ts:Ts*(N-1);

%% *BITSTREAM - convert levels to bits
clear bitstream;
bitstream = uint8(0);
idx2 = 1;
for idx = 1:numel(data)
    
    if(data(idx) > 0) %reverse if sync words are inverted
        bitstream(idx2) = '1';
        idx2 = idx2 + 1;
        
    elseif(data(idx) < -0)
        bitstream(idx2) = '0';
        idx2 = idx2 + 1;
        
    end
end
fprintf(['Symbols found: ' num2str(numel(bitstream)) ' or ' num2str(numel(bitstream)/16) ' Bytes \n' ]);

%% *BITSTREAM - Manchester Decode
bitThreshold = 0.05;
idx2=1;
clockmod = 1;
idxerr=1;
%idxerr2=1;
clear errx erry bitstream_manchester BitTime;
bitstream_manchester(1) = uint8(0);
for idx = 2:numel(bitstream)-1
    %check if we need to re-sync the bit window 
    if(mod(idx,2) ~= clockmod)    
        %We should never see two 1's or two 0's in a row during a bit window
        if(bitstream(idx-1) == bitstream(idx))
            
            %log sync errors
            errx(idxerr)=idx2;
            erry(idxerr)=data(idx);
            idxerr=idxerr+1;   
            
            %But only resync if we have confidence in the bit decisions, if one of the values is weak, let it slide. 
            if(abs(data(idx-1)) > bitThreshold && abs(data(idx)) > bitThreshold)                
                clockmod = mod(idx,2); 
            end
            
        end        
    end
    
    %check for bit boundary, if true, then make decision the bit
    if(mod(idx,2) == clockmod)
        %use the strongest symbol of the pair to determine bit, to be more
        %better sure N stuff. 
        if(abs(data(idx)) > abs(data(idx+1))) 
            if(bitstream(idx) == '1')                
                bitstream_manchester(idx2) = '1';
            else
                bitstream_manchester(idx2) = '0';
            end
        else
            if(bitstream(idx+1) == '1')                
                bitstream_manchester(idx2) = '0';
            else
                bitstream_manchester(idx2) = '1';
            end        
        end
        
        %Track new stream time
        BitTime(idx2)=RawTime(idx);    
        idx2 = idx2+1;
    end            
end
 
 
%% BITSTREAM - Test Differential Manchester Decode
testStream = '11000101110';
clear bitstreamDManchester;
for idx = 1:numel(testStream)    
    if(idx == 1) %init the loop
        bitstreamDManchester(idx) = testStream(idx);
    elseif(testStream(idx) == testStream(idx-1))
        bitstreamDManchester(idx) = '0';
    else
        bitstreamDManchester(idx) = '1';
    end
end

if(strcmp(bitstreamDManchester,'10100111001'))
    fprintf('success!\n');
else
    fprintf(bitstreamDManchester);
    fprintf('\n');
    fprintf('failure!\n');   
end


%% *BITSTREAM - Differential Manchester Decode
clear bitstreamDManchester;
for idx = 1:numel(bitstream_manchester)    
    if(idx == 1) %init the loop
        bitstreamDManchester(idx) = bitstream_manchester(idx);
    elseif(bitstream_manchester(idx) == bitstream_manchester(idx-1))
        bitstreamDManchester(idx) = '0'; %0 indicates no change
    else
        bitstreamDManchester(idx) = '1'; %1 indicates a change
    end
end


%% *BITSTREAM - convert ascii binary stream to actual binary at matched syncword locations
%Includes correct and inverted syncwords for periods of constellation reversal. 

SyncWord        = '1010101001111110';
SyncWordInverse = '0101010110000001';

SyncWordIndex = strfind(bitstreamDManchester, SyncWord);
SyncWordInvIndex = strfind(bitstreamDManchester, SyncWordInverse);
fprintf([ '\n' num2str(numel(SyncWordInvIndex)+numel(SyncWordIndex)) ' detected\n' num2str(idxerr) ' errors\n']);
SyncWordAllIndex = sort(cat(2,SyncWordIndex,SyncWordInvIndex));

clear minorFrames FrameTime;

packetLength = 32;

for frameIdx=1:numel(SyncWordAllIndex)
    %See if the frame is normal or inverted bits
    if isempty(find(SyncWordInvIndex == SyncWordAllIndex(frameIdx),1))
        for frameByteIdx=0:packetLength
            byte=0;
            %Start of byte time
            FrameTime(frameIdx,frameByteIdx+1)=BitTime(SyncWordAllIndex(frameIdx)+frameByteIdx*8);
            %if this is a normal sync word, use normal bits
            for bit_idx=0:7  %bytes are 8 bits long ;)     
                if(SyncWordAllIndex(frameIdx)+frameByteIdx*8+bit_idx > numel(bitstreamDManchester))
                    return;
                end
                
                if(bitstreamDManchester(SyncWordAllIndex(frameIdx)+frameByteIdx*8+bit_idx)=='0')               
                    byte = bitshift(byte,1); %This is a zero, just shift           
                else                    
                    byte = bitshift(byte,1); %This is a one, set the bit then shift               
                    byte = bitor(byte,1);              
                end                
            end        
        minorFrames(frameIdx,frameByteIdx+1)=byte;    
        end
    else %this minor frame is inverted
        for frameByteIdx=0:packetLength 
            byte=0;
            %Start of byte time
            FrameTime(frameIdx,frameByteIdx+1)=BitTime(SyncWordAllIndex(frameIdx)+frameByteIdx*8);            
            for bit_idx=0:7  %bytes are 8 bits long ;)                            
                if(SyncWordAllIndex(frameIdx)+frameByteIdx*8+bit_idx > numel(bitstreamDManchester))
                    return;
                end
                
                if(bitstreamDManchester(SyncWordAllIndex(frameIdx)+frameByteIdx*8+bit_idx)=='0')               
                    byte = bitshift(byte,1); %This is a zero, just shift
                    byte = bitor(byte,1);              
                else                    
                    byte = bitshift(byte,1); %This is a one, set the bit then shift                                  
                end                
            end        
        minorFrames(frameIdx,frameByteIdx+1)=byte;    
        end
    end
end

%% Spit out frames as hex
for idx=1:size(minorFrames,1)
   fprintf('%0.2X',minorFrames(idx,:));
   fprintf('\n');
end
    
%% Verify with CRC-16 and pull valid bytes into new array
goodData = 0;
idx2 = 1;
minorFramesValid = zeros(1,29);

for idx=1:size(minorFrames,1)
    if((minorFrames(idx,30)*2^8+minorFrames(idx,31)) == crc16(minorFrames(idx,3:29)))
        goodData = goodData+1;
        minorFramesValid(idx2,:) = minorFrames(idx,3:end-2);
        idx2 = idx2 + 1;
    end           
end;
fprintf(['Valid packets: ' num2str(size(minorFramesValid,1)) ' (' num2str(goodData/size(minorFrames,1)*100,3) '%% valid)\n' ]);

Possible Packet Structure Dissection

  • 8 bytes of transitions 0xAA for clock sync
  • packet start flag of 0x7E
  • Three bytes of, ID, status or mode
  • 24 bytes of message structure which appears to be empty in most packets (very strange!)
  • Two byte CRC maybe?
  • An end of packet marker 0x7E (also weird!)
  • another byte of transitions 0XAA, some sort of turn-off delay for the radio to avoid clipping off the end of a packet.

CRC Inspection

I had no idea if this was really CRC or not. It appeared to be, as identical packets had identical bytes at the end, without exception.

The CRC is not standard, but is close. I used the following website http://www.sunshine2k.de/coding/javascript/crc/crc_js.html and just tried the whole dropdown list until I landed on CRC16-genibus. If my bytes or polarity was off, this would not work, so this verifies the byte alignment and polarity of our bits. I actually didn't really think I would ever find the algorithm ;)

The CRC-16 settings are as follows: Polynomial: 0x1021 Initial Value: 0xFFFF Final XOR value: 0xFFFF

CRC Matlab Code

The following code works, the CRC-16 matches!

function crc_val = crc16 (message)

crc = uint16(hex2dec('FFFF'));

    for i = 1:length(message)
        crc = bitxor(crc,bitshift(message(i),8));

        for j = 1:8
            if (bitand(crc, hex2dec('8000')) > 0)
                crc = bitxor(bitshift(crc, 1), hex2dec('1021'));
            else
                crc = bitshift(crc, 1);
            end
        end
    end
    
    crc = bitxor(crc,65535);
    crc_val = dec2hex(crc,4);
end

Output:

crc16(hex2dec(['00'; '00'; '00'; '8E'; '8E'; '8E'; '8E'; '8E'; '8E'; '8E'; '8E'; '8E'; '8E'; '8E'; '8E'; '8E'; '8E'; '8E'; '8E'; '8E'; '8E'; '8E'; '8E'; '8E'; '8E'; '8E'; '8E']))

ans =

0x9780 <- MATCH!!

Data Contents

Attempts to reverse engineer the packet structure are still in work. Multiple patterns have been already noted, but as most packets are actually empty, more data is required than would be assumed at first to decode what these messages might contain.