Extracting bits from serial stream?

Im trying to parse the serial data from a sensor but i (think?) im struggling to format the angle offset correctly,
Below is the format of the packet i am receiving and relevant descriptions.

my distance and start angle seem to match up (in the real world) but my angle compensation is way off.

Im pretty sure i got the compensation method correct.


so im guessing there is an error in my bit shifting and byte masking for the angle compensation bits.
Some fresh eyes would be most welcome. i suspect the issue is in the β€œget angle compensation” but i have included the rest incase.

relevant code

 // get start angle
   float currentStartAngle = (packet[2] + ((packet[3] & 0x7F) << 8))/64.;
   println("packet count = "+count+" current start angle = " + currentStartAngle + " last start angle = "+lastStartAngle + " angle difference = "+angleDiff(lastStartAngle, currentStartAngle));
   
  //break down to 5 bit cabins starting at 4
  for(int i = 4; i<= 79; i = i+5){
      
    //get angle compensation 
    float angleComp1 = (packet[i+4] & 0x0F) + ((packet[i] & 0x03) << 4);
    float angleComp2 = (packet[i+4] >> 4) + ((packet[i+2] & 0x03) << 4);
    println(" angleComp1 = "+angleComp1+" " + " angleComp2 = " + angleComp2); 
    
    //calculate real angle
    float realAngle1 = lastStartAngle + angleDiff(lastStartAngle, currentStartAngle)/32 * count - angleComp1; 
    float realAngle2 = lastStartAngle + angleDiff(lastStartAngle, currentStartAngle)/32 * count - angleComp2; 
    println(" realAngle1 = "+realAngle1+" " + " realAngle2 = " + realAngle2); 
    
    //get distances
    float distance1 = (int(packet[i]>> 2) + int(packet[i+1]<< 6));  //shift and add packets
    float distance2 = (int(packet[i+2]>> 2) + int(packet[i+3]<< 6));
    //println(" distance1 = "+distance1+" " + " distance2 = " + distance2); 
    
    m_currScan.measurements[m_currScan.measurementCount++] = new LidarExpressMeasurement(realAngle1, distance1);
    m_currScan.measurements[m_currScan.measurementCount++] = new LidarExpressMeasurement(realAngle2, distance2);
  }
    lastStartAngle = currentStartAngle;
  }
  
  //angle difference method 
 public float angleDiff(float lSAngle, float cSAngle){
     if (lSAngle<=cSAngle) 
     return (cSAngle-lSAngle);
     
     else if(lSAngle>cSAngle)
     return (360. + cSAngle - lSAngle);
     
     return 0;
}

And the print out from a couple of scans

packet count = 244 current start angle = 163.5625 last start angle = 126.328125 angle difference = 37.234375

 angleComp1 = 62.0  angleComp2 = 46.0

 realAngle1 = 348.24023  realAngle2 = 364.24023

 angleComp1 = 62.0  angleComp2 = 46.0

 realAngle1 = 348.24023  realAngle2 = 364.24023

 angleComp1 = 61.0  angleComp2 = 45.0

 realAngle1 = 349.24023  realAngle2 = 365.24023

 angleComp1 = 62.0  angleComp2 = 46.0

 realAngle1 = 348.24023  realAngle2 = 364.24023

 angleComp1 = 62.0  angleComp2 = 45.0

 realAngle1 = 348.24023  realAngle2 = 365.24023

 angleComp1 = 61.0  angleComp2 = 45.0

 realAngle1 = 349.24023  realAngle2 = 365.24023

 angleComp1 = 62.0  angleComp2 = 45.0

 realAngle1 = 348.24023  realAngle2 = 365.24023

 angleComp1 = 62.0  angleComp2 = 46.0

 realAngle1 = 348.24023  realAngle2 = 364.24023

 angleComp1 = 62.0  angleComp2 = 46.0

 realAngle1 = 348.24023  realAngle2 = 364.24023

 angleComp1 = 62.0  angleComp2 = 46.0

 realAngle1 = 348.24023  realAngle2 = 364.24023

 angleComp1 = 62.0  angleComp2 = 46.0

 realAngle1 = 348.24023  realAngle2 = 364.24023

 angleComp1 = 62.0  angleComp2 = 46.0

 realAngle1 = 348.24023  realAngle2 = 364.24023

 angleComp1 = 62.0  angleComp2 = 46.0

 realAngle1 = 348.24023  realAngle2 = 364.24023

 angleComp1 = 62.0  angleComp2 = 46.0

 realAngle1 = 348.24023  realAngle2 = 364.24023

 angleComp1 = 62.0  angleComp2 = 46.0

 realAngle1 = 348.24023  realAngle2 = 364.24023

 angleComp1 = 0.0  angleComp2 = 45.0

 realAngle1 = 410.24023  realAngle2 = 365.24023

packet count = 246 current start angle = 204.5 last start angle = 163.5625 angle difference = 40.9375

 angleComp1 = 0.0  angleComp2 = 0.0

 realAngle1 = 478.26953  realAngle2 = 478.26953

 angleComp1 = 61.0  angleComp2 = 45.0

 realAngle1 = 417.26953  realAngle2 = 433.26953

 angleComp1 = 61.0  angleComp2 = 45.0

 realAngle1 = 417.26953  realAngle2 = 433.26953

 angleComp1 = 61.0  angleComp2 = 45.0

 realAngle1 = 417.26953  realAngle2 = 433.26953

 angleComp1 = 0.0  angleComp2 = 0.0

 realAngle1 = 478.26953  realAngle2 = 478.26953

 angleComp1 = 0.0  angleComp2 = 0.0

 realAngle1 = 478.26953  realAngle2 = 478.26953

 angleComp1 = 0.0  angleComp2 = 0.0

 realAngle1 = 478.26953  realAngle2 = 478.26953

 angleComp1 = 0.0  angleComp2 = 43.0

 realAngle1 = 478.26953  realAngle2 = 435.26953

 angleComp1 = 59.0  angleComp2 = 42.0

 realAngle1 = 419.26953  realAngle2 = 436.26953

 angleComp1 = 58.0  angleComp2 = 42.0

 realAngle1 = 420.26953  realAngle2 = 436.26953

 angleComp1 = 57.0  angleComp2 = 41.0

 realAngle1 = 421.26953  realAngle2 = 437.26953

 angleComp1 = 57.0  angleComp2 = 41.0

 realAngle1 = 421.26953  realAngle2 = 437.26953

 angleComp1 = 57.0  angleComp2 = 41.0

 realAngle1 = 421.26953  realAngle2 = 437.26953

 angleComp1 = 57.0  angleComp2 = 41.0

 realAngle1 = 421.26953  realAngle2 = 437.26953

 angleComp1 = 57.0  angleComp2 = 41.0

 realAngle1 = 421.26953  realAngle2 = 437.26953

 angleComp1 = 56.0  angleComp2 = 40.0

 realAngle1 = 422.26953  realAngle2 = 438.26953

1 Like

Hello,

This is what I did to explore and it seems fine:

int p1 = 0b00000001; // 0x01
int p2 = 0b00000010; // 0x02
int p4 = 0b01011010; // 0x5A

int ac1 = (p4 & 0x0F) + ((p1 & 0x03) << 4);
int ac2 = (p4 >>> 4) + ((p2 & 0x03) << 4);

println (ac1, ac2);
println (hex(ac1), hex(ac2));

If you don’t mind sharing I am interested in:

  • sensor
  • serial stream sent and how it is parsed into arrays

I am assuming Arduino is sending a stream of bytes to represent the data and you are parsing\storing it into an array on the Processing side.

’ :slight_smile: ’

1 Like

Hi glv, of course im happy to share, any thoughts are always apreciated

its an β€œrplidar a2” sensor, the interfacing and protocol notes are here.

http://bucket.download.slamtec.com/ccb3c2fc1e66bb00bd4370e208b670217c8b55fa/LR001_SLAMTEC_rplidar_protocol_v2.1_en.pdf

The basis of the code im using can be found here

but im trying to use it in express mode (give a higher resolution than the standard mode adams code uses)

Although not coded in yet, i have checked to see the correct response descriptor is being returned = a5 5a 54 00 00 40 82 for legacy express (see page 18 of the protocol)

So this is the logic,

serial Event

  public void serialEvent(Serial port)
  {
    if (!m_isScanning)
      return;

    final int lengthOfDataPacket = 84;
    byte[] packet = new byte[lengthOfDataPacket];

    while (m_port.available() >= lengthOfDataPacket)
    {
      m_port.readBytes(packet);
      parseCabinMeasurement(packet);
    }
  }

buffer

  protected void readBytes(byte buffer[])
  {
    int i = 0;
    while (i < buffer.length && millis() - m_startTime < m_timeout)
    {
      if (m_port.available() == 0)
        continue;

      byte currByte = byte(m_port.read());
      buffer[i++] = currByte;
    }

    if (millis() - m_startTime >= m_timeout)
    {
      println("TIMEOUT");
      throw new RuntimeException("Timed out waiting for bytes to read.");
    }
  }

check for the sync and start bits (as per page 22)

  protected boolean validPacket(byte[] packet)
  {
    
      int s1 =0;
      int s2 =0;
      binary(s1);
      s1 = (packet[0]>> 4);

      binary(s2);
      s2 = (packet[1]>> 4);
     if ( s1==-6  && s2 == 5 ) 
     {
    
       return true;
     }
     else return false;
      
  }   

The full function for parsing the packet and the angle difference method as described on page 30-31.

protected void parseCabinMeasurement(byte[] packet)
  {
    if (validPacket(packet) == true && isNewScanStart(packet)==true){
       //m_currScan.scanRateInHz = 100.0f / float((int)(System.currentTimeMillis() - m_currScan.startTime));
      m_lastScan = m_currScan;
      m_currScan = new LidarExpressScan();
     count = count +1;
    }
    
   // get start angle
   float currentStartAngle = (packet[2] + ((packet[3] & 0x7F) << 8))/64.;
   println("packet count = "+count+" current start angle = " + currentStartAngle + " last start angle = "+lastStartAngle + " angle difference = "+angleDiff(lastStartAngle, currentStartAngle));
   
  //break down to 5 bit cabins starting at 4
  for(int i = 4; i<= 79; i = i+5){
      
    //get angle compensation 
   // float angleComp1 = (packet[i+4] & 0x0F) + ((packet[i] & 0x03) << 4);
   // float angleComp2 = (packet[i+4] >> 4) + ((packet[i+2] & 0x03) << 4);
     float angleComp1 = (packet[i+4] & 0x0F) + ((packet[i] & 0x03) << 4);
    float angleComp2 = (packet[i+4] >> 4) + ((packet[i+2] & 0x03) << 4);
    
    println(" angleComp1 = "+angleComp1+" " + " angleComp2 = " + angleComp2); 
    
    //calculate real angle
    float realAngle1 = lastStartAngle + angleDiff(lastStartAngle, currentStartAngle)/32 * count - angleComp1; 
    float realAngle2 = lastStartAngle + angleDiff(lastStartAngle, currentStartAngle)/32 * count - angleComp2; 
    println(" realAngle1 = "+realAngle1+" " + " realAngle2 = " + realAngle2); 
    
    //get distances
    float distance1 = (int(packet[i]>> 2) + int(packet[i+1]<< 6));  //shift and add packets
    float distance2 = (int(packet[i+2]>> 2) + int(packet[i+3]<< 6));
    println(" distance1 = "+distance1+" " + " distance2 = " + distance2); 
    
    m_currScan.measurements[m_currScan.measurementCount++] = new LidarExpressMeasurement(realAngle1, distance1);
    m_currScan.measurements[m_currScan.measurementCount++] = new LidarExpressMeasurement(realAngle2, distance2);
  }
    lastStartAngle = currentStartAngle;
  }
  
  //angle difference method 
 public float angleDiff(float lSAngle, float cSAngle){
     if (lSAngle<=cSAngle) 
     return (cSAngle-lSAngle);
     
     else if(lSAngle>cSAngle)
     return (360. + cSAngle - lSAngle);
     
     return 0;
}
2 Likes

edit.

Think i might have found my issue, page 30
β€œSince in express scan mode, the corresponding data of measurement sampling is stored respectively and sequentially in the Cabin of each data response packet. In every 𝑃𝑖 response data, the data of the kth measurement sampling point is denoted as πΆπ‘˜. It carries thedistance value π·π‘–π‘ π‘‘π‘Žπ‘›π‘π‘’π‘˜ and the included angle compensation π‘‘πœƒπ‘˜ corresponded with the current measurement sampling point”

I had put that into the equation as the count of scans which is inccorect.
so it should look like

float realAngle1 = lastStartAngle + angleDiff(lastStartAngle, currentStartAngle)/32 * k - angleComp1;
Except im a bit confused with what k actually is? it just the position of the relvant angle value in that packet?

ie (although this didnt appear to solve my problem)

    count = count+1;
    //calculate real angle 
    float realAngle1 = lastStartAngle + angleDiff(lastStartAngle, currentStartAngle)/32 * count - angleComp1; 
    count = count+1;
    float realAngle2 = lastStartAngle + angleDiff(lastStartAngle, currentStartAngle)/32 * count - angleComp2; 
    println(" realAngle1 = "+realAngle1+" " + " realAngle2 = " + realAngle2 + " i = "+i); 

also worth noting, im 99% sure the parsing is correct for the distance and start angle.
As you can see the scan bellow about halfway between 166.67188 to 203.07812, it registers an object i placed at 180 degress and circa 40cm away.

packet count = 1219 current start angle = 166.67188 last start angle = 133.9375 angle difference = 32.734375
 angleComp1 = 61.0  angleComp2 = 45.0
 realAngle1 = 1319.9126  realAngle2 = 1335.9126
 distance1 = 3714.0  distance2 = 3679.0
 angleComp1 = 61.0  angleComp2 = 45.0
 realAngle1 = 1319.9126  realAngle2 = 1335.9126
 distance1 = 3610.0  distance2 = 3399.0
 angleComp1 = 61.0  angleComp2 = 45.0
 realAngle1 = 1319.9126  realAngle2 = 1335.9126
 distance1 = 3301.0  distance2 = 3487.0
 angleComp1 = 62.0  angleComp2 = 46.0
 realAngle1 = 1318.9126  realAngle2 = 1334.9126
 distance1 = 4058.0  distance2 = 4021.0
 angleComp1 = 37.0  angleComp2 = 35.0
 realAngle1 = 1343.9126  realAngle2 = 1345.9126
 distance1 = 363.0  distance2 = 414.0
 angleComp1 = 35.0  angleComp2 = 37.0
 realAngle1 = 1345.9126  realAngle2 = 1343.9126
 distance1 = 408.0  distance2 = 403.0
 angleComp1 = 34.0  angleComp2 = 36.0
 realAngle1 = 1346.9126  realAngle2 = 1344.9126
 distance1 = 398.0  distance2 = 403.0
 angleComp1 = 36.0  angleComp2 = 37.0
 realAngle1 = 1344.9126  realAngle2 = 1343.9126
 distance1 = 405.0  distance2 = 412.0
 angleComp1 = 36.0  angleComp2 = 0.0
 realAngle1 = 1344.9126  realAngle2 = 1380.9126
 distance1 = 358.0  distance2 = 0.0
 angleComp1 = 0.0  angleComp2 = 0.0
 realAngle1 = 1380.9126  realAngle2 = 1380.9126
 distance1 = 0.0  distance2 = 0.0
 angleComp1 = 0.0  angleComp2 = 43.0
 realAngle1 = 1380.9126  realAngle2 = 1337.9126
 distance1 = 0.0  distance2 = 1972.0
 angleComp1 = 59.0  angleComp2 = 43.0
 realAngle1 = 1321.9126  realAngle2 = 1337.9126
 distance1 = 2054.0  distance2 = 2059.0
 angleComp1 = 59.0  angleComp2 = 43.0
 realAngle1 = 1321.9126  realAngle2 = 1337.9126
 distance1 = 2020.0  distance2 = 2026.0
 angleComp1 = 59.0  angleComp2 = 43.0
 realAngle1 = 1321.9126  realAngle2 = 1337.9126
 distance1 = 2043.0  distance2 = 2126.0
 angleComp1 = 59.0  angleComp2 = 43.0
 realAngle1 = 1321.9126  realAngle2 = 1337.9126
 distance1 = 2084.0  distance2 = 2107.0
 angleComp1 = 59.0  angleComp2 = 45.0
 realAngle1 = 1321.9126  realAngle2 = 1335.9126
 distance1 = 2186.0  distance2 = 3155.0
packet count = 1220 current start angle = 203.07812 last start angle = 166.67188 angle difference = 36.40625


2 Likes

Did this end up resolving your issue, @steve_j?

Nope, got me a little bit closer, still seems to be an issue somewhere with parsing the bits.

Edit:

Maybe to define the question a bit better.

How should one approach parsing the above, start angle, distance1, and distance2, along with d01 and d02.

starting with the start angle.

I currently have for start angle
float currentStartAngle = (packet[2]+ ((packet[3] & 0x7F) << 8) )/64.; The divide by 64 is mentioned in the paper. It mentions it should be a q6 fixed point. Is a float here ok?

1 Like

Hello,

Can you please provide one sample of a packet and output\processed data from that packet\scan.

printArray(packet);

A bit under the weather at the moment but want to look further into this.

’ :slight_smile: ’

1 Like

Hi glv. Sorry to hear that and hope you feel better soon. Also thank you. I have really been struggling with this so any input you could provide would be most appreciated.

Please see bellow, i have included a printout of the packet and a past packet (because the values are calculated in retrospect) and a hex printout of each along with my output values

edit: have also attached two frames to highlight the issues im having. The lidar is inside a box with 0degrees facing into a corner, so i should get a dimond shape.

This is with no angle comp applied. its close but there is the issue with the depth, i havent been able to identify, i dont think its related to the angle comp.


with the angle comp applied as per my printout bellow, its wildly skewing everything.

// packet array as hex 
AB 51 0D 0B BF 0D E3 0D 34 3B 0E 37 0E 44 9B 0E EF 0E 44 37 0F 73 0F 55 C3 0F 23 10 55 6B 10 D3 10 56 27 11 AB 11 66 07 12 8F 12 76 1B 13 9B 13 77 2F 14 CB 14 88 77 15 1B 16 88 D3 16 AB 17 98 6B 18 EF 17 99 1B 18 AB 1B A9 DB 1C 47 1E BA F3 1F 8B 21 BB 
//past packet as hex 
AE 56 F9 01 F3 0A F3 0A 01 F2 0A F6 0A FF FF 0A 03 0B 00 0B 0B 13 0B 00 1F 0B 2B 0B 00 37 0B 3F 0B 01 4B 0B 5F 0B 10 73 0B 8B 0B 11 9B 0B A7 0B 11 00 00 B3 0A 00 BB 0A DF 0A 00 B7 0B 47 0C 21 5F 0C 87 0C 21 AB 0C CB 0C 32 EF 0C 23 0D 22 53 0D 5B 0D 33 
// print array packet 
[0] -85
[1] 81
[2] 13
[3] 11
[4] -65
[5] 13
[6] -29
[7] 13
[8] 52
[9] 59
[10] 14
[11] 55
[12] 14
[13] 68
[14] -101
[15] 14
[16] -17
[17] 14
[18] 68
[19] 55
[20] 15
[21] 115
[22] 15
[23] 85
[24] -61
[25] 15
[26] 35
[27] 16
[28] 85
[29] 107
[30] 16
[31] -45
[32] 16
[33] 86
[34] 39
[35] 17
[36] -85
[37] 17
[38] 102
[39] 7
[40] 18
[41] -113
[42] 18
[43] 118
[44] 27
[45] 19
[46] -101
[47] 19
[48] 119
[49] 47
[50] 20
[51] -53
[52] 20
[53] -120
[54] 119
[55] 21
[56] 27
[57] 22
[58] -120
[59] -45
[60] 22
[61] -85
[62] 23
[63] -104
[64] 107
[65] 24
[66] -17
[67] 23
[68] -103
[69] 27
[70] 24
[71] -85
[72] 27
[73] -87
[74] -37
[75] 28
[76] 71
[77] 30
[78] -70
[79] -13
[80] 31
[81] -117
[82] 33
[83] -69
//print array past packet
[0] -82
[1] 86
[2] -7
[3] 1
[4] -13
[5] 10
[6] -13
[7] 10
[8] 1
[9] -14
[10] 10
[11] -10
[12] 10
[13] -1
[14] -1
[15] 10
[16] 3
[17] 11
[18] 0
[19] 11
[20] 11
[21] 19
[22] 11
[23] 0
[24] 31
[25] 11
[26] 43
[27] 11
[28] 0
[29] 55
[30] 11
[31] 63
[32] 11
[33] 1
[34] 75
[35] 11
[36] 95
[37] 11
[38] 16
[39] 115
[40] 11
[41] -117
[42] 11
[43] 17
[44] -101
[45] 11
[46] -89
[47] 11
[48] 17
[49] 0
[50] 0
[51] -77
[52] 10
[53] 0
[54] -69
[55] 10
[56] -33
[57] 10
[58] 0
[59] -73
[60] 11
[61] 71
[62] 12
[63] 33
[64] 95
[65] 12
[66] -121
[67] 12
[68] 33
[69] -85
[70] 12
[71] -53
[72] 12
[73] 50
[74] -17
[75] 12
[76] 35
[77] 13
[78] 34
[79] 83
[80] 13
[81] 91
[82] 13
[83] 51
// my processed results 
 currentStartAngle = 44.203125 angleDiff = 40.3125 
 count = 1 angleComp1 = 49.0 realAngle1 = -43.84961 distance1 = 636.0
 count = 2 angleComp2 = 48.0 realAngle2 = -41.589844 distance2 = 636.0
 count = 3 angleComp1 = 47.0 realAngle1 = -39.33008 distance1 = 636.0
 count = 4 angleComp2 = 31.0 realAngle2 = -22.070312 distance2 = 637.0
 count = 5 angleComp1 = 48.0 realAngle1 = -37.810547 distance1 = 639.0
 count = 6 angleComp2 = 48.0 realAngle2 = -36.55078 distance2 = 704.0
 count = 7 angleComp1 = 48.0 realAngle1 = -35.291016 distance1 = 706.0
 count = 8 angleComp2 = 48.0 realAngle2 = -34.03125 distance2 = 708.0
 count = 9 angleComp1 = 48.0 realAngle1 = -32.771484 distance1 = 711.0
 count = 10 angleComp2 = 48.0 realAngle2 = -31.511719 distance2 = 714.0
 count = 11 angleComp1 = 49.0 realAngle1 = -31.251953 distance1 = 717.0
 count = 12 angleComp2 = 48.0 realAngle2 = -28.992188 distance2 = 719.0
 count = 13 angleComp1 = 48.0 realAngle1 = -27.732422 distance1 = 722.0
 count = 14 angleComp2 = 49.0 realAngle2 = -27.472656 distance2 = 727.0
 count = 15 angleComp1 = 49.0 realAngle1 = -26.21289 distance1 = 732.0
 count = 16 angleComp2 = 49.0 realAngle2 = -24.953125 distance2 = 674.0
 count = 17 angleComp1 = 49.0 realAngle1 = -23.69336 distance1 = 678.0
 count = 18 angleComp2 = 49.0 realAngle2 = -22.433594 distance2 = 681.0
 count = 19 angleComp1 = 0.0 realAngle1 = 27.826172 distance1 = 0.0
 count = 20 angleComp2 = 48.0 realAngle2 = -18.914062 distance2 = 620.0
 count = 21 angleComp1 = 48.0 realAngle1 = -17.654297 distance1 = 622.0
 count = 22 angleComp2 = 48.0 realAngle2 = -16.394531 distance2 = 631.0
 count = 23 angleComp1 = 49.0 realAngle1 = -16.134766 distance1 = 685.0
 count = 24 angleComp2 = 50.0 realAngle2 = -15.875 distance2 = 785.0
 count = 25 angleComp1 = 49.0 realAngle1 = -13.615234 distance1 = 791.0
 count = 26 angleComp2 = 50.0 realAngle2 = -13.355469 distance2 = 737.0
 count = 27 angleComp1 = 50.0 realAngle1 = -12.095703 distance1 = 746.0
 count = 28 angleComp2 = 51.0 realAngle2 = -11.8359375 distance2 = 754.0
 count = 29 angleComp1 = 50.0 realAngle1 = -9.576172 distance1 = 763.0
 count = 30 angleComp2 = 50.0 realAngle2 = -8.316406 distance2 = 840.0
 count = 31 angleComp1 = 51.0 realAngle1 = -8.056641 distance1 = 852.0
 count = 32 angleComp2 = 51.0 realAngle2 = -6.796875 distance2 = 854.0

Hello,

Looking through your angleComp data there are a few that are really off.
They range from 47 to 51 and a I see 2 exceptions of 0 and 31; these may be causing the skewing.

’ :slight_smile: ’