CRC algorithm in ESP32

User avatar
mzimmers
Posts: 643
Joined: Wed Mar 07, 2018 11:54 pm
Location: USA

CRC algorithm in ESP32

Postby mzimmers » Fri Sep 21, 2018 9:10 pm

Hi all -

I've been trying to determine exactly what algorithm the ESP-32 uses for its CRC calculations. (I want to use this for checking the validity of messages passed in through a serial port.) According to the rom/crc.h file,

Code: Select all

// CRC16-CCITT  x16+x12+x5+1   1021   ISO HDLC, ITU X.25, V.34/V.41/V.42, PPP-FCS
but I have reason to doubt this: my test results don't match up with anything I generate, nor with the CRC16-CCITT row from this page:

https://crccalc.com/

It DOES, however, seem to agree with the results from the CRC-16/GENIBUS row, when I use a 0 initializer. (Why the implementers would choose this particular version of CRC is a mystery to me, but that's another story.)

BUT: after lifting a copy of the GENIBUS algorithm from github and running it on my host computer (not the ESP32), the results don't agree.

So, now I'm thoroughly confused and have no idea what to use. Can someone shed some light on this for me? If necessary, I can do my CRC in software on the ESP, but I'd prefer not to if possible.

Thanks.
Last edited by mzimmers on Mon Sep 24, 2018 4:54 pm, edited 1 time in total.

User avatar
fly135
Posts: 606
Joined: Wed Jan 03, 2018 8:33 pm
Location: Orlando, FL

Re: CRC algorithmin ESP32

Postby fly135 » Fri Sep 21, 2018 11:25 pm

I'm thinking you could use the code in...

E:\Leeo\esp-idf-v3.1\components\wear_levelling\test_wl_host\stubs\esp32\crc.cpp

I'm guessing that this test code is intended to give the same results as the crc in rom, which has no source provided.

BTW, I searched and didn't find a 16 bit version in any test apps.

John A

tommeyers
Posts: 184
Joined: Tue Apr 17, 2018 1:51 pm
Location: Santiago, Dominican Republic

Re: CRC algorithmin ESP32

Postby tommeyers » Fri Sep 21, 2018 11:54 pm

Here is my CRC for serial per maxim

I have the other end in Java.

Code: Select all

/*
    FromArduino
      Get x,y and z aceleration values and write them to the serial port
      where they will be received by the serial monitor and written to disk.
      Given this nodes limted processing power and memory the data 
      will not be processed here.
      Message format:
        #<type of message>,<source node>,<sequence>,<Milliseconds>,<YYMMDD>,<HHMMSS.ssss>,<TimeZone>,<data>,<data>,...
        type of message: 
          start: first output when a node starts (#start,<node>,<sequence>,<milliseconds>,, 
          raw: raw data eg: #raw,<node>,<sequence>,<Miliiseconds>,,,,MeasureID,<data>,<data>,...
          event: event data eg: #event,<node>,<sequence>,<Miliiseconds>,,,,From Row, To Row,MeasureID,<data>,<data>,...
              The event can reference other rows and in this example program the events refer to the raw row that follows.
              it also has the same data as the raw row but that is not the requirement. The from Row and To Row can be empty.
          (so you can see that a node can have multiple type of measures and muultiple types of events.
              these each can have multiple measures.  Later those that have multiple measures will be 
              stored as multiple rows in the db to make access to the data very direct in a SQL way)
          delay: from a server the delay to be used eg: #delay,500
          delayis: from a client in response to a #delay eg. #delayis,01,<sequence>,<Miliseconds>,,,,500 
          comment: for notes eg: #comment,01,<sequence>,<milliseconds>,,,,"I am an accelerometer node"
          nodeexception: exceptions detected  eg. #nodeexception,01,<sequence>,<Mlliseconds>,,,,"java stack captured by catch"
          value: set the value in the client
          valueis: from the client  in response to a #value eg. #valueis,01,<sequence>,<Miliseconds>,,,,name,value
        source node: the sending nodes id
        sequence: start at 0 and incremented when row is sent
        Milliseconds: since start of node
        YYMMDD: when available otherwise 0 length string (just ending comma)
        HHMMSS.ssss: when available otherwise 0 length string (just ending comma)
        TimeZone: for the data time given
        data: quoted text or numbers
        
        examples: #raw,01,1,3000,,,,111,222,333 
                  #comment,01,2,3100,,,,"This is a comment"
*/
#define NODENAME "01" // Identify the name of the node where the data came from 
int sensorXPin = A0;    
int sensorYPin = A1;    
int sensorZPin = A2;    
int sensorXValue = 0;  //Acceleration values [0 .. 1023]
int sensorYValue = 0;  
int sensorZValue = 0;  
unsigned long rowCount = 0;
char request;
String requestString = "";
int delayMS = 0;
String valueName;
String valueValue;
String outputString;
int CRC;
String  CRCString;
char CRCChar[12];
int seismicEventThreshold = 170;
/********************************************************* 
  Send a start row sart rows contain simply NodeName,"#start, node name"
*********************************************************/  
void setup() {
  Serial.begin(9600);
  outputString = "#start,";
    outputString += NODENAME;
    outputString += ",";
    outputString += rowCount;
    outputString += ",";
    outputString += millis();
    outputString += ",,,,";                         //This only looks like it should be ",,,,,"
    itoa(int(CRC8(outputString)),CRCChar,10);  
    CRCString = CRCChar;
    CRCString = "000" + CRCString;
    outputString += CRCString.substring(CRCString.length() - 4);
    outputString += "\n";
  Serial.print(outputString);
  rowCount += 1; 
  outputString = "#comment,";
    outputString += (NODENAME);
    outputString += ",";
    outputString += rowCount;
    outputString += ",";
    outputString += millis();
    outputString += ",,,,";
    outputString += "I am an accelerometer node,";
    itoa(int(CRC8(outputString)),CRCChar,10);  
    CRCString = CRCChar;
    CRCString = "000" + CRCString;
    outputString += CRCString.substring(CRCString.length() - 4);
    outputString += "\n";
  Serial.print(outputString);
  rowCount += 1; 
}
/********************************************************* 
  Start immediately then between sending values measured 
  look fort a #delay #\n command on the serial port 
  to set the delay interval the default interval is 0;
  echo the command so it can be logged at the server
*********************************************************/  
void loop() {
    while (Serial.available()) {
      char inChar = (char)Serial.read();
      if ((requestString.length() == 0) & (inChar == '#')) {
          requestString = "#";
      } else if (requestString.length() > 0 and inChar == '\n') {
        if (requestString.length() > 7) {
          if (requestString.substring(0,7) == "#delay,") {
            delayMS = requestString.substring(7).toInt();
            writeDelayIs();
          } else if  (requestString.substring(0,7) == "#value,") {
            valueName = "";
            valueValue = "";
            int valueNameStart;
            int valueNameEnd;
            int valueValueStart;
            if (requestString.indexOf(",") > 1) {
                if (requestString.indexOf(",",requestString.indexOf(",")) > 1) {
                  valueNameStart = requestString.indexOf(",") + 1;
                  valueNameEnd = requestString.indexOf(",", (valueNameStart + 1 ));
                  if (valueNameEnd > valueNameStart) {
                    valueName = requestString.substring(valueNameStart, valueNameEnd);
                    valueValueStart = requestString.indexOf(",",valueNameEnd) + 1;
                    valueValue =  requestString.substring(valueValueStart);
                    writeValueIs(valueName, valueValue);
                  }
                }   
            }
          }  
          requestString = "";
        }
      } else if (requestString.length() > 0) {
          requestString = requestString + inChar;
          if (requestString.length() > 100) {
            requestString = "";
          }
      }
    
    }
    delay(delayMS); 
    sensorXValue = analogRead(sensorXPin);
    sensorYValue = analogRead(sensorYPin);
    sensorZValue = analogRead(sensorZPin);
    if (sensorXValue > (511 + seismicEventThreshold) 
        or sensorYValue > (511 + seismicEventThreshold)
        or sensorZValue > (511 + seismicEventThreshold)
        or sensorXValue < (511 - seismicEventThreshold) 
        or sensorYValue < (511 - seismicEventThreshold)
        or sensorZValue < (511 - seismicEventThreshold)
        ) {
      writeEvent();
    }
    writeRawValues();
} 
/*
//CRC-8 - based on the CRC8 formulas by Dallas/Maxim
//code released under the the terms of the GNU GPL 3.0 license
//this is the original function that is adapted for this sketch
byte CRC8Original(const byte *data, byte len) {
  byte crc = 0x00;
  while (len--) {
    byte extract = *data++;
    for (byte tempI = 8; tempI; tempI--) {
      byte sum = (crc ^ extract) & 0x01;
      crc >>= 1;
      if (sum) {
        crc ^= 0x8C;
      }
      extract >>= 1;
    }
  }
 return crc;
}
*/
//CRC-8 - based on the CRC8 formulas by Dallas/Maxim
//code released under the the terms of the GNU GPL 3.0 license
byte CRC8(String stringData) {
  int len = stringData.length();
  int i = 0;
  byte crc = 0x00;
  //Serial.print("(0:" + stringData + ")\n");
  while (len--) {
    byte extract;
    extract = (byte) stringData.charAt(i++);
    for (byte tempI = 8; tempI; tempI--) {
      byte sum = (crc ^ extract) & 0x01;
      crc >>= 1;
      if (sum) {
        crc ^= 0x8C;
      }
      extract >>= 1;
    }
  }
  return crc;
}
/********************************************************* 
  Format a raw row and send it to the serial port
  raw rows with more one or more measures contains  "#raws", node name, sample count since program start, 
                  milli seconds since program start,MeasureID, x value, y value, z value                
  terminate the line with a 4 digit CRC8 and a line feed (no comma)
*********************************************************/  
void writeRawValues() {
  outputString = "#raw,";
      outputString +=  NODENAME;
      outputString += ",";
      outputString += rowCount;
      outputString += ",";
      outputString += millis();
      outputString += ",,,,"  ;
      outputString += "Accel";
      outputString += ",";
      outputString += sensorXValue;   
      outputString += ",";
      outputString += sensorYValue; 
      outputString += ",";                        
      outputString += sensorZValue;
      outputString += ","; 
      itoa(int(CRC8(outputString)),CRCChar,10);  
      CRCString = CRCChar;
      CRCString = "000" + CRCString;
      outputString += CRCString.substring(CRCString.length() - 4);
      outputString += "\n";     
  Serial.print(outputString);
  rowCount += 1;
}
/********************************************************* 
  Format an event row and send it to the serial port
  event rows with one or more measures contains:  "#event", node name, sample count since program start, 
    milli seconds since program start,Date, Time, TZ,Event Type,MeasureID, x value, y value, z value                
  terminate the line with a 4 digit CRC8 and a line feed (no comma)
*********************************************************/  
void writeEvent() {
  outputString = "#event,";
      outputString +=  NODENAME;
      outputString += ",";
      outputString += rowCount;
      outputString += ",";
      outputString += millis();
      outputString += ",,,,"  ;
      outputString += "AccEvt";
      outputString += ",";
      outputString += "Accel";
      outputString += ",";
      outputString += sensorXValue;   
      outputString += ",";
      outputString += sensorYValue; 
      outputString += ",";                        
      outputString += sensorZValue;
      outputString += ","; 
      itoa(int(CRC8(outputString)),CRCChar,10);  
      CRCString = CRCChar;
      CRCString = "000" + CRCString;
      outputString += CRCString.substring(CRCString.length() - 4);
      outputString += "\n";     
  Serial.print(outputString);
  rowCount += 1;
}
void writeDelayIs() {
            outputString = "#delayis,";
            outputString += NODENAME;
            outputString += ",";
            outputString += rowCount;
            outputString += ",";
            outputString += millis();
            outputString += ",,,,";
            outputString += delayMS;
            outputString += ",";
            itoa(int(CRC8(outputString)),CRCChar,10);  
            CRCString = CRCChar;
            CRCString = "000" + CRCString;
            outputString += CRCString.substring(CRCString.length() - 4);
            outputString += "\n";
            Serial.print(outputString);
            rowCount += 1; 
}
void writeValueIs(String valueName, String valueValue) {
            outputString = "#valueis,";
            outputString += NODENAME;
            outputString += ",";
            outputString += rowCount;
            outputString += ",";
            outputString += millis();
            outputString += ",,,,";
            outputString += valueName;
            outputString += ",";
            outputString += valueValue;
            outputString += ",";
            itoa(int(CRC8(outputString)),CRCChar,10);  
            CRCString = CRCChar;
            CRCString = "000" + CRCString;
            outputString += CRCString.substring(CRCString.length() - 4);
            outputString += "\n";
            Serial.print(outputString);
            rowCount += 1; 
}
IT Professional, Maker
Santiago, Dominican Republic

tommeyers
Posts: 184
Joined: Tue Apr 17, 2018 1:51 pm
Location: Santiago, Dominican Republic

Re: CRC algorithmin ESP32

Postby tommeyers » Sat Sep 22, 2018 12:43 am

Here is the other end in Java (The while not cancelled is because this ran in another thread):

Code: Select all

while (!isCancelled()) {
			try {
				if (inputStream.available() > 0) {
					int len = inputStream.read(buffer);
					System.out.println("Received(" + socketNumber + "):" + new String(buffer,0,len));
					receivedCharactersCount = receivedCharactersCount + len;
					receivedCharacters.setText(String.valueOf(receivedCharactersCount));
					for (int i = 0; i < len ; i++) {
						if (buffer[i] == '\n') {
							if (receivedLine.substring(0,1).equals("#")) {
								if (FirstLine) {
									try { 
										receivedCharactersCount = 0;
										dateTimeNow = Calendar.getInstance();
										openTime = dateTimeNow.getTime();
										dateTimeNow.add(Calendar.MINUTE, getMaximumFileDuration());
										closeTime = dateTimeNow.getTime();
										file = new File("/Users/Tom Meyers/workspace/Data/" 
										        + dateNowFormat.format(openTime)
										        + " "
										        + timeNowShortFormat.format(openTime)
												+ " RawWriting.csv");
										while (file.exists()) {
											Thread.sleep(1000);
											dateTimeNow = Calendar.getInstance();
											openTime = dateTimeNow.getTime();
											dateTimeNow.add(Calendar.MINUTE, getMaximumFileDuration());
											closeTime = dateTimeNow.getTime();
											file = new File("/Users/Tom Meyers/workspace/Data/" 
											        + dateNowFormat.format(openTime)
											        + " "
											        + timeNowShortFormat.format(openTime)
													+ " RawWriting.csv");
										}
										file.createNewFile();
										fileOutputStream = new FileOutputStream(file);
										dateTimeNow = Calendar.getInstance();
										writeToLog( "#comment,00,,,"
												+ dateNowFormat.format(dateTimeNow.getTime()) + "," 
									            + timeNowFormat.format(dateTimeNow.getTime()) + "," 
									            + "UTC,"
									            + "Opened output file");
										logDataToScreen("#comment,00,,,"
												+ dateNowFormat.format(dateTimeNow.getTime()) + "," 
									            + timeNowFormat.format(dateTimeNow.getTime()) + "," 
									            + "UTC,"
									            + "Opened output file");
										outputFileName.setText(file.getName());
									} catch (Exception e){
										e.printStackTrace();
										dateTimeNow = Calendar.getInstance();
										writeToLog("#exception,00,,,"
												+ dateNowFormat.format(dateTimeNow.getTime()) + "," 
									            + timeNowFormat.format(dateTimeNow.getTime()) + "," 
									            + "UTC,"
									            + "Exception when opening file" + e.getMessage());
										logDataToScreen("#exception,00,,,"
												+ dateNowFormat.format(dateTimeNow.getTime()) + "," 
									            + timeNowFormat.format(dateTimeNow.getTime()) + "," 
									            + "UTC,"
									            + "Exception when opening file" + e.getMessage());
									}
									FirstLine = false;
								} 
								receivedLineSplit = receivedLine.split(",");
								if (receivedLineSplit.length > 0) {
									receivedLineCRC = receivedLineSplit[receivedLineSplit.length - 1];
									receivedLineLessCRC = receivedLine.substring(0, receivedLine.lastIndexOf(',') + 1);	
								}
								else {
									receivedLineCRC = "9999";
									receivedLineLessCRC = "";
								}
								
								if (CRC8(receivedLineLessCRC) != Integer.parseInt(receivedLineCRC)) {
									writeToLog("#exception,00,,,"
											+ dateNowFormat.format(dateTimeNow.getTime()) + "," 
								            + timeNowFormat.format(dateTimeNow.getTime()) + "," 
								            + "UTC,"
								            + "CRC Error: " + receivedLine + "/" + "/" + receivedLineCRC + "/" + CRC8(receivedLineLessCRC));	
									logDataToScreen("#exception,00,,,"
											+ dateNowFormat.format(dateTimeNow.getTime()) + "," 
								            + timeNowFormat.format(dateTimeNow.getTime()) + "," 
								            + "UTC,"
								            + "CRC Error: " + receivedLine+ "/" + CRC8(receivedLineLessCRC));	
								}  
								else {
								  writeToLog(receivedLineLessCRC.substring(0,receivedLineLessCRC.lastIndexOf(",")));
								  logDataToScreen(receivedLine);
								}
								receivedLine = "";
						} else {
							/*The output file is not opened yet - can't write to it
							 * Maybe re-think this in the future; 
							 * right now it opens when it receives the first valid data row 
							 writeToLog( "#exception,00,,,"
									+ dateNowFormat.format(dateTimeNow.getTime()) + "," 
						            + timeNowFormat.format(dateTimeNow.getTime()) + "," 
						            + "UTC,"
						            + "Unrecognized Data Format:" + receivedLine);
						    */
							dateTimeNow = Calendar.getInstance();
							logDataToScreen("#exception,00,,,"
									+ dateNowFormat.format(dateTimeNow.getTime()) + "," 
						            + timeNowFormat.format(dateTimeNow.getTime()) + "," 
						            + "UTC,"
						            + "Unrecognized Data Format: " + receivedLine);
						    receivedLine = "";
						}
					} else if (buffer[i] == '#')  { 
							receivedLine = "";
					}
						receivedLine = receivedLine + new String(buffer,i,1);
					}
				}
				System.out.println("(" + socketNumber + ") Sleep 1000");
				Thread.sleep(1000L);
				
			}
			catch ( IOException e ) {
					e.printStackTrace();
					dateTimeNow = Calendar.getInstance();
					logDataToScreen("#exception,00,,,"
							+ dateNowFormat.format(dateTimeNow.getTime()) + "," 
				            + timeNowFormat.format(dateTimeNow.getTime()) + "," 
				            + "UTC,"
				            + "IO exception when getting tcp data" + e.getMessage());
					break;
			}   
			catch (Exception e) { 
				e.printStackTrace();
				logDataToScreen("#exception,00,,,"
						+ dateNowFormat.format(dateTimeNow.getTime()) + "," 
			            + timeNowFormat.format(dateTimeNow.getTime()) + "," 
			            + "UTC,"
			            + "Exception when getting tcp data" + e.getMessage());
				break;
			}
	 		dateTimeNow = Calendar.getInstance();
			if (dateTimeNow.getTime().after(closeTime)) {
				try {
					FirstLine = true;
					closeTime = null;
					dateTimeNow = Calendar.getInstance();
					logDataToScreen("#comment,00,,,"
							+ dateNowFormat.format(dateTimeNow.getTime()) + "," 
				            + timeNowFormat.format(dateTimeNow.getTime()) + "," 
				            + "UTC,"
				            + "closed output file");
					outputFileName.setText("(none)");
					writeToLog( "#comment,00,,,"
							+ dateNowFormat.format(dateTimeNow.getTime()) + "," 
				            + timeNowFormat.format(dateTimeNow.getTime()) + "," 
				            + "UTC,"
				            + "Closed output file");
					fileOutputStream.flush();
					fileOutputStream.close();
					File newFile = new File("/Users/Tom Meyers/workspace/Data/" 
					        + dateNowFormat.format(openTime)
					        + " "
					        + timeNowShortFormat.format(openTime)
							+ " Raw.csv");
					file.renameTo(newFile);
					Thread.sleep(1000); //assure time change
				} catch (Exception e) {
					e.printStackTrace();
					logDataToScreen("#exception,00,,,"
							+ dateNowFormat.format(dateTimeNow.getTime()) + "," 
				            + timeNowFormat.format(dateTimeNow.getTime()) + "," 
				            + "UTC,"
				            + "Exception when closing/opening" + e.getMessage());
				}
			}
		}	// End of the while not cancelled
IT Professional, Maker
Santiago, Dominican Republic

User avatar
mzimmers
Posts: 643
Joined: Wed Mar 07, 2018 11:54 pm
Location: USA

Re: CRC algorithmin ESP32

Postby mzimmers » Mon Sep 24, 2018 3:23 pm

Thanks for the suggestions, guys. After reading fly's post, I decided that there's no real reason I have to use CRC16, so I converted to CRC32. The built-in CRC32 LE algorithm produces results consistent with the first row in the link I posted above, so I just found a code snippet for my host, and all is well now.

I guess CRC16 isn't in much use these days...I'm still curious as to how the GENIBUS algorithm was chosen for the ESP, but I guess that'll remain a mystery.

Who is online

Users browsing this forum: No registered users and 82 guests