When communication is limited to text, it’s hard to know how your colleagues are really feeling, which can lead to misunderstandings.
"Remote Kun" offers a new way to connect with teammates working from different locations.
Note: "Kun" is a Japanese honorific that addresses someone with familiarity and warmth.
How does this work:
The robot’s face and movements change according to your facial expressions, such as happy or angry, which are detected by your webcam.
If you appear distressed for a certain period, an alert light will turn on.
Gently tapping the robot’s back sends a notification to the remote worker via Slack.
Device Configuration:
Arm movement control
In response to the MQTT message, M5Stack Core outputs a pulse corresponding to the specified angle to the servo motor.
Message='happy': raises (90 degrees) arms three times to express 'happiness'
Message='angry': raises (90 degrees) and lower (-90 degrees) arms three times to express 'anger'
How to make a rotating red light
Materials: four red LEDs, four resistors, a universal board with a diameter of about 20 mm, and a transparent resin cap with a diameter of about 20 mm.
The LEDs and resistors are mounted on the circuit board. The LED legs are bent 90 degrees so they face horizontally.
The resin cap is then placed over the circuit board.
M5Stack Core turns on the LEDs in sequence every about 100 milliseconds. It looks like they are rotating.
Touch sensor modulemodification
TTP223 The touch sensor module has a small detection area, so it is connected to an aluminum plate about 10 mm wide and 50 mm long, and a capacitor for adjusting sensitivity is added, the detection area is expanded, making it easier to touch.
System Architecture:
Overall Summary
This system is configured to control a robot (RemoteKun) based on emotion data sent from a PC, while also detecting reactions from a touch sensor on the robot and sending notifications to Slack.
Role of Each Component and Data Flow
PC (Client-side):- An application installed on the PC uses the facial analysis library DeepFace to estimate emotions and generate the results as an Emotion Message.- This data is transmitted to the AWS cloud using the HTTP protocol.
API Gateway:- It functions as the entry point for HTTP requests (emotion data) sent from the PC.- The API Gateway passes the received requests to the first AWS Lambda Function.
Lambda Function (1st):- This is a serverless function that processes the emotion data passed from the API Gateway.- Based on the received emotion data, this function sends control commands to the robot (RemoteKun) to AWS IoT.
AWS IoT:- This service manages communication with IoT devices (in this case, RemoteKun).- It sends messages received from the Lambda Function to RemoteKun via the MQTT protocol.- When the touch sensor on RemoteKun reacts, it transmits that information to AWS IoT.
Lambda Function (2nd):- The touch sensor on RemoteKun reacts, and that information is passed to this Lambda Function via AWS IoT.- This function is responsible for sending a notification message to Slack based on the received sensor reaction.
RemoteKun (Robot):- It receives the emotion data sent via MQTT from AWS IoT and executes actions based on that data.- When the touch sensor reacts, it transmits that information to AWS IoT.
Slack:- It receives messages sent from the second Lambda Function and displays and records the robot's touch events in a chat.
Creating the body:
The almost parts of "Remote Kun" were printed with 3D printer.
All parts made with 3D printerThe list of parts made with 3D printer.・Head (with back cover)・Body (with front/back cover)・Arm (right and left)・Leg (×2)To conceal the wiring, both the head and the body have wire holes that allow the cables to pass through.The back cover of the body has a cutout for a USB connector that supplies power, as well as for mounting the touch sensor on the back.
#include<M5Unified.h>#include<WiFi.h>#include"remotekun-arm.h"#include"remotekun-communication.h"#include"remotekun-face.h"#include"remotekun-patlump.h"#include"remotekun-touch.h"// constant ---------------------------------------// defined by .env#ifndef WIFI_SSID#define WIFI_SSID "WIFI_SSID_UNDEFINED"#endif#ifndef WIFI_PASSWORD#define WIFI_PASSWORD "WIFI_PASSWORD_UNDEFINED"#endif#ifndef AWS_IOT_ENDPOINT#define AWS_IOT_ENDPOINT "AWS_IOT_ENDPOINT_UNDEFINED"#endif#ifndef DEVICE_NAME#define DEVICE_NAME "DEVICE_NAME_UNDEFINED"#endifconstuint8_tPIN_ARM_L=5;// left armconstuint8_tPIN_ARM_R=19;// right armconstuint8_tPIN_TOUCH=15;// touch sensorconstuint8_tPIN_LED0=16;// patrol lampconstuint8_tPIN_LED1=17;constuint8_tPIN_LED2=21;constuint8_tPIN_LED3=22;constunsignedlongMAIN_LOOP_WAIT_TIME_MS=10;// main loop wait [msec]constunsignedlongAUTO_TURNOFF_INTERVAL=10*1000;// patrol lamp auto off timeconstunsignedlongMESSAGE_DISABLE_INTERVAL=6*1000;// emotion continuation timeconstunsignedlongEMOTION_EXPLOSION_TIME=5*60*1000;// 5 * 60 * 1000; // dissatisfaction explosion time = standard 5 minutes [msec]// global objects --------------------------------------RemoteKunArmarm(PIN_ARM_L,PIN_ARM_R);RemoteKunCommunicationcommunication;RemoteKunFaceface;RemoteKunPatlumppatlump(PIN_LED0,PIN_LED1,PIN_LED2,PIN_LED3);RemoteKunTouchremoteKunTouch(PIN_TOUCH);// callbacks --------------------------------------// MQTT topic reception processingvoidonEmotionMessage(constString&payload,constsize_tsize){// message analysischarbuf[256];payload.toCharArray(buf,256);JsonDocumentrecievedJson;DeserializationErrorerror=deserializeJson(recievedJson,buf);if(error){Serial.println("onEmotionMessage Deserialization failed!");return;}// ignore message that is received multiple times within a certain periodif(!communication.isMessageEnabled())return;// face, armsif(strcmp(recievedJson["emotion"],"angry")==0){if(face.emotion==Expression::Angry)return;face.angry();arm.angry();communication.updateLastMassageRecievedTime();}elseif(strcmp(recievedJson["emotion"],"disgusted")==0){if(face.emotion==Expression::Angry)return;face.angry();arm.angry();communication.updateLastMassageRecievedTime();}elseif(strcmp(recievedJson["emotion"],"happy")==0){if(face.emotion==Expression::Happy)return;face.happy();arm.happy();communication.updateLastMassageRecievedTime();}else{if(face.emotion==Expression::Neutral)return;face.neutral();arm.neutral();communication.updateLastMassageRecievedTime();}}// touch detectionvoidIRAM_ATTRonTouchDetected(){if(!remoteKunTouch.isTouchEnabled())return;communication.requestPublish();remoteKunTouch.updateLastTouchedTime();face.happy();face.speech("Thank you!");patlump.off();}// emotion explosionvoidonEmotionExplosion(){patlump.on();Serial.println("emotion explosion!");}voidsetup(){Serial.begin(115200);M5.begin();// Wi-FiSerial.println(WIFI_SSID);Serial.println(WIFI_PASSWORD);Serial.println("Connecting to WiFi");M5.Lcd.printf("Connecting to WiFi(%s)",WIFI_SSID);WiFi.begin(WIFI_SSID,WIFI_PASSWORD);while(WiFi.status()!=WL_CONNECTED){delay(500);M5.Lcd.print('.');}Serial.println("Succeessfully connected to WiFi");M5.Lcd.printf("Succeessfully connected to WiFi");// armsarm.begin();arm.neutral();// MQTTcommunication.begin(AWS_IOT_ENDPOINT,8883,"RemoteKun-Device1");Serial.println("Connecting to AWS IoT...");M5.Lcd.printf("Connecting to AWS IoT...");boolconnectionResult=communication.connect();M5.Lcd.printf("MQTT Connection Result: %d\n",connectionResult);Serial.printf("MQTT Connection Result: %d\n",connectionResult);communication.onEmotionMessage=onEmotionMessage;communication.messageDisableInterval=MESSAGE_DISABLE_INTERVAL;communication.subscribe();// faceface.onEmotionExplosion=onEmotionExplosion;face.emotionPeriod=MESSAGE_DISABLE_INTERVAL;face.emotionExplosionTime=EMOTION_EXPLOSION_TIME;face.autoRecovery=false;face.begin();// touchremoteKunTouch.begin(onTouchDetected);// patlumppatlump.activeLevel=LOW;patlump.inactiveLevel=HIGH;patlump.autoTurnOffInterval=AUTO_TURNOFF_INTERVAL;patlump.begin();}voidloop(){// MQTTcommunication.action();// armsarm.action();// faceface.action();// patlumppatlump.action();// adjustmentdelay(MAIN_LOOP_WAIT_TIME_MS);}
M5Stack Program(arm)
Arduino
Arm control module(PlatformIO)
/*arm control- SG90- 50Hz -> 20msec- -90deg: 0.5msec = 6.4 / 256- 0deg: 1.45msec = 18.56 / 256- 90deg: 2.4msec = 30.72 / 256action(): Queues are executed sequentially according to their scheduled execution timesWhen methods executed, the action is added to the queue.angry()disgusted()happy()*/#include<M5Unified.h>classRemoteKunArm{public:staticconstuint8_tMAX_ACTIONS=16;doublePWM_FREQUENCY=50;doublePWM_RESOLUTION=8;uint8_tRIGHT_ARM_CHANNEL=0;uint8_tLEFT_ARM_CHANNEL=1;RemoteKunArm(uint8_tpin_right_arm,uint8_tpin_left_arm){_pin_right_arm=pin_right_arm;_pin_left_arm=pin_left_arm;_pwm_resolution_pulse=pow(2,PWM_RESOLUTION);}voidbegin(){ledcSetup(0,PWM_FREQUENCY,PWM_RESOLUTION);ledcSetup(1,PWM_FREQUENCY,PWM_RESOLUTION);ledcAttachPin(_pin_right_arm,RIGHT_ARM_CHANNEL);ledcAttachPin(_pin_left_arm,LEFT_ARM_CHANNEL);}voidangry(){unsignedlongnow=millis();uint8_taction_num=5;intangles[action_num]={90,-90,90,-90,0};for(uint8_ti=0;i<action_num;i++){_right_arm_action_queue[i]=angles[i];_left_arm_action_queue[i]=-angles[i];_action_timeout_queue[i]=now+i*500;}_action_index=0;_action_count=action_num;}voidhappy(){unsignedlongt=millis()-800;uint8_taction_num=6;intangles[action_num]={90,0,90,0,90,0};for(uint8_ti=0;i<action_num;i++){_right_arm_action_queue[i]=angles[i];_left_arm_action_queue[i]=-angles[i];if(i%2==0){t+=800;}else{t+=400;}_action_timeout_queue[i]=t;}_action_index=0;_action_count=action_num;}voiddisgusted(){angry();}voidarbitrary(intright_arm_angless[],intleft_arm_angles[],inttimeouts[],intaction_num){unsignedlongnow=millis();for(uint8_ti=0;i<action_num;i++){_right_arm_action_queue[i]=right_arm_angless[i];_left_arm_action_queue[i]=left_arm_angles[i];_action_timeout_queue[i]=now+i*timeouts[i];}_action_index=0;_action_count=action_num;}voidneutral(){unsignedlongnow=millis();_right_arm_action_queue[0]=0;_left_arm_action_queue[0]=0;_action_timeout_queue[0]=now;_action_index=0;_action_count=1;}voidaction(){if(_action_index>=_action_count){return;}if(millis()>=_action_timeout_queue[_action_index]){ledcWrite(RIGHT_ARM_CHANNEL,_angle2pulse(_right_arm_action_queue[_action_index]));ledcWrite(LEFT_ARM_CHANNEL,_angle2pulse(_left_arm_action_queue[_action_index]));_action_index++;}}private:uint8_t_pin_right_arm=19;uint8_t_pin_left_arm=26;uint32_t_pwm_resolution_pulse=256;uint32_t_right_arm_action_queue[MAX_ACTIONS]={0};// right arm angle queueuint32_t_left_arm_action_queue[MAX_ACTIONS]={0};// left arm angle queueuint32_t_action_timeout_queue[MAX_ACTIONS]={0};// scheduled execution time queueuint8_t_action_index=0;// queue indexuint8_t_action_count=0;// queue length// convert angle [deg] to pulse width// (1.9 / 180.0 * (angle + 90) + 0.5) * _pwm_resolution_pulseuint32_t_angle2pulse(intangle){return(uint32_t)((0.0105555555*(angle+90)+0.5)/20.0*_pwm_resolution_pulse);}};
M5Stack Program(communication)
Arduino
MQTT communication with AWS IoT(PlatformIO)
/*MQTT communication with AWS IoTDefine the content of the environment variables by concatenating all lines except for the BEGIN CERTIFICATE and END CERTIFICATE lines into a single lineAWS_CERT_CA_CONTENTAWS_CERT_PRIVATE_CONTENTAWS_CERT_CRT_CONTENTTopics/RemoteKun/worker/{device_name}/emotion/RemoteKun/device/{device_name}/slack*/#include<WiFiClientSecure.h>#include<MQTTPubSubClient.h>#include<ArduinoJson.h>// ---------------------------------------#ifndef AWS_CERT_CA_CONTENT#define AWS_CERT_CA_CONTENT "AWS_CERT_CA_UNDEFINED"#endif#ifndef AWS_CERT_PRIVATE_CONTENT#define AWS_CERT_PRIVATE_CONTENT "AWS_CERT_PRIVATE_UNDEFINED"#endif#ifndef AWS_CERT_CRT_CONTENT#define AWS_CERT_CRT_CONTENT "AWS_CERT_CRT_UNDEFINED"#endifconstcharAWS_CERT_CA[]="-----BEGIN CERTIFICATE-----\n"AWS_CERT_CA_CONTENT"-----END CERTIFICATE-----\n";constcharAWS_CERT_PRIVATE[]="-----BEGIN RSA PRIVATE KEY-----\n"AWS_CERT_PRIVATE_CONTENT"-----END RSA PRIVATE KEY-----\n";constcharAWS_CERT_CRT[]="-----BEGIN CERTIFICATE-----\n"AWS_CERT_CRT_CONTENT"-----END CERTIFICATE-----\n";classRemoteKunCommunication{public:unsignedinttimeout=3;unsignedlongmessageDisableInterval=10000;// Message reception disable period [msec]WiFiClientSecureclient;// SSL clientMQTTPubSubClientmqtt;// MQTT clientstd::function<void(constString&payload,constsize_tsize)>onEmotionMessage;voidbegin(constchar*host,uint16_tport,constchar*device_name){_host=host;_port=port;_device_name=device_name;mqtt.begin(client);}boolconnect(){// Connect to AWS IoT endpointSerial.print("connecting to host...");client.stop();// connect to aws endpoint with certificates and keysclient.setCACert(AWS_CERT_CA);client.setCertificate(AWS_CERT_CRT);client.setPrivateKey(AWS_CERT_PRIVATE);while(!client.connect(_host,_port)){Serial.print(".");delay(1000);}Serial.println(" OK!");// Connect to AWS MQTT brokerSerial.print("connecting to aws mqtt broker...");mqtt.disconnect();unsignedintt=0;while(!mqtt.connect("RemoteKun-Device1")){Serial.print(".");delay(1000);t++;if(t>timeout){Serial.println("timeout");returnfalse;}mqtt.update();}Serial.println(" OK!");returntrue;}voidsubscribe(){charTOPIC_ACTION[64]="";sprintf(TOPIC_ACTION,"/RemoteKun/worker/%s/emotion",_device_name);mqtt.subscribe(TOPIC_ACTION,onEmotionMessage);}// MQTT publish requestvoidrequestPublish(){_publishRequest=true;}voidaction(){mqtt.update();if(!_publishRequest)return;publish();}// Publish// Do not call directly from interrupt handlers (doing so may cause WDT timeout and restart)voidpublish(){_publishRequest=false;JsonDocumentdoc;doc["message"]="hello from RemoteKun!";charbuffer[128];serializeJson(doc,buffer);charTOPIC_SLACK[128]="";sprintf(TOPIC_SLACK,"/RemoteKun/device/%s/slack",_device_name);mqtt.publish(TOPIC_SLACK,buffer);}voidupdateLastMassageRecievedTime(){_lastRecievedTime=millis();}boolisMessageEnabled(){unsignedlongnow=millis();if(now-_lastRecievedTime>messageDisableInterval){returntrue;}else{Serial.printf("Message DISable\n");// returnfalse;}}private:constchar*_host;uint16_t_port=8883;constchar*_device_name;bool_publishRequest=false;unsignedlong_lastRecievedTime=0;};
/*Rotating red lights control*/classRemoteKunPatlump{public:uint8_tactiveLevel=HIGH;uint8_tinactiveLevel=LOW;unsignedlonginterval=75;// LED switching period [msec]unsignedlongautoTurnOffInterval=10000;// Auto-off time [msec]RemoteKunPatlump(uint8_tpin_led0,uint8_tpin_led1,uint8_tpin_led2,uint8_tpin_led3){_pin_leds[0]=pin_led0;_pin_leds[1]=pin_led1;_pin_leds[2]=pin_led2;_pin_leds[3]=pin_led3;}voidbegin(){pinMode(_pin_leds[0],OUTPUT);pinMode(_pin_leds[1],OUTPUT);pinMode(_pin_leds[2],OUTPUT);pinMode(_pin_leds[3],OUTPUT);action();}voidoff(){_on=false;}voidon(){_on=true;_lastTurnOnTime=millis();}voidaction(){unsignedlongnow=millis();// Wait for switching periodif(now<_lastSwitchedTime+interval)return;// Auto-off after a certain periodif(now>_lastTurnOnTime+autoTurnOffInterval){off();}for(intk=0;k<4;k++){if(k==_flashIndex){if(_on){digitalWrite(_pin_leds[k],activeLevel);}else{digitalWrite(_pin_leds[k],inactiveLevel);}}else{digitalWrite(_pin_leds[k],inactiveLevel);}}_lastSwitchedTime=now;_flashIndex=(_flashIndex+1)%4;}private:uint8_t_pin_leds[4]={0,1,2,3};unsignedint_flashIndex=0;unsignedlong_lastSwitchedTime=0;unsignedlong_lastTurnOnTime=0;bool_on=false;};
Comments