Hardware components | ||||||
| × | 1 | ||||
| × | 1 | ||||
 |
| × | 1 | |||
Software apps and online services | ||||||
 |
| |||||
In this lesson, we’ll begin connecting your BallyBot’s camera to projects, letting you stream live video over WiFi. This will take your robot to the next level, allowing it to perceive and interact with its environment.
Prerequisites
- Complete Lesson 6-7 (WiFi Hosting on BallyBot)
- Basic understanding of video
Before going too deep into anything else, it is good to know the hardware you are working with. For this project, we are using the OV2640 camera module which is already integrated into your BallyBot's microcontroller board. There are various settings unique to this module we will see in later code.
Adding #include's is a simple but a vital part of your code as majority of the functions we use are being pulled from these libraries downloaded into your IDE.
#include "esp_camera.h"
#include <WiFi.h> /* this one should be familiar! */
#include <WebServer.h> /* this one should be familiar! */
#include <ArduinoWebsockets.h>With our libraries in place, let's start by recreating the familiar set up of a web server:
const char* ssid = "BallyBot_AP";
const char* password = "12345678";
void setup() {
Serial.begin(115200);
WiFi.mode(WIFI_AP);
WiFi.softAP(ssid, password);
server.on("/", handle_OnConnect);
server.begin();
}
void loop() {
server.handleClient();
}
void handle_OnConnect() {
Serial.println("a client connected to website url");
server.send(200, "text/html", SendHTML());
}
String SendHTML(){
const char index_html[] PROGMEM = R"rawliteral(
<!DOCTYPE HTML>
<html>
This is the basic set up by now
</html>
)rawliteral";
return index_html;
}The next step is to import/paste in the settings for your built-in camera. These are created by the official Chip maker and it is generally expect to paste this from elsewhere:
/* place After your includes and before setup() */
#define CAMERA_MODEL_AI_THINKER
#define PWDN_GPIO_NUM 32
#define RESET_GPIO_NUM -1
#define XCLK_GPIO_NUM 0
#define SIOD_GPIO_NUM 26
#define SIOC_GPIO_NUM 27
#define Y9_GPIO_NUM 35
#define Y8_GPIO_NUM 34
#define Y7_GPIO_NUM 39
#define Y6_GPIO_NUM 36
#define Y5_GPIO_NUM 21
#define Y4_GPIO_NUM 19
#define Y3_GPIO_NUM 18
#define Y2_GPIO_NUM 5
#define VSYNC_GPIO_NUM 25
#define HREF_GPIO_NUM 23
#define PCLK_GPIO_NUM 22Then we will need a function that activates these settings which looks like this:
void CameraSetup(){
camera_config_t config;
config.ledc_channel = LEDC_CHANNEL_0;
config.ledc_timer = LEDC_TIMER_0;
config.pin_d0 = Y2_GPIO_NUM;
config.pin_d1 = Y3_GPIO_NUM;
config.pin_d2 = Y4_GPIO_NUM;
config.pin_d3 = Y5_GPIO_NUM;
config.pin_d4 = Y6_GPIO_NUM;
config.pin_d5 = Y7_GPIO_NUM;
config.pin_d6 = Y8_GPIO_NUM;
config.pin_d7 = Y9_GPIO_NUM;
config.pin_xclk = XCLK_GPIO_NUM;
config.pin_pclk = PCLK_GPIO_NUM;
config.pin_vsync = VSYNC_GPIO_NUM;
config.pin_href = HREF_GPIO_NUM;
config.pin_sscb_sda = SIOD_GPIO_NUM;
config.pin_sscb_scl = SIOC_GPIO_NUM;
config.pin_pwdn = PWDN_GPIO_NUM;
config.pin_reset = RESET_GPIO_NUM;
config.xclk_freq_hz = 10000000;
config.pixel_format = PIXFORMAT_JPEG;
//init with high specs to pre-allocate larger buffers
if(psramFound()){
config.frame_size = FRAMESIZE_VGA;
config.jpeg_quality = 40;
config.fb_count = 2;
} else {
config.frame_size = FRAMESIZE_SVGA;
config.jpeg_quality = 12;
config.fb_count = 1;
}
// camera init
esp_err_t err = esp_camera_init(&config);
if (err != ESP_OK) {
Serial.printf("Camera init failed with error 0x%x", err);
return;
}
}NOTE: You will also need to actually call this CameraSetup() function by adding it near the top of your Setup() function.
Up till now we have been working with a server, called an HTTP server, that can host a website and have been creating it with commands like server.on("/", handle_OnConnect). Now we have the need for another type of server called a web-socket server. The web-socket server will be used in order stream video from the robot to our website.
We start by adding these variables for the web-socket server:
/* WebSocket variables */
using namespace websockets;
WebsocketsServer wserver;
WebsocketsClient wclient;
bool ws_client_connected = false;
const uint16_t ws_server_port = 65080;Next in order to set up the web-socket server we are going to have to shift the HTTP server's server.handleClient(); command into setup() so they do not interfere with each other.
We will move:
void loop() {
server.handleClient();
}into the Setup function with a while loop like this:
void setup() {
Serial.begin(115200);
Serial.setDebugOutput(true);
CameraSetup();
WiFi.mode(WIFI_AP);
WiFi.softAP(ssid, password);
server.on("/", handle_OnConnect);
server.begin();
while(!ws_client_connected){
server.handleClient();
}
}Next will put the web-socket configuration commands into the setup() function.
place before the while() loop:
wserver.listen(ws_server_port);and after the while() loop:
// WebsocketsServer.accept() will delay everything until recieving a connection
wclient = wserver.accept();
Serial.print("wclient: ");
Serial.println(wclient.available());
Serial.println("Websocket Connected!");
Serial.print(WiFi.localIP());
} /* end of setup() */The web-socket sever lastly, will need code for transmitting the framebuffer(fb) of video to the connected client:
void loop() {
camera_fb_t *fb = esp_camera_fb_get();
if(!fb){
Serial.println("Camera capture failed");
esp_camera_fb_return(fb);
return;
}
if(fb->format != PIXFORMAT_JPEG){
Serial.println("Non-JPEG data not implemented");
return;
}
/* v core command v */
wclient.sendBinary((const char*) fb->buf, fb->len);
esp_camera_fb_return(fb);
}Now we have successfully built both the HTTP and web-socket server, all that is left is the web-socket client. In this case the HTTP server hosts a website, and that website will be the web-socket client connecting back to the BallyBot's web-socket server.
All we have to do is place this in for the HTTP server's website HTML:
String SendHTML(){
const char index_html[] PROGMEM = R"rawliteral(
<!DOCTYPE HTML>
<html><head>
<title>ESP Input Form</title>
<meta name="viewport" content="width=device-width, initial-scale=1">
</head>
<body>
Hello Client
<img id="stream" src="PlaceHolder.jpg">
<script>
const WS_IP = '192.168.4.1'
const WS_PORT = '65080'
const img = document.getElementById("stream")
const WS_URL = `ws:///${WS_IP}:${WS_PORT}`;
const ws = new WebSocket(WS_URL);
let urlObject;
ws.onopen = () => {
};
ws.onclose = () => {
};
ws.onmessage = message => {
const arrayBuffer = message.data;
if(urlObject){
URL.revokeObjectURL(urlObject);
}
urlObject = URL.createObjectURL(new Blob([arrayBuffer]));
img.src = urlObject;
}
</script>
</body></html>
)rawliteral";
return index_html;
}You might notice in HTML it uses the BallyBots IP address of 192.168.4.1 and The same port we specify in the web-socket server variables "65080". These are the 2 pieces of info the web-socket client needs in order to create the socket connection.
To finally stream video, you'll upload the code and then connect the the hotspot created by the Ballybot. Then you will open any internet browser and go to the URL of 192.168.4.1 once there you will see what should be a simple website with you video stream!
Having an understanding of these 4 pieces being Camera,HTTPserver,web-socket server and web-socket client, you now can learn to use the camera in combination with other projects! There are still a few minor details to know more on like sending info from client to server via web-socket but you're now covered on the core concepts of how this works.
Here is the full code of this lesson for reference in future projects:
Example Code: Full Code for The BallyBot with Camera
C++
#include "esp_camera.h"
#include <WiFi.h>
#include <WebServer.h>
#include <ArduinoWebsockets.h>
#define CAMERA_MODEL_AI_THINKER
#define PWDN_GPIO_NUM 32
#define RESET_GPIO_NUM -1
#define XCLK_GPIO_NUM 0
#define SIOD_GPIO_NUM 26
#define SIOC_GPIO_NUM 27
#define Y9_GPIO_NUM 35
#define Y8_GPIO_NUM 34
#define Y7_GPIO_NUM 39
#define Y6_GPIO_NUM 36
#define Y5_GPIO_NUM 21
#define Y4_GPIO_NUM 19
#define Y3_GPIO_NUM 18
#define Y2_GPIO_NUM 5
#define VSYNC_GPIO_NUM 25
#define HREF_GPIO_NUM 23
#define PCLK_GPIO_NUM 22
/* Settings */
const char* ssid = "BallyBot_AP";
const char* password = "12345678";
/* WebSocket variables */
using namespace websockets;
WebsocketsServer wserver;
WebsocketsClient wclient;
bool ws_client_connected = false;
const uint16_t ws_server_port = 65080;
/* Normal Server */
WebServer server(80);
void setup() {
Serial.begin(115200);
Serial.setDebugOutput(true);
Serial.println("hello world");
CameraSetup();
WiFi.mode(WIFI_AP);
WiFi.softAP(ssid, password);
Serial.print("AP IP address: "); Serial.println(WiFi.softAPIP());
wserver.listen(ws_server_port);
server.on("/", handle_OnConnect);
server.begin();
while(!ws_client_connected){
server.handleClient();
}
/* WebsocketsServer.accept() will delay everything until recieving a connection */
wclient = wserver.accept();
Serial.print("wclient: ");
Serial.println(wclient.available());
Serial.println("Websocket Connected!");
Serial.print(WiFi.localIP());
}
void loop() {
camera_fb_t *fb = esp_camera_fb_get();
if(!fb){
Serial.println("Camera capture failed");
esp_camera_fb_return(fb);
return;
}
if(fb->format != PIXFORMAT_JPEG){
Serial.println("Non-JPEG data not implemented");
return;
}
wclient.sendBinary((const char*) fb->buf, fb->len);
esp_camera_fb_return(fb);
}
void handle_OnConnect() {
Serial.println("a client connected to website url");
server.send(200, "text/html", SendHTML());
ws_client_connected = true;
}
String SendHTML(){
const char index_html[] PROGMEM = R"rawliteral(
<!DOCTYPE HTML>
<html><head>
<title>ESP Input Form</title>
<meta name="viewport" content="width=device-width, initial-scale=1">
</head>
<body>
Hello Client
<img id="stream" src="PlaceHolder.jpg">
<script>
const WS_IP = '192.168.4.1'
const WS_PORT = '65080'
const img = document.getElementById("stream")
const WS_URL = `ws:///${WS_IP}:${WS_PORT}`;
const ws = new WebSocket(WS_URL);
let urlObject;
ws.onopen = () => {
};
ws.onclose = () => {
};
ws.onmessage = message => {
const arrayBuffer = message.data;
if(urlObject){
URL.revokeObjectURL(urlObject);
}
urlObject = URL.createObjectURL(new Blob([arrayBuffer]));
img.src = urlObject;
}
</script>
</body></html>
)rawliteral";
return index_html;
}
void CameraSetup(){
camera_config_t config;
config.ledc_channel = LEDC_CHANNEL_0;
config.ledc_timer = LEDC_TIMER_0;
config.pin_d0 = Y2_GPIO_NUM;
config.pin_d1 = Y3_GPIO_NUM;
config.pin_d2 = Y4_GPIO_NUM;
config.pin_d3 = Y5_GPIO_NUM;
config.pin_d4 = Y6_GPIO_NUM;
config.pin_d5 = Y7_GPIO_NUM;
config.pin_d6 = Y8_GPIO_NUM;
config.pin_d7 = Y9_GPIO_NUM;
config.pin_xclk = XCLK_GPIO_NUM;
config.pin_pclk = PCLK_GPIO_NUM;
config.pin_vsync = VSYNC_GPIO_NUM;
config.pin_href = HREF_GPIO_NUM;
config.pin_sscb_sda = SIOD_GPIO_NUM;
config.pin_sscb_scl = SIOC_GPIO_NUM;
config.pin_pwdn = PWDN_GPIO_NUM;
config.pin_reset = RESET_GPIO_NUM;
config.xclk_freq_hz = 10000000;
config.pixel_format = PIXFORMAT_JPEG;
/* init with high specs to pre-allocate larger buffers */
if(psramFound()){
config.frame_size = FRAMESIZE_VGA;
config.jpeg_quality = 40;
config.fb_count = 2;
} else {
config.frame_size = FRAMESIZE_SVGA;
config.jpeg_quality = 12;
config.fb_count = 1;
}
/* camera init */
esp_err_t err = esp_camera_init(&config);
if (err != ESP_OK) {
Serial.printf("Camera init failed with error 0x%x", err);
return;
}
}You've successfully integrated a camera into your BallyBot project, enabling real-time video streaming and object detection. This opens up new possibilities for robotics projects, such as surveillance, tracking, and more.
What's Next?
In the next lesson, we'll explore object detection using machine learning models with the camera module.
ResourcesPrevious Lesson:Lesson 6: Customizing websites for your access point
Next Lesson:Lesson 8: Customizing websites for your access point
Comments