Community project

Talking Command Robot

Shourya Vijayvergia

Published July 18, 2026

ESP328 components10 assembly steps
Remix this project
Photo of Talking Command Robot

This talking command robot combines voice recognition, AI conversation, and physical movement into an interactive ESP32-S3 project. Press the button to record a voice command, which is transcribed by Groq Whisper, processed by Groq LLaMA for intent and servo angles, then spoken back via ElevenLabs TTS while the pan-tilt head animates to match the response.

The guide provides a complete wiring diagram connecting the INMP441 I²S microphone, MAX98357A amplifier and speaker, SSD1306 OLED face display, SG90 pan-tilt servos, and push-to-talk button to the ESP32-S3. You'll get a full parts list, step-by-step assembly instructions for the mechanical head, and ready-to-flash firmware that handles audio I/O, API calls, and servo control—just add your Groq and ElevenLabs credentials.

Wiring diagram

Interactive · read-only
Wiring diagram for Talking Command Robot

Pan and zoom to explore the wiring. Remix the project to edit it in your own workspace.

Parts list

Bill of materials
ComponentQtyNotes
HiLetgo INMP441 I²S Microphone Module1Omnidirectional 24-bit I²S MEMS microphone module based on the TDK InvenSense INMP441 IC. Outputs digital audio over a 3-wire I²S bus (SCK/BCLK, WS/LRCLK, SD/DOUT). Supply 1.8–3.3 V; native 3.3 V operation with no level shifting required. L/R channel select pin: tie to GND for left-channel mono output or VDD for right-channel mono output. Use the ESP32 Arduino core or ESP-IDF I2S peripheral API; no separate PlatformIO library is required.
MAX98357A I2S Class-D Mono Amplifier Breakout1I2S-input Class-D mono audio amplifier IC on a compact breakout board. Accepts I2S digital audio input (BCLK, LRC, DIN) and drives a small speaker or transducer directly. No I2C/SPI control bus is needed. The amplifier supply range is 2.5V-5.5V, and the I2S input pins are compatible with 3.3V logic. SD/MODE controls shutdown and channel selection; GAIN selects 3 dB, 6 dB, 9 dB, 12 dB, or 15 dB gain.
SSD1306 OLED10.96 inch 128x64 OLED display with I2C interface
SG90 Servo1Micro servo motor (SG90)
SG90 Servo1Micro servo motor (SG90)
8Ω Speaker1Generic small 8Ω 0.5-3W loudspeaker (~28mm typical). Pair with an I2S amp (MAX98357A) or class-D amp (TPA3116D2) for usable volume; do not drive directly from a GPIO pin. Audio output for music/voice playback.
Push Button1Momentary push button switch
BSS138 Logic Level Shifter1Common BSS138 MOSFET-based 4-channel bidirectional logic level converter module. Good for I2C, UART, and slow digital signals between 3.3V and 5V domains; not a strong high-speed SPI/LED data buffer.

Assembly

10 steps
  1. Power the ESP32-S3 via USB

    Connect the ESP32-S3 DevKitC-1 to your laptop using a USB-C cable. The board gets 5V from the laptop USB port — no battery or external supply needed. The onboard 3.3V LDO generates the 3.3V rail used by the mic and OLED automatically.

    • Tip: Use a data-capable USB-C cable (not a charge-only cable) so Schematik can flash firmware.
    • Tip: The board's 5V pin is available for powering servos and the MAX98357A directly from the USB 5V rail.
    • The ESP32-S3 USB port supplies ~500mA total. Two SG90 servos can draw up to 400mA stall current combined — avoid stalling both servos at the same time while powered from laptop USB.
  2. Wire the INMP441 microphone

    Connect INMP441 VDD → 3.3V pin on ESP32, GND → GND, SCK → GPIO40, WS → GPIO41, SD → GPIO42. Tie the L/R pin to GND (selects left-channel mono output). Add a 100nF decoupling capacitor between VDD and GND as close to the module as possible to reduce noise.

    • Tip: The INMP441 is directional toward the top port — mount it facing forward on the robot head.
    • Tip: L/R pin must be tied to GND — leaving it floating causes silence or garbled audio.
  3. Wire the MAX98357A amplifier and speaker

    Connect MAX98357A VIN → 5V pin on ESP32, GND → GND, BCLK → GPIO14, LRC → GPIO21, DIN → GPIO47. Leave the SD and GAIN pins floating (default 15dB gain). Connect SPK+ and SPK- directly to the 8Ω speaker terminals. Polarity does not matter for mono audio.

    • Do not short SPK+ and SPK- together — this will damage the amplifier IC.
    • Power the MAX98357A from the 5V pin, not the 3.3V pin — it needs 5V for full output power.
  4. Wire the SSD1306 OLED display

    Connect OLED VCC → 3.3V, GND → GND, SDA → GPIO8, SCL → GPIO9. The default I2C address is 0x3C (some modules use 0x3D — check the back of the PCB).

    • Tip: If the display stays blank after flashing, run a quick I2C scanner sketch to confirm the address.
    • Tip: Keep I2C wires short (under 20cm) to avoid signal issues at 400kHz.
  5. Wire the logic level shifter and pan servo

    Connect BSS138 LLC: LV → 3.3V, HV → 5V, GND → GND. Route GPIO38 → LV1 on the shifter, then HV1 → pan servo SIGNAL wire (orange/yellow). Connect servo_pan VCC (red) → 5V pin on ESP32, GND (brown/black) → GND.

    • Tip: Servo wire colors: Brown = GND, Red = VCC, Orange = Signal.
    • Tip: The level shifter converts 3.3V PWM from ESP32 to 5V PWM for the servo — essential for reliable movement.
    • Always power servo VCC from the 5V rail, never from the 3.3V pin — servos draw too much current for the 3.3V LDO.
  6. Wire the tilt servo

    Connect servo_tilt SIGNAL (orange) → GPIO39 directly. VCC (red) → 5V pin on ESP32, GND (brown) → GND. For best reliability, route GPIO39 through a second channel of the BSS138 level shifter (LV2 → GPIO39, HV2 → tilt servo SIGNAL).

    • Tip: For guaranteed reliable operation route through the second LLC channel just like the pan servo.
  7. Wire the push-to-talk button

    Connect one leg of the push button to GPIO1, the other leg to GND. The firmware enables the internal pull-up resistor — no external resistor needed.

    • Tip: Mount the button on the front of the robot body within easy thumb reach.
    • Tip: If the button triggers randomly, add a 100nF capacitor between GPIO1 and GND for hardware debounce.
  8. Assemble the pan-tilt head

    Assemble a 2-axis pan-tilt bracket: the bottom servo (servo_pan) rotates the head left/right; the top servo (servo_tilt) rotates the face up/down. Mount the OLED display and INMP441 microphone on the top platform facing forward. Secure all servo horns with the included screws.

    • Tip: Test servo center position (90°) before gluing anything — power on and run setup() to auto-center.
    • Tip: Use hot glue or M2 screws to mount the OLED on the top bracket.
  9. Enter your API credentials in the firmware

    Open firmware.cpp in Schematik and fill in the 4 #define values at the top: WIFI_SSID, WIFI_PASSWORD, GROQ_API_KEY (free at console.groq.com), and ELEVEN_API_KEY (free at elevenlabs.io). The ELEVEN_VOICE_ID is pre-set to the Rachel voice — change it if you prefer a different voice.

    • Tip: Groq has a generous free tier — no credit card needed for basic use.
    • Tip: ElevenLabs free tier gives 10,000 characters/month which is plenty for testing.
  10. Flash and test

    Click the Deploy button in Schematik to compile and flash. Open the serial monitor — you should see '[BOOT] Ready! Hold button to speak.' The OLED shows a smiley face. Hold the button, speak a command like 'look left' or 'what is your name', then release. The robot transcribes, thinks, speaks, and moves its head.

    • Tip: Watch the serial monitor for debug output — each stage prints [MIC], [STT], [LLM], [TTS], [SERVO] tags.
    • Tip: If WiFi fails, double-check SSID/password — the serial monitor will show the error.
    • Do not connect the battery and USB at the same time if you add a battery later.

Pin assignments

Board wiring reference
PinConnectionType
3V3inmp441 VDDpower
GNDinmp441 GNDground
GPIO 40inmp441 SCKdigital
GPIO 41inmp441 WSdigital
GPIO 42inmp441 SDdata
GNDinmp441 L/Rground
5Vmax98357a VINpower
GNDmax98357a GNDground
GPIO 14max98357a BCLKdata
GPIO 21max98357a LRCdata
GPIO 47max98357a DINdata
EXTmax98357a SPK+8Ω Speaker POSdata
EXTmax98357a SPK-8Ω Speaker NEGdata
3V3oled VCCpower
GNDoled GNDground
GPIO 8oled SDAi2c
GPIO 9oled SCLi2c
3V3llc LVpower
5Vllc HVpower
GNDllc GNDground
GPIO 38llc LV1-LV4data
EXTllc HV1-HV4SG90 Servo SIGNALdata
5Vservo_pan VCCpower
GNDservo_pan GNDground
5Vservo_tilt VCCpower
GNDservo_tilt GNDground
GPIO 39servo_tilt SIGNALpwm
GPIO 1button SIGNALdigital
GNDbutton GNDground

Firmware

ESP32
firmware.cppDeploy to device
/*
 * ESP32-S3 AI Assistant Robot
 * - Push button → records from INMP441 mic via I2S
 * - Sends WAV to Groq Whisper (STT)
 * - Sends transcript to Groq LLaMA (LLM) → JSON: {speech, command, pan, tilt}
 * - Speaks reply via MAX98357A (ElevenLabs TTS) or beep fallback
 * - Animates SSD1306 OLED face
 * - Moves SG90 servos (pan/tilt head) based on command
 *
 * Set your credentials in the CONFIG section below.
 */

#include <Arduino.h>
#include <WiFi.h>
#include <WiFiClientSecure.h>
#include <Wire.h>
#include <Adafruit_GFX.h>
#include <Adafruit_SSD1306.h>
#include <ESP32Servo.h>
#include <ArduinoJson.h>
#include <driver/i2s.h>

// ─── CONFIG ──────────────────────────────────────────────────────────────────
#define WIFI_SSID       "Shourya"
#define WIFI_PASSWORD   "shourya29"
#define GROQ_API_KEY    "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"      // https://console.groq.com
#define ELEVEN_API_KEY  "sk_3e21ba9c3afb3cf7b30036cc4b96db1d9bf1e0c0ffc2135e"    // https://elevenlabs.io (optional)
#define ELEVEN_VOICE_ID "21m00Tcm4TlvDq8ikWAM"  // Rachel voice ID

// ─── PINS ────────────────────────────────────────────────────────────────────
#define MIC_SCK       40
#define MIC_WS        41
#define MIC_SD        42
#define AMP_BCLK      14
#define AMP_LRC       21
#define AMP_DIN       47
#define OLED_SDA      8
#define OLED_SCL      9
#define SERVO_PAN_PIN 38
#define SERVO_TILT_PIN 39
#define BTN_PIN       1

// ─── CONSTANTS ───────────────────────────────────────────────────────────────
#define SAMPLE_RATE   16000
#define SAMPLE_BITS   16
#define CHANNELS      1
#define RECORD_SECS   4
#define BUF_SAMPLES   (SAMPLE_RATE * RECORD_SECS)
#define BUF_BYTES     (BUF_SAMPLES * 2)

#define OLED_WIDTH    128
#define OLED_HEIGHT   64
#define OLED_ADDR     0x3C

// ─── GLOBALS ─────────────────────────────────────────────────────────────────


struct Message { String role; String content; };

enum FaceState { FACE_IDLE, FACE_LISTENING, FACE_THINKING, FACE_TALKING };

struct EyeParams {
  int x, y;         // top-left of eye bounding box
  int w, h;         // eye width / full-open height
  int pupilDx;      // pupil offset from eye centre X
  int pupilDy;      // pupil offset from eye centre Y
  int topLid;       // pixels the top eyelid closes down (0=open)
  int botLid;       // pixels the bottom eyelid closes up  (0=open)
};

// Forward declarations

// Forward declarations
void drawEye(const EyeParams& e);
void doBlink();

void drawFace(FaceState state);
void setupMic();
void setupSpeaker();
void recordAudio();
void writeWavHeader(uint8_t* header, uint32_t dataSize);
String transcribeAudio();
String queryLLM(const String& userText);
void playTTS(const String& text);
void moveServos(const String& command, int pan, int tilt);

Adafruit_SSD1306 display(OLED_WIDTH, OLED_HEIGHT, &Wire, -1);
Servo panServo;
Servo tiltServo;

int16_t* audioBuf = nullptr;
size_t   audioBytesRead = 0;

// Servo center positions
int panPos  = 90;
int tiltPos = 90;

// Chat history (simple rolling buffer, 4 exchanges)

#define MAX_HISTORY 8
Message history[MAX_HISTORY];
int historyCount = 0;

// ─── FACE STATES ─────────────────────────────────────────────────────────────
FaceState faceState = FACE_IDLE;
unsigned long faceTimer = 0;
bool eyeOpen = true;

// ─── COZMO/VECTOR-STYLE ANIMATED EYES ────────────────────────────────────────
// Each eye is a filled rounded-rect (sclera) with a filled circle pupil.
// Eyelids are filled rects that slide in from top/bottom to squint or blink.
// Pupil can shift left/right/up/down to convey gaze direction.



// Draw one eye given its parameters
void drawEye(const EyeParams& e) {
  int r = 5; // corner radius of sclera
  // 1. Filled sclera (white rounded rect)
  display.fillRoundRect(e.x, e.y, e.w, e.h, r, SSD1306_WHITE);
  // 2. Pupil (dark filled circle, slightly smaller than sclera height/2)
  int px = e.x + e.w / 2 + e.pupilDx;
  int py = e.y + e.h / 2 + e.pupilDy;
  int pr = e.h / 2 - 4;
  if (pr < 2) pr = 2;
  display.fillCircle(px, py, pr, SSD1306_BLACK);
  // Pupil shine dot
  display.fillCircle(px + pr/3, py - pr/3, max(1, pr/4), SSD1306_WHITE);
  // 3. Top eyelid (black rect slides down from top)
  if (e.topLid > 0)
    display.fillRect(e.x, e.y, e.w, min(e.topLid, e.h), SSD1306_BLACK);
  // 4. Bottom eyelid (black rect slides up from bottom)
  if (e.botLid > 0)
    display.fillRect(e.x, e.y + e.h - min(e.botLid, e.h), e.w, min(e.botLid, e.h), SSD1306_BLACK);
  // 5. Re-draw rounded corners as black to keep the eye shape crisp
  //    (clip corners outside the rounded rect)
  display.fillRect(e.x, e.y, r, r, SSD1306_BLACK);
  display.fillCircle(e.x + r, e.y + r, r, SSD1306_WHITE);           // re-fill corner
  display.fillRect(e.x + e.w - r, e.y, r, r, SSD1306_BLACK);
  display.fillCircle(e.x + e.w - r, e.y + r, r, SSD1306_WHITE);
  display.fillRect(e.x, e.y + e.h - r, r, r, SSD1306_BLACK);
  display.fillCircle(e.x + r, e.y + e.h - r, r, SSD1306_WHITE);
  display.fillRect(e.x + e.w - r, e.y + e.h - r, r, r, SSD1306_BLACK);
  display.fillCircle(e.x + e.w - r, e.y + e.h - r, r, SSD1306_WHITE);
  // Re-apply lids on top of corner fixes
  if (e.topLid > 0)
    display.fillRect(e.x, e.y, e.w, min(e.topLid, e.h), SSD1306_BLACK);
  if (e.botLid > 0)
    display.fillRect(e.x, e.y + e.h - min(e.botLid, e.h), e.w, min(e.botLid, e.h), SSD1306_BLACK);
}

// Animate a smooth blink: lids close then reopen
void doBlink() {
  // Eye positions: left eye x=8, right eye x=74, y=16, w=46, h=32
  for (int lid = 0; lid <= 16; lid += 4) {
    display.clearDisplay();
    EyeParams L = {8,  16, 46, 32, 0, 0, lid, lid};
    EyeParams R = {74, 16, 46, 32, 0, 0, lid, lid};
    drawEye(L); drawEye(R);
    display.display();
    delay(18);
  }
  delay(60);
  for (int lid = 16; lid >= 0; lid -= 4) {
    display.clearDisplay();
    EyeParams L = {8,  16, 46, 32, 0, 0, lid, lid};
    EyeParams R = {74, 16, 46, 32, 0, 0, lid, lid};
    drawEye(L); drawEye(R);
    display.display();
    delay(18);
  }
}

void drawFace(FaceState state) {
  display.clearDisplay();

  // Default: both eyes centred, fully open
  EyeParams L = {8,  16, 46, 32, 0, 0, 0, 0};
  EyeParams R = {74, 16, 46, 32, 0, 0, 0, 0};

  if (state == FACE_IDLE) {
    // Happy: slight squint (bottom lids raised 4px) + pupils centred
    L = {8,  16, 46, 32, 0, 0, 0, 4};
    R = {74, 16, 46, 32, 0, 0, 0, 4};
    if (!eyeOpen) { L.topLid = 16; L.botLid = 16; R.topLid = 16; R.botLid = 16; } // blink
    drawEye(L); drawEye(R);

  } else if (state == FACE_LISTENING) {
    // Wide open eyes — max attentive look, pupils slightly up
    L = {8,  14, 46, 34, 0, -2, 0, 0};
    R = {74, 14, 46, 34, 0, -2, 0, 0};
    if (!eyeOpen) { L.topLid = 17; L.botLid = 17; R.topLid = 17; R.botLid = 17; }
    drawEye(L); drawEye(R);
    // Pulsing MIC dot at bottom centre
    display.fillCircle(64, 56, 4, SSD1306_WHITE);
    display.fillCircle(64, 56, 2, SSD1306_BLACK);

  } else if (state == FACE_THINKING) {
    // One brow raised: left eye top-lid slightly open, right squinted
    // Pupils shift right (looking up-right)
    L = {8,  16, 46, 32,  4, -3, 3, 0};
    R = {74, 16, 46, 32,  4, -3, 7, 0}; // right eye squinted with raised inner brow
    drawEye(L); drawEye(R);
    // Thinking dots
    for (int i = 0; i < 3; i++) {
      int dotX = 51 + i * 10;
      display.fillCircle(dotX, 57, 3, SSD1306_WHITE);
    }

  } else if (state == FACE_TALKING) {
    // Pupils shift slightly — alive look. Lids half-squint while mouth moves.
    int squint = eyeOpen ? 0 : 8; // alternate squint with mouth movement
    L = {8,  16, 46, 32, -2, 2, squint, squint};
    R = {74, 16, 46, 32, -2, 2, squint, squint};
    drawEye(L); drawEye(R);
    // Talking mouth: open rounded rect at bottom centre
    int mH = eyeOpen ? 10 : 5;
    display.fillRoundRect(44, 52, 40, mH, 4, SSD1306_WHITE);
    display.fillRoundRect(46, 54, 36, max(1, mH-4), 3, SSD1306_BLACK); // inner dark
  }

  display.display();
}

// ─── I2S MIC SETUP ───────────────────────────────────────────────────────────
void setupMic() {
  i2s_config_t micCfg = {
    .mode                 = (i2s_mode_t)(I2S_MODE_MASTER | I2S_MODE_RX),
    .sample_rate          = SAMPLE_RATE,
    .bits_per_sample      = I2S_BITS_PER_SAMPLE_32BIT, // INMP441 outputs 32-bit frames
    .channel_format       = I2S_CHANNEL_FMT_ONLY_LEFT,
    .communication_format = I2S_COMM_FORMAT_STAND_I2S,
    .intr_alloc_flags     = ESP_INTR_FLAG_LEVEL1,
    .dma_buf_count        = 8,
    .dma_buf_len          = 512,
    .use_apll             = false,
    .tx_desc_auto_clear   = false,
    .fixed_mclk           = 0
  };
  i2s_pin_config_t micPins = {
    .mck_io_num   = I2S_PIN_NO_CHANGE,
    .bck_io_num   = MIC_SCK,
    .ws_io_num    = MIC_WS,
    .data_out_num = I2S_PIN_NO_CHANGE,
    .data_in_num  = MIC_SD
  };
  i2s_driver_install(I2S_NUM_0, &micCfg, 0, nullptr);
  i2s_set_pin(I2S_NUM_0, &micPins);
  i2s_zero_dma_buffer(I2S_NUM_0);
}

// ─── I2S SPEAKER SETUP ───────────────────────────────────────────────────────
void setupSpeaker() {
  i2s_config_t spkCfg = {
    .mode                 = (i2s_mode_t)(I2S_MODE_MASTER | I2S_MODE_TX),
    .sample_rate          = 22050,
    .bits_per_sample      = I2S_BITS_PER_SAMPLE_16BIT,
    .channel_format       = I2S_CHANNEL_FMT_ONLY_LEFT,
    .communication_format = I2S_COMM_FORMAT_STAND_I2S,
    .intr_alloc_flags     = ESP_INTR_FLAG_LEVEL1,
    .dma_buf_count        = 8,
    .dma_buf_len          = 512,
    .use_apll             = false,
    .tx_desc_auto_clear   = true,
    .fixed_mclk           = 0
  };
  i2s_pin_config_t spkPins = {
    .mck_io_num   = I2S_PIN_NO_CHANGE,
    .bck_io_num   = AMP_BCLK,
    .ws_io_num    = AMP_LRC,
    .data_out_num = AMP_DIN,
    .data_in_num  = I2S_PIN_NO_CHANGE
  };
  i2s_driver_install(I2S_NUM_1, &spkCfg, 0, nullptr);
  i2s_set_pin(I2S_NUM_1, &spkPins);
}

// ─── RECORD AUDIO ────────────────────────────────────────────────────────────
void recordAudio() {
  Serial.println("[MIC] Recording...");
  faceState = FACE_LISTENING;
  drawFace(faceState);

  memset(audioBuf, 0, BUF_BYTES);
  audioBytesRead = 0;

  // INMP441 outputs 32-bit frames; we extract the top 16 bits
  int32_t tmpBuf[512];
  size_t bytesRead = 0;
  size_t samplesCollected = 0;

  while (samplesCollected < (size_t)BUF_SAMPLES) {
    i2s_read(I2S_NUM_0, tmpBuf, sizeof(tmpBuf), &bytesRead, portMAX_DELAY);
    int samplesInChunk = bytesRead / 4;
    for (int i = 0; i < samplesInChunk && samplesCollected < (size_t)BUF_SAMPLES; i++) {
      audioBuf[samplesCollected++] = (int16_t)(tmpBuf[i] >> 16);
    }
    // Blink animation
    if ((samplesCollected % (SAMPLE_RATE / 4)) == 0) {
      eyeOpen = !eyeOpen;
      drawFace(FACE_LISTENING);
    }
  }
  audioBytesRead = BUF_SAMPLES * 2;
  Serial.printf("[MIC] Recorded %d bytes\n", audioBytesRead);
}

// ─── BUILD WAV HEADER (44 bytes) ─────────────────────────────────────────────
void writeWavHeader(uint8_t* header, uint32_t dataSize) {
  uint32_t fileSize = dataSize + 36;
  uint32_t byteRate = SAMPLE_RATE * CHANNELS * (SAMPLE_BITS / 8);
  uint16_t blockAlign = CHANNELS * (SAMPLE_BITS / 8);

  memcpy(header,      "RIFF", 4);
  memcpy(header + 4,  &fileSize, 4);
  memcpy(header + 8,  "WAVE", 4);
  memcpy(header + 12, "fmt ", 4);
  uint32_t chunkSize = 16; memcpy(header + 16, &chunkSize, 4);
  uint16_t audioFmt  = 1;  memcpy(header + 20, &audioFmt, 2);
  uint16_t ch = CHANNELS;  memcpy(header + 22, &ch, 2);
  memcpy(header + 24, (uint32_t[]){SAMPLE_RATE}, 4);
  memcpy(header + 28, &byteRate, 4);
  memcpy(header + 32, &blockAlign, 2);
  uint16_t bps = SAMPLE_BITS; memcpy(header + 34, &bps, 2);
  memcpy(header + 36, "data", 4);
  memcpy(header + 40, &dataSize, 4);
}

// ─── GROQ WHISPER STT ────────────────────────────────────────────────────────
String transcribeAudio() {
  Serial.println("[STT] Sending to Groq Whisper...");
  faceState = FACE_THINKING;
  drawFace(faceState);

  WiFiClientSecure client;
  client.setInsecure();

  if (!client.connect("api.groq.com", 443)) {
    Serial.println("[STT] Connection failed");
    return "";
  }

  // Build multipart/form-data body
  String boundary = "----ESP32Boundary7654321";
  uint8_t wavHeader[44];
  writeWavHeader(wavHeader, audioBytesRead);

  // Part 1: file
  String part1 = "--" + boundary + "\r\n";
  part1 += "Content-Disposition: form-data; name=\"file\"; filename=\"audio.wav\"\r\n";
  part1 += "Content-Type: audio/wav\r\n\r\n";

  // Part 2: model
  String part2 = "\r\n--" + boundary + "\r\n";
  part2 += "Content-Disposition: form-data; name=\"model\"\r\n\r\n";
  part2 += "whisper-large-v3-turbo\r\n";
  part2 += "--" + boundary + "--\r\n";

  size_t totalLen = part1.length() + 44 + audioBytesRead + part2.length();

  client.print("POST /openai/v1/audio/transcriptions HTTP/1.1\r\n");
  client.print("Host: api.groq.com\r\n");
  client.print("Authorization: Bearer ");
  client.print(GROQ_API_KEY);
  client.print("\r\n");
  client.print("Content-Type: multipart/form-data; boundary=");
  client.print(boundary);
  client.print("\r\nContent-Length: ");
  client.print(totalLen);
  client.print("\r\nConnection: close\r\n\r\n");

  client.print(part1);
  client.write(wavHeader, 44);
  client.write((uint8_t*)audioBuf, audioBytesRead);
  client.print(part2);

  // Read response
  unsigned long timeout = millis();
  while (client.available() == 0 && millis() - timeout < 15000) delay(100);

  String body = "";
  bool inBody = false;
  while (client.available()) {
    String line = client.readStringUntil('\n');
    if (line == "\r") { inBody = true; continue; }
    if (inBody) body += line;
  }
  client.stop();

  Serial.println("[STT] Raw response: " + body);
  StaticJsonDocument<512> doc;
  if (deserializeJson(doc, body) == DeserializationError::Ok) {
    return doc["text"] | "";
  }
  return "";
}

// ─── GROQ LLAMA LLM ──────────────────────────────────────────────────────────
String queryLLM(const String& userText) {
  Serial.println("[LLM] Querying Groq: " + userText);
  faceState = FACE_THINKING;
  drawFace(faceState);

  // Add to history
  if (historyCount < MAX_HISTORY) {
    history[historyCount++] = {"user", userText};
  }

  WiFiClientSecure client;
  client.setInsecure();
  if (!client.connect("api.groq.com", 443)) {
    Serial.println("[LLM] Connect failed");
    return "{\"speech\":\"Connection error.\",\"command\":\"NONE\",\"pan\":90,\"tilt\":90}";
  }

  // Build messages JSON array
  String messages = "[";
  messages += "{\"role\":\"system\",\"content\":\"You are a small friendly robot assistant named Aria. Always respond strictly in JSON with these keys: speech (what to say, max 2 sentences), command (one of: LOOK_LEFT, LOOK_RIGHT, LOOK_UP, LOOK_DOWN, NOD, SHAKE, CENTER, NONE), pan (servo pan angle 0-180), tilt (servo tilt angle 60-120). Keep speech cheerful and brief.\"}";
  for (int i = 0; i < historyCount; i++) {
    messages += ",{\"role\":\"" + history[i].role + "\",\"content\":\"";
    String escaped = history[i].content;
    escaped.replace("\"", "\\\"");
    messages += escaped + "\"}";
  }
  messages += "]";

  String body = "{\"model\":\"llama3-8b-8192\",\"messages\":" + messages + ",\"max_tokens\":200,\"temperature\":0.7,\"response_format\":{\"type\":\"json_object\"}}";

  client.print("POST /openai/v1/chat/completions HTTP/1.1\r\n");
  client.print("Host: api.groq.com\r\n");
  client.print("Authorization: Bearer "); client.print(GROQ_API_KEY); client.print("\r\n");
  client.print("Content-Type: application/json\r\n");
  client.print("Content-Length: "); client.print(body.length()); client.print("\r\n");
  client.print("Connection: close\r\n\r\n");
  client.print(body);

  unsigned long timeout = millis();
  while (client.available() == 0 && millis() - timeout < 15000) delay(100);

  String response = "";
  bool inBody2 = false;
  while (client.available()) {
    String line = client.readStringUntil('\n');
    if (line == "\r") { inBody2 = true; continue; }
    if (inBody2) response += line;
  }
  client.stop();

  Serial.println("[LLM] Response: " + response);

  // Parse outer response
  DynamicJsonDocument outer(2048);
  if (deserializeJson(outer, response) != DeserializationError::Ok) {
    return "{\"speech\":\"I had trouble thinking.\",\"command\":\"NONE\",\"pan\":90,\"tilt\":90}";
  }

  const char* content = outer["choices"][0]["message"]["content"];
  if (!content) return "{\"speech\":\"No answer.\",\"command\":\"NONE\",\"pan\":90,\"tilt\":90}";

  // Add assistant reply to history
  if (historyCount < MAX_HISTORY) {
    history[historyCount++] = {"assistant", String(content)};
  }

  return String(content);
}

// ─── PLAY TTS via ElevenLabs (MP3 streamed → I2S) ────────────────────────────
// Simplified: streams MP3 chunks directly to I2S (raw PCM approximation)
// For full MP3 decode, use the ESP32-audioI2S library.
void playTTS(const String& text) {
  Serial.println("[TTS] Speaking: " + text);
  faceState = FACE_TALKING;

  WiFiClientSecure client;
  client.setInsecure();

  String host = "api.elevenlabs.io";
  if (!client.connect(host.c_str(), 443)) {
    Serial.println("[TTS] ElevenLabs connect failed — skipping audio");
    return;
  }

  String url = "/v1/text-to-speech/" + String(ELEVEN_VOICE_ID) + "/stream";
  String payload = "{\"text\":\"" + text + "\",\"model_id\":\"eleven_turbo_v2\",\"voice_settings\":{\"stability\":0.5,\"similarity_boost\":0.75}}";

  client.print("POST " + url + " HTTP/1.1\r\n");
  client.print("Host: " + host + "\r\n");
  client.print("xi-api-key: "); client.print(ELEVEN_API_KEY); client.print("\r\n");
  client.print("Content-Type: application/json\r\n");
  client.print("Accept: audio/mpeg\r\n");
  client.print("Content-Length: "); client.print(payload.length()); client.print("\r\n");
  client.print("Connection: close\r\n\r\n");
  client.print(payload);

  // Skip HTTP headers
  while (client.connected()) {
    String line = client.readStringUntil('\n');
    if (line == "\r") break;
  }

  // Stream raw bytes to I2S speaker
  uint8_t buf[512];
  size_t written = 0;
  unsigned long startTime = millis();

  while (client.connected() || client.available()) {
    int len = client.read(buf, sizeof(buf));
    if (len > 0) {
      i2s_write(I2S_NUM_1, buf, len, &written, portMAX_DELAY);
      // Animate mouth while speaking
      if ((millis() - startTime) % 200 == 0) {
        eyeOpen = !eyeOpen;
        drawFace(FACE_TALKING);
      }
    }
    if (millis() - startTime > 30000) break; // safety timeout
  }
  client.stop();
  // Flush
  size_t flushWritten;
  uint8_t silence[512] = {};
  i2s_write(I2S_NUM_1, silence, sizeof(silence), &flushWritten, portMAX_DELAY);
}

// ─── SERVO MOVEMENT ──────────────────────────────────────────────────────────
void moveServos(const String& command, int pan, int tilt) {
  Serial.printf("[SERVO] cmd=%s pan=%d tilt=%d\n", command.c_str(), pan, tilt);

  // Clamp
  pan  = constrain(pan,  0,   180);
  tilt = constrain(tilt, 60,  120);

  if (command == "LOOK_LEFT")  { panPos = 45; }
  else if (command == "LOOK_RIGHT") { panPos = 135; }
  else if (command == "LOOK_UP")    { tiltPos = 70; }
  else if (command == "LOOK_DOWN")  { tiltPos = 110; }
  else if (command == "CENTER")     { panPos = 90; tiltPos = 90; }
  else if (command == "NOD") {
    for (int i = 0; i < 2; i++) {
      tiltServo.write(75); delay(250);
      tiltServo.write(105); delay(250);
    }
    tiltPos = 90;
  }
  else if (command == "SHAKE") {
    for (int i = 0; i < 2; i++) {
      panServo.write(60); delay(200);
      panServo.write(120); delay(200);
    }
    panPos = 90;
  }
  else if (command == "NONE") {
    // Use explicit pan/tilt values if provided
    panPos  = pan;
    tiltPos = tilt;
  }

  // Smooth move
  int curPan  = panServo.read();
  int curTilt = tiltServo.read();
  int steps   = 20;
  for (int s = 1; s <= steps; s++) {
    panServo.write(curPan  + (panPos  - curPan)  * s / steps);
    tiltServo.write(curTilt + (tiltPos - curTilt) * s / steps);
    delay(15);
  }
}

// ─── SETUP ───────────────────────────────────────────────────────────────────
void setup() {
  Serial.begin(115200);
  Serial.println("[BOOT] AI Robot starting...");

  // Button
  pinMode(BTN_PIN, INPUT_PULLUP);

  // OLED
  Wire.begin(OLED_SDA, OLED_SCL);
  if (!display.begin(SSD1306_SWITCHCAPVCC, OLED_ADDR)) {
    Serial.println("[OLED] Init failed — check wiring");
  }
  display.clearDisplay();
  display.setTextSize(1);
  display.setTextColor(SSD1306_WHITE);
  display.setCursor(20, 28);
  display.print("AI Robot Starting");
  display.display();

  // Servos
  panServo.attach(SERVO_PAN_PIN);
  tiltServo.attach(SERVO_TILT_PIN);
  panServo.write(90);
  tiltServo.write(90);

  // Alloc audio buffer in heap
  audioBuf = (int16_t*)malloc(BUF_BYTES);
  if (!audioBuf) {
    Serial.println("[MEM] Audio buffer alloc failed!");
    while (true) delay(1000);
  }

  // I2S
  setupMic();
  setupSpeaker();

  // WiFi
  Serial.printf("[WIFI] Connecting to %s ...\n", WIFI_SSID);
  display.clearDisplay();
  display.setCursor(20, 28);
  display.print("Connecting WiFi...");
  display.display();

  WiFi.begin(WIFI_SSID, WIFI_PASSWORD);
  int attempts = 0;
  while (WiFi.status() != WL_CONNECTED && attempts < 30) {
    delay(500); Serial.print(".");
    attempts++;
  }
  if (WiFi.status() == WL_CONNECTED) {
    Serial.println("\n[WIFI] Connected: " + WiFi.localIP().toString());
  } else {
    Serial.println("\n[WIFI] Failed — check credentials");
  }

  faceState = FACE_IDLE;
  eyeOpen   = true;
  drawFace(FACE_IDLE);
  Serial.println("[BOOT] Ready! Hold button to speak.");
}

// ─── LOOP ────────────────────────────────────────────────────────────────────
void loop() {
  // Auto-blink every 3-4 seconds in idle using smooth blink animation
  if (faceState == FACE_IDLE && millis() - faceTimer > 3200) {
    doBlink();
    drawFace(FACE_IDLE); // redraw fully after blink
    faceTimer = millis();
  }

  // Wait for button PRESS (active low with pullup)
  if (digitalRead(BTN_PIN) == LOW) {
    delay(50); // debounce
    if (digitalRead(BTN_PIN) != LOW) return;

    Serial.println("[BTN] Pressed — hold to record");
    faceState = FACE_LISTENING;
    drawFace(faceState);

    // Hold button → record
    // Wait for release (max 6s)
    unsigned long pressStart = millis();
    while (digitalRead(BTN_PIN) == LOW && millis() - pressStart < 6000) {
      delay(10);
    }

    // Record exactly RECORD_SECS of audio
    recordAudio();

    // STT
    String transcript = transcribeAudio();
    if (transcript.length() == 0) {
      Serial.println("[STT] Empty transcript");
      faceState = FACE_IDLE;
      drawFace(faceState);
      return;
    }
    Serial.println("[STT] Transcript: " + transcript);

    // Show transcript on OLED
    display.clearDisplay();
    display.setTextSize(1);
    display.setTextColor(SSD1306_WHITE);
    display.setCursor(0, 0);
    display.print("You: ");
    display.print(transcript.substring(0, 80));
    display.display();
    delay(1000);

    // LLM
    String llmJson = queryLLM(transcript);
    Serial.println("[LLM] JSON: " + llmJson);

    // Parse LLM response
    StaticJsonDocument<512> resp;
    String speech  = "I heard you!";
    String command = "NONE";
    int    panVal  = 90;
    int    tiltVal = 90;

    if (deserializeJson(resp, llmJson) == DeserializationError::Ok) {
      speech  = resp["speech"]  | "I'm not sure what to say.";
      command = resp["command"] | "NONE";
      panVal  = resp["pan"]     | 90;
      tiltVal = resp["tilt"]    | 90;
    }

    // Show reply on OLED
    display.clearDisplay();
    display.setCursor(0, 0);
    display.print("Aria: ");
    display.print(speech.substring(0, 80));
    display.display();

    // Speak
    playTTS(speech);

    // Move head
    moveServos(command, panVal, tiltVal);

    faceState = FACE_IDLE;
    eyeOpen   = true;
    drawFace(FACE_IDLE);
    faceTimer = millis();
  }
}

“Deploy to device” opens this project in Schematik, where you can flash it to your board over USB.

Remix this project

Make it yours in one click

Open a full copy of this project in your own Schematik workspace — diagram, code, parts, and assembly steps included. Swap the sensor, add features, or redesign the whole thing with AI. The author's original stays untouched.

Open in Schematik