In this project, we are going to build our own AI Voice Assistant using ESP32, Gemini AI, INMP441 microphone and MAX98357A amplifier.

Our assistant will listen for the wake word:

“Bluedot”

You can also change this wake word later in the Python code.

Once it detects the wake word, we can ask normal questions such as:

• “What is artificial intelligence?”
• “Explain quantum computing like I am five.”
• “What are the must-visit places in India?”
• “Can you tell me a story?”
• “Tell me something interesting about space.”
• “Can you speak Telugu?”

The assistant records our voice using the INMP441 microphone and sends the audio from the ESP32 to our computer through Wi-Fi.

On the computer, Faster-Whisper converts our speech into text.

The text is then sent to Google Gemini AI, which understands the question and generates an answer.

After that, the Gemini response is converted back into speech using Edge TTS.

FFmpeg converts the generated audio into the format required by our ESP32.

The audio is then sent back to the ESP32 through Wi-Fi.

Finally, the ESP32 sends the audio to the MAX98357A amplifier, and we hear the AI response through the speaker.

The complete process is:

Our Voice → INMP441 → ESP32 → Wi-Fi → PC → Whisper → Gemini AI → Edge TTS → Wi-Fi → ESP32 → MAX98357A → Speaker

---

Components Required

For this project, we need the following components:

Component	Quantity
ESP32 Development Board	1
INMP441 I2S Microphone	1
MAX98357A I2S Audio Amplifier	1
Small Speaker	1
Female-to-Female Jumper Wires	Around 11
USB Cable	1
Computer / Laptop	1
Wi-Fi Connection	1

The computer or laptop is required because speech recognition, Gemini communication and text-to-speech processing are handled using Python.

---

Circuit Connections

There are mainly two sections in our circuit:

1. INMP441 microphone connection
2. MAX98357A amplifier and speaker connection

---

INMP441 Microphone to ESP32

Connect the INMP441 microphone to the ESP32 as shown below.

INMP441	ESP32
VDD	3.3V
GND	GND
SCK / BCLK	GPIO 26
WS / LRC	GPIO 25
SD	GPIO 33
L/R	GND

Important

Power the INMP441 using 3.3V.

The INMP441 is an I2S digital microphone. It captures our voice and sends digital audio data to the ESP32.

---

MAX98357A to ESP32

Now connect the MAX98357A amplifier.

MAX98357A	ESP32
VIN	Vin
GND	GND
BCLK	GPIO 27
LRC / LCK	GPIO 14
DIN	GPIO 22
SD	GPIO 32
GAIN	Leave unconnected

The MAX98357A receives digital audio from the ESP32 and amplifies it so that we can hear the AI response through the speaker.

The SD pin is connected to GPIO 32 so that the ESP32 can enable and mute the amplifier when required.

---

Speaker Connection

Connect the speaker directly to the MAX98357A.

MAX98357A	Speaker
SPK+	Speaker any pin/wire
SPK−	Speaker any pin/wire

Do not connect either speaker terminal directly to ESP32 GND.

The speaker should be connected between SPK+ and SPK− of the MAX98357A.

---

Software Required

Now that the hardware connections are completed, we need to prepare our computer.

For this project we use:

• Python
• Faster-Whisper
• Google GenAI Python library
• Edge TTS
• FFmpeg

---

Download and Install Python

First, download and install Python on your Windows computer.

During installation, it is a good idea to enable the option to add Python to PATH.

After installing Python, press:

Windows + R

Type:

cmd

and press Enter.

Now check whether Python is installed correctly.

python --version

You can also check pip using:

pip --version

Pip is used to install the Python libraries required for our project.

---

Install Faster-Whisper

Faster-Whisper is used to convert our voice into text.

For example, if we say:

“What is an ESP32?”

Whisper converts the recorded speech into text so that it can be sent to Gemini AI.

Install Faster-Whisper using:

pip install faster-whisper

---

Install Google Gemini Library

Next, install the Google GenAI Python library.

This library allows our Python program to communicate with Google’s Gemini API.

pip install google-genai

Google’s current Gemini API documentation uses the google-genai Python SDK.

---

Install Edge TTS

Gemini gives us its answer as text.

But because we are building a voice assistant, we need to convert that text into speech.

For this we use Edge TTS.

Install it using:

pip install edge-tts

---

Install All Python Libraries With One Command

Instead of installing them separately, you can install all the required Python packages using:

pip install faster-whisper google-genai edge-tts

The main libraries used in our project are:

Library	Purpose
faster-whisper	Converts speech into text
google-genai	Communicates with Gemini AI
edge-tts	Converts Gemini’s answer into speech

Other modules used in our Python program, such as:

socket
wave
os
asyncio
subprocess
threading
time

are included with Python and normally do not require separate installation.

---

Download FFmpeg

We also use FFmpeg in this project.

Edge TTS generates the AI speech response, but we need to convert that audio into the correct format before sending it to the ESP32.

FFmpeg is used to convert the generated speech into:

• 16 kHz
• Mono
• 16-bit PCM audio

FFmpeg provides its official download page and links to compiled builds for Windows.

After downloading FFmpeg:

1. Extract the downloaded file.
2. Open the extracted folder.
3. Find the bin folder.
4. Inside the bin folder, locate:

ffmpeg.exe

Copy the full path of ffmpeg.exe.

For example:

D:\FFmpeg\bin\ffmpeg.exe

Later, we will enter this path in our Python code.

---

Getting the Gemini API Key

Now we need a Gemini API key.

The API key allows our Python program to communicate with Gemini AI.

Gemini API keys can be created and managed through Google AI Studio.

Step 1

Open Google AI Studio and sign in using your Google account.

Step 2

Open the API Key section.

Step 3

Create an API key.

Step 4

Copy the generated API key.

Important

Never share your actual Gemini API key publicly.

Instead of writing the API key directly inside our Python program, we will save it as a Windows environment variable.

Open CMD and enter:

setx GEMINI_API_KEY "YOUR_GEMINI_API_KEY"

Replace:

YOUR_GEMINI_API_KEY

with your actual key.

For example:

setx GEMINI_API_KEY "YOUR_ACTUAL_KEY_HERE"

After running the command, close Command Prompt and open it again.

Our Python program will read the API key automatically using:

API_KEY = os.getenv("GEMINI_API_KEY")

Using an environment variable such as GEMINI_API_KEY is the recommended approach in Google’s Gemini API documentation.

---

Find Your PC IP Address

The ESP32 needs to know the IP address of our computer because it sends microphone audio to the Python program.

Open CMD and type:

ipconfig

Look under your active Wi-Fi adapter.

Find:

IPv4 Address

For example:

IPv4 Address . . . . . . . . : 192.168.1.5

In that case, we would use:

const char* PC_IP = "192.168.1.5";

in our ESP32 code.

Your IP address will probably be different.

---

Main Python Code

Create a folder for the project on your computer.

Inside that folder, create a Python file named:

yourname_assistant.py

Paste the following code into it.

Before running the code, you need to change:

ESP32_IP = "YOUR_ESP32_IP_ADDRESS"

and:

FFMPEG_PATH = r"YOUR_FFMPEG_PATH\bin\ffmpeg.exe"

We will get the ESP32 IP address after uploading the ESP32 code.

import socket
import wave
import os
import asyncio
import subprocess
import threading
import time

from faster_whisper import WhisperModel
from google import genai
import edge_tts


# Network settings
HOST = "0.0.0.0"
MIC_PORT = 5002
SPEAKER_PORT = 5001
ESP32_IP = "YOUR_ESP32_IP_ADDRESS"


# Audio settings
SAMPLE_RATE = 16000
CHANNELS = 1
SAMPLE_WIDTH = 2

WAKE_SECONDS = 3
COMMAND_SECONDS = 5

WAKE_BYTES = SAMPLE_RATE * SAMPLE_WIDTH * WAKE_SECONDS
COMMAND_BYTES = SAMPLE_RATE * SAMPLE_WIDTH * COMMAND_SECONDS

WAKE_WORD = "your name"


# Temporary audio files , replace bluedot with your name
WAKE_FILE = "wake_chunk.wav"
COMMAND_FILE = "command.wav"
TTS_MP3 = "bluedot_response.mp3"
TTS_WAV = "bluedot_response.wav"


# Enter the location of ffmpeg.exe on your computer
FFMPEG_PATH = r"YOUR_FFMPEG_PATH\bin\ffmpeg.exe"


# Read Gemini API key from Windows environment variable
API_KEY = os.getenv("GEMINI_API_KEY")

if not API_KEY:
    print("\nERROR: GEMINI_API_KEY not found.")
    print("Check your Windows environment variable.")
    raise SystemExit


gemini = genai.Client(api_key=API_KEY)


SYSTEM_PROMPT = """
You are BlueDot, a friendly AI voice assistant.

Keep answers natural, friendly, conversational, and reasonably concise
because the response will be spoken aloud.

Do not use markdown.
Do not use tables.
Do not use emojis.
Do not use complicated symbols.
"""


# Load Whisper once when the program starts
print("\nLoading Whisper...")

whisper_model = WhisperModel(
    "base",
    device="cpu",
    compute_type="int8"
)

print("Whisper loaded!")


class MicrophoneStream:
    """
    Continuously reads microphone audio coming from the ESP32.

    This keeps the TCP connection moving while Gemini or TTS
    is processing the previous audio.
    """

    def __init__(self, connection):
        self.connection = connection
        self.buffer = bytearray()

        self.lock = threading.Lock()
        self.condition = threading.Condition(self.lock)

        self.running = True

        self.thread = threading.Thread(
            target=self._reader,
            daemon=True
        )

        self.thread.start()

    def _reader(self):
        try:
            while self.running:
                data = self.connection.recv(4096)

                if not data:
                    break

                with self.condition:
                    self.buffer.extend(data)

                    # Keep only the latest 10 seconds of microphone audio
                    max_buffer = SAMPLE_RATE * SAMPLE_WIDTH * 10

                    if len(self.buffer) > max_buffer:
                        excess = len(self.buffer) - max_buffer
                        del self.buffer[:excess]

                    self.condition.notify_all()

        except Exception as error:
            print("\nMicrophone reader error:")
            print(error)

        finally:
            self.running = False

            with self.condition:
                self.condition.notify_all()

    def clear(self):
        with self.condition:
            self.buffer.clear()

    def get_audio(self, required_bytes):
        with self.condition:
            while len(self.buffer) < required_bytes and self.running:
                self.condition.wait(timeout=1)

            if len(self.buffer) < required_bytes:
                return None

            audio = bytes(self.buffer[:required_bytes])
            del self.buffer[:required_bytes]

            return audio


def save_wav(pcm_data, filename):
    with wave.open(filename, "wb") as wav:
        wav.setnchannels(CHANNELS)
        wav.setsampwidth(SAMPLE_WIDTH)
        wav.setframerate(SAMPLE_RATE)
        wav.writeframes(pcm_data)


def transcribe(filename):
    segments, info = whisper_model.transcribe(
        filename,
        language=None,
        beam_size=1,
        vad_filter=True
    )

    text = ""

    for segment in segments:
        text += segment.text

    return text.strip()


def ask_gemini(question):
    response = gemini.models.generate_content(
        model="gemini-3.5-flash",
        contents=SYSTEM_PROMPT + "\n\nUser: " + question
    )

    return response.text.strip()


def detect_tts_language(text):
    telugu_count = 0
    hindi_count = 0

    for char in text:
        code = ord(char)

        # Telugu Unicode block
        if 0x0C00 <= code <= 0x0C7F:
            telugu_count += 1

        # Devanagari Unicode block
        elif 0x0900 <= code <= 0x097F:
            hindi_count += 1

    if telugu_count > hindi_count and telugu_count > 0:
        return "telugu"

    if hindi_count > telugu_count and hindi_count > 0:
        return "hindi"

    return "english"


def select_tts_voice(text):
    language = detect_tts_language(text)

    if language == "telugu":
        voice = "te-IN-ShrutiNeural"

    elif language == "hindi":
        voice = "hi-IN-SwaraNeural"

    else:
        voice = "en-US-AriaNeural"

    return language, voice


async def generate_tts(text, filename):
    language, voice = select_tts_voice(text)

    print("\nDetected language:", language.upper())
    print("Female TTS voice:", voice)

    # Delete previous file so old audio is never played by mistake
    if os.path.exists(filename):
        os.remove(filename)

    communicate = edge_tts.Communicate(
        text,
        voice
    )

    await communicate.save(filename)


def create_tts_with_retry(text):
    attempts = 2

    for attempt in range(1, attempts + 1):
        try:
            print(
                f"\nCreating speech... Attempt {attempt}/{attempts}"
            )

            asyncio.run(
                generate_tts(
                    text,
                    TTS_MP3
                )
            )

            if not os.path.exists(TTS_MP3):
                raise RuntimeError(
                    "TTS MP3 file was not created."
                )

            if os.path.getsize(TTS_MP3) < 100:
                raise RuntimeError(
                    "TTS audio file is empty."
                )

            print("TTS created successfully!")
            return True

        except Exception as error:
            print("\nTTS attempt failed:")
            print(error)

            if attempt < attempts:
                print("Retrying TTS...")
                time.sleep(1)

    return False


def convert_to_wav():
    print("\nConverting TTS to PCM...")

    if os.path.exists(TTS_WAV):
        os.remove(TTS_WAV)

    command = [
        FFMPEG_PATH,
        "-y",
        "-i",
        TTS_MP3,
        "-ar",
        "16000",
        "-ac",
        "1",
        "-c:a",
        "pcm_s16le",
        TTS_WAV
    ]

    result = subprocess.run(
        command,
        stdout=subprocess.DEVNULL,
        stderr=subprocess.PIPE
    )

    if result.returncode != 0:
        print("\nFFmpeg conversion failed:")

        print(
            result.stderr.decode(
                errors="ignore"
            )
        )

        return False

    if not os.path.exists(TTS_WAV):
        print("ERROR: WAV file not created.")
        return False

    print("PCM conversion successful!")
    return True


def send_audio_to_esp32():
    print("\nOpening PCM audio...")

    with wave.open(TTS_WAV, "rb") as wav:
        channels = wav.getnchannels()
        rate = wav.getframerate()
        width = wav.getsampwidth()
        frames = wav.getnframes()

        print("\nSpeaker audio:")
        print("Channels:", channels)
        print("Sample rate:", rate)
        print("Sample width:", width)

        if channels != 1:
            raise RuntimeError(
                "Speaker WAV must be mono."
            )

        if rate != 16000:
            raise RuntimeError(
                "Speaker WAV must be 16000 Hz."
            )

        if width != 2:
            raise RuntimeError(
                "Speaker WAV must be 16-bit."
            )

        pcm_data = wav.readframes(frames)

    print("RAW PCM bytes:", len(pcm_data))

    if not pcm_data:
        raise RuntimeError(
            "No PCM audio generated."
        )

    print("\nConnecting to ESP32 speaker...")

    sock = socket.socket(
        socket.AF_INET,
        socket.SOCK_STREAM
    )

    sock.settimeout(15)

    try:
        sock.connect(
            (
                ESP32_IP,
                SPEAKER_PORT
            )
        )

        print("ESP32 speaker connected!")
        print("Sending RAW PCM...")

        chunk_size = 1024
        total_sent = 0

        while total_sent < len(pcm_data):
            chunk = pcm_data[
                total_sent:
                total_sent + chunk_size
            ]

            sock.sendall(chunk)

            total_sent += len(chunk)

        print("Audio sent successfully!")

    finally:
        sock.close()


def speak_answer(answer):
    if not create_tts_with_retry(answer):
        print("\nUnable to create speech.")
        return

    if not convert_to_wav():
        return

    send_audio_to_esp32()


# Create microphone TCP server
mic_server = socket.socket(
    socket.AF_INET,
    socket.SOCK_STREAM
)

mic_server.setsockopt(
    socket.SOL_SOCKET,
    socket.SO_REUSEADDR,
    1
)

mic_server.bind(
    (
        HOST,
        MIC_PORT
    )
)

mic_server.listen(1)


print("\n===================================")
print("       BLUEDOT AI ASSISTANT")
print("===================================")

print("\nMicrophone port:", MIC_PORT)
print("Speaker port:", SPEAKER_PORT)
print("ESP32 IP:", ESP32_IP)

print("\nFemale voices:")
print("English : en-US-AriaNeural")
print("Telugu  : te-IN-ShrutiNeural")
print("Hindi   : hi-IN-SwaraNeural")

print("\nWaiting for ESP32 microphone...")


connection = None


try:
    # Wait until the ESP32 microphone connects
    connection, address = mic_server.accept()

    connection.setsockopt(
        socket.IPPROTO_TCP,
        socket.TCP_NODELAY,
        1
    )

    print("\nESP32 microphone connected!")
    print("ESP32 IP:", address[0])

    microphone = MicrophoneStream(
        connection
    )

    # Remove any startup audio
    microphone.clear()

    while True:
        print('\nListening for "Bluedot"...')

        wake_audio = microphone.get_audio(
            WAKE_BYTES
        )

        if wake_audio is None:
            print("Microphone disconnected.")
            break

        save_wav(
            wake_audio,
            WAKE_FILE
        )

        wake_text = transcribe(
            WAKE_FILE
        )

        print(
            "Heard:",
            wake_text
        )

        if WAKE_WORD not in wake_text.lower():
            print(
                "No wake word detected."
            )
            continue

        print()
        print("================================")
        print("      BLUEDOT DETECTED!")
        print("================================")

        # Remove audio collected while Whisper was processing
        microphone.clear()

        print("\nCOMMAND MODE")
        print("Tell me your command...")

        command_audio = microphone.get_audio(
            COMMAND_BYTES
        )

        if command_audio is None:
            print("Microphone disconnected.")
            break

        save_wav(
            command_audio,
            COMMAND_FILE
        )

        command_text = transcribe(
            COMMAND_FILE
        )

        print()
        print("================================")
        print("YOU SAID:")
        print(command_text)
        print("================================")

        if not command_text:
            print("No command detected.")
            microphone.clear()
            continue

        print("\nAsking Gemini...")

        try:
            answer = ask_gemini(
                command_text
            )

        except Exception as error:
            print("\nGemini error:")
            print(error)

            microphone.clear()
            continue

        print()
        print("================================")
        print("GEMINI:")
        print(answer)
        print("================================")

        try:
            speak_answer(
                answer
            )

        except Exception as error:
            print("\nSpeaker/TTS error:")
            print(error)

        # Remove audio captured while the speaker was playing
        microphone.clear()

        print(
            '\nReady. Say "Bluedot" again.'
        )


except KeyboardInterrupt:
    print("\nBlueDot stopped.")


finally:
    if connection:
        connection.close()

    mic_server.close()


print("\nBlueDot server closed.")

---

ESP32 Code

Now open the Arduino IDE and create a new sketch.

Before uploading the code, change these three lines:

const char* ssid = "YOUR_WIFI_NAME";
const char* password = "YOUR_WIFI_PASSWORD";
const char* PC_IP = "YOUR_PC_IP_ADDRESS";

Enter:

• Your Wi-Fi name
• Your Wi-Fi password
• The PC IPv4 address that we found using ipconfig

Then upload the following code to your ESP32.

#include <WiFi.h>
#include "driver/i2s.h"


// Wi-Fi and PC settings
const char* ssid = "YOUR_WIFI_NAME";
const char* password = "YOUR_WIFI_PASSWORD";
const char* PC_IP = "YOUR_PC_IP_ADDRESS";


// Network ports
const uint16_t MIC_PORT = 5002;
const uint16_t SPEAKER_PORT = 5001;


// Wi-Fi clients
WiFiClient micClient;
WiFiServer speakerServer(SPEAKER_PORT);


// Audio settings
#define SAMPLE_RATE 16000


// INMP441 microphone pins
#define MIC_I2S I2S_NUM_0
#define MIC_SCK 26
#define MIC_WS  25
#define MIC_SD  33


// MAX98357A speaker pins
#define SPK_I2S I2S_NUM_1
#define SPK_BCLK 27
#define SPK_LRC  14
#define SPK_DIN  22
#define SPK_SD   32


bool micInstalled = false;
bool speakerInstalled = false;


// Install INMP441 microphone
void installMicrophone() {
  if (micInstalled) {
    return;
  }

  Serial.println();
  Serial.println("Installing INMP441 I2S...");

  i2s_config_t config = {
    .mode = (i2s_mode_t)(
      I2S_MODE_MASTER |
      I2S_MODE_RX
    ),

    .sample_rate = SAMPLE_RATE,
    .bits_per_sample = I2S_BITS_PER_SAMPLE_32BIT,
    .channel_format = I2S_CHANNEL_FMT_ONLY_LEFT,
    .communication_format = I2S_COMM_FORMAT_I2S,
    .intr_alloc_flags = ESP_INTR_FLAG_LEVEL1,
    .dma_buf_count = 8,
    .dma_buf_len = 256,
    .use_apll = false,
    .tx_desc_auto_clear = false,
    .fixed_mclk = 0
  };

  i2s_pin_config_t pins = {
    .bck_io_num = MIC_SCK,
    .ws_io_num = MIC_WS,
    .data_out_num = I2S_PIN_NO_CHANGE,
    .data_in_num = MIC_SD
  };

  esp_err_t result = i2s_driver_install(
    MIC_I2S,
    &config,
    0,
    NULL
  );

  if (result != ESP_OK) {
    Serial.print(
      "Microphone install error: "
    );
    Serial.println(result);
    return;
  }

  result = i2s_set_pin(
    MIC_I2S,
    &pins
  );

  if (result != ESP_OK) {
    Serial.print(
      "Microphone pin error: "
    );
    Serial.println(result);

    i2s_driver_uninstall(
      MIC_I2S
    );

    return;
  }

  i2s_set_clk(
    MIC_I2S,
    SAMPLE_RATE,
    I2S_BITS_PER_SAMPLE_32BIT,
    I2S_CHANNEL_MONO
  );

  micInstalled = true;

  Serial.println(
    "INMP441 ready."
  );
}


// Remove microphone I2S driver
void uninstallMicrophone() {
  if (!micInstalled) {
    return;
  }

  Serial.println();
  Serial.println(
    "Stopping microphone..."
  );

  i2s_driver_uninstall(
    MIC_I2S
  );

  micInstalled = false;

  Serial.println(
    "Microphone stopped."
  );
}


// Install MAX98357A speaker
void installSpeaker() {
  if (speakerInstalled) {
    return;
  }

  Serial.println();
  Serial.println(
    "Installing MAX98357A I2S..."
  );

  i2s_config_t config = {
    .mode = (i2s_mode_t)(
      I2S_MODE_MASTER |
      I2S_MODE_TX
    ),

    .sample_rate = SAMPLE_RATE,
    .bits_per_sample = I2S_BITS_PER_SAMPLE_16BIT,
    .channel_format = I2S_CHANNEL_FMT_RIGHT_LEFT,
    .communication_format = I2S_COMM_FORMAT_I2S,
    .intr_alloc_flags = ESP_INTR_FLAG_LEVEL1,
    .dma_buf_count = 8,
    .dma_buf_len = 256,
    .use_apll = false,
    .tx_desc_auto_clear = true,
    .fixed_mclk = 0
  };

  i2s_pin_config_t pins = {
    .bck_io_num = SPK_BCLK,
    .ws_io_num = SPK_LRC,
    .data_out_num = SPK_DIN,
    .data_in_num = I2S_PIN_NO_CHANGE
  };

  esp_err_t result = i2s_driver_install(
    SPK_I2S,
    &config,
    0,
    NULL
  );

  if (result != ESP_OK) {
    Serial.print(
      "Speaker install error: "
    );
    Serial.println(result);
    return;
  }

  result = i2s_set_pin(
    SPK_I2S,
    &pins
  );

  if (result != ESP_OK) {
    Serial.print(
      "Speaker pin error: "
    );
    Serial.println(result);

    i2s_driver_uninstall(
      SPK_I2S
    );

    return;
  }

  result = i2s_set_clk(
    SPK_I2S,
    SAMPLE_RATE,
    I2S_BITS_PER_SAMPLE_16BIT,
    I2S_CHANNEL_STEREO
  );

  if (result != ESP_OK) {
    Serial.print(
      "Speaker clock error: "
    );
    Serial.println(result);
  }

  i2s_zero_dma_buffer(
    SPK_I2S
  );

  speakerInstalled = true;

  Serial.println(
    "MAX98357A ready."
  );
}


// Remove speaker I2S driver
void uninstallSpeaker() {
  if (!speakerInstalled) {
    return;
  }

  i2s_driver_uninstall(
    SPK_I2S
  );

  speakerInstalled = false;

  Serial.println(
    "MAX98357A stopped."
  );
}


// Connect the microphone stream to the Python server
void connectMic() {
  if (micClient.connected()) {
    return;
  }

  micClient.stop();

  Serial.println();
  Serial.println(
    "Connecting microphone to Python..."
  );

  while (
    !micClient.connect(
      PC_IP,
      MIC_PORT
    )
  ) {
    Serial.println(
      "Waiting for Python..."
    );

    delay(1000);
  }

  micClient.setNoDelay(true);

  Serial.println(
    "Microphone connected to Python."
  );
}


// Read microphone audio and send it to the computer
void sendMicrophoneAudio() {
  if (!micInstalled) {
    return;
  }

  connectMic();

  int32_t rawSamples[256];
  int16_t pcmSamples[256];

  size_t bytesRead = 0;

  esp_err_t result = i2s_read(
    MIC_I2S,
    rawSamples,
    sizeof(rawSamples),
    &bytesRead,
    portMAX_DELAY
  );

  if (
    result != ESP_OK ||
    bytesRead == 0
  ) {
    return;
  }

  int sampleCount =
    bytesRead /
    sizeof(int32_t);

  for (
    int i = 0;
    i < sampleCount;
    i++
  ) {
    // This value gave clear microphone audio in our testing
    pcmSamples[i] =
      (int16_t)(
        rawSamples[i] >> 14
      );
  }

  if (micClient.connected()) {
    micClient.write(
      (uint8_t*)pcmSamples,
      sampleCount *
      sizeof(int16_t)
    );
  }
}


// Receive Gemini speech from Python and play it
void playGeminiAudio(
  WiFiClient &client
) {
  Serial.println();

  Serial.println(
    "================================"
  );

  Serial.println(
    "       GEMINI SPEAKING"
  );

  Serial.println(
    "================================"
  );

  // Mute the amplifier while switching I2S
  digitalWrite(
    SPK_SD,
    LOW
  );

  uninstallMicrophone();

  delay(30);

  installSpeaker();

  if (!speakerInstalled) {
    Serial.println(
      "Speaker I2S failed."
    );

    installMicrophone();
    return;
  }

  // Send silence first so the I2S clocks become stable
  int16_t silence[512];

  memset(
    silence,
    0,
    sizeof(silence)
  );

  size_t silenceWritten = 0;

  i2s_write(
    SPK_I2S,
    silence,
    sizeof(silence),
    &silenceWritten,
    portMAX_DELAY
  );

  delay(40);

  Serial.println(
    "Speaker clocks stable."
  );

  uint8_t monoBuffer[1024];
  int16_t stereoBuffer[1024];

  bool pendingByteExists = false;
  uint8_t pendingByte = 0;

  bool amplifierEnabled = false;
  uint32_t totalReceived = 0;

  Serial.println(
    "Waiting for audio..."
  );

  while (
    client.connected() ||
    client.available() > 0
  ) {
    int available =
      client.available();

    if (available <= 0) {
      delay(1);
      continue;
    }

    int amount = available;

    if (
      amount >
      (int)sizeof(monoBuffer)
    ) {
      amount =
        sizeof(monoBuffer);
    }

    int received = client.read(
      monoBuffer,
      amount
    );

    if (received <= 0) {
      continue;
    }

    totalReceived += received;

    // Enable amplifier only when real audio arrives
    if (!amplifierEnabled) {
      digitalWrite(
        SPK_SD,
        HIGH
      );

      delay(20);

      amplifierEnabled = true;

      Serial.println(
        "MAX98357A enabled."
      );
    }

    int outputIndex = 0;
    int position = 0;

    // Handle a sample split between two TCP packets
    if (
      pendingByteExists &&
      received > 0
    ) {
      uint16_t value =
        (
          (uint16_t)monoBuffer[0]
          << 8
        )
        |
        pendingByte;

      int16_t sample =
        (int16_t)value;

      stereoBuffer[
        outputIndex++
      ] = sample;

      stereoBuffer[
        outputIndex++
      ] = sample;

      position = 1;
      pendingByteExists = false;
    }

    // Convert mono audio to stereo
    while (
      position + 1 <
      received
    ) {
      uint16_t value =
        (
          (uint16_t)
          monoBuffer[
            position + 1
          ]
          << 8
        )
        |
        monoBuffer[position];

      int16_t sample =
        (int16_t)value;

      stereoBuffer[
        outputIndex++
      ] = sample;

      stereoBuffer[
        outputIndex++
      ] = sample;

      position += 2;
    }

    // Save an odd leftover byte for the next TCP packet
    if (position < received) {
      pendingByte =
        monoBuffer[position];

      pendingByteExists = true;
    }

    if (outputIndex > 0) {
      size_t bytesWritten = 0;

      esp_err_t result = i2s_write(
        SPK_I2S,
        stereoBuffer,
        outputIndex *
        sizeof(int16_t),
        &bytesWritten,
        portMAX_DELAY
      );

      if (result != ESP_OK) {
        Serial.print(
          "I2S write error: "
        );

        Serial.println(
          result
        );
      }
    }
  }

  Serial.println();

  Serial.print(
    "Audio bytes received: "
  );

  Serial.println(
    totalReceived
  );

  // Send silence before muting the amplifier
  silenceWritten = 0;

  i2s_write(
    SPK_I2S,
    silence,
    sizeof(silence),
    &silenceWritten,
    portMAX_DELAY
  );

  delay(40);

  digitalWrite(
    SPK_SD,
    LOW
  );

  delay(20);

  Serial.println(
    "MAX98357A muted."
  );

  uninstallSpeaker();

  delay(30);

  installMicrophone();

  Serial.println();

  Serial.println(
    "================================"
  );

  Serial.println(
    "       LISTENING AGAIN"
  );

  Serial.println(
    "================================"
  );
}


void setup() {
  Serial.begin(115200);

  delay(1000);

  Serial.println();

  Serial.println(
    "================================"
  );

  Serial.println(
    "       BLUEDOT ASSISTANT"
  );

  Serial.println(
    "================================"
  );

  // Keep amplifier muted at startup
  pinMode(
    SPK_SD,
    OUTPUT
  );

  digitalWrite(
    SPK_SD,
    LOW
  );

  WiFi.begin(
    ssid,
    password
  );

  Serial.println(
    "Connecting to Wi-Fi..."
  );

  while (
    WiFi.status() !=
    WL_CONNECTED
  ) {
    delay(500);
    Serial.print(".");
  }

  Serial.println();

  Serial.println(
    "Wi-Fi connected."
  );

  Serial.print(
    "ESP32 IP: "
  );

  Serial.println(
    WiFi.localIP()
  );

  speakerServer.begin();

  Serial.println(
    "Speaker TCP port 5001 ready."
  );

  // Microphone is active during listening mode
  installMicrophone();

  Serial.println();

  Serial.println(
    "================================"
  );

  Serial.println(
    "       LISTENING MODE"
  );

  Serial.println(
    "================================"
  );
}


void loop() {
  // Check whether Python is sending Gemini audio
  WiFiClient speakerClient =
    speakerServer.available();

  if (speakerClient) {
    playGeminiAudio(
      speakerClient
    );

    speakerClient.stop();

    return;
  }

  // Normal mode: send microphone audio to Python
  sendMicrophoneAudio();
}

---

Find the ESP32 IP Address

After uploading the ESP32 code, open the Serial Monitor.

Set the baud rate to:

115200

After the ESP32 connects to Wi-Fi, you should see something similar to:

Wi-Fi connected.
ESP32 IP: 192.168.1.14

Copy the IP address shown on your Serial Monitor.

Now open the Python code and change:

ESP32_IP = "YOUR_ESP32_IP_ADDRESS"

For example:

ESP32_IP = "192.168.1.14"

Do not simply copy the example IP address. Use the IP address shown by your ESP32.

---

Set the FFmpeg Path

Now find the location of ffmpeg.exe on your computer.

For example:

D:\FFmpeg\bin\ffmpeg.exe

Then change:

FFMPEG_PATH = r"YOUR_FFMPEG_PATH\bin\ffmpeg.exe"

to your actual path.

For example:

FFMPEG_PATH = r"D:\FFmpeg\bin\ffmpeg.exe"

---

Before Running the Project

Before starting the assistant, check the following:

• ESP32 and computer are connected to the same Wi-Fi network.
• Correct Wi-Fi name is entered in the ESP32 code.
• Correct Wi-Fi password is entered in the ESP32 code.
• Correct PC IPv4 address is entered in the ESP32 code.
• Correct ESP32 IP address is entered in the Python code.
• Gemini API key is stored in the GEMINI_API_KEY environment variable.
• FFmpeg path is correctly entered in the Python code.
• ESP32 code has been successfully uploaded.
• Required Python libraries are installed.

---

Run the AI Assistant

First, upload the ESP32 code.

After confirming its IP address and updating the Python file, open CMD.

Go to the folder where you saved:

bluedot_assistant.py

For example:

cd "D:\BlueDot_AI_Assistant"

Then run:

python bluedot_assistant.py

Whisper will load first.

You should see something similar to:

Loading Whisper...
Whisper loaded!

The Python program will then wait for the ESP32 microphone connection.

Once everything is connected, you should see:

Listening for "Bluedot"...

Now say:

“Bluedot” (/or replace your name)

Once the wake word is detected, the program will enter command mode.

You can then ask your question.

---

How Our AI Assistant Works

After completing the hardware and software setup, the working process is simple.

First, the assistant waits for us to say:

“Bluedot”

The INMP441 microphone continuously sends audio to the ESP32.

The ESP32 sends that microphone audio to our computer through Wi-Fi.

Faster-Whisper checks the audio for the wake word.

Once:

“Bluedot”

is detected, the assistant enters command mode.

Now we can ask something like:

“Explain quantum computing like I am five.”

The INMP441 captures our question.

The ESP32 sends the audio to the computer.

Faster-Whisper converts the speech into text.

That text is sent to Gemini AI.

Gemini understands the question and generates an answer.

The answer is then converted into speech using Edge TTS.

Depending on the response language, our program selects a female voice for:

• English
• Hindi

FFmpeg converts the generated speech into 16 kHz mono 16-bit PCM audio.

The computer then sends this audio back to the ESP32 through Wi-Fi.

The ESP32 temporarily stops the microphone I2S driver and switches to the speaker I2S driver.

It sends the audio to the MAX98357A.

Finally:

ESP32 → MAX98357A → Speaker

and we hear the AI-generated response.

After the answer finishes, the speaker is muted, the microphone is enabled again, and BlueDot returns to listening mode.

---

Changing the Wake Word

By default, our assistant listens for:

WAKE_WORD = "blue dot"

If you want to use a different wake word, change this line in the Python program.

For example:

WAKE_WORD = "jarvis"

You can then say:

“Jarvis”

instead of:

“Bluedot”

---

Conclusion

In this project, we built our own AI Voice Assistant using ESP32 and Gemini AI.

We used the INMP441 I2S microphone to capture our voice and the MAX98357A I2S amplifier to play the AI response through a small speaker.

The ESP32 communicates with our computer through Wi-Fi.

On the computer, Faster-Whisper converts our speech into text.

That text is sent to Gemini AI, which understands the question and generates a response.

Edge TTS converts Gemini’s response into speech, and FFmpeg converts the generated audio into the format required by our ESP32.

The final speech is sent back to the ESP32 through Wi-Fi and played through the MAX98357A and speaker.

Unlike a normal voice-control project where we program a few fixed commands, this AI assistant can understand and answer many different questions naturally.

This project is also a good example of how ESP32, electronics, Python and Artificial Intelligence can work together.

And this is only the beginning.

In future projects, we can further improve our BlueDot AI Assistant by adding sensors, displays, relays and other electronic devices so that the assistant can not only answer questions but also interact with the physical world.